- Email: [email protected]
Adaptive regularized image interpolation using a probabilistic gradient measure
Optics Communications 285 (2012) 245–248
Contents lists available at SciVerse ScienceDirect
Optics Communications journal homepage: www.elsevier.com/locate/optcom
Adaptive regularized image interpolation using a probabilistic gradient measure Zhenyu Liu School of Mathematics and Statistics, Zaozhuang University, 277160 PR China
a r t i c l e
i n f o
Article history: Received 29 August 2011 Received in revised form 19 September 2011 Accepted 21 September 2011 Available online 12 October 2011 Keywords: Image interpolation Gradient Regularization
a b s t r a c t An adaptive image interpolation approach is proposed in this paper. The proposed approach imposes a regularization on the reconstructed high-resolution image to suppress the noise and blurring incurred in the observed low-resolution image. Furthermore, the proposed regularization scheme is steered by the local gradient information of the image, which is evaluated using a probabilistic measure. Experiments are conducted to demonstrate the superior performance of the proposed approach. © 2011 Elsevier B.V. All rights reserved.
1. Introduction Image interpolation has a capability to enlarge a lower resolution image to a higher resolution version [1,2]. It has a wide range of applications, including digital photography, video communications, object recognition, and so on. Its main purpose is to provide more details in images so as to improve content recognition ability [3,4]. The existing image interpolation approaches can be classified into two categories: deterministic-based interpolation and regularizationbased interpolation. The first approach uses the single input image only to perform image interpolation. On the contrary, the second approach imposes certain prior information on the desired reconstructed high-resolution image. There are several spatially-adaptive image interpolation algorithms, which can adjust the interpolation coefficients to preserve the geometric constraint of images, such as the local edge directions. Li and Orchard [5] proposed to adjust the interpolation based on the geometric duality between low-resolution and high-resolution covariances. Zhang and Wu [6] proposed to perform interpolation on two directions and adaptively fuse their results to be single image. Liu et al. [7] proposed an image interpolation approach using regularized local linear regression. They used the moving least square method to robustly estimate local image structure, and then estimate the geometric structure of the marginal probability distribution of the missing pixels. Hung and Siu [8] proposed to use weighted least square estimator to perform unequal variance preserving for estimating both model parameters and missing pixel intensity values. The aforementioned deterministic image interpolation approaches cannot suppress the noise or blurring incurred in the observed image. In view of this, the prior image model can be further used as the regularization for the unknown reconstructed high-resolution image. The major challenge is the determination of the degree of regularization [9−11], E-mail address: [email protected]. 0030-4018/$ – see front matter © 2011 Elsevier B.V. All rights reserved. doi:10.1016/j.optcom.2011.09.065
since it determines the quality of the reconstructed high-resolution image. Conventional approaches impose a spatially-invariant regularization parameter to the whole image. However, these approaches have limited adaptive capability in the process of image reconstruction and cannot balance the suppression of noise against the preservation of image details. In view of this, a regularized image interpolation is proposed in this paper by exploiting the local gradient measure to adjust the degree of regularization according to the local content of the image. Motivated by the fact that the local gradient is critical to determine the degree of image's sharpness [12,13], for the smooth region with smaller gradient, the proposed approach will impose a stronger regularization. On the other hand, for the edge/texture region with larger gradient, the proposed approach will impose a softer regularization. Furthermore, the proposed approach exploits a probabilistic scheme to measure the gradient information. The rest of this paper is organized as follows. Section 2 presents the proposed adaptive regularization approach. Experimental results are presented in Section 3. Finally, Section 4 concludes this paper. 2. Proposed regularized image interpolation approach The given low-resolution image can be viewed as warped, blurred, down-sampled and noisy version from the original (unknown) highresolution image (denoted as f). That is, their relationship can be mathematically expressed as g ¼ hf þ v;
ð1Þ
where g represents the observed low-resolution image, h represents the above-mentioned convolving and downsampling operations, and v represents the additive white Gaussian noise. With such establishment, the goal of image reconstruction is to produce a single high-resolution image based on its low-resolution observation.
246
Z. Liu / Optics Communications 285 (2012) 245–248
Fig. 1. Test images used in this paper: (a). cartoon; (b). boat; (c). text; (d). cameraman.
Fig. 2. Various reconstructed images of Cartoon: (a)–(e). Refs. [5,6,18−20], respectively; (f). Proposed approach.
The proposed approach estimates the unknown high-resolution image (denoted as ^ f ) by minimizing the following cost function ^ f ¼ argmin fjf −gj þ Γðf Þg: f
ð2Þ
where |f − g| is the data term, representing the fidelity between the estimated high-resolution image with the observed low-resolution images. On the other hand, Γ(f) is the regularization term, which can be formulated using a conventional total variation model [14,15] as Γðf Þ ¼ ∑i ∑j ðjfi;j −fi−1;j j þ jfi;j −fiþ1;j j þ fi;j −fi;j−1 þ jfi;j −fi;jþ1 jÞ: ð3Þ The novelty of the proposed approach is to adaptively adjust the regularization according to the local content of the image as Γðf Þ ¼ ∑i ∑j λi−1;j fi;j −fi−1;j j þ λiþ1;j jf; i;j −fiþ1;j j ð4Þ þλi;j−1 fi;j −fi;j−1 þ λi;jþ1 jfi;j −fi;jþ1 jÞ where the weighting factor λ is measured in terms of edge strength of the local neighbor for each pixel. Comparing Eqs. (3) and (4), one can see that the degree of regularization is spatially adjusted in the proposed approach, rather than being spatially-invariant in conventional
approach. Therefore, the proposed approach will impose a stronger regularization for the smoother region, while imposing softer regularization for edge/texture region. The weighting factor λ is calculated based on a probabilistic measure using local gradient information. ! 1 1 λi;j ¼ ∑ ∑ ϕ λi;j − ; ð5Þ Z i δi;j j where Z is normalizing factor, vi, j measures local gradient defined as ð6Þ δi;j ¼ fi;j −fi−1;j þ jfi;j −fi;j−1 j þ ξ; in which ξ = 10 − 3, the function ϕ(⋅) is a non-negative kernel function that integrated to one [16], and a Gaussian function with a standard deviation 1 is used in this paper. Finally, the steepest descent algorithm [17] is used in this paper to find the closed-form solution for estimating the full-resolution image at each iteration step n as n o ðnþ1Þ ðnÞ ðnÞ ðnÞ T −j −i i j ðn Þ ^ f ; ¼^ f −α h sign h^ f −g þ C R sign ^ f −λR C ^ f ð7Þ
Z. Liu / Optics Communications 285 (2012) 245–248
247
Fig. 3. Various reconstructed images of Boat: (a)–(e). Refs. [5,6,18−20], respectively; (f). Proposed approach.
Fig. 4. Various reconstructed images of Text: (a)–(e). Refs. [5,6,18−20], respectively; (f). Proposed approach.
where α is a scalar defining the step size in the direction of the gradient, λ is a square matrix with the diagonal element as λi, j while all rest elements being zero values, R− i and C− j define the transposes of matrices Ri and C j, respectively and have a shifting effect in the opposite directions as Ri and Cj. The above iteration will be terminated until ^ðnþ1Þ ^ðnÞ ^ðnÞ −f =jf j≤10−4 . f
3. Experimental results In our simulations, four images are used as ground-truth test images, as shown in Fig. 1. They are convoluted with a point spread function, which is a Gaussian low-pass filter with a window size of 4 × 4 and the standard deviation of 1, followed by a down-sampling operation with a decimation factor of two in both horizontal and vertical directions, respectively. Lastly, each processed image is added
with a zero-mean white Gaussian noise to yield a noisy lowresolution image with a SNR 20 dB. In the first experiment, the proposed regularized image interpolation approach is compared with conventional image interpolation approaches, [5,6,18−20]. Each algorithm is independently exploited to utilize four low-resolution images to produce a 2 × 2 high-resolution image and compare it with the ground-truth image to calculate the PSNR performance. Furthermore, the subjective performance is compared in Figs. 2, 3, 4, and 5, where one can see that the proposed approach outperforms the conventional approaches to produce highest PSNR values (see Table 1) and most clear high-resolution images. The second experiment is to evaluate the computational complexity of the proposed algorithm in terms run time. All these algorithms are implemented using Matlab programming language, and run on a desktop with a 2.4 GHz CPU and a 3 GB RAM. Their running times (averaged over four test images) are compared in Table 2.
248
Z. Liu / Optics Communications 285 (2012) 245–248
Fig. 5. Various reconstructed images of Cameraman: (a)–(e). Refs. [5,6,18−20], respectively; (f). Proposed approach.
Table 1 The PSNR (in dB) performance comparison. Test image
Ref. [5]
Ref. [6]
Ref. [18]
Ref. [19]
Ref. [20]
Proposed approach
Cartoon Boat Text Cameraman
21.94 26.65 22.47 24.46
22.10 26.73 22.81 24.57
21.86 26.67 22.72 24.51
21.81 26.52 22.49 24.34
20.77 24.47 22.74 22.77
23.14 26.95 24.44 24.92
Table 2 The run-time (in second) performance comparison. Ref. [5]
Ref. [6]
Ref. [18]
Ref. [19]
Ref. [20]
Proposed approach
6.82
4.38
0.05
0.08
6.20
3.14
4. Conclusions A regularized image interpolation approach has been proposed in this paper, by adaptively adjusting the degree of regularization based on local content of the image. The proposed approach outperforms the conventional image interpolation approach, as verified in our experiments. Acknowledgements I would like to thank authors of Refs. [5,6,19,20] for providing their respective programs online. References [1] [2] [3] [4] [5]
K. Cai, J. Shi, S. Xiong, G. Wei, Optics Communications 284 (Sep. 2011) 4446. J. Tian, L. Ma, W. Yu, Expert Systems with Applications 38 (Sep. 2011) 12 514. J. Tian, K.-K. Ma, Signal, Image and Video Processing 3 (Sep. 2009) 217. J. Tian, K.-K. Ma, Signal, Image and Video Processing 5 (Sep. 2011) 329. X. Li, M. Orchard, IEEE Transactions on Image Processing 10 (Oct. 2001) 1521.
[6] L. Zhang, X. Wu, IEEE Transactions on Image Processing 15 (Aug. 2006) 2226. [7] X. Liu, D. Zhao, R. Xiong, S. Ma, W. Gao, Proc. Picture Coding Symposium, Nagoya, Japan, Dec. 2010, p. 118. [8] K.-W. Hung, W.-C. Siu, Proc. IEEE Int. Conf. on Image Processing, Hong Kong, Sep. 2010, p. 3297. [9] M. Zibetti, F. Bazan, J. Mayer, Signal Processing 88 (Dec. 2008) 2890. [10] J. Tian, K.-K. Ma, Journal of Visual Communication and Image Representation 21 (Apr. 2010) 232. [11] J. Tian, L. Chen, Z. Liu, Signal Processing 92 (2) (Feb. 2012) 490. [12] J. Tian, L. Chen, L. Ma, W. Yu, Optics Communications 284 (Jan. 2011) 80. [13] J. Tian, L. Chen, Proc. IEEE Int. Conf. on Image Processing, Hong Kong, Sep. 2010, p. 1205. [14] T.F. Chan, C.-K. Wong, IEEE Transactions on Image Processing 7 (Mar. 1998) 370. [15] M.K. Ng, H.-F. Shen, L.-P. Zhang, E. Lam, EURASIP Journal on Applied Signal Processing 2007 (2007) Article ID 74585. [16] M.C. Jones, J.S. Marron, S.J. Sheather, Journal of the American Statistical Association 91 (Mar. 1996) 401. [17] J. Tian, L. Chen, W. Yu, L. Ma, International Journal of Electronics and Communications 65 (Nov. 2011) 975. [18] J.W. Hwang, H.S. Lee, IEEE Signal Processing Letters 11 (2004) 359. [19] P. Getreuer, Proc. SPIE Visual Communications and Image Processing, San Jose, CA, Jan. 2009, p. 1. [20] W. Dong, L. Zhang, G. Shi, X. Wu, Proc. IEEE Int. Conf. on Image Processing, Hong Kong, Sep. 2010, p. 349.
Contents lists available at SciVerse ScienceDirect
Optics Communications journal homepage: www.elsevier.com/locate/optcom
Adaptive regularized image interpolation using a probabilistic gradient measure Zhenyu Liu School of Mathematics and Statistics, Zaozhuang University, 277160 PR China
a r t i c l e
i n f o
Article history: Received 29 August 2011 Received in revised form 19 September 2011 Accepted 21 September 2011 Available online 12 October 2011 Keywords: Image interpolation Gradient Regularization
a b s t r a c t An adaptive image interpolation approach is proposed in this paper. The proposed approach imposes a regularization on the reconstructed high-resolution image to suppress the noise and blurring incurred in the observed low-resolution image. Furthermore, the proposed regularization scheme is steered by the local gradient information of the image, which is evaluated using a probabilistic measure. Experiments are conducted to demonstrate the superior performance of the proposed approach. © 2011 Elsevier B.V. All rights reserved.
1. Introduction Image interpolation has a capability to enlarge a lower resolution image to a higher resolution version [1,2]. It has a wide range of applications, including digital photography, video communications, object recognition, and so on. Its main purpose is to provide more details in images so as to improve content recognition ability [3,4]. The existing image interpolation approaches can be classified into two categories: deterministic-based interpolation and regularizationbased interpolation. The first approach uses the single input image only to perform image interpolation. On the contrary, the second approach imposes certain prior information on the desired reconstructed high-resolution image. There are several spatially-adaptive image interpolation algorithms, which can adjust the interpolation coefficients to preserve the geometric constraint of images, such as the local edge directions. Li and Orchard [5] proposed to adjust the interpolation based on the geometric duality between low-resolution and high-resolution covariances. Zhang and Wu [6] proposed to perform interpolation on two directions and adaptively fuse their results to be single image. Liu et al. [7] proposed an image interpolation approach using regularized local linear regression. They used the moving least square method to robustly estimate local image structure, and then estimate the geometric structure of the marginal probability distribution of the missing pixels. Hung and Siu [8] proposed to use weighted least square estimator to perform unequal variance preserving for estimating both model parameters and missing pixel intensity values. The aforementioned deterministic image interpolation approaches cannot suppress the noise or blurring incurred in the observed image. In view of this, the prior image model can be further used as the regularization for the unknown reconstructed high-resolution image. The major challenge is the determination of the degree of regularization [9−11], E-mail address: [email protected]. 0030-4018/$ – see front matter © 2011 Elsevier B.V. All rights reserved. doi:10.1016/j.optcom.2011.09.065
since it determines the quality of the reconstructed high-resolution image. Conventional approaches impose a spatially-invariant regularization parameter to the whole image. However, these approaches have limited adaptive capability in the process of image reconstruction and cannot balance the suppression of noise against the preservation of image details. In view of this, a regularized image interpolation is proposed in this paper by exploiting the local gradient measure to adjust the degree of regularization according to the local content of the image. Motivated by the fact that the local gradient is critical to determine the degree of image's sharpness [12,13], for the smooth region with smaller gradient, the proposed approach will impose a stronger regularization. On the other hand, for the edge/texture region with larger gradient, the proposed approach will impose a softer regularization. Furthermore, the proposed approach exploits a probabilistic scheme to measure the gradient information. The rest of this paper is organized as follows. Section 2 presents the proposed adaptive regularization approach. Experimental results are presented in Section 3. Finally, Section 4 concludes this paper. 2. Proposed regularized image interpolation approach The given low-resolution image can be viewed as warped, blurred, down-sampled and noisy version from the original (unknown) highresolution image (denoted as f). That is, their relationship can be mathematically expressed as g ¼ hf þ v;
ð1Þ
where g represents the observed low-resolution image, h represents the above-mentioned convolving and downsampling operations, and v represents the additive white Gaussian noise. With such establishment, the goal of image reconstruction is to produce a single high-resolution image based on its low-resolution observation.
246
Z. Liu / Optics Communications 285 (2012) 245–248
Fig. 1. Test images used in this paper: (a). cartoon; (b). boat; (c). text; (d). cameraman.
Fig. 2. Various reconstructed images of Cartoon: (a)–(e). Refs. [5,6,18−20], respectively; (f). Proposed approach.
The proposed approach estimates the unknown high-resolution image (denoted as ^ f ) by minimizing the following cost function ^ f ¼ argmin fjf −gj þ Γðf Þg: f
ð2Þ
where |f − g| is the data term, representing the fidelity between the estimated high-resolution image with the observed low-resolution images. On the other hand, Γ(f) is the regularization term, which can be formulated using a conventional total variation model [14,15] as Γðf Þ ¼ ∑i ∑j ðjfi;j −fi−1;j j þ jfi;j −fiþ1;j j þ fi;j −fi;j−1 þ jfi;j −fi;jþ1 jÞ: ð3Þ The novelty of the proposed approach is to adaptively adjust the regularization according to the local content of the image as Γðf Þ ¼ ∑i ∑j λi−1;j fi;j −fi−1;j j þ λiþ1;j jf; i;j −fiþ1;j j ð4Þ þλi;j−1 fi;j −fi;j−1 þ λi;jþ1 jfi;j −fi;jþ1 jÞ where the weighting factor λ is measured in terms of edge strength of the local neighbor for each pixel. Comparing Eqs. (3) and (4), one can see that the degree of regularization is spatially adjusted in the proposed approach, rather than being spatially-invariant in conventional
approach. Therefore, the proposed approach will impose a stronger regularization for the smoother region, while imposing softer regularization for edge/texture region. The weighting factor λ is calculated based on a probabilistic measure using local gradient information. ! 1 1 λi;j ¼ ∑ ∑ ϕ λi;j − ; ð5Þ Z i δi;j j where Z is normalizing factor, vi, j measures local gradient defined as ð6Þ δi;j ¼ fi;j −fi−1;j þ jfi;j −fi;j−1 j þ ξ; in which ξ = 10 − 3, the function ϕ(⋅) is a non-negative kernel function that integrated to one [16], and a Gaussian function with a standard deviation 1 is used in this paper. Finally, the steepest descent algorithm [17] is used in this paper to find the closed-form solution for estimating the full-resolution image at each iteration step n as n o ðnþ1Þ ðnÞ ðnÞ ðnÞ T −j −i i j ðn Þ ^ f ; ¼^ f −α h sign h^ f −g þ C R sign ^ f −λR C ^ f ð7Þ
Z. Liu / Optics Communications 285 (2012) 245–248
247
Fig. 3. Various reconstructed images of Boat: (a)–(e). Refs. [5,6,18−20], respectively; (f). Proposed approach.
Fig. 4. Various reconstructed images of Text: (a)–(e). Refs. [5,6,18−20], respectively; (f). Proposed approach.
where α is a scalar defining the step size in the direction of the gradient, λ is a square matrix with the diagonal element as λi, j while all rest elements being zero values, R− i and C− j define the transposes of matrices Ri and C j, respectively and have a shifting effect in the opposite directions as Ri and Cj. The above iteration will be terminated until ^ðnþ1Þ ^ðnÞ ^ðnÞ −f =jf j≤10−4 . f
3. Experimental results In our simulations, four images are used as ground-truth test images, as shown in Fig. 1. They are convoluted with a point spread function, which is a Gaussian low-pass filter with a window size of 4 × 4 and the standard deviation of 1, followed by a down-sampling operation with a decimation factor of two in both horizontal and vertical directions, respectively. Lastly, each processed image is added
with a zero-mean white Gaussian noise to yield a noisy lowresolution image with a SNR 20 dB. In the first experiment, the proposed regularized image interpolation approach is compared with conventional image interpolation approaches, [5,6,18−20]. Each algorithm is independently exploited to utilize four low-resolution images to produce a 2 × 2 high-resolution image and compare it with the ground-truth image to calculate the PSNR performance. Furthermore, the subjective performance is compared in Figs. 2, 3, 4, and 5, where one can see that the proposed approach outperforms the conventional approaches to produce highest PSNR values (see Table 1) and most clear high-resolution images. The second experiment is to evaluate the computational complexity of the proposed algorithm in terms run time. All these algorithms are implemented using Matlab programming language, and run on a desktop with a 2.4 GHz CPU and a 3 GB RAM. Their running times (averaged over four test images) are compared in Table 2.
248
Z. Liu / Optics Communications 285 (2012) 245–248
Fig. 5. Various reconstructed images of Cameraman: (a)–(e). Refs. [5,6,18−20], respectively; (f). Proposed approach.
Table 1 The PSNR (in dB) performance comparison. Test image
Ref. [5]
Ref. [6]
Ref. [18]
Ref. [19]
Ref. [20]
Proposed approach
Cartoon Boat Text Cameraman
21.94 26.65 22.47 24.46
22.10 26.73 22.81 24.57
21.86 26.67 22.72 24.51
21.81 26.52 22.49 24.34
20.77 24.47 22.74 22.77
23.14 26.95 24.44 24.92
Table 2 The run-time (in second) performance comparison. Ref. [5]
Ref. [6]
Ref. [18]
Ref. [19]
Ref. [20]
Proposed approach
6.82
4.38
0.05
0.08
6.20
3.14
4. Conclusions A regularized image interpolation approach has been proposed in this paper, by adaptively adjusting the degree of regularization based on local content of the image. The proposed approach outperforms the conventional image interpolation approach, as verified in our experiments. Acknowledgements I would like to thank authors of Refs. [5,6,19,20] for providing their respective programs online. References [1] [2] [3] [4] [5]
K. Cai, J. Shi, S. Xiong, G. Wei, Optics Communications 284 (Sep. 2011) 4446. J. Tian, L. Ma, W. Yu, Expert Systems with Applications 38 (Sep. 2011) 12 514. J. Tian, K.-K. Ma, Signal, Image and Video Processing 3 (Sep. 2009) 217. J. Tian, K.-K. Ma, Signal, Image and Video Processing 5 (Sep. 2011) 329. X. Li, M. Orchard, IEEE Transactions on Image Processing 10 (Oct. 2001) 1521.
[6] L. Zhang, X. Wu, IEEE Transactions on Image Processing 15 (Aug. 2006) 2226. [7] X. Liu, D. Zhao, R. Xiong, S. Ma, W. Gao, Proc. Picture Coding Symposium, Nagoya, Japan, Dec. 2010, p. 118. [8] K.-W. Hung, W.-C. Siu, Proc. IEEE Int. Conf. on Image Processing, Hong Kong, Sep. 2010, p. 3297. [9] M. Zibetti, F. Bazan, J. Mayer, Signal Processing 88 (Dec. 2008) 2890. [10] J. Tian, K.-K. Ma, Journal of Visual Communication and Image Representation 21 (Apr. 2010) 232. [11] J. Tian, L. Chen, Z. Liu, Signal Processing 92 (2) (Feb. 2012) 490. [12] J. Tian, L. Chen, L. Ma, W. Yu, Optics Communications 284 (Jan. 2011) 80. [13] J. Tian, L. Chen, Proc. IEEE Int. Conf. on Image Processing, Hong Kong, Sep. 2010, p. 1205. [14] T.F. Chan, C.-K. Wong, IEEE Transactions on Image Processing 7 (Mar. 1998) 370. [15] M.K. Ng, H.-F. Shen, L.-P. Zhang, E. Lam, EURASIP Journal on Applied Signal Processing 2007 (2007) Article ID 74585. [16] M.C. Jones, J.S. Marron, S.J. Sheather, Journal of the American Statistical Association 91 (Mar. 1996) 401. [17] J. Tian, L. Chen, W. Yu, L. Ma, International Journal of Electronics and Communications 65 (Nov. 2011) 975. [18] J.W. Hwang, H.S. Lee, IEEE Signal Processing Letters 11 (2004) 359. [19] P. Getreuer, Proc. SPIE Visual Communications and Image Processing, San Jose, CA, Jan. 2009, p. 1. [20] W. Dong, L. Zhang, G. Shi, X. Wu, Proc. IEEE Int. Conf. on Image Processing, Hong Kong, Sep. 2010, p. 349.