- Email: [email protected]
Diffraction-limited image reconstruction with SURE for atmospheric turbulence removal
Infrared Physics & Technology 71 (2015) 171–174
Contents lists available at ScienceDirect
Infrared Physics & Technology journal homepage: www.elsevier.com/locate/infrared
Diffraction-limited image reconstruction with SURE for atmospheric turbulence removal Changxin Song ⇑, Ke Ma, Anqiang Li, Xiaofang Chen, Xing Xu Department of Computer, Qinghai Normal University, Xining 810008, China
h i g h l i g h t s We proposed a new method of diffraction-limited image reconstruction. We employ the method of image processing to restore a high-quality image. This method can remove the atmospheric turbulence effect on imaging system. The experiments on real data demonstrate the effectiveness of our method.
a r t i c l e
i n f o
Article history: Received 18 February 2015 Available online 27 March 2015 Keywords: Atmospheric turbulence Image restoration Blind deconvolution Stein’s unbiased risk estimate
a b s t r a c t In order to remove the atmospheric turbulence effects in long-distance imaging systems, a novel method of diffraction-limited image reconstruction from multiframe image based on Stein’s unbiased risk estimate (SURE) is proposed. A diffraction-limited image reconstruction objective function is designed by combining the temporal and spatial information. The SURE is employed to optimize the objective function. The experimental results on real data demonstrate that a single high-quality image with more details of the scene can be restored from a degraded image sequence. The performance of the proposed algorithm is superior to the method of near-diffraction-limited image reconstruction in visual quality. Ó 2015 Elsevier B.V. All rights reserved.
1. Introduction For long-distance imaging systems, the acquired images are often corrupted by the atmospheric turbulence. The effect of turbulence is caused by the time- and space-varying refraction index of the air which is due to temperature, air pressure, humidity, and wind conditions between the scene and the camera [1–3]. It makes the image geometric distortion, motion blur, and sometimes out-of-focus blur [4]. This phenomenon is usually seen in ground-based surveillance, astronomy, and so on. The degraded image makes the image visual quality and the performance of image understanding significantly worse. In order to restore a single high-quality image, many methods have been employed to remove the effects of the atmospheric turbulence from an observed frame sequence, such as efficient filter flow [5], and lucky imaging [6]. Recently, an effective method of atmospheric turbulence removal is introduced by Zhu and Milanfar [3]. This method first registers each frame to suppress geometric deformation using ⇑ Corresponding author. Tel.: +86 971 6309545. E-mail address: [email protected] (C. Song). http://dx.doi.org/10.1016/j.infrared.2015.03.009 1350-4495/Ó 2015 Elsevier B.V. All rights reserved.
B-spline based non-rigid image registration. Next, a patch-wise temporal regression process is carried out to produce an image from the registered frames. Finally, a blind deconvolution algorithm is implemented to deblur the fused image and generate the final restored image. Since all the weights in the temporal regression are positive, the restored image is always more blurry than the observed when restoring the diffraction-limited value. This patch reconstruction method can be viewed as a space invariant near-diffraction-limited reconstruction. In the paper, we propose a method of diffraction-limited image reconstruction based on Stein’s unbiased risk estimate (SURE). SURE is an unbiased estimator of mean squared error (MSE), which is applied in denoising [7–9] and deblurring [10]. Here, we use SURE for diffraction-limited image reconstruction. This proposed approach can effectively restore more image detail. 2. Problem statement In general, the atmospheric turbulence can be efficiently simulated by local blurring and warping with possibly additive noise [11]. For the distorted image sequence caused by atmospheric turbulence, the imaging model can be denoted as:
172
C. Song et al. / Infrared Physics & Technology 71 (2015) 171–174
(a) The first frame
(b) The second frame
(d) Near-diffraction-limited
(c) The third frame
(e) Diffraction-limited
(f) NDLR restoration 0.05 0.04 0.03 0.02
αt
0.01 0 -0.01 -0.02 -0.03 -0.04
0
10
20
30
40
50
60
70
80
90
100
Frame index
(g) Our method’s restoration
(h) Difference image
(i) One weight coefficient
Fig. 1. Results on Building image sequence degraded by real atmospheric turbulence.
Y t ½p ¼ ðX ht;p gÞ½p þ Nt ½p
ð1Þ
where X is the latent image, Yt is the t-th observed image. Nt is the noise in the t-th frame, usually assumed as additive white Gaussian noise. p = (m, n)T represents a 2D spatial location in the image with m and n are the pixel coordinates. The denotes a 2D convolution operator. ht,p is the space-varying point spread function (PSF) for the position p in the t-th frame. g is a space-invariant diffractionlimited PSF. The g, ht,p and Nt in the imaging model are unknown. Moreover, h is a time- and space-varying PSF. It is a difficult task to estimate the original image X. According to [3], a registration algorithm [12] based B-spline with symmetry constraint is used to remove geometric deformation firstly. For a given pixel located ! at a position p, the deformed position Wðp; u Þ can be denoted by ! ! ! Wðp; u Þ ¼ p þ AðpÞ u , in which u is the displacement from the initial position, A(p) denotes the basis function matrix for p. The cost function can be denoted as follows:
X 2 X YðWðp; ! RðWðp; u ÞÞ YðpÞ2 JðuÞ ¼ u ÞÞ RðpÞ þ p T ! ! þ cð u þ u Þ ð u þ u Þ
p
ð2Þ
where Y represents a given observed distorted image, R denotes a ! u ¼ u denotes the inverse vector that
reference image,
!T
T
transforms R into Y, uT ¼ ½ u ; u , c is a positive parameter. The Gauss–Newton method is used to minimize the Eq. (2). Once the deformation vector u is estimated for each frame, a registered sequence can be generated through bilinear interpolation. This process makes sure that for a given pixel located at a position p, the most correlated pixels in each frame across time are aligned at this same position. 3. Diffraction-limited image reconstruction Let rt be local patch of size L L extracted from the t-th registered frame, x be the corresponding latent patch. Then the patch-wise imaging model is:
rt ¼ x ht g þ nt ¼ z ht þ nt ¼ st þ nt
ð3Þ
where ht is the shifted PSF version of ht, which is patch-wise constant and time-varying. nt represents the shifted local noise patch, which is assumed to be white Gaussian noise. z ¼ x g is the unknown diffraction-limited patch, which is convolved by the space and time invariant PSF g. st ¼ z ht denotes the noise-free local patch. For each local region, the sharpest patch rl (convolved by a diffraction-limited PSF) can be detected from the patch sequence based on the intensity variance of each patch [3,13]. Next,
173
C. Song et al. / Infrared Physics & Technology 71 (2015) 171–174
(a) The first frame
(b) The second frame
(d) Near-diffraction-limited
(c) The third frame
(e) Diffraction-limited
(f) NDLR restoration 0.08 0.06 0.04
αt
0.02 0 -0.02 -0.04 -0.06
0
10
20
30
40
50
60
70
80
Frame index
(g) Our method’s restoration
(h) Difference image
(i) One weight coefficient
Fig. 2. Results on Moon’s surface image sequence degraded by real atmospheric turbulence.
patch-wise temporal kernel regression is carried out to reduce the P P r t ½pUðp; t; l; lk Þ Uðp; t; l; lk Þ, in noise level. That is, bs l ½p ¼ t
t
which sl is the corresponding denoised pixel at p, the weight Uðp; t; l; lk Þ is a kernel function measuring the similarity between st ½p and sl ½p, lk is kernel parameter. The weight is defined by: . Uðp; t; l; lk Þ ¼ exp kst sl k2 ðlk LÞ2 . As pointed out by Zhu and Milanfar [3], one limitation is that all the weights are positive, the restored image is always more blurry than the observed when restoring the diffraction-limited value. It can be considered as the near-diffraction-limited image reconstruction. The image edge detail may be destroyed. Here, we consider a strategy by expressing sbl ½p as a linear combination of the observed pixels at p across frames: sbl ½p ¼ P Fðr½pÞ ¼ t at rt ½p ¼ aT r½p, in which at is the unknown weight
where Np is the local neighborhood with centered pixel at p, wi is the spatial weight of the i-th pixel. The second part kkak2 is regularization. k is a positive parameter. The pixel far away from pixel p, the less important to estimate a. The pixel close to pixel p the more important to estimate a. Thus, wi can be defined as: 2
wi ¼ expðki pk =g2 Þ, where g is the spatial weight parameter. However, we cannot solve Eq. (4) without the original signal sl . SURE provides a means for unbiased estimation of the true MSE without any assumptions on the original signal. The optimization with SURE is equivalent to the following expression:
argminU ¼ kkak2 þ a
X 2 wi kFðr½iÞ r l ½ik þ 2r2 div rl ½i fFðr½iÞg r2 i2N p
ð5Þ
coefficient, a ¼ ½a1 ; a2 ; . . . ; aT T is coefficient vector, r½p ¼ ½r1 ½p; r 2 ½p; . . . ; rT ½pT is pixel vector at p, T is the number of frames. The 2
optimal a should be minimizing the MSE kFðr½pÞ sl ½pk . Moreover, we introduce the spatial information around pixel at p to remove the outliers. The a can be estimated jointly by minimizing the following MSE cost function:
arg min a
X 2 wi kFðr½iÞ sl ½ik þ kkak2 i2Np
ð4Þ
where div rl ½i fFðr½iÞg ¼ @Fðr½iÞ is the divergence of the output estimate @r ½i l
with respect to the measurements. r2 is the noise variance can be estimated using PCA technique [14]. In order to guarantee the image uniform, we construct the noisy diffraction-limited image S, which is formed by sharpest detection via intensity variance without temporal kernel regression. Here, the r l is local patch extracted from the noisy construction image S. The minimization P problem can be solved by @@aU ¼ 0. That is, 2kak þ i2Np k
174
C. Song et al. / Infrared Physics & Technology 71 (2015) 171–174
wi 2r k ½iðFðr½iÞ r l ½iÞ þ 2r2 dðk; lÞ ¼ 0. Thus, the solution of Eq. (5) is:
a ¼ ðA þ kIÞ1 d
ð6Þ P
P j ðr k ½i r j ½iÞ , dk ¼ i2Np
P where a ¼ ½a1 ; . . . ; aK , Ak;j ¼ i2Np wi wi r k ½i rl ½i r2 dðk; lÞ ; in which dðk; lÞ is delta function, defined
1; k ¼ l . I is T T unit matrix. The l is decided by by: dðk; lÞ ¼ 0; k–l the image frame when constructing the pixel l of S. Then, an image Z is generated by fusing all the denoised reconstruction pixels together. Finally, a single-image deblurring algorithm [15] is implemented to deblur the fused image Z, in which the parameters are set at kernel Width = 9 kernel Height = 9, noise Str = 0.1, deblur Strength = 0.3. 4. Experimental results In order to illustrate the performance of the proposed approach, two real videos are tested (https://users.soe.ucsc.edu/~xzhu/doc/ turbulence.html), shown in Figs. 1 and 2. In our method, the patch size L = 9, g ¼ 10, k ¼ 20. The method of near-diffraction-limited reconstruction approach (NDLR) [3] is used for comparison. The first dataset is Building of size 240 240 with 100 frames, shown in Fig. 1. The videos were captured through hot air exhausted by a building’s vent. The sequence was degraded by strong turbulence effects. The image reconstruction results are shown in Fig. 1(d)–(e). The image restoration results are shown in Fig. 1(f)–(g). The difference image between NDLR restoration and our method’s restoration is shown in Fig. 1(i). The difference main exists in the image edge details. One weight coefficient for local patch reconstruction is shown in Fig. 1(h). The weight coefficient at could be positive or negative. The proposed restoration result looks over NDLR in the edge regions. The second dataset is Moon’s surface of size 376 408 with 80 frames taken from a ground-based telescope, shown in Fig. 2. This video data is taken by long-distance imaging systems. From Fig. 2 we can see that the output image with our method also looks slightly sharper than NDLR. For all two image sequences, the proposed method gives an improvement in visual quality. It removed noise and blur successfully. Compared to the previous method, our reconstructed image shows even more detail and structure. 5. Conclusion In this paper, we proposed an effective method of diffractionlimited image reconstruction for restoring a single high-quality image from an image sequence distorted by air turbulence. The image geometric distortion, space–time varying blur, and noise
can be suppressed effectively. Experimental results on real data illustrate that the proposed method can restore the latent static image with more details of the scene from a degraded and noisy image sequence caused by turbulence. The proposed approach gives rise to superior results over the method of near-diffractionlimited image reconstruction. Conflict of interest There is no conflict of interest. Acknowledgments This work was supported in part by National Natural Science Foundation of China under Grant 61450006, Natural Science Foundation of Qinghai Province of China under Grant 2014-ZJ724 and the Ministry of National Education Chunhui Project of China under Grant Z2012099. References [1] D. Shi, C. Fan, P. Zhang, H. Shen, J. Zhang, C. Qiao, Y. Wang, Two-wavelength ghost imaging through atmospheric turbulence, Opt. Express 21 (2) (2013) 2050–2064. [2] M. Micheli, Y. Lou, S. Soatto, A. Bertozzi, A linear systems approach to imaging through turbulence, J. Math. Imaging Vis. 48 (1) (2014) 185–201. [3] X. Zhu, P. Milanfar, Removing atmospheric turbulence via space-invariant deconvolution, IEEE Trans. Pattern Anal. Mach. Intell. 35 (1) (2013) 157–170. [4] X. Zhu, P. Milanfar, Image reconstruction from videos distorted by atmospheric turbulence, in: Proceedings of Visual Information Processing and Communication, 2010. pp. 75430S–75430S-8. [5] M. Hirsch, S. Sra, B. Schölkopf, S. Harmeling, Efficient filter flow for spacevariant multiframe blind deconvolution, in: IEEE International Conference on Computer Vision and Pattern Recognition, 2010, pp. 607–614. [6] Y. Lou, S.H. Kang, S. Soatto, A. Bertozzi, Video stabilization of atmospheric turbulence distortion, Inverse Probl. Imaging 7 (2013) 839–861. [7] T. Blu, F. Luisier, The SURE-LET approach to image denoising, IEEE Trans. Image Process. 16 (11) (2007) 2778–2786. [8] S.R. Krishnan, C.S. Seelamantula, P. Chakravarti, Spatially adaptive kernel regression using risk estimation, IEEE Signal Process. Lett. 21 (4) (2014) 445– 448. [9] F. Luisier, T. Blu, M. Unser, SURE-LET for orthonormal wavelet-domain video denoising, IEEE Trans. Circuits Syst. Video Technol. 20 (6) (2010) 913–919. [10] F. Xue, F. Luisier, T. Blu, Multi-wiener sure-let deconvolution, IEEE Trans. Image Process. 22 (5) (2013) 1954–1968. [11] M. Lemaitrea, J. Blanc-Talona, F. Mériaudeaub, O. Laligant, Evaluation of infrared image restoration techniques, in: Proceedings of Electro-Optical and Infrared Systems: Technology and Applications III, 2006, pp. 63950R.163950R.9. [12] Y. Mao, J. Gilles, Non rigid geometric distortions correction-application to atmospheric turbulence stabilization, Inverse Probl. Imaging 6 (3) (2012) 531– 546. [13] D. Gong, Y. Zhang, S. Dang, J. Sun, Neighbor combination for atmospheric turbulence image reconstruction, in: IEEE International Conference on Image Processing, 2013, pp. 1361–1365. [14] L. Xinhao, M. Tanaka, M. Okutomi, Single-image noise level estimation for blind denoising, IEEE Trans. Image Process. 22 (12) (2013) 5226–5237. [15] Q. Shan, J. Jia, A. Agarwala, High-quality motion deblurring from a single image, ACM Trans. Graph. 27 (3) (2008) 73.
Contents lists available at ScienceDirect
Infrared Physics & Technology journal homepage: www.elsevier.com/locate/infrared
Diffraction-limited image reconstruction with SURE for atmospheric turbulence removal Changxin Song ⇑, Ke Ma, Anqiang Li, Xiaofang Chen, Xing Xu Department of Computer, Qinghai Normal University, Xining 810008, China
h i g h l i g h t s We proposed a new method of diffraction-limited image reconstruction. We employ the method of image processing to restore a high-quality image. This method can remove the atmospheric turbulence effect on imaging system. The experiments on real data demonstrate the effectiveness of our method.
a r t i c l e
i n f o
Article history: Received 18 February 2015 Available online 27 March 2015 Keywords: Atmospheric turbulence Image restoration Blind deconvolution Stein’s unbiased risk estimate
a b s t r a c t In order to remove the atmospheric turbulence effects in long-distance imaging systems, a novel method of diffraction-limited image reconstruction from multiframe image based on Stein’s unbiased risk estimate (SURE) is proposed. A diffraction-limited image reconstruction objective function is designed by combining the temporal and spatial information. The SURE is employed to optimize the objective function. The experimental results on real data demonstrate that a single high-quality image with more details of the scene can be restored from a degraded image sequence. The performance of the proposed algorithm is superior to the method of near-diffraction-limited image reconstruction in visual quality. Ó 2015 Elsevier B.V. All rights reserved.
1. Introduction For long-distance imaging systems, the acquired images are often corrupted by the atmospheric turbulence. The effect of turbulence is caused by the time- and space-varying refraction index of the air which is due to temperature, air pressure, humidity, and wind conditions between the scene and the camera [1–3]. It makes the image geometric distortion, motion blur, and sometimes out-of-focus blur [4]. This phenomenon is usually seen in ground-based surveillance, astronomy, and so on. The degraded image makes the image visual quality and the performance of image understanding significantly worse. In order to restore a single high-quality image, many methods have been employed to remove the effects of the atmospheric turbulence from an observed frame sequence, such as efficient filter flow [5], and lucky imaging [6]. Recently, an effective method of atmospheric turbulence removal is introduced by Zhu and Milanfar [3]. This method first registers each frame to suppress geometric deformation using ⇑ Corresponding author. Tel.: +86 971 6309545. E-mail address: [email protected] (C. Song). http://dx.doi.org/10.1016/j.infrared.2015.03.009 1350-4495/Ó 2015 Elsevier B.V. All rights reserved.
B-spline based non-rigid image registration. Next, a patch-wise temporal regression process is carried out to produce an image from the registered frames. Finally, a blind deconvolution algorithm is implemented to deblur the fused image and generate the final restored image. Since all the weights in the temporal regression are positive, the restored image is always more blurry than the observed when restoring the diffraction-limited value. This patch reconstruction method can be viewed as a space invariant near-diffraction-limited reconstruction. In the paper, we propose a method of diffraction-limited image reconstruction based on Stein’s unbiased risk estimate (SURE). SURE is an unbiased estimator of mean squared error (MSE), which is applied in denoising [7–9] and deblurring [10]. Here, we use SURE for diffraction-limited image reconstruction. This proposed approach can effectively restore more image detail. 2. Problem statement In general, the atmospheric turbulence can be efficiently simulated by local blurring and warping with possibly additive noise [11]. For the distorted image sequence caused by atmospheric turbulence, the imaging model can be denoted as:
172
C. Song et al. / Infrared Physics & Technology 71 (2015) 171–174
(a) The first frame
(b) The second frame
(d) Near-diffraction-limited
(c) The third frame
(e) Diffraction-limited
(f) NDLR restoration 0.05 0.04 0.03 0.02
αt
0.01 0 -0.01 -0.02 -0.03 -0.04
0
10
20
30
40
50
60
70
80
90
100
Frame index
(g) Our method’s restoration
(h) Difference image
(i) One weight coefficient
Fig. 1. Results on Building image sequence degraded by real atmospheric turbulence.
Y t ½p ¼ ðX ht;p gÞ½p þ Nt ½p
ð1Þ
where X is the latent image, Yt is the t-th observed image. Nt is the noise in the t-th frame, usually assumed as additive white Gaussian noise. p = (m, n)T represents a 2D spatial location in the image with m and n are the pixel coordinates. The denotes a 2D convolution operator. ht,p is the space-varying point spread function (PSF) for the position p in the t-th frame. g is a space-invariant diffractionlimited PSF. The g, ht,p and Nt in the imaging model are unknown. Moreover, h is a time- and space-varying PSF. It is a difficult task to estimate the original image X. According to [3], a registration algorithm [12] based B-spline with symmetry constraint is used to remove geometric deformation firstly. For a given pixel located ! at a position p, the deformed position Wðp; u Þ can be denoted by ! ! ! Wðp; u Þ ¼ p þ AðpÞ u , in which u is the displacement from the initial position, A(p) denotes the basis function matrix for p. The cost function can be denoted as follows:
X 2 X YðWðp; ! RðWðp; u ÞÞ YðpÞ2 JðuÞ ¼ u ÞÞ RðpÞ þ p T ! ! þ cð u þ u Þ ð u þ u Þ
p
ð2Þ
where Y represents a given observed distorted image, R denotes a ! u ¼ u denotes the inverse vector that
reference image,
!T
T
transforms R into Y, uT ¼ ½ u ; u , c is a positive parameter. The Gauss–Newton method is used to minimize the Eq. (2). Once the deformation vector u is estimated for each frame, a registered sequence can be generated through bilinear interpolation. This process makes sure that for a given pixel located at a position p, the most correlated pixels in each frame across time are aligned at this same position. 3. Diffraction-limited image reconstruction Let rt be local patch of size L L extracted from the t-th registered frame, x be the corresponding latent patch. Then the patch-wise imaging model is:
rt ¼ x ht g þ nt ¼ z ht þ nt ¼ st þ nt
ð3Þ
where ht is the shifted PSF version of ht, which is patch-wise constant and time-varying. nt represents the shifted local noise patch, which is assumed to be white Gaussian noise. z ¼ x g is the unknown diffraction-limited patch, which is convolved by the space and time invariant PSF g. st ¼ z ht denotes the noise-free local patch. For each local region, the sharpest patch rl (convolved by a diffraction-limited PSF) can be detected from the patch sequence based on the intensity variance of each patch [3,13]. Next,
173
C. Song et al. / Infrared Physics & Technology 71 (2015) 171–174
(a) The first frame
(b) The second frame
(d) Near-diffraction-limited
(c) The third frame
(e) Diffraction-limited
(f) NDLR restoration 0.08 0.06 0.04
αt
0.02 0 -0.02 -0.04 -0.06
0
10
20
30
40
50
60
70
80
Frame index
(g) Our method’s restoration
(h) Difference image
(i) One weight coefficient
Fig. 2. Results on Moon’s surface image sequence degraded by real atmospheric turbulence.
patch-wise temporal kernel regression is carried out to reduce the P P r t ½pUðp; t; l; lk Þ Uðp; t; l; lk Þ, in noise level. That is, bs l ½p ¼ t
t
which sl is the corresponding denoised pixel at p, the weight Uðp; t; l; lk Þ is a kernel function measuring the similarity between st ½p and sl ½p, lk is kernel parameter. The weight is defined by: . Uðp; t; l; lk Þ ¼ exp kst sl k2 ðlk LÞ2 . As pointed out by Zhu and Milanfar [3], one limitation is that all the weights are positive, the restored image is always more blurry than the observed when restoring the diffraction-limited value. It can be considered as the near-diffraction-limited image reconstruction. The image edge detail may be destroyed. Here, we consider a strategy by expressing sbl ½p as a linear combination of the observed pixels at p across frames: sbl ½p ¼ P Fðr½pÞ ¼ t at rt ½p ¼ aT r½p, in which at is the unknown weight
where Np is the local neighborhood with centered pixel at p, wi is the spatial weight of the i-th pixel. The second part kkak2 is regularization. k is a positive parameter. The pixel far away from pixel p, the less important to estimate a. The pixel close to pixel p the more important to estimate a. Thus, wi can be defined as: 2
wi ¼ expðki pk =g2 Þ, where g is the spatial weight parameter. However, we cannot solve Eq. (4) without the original signal sl . SURE provides a means for unbiased estimation of the true MSE without any assumptions on the original signal. The optimization with SURE is equivalent to the following expression:
argminU ¼ kkak2 þ a
X 2 wi kFðr½iÞ r l ½ik þ 2r2 div rl ½i fFðr½iÞg r2 i2N p
ð5Þ
coefficient, a ¼ ½a1 ; a2 ; . . . ; aT T is coefficient vector, r½p ¼ ½r1 ½p; r 2 ½p; . . . ; rT ½pT is pixel vector at p, T is the number of frames. The 2
optimal a should be minimizing the MSE kFðr½pÞ sl ½pk . Moreover, we introduce the spatial information around pixel at p to remove the outliers. The a can be estimated jointly by minimizing the following MSE cost function:
arg min a
X 2 wi kFðr½iÞ sl ½ik þ kkak2 i2Np
ð4Þ
where div rl ½i fFðr½iÞg ¼ @Fðr½iÞ is the divergence of the output estimate @r ½i l
with respect to the measurements. r2 is the noise variance can be estimated using PCA technique [14]. In order to guarantee the image uniform, we construct the noisy diffraction-limited image S, which is formed by sharpest detection via intensity variance without temporal kernel regression. Here, the r l is local patch extracted from the noisy construction image S. The minimization P problem can be solved by @@aU ¼ 0. That is, 2kak þ i2Np k
174
C. Song et al. / Infrared Physics & Technology 71 (2015) 171–174
wi 2r k ½iðFðr½iÞ r l ½iÞ þ 2r2 dðk; lÞ ¼ 0. Thus, the solution of Eq. (5) is:
a ¼ ðA þ kIÞ1 d
ð6Þ P
P j ðr k ½i r j ½iÞ , dk ¼ i2Np
P where a ¼ ½a1 ; . . . ; aK , Ak;j ¼ i2Np wi wi r k ½i rl ½i r2 dðk; lÞ ; in which dðk; lÞ is delta function, defined
1; k ¼ l . I is T T unit matrix. The l is decided by by: dðk; lÞ ¼ 0; k–l the image frame when constructing the pixel l of S. Then, an image Z is generated by fusing all the denoised reconstruction pixels together. Finally, a single-image deblurring algorithm [15] is implemented to deblur the fused image Z, in which the parameters are set at kernel Width = 9 kernel Height = 9, noise Str = 0.1, deblur Strength = 0.3. 4. Experimental results In order to illustrate the performance of the proposed approach, two real videos are tested (https://users.soe.ucsc.edu/~xzhu/doc/ turbulence.html), shown in Figs. 1 and 2. In our method, the patch size L = 9, g ¼ 10, k ¼ 20. The method of near-diffraction-limited reconstruction approach (NDLR) [3] is used for comparison. The first dataset is Building of size 240 240 with 100 frames, shown in Fig. 1. The videos were captured through hot air exhausted by a building’s vent. The sequence was degraded by strong turbulence effects. The image reconstruction results are shown in Fig. 1(d)–(e). The image restoration results are shown in Fig. 1(f)–(g). The difference image between NDLR restoration and our method’s restoration is shown in Fig. 1(i). The difference main exists in the image edge details. One weight coefficient for local patch reconstruction is shown in Fig. 1(h). The weight coefficient at could be positive or negative. The proposed restoration result looks over NDLR in the edge regions. The second dataset is Moon’s surface of size 376 408 with 80 frames taken from a ground-based telescope, shown in Fig. 2. This video data is taken by long-distance imaging systems. From Fig. 2 we can see that the output image with our method also looks slightly sharper than NDLR. For all two image sequences, the proposed method gives an improvement in visual quality. It removed noise and blur successfully. Compared to the previous method, our reconstructed image shows even more detail and structure. 5. Conclusion In this paper, we proposed an effective method of diffractionlimited image reconstruction for restoring a single high-quality image from an image sequence distorted by air turbulence. The image geometric distortion, space–time varying blur, and noise
can be suppressed effectively. Experimental results on real data illustrate that the proposed method can restore the latent static image with more details of the scene from a degraded and noisy image sequence caused by turbulence. The proposed approach gives rise to superior results over the method of near-diffractionlimited image reconstruction. Conflict of interest There is no conflict of interest. Acknowledgments This work was supported in part by National Natural Science Foundation of China under Grant 61450006, Natural Science Foundation of Qinghai Province of China under Grant 2014-ZJ724 and the Ministry of National Education Chunhui Project of China under Grant Z2012099. References [1] D. Shi, C. Fan, P. Zhang, H. Shen, J. Zhang, C. Qiao, Y. Wang, Two-wavelength ghost imaging through atmospheric turbulence, Opt. Express 21 (2) (2013) 2050–2064. [2] M. Micheli, Y. Lou, S. Soatto, A. Bertozzi, A linear systems approach to imaging through turbulence, J. Math. Imaging Vis. 48 (1) (2014) 185–201. [3] X. Zhu, P. Milanfar, Removing atmospheric turbulence via space-invariant deconvolution, IEEE Trans. Pattern Anal. Mach. Intell. 35 (1) (2013) 157–170. [4] X. Zhu, P. Milanfar, Image reconstruction from videos distorted by atmospheric turbulence, in: Proceedings of Visual Information Processing and Communication, 2010. pp. 75430S–75430S-8. [5] M. Hirsch, S. Sra, B. Schölkopf, S. Harmeling, Efficient filter flow for spacevariant multiframe blind deconvolution, in: IEEE International Conference on Computer Vision and Pattern Recognition, 2010, pp. 607–614. [6] Y. Lou, S.H. Kang, S. Soatto, A. Bertozzi, Video stabilization of atmospheric turbulence distortion, Inverse Probl. Imaging 7 (2013) 839–861. [7] T. Blu, F. Luisier, The SURE-LET approach to image denoising, IEEE Trans. Image Process. 16 (11) (2007) 2778–2786. [8] S.R. Krishnan, C.S. Seelamantula, P. Chakravarti, Spatially adaptive kernel regression using risk estimation, IEEE Signal Process. Lett. 21 (4) (2014) 445– 448. [9] F. Luisier, T. Blu, M. Unser, SURE-LET for orthonormal wavelet-domain video denoising, IEEE Trans. Circuits Syst. Video Technol. 20 (6) (2010) 913–919. [10] F. Xue, F. Luisier, T. Blu, Multi-wiener sure-let deconvolution, IEEE Trans. Image Process. 22 (5) (2013) 1954–1968. [11] M. Lemaitrea, J. Blanc-Talona, F. Mériaudeaub, O. Laligant, Evaluation of infrared image restoration techniques, in: Proceedings of Electro-Optical and Infrared Systems: Technology and Applications III, 2006, pp. 63950R.163950R.9. [12] Y. Mao, J. Gilles, Non rigid geometric distortions correction-application to atmospheric turbulence stabilization, Inverse Probl. Imaging 6 (3) (2012) 531– 546. [13] D. Gong, Y. Zhang, S. Dang, J. Sun, Neighbor combination for atmospheric turbulence image reconstruction, in: IEEE International Conference on Image Processing, 2013, pp. 1361–1365. [14] L. Xinhao, M. Tanaka, M. Okutomi, Single-image noise level estimation for blind denoising, IEEE Trans. Image Process. 22 (12) (2013) 5226–5237. [15] Q. Shan, J. Jia, A. Agarwala, High-quality motion deblurring from a single image, ACM Trans. Graph. 27 (3) (2008) 73.