- Email: [email protected]
Image restoration for interlaced scan CCD image with space-variant motion blurs
ARTICLE IN PRESS Optics & Laser Technology 42 (2010) 894–901
Contents lists available at ScienceDirect
Optics & Laser Technology journal homepage: www.elsevier.com/locate/optlastec
Image restoration for interlaced scan CCD image with space-variant motion blurs Zhao Peng a,b,, Ni Guo-Qiang a, Xu Ting-Fa a a b
Department of Photoelectric Engineering, Beijing Institute of Technology, Beijing 100081, China Information and Computer Engineering Institute, Northeast Forestry University, Harbin City, 150040, China
a r t i c l e in f o
a b s t r a c t
Article history: Received 2 May 2009 Received in revised form 5 December 2009 Accepted 6 January 2010 Available online 27 January 2010
When the speeds of objects in a scene exceed the temporal resolution of the camera shutter, motion blurs will occur. Since objects are often moving in different directions at different speeds, the degradation of a CCD image is often characterized by space-variant motion blurs. Image restoration algorithms for space-variant motion blurs are available for progressive scan CCD images, but not for interlaced scan images. To address the space-variant image restoration for interlaced scan images, a novel three-step image restoration scheme is proposed. Firstly, one interlaced scan image is divided into odd field and even field images. Secondly, these two field images are further segmented into rectangular blocks and the motion vectors are computed based on these rectangular blocks using an efficient block matching algorithm. Thirdly, image restoration is performed using a blind deconvolution algorithm in the odd or even field image. The final restored image is obtained by combining the restored odd and even field images. The scheme is illustrated by restoring a space-variant blurred moving vehicle image and a synthetic blurred image. & 2010 Elsevier Ltd. All rights reserved.
Keywords: Image processing Image restoration Motion blur
1. Introduction When the speeds of different objects in a scene exceed the temporal resolution of the camera shutter, the output of a CCD camera is usually degraded by motion blurs subject to physical and technical limitations. Depending on the imaging process, image degradation caused by motion blur can be classified as either a space-invariant or a space-variant distortion. The spaceinvariant motion blur corresponds to the cases in which the image degradation model does not depend on the positions in the CCD image. This type of motion blurred image is often a result of camera motion during the imaging process. Image restorations for space-invariant motion blurs have been addressed by many schemes [1–4]. The goal is to find the point spread function (PSF) of the blurring system and then apply deconvolution techniques to restore the ideal images. For space-variant motion blur, the PSF, which causes the degradation, is a function of position in the image. This type of motion blur usually appears in an image containing fast-moving objects with different motions recorded by a static CCD camera. Image restoration for space-
Corresponding author at: Department of Photoelectric Engineering, Beijing Institute of Technology, Beijing 100081, China. Tel.: + 86 010 68912569; fax: + 86 010 68912569 816. E-mail address: [email protected] (Z. Peng).
0030-3992/$ - see front matter & 2010 Elsevier Ltd. All rights reserved. doi:10.1016/j.optlastec.2010.01.006
variant motion blur remains a big challenge and to date has been addressed only by a few researchers. Previous work on space-variant image restoration is relatively sparse. The main space-variant image restoration schemes include coordinate transforms [5], sectional processing [6–8], iterative filters [9,10], 2D Kalman filters [11,12], projection onto convex sets (POCS) [13], and multi-channel variational [14,15]. The coordinate transformation scheme is limited to types of motion for which transformations can be derived to change the space-variant problem into a space-invariant one. This restoration has been done for rotational motion blurs and some types of geometrically induced motion blurs such as produced in satellite photography. In sectional schemes, the image is sectioned into rectangular regions, where each section is restored using a spaceinvariant method, such as the maximum a posterior filter [6] or the modified Landweber iterative filter [7]. An accurate estimation of the local motion vector for each section is required in sectional schemes, and this estimation is often performed using adjacent image frames from an image sequence. An iterative filter is presented for space-variant image restoration in Ref. [10], where the distortion process was separated for several layers or images, and a regularized iterative filter has been used. Patti et al. applied the reduced order model Kalman filter (ROMKF) to spacevariant image restoration and found that the scheme was computationally expensive for a motion blurred image even with moderate-size blurs. The multi-channels [14,15] are applicable to
ARTICLE IN PRESS Z. Peng et al. / Optics & Laser Technology 42 (2010) 894–901
degraded images with complicated motion blurs and out-of-focus blurs when the scene has depth, using multiple blurred images. All of these image restoration schemes do not specifically consider the effects of interlaced field images (i.e., the odd and even field images of one interlaced scan CCD image), although Fogel et al. indicated the need for the consideration of interlaced scan effects as a future research topic [7]. Therefore, these image restoration schemes may not provide satisfactory restoration results for interlaced scan CCD images, while good restoration results can be achieved for progressive scan CCD images. To address this problem, we propose a novel space-variant image restoration scheme for interlaced scan CCD images based on sectional processing. This novel scheme not only outperforms sectional processing schemes proposed for interlaced scan CCD image restoration [6–8] but also is time efficient since only one image frame is required in this scheme, in contrast to image sequence required in those sectional schemes [6–8]. In our scheme, only one image frame is required for the image restoration. In fact, if multiple image frames are used in a motion estimation procedure, the motion offset between the odd and the even fields of one image frame and the motion offset between the adjacent image frames will mix together. This motion offset mixture may result in large motion estimation errors. Therefore, only one image frame is used in our scheme. 2. Image restoration Our image restoration scheme for space-variant motion blurs is based on the sectional processing scheme. As is the case of all sectional image restorations, our novel scheme assumes the motion to be locally linear in a region of a reasonable size (e.g. a small rectangular block region). This assumption is reasonable since neighboring points will usually be moving at similar speeds. Therefore, the PSF in every small rectangular block can be approximately space-invariant and then some image restoration schemes for space-invariant linear motion blur can be used in every small rectangular block. Our scheme consists of three steps. First, only one interlaced scan image frame is required to extract the odd field image and the even field image. Second, the motion vector estimation is performed in every rectangular block unit using the odd field and the even field images, which requires a time-efficient block matching algorithm. Third, the PSF computed with the estimated motion vector in every rectangular block unit will be used as an initial PSF estimation for the subsequent blind deconvolution image restoration. This PSF initialization can obviously both accelerate the blind deconvolution speed and improve the image restoration result.
895
rectangular block unit is also called as macro block. Using a block matching algorithm, every macro block in the resized even field image is then compared with the corresponding macro block and its adjacent neighbors in the resized odd field image to create a motion vector that stipulates the movement of a macro block from one location to another. The computed motion vector in every macro block will be used in the subsequent PSF estimation as described in Section 2.3. In a block matching algorithm, two parameters, the size of macro block and the search parameter p, must be set with reasonable values. For example, one macro block is taken as a square of side 16 pixels, and the search parameter is 7 pixels in Fig. 1. The matching of one macro block with another is based on the output of a cost function such as the mean absolute difference (MAD) or the mean squared error (MSE). The macro block that results in the least cost is the one that matches the closest to current block. There are some fundamental block matching algorithms, which include the exhaustive search (ES), three step search (TSS), new three step search (NTSS) [16], simple and efficient search (SES) [17], four step search (FSS) [18], diamond search (DS) [19] and adaptive rood pattern search (ARPS) [20]. In general, DS and ARPS algorithms prove to be good block matching solutions, since they provide excellent Peak-Signal-to-Noise-Ratio (PSNR) results, close to those results of ES, while they drop the searches by more than an order of magnitude. Therefore, the DS algorithm is employed in the block matching in every macro block of the resized odd or even field image.
Fig. 1. One macro block for block matching.
2.1. Odd and even field image extraction The extraction of odd or even field image is the first step in our scheme. The odd or even field image is extracted from the interlaced scan image frame I(x,y), y= 0,1, y, N 1 so that Io(x,y) = I(x,y), y= 2k, k= 0,2, y, N/2 1; Ie(x,y)= I(x,y), y= 2k+1, k= 0,1,2, y, N/2 1. Then Io(x,y) or Ie(x,y) is enlarged to form Io1(x,y) or Ie1(x,y) that is identical with I(x,y) in size using an interpolation algorithm. A nearest-neighbor interpolation, a bilinear interpolation or a bicubic interpolation can be used in the interpolation algorithm. 2.2. Motion vector estimations After the odd and even field images are extracted and resized, they are divided into a matrix of rectangular block units. Each
Fig. 2. DS procedure; this figure shows the LDSP and SDSP and also shows an example path to motion vector ( 4, 2) in five search steps, which include four times of LDSP and one time of SDSP.
ARTICLE IN PRESS 896
Z. Peng et al. / Optics & Laser Technology 42 (2010) 894–901
In the DS algorithm, the search point pattern is changed from a square to a diamond, and there is no limit on the number of steps that the algorithm can take. DS uses two different types of fixed patterns: Large diamond search pattern (LDSP) and small diamond search pattern (SDSP). These two patterns and the DS procedure are illustrated in Fig. 2. The first step uses LDSP, and if the least cost is at the center location, we jump to the fourth step. The consequent steps, except for the last step, are similar to step 1 and use LDSP, while the number of points where the cost function is checked is either 3 or 5, as illustrated in Fig. 2. The final step uses SDSP around the new search origin, and the location with the least cost is the best match. Since the search pattern is neither too small nor too big and there is no limit to the number of steps, this algorithm can find global minimum very accurately. The end result can achieve a PSNR close to that of ES while computational expense is significantly decreased. 2.3. Blind deconvolution image restoration
computed using the above-mentioned motion estimation is set as an accurate initial PSF in the blind deconvolution restoration in every macro block, which not only accelerates restoration speed but also improves the restoration result. In fact, single-image blind deconvolution is considered the most ill-posed problem, since it must estimate the PSF and the original image. Previous schemes assume parametric models for the PSF such as a low-pass filter [24] or a sum of normal distributions [25]. Fergus et al. [26] shows that blur kernels are often complex and sharp and uses ensemble learning to recover a blur kernel. A variational method is used to approximate the posterior distribution and then Richardson–Lucy is applied for deconvolution. Shan et al. [27] create a unified probabilistic framework for both blur kernel estimation and original image restoration, which effectively avoids local minima and ringing artifacts. However, these two blind deconvolution schemes based on probabilistic frameworks fail to restore the blurred image from slight camera rotation or non-uniform object motion (i.e. spacevariant motion blur).
A motion vector (D1x, D1y) for every macro block between the resized odd and even field images (i.e.Io1(x,y) and Ie1(x,y)) is computed based on the above-mentioned DS algorithm. According to the CCD imaging principle, the time interval between the odd and even field images is the summation of the charge integration time of the odd or even field image and its blanking time. Therefore, the charge integration time of the odd or even field image is 0.92 times the time interval between the odd and even field images, subject to the PAL mode standard. Consequently, one motion vector for every macro block in the odd or even field image can be computed approximately as follows:
Do1 x ¼ De1 x ¼ 0:92D1 x;
Do1 y ¼ De1 y ¼ 0:92D1 y
ð1Þ
where (D1x, D1y) is the motion vector for the corresponding macro block between the resized odd and even field images. On the other hand, image blur arises from the relative linear motion between the CCD camera and the moving object, resulting in a degraded version of the same image. Generally, the relation between the observed image g(x,y) and its uncorrupted version f(x,y) is as follows [21]: gðx; yÞ ¼ f ðx; yÞhðx; yÞ þ nðx; yÞ
ð2Þ
where h is the point spread function (PSF) that convolves an original image and n is the additive noise function. The general form of linear motion blur function is given as follows [22]: ( pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1=L; if x2 þy2 r L=2; x=y ¼ tanðfÞ ð3Þ hðx; yÞ ¼ 0; otherwise As seen in Eq. (3), linear motion blur depends on two parameters: motion length L and motion direction f. In every macro block of the resized odd or even field image, these two parameters can be computed simply by the motion vector obtained with Eq. (1). After the PSF is obtained in every macro block of the resized odd or even field image, each macro block is restored by a blind deconvolution image restoration [23]. This restoration scheme maximizes the likelihood that the resulting image, when convolved with the resulting PSF, is an instance of the blurred image, assuming Poisson noise statistics. This blind deconvolution can be used effectively when no information about the blurring and noise is known. This scheme restores the degraded image and the PSF simultaneously, by using the degraded image and the initial estimated PSF as two input parameters and an iterative process similar to the Lucy–Richardson algorithm. In this scheme, it is very important to use the initial PSF estimation as one input parameter. In fact, an accurate initial PSF estimation can achieve a good image restoration result with only a few iterations. Therefore, the PSF
Fig. 3. One motion blurred moving car image.
Fig. 4. Restored car image using our scheme with 5 blind deconvolution iterations and block size 16 16.
ARTICLE IN PRESS Z. Peng et al. / Optics & Laser Technology 42 (2010) 894–901
In our scheme, the complex probabilistic-framework-based PSF estimations [26,27] are not employed, since the PSF computed using the DS motion estimation algorithm and the odd and even field images are usually accurate enough to be an initial PSF in the blind deconvolution restoration in every macro block. Therefore, this scheme not only accelerates restoration speed but also improves the restoration result. After the blind deconvolution restoration is performed in every macro block of the resized odd or even field image, we can get the restored, resized odd or even field image denoted as R R ðx; yÞorIe1 ðx; yÞ. Then each macro block after the motion Io1 R ðx; yÞ can compensation with the motion vector (D1x, D1y) in Io1 be obtained as follows: 2
3
2
1 xRo2 6 R 7 6 4 yo2 5 ¼ 4 0 0 1
0 1 0
32
3
D1 x xRo1 6 7 D1 y 7 54 yRo1 5 1
ð4Þ
1
897
Therefore, the motion compensated odd field image obtained R in this way is denoted as Io2 ðx; yÞ. Finally, the restored interlaced R scan CCD image I (x,y) is formed as follows: ( R Io2 ðx; yÞ; y ¼ 2k; k ¼ 0; 1; . . . ; N=21 ð5Þ IR ðx; yÞ ¼ R ðx; yÞ; y ¼ 2k þ1; k ¼ 0; 1; . . . ; N=21 Ie1
3. Results and comparisons 3.1. Real image restoration In this Section, a comparative image restoration experiment is performed, using our image restoration scheme and for comparison using another sectional processing [7]. A fast-moving car image sequence is taken and recorded by an interlaced scan CCD camera. One image frame of this image sequence is selected and shown in
Fig. 5. Restored car image using our scheme with 10 blind deconvolution iterations and block size 16 16.
Fig. 7. Restored car image using scheme of Ref. [7].
Fig. 6. Restored car image using our scheme with macro block size 32 32and 5 iterations.
Fig. 8. Restored car image using odd and even field extraction, motion computation and scheme of Ref. [27].
ARTICLE IN PRESS 898
Z. Peng et al. / Optics & Laser Technology 42 (2010) 894–901
Fig. 9. Image restoration comparison for the wheel sub-image: (a) original wheel image; (b) restored wheel image by scheme of Ref. [7]; (c) restored wheel image by scheme of Ref. [27]; (d) restored odd field image by our scheme; (e) restored even field image by our scheme and (f) restored final image by our scheme.
Fig. 10. Computed motion vectors with respect to different 1584 macro blocks: (a) vertical components of the computed motion vectors and (b) horizontal components of the computed motion vectors.
ARTICLE IN PRESS Z. Peng et al. / Optics & Laser Technology 42 (2010) 894–901
Fig. 3 (the image size is 704 576). In Fig. 3, a fast-moving car is moving horizontally in front of the static background wall. This moving car can be approximately divided into two different parts with different motions. The large part with a horizontal linear motion is the car body’s side face except the wheel. The small part is the moving wheel with a complicated motion, which combines a horizontal linear motion with a rotation motion. First, our scheme is used to restore this degraded image with space-variant motion blurs. The DS block matching algorithm is used here in the motion vector estimation for each macro block. The size of macro block is 16 16 and the search parameter p = 20. In our scheme, the size of macro block should be small enough to ensure that each macro block in the wheel part can be restored approximately with a space-invariant linear motion blur model. In fact, it is assumed that in each macro block in the wheel part, the combined motion can approximately be seen as a spaceinvariant linear motion, if the size of each macro block is small enough. Moreover, it can also be assumed that the curve motion between the matching blocks in the wheel part can approximately be seen as a linear motion. Although these approximations will result in PSF computation error in every macro block, as described in the third step of our scheme, the PSF computed using the above-mentioned linear motion approximations is accurate enough to be set as an initial PSF in the blind deconvolution restoration for every macro block. Again, our scheme not only accelerates restoration speed but also improves the restoration result. The restored final image is illustrated in Fig. 4 with only 5 blind deconvolution iterations. The mean time requirement for
899
our image restoration scheme is 36.7 s, while the mean time requirements for the first step, second step and third step are 1.0, 2.7 and 33.0 s, respectively. All image restorations are performed using the Matlab 6.5 on an AMD Sempron 2400 +1.66 GHz computer with an internal memory of 512 M and a Windows XP system. Second, to discover the influences of the macro block size and the blind deconvolution iterations on our image restoration, some comparative image restorations are also performed. One restored final image is illustrated in Fig. 5 with 10 blind deconvolution iterations. The mean time requirement for this image restoration scheme is 63.4s, while the mean time requirements for the first step, second step and third step are 1.0, 2.7 and 59.7 s, respectively. It can be seen that the time requirement for the third step increases dramatically, when compared with that with only 5 blind deconvolution iterations. However, image restoration results in Figs. 4 and 5 are approximately the same, since the PSF computed using the above-mentioned linear motion approximations is accurate enough to be set as an initial PSF in the blind deconvolution restoration in every macro block, which ensures that good image restoration can be achieved with only a few iterations. Another restored final image is illustrated in Fig. 6 with a macro block size of 32 32. The mean time requirement for this image restoration scheme is 13.4 s, while the mean time requirements for the first step, second step and third step are 1.0, 1.5 and 10.9 s, respectively. It can be seen that time requirement for the third step decreases greatly, when compared with that with the macro block size of 16 16. The image restoration result
Fig. 11. Synthetic space-variant blurred image: (a) original image 1; (b) original image 2; (c) synthetic blurred image 1; (d) synthetic blurred image 2 and (e) final synthetic interlaced scan blurred image.
ARTICLE IN PRESS 900
Z. Peng et al. / Optics & Laser Technology 42 (2010) 894–901
in Fig. 4 is better than that in Fig. 6. In fact, there are usually some blocking artifacts in the image restoration caused by sectional processing. These artifacts can be reduced by the overlap of sections or by decreasing the block size, at the cost of more time requirement for image restoration. Third, the image restoration schemes described in Refs. [7,27] are also performed here for the restoration performance comparison. A modified Landweber iteration [7] is used in every sectional region (i.e. one macro block of size 16 16). In this scheme, the fast-moving car image sequence is used in the motion vector estimation in every macro block, while in our novel scheme only one image frame is required for motion vector estimation. The final restored image is shown in Fig. 7, and the mean time requirement for this scheme is 55.7s. We can see that the image restoration in Fig. 4 outperforms that in Fig. 7, because the restoration scheme in Ref. [7] does not compensate the motion offset between the odd and even field images. In order to compare with the scheme in Ref. [27], we first extract the odd and even field images, followed by the DS motion estimation algorithm. We
then apply the probabilistic-framework-based deblur [27] to remove motion blurs. The final restored image is shown in Fig. 8, and the mean time requirement for this scheme is approximately 4 min. We can see that the image restoration in Fig. 4 also outperforms that in Fig. 8. Moreover, one restoration comparison in the image region of the car’s wheel is illustrated in Fig. 9, which also indicates that our scheme outperforms those image restoration schemes in Refs. [7,27]. Finally, to find how stable the DS algorithm is on patches with uniform colors or textured structures, the vertical components and the horizontal components of the computed motion vectors in Fig. 3 are illustrated in Fig. 10, with respect to different macro blocks. It can be seen that some motion computation errors do occur at those patches, which may propagate to the PSF computation and the image deconvolution as well. In our future work, we will propose some improvement solutions to the DS algorithm (e.g. we may impose constraints on the block matching, apply a multi-scale approach in a coarse-to-fine manner, or manually select patches with rich edge structures).
Fig. 12. Synthetic image restoration comparison: (a) image restoration by scheme of Ref. [7]; (b) image restoration by scheme of Ref. [27] and (c) image restoration by our scheme.
ARTICLE IN PRESS Z. Peng et al. / Optics & Laser Technology 42 (2010) 894–901
Acknowledgement
Table 1 Image restoration comparisons for different schemes.
Ref. [7] scheme Ref. [27] scheme Our scheme
901
ABSDIFF
SNR
PSNR
IMFID
MSE
0.038677 0.034795 0.027160
19.160 20.522 22.828
22.397 23.758 26.064
0.012105 0.008839 0.005187
0.0042865 0.0031329 0.0018423
This research is supported by the National Fundamental Research Plan Project (973 Project) with grant No. 2009CB72400603. It is also supported by the Harbin Youth Originality Science Foundation with grant No. 2008RFQXG005 and the National Natural Science Foundation with grant No. 60972100. References
3.2. Synthetic image restoration In order to quantitatively compare our scheme with those schemes in Refs. [7,27], one image restoration comparison for a space-variant blurred synthetic image is also performed, although the synthetic motion blurs are not absolutely identical to motion blurs in real life. One sub-image sequence of the ‘‘caltrain’’ sequence is used to produce a synthetic space-variant blurred image sequence. For example, one synthetic blurred image is illustrated in Fig. 11, which consists of two parts with different motion blurs. One part is the left part from column 1 to column 220 with a horizontal motion blur. The other part is the right part from column 221 to column 512 with a vertical motion blur. This synthetic blurred image is produced by using the following steps. First, two images are selected from the above image sequence. The left parts of these two images have a horizontal motion, while the right parts have a vertical motion. Therefore, the horizontal and the vertical motion blurs are produced in the left part and the right part, respectively, using the standard Matlab filter functions. Second, these two synthetic blurred images are used as the odd and even field images to synthesize one interlaced scan blurred image. As for the motion computation, the DS algorithm is again used in this synthetic image. In fact, as pointed out in Ref. [19], the DS algorithm has been implemented in MPEG-4 video-encoding environment and its efficacy has been demonstrated in the core experimental results [28]. Based on these results, the DS algorithm is adopted and incorporated into the MPEG-4 verification model [29]. The final image restoration comparison is illustrated in Fig. 12, and the quantitative comparison for different schemes is illustrated in Table 1. In Table 1, the ABSDIFF (Average Absolute Difference), the SNR (Signal to Noise Ratio), the PSNR (Peak Signal to Noise Ratio), the IMFID (Image Fidelity), and the MSE (Mean Square Error) are used as the comparison standards for evaluating image restoration performance. Since, in general, a better restored image will have the large SNR, PSNR and IMFID values with small ABSDIFF and MSE values, we can see that our scheme outperforms again those image restoration schemes in Refs. [7,27].
4. Conclusions In this paper, a novel image restoration scheme for interlaced scan CCD image with space-variant motion blurs is proposed. In this scheme, only one image frame is used to perform the motion vector estimation in every macro block. The PSF computed with the motion vector estimation is set as an initial PSF in the blind deconvolution restoration, which not only accelerates the restoration speed but also improves the restoration result. This restoration scheme can be used in the practical interlaced scan CCD image restoration work.
[1] Sondhi M. Image restoration: the removal of spatially invariant degradations. Proceedings of IEEE 1972;60(7):842–53. [2] Slepian D. Restoration of photographs blurred by image motion. Bell System Technical Journal 1967;46(10):2353–62. [3] Cannon M. Blind deconvolution of spatially invariant image blurs with phase. IEEE Transactions on ASSP 1976;24(1):58–63. [4] Yitzhaky Y, Mor I, Lantzman A, et al. Direct method for restoration of motion blurred images. Journal of OSA 1998;15(6):1512–9. [5] Sawchuk A. Space-variant image restoration by coordinate transformation. Journal of OSA 1974;60(2):138–44. [6] Trussell HJ, Hunt BR. Image restoration of space-variant blurs by sectional methods. IEEE Transactions on ASSP 1978;26(10):608–9. [7] Trussell HJ, Fogel S. Identification and restoration of spatially variant motion blurs in sequential images. IEEE Transactions on Image Processing 1992;1(1):123–6. [8] Kang S, Choung Y, Paik J. Segmentation-based image restoration for multiple moving objects with different motions. Proceedings of IEEE ICIP 1999;1: 376–80. [9] Tull DT, Katsaggelos AK. Iterative restoration of fast-moving objects in dynamic image sequences. Optical Engineering 1996;35(12):3460–9. [10] Tull DT, Katsaggelos AK. Regularized restoration of partial-response distortions in sporadically degraded images. Proceedings of IEEE ICIP 1998;3: 732–6. [11] Tekalp AM, Kaufman H, Woods JW. Model-based segmentation and spacevariant restoration of blurred images by decision-directed filtering. Signal Processing 1988:259–69. [12] Patti AJ, Ozkan MK, Tekalp AM, et al. New approaches for space-variant image restoration. Proceedings of IEEE ICASSP 1993;2:261–4. [13] Ozkan MK, Tekalp AM, Sezan MI. POCS-Based restoration of space-varying blurred images. IEEE Transactions on Image Processing 1994;3(4):450–4. [14] Rajagopalan AN, Chaudhuri S. A MRF-model based approach to simultaneous recovery of depth and restoration from defocused images. IEEE Transactions on PAMI 1999;21(7):577–89. [15] Sorel M, Flusser J. Space-variant restorations of images degraded by camera motion blur. IEEE Transactions on Image Processing 2008;17(2):105–16. [16] Xiang LR, Bing Z, Ming LL. A new three-step search algorithm for block motion estimation. IEEE Transactions on Circuits and Systems for Video Technology 1994;4(4):438–42. [17] Hua LJ, Ming LL. A simple and efficient search algorithm for block matching motion estimation. IEEE Transactions on Circuits and Systems for Video Technology 1997;7(2):429–33. [18] Man PL, Chung MW. A novel four-step search algorithm for fast block motion estimation. IEEE Transactions on Circuits and Systems for Video Technology 1996;6(3):313–7. [19] Shan Z, Kuang MK. A new diamond search algorithm for fast block matching motion estimation. IEEE Transactions on Image Processing 2000;9(2): 287–90. [20] Yao N, Kuang MK. Adaptive rood pattern search for fast block matching motion estimation. IEEE Transactions on Image Processing 2002;11(12) 1442–8. [21] Banham MR, Katsaggleos AK. Digital image restoration. IEEE Signal Process Magazine 1997;24(1):24–41. [22] Moghaddam ME, Jamzad M. Motion blur identification in noisy images using mathematical models and statistical measures. Pattern Recognition 2007;40(11):1946–57. [23] Kundur D, Hatzinakos D. Blind image deconvolution. IEEE Signal Process Magazine 1996;23(1):43–64. [24] Kim SK, Paik JK. Out-of-focus blur estimation and restoration for digital autofocusing system. Electronics Letters 1998;34(12):1217–9. [25] Likas A, Galatsanos N. A variational approach for Bayesian blind image deconvolution. IEEE Transactions on Signal Processing 2004;52(8):2222–33. [26] Fergus R, Singh B, Hertzmann A, et al. Removing camera shake from a single photograph. ACM Transactions on Graphics 2006;25:787–94. [27] Shan Q, Jia J, Agarwala A. High-quality motion deblurring from a single image. ACM Transactions on Graphics 2008;27(3):73:1–10. [28] Kuang MK, Hosur PI. Core experimental results of fast motion estimation based on new test conditions (Q4a). ISO/IECJTC1/SC29/WG11/M4934, July 1999. [29] MPEG-4 video verification model, version. 14.0, ISO/IECJTC1/SC29/WG11/ N2932, October 1999.
Contents lists available at ScienceDirect
Optics & Laser Technology journal homepage: www.elsevier.com/locate/optlastec
Image restoration for interlaced scan CCD image with space-variant motion blurs Zhao Peng a,b,, Ni Guo-Qiang a, Xu Ting-Fa a a b
Department of Photoelectric Engineering, Beijing Institute of Technology, Beijing 100081, China Information and Computer Engineering Institute, Northeast Forestry University, Harbin City, 150040, China
a r t i c l e in f o
a b s t r a c t
Article history: Received 2 May 2009 Received in revised form 5 December 2009 Accepted 6 January 2010 Available online 27 January 2010
When the speeds of objects in a scene exceed the temporal resolution of the camera shutter, motion blurs will occur. Since objects are often moving in different directions at different speeds, the degradation of a CCD image is often characterized by space-variant motion blurs. Image restoration algorithms for space-variant motion blurs are available for progressive scan CCD images, but not for interlaced scan images. To address the space-variant image restoration for interlaced scan images, a novel three-step image restoration scheme is proposed. Firstly, one interlaced scan image is divided into odd field and even field images. Secondly, these two field images are further segmented into rectangular blocks and the motion vectors are computed based on these rectangular blocks using an efficient block matching algorithm. Thirdly, image restoration is performed using a blind deconvolution algorithm in the odd or even field image. The final restored image is obtained by combining the restored odd and even field images. The scheme is illustrated by restoring a space-variant blurred moving vehicle image and a synthetic blurred image. & 2010 Elsevier Ltd. All rights reserved.
Keywords: Image processing Image restoration Motion blur
1. Introduction When the speeds of different objects in a scene exceed the temporal resolution of the camera shutter, the output of a CCD camera is usually degraded by motion blurs subject to physical and technical limitations. Depending on the imaging process, image degradation caused by motion blur can be classified as either a space-invariant or a space-variant distortion. The spaceinvariant motion blur corresponds to the cases in which the image degradation model does not depend on the positions in the CCD image. This type of motion blurred image is often a result of camera motion during the imaging process. Image restorations for space-invariant motion blurs have been addressed by many schemes [1–4]. The goal is to find the point spread function (PSF) of the blurring system and then apply deconvolution techniques to restore the ideal images. For space-variant motion blur, the PSF, which causes the degradation, is a function of position in the image. This type of motion blur usually appears in an image containing fast-moving objects with different motions recorded by a static CCD camera. Image restoration for space-
Corresponding author at: Department of Photoelectric Engineering, Beijing Institute of Technology, Beijing 100081, China. Tel.: + 86 010 68912569; fax: + 86 010 68912569 816. E-mail address: [email protected] (Z. Peng).
0030-3992/$ - see front matter & 2010 Elsevier Ltd. All rights reserved. doi:10.1016/j.optlastec.2010.01.006
variant motion blur remains a big challenge and to date has been addressed only by a few researchers. Previous work on space-variant image restoration is relatively sparse. The main space-variant image restoration schemes include coordinate transforms [5], sectional processing [6–8], iterative filters [9,10], 2D Kalman filters [11,12], projection onto convex sets (POCS) [13], and multi-channel variational [14,15]. The coordinate transformation scheme is limited to types of motion for which transformations can be derived to change the space-variant problem into a space-invariant one. This restoration has been done for rotational motion blurs and some types of geometrically induced motion blurs such as produced in satellite photography. In sectional schemes, the image is sectioned into rectangular regions, where each section is restored using a spaceinvariant method, such as the maximum a posterior filter [6] or the modified Landweber iterative filter [7]. An accurate estimation of the local motion vector for each section is required in sectional schemes, and this estimation is often performed using adjacent image frames from an image sequence. An iterative filter is presented for space-variant image restoration in Ref. [10], where the distortion process was separated for several layers or images, and a regularized iterative filter has been used. Patti et al. applied the reduced order model Kalman filter (ROMKF) to spacevariant image restoration and found that the scheme was computationally expensive for a motion blurred image even with moderate-size blurs. The multi-channels [14,15] are applicable to
ARTICLE IN PRESS Z. Peng et al. / Optics & Laser Technology 42 (2010) 894–901
degraded images with complicated motion blurs and out-of-focus blurs when the scene has depth, using multiple blurred images. All of these image restoration schemes do not specifically consider the effects of interlaced field images (i.e., the odd and even field images of one interlaced scan CCD image), although Fogel et al. indicated the need for the consideration of interlaced scan effects as a future research topic [7]. Therefore, these image restoration schemes may not provide satisfactory restoration results for interlaced scan CCD images, while good restoration results can be achieved for progressive scan CCD images. To address this problem, we propose a novel space-variant image restoration scheme for interlaced scan CCD images based on sectional processing. This novel scheme not only outperforms sectional processing schemes proposed for interlaced scan CCD image restoration [6–8] but also is time efficient since only one image frame is required in this scheme, in contrast to image sequence required in those sectional schemes [6–8]. In our scheme, only one image frame is required for the image restoration. In fact, if multiple image frames are used in a motion estimation procedure, the motion offset between the odd and the even fields of one image frame and the motion offset between the adjacent image frames will mix together. This motion offset mixture may result in large motion estimation errors. Therefore, only one image frame is used in our scheme. 2. Image restoration Our image restoration scheme for space-variant motion blurs is based on the sectional processing scheme. As is the case of all sectional image restorations, our novel scheme assumes the motion to be locally linear in a region of a reasonable size (e.g. a small rectangular block region). This assumption is reasonable since neighboring points will usually be moving at similar speeds. Therefore, the PSF in every small rectangular block can be approximately space-invariant and then some image restoration schemes for space-invariant linear motion blur can be used in every small rectangular block. Our scheme consists of three steps. First, only one interlaced scan image frame is required to extract the odd field image and the even field image. Second, the motion vector estimation is performed in every rectangular block unit using the odd field and the even field images, which requires a time-efficient block matching algorithm. Third, the PSF computed with the estimated motion vector in every rectangular block unit will be used as an initial PSF estimation for the subsequent blind deconvolution image restoration. This PSF initialization can obviously both accelerate the blind deconvolution speed and improve the image restoration result.
895
rectangular block unit is also called as macro block. Using a block matching algorithm, every macro block in the resized even field image is then compared with the corresponding macro block and its adjacent neighbors in the resized odd field image to create a motion vector that stipulates the movement of a macro block from one location to another. The computed motion vector in every macro block will be used in the subsequent PSF estimation as described in Section 2.3. In a block matching algorithm, two parameters, the size of macro block and the search parameter p, must be set with reasonable values. For example, one macro block is taken as a square of side 16 pixels, and the search parameter is 7 pixels in Fig. 1. The matching of one macro block with another is based on the output of a cost function such as the mean absolute difference (MAD) or the mean squared error (MSE). The macro block that results in the least cost is the one that matches the closest to current block. There are some fundamental block matching algorithms, which include the exhaustive search (ES), three step search (TSS), new three step search (NTSS) [16], simple and efficient search (SES) [17], four step search (FSS) [18], diamond search (DS) [19] and adaptive rood pattern search (ARPS) [20]. In general, DS and ARPS algorithms prove to be good block matching solutions, since they provide excellent Peak-Signal-to-Noise-Ratio (PSNR) results, close to those results of ES, while they drop the searches by more than an order of magnitude. Therefore, the DS algorithm is employed in the block matching in every macro block of the resized odd or even field image.
Fig. 1. One macro block for block matching.
2.1. Odd and even field image extraction The extraction of odd or even field image is the first step in our scheme. The odd or even field image is extracted from the interlaced scan image frame I(x,y), y= 0,1, y, N 1 so that Io(x,y) = I(x,y), y= 2k, k= 0,2, y, N/2 1; Ie(x,y)= I(x,y), y= 2k+1, k= 0,1,2, y, N/2 1. Then Io(x,y) or Ie(x,y) is enlarged to form Io1(x,y) or Ie1(x,y) that is identical with I(x,y) in size using an interpolation algorithm. A nearest-neighbor interpolation, a bilinear interpolation or a bicubic interpolation can be used in the interpolation algorithm. 2.2. Motion vector estimations After the odd and even field images are extracted and resized, they are divided into a matrix of rectangular block units. Each
Fig. 2. DS procedure; this figure shows the LDSP and SDSP and also shows an example path to motion vector ( 4, 2) in five search steps, which include four times of LDSP and one time of SDSP.
ARTICLE IN PRESS 896
Z. Peng et al. / Optics & Laser Technology 42 (2010) 894–901
In the DS algorithm, the search point pattern is changed from a square to a diamond, and there is no limit on the number of steps that the algorithm can take. DS uses two different types of fixed patterns: Large diamond search pattern (LDSP) and small diamond search pattern (SDSP). These two patterns and the DS procedure are illustrated in Fig. 2. The first step uses LDSP, and if the least cost is at the center location, we jump to the fourth step. The consequent steps, except for the last step, are similar to step 1 and use LDSP, while the number of points where the cost function is checked is either 3 or 5, as illustrated in Fig. 2. The final step uses SDSP around the new search origin, and the location with the least cost is the best match. Since the search pattern is neither too small nor too big and there is no limit to the number of steps, this algorithm can find global minimum very accurately. The end result can achieve a PSNR close to that of ES while computational expense is significantly decreased. 2.3. Blind deconvolution image restoration
computed using the above-mentioned motion estimation is set as an accurate initial PSF in the blind deconvolution restoration in every macro block, which not only accelerates restoration speed but also improves the restoration result. In fact, single-image blind deconvolution is considered the most ill-posed problem, since it must estimate the PSF and the original image. Previous schemes assume parametric models for the PSF such as a low-pass filter [24] or a sum of normal distributions [25]. Fergus et al. [26] shows that blur kernels are often complex and sharp and uses ensemble learning to recover a blur kernel. A variational method is used to approximate the posterior distribution and then Richardson–Lucy is applied for deconvolution. Shan et al. [27] create a unified probabilistic framework for both blur kernel estimation and original image restoration, which effectively avoids local minima and ringing artifacts. However, these two blind deconvolution schemes based on probabilistic frameworks fail to restore the blurred image from slight camera rotation or non-uniform object motion (i.e. spacevariant motion blur).
A motion vector (D1x, D1y) for every macro block between the resized odd and even field images (i.e.Io1(x,y) and Ie1(x,y)) is computed based on the above-mentioned DS algorithm. According to the CCD imaging principle, the time interval between the odd and even field images is the summation of the charge integration time of the odd or even field image and its blanking time. Therefore, the charge integration time of the odd or even field image is 0.92 times the time interval between the odd and even field images, subject to the PAL mode standard. Consequently, one motion vector for every macro block in the odd or even field image can be computed approximately as follows:
Do1 x ¼ De1 x ¼ 0:92D1 x;
Do1 y ¼ De1 y ¼ 0:92D1 y
ð1Þ
where (D1x, D1y) is the motion vector for the corresponding macro block between the resized odd and even field images. On the other hand, image blur arises from the relative linear motion between the CCD camera and the moving object, resulting in a degraded version of the same image. Generally, the relation between the observed image g(x,y) and its uncorrupted version f(x,y) is as follows [21]: gðx; yÞ ¼ f ðx; yÞhðx; yÞ þ nðx; yÞ
ð2Þ
where h is the point spread function (PSF) that convolves an original image and n is the additive noise function. The general form of linear motion blur function is given as follows [22]: ( pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1=L; if x2 þy2 r L=2; x=y ¼ tanðfÞ ð3Þ hðx; yÞ ¼ 0; otherwise As seen in Eq. (3), linear motion blur depends on two parameters: motion length L and motion direction f. In every macro block of the resized odd or even field image, these two parameters can be computed simply by the motion vector obtained with Eq. (1). After the PSF is obtained in every macro block of the resized odd or even field image, each macro block is restored by a blind deconvolution image restoration [23]. This restoration scheme maximizes the likelihood that the resulting image, when convolved with the resulting PSF, is an instance of the blurred image, assuming Poisson noise statistics. This blind deconvolution can be used effectively when no information about the blurring and noise is known. This scheme restores the degraded image and the PSF simultaneously, by using the degraded image and the initial estimated PSF as two input parameters and an iterative process similar to the Lucy–Richardson algorithm. In this scheme, it is very important to use the initial PSF estimation as one input parameter. In fact, an accurate initial PSF estimation can achieve a good image restoration result with only a few iterations. Therefore, the PSF
Fig. 3. One motion blurred moving car image.
Fig. 4. Restored car image using our scheme with 5 blind deconvolution iterations and block size 16 16.
ARTICLE IN PRESS Z. Peng et al. / Optics & Laser Technology 42 (2010) 894–901
In our scheme, the complex probabilistic-framework-based PSF estimations [26,27] are not employed, since the PSF computed using the DS motion estimation algorithm and the odd and even field images are usually accurate enough to be an initial PSF in the blind deconvolution restoration in every macro block. Therefore, this scheme not only accelerates restoration speed but also improves the restoration result. After the blind deconvolution restoration is performed in every macro block of the resized odd or even field image, we can get the restored, resized odd or even field image denoted as R R ðx; yÞorIe1 ðx; yÞ. Then each macro block after the motion Io1 R ðx; yÞ can compensation with the motion vector (D1x, D1y) in Io1 be obtained as follows: 2
3
2
1 xRo2 6 R 7 6 4 yo2 5 ¼ 4 0 0 1
0 1 0
32
3
D1 x xRo1 6 7 D1 y 7 54 yRo1 5 1
ð4Þ
1
897
Therefore, the motion compensated odd field image obtained R in this way is denoted as Io2 ðx; yÞ. Finally, the restored interlaced R scan CCD image I (x,y) is formed as follows: ( R Io2 ðx; yÞ; y ¼ 2k; k ¼ 0; 1; . . . ; N=21 ð5Þ IR ðx; yÞ ¼ R ðx; yÞ; y ¼ 2k þ1; k ¼ 0; 1; . . . ; N=21 Ie1
3. Results and comparisons 3.1. Real image restoration In this Section, a comparative image restoration experiment is performed, using our image restoration scheme and for comparison using another sectional processing [7]. A fast-moving car image sequence is taken and recorded by an interlaced scan CCD camera. One image frame of this image sequence is selected and shown in
Fig. 5. Restored car image using our scheme with 10 blind deconvolution iterations and block size 16 16.
Fig. 7. Restored car image using scheme of Ref. [7].
Fig. 6. Restored car image using our scheme with macro block size 32 32and 5 iterations.
Fig. 8. Restored car image using odd and even field extraction, motion computation and scheme of Ref. [27].
ARTICLE IN PRESS 898
Z. Peng et al. / Optics & Laser Technology 42 (2010) 894–901
Fig. 9. Image restoration comparison for the wheel sub-image: (a) original wheel image; (b) restored wheel image by scheme of Ref. [7]; (c) restored wheel image by scheme of Ref. [27]; (d) restored odd field image by our scheme; (e) restored even field image by our scheme and (f) restored final image by our scheme.
Fig. 10. Computed motion vectors with respect to different 1584 macro blocks: (a) vertical components of the computed motion vectors and (b) horizontal components of the computed motion vectors.
ARTICLE IN PRESS Z. Peng et al. / Optics & Laser Technology 42 (2010) 894–901
Fig. 3 (the image size is 704 576). In Fig. 3, a fast-moving car is moving horizontally in front of the static background wall. This moving car can be approximately divided into two different parts with different motions. The large part with a horizontal linear motion is the car body’s side face except the wheel. The small part is the moving wheel with a complicated motion, which combines a horizontal linear motion with a rotation motion. First, our scheme is used to restore this degraded image with space-variant motion blurs. The DS block matching algorithm is used here in the motion vector estimation for each macro block. The size of macro block is 16 16 and the search parameter p = 20. In our scheme, the size of macro block should be small enough to ensure that each macro block in the wheel part can be restored approximately with a space-invariant linear motion blur model. In fact, it is assumed that in each macro block in the wheel part, the combined motion can approximately be seen as a spaceinvariant linear motion, if the size of each macro block is small enough. Moreover, it can also be assumed that the curve motion between the matching blocks in the wheel part can approximately be seen as a linear motion. Although these approximations will result in PSF computation error in every macro block, as described in the third step of our scheme, the PSF computed using the above-mentioned linear motion approximations is accurate enough to be set as an initial PSF in the blind deconvolution restoration for every macro block. Again, our scheme not only accelerates restoration speed but also improves the restoration result. The restored final image is illustrated in Fig. 4 with only 5 blind deconvolution iterations. The mean time requirement for
899
our image restoration scheme is 36.7 s, while the mean time requirements for the first step, second step and third step are 1.0, 2.7 and 33.0 s, respectively. All image restorations are performed using the Matlab 6.5 on an AMD Sempron 2400 +1.66 GHz computer with an internal memory of 512 M and a Windows XP system. Second, to discover the influences of the macro block size and the blind deconvolution iterations on our image restoration, some comparative image restorations are also performed. One restored final image is illustrated in Fig. 5 with 10 blind deconvolution iterations. The mean time requirement for this image restoration scheme is 63.4s, while the mean time requirements for the first step, second step and third step are 1.0, 2.7 and 59.7 s, respectively. It can be seen that the time requirement for the third step increases dramatically, when compared with that with only 5 blind deconvolution iterations. However, image restoration results in Figs. 4 and 5 are approximately the same, since the PSF computed using the above-mentioned linear motion approximations is accurate enough to be set as an initial PSF in the blind deconvolution restoration in every macro block, which ensures that good image restoration can be achieved with only a few iterations. Another restored final image is illustrated in Fig. 6 with a macro block size of 32 32. The mean time requirement for this image restoration scheme is 13.4 s, while the mean time requirements for the first step, second step and third step are 1.0, 1.5 and 10.9 s, respectively. It can be seen that time requirement for the third step decreases greatly, when compared with that with the macro block size of 16 16. The image restoration result
Fig. 11. Synthetic space-variant blurred image: (a) original image 1; (b) original image 2; (c) synthetic blurred image 1; (d) synthetic blurred image 2 and (e) final synthetic interlaced scan blurred image.
ARTICLE IN PRESS 900
Z. Peng et al. / Optics & Laser Technology 42 (2010) 894–901
in Fig. 4 is better than that in Fig. 6. In fact, there are usually some blocking artifacts in the image restoration caused by sectional processing. These artifacts can be reduced by the overlap of sections or by decreasing the block size, at the cost of more time requirement for image restoration. Third, the image restoration schemes described in Refs. [7,27] are also performed here for the restoration performance comparison. A modified Landweber iteration [7] is used in every sectional region (i.e. one macro block of size 16 16). In this scheme, the fast-moving car image sequence is used in the motion vector estimation in every macro block, while in our novel scheme only one image frame is required for motion vector estimation. The final restored image is shown in Fig. 7, and the mean time requirement for this scheme is 55.7s. We can see that the image restoration in Fig. 4 outperforms that in Fig. 7, because the restoration scheme in Ref. [7] does not compensate the motion offset between the odd and even field images. In order to compare with the scheme in Ref. [27], we first extract the odd and even field images, followed by the DS motion estimation algorithm. We
then apply the probabilistic-framework-based deblur [27] to remove motion blurs. The final restored image is shown in Fig. 8, and the mean time requirement for this scheme is approximately 4 min. We can see that the image restoration in Fig. 4 also outperforms that in Fig. 8. Moreover, one restoration comparison in the image region of the car’s wheel is illustrated in Fig. 9, which also indicates that our scheme outperforms those image restoration schemes in Refs. [7,27]. Finally, to find how stable the DS algorithm is on patches with uniform colors or textured structures, the vertical components and the horizontal components of the computed motion vectors in Fig. 3 are illustrated in Fig. 10, with respect to different macro blocks. It can be seen that some motion computation errors do occur at those patches, which may propagate to the PSF computation and the image deconvolution as well. In our future work, we will propose some improvement solutions to the DS algorithm (e.g. we may impose constraints on the block matching, apply a multi-scale approach in a coarse-to-fine manner, or manually select patches with rich edge structures).
Fig. 12. Synthetic image restoration comparison: (a) image restoration by scheme of Ref. [7]; (b) image restoration by scheme of Ref. [27] and (c) image restoration by our scheme.
ARTICLE IN PRESS Z. Peng et al. / Optics & Laser Technology 42 (2010) 894–901
Acknowledgement
Table 1 Image restoration comparisons for different schemes.
Ref. [7] scheme Ref. [27] scheme Our scheme
901
ABSDIFF
SNR
PSNR
IMFID
MSE
0.038677 0.034795 0.027160
19.160 20.522 22.828
22.397 23.758 26.064
0.012105 0.008839 0.005187
0.0042865 0.0031329 0.0018423
This research is supported by the National Fundamental Research Plan Project (973 Project) with grant No. 2009CB72400603. It is also supported by the Harbin Youth Originality Science Foundation with grant No. 2008RFQXG005 and the National Natural Science Foundation with grant No. 60972100. References
3.2. Synthetic image restoration In order to quantitatively compare our scheme with those schemes in Refs. [7,27], one image restoration comparison for a space-variant blurred synthetic image is also performed, although the synthetic motion blurs are not absolutely identical to motion blurs in real life. One sub-image sequence of the ‘‘caltrain’’ sequence is used to produce a synthetic space-variant blurred image sequence. For example, one synthetic blurred image is illustrated in Fig. 11, which consists of two parts with different motion blurs. One part is the left part from column 1 to column 220 with a horizontal motion blur. The other part is the right part from column 221 to column 512 with a vertical motion blur. This synthetic blurred image is produced by using the following steps. First, two images are selected from the above image sequence. The left parts of these two images have a horizontal motion, while the right parts have a vertical motion. Therefore, the horizontal and the vertical motion blurs are produced in the left part and the right part, respectively, using the standard Matlab filter functions. Second, these two synthetic blurred images are used as the odd and even field images to synthesize one interlaced scan blurred image. As for the motion computation, the DS algorithm is again used in this synthetic image. In fact, as pointed out in Ref. [19], the DS algorithm has been implemented in MPEG-4 video-encoding environment and its efficacy has been demonstrated in the core experimental results [28]. Based on these results, the DS algorithm is adopted and incorporated into the MPEG-4 verification model [29]. The final image restoration comparison is illustrated in Fig. 12, and the quantitative comparison for different schemes is illustrated in Table 1. In Table 1, the ABSDIFF (Average Absolute Difference), the SNR (Signal to Noise Ratio), the PSNR (Peak Signal to Noise Ratio), the IMFID (Image Fidelity), and the MSE (Mean Square Error) are used as the comparison standards for evaluating image restoration performance. Since, in general, a better restored image will have the large SNR, PSNR and IMFID values with small ABSDIFF and MSE values, we can see that our scheme outperforms again those image restoration schemes in Refs. [7,27].
4. Conclusions In this paper, a novel image restoration scheme for interlaced scan CCD image with space-variant motion blurs is proposed. In this scheme, only one image frame is used to perform the motion vector estimation in every macro block. The PSF computed with the motion vector estimation is set as an initial PSF in the blind deconvolution restoration, which not only accelerates the restoration speed but also improves the restoration result. This restoration scheme can be used in the practical interlaced scan CCD image restoration work.
[1] Sondhi M. Image restoration: the removal of spatially invariant degradations. Proceedings of IEEE 1972;60(7):842–53. [2] Slepian D. Restoration of photographs blurred by image motion. Bell System Technical Journal 1967;46(10):2353–62. [3] Cannon M. Blind deconvolution of spatially invariant image blurs with phase. IEEE Transactions on ASSP 1976;24(1):58–63. [4] Yitzhaky Y, Mor I, Lantzman A, et al. Direct method for restoration of motion blurred images. Journal of OSA 1998;15(6):1512–9. [5] Sawchuk A. Space-variant image restoration by coordinate transformation. Journal of OSA 1974;60(2):138–44. [6] Trussell HJ, Hunt BR. Image restoration of space-variant blurs by sectional methods. IEEE Transactions on ASSP 1978;26(10):608–9. [7] Trussell HJ, Fogel S. Identification and restoration of spatially variant motion blurs in sequential images. IEEE Transactions on Image Processing 1992;1(1):123–6. [8] Kang S, Choung Y, Paik J. Segmentation-based image restoration for multiple moving objects with different motions. Proceedings of IEEE ICIP 1999;1: 376–80. [9] Tull DT, Katsaggelos AK. Iterative restoration of fast-moving objects in dynamic image sequences. Optical Engineering 1996;35(12):3460–9. [10] Tull DT, Katsaggelos AK. Regularized restoration of partial-response distortions in sporadically degraded images. Proceedings of IEEE ICIP 1998;3: 732–6. [11] Tekalp AM, Kaufman H, Woods JW. Model-based segmentation and spacevariant restoration of blurred images by decision-directed filtering. Signal Processing 1988:259–69. [12] Patti AJ, Ozkan MK, Tekalp AM, et al. New approaches for space-variant image restoration. Proceedings of IEEE ICASSP 1993;2:261–4. [13] Ozkan MK, Tekalp AM, Sezan MI. POCS-Based restoration of space-varying blurred images. IEEE Transactions on Image Processing 1994;3(4):450–4. [14] Rajagopalan AN, Chaudhuri S. A MRF-model based approach to simultaneous recovery of depth and restoration from defocused images. IEEE Transactions on PAMI 1999;21(7):577–89. [15] Sorel M, Flusser J. Space-variant restorations of images degraded by camera motion blur. IEEE Transactions on Image Processing 2008;17(2):105–16. [16] Xiang LR, Bing Z, Ming LL. A new three-step search algorithm for block motion estimation. IEEE Transactions on Circuits and Systems for Video Technology 1994;4(4):438–42. [17] Hua LJ, Ming LL. A simple and efficient search algorithm for block matching motion estimation. IEEE Transactions on Circuits and Systems for Video Technology 1997;7(2):429–33. [18] Man PL, Chung MW. A novel four-step search algorithm for fast block motion estimation. IEEE Transactions on Circuits and Systems for Video Technology 1996;6(3):313–7. [19] Shan Z, Kuang MK. A new diamond search algorithm for fast block matching motion estimation. IEEE Transactions on Image Processing 2000;9(2): 287–90. [20] Yao N, Kuang MK. Adaptive rood pattern search for fast block matching motion estimation. IEEE Transactions on Image Processing 2002;11(12) 1442–8. [21] Banham MR, Katsaggleos AK. Digital image restoration. IEEE Signal Process Magazine 1997;24(1):24–41. [22] Moghaddam ME, Jamzad M. Motion blur identification in noisy images using mathematical models and statistical measures. Pattern Recognition 2007;40(11):1946–57. [23] Kundur D, Hatzinakos D. Blind image deconvolution. IEEE Signal Process Magazine 1996;23(1):43–64. [24] Kim SK, Paik JK. Out-of-focus blur estimation and restoration for digital autofocusing system. Electronics Letters 1998;34(12):1217–9. [25] Likas A, Galatsanos N. A variational approach for Bayesian blind image deconvolution. IEEE Transactions on Signal Processing 2004;52(8):2222–33. [26] Fergus R, Singh B, Hertzmann A, et al. Removing camera shake from a single photograph. ACM Transactions on Graphics 2006;25:787–94. [27] Shan Q, Jia J, Agarwala A. High-quality motion deblurring from a single image. ACM Transactions on Graphics 2008;27(3):73:1–10. [28] Kuang MK, Hosur PI. Core experimental results of fast motion estimation based on new test conditions (Q4a). ISO/IECJTC1/SC29/WG11/M4934, July 1999. [29] MPEG-4 video verification model, version. 14.0, ISO/IECJTC1/SC29/WG11/ N2932, October 1999.