- Email: [email protected]
Adaptive non-local means filter for image deblocking
Signal Processing: Image Communication 28 (2013) 522–530
Contents lists available at SciVerse ScienceDirect
Signal Processing: Image Communication journal homepage: www.elsevier.com/locate/image
Adaptive non-local means filter for image deblocking$ Ci Wang a,b,n, Jun Zhou a,b, Shu Liu a,b a b
SEIEE Building 1-316, Dongchuan Road 800, Shanghai 200240, China Department of Electrical Engineering, School of Electronic, Information and Electrical Engineering, Shanghai Jiaotong University, China
a r t i c l e in f o
abstract
Article history: Received 13 January 2012 Accepted 25 January 2013 Available online 7 February 2013
Blocking artifacts often exist in the images compressed by standards, such as JPEG and MPEG, which causes serious image degradation. Many algorithms have been proposed in the last decade to alleviate this degradation by reducing the quantization noise. Unfortunately, these algorithms only produce satisfying results under an unreasonable assumption that noise magnitude has been given. However, in most applications, the user only gets inferior image copy, without any side information about noise distribution, therefore the efficiency of existing denoise algorithms is significantly reduced. In this paper, a new metric is first given to evaluate the blocking artifacts; and then non-local means filter is applied to remove quantization noise on the blocks. During the process, nonlocal means filters with different variances are used to do deblocking, and their efficiencies are recorded as the references. The deblocked image is finally the one combined with all blocks filtered with the optimal parameters. We prove with experimental results that the proposed algorithm constantly outperforms the peer ones on all kinds of images. Published by Elsevier B.V.
Keywords: Adaptive nonlocal means filter Deblocking Subjective image quality
1. Introduction To get higher transmission and storage efficiency, image and video are often compressed by JPEG, MPEG-x and H.26x, etc. Most of them adopt each block as a processing unit for simplifying software/hardware architecture. By compression, we lose detailed information within each block and produce visual discontinuity around their boundaries. Blocking artifact is the most annoying distortion since it destroys the correlation between adjacent pixels or blocks. To reduce blocking artifact, some postprocessing methods have been proposed in the last decades, which are classified into two categories: iterative and non-iterative ones. Based
$ The work was supported by the National Science Foundation 60902072 and New Teachers’ Fund for Doctor Stations, Ministry of Education 20090073120030. n Corresponding author at: SEIEE Building 1-316, Dongchuan Road 800, Shanghai 200240, China. Tel./fax: þ86 2134204155. E-mail addresses: [email protected] (C. Wang), [email protected] (J. Zhou), [email protected] (S. Liu).
0923-5965/$ - see front matter Published by Elsevier B.V. http://dx.doi.org/10.1016/j.image.2013.01.006
on theory of projection onto convex sets (POCS), the iterative one is to project the compressed image onto quantization constraint set and some prior image knowledge constraint sets [1,2]. Given their high computational cost, methods with iteration are difficult to be implemented in real time. On the other hand, non-iterative methods are designed on the exclusive characteristic of the image or noise in spatial or transform domain. In general, transform domain algorithms are executed much faster than that on spatial domain, but spatial domain algorithms perform better because they can directly alter intensity of each pixel within images. In spatial domain, the simplest deblocking way is to apply 3 3 Gaussian filter along the block boundaries [3], which results in blurring around true edges. Minami and Zakhor [4] reduce blocking artifacts under the criterion of the minimum mean squared difference of slope (MSDS) without considering about the information loss inside blocks. Sun and Cham [5] treat the blocking artifacts as additive Gaussian noise, and use this noise assumption as prior to remove noise. Under the same assumption, Averbuch et al. [6] use the weighted sum on
C. Wang et al. / Signal Processing: Image Communication 28 (2013) 522–530
the pixel quartet, symmetrically around block boundary, for deblocking. However, quantization noise in spatial domain is neither even nor exactly follows Gaussian distribution, and this phenomenon is especially obvious on heavily compressed images; hence their deblocking performance are not guaranteed. Besides these, Vo et al. [7] design an adaptive fuzzy filter to fast remove blocking and ringing artifacts with consideration of edge and texture direction. As for transform domain processing, Kim [8] adopts direction filter, referred to directional activity of block, to reduce blocking artifacts. Wu et al. [9] use soft threshold for deblocking on the wavelet domain, and Xu et al. [10] classify image into different kinds of regions through calculating the block activities in discrete hadamard transform (DHT) domain and process them respectively. These methods do not suit for postprocessing as the fact that extra computation is required for transforming image from spatial domain into a specific domain. Recently, new branch-training based deblocking algorithm has been reported, which training and matching are done in any given domain. For example, Freeman et al. [11] propose a learning based approach to distill the detail information from prior database lost during compression. However, this kind of algorithm cannot work effectively if the input image is significantly different from the trained data. Above methods use different deblocking strategies on various areas to keep edges from over-blurring, which are more complex than the ubiquitous algorithm, such as nonlocal means filter. Nonlocal means filter is a nonlinear, edge preserving smoothing filter, and it can smooth blocking artifact and preserve the edge details simultaneously. It especially suits for ASIC (Application Specific Integrated Circuit) application, where logical resource is expensive. However, its control parameter selection is still an unsolved question [12]. In [13], Nath et al. empirically calculate the filter parameter from image quantization parameter (QP) to improve deblocking performance. However, quantization noise distribution is related with quantization parameter as well as image content, so the results of this method are suboptimal. Recently, Chatterjee and Milanfar [14] propose a patch-based Wiener filter that exploits patch redundancy for image denoising. Their method outperforms peer ones at the cost of extremely enormous amount of computation. Hereinafter, we must introduce a new scalar to precisely measure the visibility of the blocking artifacts, which is then used to adjust the filter parameter. As for visual quality assessment, Wu et al. [15] propose one general block-edge impairment metric (GBIM), which is mostly consistent with human subjective evaluation. Recently, Yim and Bovik [16] propose a block-sensitive index, named PSNR-B, to produce objective quality judgment according to observations. These methods require original image as the reference, and with this requirement, they are too harsh to be used. In this paper, we propose a metric to measure the blocking artifacts, which is a cost function of two factors: MSDS and high frequency energy within the block. Compared with the others, this metric is simple and easy to be implemented for it does not require any accessorial codec information, such as quantization parameter. We examine deblocking
523
efficiency through checking this metric change before and after deblocking, and then select the optimal filter parameter for deblocking. Therefore, the proposed algorithm especially suits the applications, where only the distorted image is available, for example, as the postprocessor connected to decoder. Section 2 reviews the traditional nonlocal means filter we will use for deblocking, and gives example to illustrate its shortcoming. In Section 3, we introduce metrics to measure visual block impairment and deblocking efficiency. Section 4 gives the details about how to select control parameters for nonlocal means filter. In Section 5, we present the simulation results of various deblocking algorithms from objective and subjective views. Finally, we give conclusion in Section 6.
2. Nonlocal means filter and its application for deblocking Nonlocal means filter is first proposed by Buades in [17], which is a non-linear, edge preserving smoothing filter. The filter can be applied to the image block and modify its pixels as the weighted sum of its neighborhood pixels, whose weighted parameter is determined by the similarity of image block neighborhoods. Let f ðiÞ be the i-th observed pixel of the decoded image f , and Ni be any given size square area centered at the i-th pixel. If S is the square search window, nonlocal means filter searches the block similarity within S. Then, the i-th filtered pixel is given 1 JvððNi ÞvðNj Þ2 J=h2 Þ f~ ¼ ðiÞ ¼ Sj2S e f ðjÞ ZðiÞ
ð1Þ
where vðN i Þ is the vectorization of N i , ZðiÞ ¼ 2 2 Sj2S eJðvðNi ÞvðNj Þ J=h Þ is the normalized term, and h is the parameter to control smooth degree. Several parameters of nonlocal means filter can be adjusted to achieve the best deblocking effects. Among them, searching window size S will be the inconsequential one when it is larger than a threshold, such as 5 pixels in width and height. Although increasing size S can marginally enhance denoise performance, it also incurs the unwanted computational costs, so S selection actually rests with the computational capability of the specific applications. On the contrary, h selection has great impact on denoise performance, which is the pivot of this paper. To demonstrate above statements, ‘‘Lena’’ is first compressed with quality score (QS) 10 to produce the distorted image with blocking artifacts. Then, we apply nonlocal means filters with different parameters on it, such as h¼30, 60, 90, 120, 150, and 180, to produce the deblocked copies. It is shown that blocking artifacts cannot be efficiently suppressed when h¼30. Whereas, the detailed information will be heavily lost if it is deblocked by a smooth filter with h¼180, which is especially obvious on the Lena’s hat. Even if moderate h is adopted, for example h¼90, deblocking still cannot bring about approving results on whole image. For instance, blurring still exists on the plume and blocking artifacts also appear at the ycanthus, as shown in Fig. 1(b).
524
C. Wang et al. / Signal Processing: Image Communication 28 (2013) 522–530
Fig. 1. Deblocked images with different hs. (a) h ¼30, (b) h ¼90, and (c) h ¼180.
Fig. 2. Deblocking performances vs. filter parameters. All images are compressed with same QS.
We also conduct similar experiments on other images, such as Barbara and Baboon. These images are compressed with the same QS¼10, i.e. the same QP. After which, they are deblocked by the filters with various parameter hs, as marked in Fig. 2. It is shown that the curves of PSNR vs. h are also different, and the optimal hs of these images are hLena ¼ 80, hBabara ¼ 150, hBaboon ¼ 100 respectively. It indicates that h selection not only relies on quantization parameter (QP), but also is affected by image content. Best performance is got only if deblocking parameters are flexibly adjusted according with image contents. 3. Deblocking evaluation Blocking often occurs in the area where the smooth change of luminance goes across a block border and the neighboring pixels around this border fall into different quantization intervals. Blocking artifacts are heavier if DCT coefficients are quantized by a large scalar. We can compare current block with its 8 neighboring blocks in spatial or DCT domain to detect its blocking artifacts. In [18], Queiroz assume that DCT coefficients follow the Laplacian probability distribution, and blocking artifacts is then detected in the DCT subbands. Besides these, blocking artifact is detectable in the spatial domain by calculating the MSDS of the adjacent blocks as in [19]. In this section, we consider a no-reference assessment to
judge blocking artifacts from image itself by calculating MSDS and residual information energy of the compressed block. In [20], Meier et al. simply treat compression noise as independent and identically-distributed (IID) Gaussian random variables. From this property, Mateos et al. [21] use IID Gaussian noise to analyze the block boundary pixels. Because compression process is not considered in deduction, quantization noise is not described precisely in these models. Yang et al. [22] work with a training procedure to determine noise experimentally, and find the quantization noise has large magnitude near block boundaries. In [23], Robertson and Stevenson prove that the quantization error is not spatially invariant and is especially large on pixels near block boundaries and corners. Therefore, we use pixels around block boundary to evaluate noise magnitude. MSDS is defined as the intensity gradient of the pixels close to the boundary of the adjacent blocks. It is a good measure of the impact of quantization noise on block boundary, and is widely used as the criterion for deblocking. Let i and j be the block coordinate in horizontal and vertical directions. Suppose that blocks L and R are the left and right blocks of the block (i,j), and U and D are the upper and down blocks of block (i,j). MSDS of the block (i,j) is defined as MSDSi,j ¼ MSDSL,i,j þ MSDSR,i,j þMSDSU,i,j þMSDSD,i,j
ð2Þ
MSDS of block (i,j) and its left neighbor is given in (2), and MSDS in the other directions can be deduced in the same way. MSDSL,i,j ¼
N1 X k¼0
1 1 ð3xi,j ðk,0Þxi,j ðk,1ÞÞ ð3xi,j1 ðk,N1Þxi,j1 ðk,N2ÞÞ 2 2
2
ð3Þ N is the block size in horizontal or vertical direction. xi,j ðm,nÞ is the pixel value, where (m,n) are the relative coordinate of pixel in block (i,j). Although MSDS is a good metric for blocking artifact evaluation, it is far from enough when it comes to filter parameter selection, the reason of which is that the value of MSDS is not proportional to actual image quality. For example, over-blurring images having smaller MSDS is contradictory to our expectation. If only referred to MSDS, deblocking filter will produce the over-smoothed results. As we have known, the visibility of blocking artifacts is also related with image content. In general,
C. Wang et al. / Signal Processing: Image Communication 28 (2013) 522–530
blocking artifacts on flat area are more annoying than that on the area with much more details. For this reason, we must include a simple term Di,j to reflect image difference, which is given as
Di,j ¼ HðBi,j Þ
ð4Þ
where Bi,j is the block with index (i,j) and H is a high pass filter. In this paper, we use AC energy of the block as its activity measurement. Then, visibility of blocking artifacts T i,j will be a function of MSDSi,j and Di,j . This function cannot be precisely described for human apperceive is ambiguous so far. In this paper, we simplify this relationship as a linear function, and then deblocking efficiency is the function of T i,j change before and after deblocking. D Deblocking produces new MSDSD i,j and image content Bi,j , where the superscript D stands for the one after deblocking. There are two terms to reflect these changes C 1,i,j ¼ MSDSD i,j MSDSi,j
ð5Þ
C 2,i,j ¼ JHðBD i,j Bi,j Þ2 J
ð6Þ
where 99 992 is the l2 norm to calculate high frequency power changes in the block. In general, MSDS will decrease after deblocking, and this decreasing is described by variable C 1,i,j . Deblocking also changes image content within the block, where quantization noise is much smaller than that around block border. To avoid over-smoothness, BD ij should close to Bij , so we introduce C 2,i,j to measure their divergence. Intuitively, optimal filter should significantly reduce C 1,i,j and slightly increase C 2,i,j simultaneously. Then, a criterion is designed to measure deblocking performance as: C i,j ¼ lC 1,i,j þ C 2,i,j
ð7Þ
where l is the Lagrange multiplier. From research results of human visual system, we know that l should vary with image content. The blocking artifacts are more obvious on the flat areas than that on texture areas [24]. Therefore, optimal l of block is set according to its pattern. Either, empirical l is set for all blocks and images through numerous experiments for simplicity. In this paper, we do not pay effort on l selection, but use a constant l ¼ 0:2, for it is enough to show deblocking performance improvement. Then, C i, j is the cost function and optimal deblocking result is the solution with minimum C i,j . In monotone area, the second term of C 2,i,j is significantly smaller than the first term lC 1,i,j , so we adopt heavy smooth filter to reduce lC 1,i,j . It can efficiently reduce the blocking artifacts. On the contrary, heavy smoothing enlarges C 2,i,j in the texture area, which may be faster that lC 1,i,j decreasing. To get minimum C i,j , we prefer light smooth filter in these area to avoid over-smoothness. Therefore, Eq. (7) actually represents the tradeoff between grid noise suppression and image detail preservation.
525
means filter with various hs for deblocking. From many deblocking experiments, we find that the optimal hs are distributed within a short range under certain compressed ratio, even if the image contents are greatly varied. Fig. 2 proves this phenomenon in which hoptimal ¼ 70 for flat image Lena and hoptimal ¼ 170 for the complicated image Baboon. It provides us the possibility to go through less hs to select optimal parameter. We first get the rough h estimate as hinitial ¼
1
ð8Þ
ðQS=100Þ2
where 100 is the upper range value of QS. By (8), large hinitial is used for highly compressed image, i.e. with small QS, and small hinitial is used for slightly compressed image, i.e. with large QS. After getting the rough h estimate, we search the optimal h within small range ð0:5hinitial , 1:5hinitial Þ. By the proposed method, we do deblocking with different hs to produce a series of deblocked image blocks. Then, the inferior block in the compressed images is replaced by these deblocked blocks in turn for MSDSD i,j D D and BD i,j calculation. After MSDSi,j , MSDSi,j , Bi,j and Bi,j collected, we calculate the deblocking efficiency C i,j by (7) for these hs, and find the optimal h, i.e. the one with the minimum C i,j . Finally, distorted block is processed by non local means filter with this optimal h. The proposed algorithm brings in performance improvement over the peer ones, but it has yet to be improved at computational cost for find optimal h. First, we reduce computation of the essential unit, i.e. the cost function of (7). Its computation is consisted of three parts: (1) image deblocking; (2) high frequency energy extraction; (3) MSDS calculation. Let N N be the processing unit and searching window be [ N, þN] in horizontal and vertical directions (Fig. 3). Deblocking consists of operations, such as addition and exponent calculation. There are some fast algorithms to do exponent calculation, such as the look-up table [27] with accuracy of 108 . As for our application, pixels are represented by 8 bits, so that it requires the accuracy only 102 for exponent calculation. Furthermore, we
(i,j+1)
L
U
(i,j)
R
4. Adaptive nonlocal means deblocking Because processing unit of most compression standards are 8 8, we use this block size as the processing unit for nonlocal means filtering. After image decomposition, some non-overlapped blocks with the same size N N is produced. First, we measure blocking degree through calculating their MSDSi,j and HðBi,j Þ, and then use nonlocal
(i-1,j) MSDS
(i+1,j) D
(i,j+1)
Fig. 3. Demonstration of MSDS calculation.
526
C. Wang et al. / Signal Processing: Image Communication 28 (2013) 522–530
pursue a relative scale, rather than the exact exponent value. Consequently, look-up table with 256 vectors is used here and 8 comparisons are enough for exponent calculation, which is significantly lower than 256 additions in [27]. For simplification, exponent calculation is ignored here for it can be done by look-up table and its operation frequency is far less than that of addition. It requires N2 subtractions to calculate the weight of each pixel in searching window, so filtering a block need 4N4 additions. As for high frequency energy extraction, we 0 1 0 apply high pass filter 1 4 1 on each pixels, and 0 1 0 then it require 5N 2 additions for each block processing. The computational cost of MSDS is negligible, which is 6N additions for each block. From above analysis, we get that the dominant bottleneck of computation complexity is deblocking. Fortunately, a majority of the intermediate results of deblocking with certain h can be reused by the others, so the computational cost of h optimization will not linearly increase with hs. For different h, vðNi ÞvðNj Þ of (1) are not re-calculated, and the weight 2
quality score (QS), ranging from 1 to 100, to adjust quantization parameter and compression ratio. Usually, there is no obvious blocking artifacts when QS 430. In this paper, we emphasize on the case where QSo30. We use all kinds of images compressed with different QSs as the test data, and examine deblocking efficiency on these data from the subjective and objective quality evaluation. We use PSNR as the objective score to present image quality. In the first part of our experiment, four 512 512 images, including Lena, Pepper, Barbara and Jet are used as the raw image. Among them, Lena and Pepper are the flat images; Barbara is with abundant texture. Compared with them, Jet is a neutral one. These images are compressed with QS1¼30, QS2¼25 and QS3¼20 to produce blocking images. There are slight blocking artifacts on the compressed
Table 1 PSNR comparison of different deblocking methods. Image
Quality Method Liew and Hong [25]
Zhai Chatterjee and Proposed et al. [26] Milafar [14]
Barbara QS1 QS2 QS3
29.72 29.02 28.15
30.25 29.42 28.46
31.22 30.59 29.58
31.43 30.53 29.39
Lena
QS1 QS2 QS3
34.51 34.04 33.42
34.57 34.08 33.50
32.61 32.43 32.15
34.96 34.46 33.79
Pepper
QS1 QS2 QS3
33.15 32.89 32.51
33.17 32.87 32.48
35.16 34.85 34.41
33.33 33.03 32.60
Jet
QS1 QS2 QS3
34.93 34.34 33.61
34.86 34.22 33.45
33.74 33.52 33.18
35.25 34.58 33.76
2
of neighbor pixel e:vðNi ÞvðNj Þ :=h is easily calculated through inquiring predefined table with two input vari2
ables :vðNi ÞvðNj Þ: and h. By this arrangement, only 4N additions are required for filtering block with other hs. If N ¼ 8, the computational cost of state-of-art nonlocal means filter are 16,384 times of addition. By the proposed algorithm, it requires 16,752 times of addition to measure deblocking performance of the first h, and 624 times of addition for the other hs. Note that optimal deblocking results have been produced when we find the optimal h. Therefore, there are about 3:5 m% computational increase over standard nonlocal means filter, where m is the number of h to be examined. Besides simplifying the computation of cost function (7), we further consider reducing the number of the candidate hs to be tested. Because MSDS is gradually reduced with h increasing, C 1,i,j in (7) is a decreasing function. On the contrary, high frequency information will be lost more if larger h is used, so C 2,i,j is an increasing function of hs. Therefore, their combination C i,j is a convex function. With this convexity, we develop fast algorithm to find h with minimum C i,j . In the range [0.5hinitial , 1.5hinitial ], we evenly sample 2 hs, i.e. h1¼ 0.83hinitial and h2¼1.16hinitial , and then calculate their costs by (7). If C i,j ðh1Þ o C i,j ðh2Þ, the optimal h is further searched within [0.5hinitial , 1.16hinitial ] through resampling and range decomposition. We use QS¼10 as example, which corresponds to large potential h range. To get the optimal h within 0.1hinitial precision, we need 3 range decomposition and 6 tests with different h, so that the computational cost of the proposed algorithm is only about 20% higher than that of the nonlocal means filtering.
Table 2 VIF comparison of different deblocking methods. Image
Liew and Hong [25]
Zhai Chatterjee and Proposed et al. [26] Milafar [14]
Barbara QS1 QS2 QS3
0.48 0.43 0.37
0.52 0.46 0.39
0.52 0.50 0.47
0.52 0.47 0.40
Lena
QS1 QS2 QS3
0.49 0.44 0.39
0.51 0.46 0.40
0.40 0.40 0.37
0.51 0.46 0.40
Pepper
QS1 QS2 QS3
0.47 0.42 0.37
0.48 0.43 0.38
0.49 0.47 0.45
0.48 0.43 0.38
Jet
QS1 QS2 QS3
0.49 0.45 0.39
0.51 0.46 0.40
0.47 0.46 0.43
0.51 0.46 0.40
5. Simulation results In the following experiments, we first compress the raw images with JPEG standard to form images with different degree blocking artifacts. JPEG algorithm uses
Quality Method
C. Wang et al. / Signal Processing: Image Communication 28 (2013) 522–530
image with QS1¼ 30 and heavy blocking artifacts on the compressed one with QS3¼20. Searching window size of the nonlocal means filter is a 7 7 square area. PSNR comparison of different deblocking methods [14,25,26] is given in Table 1. In [25], Liew et al. analyze the image and the quantization noise in wavelet domain, and then suppress noise through changing wavelet coefficients. Zhai et al. use shape-adaptive filter to eliminate quantization noise, and utilize coefficient regularization and quantization constraint to confine deblocking solution within rational range. Recently, Chatterjee and Milafar [14]
527
propose a patch-based Wiener filter that exploits patch redundancy for image denoising. We select the first two algorithms as the reference for they are, adaptive ones and their codes are available on web, and the last one for it is up-to-date and with the highest performance. It is shown that the proposed algorithm is consistently better than the two methods (i.e. [25,26]) on all test images compressed with different QSs. On the images, which all kinds of textures, such as Barbara, the proposed algorithm gets larger PSNR improvement, i.e. about 1.2 dB, than the other two methods. It is due to the fact that our algorithm can keep
Fig. 4. Zoomed parts of Barbara. (a) Original image, (b) processed by Liew, (c) processed by Zhai, (c) processed by Milanfar, and (e) processed by the proposed method.
528
C. Wang et al. / Signal Processing: Image Communication 28 (2013) 522–530
the detailed information, such as stripe of Barbara trousers, during deblocking. Even on flat images, such as Pepper and Lena, the proposed images can still gets some PSNR improvement, i.e. about 0.2–0.3 dB. From Table 1, we find the proposed algorithm is generally better and a robust one for it gets approximate PSNR improvement even at different compression ratios. To compare with more sophisticate algorithm, we also use [14] as the reference. Although this algorithm is flexibly adjusted with image content, it still requires noise amplitude as known parameter, which is too eager. Instead, we adopt noise amplitude E ¼ 15 here, for it statistically gets the best results on all test images. Table 1
shows that the proposed algorithm has comparable performance as [14] in that the number of getting the best PSNR, but the former has 0.3 dB PSNR improvement over the latter on average. We also show validity of the proposed algorithm on the subjective view, and use visual information fidelity (VIF) as the evaluation score. VIF shares interesting similarities with human visual system (HVS) based quality assessment methods. HVS includes channel decomposition and exponential response. The distorted image information is described to a divisive-normalization based masking model and the numerator of VIF is a HVS based
Fig. 5. Zoomed parts of Lena. (a) Original image, (b) processed by Liew, (c) processed by Zhai, (c) processed by Milanfar, and (e) processed by the proposed method.
C. Wang et al. / Signal Processing: Image Communication 28 (2013) 522–530
perceptual distortion metric, so that VIF is given as VIF ¼
Distorted Image Information Reference Image Information
ð8Þ
Until now, VIF has the highest correlation with human visual judgment, compared with peering algorithms. In this paper, we use it as the criterion to subjectively measure deblocking performance. In Table 2, we list VIF values of four algorithms, where QS1¼30, QS2¼25 and QS3¼20. It is shown that the proposed and Milanfar algorithms get the best VIFs in most cases. Our algorithm is better in the case where blocking artifacts are heavy or QS is small, which is just the case required to do deblocking. On the contrary, Milanfar algorithm outperforms ours on the images with slightly compressed, because quantization noise in slightly compressed image is closer to Gaussian distribution. Milanfar algorithm finds similar blocks within whole image, and uses them to estimate filter parameters, so that more information redundancy can be used to suppress Gaussian noise. Its performance will be reduced if patch similarity has been destroyed by heavy compression. In Fig. 4, we compare subjective quality of the zoomed image part Barbara before and after deblocking, which is compressed with QS ¼20. It is shown that our resultant image is of high perceptual quality as the proposed algorithm smoothes most grid noise and suppresses the edge-related noise. Liew algorithm and Zhai algorithm can repair these disfigurements as well, but they still produce discontinuousness on the neckcloth stripe. Blocking artifacts still exist on the neckcloth, as marked in Fig. 4(b). Zhai algorithm excels in removing quantization noise on flat and texture areas, but it is weak at edgerelated noise elimination, for example, the areas marked in left-down and right-up corners. Compared with the others, Milanfar is the best one in removing noise, which is especially obvious on the right-down corner, i.e. around Babara’s knee. Unfortunately, its result is excessively processed, and parts of texture, is removed during deblocking. In this algorithm, stripe on neckcloth is unnatural, and that on cloth is lost, for its imperfect threshold selection (Fig. 4). We also conduct experiments on Lena with heavier compression, and their results are presented in Fig. 5. It is also shown that Liew and Zhai algorithms cannot produce the satisfied results on Lena’s mouth and nose, and remain slight blocking artifacts on Lena’s face. Meanwhile, Minanfar algorithm is better in these areas, but it completely removes hair’s details. Compared with them, the proposed algorithm takes their strong points and gets some improvements in these parts. The above experimental results demonstrate that the proposed deblocking measure has competitive performance with [14] and better than [25,26] on both objective and subjective aspects without any requirement of side information of the distortion, such as the quantization step size. Among these algorithms, Liew and Zhai algorithms are completely adaptive, and their speed is also faster. To deblock 512 512 image, they only require 20s on Matlab platform. After optimization, the proposed algorithm is slightly slower than Zhai algorithm, and
529
takes 24s to do the same work on the same platform. Moreover, Milanfar algorithm is half-adaptive, and its execution is the slowest. Even with C language optimization, it still requires 120s to do deblocking. Therefore, the proposed algorithm is promising one for it balances performance and execution speed. 6. Conclusion In this paper, we first introduce a no-reference metric to evaluate deblocking efficiency, which involves two factors, i.e. the smoothness of the block boundaries and image content fidelity. Block smoothness is measured by mean squared difference of slope, and image content fidelity is simply described as the high frequency degradation of the filtered blocks. Using this metric, we design a patch-adaptive non-local means filter to optimize the parameter selection for deblocking under unsupervised way. In the proposed method, patch and searching window sizes are fixed for simplifying implementation architecture, and filter parameter optimization is done through traversal algorithm. This optimization can be more flexibly implemented by other methods, such as linear regression, which has been carried out recently. Simulation results demonstrate that the proposed metric depicts deblocking performance well. Benefit from the metric, our deblocking algorithm outperforms the other three stateof-the-art methods from the aspects of objective and subjective quality as well as execution speed. References [1] S.K. Lee, Edge statistics-based image scale ratio and noise strength estimation in DCT-coded images, IEEE Transactions on Consumer Electronics 55 (4) (2009) 2139–2144. [2] I.H. Jang, N.C. Kim, H.J So, Iterative blocking artifact reduction using a minimum mean square error filters in wavelet domain, Signal Processing 83 (12) (2003) 2607–2619. [3] H.C. Reeve, J.S. Lim, Reduction of blocking artifacts in image coding, Optical Engineering 23 (1) (1984) 34–37. [4] S. Minami, A. Zakhor, An optimization approach for removing blocking artifacts in transform coding, IEEE Transactions on Circuit and Systems for Video Technology 5 (2) (1995) 74–82. [5] D. Sun, W.K. Cham, Postprocessing of low bitrate block DCT coded images based on a fields of experts prior, IEEE Transactions on Image Processing 16 (11) (2007) 2743–2751. [6] A.Z. Averbuch, A. Schclar, D.L. Donoho, Deblocking of block transform compressed images using weighted sums of symmetrically aligned pixels, IEEE Transactions on Image Processing 14 (2) (2005) 200–212. [7] D.T. Vo, T.Q. Nguyen, Yea Sehoon, A. Vetro, Adaptive fuzzy filtering for artifact reduction in compressed images and videos, IEEE Transactions on Image Processing 18 (6) (2009) 1166–1178. [8] I. Kim, Adaptive blocking artifact reduction using wavelet-based block analysis, IEEE Transactions on Consumer Electronics 55 (2) (2009) 933–940. [9] S. Wu, H. Yan, Z. Tan, An efficient wavelet-based deblocking algorithm for highly compressed images, IEEE Transactions on Circuits and Systems for Video Technology 11 (11) (2001) 1193–1198. [10] J. Xu, S. Zheng, X. Yang, Adaptive video blocking artifact removal in Discrete Hadamard Transform domain, Optical Engineering 45 (8) (2006) 080501. [11] W.T. Freeman, E.C. Pasztor, O.T. Carmichael, Learning low-level vision, International Journal of Computer Vision 40 (1) (2000) 25–47. [12] M. Zhang, B.K. Gunturk, Multiresolution bilateral filtering for image denoising, IEEE Transactions on Image Processing 17 (12) (2008) 2324–2333.
530
C. Wang et al. / Signal Processing: Image Communication 28 (2013) 522–530
[13] V.K. Nath, D. Hazarika, A. Mahanta, Blocking artifacts reduction using adaptive bilateral filtering, 2010 International Conference on Signal Processing and Communications, 2010, pp.1–5. [14] P. Chatterjee, P. Milafar, Patch-based near-optimal image denoising, IEEE Transactions on Image Processing 21 (4) (2012) 1635–1649. [15] H.R. Wu, et al., A generalized block-edge impairment metric for video coding, IEEE Signal Processing Letters 4 (11) (1997). [16] C.H. Yim, A.C. Bovik, Quality assessment of deblocked images, IEEE Transactions on Image Processing 20 (1) (2011) 88–98. [17] A. Buades, B. Coll, J.-M. Morel, A non-local algorithm for image denoising, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2005, pp. 60–65. [18] R.L. de Queiroz, Processing JPEG-compressed images and documents, IEEE Transactions on Image Processing 7 (1998) 1661–1672. [19] S. Minami, A. Zakhor, An optimization approach for removing blocking effects in transform coding, IEEE Transactions on Circuits and Systems for Video Technology 5 (1995) 74–82. [20] T. Meier, K.N. Ngan, G. Crebbin, Reduction of blocking artifacts in image and video coding, IEEE Transactions on Circuits and Systems for Video Technology 9 (3) (1999) 490–500.
[21] J. Mateos, A.K. Katsaggelos, R. Molina., A Bayesian approach for the estimation and transmission of regularization parameters for reducing blocking artifacts, IEEE Transactions on Image Processing 9 (7) (2000) 1200–1215. [22] J. Yang, H. Choi, T. Kim, Noise estimation for blocking artifacts reduction in DCT coded images, IEEE Transactions on Circuits and Systems for Video Technology 10 (7) (2000) 1116–1120. [23] M.A. Robertson, R.L. Stevenson, DCT quantization noise in compressed images, International Conference on Image Processing 1 (2001) 185–188. [24] H.T. Soon, K.K. Pang, K.N. Ngan, Classified perceptual coding with adaptive quantization, IEEE Transactions on Circuits and Systems for Video Technology 6 (1996) 375–388. [25] A.W.C. Liew, Y. Hong, Blocking artifacts suppression in block-coded images using overcomplete wavelet representation, IEEE Transactions on Circuits and Systems for Video Technology 14 (2004) 450–461. [26] G. Zhai, W. Zhang, X. Yang, W. Lin, Y. Xu, Efficient deblocking with coefficient regularization, shap-adaptive filtering, and quantization constraint, IEEE Transactions on Multimedia 10 (5) (2008) 735–745. [27] /http://www.convict.lu/Jeunes/ultimate_stuff/exp_ln_2.htmS.
Contents lists available at SciVerse ScienceDirect
Signal Processing: Image Communication journal homepage: www.elsevier.com/locate/image
Adaptive non-local means filter for image deblocking$ Ci Wang a,b,n, Jun Zhou a,b, Shu Liu a,b a b
SEIEE Building 1-316, Dongchuan Road 800, Shanghai 200240, China Department of Electrical Engineering, School of Electronic, Information and Electrical Engineering, Shanghai Jiaotong University, China
a r t i c l e in f o
abstract
Article history: Received 13 January 2012 Accepted 25 January 2013 Available online 7 February 2013
Blocking artifacts often exist in the images compressed by standards, such as JPEG and MPEG, which causes serious image degradation. Many algorithms have been proposed in the last decade to alleviate this degradation by reducing the quantization noise. Unfortunately, these algorithms only produce satisfying results under an unreasonable assumption that noise magnitude has been given. However, in most applications, the user only gets inferior image copy, without any side information about noise distribution, therefore the efficiency of existing denoise algorithms is significantly reduced. In this paper, a new metric is first given to evaluate the blocking artifacts; and then non-local means filter is applied to remove quantization noise on the blocks. During the process, nonlocal means filters with different variances are used to do deblocking, and their efficiencies are recorded as the references. The deblocked image is finally the one combined with all blocks filtered with the optimal parameters. We prove with experimental results that the proposed algorithm constantly outperforms the peer ones on all kinds of images. Published by Elsevier B.V.
Keywords: Adaptive nonlocal means filter Deblocking Subjective image quality
1. Introduction To get higher transmission and storage efficiency, image and video are often compressed by JPEG, MPEG-x and H.26x, etc. Most of them adopt each block as a processing unit for simplifying software/hardware architecture. By compression, we lose detailed information within each block and produce visual discontinuity around their boundaries. Blocking artifact is the most annoying distortion since it destroys the correlation between adjacent pixels or blocks. To reduce blocking artifact, some postprocessing methods have been proposed in the last decades, which are classified into two categories: iterative and non-iterative ones. Based
$ The work was supported by the National Science Foundation 60902072 and New Teachers’ Fund for Doctor Stations, Ministry of Education 20090073120030. n Corresponding author at: SEIEE Building 1-316, Dongchuan Road 800, Shanghai 200240, China. Tel./fax: þ86 2134204155. E-mail addresses: [email protected] (C. Wang), [email protected] (J. Zhou), [email protected] (S. Liu).
0923-5965/$ - see front matter Published by Elsevier B.V. http://dx.doi.org/10.1016/j.image.2013.01.006
on theory of projection onto convex sets (POCS), the iterative one is to project the compressed image onto quantization constraint set and some prior image knowledge constraint sets [1,2]. Given their high computational cost, methods with iteration are difficult to be implemented in real time. On the other hand, non-iterative methods are designed on the exclusive characteristic of the image or noise in spatial or transform domain. In general, transform domain algorithms are executed much faster than that on spatial domain, but spatial domain algorithms perform better because they can directly alter intensity of each pixel within images. In spatial domain, the simplest deblocking way is to apply 3 3 Gaussian filter along the block boundaries [3], which results in blurring around true edges. Minami and Zakhor [4] reduce blocking artifacts under the criterion of the minimum mean squared difference of slope (MSDS) without considering about the information loss inside blocks. Sun and Cham [5] treat the blocking artifacts as additive Gaussian noise, and use this noise assumption as prior to remove noise. Under the same assumption, Averbuch et al. [6] use the weighted sum on
C. Wang et al. / Signal Processing: Image Communication 28 (2013) 522–530
the pixel quartet, symmetrically around block boundary, for deblocking. However, quantization noise in spatial domain is neither even nor exactly follows Gaussian distribution, and this phenomenon is especially obvious on heavily compressed images; hence their deblocking performance are not guaranteed. Besides these, Vo et al. [7] design an adaptive fuzzy filter to fast remove blocking and ringing artifacts with consideration of edge and texture direction. As for transform domain processing, Kim [8] adopts direction filter, referred to directional activity of block, to reduce blocking artifacts. Wu et al. [9] use soft threshold for deblocking on the wavelet domain, and Xu et al. [10] classify image into different kinds of regions through calculating the block activities in discrete hadamard transform (DHT) domain and process them respectively. These methods do not suit for postprocessing as the fact that extra computation is required for transforming image from spatial domain into a specific domain. Recently, new branch-training based deblocking algorithm has been reported, which training and matching are done in any given domain. For example, Freeman et al. [11] propose a learning based approach to distill the detail information from prior database lost during compression. However, this kind of algorithm cannot work effectively if the input image is significantly different from the trained data. Above methods use different deblocking strategies on various areas to keep edges from over-blurring, which are more complex than the ubiquitous algorithm, such as nonlocal means filter. Nonlocal means filter is a nonlinear, edge preserving smoothing filter, and it can smooth blocking artifact and preserve the edge details simultaneously. It especially suits for ASIC (Application Specific Integrated Circuit) application, where logical resource is expensive. However, its control parameter selection is still an unsolved question [12]. In [13], Nath et al. empirically calculate the filter parameter from image quantization parameter (QP) to improve deblocking performance. However, quantization noise distribution is related with quantization parameter as well as image content, so the results of this method are suboptimal. Recently, Chatterjee and Milanfar [14] propose a patch-based Wiener filter that exploits patch redundancy for image denoising. Their method outperforms peer ones at the cost of extremely enormous amount of computation. Hereinafter, we must introduce a new scalar to precisely measure the visibility of the blocking artifacts, which is then used to adjust the filter parameter. As for visual quality assessment, Wu et al. [15] propose one general block-edge impairment metric (GBIM), which is mostly consistent with human subjective evaluation. Recently, Yim and Bovik [16] propose a block-sensitive index, named PSNR-B, to produce objective quality judgment according to observations. These methods require original image as the reference, and with this requirement, they are too harsh to be used. In this paper, we propose a metric to measure the blocking artifacts, which is a cost function of two factors: MSDS and high frequency energy within the block. Compared with the others, this metric is simple and easy to be implemented for it does not require any accessorial codec information, such as quantization parameter. We examine deblocking
523
efficiency through checking this metric change before and after deblocking, and then select the optimal filter parameter for deblocking. Therefore, the proposed algorithm especially suits the applications, where only the distorted image is available, for example, as the postprocessor connected to decoder. Section 2 reviews the traditional nonlocal means filter we will use for deblocking, and gives example to illustrate its shortcoming. In Section 3, we introduce metrics to measure visual block impairment and deblocking efficiency. Section 4 gives the details about how to select control parameters for nonlocal means filter. In Section 5, we present the simulation results of various deblocking algorithms from objective and subjective views. Finally, we give conclusion in Section 6.
2. Nonlocal means filter and its application for deblocking Nonlocal means filter is first proposed by Buades in [17], which is a non-linear, edge preserving smoothing filter. The filter can be applied to the image block and modify its pixels as the weighted sum of its neighborhood pixels, whose weighted parameter is determined by the similarity of image block neighborhoods. Let f ðiÞ be the i-th observed pixel of the decoded image f , and Ni be any given size square area centered at the i-th pixel. If S is the square search window, nonlocal means filter searches the block similarity within S. Then, the i-th filtered pixel is given 1 JvððNi ÞvðNj Þ2 J=h2 Þ f~ ¼ ðiÞ ¼ Sj2S e f ðjÞ ZðiÞ
ð1Þ
where vðN i Þ is the vectorization of N i , ZðiÞ ¼ 2 2 Sj2S eJðvðNi ÞvðNj Þ J=h Þ is the normalized term, and h is the parameter to control smooth degree. Several parameters of nonlocal means filter can be adjusted to achieve the best deblocking effects. Among them, searching window size S will be the inconsequential one when it is larger than a threshold, such as 5 pixels in width and height. Although increasing size S can marginally enhance denoise performance, it also incurs the unwanted computational costs, so S selection actually rests with the computational capability of the specific applications. On the contrary, h selection has great impact on denoise performance, which is the pivot of this paper. To demonstrate above statements, ‘‘Lena’’ is first compressed with quality score (QS) 10 to produce the distorted image with blocking artifacts. Then, we apply nonlocal means filters with different parameters on it, such as h¼30, 60, 90, 120, 150, and 180, to produce the deblocked copies. It is shown that blocking artifacts cannot be efficiently suppressed when h¼30. Whereas, the detailed information will be heavily lost if it is deblocked by a smooth filter with h¼180, which is especially obvious on the Lena’s hat. Even if moderate h is adopted, for example h¼90, deblocking still cannot bring about approving results on whole image. For instance, blurring still exists on the plume and blocking artifacts also appear at the ycanthus, as shown in Fig. 1(b).
524
C. Wang et al. / Signal Processing: Image Communication 28 (2013) 522–530
Fig. 1. Deblocked images with different hs. (a) h ¼30, (b) h ¼90, and (c) h ¼180.
Fig. 2. Deblocking performances vs. filter parameters. All images are compressed with same QS.
We also conduct similar experiments on other images, such as Barbara and Baboon. These images are compressed with the same QS¼10, i.e. the same QP. After which, they are deblocked by the filters with various parameter hs, as marked in Fig. 2. It is shown that the curves of PSNR vs. h are also different, and the optimal hs of these images are hLena ¼ 80, hBabara ¼ 150, hBaboon ¼ 100 respectively. It indicates that h selection not only relies on quantization parameter (QP), but also is affected by image content. Best performance is got only if deblocking parameters are flexibly adjusted according with image contents. 3. Deblocking evaluation Blocking often occurs in the area where the smooth change of luminance goes across a block border and the neighboring pixels around this border fall into different quantization intervals. Blocking artifacts are heavier if DCT coefficients are quantized by a large scalar. We can compare current block with its 8 neighboring blocks in spatial or DCT domain to detect its blocking artifacts. In [18], Queiroz assume that DCT coefficients follow the Laplacian probability distribution, and blocking artifacts is then detected in the DCT subbands. Besides these, blocking artifact is detectable in the spatial domain by calculating the MSDS of the adjacent blocks as in [19]. In this section, we consider a no-reference assessment to
judge blocking artifacts from image itself by calculating MSDS and residual information energy of the compressed block. In [20], Meier et al. simply treat compression noise as independent and identically-distributed (IID) Gaussian random variables. From this property, Mateos et al. [21] use IID Gaussian noise to analyze the block boundary pixels. Because compression process is not considered in deduction, quantization noise is not described precisely in these models. Yang et al. [22] work with a training procedure to determine noise experimentally, and find the quantization noise has large magnitude near block boundaries. In [23], Robertson and Stevenson prove that the quantization error is not spatially invariant and is especially large on pixels near block boundaries and corners. Therefore, we use pixels around block boundary to evaluate noise magnitude. MSDS is defined as the intensity gradient of the pixels close to the boundary of the adjacent blocks. It is a good measure of the impact of quantization noise on block boundary, and is widely used as the criterion for deblocking. Let i and j be the block coordinate in horizontal and vertical directions. Suppose that blocks L and R are the left and right blocks of the block (i,j), and U and D are the upper and down blocks of block (i,j). MSDS of the block (i,j) is defined as MSDSi,j ¼ MSDSL,i,j þ MSDSR,i,j þMSDSU,i,j þMSDSD,i,j
ð2Þ
MSDS of block (i,j) and its left neighbor is given in (2), and MSDS in the other directions can be deduced in the same way. MSDSL,i,j ¼
N1 X k¼0
1 1 ð3xi,j ðk,0Þxi,j ðk,1ÞÞ ð3xi,j1 ðk,N1Þxi,j1 ðk,N2ÞÞ 2 2
2
ð3Þ N is the block size in horizontal or vertical direction. xi,j ðm,nÞ is the pixel value, where (m,n) are the relative coordinate of pixel in block (i,j). Although MSDS is a good metric for blocking artifact evaluation, it is far from enough when it comes to filter parameter selection, the reason of which is that the value of MSDS is not proportional to actual image quality. For example, over-blurring images having smaller MSDS is contradictory to our expectation. If only referred to MSDS, deblocking filter will produce the over-smoothed results. As we have known, the visibility of blocking artifacts is also related with image content. In general,
C. Wang et al. / Signal Processing: Image Communication 28 (2013) 522–530
blocking artifacts on flat area are more annoying than that on the area with much more details. For this reason, we must include a simple term Di,j to reflect image difference, which is given as
Di,j ¼ HðBi,j Þ
ð4Þ
where Bi,j is the block with index (i,j) and H is a high pass filter. In this paper, we use AC energy of the block as its activity measurement. Then, visibility of blocking artifacts T i,j will be a function of MSDSi,j and Di,j . This function cannot be precisely described for human apperceive is ambiguous so far. In this paper, we simplify this relationship as a linear function, and then deblocking efficiency is the function of T i,j change before and after deblocking. D Deblocking produces new MSDSD i,j and image content Bi,j , where the superscript D stands for the one after deblocking. There are two terms to reflect these changes C 1,i,j ¼ MSDSD i,j MSDSi,j
ð5Þ
C 2,i,j ¼ JHðBD i,j Bi,j Þ2 J
ð6Þ
where 99 992 is the l2 norm to calculate high frequency power changes in the block. In general, MSDS will decrease after deblocking, and this decreasing is described by variable C 1,i,j . Deblocking also changes image content within the block, where quantization noise is much smaller than that around block border. To avoid over-smoothness, BD ij should close to Bij , so we introduce C 2,i,j to measure their divergence. Intuitively, optimal filter should significantly reduce C 1,i,j and slightly increase C 2,i,j simultaneously. Then, a criterion is designed to measure deblocking performance as: C i,j ¼ lC 1,i,j þ C 2,i,j
ð7Þ
where l is the Lagrange multiplier. From research results of human visual system, we know that l should vary with image content. The blocking artifacts are more obvious on the flat areas than that on texture areas [24]. Therefore, optimal l of block is set according to its pattern. Either, empirical l is set for all blocks and images through numerous experiments for simplicity. In this paper, we do not pay effort on l selection, but use a constant l ¼ 0:2, for it is enough to show deblocking performance improvement. Then, C i, j is the cost function and optimal deblocking result is the solution with minimum C i,j . In monotone area, the second term of C 2,i,j is significantly smaller than the first term lC 1,i,j , so we adopt heavy smooth filter to reduce lC 1,i,j . It can efficiently reduce the blocking artifacts. On the contrary, heavy smoothing enlarges C 2,i,j in the texture area, which may be faster that lC 1,i,j decreasing. To get minimum C i,j , we prefer light smooth filter in these area to avoid over-smoothness. Therefore, Eq. (7) actually represents the tradeoff between grid noise suppression and image detail preservation.
525
means filter with various hs for deblocking. From many deblocking experiments, we find that the optimal hs are distributed within a short range under certain compressed ratio, even if the image contents are greatly varied. Fig. 2 proves this phenomenon in which hoptimal ¼ 70 for flat image Lena and hoptimal ¼ 170 for the complicated image Baboon. It provides us the possibility to go through less hs to select optimal parameter. We first get the rough h estimate as hinitial ¼
1
ð8Þ
ðQS=100Þ2
where 100 is the upper range value of QS. By (8), large hinitial is used for highly compressed image, i.e. with small QS, and small hinitial is used for slightly compressed image, i.e. with large QS. After getting the rough h estimate, we search the optimal h within small range ð0:5hinitial , 1:5hinitial Þ. By the proposed method, we do deblocking with different hs to produce a series of deblocked image blocks. Then, the inferior block in the compressed images is replaced by these deblocked blocks in turn for MSDSD i,j D D and BD i,j calculation. After MSDSi,j , MSDSi,j , Bi,j and Bi,j collected, we calculate the deblocking efficiency C i,j by (7) for these hs, and find the optimal h, i.e. the one with the minimum C i,j . Finally, distorted block is processed by non local means filter with this optimal h. The proposed algorithm brings in performance improvement over the peer ones, but it has yet to be improved at computational cost for find optimal h. First, we reduce computation of the essential unit, i.e. the cost function of (7). Its computation is consisted of three parts: (1) image deblocking; (2) high frequency energy extraction; (3) MSDS calculation. Let N N be the processing unit and searching window be [ N, þN] in horizontal and vertical directions (Fig. 3). Deblocking consists of operations, such as addition and exponent calculation. There are some fast algorithms to do exponent calculation, such as the look-up table [27] with accuracy of 108 . As for our application, pixels are represented by 8 bits, so that it requires the accuracy only 102 for exponent calculation. Furthermore, we
(i,j+1)
L
U
(i,j)
R
4. Adaptive nonlocal means deblocking Because processing unit of most compression standards are 8 8, we use this block size as the processing unit for nonlocal means filtering. After image decomposition, some non-overlapped blocks with the same size N N is produced. First, we measure blocking degree through calculating their MSDSi,j and HðBi,j Þ, and then use nonlocal
(i-1,j) MSDS
(i+1,j) D
(i,j+1)
Fig. 3. Demonstration of MSDS calculation.
526
C. Wang et al. / Signal Processing: Image Communication 28 (2013) 522–530
pursue a relative scale, rather than the exact exponent value. Consequently, look-up table with 256 vectors is used here and 8 comparisons are enough for exponent calculation, which is significantly lower than 256 additions in [27]. For simplification, exponent calculation is ignored here for it can be done by look-up table and its operation frequency is far less than that of addition. It requires N2 subtractions to calculate the weight of each pixel in searching window, so filtering a block need 4N4 additions. As for high frequency energy extraction, we 0 1 0 apply high pass filter 1 4 1 on each pixels, and 0 1 0 then it require 5N 2 additions for each block processing. The computational cost of MSDS is negligible, which is 6N additions for each block. From above analysis, we get that the dominant bottleneck of computation complexity is deblocking. Fortunately, a majority of the intermediate results of deblocking with certain h can be reused by the others, so the computational cost of h optimization will not linearly increase with hs. For different h, vðNi ÞvðNj Þ of (1) are not re-calculated, and the weight 2
quality score (QS), ranging from 1 to 100, to adjust quantization parameter and compression ratio. Usually, there is no obvious blocking artifacts when QS 430. In this paper, we emphasize on the case where QSo30. We use all kinds of images compressed with different QSs as the test data, and examine deblocking efficiency on these data from the subjective and objective quality evaluation. We use PSNR as the objective score to present image quality. In the first part of our experiment, four 512 512 images, including Lena, Pepper, Barbara and Jet are used as the raw image. Among them, Lena and Pepper are the flat images; Barbara is with abundant texture. Compared with them, Jet is a neutral one. These images are compressed with QS1¼30, QS2¼25 and QS3¼20 to produce blocking images. There are slight blocking artifacts on the compressed
Table 1 PSNR comparison of different deblocking methods. Image
Quality Method Liew and Hong [25]
Zhai Chatterjee and Proposed et al. [26] Milafar [14]
Barbara QS1 QS2 QS3
29.72 29.02 28.15
30.25 29.42 28.46
31.22 30.59 29.58
31.43 30.53 29.39
Lena
QS1 QS2 QS3
34.51 34.04 33.42
34.57 34.08 33.50
32.61 32.43 32.15
34.96 34.46 33.79
Pepper
QS1 QS2 QS3
33.15 32.89 32.51
33.17 32.87 32.48
35.16 34.85 34.41
33.33 33.03 32.60
Jet
QS1 QS2 QS3
34.93 34.34 33.61
34.86 34.22 33.45
33.74 33.52 33.18
35.25 34.58 33.76
2
of neighbor pixel e:vðNi ÞvðNj Þ :=h is easily calculated through inquiring predefined table with two input vari2
ables :vðNi ÞvðNj Þ: and h. By this arrangement, only 4N additions are required for filtering block with other hs. If N ¼ 8, the computational cost of state-of-art nonlocal means filter are 16,384 times of addition. By the proposed algorithm, it requires 16,752 times of addition to measure deblocking performance of the first h, and 624 times of addition for the other hs. Note that optimal deblocking results have been produced when we find the optimal h. Therefore, there are about 3:5 m% computational increase over standard nonlocal means filter, where m is the number of h to be examined. Besides simplifying the computation of cost function (7), we further consider reducing the number of the candidate hs to be tested. Because MSDS is gradually reduced with h increasing, C 1,i,j in (7) is a decreasing function. On the contrary, high frequency information will be lost more if larger h is used, so C 2,i,j is an increasing function of hs. Therefore, their combination C i,j is a convex function. With this convexity, we develop fast algorithm to find h with minimum C i,j . In the range [0.5hinitial , 1.5hinitial ], we evenly sample 2 hs, i.e. h1¼ 0.83hinitial and h2¼1.16hinitial , and then calculate their costs by (7). If C i,j ðh1Þ o C i,j ðh2Þ, the optimal h is further searched within [0.5hinitial , 1.16hinitial ] through resampling and range decomposition. We use QS¼10 as example, which corresponds to large potential h range. To get the optimal h within 0.1hinitial precision, we need 3 range decomposition and 6 tests with different h, so that the computational cost of the proposed algorithm is only about 20% higher than that of the nonlocal means filtering.
Table 2 VIF comparison of different deblocking methods. Image
Liew and Hong [25]
Zhai Chatterjee and Proposed et al. [26] Milafar [14]
Barbara QS1 QS2 QS3
0.48 0.43 0.37
0.52 0.46 0.39
0.52 0.50 0.47
0.52 0.47 0.40
Lena
QS1 QS2 QS3
0.49 0.44 0.39
0.51 0.46 0.40
0.40 0.40 0.37
0.51 0.46 0.40
Pepper
QS1 QS2 QS3
0.47 0.42 0.37
0.48 0.43 0.38
0.49 0.47 0.45
0.48 0.43 0.38
Jet
QS1 QS2 QS3
0.49 0.45 0.39
0.51 0.46 0.40
0.47 0.46 0.43
0.51 0.46 0.40
5. Simulation results In the following experiments, we first compress the raw images with JPEG standard to form images with different degree blocking artifacts. JPEG algorithm uses
Quality Method
C. Wang et al. / Signal Processing: Image Communication 28 (2013) 522–530
image with QS1¼ 30 and heavy blocking artifacts on the compressed one with QS3¼20. Searching window size of the nonlocal means filter is a 7 7 square area. PSNR comparison of different deblocking methods [14,25,26] is given in Table 1. In [25], Liew et al. analyze the image and the quantization noise in wavelet domain, and then suppress noise through changing wavelet coefficients. Zhai et al. use shape-adaptive filter to eliminate quantization noise, and utilize coefficient regularization and quantization constraint to confine deblocking solution within rational range. Recently, Chatterjee and Milafar [14]
527
propose a patch-based Wiener filter that exploits patch redundancy for image denoising. We select the first two algorithms as the reference for they are, adaptive ones and their codes are available on web, and the last one for it is up-to-date and with the highest performance. It is shown that the proposed algorithm is consistently better than the two methods (i.e. [25,26]) on all test images compressed with different QSs. On the images, which all kinds of textures, such as Barbara, the proposed algorithm gets larger PSNR improvement, i.e. about 1.2 dB, than the other two methods. It is due to the fact that our algorithm can keep
Fig. 4. Zoomed parts of Barbara. (a) Original image, (b) processed by Liew, (c) processed by Zhai, (c) processed by Milanfar, and (e) processed by the proposed method.
528
C. Wang et al. / Signal Processing: Image Communication 28 (2013) 522–530
the detailed information, such as stripe of Barbara trousers, during deblocking. Even on flat images, such as Pepper and Lena, the proposed images can still gets some PSNR improvement, i.e. about 0.2–0.3 dB. From Table 1, we find the proposed algorithm is generally better and a robust one for it gets approximate PSNR improvement even at different compression ratios. To compare with more sophisticate algorithm, we also use [14] as the reference. Although this algorithm is flexibly adjusted with image content, it still requires noise amplitude as known parameter, which is too eager. Instead, we adopt noise amplitude E ¼ 15 here, for it statistically gets the best results on all test images. Table 1
shows that the proposed algorithm has comparable performance as [14] in that the number of getting the best PSNR, but the former has 0.3 dB PSNR improvement over the latter on average. We also show validity of the proposed algorithm on the subjective view, and use visual information fidelity (VIF) as the evaluation score. VIF shares interesting similarities with human visual system (HVS) based quality assessment methods. HVS includes channel decomposition and exponential response. The distorted image information is described to a divisive-normalization based masking model and the numerator of VIF is a HVS based
Fig. 5. Zoomed parts of Lena. (a) Original image, (b) processed by Liew, (c) processed by Zhai, (c) processed by Milanfar, and (e) processed by the proposed method.
C. Wang et al. / Signal Processing: Image Communication 28 (2013) 522–530
perceptual distortion metric, so that VIF is given as VIF ¼
Distorted Image Information Reference Image Information
ð8Þ
Until now, VIF has the highest correlation with human visual judgment, compared with peering algorithms. In this paper, we use it as the criterion to subjectively measure deblocking performance. In Table 2, we list VIF values of four algorithms, where QS1¼30, QS2¼25 and QS3¼20. It is shown that the proposed and Milanfar algorithms get the best VIFs in most cases. Our algorithm is better in the case where blocking artifacts are heavy or QS is small, which is just the case required to do deblocking. On the contrary, Milanfar algorithm outperforms ours on the images with slightly compressed, because quantization noise in slightly compressed image is closer to Gaussian distribution. Milanfar algorithm finds similar blocks within whole image, and uses them to estimate filter parameters, so that more information redundancy can be used to suppress Gaussian noise. Its performance will be reduced if patch similarity has been destroyed by heavy compression. In Fig. 4, we compare subjective quality of the zoomed image part Barbara before and after deblocking, which is compressed with QS ¼20. It is shown that our resultant image is of high perceptual quality as the proposed algorithm smoothes most grid noise and suppresses the edge-related noise. Liew algorithm and Zhai algorithm can repair these disfigurements as well, but they still produce discontinuousness on the neckcloth stripe. Blocking artifacts still exist on the neckcloth, as marked in Fig. 4(b). Zhai algorithm excels in removing quantization noise on flat and texture areas, but it is weak at edgerelated noise elimination, for example, the areas marked in left-down and right-up corners. Compared with the others, Milanfar is the best one in removing noise, which is especially obvious on the right-down corner, i.e. around Babara’s knee. Unfortunately, its result is excessively processed, and parts of texture, is removed during deblocking. In this algorithm, stripe on neckcloth is unnatural, and that on cloth is lost, for its imperfect threshold selection (Fig. 4). We also conduct experiments on Lena with heavier compression, and their results are presented in Fig. 5. It is also shown that Liew and Zhai algorithms cannot produce the satisfied results on Lena’s mouth and nose, and remain slight blocking artifacts on Lena’s face. Meanwhile, Minanfar algorithm is better in these areas, but it completely removes hair’s details. Compared with them, the proposed algorithm takes their strong points and gets some improvements in these parts. The above experimental results demonstrate that the proposed deblocking measure has competitive performance with [14] and better than [25,26] on both objective and subjective aspects without any requirement of side information of the distortion, such as the quantization step size. Among these algorithms, Liew and Zhai algorithms are completely adaptive, and their speed is also faster. To deblock 512 512 image, they only require 20s on Matlab platform. After optimization, the proposed algorithm is slightly slower than Zhai algorithm, and
529
takes 24s to do the same work on the same platform. Moreover, Milanfar algorithm is half-adaptive, and its execution is the slowest. Even with C language optimization, it still requires 120s to do deblocking. Therefore, the proposed algorithm is promising one for it balances performance and execution speed. 6. Conclusion In this paper, we first introduce a no-reference metric to evaluate deblocking efficiency, which involves two factors, i.e. the smoothness of the block boundaries and image content fidelity. Block smoothness is measured by mean squared difference of slope, and image content fidelity is simply described as the high frequency degradation of the filtered blocks. Using this metric, we design a patch-adaptive non-local means filter to optimize the parameter selection for deblocking under unsupervised way. In the proposed method, patch and searching window sizes are fixed for simplifying implementation architecture, and filter parameter optimization is done through traversal algorithm. This optimization can be more flexibly implemented by other methods, such as linear regression, which has been carried out recently. Simulation results demonstrate that the proposed metric depicts deblocking performance well. Benefit from the metric, our deblocking algorithm outperforms the other three stateof-the-art methods from the aspects of objective and subjective quality as well as execution speed. References [1] S.K. Lee, Edge statistics-based image scale ratio and noise strength estimation in DCT-coded images, IEEE Transactions on Consumer Electronics 55 (4) (2009) 2139–2144. [2] I.H. Jang, N.C. Kim, H.J So, Iterative blocking artifact reduction using a minimum mean square error filters in wavelet domain, Signal Processing 83 (12) (2003) 2607–2619. [3] H.C. Reeve, J.S. Lim, Reduction of blocking artifacts in image coding, Optical Engineering 23 (1) (1984) 34–37. [4] S. Minami, A. Zakhor, An optimization approach for removing blocking artifacts in transform coding, IEEE Transactions on Circuit and Systems for Video Technology 5 (2) (1995) 74–82. [5] D. Sun, W.K. Cham, Postprocessing of low bitrate block DCT coded images based on a fields of experts prior, IEEE Transactions on Image Processing 16 (11) (2007) 2743–2751. [6] A.Z. Averbuch, A. Schclar, D.L. Donoho, Deblocking of block transform compressed images using weighted sums of symmetrically aligned pixels, IEEE Transactions on Image Processing 14 (2) (2005) 200–212. [7] D.T. Vo, T.Q. Nguyen, Yea Sehoon, A. Vetro, Adaptive fuzzy filtering for artifact reduction in compressed images and videos, IEEE Transactions on Image Processing 18 (6) (2009) 1166–1178. [8] I. Kim, Adaptive blocking artifact reduction using wavelet-based block analysis, IEEE Transactions on Consumer Electronics 55 (2) (2009) 933–940. [9] S. Wu, H. Yan, Z. Tan, An efficient wavelet-based deblocking algorithm for highly compressed images, IEEE Transactions on Circuits and Systems for Video Technology 11 (11) (2001) 1193–1198. [10] J. Xu, S. Zheng, X. Yang, Adaptive video blocking artifact removal in Discrete Hadamard Transform domain, Optical Engineering 45 (8) (2006) 080501. [11] W.T. Freeman, E.C. Pasztor, O.T. Carmichael, Learning low-level vision, International Journal of Computer Vision 40 (1) (2000) 25–47. [12] M. Zhang, B.K. Gunturk, Multiresolution bilateral filtering for image denoising, IEEE Transactions on Image Processing 17 (12) (2008) 2324–2333.
530
C. Wang et al. / Signal Processing: Image Communication 28 (2013) 522–530
[13] V.K. Nath, D. Hazarika, A. Mahanta, Blocking artifacts reduction using adaptive bilateral filtering, 2010 International Conference on Signal Processing and Communications, 2010, pp.1–5. [14] P. Chatterjee, P. Milafar, Patch-based near-optimal image denoising, IEEE Transactions on Image Processing 21 (4) (2012) 1635–1649. [15] H.R. Wu, et al., A generalized block-edge impairment metric for video coding, IEEE Signal Processing Letters 4 (11) (1997). [16] C.H. Yim, A.C. Bovik, Quality assessment of deblocked images, IEEE Transactions on Image Processing 20 (1) (2011) 88–98. [17] A. Buades, B. Coll, J.-M. Morel, A non-local algorithm for image denoising, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2005, pp. 60–65. [18] R.L. de Queiroz, Processing JPEG-compressed images and documents, IEEE Transactions on Image Processing 7 (1998) 1661–1672. [19] S. Minami, A. Zakhor, An optimization approach for removing blocking effects in transform coding, IEEE Transactions on Circuits and Systems for Video Technology 5 (1995) 74–82. [20] T. Meier, K.N. Ngan, G. Crebbin, Reduction of blocking artifacts in image and video coding, IEEE Transactions on Circuits and Systems for Video Technology 9 (3) (1999) 490–500.
[21] J. Mateos, A.K. Katsaggelos, R. Molina., A Bayesian approach for the estimation and transmission of regularization parameters for reducing blocking artifacts, IEEE Transactions on Image Processing 9 (7) (2000) 1200–1215. [22] J. Yang, H. Choi, T. Kim, Noise estimation for blocking artifacts reduction in DCT coded images, IEEE Transactions on Circuits and Systems for Video Technology 10 (7) (2000) 1116–1120. [23] M.A. Robertson, R.L. Stevenson, DCT quantization noise in compressed images, International Conference on Image Processing 1 (2001) 185–188. [24] H.T. Soon, K.K. Pang, K.N. Ngan, Classified perceptual coding with adaptive quantization, IEEE Transactions on Circuits and Systems for Video Technology 6 (1996) 375–388. [25] A.W.C. Liew, Y. Hong, Blocking artifacts suppression in block-coded images using overcomplete wavelet representation, IEEE Transactions on Circuits and Systems for Video Technology 14 (2004) 450–461. [26] G. Zhai, W. Zhang, X. Yang, W. Lin, Y. Xu, Efficient deblocking with coefficient regularization, shap-adaptive filtering, and quantization constraint, IEEE Transactions on Multimedia 10 (5) (2008) 735–745. [27] /http://www.convict.lu/Jeunes/ultimate_stuff/exp_ln_2.htmS.