- Email: [email protected]
Iterative non-local means filter for salt and pepper noise removal
J. Vis. Commun. Image R. 38 (2016) 440–450
Contents lists available at ScienceDirect
J. Vis. Commun. Image R. journal homepage: www.elsevier.com/locate/jvci
Iterative non-local means filter for salt and pepper noise removal q Xiaotian Wang a,⇑, Shanshan Shen a, Guangming Shi a, Yuannan Xu b, Peiyu Zhang a a b
Xidian University, Xi’an, Shaanxi 710071, PR China Science and Technology on Optical Radiation Laboratory, Beijing 100854, PR China
a r t i c l e
i n f o
Article history: Received 26 November 2015 Revised 28 February 2016 Accepted 27 March 2016 Available online 31 March 2016 Keywords: Salt and pepper noise removal Non-local means Iterative filtering
a b s t r a c t Salt and Pepper noise (S&P noise) removal is an active research area in digital image processing. Existing techniques commonly use the local statistics within a neighborhood to estimate the centered noisy pixel, and tend to damage image details due to the image local diversity singularity and non-stationarity. To address this problem, in this paper, iterative nonlocal means filter (INLM) is proposed to exploit the image non-local similarity feature in the S&P noise removal procedure. Moreover, the proposed iterative framework update the similarity weights and the estimated values for higher accuracy. The experimental results show that the proposed INLM produces better results than state-of-art methods over a wide range of scenes both subjectively and objectively, and it is robust to the detection results. Ó 2016 Elsevier Inc. All rights reserved.
1. Introduction Digital images are usually contaminated by S&P noise which is introduced during image acquisition, transmission and recording process. S&P noise ruins image signal in terms of short-duration, discontinuous, noise spikes, which severely affects the subsequent image processing operations. Hence, S&P noise removal is crucial in digital image processing. Many filtering techniques had been proposed for this specific problem. The most popular and robust nonlinear filters were median filter (MF) [1] and its variations [2–4]. However, these kinds of filters treated all pixels with the same manner and replaced all the noise free pixels with the estimated values. As the result, the details of the image would be damaged, especially in the case of low signal noise ratio (SNR). To avoid this situation, various switching based filters, which introduced noise detection process before image restoration were proposed, such as [5–21]. Among these techniques, boundary discriminative noise detection filter (BDND) [12], modified BDND (MBDND) [13] and the sorted switching median filter (SSMF) [21] achieved widely acknowledged for the abilities of protecting the image detail information. Another kinds of methods were committing to find more effective filters. Yuksel [22], Toh and Isa [23], and Civicioglu [24] introduced a kind of hybrid filters, which combined median filter with other process, such as edge detector and neuro-fuzzy network in the denoising q
This paper has been recommended for acceptance by Hongliang Li.
⇑ Corresponding author.
E-mail addresses: [email protected] (X. Wang), [email protected] (S. Shen), [email protected] (G. Shi), [email protected] (Y. Xu), [email protected] (P. Zhang). http://dx.doi.org/10.1016/j.jvcir.2016.03.024 1047-3203/Ó 2016 Elsevier Inc. All rights reserved.
procedure, Morillas et al. [25], Camarena et al. [26], Morillas et al. [27], Smolka and Chydzinski [28], and Smolka [29]used peer group filters and fuzzy metrics for color images denoising. Recently, to get better denoising effect, many methods had been proposed based on a kind of new ideas that took full advantage of the different characteristics of the image and noise, and obtained a good denoising effect. For example, Zuo et al. [30] introduced a denoising procedure based on noise space characteristic, Zhang and Xiong [15] used image directional difference detector on the basis of adaptive weighted mean filter (SAWM). Zhou [31] proposed a novel effective filter based on the cloud modeling (CM) which used the uncertainties of the noise. By taking different kinds of characteristic into account, these kinds of filters restored the image with better performance and achieved wide acception. Although existing techniques have been widely accepted and achieved lots of remarkable results, there is still much room for improvement. Classic techniques in image S&P noise removal strongly rely on exploiting information in a local window around the estimated pixel for this specific restoration task. However, considering the diversity singularity and nonstationary feature of the image signal in a local window, the estimation result could easily diverge from the true value and cause ugly visual effects in texture and edge regions, especially in the case of high noise density. Therefore, local information is insufficient for high quality image restoration. It has been proved and widely accepted that one can expect better denoising performance by exploiting the nonlocal information of the image signal in the case of gaussian distributed noise suppression [32,33]. This inspired us that can we achieve better restoration performance in S&P noise removal by exploiting image non-local information during the denoising procedure? In
X. Wang et al. / J. Vis. Commun. Image R. 38 (2016) 440–450
this paper we will discuss this question and propose an iterative nonlocal means filter (INLM) for image S&P noise removal. The concept of nonlocal means filter (NLM) is based on the fact that there exists lots of similar patches with repeat patterns in natural images, and the central pixels of these similar patches share the same gray value distribution. Instead of estimating the pixel value within a local window, NLM replaces the considered pixel by the weighted mean of all the similar patterns’ central pixels. NLM has been widely accepted in image Gaussian noise suppression, however, its application in S&P noise removal has been barely discussed. In this paper, we will integrate the NLM algorithm into a S&P noise removal problem. Moreover, an iterative weighted average scheme is carried out and an iterative nonlocal means filter is proposed for further improvement. The simulation results show that the proposed INLM is capable of recovering image fine details and textures reliably at a wide range of S&P noise densities so that it outperforms other state-of-art denoising methods in terms of both subjective and objective qualities. The outline of this paper is organized as follows. In Section 2, we analyze the characteristics of both Gaussian noise and S&P noise, and explain why the NLM algorithm for Gaussian noise filtering can be integrated into a S&P noise suppression problem. In Section 3, we will develop the proposed INLM for S&P noise removal in three stages. In Section 4, experimental results of our method and a comparison study with some state-of-art techniques are presented. Finally, we conclude the paper in Section 5.
441
The amplitude and distribution characteristics of these two kinds of noise are so different that the processing methods on these two kinds of noise in principle are also different from each other. However, it occurs to us that what is the amplitude and distribution characteristic of S&P noise after prefiltering? To find out the answer to this question, we implemented switching based median filter (SMF, introduced in Section 3, Part 1) on S&P noised images and analyzed the histogram of the noisy pixels after filtering. An example of the noise histogram of Lena is given in Fig. 2(a). In this example, the testing image corrupted by S&P noise with different noise densities was filtered by SMF. The noise histogram after prefiltering is represented by green curves in Fig. 2(a). The red line in the figure represents the Gaussian distribution curve. We find that the noise histogram of each pre-filtered image approximately obeys Gaussian distribution. To make the discovery more convincing, we tested a bunch of images and showed the noise histogram after prefiltering in Fig. 2(b). It can be seen that the green curves are all very close to the red curve, which means that after SMF, the model of the noise remained in the image is very close to Gaussian distribution. This inspires us that can we achieve better recovering denoising performance by successfully fitting a Gaussian noise filter into an impulse noise removal problem? In this paper, we will discuss about this question and give an affirmative answer by applying NLM filter into S&P noise removal.
2.2. NLM algorithm for Gaussian noise filtering 2. Noise models and NLM for image denoising 2.1. Noise model analysis In the process of image acquisition, transmission and storage, different mechanisms would introduce different types of noise. The two most common noise models are Gaussian noise and S&P noise. Gaussian noise is a kind of additive noise whose amplitude obeys Gaussian distribution as follows: ðnlÞ2 1 pðnÞ ¼ pffiffiffiffiffiffiffi e 2r2 ; 2pr
ð1Þ
where n denotes the noise value, l and r denote the noise mean and standard deviation respectively. S&P noise is another commonly encountered noise type which appears as black and white spots randomly distributed over the image. When an image is corrupted by S&P noise, only a certain portion of the image pixels are replaced with noise values while the remaining pixels are unchanged. In this case, the values of the noisy pixels have either maximum value (rmax ) or minimum value (rmin ) of the image intensity range. The S&P noise model with probability q can be defined as follows:
8 with probability a > < r max ; Xði; jÞ ¼ r min ; ; with probability b > : xði; jÞ; with probability 1 q
ð2Þ
where Xði; jÞ denotes the luminance values of the noisy image at the location ði; jÞ, and xði; jÞ denotes noise-free pixel values at the location ði; jÞ with probability 1 q, and a þ b ¼ q. For further analysis, we visualize the histogram of those noise model and their influence on a clear image. Fig. 1(a) shows a histogram image of a testing image Lena. Fig. 1(b) and (c) are the histogram of noised image contaminated by S&P noise and Gaussian noise respectively. In Fig. 1(b), S&P noise falls into the red1 rectangle. And the histogram of Gaussian noise is given in Fig. 1(d). 1 For interpretation of color in Fig. 1, the reader is referred to the web version of this article.
The NLM algorithm was proposed to process images contaminated by Gaussian noise. For each pixel to be estimated, the NLM takes into consideration the similarity between the neighborhood configuration of this pixel and all the pixels in every neighborhood. ^ i;j can be computed as a Mathematically, the estimated pixel X weighted average of all the neighborhood pixels in the noisy image as follows:
P k;l2Xi;j wi;j;k;l Y k;l ^ i;j ¼ P X ; k;l2Xi;j wi;j;k;l
ð3Þ
where Xi;j is the patch group similar to the current patch centered ^ i;j ; Y k;l is the center pixel of the similar patches in Xi;j . The canby X didates in Xi;j are similar patches whose Euclidean distance is close enough to the current patch centered by X i;j . In Eq. (3), weights wi;j;k;l defines the weight for each candidate Y k;l , which can be calculated as follows:
wi;j;k;l ¼ e
kPðX i;j ÞPðY k;l Þk2 2;a h2
;
ð4Þ
where PðX i;j Þ and PðY k;l Þ represent the local windows centered by pixel X i;j and Y k;l in the noised image. And h is the smoothing parameter, which controls the decay of the exponential function. The norm used in Eq. (4) is simply the Euclidean difference, weighted by Gaussian kernel of zero mean and variance a. NLM has achieved satisfactory performance in Gaussian noise suppression because during denoising, not only local statistical information but also global structure similarity are taken into consideration. However, it could not be implemented in S&P noise removal directly because pixels corrupted by S&P noise are significantly different from their neighbors. This will lead to significant mistakes during similarity weights calculation. To solve this problem, in this paper, a new framework is constructed. The S&P noise are first pre-filtered via statistical filter and then NLM is implemented on noised pixels iteratively to refine the denoising results.
442
X. Wang et al. / J. Vis. Commun. Image R. 38 (2016) 440–450
Fig. 1. Histograms of original image and noised image on lena. (a) Original image. (b) Noised image contaminated by 20% S&P noise. (c) Noised image contaminated by Gaussian noise of l ¼ 30, r ¼ 20. (d) Gaussian noise of l ¼ 30, r ¼ 20.
Fig. 2. Noise histogram of images after prefiltering. (a) Lena. (b) A bunch of images.
443
X. Wang et al. / J. Vis. Commun. Image R. 38 (2016) 440–450
Fig. 3. The flowchart of the INLM.
framework is used to approximate the optimal value. The flowchart of the INLM is shown as in Fig. 3.
Table 1 Comparison of PSNR values on different iteration times (dB). image
Pre de
INLM 1
INLM2
INLM3
INLM4
Lena Monarch Rice Parrot
32.41 32.05 34.61 33.61
34.46 32.97 35.24 34.40
34.77 33.51 35.60 34.93
34.85 33.72 35.68 35.12
34.89 33.83 35.74 35.29
3. Proposed INLM for S&P noise removal The proposed method mainly contains three stages. (1) A preprocessing procedure based on noise detection is performed to construct a noise map N map and an intermediate pre-processed result Pre de is obtained by using switching based mediain filter (SMF) combined with the location information marked in N map. (2) Patch matching procedure in NLM is performed on and only on each noisy pixel centered local window. (3) An iterative
3.1. Pre-processing (SMF) In this section, SMF is used for enhancing the performance of S&P noise, which first marks the pixels as noisy and noise-free pixels and then performs filtering only on the noisy pixels. Moveover, as we have introduced in Section 2.1, S&P noise model is formulated based on the assumption that the amplitudes of noisy pixels fall into two small intervals around the two ends of the image histogram. In our proposed method, only noise pixels are filtered while noise-free pixels are left unchanged. According to the noise detection result, it uses ‘1’ and ‘0’ to represent the positions of noise-free and noisy pixels, respectively. Therefore, noise map N mapcan be formed as follows:
N mapði; jÞ ¼
1 min þb1 6 X i;j 6 max b2 0
otherwise
Fig. 4. Lena’s hat. (a) Original image. (b) Pre de after SMF. (c) INLM1 . (d) INLM4 .
:
ð5Þ
444
X. Wang et al. / J. Vis. Commun. Image R. 38 (2016) 440–450
Table 2 PSNR of INLM with different detecting precision. Detecting precision
b1 =b2 =5
b1 =b2 =10
b1 =b2 =15
b1 =b2 =20
Boat
(FA, MD) INLM1 (dB) INLM5 (dB)
(0,0) 29.4846 30.0381
(1,0) 29.4834 30.0372
(24,0) 29.4762 30.0290
(996,0) 29.4664 30.0184
Baboon
(FA, MD) INLM1 (dB) INLM5 (dB)
(48,0) 21.2785 21.3911
(86,0) 21.2634 21.3731
(166,0) 21.2350 21.3423
(331,0) 21.1688 21.2648
In this formula, ½min; max describes the dynamic range of the digital image. For the commonly used 8-bit testing images, ½min; max ¼ ½0; 255. b1 and b2 are two positive numbers, which determine the length of the intervals around the two ends of the histogram. In this paper, we set b1 ¼ b2 ¼ 5. All noisy pixels are estimated by the median of the uncontaminated ones from its neighborhood in a local window. Let Pre deði; jÞand Xði; jÞ denote the pixels of the prefiltered image and the noisy image at the location ði; jÞ, respectively. And let Xðk; lÞ be a set of pixels in the window of size ð2s þ 1Þ ð2s þ 1Þ centered at the location ðk; lÞ. We can write Xðk; lÞ as:
Fig. 5. Testing images. (a) Original images. (b) Noised images contaminated by 70% S&P noise.
X. Wang et al. / J. Vis. Commun. Image R. 38 (2016) 440–450
445
Fig. 6. Denoised images recovered from 70% S&P noise contaminated testing images. From top left to bottom right: Lena (34.93 dB), Boat (29.95 dB), Monarch (33.92 dB), Pepper (31.08 dB), Baboon (21.59 dB), Rice (35.77 dB), Parrot (35.39 dB) and Port (34.15 dB).
Table 3 PSNR results of restored images with 70% S&P noise (dB). Testing image
IBINR [34]
MBDND [13]
SAWM [15]
SDTF [30]
SSMF [21]
CM [31]
INLM
Lena Boat Monarch Pepper Baboon Rice Parrot Port
30.82 27.86 29.56 29.15 21.67 31.95 31.39 32.17
32.75 27.98 30.50 30.28 21.12 32.12 30.84 32.64
32.92 28.08 30.87 30.40 21.15 32.78 29.40 32.87
32.41 27.78 30.13 29.76 20.75 32.19 32.21 32.55
31.77 27.17 28.95 29.23 20.80 31.30 30.35 31.58
32.72 27.99 30.52 30.00 20.91 32.52 32.67 32.84
34.93 29.95 33.92 31.08 21.59 35.77 35.39 34.15
These bold values demonstrate the denoising performance of the proposed INLM.
Xðk; lÞ ¼ fk 2 ði s; i þ sÞ; l 2 ðj s; j þ sÞg:
ð6Þ
Then, a prefiltering procedure is performed by utilizing the local statistics informating as follows:
( Pre deði; jÞ ¼
median ðXðk; lÞÞ N mapði; jÞ ¼ 0
mapðk;lÞ–0
Xði; jÞ
N mapði; jÞ ¼ 1
:
ð7Þ
Although the detection procedure is very simple, INLM is robust to the detection results. We will discuss the detection results detailedly in the experimental section. 3.2. Patch matching
feature of the original image, the precision of the noisy pixel estimation is poor. This will affect the restoration precision and lead to obvious jagged artifacts around large scale edges. In this paper, we propose to implement the weight calculation and weighted averaging iteratively to pursue better denoising performance. The algorithm flow is described as follows: Algorithm. Iterative NLM for S&P noise removal (1) Initialization (a) Initialize the input image using the prefiltered result ð0Þ
X i;j ¼ Pre dei;j (b) Construct the similar patch group Xi;j for each noisy pixel; (c) Initialize the similar weight for each noisy pixel and its similar candidates Y k;l in Xi;j ð0Þ
ð0Þ ÞPðY k;l Þjj2 2 i;j h2
(2) Outer loop, t = 1,2,. . . (a) Update the averaging weight ðtÞ
jjPðX
wi;j;k;l ¼ e
ðt1Þ ÞPðY Þjj2 k;l 2 i;j h2
(b) Update the denoised result P ðtÞ w Y k;l2Xi;j i;j;k;l k;l ^ ðtÞ ¼ P X ðtÞ i;j
When using the traditional NLM dealing with Gaussian noise, all pixels are processed in this step. In our method, under the guidance of N map, only the noisy pixels Xði; jÞ in Pre de need to search their similar patch group Xi;j and be processed by INLM. The similar patch searching and similarity measurement are the same as the way carried out in traditional NLM. The similarity between two pixels is measured by evaluating the Gaussian weighted Euclidean distance between two image patches surrounding the pixels as in Eq. (4).
jjPðX
wi;j;k;l ¼ e
k;l2Xi;j
wi;j;k;l
(c) If (MSEðX ðtÞ ; X ðt1Þ Þ < Thr or t > 5), go to step (3). Else, go back to step 2-(a). ^ (3) Output the recovered image X.
In step 2-(c), the expression MSEðX ðtÞ ; X ðt1Þ Þ is the mean square
3.3. Iteratively estimation
error between the two variables X ðtÞ and X ðt1Þ . The parameter Thr is the threshold which determines the termination condition of the iteration. This formular means that when the difference between
Although, the prefiltered result Pre de obtained from switching based median filtering contains the coarse gray scale and structure
X ðtÞ and X ðt1Þ is small enough, this iteration should be stopped. Based on our experience on a bunch of test images contaminated by different noise densities, Thr can be calculated as follows:
446
X. Wang et al. / J. Vis. Commun. Image R. 38 (2016) 440–450
Table 4 SSIM results of restored images with 70% S&P noise. Testing image
IBINR [34]
MBDND [13]
SAWM [15]
SDTF [30]
SSMF [21]
CM [31]
INLM
Lena Boat Monarch Pepper Baboon Rice Parrot Port
0.9630 0.9575 0.9812 0.9733 0.9004 0.9692 0.9820 0.9741
0.9790 0.9644 0.9884 0.9625 0.8936 0.9760 0.9835 0.9783
0.9794 0.9645 0.9889 0.9639 0.8943 0.9761 0.9840 0.9810
0.9793 0.9654 0.9897 0.9631 0.8930 0.9778 0.9890 0.9823
0.9734 0.9544 0.9829 0.9621 0.8762 0.9696 0.9809 0.9739
0.9784 0.9656 0.9893 0.9627 0.8954 0.9778 0.9638 0.9813
0.9844 0.9760 0.9927 0.9808 0.8993 0.9783 0.9902 0.9834
These bold values demonstrate the denoising performance of the proposed INLM.
Thr ¼
MSEðX ð0Þ ; X ð1Þ Þ : 20
ð8Þ
It should be noticed that the most time consuming part of the non-local means algorithm is searching similar windows in the image. However, in our method, the iterative part only updates the similarity weight and calculates the weighted average value. Therefore, the computational complexity of this part is negligible compared with the similar patch searching procedure. To show the effectiveness of the iteration procedure, INLM is implemented on four commonly used 512 512 testing images Lena, Monarch, Rice and Parrot, which are contaminated by 70% S&P noise. The peak signal-to-noise ratio (PSNR) evaluation of our method with different iteration times is summarized in Table 1. In Fig. 4, we visualize the effect of INLM by zooming into Lena’s hat. In Table 1 and Fig. 4, INLMi means the iteration was implemented i times. It can be seen from the experimental results that the restoration performance improved significantly with the iteration filtering both subjectively and objectively. 4. Experiments and discussions In this section, we assess the performance of the proposed INLM by carring out a number of experiments, and in NLM, we set the smoothing parameter h = 2, the similar patch size K = 3 (the actual size is ð2K þ 1Þ ð2K þ 1Þ), and the searching window W = 10 (the actual size is ð2W þ 1Þ ð2W þ 1Þ). In order to have a better understanding of the performance of INLM, our experiments are divided into three parts. The first part is to analyze the noise detecting performance, which is discussed to show the robustness of INLM to the detecting false alarm. The second part is image denoising performance exhibition and comparison with some selected state-of-art filters. The third part is the computational complexity analysis.
the histogram, which is defined by b1 and b2 . And larger b1 and b2 lead to more FA, vice versa. To further prove the robustness of INLM with respect to noise detecting accuracy, an example of the INLM with different detecting accuracy by setting different b1 and b2 on Boat and Baboon contaminated by 70% S&P noise is given in Table 2. It can be seen that the decrease of denoising performance with the increasing of FA is very inconspicuous. Therefore, in INLM the choice of b1 and b2 is not crucial. 4.2. Comparison of image denoising performance In this section, we verify our algorithm via experimental simulations. We test the proposed INLM on a bunch of popular testing images (Lena, Boat, Monarch, Pepper, Baboon, Rice, Parrot) and a remote sensing image Port as shown in Fig. 5(a). We carry out the experiments in three ways to make a comprehensive comparison. First, all testing images contaminated by 70% S&P noise are processed by INLM and other well-known filters such as the interpolation-based denoising (IBINR, 2013) [34], the modified BDND (MBDND, 2013) [13], the switching-based adaptive weighted mean filter (SAWM, 2009) [15], the noise space characteristic based filter (SDTF, 2013) [30], the sorted switching median filter (SSMF, 2013) [21] and the cloud model based denoising (CM, 2012) [31]. We use peak signal to noise ratio (PSNR) and structural similarity index (SSIM) as the measure of evaluation to compare the denoising performance of each method. Second, we show the restoration performance of our method in different noise densities. We compare the average denoising results (PSNR) of each denoising method for 100 testing images which are downloaded from IMAGE NET2 with noise density ranging from 10% to 90%. Moreover, a testing image Monarch is used to exhibit the visual effect of our method for different noise densities. Third, we show the denoising results of INLM in an extreme case, in which the noise level goes up to 90%.
4.1. Analysis of noise detecting performance The noise detection is very important in all switching based S&P noise removal strategies. The accuracy of the detecting method affects the denoising performance severely. So far, the bottleneck of noise detection is how to decrease false alarm (FA), while miss detection (MD) barely exists. Lots of efforts have been paid for decreasing FA. However, high accuracy normally comes from elaborately designed detecting strategy with high computational complexity. Moreover, to the best of our knowledge, no method in published literatures can decrease the FA to zero. Fortunately, in the proposed INLM, this problem can be avoided because INLM is quite robust to FA. In our method, the value of estimated pixels is calculated by a group of pixels from the similar patches. Therefore, a few lacks of the group candidates does not influence the solution accuracy obviously. During histogram analyzing, the noise detecting accuracy is determined by the length of the intervals around the two ends of
4.2.1. Denoising performance with 70% S&P noise In this part, the method is tested on images contaminated by 70% S&P noise. The corrupted images are shown in Fig. 5(b), and the denoised results of INLM for each image are shown in Fig. 6. It can be seen that our method can effectively reconstruct clear images from the high density noise corrupted images. For comparison, these noised images are also processed by the six state-of-art denoising methods mentioned above. The PSNR and SSIM evaluation of each denoising method are shown in Tables 3 and 4. For subjective comparison, the denoising results of filtered Lena and Parrot are shown in Figs. 7 and 8. And we compare the detail visual effects of different methods for 70% S&P noise by zooming into the faces of Lena and Parrot. It can be seen from the experimental results that INLM outperforms those state-ofart filters both objectively and subjectively. 2
http://www.image-net.org/.
X. Wang et al. / J. Vis. Commun. Image R. 38 (2016) 440–450
447
Fig. 7. Denoising results of Lena contaminated by 70% S&P noise from different filters. From top left to bottom right: original image, IBINR (30.82 dB), MBDND (32.75 dB), SAWM (32.92 dB), SDTF (32.41 dB), SSMF (31.77 dB), CM (32.72 dB) and INLM (34.93 dB).
Fig. 8. Denoising results of Parrot contaminated by 70% S&P noise from different filters. From top left to bottom right: original image, IBINR (31.39 dB), MBDND (30.84 dB), SAWM (29.40 dB), SDTF (32.21 dB), SSMF (30.35 dB), CM (32.67 dB) and INLM (35.39 dB).
In order to make the experimental results more convincing, we introduce another evaluation index by analyzing the amplitude difference between the original image and the restored image. A difference image dif im is constructed by subtracting the denoised image de im from the original image or im as shown in Eq. (9):
dif im ¼ jor im de imj:
ð9Þ
For a successful impulse denoising method, the recovered result should be as close to the original image as possible. This point has two fold meanings: (1) in dif im, the number of the elements with large scale amplitude should be as small as possible; (2) the
maximum value of dif im should be as small as possible. Therefore, the amplitude distribution of dif im is analyzed in this part. In each dif im, we summarize the number of the pixels whose intensities fall into the ranges of [30, 40), [40, 50) and [50, 255] to show the estimating error of each method in a statistical way. In addition, the maximum value (Max dif ) of each dif im is also recorded for comparison. Lena, Parrot and Monarch with 70% S&P noise are used as testing images. The comparison results are shown in Table 5. It can be seen that INLM outperforms other denoising techniques by producing both least large scale pixels and smallest Max dif in dif im.
448
X. Wang et al. / J. Vis. Commun. Image R. 38 (2016) 440–450
Table 5 Amplitude distribution analysis of dif im. Filters
IBINR MBDND SAWM SDTF SSMF CM INLM1 INLM5
Difference level 30–40 40–50 >50 30–40 40–50 >50
30–40 40–50 >50
Lena
Monarch
933 581 517 503 682 475 347 330
Parrot 297 232 192 216 278 185 131 118
158 1058 160 497 98 484 161 550 202 842 168 508 66 361 60 311
549 275 225 282 654 257 216 145
510 434 497 469 974 2855 244 171
1856 1057 941 1029 1237 923 649 601
880 640 510 607 782 544 324 254
553 691 514 668 1056 621 267 190
These bold values demonstrate the denoising performance of the proposed INLM.
4.2.2. Denoising performance in different noise densities In this part, testing images are contaminated by a large range of noise densities (10–90%). For the purpose of comprehensive comparison, we visualize the denoising performance of each method by plotting the average PSNR values obtained from 100 denoised testing images corrupted by different noise densities, as shown in Fig. 9. It can be seen that the proposed method outperforms other state-of-art filters by having the highest PSNR values in different noise levels. Except for the average PSNR comparison, an example of the proposed INLM on Monarch is taken to show the denoising results in different noise densities (10%, 30%, 50%, 70%, 90%) in Fig. 10. Note that the proposed method is still capable of restoring the images with good visual effects even in strongly corrupted cases. Moreover, we also make detail objective comparison with other methods. Table 6 shows all denoising performances of each denoising technique for Monarch with different noise levels (10–90%). 4.2.3. Denoising results under extreme cases Finally, to further demonstrate the excellent performance of the proposed INLM, the denoising results on images Pepper, Baboon, Rice and Port contaminated by 90% S&P noise are given in Fig. 11. In such an extreme case, the restored images have some blurring edges in some local areas, however, the proposed INLM is still capable of restoring the images with good visual effects. 4.3. Computational complexity
Fig. 9. Graph of average denoised PSNR versus S&P noise density obtained by using a total of 100 testing images.
NLM contains two steps, similar patch matching and weighed mean of centered pixels in similar patches. In NLM, if we set the search window of 21 21 , and the similar patch of 7 7, for an image with N pixels, the computational cost of similar patch matching is 49 441 N, and the computational cost of weighed mean procedure is n N (n 6 441 is the number of the most similar patches). It is clear that the major computational cost of NLM comes from the similar patch matching. In our method, although iterative frame is used, the iterative procedure only contains the weighed mean of centered pixels in similar
Fig. 10. Denoising results of Monarch in different noise densities. From top left to bottom right: original image, noise density equals to 10%, 30%, 50%, 70% and 90%.
449
X. Wang et al. / J. Vis. Commun. Image R. 38 (2016) 440–450 Table 6 PSNR comparison for Monarch in noise density 10–90% (dB).
IBINR MBDND SAWM SDTF SSMF CM INLM
10
20
30
40
50
60
70
80
90
43.43 46.15 46.40 45.64 46.24 46.61 48.35
40.12 41.56 42.35 40.91 41.29 42.20 44.41
37.89 38.71 39.38 37.95 38.21 39.18 41.39
34.71 36.25 37.08 35.66 35.63 36.79 39.85
33.60 34.14 34.88 33.66 33.26 34.56 37.90
32.16 32.21 32.92 31.86 31.13 32.50 35.87
29.56 30.50 30.87 30.13 28.95 30.52 33.85
27.09 28.13 28.40 27.83 26.58 28.15 30.57
24.17 25.02 25.02 24.54 22.81 24.98 26.46
These bold values demonstrate the denoising performance of the proposed INLM.
Fig. 11. Denoised images recovered from 90% S&P noise. From left to right: Pepper (26.42 dB), Baboon (18.80 dB), Rice (29.40 dB) and boat (25.13 dB).
patches. This means, the most computational consuming procedure, similar patch matching, was carried out only once for each image. Therefore, the computational complexity of our method is 49 441 N þ I n N (I 6 5, is the number of iterations). 5. Conclusion In this paper, we have demonstrated the effectiveness of our proposed INLM algorithm with weight update for S&P noise removal. Instead of local estimation, non-local information is exploited to preserve the image detail. After similar patch searching, the weight and denoising result are calculated iteratively. Experiments showed that both the visual quality and evaluation indexes of the proposed algorithm outperform those state-of-art denoising techniques. Acknowledgements This work is supported by NSF of China (Nos. 61401333, 61401325, 61201289, 61372071, 61100155), Shaanxi province natural science foundation of China (No. 2014JQ8296), Research Fund for the Doctoral Program of Higher Education of China (No. 20130203120009), and Fundamental Research Funds for the Central Universities of China (Nos. JB140227, K5051302096, JB140207), and the National Defense Fund (No. 9140C610304150C61268). References [1] R.C. Gonzalez, R.E. Woods, Digital Image Processing, Prentice-Hall, Englewood Cliffs, NJ, 2002. [2] D.R.K. Brownrigg, The weighted median filter, Commun. ACM 27 (8) (1984) 807–818. [3] T. Chen, H.R. Wu, Space variant median filters for the restoration of impulse noise corrupted images, IEEE Trans. Circ. Syst. II: Anal. Digit. Signal Process. 48 (8) (2001) 784–789. [4] S.J. Ko, Y.H. Lee, Center weighted median filters and their applications to image enhancement, IEEE Trans. Circ. Syst. 38 (9) (1991) 984–993. [5] H.L. Eng, K.K. Ma, Noise adaptive soft-switching median filter, IEEE Trans. Image Process. 10 (2) (2001) 242–251. [6] G. Pok, J.C. Liu, A.S. Nair, Selective removal of impulse noise based on homogeneity level information, IEEE Trans. Image Process. 12 (1) (2003) 85– 92.
[7] S. Schulte, M. Nachtegael, V. De Witte, D. Van der Weken, E.E. Kerre, A fuzzy impulse noise detection and reduction method, IEEE Trans. Image Process. 15 (5) (2006) 1153–1162. [8] W.B. Luo, Efficient removal of impulse noise from digital images, IEEE Trans. Consum. Electron. 52 (2) (2006) 523–527. [9] I. Aizenberg, C. Butakoff, D. Paliy, Impulsive noise removal using threshold Boolean filtering based on the impulse detecting functions, IEEE Signal Process. Lett. 12 (1) (2005) 63–66. [10] V. Crnojevic, V. Senk, Z. Trpovski, Advanced impulse detection based on pixelwise MAD, IEEE Signal Process. Lett. 11 (7) (2004) 589–592. [11] P. Civicioglu, M. Alci, E. Besdok, Impulsive noise suppression from images with the noise exclusive filter, EURASIP J. Appl. Signal Process. 16 (2004) 2434– 2440. [12] P.E. Ng, K.K. Ma, A switching median filter with boundary discriminative noise detection for extremely corrupted images, IEEE Trans. Image Process. 15 (6) (2006) 1506–1516. [13] I.F. Jafar, R.A. AlNa’mneh, K.A. Darabkh, Efficient improvements on the BDND filtering algorithm for the removal of high-density impulse noise, IEEE Trans. Image Process. 22 (3) (2013) 1223–1232. [14] Z.F. Deng, Z.P. Yin, Y.L. Xiong, High probability impulse noise-removing algorithm based on mathematical morphology, IEEE Signal Process. Lett. 14 (1) (2007) 31–34. [15] Xu Ming Zhang, You Lun Xiong, Impulse noise removal using directional difference based noise detector and adaptive weighted mean filter, IEEE Signal Process. Lett. 16 (4) (2009) 295–298. [16] K.S. Srinivasan, D. Ebenezer, A new fast and efficient decision-based algorithm for removal of high-density impulse noises, IEEE Signal Process. Lett. 14 (3) (2007) 189–192. [17] P. Wei, J.L. Li, D.M. Lu, G. Chen, A fast and reliable switching median filter for highly corrupted images by impulse noise, in: Proc. IEEE Int. Symposium on Circuits and Systems (ISCAS’07), 2007, pp. 3427–3430. [18] W. Zhou, D. Zhang, Progressive switching median filter for the removal of impulse noise from highly corrupted images, IEEE Trans. Circ. Syst. II: Anal. Digit. Signal Process. 46 (1) (1999) 78–80. [19] V. Gupta, V. Chaurasia, M. Shandilya, Random-valued impulse noise removal using adaptive dual threshold median filter, J. Vis. Commun. Image Represent. 26 (9) (2015) 296–304. [20] Guorong Gao, Yanping Liu, Demetrio Labate, A two-stage shearlet-based approach for the removal of random-valued impulse noise in images, J. Vis. Commun. Image Represent. 32 (12) (2015) 83–94. [21] Shi Jinn Horng, Ling Yuan Hsu, Tian Rui Li, Shao Jie Qiao, Xun Gong, Hsien Hsin Chou, Muhammad Khurram Khan, Using sorted switching median filter to remove high-density impulse noises, J. Vis. Commun. Image Represent. 24 (7) (2013) 956–967. [22] M.E. Yuksel, A hybrid neuro-fuzzy filter for edge preserving restoration of images corrupted by impulse noise, IEEE Trans. Image Process. 15 (4) (2006) 928–936. [23] K.K.V. Toh, N.A.M. Isa, Noise adaptive fuzzy switching median filter for saltand-pepper noise reduction, IEEE Signal Process. Lett. 17 (3) (2010) 281–284. [24] P. Civicioglu, Using uncorrupted neighborhoods of the pixels for impulsive noise suppression with ANFIS, IEEE Trans. Image Process. 16 (3) (2007) 759– 773.
450
X. Wang et al. / J. Vis. Commun. Image R. 38 (2016) 440–450
[25] S. Morillas, V. Gregori, G.P. Fajarnes, Local self-adaptive fuzzy filter for impulse noise removal in color images, IEEE Trans. Signal Process. 88 (2) (2008) 390– 398. [26] J.G. Camarena, V. Gregori, S. Morillas, A. Sapena, Fast detection and removal of impulsive noise using peer groups and fuzzy metrics, J. Vis. Commun. Image Represent. 19 (1) (2008) 20–29. [27] S. Morillas, V. Gregori, G.P. Fajarnes, A. Sapena, New adaptive vector filter using fuzzy metrics, J. Electron. Imag. 16 (3) (2007) 1–15. [28] B. Smolka, A. Chydzinski, Peer group switching filter for impulse noise reduction in color images, Pattern Recogn. Lett. (2010) 489–495. [29] B. Smolka, Peer group filter for impulse noise removal in color images, Proc. CIARP’08 (2008) 699–707.
[30] Zhi Yong Zuo, Tian Xu Zhang, Jing Hu, Gang Zhou, A new method for removing impulse noise based on noise space characteristic, Optik – Int. J. Light Electron Opt. 124 (18) (2013) 3503–3509. [31] Zhe Zhou, Cognition and removal of impulse noise with uncertainty, IEEE Trans. Image Process. 21 (7) (2012) 3157–3167. [32] T. Brox, O. Kleinschmidt, D. Cremers, Efficient nonlocal means for denoising of textural patterns, IEEE Trans. Image Process. 17 (7) (2008) 1083–1092. [33] A. Buades, B. Coll, J. Morel, A review of image denoising algorithms, with a new one, SIAM Interdisc. J., Multiscale Model. Simul. 4 (2) (2005) 290–530. [34] C. Kalyoncu, O. Toygar, H. Demirel, Interpolation-based impulse noise removal, IET Image Process. 7 (8) (2013) 777–785.
Contents lists available at ScienceDirect
J. Vis. Commun. Image R. journal homepage: www.elsevier.com/locate/jvci
Iterative non-local means filter for salt and pepper noise removal q Xiaotian Wang a,⇑, Shanshan Shen a, Guangming Shi a, Yuannan Xu b, Peiyu Zhang a a b
Xidian University, Xi’an, Shaanxi 710071, PR China Science and Technology on Optical Radiation Laboratory, Beijing 100854, PR China
a r t i c l e
i n f o
Article history: Received 26 November 2015 Revised 28 February 2016 Accepted 27 March 2016 Available online 31 March 2016 Keywords: Salt and pepper noise removal Non-local means Iterative filtering
a b s t r a c t Salt and Pepper noise (S&P noise) removal is an active research area in digital image processing. Existing techniques commonly use the local statistics within a neighborhood to estimate the centered noisy pixel, and tend to damage image details due to the image local diversity singularity and non-stationarity. To address this problem, in this paper, iterative nonlocal means filter (INLM) is proposed to exploit the image non-local similarity feature in the S&P noise removal procedure. Moreover, the proposed iterative framework update the similarity weights and the estimated values for higher accuracy. The experimental results show that the proposed INLM produces better results than state-of-art methods over a wide range of scenes both subjectively and objectively, and it is robust to the detection results. Ó 2016 Elsevier Inc. All rights reserved.
1. Introduction Digital images are usually contaminated by S&P noise which is introduced during image acquisition, transmission and recording process. S&P noise ruins image signal in terms of short-duration, discontinuous, noise spikes, which severely affects the subsequent image processing operations. Hence, S&P noise removal is crucial in digital image processing. Many filtering techniques had been proposed for this specific problem. The most popular and robust nonlinear filters were median filter (MF) [1] and its variations [2–4]. However, these kinds of filters treated all pixels with the same manner and replaced all the noise free pixels with the estimated values. As the result, the details of the image would be damaged, especially in the case of low signal noise ratio (SNR). To avoid this situation, various switching based filters, which introduced noise detection process before image restoration were proposed, such as [5–21]. Among these techniques, boundary discriminative noise detection filter (BDND) [12], modified BDND (MBDND) [13] and the sorted switching median filter (SSMF) [21] achieved widely acknowledged for the abilities of protecting the image detail information. Another kinds of methods were committing to find more effective filters. Yuksel [22], Toh and Isa [23], and Civicioglu [24] introduced a kind of hybrid filters, which combined median filter with other process, such as edge detector and neuro-fuzzy network in the denoising q
This paper has been recommended for acceptance by Hongliang Li.
⇑ Corresponding author.
E-mail addresses: [email protected] (X. Wang), [email protected] (S. Shen), [email protected] (G. Shi), [email protected] (Y. Xu), [email protected] (P. Zhang). http://dx.doi.org/10.1016/j.jvcir.2016.03.024 1047-3203/Ó 2016 Elsevier Inc. All rights reserved.
procedure, Morillas et al. [25], Camarena et al. [26], Morillas et al. [27], Smolka and Chydzinski [28], and Smolka [29]used peer group filters and fuzzy metrics for color images denoising. Recently, to get better denoising effect, many methods had been proposed based on a kind of new ideas that took full advantage of the different characteristics of the image and noise, and obtained a good denoising effect. For example, Zuo et al. [30] introduced a denoising procedure based on noise space characteristic, Zhang and Xiong [15] used image directional difference detector on the basis of adaptive weighted mean filter (SAWM). Zhou [31] proposed a novel effective filter based on the cloud modeling (CM) which used the uncertainties of the noise. By taking different kinds of characteristic into account, these kinds of filters restored the image with better performance and achieved wide acception. Although existing techniques have been widely accepted and achieved lots of remarkable results, there is still much room for improvement. Classic techniques in image S&P noise removal strongly rely on exploiting information in a local window around the estimated pixel for this specific restoration task. However, considering the diversity singularity and nonstationary feature of the image signal in a local window, the estimation result could easily diverge from the true value and cause ugly visual effects in texture and edge regions, especially in the case of high noise density. Therefore, local information is insufficient for high quality image restoration. It has been proved and widely accepted that one can expect better denoising performance by exploiting the nonlocal information of the image signal in the case of gaussian distributed noise suppression [32,33]. This inspired us that can we achieve better restoration performance in S&P noise removal by exploiting image non-local information during the denoising procedure? In
X. Wang et al. / J. Vis. Commun. Image R. 38 (2016) 440–450
this paper we will discuss this question and propose an iterative nonlocal means filter (INLM) for image S&P noise removal. The concept of nonlocal means filter (NLM) is based on the fact that there exists lots of similar patches with repeat patterns in natural images, and the central pixels of these similar patches share the same gray value distribution. Instead of estimating the pixel value within a local window, NLM replaces the considered pixel by the weighted mean of all the similar patterns’ central pixels. NLM has been widely accepted in image Gaussian noise suppression, however, its application in S&P noise removal has been barely discussed. In this paper, we will integrate the NLM algorithm into a S&P noise removal problem. Moreover, an iterative weighted average scheme is carried out and an iterative nonlocal means filter is proposed for further improvement. The simulation results show that the proposed INLM is capable of recovering image fine details and textures reliably at a wide range of S&P noise densities so that it outperforms other state-of-art denoising methods in terms of both subjective and objective qualities. The outline of this paper is organized as follows. In Section 2, we analyze the characteristics of both Gaussian noise and S&P noise, and explain why the NLM algorithm for Gaussian noise filtering can be integrated into a S&P noise suppression problem. In Section 3, we will develop the proposed INLM for S&P noise removal in three stages. In Section 4, experimental results of our method and a comparison study with some state-of-art techniques are presented. Finally, we conclude the paper in Section 5.
441
The amplitude and distribution characteristics of these two kinds of noise are so different that the processing methods on these two kinds of noise in principle are also different from each other. However, it occurs to us that what is the amplitude and distribution characteristic of S&P noise after prefiltering? To find out the answer to this question, we implemented switching based median filter (SMF, introduced in Section 3, Part 1) on S&P noised images and analyzed the histogram of the noisy pixels after filtering. An example of the noise histogram of Lena is given in Fig. 2(a). In this example, the testing image corrupted by S&P noise with different noise densities was filtered by SMF. The noise histogram after prefiltering is represented by green curves in Fig. 2(a). The red line in the figure represents the Gaussian distribution curve. We find that the noise histogram of each pre-filtered image approximately obeys Gaussian distribution. To make the discovery more convincing, we tested a bunch of images and showed the noise histogram after prefiltering in Fig. 2(b). It can be seen that the green curves are all very close to the red curve, which means that after SMF, the model of the noise remained in the image is very close to Gaussian distribution. This inspires us that can we achieve better recovering denoising performance by successfully fitting a Gaussian noise filter into an impulse noise removal problem? In this paper, we will discuss about this question and give an affirmative answer by applying NLM filter into S&P noise removal.
2.2. NLM algorithm for Gaussian noise filtering 2. Noise models and NLM for image denoising 2.1. Noise model analysis In the process of image acquisition, transmission and storage, different mechanisms would introduce different types of noise. The two most common noise models are Gaussian noise and S&P noise. Gaussian noise is a kind of additive noise whose amplitude obeys Gaussian distribution as follows: ðnlÞ2 1 pðnÞ ¼ pffiffiffiffiffiffiffi e 2r2 ; 2pr
ð1Þ
where n denotes the noise value, l and r denote the noise mean and standard deviation respectively. S&P noise is another commonly encountered noise type which appears as black and white spots randomly distributed over the image. When an image is corrupted by S&P noise, only a certain portion of the image pixels are replaced with noise values while the remaining pixels are unchanged. In this case, the values of the noisy pixels have either maximum value (rmax ) or minimum value (rmin ) of the image intensity range. The S&P noise model with probability q can be defined as follows:
8 with probability a > < r max ; Xði; jÞ ¼ r min ; ; with probability b > : xði; jÞ; with probability 1 q
ð2Þ
where Xði; jÞ denotes the luminance values of the noisy image at the location ði; jÞ, and xði; jÞ denotes noise-free pixel values at the location ði; jÞ with probability 1 q, and a þ b ¼ q. For further analysis, we visualize the histogram of those noise model and their influence on a clear image. Fig. 1(a) shows a histogram image of a testing image Lena. Fig. 1(b) and (c) are the histogram of noised image contaminated by S&P noise and Gaussian noise respectively. In Fig. 1(b), S&P noise falls into the red1 rectangle. And the histogram of Gaussian noise is given in Fig. 1(d). 1 For interpretation of color in Fig. 1, the reader is referred to the web version of this article.
The NLM algorithm was proposed to process images contaminated by Gaussian noise. For each pixel to be estimated, the NLM takes into consideration the similarity between the neighborhood configuration of this pixel and all the pixels in every neighborhood. ^ i;j can be computed as a Mathematically, the estimated pixel X weighted average of all the neighborhood pixels in the noisy image as follows:
P k;l2Xi;j wi;j;k;l Y k;l ^ i;j ¼ P X ; k;l2Xi;j wi;j;k;l
ð3Þ
where Xi;j is the patch group similar to the current patch centered ^ i;j ; Y k;l is the center pixel of the similar patches in Xi;j . The canby X didates in Xi;j are similar patches whose Euclidean distance is close enough to the current patch centered by X i;j . In Eq. (3), weights wi;j;k;l defines the weight for each candidate Y k;l , which can be calculated as follows:
wi;j;k;l ¼ e
kPðX i;j ÞPðY k;l Þk2 2;a h2
;
ð4Þ
where PðX i;j Þ and PðY k;l Þ represent the local windows centered by pixel X i;j and Y k;l in the noised image. And h is the smoothing parameter, which controls the decay of the exponential function. The norm used in Eq. (4) is simply the Euclidean difference, weighted by Gaussian kernel of zero mean and variance a. NLM has achieved satisfactory performance in Gaussian noise suppression because during denoising, not only local statistical information but also global structure similarity are taken into consideration. However, it could not be implemented in S&P noise removal directly because pixels corrupted by S&P noise are significantly different from their neighbors. This will lead to significant mistakes during similarity weights calculation. To solve this problem, in this paper, a new framework is constructed. The S&P noise are first pre-filtered via statistical filter and then NLM is implemented on noised pixels iteratively to refine the denoising results.
442
X. Wang et al. / J. Vis. Commun. Image R. 38 (2016) 440–450
Fig. 1. Histograms of original image and noised image on lena. (a) Original image. (b) Noised image contaminated by 20% S&P noise. (c) Noised image contaminated by Gaussian noise of l ¼ 30, r ¼ 20. (d) Gaussian noise of l ¼ 30, r ¼ 20.
Fig. 2. Noise histogram of images after prefiltering. (a) Lena. (b) A bunch of images.
443
X. Wang et al. / J. Vis. Commun. Image R. 38 (2016) 440–450
Fig. 3. The flowchart of the INLM.
framework is used to approximate the optimal value. The flowchart of the INLM is shown as in Fig. 3.
Table 1 Comparison of PSNR values on different iteration times (dB). image
Pre de
INLM 1
INLM2
INLM3
INLM4
Lena Monarch Rice Parrot
32.41 32.05 34.61 33.61
34.46 32.97 35.24 34.40
34.77 33.51 35.60 34.93
34.85 33.72 35.68 35.12
34.89 33.83 35.74 35.29
3. Proposed INLM for S&P noise removal The proposed method mainly contains three stages. (1) A preprocessing procedure based on noise detection is performed to construct a noise map N map and an intermediate pre-processed result Pre de is obtained by using switching based mediain filter (SMF) combined with the location information marked in N map. (2) Patch matching procedure in NLM is performed on and only on each noisy pixel centered local window. (3) An iterative
3.1. Pre-processing (SMF) In this section, SMF is used for enhancing the performance of S&P noise, which first marks the pixels as noisy and noise-free pixels and then performs filtering only on the noisy pixels. Moveover, as we have introduced in Section 2.1, S&P noise model is formulated based on the assumption that the amplitudes of noisy pixels fall into two small intervals around the two ends of the image histogram. In our proposed method, only noise pixels are filtered while noise-free pixels are left unchanged. According to the noise detection result, it uses ‘1’ and ‘0’ to represent the positions of noise-free and noisy pixels, respectively. Therefore, noise map N mapcan be formed as follows:
N mapði; jÞ ¼
1 min þb1 6 X i;j 6 max b2 0
otherwise
Fig. 4. Lena’s hat. (a) Original image. (b) Pre de after SMF. (c) INLM1 . (d) INLM4 .
:
ð5Þ
444
X. Wang et al. / J. Vis. Commun. Image R. 38 (2016) 440–450
Table 2 PSNR of INLM with different detecting precision. Detecting precision
b1 =b2 =5
b1 =b2 =10
b1 =b2 =15
b1 =b2 =20
Boat
(FA, MD) INLM1 (dB) INLM5 (dB)
(0,0) 29.4846 30.0381
(1,0) 29.4834 30.0372
(24,0) 29.4762 30.0290
(996,0) 29.4664 30.0184
Baboon
(FA, MD) INLM1 (dB) INLM5 (dB)
(48,0) 21.2785 21.3911
(86,0) 21.2634 21.3731
(166,0) 21.2350 21.3423
(331,0) 21.1688 21.2648
In this formula, ½min; max describes the dynamic range of the digital image. For the commonly used 8-bit testing images, ½min; max ¼ ½0; 255. b1 and b2 are two positive numbers, which determine the length of the intervals around the two ends of the histogram. In this paper, we set b1 ¼ b2 ¼ 5. All noisy pixels are estimated by the median of the uncontaminated ones from its neighborhood in a local window. Let Pre deði; jÞand Xði; jÞ denote the pixels of the prefiltered image and the noisy image at the location ði; jÞ, respectively. And let Xðk; lÞ be a set of pixels in the window of size ð2s þ 1Þ ð2s þ 1Þ centered at the location ðk; lÞ. We can write Xðk; lÞ as:
Fig. 5. Testing images. (a) Original images. (b) Noised images contaminated by 70% S&P noise.
X. Wang et al. / J. Vis. Commun. Image R. 38 (2016) 440–450
445
Fig. 6. Denoised images recovered from 70% S&P noise contaminated testing images. From top left to bottom right: Lena (34.93 dB), Boat (29.95 dB), Monarch (33.92 dB), Pepper (31.08 dB), Baboon (21.59 dB), Rice (35.77 dB), Parrot (35.39 dB) and Port (34.15 dB).
Table 3 PSNR results of restored images with 70% S&P noise (dB). Testing image
IBINR [34]
MBDND [13]
SAWM [15]
SDTF [30]
SSMF [21]
CM [31]
INLM
Lena Boat Monarch Pepper Baboon Rice Parrot Port
30.82 27.86 29.56 29.15 21.67 31.95 31.39 32.17
32.75 27.98 30.50 30.28 21.12 32.12 30.84 32.64
32.92 28.08 30.87 30.40 21.15 32.78 29.40 32.87
32.41 27.78 30.13 29.76 20.75 32.19 32.21 32.55
31.77 27.17 28.95 29.23 20.80 31.30 30.35 31.58
32.72 27.99 30.52 30.00 20.91 32.52 32.67 32.84
34.93 29.95 33.92 31.08 21.59 35.77 35.39 34.15
These bold values demonstrate the denoising performance of the proposed INLM.
Xðk; lÞ ¼ fk 2 ði s; i þ sÞ; l 2 ðj s; j þ sÞg:
ð6Þ
Then, a prefiltering procedure is performed by utilizing the local statistics informating as follows:
( Pre deði; jÞ ¼
median ðXðk; lÞÞ N mapði; jÞ ¼ 0
mapðk;lÞ–0
Xði; jÞ
N mapði; jÞ ¼ 1
:
ð7Þ
Although the detection procedure is very simple, INLM is robust to the detection results. We will discuss the detection results detailedly in the experimental section. 3.2. Patch matching
feature of the original image, the precision of the noisy pixel estimation is poor. This will affect the restoration precision and lead to obvious jagged artifacts around large scale edges. In this paper, we propose to implement the weight calculation and weighted averaging iteratively to pursue better denoising performance. The algorithm flow is described as follows: Algorithm. Iterative NLM for S&P noise removal (1) Initialization (a) Initialize the input image using the prefiltered result ð0Þ
X i;j ¼ Pre dei;j (b) Construct the similar patch group Xi;j for each noisy pixel; (c) Initialize the similar weight for each noisy pixel and its similar candidates Y k;l in Xi;j ð0Þ
ð0Þ ÞPðY k;l Þjj2 2 i;j h2
(2) Outer loop, t = 1,2,. . . (a) Update the averaging weight ðtÞ
jjPðX
wi;j;k;l ¼ e
ðt1Þ ÞPðY Þjj2 k;l 2 i;j h2
(b) Update the denoised result P ðtÞ w Y k;l2Xi;j i;j;k;l k;l ^ ðtÞ ¼ P X ðtÞ i;j
When using the traditional NLM dealing with Gaussian noise, all pixels are processed in this step. In our method, under the guidance of N map, only the noisy pixels Xði; jÞ in Pre de need to search their similar patch group Xi;j and be processed by INLM. The similar patch searching and similarity measurement are the same as the way carried out in traditional NLM. The similarity between two pixels is measured by evaluating the Gaussian weighted Euclidean distance between two image patches surrounding the pixels as in Eq. (4).
jjPðX
wi;j;k;l ¼ e
k;l2Xi;j
wi;j;k;l
(c) If (MSEðX ðtÞ ; X ðt1Þ Þ < Thr or t > 5), go to step (3). Else, go back to step 2-(a). ^ (3) Output the recovered image X.
In step 2-(c), the expression MSEðX ðtÞ ; X ðt1Þ Þ is the mean square
3.3. Iteratively estimation
error between the two variables X ðtÞ and X ðt1Þ . The parameter Thr is the threshold which determines the termination condition of the iteration. This formular means that when the difference between
Although, the prefiltered result Pre de obtained from switching based median filtering contains the coarse gray scale and structure
X ðtÞ and X ðt1Þ is small enough, this iteration should be stopped. Based on our experience on a bunch of test images contaminated by different noise densities, Thr can be calculated as follows:
446
X. Wang et al. / J. Vis. Commun. Image R. 38 (2016) 440–450
Table 4 SSIM results of restored images with 70% S&P noise. Testing image
IBINR [34]
MBDND [13]
SAWM [15]
SDTF [30]
SSMF [21]
CM [31]
INLM
Lena Boat Monarch Pepper Baboon Rice Parrot Port
0.9630 0.9575 0.9812 0.9733 0.9004 0.9692 0.9820 0.9741
0.9790 0.9644 0.9884 0.9625 0.8936 0.9760 0.9835 0.9783
0.9794 0.9645 0.9889 0.9639 0.8943 0.9761 0.9840 0.9810
0.9793 0.9654 0.9897 0.9631 0.8930 0.9778 0.9890 0.9823
0.9734 0.9544 0.9829 0.9621 0.8762 0.9696 0.9809 0.9739
0.9784 0.9656 0.9893 0.9627 0.8954 0.9778 0.9638 0.9813
0.9844 0.9760 0.9927 0.9808 0.8993 0.9783 0.9902 0.9834
These bold values demonstrate the denoising performance of the proposed INLM.
Thr ¼
MSEðX ð0Þ ; X ð1Þ Þ : 20
ð8Þ
It should be noticed that the most time consuming part of the non-local means algorithm is searching similar windows in the image. However, in our method, the iterative part only updates the similarity weight and calculates the weighted average value. Therefore, the computational complexity of this part is negligible compared with the similar patch searching procedure. To show the effectiveness of the iteration procedure, INLM is implemented on four commonly used 512 512 testing images Lena, Monarch, Rice and Parrot, which are contaminated by 70% S&P noise. The peak signal-to-noise ratio (PSNR) evaluation of our method with different iteration times is summarized in Table 1. In Fig. 4, we visualize the effect of INLM by zooming into Lena’s hat. In Table 1 and Fig. 4, INLMi means the iteration was implemented i times. It can be seen from the experimental results that the restoration performance improved significantly with the iteration filtering both subjectively and objectively. 4. Experiments and discussions In this section, we assess the performance of the proposed INLM by carring out a number of experiments, and in NLM, we set the smoothing parameter h = 2, the similar patch size K = 3 (the actual size is ð2K þ 1Þ ð2K þ 1Þ), and the searching window W = 10 (the actual size is ð2W þ 1Þ ð2W þ 1Þ). In order to have a better understanding of the performance of INLM, our experiments are divided into three parts. The first part is to analyze the noise detecting performance, which is discussed to show the robustness of INLM to the detecting false alarm. The second part is image denoising performance exhibition and comparison with some selected state-of-art filters. The third part is the computational complexity analysis.
the histogram, which is defined by b1 and b2 . And larger b1 and b2 lead to more FA, vice versa. To further prove the robustness of INLM with respect to noise detecting accuracy, an example of the INLM with different detecting accuracy by setting different b1 and b2 on Boat and Baboon contaminated by 70% S&P noise is given in Table 2. It can be seen that the decrease of denoising performance with the increasing of FA is very inconspicuous. Therefore, in INLM the choice of b1 and b2 is not crucial. 4.2. Comparison of image denoising performance In this section, we verify our algorithm via experimental simulations. We test the proposed INLM on a bunch of popular testing images (Lena, Boat, Monarch, Pepper, Baboon, Rice, Parrot) and a remote sensing image Port as shown in Fig. 5(a). We carry out the experiments in three ways to make a comprehensive comparison. First, all testing images contaminated by 70% S&P noise are processed by INLM and other well-known filters such as the interpolation-based denoising (IBINR, 2013) [34], the modified BDND (MBDND, 2013) [13], the switching-based adaptive weighted mean filter (SAWM, 2009) [15], the noise space characteristic based filter (SDTF, 2013) [30], the sorted switching median filter (SSMF, 2013) [21] and the cloud model based denoising (CM, 2012) [31]. We use peak signal to noise ratio (PSNR) and structural similarity index (SSIM) as the measure of evaluation to compare the denoising performance of each method. Second, we show the restoration performance of our method in different noise densities. We compare the average denoising results (PSNR) of each denoising method for 100 testing images which are downloaded from IMAGE NET2 with noise density ranging from 10% to 90%. Moreover, a testing image Monarch is used to exhibit the visual effect of our method for different noise densities. Third, we show the denoising results of INLM in an extreme case, in which the noise level goes up to 90%.
4.1. Analysis of noise detecting performance The noise detection is very important in all switching based S&P noise removal strategies. The accuracy of the detecting method affects the denoising performance severely. So far, the bottleneck of noise detection is how to decrease false alarm (FA), while miss detection (MD) barely exists. Lots of efforts have been paid for decreasing FA. However, high accuracy normally comes from elaborately designed detecting strategy with high computational complexity. Moreover, to the best of our knowledge, no method in published literatures can decrease the FA to zero. Fortunately, in the proposed INLM, this problem can be avoided because INLM is quite robust to FA. In our method, the value of estimated pixels is calculated by a group of pixels from the similar patches. Therefore, a few lacks of the group candidates does not influence the solution accuracy obviously. During histogram analyzing, the noise detecting accuracy is determined by the length of the intervals around the two ends of
4.2.1. Denoising performance with 70% S&P noise In this part, the method is tested on images contaminated by 70% S&P noise. The corrupted images are shown in Fig. 5(b), and the denoised results of INLM for each image are shown in Fig. 6. It can be seen that our method can effectively reconstruct clear images from the high density noise corrupted images. For comparison, these noised images are also processed by the six state-of-art denoising methods mentioned above. The PSNR and SSIM evaluation of each denoising method are shown in Tables 3 and 4. For subjective comparison, the denoising results of filtered Lena and Parrot are shown in Figs. 7 and 8. And we compare the detail visual effects of different methods for 70% S&P noise by zooming into the faces of Lena and Parrot. It can be seen from the experimental results that INLM outperforms those state-ofart filters both objectively and subjectively. 2
http://www.image-net.org/.
X. Wang et al. / J. Vis. Commun. Image R. 38 (2016) 440–450
447
Fig. 7. Denoising results of Lena contaminated by 70% S&P noise from different filters. From top left to bottom right: original image, IBINR (30.82 dB), MBDND (32.75 dB), SAWM (32.92 dB), SDTF (32.41 dB), SSMF (31.77 dB), CM (32.72 dB) and INLM (34.93 dB).
Fig. 8. Denoising results of Parrot contaminated by 70% S&P noise from different filters. From top left to bottom right: original image, IBINR (31.39 dB), MBDND (30.84 dB), SAWM (29.40 dB), SDTF (32.21 dB), SSMF (30.35 dB), CM (32.67 dB) and INLM (35.39 dB).
In order to make the experimental results more convincing, we introduce another evaluation index by analyzing the amplitude difference between the original image and the restored image. A difference image dif im is constructed by subtracting the denoised image de im from the original image or im as shown in Eq. (9):
dif im ¼ jor im de imj:
ð9Þ
For a successful impulse denoising method, the recovered result should be as close to the original image as possible. This point has two fold meanings: (1) in dif im, the number of the elements with large scale amplitude should be as small as possible; (2) the
maximum value of dif im should be as small as possible. Therefore, the amplitude distribution of dif im is analyzed in this part. In each dif im, we summarize the number of the pixels whose intensities fall into the ranges of [30, 40), [40, 50) and [50, 255] to show the estimating error of each method in a statistical way. In addition, the maximum value (Max dif ) of each dif im is also recorded for comparison. Lena, Parrot and Monarch with 70% S&P noise are used as testing images. The comparison results are shown in Table 5. It can be seen that INLM outperforms other denoising techniques by producing both least large scale pixels and smallest Max dif in dif im.
448
X. Wang et al. / J. Vis. Commun. Image R. 38 (2016) 440–450
Table 5 Amplitude distribution analysis of dif im. Filters
IBINR MBDND SAWM SDTF SSMF CM INLM1 INLM5
Difference level 30–40 40–50 >50 30–40 40–50 >50
30–40 40–50 >50
Lena
Monarch
933 581 517 503 682 475 347 330
Parrot 297 232 192 216 278 185 131 118
158 1058 160 497 98 484 161 550 202 842 168 508 66 361 60 311
549 275 225 282 654 257 216 145
510 434 497 469 974 2855 244 171
1856 1057 941 1029 1237 923 649 601
880 640 510 607 782 544 324 254
553 691 514 668 1056 621 267 190
These bold values demonstrate the denoising performance of the proposed INLM.
4.2.2. Denoising performance in different noise densities In this part, testing images are contaminated by a large range of noise densities (10–90%). For the purpose of comprehensive comparison, we visualize the denoising performance of each method by plotting the average PSNR values obtained from 100 denoised testing images corrupted by different noise densities, as shown in Fig. 9. It can be seen that the proposed method outperforms other state-of-art filters by having the highest PSNR values in different noise levels. Except for the average PSNR comparison, an example of the proposed INLM on Monarch is taken to show the denoising results in different noise densities (10%, 30%, 50%, 70%, 90%) in Fig. 10. Note that the proposed method is still capable of restoring the images with good visual effects even in strongly corrupted cases. Moreover, we also make detail objective comparison with other methods. Table 6 shows all denoising performances of each denoising technique for Monarch with different noise levels (10–90%). 4.2.3. Denoising results under extreme cases Finally, to further demonstrate the excellent performance of the proposed INLM, the denoising results on images Pepper, Baboon, Rice and Port contaminated by 90% S&P noise are given in Fig. 11. In such an extreme case, the restored images have some blurring edges in some local areas, however, the proposed INLM is still capable of restoring the images with good visual effects. 4.3. Computational complexity
Fig. 9. Graph of average denoised PSNR versus S&P noise density obtained by using a total of 100 testing images.
NLM contains two steps, similar patch matching and weighed mean of centered pixels in similar patches. In NLM, if we set the search window of 21 21 , and the similar patch of 7 7, for an image with N pixels, the computational cost of similar patch matching is 49 441 N, and the computational cost of weighed mean procedure is n N (n 6 441 is the number of the most similar patches). It is clear that the major computational cost of NLM comes from the similar patch matching. In our method, although iterative frame is used, the iterative procedure only contains the weighed mean of centered pixels in similar
Fig. 10. Denoising results of Monarch in different noise densities. From top left to bottom right: original image, noise density equals to 10%, 30%, 50%, 70% and 90%.
449
X. Wang et al. / J. Vis. Commun. Image R. 38 (2016) 440–450 Table 6 PSNR comparison for Monarch in noise density 10–90% (dB).
IBINR MBDND SAWM SDTF SSMF CM INLM
10
20
30
40
50
60
70
80
90
43.43 46.15 46.40 45.64 46.24 46.61 48.35
40.12 41.56 42.35 40.91 41.29 42.20 44.41
37.89 38.71 39.38 37.95 38.21 39.18 41.39
34.71 36.25 37.08 35.66 35.63 36.79 39.85
33.60 34.14 34.88 33.66 33.26 34.56 37.90
32.16 32.21 32.92 31.86 31.13 32.50 35.87
29.56 30.50 30.87 30.13 28.95 30.52 33.85
27.09 28.13 28.40 27.83 26.58 28.15 30.57
24.17 25.02 25.02 24.54 22.81 24.98 26.46
These bold values demonstrate the denoising performance of the proposed INLM.
Fig. 11. Denoised images recovered from 90% S&P noise. From left to right: Pepper (26.42 dB), Baboon (18.80 dB), Rice (29.40 dB) and boat (25.13 dB).
patches. This means, the most computational consuming procedure, similar patch matching, was carried out only once for each image. Therefore, the computational complexity of our method is 49 441 N þ I n N (I 6 5, is the number of iterations). 5. Conclusion In this paper, we have demonstrated the effectiveness of our proposed INLM algorithm with weight update for S&P noise removal. Instead of local estimation, non-local information is exploited to preserve the image detail. After similar patch searching, the weight and denoising result are calculated iteratively. Experiments showed that both the visual quality and evaluation indexes of the proposed algorithm outperform those state-of-art denoising techniques. Acknowledgements This work is supported by NSF of China (Nos. 61401333, 61401325, 61201289, 61372071, 61100155), Shaanxi province natural science foundation of China (No. 2014JQ8296), Research Fund for the Doctoral Program of Higher Education of China (No. 20130203120009), and Fundamental Research Funds for the Central Universities of China (Nos. JB140227, K5051302096, JB140207), and the National Defense Fund (No. 9140C610304150C61268). References [1] R.C. Gonzalez, R.E. Woods, Digital Image Processing, Prentice-Hall, Englewood Cliffs, NJ, 2002. [2] D.R.K. Brownrigg, The weighted median filter, Commun. ACM 27 (8) (1984) 807–818. [3] T. Chen, H.R. Wu, Space variant median filters for the restoration of impulse noise corrupted images, IEEE Trans. Circ. Syst. II: Anal. Digit. Signal Process. 48 (8) (2001) 784–789. [4] S.J. Ko, Y.H. Lee, Center weighted median filters and their applications to image enhancement, IEEE Trans. Circ. Syst. 38 (9) (1991) 984–993. [5] H.L. Eng, K.K. Ma, Noise adaptive soft-switching median filter, IEEE Trans. Image Process. 10 (2) (2001) 242–251. [6] G. Pok, J.C. Liu, A.S. Nair, Selective removal of impulse noise based on homogeneity level information, IEEE Trans. Image Process. 12 (1) (2003) 85– 92.
[7] S. Schulte, M. Nachtegael, V. De Witte, D. Van der Weken, E.E. Kerre, A fuzzy impulse noise detection and reduction method, IEEE Trans. Image Process. 15 (5) (2006) 1153–1162. [8] W.B. Luo, Efficient removal of impulse noise from digital images, IEEE Trans. Consum. Electron. 52 (2) (2006) 523–527. [9] I. Aizenberg, C. Butakoff, D. Paliy, Impulsive noise removal using threshold Boolean filtering based on the impulse detecting functions, IEEE Signal Process. Lett. 12 (1) (2005) 63–66. [10] V. Crnojevic, V. Senk, Z. Trpovski, Advanced impulse detection based on pixelwise MAD, IEEE Signal Process. Lett. 11 (7) (2004) 589–592. [11] P. Civicioglu, M. Alci, E. Besdok, Impulsive noise suppression from images with the noise exclusive filter, EURASIP J. Appl. Signal Process. 16 (2004) 2434– 2440. [12] P.E. Ng, K.K. Ma, A switching median filter with boundary discriminative noise detection for extremely corrupted images, IEEE Trans. Image Process. 15 (6) (2006) 1506–1516. [13] I.F. Jafar, R.A. AlNa’mneh, K.A. Darabkh, Efficient improvements on the BDND filtering algorithm for the removal of high-density impulse noise, IEEE Trans. Image Process. 22 (3) (2013) 1223–1232. [14] Z.F. Deng, Z.P. Yin, Y.L. Xiong, High probability impulse noise-removing algorithm based on mathematical morphology, IEEE Signal Process. Lett. 14 (1) (2007) 31–34. [15] Xu Ming Zhang, You Lun Xiong, Impulse noise removal using directional difference based noise detector and adaptive weighted mean filter, IEEE Signal Process. Lett. 16 (4) (2009) 295–298. [16] K.S. Srinivasan, D. Ebenezer, A new fast and efficient decision-based algorithm for removal of high-density impulse noises, IEEE Signal Process. Lett. 14 (3) (2007) 189–192. [17] P. Wei, J.L. Li, D.M. Lu, G. Chen, A fast and reliable switching median filter for highly corrupted images by impulse noise, in: Proc. IEEE Int. Symposium on Circuits and Systems (ISCAS’07), 2007, pp. 3427–3430. [18] W. Zhou, D. Zhang, Progressive switching median filter for the removal of impulse noise from highly corrupted images, IEEE Trans. Circ. Syst. II: Anal. Digit. Signal Process. 46 (1) (1999) 78–80. [19] V. Gupta, V. Chaurasia, M. Shandilya, Random-valued impulse noise removal using adaptive dual threshold median filter, J. Vis. Commun. Image Represent. 26 (9) (2015) 296–304. [20] Guorong Gao, Yanping Liu, Demetrio Labate, A two-stage shearlet-based approach for the removal of random-valued impulse noise in images, J. Vis. Commun. Image Represent. 32 (12) (2015) 83–94. [21] Shi Jinn Horng, Ling Yuan Hsu, Tian Rui Li, Shao Jie Qiao, Xun Gong, Hsien Hsin Chou, Muhammad Khurram Khan, Using sorted switching median filter to remove high-density impulse noises, J. Vis. Commun. Image Represent. 24 (7) (2013) 956–967. [22] M.E. Yuksel, A hybrid neuro-fuzzy filter for edge preserving restoration of images corrupted by impulse noise, IEEE Trans. Image Process. 15 (4) (2006) 928–936. [23] K.K.V. Toh, N.A.M. Isa, Noise adaptive fuzzy switching median filter for saltand-pepper noise reduction, IEEE Signal Process. Lett. 17 (3) (2010) 281–284. [24] P. Civicioglu, Using uncorrupted neighborhoods of the pixels for impulsive noise suppression with ANFIS, IEEE Trans. Image Process. 16 (3) (2007) 759– 773.
450
X. Wang et al. / J. Vis. Commun. Image R. 38 (2016) 440–450
[25] S. Morillas, V. Gregori, G.P. Fajarnes, Local self-adaptive fuzzy filter for impulse noise removal in color images, IEEE Trans. Signal Process. 88 (2) (2008) 390– 398. [26] J.G. Camarena, V. Gregori, S. Morillas, A. Sapena, Fast detection and removal of impulsive noise using peer groups and fuzzy metrics, J. Vis. Commun. Image Represent. 19 (1) (2008) 20–29. [27] S. Morillas, V. Gregori, G.P. Fajarnes, A. Sapena, New adaptive vector filter using fuzzy metrics, J. Electron. Imag. 16 (3) (2007) 1–15. [28] B. Smolka, A. Chydzinski, Peer group switching filter for impulse noise reduction in color images, Pattern Recogn. Lett. (2010) 489–495. [29] B. Smolka, Peer group filter for impulse noise removal in color images, Proc. CIARP’08 (2008) 699–707.
[30] Zhi Yong Zuo, Tian Xu Zhang, Jing Hu, Gang Zhou, A new method for removing impulse noise based on noise space characteristic, Optik – Int. J. Light Electron Opt. 124 (18) (2013) 3503–3509. [31] Zhe Zhou, Cognition and removal of impulse noise with uncertainty, IEEE Trans. Image Process. 21 (7) (2012) 3157–3167. [32] T. Brox, O. Kleinschmidt, D. Cremers, Efficient nonlocal means for denoising of textural patterns, IEEE Trans. Image Process. 17 (7) (2008) 1083–1092. [33] A. Buades, B. Coll, J. Morel, A review of image denoising algorithms, with a new one, SIAM Interdisc. J., Multiscale Model. Simul. 4 (2) (2005) 290–530. [34] C. Kalyoncu, O. Toygar, H. Demirel, Interpolation-based impulse noise removal, IET Image Process. 7 (8) (2013) 777–785.