- Email: [email protected]
Image denoising feedback framework using split Bregman approach
Accepted Manuscript
Image Denoising Feedback Framework Using Split Bregman Approach Jeong Heon Kim, Farhan Akram, Kwang Nam Choi PII: DOI: Reference:
S0957-4174(17)30433-5 10.1016/j.eswa.2017.06.015 ESWA 11386
To appear in:
Expert Systems With Applications
Received date: Revised date: Accepted date:
10 March 2017 10 June 2017 11 June 2017
Please cite this article as: Jeong Heon Kim, Farhan Akram, Kwang Nam Choi, Image Denoising Feedback Framework Using Split Bregman Approach, Expert Systems With Applications (2017), doi: 10.1016/j.eswa.2017.06.015
This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT
Highlights • An image denoising feedback framework using split Bregman method is
CR IP T
proposed. • A feedback function is used in an iterative manner to minimize the error.
• In the denoising process, the proposed method preserves edges affected by the noise.
AC
CE
PT
ED
M
AN US
• The proposed method is tested on both color and range images.
1
ACCEPTED MANUSCRIPT
Image Denoising Feedback Framework Using Split Bregman Approach
CR IP T
Jeong Heon Kima , Farhan Akramb , Kwang Nam Choic,∗ a National
Institute of Supercomputing and Networking, Korea Institute of Science and Technology Information, Daejeon, Korea b Department of Computer Engineering and Mathematics, Rovira i Virgili University, Tarragona, Spain c Department of Computer Science and Engineering, Chung-Ang University, Seoul, Korea
AN US
Abstract
In this paper, an image denoising feedback framework is proposed for both color and range images. The proposed method works on an error minimization principle using split Bregman method. At first image is denoised by computing means in the local neighborhood. The pixels that have big differences from the center
M
of the local neighborhood compared to the noise variance are then extracted from the denoised image. There is a low correlation between the extracted pixels and their local neighborhood. This information is fed to the feedback
ED
function and denoising is performed again, iteratively, to minimize the error. In most cases, the proposed framework yields best results both qualitatively and quantitatively. It shows better denoising results than the bilateral filtering when
PT
the edge information in the input images is affected by intense noise. Moreover, during the denoising process feeback function ensures that the edges are not
CE
over smoothed. The proposed framework is applied to denoise both color and range images, which shows it works effectively on a wide variety of images unlike the evaluated state-of-the-art denoising methods.
AC
Keywords: Image denoising, split Bregman, bilateral filtering, image derivative, color image, range image
∗ Corresponding
author Email addresses: [email protected] (Jeong Heon Kim), [email protected] (Farhan Akram), [email protected] (Kwang Nam Choi)
Preprint submitted to Expert Systems with Applications
June 15, 2017
ACCEPTED MANUSCRIPT
Responses to the reviewer’s comments Reviewer 1
CR IP T
Comment The authors answered most of my questions. However, the novelty of this 5
paper is still unclear to me (Please see the general comments to your first submission). The authors need to explain that in the Introduction. Response
AN US
We have added a new paragraph in introduction section, which states the major contributions of the proposed framework.
Line 109 on pp. 5: There are two main contributions of this work. First, a
10
novel denoising method is proposed, which introduces a two-stage convolution based on the split Bregman solution. Second, unlike the traditional methods, it can denoise both color and range images with a simple change in the convolu-
Reviewer 2 Comment
ED
15
M
tional method.
The manuscript has been improved significantly and my concerns have been
PT
addressed. However, I would ask that the authors continue to check their work
CE
for spelling errors before any publication. 20
Response
We have tried to improve the readability of the paper by thoroughly revising
AC
and correcting the grammatical mistakes, which is shown by the highlighted parts in the revised manuscript.
25
We would like to thank both the reviewers for their insightful suggestions
and comments. It helped us a lot to improve the paper. We tried to reply all the comments and questions raised by reviewers and hope now everything is clear. 3
ACCEPTED MANUSCRIPT
1. Introduction Image denoising is an important and long standing preprocessing problem 30
in the areas of computer vision and image processing. It not only provides
CR IP T
a concrete baseline for better information analysis but also bestows information analysts with a better visual experience. It is necessary to have a better
understanding about the level of information to denoise an image while preserving important information. Therefore, the structural components of the image, 35
which contain useful information, should be preserved so that we do not draw an abstract image as a the result of denoising algorithm.
AN US
Numerous implementations of image feature descriptors for information processing are based on image gradient (Lowe, 1999; Mikolajczyk & Schmid, 2005; Dalal & Triggs, 2005; Bay et al., 2008) and structural information (Wolfson & 40
Rigoutsos, 1997; Belongie et al., 2002). An image gradient (Canny, 1986; Deriche, 1987; Smith & Brady, 1997) is one of the important image characteristics,
M
which is used to compute edges or boundaries of objects in an image. In color images, it is computed from the rate of change in an image coordinate system. In turn, in range images, it is computed in three-dimensional camera coordinate system. In this paper, a fine adjustment of image gradient is computed in
ED
45
the camera coordinate system, which positively affects gradient-based feature
PT
descriptors.
Image processing is an area of computer science in which information is extracted and processed from the given images. In the process of a new information generation from images, either external information(Hertzmann et al.,
CE
50
2001), internal information(Kyprianidis et al., 2009) or both are utilized. The elements which contribute in new information generation either employ local
AC
neighborhood (Harris & Stephens, 1988) or entire image domain (Pizer et al., 1987). A new perspective was introduced by changing the image domain into
55
frequency domain (Shensa, 1992). Moreover, the extracted information from the image was further used to revise previously extracted information (Kass et al., 1988). These approaches were applied in the area of denoising and later have
4
ACCEPTED MANUSCRIPT
been diversified (Roth & Black, 2009). In this paper, both color and range images are studied but the main focus is 60
on range image denoising. Range images have a considerable amount of noise.
CR IP T
It depends on the size of the image acquisition device; the smaller a device is, the higher it produces noise. Moreover, the distance estimation methods also generate noise when object is far distanced from the image acquisition device.
Therefore, different types of noise may infuse depending on the distance in a 65
single scene, which is one of the major problems in range images. It is very critical to preserve features, such as image gradient, in the denoising process.
AN US
In the proposed work, well-adjusted gradients are used along with the curvature information to denoise the range images and preserve their edges. A Gaussian filter suppresses noise by convolving a given image with a Gaus70
sian function ( also called Gaussian smoothing operator). Range images that contain considerate amount of noise can be regularized by the Gaussian function. However, a big convolutional window is needed to properly regularize
M
them, which results in high time complexity. Non-data regions that do not contain any useful information worsen the denoising problem. Transforming the data from the range images into data mesh or scatter point set is one of the
ED
75
methods to avoid non-data regions. However, it requires additional processing and mislays the important information during three-dimensional space forma-
PT
tion. Therefore, a starting point restriction is imposed on the diffusion equation to avoid two-dimensional vector-valued processing. Partial differential equations (PDEs) based methods work on a point-based processing mechanism and
CE
80
have high time complexity. Therefore, discretized linear methods are prioritized over PDEs based methods. Another advantage of linear methods is that their
AC
solution does not require any additional parameter adjustment even under the influence of noise.
85
A new perspective to preserve the edges in the color images is discussed
in (Perona & Malik, 1990; Rudin et al., 1992; Weickert, 1996) using diffusionbased denoising techniques. An approach of total variation using split Bregman method is extended by introducing an anisotropic diffusion and the local con5
ACCEPTED MANUSCRIPT
volution mask(Goldstein & Osher, 2009). From the diffusion-based methods 90
discussed above, it can be assumed that there is a sharp edge when the derivative of the denoised image is fairly large.
CR IP T
Based on the above assumption, an image denoising feedback framework for both color and range images is proposed in this paper. For color images denoising, in the first step either isotropic or anisotropic convolution mask is 95
applied for smoothing to remove noise from the edges. After removing noise from the given image, useful information is taken and fed to the feedback function, which calibrates the old convolution mask. The calibrated denoising function is
AN US
then applied to the original image in an iterative manner. On the other hand,
for range image denoising, first smoothing is based on the regularization scheme 100
from (Kim & Choi, 2012), which is improved by a new neighborhood selection using the feedback function. For color images, the feedback function is based on the color distance in the CIE LAB color space. In turn, for range images it is based on a linear system. The proposed method has an advantage over the
105
M
bilateral filtering method when the given image is influenced by the intense noise. With the use of an adapted feedback strategy, it assures better denoising results
ED
and low time complexity compared to non-local approaches. The proposed method targets both color and range images; therefore, it works effectively on a numerous types of images unlike the other state-of-the-art denoising methods.
110
PT
There are two main contributions of this work. First, a novel denoising method is proposed, which introduces a two-stage convolution based on the
CE
split Bregman solution. Second, unlike the traditional methods, it can denoise both color and range images with a simple change in the convolutional method. The paper is organized as follows. Related work is discussed in section 2.
AC
Split Bregman iteration is described in section 3. The feedback approach using
115
the first denoised image is explained in section 4. Color image denoising for the proposed framework is described and respective results are shown in section 5. Range image denoising for the proposed framework is described and respective results are shown in section 6. Finally, the proposed work is concluded in section 7. 6
ACCEPTED MANUSCRIPT
120
2. Related work A range image is a two-dimensional image, which shows the distance points in a scene from a specific point of view. Normally, it is dealt in a three-dimensional
CR IP T
(3D) space. It can be represented as a collection of two-dimensional vector-
valued images, which can be approximated as a collection of two-dimensional 125
color images. Therefore, the color denoising methods can also be used on range
images. In (Katkovnik et al., 2010), the development and classification of image denoising techniques are analyzed using different standards.
Image denoising techniques are classified into local and non-local averaging
130
AN US
methods depending on the distance of the information provider from the coor-
dinates to the estimated point. Local techniques formulate an estimation using close points on the coordinates. The area that represents the close points is normally defined by a local neighborhood with a Gaussian kernel of a particular window size. In turn, the non-local averages are computed throughout the
M
image.
Image denoising techniques are further classified into pointwise and multi-
135
point methods in which number of points are used to make a point estimation.
ED
Pointwise techniques estimate a point by computing a weighted sum. In turn, the multipoint techniques estimate unit blocks by changing the spectrum do-
140
PT
main. The estimated blocks are then clustered (fused) into a single representative block.
Image denoising techniques are further classified into the weighted means and
CE
the partial differential equation (PDE) based solutions. The weighted means techniques calculate an average by multiplying a weight with each participating element. In turn, the PDE based techniques minimize the error in a given image with respect to an artificial time. These types of methods normally have high
AC 145
time complexity and require a lot of iterations to reach an optimal solution. Range images contain a huge amount of unnecessary information, which
is considered noise. It is necessary to detect similar blocks, which are nondata regions, to separate them from useful information. Multipoint methods
7
ACCEPTED MANUSCRIPT
150
require a special process of element aggregation during domain transformation to extract region without information, which can increase the time complexity. Consequently, in this paper, a local pointwise method is employed to separate
CR IP T
non-data regions from regions with useful information. Numerous local-pointwise methods have been devised to control the weight 155
of a Gaussian kernel, for example, sigma-filter (Lee, 1983), Yaroslavsky filter (Yaroslavsky, 1985), and bilateral filter (Tomasi & Manduchi, 1998). These
methods determine the weight by using similarities between pixels of a vectorT
valued image I (x) = (I1 (x) , I2 (x) , I3 (x)) .
160
AN US
Numerous methods have been devised to expand the diffusion equation in the context of image denoising. Perona & Malik (1990) proposed a nonlinear expansion of the diffusion equation. Rudin-Osher-Fatemi (ROF) method deployed a total variation concept for diffusion (Rudin et al., 1992). Weickert (1996) proposed a generalized form of the diffusion equation with a diffusion tensor field.
Nonlinear diffusion processes (Brox et al., 2006; Rudin et al., 1992) are pro-
M
165
posed to preserve image details during image smoothing process. These meth-
ED
ods usually require a huge number of iterations to converge. However, the split Bregman method is an efficient solver for L1-regularized problems such as total variation (TV) denoising (Goldstein & Osher, 2009).
PT
Let I¯ (x) : Ω ⊂ R2 → R be an original image and I an observed image. In order to reach an optimal solution using split Bregman method, an iterative
AC
CE
minimization scheme using the TV model involves the following three steps:
2
2 λ µ Iˆk+1 = min I¯ − I 2 + dkx − ∇x I¯ − bkx 2 ¯ 2 I 2 (1)
2 λ + dky − ∇y I¯ − bky 2 , 2 dk+1 = shrink ∇x Iˆk+1 + bkx , 1/λ , x (2) dk+1 = shrink ∇y Iˆk+1 + bky , 1/λ , y bk+1 = bkx + ∇x Iˆk+1 − dk+1 , x x bk+1 = bky + ∇y Iˆk+1 − dk+1 , y y 8
(3)
ACCEPTED MANUSCRIPT
where Iˆ0 = I, d0x = d0y = b0x = b0y = 0 and λ is a scaling parameter (Goldstein & Osher, 2009). The Gauss-Seidel solution to the optimality condition of (1)
Iˆk+1 = Gk =
k k ˆ ˆ ∆I + 4 I ! + µI λ k k ∇ x d x − bx − + ∇y dky − bky (µ + 4λ)
CR IP T
is:
,
(4)
where ∆ is a discrete Laplace operator without diagonals. The shrinkage operator is defined as:
170
x ∗ max (|x| − γ, 0) , |x|
AN US
shrink (x, γ) =
(5)
The edge direction deficiency in (4) induces edge disintegration. The disintegrated edges lead to staircasing and the image looks like a painting. Several methods have been proposed to avoid staircasing based on orientation (Brox et al., 2006; Tai et al., 2009). Selection of parameters is very critical and plays
175
M
an important role in accurate image denoising, which can vary for different types of images. Numerous algorithms have been proposed to improve the accuracy
et al., 2014).
ED
and predictability of correct parameter selection (Chu & Mak, 2016; Montagner
Orthogonal based functions are used in full-rank high-order approximation
180
PT
for multipoint estimation of each image patch. This multipoint modeling is expressed as a corresponding transform-domain representation. In the local modeling category, the shape-adaptive transform domain filtering developed by
CE
(Foi et al., 2007) is one of most successful filter. The non-local approach is employed in the block-matching and 3D (BM3D) filtering by (Dabov et al.,
AC
2007). Foi et al. (2007) used the intersection of confidence intervals (ICI) rule
185
in choosing scale parameters of each directional window of local polynomial approximation (LPA) with anisotropic support. Anisotropic local polynomial approximation using intersection of confidence
intervals (LPA-ICI) based method in (Foi et al., 2007) builds an adaptive neighborhood Ux+ , which is constructed as the union of supports for the directional 9
ACCEPTED MANUSCRIPT
190
adaptive-scale kernels. The shape adaptive discrete cosine transform (SA-DCT), denoted as TU˜x+ , is computed by the cascaded application of 1D varying-length discrete cosine transform (DCT). DCT first transforms columns and then rows
CR IP T
of the centered adaptive neighborhood U˜x+ = {v|v ∈ Ω , (x − v) ∈ Ux+ } of Ux+ .
The non-local approach works on a principle which starts by finding matched 195
block. Dabov et al. (2007) selects similar blocks and arranges them in 3D arrays IUh,x . The 3D spectrum is calculated by the adopted normalized 3D linear transform T3D from the 3D arrays IUh,x . The global basic estimate is aggregated by the weighted means. BM3D computes the final estimate by using
200
AN US
the values from the basic estimate with the improved grouping and by using the collaborative Wiener filtering. The extension of BM3D algorithm with the shape-adaptive principal component analysis (BM3D-SAPCA) is proposed by (Dabov et al., 2009).
Tasdizen et al. (2002) developed a surface smoothing method, which incorporates both level set surface models and anisotropic diffusion of normals. Anisotropic diffusion is a better denoising method than isotropic diffusion be-
M
205
cause it preserves important image components, such as edges and structural
ED
details, which are very important for image interpolation. The three-dimensional model-based techniques have an additional overhead that originates by discarding the original structure of the range images and converting them into a three-dimensional structure for further processing. More-
PT
210
over, these methods are not designed for the range images from mobile devices,
CE
which are very hard to deal because of various noise levels in a scene. Therefore, open loop smoothing methods which perform single level smoothing yield
AC
unsatisfactory denoising results.
215
3. Split Bregman iteration Image denoising by using the thermal diffusion function can be defined with
a Gaussian filtering. The conceptual elements of an anisotropic diffusion are formulated in the form of bilateral filtering (Tomasi & Manduchi, 1998) and
10
ACCEPTED MANUSCRIPT
Kuwahara filtering (Kyprianidis et al., 2010). It is essential to study a correla220
tion between the methods that employ an iterative process and the ones use a single or fixed loop (open loop) to obtain a solution. In this section, meaning
CR IP T
of an iterative process is discussed and a strategy to formulate such process for the proposed framework is devised.
The split Bregman method is used to solve a variety of L1-regularized prob225
lems, such as total variation regularization. It solves a problem in an iterative
manner using multiple steps. The diffusion equation, which yields Gauss-Seidal
solution with an optimality condition is defined in (4). The discrete Laplace
AN US
operator plays an important role in iteratively solving the diffusion equation. In the diffusion computation, central pixel of the original image is used with 230
an additional (diffusion impeding) term that hinders diffusion. The diffusion impeding term preserves the edges just like a total variation (TV) model does; however, with less overhead. In (2) and (3), a shrink function is used to compute diffusion impeding term, which is defined in (5). A shrink function is zero
235
M
when the input value is smaller than the given parameter. Otherwise, it yields a smaller value than the input value. Where the given parameter comes from
ED
the noise variance. The input values that are greater than a certain threshold compared to the noise variance act as a diffusion impeding term. Furthermore, the shrink function employs the input values from the first derivative of the
240
PT
given image, on which diffusion is already applied in the previous step, instead of using them from the original image. This selection of parameters plays a key
CE
role in the edge preservation step. This process becomes clearer with the step-by-step output images shown in
Fig. 1. Fig. 1(a) shows an image that is regularized by the diffusion term in the
AC
first step. Fig. 1(b) shows that the points selected by the diffusion impeding
245
term impose a crisp look to the denoised image by preserving the sharpness of the edges. Diffusion and inverse diffusion are carried out simultaneously and as a result of successive diffusion, stable values are obtained. In range image denoising, the edge preservation using the split Bregman method can also be used to compute the neighbor points. First derivative image 11
CR IP T
ACCEPTED MANUSCRIPT
(b)
AN US
(a)
Figure 1 TV by using split Bregman method. (a) Regularized image with same amount of smoothness throughout. (b) Regularized image with sharp edges because of diffusion impeding term.
used in the shrink function indicates the closeness of two points in terms of
M
250
color. In the range images, the neighbor points are computed by comparing the closeness of the distance between them.
It ← I0
PT
St ← S0
ED
function f eedback smoothing
for i = 0 to n do
I t ← smoothing I 0 , S t
CE
S t ← f eedback I t , I 0 , S t , S 0 , κ
end for
AC
Figure 2 Image denoising feedback framework.
12
ACCEPTED MANUSCRIPT
4. Feedback strategy The proposed feedback framework has two main components. One compo255
nent maintains and applies a dedicated transformation function on each pixel. In
CR IP T
turn, the second component deals with the smoothing. Where the old smoothing information is fed to the feedback function in the next step. The transformation
value of each pixel is different than its feedback value; therefore, both types of values are managed separately. Let I 0 (x) : Ω ⊂ R2 → R be a noisy input image 260
and S 0 an initial smoothing function. Where the smoothing function is used to
generate both the denoised image I t from the input image I 0 and the current
AN US
smoothing information function S t as shown in Fig. 2. The feedback function formulates an improved smoothing function S t from the initial I 0 , S 0 and the updated I t , S t information. Here, κ is the feedback parameter. This process is 265
repeated for n times (in this paper n = 1).
The proposed framework defines a smoothing function in a large viewpoint
M
environment. It appears similar to bilateral filtering (Tomasi & Manduchi, 1998); however, there is a clear difference in terms of the timing of interference. The interference of the second factor is determined after first smoothing; therefore, it plays a key role during the computation of interference. If the image
ED
270
derivative generates a high value at a point then there is a high probability that
PT
it is an edge. The noise on the edges is also suppressed during the first smoothing. Although, when the image derivative decreases still the participation rate of smoothing is not greatly reduced because it is also strong for the noise on the edges. In the proposed framework, an effective coupling with an anisotropic
CE
275
filter without any component clash is possible because all components are pri-
AC
marily fed before the coupling process. Important information from the target image and the denoising method
using both the smoothing and feedback function is injected to the proposed
280
framework. For color images, smoothing based on an isotropic, anisotropic and SBTV filtering is used. Whereas, the feedback function based on the color distance in the CIE LAB color space is used to compute the difference between
13
ACCEPTED MANUSCRIPT
the local pixels and the central point. On the other hand, for range images smoothing based on a regularization strategy from (Kim & Choi, 2012) is used. 285
Whereas, the feedback function is based on a linear function to compute the
CR IP T
distance between the local pixels and the central point. Moreover, for range images, the selection and computation of the pixels in the local neighborhood is also improved by the feedback strategy.
5. Color image denoising
Local filtering is divided into two types: isotropic and anisotropic filtering.
290
AN US
Smoothing components are separately constructed using one of the aforemen-
tioned filtering or both. In turn, the feedback component is constructed from the open loop filtered images. It computes the color distance between each local pixel and the central point in the local neighborhood.
The initial smoothing function using an isotropic or anisotropic filtering
M
is defined by the coordinate pair xs of the convolution mask and the pixel coordinates as:
ED
S 0 (xs ) : Ωs ⊂ R4 → R .
(6)
The convolution mask of each pixel using an isotropic filtering can be defined by the following Gaussian kernel:
PT
G (xc ; σc ) =
σc
1 √
xc T xc , exp − 2σc2 2π
(7)
CE
where xc is the coordinate of the convolution mask and σc2 is the variance that adjusts Gaussian distribution. The convolution mask of an anisotropic filter tuned to the diffusion tensor T (x) of each pixel (Tschumperl´e, 2006) is defined
AC
as:
G (xc ; σc , T) =
σc
1 √
xc T T−1 xc exp − . 2σc2 2π
(8)
∗ ∗ The diffusion tensor T is computed using the eigen vectors θ− , θ+ of the smoothed ∗ ∗ structure tensor Gσ and their diffusivity values l− , l+ as: ∗ ∗ ∗T ∗ ∗ ∗T T = l− θ− θ− + l+ θ+ θ+ .
14
(9)
ACCEPTED MANUSCRIPT
∗T ∗T where θ− and θ+ are matrix transpose operators. The eigen values λ∗− , λ∗+
of the smoothed structure tensor Gσ are frequently used for diffusivity. The
are defined as:
1 , 1 + s2 1 ∗ l+ = , (1 + s2 )
∗ l− =√
where s is defined as: s=
λ∗− λ∗+ + . 2552 2552
The smoothed structure tensor Gσ for the color image is defined as: P Pn n Ii,x Ii,x Ii,x Ii,y i=1 i=1 ∗ Gσ , Gσ = P Pn n I I I I i,x i,y i,y i,y i=1 i=1
AN US
295
r
CR IP T
∗ ∗ diffusivity values l− , l+ , which change the characteristics of an anisotropic filter
(10)
(11)
(12)
where Ii represents an ith color band, Ii,x , Ii,y are the partial derivatives of ith color band with respect to x and y, ∗ is a convolution operator and Gσ is a
M
Gaussian function to smooth the structure tensor.
The feedback function uses the color distance between each local pixel and
ED
a central point of the area of the convolution mask. The weight W is used to adjust the old convolution mask. It is defined using the color distance Dw between each pixel and the central point of area of the convolution mask in the
PT
CIE LAB color space as:
Dw (xc ) , W (xc ; σw ) = exp − 2σw
(13)
CE
2 where σw is a variance. By using the weight W the convolution mask of the
AC
pixel x is adjusted as: G (xc ) =
1 G (xc ; σc , T) W (xc ; σw ) , normalizer
(14)
where normalizer is the value that makes the sum of convolution masks to 1.
300
In this section, the proposed feedback framework with an isotropic filtering
(FFIF), the proposed feedback framework with an anisotropic filtering (FFAF), the proposed feedback framework with SBTV (FFSBTV), SBTV (Goldstein & 15
ACCEPTED MANUSCRIPT
Osher, 2009), bilateral filtering (Tomasi & Manduchi, 1998), BM3D (Dabov et al., 2007) and SADCT (Foi et al., 2007) are tested for both qualitative and quantitative analysis. The standard deviation related to the local convolution was set to σc = σn /15 using noise variance of σn2 , which is estimated from
CR IP T
305
the input image. For an anisotropic filtering, the standard deviation of the
smoothed structure tensor Gσ was set to 0.6. In order to evaluate the pixel
similarity using the bilateral filtering, the standard deviation was set to 2σn . A small value must be assigned to the standard deviation for weight evaluation 310
of the feedback; therefore, it is set to σw = σn . The original Lena and noisy
AN US
images with various noise variances are shown in Fig. 3.
Image denoising results using Lena image with an additive Gaussian noise of the standard deviation 15 is shown in Fig. 4. The noisy image shown in Fig. 4(a) has a peak signal to noise ratio (PSNR) of 24.69db. Fig. 4(b) shows denoised 315
image using the feedback framework with an isotropic filtering (FFIF). Fig. 4(c) shows denoised image using the feedback framework with an anisotropic filtering
M
(FFAF). Fig. 4(b) has PSNR of 31.78db, which is not much different from the PSNR of Fig. 4(c), that is 31.80db. The PSNR difference is small because
320
ED
image shown in Fig. 4(b) has weak level of noise and the feedback function is created from the results of the first smoothing. Fig. 4(d) shows that in the case of bilateral filtering there are points on which smoothing is not applied. This
PT
undesirable behavior is not rectified even by increasing the variance for similarity evaluation. The proposed method is also able to remove salt and pepper noise
CE
by using the iterative edge detection and smoothing process in the feedback 325
strategy. For the color image, an anisotropic filter is used in the smoothing stage, which is able to deal with this type of noise by using the directional
AC
gradients. It smooths the noise part and maintains the image information. Image denoising results using noisy images contaminated by the Gaussian
noise of standard deviations of 20 and 25 are shown in Fig. 5 and Fig. 6, re-
330
spectively. In the bilateral filtering, similarity function yields small value when one point is affected by the intense noise in the local neighborhood. Therefore, during denoising process weak smoothing is applied continuously to the relative 16
ACCEPTED MANUSCRIPT
Table 1 PSNR comparison of various denoising algorithms: proposed feedback framework with SBTV (FFSBTV), SBTV (Goldstein & Osher, 2009), proposed feedback framework with an isotropic filtering (FFIF), proposed feedback framework with an anisotropic filtering (FFAF),
(Foi et al., 2007) (best results are boldfaced). FFSBTV
FFAF
BF
BM3D
SADCT
36.70
36.67
36.70
36.70
36.83
37.64
37.32
Barbara σ = 5
35.84
35.88
36.54
36.54
36.74
38.97
37.35
Pepper1 σ = 5
35.66
35.48
35.28
35.28
35.59
36.59
36.24
F16 σ = 5
37.12
37.37
37.80
37.81
37.63
39.46
38.07
House σ = 5
36.82
36.94
37.00
37.00
37.03
38.69
38.12
Peppers2 σ = 5
39.14
39.10
39.73
39.74
39.44
40.86
40.21
33.47
33.48
33.47
35.16
34.36
33.24
33.16
30.86
30.58
31.75
31.82
32.25
36.06
Pepper1 σ = 10
32.28
32.28
32.19
32.19
32.31
34.11
33.10
F16 σ = 10
33.46
33.66
34.20
34.22
33.94
36.26
34.77
House σ = 10
33.35
33.47
33.55
33.55
33.46
35.95
34.80
Peppers2 σ = 10
35.63
35.65
36.31
36.32
35.79
37.98
36.99
32.83
AN US
Lena σ = 10 Barbara σ = 10
33.63
Lena σ = 15
31.45
31.29
31.88
31.90
31.70
33.80
Barbara σ = 15
28.08
27.76
28.95
29.04
29.72
34.36
31.57
Pepper1 σ = 15
30.57
30.63
30.90
30.91
30.76
33.06
31.60
31.38
31.49
32.19
32.22
31.85
34.50
32.92
House σ = 15
31.53
31.65
31.85
31.87
31.54
34.56
33.13
Peppers2 σ = 15
33.59
33.54
34.28
34.28
33.68
36.10
35.03
Lena σ = 20
30.33
30.15
30.88
30.90
30.56
32.87
31.80
M
F16 σ = 15
26.46
ED
Pepper1 σ = 20
29.36
26.20
26.88
27.04
27.99
33.20
30.16
29.45
29.96
29.96
29.68
32.02
30.57
F16 σ = 20
29.89
30.03
30.84
30.87
30.40
33.25
31.63
House σ = 20
30.28
30.42
30.80
30.81
30.21
33.55
31.98
Peppers2 σ = 20
32.10
31.99
32.79
32.79
32.16
34.57
33.52
Lena σ = 25
29.54
29.36
30.17
30.19
29.74
32.22
31.02
PT CE
FFIF
Lena σ = 5
Barbara σ = 20
AC
SBTV
CR IP T
bilateral filtering (BF) (Tomasi & Manduchi, 1998), BM3D (Dabov et al., 2007) and SADCT
Barbara σ = 25
25.43
25.22
25.75
25.88
26.68
32.12
29.07
Pepper1 σ = 25
28.36
28.49
29.17
29.17
28.80
30.90
29.73
F16 σ = 25
28.75
28.92
29.82
29.86
29.28
32.28
30.62
House σ = 25
29.29
29.47
29.95
29.96
29.15
32.75
31.10
Peppers2 σ = 25
30.86
30.73
31.59
31.59
30.93
33.28
32.25
Lena σ = 30
28.77
28.58
29.43
29.46
28.94
31.46
30.21
Barbara σ = 30
24.68
24.52
25.02
25.12
25.60
31.17
28.14
Pepper1 σ = 30
27.49
27.65
28.47
28.46
28.02
29.91
28.99
F16 σ = 30
27.80
28.01
29.00
29.04
28.35
31.34
29.79
House σ = 30
28.45
28.65
29.22
29.23
28.26
32.01
30.31
Peppers2 σ = 30
29.78
29.64
30.55
30.55
29.90
32.10
31.14
Lena σ = 35
28.08
27.89
28.77
28.80
28.24
30.68
29.46
Barbara σ = 35
24.10
23.97
24.54
24.59
24.75
30.01
27.32
Pepper1 σ = 35
26.72
26.88
27.82
27.81
27.30
29.44
28.29
F16 σ = 35
26.97
27.20
28.28
28.33
27.54
30.47
29.02
House σ = 35
27.68
27.91
28.55
28.57
27.48
31.22
29.59
Peppers2 σ = 35
28.82
28.67
29.61
29.60
28.98
30.93
30.13
17
ACCEPTED MANUSCRIPT
points. Results generated by the proposed method show that the big speckles of noise are effectively reduced with the use of first smoothing. Fig. 6(d) has a 335
PSNR of 29.47db which is higher than PSNR of Fig. 6(b) because the isotropic
M
(b)
AC
CE
PT
ED
(a)
AN US
CR IP T
filtering used in Fig. 6(d) denoises an image more effectively when it has intense
(c)
(d)
Figure 3 Lena image with different level of Gaussian noise. (a) Original image. (b) Image with the Gaussian noise of standard deviation 15 (PSNR: 24.69db). (c) Image with the Gaussian noise of standard deviation 20 (PSNR: 22.25db). (d) Image with the Gaussian noise of standard deviation 25 (PSNR: 20.37db).
18
AN US
CR IP T
ACCEPTED MANUSCRIPT
(b)
CE
PT
ED
M
(a)
(c)
(d)
Figure 4 Image denoising by using noisy image with an additive Gaussian noise of standard
AC
deviation 15. (a) Noisy image (PSNR: 24.69db). (b) Denoised image using the proposed feedback framework with an isotropic filtering (FFIF) (PSNR: 31.78db). (c) Denoised image using the proposed feedback framework with an anisotropic filtering (FFAF) (PSNR: 31.80db). (d) Denoised image using the bilateral filtering (PSNR: 31.32db).
19
AN US
CR IP T
ACCEPTED MANUSCRIPT
(b)
CE
PT
ED
M
(a)
(c)
(d)
Figure 5 Image denoising by using noisy image with an additive Gaussian noise of standard
AC
deviation 20. (a) Noisy image (PSNR: 22.25db). (b) Denoised image using the proposed feedback framework with an isotropic filtering (FFIF) (PSNR: 30.53db). (c) Denoised image using the proposed feedback framework with an anisotropic filtering (FFAF) (PSNR: 30.59db). (d) Denoised image using the bilateral filtering (PSNR: 30.38db).
20
AN US
CR IP T
ACCEPTED MANUSCRIPT
(b)
CE
PT
ED
M
(a)
(c)
(d)
Figure 6 Image denoising by using noisy image with an additive Gaussian noise of standard
AC
deviation 25. (a) Noisy image (PSNR: 20.37db). (b) Denoised image using the proposed feedback framework with an isotropic filtering (FFIF) (PSNR: 29.42db). (c) Denoised image using the proposed feedback framework with an anisotropic filtering (FFAF) (PSNR: 29.50db). (d) Denoised image using the bilateral filtering (PSNR: 29.47db).
21
AN US
CR IP T
ACCEPTED MANUSCRIPT
(b)
(c)
M
(a)
(e)
(f)
AC
CE
PT
ED
(d)
Figure 7 Set of 6 color images. (a) Lena. (b) Barbara. (c) Peppers1. (d) F16. (e) House. (d) Peppers2.
22
=05 =10 =15 =20 =25 =30 =35
Pepper1 Pepper1 Pepper1 Pepper1 Pepper1 Pepper1 Pepper1
=05 =10 =15 =20 =25 =30 =35
F16 F16 F16 F16 F16 F16 F16
=05 =10 =15 =20 =25 =30 =35
House House House House House House House
=05 =10 =15 =20 =25 =30 =35
AN US
Barbara Barbara Barbara Barbara Barbara Barbara Barbara
M
=05 =10 =15 =20 =25 =30 =35
PT
ED
Lena Lena Lena Lena Lena Lena Lena
CR IP T
ACCEPTED MANUSCRIPT
AC
CE
Peppers2 Peppers2 Peppers2 Peppers2 Peppers2 Peppers2 Peppers2
=05 =10 =15 =20 =25 =30 =35
20
Proposed FFSBTV SBTV Proposed FFIF Proposed FFAF Bilateral filtering BM3D SADCT 22
24
26
28
30
32
34
36
38
PSNR Figure 8 PSNR comparison using different noisy images.
23
40
42
ACCEPTED MANUSCRIPT
CR IP T
Lena =20 Lena =25 Lena =30 Lena =35
Barbara =20 Barbara
=25
Barbara
=30
AN US
Barbara =35
Pepper1 =20 Pepper1 =25 Pepper1 =30 Pepper1 =35
F16 =20
M
F16 =25 F16 =30
ED
F16 =35
House =20 House =25 House =30
PT
House =35
Peppers2 =20
Peppers2 =25
CE
Peppers2 =30
Proposed FFSBTV SBTV Proposed FFIF Proposed FFAF Bilateral filtering BM3D SADCT
AC
Peppers2 =35 23
25
27
29
31
33
35
PSNR
Figure 9 PSNR comparison using different noisy images with heavy noise.
24
ACCEPTED MANUSCRIPT
ED
M
AN US
CR IP T
=5 =5 =5 =5 =5 =5 =10 =10 =10 =10 =10 =10 =15 =15 =15 =15 =15 =15 =20 =20 =20 =20 =20 =20 =25 =25 =25 =25 =25 =25 =30 =30 =30 =30 =30 =30 =35 =35 =35 =35 =35 =35
Proposed FFSBTV SBTV Proposed FFIF Proposed FFAF Bilateral filtering BM3D SADCT
AC
CE
PT
Lena Barbara Pepper1 F16 House Peppers2 Lena Barbara Pepper1 F16 House Peppers2 Lena Barbara Pepper1 F16 House Peppers2 Lena Barbara Pepper1 F16 House Peppers2 Lena Barbara Pepper1 F16 House Peppers2 Lena Barbara Pepper1 F16 House Peppers2 Lena Barbara Pepper1 F16 House Peppers2
20
22
24
26
28
30
32
34
36
38
40
PSNR
Figure 10 PSNR comparison using different noisy images by name.
25
42
ACCEPTED MANUSCRIPT
noise. Fig. 7 shows a set of 6 different test images, which are used for the quantitative analysis shown in Fig. 8, Fig. 9, Fig. 10 and Table 1. These test images were corrupted by an additive white Gaussian noise with different standard deviations
CR IP T
340
σ = 5, 10, 15, 20, 25, 30, 35 and used in the PSNR comparison of the denoised
images using the discussed state-of-the-art algorithms as shown in Fig. 8, Fig. 9,
Fig. 10 and Table 1. It can be seen that proposed method outperforms the bilat-
eral filtering in most cases. In turn, the bilateral filtering yields the best results 345
for all noise variations on Barbara. However, the proposed method yields better
AN US
result compared to the bilateral filtering in the presence of intense noise. Results from Table 1 conclude that the proposed framework properly determines
the neighbors in heavy noise situation, which ensures higher denoising accuracy. Non-local approaches, such as BM3D yield better denoising results; however, 350
the proposed method yields low time complexity than BM3D because it applies
M
a simple convolutional method for regularization.
6. Range image denoising
ED
Kim & Choi (2012) proposed a denoising algorithm for range images which requires a thresholding process to determine neighbor points in a local neighborhood. This method is similar to a technique for color images in which the
PT
355
viewpoint of edges are checked. In this section, a color image denoising based strategy is adapted for the proposed range image denosing framework, which is
CE
more effective than a simple thresholding. A linear system based range image denoising technique does not require a
convolution mask unlike a color image denoising technique, hence yields low
AC
time complexity. In range images, there are pixels with no value (non-data regions). Therefore, a linear system is used to reconstruct the range images, which will not spend much time on the non-data regions. The initial smoothing information which is treated as a set of matrices for a linear system is defined
26
(a)
(b)
AN US
CR IP T
ACCEPTED MANUSCRIPT
(c)
(d)
Figure 11 Range image denoising. (a) Original range image (happy Buddha). (b) Image contaminated by the Gaussian noise along the Y axis (PSNR: 55.81db). (c) Image denoising using regularization (Kim & Choi, 2012) (PSNR: 56.58db). (d) Image denoising using the
M
proposed feedback framework with improved regularization (PSNR: 57.68db).
ED
as:
S 0 = {A, G} ,
(15)
where A and G are defined in (Kim & Choi, 2012). The feedback function reconstructs A and G by discarding the non-data
PT
region from the smoothed images. In matrix A, if the difference between a pixel xp and the surrounding pixels of the smoothed image within the radius of r is
CE
greater than the threshold h based on the noise variance it is excluded from the
AC
neighbor set N :
360
N = xp , x| kx − xp k < r and I t (x) − I t (xp ) < h .
(16)
where h is smaller than or equal to the threshold used to create the initial smoothing function because the noise variance decreases at each smoothing
step. In turn, G employs a feedback strategy to estimate the ˆ I,u (u0 , v0 )
and ˆ I,u (u0 + 1, v0 ) from (Kim & Choi, 2012). 27
(a)
(b)
AN US
CR IP T
ACCEPTED MANUSCRIPT
(c)
(d)
Figure 12 Image denoising using the proposed feedback framework with improved regularization, bilateral filtering, BM3D and SA-DCT. (a) Denoised image using the proposed feedback framework with improved regularization at λ = 7. (b) Denoised image using the bilateral filtering at {σc = 1.0, σs = 0.002, k = 12}. (c) Denoised image using BM3D at σ = 3. (d)
M
Denoised image using SA-DCT at σ = 2.
A Gaussian noise with different values of variance is applied on the range
ED
image, which affects its accuracy along the Y-axis. The standard deviation of
PT
the Gaussian kernel is computed as: 0.0003x2 , σd (x) = 3
(17)
Only the size parameter was adjusted based on the specification of actual 3D imaging device in line with the experimental data. Regularization parameter λ
CE
365
was set to 10 and the surrounding pixel mask size to find neighbors was set to 5 × 5. In order to define local neighborhood, the threshold value h was set to
AC
6σd .
370
Fig. 11(a) shows an image obtained from a laser scanner. Fig. 11(b) shows
the input image after it was contaminated by the Gaussian noise along the Yaxis. Fig. 11(c) shows the denoising result using the regularization method of Kim & Choi (2012). The denoising result using the proposed feedback framework with a modified regularization strategy is shown in Fig. 11(d). It yields 28
(a)
(b)
AN US
CR IP T
ACCEPTED MANUSCRIPT
(c)
(d)
Figure 13 Boundary comparison using the proposed feedback framework with improved regularization, bilateral filtering, BM3D and SA-DCT. (a) Object boundary of the denoised image using the proposed feedback framework with improved regularization at λ = 7. (b) Object boundary of the denoised image using the bilateral filtering at {σc = 1.0, σs = 0.002, k = 12}. (c) Object boundary of the denoised image using BM3D at σ = 3. (d) Object boundary of
M
the denoised image using SA-DCT at σ = 2.
375
ED
a PSNR of 57.68db, which is at least 1 db more than PSNR of 56.58db from (Kim & Choi, 2012). Furthermore, the contraction between the floor and foot areas is remarkably decreased.
PT
Visual comparison of the proposed feedback framework, bilateral filtering, BM3D and SA-DCT are shown in Fig. 12. Fig. 12(a) shows that the pro-
CE
posed feedback framework yields the best denoised image by performing better 380
smoothing throughout the image. Fig. 12(b) shows that the bilateral filtering generates an over smoothed image from the top and introduces shrinkage over
AC
various parts of image. BM3D yields sharper and crispier result but it creates artifacts at the object boundaries as shown in Fig. 12(c). SA-DCT yields result with smoother surface but blurrier edges compared to BM3D as shown in
385
Fig. 12(d). Fig. 13 shows a boundary comparison of the denoised object using the proposed feedback framework, bilateral filtering, BM3D and SA-DCT. Fig. 13(a) 29
(a)
(b)
CR IP T
ACCEPTED MANUSCRIPT
(c)
(d)
AN US
Figure 14 SSIM comparison using the proposed feedback framework with improved regu-
larization, bilateral filtering, BM3D and SA-DCT. (a) Result using the proposed feedback framework with improved regularization at λ = 7. (b) Result using the bilateral filtering at {σc = 1.0, σs = 0.002, k = 12}. (c) Result using BM3D at σ = 3. (d) Result using SA-DCT
M
at σ = 2.
68
64 62
Kim and Choi 2012 Proposed feedback framework
N
PT
60
ED
66
58
Bilateral, c=1.0, s=0.002 BM3D BM3D-SAPCA
56
CE
54
SADCT
52 50
AC
48 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Smoothness
Figure 15 PSNR comparison using regularization method (Kim & Choi, 2012), the proposed feedback framework with improved regularization, BM3D, BM3D-SAPCA and SA-DCT on a noisy range image.
30
AN US
(a)
CR IP T
ACCEPTED MANUSCRIPT
(c)
CE
PT
ED
M
(b)
Figure 16 Range image denoising by using image from the Bumblebee dataset provided by the Point Grey Research. (a) Noisy range image. (b) Denoised image using the proposed
AC
feedback framework with improved regularization at λ = 1. (c) Denoised image using BM3D at σ = 3.99.
shows that the result generated by the proposed feedback framework offers less shrinkage compared to the result of the bilateral filtering, which is shown in
390
Fig. 13(b). Figs. 13(c) and (d) show the object boundaries using BM3D and 31
ACCEPTED MANUSCRIPT
SA-DCT, respectively. It can be seen that both methods yield less shrinkage at the boundary; however, the object is missing some of the import edge information.
395
CR IP T
Fig. 14 shows a qualitative analysis using the structural similarity index map (SSIM) (Wang et al., 2004) in log scale of power four to compare distortion along
the edges of the object. Fig. 14(a) shows that the proposed image denoising
feedback framework yields least distortion along the edges throughout the object compared to the methods shown in Figs. 14(b), (c) and (d). Fig. 15 shows a
quantitative comparison between the proposed feedback framework and stateof-the-art methods in terms of PSNR. It shows that the proposed feedback framework yields the best results.
AN US
400
The selection of correct parameter plays an important role in accurate image denoising. However, there are many methods to predict right parameters (Chu & Mak, 2016; Montagner et al., 2014). Throughout this work, same stategy is 405
used while selecting the parameter. The slope in Fig. 15 shows that the proposed
M
method is relatively insensitive to the parameters.
The proposed feedback framework and BM3D are applied to a noisy range
ED
image from the Bumblebee dataset provided by the Point Grey Research as shown in Fig. 16. The noisy range image is shown in Fig. 16(a). Denoised 410
image using the proposed feedback framework is shown in Fig. 16(b), which
PT
yields better denoised image by reconstructing inaccurate information on distant pixels. On the other hand, BM3D yields an artifact on the object boundary as
CE
shown in Fig. 16(c).
AC
7. Conclusion
415
In this paper, an image denoising framework is proposed which uses a feed-
back approach with a successive denoising strategy. It effectively improves existing smoothing techniques by feeding the results of first denoised image to the feedback function. Effective edge detection under intense noise is a major problem in image denoising. In this work, a new technique is presented to
32
ACCEPTED MANUSCRIPT
420
detect edges, which is based on the assumption that the points with large variations should be edges. The first smoothing of the feedback function removes noise from the edges and with the feedback it is assured that edges are not over
CR IP T
smoothed. The proposed feedback framework is applied to numerous denoising tech425
niques to formulate new algorithms for both color and range images. For color
images, smoothing is implemented either through an isotropic or anisotropic filtering and the feedback strategy is based on the color distance in the CIE LAB color space. For range images, neighbors are determined in an effective way
430
AN US
using a feedback function based on a linear system. Moreover, the tendency
of surface contractions in the area with strong noise is decreased. The proposed image denoising feedback framework performs better, both qualitatively and quantitatively, as compared to the discussed state-of-the-art denoising techniques for both color and range images.
It is difficult to accurately detect the edges when the image is influenced by the intense noise. In future work, we aim to formulate a new method us-
M
435
ing sequential filtering with multiple layers or by remodeling methods, such as
ED
bilateral filtering, which assesses similarity before smoothing to solve edge detection problem under intense noise. Furthermore, an advanced image denoising feedback framework using non-local means is also an interesting subject for the future work.
PT
440
CE
Declaration of interest The authors declare that there is no conflict of interests regarding the pub-
AC
lication of this paper. References
445
References Bay, H., Ess, A., Tuytelaars, T., & Gool, L. V. (2008). Speeded-up robust features (surf). Computer Vision and Image Understanding, 110 , 346–359. 33
ACCEPTED MANUSCRIPT
doi:10.1016/j.cviu.2007.09.014. Belongie, S., Malik, J., & Puzicha, J. (2002). Shape matching and object recognition using shape contexts. IEEE Transactions on Pattern Analysis and
450
CR IP T
Machine Intelligence, 24 , 509–522. doi:10.1109/34.993558.
Brox, T., Weickert, J., Burgeth, B., & Mrzek, P. (2006). Nonlinear structure
tensors. Image and Vision Computing, 24 , 41–55. doi:10.1016/j.imavis. 2005.09.010. 455
Canny, J. (1986).
A computational approach to edge detection.
IEEE
AN US
Transactions on Pattern Analysis and Machine Intelligence, 8 , 679–698. doi:10.1109/TPAMI.1986.4767851. Chu,
Y.,
& Mak,
A new {QR} decomposition-based
C. (2016).
{RLS} algorithm using the split bregman method for l1-regularized problems.
460
Signal Processing,
128 ,
303–308.
URL:
http://www.
M
sciencedirect.com/science/article/pii/S0165168416300524. doi:https://doi.org/10.1016/j.sigpro.2016.04.013.
ED
Dabov, K., Foi, A., Katkovnik, V., & Egiazarian, K. (2007). Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Transactions on Image Processing, 16 , 2080–2095. doi:10.1109/TIP.2007.901238.
PT
465
Dabov, K., Foi, A., Katkovnik, V., & Egiazarian, K. (2009). Bm3d image denoising with shape-adaptive principal component analysis. In R. Gribonval
CE
(Ed.), Proceedings of the 2009 Workshop on Signal Processing with Adaptive Sparse Structured Representations SPARS 2009. Inria Rennes - Bretagne Atlantique Saint Malo, France: INRIA.
AC
470
Dalal, N., & Triggs, B. (2005). Histograms of oriented gradients for human detection. In Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (pp. 886–893). San Diego, CA, USA: IEEE Computer Society volume 1 of CVPR 2005 . doi:10.1109/CVPR.
475
2005.177. 34
ACCEPTED MANUSCRIPT
Deriche, R. (1987). Using canny’s criteria to derive a recursively implemented optimal edge detector. International Journal of Computer Vision, 1 , 167–187. doi:10.1007/BF00123164.
CR IP T
Foi, A., Katkovnik, V., & Egiazarian, K. (2007). Pointwise shape-adaptive dct for high-quality denoising and deblocking of grayscale and color images. IEEE
480
Transactions on Image Processing, 16 , 1395–1411. doi:10.1109/TIP.2007. 891788.
Goldstein, T., & Osher, S. (2009). The split bregman method for l1-regularized
AN US
problems. SIAM Journal on Imaging Sciences, 2 , 323–343. doi:10.1137/ 080725891.
485
Harris, C., & Stephens, M. (1988). A combined corner and edge detector. In Proceedings of the Alvey Vision Conference AVC 1988 (pp. 147–152). University of Manchester, Manchester, UK: BMVA.
M
Hertzmann, A., Jacobs, C. E., Oliver, N., Curless, B., & Salesin, D. H. (2001). Image analogies. In Proceedings of the 28th Annual Conference on Computer
490
ED
Graphics and Interactive Techniques SIGGRAPH 2001 (pp. 327–340). Los Angeles, California, USA: ACM. doi:10.1145/383259.383295. Kass, M., Witkin, A., & Terzopoulos, D. (1988). Snakes: Active contour mod-
PT
els. International Journal of Computer Vision, 1 , 321–331. doi:10.1007/ BF00133570.
495
CE
Katkovnik, V., Foi, A., Egiazarian, K., & Astola, J. (2010). From local kernel to nonlocal multiple-model image denoising. International Journal of Computer
AC
Vision, 86 , 1–32. doi:10.1007/s11263-009-0272-7.
Kim, J., & Choi, K. (2012).
500
Range image denoising using a con-
strained local gaussian model for 3d object query service in the smart space.
Personal and Ubiquitous Computing, (electronic published first,
July 2012), 1–7. doi:10.1007/s00779-012-0575-5. http://dx.doi.org/ 10.1007/s00779-012-0575-5. 35
ACCEPTED MANUSCRIPT
Kyprianidis, J., Kang, H., & D¨ ollner, J. (2010). Anisotropic kuwahara filtering on the gpu. In W. Engel (Ed.), GPU Pro - Advanced Rendering Techniques
505
(pp. 247–264). AK Peters.
CR IP T
Kyprianidis, J. E., Kang, H., & D¨ ollner, J. (2009). Image and video abstraction
by anisotropic kuwahara filtering. In Posters at the 7th Symposium on NonPhotorealistic Animation and Rendering NPAR 2007. San Diego, California, USA: ACM.
510
Lee, J.-S. (1983).
Digital image smoothing and the sigma filter.
Com-
AN US
puter Vision, Graphics, and Image Processing, 24 , 255–269. doi:10.1016/ 0734-189X(83)90047-6.
Lowe, D. (1999). Object recognition from local scale-invariant features. In Proceedings of the 7th IEEE International Conference on Computer Vision
515
(pp. 1150–1157). Kerkyra, Corfu, Greece: IEEE Computer Society volume 2
M
of ICCV 1999 . doi:10.1109/ICCV.1999.790410.
Mikolajczyk, K., & Schmid, C. (2005). A performance evaluation of local de-
ED
scriptors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27 , 1615–1630. doi:10.1109/TPAMI.2005.188.
520
Montagner, Y. L., Angelini, E. D., & Olivo-Marin, J. C. (2014). An unbiased
PT
risk estimator for image denoising in the presence of mixed poisson gaussian noise. IEEE Transactions on Image Processing, 23 , 1255–1268. doi:10.1109/
CE
TIP.2014.2300821. 525
Perona, P., & Malik, J. (1990). Scale-space and edge detection using anisotropic
AC
diffusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12 , 629–639. doi:10.1109/34.56205.
Pizer, S. M., Amburn, E. P., Austin, J. D., Cromartie, R., Geselowitz, A., Greer,
530
T., ter Haar Romeny, B., Zimmerman, J. B., & Zuiderveld, K. (1987). Adaptive histogram equalization and its variations. Computer Vision, Graphics, and Image Processing, 39 , 355–368. doi:10.1016/S0734-189X(87)80186-X. 36
ACCEPTED MANUSCRIPT
Roth, S., & Black, M. J. (2009). Fields of experts. International Journal of Computer Vision, 82 , 205–229. doi:10.1007/s11263-008-0197-6. Rudin, L. I., Osher, S., & Fatemi, E. (1992). Nonlinear total variation based
doi:10.1016/0167-2789(92)90242-F.
CR IP T
noise removal algorithms. Physica D: Nonlinear Phenomena, 60 , 259–268.
535
Shensa, M. (1992). The discrete wavelet transform: wedding the a trous and mallat algorithms. IEEE Transactions on Signal Processing, 40 , 2464–2482. doi:10.1109/78.157290.
Smith, S. M., & Brady, J. M. (1997). Susana new approach to low level image
AN US
540
processing. International Journal of Computer Vision, 23 , 45–78. doi:10. 1023/A:1007963824710.
Tai, X.-C., Borok, S., & Hahn, J. (2009). Image denoising using tv-stokes equation with an orientation-matching minimization. In Proceedings of the 2nd International Conference on Scale Space and Variational Methods in Com-
M
545
puter Vision SSVM 2009 (pp. 490–501). Berlin, Heidelberg: Springer-Verlag.
ED
doi:10.1007/978-3-642-02256-2\_41. Tasdizen, T., Whitaker, R., Burchard, P., & Osher, S. (2002). Geometric surface smoothing via anisotropic diffusion of normals. In Proceedings of the 2002 IEEE Visualization Conference VIS 2002 (pp. 125–132). Boston, MA, USA:
PT
550
IEEE Computer Society. doi:http://doi.ieeecomputersociety.org/10.
CE
1109/VISUAL.2002.1183766. Tomasi, C., & Manduchi, R. (1998). Bilateral filtering for gray and color images.
AC
In Proceedings of the 6th IEEE International Conference on Computer Vision
555
ICCV 1998 (pp. 839–846). Bombay, India: IEEE Computer Society. doi:10. 1109/ICCV.1998.710815.
Tschumperl´e, D. (2006). Fast anisotropic smoothing of multi-valued images using curvature-preserving pde’s. International Journal of Computer Vision, 68 , 65–82. doi:10.1007/s11263-006-5631-z. 37
ACCEPTED MANUSCRIPT
560
Wang, Z., Bovik, A., Sheikh, H., & Simoncelli, E. (2004). Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13 , 600–612. doi:10.1109/TIP.2003.819861.
CR IP T
Weickert, J. (1996). Anisotropic Diffusion in Image Processing. Ph.D. thesis University of Kaiserslautern, Laboratory of Technomathematics. 565
Wolfson, H., & Rigoutsos, I. (1997). Geometric hashing: an overview. IEEE Computational Science & Engineering, 4 , 10–21. doi:10.1109/99.641604.
Yaroslavsky, L. P. (1985). Digital picture processingan introduction volume 9 of
AC
CE
PT
ED
M
AN US
Series in Information Sciences. New York: Springer.
38
Image Denoising Feedback Framework Using Split Bregman Approach Jeong Heon Kim, Farhan Akram, Kwang Nam Choi PII: DOI: Reference:
S0957-4174(17)30433-5 10.1016/j.eswa.2017.06.015 ESWA 11386
To appear in:
Expert Systems With Applications
Received date: Revised date: Accepted date:
10 March 2017 10 June 2017 11 June 2017
Please cite this article as: Jeong Heon Kim, Farhan Akram, Kwang Nam Choi, Image Denoising Feedback Framework Using Split Bregman Approach, Expert Systems With Applications (2017), doi: 10.1016/j.eswa.2017.06.015
This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT
Highlights • An image denoising feedback framework using split Bregman method is
CR IP T
proposed. • A feedback function is used in an iterative manner to minimize the error.
• In the denoising process, the proposed method preserves edges affected by the noise.
AC
CE
PT
ED
M
AN US
• The proposed method is tested on both color and range images.
1
ACCEPTED MANUSCRIPT
Image Denoising Feedback Framework Using Split Bregman Approach
CR IP T
Jeong Heon Kima , Farhan Akramb , Kwang Nam Choic,∗ a National
Institute of Supercomputing and Networking, Korea Institute of Science and Technology Information, Daejeon, Korea b Department of Computer Engineering and Mathematics, Rovira i Virgili University, Tarragona, Spain c Department of Computer Science and Engineering, Chung-Ang University, Seoul, Korea
AN US
Abstract
In this paper, an image denoising feedback framework is proposed for both color and range images. The proposed method works on an error minimization principle using split Bregman method. At first image is denoised by computing means in the local neighborhood. The pixels that have big differences from the center
M
of the local neighborhood compared to the noise variance are then extracted from the denoised image. There is a low correlation between the extracted pixels and their local neighborhood. This information is fed to the feedback
ED
function and denoising is performed again, iteratively, to minimize the error. In most cases, the proposed framework yields best results both qualitatively and quantitatively. It shows better denoising results than the bilateral filtering when
PT
the edge information in the input images is affected by intense noise. Moreover, during the denoising process feeback function ensures that the edges are not
CE
over smoothed. The proposed framework is applied to denoise both color and range images, which shows it works effectively on a wide variety of images unlike the evaluated state-of-the-art denoising methods.
AC
Keywords: Image denoising, split Bregman, bilateral filtering, image derivative, color image, range image
∗ Corresponding
author Email addresses: [email protected] (Jeong Heon Kim), [email protected] (Farhan Akram), [email protected] (Kwang Nam Choi)
Preprint submitted to Expert Systems with Applications
June 15, 2017
ACCEPTED MANUSCRIPT
Responses to the reviewer’s comments Reviewer 1
CR IP T
Comment The authors answered most of my questions. However, the novelty of this 5
paper is still unclear to me (Please see the general comments to your first submission). The authors need to explain that in the Introduction. Response
AN US
We have added a new paragraph in introduction section, which states the major contributions of the proposed framework.
Line 109 on pp. 5: There are two main contributions of this work. First, a
10
novel denoising method is proposed, which introduces a two-stage convolution based on the split Bregman solution. Second, unlike the traditional methods, it can denoise both color and range images with a simple change in the convolu-
Reviewer 2 Comment
ED
15
M
tional method.
The manuscript has been improved significantly and my concerns have been
PT
addressed. However, I would ask that the authors continue to check their work
CE
for spelling errors before any publication. 20
Response
We have tried to improve the readability of the paper by thoroughly revising
AC
and correcting the grammatical mistakes, which is shown by the highlighted parts in the revised manuscript.
25
We would like to thank both the reviewers for their insightful suggestions
and comments. It helped us a lot to improve the paper. We tried to reply all the comments and questions raised by reviewers and hope now everything is clear. 3
ACCEPTED MANUSCRIPT
1. Introduction Image denoising is an important and long standing preprocessing problem 30
in the areas of computer vision and image processing. It not only provides
CR IP T
a concrete baseline for better information analysis but also bestows information analysts with a better visual experience. It is necessary to have a better
understanding about the level of information to denoise an image while preserving important information. Therefore, the structural components of the image, 35
which contain useful information, should be preserved so that we do not draw an abstract image as a the result of denoising algorithm.
AN US
Numerous implementations of image feature descriptors for information processing are based on image gradient (Lowe, 1999; Mikolajczyk & Schmid, 2005; Dalal & Triggs, 2005; Bay et al., 2008) and structural information (Wolfson & 40
Rigoutsos, 1997; Belongie et al., 2002). An image gradient (Canny, 1986; Deriche, 1987; Smith & Brady, 1997) is one of the important image characteristics,
M
which is used to compute edges or boundaries of objects in an image. In color images, it is computed from the rate of change in an image coordinate system. In turn, in range images, it is computed in three-dimensional camera coordinate system. In this paper, a fine adjustment of image gradient is computed in
ED
45
the camera coordinate system, which positively affects gradient-based feature
PT
descriptors.
Image processing is an area of computer science in which information is extracted and processed from the given images. In the process of a new information generation from images, either external information(Hertzmann et al.,
CE
50
2001), internal information(Kyprianidis et al., 2009) or both are utilized. The elements which contribute in new information generation either employ local
AC
neighborhood (Harris & Stephens, 1988) or entire image domain (Pizer et al., 1987). A new perspective was introduced by changing the image domain into
55
frequency domain (Shensa, 1992). Moreover, the extracted information from the image was further used to revise previously extracted information (Kass et al., 1988). These approaches were applied in the area of denoising and later have
4
ACCEPTED MANUSCRIPT
been diversified (Roth & Black, 2009). In this paper, both color and range images are studied but the main focus is 60
on range image denoising. Range images have a considerable amount of noise.
CR IP T
It depends on the size of the image acquisition device; the smaller a device is, the higher it produces noise. Moreover, the distance estimation methods also generate noise when object is far distanced from the image acquisition device.
Therefore, different types of noise may infuse depending on the distance in a 65
single scene, which is one of the major problems in range images. It is very critical to preserve features, such as image gradient, in the denoising process.
AN US
In the proposed work, well-adjusted gradients are used along with the curvature information to denoise the range images and preserve their edges. A Gaussian filter suppresses noise by convolving a given image with a Gaus70
sian function ( also called Gaussian smoothing operator). Range images that contain considerate amount of noise can be regularized by the Gaussian function. However, a big convolutional window is needed to properly regularize
M
them, which results in high time complexity. Non-data regions that do not contain any useful information worsen the denoising problem. Transforming the data from the range images into data mesh or scatter point set is one of the
ED
75
methods to avoid non-data regions. However, it requires additional processing and mislays the important information during three-dimensional space forma-
PT
tion. Therefore, a starting point restriction is imposed on the diffusion equation to avoid two-dimensional vector-valued processing. Partial differential equations (PDEs) based methods work on a point-based processing mechanism and
CE
80
have high time complexity. Therefore, discretized linear methods are prioritized over PDEs based methods. Another advantage of linear methods is that their
AC
solution does not require any additional parameter adjustment even under the influence of noise.
85
A new perspective to preserve the edges in the color images is discussed
in (Perona & Malik, 1990; Rudin et al., 1992; Weickert, 1996) using diffusionbased denoising techniques. An approach of total variation using split Bregman method is extended by introducing an anisotropic diffusion and the local con5
ACCEPTED MANUSCRIPT
volution mask(Goldstein & Osher, 2009). From the diffusion-based methods 90
discussed above, it can be assumed that there is a sharp edge when the derivative of the denoised image is fairly large.
CR IP T
Based on the above assumption, an image denoising feedback framework for both color and range images is proposed in this paper. For color images denoising, in the first step either isotropic or anisotropic convolution mask is 95
applied for smoothing to remove noise from the edges. After removing noise from the given image, useful information is taken and fed to the feedback function, which calibrates the old convolution mask. The calibrated denoising function is
AN US
then applied to the original image in an iterative manner. On the other hand,
for range image denoising, first smoothing is based on the regularization scheme 100
from (Kim & Choi, 2012), which is improved by a new neighborhood selection using the feedback function. For color images, the feedback function is based on the color distance in the CIE LAB color space. In turn, for range images it is based on a linear system. The proposed method has an advantage over the
105
M
bilateral filtering method when the given image is influenced by the intense noise. With the use of an adapted feedback strategy, it assures better denoising results
ED
and low time complexity compared to non-local approaches. The proposed method targets both color and range images; therefore, it works effectively on a numerous types of images unlike the other state-of-the-art denoising methods.
110
PT
There are two main contributions of this work. First, a novel denoising method is proposed, which introduces a two-stage convolution based on the
CE
split Bregman solution. Second, unlike the traditional methods, it can denoise both color and range images with a simple change in the convolutional method. The paper is organized as follows. Related work is discussed in section 2.
AC
Split Bregman iteration is described in section 3. The feedback approach using
115
the first denoised image is explained in section 4. Color image denoising for the proposed framework is described and respective results are shown in section 5. Range image denoising for the proposed framework is described and respective results are shown in section 6. Finally, the proposed work is concluded in section 7. 6
ACCEPTED MANUSCRIPT
120
2. Related work A range image is a two-dimensional image, which shows the distance points in a scene from a specific point of view. Normally, it is dealt in a three-dimensional
CR IP T
(3D) space. It can be represented as a collection of two-dimensional vector-
valued images, which can be approximated as a collection of two-dimensional 125
color images. Therefore, the color denoising methods can also be used on range
images. In (Katkovnik et al., 2010), the development and classification of image denoising techniques are analyzed using different standards.
Image denoising techniques are classified into local and non-local averaging
130
AN US
methods depending on the distance of the information provider from the coor-
dinates to the estimated point. Local techniques formulate an estimation using close points on the coordinates. The area that represents the close points is normally defined by a local neighborhood with a Gaussian kernel of a particular window size. In turn, the non-local averages are computed throughout the
M
image.
Image denoising techniques are further classified into pointwise and multi-
135
point methods in which number of points are used to make a point estimation.
ED
Pointwise techniques estimate a point by computing a weighted sum. In turn, the multipoint techniques estimate unit blocks by changing the spectrum do-
140
PT
main. The estimated blocks are then clustered (fused) into a single representative block.
Image denoising techniques are further classified into the weighted means and
CE
the partial differential equation (PDE) based solutions. The weighted means techniques calculate an average by multiplying a weight with each participating element. In turn, the PDE based techniques minimize the error in a given image with respect to an artificial time. These types of methods normally have high
AC 145
time complexity and require a lot of iterations to reach an optimal solution. Range images contain a huge amount of unnecessary information, which
is considered noise. It is necessary to detect similar blocks, which are nondata regions, to separate them from useful information. Multipoint methods
7
ACCEPTED MANUSCRIPT
150
require a special process of element aggregation during domain transformation to extract region without information, which can increase the time complexity. Consequently, in this paper, a local pointwise method is employed to separate
CR IP T
non-data regions from regions with useful information. Numerous local-pointwise methods have been devised to control the weight 155
of a Gaussian kernel, for example, sigma-filter (Lee, 1983), Yaroslavsky filter (Yaroslavsky, 1985), and bilateral filter (Tomasi & Manduchi, 1998). These
methods determine the weight by using similarities between pixels of a vectorT
valued image I (x) = (I1 (x) , I2 (x) , I3 (x)) .
160
AN US
Numerous methods have been devised to expand the diffusion equation in the context of image denoising. Perona & Malik (1990) proposed a nonlinear expansion of the diffusion equation. Rudin-Osher-Fatemi (ROF) method deployed a total variation concept for diffusion (Rudin et al., 1992). Weickert (1996) proposed a generalized form of the diffusion equation with a diffusion tensor field.
Nonlinear diffusion processes (Brox et al., 2006; Rudin et al., 1992) are pro-
M
165
posed to preserve image details during image smoothing process. These meth-
ED
ods usually require a huge number of iterations to converge. However, the split Bregman method is an efficient solver for L1-regularized problems such as total variation (TV) denoising (Goldstein & Osher, 2009).
PT
Let I¯ (x) : Ω ⊂ R2 → R be an original image and I an observed image. In order to reach an optimal solution using split Bregman method, an iterative
AC
CE
minimization scheme using the TV model involves the following three steps:
2
2 λ µ Iˆk+1 = min I¯ − I 2 + dkx − ∇x I¯ − bkx 2 ¯ 2 I 2 (1)
2 λ + dky − ∇y I¯ − bky 2 , 2 dk+1 = shrink ∇x Iˆk+1 + bkx , 1/λ , x (2) dk+1 = shrink ∇y Iˆk+1 + bky , 1/λ , y bk+1 = bkx + ∇x Iˆk+1 − dk+1 , x x bk+1 = bky + ∇y Iˆk+1 − dk+1 , y y 8
(3)
ACCEPTED MANUSCRIPT
where Iˆ0 = I, d0x = d0y = b0x = b0y = 0 and λ is a scaling parameter (Goldstein & Osher, 2009). The Gauss-Seidel solution to the optimality condition of (1)
Iˆk+1 = Gk =
k k ˆ ˆ ∆I + 4 I ! + µI λ k k ∇ x d x − bx − + ∇y dky − bky (µ + 4λ)
CR IP T
is:
,
(4)
where ∆ is a discrete Laplace operator without diagonals. The shrinkage operator is defined as:
170
x ∗ max (|x| − γ, 0) , |x|
AN US
shrink (x, γ) =
(5)
The edge direction deficiency in (4) induces edge disintegration. The disintegrated edges lead to staircasing and the image looks like a painting. Several methods have been proposed to avoid staircasing based on orientation (Brox et al., 2006; Tai et al., 2009). Selection of parameters is very critical and plays
175
M
an important role in accurate image denoising, which can vary for different types of images. Numerous algorithms have been proposed to improve the accuracy
et al., 2014).
ED
and predictability of correct parameter selection (Chu & Mak, 2016; Montagner
Orthogonal based functions are used in full-rank high-order approximation
180
PT
for multipoint estimation of each image patch. This multipoint modeling is expressed as a corresponding transform-domain representation. In the local modeling category, the shape-adaptive transform domain filtering developed by
CE
(Foi et al., 2007) is one of most successful filter. The non-local approach is employed in the block-matching and 3D (BM3D) filtering by (Dabov et al.,
AC
2007). Foi et al. (2007) used the intersection of confidence intervals (ICI) rule
185
in choosing scale parameters of each directional window of local polynomial approximation (LPA) with anisotropic support. Anisotropic local polynomial approximation using intersection of confidence
intervals (LPA-ICI) based method in (Foi et al., 2007) builds an adaptive neighborhood Ux+ , which is constructed as the union of supports for the directional 9
ACCEPTED MANUSCRIPT
190
adaptive-scale kernels. The shape adaptive discrete cosine transform (SA-DCT), denoted as TU˜x+ , is computed by the cascaded application of 1D varying-length discrete cosine transform (DCT). DCT first transforms columns and then rows
CR IP T
of the centered adaptive neighborhood U˜x+ = {v|v ∈ Ω , (x − v) ∈ Ux+ } of Ux+ .
The non-local approach works on a principle which starts by finding matched 195
block. Dabov et al. (2007) selects similar blocks and arranges them in 3D arrays IUh,x . The 3D spectrum is calculated by the adopted normalized 3D linear transform T3D from the 3D arrays IUh,x . The global basic estimate is aggregated by the weighted means. BM3D computes the final estimate by using
200
AN US
the values from the basic estimate with the improved grouping and by using the collaborative Wiener filtering. The extension of BM3D algorithm with the shape-adaptive principal component analysis (BM3D-SAPCA) is proposed by (Dabov et al., 2009).
Tasdizen et al. (2002) developed a surface smoothing method, which incorporates both level set surface models and anisotropic diffusion of normals. Anisotropic diffusion is a better denoising method than isotropic diffusion be-
M
205
cause it preserves important image components, such as edges and structural
ED
details, which are very important for image interpolation. The three-dimensional model-based techniques have an additional overhead that originates by discarding the original structure of the range images and converting them into a three-dimensional structure for further processing. More-
PT
210
over, these methods are not designed for the range images from mobile devices,
CE
which are very hard to deal because of various noise levels in a scene. Therefore, open loop smoothing methods which perform single level smoothing yield
AC
unsatisfactory denoising results.
215
3. Split Bregman iteration Image denoising by using the thermal diffusion function can be defined with
a Gaussian filtering. The conceptual elements of an anisotropic diffusion are formulated in the form of bilateral filtering (Tomasi & Manduchi, 1998) and
10
ACCEPTED MANUSCRIPT
Kuwahara filtering (Kyprianidis et al., 2010). It is essential to study a correla220
tion between the methods that employ an iterative process and the ones use a single or fixed loop (open loop) to obtain a solution. In this section, meaning
CR IP T
of an iterative process is discussed and a strategy to formulate such process for the proposed framework is devised.
The split Bregman method is used to solve a variety of L1-regularized prob225
lems, such as total variation regularization. It solves a problem in an iterative
manner using multiple steps. The diffusion equation, which yields Gauss-Seidal
solution with an optimality condition is defined in (4). The discrete Laplace
AN US
operator plays an important role in iteratively solving the diffusion equation. In the diffusion computation, central pixel of the original image is used with 230
an additional (diffusion impeding) term that hinders diffusion. The diffusion impeding term preserves the edges just like a total variation (TV) model does; however, with less overhead. In (2) and (3), a shrink function is used to compute diffusion impeding term, which is defined in (5). A shrink function is zero
235
M
when the input value is smaller than the given parameter. Otherwise, it yields a smaller value than the input value. Where the given parameter comes from
ED
the noise variance. The input values that are greater than a certain threshold compared to the noise variance act as a diffusion impeding term. Furthermore, the shrink function employs the input values from the first derivative of the
240
PT
given image, on which diffusion is already applied in the previous step, instead of using them from the original image. This selection of parameters plays a key
CE
role in the edge preservation step. This process becomes clearer with the step-by-step output images shown in
Fig. 1. Fig. 1(a) shows an image that is regularized by the diffusion term in the
AC
first step. Fig. 1(b) shows that the points selected by the diffusion impeding
245
term impose a crisp look to the denoised image by preserving the sharpness of the edges. Diffusion and inverse diffusion are carried out simultaneously and as a result of successive diffusion, stable values are obtained. In range image denoising, the edge preservation using the split Bregman method can also be used to compute the neighbor points. First derivative image 11
CR IP T
ACCEPTED MANUSCRIPT
(b)
AN US
(a)
Figure 1 TV by using split Bregman method. (a) Regularized image with same amount of smoothness throughout. (b) Regularized image with sharp edges because of diffusion impeding term.
used in the shrink function indicates the closeness of two points in terms of
M
250
color. In the range images, the neighbor points are computed by comparing the closeness of the distance between them.
It ← I0
PT
St ← S0
ED
function f eedback smoothing
for i = 0 to n do
I t ← smoothing I 0 , S t
CE
S t ← f eedback I t , I 0 , S t , S 0 , κ
end for
AC
Figure 2 Image denoising feedback framework.
12
ACCEPTED MANUSCRIPT
4. Feedback strategy The proposed feedback framework has two main components. One compo255
nent maintains and applies a dedicated transformation function on each pixel. In
CR IP T
turn, the second component deals with the smoothing. Where the old smoothing information is fed to the feedback function in the next step. The transformation
value of each pixel is different than its feedback value; therefore, both types of values are managed separately. Let I 0 (x) : Ω ⊂ R2 → R be a noisy input image 260
and S 0 an initial smoothing function. Where the smoothing function is used to
generate both the denoised image I t from the input image I 0 and the current
AN US
smoothing information function S t as shown in Fig. 2. The feedback function formulates an improved smoothing function S t from the initial I 0 , S 0 and the updated I t , S t information. Here, κ is the feedback parameter. This process is 265
repeated for n times (in this paper n = 1).
The proposed framework defines a smoothing function in a large viewpoint
M
environment. It appears similar to bilateral filtering (Tomasi & Manduchi, 1998); however, there is a clear difference in terms of the timing of interference. The interference of the second factor is determined after first smoothing; therefore, it plays a key role during the computation of interference. If the image
ED
270
derivative generates a high value at a point then there is a high probability that
PT
it is an edge. The noise on the edges is also suppressed during the first smoothing. Although, when the image derivative decreases still the participation rate of smoothing is not greatly reduced because it is also strong for the noise on the edges. In the proposed framework, an effective coupling with an anisotropic
CE
275
filter without any component clash is possible because all components are pri-
AC
marily fed before the coupling process. Important information from the target image and the denoising method
using both the smoothing and feedback function is injected to the proposed
280
framework. For color images, smoothing based on an isotropic, anisotropic and SBTV filtering is used. Whereas, the feedback function based on the color distance in the CIE LAB color space is used to compute the difference between
13
ACCEPTED MANUSCRIPT
the local pixels and the central point. On the other hand, for range images smoothing based on a regularization strategy from (Kim & Choi, 2012) is used. 285
Whereas, the feedback function is based on a linear function to compute the
CR IP T
distance between the local pixels and the central point. Moreover, for range images, the selection and computation of the pixels in the local neighborhood is also improved by the feedback strategy.
5. Color image denoising
Local filtering is divided into two types: isotropic and anisotropic filtering.
290
AN US
Smoothing components are separately constructed using one of the aforemen-
tioned filtering or both. In turn, the feedback component is constructed from the open loop filtered images. It computes the color distance between each local pixel and the central point in the local neighborhood.
The initial smoothing function using an isotropic or anisotropic filtering
M
is defined by the coordinate pair xs of the convolution mask and the pixel coordinates as:
ED
S 0 (xs ) : Ωs ⊂ R4 → R .
(6)
The convolution mask of each pixel using an isotropic filtering can be defined by the following Gaussian kernel:
PT
G (xc ; σc ) =
σc
1 √
xc T xc , exp − 2σc2 2π
(7)
CE
where xc is the coordinate of the convolution mask and σc2 is the variance that adjusts Gaussian distribution. The convolution mask of an anisotropic filter tuned to the diffusion tensor T (x) of each pixel (Tschumperl´e, 2006) is defined
AC
as:
G (xc ; σc , T) =
σc
1 √
xc T T−1 xc exp − . 2σc2 2π
(8)
∗ ∗ The diffusion tensor T is computed using the eigen vectors θ− , θ+ of the smoothed ∗ ∗ structure tensor Gσ and their diffusivity values l− , l+ as: ∗ ∗ ∗T ∗ ∗ ∗T T = l− θ− θ− + l+ θ+ θ+ .
14
(9)
ACCEPTED MANUSCRIPT
∗T ∗T where θ− and θ+ are matrix transpose operators. The eigen values λ∗− , λ∗+
of the smoothed structure tensor Gσ are frequently used for diffusivity. The
are defined as:
1 , 1 + s2 1 ∗ l+ = , (1 + s2 )
∗ l− =√
where s is defined as: s=
λ∗− λ∗+ + . 2552 2552
The smoothed structure tensor Gσ for the color image is defined as: P Pn n Ii,x Ii,x Ii,x Ii,y i=1 i=1 ∗ Gσ , Gσ = P Pn n I I I I i,x i,y i,y i,y i=1 i=1
AN US
295
r
CR IP T
∗ ∗ diffusivity values l− , l+ , which change the characteristics of an anisotropic filter
(10)
(11)
(12)
where Ii represents an ith color band, Ii,x , Ii,y are the partial derivatives of ith color band with respect to x and y, ∗ is a convolution operator and Gσ is a
M
Gaussian function to smooth the structure tensor.
The feedback function uses the color distance between each local pixel and
ED
a central point of the area of the convolution mask. The weight W is used to adjust the old convolution mask. It is defined using the color distance Dw between each pixel and the central point of area of the convolution mask in the
PT
CIE LAB color space as:
Dw (xc ) , W (xc ; σw ) = exp − 2σw
(13)
CE
2 where σw is a variance. By using the weight W the convolution mask of the
AC
pixel x is adjusted as: G (xc ) =
1 G (xc ; σc , T) W (xc ; σw ) , normalizer
(14)
where normalizer is the value that makes the sum of convolution masks to 1.
300
In this section, the proposed feedback framework with an isotropic filtering
(FFIF), the proposed feedback framework with an anisotropic filtering (FFAF), the proposed feedback framework with SBTV (FFSBTV), SBTV (Goldstein & 15
ACCEPTED MANUSCRIPT
Osher, 2009), bilateral filtering (Tomasi & Manduchi, 1998), BM3D (Dabov et al., 2007) and SADCT (Foi et al., 2007) are tested for both qualitative and quantitative analysis. The standard deviation related to the local convolution was set to σc = σn /15 using noise variance of σn2 , which is estimated from
CR IP T
305
the input image. For an anisotropic filtering, the standard deviation of the
smoothed structure tensor Gσ was set to 0.6. In order to evaluate the pixel
similarity using the bilateral filtering, the standard deviation was set to 2σn . A small value must be assigned to the standard deviation for weight evaluation 310
of the feedback; therefore, it is set to σw = σn . The original Lena and noisy
AN US
images with various noise variances are shown in Fig. 3.
Image denoising results using Lena image with an additive Gaussian noise of the standard deviation 15 is shown in Fig. 4. The noisy image shown in Fig. 4(a) has a peak signal to noise ratio (PSNR) of 24.69db. Fig. 4(b) shows denoised 315
image using the feedback framework with an isotropic filtering (FFIF). Fig. 4(c) shows denoised image using the feedback framework with an anisotropic filtering
M
(FFAF). Fig. 4(b) has PSNR of 31.78db, which is not much different from the PSNR of Fig. 4(c), that is 31.80db. The PSNR difference is small because
320
ED
image shown in Fig. 4(b) has weak level of noise and the feedback function is created from the results of the first smoothing. Fig. 4(d) shows that in the case of bilateral filtering there are points on which smoothing is not applied. This
PT
undesirable behavior is not rectified even by increasing the variance for similarity evaluation. The proposed method is also able to remove salt and pepper noise
CE
by using the iterative edge detection and smoothing process in the feedback 325
strategy. For the color image, an anisotropic filter is used in the smoothing stage, which is able to deal with this type of noise by using the directional
AC
gradients. It smooths the noise part and maintains the image information. Image denoising results using noisy images contaminated by the Gaussian
noise of standard deviations of 20 and 25 are shown in Fig. 5 and Fig. 6, re-
330
spectively. In the bilateral filtering, similarity function yields small value when one point is affected by the intense noise in the local neighborhood. Therefore, during denoising process weak smoothing is applied continuously to the relative 16
ACCEPTED MANUSCRIPT
Table 1 PSNR comparison of various denoising algorithms: proposed feedback framework with SBTV (FFSBTV), SBTV (Goldstein & Osher, 2009), proposed feedback framework with an isotropic filtering (FFIF), proposed feedback framework with an anisotropic filtering (FFAF),
(Foi et al., 2007) (best results are boldfaced). FFSBTV
FFAF
BF
BM3D
SADCT
36.70
36.67
36.70
36.70
36.83
37.64
37.32
Barbara σ = 5
35.84
35.88
36.54
36.54
36.74
38.97
37.35
Pepper1 σ = 5
35.66
35.48
35.28
35.28
35.59
36.59
36.24
F16 σ = 5
37.12
37.37
37.80
37.81
37.63
39.46
38.07
House σ = 5
36.82
36.94
37.00
37.00
37.03
38.69
38.12
Peppers2 σ = 5
39.14
39.10
39.73
39.74
39.44
40.86
40.21
33.47
33.48
33.47
35.16
34.36
33.24
33.16
30.86
30.58
31.75
31.82
32.25
36.06
Pepper1 σ = 10
32.28
32.28
32.19
32.19
32.31
34.11
33.10
F16 σ = 10
33.46
33.66
34.20
34.22
33.94
36.26
34.77
House σ = 10
33.35
33.47
33.55
33.55
33.46
35.95
34.80
Peppers2 σ = 10
35.63
35.65
36.31
36.32
35.79
37.98
36.99
32.83
AN US
Lena σ = 10 Barbara σ = 10
33.63
Lena σ = 15
31.45
31.29
31.88
31.90
31.70
33.80
Barbara σ = 15
28.08
27.76
28.95
29.04
29.72
34.36
31.57
Pepper1 σ = 15
30.57
30.63
30.90
30.91
30.76
33.06
31.60
31.38
31.49
32.19
32.22
31.85
34.50
32.92
House σ = 15
31.53
31.65
31.85
31.87
31.54
34.56
33.13
Peppers2 σ = 15
33.59
33.54
34.28
34.28
33.68
36.10
35.03
Lena σ = 20
30.33
30.15
30.88
30.90
30.56
32.87
31.80
M
F16 σ = 15
26.46
ED
Pepper1 σ = 20
29.36
26.20
26.88
27.04
27.99
33.20
30.16
29.45
29.96
29.96
29.68
32.02
30.57
F16 σ = 20
29.89
30.03
30.84
30.87
30.40
33.25
31.63
House σ = 20
30.28
30.42
30.80
30.81
30.21
33.55
31.98
Peppers2 σ = 20
32.10
31.99
32.79
32.79
32.16
34.57
33.52
Lena σ = 25
29.54
29.36
30.17
30.19
29.74
32.22
31.02
PT CE
FFIF
Lena σ = 5
Barbara σ = 20
AC
SBTV
CR IP T
bilateral filtering (BF) (Tomasi & Manduchi, 1998), BM3D (Dabov et al., 2007) and SADCT
Barbara σ = 25
25.43
25.22
25.75
25.88
26.68
32.12
29.07
Pepper1 σ = 25
28.36
28.49
29.17
29.17
28.80
30.90
29.73
F16 σ = 25
28.75
28.92
29.82
29.86
29.28
32.28
30.62
House σ = 25
29.29
29.47
29.95
29.96
29.15
32.75
31.10
Peppers2 σ = 25
30.86
30.73
31.59
31.59
30.93
33.28
32.25
Lena σ = 30
28.77
28.58
29.43
29.46
28.94
31.46
30.21
Barbara σ = 30
24.68
24.52
25.02
25.12
25.60
31.17
28.14
Pepper1 σ = 30
27.49
27.65
28.47
28.46
28.02
29.91
28.99
F16 σ = 30
27.80
28.01
29.00
29.04
28.35
31.34
29.79
House σ = 30
28.45
28.65
29.22
29.23
28.26
32.01
30.31
Peppers2 σ = 30
29.78
29.64
30.55
30.55
29.90
32.10
31.14
Lena σ = 35
28.08
27.89
28.77
28.80
28.24
30.68
29.46
Barbara σ = 35
24.10
23.97
24.54
24.59
24.75
30.01
27.32
Pepper1 σ = 35
26.72
26.88
27.82
27.81
27.30
29.44
28.29
F16 σ = 35
26.97
27.20
28.28
28.33
27.54
30.47
29.02
House σ = 35
27.68
27.91
28.55
28.57
27.48
31.22
29.59
Peppers2 σ = 35
28.82
28.67
29.61
29.60
28.98
30.93
30.13
17
ACCEPTED MANUSCRIPT
points. Results generated by the proposed method show that the big speckles of noise are effectively reduced with the use of first smoothing. Fig. 6(d) has a 335
PSNR of 29.47db which is higher than PSNR of Fig. 6(b) because the isotropic
M
(b)
AC
CE
PT
ED
(a)
AN US
CR IP T
filtering used in Fig. 6(d) denoises an image more effectively when it has intense
(c)
(d)
Figure 3 Lena image with different level of Gaussian noise. (a) Original image. (b) Image with the Gaussian noise of standard deviation 15 (PSNR: 24.69db). (c) Image with the Gaussian noise of standard deviation 20 (PSNR: 22.25db). (d) Image with the Gaussian noise of standard deviation 25 (PSNR: 20.37db).
18
AN US
CR IP T
ACCEPTED MANUSCRIPT
(b)
CE
PT
ED
M
(a)
(c)
(d)
Figure 4 Image denoising by using noisy image with an additive Gaussian noise of standard
AC
deviation 15. (a) Noisy image (PSNR: 24.69db). (b) Denoised image using the proposed feedback framework with an isotropic filtering (FFIF) (PSNR: 31.78db). (c) Denoised image using the proposed feedback framework with an anisotropic filtering (FFAF) (PSNR: 31.80db). (d) Denoised image using the bilateral filtering (PSNR: 31.32db).
19
AN US
CR IP T
ACCEPTED MANUSCRIPT
(b)
CE
PT
ED
M
(a)
(c)
(d)
Figure 5 Image denoising by using noisy image with an additive Gaussian noise of standard
AC
deviation 20. (a) Noisy image (PSNR: 22.25db). (b) Denoised image using the proposed feedback framework with an isotropic filtering (FFIF) (PSNR: 30.53db). (c) Denoised image using the proposed feedback framework with an anisotropic filtering (FFAF) (PSNR: 30.59db). (d) Denoised image using the bilateral filtering (PSNR: 30.38db).
20
AN US
CR IP T
ACCEPTED MANUSCRIPT
(b)
CE
PT
ED
M
(a)
(c)
(d)
Figure 6 Image denoising by using noisy image with an additive Gaussian noise of standard
AC
deviation 25. (a) Noisy image (PSNR: 20.37db). (b) Denoised image using the proposed feedback framework with an isotropic filtering (FFIF) (PSNR: 29.42db). (c) Denoised image using the proposed feedback framework with an anisotropic filtering (FFAF) (PSNR: 29.50db). (d) Denoised image using the bilateral filtering (PSNR: 29.47db).
21
AN US
CR IP T
ACCEPTED MANUSCRIPT
(b)
(c)
M
(a)
(e)
(f)
AC
CE
PT
ED
(d)
Figure 7 Set of 6 color images. (a) Lena. (b) Barbara. (c) Peppers1. (d) F16. (e) House. (d) Peppers2.
22
=05 =10 =15 =20 =25 =30 =35
Pepper1 Pepper1 Pepper1 Pepper1 Pepper1 Pepper1 Pepper1
=05 =10 =15 =20 =25 =30 =35
F16 F16 F16 F16 F16 F16 F16
=05 =10 =15 =20 =25 =30 =35
House House House House House House House
=05 =10 =15 =20 =25 =30 =35
AN US
Barbara Barbara Barbara Barbara Barbara Barbara Barbara
M
=05 =10 =15 =20 =25 =30 =35
PT
ED
Lena Lena Lena Lena Lena Lena Lena
CR IP T
ACCEPTED MANUSCRIPT
AC
CE
Peppers2 Peppers2 Peppers2 Peppers2 Peppers2 Peppers2 Peppers2
=05 =10 =15 =20 =25 =30 =35
20
Proposed FFSBTV SBTV Proposed FFIF Proposed FFAF Bilateral filtering BM3D SADCT 22
24
26
28
30
32
34
36
38
PSNR Figure 8 PSNR comparison using different noisy images.
23
40
42
ACCEPTED MANUSCRIPT
CR IP T
Lena =20 Lena =25 Lena =30 Lena =35
Barbara =20 Barbara
=25
Barbara
=30
AN US
Barbara =35
Pepper1 =20 Pepper1 =25 Pepper1 =30 Pepper1 =35
F16 =20
M
F16 =25 F16 =30
ED
F16 =35
House =20 House =25 House =30
PT
House =35
Peppers2 =20
Peppers2 =25
CE
Peppers2 =30
Proposed FFSBTV SBTV Proposed FFIF Proposed FFAF Bilateral filtering BM3D SADCT
AC
Peppers2 =35 23
25
27
29
31
33
35
PSNR
Figure 9 PSNR comparison using different noisy images with heavy noise.
24
ACCEPTED MANUSCRIPT
ED
M
AN US
CR IP T
=5 =5 =5 =5 =5 =5 =10 =10 =10 =10 =10 =10 =15 =15 =15 =15 =15 =15 =20 =20 =20 =20 =20 =20 =25 =25 =25 =25 =25 =25 =30 =30 =30 =30 =30 =30 =35 =35 =35 =35 =35 =35
Proposed FFSBTV SBTV Proposed FFIF Proposed FFAF Bilateral filtering BM3D SADCT
AC
CE
PT
Lena Barbara Pepper1 F16 House Peppers2 Lena Barbara Pepper1 F16 House Peppers2 Lena Barbara Pepper1 F16 House Peppers2 Lena Barbara Pepper1 F16 House Peppers2 Lena Barbara Pepper1 F16 House Peppers2 Lena Barbara Pepper1 F16 House Peppers2 Lena Barbara Pepper1 F16 House Peppers2
20
22
24
26
28
30
32
34
36
38
40
PSNR
Figure 10 PSNR comparison using different noisy images by name.
25
42
ACCEPTED MANUSCRIPT
noise. Fig. 7 shows a set of 6 different test images, which are used for the quantitative analysis shown in Fig. 8, Fig. 9, Fig. 10 and Table 1. These test images were corrupted by an additive white Gaussian noise with different standard deviations
CR IP T
340
σ = 5, 10, 15, 20, 25, 30, 35 and used in the PSNR comparison of the denoised
images using the discussed state-of-the-art algorithms as shown in Fig. 8, Fig. 9,
Fig. 10 and Table 1. It can be seen that proposed method outperforms the bilat-
eral filtering in most cases. In turn, the bilateral filtering yields the best results 345
for all noise variations on Barbara. However, the proposed method yields better
AN US
result compared to the bilateral filtering in the presence of intense noise. Results from Table 1 conclude that the proposed framework properly determines
the neighbors in heavy noise situation, which ensures higher denoising accuracy. Non-local approaches, such as BM3D yield better denoising results; however, 350
the proposed method yields low time complexity than BM3D because it applies
M
a simple convolutional method for regularization.
6. Range image denoising
ED
Kim & Choi (2012) proposed a denoising algorithm for range images which requires a thresholding process to determine neighbor points in a local neighborhood. This method is similar to a technique for color images in which the
PT
355
viewpoint of edges are checked. In this section, a color image denoising based strategy is adapted for the proposed range image denosing framework, which is
CE
more effective than a simple thresholding. A linear system based range image denoising technique does not require a
convolution mask unlike a color image denoising technique, hence yields low
AC
time complexity. In range images, there are pixels with no value (non-data regions). Therefore, a linear system is used to reconstruct the range images, which will not spend much time on the non-data regions. The initial smoothing information which is treated as a set of matrices for a linear system is defined
26
(a)
(b)
AN US
CR IP T
ACCEPTED MANUSCRIPT
(c)
(d)
Figure 11 Range image denoising. (a) Original range image (happy Buddha). (b) Image contaminated by the Gaussian noise along the Y axis (PSNR: 55.81db). (c) Image denoising using regularization (Kim & Choi, 2012) (PSNR: 56.58db). (d) Image denoising using the
M
proposed feedback framework with improved regularization (PSNR: 57.68db).
ED
as:
S 0 = {A, G} ,
(15)
where A and G are defined in (Kim & Choi, 2012). The feedback function reconstructs A and G by discarding the non-data
PT
region from the smoothed images. In matrix A, if the difference between a pixel xp and the surrounding pixels of the smoothed image within the radius of r is
CE
greater than the threshold h based on the noise variance it is excluded from the
AC
neighbor set N :
360
N = xp , x| kx − xp k < r and I t (x) − I t (xp ) < h .
(16)
where h is smaller than or equal to the threshold used to create the initial smoothing function because the noise variance decreases at each smoothing
step. In turn, G employs a feedback strategy to estimate the ˆ I,u (u0 , v0 )
and ˆ I,u (u0 + 1, v0 ) from (Kim & Choi, 2012). 27
(a)
(b)
AN US
CR IP T
ACCEPTED MANUSCRIPT
(c)
(d)
Figure 12 Image denoising using the proposed feedback framework with improved regularization, bilateral filtering, BM3D and SA-DCT. (a) Denoised image using the proposed feedback framework with improved regularization at λ = 7. (b) Denoised image using the bilateral filtering at {σc = 1.0, σs = 0.002, k = 12}. (c) Denoised image using BM3D at σ = 3. (d)
M
Denoised image using SA-DCT at σ = 2.
A Gaussian noise with different values of variance is applied on the range
ED
image, which affects its accuracy along the Y-axis. The standard deviation of
PT
the Gaussian kernel is computed as: 0.0003x2 , σd (x) = 3
(17)
Only the size parameter was adjusted based on the specification of actual 3D imaging device in line with the experimental data. Regularization parameter λ
CE
365
was set to 10 and the surrounding pixel mask size to find neighbors was set to 5 × 5. In order to define local neighborhood, the threshold value h was set to
AC
6σd .
370
Fig. 11(a) shows an image obtained from a laser scanner. Fig. 11(b) shows
the input image after it was contaminated by the Gaussian noise along the Yaxis. Fig. 11(c) shows the denoising result using the regularization method of Kim & Choi (2012). The denoising result using the proposed feedback framework with a modified regularization strategy is shown in Fig. 11(d). It yields 28
(a)
(b)
AN US
CR IP T
ACCEPTED MANUSCRIPT
(c)
(d)
Figure 13 Boundary comparison using the proposed feedback framework with improved regularization, bilateral filtering, BM3D and SA-DCT. (a) Object boundary of the denoised image using the proposed feedback framework with improved regularization at λ = 7. (b) Object boundary of the denoised image using the bilateral filtering at {σc = 1.0, σs = 0.002, k = 12}. (c) Object boundary of the denoised image using BM3D at σ = 3. (d) Object boundary of
M
the denoised image using SA-DCT at σ = 2.
375
ED
a PSNR of 57.68db, which is at least 1 db more than PSNR of 56.58db from (Kim & Choi, 2012). Furthermore, the contraction between the floor and foot areas is remarkably decreased.
PT
Visual comparison of the proposed feedback framework, bilateral filtering, BM3D and SA-DCT are shown in Fig. 12. Fig. 12(a) shows that the pro-
CE
posed feedback framework yields the best denoised image by performing better 380
smoothing throughout the image. Fig. 12(b) shows that the bilateral filtering generates an over smoothed image from the top and introduces shrinkage over
AC
various parts of image. BM3D yields sharper and crispier result but it creates artifacts at the object boundaries as shown in Fig. 12(c). SA-DCT yields result with smoother surface but blurrier edges compared to BM3D as shown in
385
Fig. 12(d). Fig. 13 shows a boundary comparison of the denoised object using the proposed feedback framework, bilateral filtering, BM3D and SA-DCT. Fig. 13(a) 29
(a)
(b)
CR IP T
ACCEPTED MANUSCRIPT
(c)
(d)
AN US
Figure 14 SSIM comparison using the proposed feedback framework with improved regu-
larization, bilateral filtering, BM3D and SA-DCT. (a) Result using the proposed feedback framework with improved regularization at λ = 7. (b) Result using the bilateral filtering at {σc = 1.0, σs = 0.002, k = 12}. (c) Result using BM3D at σ = 3. (d) Result using SA-DCT
M
at σ = 2.
68
64 62
Kim and Choi 2012 Proposed feedback framework
N
PT
60
ED
66
58
Bilateral, c=1.0, s=0.002 BM3D BM3D-SAPCA
56
CE
54
SADCT
52 50
AC
48 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Smoothness
Figure 15 PSNR comparison using regularization method (Kim & Choi, 2012), the proposed feedback framework with improved regularization, BM3D, BM3D-SAPCA and SA-DCT on a noisy range image.
30
AN US
(a)
CR IP T
ACCEPTED MANUSCRIPT
(c)
CE
PT
ED
M
(b)
Figure 16 Range image denoising by using image from the Bumblebee dataset provided by the Point Grey Research. (a) Noisy range image. (b) Denoised image using the proposed
AC
feedback framework with improved regularization at λ = 1. (c) Denoised image using BM3D at σ = 3.99.
shows that the result generated by the proposed feedback framework offers less shrinkage compared to the result of the bilateral filtering, which is shown in
390
Fig. 13(b). Figs. 13(c) and (d) show the object boundaries using BM3D and 31
ACCEPTED MANUSCRIPT
SA-DCT, respectively. It can be seen that both methods yield less shrinkage at the boundary; however, the object is missing some of the import edge information.
395
CR IP T
Fig. 14 shows a qualitative analysis using the structural similarity index map (SSIM) (Wang et al., 2004) in log scale of power four to compare distortion along
the edges of the object. Fig. 14(a) shows that the proposed image denoising
feedback framework yields least distortion along the edges throughout the object compared to the methods shown in Figs. 14(b), (c) and (d). Fig. 15 shows a
quantitative comparison between the proposed feedback framework and stateof-the-art methods in terms of PSNR. It shows that the proposed feedback framework yields the best results.
AN US
400
The selection of correct parameter plays an important role in accurate image denoising. However, there are many methods to predict right parameters (Chu & Mak, 2016; Montagner et al., 2014). Throughout this work, same stategy is 405
used while selecting the parameter. The slope in Fig. 15 shows that the proposed
M
method is relatively insensitive to the parameters.
The proposed feedback framework and BM3D are applied to a noisy range
ED
image from the Bumblebee dataset provided by the Point Grey Research as shown in Fig. 16. The noisy range image is shown in Fig. 16(a). Denoised 410
image using the proposed feedback framework is shown in Fig. 16(b), which
PT
yields better denoised image by reconstructing inaccurate information on distant pixels. On the other hand, BM3D yields an artifact on the object boundary as
CE
shown in Fig. 16(c).
AC
7. Conclusion
415
In this paper, an image denoising framework is proposed which uses a feed-
back approach with a successive denoising strategy. It effectively improves existing smoothing techniques by feeding the results of first denoised image to the feedback function. Effective edge detection under intense noise is a major problem in image denoising. In this work, a new technique is presented to
32
ACCEPTED MANUSCRIPT
420
detect edges, which is based on the assumption that the points with large variations should be edges. The first smoothing of the feedback function removes noise from the edges and with the feedback it is assured that edges are not over
CR IP T
smoothed. The proposed feedback framework is applied to numerous denoising tech425
niques to formulate new algorithms for both color and range images. For color
images, smoothing is implemented either through an isotropic or anisotropic filtering and the feedback strategy is based on the color distance in the CIE LAB color space. For range images, neighbors are determined in an effective way
430
AN US
using a feedback function based on a linear system. Moreover, the tendency
of surface contractions in the area with strong noise is decreased. The proposed image denoising feedback framework performs better, both qualitatively and quantitatively, as compared to the discussed state-of-the-art denoising techniques for both color and range images.
It is difficult to accurately detect the edges when the image is influenced by the intense noise. In future work, we aim to formulate a new method us-
M
435
ing sequential filtering with multiple layers or by remodeling methods, such as
ED
bilateral filtering, which assesses similarity before smoothing to solve edge detection problem under intense noise. Furthermore, an advanced image denoising feedback framework using non-local means is also an interesting subject for the future work.
PT
440
CE
Declaration of interest The authors declare that there is no conflict of interests regarding the pub-
AC
lication of this paper. References
445
References Bay, H., Ess, A., Tuytelaars, T., & Gool, L. V. (2008). Speeded-up robust features (surf). Computer Vision and Image Understanding, 110 , 346–359. 33
ACCEPTED MANUSCRIPT
doi:10.1016/j.cviu.2007.09.014. Belongie, S., Malik, J., & Puzicha, J. (2002). Shape matching and object recognition using shape contexts. IEEE Transactions on Pattern Analysis and
450
CR IP T
Machine Intelligence, 24 , 509–522. doi:10.1109/34.993558.
Brox, T., Weickert, J., Burgeth, B., & Mrzek, P. (2006). Nonlinear structure
tensors. Image and Vision Computing, 24 , 41–55. doi:10.1016/j.imavis. 2005.09.010. 455
Canny, J. (1986).
A computational approach to edge detection.
IEEE
AN US
Transactions on Pattern Analysis and Machine Intelligence, 8 , 679–698. doi:10.1109/TPAMI.1986.4767851. Chu,
Y.,
& Mak,
A new {QR} decomposition-based
C. (2016).
{RLS} algorithm using the split bregman method for l1-regularized problems.
460
Signal Processing,
128 ,
303–308.
URL:
http://www.
M
sciencedirect.com/science/article/pii/S0165168416300524. doi:https://doi.org/10.1016/j.sigpro.2016.04.013.
ED
Dabov, K., Foi, A., Katkovnik, V., & Egiazarian, K. (2007). Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Transactions on Image Processing, 16 , 2080–2095. doi:10.1109/TIP.2007.901238.
PT
465
Dabov, K., Foi, A., Katkovnik, V., & Egiazarian, K. (2009). Bm3d image denoising with shape-adaptive principal component analysis. In R. Gribonval
CE
(Ed.), Proceedings of the 2009 Workshop on Signal Processing with Adaptive Sparse Structured Representations SPARS 2009. Inria Rennes - Bretagne Atlantique Saint Malo, France: INRIA.
AC
470
Dalal, N., & Triggs, B. (2005). Histograms of oriented gradients for human detection. In Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (pp. 886–893). San Diego, CA, USA: IEEE Computer Society volume 1 of CVPR 2005 . doi:10.1109/CVPR.
475
2005.177. 34
ACCEPTED MANUSCRIPT
Deriche, R. (1987). Using canny’s criteria to derive a recursively implemented optimal edge detector. International Journal of Computer Vision, 1 , 167–187. doi:10.1007/BF00123164.
CR IP T
Foi, A., Katkovnik, V., & Egiazarian, K. (2007). Pointwise shape-adaptive dct for high-quality denoising and deblocking of grayscale and color images. IEEE
480
Transactions on Image Processing, 16 , 1395–1411. doi:10.1109/TIP.2007. 891788.
Goldstein, T., & Osher, S. (2009). The split bregman method for l1-regularized
AN US
problems. SIAM Journal on Imaging Sciences, 2 , 323–343. doi:10.1137/ 080725891.
485
Harris, C., & Stephens, M. (1988). A combined corner and edge detector. In Proceedings of the Alvey Vision Conference AVC 1988 (pp. 147–152). University of Manchester, Manchester, UK: BMVA.
M
Hertzmann, A., Jacobs, C. E., Oliver, N., Curless, B., & Salesin, D. H. (2001). Image analogies. In Proceedings of the 28th Annual Conference on Computer
490
ED
Graphics and Interactive Techniques SIGGRAPH 2001 (pp. 327–340). Los Angeles, California, USA: ACM. doi:10.1145/383259.383295. Kass, M., Witkin, A., & Terzopoulos, D. (1988). Snakes: Active contour mod-
PT
els. International Journal of Computer Vision, 1 , 321–331. doi:10.1007/ BF00133570.
495
CE
Katkovnik, V., Foi, A., Egiazarian, K., & Astola, J. (2010). From local kernel to nonlocal multiple-model image denoising. International Journal of Computer
AC
Vision, 86 , 1–32. doi:10.1007/s11263-009-0272-7.
Kim, J., & Choi, K. (2012).
500
Range image denoising using a con-
strained local gaussian model for 3d object query service in the smart space.
Personal and Ubiquitous Computing, (electronic published first,
July 2012), 1–7. doi:10.1007/s00779-012-0575-5. http://dx.doi.org/ 10.1007/s00779-012-0575-5. 35
ACCEPTED MANUSCRIPT
Kyprianidis, J., Kang, H., & D¨ ollner, J. (2010). Anisotropic kuwahara filtering on the gpu. In W. Engel (Ed.), GPU Pro - Advanced Rendering Techniques
505
(pp. 247–264). AK Peters.
CR IP T
Kyprianidis, J. E., Kang, H., & D¨ ollner, J. (2009). Image and video abstraction
by anisotropic kuwahara filtering. In Posters at the 7th Symposium on NonPhotorealistic Animation and Rendering NPAR 2007. San Diego, California, USA: ACM.
510
Lee, J.-S. (1983).
Digital image smoothing and the sigma filter.
Com-
AN US
puter Vision, Graphics, and Image Processing, 24 , 255–269. doi:10.1016/ 0734-189X(83)90047-6.
Lowe, D. (1999). Object recognition from local scale-invariant features. In Proceedings of the 7th IEEE International Conference on Computer Vision
515
(pp. 1150–1157). Kerkyra, Corfu, Greece: IEEE Computer Society volume 2
M
of ICCV 1999 . doi:10.1109/ICCV.1999.790410.
Mikolajczyk, K., & Schmid, C. (2005). A performance evaluation of local de-
ED
scriptors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27 , 1615–1630. doi:10.1109/TPAMI.2005.188.
520
Montagner, Y. L., Angelini, E. D., & Olivo-Marin, J. C. (2014). An unbiased
PT
risk estimator for image denoising in the presence of mixed poisson gaussian noise. IEEE Transactions on Image Processing, 23 , 1255–1268. doi:10.1109/
CE
TIP.2014.2300821. 525
Perona, P., & Malik, J. (1990). Scale-space and edge detection using anisotropic
AC
diffusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12 , 629–639. doi:10.1109/34.56205.
Pizer, S. M., Amburn, E. P., Austin, J. D., Cromartie, R., Geselowitz, A., Greer,
530
T., ter Haar Romeny, B., Zimmerman, J. B., & Zuiderveld, K. (1987). Adaptive histogram equalization and its variations. Computer Vision, Graphics, and Image Processing, 39 , 355–368. doi:10.1016/S0734-189X(87)80186-X. 36
ACCEPTED MANUSCRIPT
Roth, S., & Black, M. J. (2009). Fields of experts. International Journal of Computer Vision, 82 , 205–229. doi:10.1007/s11263-008-0197-6. Rudin, L. I., Osher, S., & Fatemi, E. (1992). Nonlinear total variation based
doi:10.1016/0167-2789(92)90242-F.
CR IP T
noise removal algorithms. Physica D: Nonlinear Phenomena, 60 , 259–268.
535
Shensa, M. (1992). The discrete wavelet transform: wedding the a trous and mallat algorithms. IEEE Transactions on Signal Processing, 40 , 2464–2482. doi:10.1109/78.157290.
Smith, S. M., & Brady, J. M. (1997). Susana new approach to low level image
AN US
540
processing. International Journal of Computer Vision, 23 , 45–78. doi:10. 1023/A:1007963824710.
Tai, X.-C., Borok, S., & Hahn, J. (2009). Image denoising using tv-stokes equation with an orientation-matching minimization. In Proceedings of the 2nd International Conference on Scale Space and Variational Methods in Com-
M
545
puter Vision SSVM 2009 (pp. 490–501). Berlin, Heidelberg: Springer-Verlag.
ED
doi:10.1007/978-3-642-02256-2\_41. Tasdizen, T., Whitaker, R., Burchard, P., & Osher, S. (2002). Geometric surface smoothing via anisotropic diffusion of normals. In Proceedings of the 2002 IEEE Visualization Conference VIS 2002 (pp. 125–132). Boston, MA, USA:
PT
550
IEEE Computer Society. doi:http://doi.ieeecomputersociety.org/10.
CE
1109/VISUAL.2002.1183766. Tomasi, C., & Manduchi, R. (1998). Bilateral filtering for gray and color images.
AC
In Proceedings of the 6th IEEE International Conference on Computer Vision
555
ICCV 1998 (pp. 839–846). Bombay, India: IEEE Computer Society. doi:10. 1109/ICCV.1998.710815.
Tschumperl´e, D. (2006). Fast anisotropic smoothing of multi-valued images using curvature-preserving pde’s. International Journal of Computer Vision, 68 , 65–82. doi:10.1007/s11263-006-5631-z. 37
ACCEPTED MANUSCRIPT
560
Wang, Z., Bovik, A., Sheikh, H., & Simoncelli, E. (2004). Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13 , 600–612. doi:10.1109/TIP.2003.819861.
CR IP T
Weickert, J. (1996). Anisotropic Diffusion in Image Processing. Ph.D. thesis University of Kaiserslautern, Laboratory of Technomathematics. 565
Wolfson, H., & Rigoutsos, I. (1997). Geometric hashing: an overview. IEEE Computational Science & Engineering, 4 , 10–21. doi:10.1109/99.641604.
Yaroslavsky, L. P. (1985). Digital picture processingan introduction volume 9 of
AC
CE
PT
ED
M
AN US
Series in Information Sciences. New York: Springer.
38