Gradient entropy metric and p-Laplace diffusion constraint-based algorithm for noisy multispectral image fusion

Accepted Manuscript Gradient entropy metric and p-Laplace diffusion constraint-based algorithm for noisy multispectral image fusion Wenda Zhao, Zhijun Xu, Jian Zhao PII: DOI: Reference:

S1566-2535(15)00054-8 http://dx.doi.org/10.1016/j.inffus.2015.06.003 INFFUS 713

To appear in:

Information Fusion

Received Date: Revised Date: Accepted Date:

31 August 2014 18 May 2015 8 June 2015

Please cite this article as: W. Zhao, Z. Xu, J. Zhao, Gradient entropy metric and p-Laplace diffusion constraintbased algorithm for noisy multispectral image fusion, Information Fusion (2015), doi: http://dx.doi.org/10.1016/ j.inffus.2015.06.003

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Gradient entropy metric and ρ-Laplace diffusion constraint-based algorithm for noisy multispectral image fusion Wenda Zhao a,b, Zhijun Xu a,*, Jian Zhao a a

Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences,

Changchun 130033, China

b

University of Chinese Academy of Sciences, Beijing 100049, China

Noise is easily mistaken as useful features of input images, and therefore, significantly reducing image fusion quality. In this paper, we propose a novel gradient entropy metric and p -Laplace diffusion constraint-based method. Specifically, the method is based on the matrix of structure tensor to fuse the gradient information. To minimize the negative effects of noise on the selections of image features, the gradient entropy metric is proposed to construct the weight for each gradient of input images. Particularly, the local adaptive p -Laplace diffusion constraint is constructed to further suppress noise when rebuilding the fused image from the fused gradient field. Experimental results show that the proposed method effectively preserves edge detail features of multispectral images while suppressing noise, achieving an optimal visual effect and more comprehensive quantitative assessments compared to other existing methods. Keywords: Images fusion, noisy multispectral image, matrix of structure tensor, gradient entropy metric, p -Laplace diffusion constraint *Corresponding author. Tel.: +86 0431-86176577 E-mail address: [email protected]

1. Introduction Image fusion is crucial for comprehensive utilization of multisensor image information. With the rapid development of the sensor technology [1,2] and information processing theory [3], image fusion technique has been widely used in the medical, infrared and remote sensing field. Nevertheless, image sensors could produce noise in many applications. Through a large number of experiments, we find that not all multiband images acquired at the same time contain noise. For example, considering a three-band image composed of a thermal infrared band, a range scanner

band and an intensified near infrared band acquired under extremely low light conditions, the intensified band will be noisy while the other two bands will not be adversely affected by the illumination [4]. Therefore, gradient entropy metric and p-Laplace diffusion constraint-based algorithm is proposed for noisy multispectral image fusion. Noise, generally concentrating in high frequency part of the image, affects the extraction of image details and edge features. Existing image fusion algorithms [5-17] often mistake the noise signal as the useful characterizations of the image, reducing the effect of image fusion significantly. Therefore, noisy image fusion has become challenging work in the field of computer vision. For noisy image fusion, a straightforward method is to average the multiband images. It will reduce the noise of the fused image, but it cannot effectively extract the edge detail information of the input images. Nowadays, more robust algorithms are based on the wavelet transform [18-20]. Especially in [19], a noisy image fusion method based on non-Gaussian model in the wavelet domain was proposed to construct wavelet coefficients with the generalised Gaussian and alpha-stable distribution. Although suppressed noise effectively, the method generated the ringing effect. For image fusion and de-noising, another idea is to focus on the effective image de-noising algorithms. An interesting approach is to adopt the total variation (TV) model [21,22]. The TV model is a data driven approach and does not require knowledge of the probability density function of the pixels. It can preserve the edge details of the fused image while suppressing noise. However, there normally exists the staircase effect in TV model [23]. In recent years, researchers propose the image fusion methods in the gradient domain [24-26]. Human vision is sensitive to the changes of image local details, which are actually image gradients. Image fusion in the gradient domain easily incorporate edge details of multiband images into a gradient field. Therefore, the fused image preserves important features of the input images. Piella [26] used the matrix of structure tensor to fuse the contrast (i.e., gradient) information for the visualization of multispectral images. With the principle that big contrast correlates with visually relevant image features, Piella adopted the maximum contrast among

multispectral images as the contrast of the fused image. For clear images, the algorithm effectively preserves edge details of the input images. However, due to the fact that maximum contrast may correspond to noise, the fusion effect is not satisfactory for noisy images. To solve the problems mentioned above, the gradient entropy metric and

p -Laplace diffusion constraint-based method for the fusion of noisy multispectral image is proposed in this paper. Several contributions of the proposed method are as follows. 1) An effective variation model based on the matrix of structure tensor in the gradient domain is proposed. The proposed fusion framework is used for multispectral image fusion and de-noising. 2) Gradient entropy metric is proposed to construct the weight for each gradient of the input images. This can extract the useful feature (e.g., details and edges) from input noisy images, immune to noise. 3) A local adaptive p -Laplace regularization variation model is constructed to further suppress noise. According to different local information of the image, p is with different values adaptively. With different diffusions at different pixel locations, noise is suppressed while avoiding the edge effect. This paper is organized as follows. Section 2 gives the fusion method of noisy multispectral images based on gradient entropy metric and p-Laplace diffusion constraint. Experimental results and related discussions are provided in section 3, followed by a brief conclusion.

2. Proposed algorithm What we concern here is how to integrate noisy multispectral images into an image, preserving edge details of the input images while reducing noise. The process of the proposed fusion algorithm is shown in Fig. 1. Each step will be introduced in the following contents.

Fig. 1. Overall process of the proposed fusion algorithm.

2.1 Structure tensor-based image gradient fusion x⊂Ω As a pixel location of

the

input

image

I ( x)(x = ( x1 , x2 ) ⊂ Ω = [0, N − 1] × [0, M − 1]) , the gradient ∇I (x) is defined as: ∇I ( x) = (

∂I ∂I , ) ∂x1 ∂x2

(1)

where ∇ is the gradient operator. So gradients of all pixel locations of the image form a vector field, which is called as the gradient field. If I : Ω → [1, M ]N is a

multispectral image ( I n (n = 1, 2,..., N ) ), its gray level is I ( x) = ( I1 ( x), I 2 ( x),..., I N ( x )) , where x = ( x1 , x2 ) ⊂ Ω . And I n ( x ) denotes the gray level of the input image I n .To describe the gradient of the multispectral image of I , we firstly consider its differential: dI =

∂I ∂I dx1 + dx2 ∂x1 ∂x2

(2)

The squared norm of dI is expressed in Eq. (3): 2

2

dI = ∑∑ ( 2

i =1 j =1

= [dx1

∂I ∂I ⋅ )dxi dx j ∂xi ∂x j

 dx  dx2 ]G  1   dx2 

(3)

where G is called as the structure tensor, which is represented as:  N ∂I n 2  ∑ ( ∂x ) n =1 1 G ( x) =  N  ∂I n ∂I n ∑  n =1 ∂x1 ∂x2

∂I n ∂I n   n =1 1 ∂x2  N  ∂I ( n )2  ∑ n =1 ∂x2  N

∑ ∂x

(4)

In equation (4), G is a semi-definite matrix with nonnegative eigenvalues of

λ1 and λ2 , and the corresponding eigenvectors are e1 and e2 . If λ1 is the biggest eigenvalue of G , it represents the maximum contrast. G can be decomposed as:

G = QΛQT

(5)

where Q is an orthogonal matrix with the size 2 × 2 , and Λ is a diagonal matrix of 2 × 2 . If a fused image I′ preserves the main geometry of the multi-value image of I ,

its G ′ should be the best approximation matrix of G , which is:

G′ = QΛ′QT

(6)

λ 0  T where Λ′ =  1  . From (6), we can deduce G ′ = λ1e1e1 and V′(x) = λ1 (x) ⋅ e1 (x) . 0 0  

The direction of the corresponding eigenvector of λ1 is either +e1 (x) or −e1 (x) , and we take the average of all gradient directions of input images as the gradient direction of the fused image, which is expressed as follows:

V′(x) = λ1 (x) ⋅ e1 (x)sign e1 (x),

1 N

N

∑ ∇I

n

( x)

(7)

n =1

where ⋅, ⋅ represents the inner product of two vectors, and  1 if x ≥ 0 sign( x) =  else  −1

(8)

Structure tensor-based fusion algorithm chooses the maximum contrast as the feature of the fused image. However, for noisy image fusion, the noise information may correspond to the greatest contrast, and therefore, significantly reducing image fusion quality. An example for the fusion of thoracic CT scans of different windows is provided in Fig. 2. Fig. 2 (a) and (b) show the thoracic CT images (spectral band: 0.01-10 nm) of mediastinal window and lung window. Fig. 2 (c) is a noisy

interference image by adding zero mean white Gaussian noise. Fig. 2 (d) is the fused image of Figs. 2 (a) and (b) using structure tensor-based fusion algorithm. And Fig. 2 (e) is the fused image of Figs. 2 (a)-(c) using structure tensor-based fusion algorithm. Fig. 2 (f) is the fused image of Figs. 2 (a)-(c) by constructing a weight for each gradient of input images (will be introduced in section 2.2), and Fig. 2 (g) is the final fused image by the proposed algorithm. Figs. 2 (h)-(k) show the gradient magnitude fields of Figs. 2 (d)-(g), respectively.

(a)

(d)

(b)

(e)

(c)

(f)

(g)

(h) (i) (j) (k) Fig. 2. The fusion of thoracic CT scans of different windows (a) Thoracic CT scans of mediastinal window; (b) Thoracic CT scans of lung window; (c) Noisy interference image; (d) The fused image of (a) and (b); (e) The fused image of (a)-(c); (f) The fused image of (a)-(c) by constructing a weight for each gradient; (g) The final fused image by the proposed algorithm; (h)-(k) The corresponding gradient magnitude field of (d) - (g), respectively.

Using the matrix of structure tensor in gradient domain can effectively fuse the gradient information of thoracic CT scans, as shown in Fig. 2 (h). The gray image reconstructed from the fused gradient field preserves edge details of input images, as shown in Fig. 2 (d). But for the fusion of noisy multispectral images, it incorrectly fuses the gradient of noise instead of the structure of input images, as shown in Fig. 2

(i). Thus, the corresponding gray image contains a lot of noise, as shown in Fig. 2 (e). Figs. 2(f) and (j) are the results adopting the gradient entropy metric to construct a weight for each gradient of input images. The fused gradient field preserves structures of input images while reducing the interference of noise, and edge details of the fused image are clearer than Fig. 2 (e). In order to further suppress noise of the fused image, p -Laplace diffusion constraint is adopted in rebuilding the fused image from the

gradient field. The final fused image and the corresponding gradient magnitude field are shown in Figs. 2(g) and (k), respectively. The fused image preserves edge details of the input images while reducing noise effectively.

2.2 Gradient entropy metric for the weighted gradient

In this section, what we will do is combining the useful gradients of every band, making the fused image preserve the main features of the input images without impact of noise. According to [4], in the same area of multispectral image, the large gradients of one band may be noise, but the weaker gradients of another band may be features (i.e., edges). Therefore, structure tensor-based method for image gradient fusion (introduced in section 2.1) easily mistake noise as the useful feature. In order to make the gradients of the useful characteristics stand out from these bands without interference of noise, gradient entropy metric is adopted to structure a weight for the gradient

of each

band.

For

each band

of I n (n = 1, 2,..., N )

and

each

pixel x = ( x1 , x2 ) ⊂ Ω , the entropy metric of the gradient ∇I n ( x1 , x2 ) is in the neighborhood of ( x1 , x2 ) . The gradient entropy metric of S n (x) is structured as follows: max ΘW ( x )

S n ( x) = −

∑

p g (i ) log 2 p g (i )

(9)

i = min ΘW ( x )

where W (x) represents the neighborhood of x with the size of 7 × 7 , min ΘW (x) and max ΘW (x) represents the maximum and minimum gradient values in the

window W (x) , respectively, and p g (i ) represents the ratio of the number of the pixels with gradient value of i and the total number of the pixels in the window W (x) . For a band with useful features, its gradients are usually gathered at the edge

and orderly distributed. But for a noisy band, its gradients are distributed randomly in a large region. Therefore, based on equation (9), the gradient entropy of the noisy band is larger than that of the noise-free band. To prevent the fused gradient field from being affected by noise, the band with larger gradient entropy should be weighted less as it is noisier. There is another fact that the gradient entropy of a band without features is smaller than that of the featured band. And to extract the image features, the band with smaller gradient entropy should also be weighted less. Synthetically considering the above situations, the weight of wn (x) is constructed as follows: wn′ (x) = wn (x ) =

1

σ

−

( S n ( x ) − mid ( Sn ( x )))2 2σ 2

e

(10)

wn′ (x)

(11)

N

∑ w′(x) l

l =1

where mid ( S n (x)) represents the median of S n (x) (n = 1, 2,..., N ) if N being odd, otherwise, mid ( S n (x)) represents the average of median of S n (x) , and σ is the variance. Here we take N  1 σ =  ∑  S n ( x) − N  n =1 

1

2  2 Sl ( x)   ∑ l =1   N

(12)

Fig. 3 shows an example. Figs. 3(a)-(c) are the partial enlarged views of gradient magnitude fields of Figs. 2(a)-(c), respectively. Fig. 3(d) is the fused gradient magnitude field by constructing a weight for each gradient of Figs. 2(a)-(c), and Fig. 3(e) is the gray image reconstructed from Fig. 3(d). Corresponding to the red windows of Figs. 3(a)-(c), the gradient entropy is expressed as S1 , S 2 and S3 , respectively. Based on equation (9), we can get S1 < S 2 < S 3 . According to equations

(10)-(12), the gradient of the center of red window in Fig. 3(b) is weighted more heavily. Therefore, the main features of Figs. 3(a)-(c) are preserved with noise suppressed, as shown in Figs. 3(d) and (e).

(a)

(b)

(c)

(d) (e) Fig. 3. Example of gradient entropy metric for the weighted gradient. (a)-(c) The partial enlarged view of gradient magnitude field of Figs. 2(a)-(c), respectively; (d) fused gradient magnitude field; (e) reconstructed gray image.

Therefore, the structure tensor with weighted gradient can be expressed as:

∂I n 2  N  ∑ (wn (x) ∂x ) n =1 1 G ′(x) =  N  ∂I n ∂I n 2  ∑ wn (x ) ∂x1 ∂x2  n =1

∂I n ∂I n   n =1 1 ∂x2  N  ∂I ( wn (x) n )2  ∑ ∂x2  n =1 N

∑ w (x) ∂x 2 n

(13)

where G ′(x) represents the structure tensor with weighted gradient. 2.3 Visualization of fused gradient field with p-Laplace diffusion constraint Based on section 2.1 and 2.2, the fused gradient field of V ′ has been constructed. What we will do next is rebuilding the fused fusion image from V ′ , that is, we find an image u whose gradient field is the closest to V ′ . A common method for this is using the least squares [27-29]. The visualization can be mathematically expressed as:

min

{∫∫ ∇u − V′ dxdy} 2

Ω

(14)

For noisy multispectral image fusion, we have adopted the gradient entropy metric to construct a weight for each gradient of input images to minimize the negative effects of noise. However, the image reconstructed by equation (14) still contains noise, as shown in Fig. 2(f). An interesting approach is using the TV semi-norm to smooth noise [21, 22, 28, 30]. However, there normally exists the staircase effect in TV model. In order to overcome the shortcoming, a local adaptive p -Laplace regularization variation model is proposed: min

{∫∫ ∇u

p ( ∇u )

Ω

dxdy

}

(15)

where p is diffusion control factor. According to different local information of the image, p is taken by different values adaptively: p ( ∇u ) = 1 + cos(

∇u − min ∇u

max ∇u − min ∇u

π

⋅ ) 2

(16)

where max ∇u and min ∇u represent the maximum and minimum gradient values of the image u , respectively. At the edge of the image, the gradient of ∇u is big. Based on equation (16), p is small with approximate value of 1. So p -Laplace diffusion is similar to TV model, which keeps the edge details of the image. At the smooth region of the image, the gradient of ∇u is small. Based on equation (16), p is large with approximate value of 2. So p -Laplace diffuses isotropically, which

overcomes the staircase effect. Adopting the p -Laplace regularization variation model, we minimize the following function instead of equation (14). g (u ) = min

{∫∫ ∇u − V′ dxdy + λ ∫∫ ∇u 2

Ω

Ω

p ( ∇u )

dxdy

}

(17)

Where λ is a positive parameter weighting the effect of the fusion and de-noising. In image processing, the variation method [30] is a well-known to solve equation (17). Using the variation method, we can deduce the corresponding partial differential equation:

 ∇u  −2(∆u − divV′) − λ∇ ⋅  q ( ∇u ) =0 ∇u  

(18)

where ∆ represents the Laplace operator, div represents the divergence operator, and q ( ∇u ) = p ( ∇u ) ∇u

p ( ∇u ) −1

+ ln p ( ∇u ) p′( ∇u ) ∇u

p ( ∇u )

(19)

Then, using the gradient descent procedure, the solution of equation (18) is:

 ∂u ∇u  = 2(∆u − divV ′) + λ∇ ⋅  q( ∇u )  ∂t ∇u  

(20)

In image processing, equation (20) should be utilized for discrete data set. In this research we use the finite difference method to obtain the numerical solution. In order to solve equation (20), the semi-implicit method [31] is used to ensure the convergence. So, equation (20) is converted into the evolution with the step-size t liking equation (21). To ensure the convergence, t should be a small positive value.       2(uin+1, j + uin−1, j + uin, j +1 + uin, j −1 − 4uin, j − divV′)        ∆ +x uin, +j 1    n +1 n n x  u = u + t  + λ q ( ∇u ) ∆ −    2 2 x n y n   ( ∆ + ui , j ) + ( ∆ 0 ui , j 2 )         y n +1  ∆ u    + i , j n y  + λ q ( ∇u ) ∆ −    2 2 y n x n  ∆ u + ∆ u 2  ( + i , j ) ( 0 i , j )   

(21)

where n represents the number of iterations. ∆ ±x ui , j = ±(ui ±1, j − ui , j ) , ∆ ±y ui , j = ± (ui , j ±1 − ui , j )

(22)

∆ 0x ui , j = ui +1, j − ui −1, j , ∆ 0y ui , j = ui , j +1 − ui , j −1

(23)

where i ( 0 ≤ i ≤ N − 1 ) and j ( 0 ≤ j ≤ M − 1 ) represent the coordinates of the length and width of the pixel location, respectively.

3. Experiments

In the proposed method, λ in (21) is a scalar of trade-off between the fidelity and the smoothness of the fused image. The number of iterations n is an adjustable parameter, making the result tend to the optimal solution in different degrees. Fig. 4 presents the influences of different n and λ . The source images are shown in Figs. 2(a)-(c).

(a)

(e)

(b)

(f)

(c)

(g)

(d)

(h)

(i) (j) (k) (l) Fig. 4. The influences of different λ and n on the results. (a)-(d) The results when λ takes 0.1 and n takes 50, 100, 150, 200, respectively; (e)-(h) The results when λ takes 0.35 and n takes 50, 100, 150, 200, respectively; (i)-(l) The results when λ takes 0.5 and n takes 50, 100, 150, 200, respectively.

It can be seen from Fig. 4 that when n is small (taking 50 and 100, respectively), with the decrease of n , the details of the fused image become increasingly clear. However, when n is big (taking 150 and 200, respectively), the improvement of the quality is weakened. With decrease of λ , the capability of noise suppression of the method is improved. However, the details become blurred. Comprehensively considering the detail preservation and noise suppression, we take λ = 0.35 and

n = 150 , and the result is shown in fig. 4(g). To verify the effectiveness of the proposed fusion method (GEMPL), comparing it with typical methods described above, namely image fusion with guided filtering

(IFGF) [12], multi-level local extreme-based fusion (MLLE) [9], contrast fusion based on structure tensor (CFST) [26], non-Gaussian model-based fusion in the wavelet domain (NGMF) [19], and the TV-based fusion (TVF) [21], we conduct a lot of experiments. In part 2.2 of this paper, the gradient entropy metric-based weight structure is proposed. To verify the effectiveness, we conduct experiments respectively using average gradient (AGPL) and median gradient (MGPL)-based weight structure instead, in comparison. Before the fusion, we assume that the input images to be fused are registered. Here are some experimental results.

(a)

(d)

(b)

(e)

(c)

(f)

(g) (h) (i) Fig. 5. Fusion results of noisy Lena images. (a and b) Noisy Lena images; (c) IFGF; (d) MLLE; (e) CFST; (f) NGMF; (g) TVF; (h) AGPL; (i) GEMPL.

Figs. 5(a) and (b) are partially noisy Lena images (spectral band: 400-760nm) by adding zero mean white Gaussian noise. Due to the fact that IFGF, MLLE and CFST mistakenly take noise as the useful characterization and do not suppress noise, the

fused images contain a lot of noise, as shown in Fig. 5 (c)-(e). GFMF effectively suppresses noise, but generates the ringing effect, as shown in Fig. 5 (f). Fig. 5 (g) is the result of TVF. It preserves edge details of input images and reduces noise, but produces the staircase effect. The result of AGPL is shown in Fig. 5(h). Because of using average gradient to construct the weight, image details become blurred. It is observed from Fig. 5 (i), that image obtained by GEMPL preserves edge detail features of the input images while suppressing noise effectively. Peak signal to noise ratio (PSNR) and signal to noise ratio (SNR) [32] are used as the quality criterions to evaluate the capability of the fusion and de-noising of the proposed method, as shown in Table 1. The best value of assessment is marked in bold. In comparison with noisy Lena image, PSNR of the fused image by GEMPL increases about 60.8% and SNR increases about 100.7%, which indicates good performance of proposed algorithm for image fusion and de-noising. Table 1 Quantitative criterions of different image fusion methods using SNR and PSNR noisy Lena noisy Lena image (a) image (b)

IFGF

MLLE

CFST

NGMF

TVF

AGPL

GEMPL

PSNR

20.9448

20.9448

19.6187

21.8944

24.0482

31.0644 28.6400 21.6932

33.6734

SNR

14.0642

14.0642

12.7381

15.0138

20.1204

27.1453 25.7001 14.8127

28.2255

(a)

(d)

(b)

(e)

(c)

(f)

(g)

(h) (i) (j) (k) Fig. 6. Fusion results of infrared, visible and noisy images. (a and b) Infrared and visible images; (c) noise image; (d) IFGF; (e) MLLE; (f) CFST; (g) NGMF; (h) TVF; (i) MGPL; (j) AGPL; (k) GEMPL.

Fig. 6 is the fusion results of infrared (spectral band: 0.78-1000 um), visible (spectral band: 400-760 nm) and noisy images. IFGF, MLLE and CFST are respectively used to fuse the detail features of Figs. 6(a) and (b) while incorporating noise, as shown in Fig. 6(d)-(f). Edge detail feature in Fig. 6(h) is clearer than that in Fig. 6(g), but it has significant staircase effect. Fig. 6 (i) is the result of MGPL. The median gradient may be the image detail, but also may be the noise. Therefore, using median gradient to construct the weight, the fused image is with spots. AGPL smoothes image details, as shown in Fig. 6 (j). In comparison, the proposed method effectively fuses detail features of the original images and suppresses noise, as shown in Fig. 6(k).

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i) (j) (k) (l) Fig. 7. Fusion results of noisy medical images. (a and b) MRI and CT images; (c and d) noisy MRI and CT images; (e) IFGF; (f) MLLE; (g) CFST; (h) NGMF; (i) TVF; (j) MGPL; (k) AGPL; (l) GEMPL.

Fig. 7 is the fusion results of noisy medical images, MRI and CT images (spectral band: 0.01-10 nm). Figs. 7(e)-(g) are the fusion of Figs. 7(a)-(d) by IFGF, MLLE and CFST, respectively. Because there is no noise suppression, fused image contains a lot of noise. In Figs. 7(h)-(k), noise is suppressed, but image detail features are fuzzy. Fig. 7 (l) is the fusion result by GEMPL. The edge detail features are clear and the visual effect is the best.

(a)

(b)

(d)

(e)

(h)

(i)

(c)

(f)

(j)

(g)

(k)

Fig. 8. Fusion results of multispectral images. (a) Panchromatic image; (b) red light image; (c) noisy green light image; (d) IFGF; (e) MLLE; (f) CFST; (g) NGMF; (h) TVF; (i) MGPL; (j) AGPL; (k) GEMPL.

(a)

(d)

(b)

(e)

(c)

(f)

(g)

(h) (i) (j) (k) Fig. 9. Multispectral images and the fused images obtained by different methods. (a) Panchromatic image; (b) red light image; (c) noisy green light image; (d) IFGF; (e) MLLE; (f) CFST; (g) NGMF; (h) TVF; (i) MGPL; (j) AGPL; (k) GEMPL.

Figs. 8 and 9 show two multispectral image fusion: panchromatic image (spectral band: 450-900 nm), red light image (spectral band: 630-690 nm) and green light image (spectral band: 520-660 nm) added noise of 17.08 and 18.56 dB in SNR, respectively. It can be seen from Figs. 8(d)-(f) and 9(d)-(f) that the images produced by IFGF, MLLE and CFST are with fused detail features but a lot of noise. NGMF over smoothens some visually important features, as shown in Figs. 8(g) and 9(g). As shown in Figs. 8(h) and 9(h), TVF produces the staircase effect. Figs. 8(i) and 9(i) are the results of MGPL. The fused images are with spots, and thus have bad visual effect. As shown in Figs. 8(j) and 9(j), AGPL makes image details blurred. However,

GEMPL can preserve image features and details, and suppress noise effectively as well, as shown in Figs. 8(k) and 9(k). Different types of images have been applied to be fused for the visual comparison, and the results show that the proposed method effectively preserves edge detail features of the input images while suppressing noise. For the objective evaluation of the fused results, peak signal to noise ratio (PSNR), signal to noise ratio (SNR), mean absolute error (MAE), root mean square error (RMSE) and correlation (CORR) [32,33] are selected. The reference images for the quantitative measurements are shown in Fig. 10. The quantitative assessments of different methods for Figs. 6-9 are shown in table 2, and are represented graphically in Fig. 11.

(a) (b) (c) (d) Fig. 10. Reference images for the quantitative measurements. (a)-(d) are the reference images of Figs. 6-9, respectively. Table 2 The quantitative assessments of different methods.

Fig. 6

Fig. 7

Fig. 8

Index

IFGF

MLLE

CFST

TVF

MGPL

AGPL

GEMPL

PSNR

21.6814

19.3720

21.5896

24.2916

NGMF

21.4155

19.6719

21.7072

27.9100

7.1090

9.3266

12.0286

9.1525

7.4089

9.4442

15.6470

SNR

9.4184

MAE

20.4592

12.7147

16.1688

9.2374

11.2028

16.4499

11.4300

6.3900

RMSE

27.4119

21.0121

21.2355

15.5582

21.6654

26.4816

20.9498

10.2575

CORR

0.8804

0.9422

0.9415

0.9666

0.9393

0.9179

0.9399

0.9859

PSNR

15.6453

15.8778

18.8953

20.7790

23.4925

17.9458

16.5683

24.5299

SNR

2.9976

3.2301

6.2477

8.1313

10.8449

5.2982

3.9206

11.8822

MAE

24.3260

22.7093

16.9711

15.4761

13.2833

21.3824

RMSE

42.0993

40.9876

28.9583

23.3126

17.0574

32.3035

27.8552

15.1372

CORR

0.7620

0.8057

0.8769

0.9166

0.9583

0.8310

0.8701

0.9876

PSNR

13.8633

15.8760

25.4522

18.1900

17.0774

26.6100

8.6437

7.5311

17.0637

16.8611

24.4939

SNR

4.3170

6.3297

7.3148

14.9476

15.9059

MAE

45.0270

32.5501

31.1465

9.8050

7.9136

RMSE

51.6864

40.9957

36.6002

15.2000

CORR

0.8885

0.9195

0.9325

0.9840

19.5511 11.8378

19.1610

30.5565

5.7276

13.6123

31.4079

35.7002

11.9135

0.9873

0.9462

0.9359

0.9902

Fig. 9

PSNR

13.4038

14.0487

15.2366

23.8745

22.5830

13.2474

15.3987

30.2449

SNR

3.4108

4.0557

5.2436

13.8815

12.5900

3.2544

5.4057

20.2519

35.9621

5.2892

MAE

47.4645

39.8338

36.4555

9.6560

11.5552

44.1339

RMSE

54.4943

50.5947

44.1275

16.3236

18.9405

55.4845

CORR

0.8701

0.8728

0.8957

0.9792

0.9732

0.8576

43.3119 0.9004

7.8396 0.9954

Table 2 objectively reflects the effect of different fusion methods with the best value of assessment marked by bold. The values of quantitative assessment show that the proposed method is the best among the fusion methods. Fig. 11 is the graphic representation of table 2.

(a)

(b)

(c) (d) Fig. 11. Graphical representation of quantitative assessments of different fusion methods. (a)-(d) are the graphical representations of quantitative assessments for Figs. 6-9, respectively.

4. Conclusion

In this paper, a novel method for noisy multispectral image fusion in the gradient domain is proposed. The gradients can reflect the changes of local details of images, therefore becoming the best candidates for feature selections for image fusion. In order to prevent the fused gradient field based on the matrix of structure tensor from being affected by noise, the gradient entropy metric-based weights for each gradient of input images are constructed. This method effectively prevents the extractions of image details and edge features from noise interference. Additionally, adding

p -Laplace diffusion constraint in reconstruction of the fused image suppresses the noise further. The results show that the proposed method can be effectively used for multispectral image fusion and de-noising, with better performance compared to other existing methods. Future research is expected to test the adaptability of the parameters.

Acknowledgements The authors thank the reviewers and editors for their helpful and constructive suggestions. This work is supported by National Natural Science Foundation of China (Grant Nos. 61137001).

References [1] S. Witte, V. T. Tenner, D. W. Noom, and K. S. Eikema, Lensless diffractive imaging with ultra-broadband table-top sources: from infrared to extreme-ultraviolet wavelengths, Light: Science & Applications 3 (2014) e163:1-8. [2] P. Girshovitz, and N. T. Shaked, Doubling the field of view in off-axis low-coherence interferometric imaging, Light: Science & Applications 3 (2014) e151:1-9. [3] G. Piella, A general framework for multiresolution image fusion: from pixels to regions, Inf. Fus. 4 (4) (2003) 259-280. [4] D. A. Socolinsky, A locally-adaptive metric for contrast-fusion of noisy multimodal imagery, Proc. Signal and Image Processing 2000, IASTED, (2000) 1-7.

[5] X. Z. Bai, and Y. Zhang, Detail preserved fusion of infrared and visual images by using opening and closing based toggle operator, Optics & Laser Technology 63 (2014) 105-113. [6] R. Singh, and A. Khare, Fusion of multimodal medical images using Daubechies complex wavelet transform-A multiresolution approach, Information Fusion 19 (2014) 49-60. [7] G. M. Cui, H. J. Feng, Z. H. Xu, Q. Li, and Y. T. Chen, Detail preserved fusion of visible and infrared images using regional saliency extraction and multi-scale image decomposition, Optics Communications 341 (2015) 199-209. [8] G. Bhatnagar, Q. M. J. Wu, and Z. Liu, A new contrast based multimodal medical image fusion framework, Neurocomputing 157 (2015) 143-152. [9] Z. P. Xu, Medical image fusion using multi-level local extrema, Information Fusion 19 (2014) 38-48. [10] Y. Jiang, and M. H. Wang, Image fusion with morphological component analysis, Information Fusion 18 (2014) 107-118. [11] P. Balasubramaniam, and V. P. Ananthi, Image fusion using intuitionistic fuzzy sets, Information Fusion 20 (2014) 21-30. [12] S. T. Li, X. D. Kang, and J. W. Hu, Image Fusion with Guided Filtering, IEEE Trans. Image Process. 22 (7) (2013) 2864-2875. [13] Z. F. Shao, J. Liu, and Q. M. Cheng, Fusion of infrared and visible images based on focus measure operators in the curvelet domain, Appl. Opt. 51 (12) (2012) 1910-1921. [14] X. Z. Bai, Image fusion through feature extraction by using sequentially combined toggle and top-hat based contrast operator, Appl. Opt. 51 (31) (2012) 7566-7575. [15] A. Goshtasby, and S. Nikolov, Image fusion: advances in the state of the art, Inf. Fus. 8 (2) (2007) 114-118. [16] Y. X. Zhang , L. Chen , J. Jia , and Z. H. Zhao, Multi-focus image fusion based on non-negative matrix factorization and difference images, Signal Processing 105 (2014) 84-97. [17] J. Lewis, R. O. Callaghan, S. Nikolov, D. Bull, and N. Canagarajah, Pixel- and region-based image fusion with complex wavelets, Inf. Fus. 8 (2) (2007) 119-130. [18] P. Scheunders, and S. D. Backer, Wavelet Denoising of Multicomponent Images Using Gaussian Scale Mixture Models and a Noise-Free Image as Priors, IEEE Trans. Image Process. 16 (7) (2007) 1865-1872. [19] A. Loza, D. Bull, N. Canagarajah, and A. Achim, Non-Gaussian model-based fusion of noisy images in the wavelet domain, Computer Vision and Image Understanding 114 (1) (2010) 54-65. [20] A. Loza, A. Achim, D. Bull, and N. Canagarajah, Statistical Model-based Fusion of Noisy Multi-band Images in the Wavelet Domain, International Conference on Information Fusion (2007) 1-6. [21] M. Kumar, and S. Dass, A Total Variation-Based Algorithm for Pixel-Level Image Fusion, IEEE Trans. Image Process. 18 (9) (2009) 2137-2143.

[22] W. W. Wang, P. L. Shui, and X. C. Feng, Variational Models for Fusion and Denoising of Multifocus Images, IEEE Signal Processing Letters 15 (2008) 65-68. [23] T. F. Chan, S. Esedoglu, and F. Park, A fourth order dual method for staircase reduction in texture extraction and image restoration problems, IEEE Int. Conf. Image Process. (2010) 4137-4140. [24] V. S. Petrovic, and C. S. Xydeas, Gradient-based multiresolution image fusion, IEEE Trans. Image Process. 13 (2) (2004) 228-237. [25] D. A. Socolinsky, and L.B. Wolff, Multispectral image visualization through first-order fusion, IEEE Trans. Image Process. 11 (8) (2002) 923-931. [26] G. Piella, Image fusion for enhanced visualization: a variational approach, Int. J. Comput. Vision 83 (1) (2009) 1-11. [27] W. D. Zhao, Z. J. Xu, J. Zhao, F. Zhao, and X. Z. Han, Infrared image detail enhancement based on the gradient field specification, Appl. Opt. 53 (19) (2014) 4141-4149. [28] C. Wang, and Z. F. Ye, Perceptual Contrast-Based Image Fusion: A Variational Approach, Acta Automatica Sinica 33 (2) (2007) 132-137. [29] C. Wang, and Z. F. Ye, First-order fusion of volumetric medical imagery, IEE Proc.-Vis. Image Signal Process. 153 (2) (2006) 191-198. [30] L. Rudin, S. Osher, and E. Fatemi, Nonlinear total variation based noise removal algorithms, Physica D 60 (1) (1992) 259-268. [31] C. Samson, L. B. Feraud, G. Aubert, et al., A level set model for image classification, International Journal of Computer Vision 40 (3) (2000) 187-197. [32] P. Balasubramaniam, and V. P. Ananthi, Image fusion using intuitionistic fuzzy sets, Inf. Fus. 20 (2014) 21-30. [33] S. Daneshvar, and H. Ghassemian, MRI and PET image fusion by combining IHS and retina-inspired models, Inf. Fus. 11 (2) (2010) 114-123.

Graphical Abstract:




Highly promising and applied noisy multispectral image fusion




Adopting the matrix of structure tensor to fuse the gradient information




Gradient entropy metric-based weighted gradient to extract image features without noise interference




Local adaptive p -Laplace diffusion constraint is constructed while rebuilding the fused gradient field