Multi-focus image fusion algorithm based on compound PCNN in Surfacelet domain

Optik 125 (2014) 296–300

Contents lists available at ScienceDirect

Optik journal homepage: www.elsevier.de/ijleo

Multi-focus image fusion algorithm based on compound PCNN in Surfacelet domain Baohua Zhang ∗ , Chuanting Zhang, Liu Yuanyuan, Wu Jianshuai, Liu He School of Information Engineering, Inner Mongolia University of Science and Technology, Baotou, Inner Mongolia 014010, PR China

a r t i c l e

i n f o

Article history: Received 14 January 2013 Accepted 21 June 2013

Keywords: Compound PCNN Surfacelet Local sum-modified-Laplacian Multi-focus image fusion

a b s t r a c t In order to effectively retain details and suppress noise, a multi-focus image fusion method based on Surfacelet transform and compound PCNN is proposed. Surfacelet transform is a powerful multi-resolution analysis tool which is able to decompose the original image into a number of different frequency band sub-images, compound PCNN model is a combined model of PCNN and dual-channel PCNN which is to select the fusion coefficients from the decomposed coefficients, the Local sum-modified-Laplacian (LSML) is selected as external stimulus of compound PCNN, fusion coefficients are decided by compound PCNN. The experimental results show that the new method has a good performance, fusion image has more texture details and it is more similar to the original images, the objective evaluation indexes show that this method is superior to the traditional image fusion methods. © 2013 Elsevier GmbH. All rights reserved.

1. Introduction Image fusion is a process of combining complementary information from multiple sensor images to generate a single image that contains a more accurate description of the scene than any of the individual images. The image fusion methods mainly include the fusion methods based on space domain and transform domain [1–4]. Image fusion methods need to consider the characteristics of human visual system [5–7]. The visual receptive field of cerebral cortex has the characteristics of locality, directionality and band-pass. These characteristics make human get the key information using the least neurons in a natural scene, which is sparsely represented or it means the sparsest coding for natural scene. So efficient nonlinear image approximation should be achieved to detect, classify, estimate and study natural images. For 2D image, the edge of the image includes much information and wavelet transform (WT) extracts the discrete parts of the image edge well while it is not ideal at the continuous parts. The edge detected method based on Surfacelet overcomes the limitation of insufficient decomposition angles. Decomposition with DTWT (Discrete Time wavelet Transform) only has 28 directional sub-bands [8], but the number of sub-band in the most elaborate direction of Surfacelet Transform can be up to 192 or more. The characteristics of Surfacelet coefficients and edges are effectively used for extracting the edges of high frequency sub-band [9].

∗ Corresponding author. E-mail address: zbh [email protected] (B. Zhang). 0030-4026/$ – see front matter © 2013 Elsevier GmbH. All rights reserved. http://dx.doi.org/10.1016/j.ijleo.2013.07.002

On the other hand, the edge detection method based on DWT discards the low frequency information and only processes the high frequency coefficients, so the achieved edge of the image often shows cracking partly and cannot describe the edge of image consistently. Although the modified method can mend the cracked parts using the breakpoint connecting algorithm and can accurately connect the minimum distance gaps, it cannot solve the problems that how to connect the long-path gap accurately and effectively using the low frequency information. In Surfacelet, anisotropic basis functions at different scales and the edge singularity in the image are proposed to process the singularity. The coefficients in the high frequency sub-band decomposed by Surfacelet transform are divided into three categories: strong fringing coefficient, weak fringing coefficient and noise coefficient. In the same scale, strong fringing coefficients in the same position of all sub-sands present a larger modulus; weak fringing coefficients in some certain directions presents a larger modulus, but in other directions show a smaller modulus. As to noise coefficients, the coefficients in all directional sub-sands present a smaller modulus. The noises can be conveniently detached with the absolute value of the modulus. In the image fusion algorithm based on MSD (multiscale decomposition) field [10,11], the quality of image fusion largely depends on the selection of the fusion rules. Pulse-coupled neural network (PCNN) is a novel artificial neural network model developed by Eckhorn in 1990 and based on the experimental observations of synchronous pulse bursts in cat and monkey visual cortex [12,13]. It has some characteristics, such as: pulse coupling, variable threshold, generating synchronous pulse and multiplication modulation, etc. The same image which is proposed using PCNN has good

B. Zhang et al. / Optik 125 (2014) 296–300

297

Fig. 2. The model of Dual-channel PCNN.

Fig. 1. The frequency decomposing image of Surfacelet transform in two dimensions.

upsample and downsample, which will finally achieve the condition of reconstruction, as shown in formula (1): |Li (ω)|2 di2

stability in certain conditions. For the advantages of PCNN, it can be used for image fusion well. However, there are some factors which lead to the difficulty of the mathematical analysis to the network in the standard PCNN, such as nonlinear, leakage integration, numerous uncertain parameters and so on. The dual-channel PCNN used for medical image fusion is proposed by Wang and Ma to solve the problem of leakage integration [14]. In order to solve the other problems of standard PCNN, another dual-channel PCNN is proposed by Chai and Li. However, the fusion image in the paper [14] cannot be output directly and it must be linearly transformed. In paper [7], the direction information measure of decomposition coefficient is selected as the linking weight (the linking weights in linking input and feeding input are 0.5). It approximates the feedback essence of neural network. Firing position is changed after iterations, when the next iteration starts; the linking weight depends on the output of last iteration. The outline of the paper is as follows. We introduce the Surfacelet transform in Section 2 to state the advantages of Surfacelet transform and the compound PCNN model. In Section 3, the fusion method based on the compound PCNN model is proposed and the fusion steps are introduced in details. In Section 4, we detailed analyze the advantages of the algorithm proposed in the paper and comparing with classical methods of multi-focus image fusion. In Section 5, it is conclusions.

2. Surfacelet transform and compound PCNN 2.1. N dimensional directional filter bank Bamberger proposed the directional filter bank (DFB) for a directional decomposition of 2D signals. On this basis, Yue Lu proposed new filter banks for N-dimensional (N ≥ 2) signals, named NDFB, that can achieve the N-directional decomposition with a simple and efficient tree-structured construction. Fig. 1 is the sketch maps of frequency splitting when NDFB is in the condition of two dimensions respectively. Although this method has much redundancy information than LP, it avoids the mixing phenomenon that NDFB produces in the frequency domain. The mixing phenomenon mainly focuses on the boundary of the frequency unit (−, )2 , which results from the periodicity in 2 of discrete signal spectrum. This specific sampling method also makes the decomposing coefficient of Surfacelet well express in the space and frequency domain. The main advantage of designing this filter in the space domain is that the frequency response of the image can be strictly controlled (once the frequency response exceeds the terminational frequency, it will be zero). ωs,i (i = 1, 2), ωp,A and ωs,A are selected reasonably can totally eliminate the mixing distortion phenomenon that results from the

+ |Di (ω)|2 ≡ 1,

i = 0, 1

(1)

In order to avoid mixing, the channel and stop-band frequency are selected as ωs,i (i = 1, 2) = 23 , ωp,1 = 4 and ωs,1 = 2 , and the frequency parameters of S (ω) to be ωp,A = 3 and ωs,A = 2 . Once 3 the lowpass filters have been chosen, the highpass filters Di (ω) can be obtained to ensure perfect reconstruction. 2.2. Compound PCNN PCNN has complex structure and computation time long, at the same time improved simplified model has shortcomings of uneven brightness zone resolution, and it is not conducive to dim image information extracting. In contrast, dual-channel PCNN has an advantage in this regard. The low-frequency sub-image is significantly dim after Surfacelet transform decomposition; Combination model has the advantage to select different fusion coefficients. Aim to shortcomings of PCNN model in image fusion domain, combines with the characteristics of multi-focus image, this paper proposes compound PCNN. Compound PCNN is motivated by the Local summodified-Laplacian which is based on the dual-channel PCNN and PCNN. The composition of the simplified PCNN neurons include: the receptive field, modulation domain and pulse generating domain. The simplified PCNN and the dual-channel PCNN can be expressed as follows: The receptive field: Fij [n] = Sij [n] Lij [n] = e−˛L Lij [n − 1] + VL

(2)



Wijkl Ykl [n − 1]

(3)

kl

Modulation domain: Uij [n] = Fij [n](1 + ˇLij [n])

(4)

Pulse generating domain:



Yij [n] =

1, Uij [n] > Tij [n]

(5)

0, otherwise

Tij [n] = e−˛T Tij [n − 1] + VT Yij [n]

(6)

The dual-channel PCNN is different from PCNN in composition of neurons, as shown in Fig. 2: The receptive field: FijA [n] = SijA [n]

(7)

FijB [n] = SijB [n]

(8)

298

B. Zhang et al. / Optik 125 (2014) 296–300

LFA

PCNN Fused Coefs

LFB

PCNN

Compound PCNN HFA dual channel PCNN

Fused Coefs

HFB

Fig. 3. The compound PCNN model.

Lij [n] = e

−˛L

Lij [n − 1] + VL



averagely and redundant information is generated, images are distorted, the resolution of target reduces, all of which are inconvenient to process images. The characteristics of an image cannot be represented by pixel values. In high frequency sub-bands, the larger coefficients correspond to some extremums, such as edges and textures and other image features. The Local sum-modified-Laplacian is regarded as a measure metrics of high frequency component. The spatial frequency reflects the activity of an image in space domain. In order to meet the Human Visual System (HVS), the proposed method uses the Local sum-modified-Laplacian to select low frequency and high frequency sub-bands which can show the edgecharacteristics of an image well. It is also the external incentive of dual-channel PCNN. The followings are the definitions of the Local sum-modified-Laplacian: ML(i, j) = |2I(i, j) − I(i + 1, j) − I(i, j − 1)|

Wijkl Ykl [n − 1]

(9)

+ |2I(i, j) − I(i, j − 1) − I(i, j + 1)|

(13)

kl

Modulation domain: Uij [n] = max(FijA [n](1 + ˇijA [n]Lij [n]), FijB [n](1 + ˇijB [n]Lij [n]))

(10)

Pulse generating domain:



Yij [n] =

1, Uij [n] > Tij [n] 0,

 x

Wl (x, y)[ML(i + x, j + y)]

2

(14)

y

I(i,j) is the decomposition coefficient  in the coordinate (i,j). Wl

Wl (a, b) = 1. The size of Wl

is a template and its value is (11)

otherwise

Tij [n] = e−˛T Tij [n − 1] + VT Yij [n]

LSML =

(12)

where FijA and FijB are external outputs, Sij , SijA [n], S Bij [n] represent the input stimulus, n represents the number of iterations, Lij is linking input. The model that the paper proposed is the simplified PCNN, which deletes the feedback loop of link input and makes calculations easier. ˇ is the linking weight. wijld is synaptic connections, VL , VT is a normalization constant, Yij is pulse output of the neuron, its value is 0 or 1, Tij is the dynamic threshold, ˛L , ˛T is the constant, Uij is the firing mapping image, and it is the internal activity of neuron. The final images can be achieved using Uij . If Uij [n] > Tij [n], the neuron generates a pulse, called a firing. In fact, after n iterations, the number of firings of the neuron is used to represent the information of the image at the corresponding point. Firing map is constructed by the number of firings and is regarded as output of PCNN. The compound PCNN model used in this paper is shown as Fig. 3. The model includes two PCNN and a dual channel PCNN. The low frequency (LF) coefficients of the source images are input into PCNN respectively, the high frequency (HF) coefficients are input into the two channels of dual channel PCNN. 3. Image fusion algorithms 3.1. Fusion rules The fusion rules have an important impact on fusion results. In the existing PCNN multi-focus image fusion algorithm, there are many ways to process external stimulation. In the paper [13], Summodified-Laplacian (SML) [15] is used for processing low and the high frequency coefficients. And in the paper [16], low frequency coefficients are processed using image visibility and high frequency coefficients is processed using spatial frequency. Above all, different fusion rules have different impacts on fusion results. Most of the original image information is retained by low frequency sub-band. The averaging method is used for most of fusion rules of low frequency images. It fits to process some images which are extracted and shrunk; the size of original images is the same with the low frequency sub-band images after Surfacelet transform. Weighted average fusion rules can compute image information

is 3 × 3. In order to template ⎛ is adopted √ 2 ⎜ 2 1 1 1 √ ⎜ 1 5 + 4 2 ⎝ √2 1 2

a

b

emphasize the central pixels, the weighted and its value is shown as follows: Wl (x, y) = √ ⎞ 2 2 ⎟ ⎟ √1 ⎠ 2 2

3.2. Fusion steps Suppose N is the time of iteration about compound PCNN and detail fusion steps are as follows: 1) Every image can get a series of high and low frequency sub-bands after decomposing image A and B using Surfacelet transform. 2) The normalization of high and low frequency sub-bands. For the coefficients of sub-bands about the image, according to formula (13), window mechanics is introduced to achieve the Local sum-modified-Laplacian of the coefficients. 3) Low and high frequency coefficients are selected after normalization using compound PCNN. 3.1) Initialization: Lijl [0] = 0, Uijl [0] = 0, ijl [0] = 0, Yijl [0] = 0. l is the lth directional sub-band. 3.2) According to formulas (2)–(6) and formulas (7)–(12), respectively calculate Lijl [n], Uijl [n], ijl [n], Yijl [n]. 3.3) The Local sum-modified-Laplacian of decomposed coefficients achieved from step 2 are regarded as the external simulation of PCNN and the dual-channel PCNN. 3.4) If n = N is satisfied, the iteration finishes. The selection rule of fusion coefficients is as follows:



CoefFijk,l

=

CoefAk,l , if(Uijl [N] = FijA,l [N](1 + ˇijA,l Lijl [N]) ij CoefBijk,l , if(Uijl [N] = FijB,l [N](1 + ˇijB,l Lijl [N]) (15)

CoefFijl,k

CoefAk,l ij

where is the fusion coefficient. and CoefBijk,l are respectively the corresponding directional sub-band coefficients in the initial images A and B. In step 3, all high frequency sub-band fusion coefficients and a low frequency sub-band fusion coefficient can be got. The achieved

B. Zhang et al. / Optik 125 (2014) 296–300

Fig. 5. (a)–(d) are fusion images using the proposed method, Laplacian-based method, DWT-based method, PCA-based method, respectively.

Fig. 4. Source images and reference images in this experiment: DISK.

coefficients are inversed based on Surfacelet inverse transform and the fusion image is achieved.

N M Q

AB/F

=

n=1

4.1. Objectivity evaluation index 4.1.1. Mutual information Mutual information expresses how much information of source image is fused into the result image [17]. The larger the mutual information is, the better the fusion effect. Supposed that A and B are source images and F is the fused image, then the calculation of mutual information between A and F, B and F are respectively shown as formulas (16) and (17): pAF (a, f ) log

pAF (a, f ) pA (a)pF (f )

(16)

pBF (b, f ) log

pBF (b, f ) pB (b)pF (f )

(17)

a,f

IBF =

 b,f

(Q AF (n, m)wA (n, m) + Q BF (n, m)wB (n, m))

N M

j=1

(wA (i, j) + wB (i, j)) (19)

Experiments are done to verify the algorithm proposed in the paper. Experimental images are shown in Fig. 4. Fig. 4(a)–(c) are the source image and reference image “DISK” in the experiment. As shown in Fig. 4(a) and (b), each image contains multiple objects at different distances from the camera. The focus in Fig. 4(a) is on the clock, while that in Fig. 4(b) is on the book. In order to evaluate the fusion result, a reference image is introduced in the experiment. Fig. 4(c) is the perfect-focus fusion images in the laboratory of Lehigh University. All these images are considered as referenced images to compare with fusion images attained from the proposed algorithm in the paper and other algorithms. In order to evaluate advantages of the proposed algorithm in the paper, the proposed algorithm is respectively compared with PCA, DWT and Laplacian method. In the experiment, parameters of compound PCNN are as follows: ␣L = 0.001, ␣ = 0.1, VL = 0.2, V␪ = 25,

 0.707 1 0.707 and W = . 1 0 1 0.707 1 0.707



m=1

i=1

4. Experimental results and analysis

IAF =

299

Q AF (n, m) = QgAF (n, m)QaAF (n, m) with 0 ≤ Q AF (n, m) ≤ 1, QgAF (n, m) and QaAF (n, m) model perceptual loss of information in F, in terms of how well the strength and orientational values of a pixel (n, m) in A or B are represented in the fusion image. If A = 0, it corresponds to the complete loss of edge information at location (n, m) as transferred from A or B into F. If A = 1, it indicates “fusion” from A to F with no loss of information. wA (n, m) and wB (n, m) reflect the importance of QAF (n, m) and QBF (n, m). 4.1.3. Structural similarity SSIM includes three parts: luminance, contrast ratio and structure [19]. The formula is shown as formula (20): SSIM(x, y) = l(x, y)c(x, y)s(x, y) where l(x, y) =

2ux uy +C1 2 u2 x +uy +C1

, c(x, y) =

(20) 2x y +C2 x2 +y2 +C2

, s(x, y) =

xy +C3 x y +C3

are

respectively the luminance, contrast ratio and structure function of x and y, x and y are the source image block and degraded image block. ux and uy are the average value of x and y;  x and  y are the standard deviation of x and y;  xy is the covariance of x and y; C1 , C2 , C3 are constants. The higher the structural similarity is, the image x and y are similar and the fusion result is better. 4.2. Analysis of fusion results Fig. 5 illustrates the fusion results obtained by the above methods. For a clearer comparison, Fig. 6 magnifies parts of the fusion results to show their differences. Fig. 6(a) is the reference image; the labeled region is selected in Fig. 6(a), Fig. 6(b)–(e) are the magnified images of Fig. 5(a)–(d). In Fig. 6(b)–(e), it is evidently to find

where pAF and pBF are Joint Histogram of the source images and F, pA , pB , pF are respectively the histogram of A, B, C and a, b, f are the pixel values of respective image. The calculation of mutual information is shown as formula (18): MI = IAF + IBF

(18)

4.1.2. QAB/F QAB/F expresses how much edge information is fused from the source image into the fusion image [18]. The larger the value of QAB/F is, the more edge information will be fused into the fusion image. The better fusion result is achieved. QAB/F is shown as formula (19):

Fig. 6. Parts of the images of Fig. 5, (a) is the reference image, (b)–(e) are taken from Fig. 5(a)–(d), respectively.

300

B. Zhang et al. / Optik 125 (2014) 296–300

analysis above, the conclusion is that the proposed algorithm also shows significant improvement over the traditional fusion methods. Acknowledgements This work was supported by the National Natural Science Foundation of China (61179019), (61261028), Ministry of Education project (Z2009-1-01033), Inner Mongolian Natural Science Foundation (2010MS0907), Inner Mongolian College research projects (NJ10097). Fig. 7. Fusion method objective evaluation comparison.

that the fusion image of the proposed method is clearer than the other methods. Moreover, the fusion result of DWT-based method, as shown in Fig. 6(d), introduces many ‘artifacts’ around edges because DWT lacks shift-invariance. It is proven that the proposed method can decrease the pseudo-Gibbs phenomena successfully and improve the quality of the fusion image. Fig. 6(c) indicates that the Laplacian-based method provides better performance in fusion multi-focus images compares with DWT-based and PCAbased methods. However, from Fig. 6(c) it is not extracted all useful information of the source images perfectly. Fig. 6(b)–(e) show that the fusion images attained by the proposed method has the best visual quality. So Fig. 5(a) contains most of the useful information of the source images, and meantime, fewer artifacts are introduced during the fusion process. For further comparison, besides visual observation, three objective criteria are used to compare the fusion results. The first criterion is the mutual information (MI). It is a metric defined as the sum of mutual information between each input image and the fused image. The second criterion is QAB/F metric, which considers the amount of edge information transferred from the input images to the fused image. This method uses a Sobel edge detector to calculate strength and orientation information at each pixel in both source and the fusion images. In comparison, Fig. 7 shows the objective criteria on mutual information (MI), QAB/F and SSIM of Fig. 5(a)–(d). 5. Conclusion In this paper, we present a new multi-focus image fusion method based on Surfacelet and compound PCNN. All the objective criteria above prove that the fusion images of the proposed method contain more image features, i.e., edges, are preserved in the fusion process. All of the comparisons indicate that the proposed method outperforms Laplacian, DWT and PCA fusion methods, whether in subjective evaluation or objective evaluation criterion. From

References [1] G. Piella, A general framework for multiresolution image fusion: from pixels to regions, Inf. Fusion 4 (2003) 259–280. [2] G. Pajares, J. Manuel de la Cruz, A wavelet-based image fusion tutorial, Pattern Recogn. 37 (2004) 1855–1872. [3] Y. Chai, H. Li, M. Guo, Multifocus image fusion scheme based on features of multiscale products and PCNN in lifting stationary wavelet domain, Opt. Commun. 284 (2011) 1146–1158. [4] S. Li, B. Yang, Hybrid multiresolution method for multisensor multimodal image fusion, Sens. J. IEEE 10 (2010) 1519–1526. [5] Y. Chai, H. Li, X. Zhang, Multifocus image fusion based on features contrast of multiscale products in nonsubsampled contourlet transform domain, Optik 123 (2012) 569–581. [6] X. Lu, H. Ma, B. Zhang, A non-rigid medical image registration method based on improved linear elastic model, Optik 123 (2012) 1867–1873. [7] Y. Chai, H. Li, J. Qu, Image fusion scheme using a novel dual-channel PCNN in lifting stationary wavelet domain, Opt. Commun. 283 (2010) 3591–3602. [8] R.H. Bamberger, M.J. Smith, A filter bank for the directional decomposition of images: theory and design, IEEE Trans. Signal Process. 40 (1992) 882–893. [9] Y.M. Lu, M.N. Do, Multidimensional directional filter banks and surfacelets, IEEE Trans. Image Process. 16 (2007) 918–931. [10] M.N. Do, M. Vetterli, The contourlet transform: an efficient directional multiresolution image representation, IEEE Trans. Image Process. 14 (2005) 2091–2106. [11] A.L. Da Cunha, J. Zhou, M.N. Do, The nonsubsampled contourlet transform: theory, design, and applications, IEEE Trans. Image Process. 15 (2006) 3089–3101. [12] R. Eckhorn, H. Reitboeck, M. Arndt, P. Dicke, Feature linking via synchronization among distributed assemblies: simulations of results from cat visual cortex, Neural Comput. 2 (1990) 293–307. [13] J.L. Johnson, M.L. Padgett, PCNN models and applications, IEEE Trans. Neural Netw. 10 (1999) 480–498. [14] Z. Wang, Y. Ma, J. Gu, Multi-focus image fusion using PCNN, Pattern Recogn. 43 (2010) 2003–2016. [15] D. Agrawal, J. Singhai, Multifocus image fusion using modified pulse coupled neural network for improved image quality, IET Image Process. 4 (2010) 443–451. [16] X.-B. Qu, J.-W. Yan, H.-Z. Xiao, Z.-Q. Zhu, Image fusion algorithm based on spatial frequency-motivated pulse coupled neural networks in nonsubsampled contourlet transform domain, Acta Automat. Sin. 34 (2008) 1508–1514. [17] G. Qu, D. Zhang, P. Yan, Information measure for performance of image fusion, Electron. Lett. 38 (2002) 313–315. [18] C. Xydeas, V. Petrovic, Objective image fusion performance measure, Electron. Lett. 36 (2000) 308–309. [19] Z. Wang, A.C. Bovik, H.R. Sheikh, E.P. Simoncelli, Image quality assessment: from error visibility to structural similarity, IEEE Trans. Image Process. 13 (2004) 600–612.