Fusion of multifocus images by lattice structures

J. Vis. Commun. Image R. 38 (2016) 848–857

Contents lists available at ScienceDirect

J. Vis. Commun. Image R. journal homepage: www.elsevier.com/locate/jvci

Fusion of multifocus images by lattice structures q N.H. Kaplan a, I. Erer b,⇑, O. Ersoy c a

Department of Electrical and Electronical Engineering, Erzurum Technical University, Yakutiye 25700, Erzurum, Turkey Department of Electronics and Communications Engineering, Istanbul Technical University, Maslak 34469, Istanbul, Turkey c Purdue University, School of Electrical and Computer Engineering, West Lafayette, IN 47907, USA b

a r t i c l e

i n f o

Article history: Received 10 February 2016 Revised 11 April 2016 Accepted 20 April 2016 Available online 22 April 2016 Keywords: Image fusion Lifting wavelet transform Multiscale products Lattice filters QMF filterbanks

a b s t r a c t Image fusion methods based on multiscale transform (MST) suffer from high computational load due to the use of fast Fourier transforms (ffts) in the lowpass and highpass filtering steps. Lifting wavelet scheme which is based on second generation wavelets has been proposed as a solution to this issue. Lifting Wavelet Transform (LWT) is composed of split, prediction and update operations all implemented in the spatial domain using multiplications and additions, thus computation time is highly reduced. Since image fusion performance benefits from undecimated transform, it has later been extended to Stationary Lifting Wavelet Transform (SLWT). In this paper, we propose to use the lattice filter for the MST analysis step. Lattice filter is composed of analysis and synthesis parts where simultaneous lowpass and highpass operations are performed in spatial domain with the help of additions/multiplications and delay operations, in a recursive structure which increases robustness to noise. Since the original filter is designed for the undecimated case, we have developed undecimated lattice structures, and applied them to the fusion of multifocus images. Fusion results and evaluation metrics show that the proposed method has better performance especially with noisy images while having similar computational load with LSWT based fusion method. Ó 2016 Elsevier Inc. All rights reserved.

1. Introduction Digital cameras, focused at a specific distance, yields sharper objects in the focused area and blurry objects otherwise. Multifocus image fusion is the process of obtaining a new and improved image from different forms of the same image [1–3]. The goal of fusion is to include all the important information from input images in the fused image, while avoiding artifacts and noise [4–6]. The simplest fusion method takes the average of the pixel values of the source images. However, this approach yields reduced contrast in the fusion result. Multiscale transforms (MST) such as Laplacian pyramid, gradient pyramid, morphological pyramid, discrete wavelet transform (DWT), and stationary wavelet transform (SWT) have been widely used in image fusion tasks [7–12]. In these methods, the source images are decomposed into their subbands (named as lowpass and highpass subbands) by the MST. Then the subbands are merged by a predefined rule, and finally the fused image is obtained by the inverse MST. The most popular MST methods are based on DWT. In DWT based methods, the calcula-

q

This paper has been recommended for acceptance by M.T. Sun.

⇑ Corresponding author.

E-mail address: [email protected] (I. Erer). http://dx.doi.org/10.1016/j.jvcir.2016.04.017 1047-3203/Ó 2016 Elsevier Inc. All rights reserved.

tion cost is very high due to the convolution operation which requires the use of ffts. To make the calculations faster, some methods are developed, such as the lifting wavelet transform (LWT) [13]. Because of the decimation process during decomposition, both DWT and LWT are not shift-invariant, thus affecting the quality of the fused image [14]. To overcome this problem, the lifting stationary wavelet transform (LSWT) is used for image fusion processes [8,9]. Recently, multidirectional MST methods such as curvelets, or nonsubsampled contourlet method (NSCT) based fusion methods have been proposed [15–17]. Although, their performance is better than the method with LSWT due to multidirection information, their complexity prevents them to be appropriate for real time implementations. Another approach for fast processing of image sequences is by parallel implementation in many core processor systems [18,19]. In this paper, a new image fusion algorithm based on the subband decomposition of the source images using 1-D lattice filter structures is proposed. Lattice filters are used in digital filter implementations because they have a number of interesting and important properties including modularity, low sensitivity to parameter quantization effects and a simple stability test [20]. Two channel QMF (quadrature mirror filter) bank with lattice structure has perfect reconstruction property, and it guarantees good stopband attenuation for each of the analysis filters. A higher order QMF

N.H. Kaplan et al. / J. Vis. Commun. Image R. 38 (2016) 848–857

bank with lattice structure may be obtained from a lower order one simply by adding more lattice sections. The analysis section of the complete filter bank is represented by a lattice filter while the synthesis part is represented by the inverse lattice filter [20]. The lattice filters described above also includes decimation process, as in DWT and LWT, with similar problems. So, an undecimated lattice structure is proposed in this paper to avoid this problem. In MST based methods, another important problem is the rule selection for merging the subbands. The basic method is averaging lowpass subbands and using the absolute maximum of the highpass subbands. Taking into consideration the human vision system (HVS), local luminance contrast is measured by rationing the highpass subband to the local luminance [7]. Variations of this contrast measure are also proposed [16,17]. However, these measures use a single pixel value. A single pixel is usually not enough to determine whether it represents enough information about the subbands, and is also not good enough to eliminate the effects of noise. In wavelet subbands, there are dependencies between the scales, i.e., a large magnitude in finer scale yields a large magnitude in coarser scale. If the pixel is affected by noise, the magnitude decays. So, multiscale products (MSP), which is the multiplication of adjacent subbands, is used to eliminate effects of noise [21,22]. The paper is organized as follows: Section 2 reviews the background of the lifting wavelet transform and 1-D filter bank with attice structure. In Section 3, the new undecimated case for 1-D lattice structures is developed, and a new fusion method based on the proposed undecimated lattice structures is presented. Fusion results and evaluation metric results are given in Section 4. General conclusions are provided in Section 5.

849

where pr is the prediction coefficients of the lifting scheme and M is the number of the coefficients. 2.1.3. Update An approximate signal is obtained by using the detail signal d1 ðnÞ to update the even array xe ðnÞ using an update operator U½ as

s1 ðnÞ ¼ xe ðnÞ  U½d1 ðnÞ

ð4Þ

s1 ðnÞ is again divided into even and odd arrays, and the recursion steps are repeated until we obtain sj ; dj ; dj1 ; . . . ; d2 ; d1 The inverse lifting scheme is obtained by reversing the order of the decomposition steps and changing + by  and vice versa. The split step is replaced by a merge step where the odd and even signals are fused together at each resolution level. 2.1.4. Undecimated case The undecimated lifting scheme [23] consists only of two stages predict and update where the filter weight vectors are extended by inserting 2j1 zeros between their samples at each level j as

P j ¼ fP00 ; 0; . . . ; 0; P01 ; 0; . . . ; 0; P02 ; 0; . . . ; 0; P0M1 g U j ¼ fU 00 ; 0; . . . ; 0; U 01 ; 0; . . . ; 0; U 02 ; 0; . . . ; 0; U 0N1 g

ð5Þ

where M and N are the numbers of weight coefficients for filters P and U, respectively. In the reconstruction step, similar to the decimated case, the order of the steps are reversed, the signs are changed, and the merging step is replaced by an averaging step. 2.2. Background on 1-D PR filter bank with lattice structure

2. Background information In Section 2.1, the LWT method for signal decomposition and reconstruction is reviewed. In Section 2.2, signal decomposition and reconstruction with the lattice filterbank with the perfect reconstruction (PR) is reviewed. 2.1. Background on lifting wavelet transform The lifting wavelet transform (LWT), also known as the second generation wavelet transform, consists of split, predict and update steps [13]. Since it requires only shifts, additions and scalar multiplications instead of fft operations required for the convolution steps of the classical wavelet transform, it leads to an easier hardware implementation, as well as less storage space and computation time [13]. The lifting steps for decomposition in the decimated case are as follows: 2.1.1. Split The original signal sðnÞ is divided into an even and an odd array as

xe ðnÞ ¼ sð2nÞ;

x0 ðnÞ ¼ sð2n þ 1Þ

ð1Þ

2.1.2. Predict The odd array is predicted by the even array using a prediction operator P½. The detail information is obtained between x0 ðnÞ and its predicted signal P½xe ðnÞ as

d1 ðnÞ ¼ x0 ðnÞ  P½xe ðnÞ

ð2Þ

The predicted signal P½xe ðnÞ is obtained as follows:

P½xe ðnÞ ¼

M=2þ1 X M=2þ1

pr xe ðn þ rÞ

ð3Þ

A two channel filterbank is composed of analysis and synthesis sections. In the analysis section, the input signal is split into low and high pass components, and in the synthesis section, it is reconstructed from its components. Vaidyanathan [20] has proposed the use of quadrature mirror filters (QMF) with lattice structure for the design of two channel filterbanks with perfect reconstruction. The hierarchical property of lattice structures enables the design of higher order perfect reconstruction quadrature mirror filter banks (PR-QMFB) from lower order PR-QMFB simply by adding more lattice sections. QMFB with lattice structure involves a cascade of lattice structures, and each lattice structure is associated with a lattice coefficient. They have two special characteristics:  In each stage of the lattice, one coefficient is positive and the other one is negative, but both have the same magnitude.  All coefficients with even valued indices are zero. Analysis and synthesis lattice structures used in the single level decomposition and reconstruction for 1D signals are shown in Fig. 1a and b, respectively. As seen in Fig. 1a, the analysis lattice structure divides the input signal xðnÞ into its low-pass and high-pass components: xL ðnÞ and xH ðnÞ. The synthesis structure reconstructs the signal x0 ðnÞ from these components. In Table 1, the lattice parameters determined for given stopband frequencies and corresponding average stoppand powers are shown. The average stoppand power decreases with the increase of lattice order. Using (8), it is possible to find lattice parameters corresponding to different stopband frequencies and to construct different lattice structures. The input output relations for the first stage and succeeding stages are given by

850

N.H. Kaplan et al. / J. Vis. Commun. Image R. 38 (2016) 848–857

Fig. 1. Decimated 1-D PR lattice structure (a) analysis structure (subband decomposition) (b) synthesis structure (reconstruction).

Table 1 Lattice filter coefficients.

"

Stopband frequency

Lattice parameters

0:25p

a1 a3 a5

#

1

hL ðnÞ 1

hH ðnÞ "

 ¼

2mþ1

ðnÞ

2mþ1

ðnÞ

hL

hH

1 0.060944 0.000066

"

a1

a1

1

hH ðn  1Þ

1

a2mþ1

a2mþ11

1

 ¼

0

0.1415 0.0247 0.0005

#

0

1

#

Stopband average power

hL ðnÞ

ð6Þ

0

"

2m1

hL

ðnÞ

2m1

ðn  2Þ

hH

0

jþ1

#

jþ1

ð7Þ 0

0

with hL ðnÞ ¼ hH ðnÞ ¼ dðnÞ, and hL ðnÞ; hH ðnÞ and hL ðnÞ; hL ðnÞ for j ¼ 0; 1; 2; . . . are the filter impulse responses of the lowpass and high pass filters of order 2j þ 1 and 0, respectively; aj is the lattice coefficient at level j. With the help of these equations, it can be shown that the PR conditions are satisfied by 2mþ1

ðnÞ ¼ hL

2mþ1

ðnÞhL

hH hL

2mþ1

2mþ1

ðn þ 2m  1Þ 2mþ1

ðnÞ þ hH

2mþ1

ðnÞhH

ð8Þ

ðnÞ ¼ constant

The 1D PR lattice filter structure defined above has to be optimized to determine the lattice filter parameters. To achieve this in 1D frequency domain, the stopband energy is minimized at specific frequencies [20]. The lattice coefficients aj for j ¼ 0; 1; 2; . . . are computed by the minimization of the stopband energy as

a^ ¼ argmina jjHL2mþ1 ðejw Þjj22 ; subject to ws < w < p

ð9Þ

where ws is the desired stopband frequency. The objective function is the stopband energy of the frequency response of the lowpass filter H2mþ1 ðejw Þ. Since HL ðejw Þ and HH ðejw Þ are the frequency responses L of the mirror filters, the minimization of the objective function 2mþ1

results in the corresponding lowpass and highpass filters. hL 2mþ1 hH ðnÞ

3. Image fusion using undecimated QMF bank with lattice structure Since the decimated algorithm lacks the property of shift-invariance, its performance is affected by the shift of the input image, which is a severe drawback in applications such as fusion, denoising and segmentation/classification. In the undecimated wavelet transform, the down and upsampling processes are suppressed, and at each decomposition level, zeroes are inserted between the filter coefficients [5]. Following a similar strategy, we suppress the down and upsampling processes, and replace the delay term, which remains constant for all the decomposition/reconstruction levels in the decimated case, by q ¼ 2L where L is the decomposition level. Analysis and synthesis lattice structures used in the decomposition and reconstruction for 1D signals for the undecimated case are shown in Fig. 2a and b, respectively. The pseudo-codes for decomposition and reconstruction with undecimated lattice structures are given in Table 3. Lattice filters have similar computational load as the lifting wavelet. For (4,2) lifting scheme, single level 1D decomposition via lifting stationary wavelet transform has 23 N additions. Similarly, single level decomposition with 3 stage undecimated lattice structure has 6N additions. Image fusion with lattice structure involves image decomposition into subbands, merging of subband information by a predefined rule, and the reconstruction from the new subbands to obtain the fused image. For image decomposition, the pseudo-code for decomposition given in Table 2 is applied first to the rows, and then to the columns of the image, or vice versa. So, 4 different subbands are obtained by this decomposition process, named as LL, LH, HL and HH, respectively. In order to carry out the decomposition, this structure is again applied to the LL subband. In the reconstruction process, the pseudo-code for reconstruction is applied to the subbands. The image decomposition and reconstruction steps are shown in Fig. 3. Image fusion process can be summarized as follows:

ðnÞ

of the last stage are the final lowpass and highpass and filters with good stopband attenuation. The pseudo-code for the decomposition process is given in Table 2.

 Source images are registered with each other.  Source images are decomposed into their subbands by lattice structures.  The subbands are merged by a predefined rule.

851

N.H. Kaplan et al. / J. Vis. Commun. Image R. 38 (2016) 848–857 Table 2 Pseudo-code for decimated lattice decomposition/reconstruction. Pseudo-code for decimated lattice decomposition Let x be a vector of length N, be the lattice coefficients for (2m + 1) stages Initialization ð0Þ

ð0Þ

Table 3 Pseudo-code for undecimated lattice decomposition/reconstruction. Pseudo-code for undecimated lattice decomposition Let x be a vector of length N, a be the lattice coefficients for (2m + 1) stages Initialization ð0Þ

ð0Þ

xL;0 ðnÞ ¼ xð2n  1Þ; xH;0 ðnÞ ¼ xð2nÞ where n ¼ 1; 2; . . . ; N=2

xL;0 ðnÞ ¼ xH;0 ðnÞ ¼ xðnÞ where n ¼ 1; 2; . . . ; N

1. Stage

1. Stage

ð1Þ

ð0Þ

ð0Þ

ð1Þ

xL;0 ðnÞ ¼ xL;0 ðnÞ þ a1 xH;0 ðn  1Þ ð1Þ

ð0Þ

ðnÞ ¼ xL;0

ð2m1Þ

ð2mþ1Þ

ðnÞ ¼ a2mþ1 xL;0

ð2m1Þ

ðnÞ þ a2mþ1 xH;0 ð2m1Þ

ðn  2Þ

ð2m1Þ

ðnÞ þ xH;0

ðn  2Þ

Initialization for next level ð0Þ

ð2mþ1Þ

xL;1 ðnÞ ¼ xL;0

ð0Þ

ð0Þ

For odd stages

ð2mþ1Þ

xH;0

ð0Þ

ð1Þ

xH;0 ðnÞ ¼ a1 xL;0 ðnÞ þ xH;0 ðn  q=2Þ

For odd stages xL;0

ð0Þ

xL;0 ðnÞ ¼ xL;0 ðnÞ þ a1 xH;0 ðn  q=2Þ

ð0Þ

xH;0 ðnÞ ¼ a1 xL;0 ðnÞ þ xH;0 ðn  1Þ

ð2mþ1Þ

ðnÞ ¼ xL;0

ð2mþ1Þ

ðnÞ ¼ a2mþ1 xL;0

xL;0

xH;0

ð2m1Þ

ð2m1Þ

ðnÞ þ a2mþ1 xH;0 ð2m1Þ

ðn  qÞ

ð2m1Þ

ðnÞ þ xH;0

ðn  qÞ

Initialization for next level

ð0Þ

ð2mþ1Þ

ð2n  1Þ; xH;1 ðnÞ ¼ xL;0

ð2nÞ where n ¼ 1; 2; . . . ; N=4

ð0Þ

ð0Þ

ð2mþ1Þ

xL;1 ðnÞ ¼ xH;1 ðnÞ ¼ xL;0

ðnÞ, where n ¼ 1; 2; . . . ; N

Repeat all stages for M levels.

Repeat all stages for M levels.

Pseudo-code for decimated lattice reconstruction For odd stages

Pseudo-code for undecimated lattice reconstruction For odd stages

ð2m1Þ

xL;M

ð2mþ1Þ

ðnÞ ¼ xL;M

ð2m1Þ xH;M ðnÞ

¼ a

ð2mþ1Þ

ðn  2Þ þ a2mþ1 xH;M

ð2mþ1Þ ðn 2mþ1 xL;M

 2Þ þ

ðnÞ

ð2mþ1Þ xH;M ðnÞ

1. Stage ð0Þ

ð2m1Þ

xL;M

ð2mþ1Þ

ðnÞ ¼ xL;M

ð2m1Þ xH;M ðnÞ

ð2mþ1Þ

ðn  qÞ þ a2mþ1 xH;M

ð2mþ1Þ ðn 2mþ1 xL;M

¼ a

ðnÞ

1. Stage ð1Þ

ð1Þ

ð0Þ

ð1Þ

ð1Þ

xL;M ðnÞ ¼ xL;M ðn  1Þ þ a1 xH;M ðnÞ

xL;M ðnÞ ¼ xL;M ðn  q=2Þ þ a1 xH;M ðnÞ

ð0Þ xH;M ðnÞ

ð0Þ xH;M ðnÞ

ð1Þ 1 xL;M ðn

¼ a

 1Þ þ

ð1Þ xH;M ðnÞ

Merging ð2mþ1Þ

ðnÞ

ð2mþ1Þ

 qÞ þ xH;M

ð1Þ 1 xL;M ðn

¼ a

ð1Þ

 q=2Þ þ xH;M ðnÞ

Merging ð0Þ

ð2mþ1Þ

ð0Þ

xL;M1 ð2n  1Þ ¼ xL;M ðnÞ; xL;M1 ð2nÞ ¼ xH;M ðnÞ where n ¼ 1; 2; . . . ; N=ð2MÞ

 The new subbands are reconstructed by lattice structure to obtain the fused image. The process described above is visualized in Fig. 4.

ð2mþ1Þ

ð0Þ

ð0Þ

xL;M1 ðnÞ ¼ xL;M ðnÞ þ xH;M ðnÞ, where n ¼ 1; 2; . . . ; N

highpass subbands, then chooses the coefficients that have the higher feature contrast (MSR). The rule for the lowpass subband is to choose the coefficients that have the higher improved energy of Laplacian (IEOL) [8].

4. Experimental results

4.1. Quality metrics

The proposed method is compared to commonly used image fusion methods. These methods are the LWT, LSWT-simple and LSWT-MSP-con-max. For LWT methods, the (4,2) lifting scheme is used. The number of decomposition levels for all methods are chosen as 3. Two fusion rules are used for the fusion process. First one is the simple rule, which averages the lowpass subbands, while selecting the maximum of the highpass subbands. The second rule (con-max), first calculates the multiscale products (MSP) of the

The comparison of the different fusion methods is performed quantitatively using the following indicators: Information measure (MI) gives the amount of information that the input images and the fused image share [24]. Objective image fusion performance measure (QAB/F) is a quality metric, which gives the visual information obtained from the input images [25]. When the values of MI and (QAB/F) are larger, the fusion performance is better.

Fig. 2. Undecimated 1-D PR structure (a) analysis structure (subband decomposition) (b) synthesis structure (reconstruction).

852

N.H. Kaplan et al. / J. Vis. Commun. Image R. 38 (2016) 848–857

Fig. 3. Image (a) decomposition and (b) reconstruction with lattice structure.

Fig. 4. Fusion process with lattice structure.

4.2. Fusion results for clean multifocus images The fusion methods are applied to three different clean images. The first clean image is the lab image. The original images are given in Fig. 5a and b, whereas the fusion results for LWT, Decimated Lattice, LSWT-simple, Lattice-simple, LSWT-MSP-con-max and Lattice-MSP-con-max are given in Fig. 5c–h. The artifacts are present in the results for LWT and decimated lattice. The second image used for fusion is the book image, which is a color1 image. The original images are given in Fig. 6a and b, and the 1 For interpretation of color in Figs. 6 and 10, the reader is referred to the web version of this article.

fusion results for LWT, Decimated Lattice, LSWT-simple, Latticesimple, LSWT-MSP-con-max and Lattice-MSP-con-max are given in Fig. 6c–h. LSWT-MSP-con-max and Lattice-MSP-con-max have better performances. The final image is the pepsi image. The original images are given in Fig. 7a and b, and the fusion results for LWT, Decimated Lattice, LSWT-simple, Lattice-simple, LSWT-MSP-con-max and LatticeMSP-con-max are given in Fig. 7c–h. Here, again the LSWT-MSPcon-max and Lattice-MSP-con-max results are better. To understand the fusion results better, the difference images between the original image shown in Fig. 8a and the fusion results for LWT, Decimated Lattice, LSWT-simple, Lattice-simple, LSWTMSP-con-max and Lattice-MSP-con-max are given in Fig. 8a–f.

N.H. Kaplan et al. / J. Vis. Commun. Image R. 38 (2016) 848–857

853

Fig. 5. Lab image (a) focus on right, (b) focus on left, and fusion results for (c) LWT, (d) Decimated Lattice, (e) LSWT-simple, (f) Lattice-simple, (g) LSWT-MSP-con-max, (h) Lattice-MSP-con-max.

Fig. 6. Book image (a) focus on right, (b) focus on left, and fusion results for (c) LWT, (d) Decimated Lattice, (e) LSWT-simple, (f) Lattice-simple, (g) LSWT-MSP-con-max, (h) Lattice-MSP-con-max.

The right side of the difference images should be all black for a perfect fusion result. It can be clearly seen that the best results are achieved with LSWT-MSP-con-max and the proposed method, while there is a little more white area with LSWT-MSP-con-max than Lattice-MSP-con-max, meaning the proposed method keeps more information than the former one. In order to compare more objectively, quantitative comparison for the fusion results are given in Table 4. The bold values show the best score for each metric. Both visual and quantitative comparisons show that the proposed method outperforms other used methods with all three images.

4.3. Fusion of noisy multifocus images In order to compare the performances of the methods with noisy images, Gaussian noise with sigma ¼ 0:01 is added to the clean images given above. The noisy lab images are given in Fig. 9a and b, and the fusion results are given in Fig. 9c–h. The decimated lattice and LWT have too much noise than the other methods. The noisy book images are given in Fig. 10a and b, while the fusion results are given in Fig. 10c–h. Careful inspection shows that the yellow writings below the white header of the book can be read better for Lattice, LSWT-MSP-con-max, and Lattice-MSP-con max.

854

N.H. Kaplan et al. / J. Vis. Commun. Image R. 38 (2016) 848–857

Fig. 7. Pepsi image (a) focus on right, (b) focus on left and fusion results for (c) LWT, (d) Decimated Lattice, (e) LSWT-simple, (f) Lattice-simple, (g) LSWT-MSP-con-max, (h) Lattice-MSP-con-max.

Fig. 8. Difference image for (a) LWT, (b) Decimated Lattice, (c) LSWT, (d) lattice, (e) LSWT-MSP-con-max, (f) Lattice-MSP-con-max.

N.H. Kaplan et al. / J. Vis. Commun. Image R. 38 (2016) 848–857 Table 4 Metrics for clean images. Figure

Lab

Book

Pepsi

Metric

QAB/F

MI

QAB/F

MI

QAB/F

MI

LWT Decimated Lattice LSWT-simple Lattice-simple LSWT-MSP-conmax Lattice-MSP-conmax

0.4841 0.6850 0.7210 0.7434 0.7352

0.7009 0.8465 0.8987 0.9104 0.9641

0.5496 0.7828 0.7985 0.8143 0.7901

0.7480 0.8779 0.8848 0.8924 0.8567

0.4390 0.6439 0.7162 0.7296 0.7021

0.5450 0.5634 0.5897 0.5781 0.6409

0.7512

0.9663

0.8218

0.8969

0.7332

0.6519

855

The noisy pepsi images are given in Fig. 11a and b, and the fusion results are given in Fig. 11c–h. Even though, the decimated lattice results seem to be sharper, there is a loss in contrast. Here the LWT and decimated lattice results are too noisy. In order to understand the noisy results better, a zoomed version of the pepsi image for the fusion results are also given in Fig. 12a–f. In the zoomed version of the results, the noise effect is reduced better in the proposed method. The letters are more readable then the other methods. Quantitative comparisons for noisy images are given in Table 5. The bold values show the best score for each metric. Both visual

Fig. 9. Lab image (a) Focus on right, (b) focus on left, and fusion results for (c) LWT, (d) Decimated Lattice, (e) LSWT-simple, (f) Lattice-simple, (g) LSWT-MSP-con-max, (h) Lattice-MSP-con-max.

Fig. 10. Book image (a) Focus on right, (b) focus on left, and fusion results for (c) LWT, (d) Decimated Lattice, (e) LSWT-simple, (f) Lattice-simple, (g) LSWT-MSP-con-max, (h) Lattice-MSP-con-max.

856

N.H. Kaplan et al. / J. Vis. Commun. Image R. 38 (2016) 848–857

Fig. 11. Pepsi image (a) focus on right, (b) focus on left, and fusion results for (c) LWT, (d) Decimated Lattice, (e) LSWT-simple, (f) Lattice-simple, (g) LSWT-MSP-con-max, (h) Lattice-MSP-con-max.

Fig. 12. Zoomed fusion results for Pepsi image (a) LWT, (b) Decimated Lattice, (c) LSWT-simple, (d) Lattice-simple, (e) LSWT-MSP-con-max, (f) Lattice-MSP-con-max.

N.H. Kaplan et al. / J. Vis. Commun. Image R. 38 (2016) 848–857 Table 5 Metrics for noisy images. Figure

Lab

Book

Pepsi

Metric

QAB/F

MI

QAB/F

MI

QAB/F

MI

LWT Decimated Lattice LSWT-simple Lattice-simple LSWT-MSP-conmax Lattice-MSP-conmax

0.2837 0.3802 0.3803 0.4344 0.4138

0.5409 0.5801 0.5794 0.5875 0.5985

0.2820 0.3767 0.3787 0.4424 0.4122

0.3498 0.4267 0.4346 0.4616 0.4418

0.2267 0.2968 0.2987 0.3558 0.3326

0.3179 0.3740 0.3838 0.3967 0.3958

0.4576

0.6034

0.4698

0.4653

0.4037

0.4149

Table 6 Computation times for different methods. Figure

Lab (s)

Book (s)

Pepsi (s)

LWT Decimated Lattice LSWT-simple Lattice-simple LSWT-MSP-con-max Lattice-MSP-con-max

0.099 0.045 0.203 0.156 10.299 5.634

1.047 0.625 3.717 3.531 214.834 193.394

0.245 0.140 0.757 0.559 33.956 22.091

and quantitative results show that the proposed method is better than the other used methods. Moreover, the computation times for LSWT based methods and the proposed ones are given in Table 6. The bold values show the best score for each metric. As it can be seen in Table 6, the proposed method is faster than the former methods. 5. Conclusions In this article, a new multifocus image fusion algorithm based on undecimated lattice structures is proposed. The source images are decomposed using undecimated lattice analysis filters, the subbands are fused using a predefined fusion rule, and then the fusion result is obtained from the fused subbands by using undecimated synthesis lattice filters. The decomposition and reconstruction steps are all performed in the spatial domain, and require only simple additions, multiplications and delay operations. The fusion method is tested with a simple fusion rule as well as another MSP based complicated rule which is more adopted to the human HVS system. The fusion results obtained for clean and noisy images, and the evaluation metrics obtained validate that the proposed method has a comparable complexity to LSWT based fusion methods, and a better fusion performance due to the use of lattice structures. The proposed method is also suitable for easy hardware implementation because of its simple arithmetical operations with increased robustness to noise. References [1] W.B. Seales, S. Dutta, Everywhere-in-focus image fusion using controllable cameras, in: Proceedings of SPIE, 1996, pp. 227–234. [2] S. Li, B. Yang, Multifocus image fusion using region segmentation and spatial frequency, Image Vis. Comput. 26 (7) (2008) 971–979, http://dx.doi.org/ 10.1016/j.imavis.2007.10.012.

857

[3] Z. Wang, Y. Ma, J. Gu, Multi-focus image fusion using {PCNN}, Pattern Recogn. 43 (6) (2010) 2003–2016, http://dx.doi.org/10.1016/j.patcog.2010.01.011. [4] S. Li, J.T. Kwok, Y. Wang, Multifocus image fusion using artificial neural networks, Pattern Recogn. Lett. 23 (8) (2002) 985–997, http://dx.doi.org/ 10.1016/S0167-8655(02)00029-6. [5] G. Pajares, J.M. de la Cruz, A wavelet-based image fusion tutorial, Pattern Recogn. 37 (9) (2004) 1855–1872, http://dx.doi.org/10.1016/j.patcog.2004.03. 010. [6] A. Loza, D. Bull, N. Canagarajah, A. Achim, Non-gaussian model-based fusion of noisy images in the wavelet domain, Comput. Vis. Image Underst. 114 (1) (2010) 54–65, http://dx.doi.org/10.1016/j.cviu.2009.09.002. [7] A. Toet, L.J. van Ruyven, J.M. Valeton, Merging thermal and visual images by a contrast pyramid, Opt. Eng. 28 (7) (1989) 789–792, http://dx.doi.org/10.1117/ 12.7977034. [8] H. Li, S. Wei, Y. Chai, Multifocus image fusion scheme based on feature contrast in the lifting stationary wavelet domain, EURASIP J. Adv. Signal Process. 2012 (1) (2012) 1–16, http://dx.doi.org/10.1186/1687-6180-2012-39. [9] Y. Chai, H. Li, M. Guo, Multifocus image fusion scheme based on features of multiscale products and {PCNN} in lifting stationary wavelet domain, Opt. Commun. 284 (5) (2011) 1146–1158, http://dx.doi.org/10.1016/j.optcom. 2010.10.056. [10] V.S. Petrovic, C.S. Xydeas, Gradient-based multiresolution image fusion, IEEE Trans. Image Process. 13 (2) (2004) 228–237, http://dx.doi.org/10.1109/ TIP.2004.823821. [11] H. Li, B.S. Manjunath, S.K. Mitra, Multi-sensor image fusion using the wavelet transform, in: IEEE International Conference on Image Processing – ICIP-94, IEEE, 1994, pp. 51–55, http://dx.doi.org/10.1109/ICIP.1994.413273. [12] X.-B. Qu, J.-W. Yan, H.-Z. Xiao, Z.-Q. Zhu, Image fusion algorithm based on spatial frequency-motivated pulse coupled neural networks in nonsubsampled contourlet transform domain, Acta Automat. Sinica 34 (12) (2008) 1508–1514, http://dx.doi.org/10.1016/S1874-1029(08)60174-3. [13] W. Sweldens, The lifting scheme: a construction of second generation wavelets, SIAM J. Math. Anal. 29 (2) (1998) 511–546, http://dx.doi.org/ 10.1137/S0036141095289051. [14] R.R. Coifman, D.L. Donoho, Ch. Translation-invariant de-noising, in: A. Antoniadis, G. Oppenheim (Eds.), Wavelets and Statistics, Springer, New York, New York, 1995, pp. 125–150, http://dx.doi.org/10.1007/978-1-46122544-7_9. [15] M. Wang, J.-L. Zhang, D.-Y. Cao, Fusion of multispectral and panchromatic satellite images based on ihs and curvelet transformations, Wavelet Analysis and Pattern Recognition, 2007. ICWAPR ’07. International Conference on, vol. 1, 2007, pp. 321–325, http://dx.doi.org/10.1109/ICWAPR.2007.4420687. [16] Q. Zhang, B. Guo, Fusion of multi-sensor images based on the nonsubsampled contourlet transform, Acta Automat. Sinica 34 (2008) 135–141, http://dx.doi. org/10.3724/SP.J.1004.2008.00135. [17] L. Yang, B. Guo, W. Ni, Multimodality medical image fusion based on multiscale geometric analysis of contourlet transform, Neurocomputing 72 (13) (2008) 203–211, http://dx.doi.org/10.1016/j.neucom.2008.02.025. [18] C. Yan, Y. Zhang, J. Xu, F. Dai, L. Li, Q. Dai, F. Wu, A highly parallel framework for HEVC coding unit partitioning tree decision on many-core processors, IEEE Signal Process. Lett. 21 (5) (2014) 573–576, http://dx.doi.org/10.1109/ LSP.2014.2310494. [19] C. Yan, Y. Zhang, J. Xu, F. Dai, J. Zhang, Q. Dai, F. Wu, Efficient parallel framework for HEVC motion estimation on many-core processors, IEEE Trans. Circ. Syst. Video Technol. 24 (12) (2014) 2077–2089, http://dx.doi.org/ 10.1109/TCSVT.2014.2335852. [20] P.P. Vaidyanathan, P.Q. Hoang, Lattice structures for optimal design and robust implementation of two-channel perfect-reconstruction QMF banks, IEEE Trans. Acoust. Speech Signal Process. 36 (1) (1988) 81–94, http://dx.doi.org/ 10.1109/29.1491. [21] P. Bao, L. Zhang, Noise reduction for magnetic resonance images via adaptive multiscale products thresholding, IEEE Trans. Med. Imag. 22 (9) (2003) 1089– 1099, http://dx.doi.org/10.1109/TMI.2003.816958. [22] Y. Xu, J.B. Weaver, D.M. Healy, J. Lu, Wavelet transform domain filters: a spatially selective noise filtration technique, IEEE Trans. Image Process. 3 (6) (1994) 747–758, http://dx.doi.org/10.1109/83.336245. [23] C.S. Lee, C.K. Lee, K.Y. Yoo, New lifting based structure for undecimated wavelet transform, Electron. Lett. 36 (22) (2000) 1894–1895, http://dx.doi.org/ 10.1049/el:20001294. [24] G. Qu, D. Zhang, P. Yan, Information measure for performance of image fusion, Electron. Lett. 38 (7) (2002) 313–315, http://dx.doi.org/10.1049/el:20020212. [25] C.S. Xydeas, V. Petrovic, Objective image fusion performance measure, Electron. Lett. 36 (4) (2000) 308–309, http://dx.doi.org/10.1049/el:20000267.