- Email: [email protected]
Discrete Cosine Transform based fusion of multi-focus images for visual sensor networks
Signal Processing 95 (2014) 161–170
Contents lists available at ScienceDirect
Signal Processing journal homepage: www.elsevier.com/locate/sigpro
Fast communication
Discrete Cosine Transform based fusion of multi-focus images for visual sensor networks Y. Asnath Victy Phamila n, R. Amutha SSN College of Engineering, Anna University, Chennai, India
a r t i c l e i n f o
abstract
Article history: Received 29 April 2013 Received in revised form 29 August 2013 Accepted 2 September 2013 Available online 16 September 2013
This paper presents a simple and efficient multi-focus image fusion scheme explicitly designed for wireless visual sensor systems equipped with resource constrained, battery powered image sensors employed in surveillance, hazardous environment like battlefields etc. Here the fusion of multi-focus images is based on higher valued Alternating Current (AC) coefficients calculated in Discrete Cosine Transform (DCT) domain. The proposed method overcomes the computation and energy limitation of low power devices and is investigated in terms of image quality and computation energy. Simulations are performed using Atmel Atmega128 processor of Mica 2 mote to measure the resultant energy savings. The experimental results verify the significant efficiency improvement of the proposed method in output quality and energy consumption, when compared with other fusion techniques in DCT domain. & 2013 Elsevier B.V. All rights reserved.
Keywords: Image fusion Discrete Cosine Transform Visual sensor network Fusion metrics
1. Introduction Image fusion is the process of combining multiple source images from sensor network into a single one, which contains a more accurate description of the scene, more informative and suitable for both visual perception and further processing [1]. In multi-focus image fusion technique, several images of a scene captured with focus on different objects are fused such that all the objects will be in focus in the resulting image. So far, several researches have been focused on image fusion which is performed on the images in the spatial and spectral domain [2–8]. Various multi-focus image fusion algorithms in wavelet domain are available in literature [3–5]. In [3] statistical sharpness measure based on wavelet coefficients distribution is used to perform adaptive image fusion in wavelet domain. Instead of using Discrete Wavelet Transform (DWT) to decompose images into frequency domain, Discrete Stationary Wavelet transform is used in [4] to overcome the
n
Corresponding author. Tel.: þ91 9884322100; fax: þ91 44 27469772. E-mail addresses: [email protected] (Y.A.V. Phamila), [email protected] (R. Amutha). 0165-1684/$ - see front matter & 2013 Elsevier B.V. All rights reserved. http://dx.doi.org/10.1016/j.sigpro.2013.09.001
lack of translation invariance of the Discrete Wavelet Transform. Then the transformed coefficients are fused and the fused image is constructed by applying inverse Discrete Stationary Wavelet Transform. As the wavelets do not represent long edges well in the fused results, multi-focus image fusion is performed by combining both the wavelet and curvelet transforms to improve the quality [5]. But the limitation is that it consumes more time than the wavelet-based methods because two different multiscale decomposition processes are applied. However multi-focus image fusion based on DWT has its own disadvantages. DWT needs great number of convolution calculations, and it consumes much time or memory resources, which impedes its application for resource constrained battery powered visual sensor nodes. The energy needed for DCT based fusion is less compared to the DWT based methods. Hence DCT based fusion methods are more appropriate for resource constrained devices. Since the computational energy is much less than the transmission energy, data are compressed and fused before transmission in automated battlefields, where the robots collect image data from sensor network [9]. When the source images are to be coded in Joint Photographic Experts Group (JPEG) standard or when the resultant fused
162
Y.A.V. Phamila, R. Amutha / Signal Processing 95 (2014) 161–170
image is to be saved or transmitted in JPEG format, the fusion methods in DCT domain will be more efficient [7,8]. Contrast sensitivity method is adopted in [7] to construct the fused image. The contrasts of the corresponding AC coefficients of different blurred images are compared and the AC coefficient with the largest contrast value is selected as the AC coefficient of the fused image. But the contrast calculation for each AC coefficient involves complex floating point computations and the fused image suffers from blocking artifacts due to diversity in the selection of DCT coefficients. In [6,7] the DCT representation of the fused image is obtained by taking the average of all the DCT representations of all the input images but the fused result has undesirable blurring effects. To overcome the undesirable side effects like blurring or blocking artifacts which reduce the quality of the fused image, multi-focus image fusion based on variance calculated in DCT domain is presented in [8]. However the mean, contrast and variance calculation for fusion [7,8] involves complex floating point arithmetic operations which incur high energy consumption in resource constrained battery powered sensor nodes. In this paper, a simple and efficient multi-focus image fusion scheme that is suitable for resource-constrained (processing, bandwidth, memory space, battery power) sensor network is proposed. In the proposed method, the image blocks with more number of higher valued AC coefficients is absorbed into the fused image. It is extremely fast as it does not involve any complex floating point arithmetic operations like mean or variance calculation. The proposed fusion rule considerably reduces the computational complexity without compromising image quality. Since the proposed fusion scheme is performed in DCT domain, it is time-saving and simple when the fused image needs to be saved or transmitted in JPEG format [7,8]. Simulations are performed using Atmel Atmega128 processor of Mica 2 mote to measure the resultant energy savings.
2. Image fusion In the proposed scheme, the key step is to fuse the DCT representations of multi-focus images into a single fused image. The input images are divided into blocks of size 8 8 and the DCT coefficients of each block is computed. Then the fusion rule is applied wherein the transformed block with more number of higher valued AC coefficients is absorbed into the fused image.
where u, v¼0, 1, …, N 1 and ( pffiffiffi 1= 2; if u ¼ 0 cðuÞ ¼ 1; if u a 0
ð2Þ
the inverse transform is defined as f ðx; yÞ ¼
2 N1 N1 ð2x þ 1Þuπ ∑ ∑ cðuÞcðvÞFðu; vÞ cos Nv¼0u¼0 2N ð2y þ1Þvπ cos 2N
ð3Þ
where x, y¼0, 1, …, N 1. Here F (0, 0) is the DC coefficient and it represents the mean value of that image block. Remaining coefficients are AC coefficients. 2.2. Fusion based on variance In [8] variance is used as the activity level for fusion criteria because in multi-focus images, the focused region is more informative and the information details of that region correspond to high variance. It is inferred that the variance of an N N block of pixels can be exactly calculated from its DCT coefficients by computing the sum of the squared normalized AC coefficients of the DCT block. The normalized transform coefficients are defined as ^ vÞ ¼ Fðu; vÞ Fðu; N
ð4Þ
2
variance (s ) of the image block [8] can be inferred from the transformed coefficients as follows: N 1 N 1 F 2 ðu; vÞ
s2 ¼ ∑
∑
u¼0v¼0
N2
2
F^ ð0; 0Þ
ð5Þ
^ 0Þ is the normalized DC coefficient and other where Fð0; ^ vÞs are the normalized AC coefficients. Fðu; 2.3. Fusion criteria for the proposed AC_Max fusion The advantage of DCT is that the energy of the original data may be concentrated in only a few low frequency components of DCT depending on the correlation in the data. Also the low-frequency components usually contain the most of the image information. Higher the value of AC coefficients implies finer image information. Eq. (5) implies that the variance of a block of size 8 8 is given by the sum of the squares of the normalized 63 AC coefficients. 63
^ 2 s2 ¼ ∑ Ai
ð6Þ
i¼1
2.1. Discrete Cosine Transform Two dimensional DCT transform of an N N image block f (x, y) [8] is given as Fðu; vÞ ¼
N1 N1 2 ð2x þ1Þuπ cðuÞðvÞ ∑ ∑ f ðx; yÞ cos N 2N y¼0x¼0
cos
ð2y þ 1Þvπ 2N
ð1Þ
Because of the energy compaction property of AC coefficients, only few coefficients towards the top left submatrix of the DCT transformed matrix have larger values [10] and the contribution of these coefficients to variance is more compared to other AC coefficients. Hence if the AC coefficient value is high, then the variance value is also high. Hence in our proposed method we absorb the block with more number of higher valued AC coefficients for two reasons. First is that higher the AC component value implies more fine details of the image [11]. Secondly, from
Y.A.V. Phamila, R. Amutha / Signal Processing 95 (2014) 161–170
Eq. (6) it is inferred that higher the AC component value results in higher variance. Here instead of computing variance using all the transformed AC coefficients which involves floating point multiplication and additions, the proposed algorithm checks only the number of higher valued AC coefficients that contributes to larger variance. Thus the energy needed for computation is drastically reduced. The quality of the fused image is significantly high because only the blocks with high energy where more image details are stored, are selected for fusion.
163
This is repeated for all Q blocks to fuse the DCT representations of multi-images into a single DCT representation of image. The fused DCT blocks are subject to consistency verification (CV) [12] in which if the center block comes from the source image1 and majority of the surrounding blocks come from image 2, the center block is switched to that of image 2. Here 3 3 neighborhood window is used for consistency verification. After consistency verification, the fused DCT coefficients can be easily saved or transmitted in JPEG format. The original fused image can be reconstructed by applying inverse DCT to the fused DCT coefficients.
2.4. AC_Max fusion method 3. Performance metrics Fig. 1 illustrates the general framework of our proposed image fusion scheme. The algorithm can be extended for more than two source images with the assumption that all the source images are aligned with some registration methods. Let Y¼{yi,j} (i¼0, …, N 1 and j¼0, …, M 1) be an image and it is divided into Q number of 8 8 blocks. Let Xn¼{xn,k,l} (k¼0, …, 7; l¼0, …, 7; n¼ 0, …, Q1) be the nth 8 8 block and the corresponding DCT output of the block Xn¼ {xn,k,l} be Dn¼{Dn,k,l}. Then the set D¼{D0, D1, D2, …, DQ 1} denotes the DCT representation of image Y¼{yi,j}. Let Dt ¼ fDt 0 ; Dt 1 ; Dt 2 ; …; Dt Q 1 g be the DCT representation of the tth input image and let B be the number of input source images to be fused. Then the DCT representation DF ¼ fDF 0 ; DF 1 ; DF 2 ; …; DF Q 1 g of the fused image is obtained by fusing the transformed DCT coefficients of the input set of B images. The fusion criterion is that the block with the majority of maximum valued DCT AC coefficients is absorbed into the fused image since it contributes more significant signal information to the fused image. Hence in our fusion method, the nth block of the fused image ðDF n Þ is obtained by Eqs. (7) and (8). DF n ¼ DT n
Extensive experiments are performed to demonstrate the superior performance of the proposed algorithm using
ð7Þ
where T ¼ arg max fC tn g; t ¼ 1; …; B
ð8Þ
t
C tn in Eq. (8) specifies the number of maximum valued transformed AC coefficients found in the nth block of tth image when compared with the respective blocks in other source images. For example, the fused DCT block DF n of two source image blocks (X 1 n and X 2 n ) is obtained as illustrated in the flowchart (Fig. 2).
Fig. 2. Flowchart for AC_Max fusion rule.
Fig. 1. General structure of the proposed fusion scheme.
164
Y.A.V. Phamila, R. Amutha / Signal Processing 95 (2014) 161–170
six performance metrics. The structural similarity measure (SSIM) [8,13], mean square error (MSE) and peak signal to noise ratio (PSNR) are used as quality measures for objective evaluation of results of referenced images. If MSE equals zero, it implies that the fused image is exactly the same as that of the original referenced image. The higher the values of PSNR and SSIM, the better are the quality of the fused image. To evaluate our proposed algorithm on non-referenced multi-focus images, the spatial frequency (SF) metric [14,15], the state-of-the-art fusion performance metric Petrovic [8,16,17] metric (QAB/F) and mutual information (MI) [18] are used.
3.1. Mean square error (MSE) MSE is the cumulative squared error between the fused and the referenced image and it is defined by Eq. (9) MSE ¼
N 1 M ∑ ∑ ½Iðm; nÞ Fðm; nÞ2 MN m ¼ 1 n ¼ 1
ð9Þ
where I(m, n) is the pixel value of the referenced image and F(m, n) is the pixel value of the fused image. If MSE¼0, it is a perfect fusion where the fused image exactly matches the referenced image.
Fig. 3. Fusion results of multifocus images. (a) Traffic image and (b) Battlefield image.
Fig. 4. Standard test images.
Y.A.V. Phamila, R. Amutha / Signal Processing 95 (2014) 161–170
3.2. Peak signal to noise ratio (PSNR) The quality of the image is evaluated using PSNR. The PSNR is calculated using Eq. (10) PSNR ¼ 10log 10
ð2b 1Þ2 dB MSE
ð10Þ
where b is the number of bits per pixel (bpp) of the original image, and MSE is the mean-square-error which
165
is defined by Eq. (9). Higher PSNR value indicates better quality. If MSE ¼0, then the PSNR becomes infinity which implies ideal fusion, where the fused image and the referenced test image are exactly identical. 3.3. Structural similarity index (SSIM) The structural similarity measure [8,13], as a quality criterion, is used for objective evaluation of fused image.
Fig. 5. Fusion results of Lena image. (a) Referenced image. (b) Focussed on right. (c) Focussed on left. (d) DCT þAverage result. (e) DCT þVariance result. (f) DWT result. (g) SIDWT result. (h) Proposed method.
Fig. 6. Fusion results of Pepsi image. (a) Source Image 1. (b) Source Image 2. (c) DCT þAverage result. (d) DCT þ Variance result. (e) DWT result. (f) SIDWT result. (g) Proposed method.
166
Y.A.V. Phamila, R. Amutha / Signal Processing 95 (2014) 161–170
Fig. 7. Fusion results of Clock image. (a) Source Image 1. (b) Source Image 2. (c) DCT þ Average result. (d) DCT þ Variance result. (e) DWT result. (f) SIDWT result. (g) Result of the proposed method.
The general form of the metric that is used to measure the structural similarity between two signal vectors x and y is given by Eq. (11) ! ! 2μx μy þ C 1 2sxy þC 2 ð11Þ SSIMðx; yÞ ¼ μ2x þ μ2y þ C 1 s2x þs2y þ C 2 where mx and my are the sample means of x and y respectively, sx 2 and sy 2 are the sample variances of x and y respectively, and sxy is the sample cross-covariance between x and y. The default values for C1 and C2 are 0.01 and 0.03. The average of the SSIM values across the image (mean SSIM or MSSIM) gives the final quality measure. AB/F
3.4. Petrovic metric (Q
)
This measure was proposed by Xydeas and Petrovic [16,17]. This metric is based on the assumption that fusion algorithm that transfers input gradient information into resultant image more accurately performs better. In this case, a pixel wise measure of information preservation is obtained between each input image (A and B) and the fused image (F) of size M N to compute QAB/F using simple local perceptual importance factors. It is calculated by Q AB=F ¼
AF BF M A B ∑N n ¼ 1 ∑m ¼ 1 ðQ ðn; mÞw ðn; mÞ þ Q ðn; mÞw ðn; mÞÞ N M A B ∑n ¼ 1 ∑m ¼ 1 ðw ðn; mÞ þw ðn; mÞÞ
ð12Þ AF
3.5. Spatial frequency (SF) The row and column frequencies of the fused image (F) of size M N is computed [14,15] as given in Eqs. (13) and (14). sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 M1 N1 RF ¼ ∑ ∑ ½Fðm; nÞ Fðm; n 1Þ2 ð13Þ MN m ¼ 0 n ¼ 1 sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 N1 M 1 CF ¼ ∑ ∑ ½Fðm; nÞ Fðm 1; nÞ2 MN n ¼ 0 m ¼ 1
then the total spatial frequency of the fused image which is based on edge information is computed as follows: qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi SF ¼ ðRFÞ2 þ ðCFÞ2 ð15Þ higher the spatial frequency, higher is the clarity of the image. 3.6. Mutual information (MI) This metric gives the amount of information that the fused image F has from input source images (A and B) [18]. MI represents the similarity of the image intensity distributions of the corresponding image pair. Higher the MI better is the quality of the fused image. The mutual information between source image A and the fused image F is computed as given in Eq. (16)
BF
where Q and Q are computed using edge information preservation values [14,16]. wA(n,m) And wB(n,m) are the weighted importance factors for QAF and QBF respectively. QAB/F is in the range [0, 1] where 0 means complete loss of information and 1 means ideal fusion.
ð14Þ
I AF ¼ ∑pAF ða; f Þlog a;f
pAF ða; f Þ pA ðaÞpF ðf Þ
ð16Þ
where pAF, pA and pB are computed by normalization of the joint and marginal histograms of A and F. Similarly mutual
Y.A.V. Phamila, R. Amutha / Signal Processing 95 (2014) 161–170
167
Fig. 8. Fusion results of Lake image. (a) Referenced Image. (b) Focussed on Left. (c) Focussed on Middle. (d) Focussed on Right. (e) DCT þAverage result. (f) DCT þ Variance result. (g) DWT result. (h) SIDWT result. (i) Proposed method.
Fig. 9. Fusion results of Toy image. (a) Near Focussed Image. (b) Middle Focussed Image. (c) Far Focussed Image. (d) DCT þAverage result. (e) DCT þVariance result. (f) DWT result. (g) SIDWT result. (h) Result of the proposed method.
168
Y.A.V. Phamila, R. Amutha / Signal Processing 95 (2014) 161–170
Table 1 The MSE, PSNR and MSSIM comparison of various image fusion approaches on reference images. Fusion Method
Lena MSE
DCT þAverage 39.11 DCT þ Variance 8.70 DCT þ Variance þ CV 1.88 DWT with DBSS(2,2) 5.62 SIDWT with Haar 5.48 Proposed (DCT þ AC_Max) 0 Proposed(DCTþAC_Max þCV) 0
Traffic
Bridge
PSNR (dB) MSSIM MSE
PSNR (dB) MSSIM MSE
32.20 38.73 45.38 40.63 40.74 1 1
27.62 41.89 49.12 37.05 37.99 1 1
0.9383 112.6 0.9854 4.2 0.9982 0.797 0.9906 12.81 0.9899 10.33 1 0 1 0
Lake PSNR (dB) MSSIM MSE
0.8997 28.83 33.53 0.9920 6.69 39.87 0.9997 4.89 41.23 0.9901 4.49 41.61 0.9905 4.54 41.55 1 0.03 62.91 1 0.02 64.32
PSNR (dB) MSSIM
0.9465 145.43 0.9870 13.64 0.9957 0.297 0.9895 14.58 0.9893 33.67 0.9998 0 0.9998 0
26.50 36.78 53.40 36.49 32.86 1 1
0.8343 0.9849 0.9999 0.9836 0.9759 1 1
Table 2 The QAB/F, spatial frequency (SF) and mutual information(MI) of various image fusion approaches on non-referenced images. Fusion method
Clock AB/F
DCT þAverage DCT þVariance þ CV DWT with DBSS(2,2) SIDWT with Haar Proposed(DCT þ AC_Max þCV)
Pepsi AB/F
Q
SF
MI
Q
0.65 0.73 0.67 0.71 0.74
11.56 18.39 18.38 17.24 18.46
7.13 9.04 6.50 6.75 9.04
0.63 0.77 0.73 0.74 0.78
Lab
MI ¼ I AF þ I BF
Aircraft
MI
Q
SF
MI
Q
SF
MI
QAB/F
SF
MI
10.57 13.81 13.83 13.39 13.96
6.84 8.51 6.35 6.60 8.66
0.55 0.73 0.66 0.68 0.73
7.72 13.17 13.10 12.38 13.41
7.08 8.60 6.52 6.93 8.65
0.61 0.80 0.79 0.79 0.81
9.09 16.70 16.77 16.36 16.94
4.01 5.74 4.12 4.55 5.74
0.58 0.60 0.54 0.61 0.62
9.1 13.45 13.59 13.25 13.70
9.94 10.13 7.45 8.46 10.20
ð17Þ
4. Experimental results The proposed fusion algorithm is applied on a set of nonreferenced and set of referenced images and the results are evaluated. The first experiment is conducted using an extensive set of artificially generated images with different focus levels. Standard test images like Traffic, Lena, Battlefield, Barbara, Bird etc are taken as ground truth images [19,20]. Two blurred artificial images are generated for each test image by convolving the test image with a 9 9 averaging filter centered at the left part and right part respectively [3]. Fig. 3 depicts the fusion result of the proposed method on referenced multi-focused ‘traffic’ and ‘battlefield’ images. From Fig. 3, it is seen the fused image exactly represents the original referenced image with mean SSIM value as 1. The proposed fusion method is experimented on the 48 sets of (48 2¼96) blurred images artificially generated by convolving the test images with a 7 7 averaging filter centered at the left part and right part respectively. The 48 standard grayscale test images are shown in Fig. 4. In 42 sets of images, this proposed fusion scheme results in ideal fused results with mean SSIM (MSSIM) as 1. The second experiment is conducted on sets of standard non-referenced multifocus test images [21]. The fusion result of Lena, Pepsi and Clock standard test images by applying various fusion algorithms in DCT domain [8,22] (DCTþAverage and DCTþVariance), DWT domain (DWT [12] and SIDWT [23]) and our proposed method is shown in Figs. 5–7
AB/F
Toy
SF
information IBF is computed between the source image B and the fused image. Then the mutual information between the source images (A, B) and the fused image (F) is given as follows:
AB/F
Table 3 Fusion metrics for DCT þVariance and DCT þ AC_Max with different DCT submatrix of size Ns on Barbara Image. Fusion method (Ns)
MSE
PSNR (dB) SSIM
DCT þVariance (8 8) 20.28 35.06 DCT þAC_Max(2 2) 71.27 29.60 DCT þAC_Max (3 3) 6.25 40.17 DCT þAC_Max (4 4) 2.20 44.71 DCT þAC_Max (5 5) 0.36 52.55 DCT þAC_Max (6 6) 0 1 DCT þAC_Max (7 7) 0 1 DCT þAC_Max (8 8) 0 1
0.9643 0.9032 0.9854 0.9942 0.9988 1 1 1
MI
SF
QAB/F
7.37 7.26 7.43 7.46 7.48 7.49 7.49 7.49
28.37 26.77 28.39 28.40 28.40 28.40 28.40 28.40
0.80 0.69 0.80 0.81 0.81 0.81 0.81 0.81
respectively. As the proposed fusion method selects blocks with more useful image information that corresponds to more number of higher AC coefficients, one can see that the resultant fused images obtained by the proposed method in Figs. 5–7 yields better image quality than that of the other methods. Fig. 8 shows the fusion result of three blurred lake images with focus on the left, middle and right. From Fig. 8, it is seen that the proposed fused method can be extended for more than two blurred images and it produces the ideal fused ‘lake’ image with mean SSIM as 1. Fig. 9 depicts the fusion of near focused, middle focused and far focused toy image. Here again the proposed method yields better image quality than that of the other compared methods. For the wavelet based methods, (the DWT with DBSS (2,2) and the SIDWT with Haar basis), simulation was carried out with the “Image Fusion Toolbox”, kindly provided by Rockinger [24]. Here comparison of fusion methods is done similar to [8,12]. The MSE, PSNR and SSIM performance comparison of various fusion methods on referenced images is presented in Table 1, where one can see that the proposed approach performs
Y.A.V. Phamila, R. Amutha / Signal Processing 95 (2014) 161–170
169
Table 4 Energy consumption for fusing two 8 8 blocks. Fusion scheme
Transform cycles for two 8 8 blocks
Fusion cycles
Total cycles
Execution time (ms)
Energy (mJ)
DCT þ Average DCT þ Variance DWT with DBSS(2,2) SIDWT with Haar Proposed (DCT þAC_Max)
1,160,212 1,160,212 2,414,770 3,017,868 1,160,212
243,595 488,600 28,273 31,092 5550
1,403,807 1,648,812 2,443,043 3,048,960 1,165,762
175.475 206.10 305.38 381.12 145.720
3860.47 4534.23 6718.37 8384.64 3205.84
better than other four conventional approaches by producing the best metric values. From Table 1 it is clear that the proposed method reproduces a very high quality fused image and in most cases the fused image is exactly identical to the referenced image (MSE¼0, MSSIM¼1). The performance metric comparison for fusion of nonreferenced images is given in Table 2 and from Table 2 it is inferred that the proposed approach performs better than the other four approaches by producing the best objective performance. To emphasize the performance improvement of DCTþAC_Max fusion method over DCTþVariance [8] method, a detailed analysis is given in Table 3 for fusing two blurred Barbara images. Two blurred artificial images are generated for the test image by convolving the test image with a disk averaging filter of radius 9. The blurred images are fused using DCTþ Variance Fusion method and the proposed AC_Max fusion method. Both the results are tabulated in Table 3. Here AC _Max is computed for various sizes Ns of DCT transformed (8 8) top left Zonal sub matrix.(Ns Ns where Ns ¼2,3,4,5,6,7,8) [10]. Since energy compaction is more towards top left submatrix of the transformed coefficients [10], AC_Max count starting with submatrix size Ns 42, gives a better fused result than that of the variance method. 5. Energy consumption analysis For energy consumption analysis, the ATmega128 processor of Mica 2 mote at 8 MHz with an active power consumption of 22 mW is used as the target platform [25]. Compilation is performed via WinAVR with “–O3” optimization setting. In the proposed scheme, the key step is to fuse the DCT representations of multi-focus images into a single fused image based on the AC coefficients values. The computation cost includes the Discrete Cosine Transform and the Fusion Rule. Since the fusion rule does not involve any complex arithmetic floating point operations like mean or variance calculations, it is extremely simple, fast and efficient and hence is suitable for real time applications. The computation cycles and the energy needed for fusing two 8 8 image blocks using the fusion schemes in DWT and DCT domain are given in Table 4. Since DWT involves many convolution operations, DWT based fusion methods consume huge computation time and energy when compared to DCT based methods. In DCT þAverage method, each transformed block needs 64 floating point additions and one floating point division to compute its mean value. In DCT þVariance method, each block needs 63 floating point divisions for normalization of AC coefficients, followed by 63 floating point multiplications and additions as given in Eq. (6) to compute the variance in DCT domain. But in the proposed method only 63
integer additions are used to count the number of higher valued AC coefficients. Hence from column 3 of Table 4 it is inferred that the computation cycle for fusion by the proposed fusion method is 1% and 2% of that of the DCT þVariance and DCT þAverage methods respectively. It is extremely fast as it does not involve any complex floating point arithmetic operations like mean or variance calculation. Hence from column 6 of Table 4 it is evident that this fusion scheme is energy efficient as it consumes only 38%, 48% and 71% of energy needed by SIDWT, DWT and DCT þVariance based fusion methods respectively and guarantees better fused image quality. Also the fusion method can be integrated with transform using Binary DCT [10], Speed optimized integer versions of JPEG-DCT [25,26] or Optimized Integer DCT [25] instead of floating point standard floating point DCT which will further reduce the computation complexity.
6. Conclusion In this paper, a very simple, fast and energy efficient DCT based multi-focus image fusion scheme is proposed which outperforms other DCT based fusion methods as verified in our extensive experiments. Since the fusion rule does not involve any complex arithmetic floating point operations like mean or variance calculations, it is extremely simple and energy efficient, making it more suitable for resource constrained battery powered sensors for energy efficient fusion and subsequent compression. In future it is planned to validate our approach on a sensor network testbed. References [1] D. Drajic, N. Cvejic, Adaptive fusion of multimodal surveillance image sequences in visual sensor networks, IEEE Transactions on Consumer Electronics 53 (4) (2007) 1456–1462. [2] Shutao Li, Bin Yang, Multifocus image fusion using region segmentation and spatial frequency, Image Vision Computing 26 (7) (2008) 971–979. [3] Jing Tian, Li Chen, Adaptive multi-focus image fusion using a wavelet-based statistical sharpness measure, Signal Processing 92 (9) (2012) 2137–2146. [4] Praveen Verma, Manoj Kumar, Maheedhar Dubey, Prateek Verma, Multifocus image fusion using wavelet transform, International Journal of Electronics and Computer Science Engineering 1 (4) (2012) 2112–2118. [5] Shutao Li, Bin Yang, Multifocus image fusion by combining curvelet and wavelet transform, Pattern Recognition Letters 29 (2008) 1295–1301. [6] M.A. Mohamed, R.M. EI-Den, Implementation of image fusion techniques for multi-focus images using FPGA, in: Proceedings of the National Radio Science Conference, National Telecommunication Institute, Egypt, 2011.
170
Y.A.V. Phamila, R. Amutha / Signal Processing 95 (2014) 161–170
[7] Jinshan Tang, A contrast based image fusion technique in the DCT domain, Digital Signal Processing 14 (3) (2004) 218–226. [8] Mohammad Bagher Akbari Haghighat, Ali Aghagolzadeh, Hadi Seyedarabi, Multi-focus image fusion for visual sensor networks in DCT domain, Computers and Electrical Engineering 37 (5) (2011) 789–797. [9] M. Hossny, S. Nahavandi, D. Creighton, A. Bhatti, Towards autonomous image fusion, in: Proceedings of the International Conference on Control, Automation, Robotics and Vision, 2010. [10] V. Lecuire, L. Makkaoui, J.-M. Moureaux, Fast zonal DCT for energy conservation in wireless image sensor networks, Electronic Letters 48 (2) (2012) 125–127. [11] C. Poynton, Digital Video and HDTV Algorithms and Interfaces, Elsevier, San Francisco, 2003. [12] H. Li, B. Manjunath, S.K. Mitra, Multisensor image fusion using the wavelet transform, Graph Models Image Processing 57 (3) (1995) 235–245. [13] Zhou Wang, Alan Conrad Bovik, Hamid Rahim Sheikh, Eero P. Simoncelli, Image quality assessment: from error visibility to structural similarity, IEEE Transactions on Image Processing 13 (4) (2004) 600–612. [14] Shutao Li, Bin Yang, Multifocus image fusion using region segmentation and spatial frequency, Image and Vision Computing 26 (2008) 971–979. [15] Xiangzhi Bai, Fugen Zhou, Bindang Xue, Edge preserved image fusion based on multiscale toggle contrast operator, Image and Vision Computing 29 (2011) 829–839.
[16] C.S. Xydeas, V. Petrovic, Objective image fusion performance measure, Electronic Letters 36 (4) (2000) 308–309. [17] V. Petrovic, T. Cootes, Objectively optimised multisensor image fusion, in: Proceedings of 9th International Conference on Information Fusion, Florence, 2006, pp. 1–7. [18] G. Qu., D. Zhang, P. Yan, Information measure for performance of image fusion, Electronic Letters 38 (7) (2002) 313–315. [19] 〈http://decsai.ugr.es/cvg/CG/base.htm〉 (last accessed April 2013). [20] 〈http://links.uwaterloo.ca/Repository.html〉 (last accessed April 2013). [21] 〈http://www.imagefusion.org〉 (last accessed April 2013). [22] 〈http://www.mathworks.com/matlabcentral/fileexchange/40861〉 (last accessed April 2013). [23] Rockinger O., Image sequence fusion using a shift-invariant wavelet transform, in: Proceedings of IEEE International Conference on Image Processing, Santa Barbara, CA, 1997, pp. 288–291. [24] 〈http://www.metapix.de/toolbox.htm〉 (last accessed April 2013). [25] Dong-U Lee, Hyungjin Kim, Mohammad Rahimi Estrin, Energyefficient image compression for resource-constrained platforms, IEEE Transactions on Image Processing 18 (9) (2009) 2100–2113. [26] T. Lane, P. Gladstone, L. Ortiz, J. Boucher, L. Crocker, J. Minguillon, G. Phillips, D. Rossi, G. Weijers, The Independent JPEG Group's JPEG Software Release 6b [Online]. Available From: 〈http://www.ijg.org〉 (last accessed April 2013).
Contents lists available at ScienceDirect
Signal Processing journal homepage: www.elsevier.com/locate/sigpro
Fast communication
Discrete Cosine Transform based fusion of multi-focus images for visual sensor networks Y. Asnath Victy Phamila n, R. Amutha SSN College of Engineering, Anna University, Chennai, India
a r t i c l e i n f o
abstract
Article history: Received 29 April 2013 Received in revised form 29 August 2013 Accepted 2 September 2013 Available online 16 September 2013
This paper presents a simple and efficient multi-focus image fusion scheme explicitly designed for wireless visual sensor systems equipped with resource constrained, battery powered image sensors employed in surveillance, hazardous environment like battlefields etc. Here the fusion of multi-focus images is based on higher valued Alternating Current (AC) coefficients calculated in Discrete Cosine Transform (DCT) domain. The proposed method overcomes the computation and energy limitation of low power devices and is investigated in terms of image quality and computation energy. Simulations are performed using Atmel Atmega128 processor of Mica 2 mote to measure the resultant energy savings. The experimental results verify the significant efficiency improvement of the proposed method in output quality and energy consumption, when compared with other fusion techniques in DCT domain. & 2013 Elsevier B.V. All rights reserved.
Keywords: Image fusion Discrete Cosine Transform Visual sensor network Fusion metrics
1. Introduction Image fusion is the process of combining multiple source images from sensor network into a single one, which contains a more accurate description of the scene, more informative and suitable for both visual perception and further processing [1]. In multi-focus image fusion technique, several images of a scene captured with focus on different objects are fused such that all the objects will be in focus in the resulting image. So far, several researches have been focused on image fusion which is performed on the images in the spatial and spectral domain [2–8]. Various multi-focus image fusion algorithms in wavelet domain are available in literature [3–5]. In [3] statistical sharpness measure based on wavelet coefficients distribution is used to perform adaptive image fusion in wavelet domain. Instead of using Discrete Wavelet Transform (DWT) to decompose images into frequency domain, Discrete Stationary Wavelet transform is used in [4] to overcome the
n
Corresponding author. Tel.: þ91 9884322100; fax: þ91 44 27469772. E-mail addresses: [email protected] (Y.A.V. Phamila), [email protected] (R. Amutha). 0165-1684/$ - see front matter & 2013 Elsevier B.V. All rights reserved. http://dx.doi.org/10.1016/j.sigpro.2013.09.001
lack of translation invariance of the Discrete Wavelet Transform. Then the transformed coefficients are fused and the fused image is constructed by applying inverse Discrete Stationary Wavelet Transform. As the wavelets do not represent long edges well in the fused results, multi-focus image fusion is performed by combining both the wavelet and curvelet transforms to improve the quality [5]. But the limitation is that it consumes more time than the wavelet-based methods because two different multiscale decomposition processes are applied. However multi-focus image fusion based on DWT has its own disadvantages. DWT needs great number of convolution calculations, and it consumes much time or memory resources, which impedes its application for resource constrained battery powered visual sensor nodes. The energy needed for DCT based fusion is less compared to the DWT based methods. Hence DCT based fusion methods are more appropriate for resource constrained devices. Since the computational energy is much less than the transmission energy, data are compressed and fused before transmission in automated battlefields, where the robots collect image data from sensor network [9]. When the source images are to be coded in Joint Photographic Experts Group (JPEG) standard or when the resultant fused
162
Y.A.V. Phamila, R. Amutha / Signal Processing 95 (2014) 161–170
image is to be saved or transmitted in JPEG format, the fusion methods in DCT domain will be more efficient [7,8]. Contrast sensitivity method is adopted in [7] to construct the fused image. The contrasts of the corresponding AC coefficients of different blurred images are compared and the AC coefficient with the largest contrast value is selected as the AC coefficient of the fused image. But the contrast calculation for each AC coefficient involves complex floating point computations and the fused image suffers from blocking artifacts due to diversity in the selection of DCT coefficients. In [6,7] the DCT representation of the fused image is obtained by taking the average of all the DCT representations of all the input images but the fused result has undesirable blurring effects. To overcome the undesirable side effects like blurring or blocking artifacts which reduce the quality of the fused image, multi-focus image fusion based on variance calculated in DCT domain is presented in [8]. However the mean, contrast and variance calculation for fusion [7,8] involves complex floating point arithmetic operations which incur high energy consumption in resource constrained battery powered sensor nodes. In this paper, a simple and efficient multi-focus image fusion scheme that is suitable for resource-constrained (processing, bandwidth, memory space, battery power) sensor network is proposed. In the proposed method, the image blocks with more number of higher valued AC coefficients is absorbed into the fused image. It is extremely fast as it does not involve any complex floating point arithmetic operations like mean or variance calculation. The proposed fusion rule considerably reduces the computational complexity without compromising image quality. Since the proposed fusion scheme is performed in DCT domain, it is time-saving and simple when the fused image needs to be saved or transmitted in JPEG format [7,8]. Simulations are performed using Atmel Atmega128 processor of Mica 2 mote to measure the resultant energy savings.
2. Image fusion In the proposed scheme, the key step is to fuse the DCT representations of multi-focus images into a single fused image. The input images are divided into blocks of size 8 8 and the DCT coefficients of each block is computed. Then the fusion rule is applied wherein the transformed block with more number of higher valued AC coefficients is absorbed into the fused image.
where u, v¼0, 1, …, N 1 and ( pffiffiffi 1= 2; if u ¼ 0 cðuÞ ¼ 1; if u a 0
ð2Þ
the inverse transform is defined as f ðx; yÞ ¼
2 N1 N1 ð2x þ 1Þuπ ∑ ∑ cðuÞcðvÞFðu; vÞ cos Nv¼0u¼0 2N ð2y þ1Þvπ cos 2N
ð3Þ
where x, y¼0, 1, …, N 1. Here F (0, 0) is the DC coefficient and it represents the mean value of that image block. Remaining coefficients are AC coefficients. 2.2. Fusion based on variance In [8] variance is used as the activity level for fusion criteria because in multi-focus images, the focused region is more informative and the information details of that region correspond to high variance. It is inferred that the variance of an N N block of pixels can be exactly calculated from its DCT coefficients by computing the sum of the squared normalized AC coefficients of the DCT block. The normalized transform coefficients are defined as ^ vÞ ¼ Fðu; vÞ Fðu; N
ð4Þ
2
variance (s ) of the image block [8] can be inferred from the transformed coefficients as follows: N 1 N 1 F 2 ðu; vÞ
s2 ¼ ∑
∑
u¼0v¼0
N2
2
F^ ð0; 0Þ
ð5Þ
^ 0Þ is the normalized DC coefficient and other where Fð0; ^ vÞs are the normalized AC coefficients. Fðu; 2.3. Fusion criteria for the proposed AC_Max fusion The advantage of DCT is that the energy of the original data may be concentrated in only a few low frequency components of DCT depending on the correlation in the data. Also the low-frequency components usually contain the most of the image information. Higher the value of AC coefficients implies finer image information. Eq. (5) implies that the variance of a block of size 8 8 is given by the sum of the squares of the normalized 63 AC coefficients. 63
^ 2 s2 ¼ ∑ Ai
ð6Þ
i¼1
2.1. Discrete Cosine Transform Two dimensional DCT transform of an N N image block f (x, y) [8] is given as Fðu; vÞ ¼
N1 N1 2 ð2x þ1Þuπ cðuÞðvÞ ∑ ∑ f ðx; yÞ cos N 2N y¼0x¼0
cos
ð2y þ 1Þvπ 2N
ð1Þ
Because of the energy compaction property of AC coefficients, only few coefficients towards the top left submatrix of the DCT transformed matrix have larger values [10] and the contribution of these coefficients to variance is more compared to other AC coefficients. Hence if the AC coefficient value is high, then the variance value is also high. Hence in our proposed method we absorb the block with more number of higher valued AC coefficients for two reasons. First is that higher the AC component value implies more fine details of the image [11]. Secondly, from
Y.A.V. Phamila, R. Amutha / Signal Processing 95 (2014) 161–170
Eq. (6) it is inferred that higher the AC component value results in higher variance. Here instead of computing variance using all the transformed AC coefficients which involves floating point multiplication and additions, the proposed algorithm checks only the number of higher valued AC coefficients that contributes to larger variance. Thus the energy needed for computation is drastically reduced. The quality of the fused image is significantly high because only the blocks with high energy where more image details are stored, are selected for fusion.
163
This is repeated for all Q blocks to fuse the DCT representations of multi-images into a single DCT representation of image. The fused DCT blocks are subject to consistency verification (CV) [12] in which if the center block comes from the source image1 and majority of the surrounding blocks come from image 2, the center block is switched to that of image 2. Here 3 3 neighborhood window is used for consistency verification. After consistency verification, the fused DCT coefficients can be easily saved or transmitted in JPEG format. The original fused image can be reconstructed by applying inverse DCT to the fused DCT coefficients.
2.4. AC_Max fusion method 3. Performance metrics Fig. 1 illustrates the general framework of our proposed image fusion scheme. The algorithm can be extended for more than two source images with the assumption that all the source images are aligned with some registration methods. Let Y¼{yi,j} (i¼0, …, N 1 and j¼0, …, M 1) be an image and it is divided into Q number of 8 8 blocks. Let Xn¼{xn,k,l} (k¼0, …, 7; l¼0, …, 7; n¼ 0, …, Q1) be the nth 8 8 block and the corresponding DCT output of the block Xn¼ {xn,k,l} be Dn¼{Dn,k,l}. Then the set D¼{D0, D1, D2, …, DQ 1} denotes the DCT representation of image Y¼{yi,j}. Let Dt ¼ fDt 0 ; Dt 1 ; Dt 2 ; …; Dt Q 1 g be the DCT representation of the tth input image and let B be the number of input source images to be fused. Then the DCT representation DF ¼ fDF 0 ; DF 1 ; DF 2 ; …; DF Q 1 g of the fused image is obtained by fusing the transformed DCT coefficients of the input set of B images. The fusion criterion is that the block with the majority of maximum valued DCT AC coefficients is absorbed into the fused image since it contributes more significant signal information to the fused image. Hence in our fusion method, the nth block of the fused image ðDF n Þ is obtained by Eqs. (7) and (8). DF n ¼ DT n
Extensive experiments are performed to demonstrate the superior performance of the proposed algorithm using
ð7Þ
where T ¼ arg max fC tn g; t ¼ 1; …; B
ð8Þ
t
C tn in Eq. (8) specifies the number of maximum valued transformed AC coefficients found in the nth block of tth image when compared with the respective blocks in other source images. For example, the fused DCT block DF n of two source image blocks (X 1 n and X 2 n ) is obtained as illustrated in the flowchart (Fig. 2).
Fig. 2. Flowchart for AC_Max fusion rule.
Fig. 1. General structure of the proposed fusion scheme.
164
Y.A.V. Phamila, R. Amutha / Signal Processing 95 (2014) 161–170
six performance metrics. The structural similarity measure (SSIM) [8,13], mean square error (MSE) and peak signal to noise ratio (PSNR) are used as quality measures for objective evaluation of results of referenced images. If MSE equals zero, it implies that the fused image is exactly the same as that of the original referenced image. The higher the values of PSNR and SSIM, the better are the quality of the fused image. To evaluate our proposed algorithm on non-referenced multi-focus images, the spatial frequency (SF) metric [14,15], the state-of-the-art fusion performance metric Petrovic [8,16,17] metric (QAB/F) and mutual information (MI) [18] are used.
3.1. Mean square error (MSE) MSE is the cumulative squared error between the fused and the referenced image and it is defined by Eq. (9) MSE ¼
N 1 M ∑ ∑ ½Iðm; nÞ Fðm; nÞ2 MN m ¼ 1 n ¼ 1
ð9Þ
where I(m, n) is the pixel value of the referenced image and F(m, n) is the pixel value of the fused image. If MSE¼0, it is a perfect fusion where the fused image exactly matches the referenced image.
Fig. 3. Fusion results of multifocus images. (a) Traffic image and (b) Battlefield image.
Fig. 4. Standard test images.
Y.A.V. Phamila, R. Amutha / Signal Processing 95 (2014) 161–170
3.2. Peak signal to noise ratio (PSNR) The quality of the image is evaluated using PSNR. The PSNR is calculated using Eq. (10) PSNR ¼ 10log 10
ð2b 1Þ2 dB MSE
ð10Þ
where b is the number of bits per pixel (bpp) of the original image, and MSE is the mean-square-error which
165
is defined by Eq. (9). Higher PSNR value indicates better quality. If MSE ¼0, then the PSNR becomes infinity which implies ideal fusion, where the fused image and the referenced test image are exactly identical. 3.3. Structural similarity index (SSIM) The structural similarity measure [8,13], as a quality criterion, is used for objective evaluation of fused image.
Fig. 5. Fusion results of Lena image. (a) Referenced image. (b) Focussed on right. (c) Focussed on left. (d) DCT þAverage result. (e) DCT þVariance result. (f) DWT result. (g) SIDWT result. (h) Proposed method.
Fig. 6. Fusion results of Pepsi image. (a) Source Image 1. (b) Source Image 2. (c) DCT þAverage result. (d) DCT þ Variance result. (e) DWT result. (f) SIDWT result. (g) Proposed method.
166
Y.A.V. Phamila, R. Amutha / Signal Processing 95 (2014) 161–170
Fig. 7. Fusion results of Clock image. (a) Source Image 1. (b) Source Image 2. (c) DCT þ Average result. (d) DCT þ Variance result. (e) DWT result. (f) SIDWT result. (g) Result of the proposed method.
The general form of the metric that is used to measure the structural similarity between two signal vectors x and y is given by Eq. (11) ! ! 2μx μy þ C 1 2sxy þC 2 ð11Þ SSIMðx; yÞ ¼ μ2x þ μ2y þ C 1 s2x þs2y þ C 2 where mx and my are the sample means of x and y respectively, sx 2 and sy 2 are the sample variances of x and y respectively, and sxy is the sample cross-covariance between x and y. The default values for C1 and C2 are 0.01 and 0.03. The average of the SSIM values across the image (mean SSIM or MSSIM) gives the final quality measure. AB/F
3.4. Petrovic metric (Q
)
This measure was proposed by Xydeas and Petrovic [16,17]. This metric is based on the assumption that fusion algorithm that transfers input gradient information into resultant image more accurately performs better. In this case, a pixel wise measure of information preservation is obtained between each input image (A and B) and the fused image (F) of size M N to compute QAB/F using simple local perceptual importance factors. It is calculated by Q AB=F ¼
AF BF M A B ∑N n ¼ 1 ∑m ¼ 1 ðQ ðn; mÞw ðn; mÞ þ Q ðn; mÞw ðn; mÞÞ N M A B ∑n ¼ 1 ∑m ¼ 1 ðw ðn; mÞ þw ðn; mÞÞ
ð12Þ AF
3.5. Spatial frequency (SF) The row and column frequencies of the fused image (F) of size M N is computed [14,15] as given in Eqs. (13) and (14). sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 M1 N1 RF ¼ ∑ ∑ ½Fðm; nÞ Fðm; n 1Þ2 ð13Þ MN m ¼ 0 n ¼ 1 sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 N1 M 1 CF ¼ ∑ ∑ ½Fðm; nÞ Fðm 1; nÞ2 MN n ¼ 0 m ¼ 1
then the total spatial frequency of the fused image which is based on edge information is computed as follows: qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi SF ¼ ðRFÞ2 þ ðCFÞ2 ð15Þ higher the spatial frequency, higher is the clarity of the image. 3.6. Mutual information (MI) This metric gives the amount of information that the fused image F has from input source images (A and B) [18]. MI represents the similarity of the image intensity distributions of the corresponding image pair. Higher the MI better is the quality of the fused image. The mutual information between source image A and the fused image F is computed as given in Eq. (16)
BF
where Q and Q are computed using edge information preservation values [14,16]. wA(n,m) And wB(n,m) are the weighted importance factors for QAF and QBF respectively. QAB/F is in the range [0, 1] where 0 means complete loss of information and 1 means ideal fusion.
ð14Þ
I AF ¼ ∑pAF ða; f Þlog a;f
pAF ða; f Þ pA ðaÞpF ðf Þ
ð16Þ
where pAF, pA and pB are computed by normalization of the joint and marginal histograms of A and F. Similarly mutual
Y.A.V. Phamila, R. Amutha / Signal Processing 95 (2014) 161–170
167
Fig. 8. Fusion results of Lake image. (a) Referenced Image. (b) Focussed on Left. (c) Focussed on Middle. (d) Focussed on Right. (e) DCT þAverage result. (f) DCT þ Variance result. (g) DWT result. (h) SIDWT result. (i) Proposed method.
Fig. 9. Fusion results of Toy image. (a) Near Focussed Image. (b) Middle Focussed Image. (c) Far Focussed Image. (d) DCT þAverage result. (e) DCT þVariance result. (f) DWT result. (g) SIDWT result. (h) Result of the proposed method.
168
Y.A.V. Phamila, R. Amutha / Signal Processing 95 (2014) 161–170
Table 1 The MSE, PSNR and MSSIM comparison of various image fusion approaches on reference images. Fusion Method
Lena MSE
DCT þAverage 39.11 DCT þ Variance 8.70 DCT þ Variance þ CV 1.88 DWT with DBSS(2,2) 5.62 SIDWT with Haar 5.48 Proposed (DCT þ AC_Max) 0 Proposed(DCTþAC_Max þCV) 0
Traffic
Bridge
PSNR (dB) MSSIM MSE
PSNR (dB) MSSIM MSE
32.20 38.73 45.38 40.63 40.74 1 1
27.62 41.89 49.12 37.05 37.99 1 1
0.9383 112.6 0.9854 4.2 0.9982 0.797 0.9906 12.81 0.9899 10.33 1 0 1 0
Lake PSNR (dB) MSSIM MSE
0.8997 28.83 33.53 0.9920 6.69 39.87 0.9997 4.89 41.23 0.9901 4.49 41.61 0.9905 4.54 41.55 1 0.03 62.91 1 0.02 64.32
PSNR (dB) MSSIM
0.9465 145.43 0.9870 13.64 0.9957 0.297 0.9895 14.58 0.9893 33.67 0.9998 0 0.9998 0
26.50 36.78 53.40 36.49 32.86 1 1
0.8343 0.9849 0.9999 0.9836 0.9759 1 1
Table 2 The QAB/F, spatial frequency (SF) and mutual information(MI) of various image fusion approaches on non-referenced images. Fusion method
Clock AB/F
DCT þAverage DCT þVariance þ CV DWT with DBSS(2,2) SIDWT with Haar Proposed(DCT þ AC_Max þCV)
Pepsi AB/F
Q
SF
MI
Q
0.65 0.73 0.67 0.71 0.74
11.56 18.39 18.38 17.24 18.46
7.13 9.04 6.50 6.75 9.04
0.63 0.77 0.73 0.74 0.78
Lab
MI ¼ I AF þ I BF
Aircraft
MI
Q
SF
MI
Q
SF
MI
QAB/F
SF
MI
10.57 13.81 13.83 13.39 13.96
6.84 8.51 6.35 6.60 8.66
0.55 0.73 0.66 0.68 0.73
7.72 13.17 13.10 12.38 13.41
7.08 8.60 6.52 6.93 8.65
0.61 0.80 0.79 0.79 0.81
9.09 16.70 16.77 16.36 16.94
4.01 5.74 4.12 4.55 5.74
0.58 0.60 0.54 0.61 0.62
9.1 13.45 13.59 13.25 13.70
9.94 10.13 7.45 8.46 10.20
ð17Þ
4. Experimental results The proposed fusion algorithm is applied on a set of nonreferenced and set of referenced images and the results are evaluated. The first experiment is conducted using an extensive set of artificially generated images with different focus levels. Standard test images like Traffic, Lena, Battlefield, Barbara, Bird etc are taken as ground truth images [19,20]. Two blurred artificial images are generated for each test image by convolving the test image with a 9 9 averaging filter centered at the left part and right part respectively [3]. Fig. 3 depicts the fusion result of the proposed method on referenced multi-focused ‘traffic’ and ‘battlefield’ images. From Fig. 3, it is seen the fused image exactly represents the original referenced image with mean SSIM value as 1. The proposed fusion method is experimented on the 48 sets of (48 2¼96) blurred images artificially generated by convolving the test images with a 7 7 averaging filter centered at the left part and right part respectively. The 48 standard grayscale test images are shown in Fig. 4. In 42 sets of images, this proposed fusion scheme results in ideal fused results with mean SSIM (MSSIM) as 1. The second experiment is conducted on sets of standard non-referenced multifocus test images [21]. The fusion result of Lena, Pepsi and Clock standard test images by applying various fusion algorithms in DCT domain [8,22] (DCTþAverage and DCTþVariance), DWT domain (DWT [12] and SIDWT [23]) and our proposed method is shown in Figs. 5–7
AB/F
Toy
SF
information IBF is computed between the source image B and the fused image. Then the mutual information between the source images (A, B) and the fused image (F) is given as follows:
AB/F
Table 3 Fusion metrics for DCT þVariance and DCT þ AC_Max with different DCT submatrix of size Ns on Barbara Image. Fusion method (Ns)
MSE
PSNR (dB) SSIM
DCT þVariance (8 8) 20.28 35.06 DCT þAC_Max(2 2) 71.27 29.60 DCT þAC_Max (3 3) 6.25 40.17 DCT þAC_Max (4 4) 2.20 44.71 DCT þAC_Max (5 5) 0.36 52.55 DCT þAC_Max (6 6) 0 1 DCT þAC_Max (7 7) 0 1 DCT þAC_Max (8 8) 0 1
0.9643 0.9032 0.9854 0.9942 0.9988 1 1 1
MI
SF
QAB/F
7.37 7.26 7.43 7.46 7.48 7.49 7.49 7.49
28.37 26.77 28.39 28.40 28.40 28.40 28.40 28.40
0.80 0.69 0.80 0.81 0.81 0.81 0.81 0.81
respectively. As the proposed fusion method selects blocks with more useful image information that corresponds to more number of higher AC coefficients, one can see that the resultant fused images obtained by the proposed method in Figs. 5–7 yields better image quality than that of the other methods. Fig. 8 shows the fusion result of three blurred lake images with focus on the left, middle and right. From Fig. 8, it is seen that the proposed fused method can be extended for more than two blurred images and it produces the ideal fused ‘lake’ image with mean SSIM as 1. Fig. 9 depicts the fusion of near focused, middle focused and far focused toy image. Here again the proposed method yields better image quality than that of the other compared methods. For the wavelet based methods, (the DWT with DBSS (2,2) and the SIDWT with Haar basis), simulation was carried out with the “Image Fusion Toolbox”, kindly provided by Rockinger [24]. Here comparison of fusion methods is done similar to [8,12]. The MSE, PSNR and SSIM performance comparison of various fusion methods on referenced images is presented in Table 1, where one can see that the proposed approach performs
Y.A.V. Phamila, R. Amutha / Signal Processing 95 (2014) 161–170
169
Table 4 Energy consumption for fusing two 8 8 blocks. Fusion scheme
Transform cycles for two 8 8 blocks
Fusion cycles
Total cycles
Execution time (ms)
Energy (mJ)
DCT þ Average DCT þ Variance DWT with DBSS(2,2) SIDWT with Haar Proposed (DCT þAC_Max)
1,160,212 1,160,212 2,414,770 3,017,868 1,160,212
243,595 488,600 28,273 31,092 5550
1,403,807 1,648,812 2,443,043 3,048,960 1,165,762
175.475 206.10 305.38 381.12 145.720
3860.47 4534.23 6718.37 8384.64 3205.84
better than other four conventional approaches by producing the best metric values. From Table 1 it is clear that the proposed method reproduces a very high quality fused image and in most cases the fused image is exactly identical to the referenced image (MSE¼0, MSSIM¼1). The performance metric comparison for fusion of nonreferenced images is given in Table 2 and from Table 2 it is inferred that the proposed approach performs better than the other four approaches by producing the best objective performance. To emphasize the performance improvement of DCTþAC_Max fusion method over DCTþVariance [8] method, a detailed analysis is given in Table 3 for fusing two blurred Barbara images. Two blurred artificial images are generated for the test image by convolving the test image with a disk averaging filter of radius 9. The blurred images are fused using DCTþ Variance Fusion method and the proposed AC_Max fusion method. Both the results are tabulated in Table 3. Here AC _Max is computed for various sizes Ns of DCT transformed (8 8) top left Zonal sub matrix.(Ns Ns where Ns ¼2,3,4,5,6,7,8) [10]. Since energy compaction is more towards top left submatrix of the transformed coefficients [10], AC_Max count starting with submatrix size Ns 42, gives a better fused result than that of the variance method. 5. Energy consumption analysis For energy consumption analysis, the ATmega128 processor of Mica 2 mote at 8 MHz with an active power consumption of 22 mW is used as the target platform [25]. Compilation is performed via WinAVR with “–O3” optimization setting. In the proposed scheme, the key step is to fuse the DCT representations of multi-focus images into a single fused image based on the AC coefficients values. The computation cost includes the Discrete Cosine Transform and the Fusion Rule. Since the fusion rule does not involve any complex arithmetic floating point operations like mean or variance calculations, it is extremely simple, fast and efficient and hence is suitable for real time applications. The computation cycles and the energy needed for fusing two 8 8 image blocks using the fusion schemes in DWT and DCT domain are given in Table 4. Since DWT involves many convolution operations, DWT based fusion methods consume huge computation time and energy when compared to DCT based methods. In DCT þAverage method, each transformed block needs 64 floating point additions and one floating point division to compute its mean value. In DCT þVariance method, each block needs 63 floating point divisions for normalization of AC coefficients, followed by 63 floating point multiplications and additions as given in Eq. (6) to compute the variance in DCT domain. But in the proposed method only 63
integer additions are used to count the number of higher valued AC coefficients. Hence from column 3 of Table 4 it is inferred that the computation cycle for fusion by the proposed fusion method is 1% and 2% of that of the DCT þVariance and DCT þAverage methods respectively. It is extremely fast as it does not involve any complex floating point arithmetic operations like mean or variance calculation. Hence from column 6 of Table 4 it is evident that this fusion scheme is energy efficient as it consumes only 38%, 48% and 71% of energy needed by SIDWT, DWT and DCT þVariance based fusion methods respectively and guarantees better fused image quality. Also the fusion method can be integrated with transform using Binary DCT [10], Speed optimized integer versions of JPEG-DCT [25,26] or Optimized Integer DCT [25] instead of floating point standard floating point DCT which will further reduce the computation complexity.
6. Conclusion In this paper, a very simple, fast and energy efficient DCT based multi-focus image fusion scheme is proposed which outperforms other DCT based fusion methods as verified in our extensive experiments. Since the fusion rule does not involve any complex arithmetic floating point operations like mean or variance calculations, it is extremely simple and energy efficient, making it more suitable for resource constrained battery powered sensors for energy efficient fusion and subsequent compression. In future it is planned to validate our approach on a sensor network testbed. References [1] D. Drajic, N. Cvejic, Adaptive fusion of multimodal surveillance image sequences in visual sensor networks, IEEE Transactions on Consumer Electronics 53 (4) (2007) 1456–1462. [2] Shutao Li, Bin Yang, Multifocus image fusion using region segmentation and spatial frequency, Image Vision Computing 26 (7) (2008) 971–979. [3] Jing Tian, Li Chen, Adaptive multi-focus image fusion using a wavelet-based statistical sharpness measure, Signal Processing 92 (9) (2012) 2137–2146. [4] Praveen Verma, Manoj Kumar, Maheedhar Dubey, Prateek Verma, Multifocus image fusion using wavelet transform, International Journal of Electronics and Computer Science Engineering 1 (4) (2012) 2112–2118. [5] Shutao Li, Bin Yang, Multifocus image fusion by combining curvelet and wavelet transform, Pattern Recognition Letters 29 (2008) 1295–1301. [6] M.A. Mohamed, R.M. EI-Den, Implementation of image fusion techniques for multi-focus images using FPGA, in: Proceedings of the National Radio Science Conference, National Telecommunication Institute, Egypt, 2011.
170
Y.A.V. Phamila, R. Amutha / Signal Processing 95 (2014) 161–170
[7] Jinshan Tang, A contrast based image fusion technique in the DCT domain, Digital Signal Processing 14 (3) (2004) 218–226. [8] Mohammad Bagher Akbari Haghighat, Ali Aghagolzadeh, Hadi Seyedarabi, Multi-focus image fusion for visual sensor networks in DCT domain, Computers and Electrical Engineering 37 (5) (2011) 789–797. [9] M. Hossny, S. Nahavandi, D. Creighton, A. Bhatti, Towards autonomous image fusion, in: Proceedings of the International Conference on Control, Automation, Robotics and Vision, 2010. [10] V. Lecuire, L. Makkaoui, J.-M. Moureaux, Fast zonal DCT for energy conservation in wireless image sensor networks, Electronic Letters 48 (2) (2012) 125–127. [11] C. Poynton, Digital Video and HDTV Algorithms and Interfaces, Elsevier, San Francisco, 2003. [12] H. Li, B. Manjunath, S.K. Mitra, Multisensor image fusion using the wavelet transform, Graph Models Image Processing 57 (3) (1995) 235–245. [13] Zhou Wang, Alan Conrad Bovik, Hamid Rahim Sheikh, Eero P. Simoncelli, Image quality assessment: from error visibility to structural similarity, IEEE Transactions on Image Processing 13 (4) (2004) 600–612. [14] Shutao Li, Bin Yang, Multifocus image fusion using region segmentation and spatial frequency, Image and Vision Computing 26 (2008) 971–979. [15] Xiangzhi Bai, Fugen Zhou, Bindang Xue, Edge preserved image fusion based on multiscale toggle contrast operator, Image and Vision Computing 29 (2011) 829–839.
[16] C.S. Xydeas, V. Petrovic, Objective image fusion performance measure, Electronic Letters 36 (4) (2000) 308–309. [17] V. Petrovic, T. Cootes, Objectively optimised multisensor image fusion, in: Proceedings of 9th International Conference on Information Fusion, Florence, 2006, pp. 1–7. [18] G. Qu., D. Zhang, P. Yan, Information measure for performance of image fusion, Electronic Letters 38 (7) (2002) 313–315. [19] 〈http://decsai.ugr.es/cvg/CG/base.htm〉 (last accessed April 2013). [20] 〈http://links.uwaterloo.ca/Repository.html〉 (last accessed April 2013). [21] 〈http://www.imagefusion.org〉 (last accessed April 2013). [22] 〈http://www.mathworks.com/matlabcentral/fileexchange/40861〉 (last accessed April 2013). [23] Rockinger O., Image sequence fusion using a shift-invariant wavelet transform, in: Proceedings of IEEE International Conference on Image Processing, Santa Barbara, CA, 1997, pp. 288–291. [24] 〈http://www.metapix.de/toolbox.htm〉 (last accessed April 2013). [25] Dong-U Lee, Hyungjin Kim, Mohammad Rahimi Estrin, Energyefficient image compression for resource-constrained platforms, IEEE Transactions on Image Processing 18 (9) (2009) 2100–2113. [26] T. Lane, P. Gladstone, L. Ortiz, J. Boucher, L. Crocker, J. Minguillon, G. Phillips, D. Rossi, G. Weijers, The Independent JPEG Group's JPEG Software Release 6b [Online]. Available From: 〈http://www.ijg.org〉 (last accessed April 2013).