- Email: [email protected]
Multi-focus image fusion algorithm based on pulse coupled neural networks and modified decision map
Accepted Manuscript Title: Multi-focus Image Fusion Algorithm Based on Pulse Coupled Neural Networks and Modified Decision Map Authors: Chaoben Du, Shesheng Gao PII: DOI: Reference:
S0030-4026(17)31582-6 https://doi.org/10.1016/j.ijleo.2017.11.162 IJLEO 60076
To appear in: Received date: Accepted date:
17-11-2016 22-11-2017
Please cite this article as: Chaoben D, Gao S, Multi-focus Image Fusion Algorithm Based on Pulse Coupled Neural Networks and Modified Decision Map, Optik - International Journal for Light and Electron Optics (2010), https://doi.org/10.1016/j.ijleo.2017.11.162 This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Multi-focus Image Fusion Algorithm Based on Pulse Coupled Neural Networks and Modified Decision Map Chaoben Du , Shesheng Gao （School of Automation, Northwestern Polytechnical University, Xi’an 710129, China ）
Abstract: In this paper, we present a new approach for the multi-focus image fusion, which utilizes
IP T
Orientation Information(OI) Motivated Pulse Coupled Neural Networks(PCNN) and modified decision map. First, the source multi-focus images are fused using the Orientation Information Motivated Pulse Coupled Neural Networks (OI-PCNN), and the initial decision map is obtained. Second, after
SC R
analyzing the decision map, we modified the decision map by employing a mathematical morphology post-processing technique. Finally, based on the modified decision map, the final fused image is
obtained by selecting the pixels in the focus areas. Objective performance evaluation criteria and visual observation demonstrate that the proposed method is better than various existing spatial domain and transform domain fusion methods, including NSCT, PCNN-NSCT, SF-PCNN-NSCT, and
U
EOE-PCNN-NSCT method.
Key words: Decision Map; PCNN; Image fusion; Orientation Information
N
1 Introduction
Image Fusion is a rapidly developing research area widely used in smart transportation, remote
A
sensing, computer vision, biomedical imaging and so on[1-2]. The purpose of image fusion is to
M
combine different images from several sensors to create a new image ,which will be more comprehensive and accurate. Furthermore, suitable for other image processing tasks or a human operator[3] .
ED
It is often impossible to acquire an image that contains all relevant focused objects in applications of optical microscopes, digital cameras or other equipment, because of the shortage of optical lens [4]. Therefore, some objects in focus are clear, but other objects, which at different distances from the
PT
imaging equipment will be out of focus ,are blurred [5]. However, humans prefer to obtain a clear image of all targets in reality. A possible way to overcome this shortage is to employ multi-focus image fusion methods, in which we can get one image contain all of the objects in focus by means of it
CC E
containing the best information from multiple original images [6]. Image fusion methods usually include transform domain and spatial domain fusion techniques [7].
Image fusion techniques in the transform domain based image fusion methods, each source image is firstly decomposed into a sequence of images through a particular mathematical transformation. Then, the fused coefficients are obtained through some fusion rules for combination. Finally, the fusion
A
image is obtained by means of a mathematical inverse transform. Thus, the transform domain fusion methods are also known as Multi-scale fusion methods. For spatial domain are usually directly on pixel gray level or color space from the source multi-focus images for fusion operation. Therefore, the spatial domain fusion method is also used in the single scale fusion method [4]. The Multi-scale fusion methods performs very well with both clean and noisy images. But this scheme is complicated and time-consuming. The simplest spatial-based method is to take the average of the input images pixel by pixel. Pulse Coupled Neural Networks (PCNN) was developed by Eckhorn et al in 1990 ,is one of the spatial-based fusion methods, which based on the experimental observations
of synchronous pulse bursts in cat and monkey visual cortex [8].It owns some excellent characters such as pulse synchronization of neurons and the global coupling ,and has been proven suitable for image processing. Currently, PCNN has widely used in image fusion [9-10]. In spite of many multi-focus image fusion methods based on PCNN or NSCT transform. Chun hui Zhao use redundant-lifting NSWMDA and adaptive PCNN fuse Image [10]. D. Agrawal have used spatial frequency-energy of Laplacian [11] or Y. Chai have used the improved sum-modified Laplacian (SML) [12] to motivate PCNN. But it still exist problems which cannot meet our requirements. As a matter of fact, humans are often sensitive to d directional features and edges information[13]. So another important spatial-based images fusion
IP T
method called the focused region-based method can detect the clear regions of input images, and then
directly transferred the pixels from clear regions into the fused image [14-16].However, these methods may generate discontinuous phenomena and artificial information at the boundaries of focused
SC R
regions ,because the boundary cannot be determined accurately[17].
Based on the above analysis, an image fusion method Based On Orientation Information(OI) Motivated Pulse Coupled Neural Networks(PCNN) and modified decision map is proposed in this paper. First, the source multi-focus images are fused using the Orientation Information Motivated Pulse Coupled Neural Networks (OI-PCNN), and the initial decision map is obtained. Second, after
U
analyzing the decision map, we modified the decision map by employing a mathematical morphology post-processing technique. Finally, based on the modified decision map, the final fused image is
N
obtained by selecting the pixels in the focus areas. Objective performance evaluation criteria and visual observation demonstrate that the proposed method is better than various existing spatial domain and
A
transform domain fusion methods.
M
The rest of the paper is organized as follows. The related theory of the Orientation Information is introduced in Section 2. The basic principle and improved model of PCNN is described in Section 3. The modified decision map and the proposed multi-focus image fusion framework are described in
ED
Section 4. Experimental results and analysis are given in Section 5, and the concluding remarks are described in Section 6.
2 Orientation Information
PT
2.1 The Classic Orientation Information
The classic orientation information measure of the image by was proposed by Wang et al in 2003 is shown in (1) [18]. It is an effective way to show the piecewise smoothness property of the images[19]. k
CC E
For a given image x , we use x ij to denote the pixels in the k - t h block centred at ( i , j ) pixel. The orientation information M
A
M
k
ij
d
m ax
d
m in
where d d
m ax
m in
fA L
m ax (d )
0 1 8 0
m in ( d )
0 1 8 0
( i , j ) A L
x (i, j )
k ij
in the block is defined as:
(1)
fA R
x (i, j )
( i , j ) A R
d f A f A L
R
A R and A L denotes the right and left region in a given block as shown in Figure 1. The extent of the
given block is defined as ( 2 w x 1) ( 2 w y 1) . In Figure 1, l is the line through the pixel
IP T
( i , j ) . Angle direction could be calculated in spatial domain and all the image pixels will be
detected ,through orientation information measure. In order to reduce the time-consuming and complexity, images in spatial domain are decomposed into blocks firstly, then orientation information
SC R
is measured in each block in this paper. The orientation information is considered as the feature in each
M
A
N
U
block[19].
Fig.1 Orientation measurement.
From Eq.(l),we can conclude that[18]:
ED
1) If pixel ( i , j ) is in the edge region of image, d gets the maximum value when L is in the
PT
direction of edge, and d gets the minimum value when L is in the direction orthogonal to the local edge, because the edge points are directional. Because there are sharp intensity transitions between two
CC E
sides of the edge, so M ( i , j ) is larger than pixels in other regions. Experiments have shown that M ( i , j ) is not sensitive to the changes of the neighborhood window. Within a certain range,
A
M ( i , j ) of the edge points is relatively large, regardless of the large or small size.
2) If pixel ( i , j ) is in the smooth region of image, d
is relatively small no matter which
direction L is in. Because pixel values in the smooth region are approximately the same, M ( i , j ) is relatively small. It is obviously that M ( i , j ) is also not sensitive to the analysis window size to some extent. Similarly, the value of M ( i , j ) in the smoothing point is not sensitive to the change of
the neighborhood size. In a certain range, the value of M ( i , j ) in the smoothing point is relatively small, regardless of the size of the observation scale or small. 3) If pixel ( i , j ) is in the texture region, when the observation scale is small, the neighborhood is not enough to reflect the texture on the gray change regularity, texture is more marginal, so M ( i , j ) is larger. When the scale is larger, that is to say, the current neighborhood can contain enough texture, is quite similar no matter which
IP T
texture on both sides of the structure are relatively similar, so d
direction L is in, M ( i , j ) is smaller because regular patterns are dominant. That is because in this
SC R
case, regular patterns are not stronger than edges. It is clear that M ( i , j ) is sensitive to the change of the neighborhood size. The larger the analysis window scale is, the smaller M ( i , j ) is.
U
4)Values in the same edge of M ( i , j ) is higher than that of other points of M ( i , j ) value, and information measure is close to the value, the characteristics of effectively reflects the edge continuity
N
and structural, in image restoration, of edge points and vertical edges and smooth areas point with
A
different strength recovery provides convenient conditions.
,
has the same
M
5) From a statistical point of view, noise in the two sides of line in the image, L
distribution and is nondirectional no matter which direction L is in, therefore we can make the
ED
conclusion that the value of M ( i , j ) is not sensitive to noise. From the above l), 2), 3) , 4)and 5),we can clearly see that the orientation information measure of
PT
every pixel in the image, is a measure of degree to which the pixel is in the edge regions. 2.2 Improved Orientation Information When higher levels of image noise ,the original orientation information measured can not carry out
CC E
effective edge detection, When the noise level is low, the orientation information measured of the noise is small; when the noise level is high, the orientation information measured of the noise is large. But at this time, the difference of the gray values d in all directions is relatively small . Similarly, the image
A
edge and texture area, flat gray difference of the properties of the d can also and so on .Due to the noise in the image edge region and flat region is randomly distributed, at this time, no matter how the scale changes, the orientation information of the noise point is relatively small. To overcome the defect mentioned above, inspired by the difference of the gray values d in all directions ,on the basis of each pixel in a four direction information measure and combining the concept of Shock Filter[20], a new orientation information measured rule , average value of the sum of the difference of each direction of the pixel, is designed in this article.
E (i, j )
d K , ( i 1, 2 ,
, M ; j 1, 2 ,
（2）
,N)
where， K is the number of the orientation information measure window， M 2 w x 1 ， N 2wy 1
The next step for the normalization of E ( i , j ) ， E ( i , j ) projection to the interval [ 0 , 1] ,so
m ax
1 i M ;1 i N
m in
1 i M ;1 i N
E (i, j )
E (i, j )
（3）
m in
1 i M ;1 i N
IP T
E (i, j ) E (i, j )
E (i, j )
So improvement orientation information measure is as follows： M ( i , j ) E ( i , j ) M ( i , j )
SC R
（4）
Equation (3) in the normalization process will weaken the image edge detection，In this paper, the Shock Filter is used to enhance the edge detection, improve the detection efficiency and accuracy of the
where， E M
(u )
1 u
0
k
s ig n ( ( E M
k
))( k 0 ,1,
M ( i , j ) ，
( u x u xx 2 u x u y u xy u y u yy ) 2
2
(5)
ED
1, u 0 s ig n ( u ) 0 , u 0 1, u 0
K)
N
EM
A
k
M
EM
U
image edge, and then get the orientation information measure：
PT
3.The Basic Principle and Improved model of PCNN In this section, a brief review of the original PCNN is given, and then an improved PCNN model is introduced. Firstly, we describe the model and principle of the original PCNN, which is based on the
CC E
experimental observations of synchronous pulse bursts in cat and monkey visual cortices and has a strong biological background. Then, an improved PCNN model , adaptive PCNN model is proposed, which not only possesses the properties of simplified PCNN, but also chooses the linking strength of each neuron adaptively[21].
A
3.1 The Basic PCNN model
PCNN is a feedback network and each PCNN neuron consists of three parts: the receptive field,
modulation field and pulse generator [22]. The basic PCNN neuron's structure is shown in Fig. 2. In the basic PCNN model, the neuron receives input signals from feeding and linking inputs by the receptive field. Then, input signals are divided into two channels. One channel is feeding input F i j and the other is linking input L i j . S i j is the input motivation such as the normalized gray level of image pixels in ( i , j ) position or the orientation information of the original image in this paper. U ij is the
internal activity of neuron, and i j is the dynamic threshold. Y ij is the pulse output of neuron and it gets either the binary value 0 or 1 . The inter-connections M and W are the synaptic gain strengths for the feeding and the linking inputs, which are dependent on the distance between neurons. L , F and are the attenuation time constants of L i j , F i j , and i j , respectively. i j is the
linking strength. V F , V L , V
denote the inherent voltage potentials of F i j , L i j , and i j ,
respectively. The input motivation (orientation information of the original image) is received by the
M
A
N
U
SC R
IP T
feeding element, the internal activation element combines the feeding element with the linking element.
Fig. 2. The basic model of the single PCNN neuron.
The value of the internal activation element is compared with a dynamic threshold that gradually
ED
decreases at iteration. The internal activation element accumulates the signals until it surpasses the dynamic threshold and then fires the output element and the dynamic threshold increases simultaneously strongly. The output of the neuron is then iteratively fed back to the element with a [23,24]:
PT
delay of one iteration. The basic PCNN model is described as iteration by the following equations
（6）
A
CC E
F i , j V F M ij , a b Y a b [ n 1] e x p ( F ) F i , j [ n 1] S ij ab L [n] V W ij , a b Y a b [ n 1] e x p ( L ) L i , j [ n 1] i, j L ab U i , j [ n ] F i , j [ n ] (1 L i , j [ n ] ) i , j [ n ] V Y i , j [ n 1] e x p ( ) i , j [ n 1] Y i , j [ n ] s te p (U i , j [ n ] i , j [ n ] )
In equations (6), the indexes i and j refer to the pixel location in the image, n denotes the current iteration (discrete time step), where n varies from 1 to N 1 ( N 1 is the total number of iterations). Obviously, the neural network constructed by the neurons shown in Fig. 2 has some factors of non-linear, leak integrate, and too much uncertain parameters, etc. All these uncertain factors lead to
the difficulty of the mathematical analysis to the network. In the PCNN-based image fusion, we use a simplified model as shown in Fig.3, which assumes that the feeding part F i j only accepts the external stimulus signal S i j . The mathematical model of the simplified PCNN can be described as follows: F i , j S ij k k k k L i , j [ n ] V L W ij , a b Y a b [ n 1] e x p ( L ) L i , j [ n 1] ab k k k k U i , j [ n ] F i , j [ n ] (1 i , j L i , j [ n ] ) k k k i , j [ n ] V Y i , j [ n 1] e x p ( ) i , j [ n 1] k k k Y i , j [ n ] s te p (U i , j [ n ] i , j [ n ] ) k
IP T
(7)
A
N
U
SC R
k
M
Fig. 3. Simplified model of PCNN neuron
where k denotes the pixels in the k th block of the fused image and two source images respectively. .
ED
In image processing, Pulse coupled neural networks (PCNN) is a single layer pulse coupled neural cells with a two dimensional connection as shown in Fig. 4 [25,26]. The number of neurons in the network is equal to the number of pixels in the input image. There exists a one-to-one correspondence
PT
between the input image pixels and network neurons. Every neuron is connected with neighboring neurons in linking range. The output of each neuron results in two states, namely firing( 1 state) and non-firing ( 0 state). Pulse output will be delivered to adjacent neurons. If adjacent neurons have a
CC E
similar intensity with the current neuron N ij , they will fire because of pulse coupled action. In this case, we recall that neuron N ij captures the adjacent neurons. Finally, the neuron N ij and the similar adjacent neurons will emit synchronous pulses. So, PCNN possesses the global coupling and
A
pulse synchronization characteristics. These characteristics benefit image fusion which makes use of local image information[21].
IP T SC R
Fig. 4. Connection model of PCNN neuron.
3.2. Adaptive PCNN In the basic PCNN model shown in (7), k ij
ij
reflects the pixel characteristics and the values vary
1 . It plays an important role in the proposed image fusion
U
between all pixels; usually 0
k
method in this paper, because it can vary the weighting of the linking channel in the internal activity.
k ij
should be increased to stress its importance. However, in
A
channel, its corresponding value of
N
Its value is usually dependent on different demands. For example, if much influence from the linking
linking strength
k ij
M
almost all the literature about image processing with PCNN, people often assign the same value to the of each neuron. It is chosen with experiments or experiences. This drawback is
ED
a big limit to the automatic process and the generalization for PCNN. In human vision, the responses to a region with prominent features are stronger than to a region with less prominent features. Therefore, the linking strength of each neuron in the PCNN should be related to the features of the
PT
corresponding pixels of the images. This is consistent with the fact that it is impossible for real neurons to have the same linking strength[21]. It is generally known that the clarity of each pixel is an observable characteristics of the edges of the
CC E
images, which contains a lot of orientation information. So, we can select the distinct of each pixel as the linking strength of the corresponding channel. Recall that the gradient of each pixel has relationship with the local neighborhood region and it is a distinguishing feature of the local region. In addition, the gradient reflects the local feature of the corresponding region. It can properly reflect the clarity of pixel to some extent. In either the objects in focus or the objects out of focus in a multi-focus image, the
A
gradient has an observable difference. So we can employ the gradient as a measure of each pixel clarity in this paper. Considering the diagonal information of each pixel in this paper, a new gradient measurement is proposed and used to determine the clarity. If we use Q ( i , j ) to denote the value of gradient measurement at the position ( i , j ) then it can be computed as follows: Q (i, j )
Q1 (i, j ) Q 2 (i, j )
(8)
Where Q 1 ( i , j ) I ( i 1, j ) I ( i , j )
2
I ( i , j 1) I ( i , j )
2
f 1 ( i , j ) d I ( i 1, j 1) I ( i , j )
2
f 2 ( i , j ) d I ( i 1, j 1) I ( i , j )
2
1
Where I ( i , j ) is the value at ( i , j ) ; d
is a distance weight.
2
IP T
Q 2 ( i , j ) f1 ( i , j ) f 2 ( i , j )
SC R
However, use of uniquely near pixel-level indicators, such as the image gradient, alone can make
decisions vulnerable to wide fluctuations dependent on sensor, optics and scene specific parameters. Furthermore, most image fusion methods using auto focus technology are global or semi-global in scale. So, corroboration from neighboring pixels of decision choices becomes necessary to maintain robustness of the method in the face of the above adverse effects. Adding this corroboration while
U
keeping pixel-level decisions requires summing the Q ( i , j ) over a ( 2 M 1) ( 2 N 1) region
N
surrounding each decision-point. This will produce a new definition of measurement function as
N
Q (i m , j n )
mM n N
and N
determine the window with size ( 2 M 1) ( 2 N 1) are used
ED
where the parameters M
(9)
M
M
C L (i, j )
A
follows:
to compute the measurement.
The use of such aggregation consequently improve the accuracy of decision making by ensuring that
PT
pixels with large clarity measures influence the decision of their neighbors. Finally, a sigmoid function is applied to these filtered clarity measures to transform them into a semi-hard decision map. Now, we
CC E
can give the formula of the linking strength
i, j
1
k
1 e
k ij
in the follows:
（10）
L
C L ( i , j )
From the above formulas we can see clearly that the better the clarity of the pixel, the larger the
A
value of i , j , and therefore the greater the linking strength of the corresponding neuron. Then the k
corresponding neuron is captured earlier than the corresponding neuron of the other input image, and vice versa. Obviously, i , j reflects the part characteristic of the pixel and the value is always k
different from others. It is impossible for real neurons to have the same linking strength and that the weight value of each neuron of the first and second generation of neuron network is different. Therefore, it is of practical significance.
4.Multi-focus Image Fusion Framework and Modified Decision Map 4.1 Steps of Image Fusion Based on PCNN Step1：Decompose the source images into blocks [ 3 3 ] and measure the improved orientation information of input image in each block according to Eq.(5) . Step2：Improved orientation information of each source image is input to PCNN and pulse of k
neurons generated according to (7). Then sum of firing times T i , j [ n ] is calculated as follows : k i, j
[n] T
k i, j
[ n 1] Y
k i, j
(11)
[n]
IP T
T
Step3：If n N 1 , then iteration stops. Get the decision map D i , j based on (12), which is the k
fusion rule proposed in this paper. That is block of source images with larger orientation information is
D i, j k
k
1 , if :T1 , ij ( n ) T 2 , ij ( n ) k
k
0 , if :T1 , ij ( n ) T 2 , ij ( n )
（12）
The traditional PCNN image fusion method, the next step should be according to the
U
Step4:
k
SC R
employed as the blocks of the fused images.
k
.But，in this paper,
, which is used for achieving the modified decision map
A
k
we analysis the initial decision map D i , j
k
x F , ij
N
filtered decision map D i , j by using (12) and (13) reconstruct the fused image
M
obtained by employing a mathematical morphology post-processing technique. Finally, based on the modified decision map, the final fused image is obtained by selecting the pixels in the focus areas. k
k
x1 , ij , if : D ij 1 k
ED
x F , ij k
k
（13）
x 2 , ij , if : D ij 0
k i, j
k
[ n ] 1 .also called one firing times. In fact, sum of
CC E
pulse Y
PT
where n denotes the iteration times. If U i , j is larger than
k i, j
Y
,then the neuron will generate a k i, j
[ n ] in n iteration is often
defined as (11) and employed to represent image information. One often analyse T rather than Y
k
i, j
k i, j
[ n ] instead,
k k k [ n ] . x F , ij , x 1 , ij and x 2 , ij denote the pixels in the k th block of the fused image
A
and two source images respectively. 4.2 Modified Decision Map In this section, some of the motivating factors have been discussed in the design of the proposed
approach to multi-focus image fusion. The framework of the proposed image fusion algorithm is depicted in Fig. 5.
Initial decision map Modified decision map
Final fused image
PCNN
Fig. 5. Schematic diagram of the proposed image fusion algorithm. k
IP T
Morphology processing
The binary decision map D i , j can reflect but not fully reflect the focused characteristics of the
SC R
source image, so it is called ‘the initial decision map’. However, the complexity of the image detail and
the shortcomings of the initial PCNN fusion method would cause some pixel in focus areas are not been detected. Thus, there are small holes, small black spots and narrow fault zones, etc, in the initial k
U
decision map D i , j . According to the theory of imaging, the areas, either in focus or out of focus, are always continuous [27]. With the result that, the defects of the initial decision map mentioned above
N
should be removed by a good fusion method. To remove these defects, mathematical morphology
A
methods are used in the following steps:
k
equation:
(14)
ED
e ( i , j ) im fill ( d , ' h o le s ') ( i , j )
M
Step 1: Use the filter im fill to remove the small black holes in D i , j , as shown in the following
k
To remove the small white holes in D i , j , equation (15) is firstly used to reverse the result of the
PT
above equation, and then the filter im fill is used to remove the small black holes. Finally, equation (17) is used to reverse the result of the previous step.
(15)
E E ( i , j ) im fill ( e e , ' h o le s ') ( i , j )
(16)
E (i, j ) 1 E E (i, j )
(17)
A
CC E
e e (i, j ) 1 e (i, j )
Step 2: To join the narrow breaks and fill the long thin gulfs in the result of the previous step,
E ( i , j ) , the morphological closing is employed. The result can be denoted as C ( i , j ) .
Step 3: Use the filter b w a r e a o p e n to remove the black area that cannot be removed by step 1, as shown in the follow: z ( i , j ) b w a r e a o p e n ( C ,T H ) ( i , j )
(18)
where T H is a threshold, which is set to remove the holes when they are smaller than the threshold T H . The threshold value depends on the image size. Step 4, similar to Step 1, employ the following equations to remove the white area that cannot be removed by Step 1 zz (i, j ) 1 z (i, j ) Z ( i , j ) b w a r e a o p e n ( Z Z ,T H ) ( i , j )
IP T
Z (i, j ) 1 Z Z (i, j )
Finally, we employed i m o p e n and im c lo s e remove separate black parts in the white area that
SC R
cannot be removed by Step 4 as follow: s e = s tr e l( 's q u a r e ',p a r a m e te r s )
F Z ( i,j) = im o p e n ( Z Z ,s e )
U
F Z Z ( i,j) = im c lo s e ( F Z ,s e )
N
Where, p a ra m e te rs generally control the size of SE which value depends on the image size.
A
F Z Z ( i ,j ) is a binary decision map, so it is called ‘the modified decision map’.
M
5. The Experimental Results and Analysis
In this section, we introduce the evaluation index system and analysis the experimental results as follows.
ED
5.1 Evaluation Index System
An excellent image fusion scheme is required to meet the following conditions: (1) it should be able to extract the complementarily information from two source images; (2) it should be robust and reliable;
PT
(3) it must not introduce inconsistencies or artifacts according to HVS[17]. To sum up, the measure of image fusion quality contains two categories: subjective evaluations and objective evaluations [28]. Subjective evaluations depend largely on the human visual distinguishing feature and professional
CC E
knowledge of the observer, so it is time-consuming and have poor repeatability. Objective evaluations are easily performed by computers, completely automatically, which can generally evaluate the similarity between the fused image and source images [29]. Therefore, besides visual observation, in this paper two objective criteria are used to evaluate the proposed fusion scheme. 1) Mutual Information: Mutual information (MI) [30] between fusion image F and the source images
A
A and B is defined as: MI MI
AF
MI
BF
in which L
MI
AF
L
p
AF
( a , f ) lo g 2 (
f 0 a0
L
MI
BF
L
f 0 b0
p
BF
( b , f ) lo g 2 (
p
AF
A
(a , f ) F
)
p (a ) p ( f ) p B
BF
(a, f ) F
p (a ) p ( f )
)
in which M I
AF
and M I
images; a , b
BF
denote the normalized M I between the fused image and the source
f [ 0 , L ] . p ( a ) , p ( a ) and A
and
B
AF
histograms of source images and the fused image. p
F
p ( f ) are the normalized gray level
( a , f ) and p
BF
( a , f ) are the joint gray level
histograms between the fused image and the source images A and B . M I can indicate how much information the fused image F conveys about the source images A and B . Therefore,the greater the value of M I , the better the fusion effect. 2) Edge Based Similarity Measure: The edge-based similarity measure (QAB/F) which proposed by
IP T
Xydeas and Petrovic [44] gives the similarity between the edges transferred from the input images to the fused image. Mathematically, the definition is given as 0
N0
[Q Q
AB F
i 1
AF
(i, j ) w (i, j ) Q A
BF
B
(i, j ) w (i, j )]
j 1 M
0
N0
[w i 1
SC R
M
A
(i, j ) w (i, j )] B
j 1
in which L
w (i, j )
si (i, j ) s j (i, j )
w (i, j )
si (i, j ) s j (i, j )
A
A
U
A
Q
AF
(i, j ) Q a
BF
(i , j ) Q a (i , j )Q g (i , j )
AF
A
B
AF
(i, j )Q g (i, j )
BF
M
Q
B
N
L B
BF
A
B
ED
M 0 N 0 is the size of the image, w ( i , j ) and w ( i , j ) are the gradient strengths for the source
PT
images A and B . s i ( i , j ) and s j ( i , j ) denote the horizontal and vertical Sobel edge detectors for each source image. Q xF
AF
( i , j ) and Q
BF
( i , j ) are the edge preservation values for each source
xF
CC E
image. Q a ( i , j ) and Q g ( i , j ) are the edge strength and orientation preservation values at location ( i , j ) for the source images A and B . Q
AB F
can indicate how much strength and orientation
A
information is captured at each pixel in image A , B and F value, the better the fusion result.
by a Sobel detector. So, the larger the
5.2 Experimental Results In order to show the advantages of the new image fusion method, using clock , pepsi and book
selected as test images to verify the algorithm proposed in this paper.Fig.5 (a) are (b) are source Multi-focus images with size 512×512. In the experiments, parameters of PCNN set as p q 3 3
, L
0 .0 6 9 3 1
,
1
W [ 0 .7 0 7 , 1, 0 .7 0 7 ;1, 0 , 1; 0 .7 0 7 , 1, 0 .7 0 7 ]
,
0 .2
，
V 2 0
,
VL 1
, 0 .5 ，
Table.1 Fusion results of the four methods in MI and QAB/F Image
criteria
pepsi
MI
7.2363
QAB/F
PCNN-NSCT
SF-NSCT-PCNN
EOE-NSCT-PCNN
Proposed method
6.9608
7.4827
8.3949
8.9154
0.7107
0.7745
0.7794
0.7241
0.7933
MI
7.2929
7.4640
7.9837
8.2731
9.3919
QAB/F
0.7026
0.6983
0.7064
0.7865
0.7975
(b) Source B
(c) Initial map
(e) NSCT
(f) PCNN-NSCT
CC E
PT
(d) Modified decision map
ED
M
A
N
(a) Source A
U
SC R
IP T
book
NSCT
A
(j) SF-NSCT-PCNN
(h) EOE-NSCT-PCNN
(i) proposed
(k) PCNN-NSCT-
(m) EOE-NSCT-PCNN
(l) SF-NSCT-PCNN
(n) proposed
IP T
(j) NSCT -
Fig. 6. The experimental data and results of fusing images ‘Pepsi’: (a)–(b) The source images; (c)–(d) The process of the focus area detection;(e)–(i) The fusion results of
A
N
U
SC R
different fusion algorithms; (j)–(n) The difference images between fused images and Source A. the difference image between fused images and Source A
(b) Source B
(c) Initial map
PT
ED
M
(a) Source A
(e) NSCT
(f) PCNN-NSCT
A
CC E
(d) Modified decision map
(j) SF-NSCT-PCNN
(h) EOE-NSCT-PCNN
(i) proposed
(k) PCNN-NSCT-
(l) SF-NSCT-PCNN
(n) proposed -
U
(m) EOE-NSCT-PCNN
SC R
IP T
(j) NSCT -
Fig. 7. The experimental data and results of fusing images ‘book’: (a)–(b) The source images; (c)–(d) The process of the focus area detection;(e)–(i) The fusion results of different
N
fusion algorithms; (j)–(n) The difference images between fused images and Source A. the difference image between fused images and Source A
Comparison of NSCT [29] , PCNN-NSCT [31],SF-PCNN-NSCT[26] , EOE-PCNN-NSCT[33] and
A
the proposed method in terms of both subjective and objective evaluations. Mutual information (MI)
M
and QAB/F are employed as information-based objective criteria are used to compare the fusion results. The values of MI and QAB/F of the two sets of images are listed in Table 1. From Table 1, we can see that the values of MI and QAB/F of the proposed method are all most the largest and optimal in the five
ED
fusion methods.
As shown in Fig. 6(c), the initial map has many gaps and holes in both the black area and the white area. According to the multi-focus image-forming principle, this binary image cannot perfectly show
PT
the focus characteristics of the source images. Morphology processing is used to remove the gaps and holes. The obtained decision map successfully fills the gaps and holes and reflects the focus characteristics of the source image more accurately. The white area of decision map represents the
CC E
focus area of Source A, whereas the black area represents the focus area of Source B. Focusing on Figs. 6 (e) and (i), one can obviously find that the fused image of the proposed method is clearer and more natural than the other fused results especially in the region of upper right corner. As illustrated in the difference images from Figs. 6 (j) and (n), the proposed algorithm extracts almost all of the in-focus components of the source images and preserves the useful information better than the other
A
methods(highlighted by the black squares). To illustrate that the proposed method can be widely applied to multi-focus fusion, the second
experiments are performed on the other multi-focus images “book”. From Fig. 7, a similar conclusion to those of the previous two experiments is obtained, namely that the proposed algorithm is better than the other methods. 6 Conclusion and discussion A new approach for the multi-focus image fusion, which utilizes Orientation Information(OI)
Motivated Pulse Coupled Neural Networks(PCNN) and modified decision map. First, the source multi-focus images are fused using the Orientation Information Motivated Pulse Coupled Neural Networks (OI-PCNN), and the initial decision map is obtained. Second, after analyzing the decision map, we modified the decision map by employing a mathematical morphology post-processing technique. Finally, based on the modified decision map, the final fused image is obtained by selecting the pixels in the focus areas. Objective performance evaluation criteria and visual observation demonstrate that the proposed method is better than various existing spatial domain and transform domain fusion methods.
IP T
Acknowledgments
The work of this paper was supported by the National Natural Science Foundation of China (Project Nu mber: 61174193) and the Specialized Research Fund for the Doctoral Program of Higher Education (Project
SC R
Number: 20136102110036).
References
U
[1] B. Miles, I. B. Ayed, M. W. K. Law, G. Garvin, A. Fenster, and S. Li,“Spine image fusion via graph cuts,” IEEE Trans. Biomed. Eng. vol. 60,no. 7, pp. 1841–1850, Jul. 2013.
N
[2] J. Liang, Y. He, D. Liu, and X. Zeng, “Image fusion using higher order singular value decomposition,” IEEE Trans. Image Process., vol. 21,no. 5, pp. 2898–2909, May 2012.
A
[3] S. Li and B. Yang.“Hybrid multiresolution method for multisensor multimodal image fusion,” IEEE Sensors J. vol. 10, no. 9, pp. 1519–1526,Sep. 2010.
M
[4]Yong Yang, Song Tong, Shuying Huang, and Pan Lin. Multifocus Image Fusion Based on NSCT and Focused Area Detection J. IEEE SENSORS JOURNAL, VOL. 15, NO. 5,pp.2824-2838 MAY
ED
2015
[5]B.Yang and S. Li, “Multifocus image fusion and restoration with sparse representation,” IEEE Trans. Instrum. Meas., vol. 59, no. 4, pp. 884–892,Apr. 2010. [6]W. Wang and F. Chang, “A multi-focus image fusion method based on Laplacian pyramid,” J.
PT
Comput., vol. 6, no. 12, pp. 2559–2566, Dec. 2011. [7] N. Mitianoudis and T. Stathaki, “Optimal contrast correction for ICA-based fusion of multimodal images,” IEEE Sensors J., vol. 8, no. 12,pp. 2016–2026, Dec. 2008.
CC E
[8] Eckhorn R., Reitboeck H., Arndt M., et al, “Feature linking via synchronization among
distributed assemblies: Simulations of results from cat visual cortex,” Neural Comp., vol.2, no.3, pp.293-307, 1990.
[9] Amina Jameel, Abdul Ghafoor, Muhammad Mohsin Riaz “Wavelet and guided filter based
A
multi-focus fusion for noisy images”. Optik 126 (2015) 3920–3923 [10]Chunhui Zhao, Guofeng Shao, Lijuan Ma, Xi Zhang Image fusion algorithm based on
redundant-lifting NSWMDA and adaptive PCNN. Optik 125 (2014) 6247–6255 [11] D. Agrawal, J. Singhai, Multifocus image fusion using modified pulse coupledneural network for improved image quality, Image Process. IET 4 (6) (2010)443–451. [12] Y. Chai, H.F. Li, M.Y. Guo, Multifocus image fusion scheme based on featuresof multiscale products and PCNN in lifting stationary wavelet domain, Opt.Commun. 284 (5) (2010) 1146–1158. [13] Xiaobo Qu, Changwei Hu, Jingwen Yan, Image Fusion Algorithm Based On Orientation Information Motivated Pulse Coupled Neural Networks, The seventh World Congress on Intelligent
Control and Automation WCICA2008-Chongqing, China,25-27 June 2008, pp.2437-2441 [14] G. Piella, “A general framework for multiresolution image fusion: From pixels to regions,” Inf. Fusion, vol. 4, no. 4, pp. 259–280, Dec. 2003. [15] S. Li, X. Kang, J. Hu, and B. Yang, “Image matting for fusion of multi-focus images in dynamic scenes,” Inf. Fusion, vol. 14, no. 2, pp. 147–162, Apr. 2013. [16] Y. Liu, J. Jin, Q. Wang, Y. Shen, and X. Dong, “Region level basedmulti-focus image fusion using quaternion wavelet and normalized cut,”Signal Process., vol. 97, pp. 9–30, Apr. 2014. [17] Yong Yang, Song Tong, Shuying Huang, and Pan Lin. Multi-focus Image Fusion Based on NSCT and Focused Area Detection. IEEE SENSORS JOURNAL, VOL. 15, pp.2824-2838.NO. 5,
IP T
MAY 2015.
[18] Wang Xinnian, Liang Dequn, “Orientation-information measure-based image restoration”, 2003 International Conference on Machine Learning and Cybernetics, Vol.5, pp2925-2930,2003.
SC R
[19]Xiaobo Qu，Changwei Hu，Jingwen Yan. Image Fusion Algorithm Based On Orientation
Information Motivated Pulse Coupled Neural Networks. Proceedings of the 7th World Congress on Intelligent Control and Automation June 25 - 27, 2008, Chongqing, China
[20]JIANG Ling-ling , YIN Hai-qing , FENG Xiang-chu. An algorithm for image enhancement based on shock filters and anisotropic diffusion. JOURNAL
XIDIAN
UNIVERSITY. Jun.
U
2006 Vol,33 ,No. 3.476-479.
OF
[21]Y. Chai, H.F. Li, M.Y. Guo. Multifocus image fusion scheme based on features of multiscale
N
products and PCNN in lifting stationary wavelet domain. Optics Communications 284 (2011) 1146– 1158.
A
[22] J.L. Johnson, M.L. Padgett, PCNN models and applications. IEEE Transatcion on Neural
M
Networks 10 (3) (1999) 480.
[23] H.S. Ranganath, G. Kuntimad, J.L. Johonson, Perfect image segmentation using pulse coupled neural networks. Proc.of the Southeast Conference on‘Visualize the future’, 1995, p. 37.
ED
[24] L. Johnosn, H. Ranganath, G. Kuntimad, H.J. Caulfield, Object detection using pulse coupled neural networks. IEEE Transactions on Neural Networks 10 (3) (1999) 461. [25] R.P. Broussard, S.K. Rogers, M.E. Oxley, et al., Physiologically motivated image fusion for (1999) 554.
PT
object detection using a pulse coupled neural network. IEEE Transactions on Neural Networks 10 (3) [26] Xiaobo Qu, Jingwen YAN, Hongzhi XIAO, Ziqian ZHU, “Image Fusion Algorithm Based on
CC E
Spatial Frequency Motivated Pulse Coupled Neural Networks in Nonsubsampled Contourlet Transform Domain,”Acta Automatica Sinica, vol. 34, no.12, pp. 1508-1514, 2008 [27] S. Li and B. Yang, “Hybrid multiresolution method for multisensor multimodalimage fusion,”
IEEE Sensors J., vol. 10, no. 9, pp. 1519–1526, Sep. 2010. [28] G. Piella and H. Heijmans, “A new quality metric for image fusion,”in Proc. IEEE Int. Conf.
A
Image Process., Amsterdam, The Netherlands,Sep. 2003, pp. 173–176. [29] G. Bhatnagar, Q. M. J. Wu, and Z. Liu, “Directive contrast based multimodal medical image
fusion in NSCT domain,” IEEE Trans. Multimedia, vol. 15, no. 5, pp. 1014–1024, Aug. 2013. [30] B. Yang and S. Li, “Pixel-level image fusion with simultaneous orthogonal matching pursuit,” Inf. Fusion, vol. 13, no. 1, pp. 10–19, Jan. 2012. [31]Wang Rongfeng,“Image fusion algorithm based on nonsubsampled contourlet transform and pulse coupled neural netwrork”. Computer Applications, Vol.28 ,no.12,pp.164-167, 2008. [32] Jingjing Wang, Qian Li, Zhenhong Jia, Nikola Kasabov, Jie Yang. A novel multi-focus image
fusion method using PCNN innonsubsampled contourlet transform domain. Optik 126 (2015)
A
CC E
PT
ED
M
A
N
U
SC R
IP T
2508–2511
S0030-4026(17)31582-6 https://doi.org/10.1016/j.ijleo.2017.11.162 IJLEO 60076
To appear in: Received date: Accepted date:
17-11-2016 22-11-2017
Please cite this article as: Chaoben D, Gao S, Multi-focus Image Fusion Algorithm Based on Pulse Coupled Neural Networks and Modified Decision Map, Optik - International Journal for Light and Electron Optics (2010), https://doi.org/10.1016/j.ijleo.2017.11.162 This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Multi-focus Image Fusion Algorithm Based on Pulse Coupled Neural Networks and Modified Decision Map Chaoben Du , Shesheng Gao （School of Automation, Northwestern Polytechnical University, Xi’an 710129, China ）
Abstract: In this paper, we present a new approach for the multi-focus image fusion, which utilizes
IP T
Orientation Information(OI) Motivated Pulse Coupled Neural Networks(PCNN) and modified decision map. First, the source multi-focus images are fused using the Orientation Information Motivated Pulse Coupled Neural Networks (OI-PCNN), and the initial decision map is obtained. Second, after
SC R
analyzing the decision map, we modified the decision map by employing a mathematical morphology post-processing technique. Finally, based on the modified decision map, the final fused image is
obtained by selecting the pixels in the focus areas. Objective performance evaluation criteria and visual observation demonstrate that the proposed method is better than various existing spatial domain and transform domain fusion methods, including NSCT, PCNN-NSCT, SF-PCNN-NSCT, and
U
EOE-PCNN-NSCT method.
Key words: Decision Map; PCNN; Image fusion; Orientation Information
N
1 Introduction
Image Fusion is a rapidly developing research area widely used in smart transportation, remote
A
sensing, computer vision, biomedical imaging and so on[1-2]. The purpose of image fusion is to
M
combine different images from several sensors to create a new image ,which will be more comprehensive and accurate. Furthermore, suitable for other image processing tasks or a human operator[3] .
ED
It is often impossible to acquire an image that contains all relevant focused objects in applications of optical microscopes, digital cameras or other equipment, because of the shortage of optical lens [4]. Therefore, some objects in focus are clear, but other objects, which at different distances from the
PT
imaging equipment will be out of focus ,are blurred [5]. However, humans prefer to obtain a clear image of all targets in reality. A possible way to overcome this shortage is to employ multi-focus image fusion methods, in which we can get one image contain all of the objects in focus by means of it
CC E
containing the best information from multiple original images [6]. Image fusion methods usually include transform domain and spatial domain fusion techniques [7].
Image fusion techniques in the transform domain based image fusion methods, each source image is firstly decomposed into a sequence of images through a particular mathematical transformation. Then, the fused coefficients are obtained through some fusion rules for combination. Finally, the fusion
A
image is obtained by means of a mathematical inverse transform. Thus, the transform domain fusion methods are also known as Multi-scale fusion methods. For spatial domain are usually directly on pixel gray level or color space from the source multi-focus images for fusion operation. Therefore, the spatial domain fusion method is also used in the single scale fusion method [4]. The Multi-scale fusion methods performs very well with both clean and noisy images. But this scheme is complicated and time-consuming. The simplest spatial-based method is to take the average of the input images pixel by pixel. Pulse Coupled Neural Networks (PCNN) was developed by Eckhorn et al in 1990 ,is one of the spatial-based fusion methods, which based on the experimental observations
of synchronous pulse bursts in cat and monkey visual cortex [8].It owns some excellent characters such as pulse synchronization of neurons and the global coupling ,and has been proven suitable for image processing. Currently, PCNN has widely used in image fusion [9-10]. In spite of many multi-focus image fusion methods based on PCNN or NSCT transform. Chun hui Zhao use redundant-lifting NSWMDA and adaptive PCNN fuse Image [10]. D. Agrawal have used spatial frequency-energy of Laplacian [11] or Y. Chai have used the improved sum-modified Laplacian (SML) [12] to motivate PCNN. But it still exist problems which cannot meet our requirements. As a matter of fact, humans are often sensitive to d directional features and edges information[13]. So another important spatial-based images fusion
IP T
method called the focused region-based method can detect the clear regions of input images, and then
directly transferred the pixels from clear regions into the fused image [14-16].However, these methods may generate discontinuous phenomena and artificial information at the boundaries of focused
SC R
regions ,because the boundary cannot be determined accurately[17].
Based on the above analysis, an image fusion method Based On Orientation Information(OI) Motivated Pulse Coupled Neural Networks(PCNN) and modified decision map is proposed in this paper. First, the source multi-focus images are fused using the Orientation Information Motivated Pulse Coupled Neural Networks (OI-PCNN), and the initial decision map is obtained. Second, after
U
analyzing the decision map, we modified the decision map by employing a mathematical morphology post-processing technique. Finally, based on the modified decision map, the final fused image is
N
obtained by selecting the pixels in the focus areas. Objective performance evaluation criteria and visual observation demonstrate that the proposed method is better than various existing spatial domain and
A
transform domain fusion methods.
M
The rest of the paper is organized as follows. The related theory of the Orientation Information is introduced in Section 2. The basic principle and improved model of PCNN is described in Section 3. The modified decision map and the proposed multi-focus image fusion framework are described in
ED
Section 4. Experimental results and analysis are given in Section 5, and the concluding remarks are described in Section 6.
2 Orientation Information
PT
2.1 The Classic Orientation Information
The classic orientation information measure of the image by was proposed by Wang et al in 2003 is shown in (1) [18]. It is an effective way to show the piecewise smoothness property of the images[19]. k
CC E
For a given image x , we use x ij to denote the pixels in the k - t h block centred at ( i , j ) pixel. The orientation information M
A
M
k
ij
d
m ax
d
m in
where d d
m ax
m in
fA L
m ax (d )
0 1 8 0
m in ( d )
0 1 8 0
( i , j ) A L
x (i, j )
k ij
in the block is defined as:
(1)
fA R
x (i, j )
( i , j ) A R
d f A f A L
R
A R and A L denotes the right and left region in a given block as shown in Figure 1. The extent of the
given block is defined as ( 2 w x 1) ( 2 w y 1) . In Figure 1, l is the line through the pixel
IP T
( i , j ) . Angle direction could be calculated in spatial domain and all the image pixels will be
detected ,through orientation information measure. In order to reduce the time-consuming and complexity, images in spatial domain are decomposed into blocks firstly, then orientation information
SC R
is measured in each block in this paper. The orientation information is considered as the feature in each
M
A
N
U
block[19].
Fig.1 Orientation measurement.
From Eq.(l),we can conclude that[18]:
ED
1) If pixel ( i , j ) is in the edge region of image, d gets the maximum value when L is in the
PT
direction of edge, and d gets the minimum value when L is in the direction orthogonal to the local edge, because the edge points are directional. Because there are sharp intensity transitions between two
CC E
sides of the edge, so M ( i , j ) is larger than pixels in other regions. Experiments have shown that M ( i , j ) is not sensitive to the changes of the neighborhood window. Within a certain range,
A
M ( i , j ) of the edge points is relatively large, regardless of the large or small size.
2) If pixel ( i , j ) is in the smooth region of image, d
is relatively small no matter which
direction L is in. Because pixel values in the smooth region are approximately the same, M ( i , j ) is relatively small. It is obviously that M ( i , j ) is also not sensitive to the analysis window size to some extent. Similarly, the value of M ( i , j ) in the smoothing point is not sensitive to the change of
the neighborhood size. In a certain range, the value of M ( i , j ) in the smoothing point is relatively small, regardless of the size of the observation scale or small. 3) If pixel ( i , j ) is in the texture region, when the observation scale is small, the neighborhood is not enough to reflect the texture on the gray change regularity, texture is more marginal, so M ( i , j ) is larger. When the scale is larger, that is to say, the current neighborhood can contain enough texture, is quite similar no matter which
IP T
texture on both sides of the structure are relatively similar, so d
direction L is in, M ( i , j ) is smaller because regular patterns are dominant. That is because in this
SC R
case, regular patterns are not stronger than edges. It is clear that M ( i , j ) is sensitive to the change of the neighborhood size. The larger the analysis window scale is, the smaller M ( i , j ) is.
U
4)Values in the same edge of M ( i , j ) is higher than that of other points of M ( i , j ) value, and information measure is close to the value, the characteristics of effectively reflects the edge continuity
N
and structural, in image restoration, of edge points and vertical edges and smooth areas point with
A
different strength recovery provides convenient conditions.
,
has the same
M
5) From a statistical point of view, noise in the two sides of line in the image, L
distribution and is nondirectional no matter which direction L is in, therefore we can make the
ED
conclusion that the value of M ( i , j ) is not sensitive to noise. From the above l), 2), 3) , 4)and 5),we can clearly see that the orientation information measure of
PT
every pixel in the image, is a measure of degree to which the pixel is in the edge regions. 2.2 Improved Orientation Information When higher levels of image noise ,the original orientation information measured can not carry out
CC E
effective edge detection, When the noise level is low, the orientation information measured of the noise is small; when the noise level is high, the orientation information measured of the noise is large. But at this time, the difference of the gray values d in all directions is relatively small . Similarly, the image
A
edge and texture area, flat gray difference of the properties of the d can also and so on .Due to the noise in the image edge region and flat region is randomly distributed, at this time, no matter how the scale changes, the orientation information of the noise point is relatively small. To overcome the defect mentioned above, inspired by the difference of the gray values d in all directions ,on the basis of each pixel in a four direction information measure and combining the concept of Shock Filter[20], a new orientation information measured rule , average value of the sum of the difference of each direction of the pixel, is designed in this article.
E (i, j )
d K , ( i 1, 2 ,
, M ; j 1, 2 ,
（2）
,N)
where， K is the number of the orientation information measure window， M 2 w x 1 ， N 2wy 1
The next step for the normalization of E ( i , j ) ， E ( i , j ) projection to the interval [ 0 , 1] ,so
m ax
1 i M ;1 i N
m in
1 i M ;1 i N
E (i, j )
E (i, j )
（3）
m in
1 i M ;1 i N
IP T
E (i, j ) E (i, j )
E (i, j )
So improvement orientation information measure is as follows： M ( i , j ) E ( i , j ) M ( i , j )
SC R
（4）
Equation (3) in the normalization process will weaken the image edge detection，In this paper, the Shock Filter is used to enhance the edge detection, improve the detection efficiency and accuracy of the
where， E M
(u )
1 u
0
k
s ig n ( ( E M
k
))( k 0 ,1,
M ( i , j ) ，
( u x u xx 2 u x u y u xy u y u yy ) 2
2
(5)
ED
1, u 0 s ig n ( u ) 0 , u 0 1, u 0
K)
N
EM
A
k
M
EM
U
image edge, and then get the orientation information measure：
PT
3.The Basic Principle and Improved model of PCNN In this section, a brief review of the original PCNN is given, and then an improved PCNN model is introduced. Firstly, we describe the model and principle of the original PCNN, which is based on the
CC E
experimental observations of synchronous pulse bursts in cat and monkey visual cortices and has a strong biological background. Then, an improved PCNN model , adaptive PCNN model is proposed, which not only possesses the properties of simplified PCNN, but also chooses the linking strength of each neuron adaptively[21].
A
3.1 The Basic PCNN model
PCNN is a feedback network and each PCNN neuron consists of three parts: the receptive field,
modulation field and pulse generator [22]. The basic PCNN neuron's structure is shown in Fig. 2. In the basic PCNN model, the neuron receives input signals from feeding and linking inputs by the receptive field. Then, input signals are divided into two channels. One channel is feeding input F i j and the other is linking input L i j . S i j is the input motivation such as the normalized gray level of image pixels in ( i , j ) position or the orientation information of the original image in this paper. U ij is the
internal activity of neuron, and i j is the dynamic threshold. Y ij is the pulse output of neuron and it gets either the binary value 0 or 1 . The inter-connections M and W are the synaptic gain strengths for the feeding and the linking inputs, which are dependent on the distance between neurons. L , F and are the attenuation time constants of L i j , F i j , and i j , respectively. i j is the
linking strength. V F , V L , V
denote the inherent voltage potentials of F i j , L i j , and i j ,
respectively. The input motivation (orientation information of the original image) is received by the
M
A
N
U
SC R
IP T
feeding element, the internal activation element combines the feeding element with the linking element.
Fig. 2. The basic model of the single PCNN neuron.
The value of the internal activation element is compared with a dynamic threshold that gradually
ED
decreases at iteration. The internal activation element accumulates the signals until it surpasses the dynamic threshold and then fires the output element and the dynamic threshold increases simultaneously strongly. The output of the neuron is then iteratively fed back to the element with a [23,24]:
PT
delay of one iteration. The basic PCNN model is described as iteration by the following equations
（6）
A
CC E
F i , j V F M ij , a b Y a b [ n 1] e x p ( F ) F i , j [ n 1] S ij ab L [n] V W ij , a b Y a b [ n 1] e x p ( L ) L i , j [ n 1] i, j L ab U i , j [ n ] F i , j [ n ] (1 L i , j [ n ] ) i , j [ n ] V Y i , j [ n 1] e x p ( ) i , j [ n 1] Y i , j [ n ] s te p (U i , j [ n ] i , j [ n ] )
In equations (6), the indexes i and j refer to the pixel location in the image, n denotes the current iteration (discrete time step), where n varies from 1 to N 1 ( N 1 is the total number of iterations). Obviously, the neural network constructed by the neurons shown in Fig. 2 has some factors of non-linear, leak integrate, and too much uncertain parameters, etc. All these uncertain factors lead to
the difficulty of the mathematical analysis to the network. In the PCNN-based image fusion, we use a simplified model as shown in Fig.3, which assumes that the feeding part F i j only accepts the external stimulus signal S i j . The mathematical model of the simplified PCNN can be described as follows: F i , j S ij k k k k L i , j [ n ] V L W ij , a b Y a b [ n 1] e x p ( L ) L i , j [ n 1] ab k k k k U i , j [ n ] F i , j [ n ] (1 i , j L i , j [ n ] ) k k k i , j [ n ] V Y i , j [ n 1] e x p ( ) i , j [ n 1] k k k Y i , j [ n ] s te p (U i , j [ n ] i , j [ n ] ) k
IP T
(7)
A
N
U
SC R
k
M
Fig. 3. Simplified model of PCNN neuron
where k denotes the pixels in the k th block of the fused image and two source images respectively. .
ED
In image processing, Pulse coupled neural networks (PCNN) is a single layer pulse coupled neural cells with a two dimensional connection as shown in Fig. 4 [25,26]. The number of neurons in the network is equal to the number of pixels in the input image. There exists a one-to-one correspondence
PT
between the input image pixels and network neurons. Every neuron is connected with neighboring neurons in linking range. The output of each neuron results in two states, namely firing( 1 state) and non-firing ( 0 state). Pulse output will be delivered to adjacent neurons. If adjacent neurons have a
CC E
similar intensity with the current neuron N ij , they will fire because of pulse coupled action. In this case, we recall that neuron N ij captures the adjacent neurons. Finally, the neuron N ij and the similar adjacent neurons will emit synchronous pulses. So, PCNN possesses the global coupling and
A
pulse synchronization characteristics. These characteristics benefit image fusion which makes use of local image information[21].
IP T SC R
Fig. 4. Connection model of PCNN neuron.
3.2. Adaptive PCNN In the basic PCNN model shown in (7), k ij
ij
reflects the pixel characteristics and the values vary
1 . It plays an important role in the proposed image fusion
U
between all pixels; usually 0
k
method in this paper, because it can vary the weighting of the linking channel in the internal activity.
k ij
should be increased to stress its importance. However, in
A
channel, its corresponding value of
N
Its value is usually dependent on different demands. For example, if much influence from the linking
linking strength
k ij
M
almost all the literature about image processing with PCNN, people often assign the same value to the of each neuron. It is chosen with experiments or experiences. This drawback is
ED
a big limit to the automatic process and the generalization for PCNN. In human vision, the responses to a region with prominent features are stronger than to a region with less prominent features. Therefore, the linking strength of each neuron in the PCNN should be related to the features of the
PT
corresponding pixels of the images. This is consistent with the fact that it is impossible for real neurons to have the same linking strength[21]. It is generally known that the clarity of each pixel is an observable characteristics of the edges of the
CC E
images, which contains a lot of orientation information. So, we can select the distinct of each pixel as the linking strength of the corresponding channel. Recall that the gradient of each pixel has relationship with the local neighborhood region and it is a distinguishing feature of the local region. In addition, the gradient reflects the local feature of the corresponding region. It can properly reflect the clarity of pixel to some extent. In either the objects in focus or the objects out of focus in a multi-focus image, the
A
gradient has an observable difference. So we can employ the gradient as a measure of each pixel clarity in this paper. Considering the diagonal information of each pixel in this paper, a new gradient measurement is proposed and used to determine the clarity. If we use Q ( i , j ) to denote the value of gradient measurement at the position ( i , j ) then it can be computed as follows: Q (i, j )
Q1 (i, j ) Q 2 (i, j )
(8)
Where Q 1 ( i , j ) I ( i 1, j ) I ( i , j )
2
I ( i , j 1) I ( i , j )
2
f 1 ( i , j ) d I ( i 1, j 1) I ( i , j )
2
f 2 ( i , j ) d I ( i 1, j 1) I ( i , j )
2
1
Where I ( i , j ) is the value at ( i , j ) ; d
is a distance weight.
2
IP T
Q 2 ( i , j ) f1 ( i , j ) f 2 ( i , j )
SC R
However, use of uniquely near pixel-level indicators, such as the image gradient, alone can make
decisions vulnerable to wide fluctuations dependent on sensor, optics and scene specific parameters. Furthermore, most image fusion methods using auto focus technology are global or semi-global in scale. So, corroboration from neighboring pixels of decision choices becomes necessary to maintain robustness of the method in the face of the above adverse effects. Adding this corroboration while
U
keeping pixel-level decisions requires summing the Q ( i , j ) over a ( 2 M 1) ( 2 N 1) region
N
surrounding each decision-point. This will produce a new definition of measurement function as
N
Q (i m , j n )
mM n N
and N
determine the window with size ( 2 M 1) ( 2 N 1) are used
ED
where the parameters M
(9)
M
M
C L (i, j )
A
follows:
to compute the measurement.
The use of such aggregation consequently improve the accuracy of decision making by ensuring that
PT
pixels with large clarity measures influence the decision of their neighbors. Finally, a sigmoid function is applied to these filtered clarity measures to transform them into a semi-hard decision map. Now, we
CC E
can give the formula of the linking strength
i, j
1
k
1 e
k ij
in the follows:
（10）
L
C L ( i , j )
From the above formulas we can see clearly that the better the clarity of the pixel, the larger the
A
value of i , j , and therefore the greater the linking strength of the corresponding neuron. Then the k
corresponding neuron is captured earlier than the corresponding neuron of the other input image, and vice versa. Obviously, i , j reflects the part characteristic of the pixel and the value is always k
different from others. It is impossible for real neurons to have the same linking strength and that the weight value of each neuron of the first and second generation of neuron network is different. Therefore, it is of practical significance.
4.Multi-focus Image Fusion Framework and Modified Decision Map 4.1 Steps of Image Fusion Based on PCNN Step1：Decompose the source images into blocks [ 3 3 ] and measure the improved orientation information of input image in each block according to Eq.(5) . Step2：Improved orientation information of each source image is input to PCNN and pulse of k
neurons generated according to (7). Then sum of firing times T i , j [ n ] is calculated as follows : k i, j
[n] T
k i, j
[ n 1] Y
k i, j
(11)
[n]
IP T
T
Step3：If n N 1 , then iteration stops. Get the decision map D i , j based on (12), which is the k
fusion rule proposed in this paper. That is block of source images with larger orientation information is
D i, j k
k
1 , if :T1 , ij ( n ) T 2 , ij ( n ) k
k
0 , if :T1 , ij ( n ) T 2 , ij ( n )
（12）
The traditional PCNN image fusion method, the next step should be according to the
U
Step4:
k
SC R
employed as the blocks of the fused images.
k
.But，in this paper,
, which is used for achieving the modified decision map
A
k
we analysis the initial decision map D i , j
k
x F , ij
N
filtered decision map D i , j by using (12) and (13) reconstruct the fused image
M
obtained by employing a mathematical morphology post-processing technique. Finally, based on the modified decision map, the final fused image is obtained by selecting the pixels in the focus areas. k
k
x1 , ij , if : D ij 1 k
ED
x F , ij k
k
（13）
x 2 , ij , if : D ij 0
k i, j
k
[ n ] 1 .also called one firing times. In fact, sum of
CC E
pulse Y
PT
where n denotes the iteration times. If U i , j is larger than
k i, j
Y
,then the neuron will generate a k i, j
[ n ] in n iteration is often
defined as (11) and employed to represent image information. One often analyse T rather than Y
k
i, j
k i, j
[ n ] instead,
k k k [ n ] . x F , ij , x 1 , ij and x 2 , ij denote the pixels in the k th block of the fused image
A
and two source images respectively. 4.2 Modified Decision Map In this section, some of the motivating factors have been discussed in the design of the proposed
approach to multi-focus image fusion. The framework of the proposed image fusion algorithm is depicted in Fig. 5.
Initial decision map Modified decision map
Final fused image
PCNN
Fig. 5. Schematic diagram of the proposed image fusion algorithm. k
IP T
Morphology processing
The binary decision map D i , j can reflect but not fully reflect the focused characteristics of the
SC R
source image, so it is called ‘the initial decision map’. However, the complexity of the image detail and
the shortcomings of the initial PCNN fusion method would cause some pixel in focus areas are not been detected. Thus, there are small holes, small black spots and narrow fault zones, etc, in the initial k
U
decision map D i , j . According to the theory of imaging, the areas, either in focus or out of focus, are always continuous [27]. With the result that, the defects of the initial decision map mentioned above
N
should be removed by a good fusion method. To remove these defects, mathematical morphology
A
methods are used in the following steps:
k
equation:
(14)
ED
e ( i , j ) im fill ( d , ' h o le s ') ( i , j )
M
Step 1: Use the filter im fill to remove the small black holes in D i , j , as shown in the following
k
To remove the small white holes in D i , j , equation (15) is firstly used to reverse the result of the
PT
above equation, and then the filter im fill is used to remove the small black holes. Finally, equation (17) is used to reverse the result of the previous step.
(15)
E E ( i , j ) im fill ( e e , ' h o le s ') ( i , j )
(16)
E (i, j ) 1 E E (i, j )
(17)
A
CC E
e e (i, j ) 1 e (i, j )
Step 2: To join the narrow breaks and fill the long thin gulfs in the result of the previous step,
E ( i , j ) , the morphological closing is employed. The result can be denoted as C ( i , j ) .
Step 3: Use the filter b w a r e a o p e n to remove the black area that cannot be removed by step 1, as shown in the follow: z ( i , j ) b w a r e a o p e n ( C ,T H ) ( i , j )
(18)
where T H is a threshold, which is set to remove the holes when they are smaller than the threshold T H . The threshold value depends on the image size. Step 4, similar to Step 1, employ the following equations to remove the white area that cannot be removed by Step 1 zz (i, j ) 1 z (i, j ) Z ( i , j ) b w a r e a o p e n ( Z Z ,T H ) ( i , j )
IP T
Z (i, j ) 1 Z Z (i, j )
Finally, we employed i m o p e n and im c lo s e remove separate black parts in the white area that
SC R
cannot be removed by Step 4 as follow: s e = s tr e l( 's q u a r e ',p a r a m e te r s )
F Z ( i,j) = im o p e n ( Z Z ,s e )
U
F Z Z ( i,j) = im c lo s e ( F Z ,s e )
N
Where, p a ra m e te rs generally control the size of SE which value depends on the image size.
A
F Z Z ( i ,j ) is a binary decision map, so it is called ‘the modified decision map’.
M
5. The Experimental Results and Analysis
In this section, we introduce the evaluation index system and analysis the experimental results as follows.
ED
5.1 Evaluation Index System
An excellent image fusion scheme is required to meet the following conditions: (1) it should be able to extract the complementarily information from two source images; (2) it should be robust and reliable;
PT
(3) it must not introduce inconsistencies or artifacts according to HVS[17]. To sum up, the measure of image fusion quality contains two categories: subjective evaluations and objective evaluations [28]. Subjective evaluations depend largely on the human visual distinguishing feature and professional
CC E
knowledge of the observer, so it is time-consuming and have poor repeatability. Objective evaluations are easily performed by computers, completely automatically, which can generally evaluate the similarity between the fused image and source images [29]. Therefore, besides visual observation, in this paper two objective criteria are used to evaluate the proposed fusion scheme. 1) Mutual Information: Mutual information (MI) [30] between fusion image F and the source images
A
A and B is defined as: MI MI
AF
MI
BF
in which L
MI
AF
L
p
AF
( a , f ) lo g 2 (
f 0 a0
L
MI
BF
L
f 0 b0
p
BF
( b , f ) lo g 2 (
p
AF
A
(a , f ) F
)
p (a ) p ( f ) p B
BF
(a, f ) F
p (a ) p ( f )
)
in which M I
AF
and M I
images; a , b
BF
denote the normalized M I between the fused image and the source
f [ 0 , L ] . p ( a ) , p ( a ) and A
and
B
AF
histograms of source images and the fused image. p
F
p ( f ) are the normalized gray level
( a , f ) and p
BF
( a , f ) are the joint gray level
histograms between the fused image and the source images A and B . M I can indicate how much information the fused image F conveys about the source images A and B . Therefore,the greater the value of M I , the better the fusion effect. 2) Edge Based Similarity Measure: The edge-based similarity measure (QAB/F) which proposed by
IP T
Xydeas and Petrovic [44] gives the similarity between the edges transferred from the input images to the fused image. Mathematically, the definition is given as 0
N0
[Q Q
AB F
i 1
AF
(i, j ) w (i, j ) Q A
BF
B
(i, j ) w (i, j )]
j 1 M
0
N0
[w i 1
SC R
M
A
(i, j ) w (i, j )] B
j 1
in which L
w (i, j )
si (i, j ) s j (i, j )
w (i, j )
si (i, j ) s j (i, j )
A
A
U
A
Q
AF
(i, j ) Q a
BF
(i , j ) Q a (i , j )Q g (i , j )
AF
A
B
AF
(i, j )Q g (i, j )
BF
M
Q
B
N
L B
BF
A
B
ED
M 0 N 0 is the size of the image, w ( i , j ) and w ( i , j ) are the gradient strengths for the source
PT
images A and B . s i ( i , j ) and s j ( i , j ) denote the horizontal and vertical Sobel edge detectors for each source image. Q xF
AF
( i , j ) and Q
BF
( i , j ) are the edge preservation values for each source
xF
CC E
image. Q a ( i , j ) and Q g ( i , j ) are the edge strength and orientation preservation values at location ( i , j ) for the source images A and B . Q
AB F
can indicate how much strength and orientation
A
information is captured at each pixel in image A , B and F value, the better the fusion result.
by a Sobel detector. So, the larger the
5.2 Experimental Results In order to show the advantages of the new image fusion method, using clock , pepsi and book
selected as test images to verify the algorithm proposed in this paper.Fig.5 (a) are (b) are source Multi-focus images with size 512×512. In the experiments, parameters of PCNN set as p q 3 3
, L
0 .0 6 9 3 1
,
1
W [ 0 .7 0 7 , 1, 0 .7 0 7 ;1, 0 , 1; 0 .7 0 7 , 1, 0 .7 0 7 ]
,
0 .2
，
V 2 0
,
VL 1
, 0 .5 ，
Table.1 Fusion results of the four methods in MI and QAB/F Image
criteria
pepsi
MI
7.2363
QAB/F
PCNN-NSCT
SF-NSCT-PCNN
EOE-NSCT-PCNN
Proposed method
6.9608
7.4827
8.3949
8.9154
0.7107
0.7745
0.7794
0.7241
0.7933
MI
7.2929
7.4640
7.9837
8.2731
9.3919
QAB/F
0.7026
0.6983
0.7064
0.7865
0.7975
(b) Source B
(c) Initial map
(e) NSCT
(f) PCNN-NSCT
CC E
PT
(d) Modified decision map
ED
M
A
N
(a) Source A
U
SC R
IP T
book
NSCT
A
(j) SF-NSCT-PCNN
(h) EOE-NSCT-PCNN
(i) proposed
(k) PCNN-NSCT-
(m) EOE-NSCT-PCNN
(l) SF-NSCT-PCNN
(n) proposed
IP T
(j) NSCT -
Fig. 6. The experimental data and results of fusing images ‘Pepsi’: (a)–(b) The source images; (c)–(d) The process of the focus area detection;(e)–(i) The fusion results of
A
N
U
SC R
different fusion algorithms; (j)–(n) The difference images between fused images and Source A. the difference image between fused images and Source A
(b) Source B
(c) Initial map
PT
ED
M
(a) Source A
(e) NSCT
(f) PCNN-NSCT
A
CC E
(d) Modified decision map
(j) SF-NSCT-PCNN
(h) EOE-NSCT-PCNN
(i) proposed
(k) PCNN-NSCT-
(l) SF-NSCT-PCNN
(n) proposed -
U
(m) EOE-NSCT-PCNN
SC R
IP T
(j) NSCT -
Fig. 7. The experimental data and results of fusing images ‘book’: (a)–(b) The source images; (c)–(d) The process of the focus area detection;(e)–(i) The fusion results of different
N
fusion algorithms; (j)–(n) The difference images between fused images and Source A. the difference image between fused images and Source A
Comparison of NSCT [29] , PCNN-NSCT [31],SF-PCNN-NSCT[26] , EOE-PCNN-NSCT[33] and
A
the proposed method in terms of both subjective and objective evaluations. Mutual information (MI)
M
and QAB/F are employed as information-based objective criteria are used to compare the fusion results. The values of MI and QAB/F of the two sets of images are listed in Table 1. From Table 1, we can see that the values of MI and QAB/F of the proposed method are all most the largest and optimal in the five
ED
fusion methods.
As shown in Fig. 6(c), the initial map has many gaps and holes in both the black area and the white area. According to the multi-focus image-forming principle, this binary image cannot perfectly show
PT
the focus characteristics of the source images. Morphology processing is used to remove the gaps and holes. The obtained decision map successfully fills the gaps and holes and reflects the focus characteristics of the source image more accurately. The white area of decision map represents the
CC E
focus area of Source A, whereas the black area represents the focus area of Source B. Focusing on Figs. 6 (e) and (i), one can obviously find that the fused image of the proposed method is clearer and more natural than the other fused results especially in the region of upper right corner. As illustrated in the difference images from Figs. 6 (j) and (n), the proposed algorithm extracts almost all of the in-focus components of the source images and preserves the useful information better than the other
A
methods(highlighted by the black squares). To illustrate that the proposed method can be widely applied to multi-focus fusion, the second
experiments are performed on the other multi-focus images “book”. From Fig. 7, a similar conclusion to those of the previous two experiments is obtained, namely that the proposed algorithm is better than the other methods. 6 Conclusion and discussion A new approach for the multi-focus image fusion, which utilizes Orientation Information(OI)
Motivated Pulse Coupled Neural Networks(PCNN) and modified decision map. First, the source multi-focus images are fused using the Orientation Information Motivated Pulse Coupled Neural Networks (OI-PCNN), and the initial decision map is obtained. Second, after analyzing the decision map, we modified the decision map by employing a mathematical morphology post-processing technique. Finally, based on the modified decision map, the final fused image is obtained by selecting the pixels in the focus areas. Objective performance evaluation criteria and visual observation demonstrate that the proposed method is better than various existing spatial domain and transform domain fusion methods.
IP T
Acknowledgments
The work of this paper was supported by the National Natural Science Foundation of China (Project Nu mber: 61174193) and the Specialized Research Fund for the Doctoral Program of Higher Education (Project
SC R
Number: 20136102110036).
References
U
[1] B. Miles, I. B. Ayed, M. W. K. Law, G. Garvin, A. Fenster, and S. Li,“Spine image fusion via graph cuts,” IEEE Trans. Biomed. Eng. vol. 60,no. 7, pp. 1841–1850, Jul. 2013.
N
[2] J. Liang, Y. He, D. Liu, and X. Zeng, “Image fusion using higher order singular value decomposition,” IEEE Trans. Image Process., vol. 21,no. 5, pp. 2898–2909, May 2012.
A
[3] S. Li and B. Yang.“Hybrid multiresolution method for multisensor multimodal image fusion,” IEEE Sensors J. vol. 10, no. 9, pp. 1519–1526,Sep. 2010.
M
[4]Yong Yang, Song Tong, Shuying Huang, and Pan Lin. Multifocus Image Fusion Based on NSCT and Focused Area Detection J. IEEE SENSORS JOURNAL, VOL. 15, NO. 5,pp.2824-2838 MAY
ED
2015
[5]B.Yang and S. Li, “Multifocus image fusion and restoration with sparse representation,” IEEE Trans. Instrum. Meas., vol. 59, no. 4, pp. 884–892,Apr. 2010. [6]W. Wang and F. Chang, “A multi-focus image fusion method based on Laplacian pyramid,” J.
PT
Comput., vol. 6, no. 12, pp. 2559–2566, Dec. 2011. [7] N. Mitianoudis and T. Stathaki, “Optimal contrast correction for ICA-based fusion of multimodal images,” IEEE Sensors J., vol. 8, no. 12,pp. 2016–2026, Dec. 2008.
CC E
[8] Eckhorn R., Reitboeck H., Arndt M., et al, “Feature linking via synchronization among
distributed assemblies: Simulations of results from cat visual cortex,” Neural Comp., vol.2, no.3, pp.293-307, 1990.
[9] Amina Jameel, Abdul Ghafoor, Muhammad Mohsin Riaz “Wavelet and guided filter based
A
multi-focus fusion for noisy images”. Optik 126 (2015) 3920–3923 [10]Chunhui Zhao, Guofeng Shao, Lijuan Ma, Xi Zhang Image fusion algorithm based on
redundant-lifting NSWMDA and adaptive PCNN. Optik 125 (2014) 6247–6255 [11] D. Agrawal, J. Singhai, Multifocus image fusion using modified pulse coupledneural network for improved image quality, Image Process. IET 4 (6) (2010)443–451. [12] Y. Chai, H.F. Li, M.Y. Guo, Multifocus image fusion scheme based on featuresof multiscale products and PCNN in lifting stationary wavelet domain, Opt.Commun. 284 (5) (2010) 1146–1158. [13] Xiaobo Qu, Changwei Hu, Jingwen Yan, Image Fusion Algorithm Based On Orientation Information Motivated Pulse Coupled Neural Networks, The seventh World Congress on Intelligent
Control and Automation WCICA2008-Chongqing, China,25-27 June 2008, pp.2437-2441 [14] G. Piella, “A general framework for multiresolution image fusion: From pixels to regions,” Inf. Fusion, vol. 4, no. 4, pp. 259–280, Dec. 2003. [15] S. Li, X. Kang, J. Hu, and B. Yang, “Image matting for fusion of multi-focus images in dynamic scenes,” Inf. Fusion, vol. 14, no. 2, pp. 147–162, Apr. 2013. [16] Y. Liu, J. Jin, Q. Wang, Y. Shen, and X. Dong, “Region level basedmulti-focus image fusion using quaternion wavelet and normalized cut,”Signal Process., vol. 97, pp. 9–30, Apr. 2014. [17] Yong Yang, Song Tong, Shuying Huang, and Pan Lin. Multi-focus Image Fusion Based on NSCT and Focused Area Detection. IEEE SENSORS JOURNAL, VOL. 15, pp.2824-2838.NO. 5,
IP T
MAY 2015.
[18] Wang Xinnian, Liang Dequn, “Orientation-information measure-based image restoration”, 2003 International Conference on Machine Learning and Cybernetics, Vol.5, pp2925-2930,2003.
SC R
[19]Xiaobo Qu，Changwei Hu，Jingwen Yan. Image Fusion Algorithm Based On Orientation
Information Motivated Pulse Coupled Neural Networks. Proceedings of the 7th World Congress on Intelligent Control and Automation June 25 - 27, 2008, Chongqing, China
[20]JIANG Ling-ling , YIN Hai-qing , FENG Xiang-chu. An algorithm for image enhancement based on shock filters and anisotropic diffusion. JOURNAL
XIDIAN
UNIVERSITY. Jun.
U
2006 Vol,33 ,No. 3.476-479.
OF
[21]Y. Chai, H.F. Li, M.Y. Guo. Multifocus image fusion scheme based on features of multiscale
N
products and PCNN in lifting stationary wavelet domain. Optics Communications 284 (2011) 1146– 1158.
A
[22] J.L. Johnson, M.L. Padgett, PCNN models and applications. IEEE Transatcion on Neural
M
Networks 10 (3) (1999) 480.
[23] H.S. Ranganath, G. Kuntimad, J.L. Johonson, Perfect image segmentation using pulse coupled neural networks. Proc.of the Southeast Conference on‘Visualize the future’, 1995, p. 37.
ED
[24] L. Johnosn, H. Ranganath, G. Kuntimad, H.J. Caulfield, Object detection using pulse coupled neural networks. IEEE Transactions on Neural Networks 10 (3) (1999) 461. [25] R.P. Broussard, S.K. Rogers, M.E. Oxley, et al., Physiologically motivated image fusion for (1999) 554.
PT
object detection using a pulse coupled neural network. IEEE Transactions on Neural Networks 10 (3) [26] Xiaobo Qu, Jingwen YAN, Hongzhi XIAO, Ziqian ZHU, “Image Fusion Algorithm Based on
CC E
Spatial Frequency Motivated Pulse Coupled Neural Networks in Nonsubsampled Contourlet Transform Domain,”Acta Automatica Sinica, vol. 34, no.12, pp. 1508-1514, 2008 [27] S. Li and B. Yang, “Hybrid multiresolution method for multisensor multimodalimage fusion,”
IEEE Sensors J., vol. 10, no. 9, pp. 1519–1526, Sep. 2010. [28] G. Piella and H. Heijmans, “A new quality metric for image fusion,”in Proc. IEEE Int. Conf.
A
Image Process., Amsterdam, The Netherlands,Sep. 2003, pp. 173–176. [29] G. Bhatnagar, Q. M. J. Wu, and Z. Liu, “Directive contrast based multimodal medical image
fusion in NSCT domain,” IEEE Trans. Multimedia, vol. 15, no. 5, pp. 1014–1024, Aug. 2013. [30] B. Yang and S. Li, “Pixel-level image fusion with simultaneous orthogonal matching pursuit,” Inf. Fusion, vol. 13, no. 1, pp. 10–19, Jan. 2012. [31]Wang Rongfeng,“Image fusion algorithm based on nonsubsampled contourlet transform and pulse coupled neural netwrork”. Computer Applications, Vol.28 ,no.12,pp.164-167, 2008. [32] Jingjing Wang, Qian Li, Zhenhong Jia, Nikola Kasabov, Jie Yang. A novel multi-focus image
fusion method using PCNN innonsubsampled contourlet transform domain. Optik 126 (2015)
A
CC E
PT
ED
M
A
N
U
SC R
IP T
2508–2511