- Email: [email protected]
Target extraction of blurred infrared image with an immune network template algorithm
Optics & Laser Technology 56 (2014) 102–106
Contents lists available at ScienceDirect
Optics & Laser Technology journal homepage: www.elsevier.com/locate/optlastec
Target extraction of blurred infrared image with an immune network template algorithm Dongmei Fu, Xiao Yu n, Hejun Tong School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing 100083, China
art ic l e i nf o
a b s t r a c t
Article history: Received 24 March 2013 Received in revised form 16 July 2013 Accepted 24 July 2013 Available online 23 August 2013
To extract targets from blurred infrared images efficiently, this paper presents a coordinated immune network template algorithm based on the coordination mechanism between innate immunity and adaptive immunity. First, the neighbourhood characteristics for every pixel in a blurred infrared image are computed as the template features of each pixel. Next, all of the pixels are divided into three sets according to their grey level values: a target set, a background set and a blurred set. Finally, the pixels in the target set and the background set are used for training the adaptive network, which can divide each blurred pixel into two classes: a target pixel or a background pixel. Experimental results on the hand trace infrared images verified that the proposed algorithm could efficiently extract targets from images and produce better extraction accuracy. & 2013 Elsevier Ltd. All rights reserved.
Keywords: Target extraction Blurred infrared image Immune network
1. Introduction Current latent handprint and trace evidence collection technologies are usually invasive and can be destructive to the original deposits in crime scene [1]. The temperature of the human hand is generally higher than that of its surroundings. A thermal trace of a hand will be left on surfaces if a human hand has touched its surroundings, and infrared images can describe the hand print trace. If infrared images are used to collect this hand print trace, the original deposits will not be destroyed [2]. If an infrared image of the trace is shot immediately after the hand leaves its surroundings, within one second, the image can reflect the touch contour (the contour of the hand trace) between the hand and its surroundings. However, in crime scenes, infrared images of the traces are generally shot after the hand has been gone for more than a second. In these circumstances, the infrared image will always be blurry because the grey level of its pixels will not accurately reflect the contour of the hand trace. Extracting the hand trace contour from this type of blurred infrared image is a challenging task [3]. The template extraction method, which uses template characteristics to extract targets from an image, has received considerable attention in recent years. Many template image extraction algorithms have been proposed in the literature. In general, template image extraction algorithms can be classified into two categories: boundarybased approaches and region-based approaches.
n
Corresponding author. Tel.: +86 15201157204. E-mail address: [email protected] (X. Yu).
0030-3992/$ - see front matter & 2013 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.optlastec.2013.07.019
Boundary-based approaches extract targets from images based on abrupt (local) changes in intensity, but blurring and noise will result in deviations from the ideal shape. The Robert crossgradient template [4] is one of the earliest attempts to use a 2-D template. The Prewitt template [5] is the simplest digital approximation of the partial derivatives using a template of size 3 3. The Sobel template [6] is developed from the Prewitt template. The Canny template [7] can produce high-quality edge lines with regard to their continuity, thinness and straightness. Deformable templates [8] design templates according to image features based on the results of the Canny template. Because the boundary-based approaches are more sensitive to blurring and noise, recent template image extraction algorithms have used a region-based approach. Yoon [9] proposed a feature template that extracts various features from an image, such as the shape and texture, to design a template, but the user is asked to label a certain region as the initial foreground. To achieve automatic image extraction, Bhanu [10] proposed a functional template, which encodes templates using a collection of discriminating functions associated with image features; this method is guided by a genetic algorithm (GA) that suffers the defect of a local optimum tendency. To overcome this problem, Hua Bo [11] uses a spatial matrix to describe the spatial characteristics of an image and presents a spatial matrix template based on the mechanism of adaptive immunity. In addition, certain extraction methods based on immunity networks are proposed. Younis [12] proposed an artificial immune-activated neural network (AIANN), which uses neighbouring greyscale intensities of pixels to extract targets from an MRI image. Huang [13] proposed an immune kernel clustering network (IKCN), which uses the mean deviation of a template to
D. Fu et al. / Optics & Laser Technology 56 (2014) 102–106
extract SAR images. However, these immune network methods are all based on the mechanism of adaptive immunity. Medical studies indicate that the biological immune system consists of two components, innate immunity and adaptive immunity. There is a direct signal connection between innate immunity and adaptive immunity [14,15]. To extract a hand trace contour from a blurred infrared image, a coordinated immune network template algorithm (CINTA) is proposed in this paper. Inspired by the coordination mechanism between innate immunity and adaptive immunity, we design an immune network to achieve the extraction. First, the algorithm computes the neighbourhood characteristics for every pixel in a blurred infrared image as region features of each pixel. Next, the pixels are divided into three sets based on their grey level values: the target pixel set, the background pixel set and the blurred pixel set. Finally, the target and background pixel samples are used to train the immune network, which will detect blurred pixels and divide them into target pixel or background pixel categories. Experimental results indicate that the proposed method can improve the target extraction rate and reduce the extraction error rate. The remainder of the paper is organised as follows: the next section presents our proposed coordinated immune network template algorithm. Section 3 introduces the blurred infrared image of hand trace. Section 4 presents the experimental results, and Section 5 presents the conclusions of this paper.
2. Coordinated immune network template algorithm Artificial immune networks, derived from immune network theory, are important and effective models of artificial immune systems and have been successfully applied to data analysis [16], multimodal electromagnetic problems [17] and remote sensing [18]. However, these immune network methods are based on the mechanism of adaptive immunity and cannot reflect the overall mechanisms of the biological immune system. The biological immune system consists of innate and adaptive immunity. Innate immunity can recognise pathogens according to their primitive features and change the features of the pathogens. Adaptive immunity can recognise pathogens according to the new features of the pathogens. Inspired by the coordination mechanism between innate immunity and adaptive immunity, we design a three-layer immune network, which contains an innate feature transformation layer, an innate recognition layer and an adaptive recognition layer, as shown in Fig. 1. The innate feature transformation layer uses a template to obtain the 3 3 neighbourhood characteristics for every image pixel. The innate recognition layer divides image pixels into three sets based on grey level: a target set, a background
103
set and a blurred set. The adaptive recognition layer divides the blurred pixels into target pixel or background pixel based on neighbourhood characteristics. Fig. 2 shows the entire flow chart of our algorithm. 2.1. Innate feature transformation For a blurred infrared image that has R rows and C columns, f ðu; vÞ is the grey level at pixel sample point ðu; vÞ, u ¼ 1; 2; :::R, v ¼ 1; 2; ::C. Suppose each pixel point ðu; vÞ is a sample xi ði ¼ 1; 2; ⋯R CÞ. Next, there are R C pixel samples in the blurred infrared image. The innate feature transformation layer uses a template g i ði ¼ 1; 2; ⋯R CÞ of the size 3 3 (as shown in Fig. 3) to obtain the neighbourhood characteristics of every sample point, and the sample point is the central point of the template. The neighbourhood characteristics of every pixel sample will be the template features of the pixel sample. The template features include the template mean and the high frequency wavelet coefficient of the template. The mean g i1 of template g i is given by g i1 ¼
1 ∑ f ðs; tÞ 9ðs;tÞ A gi
ð1Þ
where f ðs; tÞ is the grey value at pixel ðs; tÞ. When the wavelet resolution is 2, the wavelet coefficients of template g i are given by 8 0 dq;p ¼ ∑ϕk2q ϕl2p g i > > > > l;k > > > > 1 > d ¼ ∑ ϕk2q ψ l2p g i > q;p > < l;k ð2Þ 2 > dq;p ¼ ∑ψ k2q ϕl2p g i > > > l;k > > > > 3 > > d ¼ ∑ ψ k2q ψ l2p g i > : q;p l;k 0
1
2
3
where dq;p , dq;p , dq;p , and dq;p are the high frequency coefficient, horizontal high frequency coefficient, vertical high frequency coefficient and diagonal high frequency coefficient, respectively. ψðxÞis a one-dimensional scaling function, where q ¼ 1; 2; 3 and p ¼ 1; 2; 3, k is an integer which determines the position of ϕk2q along the q-axis, l is an integer which determines the position of Original image Innate feature transformation pixels with gray level feature and template feature Innate recognition based on gray level feature target pixel set
background pixel set
Train adaptive recognition layer based on template feature
blurred pixel set
Adaptive recognition based on template feature target pixel set
Fig. 1. The three layers immune network.
background pixel set
Fig. 2. Coordination immune network template extraction algorithm.
104
D. Fu et al. / Optics & Laser Technology 56 (2014) 102–106
gi Blurred infrared image f(u-1,v-1)
f(u-1,v)
f(u-1,v+1)
f(u,v-1)
f(u,v)
f(u,v+1)
f(u+1,v-1)
f(u+1,v)
f(u+1,v+1)
template
A pixel of the blurred infrared image
Fig. 3. Generate the template region g i of a sample point ðu; vÞ.
ϕl2q along the p-axis, and ϕ is a one-dimensional scaling function. We use the Daubechies filter to determine the ϕ. The innate feature transformation layer chooses the high frequency coeffi0 cient dq;p of g i as the frequency-domain feature:
we only use the template features fxji2 ; xji3 g of pixel samples xji here). Step 2 Randomly choose α values as weights wm ðm ¼ 1; ⋯αÞ for j
j
α hyperplanes, where 〈wm ; xkk 〉 ¼ max f〈wm ; xii s〉g, 〈; 〉 is the inner j
0
g i2 ¼ dq;p
ð3Þ
xii A M
, and ji is the class label product operator, jk is the class label of xjk k
After the template features are determined, pixel samples will have three features: the grey level, the neighbourhood mean and the high frequency wavelet coefficient of the pixel's neighbourhood. Each pixel sample is a three-dimensional vector xi ¼ fxi1 ; xi2 ; xi3 g, where i is a sequential value, i ¼ 1; 2; …R C, xi1 ; xi2 ; xi3 are the three components of the sample vector, xi1 is the grey level, and xi2 ¼ g i1 , xi3 ¼ g i2 .
of xjii . Then, calculate the threshold θm ðm ¼ 1; ⋯αÞ of α hyperplanes according to the following formulas:
2.2. Innate recognition
〈wm ; xhh 〉 þ 〈wm ; xli 〉 θm ¼ 2
The innate recognition layer determines the class label for every pixel vector xi ¼ fxi1 ; xi2 ; xi3 g based on xi1 . First, we use the Otsu method [19] to segment pixel vectors based on xi1 and obtain a grey level threshold T. Second, we set T 0 ¼ 0:1, T 1 ¼ T þ T 0 , and T 2 ¼ TT 0 . There are three nodes in the innate recognition layer, ½T 1 ; 255 are the grey level ranges of the first node, ðT 2 ; T 1 Þ are the grey level ranges of the second node, and ½0; T 2 are the grey level ranges of the third node. These three nodes determine the class label for every pixel vector. Suppose j is the class label of a pixel vector xji ¼ fxji1 ; xji2 ; xji3 g. The innate recognition layer determines j by the following formula: 8 > 1 xji1 A ½T 1 ; 255 > > < j ð4Þ j ¼ 2 xi1 A ðT 2 ; T 1 Þ > > > : 3 xj A ½0; T
Next, the α hyperplanes will be wm x þ θm ¼ 0ðm ¼ 1; ⋯αÞ. Samples that satisfy wm x þ θm 4 0 are the samples determined by the mth hyperplane, and the class label of this hyperplane is jk . Step 3 Calculate the sample number determined by each hyperplane, and obtain the best hyperplane, the one that has the largest sample number. Next, assign the weight, threshold and class label of the best hyperplane to an adaptive node, and assign a serial number to the node. The samples determined by the best hyperplane will be deleted from the training set M. Step 4 Repeat Step 2 to Step 3, until there is only one class sample in training set M. Step 5 Randomly choose w as the weight of the last adaptive
i1
2
Therefore, the innate recognition layer divides pixels into three sets, a target set whose class label is 1, a background set whose class label is 3 and a blurred set whose class label is 2. 2.3. Training the adaptive recognition layer Every node in the adaptive recognition layer has a serial number from one to N3 ; the weights and thresholds of these nodes will be trained by the target pixel set and the background pixel set. The training process is as follows: Step 1 Use pixel samples xji ¼ fxji2 ; xji3 gðj ¼ 1; 3Þ in target set and background set to constitute the training sample set M. (Note that
j
〈wm ; xhh 〉jjh a jk ¼ j
〈wm ; xll 〉jjl ¼ jk ¼ j
j
j
ð5Þ
j
ð6Þ
max
f〈wm ; xii 〉g
min
f〈wm ; xii 〉g
xjii A M; ji a jk
xii A M; ji ¼ jk j
ð7Þ
j
node and compute 〈w; xbb 〉 ¼ minf〈w; x〉g; then, jb is the class label xAM
of the node. Set θ ¼ 1 as the threshold of the node, and assign N1 as the serial number to this node. Step 6 Adaptive nodes work in accordance with their serial number, ranging from small to large; their working function is given by ( jn Z n 40 ΩðZ n Þ ¼ ð8Þ 0 Zn o 0 where Z n ¼ wn I n θn , I n is the input, n ¼ 1; ⋯; N 1 is the serial number, and jn ¼ 1; 3 is the class label. The threshold θn can be changed by θ′n ¼ θn þ λn and is set λn to zero by default. 2.4. Adaptive recognition As its structure and function are determined, the adaptive recognition layer can recognise pixels in the blurred set and divide
D. Fu et al. / Optics & Laser Technology 56 (2014) 102–106
them into target pixels or background pixels based on template features; therefore, the immune network template algorithm can be used to extract targets from a blurred infrared image. Inputting a blurred infrared image of a hand trance to the immune template network, the output is the extraction result.
105
arrive in one second. The infrared image that we often obtain is Fig. 4(b), which is a blurred infrared image, and its grey level does not accurately reflect the contour of the hand trace. The purpose of this paper is to extract an accurate hand contour from Fig. 4(b). To evaluate the extraction results, we use the manually processed result as the ground truth, as shown in Fig. 4(c).
3. Blurred infrared image and ground truth Under laboratory conditions, we shot hand trace infrared images one second and two minutes after a hand left a board, as shown in Fig. 4(a) and (b), respectively. Fig. 4(a) can reflect the touching contour between the hand and the board, but Fig. 4 (a) usually cannot be obtained in a crime scene because we cannot
4. Experimental results The proposed immune network template algorithm has been simulated in a MATLAB programme environment for target extraction from a hand trace infrared image.
Fig. 4. Blurred infrared images of hand traces shot at different times and the ground truth. (a) The hand leaves for 1 second. (b) The hand leaves for 2 min. (c) The Ground Truth.
Fig. 5. The extraction results. (a) Original image, (b) Prewitt, (c) Canny, (d) deformable template, (e) spatial matrix template, (f) artificial immune activated neural network (AIANN), (g) immune kernel clustering network(IKCN) and (h) coordination immune network template algorithm(CINTA) ðλ1 ¼ 0; λ2 ¼ 0Þ. (i) CINTA ðλ1 ¼ 0:38; λ2 ¼ 0Þ.
106
D. Fu et al. / Optics & Laser Technology 56 (2014) 102–106
Table 1 Extraction results analysis. Algorithm
TPR
J ðGT;SIÞ
DðGT;SIÞ
r err
Spatial matrix template AIANN IKCN CINTA ðλ1 ¼ 0Þ CINTA ðλ1 ¼ 0:38Þ
0.3204 0.2735 0.2735 0.6645 0.7389
0.2612 0.2228 0.2230 0.6162 0.6384
0.4143 0.3644 0.3647 0.7644 0.7793
32.15% 33.87% 33.82% 14.55% 14.71%
Fig. 5 shows the extraction results of different algorithms for the hand trace infrared image: (a) is the original image, (b) is the extraction result of the Prewitt template, (c) is the extraction result of the Canny template, (d) is the extraction result of the deformable template, (e) is the extraction result of the spatial matrix template, (f) is the extraction result of artificial immune activated neural network (AIANN), (g) is the extraction result of the immune kernel clustering network (IKCN), (h) is the extraction result of the coordinated immune network template algorithm (CINTA) ðλ1 ¼ 0; λ2 ¼ 0Þ and (i) is the extraction result of coordinated immune network template algorithm (CINTA) ðλ1 ¼ 0:38; λ2 ¼ 0Þ. It can be observed that in the cases with large unknown regions, the Prewitt, Canny and deformable template perform poorly, and cannot extract the hand area from the image. The spatial matrix template, AIANN and IKCN perform better, but they still produce inaccurate boundary contours. On the contrary, the method we proposed, CINTA, can produce accurate target contours. To evaluate the extraction performance quantitatively, the extraction algorithms are evaluated with the following parameters: true positive rate (TPR) [20], the Jaccard similarity index (J ðGT;SIÞ ) [21], the Dice similarity index (DðGT;SIÞ ) [22] and the absolute error ratio [23]. Table 1 presents the quantitative evaluation results. The TPR of the spatial matrix template, AIANN and IKCN, are much smaller than our algorithm, which indicates that the target extraction rate of these methods is far smaller than our algorithm. The J ðGT;SIÞ and DðGT;SIÞ of these three methods are much smaller than our algorithm, which indicates that overlap ratio between the ground truth and the extracted result of these three methods is far lower than the overlap degree of our algorithm. The r err of these three methods is larger than our algorithm, which illustrates that the absolute extraction error of these three methods is larger than that obtained with our algorithm. The default value of λ1 in our algorithm is 0. When we reduce the value of λ1 , such as setting λ1 ¼ 0:38, the extraction result of our algorithm will have higher TPR, J ðGT;SIÞ , DðGT;SIÞ and r err . Overall, after analysing the extraction performance of the above mentioned algorithms based on the proposed quantitative parameters, CINTA provides the best result for extracting target regions from a blurred infrared image. 5. Conclusions In this paper, a coordinated immune network template algorithm based on the immune coordination mechanism was proposed. This paper investigated the working principle of immune system; the pathogen is processed by the innate immunity first and later by the adaptive immunity. An immune network model according to the immune operating principle was presented in this paper. This model contains three layers: the innate feature transformation layer, the innate recognition layer and the adaptive recognition layer. Each pixel in a blurred infrared image of a hand trace was handled by these three layers and was finally divided into two classes, the target pixel or the background pixel. Compared to the previously reported classical template methods and immune network template methods, the coordinated
immune network template algorithm presents a better target extraction rate and overall extraction accuracy while reducing the absolute error rate significantly. From the obtained results, it is evident that the proposed extraction technique yields better extraction performance for hand trace infrared images. The algorithm is directly used for a blurred infrared image of hand trace that is shot under lab conditions. Studying the practical significance and performance of the algorithm for blurred infrared images of traces that are captured during crime scenes will be the focus of future work.
Acknowledgements This research is supported by the National Natural Science Foundation of China (No. 61272358) and the international technological project between China and the Czech Republic (No. 39-10). References [1] Bhargava Rohit, Schwartz Perlman Rebecca, Daniel C. Non-invasive detection of superimposed latent fingerprints and inter-ridge trace evidence by infrared spectroscopic imaging. Analytical and Bioanalytical Chemistry 2009;394 (8):2069–75. [2] Van Lersel Miranda. Infrared photography aids forensic research. TNO Magazine 2010;1:6–7. [3] Xiong ZL, Yang KT, Ding WX. Computational and experimental research on infrared trace by human being contact. Applied Optics 2010;49(18):3587–95. [4] Roberts LG. Machine perception of three-dimensional solids.Cambridge: MIT Press; 1965. [5] Prewitt JMS. Object enhancement and extraction.New York: Academic Press; 1970. [6] Sobel IE. Camera models and machine perception. Stanford University; 1970 (Ph.D. dissertation). [7] Canny J. A computational approach to edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 1986;8(6):679–98. [8] Garrido A, de la Blanca NP. Applying deformable templates for cell image segmentation. Pattern Recognition 2000;33(5):821–32. [9] Yoon Young-Geun, Lee Seok-Lyong, Chung Chin-Wan. An effective defect inspection system for polarized film images using image segmentation and template matching techniques. Computers and Industrial Engineering 2008;55(3):567–83. [10] Bhanu B, Fonder S. Functional template-based SAR image segmentation. Pattern Recognition 2004;37(1):61–77. [11] Bo Hua, Ma Fulong, Han Baojun SAR. Image segmentation based on immune algorithm. In: Proceedings of international conference on control and automation, Xi an, China; 2005, p. 375–378. [12] Akmal Younis Mohamed, Ibrahim Mansur, Kabuka Nigel John. An artificial immune-activated neural network applied to brain 3D MRI segmentation. Journal of Digital Imaging 2008;21(1):69–88. [13] Huang Wenlong, Jiao Licheng. Artificial immune kernel clustering network for unsupervised image segmentation. Progress in Natural Science 2008;18 (4):455–61. [14] Suzuki N, Suzuki S, Millar DGA. Critical role for the innate immune signaling molecule IRAK-4 in T cell activation. Science 2006;311(5769):1927–32. [15] Iwasaki1 A, Medzhitov R. Regulation of adaptive immunity by the innate immune system. Science 2010;327(5963):291–5. [16] De Castro, Timmis J. Artificial immune systems: a new computational intelligence approach.London: Springer-Verlag; 2002. [17] Campelo F, Guimaraes FG, Igarashi H, Ramirez JA, Noguchi S. A modified immune network algorithm for multimodal electromagnetic problems. IEEE Transactions on Magnetics 2006;42(4):1111–4. [18] PAL M. Artificial immune-based supervised classifier for land-cover classification. International Journal of Remote Sensing 2008;29(8):2273–91. [19] Xue JingHao Titterington Michael. T-tests, f-tests and otsu's methods for image thresholding. IEEE Transactions on Image Processing 2011;20(8):2392–6. [20] Jadin MS, Taib S. Infrared image enhancement and segmentation for extracting the thermal anomalies in electrical equipment. Electronics and Electrical Engineering 2012;4:107–12. [21] Rub E.C. Multimodal evaluation for medical image segmentation. In: Proceedings of international conference on computer analysis of images and patterns, Vienna, Austria;2007. p. 229–236 [22] Babalola KO, Patenaude B, Aljabar P. An evaluation of four automatic methods of segmenting the subcortical structures in the brain. Neuroimage 2009;47 (4):1435–47. [23] Wenbing Tao, Hai Jin, Liman Liu. Object segmentation using ant colony optimization algorithm and fuzzy entropy. Pattern Recognition Letters 2007;28(7):788–96.
Contents lists available at ScienceDirect
Optics & Laser Technology journal homepage: www.elsevier.com/locate/optlastec
Target extraction of blurred infrared image with an immune network template algorithm Dongmei Fu, Xiao Yu n, Hejun Tong School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing 100083, China
art ic l e i nf o
a b s t r a c t
Article history: Received 24 March 2013 Received in revised form 16 July 2013 Accepted 24 July 2013 Available online 23 August 2013
To extract targets from blurred infrared images efficiently, this paper presents a coordinated immune network template algorithm based on the coordination mechanism between innate immunity and adaptive immunity. First, the neighbourhood characteristics for every pixel in a blurred infrared image are computed as the template features of each pixel. Next, all of the pixels are divided into three sets according to their grey level values: a target set, a background set and a blurred set. Finally, the pixels in the target set and the background set are used for training the adaptive network, which can divide each blurred pixel into two classes: a target pixel or a background pixel. Experimental results on the hand trace infrared images verified that the proposed algorithm could efficiently extract targets from images and produce better extraction accuracy. & 2013 Elsevier Ltd. All rights reserved.
Keywords: Target extraction Blurred infrared image Immune network
1. Introduction Current latent handprint and trace evidence collection technologies are usually invasive and can be destructive to the original deposits in crime scene [1]. The temperature of the human hand is generally higher than that of its surroundings. A thermal trace of a hand will be left on surfaces if a human hand has touched its surroundings, and infrared images can describe the hand print trace. If infrared images are used to collect this hand print trace, the original deposits will not be destroyed [2]. If an infrared image of the trace is shot immediately after the hand leaves its surroundings, within one second, the image can reflect the touch contour (the contour of the hand trace) between the hand and its surroundings. However, in crime scenes, infrared images of the traces are generally shot after the hand has been gone for more than a second. In these circumstances, the infrared image will always be blurry because the grey level of its pixels will not accurately reflect the contour of the hand trace. Extracting the hand trace contour from this type of blurred infrared image is a challenging task [3]. The template extraction method, which uses template characteristics to extract targets from an image, has received considerable attention in recent years. Many template image extraction algorithms have been proposed in the literature. In general, template image extraction algorithms can be classified into two categories: boundarybased approaches and region-based approaches.
n
Corresponding author. Tel.: +86 15201157204. E-mail address: [email protected] (X. Yu).
0030-3992/$ - see front matter & 2013 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.optlastec.2013.07.019
Boundary-based approaches extract targets from images based on abrupt (local) changes in intensity, but blurring and noise will result in deviations from the ideal shape. The Robert crossgradient template [4] is one of the earliest attempts to use a 2-D template. The Prewitt template [5] is the simplest digital approximation of the partial derivatives using a template of size 3 3. The Sobel template [6] is developed from the Prewitt template. The Canny template [7] can produce high-quality edge lines with regard to their continuity, thinness and straightness. Deformable templates [8] design templates according to image features based on the results of the Canny template. Because the boundary-based approaches are more sensitive to blurring and noise, recent template image extraction algorithms have used a region-based approach. Yoon [9] proposed a feature template that extracts various features from an image, such as the shape and texture, to design a template, but the user is asked to label a certain region as the initial foreground. To achieve automatic image extraction, Bhanu [10] proposed a functional template, which encodes templates using a collection of discriminating functions associated with image features; this method is guided by a genetic algorithm (GA) that suffers the defect of a local optimum tendency. To overcome this problem, Hua Bo [11] uses a spatial matrix to describe the spatial characteristics of an image and presents a spatial matrix template based on the mechanism of adaptive immunity. In addition, certain extraction methods based on immunity networks are proposed. Younis [12] proposed an artificial immune-activated neural network (AIANN), which uses neighbouring greyscale intensities of pixels to extract targets from an MRI image. Huang [13] proposed an immune kernel clustering network (IKCN), which uses the mean deviation of a template to
D. Fu et al. / Optics & Laser Technology 56 (2014) 102–106
extract SAR images. However, these immune network methods are all based on the mechanism of adaptive immunity. Medical studies indicate that the biological immune system consists of two components, innate immunity and adaptive immunity. There is a direct signal connection between innate immunity and adaptive immunity [14,15]. To extract a hand trace contour from a blurred infrared image, a coordinated immune network template algorithm (CINTA) is proposed in this paper. Inspired by the coordination mechanism between innate immunity and adaptive immunity, we design an immune network to achieve the extraction. First, the algorithm computes the neighbourhood characteristics for every pixel in a blurred infrared image as region features of each pixel. Next, the pixels are divided into three sets based on their grey level values: the target pixel set, the background pixel set and the blurred pixel set. Finally, the target and background pixel samples are used to train the immune network, which will detect blurred pixels and divide them into target pixel or background pixel categories. Experimental results indicate that the proposed method can improve the target extraction rate and reduce the extraction error rate. The remainder of the paper is organised as follows: the next section presents our proposed coordinated immune network template algorithm. Section 3 introduces the blurred infrared image of hand trace. Section 4 presents the experimental results, and Section 5 presents the conclusions of this paper.
2. Coordinated immune network template algorithm Artificial immune networks, derived from immune network theory, are important and effective models of artificial immune systems and have been successfully applied to data analysis [16], multimodal electromagnetic problems [17] and remote sensing [18]. However, these immune network methods are based on the mechanism of adaptive immunity and cannot reflect the overall mechanisms of the biological immune system. The biological immune system consists of innate and adaptive immunity. Innate immunity can recognise pathogens according to their primitive features and change the features of the pathogens. Adaptive immunity can recognise pathogens according to the new features of the pathogens. Inspired by the coordination mechanism between innate immunity and adaptive immunity, we design a three-layer immune network, which contains an innate feature transformation layer, an innate recognition layer and an adaptive recognition layer, as shown in Fig. 1. The innate feature transformation layer uses a template to obtain the 3 3 neighbourhood characteristics for every image pixel. The innate recognition layer divides image pixels into three sets based on grey level: a target set, a background
103
set and a blurred set. The adaptive recognition layer divides the blurred pixels into target pixel or background pixel based on neighbourhood characteristics. Fig. 2 shows the entire flow chart of our algorithm. 2.1. Innate feature transformation For a blurred infrared image that has R rows and C columns, f ðu; vÞ is the grey level at pixel sample point ðu; vÞ, u ¼ 1; 2; :::R, v ¼ 1; 2; ::C. Suppose each pixel point ðu; vÞ is a sample xi ði ¼ 1; 2; ⋯R CÞ. Next, there are R C pixel samples in the blurred infrared image. The innate feature transformation layer uses a template g i ði ¼ 1; 2; ⋯R CÞ of the size 3 3 (as shown in Fig. 3) to obtain the neighbourhood characteristics of every sample point, and the sample point is the central point of the template. The neighbourhood characteristics of every pixel sample will be the template features of the pixel sample. The template features include the template mean and the high frequency wavelet coefficient of the template. The mean g i1 of template g i is given by g i1 ¼
1 ∑ f ðs; tÞ 9ðs;tÞ A gi
ð1Þ
where f ðs; tÞ is the grey value at pixel ðs; tÞ. When the wavelet resolution is 2, the wavelet coefficients of template g i are given by 8 0 dq;p ¼ ∑ϕk2q ϕl2p g i > > > > l;k > > > > 1 > d ¼ ∑ ϕk2q ψ l2p g i > q;p > < l;k ð2Þ 2 > dq;p ¼ ∑ψ k2q ϕl2p g i > > > l;k > > > > 3 > > d ¼ ∑ ψ k2q ψ l2p g i > : q;p l;k 0
1
2
3
where dq;p , dq;p , dq;p , and dq;p are the high frequency coefficient, horizontal high frequency coefficient, vertical high frequency coefficient and diagonal high frequency coefficient, respectively. ψðxÞis a one-dimensional scaling function, where q ¼ 1; 2; 3 and p ¼ 1; 2; 3, k is an integer which determines the position of ϕk2q along the q-axis, l is an integer which determines the position of Original image Innate feature transformation pixels with gray level feature and template feature Innate recognition based on gray level feature target pixel set
background pixel set
Train adaptive recognition layer based on template feature
blurred pixel set
Adaptive recognition based on template feature target pixel set
Fig. 1. The three layers immune network.
background pixel set
Fig. 2. Coordination immune network template extraction algorithm.
104
D. Fu et al. / Optics & Laser Technology 56 (2014) 102–106
gi Blurred infrared image f(u-1,v-1)
f(u-1,v)
f(u-1,v+1)
f(u,v-1)
f(u,v)
f(u,v+1)
f(u+1,v-1)
f(u+1,v)
f(u+1,v+1)
template
A pixel of the blurred infrared image
Fig. 3. Generate the template region g i of a sample point ðu; vÞ.
ϕl2q along the p-axis, and ϕ is a one-dimensional scaling function. We use the Daubechies filter to determine the ϕ. The innate feature transformation layer chooses the high frequency coeffi0 cient dq;p of g i as the frequency-domain feature:
we only use the template features fxji2 ; xji3 g of pixel samples xji here). Step 2 Randomly choose α values as weights wm ðm ¼ 1; ⋯αÞ for j
j
α hyperplanes, where 〈wm ; xkk 〉 ¼ max f〈wm ; xii s〉g, 〈; 〉 is the inner j
0
g i2 ¼ dq;p
ð3Þ
xii A M
, and ji is the class label product operator, jk is the class label of xjk k
After the template features are determined, pixel samples will have three features: the grey level, the neighbourhood mean and the high frequency wavelet coefficient of the pixel's neighbourhood. Each pixel sample is a three-dimensional vector xi ¼ fxi1 ; xi2 ; xi3 g, where i is a sequential value, i ¼ 1; 2; …R C, xi1 ; xi2 ; xi3 are the three components of the sample vector, xi1 is the grey level, and xi2 ¼ g i1 , xi3 ¼ g i2 .
of xjii . Then, calculate the threshold θm ðm ¼ 1; ⋯αÞ of α hyperplanes according to the following formulas:
2.2. Innate recognition
〈wm ; xhh 〉 þ 〈wm ; xli 〉 θm ¼ 2
The innate recognition layer determines the class label for every pixel vector xi ¼ fxi1 ; xi2 ; xi3 g based on xi1 . First, we use the Otsu method [19] to segment pixel vectors based on xi1 and obtain a grey level threshold T. Second, we set T 0 ¼ 0:1, T 1 ¼ T þ T 0 , and T 2 ¼ TT 0 . There are three nodes in the innate recognition layer, ½T 1 ; 255 are the grey level ranges of the first node, ðT 2 ; T 1 Þ are the grey level ranges of the second node, and ½0; T 2 are the grey level ranges of the third node. These three nodes determine the class label for every pixel vector. Suppose j is the class label of a pixel vector xji ¼ fxji1 ; xji2 ; xji3 g. The innate recognition layer determines j by the following formula: 8 > 1 xji1 A ½T 1 ; 255 > > < j ð4Þ j ¼ 2 xi1 A ðT 2 ; T 1 Þ > > > : 3 xj A ½0; T
Next, the α hyperplanes will be wm x þ θm ¼ 0ðm ¼ 1; ⋯αÞ. Samples that satisfy wm x þ θm 4 0 are the samples determined by the mth hyperplane, and the class label of this hyperplane is jk . Step 3 Calculate the sample number determined by each hyperplane, and obtain the best hyperplane, the one that has the largest sample number. Next, assign the weight, threshold and class label of the best hyperplane to an adaptive node, and assign a serial number to the node. The samples determined by the best hyperplane will be deleted from the training set M. Step 4 Repeat Step 2 to Step 3, until there is only one class sample in training set M. Step 5 Randomly choose w as the weight of the last adaptive
i1
2
Therefore, the innate recognition layer divides pixels into three sets, a target set whose class label is 1, a background set whose class label is 3 and a blurred set whose class label is 2. 2.3. Training the adaptive recognition layer Every node in the adaptive recognition layer has a serial number from one to N3 ; the weights and thresholds of these nodes will be trained by the target pixel set and the background pixel set. The training process is as follows: Step 1 Use pixel samples xji ¼ fxji2 ; xji3 gðj ¼ 1; 3Þ in target set and background set to constitute the training sample set M. (Note that
j
〈wm ; xhh 〉jjh a jk ¼ j
〈wm ; xll 〉jjl ¼ jk ¼ j
j
j
ð5Þ
j
ð6Þ
max
f〈wm ; xii 〉g
min
f〈wm ; xii 〉g
xjii A M; ji a jk
xii A M; ji ¼ jk j
ð7Þ
j
node and compute 〈w; xbb 〉 ¼ minf〈w; x〉g; then, jb is the class label xAM
of the node. Set θ ¼ 1 as the threshold of the node, and assign N1 as the serial number to this node. Step 6 Adaptive nodes work in accordance with their serial number, ranging from small to large; their working function is given by ( jn Z n 40 ΩðZ n Þ ¼ ð8Þ 0 Zn o 0 where Z n ¼ wn I n θn , I n is the input, n ¼ 1; ⋯; N 1 is the serial number, and jn ¼ 1; 3 is the class label. The threshold θn can be changed by θ′n ¼ θn þ λn and is set λn to zero by default. 2.4. Adaptive recognition As its structure and function are determined, the adaptive recognition layer can recognise pixels in the blurred set and divide
D. Fu et al. / Optics & Laser Technology 56 (2014) 102–106
them into target pixels or background pixels based on template features; therefore, the immune network template algorithm can be used to extract targets from a blurred infrared image. Inputting a blurred infrared image of a hand trance to the immune template network, the output is the extraction result.
105
arrive in one second. The infrared image that we often obtain is Fig. 4(b), which is a blurred infrared image, and its grey level does not accurately reflect the contour of the hand trace. The purpose of this paper is to extract an accurate hand contour from Fig. 4(b). To evaluate the extraction results, we use the manually processed result as the ground truth, as shown in Fig. 4(c).
3. Blurred infrared image and ground truth Under laboratory conditions, we shot hand trace infrared images one second and two minutes after a hand left a board, as shown in Fig. 4(a) and (b), respectively. Fig. 4(a) can reflect the touching contour between the hand and the board, but Fig. 4 (a) usually cannot be obtained in a crime scene because we cannot
4. Experimental results The proposed immune network template algorithm has been simulated in a MATLAB programme environment for target extraction from a hand trace infrared image.
Fig. 4. Blurred infrared images of hand traces shot at different times and the ground truth. (a) The hand leaves for 1 second. (b) The hand leaves for 2 min. (c) The Ground Truth.
Fig. 5. The extraction results. (a) Original image, (b) Prewitt, (c) Canny, (d) deformable template, (e) spatial matrix template, (f) artificial immune activated neural network (AIANN), (g) immune kernel clustering network(IKCN) and (h) coordination immune network template algorithm(CINTA) ðλ1 ¼ 0; λ2 ¼ 0Þ. (i) CINTA ðλ1 ¼ 0:38; λ2 ¼ 0Þ.
106
D. Fu et al. / Optics & Laser Technology 56 (2014) 102–106
Table 1 Extraction results analysis. Algorithm
TPR
J ðGT;SIÞ
DðGT;SIÞ
r err
Spatial matrix template AIANN IKCN CINTA ðλ1 ¼ 0Þ CINTA ðλ1 ¼ 0:38Þ
0.3204 0.2735 0.2735 0.6645 0.7389
0.2612 0.2228 0.2230 0.6162 0.6384
0.4143 0.3644 0.3647 0.7644 0.7793
32.15% 33.87% 33.82% 14.55% 14.71%
Fig. 5 shows the extraction results of different algorithms for the hand trace infrared image: (a) is the original image, (b) is the extraction result of the Prewitt template, (c) is the extraction result of the Canny template, (d) is the extraction result of the deformable template, (e) is the extraction result of the spatial matrix template, (f) is the extraction result of artificial immune activated neural network (AIANN), (g) is the extraction result of the immune kernel clustering network (IKCN), (h) is the extraction result of the coordinated immune network template algorithm (CINTA) ðλ1 ¼ 0; λ2 ¼ 0Þ and (i) is the extraction result of coordinated immune network template algorithm (CINTA) ðλ1 ¼ 0:38; λ2 ¼ 0Þ. It can be observed that in the cases with large unknown regions, the Prewitt, Canny and deformable template perform poorly, and cannot extract the hand area from the image. The spatial matrix template, AIANN and IKCN perform better, but they still produce inaccurate boundary contours. On the contrary, the method we proposed, CINTA, can produce accurate target contours. To evaluate the extraction performance quantitatively, the extraction algorithms are evaluated with the following parameters: true positive rate (TPR) [20], the Jaccard similarity index (J ðGT;SIÞ ) [21], the Dice similarity index (DðGT;SIÞ ) [22] and the absolute error ratio [23]. Table 1 presents the quantitative evaluation results. The TPR of the spatial matrix template, AIANN and IKCN, are much smaller than our algorithm, which indicates that the target extraction rate of these methods is far smaller than our algorithm. The J ðGT;SIÞ and DðGT;SIÞ of these three methods are much smaller than our algorithm, which indicates that overlap ratio between the ground truth and the extracted result of these three methods is far lower than the overlap degree of our algorithm. The r err of these three methods is larger than our algorithm, which illustrates that the absolute extraction error of these three methods is larger than that obtained with our algorithm. The default value of λ1 in our algorithm is 0. When we reduce the value of λ1 , such as setting λ1 ¼ 0:38, the extraction result of our algorithm will have higher TPR, J ðGT;SIÞ , DðGT;SIÞ and r err . Overall, after analysing the extraction performance of the above mentioned algorithms based on the proposed quantitative parameters, CINTA provides the best result for extracting target regions from a blurred infrared image. 5. Conclusions In this paper, a coordinated immune network template algorithm based on the immune coordination mechanism was proposed. This paper investigated the working principle of immune system; the pathogen is processed by the innate immunity first and later by the adaptive immunity. An immune network model according to the immune operating principle was presented in this paper. This model contains three layers: the innate feature transformation layer, the innate recognition layer and the adaptive recognition layer. Each pixel in a blurred infrared image of a hand trace was handled by these three layers and was finally divided into two classes, the target pixel or the background pixel. Compared to the previously reported classical template methods and immune network template methods, the coordinated
immune network template algorithm presents a better target extraction rate and overall extraction accuracy while reducing the absolute error rate significantly. From the obtained results, it is evident that the proposed extraction technique yields better extraction performance for hand trace infrared images. The algorithm is directly used for a blurred infrared image of hand trace that is shot under lab conditions. Studying the practical significance and performance of the algorithm for blurred infrared images of traces that are captured during crime scenes will be the focus of future work.
Acknowledgements This research is supported by the National Natural Science Foundation of China (No. 61272358) and the international technological project between China and the Czech Republic (No. 39-10). References [1] Bhargava Rohit, Schwartz Perlman Rebecca, Daniel C. Non-invasive detection of superimposed latent fingerprints and inter-ridge trace evidence by infrared spectroscopic imaging. Analytical and Bioanalytical Chemistry 2009;394 (8):2069–75. [2] Van Lersel Miranda. Infrared photography aids forensic research. TNO Magazine 2010;1:6–7. [3] Xiong ZL, Yang KT, Ding WX. Computational and experimental research on infrared trace by human being contact. Applied Optics 2010;49(18):3587–95. [4] Roberts LG. Machine perception of three-dimensional solids.Cambridge: MIT Press; 1965. [5] Prewitt JMS. Object enhancement and extraction.New York: Academic Press; 1970. [6] Sobel IE. Camera models and machine perception. Stanford University; 1970 (Ph.D. dissertation). [7] Canny J. A computational approach to edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 1986;8(6):679–98. [8] Garrido A, de la Blanca NP. Applying deformable templates for cell image segmentation. Pattern Recognition 2000;33(5):821–32. [9] Yoon Young-Geun, Lee Seok-Lyong, Chung Chin-Wan. An effective defect inspection system for polarized film images using image segmentation and template matching techniques. Computers and Industrial Engineering 2008;55(3):567–83. [10] Bhanu B, Fonder S. Functional template-based SAR image segmentation. Pattern Recognition 2004;37(1):61–77. [11] Bo Hua, Ma Fulong, Han Baojun SAR. Image segmentation based on immune algorithm. In: Proceedings of international conference on control and automation, Xi an, China; 2005, p. 375–378. [12] Akmal Younis Mohamed, Ibrahim Mansur, Kabuka Nigel John. An artificial immune-activated neural network applied to brain 3D MRI segmentation. Journal of Digital Imaging 2008;21(1):69–88. [13] Huang Wenlong, Jiao Licheng. Artificial immune kernel clustering network for unsupervised image segmentation. Progress in Natural Science 2008;18 (4):455–61. [14] Suzuki N, Suzuki S, Millar DGA. Critical role for the innate immune signaling molecule IRAK-4 in T cell activation. Science 2006;311(5769):1927–32. [15] Iwasaki1 A, Medzhitov R. Regulation of adaptive immunity by the innate immune system. Science 2010;327(5963):291–5. [16] De Castro, Timmis J. Artificial immune systems: a new computational intelligence approach.London: Springer-Verlag; 2002. [17] Campelo F, Guimaraes FG, Igarashi H, Ramirez JA, Noguchi S. A modified immune network algorithm for multimodal electromagnetic problems. IEEE Transactions on Magnetics 2006;42(4):1111–4. [18] PAL M. Artificial immune-based supervised classifier for land-cover classification. International Journal of Remote Sensing 2008;29(8):2273–91. [19] Xue JingHao Titterington Michael. T-tests, f-tests and otsu's methods for image thresholding. IEEE Transactions on Image Processing 2011;20(8):2392–6. [20] Jadin MS, Taib S. Infrared image enhancement and segmentation for extracting the thermal anomalies in electrical equipment. Electronics and Electrical Engineering 2012;4:107–12. [21] Rub E.C. Multimodal evaluation for medical image segmentation. In: Proceedings of international conference on computer analysis of images and patterns, Vienna, Austria;2007. p. 229–236 [22] Babalola KO, Patenaude B, Aljabar P. An evaluation of four automatic methods of segmenting the subcortical structures in the brain. Neuroimage 2009;47 (4):1435–47. [23] Wenbing Tao, Hai Jin, Liman Liu. Object segmentation using ant colony optimization algorithm and fuzzy entropy. Pattern Recognition Letters 2007;28(7):788–96.