- Email: [email protected]
Real-time visual enhancement for infrared small dim targets in video
Infrared Physics & Technology 83 (2017) 217–226
Contents lists available at ScienceDirect
Infrared Physics & Technology journal homepage: www.elsevier.com/locate/infrared
Regular article
Real-time visual enhancement for infrared small dim targets in video Xiaoliang Sun a,⇑, Xiaolin Liu b, Zhixuan Tang c, Gucan Long a, Qifeng Yu a a
College of Aerospace Science and Engineering, National University of Defense Technology, Changsha, China College of Mechatronics Engineering and Automation, National University of Defense Technology, Changsha, China c Reserve Officers Training Office of Ningxia University, Yinchuan, China b
h i g h l i g h t s Temporal cues are incorporated to perform visual enhancement. The target intensity is enhanced via accumulating along the trajectory extracted by DPA. An adaptive manner is presented to eliminate the sharp edge in merging. The target’s prior shape information is adopted in clutter suppression and adaptive merging.
a r t i c l e
i n f o
Article history: Received 3 October 2016 Revised 20 April 2017 Accepted 2 May 2017 Available online 8 May 2017 Keywords: Infrared dim small target Visual enhancement Dynamic programming algorithm Energy accumulation Gaussian
a b s t r a c t Visual enhancement for infrared small dim targets is a standing problem in infrared image processing. Existing approaches cannot enhance the target well and suppress the background simultaneously, especially for targets which are so faint that they are hardly visible. This paper proposes a novel real-time visual enhancement algorithm for infrared small dim targets in video by introducing temporal cues. In this work, Dynamic Programming Algorithm (DPA) is used to detect the target’s trajectory in the video and the target is enhanced through energy accumulation along the trajectory. The shape prior of the small dim target is adopted for background suppression and adaptive merging. Experimental results on real infrared small dim target videos indicate that the proposed algorithm can improve the visual quality of these types of images notably, especially for cases in which the target is hardly visible. In addition, the proposed algorithm takes on average 8.35 ms to process a 320 ⁄ 256 image, and thus meets the needs of real-time applications. Ó 2017 Elsevier B.V. All rights reserved.
1. Introduction Due to its advantages of all-weather operation, detecting and anti-jamming abilities, infrared imaging equipment has been widely used in military reconnaissance, traffic monitoring, weapon guidance, space target surveillance, etc. This paper focuses on the visual enhancement of small dim infrared targets in video captured during space target surveillance in order to obtain high-quality images. In space target surveillance, infrared imaging equipment is often used in long-distance space target detection to compensate for the deficiency of visible light equipment. As a result of long distance imaging, the target’s energy is affected heavily by air turbulence, scattering, etc., and thus it is weak, while the Signal to Noise Ratio (SNR) of the captured image is low. This results in low quality images which are difficult to interpret. Visual enhancement for
⇑ Corresponding author. E-mail address: [email protected] (X. Sun). http://dx.doi.org/10.1016/j.infrared.2017.05.002 1350-4495/Ó 2017 Elsevier B.V. All rights reserved.
infrared small dim target has a great significance in extending the observation distance of infrared imaging equipment. Infrared image enhancement has been studied for a long time. Traditional methods use the global information of the image to perform the enhancement, e.g. histogram-based methods, gradient domain-based methods, etc. These enhancement methods are not specifically designed for infrared images with small dim targets. In these types of images, the target constitutes only a small part of the infrared image while the background is the dominant component. Global information-based algorithms tend to over-enhance the background but the target is not highlighted properly. Infrared small dim target detection has been well studied. Target enhancement (e.g. mathematical morphology-based algorithms, filteringbased algorithms, etc.) is often used as a preprocessing step during detection to improve the Signal-to-Clutter Ratio (SCR) of the local region which contains the target. However, it does not focus on improving the visual quality of the infrared small dim image. The background is often completely removed in the enhanced result
218
X. Sun et al. / Infrared Physics & Technology 83 (2017) 217–226
and in turn the result cannot be visually interpreted. To the best of our knowledge, no existing visual enhancement algorithm is specially designed for infrared small dim target in video, especially for extreme cases in which the targets are so faint that hardly visible. Additionally, existing enhancement algorithms mainly depend on spatial cues captured from a single image. However, spatial cues may be inadequate to perform visual enhancement for extreme cases in which targets are so weak that hardly visible. Therefore, existing algorithms do not improve the visual quality of infrared small dim target images in a satisfactory manner. Temporal cues in image sequences should be introduced in visual enhancement for infrared small dim target. This paper tackles the problem of visual enhancement of infrared small dim targets in video by using the spatial and temporal information contained in the image sequence and the prior of the target shape. We attempt to improve the visual quality of the image further by enhancing the target and at the same time appropriately suppressing the background. In typical images of this type, the target’s energy is very weak as a result of long-distance imaging, and, in extreme cases, the observer cannot distinguish the target from the background. Determining by the characteristic of the infrared sensor, the small dim target is often modeled as a Gaussian spot. Based on the above analysis, this paper proposes a new visual enhancement algorithm for infrared small dim targets in video. First, we adopt DPA to track the target and enhance the latter through accumulation of the target-centered local regions along its trajectory. Next, the accumulated target-centered local region is weighted by a Gaussian mask to suppress clutter around the target. Finally, the processed target-centered local region is embedded back into the input image in an adaptive merging manner. During this merging operation, the intensities of the pixels in the local region are selected as adaptive weights. We evaluated the proposed algorithm on real infrared videos including small dim targets. Comparisons between our algorithm and typical enhancement algorithms are also performed. Experimental results show that the proposed algorithm can notably enhance the target and suppress the background appropriately. The visual quality of the infrared video was markedly improved, even for videos in which the targets are so faint that hardly visible. The rest of the paper is organized as follows: related works are summarized in Section 2. Section 3 contains an analysis of the characteristics of the target and the background in infrared image sequences containing small dim targets. We explain the proposed algorithm in detail in Section 4, while in Section 5 experimental results are presented. Finally, Section 6 concludes the paper.
2. Related works Histogram Equalization (HE) [1] is widely used in infrared image enhancement. However, it is unsuitable for the enhancement of infrared small dim target images, as the background is the dominant component of the image and occupies most of the valid gray levels, while the target only uses few gray levels. Thus, equalizing the histogram of the whole image causes the overenhancement of the background. Plate Histogram Equalization (PHE) [2] introduces a plate value to control the equalization. Contrast Limited Adaptive Histogram Equalization (CLAHE) [3] performs the plate histogram equalization in local parts of the input infrared image. Modifications to HE, PHE, CLAHE, etc., also tend to over-enhance the background. Hu et al. [4] adopted adaptive median and homomorphic filtering for infrared image enhancement. In [5], Kim et al. performed contrast enhancement in the gradient domain. Their algorithm enhances local contrast and preserves global contrast simultaneously.
The wavelet transformation plays an important role in image processing. Zhang et al. [6] introduced the Discrete Stationary Wavelet Transform (DSWT) to enhance the contrast of infrared image. Wang et al. [7] suppressed noise by manipulating the wavelet coefficients. Then, a nonlinear transformation was applied to the wavelet coefficients based on Weber theory to enhance the target. The wavelet transformation based methods suffer from the ringing effect and the under-enhancement of the details. The contourlet transformation is more powerful than the wavelet transformation in representing contours and textures. Shi et al. [8] use the contourlet transformation to decompose the infrared image at different scales and directions. The beta function is applied in the low frequency domain to enhance global contrast. A non-linear gain function is adopted to process the coefficients at the different scales. The enhanced image is obtained through the transformation of the processed coefficients back to the spatial domain. Lee et al. [9] tackled infrared small dim target enhancement from the perspective of saliency analysis. The authors enhanced the target based on the assumption that the target is brighter that the surrounding background and that the target’s shape can be modeled as a Gaussian distribution. The enhanced result was obtained by merging the temperature and the Difference of Gaussian (DoG) maps. Motivated by models of the visual attention mechanism, Qi et al. [10] introduced the quaternion Fourier transformation in infrared small dim target enhancement. As mentioned in Section 1, enhancement is often treated as a preprocessing step in infrared small dim target detection. Mathematical morphology operations are simple and have been widely used in small infrared target processing [11–15]. Zhou et al. [12] used the top-hat filter group to enhance the target and suppress the background at the same time. Bai et al. [11] extended the top-hat transformation to the top-hat selection transformation for infrared small dim target enhancement, where the parameters used in the transformation were calculated based on the properties of the target region. In [15], the authors pointed out that in infrared images the target’s intensity is higher than that of the surrounding background’s, and designed a structuring element used in erosion and dilation specifically for infrared small target enhancement. The hit-or-miss transformation [13] and toggle contrast operator [14] have also been used to enhance infrared small dim target images. A local kernel method for small infrared target detection was proposed in [16]. The authors trained a weighted local model to preserve background while removing target in infrared images. In order to suppress the strong edges in infrared images, modifications [17,18] were made on the infrared patch image model. Dai et al. [17] incorporated structural prior information in separating small target from background. The weight is adaptive for each column according to structure prior in the weighted infrared patch image model. [18] adopted non-negative constraint and separated the target from the background via minimizing the partial sum of singular values. Xie et al. [19] proposed a novel algorithm called accumulated center-surround difference measure to detect small target in heterogeneous area. These algorithms are designed to improve the performance of infrared small dim target detection, not the visual quality of the input image. The algorithms discussed above perform target enhancement using cues from a single image. However, the infrared small dim target may be so faint that hardly visible to the observer in extreme cases, the above mentioned algorithms may not yield satisfactory results. What’s worse, the small dim target may be lost in the output enhanced result. The spatial cues from a single image are inadequate for the visual enhancement of infrared small dim target, especially for hardly visible small dim targets. The visual enhancement for infrared small dim targets may benefit from the temporal cues contained in image sequences. Temporal cues have been widely used in visual target tracking [20,21]. Unfortunately, the
219
X. Sun et al. / Infrared Physics & Technology 83 (2017) 217–226
temporal cues have not attracted enough attentions in previous visual enhancement works. As a preprocessing step of infrared small target detection, [22–24] introduced temporal target information in enhancement. [22,23] assumed that small targets are moving in a specific direction. They have difficulties in detecting non-moving targets. In order to solve weaknesses in [22,23], Bae [24] used a 1D temporal bilateral filter to extract temporal target information. The method is not efficient under infrared images including hardly visible small dim targets. The temporal cues have been widely used in infrared small dim target detection. The Tracking-Before-Detection (TBD) algorithm first determines all possible target trajectories in an image sequence, and the true trajectory is selected through examination of the trajectory candidates. The TBD algorithm demonstrates excellent performance in infrared small target detection, especially for image sequences with low SNR. As a powerful tool, DPA was used in TBD by Barniv et al. [25,26] to reduce computational load. Arnold et al. [27] modified traditional DPA by introducing a new track scoring function and an intra-frame search procedure. In [28], the authors used the amplitude of the target signal to construct an index function, and, subsequently, several works [29– 31] on index function modification and preprocessing of the observed data were published recently. By selecting a number of points located in the target region used in energy accumulation, Zhang et al. [30] improved the performance of the detection algorithm significantly. Drawing lessons from TBD algorithms for infrared small dim target detection, this paper adopts DPA to track the small dim target in infrared video. The visual enhancement of the infrared small dim targets is performed along the extracted trajectory. 3. Analysis of infrared small dim target images This paper focuses on the enhancement of infrared small dim target videos captured in long distance observation. Fig. 1 shows a typical frame chosen from such a video. The dim target is indicated by a rectangle in Fig. 1(a) and the 3D display of the local region is shown in Fig. 1(b). Obviously, in this case the target is very faint and not easy to observe directly. Based on existing works, we model the target image consisting of noise, background and the target. The spatial and temporal model of the video can be expressed as
Iðx; y; kÞ ¼ Nðx; y; kÞ þ Bðx; y; kÞ þ Tðx; y; kÞ
ð1Þ
where k is the frame index. ðx; yÞ is the pixel’s coordinates, while Nðx; y; kÞ, Bðx; y; kÞ and Tðx; y; kÞ denote noise, background and target gray values, respectively. Previous works [31–33] modeled the noise as a zero-mean Gaussian distribution in the spatial and temporal domain, i.e.
Nðx; y;kÞ Nð0; r2N Þ
ð2Þ
where rN is the standard deviation determined by the infrared imaging sensor. The background Bðx; y; kÞ is often modeled as being heavily correlated and a large area component in the spatial domain but is assumed to be relatively stable in the temporal domain. Therefore, Bðx; y; kÞ can be modeled as
Bðx; y;kÞ ¼ Cðx; yÞ þ NB ðx; y; kÞ
ð3Þ
where Cðx; yÞ denotes the intensity of the background and NB ðx; y; kÞ Nð0; r2B Þ is the noise. The target is often presented as a bright spot in the image, as shown in Fig. 1. We adopt the model given by [32] and consider the target as a 2D Gaussian distribution in the spatial domain:
tðx; yÞ ¼
D
2pr2
2
ðx iÞ þ ðy jÞ exp 2r2
2
!
ð4Þ
where r is determined by the infrared sensor’s Point Spread Function (PSF). D denotes the target intensity at ði; jÞ. The temporal and spatial model of the target is
Tðx; y; kÞ ¼ tðx; y; k; hÞ
ð5Þ
In this case, h is a parameter related to target’s velocity, the infrared sensor’s PSF, etc. The target is often assumed to be brighter than the surrounding background. This assumption is fundamental to many algorithms, and also forms the basis of the algorithm presented here. This paper validates this assumption using the Negative Laplace of Gaussian (NLoG) operator to filter real infrared small dim target images. Using this approach, bright spots which satisfy the assumptions should produce high values in the response map. Some results are listed in Fig. 2, where the small target is indicated by a rectangle. The response maps in the bottom row of Fig. 2 show that the targets give rise to high response values in the filtered results, thus verifying the rationality of the assumptions about the target. The example images listed in Fig. 2 contain small targets that are visible to the observer. NLoG performs well in detecting these targets. However, our investigations showed that NLoG has trouble
Fig. 1. Infrared small dim target image example. (a) Original image. (b) 3D display of the local region.
220
X. Sun et al. / Infrared Physics & Technology 83 (2017) 217–226
Fig. 2. Infrared small dim target images (top row) and corresponding NLoG response maps (bottom row).
in detecting target in low SNR images, as this method only uses spatial information from a single image and cannot detect small dim targets reliably, especially for low SNR images. Therefore, temporal cues should be introduced to enhance the target. The motion of the small dim target within the image is approximate to the translation motion in long distance imaging. Following TBD, we sum the target-centered local regions, each denoted as lðx; y; kÞ, along the target’s trajectory within a time window of length X as
Lðx; yÞ ¼
X lðx; y; kÞ
ð6Þ
k2X
In this manner, zero-mean noise is suppressed and the target is enhanced significantly in Lðx; yÞ. When performed on infrared small dim target detection, this preprocessing tends to further suppress the background and noise, or even remove them completely, as shown in Eq. (7).
Lðx; yÞ ¼ T 0 ðx; yÞ
velocity in the previous frame. This target movement can be viewed as a Markov process, and DPA can be used to obtain the target’s trajectory efficiently through iterative searches. The target is presented as a bright spot in the captured image and its energy is dispersed within a certain region. We adopt the improved DPA presented in [28], which uses multi-point accumulation. In order to reduce the computational load, the local regions considered in the present approach have a square shape. Let Rt , with dimensions ð2r t þ 1Þ ð2r t þ 1Þ and Rtþb , with dimensions ð2r tþb þ 1Þ ð2rtþb þ 1Þ denote the target region and the targetcentered local region, respectively. The background in Rtþb is denoted as Rb , which is equal to ðRtþb Rt Þ. As noted in Section 2, we assume that the target is brighter than the surrounding background and that its shape can be approximated by a Gaussian function. This paper uses the intensity difference zðx; y; kÞ between Rt and Rb to define the merit function:
X X zðx; y; kÞ ¼ ð Iðx; y; kÞ Iðx; y; kÞÞ
ð7Þ
where T 0 ðx; yÞ is the aggregate of the target’s energy. The enhanced result, however, may not be suitable for the observer. This paper aims to improve the visual quality of infrared images containing small dim targets. Therefore, we seek to enhance the target and suppress the background and noise appropriately, and then merge them smoothly with the final enhanced image. This is achieved through the adaptive merging method proposed below. 4. Methodology Based on the analysis presented above, a new visual enhancement algorithm for infrared small dim targets contained in video frames is presented in this section. The algorithm consists of three main parts: energy accumulation along the target trajectory, background and noise suppression and adaptive merging. The flow chart of the proposed algorithm is shown in Fig. 3.
Rt
An iterative equation can then be defined as
FðPk Þ ¼ max½FðPk1 Þ þ zðx; y; kÞ Pk1
wk ðPk Þ ¼ argmax½FðPk1 Þ
ð9Þ
P k1
F denotes the merit-function, P k is the pixel value in frame k, while wk ðPk Þ indicates the trajectory which maximizes F. Based on the assumptions about the target, F reaches its maximum at the center of the target, but there no assumptions are made about the moving model of the target. The upper bound of the target’s velocity is determined by the radius rs of the search region. The duration of the time window is X, within which the trajectory with the maximum F is stored in w. The energy accumulation of the target-centered local region sequence along w is determined using Eq. (6). The local region’s size is ROI w ROI h. The SNR of the accumulated region is used to measure the confidence of the accumulated result, and for a point target it is defined as [34]:
4.1. Energy accumulation along the target trajectory
SNR ¼ 10lg The video considered in this discussion is assumed to have been captured during long distance imaging, and includes a small dim target moving slowly in the image. The position of the target in current frame is only determined based on target’s position and
ð8Þ
Rb
T B
rn
ð10Þ
where T and B denote the intensity of the target and the background, respectively, while rn is the standard deviation of the noise. In this paper, we use the following approximations:
X. Sun et al. / Infrared Physics & Technology 83 (2017) 217–226
221
Fig. 3. The flow chart of the proposed algorithm. (a) Original image. (b) Energy accumulation. (c) Gaussian mask. (d) Background and noise suppression. (e) Adaptive merging.
T ¼ mean ðLðx; yÞÞ ðx;yÞ2Rr t
B ¼ mean ðLðx; yÞÞ ðx;yÞRRtþb
ð11Þ
rn ¼ stdev ðLðx; yÞÞ ðx;yÞRRtþb
If the SNR of the accumulated region is greater than a pre-set threshold ThSNR, the accumulated result is considered reliable, and unreliable otherwise. If the accumulated result is unreliable, the energy accumulation process will be reapplied. The algorithm for DPA based energy accumulation is summarized in Algorithm 1. Algorithm 1. DPA based energy accumulation for k = 0 to Video length do if k ¼¼ 0 Initialize the merit-function FðP 0 Þ ¼ zðx; y; 0Þ and the backtracking function w0 ðP0 Þ ¼ ðx; yÞ for all pixels in the input image, zðx; y; 0Þ is calculated as Eq. (8) else if k P 1 Perform the iteration according to Eq. (9) if k P X Detect the pixel with the maximum F and perform the accumulation of the target centered local region along w Calculate the SNR of the accumulated region using Eqs. (10) and (11) if SNR > ThSNR Output the accumulated region else Reinitialize the whole process by setting the current frame as the first frame and k = 0 end if end if end if end for
4.2. Background and noise suppression As mentioned in Section 3, infrared small dim target images comprise the target, background and noise. The noise is modeled as a zero mean Gaussian distribution. Ideally, if X is infinitely long,
the noise can be completely removed from the accumulated result. In practice, as the background is slowly varying over frames, the noise does not fully behave as modeled, X is not infinitely long, etc. As a result, the background and noise are also enhanced during the accumulation process result, as shown in Fig. 3(b). The target is located at the center of the accumulated region. We use a 2D Gaussian distribution to approximate the target shape [26], while suppressing the background and noise in the accumulated region by weighting Lðx; yÞ using a Gaussian mask f ðx; yÞ with a size of ROI w ROI h:
f ðx; yÞ ¼
1 ððxx0 Þ2 þðyy0 Þ2 Þ=ð2r2f Þ e 2pr2f
ð12Þ
ðx0 ¼ ROIw =2; y0 ¼ ROIw =2Þ is the center of the mask, while rf denotes the standard deviation, which depends on the target’s size and is determined according to the 3 r criterion. The weighted result L0 ðx; yÞ is obtained by
L0 ðx; yÞ ¼ f ðx; yÞLðx; yÞ
ð13Þ
As Fig. 3(c) shows, using this approach, the background and noise are suppressed effectively and the target is preserved in L0 ðx; yÞ. 4.3. Adaptive merging To complete the visual enhancement of the target, the accumulated region L0 ðx; yÞ should be embedded back into the original image Iðx; yÞ. The target-centered local region and the remaining part of Iðx; yÞ are processed separately. If we use L0 ðx; yÞ to replace the corresponding part lðx; yÞ in Iðx; yÞ, sharp artifacts will occur across the boundary of L0 ðx; yÞ. The example image shown in Fig. 4(a) was selected from an infrared small dim target video. To illustrate the merging process, only part of the example image is selected and zoomed in for display. The dim target in the example image is practically invisible to the observer, but the local region used for accumulation is depicted by a rectangle. The image in Fig. 4(b) was obtained by replacing the corresponding part lðx; yÞ by L0 ðx; yÞ, and it is evident that the sharp edges in degrade the visual quality of the image. This paper proposes an adaptive merging method to tackle this problem. The enhanced target should be embedded back into the input image smoothly, while the target’s energy should be preserved for the final output. Therefore, the weights used during merging should vary according to the intensity of the target. When considering the distribution of the intensity in L0 ðx; yÞ, we take the normalized L0 ðx; yÞ, denoted by L0N ðx; yÞ, as the weighting factor for
222
X. Sun et al. / Infrared Physics & Technology 83 (2017) 217–226
Fig. 4. Example of adaptive merging. (a) Sample image. (b) Direct replacement. (c) Adaptive merging.
adaptive merging. The adaptive merging of L0 ðx; yÞ and lðx; yÞ is expressed as 0
l ðx; yÞ ¼ L0N ðx; yÞL0 ðx; yÞ þ ð1 L0N ðx; yÞÞlðx; yÞ 0
ð14Þ
0
l ðx; yÞ is the merged result. By placing l ðx; yÞ back to Iðx; yÞ, the final enhancement result is obtained. As shown in Fig. 4(c), the enhanced target is smoothly embedded into Iðx; yÞ. 5. Experiments 5.1. Experimental setup To validate the performance of the proposed algorithm qualitatively and quantitatively, a large number of experiments was conducted on real infrared small dim target videos. As mentioned above, there is no existing specially designed visual enhancement algorithm for infrared small dim target. We compared our algorithm with relevant typical enhancement algorithms for infrared small dim target images, including PHE [2], CLAHE [3] and MTH [11]. PHE and CLAHE are widely used for infrared image enhancement, while MTH is a typical algorithm used as a preprocessing method in infrared small target detection. For PHE, CLAHE and MTH, the parameters suggested by the authors were used. PHE, CLAHE and MTH perform the enhancement using spatial cues from a single image. They were applied on real videos frame by frame. The resolution of the image used in this paper was 320 ⁄ 256. For the proposed algorithm, the parameter settings used in all experiments were: the radius r t of the target region was 5 pixels, r tþb was 11 pixels, the search radius r s was 3 pixels, the time window X had a length of 10 frames, the size of the accumulation region lðx; yÞ was 60 ⁄ 60, while the SNR threshold was set to 4.77 dB. 5.2. Results We mainly reported experimental results on two representative videos from the tested infrared small dim target data, named Seq. 1 and Seq. 2. The length of Seq. 1 was 209 frames. The target in Seq. 1 is so faint that hardly visible. Seq. 2 was longer than Seq. 1, containing 3009 frames. As the sequence progresses, the target in Seq. 2 is gradually fading, becoming hardly visible during the latter part of the sequence. The presented results include the target trajectories extracted by DPA, visual comparisons, improvements of the SNR of the target-centered local region and the target area’s mean intensity, which was calculated in region Rt . The targetcentered local region’s size was ROI w ROI h. The target trajectory extraction is the key step of our algorithm and the energy accumulation is performed along the extracted tar-
get trajectory. Fig. 5 presents parts of the target trajectories (labeled by white crosses) extracted by DPA for Seq. 1 and Seq. 2. The lengths of the shown trajectories in Fig. 5(a) and (b) were 40 (11th–50th frame) and 140 (1960th–2099th frame). The movement of the target was arbitrary and DPA extracted the target trajectory reliably as Fig. 5 shows. Due to the long distance imaging, the target moved slowly in the image domain. The target in Seq. 2 moved slower than the one in Seq. 1 as shown in Fig. 5. For qualitatively evaluation, the enhancement results of several sample frames picked from Seq. 1 and Seq. 2 are shown in Figs. 6 and 7 for visual comparison. As was the case in Fig. 4, only part of the zoomed-in input images are shown in Figs. 6 and 7. The targets were labeled by rectangles. As Fig. 6(a) shows, the original target in Seq. 1 is hardly visible. The original target in Seq. 2 is gradually weakening as shown in Fig. 7(a), and hardly visible in the 2975th frame. The enhancement results shown in Figs. 6 (b) and 7(b) indicated that the targets were enhanced effectively through energy accumulation. Benefitting from the adaptive merging process, the enhanced target-centered local region was embedded back into the input image smoothly. The proposed algorithm improved the visual quality of infrared small dim target images notably. We also presented the visual comparison between PHE, CLAHE, MTH and the proposed algorithm. Sample images picked from the test infrared small dim target data were listed in Fig. 8, with the targets denoted by rectangles. As explained in Sections 1 and 2, the background is the dominant part of the image. PHE and CLAHE over-enhanced the background and degraded the visual quality of the image, as shown in Fig. 8(b) and (c), respectively. MTH estimated the background using the modified top-hat transformation where the target was extracted by subtracting the estimated background from the input image. In the corresponding result, the target’s intensity did not exceed its original value, which means that if the target is so faint that it is hardly visible, the MTH algorithm cannot produce a satisfactory result. As shown in Fig. 8(d), the background was almost completely removed from the result but the target was not enhanced effectively. This paper focuses on the visual enhancement for infrared dim small targets. In extreme cases, the target is almost flooded by the background, and information from a single image may not be enough to achieve satisfactory visual enhancement. The PHE, CLAHE and MTH algorithms are applied on single input images, and thus perform poorly in extreme cases. In the example image shown in the third column of Fig. 8, the target is invisible to observer, and PHE, CLAHE and MTH all failed to enhance the target. Our algorithm enhanced the target’s intensity through energy accumulation along the target’s trajectory, achieving notable
223
X. Sun et al. / Infrared Physics & Technology 83 (2017) 217–226
(b) Enhancement result
(a) Original images
Fig. 5. Target trajectory extracted by DPA.
27th frame
121th frame
201th frame
Fig. 6. Visual comparison for Seq. 1.
results as shown in Fig. 8(e), even for the extreme case. Based on the assumptions about the target’s shape, the accumulation region was embedded back into the input image smoothly, and the visual quality of the infrared small dim target video was markedly improved. For quantitatively evaluation, the improvements of the SNR of the accumulation region and the target area’s mean intensity before and after enhancement on Seq. 1 and Seq. 2 are given in Figs. 9and 10. In Seq. 1, the target is almost drowned out by the background in the original video, while the SNR of the targetcentered local region is low and unstable. The target area’s mean intensity is approximately equal to that of the background. As mentioned above, PHE and CLAHE tended to over-enhance the background. Although the target intensity is enhanced, the SNR of the local region is degraded as shown in Fig. 9. MTH degraded the target intensity and the SNR of the local region. Our algorithm performs better in comparing with PHE, CLAHE and MTH. As illustrated by the results in Fig. 9, the target intensity is enhanced
greatly by the proposed algorithm, while the SNR of the local region is notably improved and remains stable throughout the whole video. The evaluation results for Seq. 2 are presented in Fig. 10. The target area’s mean intensity for the original video shown in Fig. 10 clearly shows the change of the target intensity from strong to weak in the original videos. The SNR of the local region is low. Similar to the results in Fig. 9, PHE and CLAHE enhanced the target intensity notably, however, they degraded the SNR of the local region as a result of the over-enhancement of the background. MTH subtracted the estimated background from the original image. It reduced the target intensity and it had difficulties in enhancing hardly visible target as shown in Fig. 10. The target enhanced by our algorithm demonstrates a high and stable intensity, with a significant improvement in the SNR of the targetcentered local region. As evident from Figs. 9 and 10, the proposed algorithm improved the visual quality of infrared small dim target image
224
X. Sun et al. / Infrared Physics & Technology 83 (2017) 217–226
Fig. 7. Visual comparison for Seq. 2.
Fig. 8. Visual comparison between typical methods and the proposed algorithm. (a) Original image. (b) PHE. (c) CLAHE. (d) MTH. (e) Proposed algorithm.
225
X. Sun et al. / Infrared Physics & Technology 83 (2017) 217–226
SNR(dB)
Target area mean intensity
25
200
20
180
15
160
10
140
5
120 100
0
80
-5
60
-10
0
1 9 17 25 33 41 49 57 65 73 81 89 97 105 113 121 129 137 145 153 161 169 177 185
20
-20
Original
PHE
CLAHE
MTH
Proposed
1 9 17 25 33 41 49 57 65 73 81 89 97 105 113 121 129 137 145 153 161 169 177 185
40
-15
Original
PHE
CLAHE
MTH
Proposed
Fig. 9. Improvements of the SNR and the target area’s mean intensity for Seq. 1.
Fig. 10. Improvements of the SNR and the target area’s mean intensity for Seq. 2.
Table 1 Computation time comparison.
Code Time (ms)
PHE
CLAHE
MTH
Proposed
C++ 2.68
C++ 3.51
Matlab 47.55
C++ 8.35
rences are treated as targets and preserved in the final result by our algorithm. In these case, the prior of the target movement may be useful to eliminate the negative effects brought by the isolated bright point noise.
6. Conclusions notably, even for extreme cases in which the small dim targets are hardly visible. Our algorithm performed better than typical methods in the comparisons. Table 1 presents the average time of evaluated methods for processing one 320 ⁄ 256 image on an Intel i7 2.4 GHz CPU 64-bit Windows laptop computer with 12 GB memory and a Geforce GTX 860 M graphics card. PHE, CLAHE and the proposed algorithm were implemented in C++. MTH was programmed in Matlab and it should be faster if implemented in C++. Our method is slower than PHE and CLAHE. Because the proposed algorithm needs to extract the target trajectory, it takes longer computational time. With the parameter settings as described above, it can be concluded that the proposed algorithm meets the needs of real time applications. Although the proposed algorithm performed well in the above experiments, there is still additional work to be done. Due to the DPA used for this approach, which outputs only one trajectory with using the maximum merit function, the proposed algorithm can only deal with a single target, and cannot enhance multiple targets simultaneously. In our future work, the strategy used in TBD for multi-target tracking may be introduced to extend our algorithm to multi-target visual enhancement. Additionally, our algorithm cannot deal with isolated bright point noise properly; such occur-
This paper tackles the problem of infrared small dim target visual enhancement in video. The shortcomings of existing methods in enhancing such targets are analyzed, especially for cases where the target is so weak that hardly visible. The model of infrared small dim target video widely used in previous works was presented, and the corresponding assumptions about the target’s characteristics were validated using real images in this paper. It is noted that temporal cues should play a significant role in the visual enhancement for infrared small dim targets in video. The proposed algorithm adopts DPA to detect the target’s trajectory in the video firstly. Then, energy accumulation is performed along the trajectory to enhance the target’s intensity. Based on the assumption of the target’s shape, the clutter is suppressed using a Gaussian mask, while the enhanced target-centered local region is embedded back into the original background smoothly in an adaptive merging manner. Experiments on real infrared small dim target videos indicate that the proposed algorithm improves the visual quality of the video greatly, even in cases that the target is so faint that hardly visible. The proposed algorithm needs improvement when dealing with multi-target and isolated bright point noise, and these problems will be tackled in future works.
226
X. Sun et al. / Infrared Physics & Technology 83 (2017) 217–226
Acknowledgments This work is supported by grants from the National Natural Science Foundation of China (Grant No. 11472302) and Scientific Research Program of National University of Defense Technology (No. ZK16-03-27). We would like to thank all reviewers for their insights and suggestions, which were very helpful in improving the paper. References [1] R.C. Gonzalez, R.E. Woods, Digital Image Processing, vol. 28, Prentice Hall International, 2002 (484–486). [2] Y.X. Mao, C.X. Sun, Ze Yu Xu, Detail enhancement algorithm for infrared image, Comput. Eng. Appl. (2006). [3] K. Zuiderveld, Contrast limited adaptive histogram equalization, Graph. Gems (1994) 474–485. [4] Dou-Ming Hu, H.S. Zhao, Yun-Chuan Li, C. Pan, J.Y. Liu, A new approach to infrared image enhancement based on homomorphic filter, Infrared Technol. (2012). [5] J.H. Kim, Novel contrast enhancement scheme for infrared image using detailpreserving stretching, Opt. Eng. 50 (2011) 249. [6] C. Zhang, X. Wang, H. Zhang, G. Lv, H. Wei, A Reducing multi-noise contrast enhancement algorithm for infrared image 1 (2006) 632–635. [7] X.W. Wang, S.T. Liu, X.D. Zhou, New algorithm for infrared small target image enhancement based on wavelet transform and human visual properties, J. Syst. Eng. Electron. (2006) 268–273. [8] D. Shi, Infrared image nonlinear enhancement algorithm based on contourlet transform, Acta Opt. Sin. 29 (2009) 342–346. [9] E. Lee, E. Gu, K. Park, Effective small target enhancement and detection in infrared images using saliency map and image intensity, Opt. Rev. 22 (2015) 659–668. [10] S. Qi, J. Ma, H. Li, S. Zhang, J. Tian, Infrared small target enhancement via phase spectrum of Quaternion Fourier Transform, Infrared Phys. Technol. 62 (2014) 50–58. [11] X.Z. Bai, F.G. Zhou, Top-hat selection transformation for infrared dim small target enhancement, Imag. Sci. J. 58 (2010) 112–117. [12] J. Zhou, H. Lv, F. Zhou, Infrared small target enhancement by using sequential top-hat filters, Proc. SPIE – Int. Soc. Opt. Eng. 9301 (2014) 93011L. [13] X. Bai, F. Zhou, Hit-or-miss transform based infrared dim small target enhancement, Opt. Laser Technol. 43 (2011) 1084–1090. [14] X. Bai, F. Zhou, B. Xue, Infrared dim small target enhancement using toggle contrast operator, Infrared Phys. Technol. 55 (2012) 177–182. [15] X. Bai, Morphological operator for infrared dim small target enhancement using dilation and erosion through structuring element construction, Optik – Int. J. Light Electron Opt. 124 (2013) 6163–6166.
[16] K. Xie, T. Zhou, Y. Qiao, C.J. Ge, J. Yang, Learning to detect small target: a local kernel method, Infrared Phys. Technol. 69 (2015) 7–12. [17] Y.M. Dai, Y.Q. Wu, Y. Song, Infrared small target and background separation via column-wise weighted robust principal component analysis, Infrared Phys. Technol. 77 (2016) 421–430. [18] Y.M. Dai, Y.Q. Wu, Y. Song, J. Guo, Non-negative infrared patch-image model: Robust target-background separation via partial sum minimization of singular values, Infrared Phys. Technol. 81 (2017) 182–194. [19] K. Xie, K.R. Fu, T. Zhou, J.H. Zhang, J. Yang, Q. Wu, Small target detection based on accumulated center-surround difference measure, Infrared Phys. Technol. 67 (2014) 229–236. [20] T. Zhou, H. Bhaskar, F.H. Liu, J. Yang, P. Cai, Online learning and joint optimization of combined spatial-temporal models for robust visual tracking, Neurocomputing 226 (2017) 221–237. [21] T. Zhou, X.J. He, K. Xie, K.R. Fu, J.H. Zhang, J. Yang, Robust visual tracking via efficient manifold ranking with low-dimensional compressive features, Pattern Recogn. 48 (2015) 2459–2473. [22] A.P. Tzannes, D.H. Brooks, Detecting small moving objects using temporal hypothesis testing, IEEE Trans. Aerosp. Electron. Syst. 38 (2002) 570–586. [23] T.W. Bae, B.I. Kim, Y.C. Kim, K.I. Sohng, Small target detection using cross product based on temporal profile in infrared image sequences, Comput. Electr. Eng. 36 (2010) 1156–1164. [24] T.W. Bae, Spatial and temporal bilateral filter for infrared small target enhancement, Infrared Phys. Technol. 63 (2014) 42–53. [25] Y. Barniv, Dynamic programming solution for detecting dim moving targets, IEEE Trans. Aerosp. Electron. Syst. 21 (1985) 144–156. [26] Y. Barniv, O. Kella, Dynamic programming solution for detecting dim moving targets part II: analysis, IEEE Trans. Aerosp. Electron. Syst. 23 (1987) 776–788. [27] J. Arnold, S.W. Shaw, H. Pasternack, Efficient target tracking using dynamic programming, IEEE Trans. Aerosp. Electron. Syst. 29 (1993) 44–56. [28] Y.Y. Zhang, C.X. Wang, Space small targets detection based on improved DPA, Acta Electron. Sin. 38 (2010) 556–560. [29] S.J. Davey, B. Cheung, M.G. Rutten, Track-Before-Detect for sensors with complex measurements, in: International Conference on Information Fusion International Conference on Information Fusion, 2009, pp. 618–625. [30] H.H. Zhang, H. Duan, M.H. Liao, The TBD method for dim targets based on multi-level crossover and matching operator, J. Harbin Inst. Technol. 18 (2011) 57–61. [31] Y. Barshalom, H.M. Shertukde, K.R. Pattipati, Precision target tracking for small extended objects, Proc. SPIE 2015 (2015) 121–126. [32] D.S. Chan, D.A. Langan, D.A. Staver, Spatial-processing techniques for the detection of small targets in IR clutter, Proc. SPIE (1990) 53–62. [33] O.E. Drummond, Small target detection from image sequences using recursive max filter, Proc. SPIE – Int. Soc. Opt. Eng. 2561 (1995) 153–166. [34] Y. Barshalom, Tracking methods in a multitarget environment, IEEE Trans. Automat. Control 23 (1978) 618–626.
Contents lists available at ScienceDirect
Infrared Physics & Technology journal homepage: www.elsevier.com/locate/infrared
Regular article
Real-time visual enhancement for infrared small dim targets in video Xiaoliang Sun a,⇑, Xiaolin Liu b, Zhixuan Tang c, Gucan Long a, Qifeng Yu a a
College of Aerospace Science and Engineering, National University of Defense Technology, Changsha, China College of Mechatronics Engineering and Automation, National University of Defense Technology, Changsha, China c Reserve Officers Training Office of Ningxia University, Yinchuan, China b
h i g h l i g h t s Temporal cues are incorporated to perform visual enhancement. The target intensity is enhanced via accumulating along the trajectory extracted by DPA. An adaptive manner is presented to eliminate the sharp edge in merging. The target’s prior shape information is adopted in clutter suppression and adaptive merging.
a r t i c l e
i n f o
Article history: Received 3 October 2016 Revised 20 April 2017 Accepted 2 May 2017 Available online 8 May 2017 Keywords: Infrared dim small target Visual enhancement Dynamic programming algorithm Energy accumulation Gaussian
a b s t r a c t Visual enhancement for infrared small dim targets is a standing problem in infrared image processing. Existing approaches cannot enhance the target well and suppress the background simultaneously, especially for targets which are so faint that they are hardly visible. This paper proposes a novel real-time visual enhancement algorithm for infrared small dim targets in video by introducing temporal cues. In this work, Dynamic Programming Algorithm (DPA) is used to detect the target’s trajectory in the video and the target is enhanced through energy accumulation along the trajectory. The shape prior of the small dim target is adopted for background suppression and adaptive merging. Experimental results on real infrared small dim target videos indicate that the proposed algorithm can improve the visual quality of these types of images notably, especially for cases in which the target is hardly visible. In addition, the proposed algorithm takes on average 8.35 ms to process a 320 ⁄ 256 image, and thus meets the needs of real-time applications. Ó 2017 Elsevier B.V. All rights reserved.
1. Introduction Due to its advantages of all-weather operation, detecting and anti-jamming abilities, infrared imaging equipment has been widely used in military reconnaissance, traffic monitoring, weapon guidance, space target surveillance, etc. This paper focuses on the visual enhancement of small dim infrared targets in video captured during space target surveillance in order to obtain high-quality images. In space target surveillance, infrared imaging equipment is often used in long-distance space target detection to compensate for the deficiency of visible light equipment. As a result of long distance imaging, the target’s energy is affected heavily by air turbulence, scattering, etc., and thus it is weak, while the Signal to Noise Ratio (SNR) of the captured image is low. This results in low quality images which are difficult to interpret. Visual enhancement for
⇑ Corresponding author. E-mail address: [email protected] (X. Sun). http://dx.doi.org/10.1016/j.infrared.2017.05.002 1350-4495/Ó 2017 Elsevier B.V. All rights reserved.
infrared small dim target has a great significance in extending the observation distance of infrared imaging equipment. Infrared image enhancement has been studied for a long time. Traditional methods use the global information of the image to perform the enhancement, e.g. histogram-based methods, gradient domain-based methods, etc. These enhancement methods are not specifically designed for infrared images with small dim targets. In these types of images, the target constitutes only a small part of the infrared image while the background is the dominant component. Global information-based algorithms tend to over-enhance the background but the target is not highlighted properly. Infrared small dim target detection has been well studied. Target enhancement (e.g. mathematical morphology-based algorithms, filteringbased algorithms, etc.) is often used as a preprocessing step during detection to improve the Signal-to-Clutter Ratio (SCR) of the local region which contains the target. However, it does not focus on improving the visual quality of the infrared small dim image. The background is often completely removed in the enhanced result
218
X. Sun et al. / Infrared Physics & Technology 83 (2017) 217–226
and in turn the result cannot be visually interpreted. To the best of our knowledge, no existing visual enhancement algorithm is specially designed for infrared small dim target in video, especially for extreme cases in which the targets are so faint that hardly visible. Additionally, existing enhancement algorithms mainly depend on spatial cues captured from a single image. However, spatial cues may be inadequate to perform visual enhancement for extreme cases in which targets are so weak that hardly visible. Therefore, existing algorithms do not improve the visual quality of infrared small dim target images in a satisfactory manner. Temporal cues in image sequences should be introduced in visual enhancement for infrared small dim target. This paper tackles the problem of visual enhancement of infrared small dim targets in video by using the spatial and temporal information contained in the image sequence and the prior of the target shape. We attempt to improve the visual quality of the image further by enhancing the target and at the same time appropriately suppressing the background. In typical images of this type, the target’s energy is very weak as a result of long-distance imaging, and, in extreme cases, the observer cannot distinguish the target from the background. Determining by the characteristic of the infrared sensor, the small dim target is often modeled as a Gaussian spot. Based on the above analysis, this paper proposes a new visual enhancement algorithm for infrared small dim targets in video. First, we adopt DPA to track the target and enhance the latter through accumulation of the target-centered local regions along its trajectory. Next, the accumulated target-centered local region is weighted by a Gaussian mask to suppress clutter around the target. Finally, the processed target-centered local region is embedded back into the input image in an adaptive merging manner. During this merging operation, the intensities of the pixels in the local region are selected as adaptive weights. We evaluated the proposed algorithm on real infrared videos including small dim targets. Comparisons between our algorithm and typical enhancement algorithms are also performed. Experimental results show that the proposed algorithm can notably enhance the target and suppress the background appropriately. The visual quality of the infrared video was markedly improved, even for videos in which the targets are so faint that hardly visible. The rest of the paper is organized as follows: related works are summarized in Section 2. Section 3 contains an analysis of the characteristics of the target and the background in infrared image sequences containing small dim targets. We explain the proposed algorithm in detail in Section 4, while in Section 5 experimental results are presented. Finally, Section 6 concludes the paper.
2. Related works Histogram Equalization (HE) [1] is widely used in infrared image enhancement. However, it is unsuitable for the enhancement of infrared small dim target images, as the background is the dominant component of the image and occupies most of the valid gray levels, while the target only uses few gray levels. Thus, equalizing the histogram of the whole image causes the overenhancement of the background. Plate Histogram Equalization (PHE) [2] introduces a plate value to control the equalization. Contrast Limited Adaptive Histogram Equalization (CLAHE) [3] performs the plate histogram equalization in local parts of the input infrared image. Modifications to HE, PHE, CLAHE, etc., also tend to over-enhance the background. Hu et al. [4] adopted adaptive median and homomorphic filtering for infrared image enhancement. In [5], Kim et al. performed contrast enhancement in the gradient domain. Their algorithm enhances local contrast and preserves global contrast simultaneously.
The wavelet transformation plays an important role in image processing. Zhang et al. [6] introduced the Discrete Stationary Wavelet Transform (DSWT) to enhance the contrast of infrared image. Wang et al. [7] suppressed noise by manipulating the wavelet coefficients. Then, a nonlinear transformation was applied to the wavelet coefficients based on Weber theory to enhance the target. The wavelet transformation based methods suffer from the ringing effect and the under-enhancement of the details. The contourlet transformation is more powerful than the wavelet transformation in representing contours and textures. Shi et al. [8] use the contourlet transformation to decompose the infrared image at different scales and directions. The beta function is applied in the low frequency domain to enhance global contrast. A non-linear gain function is adopted to process the coefficients at the different scales. The enhanced image is obtained through the transformation of the processed coefficients back to the spatial domain. Lee et al. [9] tackled infrared small dim target enhancement from the perspective of saliency analysis. The authors enhanced the target based on the assumption that the target is brighter that the surrounding background and that the target’s shape can be modeled as a Gaussian distribution. The enhanced result was obtained by merging the temperature and the Difference of Gaussian (DoG) maps. Motivated by models of the visual attention mechanism, Qi et al. [10] introduced the quaternion Fourier transformation in infrared small dim target enhancement. As mentioned in Section 1, enhancement is often treated as a preprocessing step in infrared small dim target detection. Mathematical morphology operations are simple and have been widely used in small infrared target processing [11–15]. Zhou et al. [12] used the top-hat filter group to enhance the target and suppress the background at the same time. Bai et al. [11] extended the top-hat transformation to the top-hat selection transformation for infrared small dim target enhancement, where the parameters used in the transformation were calculated based on the properties of the target region. In [15], the authors pointed out that in infrared images the target’s intensity is higher than that of the surrounding background’s, and designed a structuring element used in erosion and dilation specifically for infrared small target enhancement. The hit-or-miss transformation [13] and toggle contrast operator [14] have also been used to enhance infrared small dim target images. A local kernel method for small infrared target detection was proposed in [16]. The authors trained a weighted local model to preserve background while removing target in infrared images. In order to suppress the strong edges in infrared images, modifications [17,18] were made on the infrared patch image model. Dai et al. [17] incorporated structural prior information in separating small target from background. The weight is adaptive for each column according to structure prior in the weighted infrared patch image model. [18] adopted non-negative constraint and separated the target from the background via minimizing the partial sum of singular values. Xie et al. [19] proposed a novel algorithm called accumulated center-surround difference measure to detect small target in heterogeneous area. These algorithms are designed to improve the performance of infrared small dim target detection, not the visual quality of the input image. The algorithms discussed above perform target enhancement using cues from a single image. However, the infrared small dim target may be so faint that hardly visible to the observer in extreme cases, the above mentioned algorithms may not yield satisfactory results. What’s worse, the small dim target may be lost in the output enhanced result. The spatial cues from a single image are inadequate for the visual enhancement of infrared small dim target, especially for hardly visible small dim targets. The visual enhancement for infrared small dim targets may benefit from the temporal cues contained in image sequences. Temporal cues have been widely used in visual target tracking [20,21]. Unfortunately, the
219
X. Sun et al. / Infrared Physics & Technology 83 (2017) 217–226
temporal cues have not attracted enough attentions in previous visual enhancement works. As a preprocessing step of infrared small target detection, [22–24] introduced temporal target information in enhancement. [22,23] assumed that small targets are moving in a specific direction. They have difficulties in detecting non-moving targets. In order to solve weaknesses in [22,23], Bae [24] used a 1D temporal bilateral filter to extract temporal target information. The method is not efficient under infrared images including hardly visible small dim targets. The temporal cues have been widely used in infrared small dim target detection. The Tracking-Before-Detection (TBD) algorithm first determines all possible target trajectories in an image sequence, and the true trajectory is selected through examination of the trajectory candidates. The TBD algorithm demonstrates excellent performance in infrared small target detection, especially for image sequences with low SNR. As a powerful tool, DPA was used in TBD by Barniv et al. [25,26] to reduce computational load. Arnold et al. [27] modified traditional DPA by introducing a new track scoring function and an intra-frame search procedure. In [28], the authors used the amplitude of the target signal to construct an index function, and, subsequently, several works [29– 31] on index function modification and preprocessing of the observed data were published recently. By selecting a number of points located in the target region used in energy accumulation, Zhang et al. [30] improved the performance of the detection algorithm significantly. Drawing lessons from TBD algorithms for infrared small dim target detection, this paper adopts DPA to track the small dim target in infrared video. The visual enhancement of the infrared small dim targets is performed along the extracted trajectory. 3. Analysis of infrared small dim target images This paper focuses on the enhancement of infrared small dim target videos captured in long distance observation. Fig. 1 shows a typical frame chosen from such a video. The dim target is indicated by a rectangle in Fig. 1(a) and the 3D display of the local region is shown in Fig. 1(b). Obviously, in this case the target is very faint and not easy to observe directly. Based on existing works, we model the target image consisting of noise, background and the target. The spatial and temporal model of the video can be expressed as
Iðx; y; kÞ ¼ Nðx; y; kÞ þ Bðx; y; kÞ þ Tðx; y; kÞ
ð1Þ
where k is the frame index. ðx; yÞ is the pixel’s coordinates, while Nðx; y; kÞ, Bðx; y; kÞ and Tðx; y; kÞ denote noise, background and target gray values, respectively. Previous works [31–33] modeled the noise as a zero-mean Gaussian distribution in the spatial and temporal domain, i.e.
Nðx; y;kÞ Nð0; r2N Þ
ð2Þ
where rN is the standard deviation determined by the infrared imaging sensor. The background Bðx; y; kÞ is often modeled as being heavily correlated and a large area component in the spatial domain but is assumed to be relatively stable in the temporal domain. Therefore, Bðx; y; kÞ can be modeled as
Bðx; y;kÞ ¼ Cðx; yÞ þ NB ðx; y; kÞ
ð3Þ
where Cðx; yÞ denotes the intensity of the background and NB ðx; y; kÞ Nð0; r2B Þ is the noise. The target is often presented as a bright spot in the image, as shown in Fig. 1. We adopt the model given by [32] and consider the target as a 2D Gaussian distribution in the spatial domain:
tðx; yÞ ¼
D
2pr2
2
ðx iÞ þ ðy jÞ exp 2r2
2
!
ð4Þ
where r is determined by the infrared sensor’s Point Spread Function (PSF). D denotes the target intensity at ði; jÞ. The temporal and spatial model of the target is
Tðx; y; kÞ ¼ tðx; y; k; hÞ
ð5Þ
In this case, h is a parameter related to target’s velocity, the infrared sensor’s PSF, etc. The target is often assumed to be brighter than the surrounding background. This assumption is fundamental to many algorithms, and also forms the basis of the algorithm presented here. This paper validates this assumption using the Negative Laplace of Gaussian (NLoG) operator to filter real infrared small dim target images. Using this approach, bright spots which satisfy the assumptions should produce high values in the response map. Some results are listed in Fig. 2, where the small target is indicated by a rectangle. The response maps in the bottom row of Fig. 2 show that the targets give rise to high response values in the filtered results, thus verifying the rationality of the assumptions about the target. The example images listed in Fig. 2 contain small targets that are visible to the observer. NLoG performs well in detecting these targets. However, our investigations showed that NLoG has trouble
Fig. 1. Infrared small dim target image example. (a) Original image. (b) 3D display of the local region.
220
X. Sun et al. / Infrared Physics & Technology 83 (2017) 217–226
Fig. 2. Infrared small dim target images (top row) and corresponding NLoG response maps (bottom row).
in detecting target in low SNR images, as this method only uses spatial information from a single image and cannot detect small dim targets reliably, especially for low SNR images. Therefore, temporal cues should be introduced to enhance the target. The motion of the small dim target within the image is approximate to the translation motion in long distance imaging. Following TBD, we sum the target-centered local regions, each denoted as lðx; y; kÞ, along the target’s trajectory within a time window of length X as
Lðx; yÞ ¼
X lðx; y; kÞ
ð6Þ
k2X
In this manner, zero-mean noise is suppressed and the target is enhanced significantly in Lðx; yÞ. When performed on infrared small dim target detection, this preprocessing tends to further suppress the background and noise, or even remove them completely, as shown in Eq. (7).
Lðx; yÞ ¼ T 0 ðx; yÞ
velocity in the previous frame. This target movement can be viewed as a Markov process, and DPA can be used to obtain the target’s trajectory efficiently through iterative searches. The target is presented as a bright spot in the captured image and its energy is dispersed within a certain region. We adopt the improved DPA presented in [28], which uses multi-point accumulation. In order to reduce the computational load, the local regions considered in the present approach have a square shape. Let Rt , with dimensions ð2r t þ 1Þ ð2r t þ 1Þ and Rtþb , with dimensions ð2r tþb þ 1Þ ð2rtþb þ 1Þ denote the target region and the targetcentered local region, respectively. The background in Rtþb is denoted as Rb , which is equal to ðRtþb Rt Þ. As noted in Section 2, we assume that the target is brighter than the surrounding background and that its shape can be approximated by a Gaussian function. This paper uses the intensity difference zðx; y; kÞ between Rt and Rb to define the merit function:
X X zðx; y; kÞ ¼ ð Iðx; y; kÞ Iðx; y; kÞÞ
ð7Þ
where T 0 ðx; yÞ is the aggregate of the target’s energy. The enhanced result, however, may not be suitable for the observer. This paper aims to improve the visual quality of infrared images containing small dim targets. Therefore, we seek to enhance the target and suppress the background and noise appropriately, and then merge them smoothly with the final enhanced image. This is achieved through the adaptive merging method proposed below. 4. Methodology Based on the analysis presented above, a new visual enhancement algorithm for infrared small dim targets contained in video frames is presented in this section. The algorithm consists of three main parts: energy accumulation along the target trajectory, background and noise suppression and adaptive merging. The flow chart of the proposed algorithm is shown in Fig. 3.
Rt
An iterative equation can then be defined as
FðPk Þ ¼ max½FðPk1 Þ þ zðx; y; kÞ Pk1
wk ðPk Þ ¼ argmax½FðPk1 Þ
ð9Þ
P k1
F denotes the merit-function, P k is the pixel value in frame k, while wk ðPk Þ indicates the trajectory which maximizes F. Based on the assumptions about the target, F reaches its maximum at the center of the target, but there no assumptions are made about the moving model of the target. The upper bound of the target’s velocity is determined by the radius rs of the search region. The duration of the time window is X, within which the trajectory with the maximum F is stored in w. The energy accumulation of the target-centered local region sequence along w is determined using Eq. (6). The local region’s size is ROI w ROI h. The SNR of the accumulated region is used to measure the confidence of the accumulated result, and for a point target it is defined as [34]:
4.1. Energy accumulation along the target trajectory
SNR ¼ 10lg The video considered in this discussion is assumed to have been captured during long distance imaging, and includes a small dim target moving slowly in the image. The position of the target in current frame is only determined based on target’s position and
ð8Þ
Rb
T B
rn
ð10Þ
where T and B denote the intensity of the target and the background, respectively, while rn is the standard deviation of the noise. In this paper, we use the following approximations:
X. Sun et al. / Infrared Physics & Technology 83 (2017) 217–226
221
Fig. 3. The flow chart of the proposed algorithm. (a) Original image. (b) Energy accumulation. (c) Gaussian mask. (d) Background and noise suppression. (e) Adaptive merging.
T ¼ mean ðLðx; yÞÞ ðx;yÞ2Rr t
B ¼ mean ðLðx; yÞÞ ðx;yÞRRtþb
ð11Þ
rn ¼ stdev ðLðx; yÞÞ ðx;yÞRRtþb
If the SNR of the accumulated region is greater than a pre-set threshold ThSNR, the accumulated result is considered reliable, and unreliable otherwise. If the accumulated result is unreliable, the energy accumulation process will be reapplied. The algorithm for DPA based energy accumulation is summarized in Algorithm 1. Algorithm 1. DPA based energy accumulation for k = 0 to Video length do if k ¼¼ 0 Initialize the merit-function FðP 0 Þ ¼ zðx; y; 0Þ and the backtracking function w0 ðP0 Þ ¼ ðx; yÞ for all pixels in the input image, zðx; y; 0Þ is calculated as Eq. (8) else if k P 1 Perform the iteration according to Eq. (9) if k P X Detect the pixel with the maximum F and perform the accumulation of the target centered local region along w Calculate the SNR of the accumulated region using Eqs. (10) and (11) if SNR > ThSNR Output the accumulated region else Reinitialize the whole process by setting the current frame as the first frame and k = 0 end if end if end if end for
4.2. Background and noise suppression As mentioned in Section 3, infrared small dim target images comprise the target, background and noise. The noise is modeled as a zero mean Gaussian distribution. Ideally, if X is infinitely long,
the noise can be completely removed from the accumulated result. In practice, as the background is slowly varying over frames, the noise does not fully behave as modeled, X is not infinitely long, etc. As a result, the background and noise are also enhanced during the accumulation process result, as shown in Fig. 3(b). The target is located at the center of the accumulated region. We use a 2D Gaussian distribution to approximate the target shape [26], while suppressing the background and noise in the accumulated region by weighting Lðx; yÞ using a Gaussian mask f ðx; yÞ with a size of ROI w ROI h:
f ðx; yÞ ¼
1 ððxx0 Þ2 þðyy0 Þ2 Þ=ð2r2f Þ e 2pr2f
ð12Þ
ðx0 ¼ ROIw =2; y0 ¼ ROIw =2Þ is the center of the mask, while rf denotes the standard deviation, which depends on the target’s size and is determined according to the 3 r criterion. The weighted result L0 ðx; yÞ is obtained by
L0 ðx; yÞ ¼ f ðx; yÞLðx; yÞ
ð13Þ
As Fig. 3(c) shows, using this approach, the background and noise are suppressed effectively and the target is preserved in L0 ðx; yÞ. 4.3. Adaptive merging To complete the visual enhancement of the target, the accumulated region L0 ðx; yÞ should be embedded back into the original image Iðx; yÞ. The target-centered local region and the remaining part of Iðx; yÞ are processed separately. If we use L0 ðx; yÞ to replace the corresponding part lðx; yÞ in Iðx; yÞ, sharp artifacts will occur across the boundary of L0 ðx; yÞ. The example image shown in Fig. 4(a) was selected from an infrared small dim target video. To illustrate the merging process, only part of the example image is selected and zoomed in for display. The dim target in the example image is practically invisible to the observer, but the local region used for accumulation is depicted by a rectangle. The image in Fig. 4(b) was obtained by replacing the corresponding part lðx; yÞ by L0 ðx; yÞ, and it is evident that the sharp edges in degrade the visual quality of the image. This paper proposes an adaptive merging method to tackle this problem. The enhanced target should be embedded back into the input image smoothly, while the target’s energy should be preserved for the final output. Therefore, the weights used during merging should vary according to the intensity of the target. When considering the distribution of the intensity in L0 ðx; yÞ, we take the normalized L0 ðx; yÞ, denoted by L0N ðx; yÞ, as the weighting factor for
222
X. Sun et al. / Infrared Physics & Technology 83 (2017) 217–226
Fig. 4. Example of adaptive merging. (a) Sample image. (b) Direct replacement. (c) Adaptive merging.
adaptive merging. The adaptive merging of L0 ðx; yÞ and lðx; yÞ is expressed as 0
l ðx; yÞ ¼ L0N ðx; yÞL0 ðx; yÞ þ ð1 L0N ðx; yÞÞlðx; yÞ 0
ð14Þ
0
l ðx; yÞ is the merged result. By placing l ðx; yÞ back to Iðx; yÞ, the final enhancement result is obtained. As shown in Fig. 4(c), the enhanced target is smoothly embedded into Iðx; yÞ. 5. Experiments 5.1. Experimental setup To validate the performance of the proposed algorithm qualitatively and quantitatively, a large number of experiments was conducted on real infrared small dim target videos. As mentioned above, there is no existing specially designed visual enhancement algorithm for infrared small dim target. We compared our algorithm with relevant typical enhancement algorithms for infrared small dim target images, including PHE [2], CLAHE [3] and MTH [11]. PHE and CLAHE are widely used for infrared image enhancement, while MTH is a typical algorithm used as a preprocessing method in infrared small target detection. For PHE, CLAHE and MTH, the parameters suggested by the authors were used. PHE, CLAHE and MTH perform the enhancement using spatial cues from a single image. They were applied on real videos frame by frame. The resolution of the image used in this paper was 320 ⁄ 256. For the proposed algorithm, the parameter settings used in all experiments were: the radius r t of the target region was 5 pixels, r tþb was 11 pixels, the search radius r s was 3 pixels, the time window X had a length of 10 frames, the size of the accumulation region lðx; yÞ was 60 ⁄ 60, while the SNR threshold was set to 4.77 dB. 5.2. Results We mainly reported experimental results on two representative videos from the tested infrared small dim target data, named Seq. 1 and Seq. 2. The length of Seq. 1 was 209 frames. The target in Seq. 1 is so faint that hardly visible. Seq. 2 was longer than Seq. 1, containing 3009 frames. As the sequence progresses, the target in Seq. 2 is gradually fading, becoming hardly visible during the latter part of the sequence. The presented results include the target trajectories extracted by DPA, visual comparisons, improvements of the SNR of the target-centered local region and the target area’s mean intensity, which was calculated in region Rt . The targetcentered local region’s size was ROI w ROI h. The target trajectory extraction is the key step of our algorithm and the energy accumulation is performed along the extracted tar-
get trajectory. Fig. 5 presents parts of the target trajectories (labeled by white crosses) extracted by DPA for Seq. 1 and Seq. 2. The lengths of the shown trajectories in Fig. 5(a) and (b) were 40 (11th–50th frame) and 140 (1960th–2099th frame). The movement of the target was arbitrary and DPA extracted the target trajectory reliably as Fig. 5 shows. Due to the long distance imaging, the target moved slowly in the image domain. The target in Seq. 2 moved slower than the one in Seq. 1 as shown in Fig. 5. For qualitatively evaluation, the enhancement results of several sample frames picked from Seq. 1 and Seq. 2 are shown in Figs. 6 and 7 for visual comparison. As was the case in Fig. 4, only part of the zoomed-in input images are shown in Figs. 6 and 7. The targets were labeled by rectangles. As Fig. 6(a) shows, the original target in Seq. 1 is hardly visible. The original target in Seq. 2 is gradually weakening as shown in Fig. 7(a), and hardly visible in the 2975th frame. The enhancement results shown in Figs. 6 (b) and 7(b) indicated that the targets were enhanced effectively through energy accumulation. Benefitting from the adaptive merging process, the enhanced target-centered local region was embedded back into the input image smoothly. The proposed algorithm improved the visual quality of infrared small dim target images notably. We also presented the visual comparison between PHE, CLAHE, MTH and the proposed algorithm. Sample images picked from the test infrared small dim target data were listed in Fig. 8, with the targets denoted by rectangles. As explained in Sections 1 and 2, the background is the dominant part of the image. PHE and CLAHE over-enhanced the background and degraded the visual quality of the image, as shown in Fig. 8(b) and (c), respectively. MTH estimated the background using the modified top-hat transformation where the target was extracted by subtracting the estimated background from the input image. In the corresponding result, the target’s intensity did not exceed its original value, which means that if the target is so faint that it is hardly visible, the MTH algorithm cannot produce a satisfactory result. As shown in Fig. 8(d), the background was almost completely removed from the result but the target was not enhanced effectively. This paper focuses on the visual enhancement for infrared dim small targets. In extreme cases, the target is almost flooded by the background, and information from a single image may not be enough to achieve satisfactory visual enhancement. The PHE, CLAHE and MTH algorithms are applied on single input images, and thus perform poorly in extreme cases. In the example image shown in the third column of Fig. 8, the target is invisible to observer, and PHE, CLAHE and MTH all failed to enhance the target. Our algorithm enhanced the target’s intensity through energy accumulation along the target’s trajectory, achieving notable
223
X. Sun et al. / Infrared Physics & Technology 83 (2017) 217–226
(b) Enhancement result
(a) Original images
Fig. 5. Target trajectory extracted by DPA.
27th frame
121th frame
201th frame
Fig. 6. Visual comparison for Seq. 1.
results as shown in Fig. 8(e), even for the extreme case. Based on the assumptions about the target’s shape, the accumulation region was embedded back into the input image smoothly, and the visual quality of the infrared small dim target video was markedly improved. For quantitatively evaluation, the improvements of the SNR of the accumulation region and the target area’s mean intensity before and after enhancement on Seq. 1 and Seq. 2 are given in Figs. 9and 10. In Seq. 1, the target is almost drowned out by the background in the original video, while the SNR of the targetcentered local region is low and unstable. The target area’s mean intensity is approximately equal to that of the background. As mentioned above, PHE and CLAHE tended to over-enhance the background. Although the target intensity is enhanced, the SNR of the local region is degraded as shown in Fig. 9. MTH degraded the target intensity and the SNR of the local region. Our algorithm performs better in comparing with PHE, CLAHE and MTH. As illustrated by the results in Fig. 9, the target intensity is enhanced
greatly by the proposed algorithm, while the SNR of the local region is notably improved and remains stable throughout the whole video. The evaluation results for Seq. 2 are presented in Fig. 10. The target area’s mean intensity for the original video shown in Fig. 10 clearly shows the change of the target intensity from strong to weak in the original videos. The SNR of the local region is low. Similar to the results in Fig. 9, PHE and CLAHE enhanced the target intensity notably, however, they degraded the SNR of the local region as a result of the over-enhancement of the background. MTH subtracted the estimated background from the original image. It reduced the target intensity and it had difficulties in enhancing hardly visible target as shown in Fig. 10. The target enhanced by our algorithm demonstrates a high and stable intensity, with a significant improvement in the SNR of the targetcentered local region. As evident from Figs. 9 and 10, the proposed algorithm improved the visual quality of infrared small dim target image
224
X. Sun et al. / Infrared Physics & Technology 83 (2017) 217–226
Fig. 7. Visual comparison for Seq. 2.
Fig. 8. Visual comparison between typical methods and the proposed algorithm. (a) Original image. (b) PHE. (c) CLAHE. (d) MTH. (e) Proposed algorithm.
225
X. Sun et al. / Infrared Physics & Technology 83 (2017) 217–226
SNR(dB)
Target area mean intensity
25
200
20
180
15
160
10
140
5
120 100
0
80
-5
60
-10
0
1 9 17 25 33 41 49 57 65 73 81 89 97 105 113 121 129 137 145 153 161 169 177 185
20
-20
Original
PHE
CLAHE
MTH
Proposed
1 9 17 25 33 41 49 57 65 73 81 89 97 105 113 121 129 137 145 153 161 169 177 185
40
-15
Original
PHE
CLAHE
MTH
Proposed
Fig. 9. Improvements of the SNR and the target area’s mean intensity for Seq. 1.
Fig. 10. Improvements of the SNR and the target area’s mean intensity for Seq. 2.
Table 1 Computation time comparison.
Code Time (ms)
PHE
CLAHE
MTH
Proposed
C++ 2.68
C++ 3.51
Matlab 47.55
C++ 8.35
rences are treated as targets and preserved in the final result by our algorithm. In these case, the prior of the target movement may be useful to eliminate the negative effects brought by the isolated bright point noise.
6. Conclusions notably, even for extreme cases in which the small dim targets are hardly visible. Our algorithm performed better than typical methods in the comparisons. Table 1 presents the average time of evaluated methods for processing one 320 ⁄ 256 image on an Intel i7 2.4 GHz CPU 64-bit Windows laptop computer with 12 GB memory and a Geforce GTX 860 M graphics card. PHE, CLAHE and the proposed algorithm were implemented in C++. MTH was programmed in Matlab and it should be faster if implemented in C++. Our method is slower than PHE and CLAHE. Because the proposed algorithm needs to extract the target trajectory, it takes longer computational time. With the parameter settings as described above, it can be concluded that the proposed algorithm meets the needs of real time applications. Although the proposed algorithm performed well in the above experiments, there is still additional work to be done. Due to the DPA used for this approach, which outputs only one trajectory with using the maximum merit function, the proposed algorithm can only deal with a single target, and cannot enhance multiple targets simultaneously. In our future work, the strategy used in TBD for multi-target tracking may be introduced to extend our algorithm to multi-target visual enhancement. Additionally, our algorithm cannot deal with isolated bright point noise properly; such occur-
This paper tackles the problem of infrared small dim target visual enhancement in video. The shortcomings of existing methods in enhancing such targets are analyzed, especially for cases where the target is so weak that hardly visible. The model of infrared small dim target video widely used in previous works was presented, and the corresponding assumptions about the target’s characteristics were validated using real images in this paper. It is noted that temporal cues should play a significant role in the visual enhancement for infrared small dim targets in video. The proposed algorithm adopts DPA to detect the target’s trajectory in the video firstly. Then, energy accumulation is performed along the trajectory to enhance the target’s intensity. Based on the assumption of the target’s shape, the clutter is suppressed using a Gaussian mask, while the enhanced target-centered local region is embedded back into the original background smoothly in an adaptive merging manner. Experiments on real infrared small dim target videos indicate that the proposed algorithm improves the visual quality of the video greatly, even in cases that the target is so faint that hardly visible. The proposed algorithm needs improvement when dealing with multi-target and isolated bright point noise, and these problems will be tackled in future works.
226
X. Sun et al. / Infrared Physics & Technology 83 (2017) 217–226
Acknowledgments This work is supported by grants from the National Natural Science Foundation of China (Grant No. 11472302) and Scientific Research Program of National University of Defense Technology (No. ZK16-03-27). We would like to thank all reviewers for their insights and suggestions, which were very helpful in improving the paper. References [1] R.C. Gonzalez, R.E. Woods, Digital Image Processing, vol. 28, Prentice Hall International, 2002 (484–486). [2] Y.X. Mao, C.X. Sun, Ze Yu Xu, Detail enhancement algorithm for infrared image, Comput. Eng. Appl. (2006). [3] K. Zuiderveld, Contrast limited adaptive histogram equalization, Graph. Gems (1994) 474–485. [4] Dou-Ming Hu, H.S. Zhao, Yun-Chuan Li, C. Pan, J.Y. Liu, A new approach to infrared image enhancement based on homomorphic filter, Infrared Technol. (2012). [5] J.H. Kim, Novel contrast enhancement scheme for infrared image using detailpreserving stretching, Opt. Eng. 50 (2011) 249. [6] C. Zhang, X. Wang, H. Zhang, G. Lv, H. Wei, A Reducing multi-noise contrast enhancement algorithm for infrared image 1 (2006) 632–635. [7] X.W. Wang, S.T. Liu, X.D. Zhou, New algorithm for infrared small target image enhancement based on wavelet transform and human visual properties, J. Syst. Eng. Electron. (2006) 268–273. [8] D. Shi, Infrared image nonlinear enhancement algorithm based on contourlet transform, Acta Opt. Sin. 29 (2009) 342–346. [9] E. Lee, E. Gu, K. Park, Effective small target enhancement and detection in infrared images using saliency map and image intensity, Opt. Rev. 22 (2015) 659–668. [10] S. Qi, J. Ma, H. Li, S. Zhang, J. Tian, Infrared small target enhancement via phase spectrum of Quaternion Fourier Transform, Infrared Phys. Technol. 62 (2014) 50–58. [11] X.Z. Bai, F.G. Zhou, Top-hat selection transformation for infrared dim small target enhancement, Imag. Sci. J. 58 (2010) 112–117. [12] J. Zhou, H. Lv, F. Zhou, Infrared small target enhancement by using sequential top-hat filters, Proc. SPIE – Int. Soc. Opt. Eng. 9301 (2014) 93011L. [13] X. Bai, F. Zhou, Hit-or-miss transform based infrared dim small target enhancement, Opt. Laser Technol. 43 (2011) 1084–1090. [14] X. Bai, F. Zhou, B. Xue, Infrared dim small target enhancement using toggle contrast operator, Infrared Phys. Technol. 55 (2012) 177–182. [15] X. Bai, Morphological operator for infrared dim small target enhancement using dilation and erosion through structuring element construction, Optik – Int. J. Light Electron Opt. 124 (2013) 6163–6166.
[16] K. Xie, T. Zhou, Y. Qiao, C.J. Ge, J. Yang, Learning to detect small target: a local kernel method, Infrared Phys. Technol. 69 (2015) 7–12. [17] Y.M. Dai, Y.Q. Wu, Y. Song, Infrared small target and background separation via column-wise weighted robust principal component analysis, Infrared Phys. Technol. 77 (2016) 421–430. [18] Y.M. Dai, Y.Q. Wu, Y. Song, J. Guo, Non-negative infrared patch-image model: Robust target-background separation via partial sum minimization of singular values, Infrared Phys. Technol. 81 (2017) 182–194. [19] K. Xie, K.R. Fu, T. Zhou, J.H. Zhang, J. Yang, Q. Wu, Small target detection based on accumulated center-surround difference measure, Infrared Phys. Technol. 67 (2014) 229–236. [20] T. Zhou, H. Bhaskar, F.H. Liu, J. Yang, P. Cai, Online learning and joint optimization of combined spatial-temporal models for robust visual tracking, Neurocomputing 226 (2017) 221–237. [21] T. Zhou, X.J. He, K. Xie, K.R. Fu, J.H. Zhang, J. Yang, Robust visual tracking via efficient manifold ranking with low-dimensional compressive features, Pattern Recogn. 48 (2015) 2459–2473. [22] A.P. Tzannes, D.H. Brooks, Detecting small moving objects using temporal hypothesis testing, IEEE Trans. Aerosp. Electron. Syst. 38 (2002) 570–586. [23] T.W. Bae, B.I. Kim, Y.C. Kim, K.I. Sohng, Small target detection using cross product based on temporal profile in infrared image sequences, Comput. Electr. Eng. 36 (2010) 1156–1164. [24] T.W. Bae, Spatial and temporal bilateral filter for infrared small target enhancement, Infrared Phys. Technol. 63 (2014) 42–53. [25] Y. Barniv, Dynamic programming solution for detecting dim moving targets, IEEE Trans. Aerosp. Electron. Syst. 21 (1985) 144–156. [26] Y. Barniv, O. Kella, Dynamic programming solution for detecting dim moving targets part II: analysis, IEEE Trans. Aerosp. Electron. Syst. 23 (1987) 776–788. [27] J. Arnold, S.W. Shaw, H. Pasternack, Efficient target tracking using dynamic programming, IEEE Trans. Aerosp. Electron. Syst. 29 (1993) 44–56. [28] Y.Y. Zhang, C.X. Wang, Space small targets detection based on improved DPA, Acta Electron. Sin. 38 (2010) 556–560. [29] S.J. Davey, B. Cheung, M.G. Rutten, Track-Before-Detect for sensors with complex measurements, in: International Conference on Information Fusion International Conference on Information Fusion, 2009, pp. 618–625. [30] H.H. Zhang, H. Duan, M.H. Liao, The TBD method for dim targets based on multi-level crossover and matching operator, J. Harbin Inst. Technol. 18 (2011) 57–61. [31] Y. Barshalom, H.M. Shertukde, K.R. Pattipati, Precision target tracking for small extended objects, Proc. SPIE 2015 (2015) 121–126. [32] D.S. Chan, D.A. Langan, D.A. Staver, Spatial-processing techniques for the detection of small targets in IR clutter, Proc. SPIE (1990) 53–62. [33] O.E. Drummond, Small target detection from image sequences using recursive max filter, Proc. SPIE – Int. Soc. Opt. Eng. 2561 (1995) 153–166. [34] Y. Barshalom, Tracking methods in a multitarget environment, IEEE Trans. Automat. Control 23 (1978) 618–626.