A novel algorithm and hardware implementation for correcting sensor non-uniformities in infrared focal plane array based staring system

Infrared Physics & Technology 50 (2007) 9–13 www.elsevier.com/locate/infrared

A novel algorithm and hardware implementation for correcting sensor non-uniformities in infrared focal plane array based staring system Ajay Kumar b

a,*

, S. Sarkar b, R.P. Agarwal

b

a Instruments Research and Development Establishment, Raipur Road, Dehradun 248 008, Uttranchal, India Department of Electronics and Computer Engineering, Indian Institute of Technology, Roorkee 246 667, India

Received 11 June 2005 Available online 14 June 2006

Abstract The spatial and temporal non-uniformities in infrared focal plane array (IRFPA) result a slowly varying pattern on the image thereby, degrading the resolving capabilities of thermal imaging system considerably. A method based upon the variation of the integration time and its implementation for calibrating the sensor non-uniformities is presented. The results are compared with the black body based calibration method. This approach results the same residual non-uniformities as obtained using black body method. However, this approach offers the much required field upgradability, making the system more robust giving same performance under all environmental conditions.  2006 Elsevier B.V. All rights reserved. Keywords: Thermal imaging; Non-uniformity correction; Residual non-uniformity; Infrared focal plane array; FPGA

1. Introduction Thermal imaging systems [1] are used in a wide range of applications both in military and civilian sectors such as night vision, surveillance, fire detection, robotics and spectral imaging. Most of these systems use infrared focal plane array (IRFPA), which consists of a mosaic of photo detectors placed at the focal plane of imaging systems [2]. The advancement in infrared sensor technologies has resulted in improvement in the performance of thermal systems due to increase in number of pixels, smaller pitch and better noise equivalent temperature difference (NETD). However, their performance is strongly affected by several degrading factors such as IRFPA temperature, finite lens aperture causing blur, finite photo responsivity of the detector, under sampling caused by the finite pixel size etc. [3]. Furthermore, the pixel to pixel fluctuations can *

Corresponding author. Tel.: +91 135 2782508. E-mail addresses: [email protected], [email protected] (A. Kumar).

1350-4495/$ - see front matter  2006 Elsevier B.V. All rights reserved. doi:10.1016/j.infrared.2006.04.002

be attributed to a number of factors such as 1/f noise associated with detector and the corresponding readout integrated circuits (ROIC), and the non-linear dependence of the detector gain on the photon flux incident on it [4]. All these factors result in spatial and temporal non-uniformities of the IRFPA thereby degrading the image quality significantly. Also, the spatial non-uniformity fluctuates slowly in time due to variations in focal plane array (FPA) temperature, unstable bias voltages and the change in scene irradiance. This temporal drift is manifested in the acquired image in the form of a slowly varying pattern superimposed on the infrared image, which reduces the resolving capability of the IRFPA [5,6]. There are mainly two types of non-uniformity correction (NUC) techniques, namely scene-based [7–10] and calibration-based techniques [11,12]. The scene-based techniques generally use an image sequence and rely on the scene parameters like motion or change in the actual scene. However, these algorithms do not provide the required radiometric accuracy and are also difficult to implement in real

10

A. Kumar et al. / Infrared Physics & Technology 50 (2007) 9–13

time. The most common calibration-based technique is two-point calibration method using black body [11,12]. In this method, the normal operation of the thermal imaging system is interrupted as the camera images a uniformly calibrated target at two distinct and known temperatures. The gain and offset of each detector are then calibrated across the array so that all the detectors produce a radiometrically accurate and uniform readout at the two reference temperatures. However, this method has its own limitations. Once calibrated, it does not result the uniform output under all environmental conditions. In this paper, we present a new approach for calibrating the sensor non-uniformities based on the variation of integration time of the detector. The sensor output is taken at different integration times by exposing the sensor with a uniform source such as sky background; non-uniformity correction is then performed using two-point calibration method. This results same radiometrically calibrated output as obtained using black body source as the target, but with better field upgradability, thus resulting similar output under all environmental conditions. This paper is organized as follows. NUC model is presented in Section 2. In Section 3, results of the algorithms and its comparison with black body method are given. Section 4 gives hardware implementation scheme of the algorithm. The conclusions are given in Section 5. 2. NUC model It assumes that the IRFPA is exposed to a uniform source of infrared (IR) radiations. The image of the scene generates a signal at each pixel that is proportional to local image incidence [13]. The total current generated by the sensor element usually consists of the photon current, dark current and stray current. The stray current is due to dewar stray emission and window stray emission and is generally negligible. The dark current is proportional to eðEg =2KT Þ where Eg is the band gap of the sensor material and T is the absolute temperature in K. The output Yij of (i, j)th pixel is proportional to the number of photoelectrons accumulated at (i, j)th pixel during the integration time, which is given as [13]:   Z k2 N ij ¼ so T int Lðk; T ij Þgij ðkÞdk Aij Xij þ Dij ð1Þ k1

and Xij is given as Xij ¼

p cos4 hij 1 þ 4ðF =#Þ

hij k1, k 2 Dij Aij gij Tint

off axis angle of pixel as seen from the exit pupil lower and upper cutoff wavelength of optical system dark charge per integration time at pixel (i, j) active area of the pixel (i, j) quantum efficiency of the pixel (i, j) versus wavelength integration time of the system

Dark current Dij in Eq. (1) results the additive offset noise, which is also non-uniform from pixel to pixel. Similarly the multiplicative non-uniformities are due to spatial variation in the detector actual area and its angular distance from the optical axis. In order to perform the non-uniformity correction, sensor output is acquired by varying the integration time. Acquisition of sensor data by varying the integration time is equivalent to acquiring the data at artificially generated different temperature. The variation in number of photoelectrons due to change in integration time is equivalent to variation in number of photoelectrons due to two artificially generated temperatures as given in Eq. (1). The integration time of the sensor is varied and the raw video data is acquired by exposing the system with a uniform and high emissivity source. To achieve this, first set of image data I1 is recorded at lower integration time and second set of image data I2 is recorded at higher integration time. Multiple frames of image, at each integration time, are recorded and averaged to reduce the temporal noise. For the (i, j)th detector in the array, the measured signal Yij (detector response) is given by the following linear relationship. Y ij ¼ Gij X ij þ Oij ð2Þ where Gij and Oij are the gain and offset non-uniformities associated with the (i, j)th detector pixel, respectively and Xij is the irradiance received by the (i, j)th detector pixel. Thus, after NUC correction, the above equation can be expressed as: ð3Þ X ij ¼ G0ij ðY ij  Oij Þ; where

G0ij ¼

1 : Gij

Defining hI 2 i  hI 1 i ; G0ij ¼ I 2ij  I 1ij Oij ¼ I 1ij ;

; 2

where L(k, Tij) spectral photon radiance from object space cell (i, j) at temperature Tij calculated from Planck’s 2Pc law and is defined as k4 expð hc kKT Þ1 F/# f number of the optics so effective transmittance of optical system Xij projected solid angle as viewed from the FPA

ð4Þ

ð5Þ ð6Þ

where I1ij and I2ij are (i, j)th pixel intensities at lower and higher integration time, respectively. hI1i and hI2i are the spatial averages of the image frames at lower and higher integration time, respectively, and are given as: hI1i ¼

m;n 1 X I 1ij ; N i¼1;j¼1

ð7Þ

hI2i ¼

m;n 1 X I 2ij ; N i¼1;j¼1

ð8Þ

A. Kumar et al. / Infrared Physics & Technology 50 (2007) 9–13

where N = m * n is the total number of pixels in a frame and m and n are number of rows and columns, respectively. Thus, from (3)–(8) the corrected output of the pixel (i, j) is given as: X ij ¼

hI 2 i  hI 1 i ðY ij  I 1ij Þ: I 2ij  I 1ij

ð9Þ

This algorithm can be further improved by taking in to account the higher order non-linear coefficients. However, it is not required as the sensor normally operates in the linear region and also it is difficult to realize it in the hardware due to real time constraints.

Fig. 1b. Image after NUC.

3. Results and discussion The proposed algorithm has been implemented on 320 · 240 elements InSb staring focal plane array based thermal imaging system operating in 3–5 lm wavelength region [14]. The video processing electronics of the system generates the sensor interface signals and performs the non-uniformity corrections, bad pixel detection and replacement, digital scan conversion, automatic gain control, dynamic range compression and several image-processing functions such as contrast and edge enhancement and histogram equalization. It then, generates a high resolution CCIR-B output, which can be displayed on standard TV monitor. The integration time of the system under normal operation is kept at 2 ms. Two sets of image data are acquired at integration time of 1.8 ms and 2.2 ms. The integration time of the IRFPA is controlled through the video processing board. Eight frames of image data are collected at each integration time and averaged to reduce the temporal noise. This data is used to compute the gain and offset coefficients. The gain and offset corrections are implemented on the incoming real image. Fig. 1(a) shows the raw image with non-uniformities and Fig. 1(b) image after NUC correction. Figs. 2(a), 2(b) and represents the 3-D plots of the image data and Figs. 3(a), 3(b) and gives the plot between the pixel number and pixel vales before and after non-uniformity correction, respectively. Figs. 2 and 3 show the variations in pixel intensities before and after performing non-uniformity correction. The proposed algorithm not only corrects the nonuniformities present in the system but, also, detects the

Fig. 1a. Raw image.

Fig. 2a. 3-D plot of raw image.

Fig. 2b. 3-D plot of image after NUC.

Fig. 3a. Variation of pixel value in raw image.

11

12

A. Kumar et al. / Infrared Physics & Technology 50 (2007) 9–13

0.18 0.16

RNU (%)

0.14

Proposed method Black body method

0.12 0.1 0.08 0.06 0.04 0.02 0 1.2 1.4 1.6 1.8 2 2.2 2.4 2.6 2.8 Integration time (msec)

Fig. 3b. Variation of pixel value after NUC.

Fig. 5. Variation of RNU with integration time.

defective pixel elements as seen from Figs. 2 and 3. To detect the defective pixels mean and standard deviation (SD) of the image is computed after non-uniformity correction and thresholding criterion (mean ± 3SD) is applied. These defective elements can be corrected by any suitable algorithm [15,16]. Figs. 4(a) and 4(b) show an image frame before and after non-uniformity correction. The performance of the algorithm is measured by residual non-uniformity (RNU) [17–20], which is given as: vffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ! u L X M X  2 1u 1 t RNU ¼ N i;j  N ; N L  M i¼1 j¼1

ð10Þ

where L and M are the dimensions of IRFPA, Nij is the current output of the pixel (i, j) and N is spatial mean of the pixels in the IRFPA, which is defined as: N¼

L X M 1 X N ij : L  M i¼i j¼1

ð11Þ

Eq. (10) is applied on image frame given in Figs. 4(a) and 4(b), respectively and RNU is calculated. It is observed that RNU of the system is reduced to 0.12% from 6% after applying the proposed non-uniformity correction algorithm. This is similar to what is achieved using standard black body method. Further, RNU of the system is calculated based upon the proposed method and standard black body method by varying the integration time. A plot of variation of residual nun uniformity (RNU) and the integration time is given in Fig. 5. 4. Hardware implementation

Fig. 4a. Image frame before NUC.

Fig. 4b. Image frame after NUC.

The proposed algorithm can be easily implemented in hardware. A scheme for hardware implementation is given in Fig. 6. The analog video signal from FPA is pre-processed and converted into a digital data using a 14-bit ADC. The 14-bit ADC is used because quantisation error should be much less than the signal corresponding to the NETD of the sensor array. This raw video digital data, at different integration time, is stored in the frame memory. Eight frames of raw video data is averaged by successively operating the frame memory in read and write mode in a single clock cycle. This is done to reduce the temporal noise in the data. The integration time of the IRFPA is controlled in the video processing board using a Xilinx FPGA (XC2S1500FG676). The gain and the offset coefficients, thus calculated, are stored in offset and gain flash memory. The gain and the offset coefficients are now read from the respective flash and corrections are applied on incoming video data in real time. The data path and the control logic is implemented in a single Xilinx FPGA. The data path requires only one multiplier and one adder/subtractor, which can be easily implemented in a FPGA.

A. Kumar et al. / Infrared Physics & Technology 50 (2007) 9–13

13

Frame Memory

FPGA (XC2V1500) IRFPA (320 x 240)

14 bit ADC

+

Offset Coeff Sensor Interface Controls

X

NUC Corrected Data

Gain Coeff

Offset Flash Gain Flash (320 x 240) (320 x 240) ADC Clock FPGA (XC2V1500)

Fig. 6. Hardware implementation of the algorithm.

5. Conclusions In this paper, a new approach of correcting the sensor non-uniformities based on variation of integration time is presented. Hardware architecture for implementing the algorithm in real time is given. The results show that this approach gives the same residual non-uniformity as good as the black body method [11,12]. This approach offers the sought-after field upgradability of gain and offset coefficients, thus making the system more robust by giving same performance under all environmental conditions. Acknowledgements The authors are thankful to Mr. J.A.R. Krishna Moorty, Director, IRDE for his encouragement and supporting this work. References [1] R.D. Hudson, Infrared Systems Engineering, John Wiley, New York, 1969. [2] P. Norton, James Campbell, Stuart Horn, Third generation infrared imagers, Proceedings of SPIE 4130 (2000) 226–235. [3] Mark D. Nelson, J.F. Johnson, T.S. Lomheim, General noise process in hybrid infrared focal plane arrays, Optical Engineering 30 (11) (1991) 1682–1693. [4] J.M. Mooney, F. Shepherd, W. Ewing, J. Murguia, J. Silverman, Responsivity non-uniformity limited performance of infrared imaging cameras, Optical Engineering 28 (11) (1989) 1151–1161. [5] A.F. Milton, F.R. Barone, M.R. Kruer, Influence of non-uniformity on infrared focal plane array performance, Optical Engineering 24 (5) (1985) 855–862. [6] M.T. Eismann, C.R. Schwartz, Focal plane array non-linearity and non-uniformity impacts to target detection with thermal infrared imaging spectrometers, Proceedings of SPIE 3063 (1997) 164–173.

[7] John G. Harris, Yu Ming Chang, Non-uniformity correction of infrared image sequence using constant statistics constraint, IEEE Transactions on Image Processing 8 (8) (1999) 1148–1151. [8] Sergio N. Torres, E.M. Vera, A. Reevs, Sergio K. Sobarzo, Adaptive scene based non-uniformity correction method for infrared focal plane arrays, Proceedings of SPIE 5076 (2003) 130–139. [9] Sergio N. Torres, Majeed M. Hayat, Kalman filtering for adaptive non-uniformity correction for infrared focal plane arrays, Journal of Optical Society of America A 20 (3) (2003) 470–480. [10] Bradley M. Ratliff, Majeed M. Hayat, Russell C. Hardie, An algebraic algorithms for non-uniformity correction in focal plane arrays, Journal of Optical Society of America A 19 (9) (2002) 1737– 1745. [11] M. Schulz, L. Caldwell, Non-uniformity correction and correctibility of focal plane arrays, Infrared Physics and Technology 36 (1995) 763– 767. [12] Yan Shi, Tianxu Zhang, Zhiguo Cao, Li Hui, A feasible approach for non uniformity correction in IRFPA with nonlinear response, Infrared Physics and Technology 46 (2005) 329–337. [13] D.L. Perry, Eustace L. Dereniak, Linear theory of non uniformity correction in infrared sensors, Optical Engineering 32 (8) (1993) 1854– 1859. [14] 320 times 256 InSb Focal Plane Array Detector user guide, M/S SCD Israel. [15] Jerry Silverman, Signal processing algorithms for display and enhancement of IR images, Proceeding of SPIE 2020 (1993) 440–450. [16] J.M. Lopez, Javier Alda, Bad pixel identification by means of principal components analysis, Optical Engineering 41 (9) (2002) 2152–2157. [17] Abraham Friedenberg, Issac Goldblatt, Nonuniformity two point linear correction errors in infrared focal plane arrays, Optical Engineering 37 (4) (1998) 1251–1253. [18] V.N. Borovytsky, Residual error after non uniformity correction, Semiconductor physics, quantum electronics and opto-electronics 3 (1) (2000) 102–105. [19] Liu Huitong, Wang Qi, Chen Sihai, Yi Xinjian, Analysis of the residual error after nonuniformity correction for Infrared focal plane array, IEEE Conferences (2000) 213–214. [20] Hui Xin Zhou, Rui Lai, Shang Qian Liu, Guang Jiang, New improved non uniformity correction for infrared focal plane arrays, Optics Communications 245 (2005) 49–53.