- Email: [email protected]
Noise-robust boundary recursive algorithm for super-resolution reconstruction of staring focal plane array micro-scanning imaging
Infrared Physics & Technology 68 (2015) 159–166
Contents lists available at ScienceDirect
Infrared Physics & Technology journal homepage: www.elsevier.com/locate/infrared
Noise-robust boundary recursive algorithm for super-resolution reconstruction of staring focal plane array micro-scanning imaging Xiong Dun ⇑, Weiqi Jin, Lu Lu MoE Key Laboratory of Photoelectronic Imaging Technology and System, School of Optoelectronics, Beijing Institute of Technology, Beijing 100089, China
h i g h l i g h t s A noise-robust boundary recursive algorithm for super-resolution reconstruction is proposed. We give out a new fill ratio based sampling model. We use a modified bilateral filter and gray statistical principle to reduce reconstruction error. The proposed algorithm particularly suitable for real-time processing.
a r t i c l e
i n f o
Article history: Received 9 September 2014 Available online 10 December 2014 Keywords: Micro-scanning Super-resolution reconstruction Recursive reconstruction
a b s t r a c t A boundary recursive algorithm for the super-resolution reconstruction of staring focal plane array (FPA) micro-scanning imaging with consideration of a fill ratio-based sampling model is presented. The reconstruction errors of the algorithm introduced by image noise and boundary approximation are analyzed. Then, a modified bilateral filter and gray statistical principle are used in the algorithm to reduce these errors. Simulation and actual imaging experiments confirm that the proposed algorithm has effective noise robustness and can achieve superior results compared to an over-sampled reconstruction in the presence of low noise. This algorithm can achieve ideal sub-pixel imaging and has excellent immunity to noise. Its application will enhance the performance of optoelectronic-imaging systems. Ó 2014 Elsevier B.V. All rights reserved.
1. Introduction Micro-scanning imaging is characterized by a high sampling ratio that can be exploited to improve spatial resolution and achieve sub-pixel imaging without reducing signal noise ratio (SNR) in an optoelectronic-imaging device [1]. It has been successfully applied in the fields of remote sensing [2], thermal imaging [3–5], and microscopy imaging [6]. The principal step of micro-scanning imaging is the superresolution (SR) reconstruction and significant achievements have been made in this area. The most common reconstruction algorithm is the direct interlaced interpolation of multi-frame image sequences, called over-sampled reconstruction (OR) [7–10]. According to [7] and as we will confirm in a subsequent section, interlaced interpolation only increases the sampling rate rather than improving the pixel integral blur effect. Consequently, the image reconstructed by OR is a convolution of the ideal SR image and pixel integral window. This means that OR is not an ideal sub-pixel ⇑ Corresponding author. E-mail address: [email protected] (X. Dun). http://dx.doi.org/10.1016/j.infrared.2014.12.004 1350-4495/Ó 2014 Elsevier B.V. All rights reserved.
imaging solution. The advantage of OR is that the processing time is significantly reduced and therefore, it is suitable for a real-time micro-scanning imaging system. In recent years, to address the problem that sub-pixel displacement is not an ideal value, Interframe difference-based OR [8] and Lagrange polynomial-based OR [9] have been presented. However, the images reconstructed by these approaches remain blurred by the pixel integral window. Other common reconstruction algorithms are the multiframe SR reconstruction (MSRR) methods. There are numerous journal papers that discuss these MSRR methods [11–16]. A sample of reviews of state-of-the-art MSRR methods can be found in [17,18]. The benefits of the MSRR methods are that they not only increase the sampling rate but also reduce the pixel integral blur effect. However, current MSRR methods are time-consuming because the calculation is complicated and the number of iterations required to converge is significant [19]. This makes MSRR methods difficult to apply to real-time micro-scanning imaging systems. In recent years, boundary recursive algorithms (BRA) have been developed [20,21]. Not only can these methods obtain the same performance as half-sized pixel imaging (ideal sub-pixel imaging),
160
X. Dun et al. / Infrared Physics & Technology 68 (2015) 159–166
but they are also computationally inexpensive and suitable for real-time application. These algorithms require a precise boundary as an initial value of the recursive process. Chen subsequently determined that an approximation boundary would introduce positive–negative alternating error and used a gray statistical principle to address the error. Then, a boundary recursive algorithm with error optimization (BRA-EO) was provided [22,23]. It can acquire approximate performance of half-sized pixel imaging without a precise boundary. However, as we will demonstrate, the BRA methods mentioned above have poor immunity to noise and can only be utilized in the case where the fill ratio of the detector is one. Because the fill ratio of a detector is always less than one, random noise is inevitable in the actual imaging. Hence, the advantages of the BRA methods cannot be realized and this method is sometimes unable to reconstruct an SR image. Therefore, it is both necessary and important to study a new boundary recursive algorithm that demonstrates superior immunity to noise and considers the fill rate of the detector. In this paper, we develop a noise-robust boundary recursive algorithm based on a new fill ratio-based sampling model and present simulation and actual imaging experiments to verify the effectiveness of the algorithm. 2. Algorithm principal 2.1. Fill ratio-based sampling model There is a gap between adjacent detector units due to the requirements of the electrode wiring and reduced lateral diffusion. This causes the fill ratio of the detector to be less than one and means that the ideal SR image will acquire a different contribution from the low-resolution (LR) image, which is directly acquired by the micro-scanning imaging device. If we use a 2 2 microscanning as an example, each LR pixel is generated by four SR pixels with weights of [q2, q; q, 1] as illustrated in Fig. 1. q is called the pixel weighting factor and is expressed as:
q¼
( pffiffiffi 2 d 1 ð0:25 6 d 6 1Þ 0
ðd < 0:25Þ
;
ð1Þ
where d is the fill ratio of the detector (the size of detector unit is d d and the gap between adjacent detector units is l l, then d = (d/l)2). As indicated in Fig. 2, for the 2 2 micro-scanning, we can acquire four LR images by moving the FPA one half of the detector unit (one SR pixel) to the right, down, left, and up relative to the target scene, assuming the image scene is fixed. Interlaced interpolation is performed on the four LR images according to the
acquiring order to create an over-sampled image P. The pixels of the over-sampled image P can be written as:
Pði; jÞ ¼
Oði; jÞ þ qOði 1; jÞ þ qOði; j 1Þ þ q2 Oði 1; j 1Þ ð1 þ qÞ2 þ nði; jÞ ð2M P i P 1; 2N P j P 1Þ;
ð2Þ
where O(i, j) are the pixels of the ideal SR image in Fig. 1, O(0, j) and O(i, 0) are the boundary pixels, and n is the image noise of the LR image. According to Eq. (2), the over-sampled image P is the convolution of the ideal SR image O and the weight function [q2, q; q, 1]. This proves that the OR method cannot obtain ideal sub-pixel imaging. 2.2. Recursive reconstruction formula based on the sampling model The boundary pixels of O can be expressed as a function of the first row pixel or first column pixel of O:
Oði; 0Þ ¼ Oði; 1Þ þ bi ; Oð0;jÞ ¼ Oð1; jÞ þ aj ; Oð0;0Þ ¼ Oð1; 1Þ þ c;
ð3Þ
where 1 6 i 6 2M, 1 6 j 6 2N, and aj, bi, and c are the corresponding boundary approximation errors. According to Eqs. (2) and (3), the ideal SR image O can be expressed as: 8 a1 b1 qa1 qb1 Oð1;1Þ ¼ Pð1;1Þ nð1; 1Þ cð1þqÞ > 2 1þq 1þq > > > > > > qa q2 aj1 > > Oð1;jÞ ¼ ð1 þ qÞðPð1;jÞ nð1;jÞÞ qOð1j 1Þ 1þqj þ 1þq > < qb
Oði;1Þ ¼ ð1 þ qÞðPði; jÞ nði; jÞÞ qOði 1; 1Þ 1þqi þ > > > > > > > Oði;jÞ ¼ ð1 þ 2q þ q2 ÞðPði; jÞ nði;jÞÞ qOði 1;jÞ > > > : qOði;j 1Þ q2 Oði 1:j 1Þ
q2 bi1 1þq
ð2 6 i 6 2M; 2 6 j 6 2NÞ:
ð4Þ If we ignore the effect of noise and boundary approximation error and use the approximation SR image O1 instead of O, then, Eq. (4) can be rewritten as:
8 O1 ð1; 1Þ ¼ Pð1; 1Þ > > > > > > < O1 ð1; jÞ ¼ ð1 þ qÞPð1; jÞ qOð1j 1Þ O1 ði; 1Þ ¼ ð1 þ qÞPði;jÞ qOði 1; 1Þ ð2 6 i 6 2M; 2 6 j 6 2NÞ: > > > > O1 ði; jÞ ¼ ð1 þ 2q þ q2 ÞPði; jÞ qOði 1; jÞ > > : qOði;j 1Þ q2 Oði 1:j 1Þ ð5Þ It can be seen from Eq. (5) that if the pixel weighting factor q is known beforehand, O1 can be sequentially reconstructed from the over-sampled image P. Considering the special case that the fill
Fig. 1. Fill ratio-based sampling model.
161
X. Dun et al. / Infrared Physics & Technology 68 (2015) 159–166
Fig. 2. (a) Micro-scanning mode, (b) over-sampled image P.
ratio of the detector is one, the recursive reconstruction formula Eq. (5) is the same as that in [22]. This means that the BRA-EO method is a special case of the proposed method with the fill ratio of the detector set to one. 2.3. Analysis of the reconstruction error In Eq. (5), the noise and boundary approximation error are set to zero, thus the O1 from Eq. (5) has some error as compared with the ideal SR image O. The key step of obtaining O is the analysis and optimization of the reconstruction error caused by noise and boundary approximation. 2.3.1. Reconstruction error caused by noise The pixels of O are subtracted from the corresponding pixels of O1, or Eq. (4) is subtracted from Eq. (5). The reconstruction error caused by noise, designated en, can be expressed as: 8 en ð1; 1Þ ¼ nði; jÞ > > > > > en ð1; jÞ ¼ ð1 þ qÞ½nð1; jÞ qnð1; j 1Þ > > > > > > þq2 nð1; j 2Þ þ þ ð1Þj1 qj1 nð1; 1Þ > > > > > e ði; 1Þ ¼ ð1 þ qÞ½nði; 1Þ qnði 1; 1Þ > n < ð2 6 i 6 2M; 2 6 j 6 2NÞ: þq2 nði 2; 1Þ þ þ ð1Þi1 qi1 nð1;1Þ > > > > en ði; jÞ ¼ ð1 þ 2q þ q2 Þ½nði; jÞ qnði 1; jÞ > > > > > qnði; j 1Þ þ q2 nði 2; jÞ þ q2 nði; j 2Þ > > > > > þq2 nði 1; j 1Þ . . . q3 nði 3; jÞ > > > : i1 j1 q3 nði; j 3Þ þ þ ð1Þ qi1 nð1; jÞ þ ð1Þ qj1 nði; 1Þ
ð6Þ
Two important discussions are obtained in Eq. (6) as follows: (1) If q is equal to one, the en of an arbitrary recursive point is influenced by the noise of all the recursive points that have been previously calculated. This means that once there is a recursive point whose noise is significant (for example, the value of the recursive point has an abnormal response), then, the en of all the following recursive points will be significant, resulting in an O1 that is a significant deviation from the ideal SR image. This is proof that the BRA-EO method discussed in [21] has poor immunity to noise. (2) If q is smaller than one, the en of an arbitrary recursive point is only influenced by the noise of the recursive points within the corresponding region. The point’s region number can be expressed as [ceiling(ln 100/ln q)]2. If we use q = 0.8 as an example, the point’s region number is 21 21, which is substantially smaller than a general image resolution (320 256). This means the accumulation of en is local and controllable.
Because the fill ratio of the detector is typically less than one, then q is certainly less than one. Therefore, according to the above analysis, the reconstruction error caused by noise is not serious in practice. Nevertheless, the noise will continue to add some reconstruction error, especially when q is close to one. Therefore, we modify the adaptive bilateral filter [24] and apply it to the recursive reconstruction formula as follows:
P0
m¼w1 2
O1 ði; jÞ ¼
Pw1 2
rði m; j nÞO1 ði m; j nÞ ; Pw1 2 rði m; j nÞ m¼w1 n¼w1
P0
n¼w1 2 2
ð7Þ
2
where w is the size of the bilateral filter and r(i m, j n) is the likelihood function, which computes the likelihood that O1(i, j) and O1(i m, j n) have the same gray value based on the difference between P(i, j) and P(i m, j n) as follows:
8 kqT > < TþðPðim;jnÞPði;jÞÞ ðm – 0; n – 0Þ rði m; i nÞ ¼ 1 : ðm ¼ 0; n ¼ 0Þ > : w1 0 ðm ¼ 0; 2 P n > 0Þ
ð8Þ
k represents the noise penalty factor; the greater the noise, the closer the penalty factor is to one. T represents the threshold and can be determined by the maximum value and minimum value of P, as follows:
T ¼ 2ðmaxðPði; jÞÞ minðPði; jÞÞÞ;
ð9Þ
where max(.) and min(.) are the maximum operand and minimum operand, respectively. The reconstructed results of the simulated data without and with the adaptive bilateral filter are presented in Fig. 3. A Gaussian noise, whose standard deviation is 0.05, is added to the oversampled image. From Fig. 3, it can be observed that the proposed modified bilateral filter can effectively reduce reconstruction error caused by noise. 2.3.2. Reconstruction error caused by boundary approximation The pixels of O are subtracted from the corresponding pixels of O1, or Eq. (4) is subtracted from Eq. (5). The reconstruction error caused by boundary approximation, designated eb, can be expressed as:
eb ði; jÞ ¼ ðqÞiþj2
qbi 1þq
c a1 b1
!
ð1 þ qÞ2 ð1 6 i 6 2M;
þ ðqÞi1
qaj þ ðqÞj1 1þq
1 6 j 6 2NÞ:
ð10Þ
162
X. Dun et al. / Infrared Physics & Technology 68 (2015) 159–166
Fig. 3. Modified bilateral filter is proposed to reduce reconstruction error caused by noise: (a) over-sampled image, (b) and (c) are reconstruction image without and with modified bilateral filtering.
If we define Aj ¼ qj1 rewritten as:
ca1 b1 ð1þqÞ2
eb ði; jÞ ¼ ðqÞi1 Aj þ ðqÞj1 Bi
qa
qbi þ 1þqj , Bj ¼ 1þq , then Eq. (10) can be
ð1 6 i 6 2M;
1 6 j 6 2NÞ:
ð11Þ
In Eq. (11), eb can be divided into two parts: (1) reconstruction error caused by row boundary approximation is positive–negative alternating qi1Aj in the odd and even rows; (2) reconstruction error caused by column boundary approximation is positive– negative alternating qj1Bi in the odd and even columns. This means the boundary approximation leads to an additional difference between adjacent rows (or columns) of the reconstructed image. However, according to mathematical statistics theory, the sum of the odd rows (or columns) of an image that has large samples is approximately equal to the sum of the even rows (or columns). Therefore, the difference between the sum of the odd rows (or columns) of the reconstructed image and the sum of the even rows (or columns) can be used to estimate Aj and Bi. According to the above analysis, the approximate solution of Aj can be expressed as:
Aj ¼
M 1q X ½O1 ð2i 1; jÞ O1 ð2i 1; jÞqð2i=cÞ ; 2M 1 q i¼1
ð12Þ
where c is the penalty factor of the pixel weighting factor, which measures the degree to which the difference between adjacent rows of the reconstructed image is influenced by the row boundary approximation; the smaller the value of q, the smaller the value of c. The experimental results confirm that when q = 0.8, c = 1 will produce an optimal result; whereas, when q = 0.98, c = 8 produces the optimal result. Using Eqs. (11) and (12), the reconstruction error caused by row boundary approximation can be optimized and the intermediate reconstructed image Ot is obtained as:
Ot ði; jÞ ¼ O1 ði; jÞ ðqÞj1 Aj :
ð13Þ
Then, using the intermediate reconstructed image Ot, the approximate solution of Bi can be obtained as:
Bi ¼
N 1q X ½Ot ði; 2j 1Þ O1 ði; 2jÞqð2i=cÞ : 2M 1 q j¼1
ð14Þ
Using Eqs. (11) and (14), the reconstruction error caused by column boundary approximation can be optimized and the final reconstructed image Os is obtained as:
Os ði; jÞ ¼ Ot ði; jÞ ðqÞi1 Bi :
After optimizing the errors qi1Aj and qj1Bi, the final reconstructed image Os becomes equal to the ideal SR image O.
ð15Þ
2.4. Proposed algorithm The proposed algorithm, named noise-robust boundary recursive algorithm (NR-BRA), is expressed in the following steps: Step 1: Create over-sampled image P from direct interlaced interpolation of LR image. If the displacement between LR images is not the ideal 0.5 pixels, the method discussed in [8] is used to amend the construction error of P. Step 2: Calculate the value of parameter q from the fill ratio of the detector and set the value of parameters k(0 6 k 6 1), w(3 6 w 6 7), and c(1 6 c 6 8). Step 3: Recursively reconstruct O1 from the over-sampled image P using Eqs. (5) and (7). Step 4: Optimize the row boundary approximation reconstruction error of O1 using Eqs. (12) and (13) and obtain the intermediate reconstructed image Ot. Step 5: Optimize the column boundary approximation reconstruction error of O1 using Eqs. (14) and (15) and obtain the final reconstructed image Os. 3. Experimental results To demonstrate the advantage of the proposed NR-BRA algorithm compared with other reconstruction algorithms such as BRA-EO [20], OR [7–9], and typical MSRR method L1-BTV [11], we apply the proposed method to both simulated and real data. The L1-BTV is selected as the typical MSRR method because it includes the feature of robust data fusion, preserves edges, and is computationally inexpensive to implement compared to other MSRR methods. 3.1. Applications to simulated data The performance of NR-BRA is studied and compared with the performance of BRA-EO, OR, and L1-BTV by applying these algorithms to 8-bit LR micro-scanning image sequences corrupted by simulated Gaussian or salt and pepper noise. The LR image sequences are generated from an original SR image (256 256) according to the sampling model discussed in Section 2.1. The metrics employed to measure the reconstruction performance are the peak signal-to-noise ratio (PSNR) [25] and universal image quality
X. Dun et al. / Infrared Physics & Technology 68 (2015) 159–166
163
Fig. 4. PSNR curves of reconstructed image of the four methods: (a) result for the first simulation sequence group, corrupted by increasing Gaussian noise, (b) result for the second simulation sequence group, corrupted by increasing salt and pepper noise.
Fig. 5. Reconstructed image of the third simulation sequence group: (a) original SR image, (b) LR image, (c)–(f) SR images reconstructed by BRA-EO, OR, L1-BTV, and NR-BRA, respectively.
index (Q) [26], which are widely used to quantify the differences between two images. PSNR is defined as: b
PSNR ¼ 10log10
!
2 1 ; RMSE
RMSE ¼
rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 X 2 ðOs ði; jÞ Oði; jÞÞ ; i;j MN
ð17Þ
ð16Þ
where b represents the number of bits per pixel in the image, which in this case is equal to eight. The root-mean square error (RMSE) is defined as:
where Os(i, j) is the (i, j)th pixel’s value of the final reconstructed image and O(i, j) is the pixel’s value in the original SR image. Greater values for PSNR indicate superior performance. Q is defined as:
164
X. Dun et al. / Infrared Physics & Technology 68 (2015) 159–166
Table 1 PSNR and Q values of the reconstructed images of the four methods. Method
PSNR
Q
BRA-EO OR NR-BRA L1-BTV
9.75 15.68 19.24 18.10
0.029 0.794 0.924 0.893
Q ¼
r
2 x
4rxy x y h i; 2 2 Þ2 þ ry ðxÞ þ ðy
ð18Þ
P P 1 1 2 2 2 where r2x ¼ MN1 i;j ðOs ði; jÞ xÞ , ry ¼ MN1 i;j ðOði; jÞ yÞ , rxy ¼ P P 1 1 1 x ¼ MN i;j Os ði; jÞ, and y ¼ MN i;j ðOs ði; jÞ xÞðOði; jÞ yÞ, MN1 P i;j Oði; jÞ. The range of Q is [1, 1]. The simulated fill ratio of the detector has the value 0.81 and the level of the simulated Gaussian noise or salt and pepper noise changes once every four LR frames. We have simulated three image sequence groups. In the first group, the LR image sequences are only corrupted by Gaussian noise and the standard deviation r increases from zero to 0.08 with a step of 0.0025. In the second group, the LR image sequences are only corrupted by salt and pepper noise and the noise density D increases from zero to 0.05 with a step of 0.0025. In the third group, the LR image sequences are corrupted by Gaussian noise (r = 0.04) and salt and pepper noise (D = 0.01) simultaneously, and their level remains unchanged. The L1-BTV method is tested with a step size of one, regularization parameter of 0.01, scalar weight of 0.75, iterations of 25, and regularization nuclear dimension of two. The bilateral filter size w is three in NR-BRA, the penalty factor of the pixel weighting factor c is set to one, and the noise penalty factor k is a function of the noise intensity (k = 1 e100s, s = r or D). Fig. 4 presents the PSNR curves of the reconstructed image of the four methods. Fig. 4(a) is the result for the first simulation sequence group; Fig. 4(b) is the result for the second simulation sequence group. It can be observed from Fig. 4 that the NR-BRA method significantly outperformed the other methods as confirmed by its higher PSNR. Although the performance of the NRBRA method decreased with increasing noise level, it remained superior to the performance of the OR and L1-BTV methods. For both sequences, the PSNR curve of the BRA-EO method was the worst. This is because the fill ratio in the BRA-EO method is assumed to be a constant value of one and this assumption causes the BRA-EO method to be extremely sensitive to noise, as discussed in Section 2.3.1. It should be noted that the BRA-EO method was the worst performer even when the LR images were not corrupted by noise. This was because the BRA-EO method could only use the case where the fill ratio of the detector was one, even though the LR images used a fill ratio of 0.81, which was equivalent to increasing the noise in the LR images.
Fig. 6. Photograph of micro-scanning imaging system.
Fig. 5 displays the reconstructed image of the third simulation sequence group. Fig. 5(a) is the original SR image. Fig. 5(b) is the LR image corrupted with simulated noise. The reconstructed results using BRA-EO, OR, L1-BTV, and NR-BRA are presented in Fig. 5(c)–(f), respectively. In the reconstructed image of BRA-EO, it is difficult to identify any information about the original SR image. However, the reconstructed images of OR, L1-BTV, and NR-BRA provide more details than the LR image. In particular, the results of the NR-BRA method have the sharpest edges and are closer to the original SR image. Table 1 presents the quantitative evaluations of the reconstructed images of the four methods. It can be observed that the performance of the NR-BRA method is significantly superior to the other methods.
3.2. Applications to real infrared data In this subsection, the proposed algorithm is applied to actual infrared data. The data was collected using a 320 256 microscanning imaging system operating in the 3.7–4.8 lm range. Fig. 6 is a photograph of the micro-scanning imaging system. The focal length was 200 mm and the pixel spacing was 30 lm 30 lm. The imaging target was a standard resolution target (USAF1951) placed on the focal plane of an infrared collimator (focus length was 1 m) and illuminated by a standard blackbody. A sample image of the test sequence is displayed in Fig. 7. It can be seen that the sample image can only distinguish the sixth target and those targets before the sixth target. An obvious mixing effect can be found in the eighth and tenth targets. (The theoretical resolution of the sample image has been calculated using the parameters of the microscanning imaging system and infrared collimator and the result was that the sample image could distinguish target frequency of 3.33 (cycle/mm), which is located between the sixth and eighth targets. This coincides with the above findings.) The USAF1951 standard resolution target used in this experiment is produced by coating on a quartz glass. The transmittance of the area of the standard resolution target without target is relatively high and the transmittance of the area of the standard resolution target with target is relatively low. Hence, it can be determined from Fig. 7 that the area of the standard resolution target without target is white and the area of the standard resolution target with target is black. The black circle around the LR image is the field diaphragm image of the infrared collimator. The SR reconstructed over the test sequence was completed using the selected algorithms with the same parameter sets as in Section 3.1. The reconstruction results are presented in Fig. 8. For an accurate visual evaluation, only the key areas of each reconstructed image are displayed. Fig. 8(a) is the LR image (real sample image). The reconstructed results using BRA-EO, OR, L1-BTV, and NR-BRA are displayed in Fig. 8(b)–(e), respectively. From Fig. 8, it is evident that the NR-BRA algorithm obtains the best image detail. Furthermore, it effectively reduces the reconstruction error caused
Fig. 7. Sample image of actual infrared data.
X. Dun et al. / Infrared Physics & Technology 68 (2015) 159–166
Fig. 8. Reconstructed image of the real infrared data: (a) LR image, (b)–(e) SR image reconstructed by BRA-EO, OR, L1-BTV, and NR-BRA, respectively.
Fig. 9. DN samples of the reconstructed image at 405th column.
Fig. 10. DN samples of the reconstructed image at 470th column.
165
166
X. Dun et al. / Infrared Physics & Technology 68 (2015) 159–166
Table 2 Comparison of processing time (in seconds) of investigated algorithms. Method
Third simulated data
Real data
Image size BRA-EO OR NR-BRA L1-BTV
128 128 0.026 0.007 0.068 0.650
240 320 0.110 0.053 0.292 3.067
by image noise and boundary approximation. Digital number (DN) samples of the reconstructed image at the 405th column and 470th column are presented in Figs. 9 and 10. Fig. 9 presents the DN of column 405 for each reconstructed image. Fig. 10 presents the DN of column 470 for each reconstructed image. The results for BRA-EO are not included in Figs. 9 and 10 because it was seriously corrupted by reconstruction error. In Figs. 9 and 10, it can be observed that compared with the OR algorithm and L1-BTV methods, the NR-BRA algorithm increased the contrast of the reconstructed image at the eighth target and tenth target more than one time and 0.5 times, respectively. This means that the NR-BRA algorithm obtained shaper edges and results closer to the sub-pixel image than the OR algorithm and L1-BTV method. Table 2 presents a comparison of the average processing time for the real and simulated data using a personal computer and Matlab 2010 (2.9 GHz Intel Core 2 CPU, 4 GB memory). Because the NR-BRA algorithm is more complicated than the OR and BRA-EO algorithms, it required more processing time. However, the time of 0.29 s to process four LR images of size of 240 320 is only one-tenth of the time required by the L1-BTV method (25 iterations). This means the proposed method is more suitable for real-time processing applications. 4. Conclusions A noise-robust boundary recursive algorithm for SR reconstruction of staring FPA micro-scanning imaging was presented. This method implemented a new fill ratio-based sampling model and a modified bilateral filter and gray statistical principle to reconstruct an SR image while suppressing reconstruction error caused by image noise and boundary approximation. The strength of the proposed algorithm is in its noise robustness and less complex algebraic calculation, making it particularly suitable for real-time processing. Experiments were performed to validate the proposed algorithm. It was demonstrated that even in the presence of noise, the proposed algorithm achieved superior results to OR. This is significant progress compared to the previous BRA method. The proposed NR-BRA method resolves the problems of the previous BRA method and confirms that sub-pixel imaging can be realized in practical applications. Conflict of interest There is no conflict of interest.
Acknowledgments This work was supported by the Key Program of National Natural Science Foundation of China (Grant No. 61231014) and the Key Program of GAD’s pre-research project (Grant No. 9140A01010113BQ01001). References [1] X.R. Wang, Z.X. Feng, J.Q. Zhang, Z.J. Yang, Characterization of the performance of micro-scanning infrared imager, Int. J. Infrared Millimeter Waves 25 (2004) 1657–1668. [2] J. Wang, F. Zhou, Y. Pan, A method for improving the effective resolution of space optical remote sensor, Spacecraft Recov. Rem. Sens. 4 (2011) 24–29. [3] X. Sui, Q. Chen, G. Gu, Micro-scanning system using flat optics for resolution improvement of infrared images, Opt.-Int. J. Light Electron Opt. 124 (2013) 2292–2297. [4] J.M. Wiltse, J.L. Miller, Imagery improvements in staring infrared imagers by employing sub-pixel micro-scan, Opt. Eng. 44 (2005) 056401–056409. [5] X. Sui, H. Gao, Y. Sun, Q. Chen, G. Gu, Infrared super-resolution imaging method based on retina micro-motion, Infrared Phys. Technol. 60 (2013) 340–345. [6] S. Li, T.P. Chu, Super-resolution image reconstruction for ultrasonic nondestructive evaluation, IEEE Trans. Ultrason., Ferroelectr. Freq. Control 60 (2013) 2575–2585. [7] K.M. Hock, Effect of oversampling in pixel arrays, Opt. Eng. 34 (1995) 1281– 1288. [8] M.J. Gao, W.Q. Jin, X. Wang, J. Yu, Y. Chen, Inter-frame difference oversample reconstruction for optical micro-scanning thermal microscope imaging system, Trans. Beijing Inst. Technol. 29 (2009) 704–707. [9] J.Q. Bai, Q. Chen, Z.Y. Wang, An improved high-resolution reconstruction algorithm for staring infrared image, Acta. Opt. Sin. 1 (2010) 86–90. [10] X. Zhao, C. Li, Study of optical micro scanning reconstruction, in: SPIE Optical Engineering Applications, International Society for Optics and Photonics, 2012, pp. 85000C. [11] S. Farsiu, M.D. Robinson, M. Elad, P. Milanfar, Fast and robust multiframe super resolution, IEEE Trans. Image Process. 13 (2004) 1327–1344. [12] H. Ur, D. Gross, Improved resolution from subpixel shifted pictures, CVGIP: Graph. Models Image Proc. 54 (1992) 181–186. [13] H. Stark, P. Oskoui, High-resolution image recovery from image-plane arrays, using convex projections, JOSA A 6 (1989) 1715–1726. [14] S.P. Belekos, N.P. Galatsanos, A.K. Katsaggelos, Maximum a posteriori video super-resolution using a new multichannel image prior, IEEE Trans. Image Process. 19 (2010) 1451–1464. [15] M.V.W. Zibetti, J. Mayer, A robust and computationally efficient simultaneous super-resolution scheme for image sequences, IEEE Trans. Circuits Syst. Video Technol. 17 (2007) 1288–1300. [16] Q. Yuan, L. Zhang, H. Shen, P. Li, Adaptive multiple-frame image superresolution based on U-curve, IEEE Trans. Image Process. 19 (2010) 3157–3170. [17] S.C. Park, M.K. Park, M.G. Kang, Super-resolution image reconstruction: a technical overview, IEEE Signal Process. Mag. 20 (2003) 21–36. [18] S. Chaudhuri, Super-Resolution Imaging, Springer, 2001. [19] C. Chen, H. Liang, S. Zhao, Z. Lyu, M. Sarem, A novel multi-image superresolution reconstruction method using anisotropic fractional order adaptive norm, Vis. Comput. (2014) 1–15. [20] W. Jin, C. Wang, N. Zhang, L. Wang, P. Lu, Study on sub-pixel processing algorithm for scanning FPA thermal imaging system, J. Infrared Millimeter Waves 24 (2005) 261–264. [21] C. Xu, W.Q. Jin, Y.Q. Li, X. Wang, An improved sub-pixel processing algorithm based on focal plane detector, Acta Electron. Sin. 35 (2007) 1608–1611. [22] Y. Chen, W. Jin, L. Wang, X. Wang, Sub-pixel processing algorithm of the reducing boundary recursive error in staring FPA micro-scanning imaging, Chin. Opt. Lett. 8 (2010) 146–150. [23] Y. Chen, W.Q. Jin, L.X. Wang, B. Liu, C.L. Liu, Multi-frame sub-pixel processing algorithm based on uncontrolled micro-scanning, Opt. Commun. 283 (2010) 3931–3936. [24] C. Tomasi, R. Manduchi, Bilateral filtering for gray and color images, in: IEEE Sixth International Conference on Computer Vision, 1998, pp. 839–846. [25] T.N. Pappas, R.J. Safranek, J. Chen, Perceptual criteria for image quality evaluation, Handbook of image and video processing, Academic Press, New York, 2000. pp. 669–684. [26] Z. Wang, A.C. Bovik, A universal image quality index, IEEE Signal Proc. Lett. 9 (2002) 81–84.
Contents lists available at ScienceDirect
Infrared Physics & Technology journal homepage: www.elsevier.com/locate/infrared
Noise-robust boundary recursive algorithm for super-resolution reconstruction of staring focal plane array micro-scanning imaging Xiong Dun ⇑, Weiqi Jin, Lu Lu MoE Key Laboratory of Photoelectronic Imaging Technology and System, School of Optoelectronics, Beijing Institute of Technology, Beijing 100089, China
h i g h l i g h t s A noise-robust boundary recursive algorithm for super-resolution reconstruction is proposed. We give out a new fill ratio based sampling model. We use a modified bilateral filter and gray statistical principle to reduce reconstruction error. The proposed algorithm particularly suitable for real-time processing.
a r t i c l e
i n f o
Article history: Received 9 September 2014 Available online 10 December 2014 Keywords: Micro-scanning Super-resolution reconstruction Recursive reconstruction
a b s t r a c t A boundary recursive algorithm for the super-resolution reconstruction of staring focal plane array (FPA) micro-scanning imaging with consideration of a fill ratio-based sampling model is presented. The reconstruction errors of the algorithm introduced by image noise and boundary approximation are analyzed. Then, a modified bilateral filter and gray statistical principle are used in the algorithm to reduce these errors. Simulation and actual imaging experiments confirm that the proposed algorithm has effective noise robustness and can achieve superior results compared to an over-sampled reconstruction in the presence of low noise. This algorithm can achieve ideal sub-pixel imaging and has excellent immunity to noise. Its application will enhance the performance of optoelectronic-imaging systems. Ó 2014 Elsevier B.V. All rights reserved.
1. Introduction Micro-scanning imaging is characterized by a high sampling ratio that can be exploited to improve spatial resolution and achieve sub-pixel imaging without reducing signal noise ratio (SNR) in an optoelectronic-imaging device [1]. It has been successfully applied in the fields of remote sensing [2], thermal imaging [3–5], and microscopy imaging [6]. The principal step of micro-scanning imaging is the superresolution (SR) reconstruction and significant achievements have been made in this area. The most common reconstruction algorithm is the direct interlaced interpolation of multi-frame image sequences, called over-sampled reconstruction (OR) [7–10]. According to [7] and as we will confirm in a subsequent section, interlaced interpolation only increases the sampling rate rather than improving the pixel integral blur effect. Consequently, the image reconstructed by OR is a convolution of the ideal SR image and pixel integral window. This means that OR is not an ideal sub-pixel ⇑ Corresponding author. E-mail address: [email protected] (X. Dun). http://dx.doi.org/10.1016/j.infrared.2014.12.004 1350-4495/Ó 2014 Elsevier B.V. All rights reserved.
imaging solution. The advantage of OR is that the processing time is significantly reduced and therefore, it is suitable for a real-time micro-scanning imaging system. In recent years, to address the problem that sub-pixel displacement is not an ideal value, Interframe difference-based OR [8] and Lagrange polynomial-based OR [9] have been presented. However, the images reconstructed by these approaches remain blurred by the pixel integral window. Other common reconstruction algorithms are the multiframe SR reconstruction (MSRR) methods. There are numerous journal papers that discuss these MSRR methods [11–16]. A sample of reviews of state-of-the-art MSRR methods can be found in [17,18]. The benefits of the MSRR methods are that they not only increase the sampling rate but also reduce the pixel integral blur effect. However, current MSRR methods are time-consuming because the calculation is complicated and the number of iterations required to converge is significant [19]. This makes MSRR methods difficult to apply to real-time micro-scanning imaging systems. In recent years, boundary recursive algorithms (BRA) have been developed [20,21]. Not only can these methods obtain the same performance as half-sized pixel imaging (ideal sub-pixel imaging),
160
X. Dun et al. / Infrared Physics & Technology 68 (2015) 159–166
but they are also computationally inexpensive and suitable for real-time application. These algorithms require a precise boundary as an initial value of the recursive process. Chen subsequently determined that an approximation boundary would introduce positive–negative alternating error and used a gray statistical principle to address the error. Then, a boundary recursive algorithm with error optimization (BRA-EO) was provided [22,23]. It can acquire approximate performance of half-sized pixel imaging without a precise boundary. However, as we will demonstrate, the BRA methods mentioned above have poor immunity to noise and can only be utilized in the case where the fill ratio of the detector is one. Because the fill ratio of a detector is always less than one, random noise is inevitable in the actual imaging. Hence, the advantages of the BRA methods cannot be realized and this method is sometimes unable to reconstruct an SR image. Therefore, it is both necessary and important to study a new boundary recursive algorithm that demonstrates superior immunity to noise and considers the fill rate of the detector. In this paper, we develop a noise-robust boundary recursive algorithm based on a new fill ratio-based sampling model and present simulation and actual imaging experiments to verify the effectiveness of the algorithm. 2. Algorithm principal 2.1. Fill ratio-based sampling model There is a gap between adjacent detector units due to the requirements of the electrode wiring and reduced lateral diffusion. This causes the fill ratio of the detector to be less than one and means that the ideal SR image will acquire a different contribution from the low-resolution (LR) image, which is directly acquired by the micro-scanning imaging device. If we use a 2 2 microscanning as an example, each LR pixel is generated by four SR pixels with weights of [q2, q; q, 1] as illustrated in Fig. 1. q is called the pixel weighting factor and is expressed as:
q¼
( pffiffiffi 2 d 1 ð0:25 6 d 6 1Þ 0
ðd < 0:25Þ
;
ð1Þ
where d is the fill ratio of the detector (the size of detector unit is d d and the gap between adjacent detector units is l l, then d = (d/l)2). As indicated in Fig. 2, for the 2 2 micro-scanning, we can acquire four LR images by moving the FPA one half of the detector unit (one SR pixel) to the right, down, left, and up relative to the target scene, assuming the image scene is fixed. Interlaced interpolation is performed on the four LR images according to the
acquiring order to create an over-sampled image P. The pixels of the over-sampled image P can be written as:
Pði; jÞ ¼
Oði; jÞ þ qOði 1; jÞ þ qOði; j 1Þ þ q2 Oði 1; j 1Þ ð1 þ qÞ2 þ nði; jÞ ð2M P i P 1; 2N P j P 1Þ;
ð2Þ
where O(i, j) are the pixels of the ideal SR image in Fig. 1, O(0, j) and O(i, 0) are the boundary pixels, and n is the image noise of the LR image. According to Eq. (2), the over-sampled image P is the convolution of the ideal SR image O and the weight function [q2, q; q, 1]. This proves that the OR method cannot obtain ideal sub-pixel imaging. 2.2. Recursive reconstruction formula based on the sampling model The boundary pixels of O can be expressed as a function of the first row pixel or first column pixel of O:
Oði; 0Þ ¼ Oði; 1Þ þ bi ; Oð0;jÞ ¼ Oð1; jÞ þ aj ; Oð0;0Þ ¼ Oð1; 1Þ þ c;
ð3Þ
where 1 6 i 6 2M, 1 6 j 6 2N, and aj, bi, and c are the corresponding boundary approximation errors. According to Eqs. (2) and (3), the ideal SR image O can be expressed as: 8 a1 b1 qa1 qb1 Oð1;1Þ ¼ Pð1;1Þ nð1; 1Þ cð1þqÞ > 2 1þq 1þq > > > > > > qa q2 aj1 > > Oð1;jÞ ¼ ð1 þ qÞðPð1;jÞ nð1;jÞÞ qOð1j 1Þ 1þqj þ 1þq > < qb
Oði;1Þ ¼ ð1 þ qÞðPði; jÞ nði; jÞÞ qOði 1; 1Þ 1þqi þ > > > > > > > Oði;jÞ ¼ ð1 þ 2q þ q2 ÞðPði; jÞ nði;jÞÞ qOði 1;jÞ > > > : qOði;j 1Þ q2 Oði 1:j 1Þ
q2 bi1 1þq
ð2 6 i 6 2M; 2 6 j 6 2NÞ:
ð4Þ If we ignore the effect of noise and boundary approximation error and use the approximation SR image O1 instead of O, then, Eq. (4) can be rewritten as:
8 O1 ð1; 1Þ ¼ Pð1; 1Þ > > > > > > < O1 ð1; jÞ ¼ ð1 þ qÞPð1; jÞ qOð1j 1Þ O1 ði; 1Þ ¼ ð1 þ qÞPði;jÞ qOði 1; 1Þ ð2 6 i 6 2M; 2 6 j 6 2NÞ: > > > > O1 ði; jÞ ¼ ð1 þ 2q þ q2 ÞPði; jÞ qOði 1; jÞ > > : qOði;j 1Þ q2 Oði 1:j 1Þ ð5Þ It can be seen from Eq. (5) that if the pixel weighting factor q is known beforehand, O1 can be sequentially reconstructed from the over-sampled image P. Considering the special case that the fill
Fig. 1. Fill ratio-based sampling model.
161
X. Dun et al. / Infrared Physics & Technology 68 (2015) 159–166
Fig. 2. (a) Micro-scanning mode, (b) over-sampled image P.
ratio of the detector is one, the recursive reconstruction formula Eq. (5) is the same as that in [22]. This means that the BRA-EO method is a special case of the proposed method with the fill ratio of the detector set to one. 2.3. Analysis of the reconstruction error In Eq. (5), the noise and boundary approximation error are set to zero, thus the O1 from Eq. (5) has some error as compared with the ideal SR image O. The key step of obtaining O is the analysis and optimization of the reconstruction error caused by noise and boundary approximation. 2.3.1. Reconstruction error caused by noise The pixels of O are subtracted from the corresponding pixels of O1, or Eq. (4) is subtracted from Eq. (5). The reconstruction error caused by noise, designated en, can be expressed as: 8 en ð1; 1Þ ¼ nði; jÞ > > > > > en ð1; jÞ ¼ ð1 þ qÞ½nð1; jÞ qnð1; j 1Þ > > > > > > þq2 nð1; j 2Þ þ þ ð1Þj1 qj1 nð1; 1Þ > > > > > e ði; 1Þ ¼ ð1 þ qÞ½nði; 1Þ qnði 1; 1Þ > n < ð2 6 i 6 2M; 2 6 j 6 2NÞ: þq2 nði 2; 1Þ þ þ ð1Þi1 qi1 nð1;1Þ > > > > en ði; jÞ ¼ ð1 þ 2q þ q2 Þ½nði; jÞ qnði 1; jÞ > > > > > qnði; j 1Þ þ q2 nði 2; jÞ þ q2 nði; j 2Þ > > > > > þq2 nði 1; j 1Þ . . . q3 nði 3; jÞ > > > : i1 j1 q3 nði; j 3Þ þ þ ð1Þ qi1 nð1; jÞ þ ð1Þ qj1 nði; 1Þ
ð6Þ
Two important discussions are obtained in Eq. (6) as follows: (1) If q is equal to one, the en of an arbitrary recursive point is influenced by the noise of all the recursive points that have been previously calculated. This means that once there is a recursive point whose noise is significant (for example, the value of the recursive point has an abnormal response), then, the en of all the following recursive points will be significant, resulting in an O1 that is a significant deviation from the ideal SR image. This is proof that the BRA-EO method discussed in [21] has poor immunity to noise. (2) If q is smaller than one, the en of an arbitrary recursive point is only influenced by the noise of the recursive points within the corresponding region. The point’s region number can be expressed as [ceiling(ln 100/ln q)]2. If we use q = 0.8 as an example, the point’s region number is 21 21, which is substantially smaller than a general image resolution (320 256). This means the accumulation of en is local and controllable.
Because the fill ratio of the detector is typically less than one, then q is certainly less than one. Therefore, according to the above analysis, the reconstruction error caused by noise is not serious in practice. Nevertheless, the noise will continue to add some reconstruction error, especially when q is close to one. Therefore, we modify the adaptive bilateral filter [24] and apply it to the recursive reconstruction formula as follows:
P0
m¼w1 2
O1 ði; jÞ ¼
Pw1 2
rði m; j nÞO1 ði m; j nÞ ; Pw1 2 rði m; j nÞ m¼w1 n¼w1
P0
n¼w1 2 2
ð7Þ
2
where w is the size of the bilateral filter and r(i m, j n) is the likelihood function, which computes the likelihood that O1(i, j) and O1(i m, j n) have the same gray value based on the difference between P(i, j) and P(i m, j n) as follows:
8 kqT > < TþðPðim;jnÞPði;jÞÞ ðm – 0; n – 0Þ rði m; i nÞ ¼ 1 : ðm ¼ 0; n ¼ 0Þ > : w1 0 ðm ¼ 0; 2 P n > 0Þ
ð8Þ
k represents the noise penalty factor; the greater the noise, the closer the penalty factor is to one. T represents the threshold and can be determined by the maximum value and minimum value of P, as follows:
T ¼ 2ðmaxðPði; jÞÞ minðPði; jÞÞÞ;
ð9Þ
where max(.) and min(.) are the maximum operand and minimum operand, respectively. The reconstructed results of the simulated data without and with the adaptive bilateral filter are presented in Fig. 3. A Gaussian noise, whose standard deviation is 0.05, is added to the oversampled image. From Fig. 3, it can be observed that the proposed modified bilateral filter can effectively reduce reconstruction error caused by noise. 2.3.2. Reconstruction error caused by boundary approximation The pixels of O are subtracted from the corresponding pixels of O1, or Eq. (4) is subtracted from Eq. (5). The reconstruction error caused by boundary approximation, designated eb, can be expressed as:
eb ði; jÞ ¼ ðqÞiþj2
qbi 1þq
c a1 b1
!
ð1 þ qÞ2 ð1 6 i 6 2M;
þ ðqÞi1
qaj þ ðqÞj1 1þq
1 6 j 6 2NÞ:
ð10Þ
162
X. Dun et al. / Infrared Physics & Technology 68 (2015) 159–166
Fig. 3. Modified bilateral filter is proposed to reduce reconstruction error caused by noise: (a) over-sampled image, (b) and (c) are reconstruction image without and with modified bilateral filtering.
If we define Aj ¼ qj1 rewritten as:
ca1 b1 ð1þqÞ2
eb ði; jÞ ¼ ðqÞi1 Aj þ ðqÞj1 Bi
qa
qbi þ 1þqj , Bj ¼ 1þq , then Eq. (10) can be
ð1 6 i 6 2M;
1 6 j 6 2NÞ:
ð11Þ
In Eq. (11), eb can be divided into two parts: (1) reconstruction error caused by row boundary approximation is positive–negative alternating qi1Aj in the odd and even rows; (2) reconstruction error caused by column boundary approximation is positive– negative alternating qj1Bi in the odd and even columns. This means the boundary approximation leads to an additional difference between adjacent rows (or columns) of the reconstructed image. However, according to mathematical statistics theory, the sum of the odd rows (or columns) of an image that has large samples is approximately equal to the sum of the even rows (or columns). Therefore, the difference between the sum of the odd rows (or columns) of the reconstructed image and the sum of the even rows (or columns) can be used to estimate Aj and Bi. According to the above analysis, the approximate solution of Aj can be expressed as:
Aj ¼
M 1q X ½O1 ð2i 1; jÞ O1 ð2i 1; jÞqð2i=cÞ ; 2M 1 q i¼1
ð12Þ
where c is the penalty factor of the pixel weighting factor, which measures the degree to which the difference between adjacent rows of the reconstructed image is influenced by the row boundary approximation; the smaller the value of q, the smaller the value of c. The experimental results confirm that when q = 0.8, c = 1 will produce an optimal result; whereas, when q = 0.98, c = 8 produces the optimal result. Using Eqs. (11) and (12), the reconstruction error caused by row boundary approximation can be optimized and the intermediate reconstructed image Ot is obtained as:
Ot ði; jÞ ¼ O1 ði; jÞ ðqÞj1 Aj :
ð13Þ
Then, using the intermediate reconstructed image Ot, the approximate solution of Bi can be obtained as:
Bi ¼
N 1q X ½Ot ði; 2j 1Þ O1 ði; 2jÞqð2i=cÞ : 2M 1 q j¼1
ð14Þ
Using Eqs. (11) and (14), the reconstruction error caused by column boundary approximation can be optimized and the final reconstructed image Os is obtained as:
Os ði; jÞ ¼ Ot ði; jÞ ðqÞi1 Bi :
After optimizing the errors qi1Aj and qj1Bi, the final reconstructed image Os becomes equal to the ideal SR image O.
ð15Þ
2.4. Proposed algorithm The proposed algorithm, named noise-robust boundary recursive algorithm (NR-BRA), is expressed in the following steps: Step 1: Create over-sampled image P from direct interlaced interpolation of LR image. If the displacement between LR images is not the ideal 0.5 pixels, the method discussed in [8] is used to amend the construction error of P. Step 2: Calculate the value of parameter q from the fill ratio of the detector and set the value of parameters k(0 6 k 6 1), w(3 6 w 6 7), and c(1 6 c 6 8). Step 3: Recursively reconstruct O1 from the over-sampled image P using Eqs. (5) and (7). Step 4: Optimize the row boundary approximation reconstruction error of O1 using Eqs. (12) and (13) and obtain the intermediate reconstructed image Ot. Step 5: Optimize the column boundary approximation reconstruction error of O1 using Eqs. (14) and (15) and obtain the final reconstructed image Os. 3. Experimental results To demonstrate the advantage of the proposed NR-BRA algorithm compared with other reconstruction algorithms such as BRA-EO [20], OR [7–9], and typical MSRR method L1-BTV [11], we apply the proposed method to both simulated and real data. The L1-BTV is selected as the typical MSRR method because it includes the feature of robust data fusion, preserves edges, and is computationally inexpensive to implement compared to other MSRR methods. 3.1. Applications to simulated data The performance of NR-BRA is studied and compared with the performance of BRA-EO, OR, and L1-BTV by applying these algorithms to 8-bit LR micro-scanning image sequences corrupted by simulated Gaussian or salt and pepper noise. The LR image sequences are generated from an original SR image (256 256) according to the sampling model discussed in Section 2.1. The metrics employed to measure the reconstruction performance are the peak signal-to-noise ratio (PSNR) [25] and universal image quality
X. Dun et al. / Infrared Physics & Technology 68 (2015) 159–166
163
Fig. 4. PSNR curves of reconstructed image of the four methods: (a) result for the first simulation sequence group, corrupted by increasing Gaussian noise, (b) result for the second simulation sequence group, corrupted by increasing salt and pepper noise.
Fig. 5. Reconstructed image of the third simulation sequence group: (a) original SR image, (b) LR image, (c)–(f) SR images reconstructed by BRA-EO, OR, L1-BTV, and NR-BRA, respectively.
index (Q) [26], which are widely used to quantify the differences between two images. PSNR is defined as: b
PSNR ¼ 10log10
!
2 1 ; RMSE
RMSE ¼
rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 X 2 ðOs ði; jÞ Oði; jÞÞ ; i;j MN
ð17Þ
ð16Þ
where b represents the number of bits per pixel in the image, which in this case is equal to eight. The root-mean square error (RMSE) is defined as:
where Os(i, j) is the (i, j)th pixel’s value of the final reconstructed image and O(i, j) is the pixel’s value in the original SR image. Greater values for PSNR indicate superior performance. Q is defined as:
164
X. Dun et al. / Infrared Physics & Technology 68 (2015) 159–166
Table 1 PSNR and Q values of the reconstructed images of the four methods. Method
PSNR
Q
BRA-EO OR NR-BRA L1-BTV
9.75 15.68 19.24 18.10
0.029 0.794 0.924 0.893
Q ¼
r
2 x
4rxy x y h i; 2 2 Þ2 þ ry ðxÞ þ ðy
ð18Þ
P P 1 1 2 2 2 where r2x ¼ MN1 i;j ðOs ði; jÞ xÞ , ry ¼ MN1 i;j ðOði; jÞ yÞ , rxy ¼ P P 1 1 1 x ¼ MN i;j Os ði; jÞ, and y ¼ MN i;j ðOs ði; jÞ xÞðOði; jÞ yÞ, MN1 P i;j Oði; jÞ. The range of Q is [1, 1]. The simulated fill ratio of the detector has the value 0.81 and the level of the simulated Gaussian noise or salt and pepper noise changes once every four LR frames. We have simulated three image sequence groups. In the first group, the LR image sequences are only corrupted by Gaussian noise and the standard deviation r increases from zero to 0.08 with a step of 0.0025. In the second group, the LR image sequences are only corrupted by salt and pepper noise and the noise density D increases from zero to 0.05 with a step of 0.0025. In the third group, the LR image sequences are corrupted by Gaussian noise (r = 0.04) and salt and pepper noise (D = 0.01) simultaneously, and their level remains unchanged. The L1-BTV method is tested with a step size of one, regularization parameter of 0.01, scalar weight of 0.75, iterations of 25, and regularization nuclear dimension of two. The bilateral filter size w is three in NR-BRA, the penalty factor of the pixel weighting factor c is set to one, and the noise penalty factor k is a function of the noise intensity (k = 1 e100s, s = r or D). Fig. 4 presents the PSNR curves of the reconstructed image of the four methods. Fig. 4(a) is the result for the first simulation sequence group; Fig. 4(b) is the result for the second simulation sequence group. It can be observed from Fig. 4 that the NR-BRA method significantly outperformed the other methods as confirmed by its higher PSNR. Although the performance of the NRBRA method decreased with increasing noise level, it remained superior to the performance of the OR and L1-BTV methods. For both sequences, the PSNR curve of the BRA-EO method was the worst. This is because the fill ratio in the BRA-EO method is assumed to be a constant value of one and this assumption causes the BRA-EO method to be extremely sensitive to noise, as discussed in Section 2.3.1. It should be noted that the BRA-EO method was the worst performer even when the LR images were not corrupted by noise. This was because the BRA-EO method could only use the case where the fill ratio of the detector was one, even though the LR images used a fill ratio of 0.81, which was equivalent to increasing the noise in the LR images.
Fig. 6. Photograph of micro-scanning imaging system.
Fig. 5 displays the reconstructed image of the third simulation sequence group. Fig. 5(a) is the original SR image. Fig. 5(b) is the LR image corrupted with simulated noise. The reconstructed results using BRA-EO, OR, L1-BTV, and NR-BRA are presented in Fig. 5(c)–(f), respectively. In the reconstructed image of BRA-EO, it is difficult to identify any information about the original SR image. However, the reconstructed images of OR, L1-BTV, and NR-BRA provide more details than the LR image. In particular, the results of the NR-BRA method have the sharpest edges and are closer to the original SR image. Table 1 presents the quantitative evaluations of the reconstructed images of the four methods. It can be observed that the performance of the NR-BRA method is significantly superior to the other methods.
3.2. Applications to real infrared data In this subsection, the proposed algorithm is applied to actual infrared data. The data was collected using a 320 256 microscanning imaging system operating in the 3.7–4.8 lm range. Fig. 6 is a photograph of the micro-scanning imaging system. The focal length was 200 mm and the pixel spacing was 30 lm 30 lm. The imaging target was a standard resolution target (USAF1951) placed on the focal plane of an infrared collimator (focus length was 1 m) and illuminated by a standard blackbody. A sample image of the test sequence is displayed in Fig. 7. It can be seen that the sample image can only distinguish the sixth target and those targets before the sixth target. An obvious mixing effect can be found in the eighth and tenth targets. (The theoretical resolution of the sample image has been calculated using the parameters of the microscanning imaging system and infrared collimator and the result was that the sample image could distinguish target frequency of 3.33 (cycle/mm), which is located between the sixth and eighth targets. This coincides with the above findings.) The USAF1951 standard resolution target used in this experiment is produced by coating on a quartz glass. The transmittance of the area of the standard resolution target without target is relatively high and the transmittance of the area of the standard resolution target with target is relatively low. Hence, it can be determined from Fig. 7 that the area of the standard resolution target without target is white and the area of the standard resolution target with target is black. The black circle around the LR image is the field diaphragm image of the infrared collimator. The SR reconstructed over the test sequence was completed using the selected algorithms with the same parameter sets as in Section 3.1. The reconstruction results are presented in Fig. 8. For an accurate visual evaluation, only the key areas of each reconstructed image are displayed. Fig. 8(a) is the LR image (real sample image). The reconstructed results using BRA-EO, OR, L1-BTV, and NR-BRA are displayed in Fig. 8(b)–(e), respectively. From Fig. 8, it is evident that the NR-BRA algorithm obtains the best image detail. Furthermore, it effectively reduces the reconstruction error caused
Fig. 7. Sample image of actual infrared data.
X. Dun et al. / Infrared Physics & Technology 68 (2015) 159–166
Fig. 8. Reconstructed image of the real infrared data: (a) LR image, (b)–(e) SR image reconstructed by BRA-EO, OR, L1-BTV, and NR-BRA, respectively.
Fig. 9. DN samples of the reconstructed image at 405th column.
Fig. 10. DN samples of the reconstructed image at 470th column.
165
166
X. Dun et al. / Infrared Physics & Technology 68 (2015) 159–166
Table 2 Comparison of processing time (in seconds) of investigated algorithms. Method
Third simulated data
Real data
Image size BRA-EO OR NR-BRA L1-BTV
128 128 0.026 0.007 0.068 0.650
240 320 0.110 0.053 0.292 3.067
by image noise and boundary approximation. Digital number (DN) samples of the reconstructed image at the 405th column and 470th column are presented in Figs. 9 and 10. Fig. 9 presents the DN of column 405 for each reconstructed image. Fig. 10 presents the DN of column 470 for each reconstructed image. The results for BRA-EO are not included in Figs. 9 and 10 because it was seriously corrupted by reconstruction error. In Figs. 9 and 10, it can be observed that compared with the OR algorithm and L1-BTV methods, the NR-BRA algorithm increased the contrast of the reconstructed image at the eighth target and tenth target more than one time and 0.5 times, respectively. This means that the NR-BRA algorithm obtained shaper edges and results closer to the sub-pixel image than the OR algorithm and L1-BTV method. Table 2 presents a comparison of the average processing time for the real and simulated data using a personal computer and Matlab 2010 (2.9 GHz Intel Core 2 CPU, 4 GB memory). Because the NR-BRA algorithm is more complicated than the OR and BRA-EO algorithms, it required more processing time. However, the time of 0.29 s to process four LR images of size of 240 320 is only one-tenth of the time required by the L1-BTV method (25 iterations). This means the proposed method is more suitable for real-time processing applications. 4. Conclusions A noise-robust boundary recursive algorithm for SR reconstruction of staring FPA micro-scanning imaging was presented. This method implemented a new fill ratio-based sampling model and a modified bilateral filter and gray statistical principle to reconstruct an SR image while suppressing reconstruction error caused by image noise and boundary approximation. The strength of the proposed algorithm is in its noise robustness and less complex algebraic calculation, making it particularly suitable for real-time processing. Experiments were performed to validate the proposed algorithm. It was demonstrated that even in the presence of noise, the proposed algorithm achieved superior results to OR. This is significant progress compared to the previous BRA method. The proposed NR-BRA method resolves the problems of the previous BRA method and confirms that sub-pixel imaging can be realized in practical applications. Conflict of interest There is no conflict of interest.
Acknowledgments This work was supported by the Key Program of National Natural Science Foundation of China (Grant No. 61231014) and the Key Program of GAD’s pre-research project (Grant No. 9140A01010113BQ01001). References [1] X.R. Wang, Z.X. Feng, J.Q. Zhang, Z.J. Yang, Characterization of the performance of micro-scanning infrared imager, Int. J. Infrared Millimeter Waves 25 (2004) 1657–1668. [2] J. Wang, F. Zhou, Y. Pan, A method for improving the effective resolution of space optical remote sensor, Spacecraft Recov. Rem. Sens. 4 (2011) 24–29. [3] X. Sui, Q. Chen, G. Gu, Micro-scanning system using flat optics for resolution improvement of infrared images, Opt.-Int. J. Light Electron Opt. 124 (2013) 2292–2297. [4] J.M. Wiltse, J.L. Miller, Imagery improvements in staring infrared imagers by employing sub-pixel micro-scan, Opt. Eng. 44 (2005) 056401–056409. [5] X. Sui, H. Gao, Y. Sun, Q. Chen, G. Gu, Infrared super-resolution imaging method based on retina micro-motion, Infrared Phys. Technol. 60 (2013) 340–345. [6] S. Li, T.P. Chu, Super-resolution image reconstruction for ultrasonic nondestructive evaluation, IEEE Trans. Ultrason., Ferroelectr. Freq. Control 60 (2013) 2575–2585. [7] K.M. Hock, Effect of oversampling in pixel arrays, Opt. Eng. 34 (1995) 1281– 1288. [8] M.J. Gao, W.Q. Jin, X. Wang, J. Yu, Y. Chen, Inter-frame difference oversample reconstruction for optical micro-scanning thermal microscope imaging system, Trans. Beijing Inst. Technol. 29 (2009) 704–707. [9] J.Q. Bai, Q. Chen, Z.Y. Wang, An improved high-resolution reconstruction algorithm for staring infrared image, Acta. Opt. Sin. 1 (2010) 86–90. [10] X. Zhao, C. Li, Study of optical micro scanning reconstruction, in: SPIE Optical Engineering Applications, International Society for Optics and Photonics, 2012, pp. 85000C. [11] S. Farsiu, M.D. Robinson, M. Elad, P. Milanfar, Fast and robust multiframe super resolution, IEEE Trans. Image Process. 13 (2004) 1327–1344. [12] H. Ur, D. Gross, Improved resolution from subpixel shifted pictures, CVGIP: Graph. Models Image Proc. 54 (1992) 181–186. [13] H. Stark, P. Oskoui, High-resolution image recovery from image-plane arrays, using convex projections, JOSA A 6 (1989) 1715–1726. [14] S.P. Belekos, N.P. Galatsanos, A.K. Katsaggelos, Maximum a posteriori video super-resolution using a new multichannel image prior, IEEE Trans. Image Process. 19 (2010) 1451–1464. [15] M.V.W. Zibetti, J. Mayer, A robust and computationally efficient simultaneous super-resolution scheme for image sequences, IEEE Trans. Circuits Syst. Video Technol. 17 (2007) 1288–1300. [16] Q. Yuan, L. Zhang, H. Shen, P. Li, Adaptive multiple-frame image superresolution based on U-curve, IEEE Trans. Image Process. 19 (2010) 3157–3170. [17] S.C. Park, M.K. Park, M.G. Kang, Super-resolution image reconstruction: a technical overview, IEEE Signal Process. Mag. 20 (2003) 21–36. [18] S. Chaudhuri, Super-Resolution Imaging, Springer, 2001. [19] C. Chen, H. Liang, S. Zhao, Z. Lyu, M. Sarem, A novel multi-image superresolution reconstruction method using anisotropic fractional order adaptive norm, Vis. Comput. (2014) 1–15. [20] W. Jin, C. Wang, N. Zhang, L. Wang, P. Lu, Study on sub-pixel processing algorithm for scanning FPA thermal imaging system, J. Infrared Millimeter Waves 24 (2005) 261–264. [21] C. Xu, W.Q. Jin, Y.Q. Li, X. Wang, An improved sub-pixel processing algorithm based on focal plane detector, Acta Electron. Sin. 35 (2007) 1608–1611. [22] Y. Chen, W. Jin, L. Wang, X. Wang, Sub-pixel processing algorithm of the reducing boundary recursive error in staring FPA micro-scanning imaging, Chin. Opt. Lett. 8 (2010) 146–150. [23] Y. Chen, W.Q. Jin, L.X. Wang, B. Liu, C.L. Liu, Multi-frame sub-pixel processing algorithm based on uncontrolled micro-scanning, Opt. Commun. 283 (2010) 3931–3936. [24] C. Tomasi, R. Manduchi, Bilateral filtering for gray and color images, in: IEEE Sixth International Conference on Computer Vision, 1998, pp. 839–846. [25] T.N. Pappas, R.J. Safranek, J. Chen, Perceptual criteria for image quality evaluation, Handbook of image and video processing, Academic Press, New York, 2000. pp. 669–684. [26] Z. Wang, A.C. Bovik, A universal image quality index, IEEE Signal Proc. Lett. 9 (2002) 81–84.