- Email: [email protected]
A sur-pixel scan method for super-resolution reconstruction
Optik 124 (2013) 6905–6909
Contents lists available at ScienceDirect
Optik journal homepage: www.elsevier.de/ijleo
A sur-pixel scan method for super-resolution reconstruction Mingjie Sun ∗ , Kanglong Yu Institution of Opto-electrionics Technology, Beihang University, Beijing, China
a r t i c l e
i n f o
Article history: Received 12 January 2013 Accepted 25 May 2013 Keywords: Super-resolution Sub-pixel scan Sur-pixel scan
a b s t r a c t The key technique in super-resolution reconstruction is sub-pixel scan, which is mainly achieved by moving detector or optic axis. Moving optic axis is performed by rotating or vibrating one optical component of the imaging lens. However, the machining requirements of the optical component for sub-pixel scan are rigorous or sometimes even impossible. A sur-pixel scan method is proposed, it can effectively restrain frequency aliasing while having lower machining requirements compared to sub-pixel scan. In this paper, the effect of sur-pixel scan on image frequency aliasing is theoretically analysed, its engineering superiority is mathematically calculated in visible spectrum and infrared spectrum. Simulations and experiments are performed to validate the effectiveness of sur-pixel scan method. © 2013 Elsevier GmbH. All rights reserved.
1. Introduction
2. Definition
A high resolution image can be reconstructed from multiple low resolution images obtained by micro scanning. This is superresolution (SR) reconstruction technology [1]. The key technique, so-called sub-pixel scan, shifts the image under pixel scale (usually microns) precisely to get a series of low resolution images. The sub-pixel displacements between these images are the source of frequency aliasing restraint, which is the essence of SR reconstruction [1–4]. Sub-pixel scan is mainly achieved by moving image detectors [5,6] or optic axis [7–9] of the imaging system. The problem of the former method is that the detectors will be degenerated from longterm oscillation performed by piezoelectricity ceramic. The latter method is performed by rotating or vibrating one optical component of the imaging lens, which won’t cause degeneration problem as it is a non-contact scan. However, the machining requirements of the optical component for the sub-pixel scan are rigorous or sometimes even impossible (see details in Section 4). In this paper, reconstruction algorithm will not be discussed. Main efforts are focused on the research of scan technique. A surpixel scan method is proposed, it can effectively restrain frequency aliasing, while having lower machining requirements compared to sub-pixel scan. Its effects on image aliasing are analysed. Simulation and experiments are performed to validate the effectiveness of the sur-pixel scan method.
“Sur-pixel” is a new term to describe the scan method proposed in this paper and to distinguish it from the conventional “sub-pixel” scan. Sub-pixel scan means image shifts within one pixel precisely, while for sur-pixel scan, it means image shifts over one or several pixels precisely, to get a series of low resolution images used for super-resolution reconstruction. 3. Sur-pixel scan theory In one detector, the number of pixels as well as the size of and between the pixels are limited, which leads to image aliasing during detector sampling [10]. With a detector illustrated in Fig. 1, the sampling process can be expressed as follow: gs (x, y) =
1 x y d
a 2
,
d d
1
gi (x, y)
D
comb 2
x y , D D
(1)
where gi (x,y) is the intensity distribution of the image, gs (x,y) is the intensity distribution of the detector output, a(x/d, y/d) is the aperture function of one pixel, comb(x/D, y/D) is the sampling function defined as follow: comb(x, y) =
∞ ∞
ı(x − m, y − n)
(2)
m=−∞n=−∞
After Fourier-transforming, Eq. (1) can be expressed as follow: Gs (u, v) = [A(du, dv)Gi (u, v)] comb(Du, Dv) ∗ Corresponding author at: #37 Xueyuan Road, Haidian District, Beijing, Post Code 100191, China. Tel.: +86 010 82316547 812. E-mail address: [email protected] (M. Sun). 0030-4026/$ – see front matter © 2013 Elsevier GmbH. All rights reserved. http://dx.doi.org/10.1016/j.ijleo.2013.05.131
(3)
with comb(Du, Dv) convolution, there is frequency aliasing. The image frequency aliasing can equally be restrained both by sub-pixel scan and sur-pixel scan. Common scan mode includes
6906
M. Sun, K. Yu / Optik 124 (2013) 6905–6909
Fig. 1. Pixels array of a detector, D is the size of pixel, d is the size of pixel effective area.
2 × 2, 3 × 3, 4 × 4 [11]. Take 2 × 2 mode as an example, the subpixel scan process is shown as Fig. 2. The intensity distribution of detector output gsub (x,y) can be derived as follow: gsub (x, y) =
comb
1 y x − , D 2 D
1 4
1 x y a gi (x, y) , d d d2
+ comb
1 x y , − D D 2
x y 1 , [comb + D D D2
+ comb
1 y 1 x − , − D 2 D 2
∞ ∞
]
ı(Du − m, Dv − n)
m=−∞n=−∞
× (1 + e−im + e−in + e−i(m+n) )
(5)
In Eq. (5), only when both m and n are even, that Gsub (u,v) = / 0, which means aliasing only occurs at even-order frequency spectrum. Therefore, image frequency aliasing is restrained. Take the same 2 × 2 mode for example, the sur-pixel scan process is shown as Fig. 3, the scan step is (k + 1/2)D rather than D/2 in sub-pixel scan, where k = 1,2,3. . .. The intensity distribution of detector output gsub (x,y), can be derived as follow: 1 4
1 x y a gi (x, y) , d d d2
+comb
+comb
x y , − D D
k+
1 2
x − D
+ comb
k+
1 2
x − D
x y 1 , {comb D D D2
,
y D
k+
(6) 1 2
,
y − D
k+
1 2
}
1 [A(du, dv)Gi (u, v)] [ 4
∞ ∞
ı(Du − m, Dv − n)
m=−∞n=−∞ (1 + e−i(2k+1)m + e−i(2k+1)n + e−i(2k+1)(m+n) )]
(4)
gsur (x, y) =
Substituting Eq. (2) into Eq. (6), after Fourier-transforming Eq. (6) can be converted into: Gsur (u, v) =
Substituting Eq. (2) into Eq. (4), after Fourier-transforming Eq. (4) can be converted into: 1 Gsub (u, v) = [A(du, dv)Gi (u, v)] 4
Fig. 3. Mode 2 × 2 sur-pixel scan pattern. The white blocks represent original pixel position, the black blocks represent pixel position after sur-pixel shift which route is shown as 1–4.Scan step is (k + 1/2)D.
(7)
In Eq. (7), only when both m and n are even, that Gsub (u,v) = / 0, which means aliasing only occurs at even-order frequency spectrum too. In other words, 2 × 2 sur-pixel scan has the same effect on frequency aliasing restraint as sub-pixel scan does. With further analysis, the same conclusion can be derived for 3 × 3 and 4 × 4 sur-pixel scan. 4. Engineering superiority Moving optic axis by rotating a sloping parallel-plate is a typical method to perform sub-pixel scan [7]. It has the advantages of noncontact, compact design and uniform imaging. Optic axis moved by sloping parallel-plate is shown in Fig. 4(a). A 2 × 2 sub-pixel scan system is shown in Fig. 4(b). When the incident light transmits the sloping parallel-plate in Position 1 (illustrated in solid line), the optic axis, as well as the image, moves to the upper part of the detector. After the plate rotating 180◦ along Z axis to Position 2(illustrated in dash line), the optic axis moves to the lower part. There are two other positions, where the rotating is of 90◦ and 270◦ , leading the optic axis moving to the right and the left part of the detector. According to Fig. 4(a) and geometrical optics, we have Eq. (8) in sub-pixel/sur-pixel scan:
⎧ ⎨ sin = n sin ˛ ⎩
d = t sin( − ˛)/ cos ˛ √ 2d = 2(2k + 1)D/2; k = 0, 1, 2, 3....
(8)
where n is the refractive index of plate, is the incident angle,˛ is the refraction angle of refraction, t is the thickness of plate, d is the optical axis shift, D is the pixel size. In sub-pixel scan, k = 0, and
Fig. 2. Mode 2 × 2 sub-pixel scan pattern. The white blocks represent original pixel position, the black blocks represent pixel position after sub-pixel shift which route is shown as 1∼4. D is the interval between pixels (numerically equal to pixel size), scan step is D/2.
Fig. 4. (a) Optic axis moved by sloping parallel-plate. (b) 2 × 2 sub-pixel scan system.
M. Sun, K. Yu / Optik 124 (2013) 6905–6909 Table 2 Entropies of simulation results.
Table 1 Parallel-plate parameters in sub-pixel and sur-pixel scan. Spectrums Materials Aperture diameter (mm)
Visible (380–760 nm) BK7 glass 6
Methods
sub-pixel
Thickness (mm) Machining requirement
0.14 impossible
sur-pixel k=2 k=1 0.42 0.70 normal easy
6907
Infrared (8–12 m) Germanium 16.8 sub-pixel 0.47 rigorous
Methods
Linear interpolation
Sub-pixel scan
Entropy
5.6775
5.9414
Sur-pixel scan k=1
k=2
5.9213
5.9359
sur-pixel k=1 k=2 1.41 2.35 easy easy
the scan step is D/2. In sur-pixel scan, k = 1, 2, 3. . ., and the scan step is (k + 1/2)D. Eq. (8) can be expressed as follow,
sin = n sin ˛ √ t = 2(2k + 1)D cos ˛/[4 sin( − ˛)] k = 0, 1, 2, 3....
(9)
Usually, the incident angle should not be larger than 2◦ to keep the scan shift error small. For visible spectrum detectors, their pixel sizes are usually small (about 5 m × 5 m). Take SONY CCD ICX204AK for example. It is a diagonal 6 mm CCD with its resolution 1024 × 768, the pixel size is 4.65 m × 4.65 m. The parallel-plate is made of BK7 glass, n = 1.519 at 550 nm wavelength. Assume = 2◦ , solving Eq. (9), we get tsub = 0.14 mm and tsur = (2k + 1) × 0.14 mm. In this case, the clear aperture of the parallel plate must be larger than the diagonal of the CCD, which is 6 mm. We note that it is impossible to machining a 0.14 mm BK7 parallel-plate to perform 2 × 2 sub-pixel scan. On the other hand, to perform sur-pixel method, a 0.42 mm or 0.7 mm BK7 parallel-plate is easier to manufacture. For infrared spectrum detectors, the pixel sizes are relatively larger (about 30 m × 30 m) than that of visible spectrum detectors. Take GWIC Inc. GWIR020131A detector for example. It is an 8–12 m infrared focal plane array with its resolution 384 × 288, the pixel size is 35 m × 35 m, the diagonal is 16.8 mm. The parallel-plate is made of germanium, n = 4.000 at 10 m wavelength. Assume = 2◦ , solving Eq. (9), we get tsub = 0.47 mm and tsur = (2k + 1) × 0.47 mm. The clear aperture of the parallel plate should be larger than 16.8 mm. We note a similar situation as in visible spectrum case. Although in this case, to perform 2 × 2 sub-pixel scan, the machining requirement is possible but quite rigorous, and it is much easier to go with sur-pixel method. The calculation results of both visible and infrared spectrum cases are listed in Table 1. Combining with the theory in Section 3, we can theoretically conclude that sur-pixel method can reduce the machining requirement significantly while achieving the same effect on frequency aliasing restraint.
Fig. 5. (a) Original image with linear interpolation, (b) sub-pixel reconstructed image, (c) sur-pixel reconstructed image (k = 1), (d) sur-pixel reconstructed image (k = 2).
Fig. 7. (a) Germanium parallel-plates, (b) infrared lens, (c) rotating structure, (d) rotating structure with a sloping parallel-plate in, (e) GWIR020131A detector, (f) experimental system.
5. Simulations and experiments 5.1. Simulations An image of 1536 × 1152 is taken to be the object in simulation. Assume D = 4 pixels and d = 3 pixels in the detector. Both sub-pixel and sur-pixel scan (k = 1,2) are performed with 2 × 2 mode on the object. Twelve original scan images of 384 × 288 are obtained and three 768 × 576 images are reconstructed from those original scan images. One of those original scan images and three reconstructed images are shown in Fig. 5. In Fig. 5, the high resolution images reconstructed from sur-pixel scan are the same as the image reconstructed from sub-pixel scan, except for those “sick pixels” on the image boundaries. “Sick pixels” refer to those pixels near the image boundaries, which are reconstructed improperly due to sur-pixel scanning. When k = 1, pixels of first row, first column, last row and last column of the sur-pixel scan reconstructed images are sick area, when k = 2, pixels of first two rows, first two columns, last two rows and last two columns of the sur-pixel scan reconstructed images are sick area. The bigger k is, the larger the sick area will be. However, the objects of interests are at the centre area of the image in most occasions. Therefore, the sick pixels caused by sur-pixel scan have only infinitesimal effect on image quality. Zoomed areas of these images are shown in Fig. 6. In Fig. 6, image (b), (c) and (d) got more details than image (a). Image (c) and (d) are the same as image (b). Entropy is a statistical measure of randomness that can be used to characterise the texture of the image. Entropy of a greyscale image is defined as n
H(X) = −
P(xi )log2 P(xi ),
(10)
i=1
Fig. 6. Zoomed areas of (a) original image with linear interpolation, (b) sub-pixel reconstructed, (c) sur-pixel reconstructed (k = 1), (d) sur-pixel reconstructed (k = 2).
where H(X) is the entropy of the image X, P(xi ) is the probability mass function of the greyscale level xi in the image X, n is the total number of greyscale levels in the image X. The value of entropy presents the amount of information, both signal and noise, contained in the image. The zoomed areas in Fig. 6 are calculated with Eq. (10), the results are listed in Table 2. In our simulation, the noises are almost
6908
M. Sun, K. Yu / Optik 124 (2013) 6905–6909
Fig. 8. (a) original image with linear interpolation, (b) sub-pixel reconstructed image, (c) sur-pixel reconstructed image (k = 1), (d) sur-pixel reconstructed image (k = 2).
Fig. 9. Zoomed areas of (a) original image with linear interpolation, (b) sub-pixel reconstructed image, (c) sur-pixel reconstructed image (k = 1), (d) sur-pixel reconstructed image (k = 2).
the same for the four images in Fig. 6, therefore the differences among the results can be used for detail evaluation. We note that the entropy of the linear interpolation of original scan image is much smaller than the other three reconstructed images which means it has much less details. The entropies of three reconstructed images have infinitesimal differences among them. The conclusion based on entropy evaluation is coincident with subjective observation of Figs. 5 and 6.
5.2. Experiments In order to validate the effectiveness of the sur-pixel optical scan method, three germanium parallel-plates with different thicknesses, which are listed in Table 1, are made, an experimental system same as the one shown in Fig. 4(b) is setup. All the components and the experimental system are shown in Fig. 7. The parallel-plates in Fig. 7(a) are made of germanium with thickness 0.47 mm, 1.41 mm and 2.35 mm respectively. The infrared lens in Fig. 7(b) works at 8–12 m, its effective focal length is 25 mm, its clear aperture is 30 mm. The rotating structure in Fig. 7(c) is used to manually rotate the parallel-plate in it. The sloping angle is 1◦ . The infrared detector in Fig. 7(e) is manufactured by GWIC Inc., the production model is GW20020131A. It works at 8–12 m. The experimental system in Fig. 7(f) is made up of the components mentioned above. The experimental images are captured by computer. During the experiments, the scan is performed by rotating the parallel-plate. In each experiment, four original low resolution scan images are captured at 45◦ , 135◦ , 225◦ and 315◦ (Position 1 in Fig. 4 is defined as 0◦ ), and a high resolution image is reconstructed from these four images. Three scan experiments are performed, each with a different parallel-plate in the rotating structure as shown in Fig. 7(d). In all, twelve original scan images and three reconstructed images are obtained. One of those original scan images with linear interpolation and three reconstructed images are shown in Fig. 8. In Fig. 8, the high resolution images reconstructed from surpixel scan are the same as the image reconstructed from sub-pixel scan, except for those sick pixels on the image boundaries. Zoomed areas of these images are in Fig. 9. The zoomed areas in Fig. 9 are calculated with Eq. (10), the results are listed in Table 3.
Table 3 Entropies of experimental results. Methods
Thickness (mm) Entropy
Linear interpolation
– 6.3027
Sub-pixel scan
0.47 6.3142
Sur-pixel scan
k=1
k=2
1.41 6.3100
2.35 6.3112
Table 3 shows that the sub-pixel reconstructed image and the sur-pixel reconstructed images have almost the same entropies, and their entropies are bigger than that of interpolated original scan image. In Fig. 9, the details of image (b), (c) and (d) are clearer than that of image (a). Image (c) and (d) are the same as image (b), except they got sick pixels at the boundaries. The experimental results are coincident with the simulation results. 6. Conclusion Based on the simulation and experimental results, concluded from both subjective observation and entropy evaluation, the sur-pixel optical scan method for super-resolution reconstruction can effectively restrain image frequency aliasing. The qualities of sur-pixel scan reconstructed images are almost the same as that of sub-pixel scan reconstructed image. In the meantime, the engineering superiorities of sur-pixel scan are respectively derived for invisible spectrum and infrared spectrum. the sur-pixel scan method can lower machining requirements from rigorous or impossible to easy, which is a major advantage for engineering application. Acknowledgements The research is supported by the National High Technology Research and Development Program of China. References [1] C.P. Sung, K.P. Min, G.K. Moon, Super-resolution image reconstruction: a technical overview, IEEE Signal Proc. Mag. 20 (3) (2003) 21–36. [2] S. Farsiu, D. Robinson, M. Elad, P. Milanfar, Advances and challenges in superresolution, Int. J. Imag. Syst. Technol. 14 (2) (2004) 47–57.
M. Sun, K. Yu / Optik 124 (2013) 6905–6909 [3] P. Milanfar, Super-Resolution Imaging, CRC Press, Boca Raton, Florida, 2010. [4] J. Forntin, P. Chevrette, R. Plante, Evaluation of the microscanning process, Proc. SPIE 2269 (1994) 271–279. [5] J.M. Wiltse, J.L. Miller, Imagery improvements in staring infrared imagers by employing subpixel microscan, Opt. Eng. 44 (5) (2005) 056401. [6] J. Fortin, P. Chevere, Realization of a fast microscanning device for infrared focal plane arrays, Proc. SPIE 2742 (1996) 185–196. [7] X. Sui, L. Bai, Q. Chen, G. Gu, Influencing factors of microscanning performance based on flat optical component, Chin. Opt. Letts. 9 (5) (2011) 052302.
6909
[8] K. Krapels, R. Driggers, R. Vollmerhausen, A performance comparison of rectangular(4-point) and diagonal(2-point) dither, Proc. SPIE 4030 (2000) 151–169. [9] L.L. Noc, B. Tremblay, A. Martel, et al., 1280 × 960 pixel microscanned infrared imaging module, Proc. SPIE 7660 (2010) 766021. [10] T. Stadtmiller, J. Gillete, R. Hardie, Reduction of aliasing in staring infrared imagers utilizing subpixel techniques, Presented at Aerospace and Electronics Conference, May 22–26, 1995. [11] X. Wang, J. Zhang, Z. Feng, Analysis of improvement amount of typical microscanning modes to IR imagery quality, Infrared Phys. & Tech. 46 (2005) 412–417.
Contents lists available at ScienceDirect
Optik journal homepage: www.elsevier.de/ijleo
A sur-pixel scan method for super-resolution reconstruction Mingjie Sun ∗ , Kanglong Yu Institution of Opto-electrionics Technology, Beihang University, Beijing, China
a r t i c l e
i n f o
Article history: Received 12 January 2013 Accepted 25 May 2013 Keywords: Super-resolution Sub-pixel scan Sur-pixel scan
a b s t r a c t The key technique in super-resolution reconstruction is sub-pixel scan, which is mainly achieved by moving detector or optic axis. Moving optic axis is performed by rotating or vibrating one optical component of the imaging lens. However, the machining requirements of the optical component for sub-pixel scan are rigorous or sometimes even impossible. A sur-pixel scan method is proposed, it can effectively restrain frequency aliasing while having lower machining requirements compared to sub-pixel scan. In this paper, the effect of sur-pixel scan on image frequency aliasing is theoretically analysed, its engineering superiority is mathematically calculated in visible spectrum and infrared spectrum. Simulations and experiments are performed to validate the effectiveness of sur-pixel scan method. © 2013 Elsevier GmbH. All rights reserved.
1. Introduction
2. Definition
A high resolution image can be reconstructed from multiple low resolution images obtained by micro scanning. This is superresolution (SR) reconstruction technology [1]. The key technique, so-called sub-pixel scan, shifts the image under pixel scale (usually microns) precisely to get a series of low resolution images. The sub-pixel displacements between these images are the source of frequency aliasing restraint, which is the essence of SR reconstruction [1–4]. Sub-pixel scan is mainly achieved by moving image detectors [5,6] or optic axis [7–9] of the imaging system. The problem of the former method is that the detectors will be degenerated from longterm oscillation performed by piezoelectricity ceramic. The latter method is performed by rotating or vibrating one optical component of the imaging lens, which won’t cause degeneration problem as it is a non-contact scan. However, the machining requirements of the optical component for the sub-pixel scan are rigorous or sometimes even impossible (see details in Section 4). In this paper, reconstruction algorithm will not be discussed. Main efforts are focused on the research of scan technique. A surpixel scan method is proposed, it can effectively restrain frequency aliasing, while having lower machining requirements compared to sub-pixel scan. Its effects on image aliasing are analysed. Simulation and experiments are performed to validate the effectiveness of the sur-pixel scan method.
“Sur-pixel” is a new term to describe the scan method proposed in this paper and to distinguish it from the conventional “sub-pixel” scan. Sub-pixel scan means image shifts within one pixel precisely, while for sur-pixel scan, it means image shifts over one or several pixels precisely, to get a series of low resolution images used for super-resolution reconstruction. 3. Sur-pixel scan theory In one detector, the number of pixels as well as the size of and between the pixels are limited, which leads to image aliasing during detector sampling [10]. With a detector illustrated in Fig. 1, the sampling process can be expressed as follow: gs (x, y) =
1 x y d
a 2
,
d d
1
gi (x, y)
D
comb 2
x y , D D
(1)
where gi (x,y) is the intensity distribution of the image, gs (x,y) is the intensity distribution of the detector output, a(x/d, y/d) is the aperture function of one pixel, comb(x/D, y/D) is the sampling function defined as follow: comb(x, y) =
∞ ∞
ı(x − m, y − n)
(2)
m=−∞n=−∞
After Fourier-transforming, Eq. (1) can be expressed as follow: Gs (u, v) = [A(du, dv)Gi (u, v)] comb(Du, Dv) ∗ Corresponding author at: #37 Xueyuan Road, Haidian District, Beijing, Post Code 100191, China. Tel.: +86 010 82316547 812. E-mail address: [email protected] (M. Sun). 0030-4026/$ – see front matter © 2013 Elsevier GmbH. All rights reserved. http://dx.doi.org/10.1016/j.ijleo.2013.05.131
(3)
with comb(Du, Dv) convolution, there is frequency aliasing. The image frequency aliasing can equally be restrained both by sub-pixel scan and sur-pixel scan. Common scan mode includes
6906
M. Sun, K. Yu / Optik 124 (2013) 6905–6909
Fig. 1. Pixels array of a detector, D is the size of pixel, d is the size of pixel effective area.
2 × 2, 3 × 3, 4 × 4 [11]. Take 2 × 2 mode as an example, the subpixel scan process is shown as Fig. 2. The intensity distribution of detector output gsub (x,y) can be derived as follow: gsub (x, y) =
comb
1 y x − , D 2 D
1 4
1 x y a gi (x, y) , d d d2
+ comb
1 x y , − D D 2
x y 1 , [comb + D D D2
+ comb
1 y 1 x − , − D 2 D 2
∞ ∞
]
ı(Du − m, Dv − n)
m=−∞n=−∞
× (1 + e−im + e−in + e−i(m+n) )
(5)
In Eq. (5), only when both m and n are even, that Gsub (u,v) = / 0, which means aliasing only occurs at even-order frequency spectrum. Therefore, image frequency aliasing is restrained. Take the same 2 × 2 mode for example, the sur-pixel scan process is shown as Fig. 3, the scan step is (k + 1/2)D rather than D/2 in sub-pixel scan, where k = 1,2,3. . .. The intensity distribution of detector output gsub (x,y), can be derived as follow: 1 4
1 x y a gi (x, y) , d d d2
+comb
+comb
x y , − D D
k+
1 2
x − D
+ comb
k+
1 2
x − D
x y 1 , {comb D D D2
,
y D
k+
(6) 1 2
,
y − D
k+
1 2
}
1 [A(du, dv)Gi (u, v)] [ 4
∞ ∞
ı(Du − m, Dv − n)
m=−∞n=−∞ (1 + e−i(2k+1)m + e−i(2k+1)n + e−i(2k+1)(m+n) )]
(4)
gsur (x, y) =
Substituting Eq. (2) into Eq. (6), after Fourier-transforming Eq. (6) can be converted into: Gsur (u, v) =
Substituting Eq. (2) into Eq. (4), after Fourier-transforming Eq. (4) can be converted into: 1 Gsub (u, v) = [A(du, dv)Gi (u, v)] 4
Fig. 3. Mode 2 × 2 sur-pixel scan pattern. The white blocks represent original pixel position, the black blocks represent pixel position after sur-pixel shift which route is shown as 1–4.Scan step is (k + 1/2)D.
(7)
In Eq. (7), only when both m and n are even, that Gsub (u,v) = / 0, which means aliasing only occurs at even-order frequency spectrum too. In other words, 2 × 2 sur-pixel scan has the same effect on frequency aliasing restraint as sub-pixel scan does. With further analysis, the same conclusion can be derived for 3 × 3 and 4 × 4 sur-pixel scan. 4. Engineering superiority Moving optic axis by rotating a sloping parallel-plate is a typical method to perform sub-pixel scan [7]. It has the advantages of noncontact, compact design and uniform imaging. Optic axis moved by sloping parallel-plate is shown in Fig. 4(a). A 2 × 2 sub-pixel scan system is shown in Fig. 4(b). When the incident light transmits the sloping parallel-plate in Position 1 (illustrated in solid line), the optic axis, as well as the image, moves to the upper part of the detector. After the plate rotating 180◦ along Z axis to Position 2(illustrated in dash line), the optic axis moves to the lower part. There are two other positions, where the rotating is of 90◦ and 270◦ , leading the optic axis moving to the right and the left part of the detector. According to Fig. 4(a) and geometrical optics, we have Eq. (8) in sub-pixel/sur-pixel scan:
⎧ ⎨ sin = n sin ˛ ⎩
d = t sin( − ˛)/ cos ˛ √ 2d = 2(2k + 1)D/2; k = 0, 1, 2, 3....
(8)
where n is the refractive index of plate, is the incident angle,˛ is the refraction angle of refraction, t is the thickness of plate, d is the optical axis shift, D is the pixel size. In sub-pixel scan, k = 0, and
Fig. 2. Mode 2 × 2 sub-pixel scan pattern. The white blocks represent original pixel position, the black blocks represent pixel position after sub-pixel shift which route is shown as 1∼4. D is the interval between pixels (numerically equal to pixel size), scan step is D/2.
Fig. 4. (a) Optic axis moved by sloping parallel-plate. (b) 2 × 2 sub-pixel scan system.
M. Sun, K. Yu / Optik 124 (2013) 6905–6909 Table 2 Entropies of simulation results.
Table 1 Parallel-plate parameters in sub-pixel and sur-pixel scan. Spectrums Materials Aperture diameter (mm)
Visible (380–760 nm) BK7 glass 6
Methods
sub-pixel
Thickness (mm) Machining requirement
0.14 impossible
sur-pixel k=2 k=1 0.42 0.70 normal easy
6907
Infrared (8–12 m) Germanium 16.8 sub-pixel 0.47 rigorous
Methods
Linear interpolation
Sub-pixel scan
Entropy
5.6775
5.9414
Sur-pixel scan k=1
k=2
5.9213
5.9359
sur-pixel k=1 k=2 1.41 2.35 easy easy
the scan step is D/2. In sur-pixel scan, k = 1, 2, 3. . ., and the scan step is (k + 1/2)D. Eq. (8) can be expressed as follow,
sin = n sin ˛ √ t = 2(2k + 1)D cos ˛/[4 sin( − ˛)] k = 0, 1, 2, 3....
(9)
Usually, the incident angle should not be larger than 2◦ to keep the scan shift error small. For visible spectrum detectors, their pixel sizes are usually small (about 5 m × 5 m). Take SONY CCD ICX204AK for example. It is a diagonal 6 mm CCD with its resolution 1024 × 768, the pixel size is 4.65 m × 4.65 m. The parallel-plate is made of BK7 glass, n = 1.519 at 550 nm wavelength. Assume = 2◦ , solving Eq. (9), we get tsub = 0.14 mm and tsur = (2k + 1) × 0.14 mm. In this case, the clear aperture of the parallel plate must be larger than the diagonal of the CCD, which is 6 mm. We note that it is impossible to machining a 0.14 mm BK7 parallel-plate to perform 2 × 2 sub-pixel scan. On the other hand, to perform sur-pixel method, a 0.42 mm or 0.7 mm BK7 parallel-plate is easier to manufacture. For infrared spectrum detectors, the pixel sizes are relatively larger (about 30 m × 30 m) than that of visible spectrum detectors. Take GWIC Inc. GWIR020131A detector for example. It is an 8–12 m infrared focal plane array with its resolution 384 × 288, the pixel size is 35 m × 35 m, the diagonal is 16.8 mm. The parallel-plate is made of germanium, n = 4.000 at 10 m wavelength. Assume = 2◦ , solving Eq. (9), we get tsub = 0.47 mm and tsur = (2k + 1) × 0.47 mm. The clear aperture of the parallel plate should be larger than 16.8 mm. We note a similar situation as in visible spectrum case. Although in this case, to perform 2 × 2 sub-pixel scan, the machining requirement is possible but quite rigorous, and it is much easier to go with sur-pixel method. The calculation results of both visible and infrared spectrum cases are listed in Table 1. Combining with the theory in Section 3, we can theoretically conclude that sur-pixel method can reduce the machining requirement significantly while achieving the same effect on frequency aliasing restraint.
Fig. 5. (a) Original image with linear interpolation, (b) sub-pixel reconstructed image, (c) sur-pixel reconstructed image (k = 1), (d) sur-pixel reconstructed image (k = 2).
Fig. 7. (a) Germanium parallel-plates, (b) infrared lens, (c) rotating structure, (d) rotating structure with a sloping parallel-plate in, (e) GWIR020131A detector, (f) experimental system.
5. Simulations and experiments 5.1. Simulations An image of 1536 × 1152 is taken to be the object in simulation. Assume D = 4 pixels and d = 3 pixels in the detector. Both sub-pixel and sur-pixel scan (k = 1,2) are performed with 2 × 2 mode on the object. Twelve original scan images of 384 × 288 are obtained and three 768 × 576 images are reconstructed from those original scan images. One of those original scan images and three reconstructed images are shown in Fig. 5. In Fig. 5, the high resolution images reconstructed from sur-pixel scan are the same as the image reconstructed from sub-pixel scan, except for those “sick pixels” on the image boundaries. “Sick pixels” refer to those pixels near the image boundaries, which are reconstructed improperly due to sur-pixel scanning. When k = 1, pixels of first row, first column, last row and last column of the sur-pixel scan reconstructed images are sick area, when k = 2, pixels of first two rows, first two columns, last two rows and last two columns of the sur-pixel scan reconstructed images are sick area. The bigger k is, the larger the sick area will be. However, the objects of interests are at the centre area of the image in most occasions. Therefore, the sick pixels caused by sur-pixel scan have only infinitesimal effect on image quality. Zoomed areas of these images are shown in Fig. 6. In Fig. 6, image (b), (c) and (d) got more details than image (a). Image (c) and (d) are the same as image (b). Entropy is a statistical measure of randomness that can be used to characterise the texture of the image. Entropy of a greyscale image is defined as n
H(X) = −
P(xi )log2 P(xi ),
(10)
i=1
Fig. 6. Zoomed areas of (a) original image with linear interpolation, (b) sub-pixel reconstructed, (c) sur-pixel reconstructed (k = 1), (d) sur-pixel reconstructed (k = 2).
where H(X) is the entropy of the image X, P(xi ) is the probability mass function of the greyscale level xi in the image X, n is the total number of greyscale levels in the image X. The value of entropy presents the amount of information, both signal and noise, contained in the image. The zoomed areas in Fig. 6 are calculated with Eq. (10), the results are listed in Table 2. In our simulation, the noises are almost
6908
M. Sun, K. Yu / Optik 124 (2013) 6905–6909
Fig. 8. (a) original image with linear interpolation, (b) sub-pixel reconstructed image, (c) sur-pixel reconstructed image (k = 1), (d) sur-pixel reconstructed image (k = 2).
Fig. 9. Zoomed areas of (a) original image with linear interpolation, (b) sub-pixel reconstructed image, (c) sur-pixel reconstructed image (k = 1), (d) sur-pixel reconstructed image (k = 2).
the same for the four images in Fig. 6, therefore the differences among the results can be used for detail evaluation. We note that the entropy of the linear interpolation of original scan image is much smaller than the other three reconstructed images which means it has much less details. The entropies of three reconstructed images have infinitesimal differences among them. The conclusion based on entropy evaluation is coincident with subjective observation of Figs. 5 and 6.
5.2. Experiments In order to validate the effectiveness of the sur-pixel optical scan method, three germanium parallel-plates with different thicknesses, which are listed in Table 1, are made, an experimental system same as the one shown in Fig. 4(b) is setup. All the components and the experimental system are shown in Fig. 7. The parallel-plates in Fig. 7(a) are made of germanium with thickness 0.47 mm, 1.41 mm and 2.35 mm respectively. The infrared lens in Fig. 7(b) works at 8–12 m, its effective focal length is 25 mm, its clear aperture is 30 mm. The rotating structure in Fig. 7(c) is used to manually rotate the parallel-plate in it. The sloping angle is 1◦ . The infrared detector in Fig. 7(e) is manufactured by GWIC Inc., the production model is GW20020131A. It works at 8–12 m. The experimental system in Fig. 7(f) is made up of the components mentioned above. The experimental images are captured by computer. During the experiments, the scan is performed by rotating the parallel-plate. In each experiment, four original low resolution scan images are captured at 45◦ , 135◦ , 225◦ and 315◦ (Position 1 in Fig. 4 is defined as 0◦ ), and a high resolution image is reconstructed from these four images. Three scan experiments are performed, each with a different parallel-plate in the rotating structure as shown in Fig. 7(d). In all, twelve original scan images and three reconstructed images are obtained. One of those original scan images with linear interpolation and three reconstructed images are shown in Fig. 8. In Fig. 8, the high resolution images reconstructed from surpixel scan are the same as the image reconstructed from sub-pixel scan, except for those sick pixels on the image boundaries. Zoomed areas of these images are in Fig. 9. The zoomed areas in Fig. 9 are calculated with Eq. (10), the results are listed in Table 3.
Table 3 Entropies of experimental results. Methods
Thickness (mm) Entropy
Linear interpolation
– 6.3027
Sub-pixel scan
0.47 6.3142
Sur-pixel scan
k=1
k=2
1.41 6.3100
2.35 6.3112
Table 3 shows that the sub-pixel reconstructed image and the sur-pixel reconstructed images have almost the same entropies, and their entropies are bigger than that of interpolated original scan image. In Fig. 9, the details of image (b), (c) and (d) are clearer than that of image (a). Image (c) and (d) are the same as image (b), except they got sick pixels at the boundaries. The experimental results are coincident with the simulation results. 6. Conclusion Based on the simulation and experimental results, concluded from both subjective observation and entropy evaluation, the sur-pixel optical scan method for super-resolution reconstruction can effectively restrain image frequency aliasing. The qualities of sur-pixel scan reconstructed images are almost the same as that of sub-pixel scan reconstructed image. In the meantime, the engineering superiorities of sur-pixel scan are respectively derived for invisible spectrum and infrared spectrum. the sur-pixel scan method can lower machining requirements from rigorous or impossible to easy, which is a major advantage for engineering application. Acknowledgements The research is supported by the National High Technology Research and Development Program of China. References [1] C.P. Sung, K.P. Min, G.K. Moon, Super-resolution image reconstruction: a technical overview, IEEE Signal Proc. Mag. 20 (3) (2003) 21–36. [2] S. Farsiu, D. Robinson, M. Elad, P. Milanfar, Advances and challenges in superresolution, Int. J. Imag. Syst. Technol. 14 (2) (2004) 47–57.
M. Sun, K. Yu / Optik 124 (2013) 6905–6909 [3] P. Milanfar, Super-Resolution Imaging, CRC Press, Boca Raton, Florida, 2010. [4] J. Forntin, P. Chevrette, R. Plante, Evaluation of the microscanning process, Proc. SPIE 2269 (1994) 271–279. [5] J.M. Wiltse, J.L. Miller, Imagery improvements in staring infrared imagers by employing subpixel microscan, Opt. Eng. 44 (5) (2005) 056401. [6] J. Fortin, P. Chevere, Realization of a fast microscanning device for infrared focal plane arrays, Proc. SPIE 2742 (1996) 185–196. [7] X. Sui, L. Bai, Q. Chen, G. Gu, Influencing factors of microscanning performance based on flat optical component, Chin. Opt. Letts. 9 (5) (2011) 052302.
6909
[8] K. Krapels, R. Driggers, R. Vollmerhausen, A performance comparison of rectangular(4-point) and diagonal(2-point) dither, Proc. SPIE 4030 (2000) 151–169. [9] L.L. Noc, B. Tremblay, A. Martel, et al., 1280 × 960 pixel microscanned infrared imaging module, Proc. SPIE 7660 (2010) 766021. [10] T. Stadtmiller, J. Gillete, R. Hardie, Reduction of aliasing in staring infrared imagers utilizing subpixel techniques, Presented at Aerospace and Electronics Conference, May 22–26, 1995. [11] X. Wang, J. Zhang, Z. Feng, Analysis of improvement amount of typical microscanning modes to IR imagery quality, Infrared Phys. & Tech. 46 (2005) 412–417.