- Email: [email protected]
MEMS-based super-resolution remote sensing system using compressive sensing
Optics Communications 426 (2018) 410–417
Contents lists available at ScienceDirect
Optics Communications journal homepage: www.elsevier.com/locate/optcom
MEMS-based super-resolution remote sensing system using compressive sensing Xudong Zhang a,b , Jianan Xie b , Chunlai Li b , Rui Xu a,b , Yue Zhang a,b , Shijie Liu a,b , Jianyu Wang a,b, * a b
University of Chinese Academy of Sciences, Yuquan Road, 100049 Beijing, China Shanghai Institute of Technical Physics of the Chinese Academy of Sciences, Yutian Road, 200083 Shanghai, China
ARTICLE
INFO
ABSTRACT
Keywords: Super resolution Remote sensing Compressive sensing Non-uniformity
In this article we present a novel remote sensing system the resolution of which exceeds the image sensor’s with the help of high-resolution MEMS device, which will reduce the cost of high-resolution image sensors. The superresolution factors of 2 × 2 and 4×4 have been tested by simulation and remote sensing experiments. By applying the theory of compressive sensing, we can compress the acquisition data rate while preserving high image quality. We employ the IRLS as the super-resolution reconstruction method. A sample rate greater than 50% guarantees that our algorithm is more accurate. Through experiments we have found that the mismatch of the image sensor and MEMS device diminishes the image uniformity. Our algorithm embedded with non-uniformity correction greatly alleviates the problem visually and quantitatively.
1. Introduction
input in the algorithms contain geometric or photometric differences that depict the details of a certain HR image [4]. The most common method is to use sequential LR frames that contain spatial differences of moving objects to reconstruct HR images [5]. However, in this method the improvement of special resolution exists only in the moving part of the image. The Compressive Sensing (CS) theory [6] has changed the signal pattern in many fields. The original goal of CS theory is to reconstruct a signal from a series of measurements whose sampling rates can be much fewer than the Shannon–Nyquist sampling theorem by applying sparsity of signal and optimization methods [6,7]. The most exciting application of this theory is the single-pixel camera, which substitutes the CCD or CMOS with a single photon detector and a digital micromirror device (DMD) [8]. However, due to the low imaging quality and long sampling time, this new type of imaging system is rarely used in the industrial field. Unlike traditional data compression method, by applying CS theory we are able to compress the data during sampling, which makes it possible to reduce the sensing cost. Moreover, recent research studies approve that CS theory containing signal sparsity has good performance in image de-noising [9]. Our SR remote sensing system is able to produce an HR image with reduced restoration noise after applying CS theory. Unlike the existing SR methods, we present a novel framework of remote sensing image system that manages to improve the resolution in
Remote sensing systems are widely applied in numerous fields, including oceanography, ecology, geology, hydrology, land surveying and military applications [1]. Higher resolution images provide more details and information, thus improving the analysis quality in most of these applications. Super-resolution (SR), the technique of enhancing the resolution of the imaging system, has attracted much research interests in the last decades [2,3]. The traditional means of attaining super resolution are hard-ware based approaches, including decreasing the pixel size and increasing the size of the area array in order to increase the number of pixels per unit area [3]. Although this kind of technique is both widely used and effective, this will result in the dramatically rising cost of the image sensor, particularly for the high-quality sensors used in remote sensing systems. Another method of SR is based on the algorithms that produce high-resolution (HR) image from one or more low-resolution (HR) images. The algorithms can be divided into two types: single-image SR algorithms and multi-frame SR algorithms. Single-image ones rely on the priori knowledge that provide high frequency information of the single image, which contains the neuralnetwork based and sparsity-based SR algorithms. However, despite of its extensibility features, single-image ones possess limited effect because information contained in single image and priori knowledge is limited. Multi-frame ones reconstruct HR images by assuming that the LR images *
Corresponding author at: Shanghai Institute of Technical Physics of the Chinese Academy of Sciences, Yutian Road, 200083 Shanghai, China. E-mail address: [email protected] (J. Wang).
https://doi.org/10.1016/j.optcom.2018.05.046 Received 2 March 2018; Received in revised form 12 May 2018; Accepted 13 May 2018 0030-4018/© 2018 Elsevier B.V. All rights reserved.
X. Zhang et al.
Optics Communications 426 (2018) 410–417
Fig. 1. Framework of the SR remote-sensing system. This system occupies an image sensor with 𝑀 ⋅ 𝑁 spatial resolution. The 𝑘 ⋅ 𝑙 times higher resolution MEMS device plays the role as coded aperture that modulate the subpixel information of the imaging scene. Simply to reconstruct a HR image, we need 𝑘 ⋅ 𝑙 frames LR image. (In Fig. 1 we show the scheme when 𝑘 = 2 and 𝑙 = 2.) However, with the help of CS theory it is possible to reduce the number of frames with high-quality HR images, thus enhancing the speed of imaging and compressing the data.
all regions of the image, possessing good imaging quality and breaking the limitation of the image sensors’ size without increasing cost constraint of the large-array sensors. The key is to apply a higher-resolution lower-cost digital micromirror device (DMD), which is a micro-electromechanical system (MEMS) and can be replaced with other MEMS like digital micro-shutter (DMS). Details of how to reduce the sensor’s cost will be discussed in the next section. With the help of Compressive Sensing (CS) theory [6], we are able to reconstruct a HR image with less sample rate. Moreover, we embedded non-uniformity correction algorithm in the CS algorithm, whereby our method is capable of relieving the fixed-pattern noise caused by the non-uniformity problem. To the best of our knowledge, the proposed architecture is firstly used for remote sensing tasks, and the algorithm in this article is novel. In summary, our method has two main advantages. Firstly, we are able to reduce the cost of HR image sensor for the remote sensing system by substituting it with LR sensor and HR DMD. Secondly, data compression is available, which may help to relieve the data transmission and storage burden for remote sensing systems. The rest of this paper is organized as follows: the next section explains the details of our hardware system. Section 3 explains the principle of the reconstruction algorithm. We show the performance of our algorithm through simulation in Section 4. Section 5 shows the actual remote sensing experiment results. Section 6 is the conclusion and expectation of future work.
Fig. 2. Aerial view of super-resolution remote sensing system in the lab. 𝐿1 (𝑓 = 50 mm, F/4.0, Carl Zeiss) is the objective lens focusing the imaging scene onto the DMD (VIALUX V-7001, DLP7000, 1024 × 768 pixels, micromirror pitch size of 13.7 × 13.7 μm2 ). 𝐿2 (Industrial lens, Primary Magnification, PMAG = 0.15X–0.5X, F/4.0, Edmund) is the secondary imaging lens establish the relationship between pixels in DMD and pixels in CCD (FLIR, 1032 × 776 pixels, pixel size of 4.65 × 4.65 μm2 ). In experiments, to produce a 512 × 512 image we only select 256 × 256 pixels (when 𝑘 = 𝑙 = 2) and 128 × 128 pixels (when 𝑘 = 𝑙 = 4) in the central area of the CCD. Meanwhile we select 512 × 512 micromirrors in the central area of the DMD both when 𝑘 = 𝑙 = 2 and when 𝑘 = 𝑙 = 4.
2. Super-resolution remote sensing system Fig. 1 shows the framework of the proposed super-resolution remote sensing system. Fig. 2 shows the aerial view of our system. Our purpose is to use the 𝑀 × 𝑁 pixels of the CCD array to produce a 𝑘𝑀 × 𝑙𝑁 size image (𝑘 and 𝑙 represent the scaling factor), by applying size of 𝑘𝑀 × 𝑙𝑁 pixels MEMS device (we use DMD in this paper) as the coded aperture. The acquisition model can be illustrated by Eq. (1). 𝐿𝑚 (𝑖, 𝑗) = 𝐷𝑚 (𝐻 (𝑥, 𝑦)) + 𝑛 (𝑖, 𝑗)
frames make it possible to solve Eq. (1) and reconstruct 𝐻 (𝑥, 𝑦) without any information loss, except for some extra noise introduced as a result of diffraction. However, if 𝑆 is large, the exposure time of the system will be long, which will reduce the frame frequency and consequently degrade the performance of this remote sensing system. Additionally, the large scaling factor will lead to a large data size, placing stress on the transmission of data, particularly for the spatial remote sensing system.
(1)
where 𝐿𝑚 is the LR image captured by CCD, 𝑚 ≤ 𝑆 is the index of LR images, 𝑛 (𝑖, 𝑗) is the additive noise, and 𝐻 (𝑥, 𝑦) is the HR image. 𝐷𝑚 (⋅) is the down-sample factor of the system representing the code loaded on the DMD. In this down-sample factor, the Point Spread Function (PSF) of the DMD should also be considered. Changing the code without any repetitive pixel of the DMD and capturing 𝑆 = 𝑘 ⋅ 𝑙
MEMS devices are widely used in various fields including telecommunications, medical applications and display industry [10]. For decades of research and industrialization, the performance of MEMS has been enhanced including lower economic cost and higher resolution. 411
X. Zhang et al.
Optics Communications 426 (2018) 410–417
Fig. 3. Example of cost reducing by substituting HR CCD with LR CCD and DMD.
DMD as the most typical example of MEMS has been applied to digital light processing (DLP) for cinema products and home-entertainment products of Texas Instruments [11]. Comparing to HR image sensors for remote sensing systems like CCD47-20 (with 1024 × 1024 pixels) from Teledyne e2v Company whose products are widely used in telescope and astronomical applications [12,13], DMD (DLP4710 with 1920 × 1080 resolution from Texas Instruments) only cost 2% of this CCD. As shown in Fig. 3, replacing a HR CCD by the combination of a LR sensor like CCD77-00 (with 512 × 512 pixels from Teledyne e2v Company) and a HR DMD makes it possible to reduce the cost by about 36% when 𝑘 = 𝑙 = 2. The percentage may change a little for the varying price in different regions. If the scaling factors 𝑘 and 𝑙 are larger than 2, our method may even perform better in cost reduction. The statistics are general because we ignore the additional cost of the optical and mechanical instruments. However it still gives us a possible method to acquire low cost remote sensing system. Although the CCD and DMD we use in experiments of this paper are not the same as the ones mentioned above, we still are able to demonstrate the performance of our method. For convenience, we select 256 × 256 pixels (when 𝑘 = 𝑙 = 2) and 128 × 128 pixels (when 𝑘 = 𝑙 = 4) of the CCD to represent two kind of LR CCDs. For DMD, the central 512 × 512 area is occupied under both circumstances. CS theory combines sampling and compression into non-adaptive linear measurement process [8] at a rate significantly below the Nyquist [6]. The classical CS acquisition approach can be stated as follows: 𝑦 = 𝛷𝑥 = 𝛷𝛹 𝑠 = 𝜃𝑠. R𝑀
Here, 𝐷𝑚 is the down-sample matrix of the 𝑚’th frame LR image, like the one in Eq. (1). Here, 𝑥 ∈ R𝑁 represents the HR image and 𝐷𝑚 𝑥 provides the 𝑚’th frame of LR image. Note that 𝛷𝑚 is the measurement ′ = 𝛷 𝐷 denotes matrix is not dependent on the signal matrix [6]. 𝛷𝑚 𝑚 𝑚 the modulated measurement matrix. We regularly set the code of DMD (shown in Fig. 4) and capture LR images first instead of randomly setting the code unlike the single-pixel camera, thus we can avoid repetition selection of the pixels. Then we will modulate the LR images with Gaussian random matrix 𝛷𝑚 in order to make it meet the RIP and ′ = 𝛷 𝐷 .𝜃 ′ denotes 𝛷′ 𝛹 . produce 𝛷𝑚 𝑚 𝑚 𝑚 𝑚 Compressible signals are well approximated by K-sparse representation [6]. Research studies have asserted that natural images tend to be compressible in the discrete cosine transform (DCT) and wavelet bases on which the JPEG and JPEG2000 compression standards are based [16]. In this work we introduce DCT as the basis matrix 𝛹 . 𝑦𝑚 is the measurements vector of the 𝑚’th LR image. Each 𝑦𝑚 is 𝑁𝑚 -dimension vector. Therefore, we will use 𝑚 × 𝑁𝑚 (𝑚 < 𝑆) equations to solve 𝑆 × 𝑁𝑚 = 𝑁 elements in 𝑥 ∈ R𝑁 . In this paper we choose Iteratively Reweighted Least-Squares (IRLS) [17] to solve this underdetermined problem, as is illustrated by the following minimization problem. min 𝑠
𝑠.𝑡. 𝑦𝑚 = 𝜃𝑚′ 𝑠,
∀𝑚 < 𝑘𝑙.
(4)
To find that 𝑥 ∈ R𝑁 is equal to solve the 𝓁𝑝 minimization of (0 < 𝑝 ≤ 1). It has been shown that by reducing the value of 𝑝 it is possible to reduce the number of required linear measurements 𝓁 with respect to that attained for 𝑝 = 1. [18]. The IRLS method is based on solving (4) with modified objective function, which at each iteration function ∑ 𝑝 approaches 𝑁 𝑘=1 |𝑠| . Simply, replacing the 𝓁𝑝 objective function in (4) by a weighted 𝓁2 norm,
(2) R𝑁 (𝑁
In this case 𝑦 ∈ is the vector of stacking measurements; 𝑥 ∈ < 𝑀) is the original compressible signal; 𝛷 is the 𝑀 × 𝑁 measurement matrix; and 𝜃 ∶= 𝛷𝛹 where 𝛹 is the 𝑁 × 𝑁 basis matrix. We choose the Gaussian random matrix as 𝛷 because of its high probability of satisfying the so-called restricted isometry property (RIP) [14]. 𝑥 in the spatial domain can be represented by vector 𝑠, which is 𝐾-sparse (𝐾 < 𝑁 coefficients in 𝑠 are non-zero) in the 𝛹 domain. To find the solution of the equation, recovering 𝑥, many reconstruction algorithms have been studied in recent years. The famous single-pixel camera is based on the CS theory, which reconstructs images from the measurement of the correlation between masks and imaging as viewed by a single photodiode [15]. However, the signal-to-noise ratio (SNR) scales poorly with increasing resolution [15]. In our system this problem can be overcome. Meanwhile, the data rate can also be reduced, which means that if 𝑆 is smaller than 𝑘 ⋅ 𝑙, the reconstructed HR image can maintain a high SNR. However our SR remote sensing architecture is not suggested on moving platforms. As the exposure time is longer than normal remote sensing system, we cannot ensure the accurate registration for every pixel in the exposure time when the platform is moving. Staring remote sensing systems, like telescopes on land or synchronous orbit satellites are recommended to use our method.
min 𝑠
𝑁 ∑
𝑤𝑖 𝑠𝑖 2 ,
𝑠.𝑡. 𝑦𝑚 = 𝜃𝑚′ 𝑠,
∀𝑚 < 𝑘𝑙
(5)
𝑖=1
| |𝑝−2 where 𝑤𝑖 = |𝑠(𝑛−1) is the first-order approximation to the 𝓁𝑝 objective | | 𝑖 | function. Changing 𝑤𝑖 at each iteration so that 𝑤𝑖 𝑠𝑖 2 is sufficiently close to ‖𝑠‖𝑝𝑝 in (4) after convergence. The solution of (5) is 𝑇
𝑇
𝑠(𝑛−1) = 𝑄𝑛 𝜃𝑚′ (𝜃𝑚′ 𝑄𝑛 𝜃𝑚′ )−1 𝑏,
(6)
where 𝑄𝑛 is the diagonal matrix with entries | |2−𝑝 1∕𝑤𝑖 = |𝑠(𝑛−1) | . | 𝑖 |
(7)
The convergence criterion for each iteration stage can be illustrated by √ ‖ 𝑛 ‖ ‖𝑠 − 𝑠𝑛−1 ‖ ‖ ‖ < 𝜇. 𝑛−1 ‖ 100 1+‖ ‖𝑠 ‖
(8)
Once (8) is attained, 𝜇 is reduced by a factor of 10, and the iterative procedure is repeated until 𝜇 < 10−8 [17]. In summary, the sparse representation 𝑠 of the HR image can be reconstructed from sufficient linear measurements 𝑦𝑚 , 𝑚 = 1, 2 … 𝑘𝑙 by solving Eq. (6). The detailed parameters in this algorithm are shown in the following table.
3. Reconstruction algorithm IRLS-SR In our SR remote sensing system, the acquisition approach should be modified as follows: ′ ′ 𝑦𝑚 = 𝛷𝑚 𝐷𝑚 𝑥 = 𝛷𝑚 𝑥 = 𝛷𝑚 𝛹 𝑠 = 𝜃𝑚′ 𝑠.
1 ‖𝑠‖𝑝𝑝 , 2
(3) 412
X. Zhang et al.
Optics Communications 426 (2018) 410–417
Fig. 4. The code aperture pattern loaded on the DMD corresponding to one pixel on image sensor array. (a) Represents the four templates used for 𝑘 = 2, 𝑙 = 2. (b) Represents the 16 templates used for 𝑘 = 4, 𝑙 = 4.
Fig. 5. Simulation on the ‘‘Airfield’’ dataset with different 𝑚, when 𝑘 = 𝑙 = 2. The upper-left image is the LR one with 256 × 256 resolution. The zoomed one is on its right. The zoomed SR images are shown under them with varying 𝑚. Algorithm 1. IRLS method for super-resolution remote sensing system.
When 𝑘 = 𝑙 = 2, we use the original image with 512 × 512 resolution as the ground truth. And based on the pattern in Fig. 4(a), we make the down-sample matrix 𝐷𝑚 . Additionally, we add Gaussian white noise 𝑛 ∼ N(0, 𝜎 2 ) to simulate the additive noise of our system where 𝜎 is the standard deviation measured by experiments. We apply the 3 × 3 Gaussian low pass filter whose deviation is 3 as the blurring operator to simulate the point spread function of the DMD. In Fig. 4 the white sub-pixels represent 1 and black ones represent 0. Figs. 5 and 6 show the results of using our algorithm on the dataset referred to as ‘‘Airfield’’. In order to show the performance of our method, we use Peak Signal to Noise Ratio (PSNR) to quantize it, and we compare it with the bilinear interpolation. The results in Table 1 show that when 𝑚 = 1 and 𝑚 = 2, our method does not perform so well as bilinear interpolation. But with the increasing 𝑚, the performance of our method becomes better, and when 𝑚 = 3 and 𝑚 = 4 (with 75% and 100% sample rate) its PSNR is 0.1–0.4 dB higher than bilinear interpolation. When 𝑘 = 𝑙 = 4, we also use the original image with 512 × 512 resolution as the ground truth of the HR image. Then the LR image will
Parameters: 𝑝 = 1, 𝜃𝑚′ , 𝑀 = 256, 𝑁 = 256, 𝑘 = 𝑙 = 2 or 𝑘 = 𝑙 = 4, 𝜇 = 1. Step 1: Initialize the image size and number of frames 𝑚. Step 2: Capture the 𝑚 LR images. Step 3: Modulate the m LR images with Gaussian Random matrix. Obtain 𝑦𝑚 , 𝑚 < 𝑘𝑙. Step 4: Do the inner loop: 4.1 Initialize 𝑛 ∶= 1, 𝑠0 = (0, 0, … , 0) and 𝑄(0) = 𝑂. 4.2 Update 𝑄(𝑛) using (7). 4.3 Compute 𝑠𝑛 using (6). 4.4 If (8) is satisfied, go to step 5; otherwise, let 𝑛 = 𝑛 + 1 and go to step 4.2. Step 5: Update the regularization parameter, 𝜇 = 𝜇∕10. Step 6: If 𝜇 < 10−8 , finish; else, go to Step 4.
4. Simulation results In order to demonstrate the performance of our proposed algorithm, comparing with the bilinear interpolation [19], we test it in two circumstances, 𝑘 = 𝑙 = 2 and 𝑘 = 𝑙 = 4. Varying the number of frames 𝑚 provides us with different sampling rate. 413
X. Zhang et al.
Optics Communications 426 (2018) 410–417
Fig. 6. Simulation on the ‘‘Airfield’’ dataset with different 𝑚, when 𝑘 = 𝑙 = 4. The upper-left image is the LR one with 128 × 128 resolution. The zoomed one is on its right. The zoomed SR images are shown under them with varying 𝑚. Table 1 PSNR of using our method on different datasets (𝑘 = 𝑙 = 2). Proposed method
Aerial Airfield Airplane Baboon Boat Bridge Goldhill Lena Man
Bilinear interpolation
𝑚=1
𝑚=2
𝑚=3
𝑚=4
𝑚=1
𝑚=2
𝑚=3
𝑚=4
21.9 dB 21.7 dB 24.7 dB 20.4 dB 24.7 dB 22.2 dB 26.0 dB 28.8 dB 25.7 dB
25.9 dB 24.9 dB 27.4 dB 23.6 dB 26.5 dB 25.4 dB 28.8 dB 31.5 dB 28.1 dB
29.2 dB 28.7 dB 32.0 dB 27.2 dB 31.5 dB 29.1 dB 32.7 dB 34.9 dB 32.2 dB
32.8 dB 32.7 dB 34.8 dB 31.7 dB 34.4 dB 33.3 dB 35.3 dB 36.2 dB 35.1 dB
23.1 dB 22.9 dB 25.7 dB 21.4 dB 25.9 dB 23.3 dB 27.1 dB 30.9 dB 26.9 dB
28.5 dB 27.1 dB 29.8 dB 25.7 dB 28.7 dB 27.4 dB 31.1 dB 34.9 dB 30.6 dB
28.9 dB 28.5 dB 31.6 dB 27.0 dB 31.3 dB 28.9 dB 32.5 dB 35.2 dB 32.0 dB
32.6 dB 32.4 dB 34.8 dB 31.3 dB 34.3 dB 33.0 dB 35.3 dB 36.3 dB 35.0 dB
Table 2 PSNR of using our method on different datasets (𝑘 = 𝑙 = 4). Proposed method
Aerial Airfield Airplane Baboon Boat Bridge Goldhill Lena Man
Bilinear interpolation
𝑚=4
𝑚=9
𝑚 = 12
𝑚 = 16
𝑚=4
𝑚=9
𝑚 = 12
𝑚 = 16
20.2 dB 19.7 dB 22.9 dB 19.0 dB 23.1 dB 20.4 dB 24.2 dB 27.2 dB 23.6 dB
25.0 dB 24.5 dB 27.7 dB 23.1 dB 27.6 dB 24.9 dB 29.0 dB 32.6 dB 28.5 dB
26.6 dB 26.8 dB 29.3 dB 25.1 dB 31.3 dB 26.7 dB 31.1 dB 34.4 dB 30.9 dB
33.4 dB 33.2 dB 34.9 dB 33.4 dB 34.9 dB 34.1 dB 35.7 dB 36.0 dB 35.5 dB
20.9 dB 20.6 dB 24.0 dB 19.5 dB 23.6 dB 21.2 dB 25.1 dB 27.0 dB 24.6 dB
24.5 dB 24.0 dB 27.3 dB 22.3 dB 27.0 dB 24.3 dB 28.4 dB 31.7 dB 28.0 dB
26.2 dB 26.2 dB 28.7 dB 24.1 dB 30.7 dB 26.1 dB 30.4 dB 34.3 dB 30.5 dB
33.3 dB 33.0 dB 34.8 dB 33.1 dB 34.8 dB 33.9 dB 35.7 dB 35.9 dB 35.4 dB
be 128 × 128 resolution and the coded aperture pattern is on Fig. 4(b). The results in Table 2 show that when the sampling rate exceeds 50% (𝑚 ≥ 9), the performance of our method is 0.1–0.9 dB higher than bilinear interpolation.
we have to make sure that changing the state of micromirrors is distinguishable by the corresponding pixel in CCD. Fig. 7 shows the responses of one pixel in CCD versus the number of micromirrors of ‘‘on’’ state, when 𝑘 = 𝑙 = 2 and 𝑘 = 𝑙 = 4. In experiments, we illuminate the DMD with area light source. We take the average of all pixels in CCD to get the response of one pixel. In that case we may avoid the influence of noise. Therefore the curves in Fig. 7 are smooth and linear. The curves convince us that CCD is able to sense the changing of DMD as the curves take up almost all the range of 8-bit quantization. The curves also convince us the assumption of linearity we made in the next part is reasonable. Secondly, we are going to discuss the fixed-pattern noise that is introduced by the mismatch of the DMD and the sensor, as shown in Fig. 8.
5. Imaging experiments In this section, we use our remote sensing system to image real scenes. But before that we are going to discuss some problems of the hardware of our remote sensing system. Firstly, we have to make sure that the hardware of the system possesses the potential to acquire HR images. It is possible that even though the resolution of DMD is higher, our system still cannot acquire HR images due to the bad performance of optical system, low signal to noise ratio or poor sensitivity of the CCD. Considering these problems, 414
X. Zhang et al.
Optics Communications 426 (2018) 410–417
(a) 𝑘 = 𝑙 = 2.
Fig. 9. The non-uniformity effect when 𝑘 = 2, 𝑙 = 2, 𝑚 = 4 and 𝑘 = 4, 𝑙 = 4, 𝑚 = 16. (a) 𝑘 = 2, 𝑙 = 2, 𝑚 = 4. (b) 𝑘 = 4, 𝑙 = 4, 𝑚 = 16. The sampling rate is 100%, and the reconstructed high-resolution image on the left is made by bilinear interpolation. The enlarged figure is shown on the right. (a) and (b) were taken with different exposure time and different angles, which causes the change of the brightness and details in the figures.
(b) 𝑘 = 𝑙 = 4. Fig. 7. The response of one pixel in CCD on average versus the number of corresponding micromirrors of ‘‘on’’ state in DMD.
non-uniformity [20] effect will cause great damage to the image quality, as shown in Fig. 9. The non-uniformity phenomenon is a common problem for image sensors, especially for the infrared focal-plane array (FPA) [20]. Although the CCD we use does not have many problems about nonuniformity, the combination of CCD and DMD regarded as a larger size image sensor will suffer from the non-uniformity phenomenon for the mismatch problem. The fixed-pattern noise can be relieved by nonuniformity correction, NUC [20], which has been studied for decades. The two-point NUC [21] is the most common method that is widely used in practice. Assuming that the response of each pixel of the image sensor is linear and stable with time, the output of the one located at (𝑥, 𝑦) can be expressed as 𝑧𝑡 (𝑥, 𝑦) = 𝑔 (𝑥, 𝑦) 𝑠𝑡 (𝑥, 𝑦) + 𝑏 (𝑥, 𝑦) + 𝑁 (𝑥, 𝑦) , Fig. 8. The effect caused by the mismatch of the CCD and DMD. In this figure, the CCD array is 2 × 2 and the DMD resolution is 4 × 4, therefore 𝑘 = 2, 𝑙 = 2. The black bar of the CCD is edge of the pixel of the CCD, because the fill factor of the sensor is not 100%. The red segments represent the energy loss of the light reflected from DMD, which causes the effect of non-uniformity. Different color represents the different coded aperture introduced in DMD. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
(9)
where 𝑁 (𝑥, 𝑦) is the random noise, 𝑧𝑡 (𝑥, 𝑦) is the observed scene value, 𝑠𝑡 (𝑥, 𝑦) is the true scene value, and 𝑔 (𝑥, 𝑦) and 𝑏 (𝑥, 𝑦) are the gain and bias/offset. By calibration of a reference source twice, the gain and bias/offset of each pixel will be solved and the output of the image sensor will be modified. As the linearity of response is testified in Fig. 7, it is reasonable to embed NUC into our algorithm. In our method, applying NUC to the output of our algorithm is not suggested especially when the sampling rate is lower than 100%. Because the output calculated by CS theory may not obey the assumption of linearity of each pixel. Instead, the NUC should be applied to the source of the data that acquired. Therefore the down-sampling factor 𝐷𝑚 , which deals with the source LR data, has to be modified as follows:
As shown in Fig. 8, the mismatch of the CCD and DMD is hard to eliminate due to the precision of the mechanical structure and the optical distortion problems. The red segments in Fig. 8 represent the energy loss caused by the mismatch. Therefore, in Fig. 8 the frame ‘‘gray’’ gets less energy than frame ‘‘purple’’, ‘‘green‘‘ and ‘‘yellow’’. This
′ 𝐷𝑚 (𝑥, 𝑦) = 𝑔 (𝑥, 𝑦) 𝐷𝑚 (𝑥, 𝑦) + 𝑏 (𝑥, 𝑦) + 𝑁 (𝑥, 𝑦) ,
415
(10)
X. Zhang et al.
Optics Communications 426 (2018) 410–417 Table 3 Non-uniformity of our method and bilinear interpolation on a testing uniform target. 𝑘 = 2, 𝑙 = 2
Our method Bilinear interpolation
𝑘 = 4, 𝑙 = 4
𝑚=1
𝑚=2
𝑚=3
𝑚=4
𝑚=4
𝑚=9
𝑚 = 12
𝑚 = 16
1.1% 14.7%
3.5% 14.6%
1.5% 14.6%
1.7% 14.7%
2.4% 14.0%
2.2% 14.2%
2.1% 14.2%
2.2% 14.3%
Fig. 10. The remote sensing results of our system when 𝑘 = 2, 𝑙 = 2. The LR image of 256 × 256 resolution is shown on the upper left with its zoomed image in the middle. The SR images are of 512 × 512 resolution. The zoomed SR image reconstructed by bilinear interpolation is on the upper right, which contains severe non-uniformity problem. Our method’s results are shown on the below with varying 𝑚. We find less non-uniformity in our method’s results visually. ′ (𝑥, 𝑦) is the modified downwhere 𝑁 (𝑥, 𝑦) is the random noise, 𝐷𝑚 sampling factor, and 𝑔 (𝑥, 𝑦) and 𝑏 (𝑥, 𝑦) are the gain and bias/offset, respectively, at this pixel. This model is based on the assumption that 𝑔 (𝑥, 𝑦) and 𝑏 (𝑥, 𝑦) slowly change, being constant during the period of calibration and imaging. In order to calculate the gain 𝑔 (𝑥, 𝑦) and the bias 𝑏 (𝑥, 𝑦), we occupy the normal two-point NUC [21] and embed it into this super-resolution algorithm, as mentioned above. We use integrating sphere as the reference sources to provide two reference point to solve 𝑔 (𝑥, 𝑦) and 𝑏 (𝑥, 𝑦). We use non-uniformity 𝑈𝑅 to judge the performance of our two-point NUC embedded algorithm, √ √ 𝑘𝑀 ∑ 𝑙𝑁 [ √ ]2 ∑ 1 1√ (11) 𝑆𝑖,𝑗 (𝜑) − 𝑆 . 𝑈𝑅 = √ 𝑘𝑀 × 𝑙𝑁 𝑆 𝑖=1 𝑗=1
𝑆=
𝑘𝑀 ∑ 𝑙𝑁 ∑ 1 𝑆 (𝜑) , 𝑘𝑀 × 𝑙𝑁 𝑖=1 𝑗=1 𝑖,𝑗
performance when the sampling rate exceeds 50%. With the help of our method, the details of the air-conditioner in the image can be distinguished in the SR images. However, the non-uniformity phenomenon still exists even though it is relieved by the modified down-sample factor of our method. We may focus on more sophisticated NUC method in the future to achieve better performance. 6. Conclusions In this paper we present a novel super-resolution remote sensing system that occupies the high resolution of the MEMS to provide image resolution which surpasses that of the original image sensor. With this method, the cost for high resolution CCD will be reduced. The algorithm inside is based on compressive sensing theory. With several frames of exposure, the system is able to reconstruct HR image. The simulation results show that our method outperforms the classical bilinear interpolation. Through simulation and remote sensing experiments, we find our method possesses better performance when the sampling rate is greater than 50% (when 𝑘 = 2, 𝑙 = 2 or 𝑘 = 4, 𝑙 = 4). With this help, the data transmission and storage burden will be released by 50%. Actual imaging experiments will introduce the non-uniformity effects caused by the hardware precision. Our method with NUC embedded is capable of relieving the fixed-pattern noise of the non-uniformity. The performance is testified quantitatively and visually. We expect more future work applying better NUC algorithm embedded in our method, as our method occupies the classical two-point NUC. As our work only analyzes the monochrome image, we anticipate that more research studies will focus on spectral problems.
(12)
where 𝜑 represents the incident irradiance on pixel (𝑖, 𝑗) and 𝑆𝑖,𝑗 (𝜑) is the output of the SR sensing system. By taking the average of all 𝑘𝑀 × 𝑙𝑁 pixels (𝑀 × 𝑁 is the original size of the CCD), we will have 𝑆 representing the mean of the output of a uniform target provided by the integrating sphere. According to the definition of 𝑈𝑅 , we may conclude that the lower value represents the better performance of uniformity. We compare our embedded method with classical bilinear interpolation. Through the results shown in Table 3, we find our method helps to improve the uniformity quantitatively. Because of lack of ground-truth, when evaluating the performance of remote sensing results we have to judge the image quality only in visual way. Figs. 10 and 11 provide the visual results when we use our remote sensing system to image the building about 125 m away from us. Comparing these images with the ones in Fig. 9, we find that the fixed-pattern noise, especially the grid noise, is relieved visually. Corresponding with the simulation results, our method possesses better
Acknowledgment This work is supported by the Chinese Academy of Sciences Innovation Fund (grant CXJJ-16S054). 416
X. Zhang et al.
Optics Communications 426 (2018) 410–417
Fig. 11. The remote sensing results of our system when 𝑘 = 2, 𝑙 = 2. The LR image of 128 × 128 resolution is shown on the upper left with its zoomed image in the middle. The SR images are of 512 × 512 resolution. The zoomed SR image reconstructed by bilinear interpolation is on the upper right, which contains severe non-uniformity problem. Our method’s results are shown on the below with varying 𝑚. We find less nonuniformity in our method’s results visually.
References
[12] P. Jorden, P. Jerram, M. Robbins, e2v CCD and CMOS sensors and systems designed for astronomical applications, in: SPIE Astronomical Telescopes + Instrumentation, 2016, 991504. [13] J. Wolf, H.P. Röser, Testing the e2v ccd47-20 as the new sensor for the sofia target acquisition and tracking cameras, Proc. SPIE - Int. Soc. Opt. Eng. 7733 (6) (2010). [14] E.J. Candes, J. Romberg, T. Tao, Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information, IEEE Trans. Inform. Theory 52 (2006) 489–509. [15] M.J. Sun, M.P. Edgar, D.B. Phillips, G.M. Gibson, M.J. Padgett, Improving the signalto-noise ratio of single-pixel imaging using digital micro-scanning, Opt. Express 24 (10) (2016) 10476. [16] M.D. Adams, R. Ward, Wavelet transforms in the JPEG-2000 standard. Communications, in: Computers and Signal Processing, PACRIM. 2001 IEEE Pacific Rim Conference, IEEE, 2001, pp. 160–163. [17] C.J. Miosso, R.V. Borries, M. Argaez, L. Velazquez, C. Quintero, C.M. Potes, Compressive sensing reconstruction with prior information by iteratively reweighted least-squares, IEEE Trans. Signal Process. 57 (6) (2009) 2424–2431. [18] R. Chartrand, W. Yin, Iteratively reweighted algorithms for compressive sensing, in: IEEE International Conference on Acoustics, Speech and Signal Processing, IEEE, 2008, pp. 3869–3872. [19] K.T. Gribbon, D.G. Bailey, A novel approach to real-time bilinear interpolation, in: IEEE International Conference on Field-Programmable Technology, 2004. Proceedings, IEEE, 2004, pp. 126–131. [20] W. Zhao, C. Zhang, Scene-based non-uniformity correction and enhancement: pixel statistics and sub-pixel motion, J. Opt. Soc. Amer. Opt. Image Sci. Vis. 25 (7) (2008) 1668. [21] H.X. Zhou, R. Lai, S.Q. Liu, G. Jiang, New improved non-uniformity correction for infrared focal plane arrays, Opt. Commun. 245 (1–6) (2005) 49–53.
[1] D.A. Stow, A. Hope, D. Mcguire, D. Verbyla, J. Gamon, F. Huemmrich, et al., Remote sensing of vegetation and land-cover change in arctic tundra ecosystems, Remote Sens. Environ. 89 (3) (2004) 281–308. [2] Na Qi, Yunhui Shi, Xiaoyan Sun, Wenpeng Ding, Baocai Yin, Single image superresolution via 2-D sparse representation, in: 2015 IEEE International Conference on Multimedia and Expo (ICME), 2015, pp. 1–6. [3] K. Nasrollahi, T.B. Moeslund, Super-resolution: a comprehensive survey, Mach. Vis. Appl. 25 (6) (2014) 1423–1468. [4] S. Zhou, Y. Yuan, L. Su, X. Ding, J. Wang, Multiframe super resolution reconstruction method based on light field angular images, Opt. Commun. 404 (1) (2017) 189–195. [5] A.W.M.V. Eekeren, K. Schutte, L.J.V. Vliet, Multi-frame super-resolution reconstruction of small moving objects, IEEE Trans. Image Process. Publ. IEEE Signal Process. Soc. 19 (11) (2010) 2901–2912. [6] D. Boufounos, D. Liu, P.T. Boufounos, A lecture on compressive sensing, IEEE Signal Process. Mag. (2007) 1–9. [7] D.L. Donoho, Compressed sensing, IEEE Trans. Inform. Theory 52 (4) (2006) 1289– 1306. [8] M.F. Duarte, M.A. Davenport, D. Takhar, J.N. Laska, T. Sun, K.F. Kelly, et al., Singlepixel imaging via compressive sampling, IEEE Signal Process. Mag. 25 (2) (2008) 83–91. [9] J. Jin, B. Yang, K. Liang, X. Wang, General image de-noising framework based on compressive sensing theory, Comput. Graph. 38 (1) (2014) 382–391. [10] J. Ma, Advanced mems-based technologies and displays, Displays 37 (2015) 2–10. [11] D.S. Dewald, D.J. Segler, S.M. Penn, Advances in contrast enhancement for dlp projection displays, J. Soc. Inf. Disp. 11 (1) (2003) 177–181.
417
Contents lists available at ScienceDirect
Optics Communications journal homepage: www.elsevier.com/locate/optcom
MEMS-based super-resolution remote sensing system using compressive sensing Xudong Zhang a,b , Jianan Xie b , Chunlai Li b , Rui Xu a,b , Yue Zhang a,b , Shijie Liu a,b , Jianyu Wang a,b, * a b
University of Chinese Academy of Sciences, Yuquan Road, 100049 Beijing, China Shanghai Institute of Technical Physics of the Chinese Academy of Sciences, Yutian Road, 200083 Shanghai, China
ARTICLE
INFO
ABSTRACT
Keywords: Super resolution Remote sensing Compressive sensing Non-uniformity
In this article we present a novel remote sensing system the resolution of which exceeds the image sensor’s with the help of high-resolution MEMS device, which will reduce the cost of high-resolution image sensors. The superresolution factors of 2 × 2 and 4×4 have been tested by simulation and remote sensing experiments. By applying the theory of compressive sensing, we can compress the acquisition data rate while preserving high image quality. We employ the IRLS as the super-resolution reconstruction method. A sample rate greater than 50% guarantees that our algorithm is more accurate. Through experiments we have found that the mismatch of the image sensor and MEMS device diminishes the image uniformity. Our algorithm embedded with non-uniformity correction greatly alleviates the problem visually and quantitatively.
1. Introduction
input in the algorithms contain geometric or photometric differences that depict the details of a certain HR image [4]. The most common method is to use sequential LR frames that contain spatial differences of moving objects to reconstruct HR images [5]. However, in this method the improvement of special resolution exists only in the moving part of the image. The Compressive Sensing (CS) theory [6] has changed the signal pattern in many fields. The original goal of CS theory is to reconstruct a signal from a series of measurements whose sampling rates can be much fewer than the Shannon–Nyquist sampling theorem by applying sparsity of signal and optimization methods [6,7]. The most exciting application of this theory is the single-pixel camera, which substitutes the CCD or CMOS with a single photon detector and a digital micromirror device (DMD) [8]. However, due to the low imaging quality and long sampling time, this new type of imaging system is rarely used in the industrial field. Unlike traditional data compression method, by applying CS theory we are able to compress the data during sampling, which makes it possible to reduce the sensing cost. Moreover, recent research studies approve that CS theory containing signal sparsity has good performance in image de-noising [9]. Our SR remote sensing system is able to produce an HR image with reduced restoration noise after applying CS theory. Unlike the existing SR methods, we present a novel framework of remote sensing image system that manages to improve the resolution in
Remote sensing systems are widely applied in numerous fields, including oceanography, ecology, geology, hydrology, land surveying and military applications [1]. Higher resolution images provide more details and information, thus improving the analysis quality in most of these applications. Super-resolution (SR), the technique of enhancing the resolution of the imaging system, has attracted much research interests in the last decades [2,3]. The traditional means of attaining super resolution are hard-ware based approaches, including decreasing the pixel size and increasing the size of the area array in order to increase the number of pixels per unit area [3]. Although this kind of technique is both widely used and effective, this will result in the dramatically rising cost of the image sensor, particularly for the high-quality sensors used in remote sensing systems. Another method of SR is based on the algorithms that produce high-resolution (HR) image from one or more low-resolution (HR) images. The algorithms can be divided into two types: single-image SR algorithms and multi-frame SR algorithms. Single-image ones rely on the priori knowledge that provide high frequency information of the single image, which contains the neuralnetwork based and sparsity-based SR algorithms. However, despite of its extensibility features, single-image ones possess limited effect because information contained in single image and priori knowledge is limited. Multi-frame ones reconstruct HR images by assuming that the LR images *
Corresponding author at: Shanghai Institute of Technical Physics of the Chinese Academy of Sciences, Yutian Road, 200083 Shanghai, China. E-mail address: [email protected] (J. Wang).
https://doi.org/10.1016/j.optcom.2018.05.046 Received 2 March 2018; Received in revised form 12 May 2018; Accepted 13 May 2018 0030-4018/© 2018 Elsevier B.V. All rights reserved.
X. Zhang et al.
Optics Communications 426 (2018) 410–417
Fig. 1. Framework of the SR remote-sensing system. This system occupies an image sensor with 𝑀 ⋅ 𝑁 spatial resolution. The 𝑘 ⋅ 𝑙 times higher resolution MEMS device plays the role as coded aperture that modulate the subpixel information of the imaging scene. Simply to reconstruct a HR image, we need 𝑘 ⋅ 𝑙 frames LR image. (In Fig. 1 we show the scheme when 𝑘 = 2 and 𝑙 = 2.) However, with the help of CS theory it is possible to reduce the number of frames with high-quality HR images, thus enhancing the speed of imaging and compressing the data.
all regions of the image, possessing good imaging quality and breaking the limitation of the image sensors’ size without increasing cost constraint of the large-array sensors. The key is to apply a higher-resolution lower-cost digital micromirror device (DMD), which is a micro-electromechanical system (MEMS) and can be replaced with other MEMS like digital micro-shutter (DMS). Details of how to reduce the sensor’s cost will be discussed in the next section. With the help of Compressive Sensing (CS) theory [6], we are able to reconstruct a HR image with less sample rate. Moreover, we embedded non-uniformity correction algorithm in the CS algorithm, whereby our method is capable of relieving the fixed-pattern noise caused by the non-uniformity problem. To the best of our knowledge, the proposed architecture is firstly used for remote sensing tasks, and the algorithm in this article is novel. In summary, our method has two main advantages. Firstly, we are able to reduce the cost of HR image sensor for the remote sensing system by substituting it with LR sensor and HR DMD. Secondly, data compression is available, which may help to relieve the data transmission and storage burden for remote sensing systems. The rest of this paper is organized as follows: the next section explains the details of our hardware system. Section 3 explains the principle of the reconstruction algorithm. We show the performance of our algorithm through simulation in Section 4. Section 5 shows the actual remote sensing experiment results. Section 6 is the conclusion and expectation of future work.
Fig. 2. Aerial view of super-resolution remote sensing system in the lab. 𝐿1 (𝑓 = 50 mm, F/4.0, Carl Zeiss) is the objective lens focusing the imaging scene onto the DMD (VIALUX V-7001, DLP7000, 1024 × 768 pixels, micromirror pitch size of 13.7 × 13.7 μm2 ). 𝐿2 (Industrial lens, Primary Magnification, PMAG = 0.15X–0.5X, F/4.0, Edmund) is the secondary imaging lens establish the relationship between pixels in DMD and pixels in CCD (FLIR, 1032 × 776 pixels, pixel size of 4.65 × 4.65 μm2 ). In experiments, to produce a 512 × 512 image we only select 256 × 256 pixels (when 𝑘 = 𝑙 = 2) and 128 × 128 pixels (when 𝑘 = 𝑙 = 4) in the central area of the CCD. Meanwhile we select 512 × 512 micromirrors in the central area of the DMD both when 𝑘 = 𝑙 = 2 and when 𝑘 = 𝑙 = 4.
2. Super-resolution remote sensing system Fig. 1 shows the framework of the proposed super-resolution remote sensing system. Fig. 2 shows the aerial view of our system. Our purpose is to use the 𝑀 × 𝑁 pixels of the CCD array to produce a 𝑘𝑀 × 𝑙𝑁 size image (𝑘 and 𝑙 represent the scaling factor), by applying size of 𝑘𝑀 × 𝑙𝑁 pixels MEMS device (we use DMD in this paper) as the coded aperture. The acquisition model can be illustrated by Eq. (1). 𝐿𝑚 (𝑖, 𝑗) = 𝐷𝑚 (𝐻 (𝑥, 𝑦)) + 𝑛 (𝑖, 𝑗)
frames make it possible to solve Eq. (1) and reconstruct 𝐻 (𝑥, 𝑦) without any information loss, except for some extra noise introduced as a result of diffraction. However, if 𝑆 is large, the exposure time of the system will be long, which will reduce the frame frequency and consequently degrade the performance of this remote sensing system. Additionally, the large scaling factor will lead to a large data size, placing stress on the transmission of data, particularly for the spatial remote sensing system.
(1)
where 𝐿𝑚 is the LR image captured by CCD, 𝑚 ≤ 𝑆 is the index of LR images, 𝑛 (𝑖, 𝑗) is the additive noise, and 𝐻 (𝑥, 𝑦) is the HR image. 𝐷𝑚 (⋅) is the down-sample factor of the system representing the code loaded on the DMD. In this down-sample factor, the Point Spread Function (PSF) of the DMD should also be considered. Changing the code without any repetitive pixel of the DMD and capturing 𝑆 = 𝑘 ⋅ 𝑙
MEMS devices are widely used in various fields including telecommunications, medical applications and display industry [10]. For decades of research and industrialization, the performance of MEMS has been enhanced including lower economic cost and higher resolution. 411
X. Zhang et al.
Optics Communications 426 (2018) 410–417
Fig. 3. Example of cost reducing by substituting HR CCD with LR CCD and DMD.
DMD as the most typical example of MEMS has been applied to digital light processing (DLP) for cinema products and home-entertainment products of Texas Instruments [11]. Comparing to HR image sensors for remote sensing systems like CCD47-20 (with 1024 × 1024 pixels) from Teledyne e2v Company whose products are widely used in telescope and astronomical applications [12,13], DMD (DLP4710 with 1920 × 1080 resolution from Texas Instruments) only cost 2% of this CCD. As shown in Fig. 3, replacing a HR CCD by the combination of a LR sensor like CCD77-00 (with 512 × 512 pixels from Teledyne e2v Company) and a HR DMD makes it possible to reduce the cost by about 36% when 𝑘 = 𝑙 = 2. The percentage may change a little for the varying price in different regions. If the scaling factors 𝑘 and 𝑙 are larger than 2, our method may even perform better in cost reduction. The statistics are general because we ignore the additional cost of the optical and mechanical instruments. However it still gives us a possible method to acquire low cost remote sensing system. Although the CCD and DMD we use in experiments of this paper are not the same as the ones mentioned above, we still are able to demonstrate the performance of our method. For convenience, we select 256 × 256 pixels (when 𝑘 = 𝑙 = 2) and 128 × 128 pixels (when 𝑘 = 𝑙 = 4) of the CCD to represent two kind of LR CCDs. For DMD, the central 512 × 512 area is occupied under both circumstances. CS theory combines sampling and compression into non-adaptive linear measurement process [8] at a rate significantly below the Nyquist [6]. The classical CS acquisition approach can be stated as follows: 𝑦 = 𝛷𝑥 = 𝛷𝛹 𝑠 = 𝜃𝑠. R𝑀
Here, 𝐷𝑚 is the down-sample matrix of the 𝑚’th frame LR image, like the one in Eq. (1). Here, 𝑥 ∈ R𝑁 represents the HR image and 𝐷𝑚 𝑥 provides the 𝑚’th frame of LR image. Note that 𝛷𝑚 is the measurement ′ = 𝛷 𝐷 denotes matrix is not dependent on the signal matrix [6]. 𝛷𝑚 𝑚 𝑚 the modulated measurement matrix. We regularly set the code of DMD (shown in Fig. 4) and capture LR images first instead of randomly setting the code unlike the single-pixel camera, thus we can avoid repetition selection of the pixels. Then we will modulate the LR images with Gaussian random matrix 𝛷𝑚 in order to make it meet the RIP and ′ = 𝛷 𝐷 .𝜃 ′ denotes 𝛷′ 𝛹 . produce 𝛷𝑚 𝑚 𝑚 𝑚 𝑚 Compressible signals are well approximated by K-sparse representation [6]. Research studies have asserted that natural images tend to be compressible in the discrete cosine transform (DCT) and wavelet bases on which the JPEG and JPEG2000 compression standards are based [16]. In this work we introduce DCT as the basis matrix 𝛹 . 𝑦𝑚 is the measurements vector of the 𝑚’th LR image. Each 𝑦𝑚 is 𝑁𝑚 -dimension vector. Therefore, we will use 𝑚 × 𝑁𝑚 (𝑚 < 𝑆) equations to solve 𝑆 × 𝑁𝑚 = 𝑁 elements in 𝑥 ∈ R𝑁 . In this paper we choose Iteratively Reweighted Least-Squares (IRLS) [17] to solve this underdetermined problem, as is illustrated by the following minimization problem. min 𝑠
𝑠.𝑡. 𝑦𝑚 = 𝜃𝑚′ 𝑠,
∀𝑚 < 𝑘𝑙.
(4)
To find that 𝑥 ∈ R𝑁 is equal to solve the 𝓁𝑝 minimization of (0 < 𝑝 ≤ 1). It has been shown that by reducing the value of 𝑝 it is possible to reduce the number of required linear measurements 𝓁 with respect to that attained for 𝑝 = 1. [18]. The IRLS method is based on solving (4) with modified objective function, which at each iteration function ∑ 𝑝 approaches 𝑁 𝑘=1 |𝑠| . Simply, replacing the 𝓁𝑝 objective function in (4) by a weighted 𝓁2 norm,
(2) R𝑁 (𝑁
In this case 𝑦 ∈ is the vector of stacking measurements; 𝑥 ∈ < 𝑀) is the original compressible signal; 𝛷 is the 𝑀 × 𝑁 measurement matrix; and 𝜃 ∶= 𝛷𝛹 where 𝛹 is the 𝑁 × 𝑁 basis matrix. We choose the Gaussian random matrix as 𝛷 because of its high probability of satisfying the so-called restricted isometry property (RIP) [14]. 𝑥 in the spatial domain can be represented by vector 𝑠, which is 𝐾-sparse (𝐾 < 𝑁 coefficients in 𝑠 are non-zero) in the 𝛹 domain. To find the solution of the equation, recovering 𝑥, many reconstruction algorithms have been studied in recent years. The famous single-pixel camera is based on the CS theory, which reconstructs images from the measurement of the correlation between masks and imaging as viewed by a single photodiode [15]. However, the signal-to-noise ratio (SNR) scales poorly with increasing resolution [15]. In our system this problem can be overcome. Meanwhile, the data rate can also be reduced, which means that if 𝑆 is smaller than 𝑘 ⋅ 𝑙, the reconstructed HR image can maintain a high SNR. However our SR remote sensing architecture is not suggested on moving platforms. As the exposure time is longer than normal remote sensing system, we cannot ensure the accurate registration for every pixel in the exposure time when the platform is moving. Staring remote sensing systems, like telescopes on land or synchronous orbit satellites are recommended to use our method.
min 𝑠
𝑁 ∑
𝑤𝑖 𝑠𝑖 2 ,
𝑠.𝑡. 𝑦𝑚 = 𝜃𝑚′ 𝑠,
∀𝑚 < 𝑘𝑙
(5)
𝑖=1
| |𝑝−2 where 𝑤𝑖 = |𝑠(𝑛−1) is the first-order approximation to the 𝓁𝑝 objective | | 𝑖 | function. Changing 𝑤𝑖 at each iteration so that 𝑤𝑖 𝑠𝑖 2 is sufficiently close to ‖𝑠‖𝑝𝑝 in (4) after convergence. The solution of (5) is 𝑇
𝑇
𝑠(𝑛−1) = 𝑄𝑛 𝜃𝑚′ (𝜃𝑚′ 𝑄𝑛 𝜃𝑚′ )−1 𝑏,
(6)
where 𝑄𝑛 is the diagonal matrix with entries | |2−𝑝 1∕𝑤𝑖 = |𝑠(𝑛−1) | . | 𝑖 |
(7)
The convergence criterion for each iteration stage can be illustrated by √ ‖ 𝑛 ‖ ‖𝑠 − 𝑠𝑛−1 ‖ ‖ ‖ < 𝜇. 𝑛−1 ‖ 100 1+‖ ‖𝑠 ‖
(8)
Once (8) is attained, 𝜇 is reduced by a factor of 10, and the iterative procedure is repeated until 𝜇 < 10−8 [17]. In summary, the sparse representation 𝑠 of the HR image can be reconstructed from sufficient linear measurements 𝑦𝑚 , 𝑚 = 1, 2 … 𝑘𝑙 by solving Eq. (6). The detailed parameters in this algorithm are shown in the following table.
3. Reconstruction algorithm IRLS-SR In our SR remote sensing system, the acquisition approach should be modified as follows: ′ ′ 𝑦𝑚 = 𝛷𝑚 𝐷𝑚 𝑥 = 𝛷𝑚 𝑥 = 𝛷𝑚 𝛹 𝑠 = 𝜃𝑚′ 𝑠.
1 ‖𝑠‖𝑝𝑝 , 2
(3) 412
X. Zhang et al.
Optics Communications 426 (2018) 410–417
Fig. 4. The code aperture pattern loaded on the DMD corresponding to one pixel on image sensor array. (a) Represents the four templates used for 𝑘 = 2, 𝑙 = 2. (b) Represents the 16 templates used for 𝑘 = 4, 𝑙 = 4.
Fig. 5. Simulation on the ‘‘Airfield’’ dataset with different 𝑚, when 𝑘 = 𝑙 = 2. The upper-left image is the LR one with 256 × 256 resolution. The zoomed one is on its right. The zoomed SR images are shown under them with varying 𝑚. Algorithm 1. IRLS method for super-resolution remote sensing system.
When 𝑘 = 𝑙 = 2, we use the original image with 512 × 512 resolution as the ground truth. And based on the pattern in Fig. 4(a), we make the down-sample matrix 𝐷𝑚 . Additionally, we add Gaussian white noise 𝑛 ∼ N(0, 𝜎 2 ) to simulate the additive noise of our system where 𝜎 is the standard deviation measured by experiments. We apply the 3 × 3 Gaussian low pass filter whose deviation is 3 as the blurring operator to simulate the point spread function of the DMD. In Fig. 4 the white sub-pixels represent 1 and black ones represent 0. Figs. 5 and 6 show the results of using our algorithm on the dataset referred to as ‘‘Airfield’’. In order to show the performance of our method, we use Peak Signal to Noise Ratio (PSNR) to quantize it, and we compare it with the bilinear interpolation. The results in Table 1 show that when 𝑚 = 1 and 𝑚 = 2, our method does not perform so well as bilinear interpolation. But with the increasing 𝑚, the performance of our method becomes better, and when 𝑚 = 3 and 𝑚 = 4 (with 75% and 100% sample rate) its PSNR is 0.1–0.4 dB higher than bilinear interpolation. When 𝑘 = 𝑙 = 4, we also use the original image with 512 × 512 resolution as the ground truth of the HR image. Then the LR image will
Parameters: 𝑝 = 1, 𝜃𝑚′ , 𝑀 = 256, 𝑁 = 256, 𝑘 = 𝑙 = 2 or 𝑘 = 𝑙 = 4, 𝜇 = 1. Step 1: Initialize the image size and number of frames 𝑚. Step 2: Capture the 𝑚 LR images. Step 3: Modulate the m LR images with Gaussian Random matrix. Obtain 𝑦𝑚 , 𝑚 < 𝑘𝑙. Step 4: Do the inner loop: 4.1 Initialize 𝑛 ∶= 1, 𝑠0 = (0, 0, … , 0) and 𝑄(0) = 𝑂. 4.2 Update 𝑄(𝑛) using (7). 4.3 Compute 𝑠𝑛 using (6). 4.4 If (8) is satisfied, go to step 5; otherwise, let 𝑛 = 𝑛 + 1 and go to step 4.2. Step 5: Update the regularization parameter, 𝜇 = 𝜇∕10. Step 6: If 𝜇 < 10−8 , finish; else, go to Step 4.
4. Simulation results In order to demonstrate the performance of our proposed algorithm, comparing with the bilinear interpolation [19], we test it in two circumstances, 𝑘 = 𝑙 = 2 and 𝑘 = 𝑙 = 4. Varying the number of frames 𝑚 provides us with different sampling rate. 413
X. Zhang et al.
Optics Communications 426 (2018) 410–417
Fig. 6. Simulation on the ‘‘Airfield’’ dataset with different 𝑚, when 𝑘 = 𝑙 = 4. The upper-left image is the LR one with 128 × 128 resolution. The zoomed one is on its right. The zoomed SR images are shown under them with varying 𝑚. Table 1 PSNR of using our method on different datasets (𝑘 = 𝑙 = 2). Proposed method
Aerial Airfield Airplane Baboon Boat Bridge Goldhill Lena Man
Bilinear interpolation
𝑚=1
𝑚=2
𝑚=3
𝑚=4
𝑚=1
𝑚=2
𝑚=3
𝑚=4
21.9 dB 21.7 dB 24.7 dB 20.4 dB 24.7 dB 22.2 dB 26.0 dB 28.8 dB 25.7 dB
25.9 dB 24.9 dB 27.4 dB 23.6 dB 26.5 dB 25.4 dB 28.8 dB 31.5 dB 28.1 dB
29.2 dB 28.7 dB 32.0 dB 27.2 dB 31.5 dB 29.1 dB 32.7 dB 34.9 dB 32.2 dB
32.8 dB 32.7 dB 34.8 dB 31.7 dB 34.4 dB 33.3 dB 35.3 dB 36.2 dB 35.1 dB
23.1 dB 22.9 dB 25.7 dB 21.4 dB 25.9 dB 23.3 dB 27.1 dB 30.9 dB 26.9 dB
28.5 dB 27.1 dB 29.8 dB 25.7 dB 28.7 dB 27.4 dB 31.1 dB 34.9 dB 30.6 dB
28.9 dB 28.5 dB 31.6 dB 27.0 dB 31.3 dB 28.9 dB 32.5 dB 35.2 dB 32.0 dB
32.6 dB 32.4 dB 34.8 dB 31.3 dB 34.3 dB 33.0 dB 35.3 dB 36.3 dB 35.0 dB
Table 2 PSNR of using our method on different datasets (𝑘 = 𝑙 = 4). Proposed method
Aerial Airfield Airplane Baboon Boat Bridge Goldhill Lena Man
Bilinear interpolation
𝑚=4
𝑚=9
𝑚 = 12
𝑚 = 16
𝑚=4
𝑚=9
𝑚 = 12
𝑚 = 16
20.2 dB 19.7 dB 22.9 dB 19.0 dB 23.1 dB 20.4 dB 24.2 dB 27.2 dB 23.6 dB
25.0 dB 24.5 dB 27.7 dB 23.1 dB 27.6 dB 24.9 dB 29.0 dB 32.6 dB 28.5 dB
26.6 dB 26.8 dB 29.3 dB 25.1 dB 31.3 dB 26.7 dB 31.1 dB 34.4 dB 30.9 dB
33.4 dB 33.2 dB 34.9 dB 33.4 dB 34.9 dB 34.1 dB 35.7 dB 36.0 dB 35.5 dB
20.9 dB 20.6 dB 24.0 dB 19.5 dB 23.6 dB 21.2 dB 25.1 dB 27.0 dB 24.6 dB
24.5 dB 24.0 dB 27.3 dB 22.3 dB 27.0 dB 24.3 dB 28.4 dB 31.7 dB 28.0 dB
26.2 dB 26.2 dB 28.7 dB 24.1 dB 30.7 dB 26.1 dB 30.4 dB 34.3 dB 30.5 dB
33.3 dB 33.0 dB 34.8 dB 33.1 dB 34.8 dB 33.9 dB 35.7 dB 35.9 dB 35.4 dB
be 128 × 128 resolution and the coded aperture pattern is on Fig. 4(b). The results in Table 2 show that when the sampling rate exceeds 50% (𝑚 ≥ 9), the performance of our method is 0.1–0.9 dB higher than bilinear interpolation.
we have to make sure that changing the state of micromirrors is distinguishable by the corresponding pixel in CCD. Fig. 7 shows the responses of one pixel in CCD versus the number of micromirrors of ‘‘on’’ state, when 𝑘 = 𝑙 = 2 and 𝑘 = 𝑙 = 4. In experiments, we illuminate the DMD with area light source. We take the average of all pixels in CCD to get the response of one pixel. In that case we may avoid the influence of noise. Therefore the curves in Fig. 7 are smooth and linear. The curves convince us that CCD is able to sense the changing of DMD as the curves take up almost all the range of 8-bit quantization. The curves also convince us the assumption of linearity we made in the next part is reasonable. Secondly, we are going to discuss the fixed-pattern noise that is introduced by the mismatch of the DMD and the sensor, as shown in Fig. 8.
5. Imaging experiments In this section, we use our remote sensing system to image real scenes. But before that we are going to discuss some problems of the hardware of our remote sensing system. Firstly, we have to make sure that the hardware of the system possesses the potential to acquire HR images. It is possible that even though the resolution of DMD is higher, our system still cannot acquire HR images due to the bad performance of optical system, low signal to noise ratio or poor sensitivity of the CCD. Considering these problems, 414
X. Zhang et al.
Optics Communications 426 (2018) 410–417
(a) 𝑘 = 𝑙 = 2.
Fig. 9. The non-uniformity effect when 𝑘 = 2, 𝑙 = 2, 𝑚 = 4 and 𝑘 = 4, 𝑙 = 4, 𝑚 = 16. (a) 𝑘 = 2, 𝑙 = 2, 𝑚 = 4. (b) 𝑘 = 4, 𝑙 = 4, 𝑚 = 16. The sampling rate is 100%, and the reconstructed high-resolution image on the left is made by bilinear interpolation. The enlarged figure is shown on the right. (a) and (b) were taken with different exposure time and different angles, which causes the change of the brightness and details in the figures.
(b) 𝑘 = 𝑙 = 4. Fig. 7. The response of one pixel in CCD on average versus the number of corresponding micromirrors of ‘‘on’’ state in DMD.
non-uniformity [20] effect will cause great damage to the image quality, as shown in Fig. 9. The non-uniformity phenomenon is a common problem for image sensors, especially for the infrared focal-plane array (FPA) [20]. Although the CCD we use does not have many problems about nonuniformity, the combination of CCD and DMD regarded as a larger size image sensor will suffer from the non-uniformity phenomenon for the mismatch problem. The fixed-pattern noise can be relieved by nonuniformity correction, NUC [20], which has been studied for decades. The two-point NUC [21] is the most common method that is widely used in practice. Assuming that the response of each pixel of the image sensor is linear and stable with time, the output of the one located at (𝑥, 𝑦) can be expressed as 𝑧𝑡 (𝑥, 𝑦) = 𝑔 (𝑥, 𝑦) 𝑠𝑡 (𝑥, 𝑦) + 𝑏 (𝑥, 𝑦) + 𝑁 (𝑥, 𝑦) , Fig. 8. The effect caused by the mismatch of the CCD and DMD. In this figure, the CCD array is 2 × 2 and the DMD resolution is 4 × 4, therefore 𝑘 = 2, 𝑙 = 2. The black bar of the CCD is edge of the pixel of the CCD, because the fill factor of the sensor is not 100%. The red segments represent the energy loss of the light reflected from DMD, which causes the effect of non-uniformity. Different color represents the different coded aperture introduced in DMD. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
(9)
where 𝑁 (𝑥, 𝑦) is the random noise, 𝑧𝑡 (𝑥, 𝑦) is the observed scene value, 𝑠𝑡 (𝑥, 𝑦) is the true scene value, and 𝑔 (𝑥, 𝑦) and 𝑏 (𝑥, 𝑦) are the gain and bias/offset. By calibration of a reference source twice, the gain and bias/offset of each pixel will be solved and the output of the image sensor will be modified. As the linearity of response is testified in Fig. 7, it is reasonable to embed NUC into our algorithm. In our method, applying NUC to the output of our algorithm is not suggested especially when the sampling rate is lower than 100%. Because the output calculated by CS theory may not obey the assumption of linearity of each pixel. Instead, the NUC should be applied to the source of the data that acquired. Therefore the down-sampling factor 𝐷𝑚 , which deals with the source LR data, has to be modified as follows:
As shown in Fig. 8, the mismatch of the CCD and DMD is hard to eliminate due to the precision of the mechanical structure and the optical distortion problems. The red segments in Fig. 8 represent the energy loss caused by the mismatch. Therefore, in Fig. 8 the frame ‘‘gray’’ gets less energy than frame ‘‘purple’’, ‘‘green‘‘ and ‘‘yellow’’. This
′ 𝐷𝑚 (𝑥, 𝑦) = 𝑔 (𝑥, 𝑦) 𝐷𝑚 (𝑥, 𝑦) + 𝑏 (𝑥, 𝑦) + 𝑁 (𝑥, 𝑦) ,
415
(10)
X. Zhang et al.
Optics Communications 426 (2018) 410–417 Table 3 Non-uniformity of our method and bilinear interpolation on a testing uniform target. 𝑘 = 2, 𝑙 = 2
Our method Bilinear interpolation
𝑘 = 4, 𝑙 = 4
𝑚=1
𝑚=2
𝑚=3
𝑚=4
𝑚=4
𝑚=9
𝑚 = 12
𝑚 = 16
1.1% 14.7%
3.5% 14.6%
1.5% 14.6%
1.7% 14.7%
2.4% 14.0%
2.2% 14.2%
2.1% 14.2%
2.2% 14.3%
Fig. 10. The remote sensing results of our system when 𝑘 = 2, 𝑙 = 2. The LR image of 256 × 256 resolution is shown on the upper left with its zoomed image in the middle. The SR images are of 512 × 512 resolution. The zoomed SR image reconstructed by bilinear interpolation is on the upper right, which contains severe non-uniformity problem. Our method’s results are shown on the below with varying 𝑚. We find less non-uniformity in our method’s results visually. ′ (𝑥, 𝑦) is the modified downwhere 𝑁 (𝑥, 𝑦) is the random noise, 𝐷𝑚 sampling factor, and 𝑔 (𝑥, 𝑦) and 𝑏 (𝑥, 𝑦) are the gain and bias/offset, respectively, at this pixel. This model is based on the assumption that 𝑔 (𝑥, 𝑦) and 𝑏 (𝑥, 𝑦) slowly change, being constant during the period of calibration and imaging. In order to calculate the gain 𝑔 (𝑥, 𝑦) and the bias 𝑏 (𝑥, 𝑦), we occupy the normal two-point NUC [21] and embed it into this super-resolution algorithm, as mentioned above. We use integrating sphere as the reference sources to provide two reference point to solve 𝑔 (𝑥, 𝑦) and 𝑏 (𝑥, 𝑦). We use non-uniformity 𝑈𝑅 to judge the performance of our two-point NUC embedded algorithm, √ √ 𝑘𝑀 ∑ 𝑙𝑁 [ √ ]2 ∑ 1 1√ (11) 𝑆𝑖,𝑗 (𝜑) − 𝑆 . 𝑈𝑅 = √ 𝑘𝑀 × 𝑙𝑁 𝑆 𝑖=1 𝑗=1
𝑆=
𝑘𝑀 ∑ 𝑙𝑁 ∑ 1 𝑆 (𝜑) , 𝑘𝑀 × 𝑙𝑁 𝑖=1 𝑗=1 𝑖,𝑗
performance when the sampling rate exceeds 50%. With the help of our method, the details of the air-conditioner in the image can be distinguished in the SR images. However, the non-uniformity phenomenon still exists even though it is relieved by the modified down-sample factor of our method. We may focus on more sophisticated NUC method in the future to achieve better performance. 6. Conclusions In this paper we present a novel super-resolution remote sensing system that occupies the high resolution of the MEMS to provide image resolution which surpasses that of the original image sensor. With this method, the cost for high resolution CCD will be reduced. The algorithm inside is based on compressive sensing theory. With several frames of exposure, the system is able to reconstruct HR image. The simulation results show that our method outperforms the classical bilinear interpolation. Through simulation and remote sensing experiments, we find our method possesses better performance when the sampling rate is greater than 50% (when 𝑘 = 2, 𝑙 = 2 or 𝑘 = 4, 𝑙 = 4). With this help, the data transmission and storage burden will be released by 50%. Actual imaging experiments will introduce the non-uniformity effects caused by the hardware precision. Our method with NUC embedded is capable of relieving the fixed-pattern noise of the non-uniformity. The performance is testified quantitatively and visually. We expect more future work applying better NUC algorithm embedded in our method, as our method occupies the classical two-point NUC. As our work only analyzes the monochrome image, we anticipate that more research studies will focus on spectral problems.
(12)
where 𝜑 represents the incident irradiance on pixel (𝑖, 𝑗) and 𝑆𝑖,𝑗 (𝜑) is the output of the SR sensing system. By taking the average of all 𝑘𝑀 × 𝑙𝑁 pixels (𝑀 × 𝑁 is the original size of the CCD), we will have 𝑆 representing the mean of the output of a uniform target provided by the integrating sphere. According to the definition of 𝑈𝑅 , we may conclude that the lower value represents the better performance of uniformity. We compare our embedded method with classical bilinear interpolation. Through the results shown in Table 3, we find our method helps to improve the uniformity quantitatively. Because of lack of ground-truth, when evaluating the performance of remote sensing results we have to judge the image quality only in visual way. Figs. 10 and 11 provide the visual results when we use our remote sensing system to image the building about 125 m away from us. Comparing these images with the ones in Fig. 9, we find that the fixed-pattern noise, especially the grid noise, is relieved visually. Corresponding with the simulation results, our method possesses better
Acknowledgment This work is supported by the Chinese Academy of Sciences Innovation Fund (grant CXJJ-16S054). 416
X. Zhang et al.
Optics Communications 426 (2018) 410–417
Fig. 11. The remote sensing results of our system when 𝑘 = 2, 𝑙 = 2. The LR image of 128 × 128 resolution is shown on the upper left with its zoomed image in the middle. The SR images are of 512 × 512 resolution. The zoomed SR image reconstructed by bilinear interpolation is on the upper right, which contains severe non-uniformity problem. Our method’s results are shown on the below with varying 𝑚. We find less nonuniformity in our method’s results visually.
References
[12] P. Jorden, P. Jerram, M. Robbins, e2v CCD and CMOS sensors and systems designed for astronomical applications, in: SPIE Astronomical Telescopes + Instrumentation, 2016, 991504. [13] J. Wolf, H.P. Röser, Testing the e2v ccd47-20 as the new sensor for the sofia target acquisition and tracking cameras, Proc. SPIE - Int. Soc. Opt. Eng. 7733 (6) (2010). [14] E.J. Candes, J. Romberg, T. Tao, Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information, IEEE Trans. Inform. Theory 52 (2006) 489–509. [15] M.J. Sun, M.P. Edgar, D.B. Phillips, G.M. Gibson, M.J. Padgett, Improving the signalto-noise ratio of single-pixel imaging using digital micro-scanning, Opt. Express 24 (10) (2016) 10476. [16] M.D. Adams, R. Ward, Wavelet transforms in the JPEG-2000 standard. Communications, in: Computers and Signal Processing, PACRIM. 2001 IEEE Pacific Rim Conference, IEEE, 2001, pp. 160–163. [17] C.J. Miosso, R.V. Borries, M. Argaez, L. Velazquez, C. Quintero, C.M. Potes, Compressive sensing reconstruction with prior information by iteratively reweighted least-squares, IEEE Trans. Signal Process. 57 (6) (2009) 2424–2431. [18] R. Chartrand, W. Yin, Iteratively reweighted algorithms for compressive sensing, in: IEEE International Conference on Acoustics, Speech and Signal Processing, IEEE, 2008, pp. 3869–3872. [19] K.T. Gribbon, D.G. Bailey, A novel approach to real-time bilinear interpolation, in: IEEE International Conference on Field-Programmable Technology, 2004. Proceedings, IEEE, 2004, pp. 126–131. [20] W. Zhao, C. Zhang, Scene-based non-uniformity correction and enhancement: pixel statistics and sub-pixel motion, J. Opt. Soc. Amer. Opt. Image Sci. Vis. 25 (7) (2008) 1668. [21] H.X. Zhou, R. Lai, S.Q. Liu, G. Jiang, New improved non-uniformity correction for infrared focal plane arrays, Opt. Commun. 245 (1–6) (2005) 49–53.
[1] D.A. Stow, A. Hope, D. Mcguire, D. Verbyla, J. Gamon, F. Huemmrich, et al., Remote sensing of vegetation and land-cover change in arctic tundra ecosystems, Remote Sens. Environ. 89 (3) (2004) 281–308. [2] Na Qi, Yunhui Shi, Xiaoyan Sun, Wenpeng Ding, Baocai Yin, Single image superresolution via 2-D sparse representation, in: 2015 IEEE International Conference on Multimedia and Expo (ICME), 2015, pp. 1–6. [3] K. Nasrollahi, T.B. Moeslund, Super-resolution: a comprehensive survey, Mach. Vis. Appl. 25 (6) (2014) 1423–1468. [4] S. Zhou, Y. Yuan, L. Su, X. Ding, J. Wang, Multiframe super resolution reconstruction method based on light field angular images, Opt. Commun. 404 (1) (2017) 189–195. [5] A.W.M.V. Eekeren, K. Schutte, L.J.V. Vliet, Multi-frame super-resolution reconstruction of small moving objects, IEEE Trans. Image Process. Publ. IEEE Signal Process. Soc. 19 (11) (2010) 2901–2912. [6] D. Boufounos, D. Liu, P.T. Boufounos, A lecture on compressive sensing, IEEE Signal Process. Mag. (2007) 1–9. [7] D.L. Donoho, Compressed sensing, IEEE Trans. Inform. Theory 52 (4) (2006) 1289– 1306. [8] M.F. Duarte, M.A. Davenport, D. Takhar, J.N. Laska, T. Sun, K.F. Kelly, et al., Singlepixel imaging via compressive sampling, IEEE Signal Process. Mag. 25 (2) (2008) 83–91. [9] J. Jin, B. Yang, K. Liang, X. Wang, General image de-noising framework based on compressive sensing theory, Comput. Graph. 38 (1) (2014) 382–391. [10] J. Ma, Advanced mems-based technologies and displays, Displays 37 (2015) 2–10. [11] D.S. Dewald, D.J. Segler, S.M. Penn, Advances in contrast enhancement for dlp projection displays, J. Soc. Inf. Disp. 11 (1) (2003) 177–181.
417