- Email: [email protected]
High-quality image retrieval by iterative total-error compensation for single-pixel imaging of random illuminations
Optics and Lasers in Engineering 124 (2020) 105852
Contents lists available at ScienceDirect
Optics and Lasers in Engineering journal homepage: www.elsevier.com/locate/optlaseng
High-quality image retrieval by iterative total-error compensation for single-pixel imaging of random illuminations Jung-Ping Liu a,∗, Kun-Chi Tsai a, Bo-Bing Luo a, Yoshio Hayasaki b a b
Department of Photonics, Feng Chia University, 100 Wenhwa Rd., Seatwen, Taichung 40724, Taiwan Center for Optical Research & Education (CORE), Utsunomiya University, 7-1-2 Yoto, Utsunomiya 321-8585, Japan
a b s t r a c t We proposed iterative total-error compensation (ITEC) for the image retrieval of single-pixel imaging using random illuminations and point light detection. ITEC is based on the compensation of the error of all pixels arose in the previous retrieved image. The merit of ITEC is that its computing complexity is relatively low, and there is no demand for the type of object. In addition, ITEC can work in the circumstance that a SLM or an arrayed light source is unavailable. Therefore, ITEC provides high flexibility on cost and complexity of single-pixel imaging system.
1. Introduction Single-pixel imaging applies structured-light illumination and a single-pixel detector to get the spatial information of an object. A twodimensional (2D) or three-dimensional (3D) image of the object can be retrieved by using different algorithms. Since single-pixel imaging does not apply an arrayed image sensor, it has application potential in circumstances that a high-resolution camera is unavailable. In addition, single-pixel imaging has been applied to 3D imaging [1] and cytometry [2]. Ghost imaging (GI) is one of the easiest methods among the single-pixel imaging techniques [3–5]. GI applies random illuminations in imaging, and the image of the object is retrieved by the correlation of power measurements and the corresponding illumination patterns. In conventional GI, the signal-to-noise ratio (SNR) of a retrieved image is linearly proportional to the number of measurements. In typical imaging of a binary object, the SNR will be only unity when the number of measurements is equal to the pixel number of the image [4]. This property not only limits the quality of retrieved image, but also prevents GI from being applied in practical applications. So far, many techniques have been proposed to deal with this problem. These methods can be classified into three. The first is in the concern of devising special illumination patterns, such as Hadamard-pattern-based illumination [6,7] or Fourier-component-based illumination [8,9], to reduce the statistical noise. These techniques indeed improve the quality of retrieved images significantly. However, they must be realized using a spatial light modulator (SLM) or an arrayed light source [10]. In the circumstance that a SLM or an arrayed light source is unavailable, the special illumination patterns cannot be generated. The second kind of single-pixel imaging is based on the use of algorithms, such as Gerchberg–Saxton (GS) algorithm, for image optimization over conventional GI image [11,12].
∗
These methods have two drawbacks. First these methods include both the conventional correlation calculation and optimization procedure. Therefore, the full computing time is further longer than that of conventional GI. Second, the quality of images retrieved by these methods is still positively proportional to the number of measurements. In other words, a lot of measurements are still demanded to achieve high quality imaging. Finally, compressive-sensing-based algorithm can reduce the demand of large measurement number [13,14]. However, it has a presupposition on the type of signal, and its computational complexity is extremely high. In this paper, we consider improving the image quality by using random illuminations. We proposed a much different strategy of image retrieval for single-pixel imaging. First, any power measurement is not regarded as the image signal directly. Instead, it is applied to correct the error arose in the previous retrieved image. Second, every measurement can be applied to correct the error in not only the open pixels, but also the closed pixels, which is the concept of totalerror compensation. Finally, each measurement value can be applied to correct the pixel error unlimited times, provided the previous retrieved image has been changed. Therefore, a high-quality image can be retrieved by our proposed method with moderate computing cost. Since the proposed method applies random-illuminations, which can be generated by scattering [13,15–17]or a multimode fiber [18], the proposed method provides high flexibility for random-illuminationbased single-pixel imaging system. The remaining of the paper is organized as follows. In Section 2, the principle of GI and our proposed method, ITEC, will be explained. In Section 3, simulations are conducted to verify the proposed method. Experimental results are provided and discussed in Section 4. Finally, concluding remarks are made in Section 5.
Corresponding author. E-mail address: [email protected] (J.-P. Liu).
https://doi.org/10.1016/j.optlaseng.2019.105852 Received 26 March 2019; Received in revised form 7 August 2019; Accepted 5 September 2019 0143-8166/© 2019 Published by Elsevier Ltd.
J.-P. Liu, K.-C. Tsai and B.-B. Luo et al.
Optics and Lasers in Engineering 124 (2020) 105852
Fig. 1. Setup of single-pixel imaging.
2. Principle The setup of single-pixel imaging is illustrated in Fig. 1. The light beam is expanded to illuminate a SLM. In the m-th measurement (m ∈ 1 ∼ M), a binary pattern 𝑇𝑚 (𝑟⃗) is displayed on the SLM, and is projected to the plane of an object with irradiance transmittance 𝑂(𝑟⃗), where 𝑟⃗ denotes the transverse coordinates. The light transmitted through the object is collected by a lens and measured by a bucket photodetector. The measured signal is expressed as 𝑆𝑚 = 𝛼
∬
𝐿0 𝑇𝑚 (𝑟⃗)𝑂(𝑟⃗)𝑑 2 𝑟
(1)
where L0 stands for the uniform irradiance at the SLM plane, and 𝛼 is the gain factor of the detector. In Eq. (1), we have assumed that the magnification of the projecting lens is unity for the simplicity without loss of generality. In conventional GI, the retrieved image is obtained by [3] ⟨ ⟩ 𝐼(𝑟⃗) = (𝑆𝑚 − 𝑆̄ )(𝑇𝑚 (𝑟⃗) − 𝑇̄ (𝑟⃗)) (2) 1 ∑𝑀 where ⟨⋅⟩ = 𝑀 𝑖=1 ⋅ denotes the average operation, and 𝑆̄ = ⟨𝑆𝑚 ⟩, 𝑇̄ (𝑟⃗) = ⟨𝑇𝑚 (𝑟⃗)⟩. Assume that the pattern 𝑇𝑚 (𝑟⃗) contains P pixels with values of either zero (off) or one (on), then the SNR of the retrieved image is M/P, provided that the object is binary [4]. In other words, every measured signal provides the same contribution (i.e. 1/P in SNR) to the whole image. Since every measured signal contributes equally, numerous measurements must be made to improve the image quality. In our proposed method, we have a different concept of the image retrieval and adopt a new calculation strategy, whose flowchart is shown in Fig. 2. It is known that any measured signal Sm provides a unique understanding of the pixels selected by 𝑇𝑚 (𝑟⃗) (i.e. the “on” pixels) on the object. If in the n-th calculation, the retrieved image 𝐼𝑛 (𝑟⃗) (n is the index of calculation number) is the same as the object, then the difference between 𝛼 ∬ 𝐿0 𝑇𝑚 (𝑟⃗)𝐼𝑛 (𝑟⃗)𝑑 2 𝑟 and Sm must be zero. Otherwise, their difference, namely the error arose in 𝐼𝑛 (𝑟⃗), is compensated, resulting in a new image. The new image is expressed as 𝐼𝑛+1 (𝑟⃗) = 𝐼𝑛 (𝑟⃗) + 𝑇𝑚 (𝑟⃗)Δ+ 𝑚,𝑛
(3)
where Δ+ 𝑚,𝑛 is the compensation term, and is determined by Δ+ 𝑚,𝑛 =
𝑆𝑚 − 𝛼𝐿0 ∬ 𝑇𝑚 (𝑟⃗)𝐼𝑛 (𝑟⃗)𝑑 2 𝑟 ∬ 𝑇𝑚 (𝑟⃗)𝑑 2 𝑟
.
(4)
The numerator in Eq. (4) is for calculating the difference between measurement (Sm ) and the integral value obtained from 𝐼𝑛 (𝑟⃗), while the denominator is for taking average over all opened pixels. It should be noted that Sm provides not only the information of the “on” pixels of 𝑇𝑚 (𝑟⃗), but also the information of the “off ” pixels. If a complementary pattern of 𝑇𝑚 (𝑟⃗), which can be expressed as 1 − 𝑇𝑚 (𝑟⃗), is applied in the same experiment, then the measured power value will be 𝑆0 − 𝑆𝑚 , where 𝑆0 = 𝛼 ∬ 𝐿0 𝑂(𝑟⃗)𝑑 2 𝑟is the total power of the object under uniform illumination. Therefore, as S0 is taken in advance, we can know the measurement value under illumination with a complementary pattern without performing experimental measurement. Based on this idea and the concept similar as Eq. (4), the “off” pixels of mask 𝑇𝑚 (𝑟⃗) can be also compensated by another term expressed as [ ] (𝑆0 − 𝑆𝑚 ) − 𝛼𝐿0 ∬ 1 − 𝑇𝑚 (𝑟⃗) 𝐼𝑛 (𝑟⃗)𝑑 2 𝑟 Δ− = (5) [ ] 𝑚,𝑛 ∬ 1 − 𝑇𝑚 (𝑟⃗) 𝑑 2 𝑟 − Accordingly, the formula for calibrating both Δ+ 𝑚,𝑛 and Δ𝑚,𝑛 is expressed as [ ] 𝐼𝑛+1 (𝑟⃗) = 𝐼𝑛 (𝑟⃗) + 𝑇𝑚 (𝑟⃗)Δ+ ⃗) Δ− (6) 𝑚,𝑛 + 1 − 𝑇𝑚 (𝑟 𝑚,𝑛 .
In Eq. (6), a new image 𝐼𝑛+1 (𝑟⃗) is obtained from the previous image 𝐼𝑛 (𝑟⃗). This process can be iteratively implemented until a satisfactory image is obtained. In every iteration, we can also enforce all negative value in 𝐼𝑛+1 (𝑟⃗) to be zero because any image value cannot be negative. This process is not necessary but can enhance the efficiency of the algorithm. Since our proposed method applies a single measured signal to compensate the error arose in all pixels in the previous retrieved image, we call this method the iterative total-error compensation (ITEC) method. In ITEC, any measurement value Sm can be applied multiple times because the image is continuously altered in the error-compensation procedure. For example, for total M measurements, the first, the second, … and the M-th measurements are applied in the first, the second, … and the M-th calculations of error compensation in sequence. In the second round of compensation, the same set of measurement data is applied again to the (M + 1)-th to the 2M-th calculations of error compensation. This procedure can be repeated until a target metrics of image quality or the number of calculations is reached. Therefore, the compensation number N can be much larger than the measurement number M. Fig. 2. Flowchart of proposed image retrieval method.
J.-P. Liu, K.-C. Tsai and B.-B. Luo et al.
The computational complexity (one numerical addition or one multiplication) of GI [Eq. (2)] is 2P for each measurement calculation. By contrast, ITEC [Eq. (6)] is roughly 7P for each iteration, which is acceptable, provided ITEC can achieve much better image quality. On the other hand, the computational complexity of compressive-sensing-based algorithm is several orders of magnitude larger than that of GI as well as ITEC [19]. Therefore, ITEC is considered one of the most efficient algorithms for single-pixel imaging of random illuminations. 3. Simulation We have performed simulation to prove the ability of ITEC. The illumination patterns contains 80 × 60 pixels, and thus P is 4800. For the convenience of comparison, we adopt peak signal-to-noise ratio (PSNR) as the metrics of image quality here. PSNR is defined as [7] 𝑀𝐴𝑋 2 , (7) 𝑀𝑆𝐸 where MAX is the maximum value in the target image, and 𝑀𝑆𝐸 = 2 1 ∑ [𝐼(𝑟⃗) − 𝑂(𝑟⃗)] is the mean square error of the retrieved image. The 𝑃 results of a set of simulation are shown in Fig. 3. Fig. 3(a) is the image obtained by GI [Eq. (2)]. Even though the measurement number is as high as 10P (4.8 × 104 ), the PSNR is only 14.8 dB. In comparison, we have applied ITEC with (𝑀 , 𝑁 ) = (3600, 1.152 × 105 )to retrieve an image, as shown in Fig. 3(b). Note in this case the measurement number 𝑀 = 3600 is only 0.75P, that is the data is slightly compressed. The SNR of retrieved image is 17.4 dB. The image quality can be further improved by using a larger M and N. Two images obtained by ITEC at (𝑀 , 𝑁 ) = (4800, 1.152 × 105 ) and (𝑀 , 𝑁 ) = (4800, 4.8 × 106 ) are shown in Fig. 3(c) and (d), respectively. It can be found that the image quality can be significantly improved by applying a larger N. Fig. 3(e) shows the image obtained by ITEC at (𝑀 , 𝑁 ) = (9600, 1.152 × 105 ). Now the retrieved image is nearly the same as the original object, which is shown in Fig. 3(f). The simulation indicates that the image quality is more sensitive to the measurement number than the compensation number. To see the dependence of M and N, the PSNR at different combinations of (M, N) is illustrated in Fig. 4. Apparently, one can get a retrieved image with acceptable PSNR by using a greater number of measurements and less number of compensations, or by using less number of measurements and a lot of number of compensations. In ITEC, there is no limitation on the initial retrieved image 𝐼0 (𝑟⃗). That is, even a full black image can be an initial image to start ITEC. 𝑃 𝑆𝑁𝑅 = 10 log10
Optics and Lasers in Engineering 124 (2020) 105852
We also found that there is a threshold value of the measurement number in ITEC. If the measurement number is less than the threshold value, the image can be still retrieved. However, the SNR will approach a limited value even though more compensations are conducted. If the measurement number is larger than the threshold value, the PSNR of the retrieved image can be continuously improved by more compensations. In our simulations, the threshold value is about P/4 for binary object and P for grayscale object. The reason of the threshold value of measurement number is explained as follows. In ITEC, every calculation [Eq. (6)] compensates only the average value of all pixels, but not any individual pixel. Therefore, in a single compensation, the error of some pixels is reduced. but that of other pixels remains or increases. Because each mask 𝑇𝑚 (𝑟⃗) selects different pixels, each measurement can be applied to compensate different group of pixels. Nevertheless, if two pixels are always selected together or unselected together in a set of 𝑇𝑚 (𝑟⃗), then only the average value of the two pixels are compensated. There is no way to correct each of the two pixels. This effect is called pixel degeneracy. Although the probability of always selecting two pixels together is very low, it is possible that multiple pixels are frequently selected together in a set of 𝑇𝑚 (𝑟⃗), i.e. partial pixel degeneracy. Partial pixel degeneracy should be the reason that the measurement number must be larger than a threshold value. Finally, we have also investigated the influence of noise in ITEC. Various amount of white Gaussian noise was added to the raw data Sm to simulate a real noisy imaging environment, and the images were retrieved by ITEC. The resulting PSNR curves are shown in Fig. 5. The PSNR for noise-free data continuously grows, as what we have seen in Fig. 4. For the raw data with signal-to-noise ratio (SNR) of 20 dB, the PSNR of retrieved image still grows with increasing compensation number. However, the slope is not as large as that of noise-free case. For the raw data with SNR 15 dB, the PSNR reaches 28 dB at 2 × 106 compensation number but decreases afterward. Similarly, for the raw data with SNR 10 dB, the maximum PSNR is about 27 dB at 9 × 105 compensation number. These simulations indicate that ITEC can still work in practical noisy environment. However, if the SNR of raw data is less than 20 dB, the PSNR of retrieved image will quickly reach a maximum value because of the influence of noise. The downward trend of the PSNR after the maximum should result from the over compensation of the retrieved image. That is, when the image is good enough, the process of more compensation cannot reduce the noise in the image; instead, more noise from Sm is added to the image. 4. Experiment
Fig. 3. Simulation results of single-pixel imaging. (a) Image retrieved by GI. (b)–(e), images retrieved by ITEC at (M, N) of (3600, 1.152 × 105 ), (4800, 1.152 × 105 ), (4800, 4.8 × 106 ), and (9600, 1.152 × 105 ), respectively. (f) Original image.
ITEC has also been tested by using optically obtained data. A LED made by CREE (XM-L) with central wavelength 530 nm is applied as the light source. The SLM is a liquid crystal (LC) SLM made by Holoeye (LC-2002), and it contains 800 × 600 pixels with 32 𝜇m pixel pitch. The sampling pixel of measurement mask is represented by 10 × 10 SLM pixels. Therefore, the mask size is still 80 × 60 pixels. The detector is made by Thorlabs (PDA-100A) with an active area of 100 mm2 . The object is a transparency film, on which a photo is half-toning printed. Fig. 6(a) shows the image of the prepared transparency film obtained by a photo scanner. The actual size of the printed photo is 19.2 × 25.6 mm2 . Therefore, the printing dot size is much smaller than the sampling pixel size of our system, and thus the object can be regarded as a grayscale image. Fig. 6(b) shows the image retrieved by GI while the measurement number is 9600. It should be noted that the PSNR in experiment is not calculated using Fig. 6(a) as the target image because of the misalignment problem and the nonlinearity of photo scanner. Instead, a highquality image retrieved by ITEC at (𝑀 , 𝑁 ) = (2 × 104 , 2 × 105 ) is first calculated and applied as the target image. By using ITEC at (𝑀 , 𝑁 ) = (4800, 9600) [Fig. 6(c)], the image quality is apparently better than that obtained by GI. The quality of the retrieved image can be continuously improved by increasing N. Fig. 6(d) shows the image retrieved by ITEC at (𝑀 , 𝑁 ) = (4800, 1.152 × 105 ). By using a larger M, says 9600, the im-
J.-P. Liu, K.-C. Tsai and B.-B. Luo et al.
Optics and Lasers in Engineering 124 (2020) 105852
Fig. 4. PSNR as a function of measurement number (M) and compensation number (N). Fig. 5. PSNR as a function of compensation number (N) for raw data with different noise level.
age retrieved at the same N can be better. The images retrieved by ITEC at (𝑀 , 𝑁 ) = (9600, 9600)and (𝑀 , 𝑁 ) = (9600, 5.76 × 104 ) are shown in Fig. 6(e) and (f), respectively. The quality of image retrieved by ITEC is very good in comparison with that obtained by GI, but cannot be continuously improved. This result agrees with our simulation of raw data containing Gaussian noise. In addition, systematic error may degrade the image further. In our experiment, we used a LED as the source to alleviate the speckle noise. However, its spatially and temporally low coherence may result in nonuni-
form illumination and light leakage of the LC-SLM. These problems can be solved by using a digital micromirror device (DMD) and careful calibration. Finally, ITEC is sensitive to the measurement value of S0 , which is taken into account in Eq. (5). Unfortunately, it is hard to measure S0 at the same level as that of Sm . The high-spatial-frequency light and high-order diffractive light cannot be completely measured by the photodetector because of the limited size of active area of the single-pixel detector. Therefore, the measured Sm will be always smaller than the theoretical value. On the other hand, there is less high-spatial-frequency
J.-P. Liu, K.-C. Tsai and B.-B. Luo et al.
Optics and Lasers in Engineering 124 (2020) 105852
method is accurate and faster. The advantage of ITEC is that it applies random illuminations, which can be generated by scattering or other means without a SLM or an arrayed light source. Therefore, it provides more application flexibility of single-pixel imaging. In addition, ITEC also makes high quality single pixel imaging at any spectral range of electromagnetic wave possible. Funding Ministry of Science and Technology of Taiwan (106-2628-E-035002-MY3) References
Fig. 6. Results of optical experiments. (a) Object photo, (b) image retrieved by GI. (c)-(f) are images retrieved by ITEC at (M, N) of (4800, 9600), (4800, 1.152 × 105 ), (9600, 9600), and (9600, 5.76 × 104 ) , respectively.
component in S0 , and thus most of the transmitted light can be collected by the detector. To solve this problem, instead of directly measurement, we have assumed 𝑆0 ≡ 2𝑆̄ in our experiment. By this way, the influence of high-spatial-frequency light and high-order diffractive light is similar in both S0 and Sm , which promised the success of ITEC. 5. Conclusions In conclusion, in this paper we have proposed iterative total-error compensation (ITEC) for image retrieval of random-illumination-based single-pixel imaging. ITEC can be easily implemented and demands much less computational complexity than compressive sensing. In principle, the quality of image retrieved by ITEC can be limitless improved if the measurement number is larger than a threshold value and the SNR of raw data is larger than 20 dB. To our knowledge, no other method possesses this unique property. Therefore, either the measurement time or the computing time can be easily optimized in ITEC. Indeed, Hadamardpattern-based illumination patterns and Fourier-coefficient-based illumination patterns can realize high-quality imaging if all Hadamard components or Fourier components are measured. Although ITEC can be also applied to the image retrieval of single pixel imaging by Hadamardpattern-based illumination patterns or Fourier-coefficient-based illumination patterns, it is not recommended because the standard retrieval
[1] Sun B, Edgar MP, Bowman R, Vittert LE, Welsh S, Bowman A, Padgett MJ. 3D Computational imaging with single-pixel detectors. Science 2013;340:844–7. [2] Ota S, Horisaki R, Kawamura Y, Ugawa M, Sato I, Hashimoto K, Kamesawa R, Setoyama K, Yamaguchi S, Fujiu K, Waki K, Noji H. Ghost cytometry. Science 2018;360:1246–51. [3] Shapiro JH. Computational ghost imaging. Phys. Rev. A 2008;78:061802. [4] Ferri F, Magatti D, Lugiato LA, Gatti A. Differential ghost imaging. Phys. Rev. Lett. 2010;104:253603. [5] Sun B, Welsh SS, Edgar MP, Shapiro JH, Padgett MJ. Normalized ghost imaging. Opt. Express 2012;20:16892–901. [6] Martínez-León L, Clemente P, Mori Y, Climent V, Lancis J, Tajahuerce E. Single-pixel digital holography with phase-encoded illumination. Opt. Express 2017;25:4975– 4984. [7] Zhang Z, Wang X, Zheng G, Zhong J. Hadamard single-pixel imaging versus fourier single-pixel imaging. Opt. Express 2017;25:19619–39. [8] Zhang Z, Ma X, Zhong J. Single-pixel imaging by means of fourier spectrum acquisition. Nat. Commun. 2015;6:6225. [9] Zhang Z, Liu S, Peng J, Yao M, Zheng G, Zhong J. Simultaneous spatial, spectral, and 3D compressive imaging via efficient fourier single-pixel measurements. Optica 2018;5:315–19. [10] Salvador-Balaguer E, Latorre-Carmona P, Chabert C, Pla F, Lancis J, Tajahuerce E. Low-cost single-pixel 3D imaging by using an led array. Opt. Express 2018;26:15623–31. [11] Yao X-R, Yu W-K, Liu X-F, Li L-Z, Li M-F, Wu L-A, Zhai G-J. Iterative denoising of ghost imaging. Opt. Express 2014;22:24268–75. [12] Wang W, Hu X, Liu J, Zhang S, Suo J, Situ G. Gerchberg-Saxton-like ghost imaging. Opt. Express 2015;23:28416–22. [13] Katz O, Bromberg Y, Silberberg Y. Compressive ghost imaging. Appl. Phys. Lett. 2009;95:131110. [14] Huang H, Zhou C, Tian T, Liu D, Song L. High-quality compressive ghost imaging. Opt. Commun. 2018;412:60–5. [15] Liu X-F, Chen X-H, Yao X-R, Yu W-K, Zhai G-J, Wu L-A. Lensless ghost imaging with sunlight. Opt. Lett 2014;39:2314–17. [16] Zhang D-J, Tang Q, Wu T-F, Qiu H-C, Xu D-Q, Li H-G, Wang H-B, Xiong J, Wang K. Lensless ghost imaging of a phase object with pseudo-thermal light. Appl. Phys. Lett. 2014;104:121113. [17] Gong W. High-resolution pseudo-inverse ghost imaging. Photon. Res. 2015;3:234–7. [18] Choi Y, Yoon C, Kim M, Yang TD, Fang-Yen C, Dasari RR, Lee KJ, Choi W. Scanner-Free and wide-field endoscopic imaging by using a single multimode optical fiber. Phys. Rev. Lett. 2012;109:203901. [19] Bian L, Suo J, Dai Q, Chen F. Experimental comparison of single-pixel imaging algorithms. J. Opt. Soc. Am. A 2018;35:78–87.
Contents lists available at ScienceDirect
Optics and Lasers in Engineering journal homepage: www.elsevier.com/locate/optlaseng
High-quality image retrieval by iterative total-error compensation for single-pixel imaging of random illuminations Jung-Ping Liu a,∗, Kun-Chi Tsai a, Bo-Bing Luo a, Yoshio Hayasaki b a b
Department of Photonics, Feng Chia University, 100 Wenhwa Rd., Seatwen, Taichung 40724, Taiwan Center for Optical Research & Education (CORE), Utsunomiya University, 7-1-2 Yoto, Utsunomiya 321-8585, Japan
a b s t r a c t We proposed iterative total-error compensation (ITEC) for the image retrieval of single-pixel imaging using random illuminations and point light detection. ITEC is based on the compensation of the error of all pixels arose in the previous retrieved image. The merit of ITEC is that its computing complexity is relatively low, and there is no demand for the type of object. In addition, ITEC can work in the circumstance that a SLM or an arrayed light source is unavailable. Therefore, ITEC provides high flexibility on cost and complexity of single-pixel imaging system.
1. Introduction Single-pixel imaging applies structured-light illumination and a single-pixel detector to get the spatial information of an object. A twodimensional (2D) or three-dimensional (3D) image of the object can be retrieved by using different algorithms. Since single-pixel imaging does not apply an arrayed image sensor, it has application potential in circumstances that a high-resolution camera is unavailable. In addition, single-pixel imaging has been applied to 3D imaging [1] and cytometry [2]. Ghost imaging (GI) is one of the easiest methods among the single-pixel imaging techniques [3–5]. GI applies random illuminations in imaging, and the image of the object is retrieved by the correlation of power measurements and the corresponding illumination patterns. In conventional GI, the signal-to-noise ratio (SNR) of a retrieved image is linearly proportional to the number of measurements. In typical imaging of a binary object, the SNR will be only unity when the number of measurements is equal to the pixel number of the image [4]. This property not only limits the quality of retrieved image, but also prevents GI from being applied in practical applications. So far, many techniques have been proposed to deal with this problem. These methods can be classified into three. The first is in the concern of devising special illumination patterns, such as Hadamard-pattern-based illumination [6,7] or Fourier-component-based illumination [8,9], to reduce the statistical noise. These techniques indeed improve the quality of retrieved images significantly. However, they must be realized using a spatial light modulator (SLM) or an arrayed light source [10]. In the circumstance that a SLM or an arrayed light source is unavailable, the special illumination patterns cannot be generated. The second kind of single-pixel imaging is based on the use of algorithms, such as Gerchberg–Saxton (GS) algorithm, for image optimization over conventional GI image [11,12].
∗
These methods have two drawbacks. First these methods include both the conventional correlation calculation and optimization procedure. Therefore, the full computing time is further longer than that of conventional GI. Second, the quality of images retrieved by these methods is still positively proportional to the number of measurements. In other words, a lot of measurements are still demanded to achieve high quality imaging. Finally, compressive-sensing-based algorithm can reduce the demand of large measurement number [13,14]. However, it has a presupposition on the type of signal, and its computational complexity is extremely high. In this paper, we consider improving the image quality by using random illuminations. We proposed a much different strategy of image retrieval for single-pixel imaging. First, any power measurement is not regarded as the image signal directly. Instead, it is applied to correct the error arose in the previous retrieved image. Second, every measurement can be applied to correct the error in not only the open pixels, but also the closed pixels, which is the concept of totalerror compensation. Finally, each measurement value can be applied to correct the pixel error unlimited times, provided the previous retrieved image has been changed. Therefore, a high-quality image can be retrieved by our proposed method with moderate computing cost. Since the proposed method applies random-illuminations, which can be generated by scattering [13,15–17]or a multimode fiber [18], the proposed method provides high flexibility for random-illuminationbased single-pixel imaging system. The remaining of the paper is organized as follows. In Section 2, the principle of GI and our proposed method, ITEC, will be explained. In Section 3, simulations are conducted to verify the proposed method. Experimental results are provided and discussed in Section 4. Finally, concluding remarks are made in Section 5.
Corresponding author. E-mail address: [email protected] (J.-P. Liu).
https://doi.org/10.1016/j.optlaseng.2019.105852 Received 26 March 2019; Received in revised form 7 August 2019; Accepted 5 September 2019 0143-8166/© 2019 Published by Elsevier Ltd.
J.-P. Liu, K.-C. Tsai and B.-B. Luo et al.
Optics and Lasers in Engineering 124 (2020) 105852
Fig. 1. Setup of single-pixel imaging.
2. Principle The setup of single-pixel imaging is illustrated in Fig. 1. The light beam is expanded to illuminate a SLM. In the m-th measurement (m ∈ 1 ∼ M), a binary pattern 𝑇𝑚 (𝑟⃗) is displayed on the SLM, and is projected to the plane of an object with irradiance transmittance 𝑂(𝑟⃗), where 𝑟⃗ denotes the transverse coordinates. The light transmitted through the object is collected by a lens and measured by a bucket photodetector. The measured signal is expressed as 𝑆𝑚 = 𝛼
∬
𝐿0 𝑇𝑚 (𝑟⃗)𝑂(𝑟⃗)𝑑 2 𝑟
(1)
where L0 stands for the uniform irradiance at the SLM plane, and 𝛼 is the gain factor of the detector. In Eq. (1), we have assumed that the magnification of the projecting lens is unity for the simplicity without loss of generality. In conventional GI, the retrieved image is obtained by [3] ⟨ ⟩ 𝐼(𝑟⃗) = (𝑆𝑚 − 𝑆̄ )(𝑇𝑚 (𝑟⃗) − 𝑇̄ (𝑟⃗)) (2) 1 ∑𝑀 where ⟨⋅⟩ = 𝑀 𝑖=1 ⋅ denotes the average operation, and 𝑆̄ = ⟨𝑆𝑚 ⟩, 𝑇̄ (𝑟⃗) = ⟨𝑇𝑚 (𝑟⃗)⟩. Assume that the pattern 𝑇𝑚 (𝑟⃗) contains P pixels with values of either zero (off) or one (on), then the SNR of the retrieved image is M/P, provided that the object is binary [4]. In other words, every measured signal provides the same contribution (i.e. 1/P in SNR) to the whole image. Since every measured signal contributes equally, numerous measurements must be made to improve the image quality. In our proposed method, we have a different concept of the image retrieval and adopt a new calculation strategy, whose flowchart is shown in Fig. 2. It is known that any measured signal Sm provides a unique understanding of the pixels selected by 𝑇𝑚 (𝑟⃗) (i.e. the “on” pixels) on the object. If in the n-th calculation, the retrieved image 𝐼𝑛 (𝑟⃗) (n is the index of calculation number) is the same as the object, then the difference between 𝛼 ∬ 𝐿0 𝑇𝑚 (𝑟⃗)𝐼𝑛 (𝑟⃗)𝑑 2 𝑟 and Sm must be zero. Otherwise, their difference, namely the error arose in 𝐼𝑛 (𝑟⃗), is compensated, resulting in a new image. The new image is expressed as 𝐼𝑛+1 (𝑟⃗) = 𝐼𝑛 (𝑟⃗) + 𝑇𝑚 (𝑟⃗)Δ+ 𝑚,𝑛
(3)
where Δ+ 𝑚,𝑛 is the compensation term, and is determined by Δ+ 𝑚,𝑛 =
𝑆𝑚 − 𝛼𝐿0 ∬ 𝑇𝑚 (𝑟⃗)𝐼𝑛 (𝑟⃗)𝑑 2 𝑟 ∬ 𝑇𝑚 (𝑟⃗)𝑑 2 𝑟
.
(4)
The numerator in Eq. (4) is for calculating the difference between measurement (Sm ) and the integral value obtained from 𝐼𝑛 (𝑟⃗), while the denominator is for taking average over all opened pixels. It should be noted that Sm provides not only the information of the “on” pixels of 𝑇𝑚 (𝑟⃗), but also the information of the “off ” pixels. If a complementary pattern of 𝑇𝑚 (𝑟⃗), which can be expressed as 1 − 𝑇𝑚 (𝑟⃗), is applied in the same experiment, then the measured power value will be 𝑆0 − 𝑆𝑚 , where 𝑆0 = 𝛼 ∬ 𝐿0 𝑂(𝑟⃗)𝑑 2 𝑟is the total power of the object under uniform illumination. Therefore, as S0 is taken in advance, we can know the measurement value under illumination with a complementary pattern without performing experimental measurement. Based on this idea and the concept similar as Eq. (4), the “off” pixels of mask 𝑇𝑚 (𝑟⃗) can be also compensated by another term expressed as [ ] (𝑆0 − 𝑆𝑚 ) − 𝛼𝐿0 ∬ 1 − 𝑇𝑚 (𝑟⃗) 𝐼𝑛 (𝑟⃗)𝑑 2 𝑟 Δ− = (5) [ ] 𝑚,𝑛 ∬ 1 − 𝑇𝑚 (𝑟⃗) 𝑑 2 𝑟 − Accordingly, the formula for calibrating both Δ+ 𝑚,𝑛 and Δ𝑚,𝑛 is expressed as [ ] 𝐼𝑛+1 (𝑟⃗) = 𝐼𝑛 (𝑟⃗) + 𝑇𝑚 (𝑟⃗)Δ+ ⃗) Δ− (6) 𝑚,𝑛 + 1 − 𝑇𝑚 (𝑟 𝑚,𝑛 .
In Eq. (6), a new image 𝐼𝑛+1 (𝑟⃗) is obtained from the previous image 𝐼𝑛 (𝑟⃗). This process can be iteratively implemented until a satisfactory image is obtained. In every iteration, we can also enforce all negative value in 𝐼𝑛+1 (𝑟⃗) to be zero because any image value cannot be negative. This process is not necessary but can enhance the efficiency of the algorithm. Since our proposed method applies a single measured signal to compensate the error arose in all pixels in the previous retrieved image, we call this method the iterative total-error compensation (ITEC) method. In ITEC, any measurement value Sm can be applied multiple times because the image is continuously altered in the error-compensation procedure. For example, for total M measurements, the first, the second, … and the M-th measurements are applied in the first, the second, … and the M-th calculations of error compensation in sequence. In the second round of compensation, the same set of measurement data is applied again to the (M + 1)-th to the 2M-th calculations of error compensation. This procedure can be repeated until a target metrics of image quality or the number of calculations is reached. Therefore, the compensation number N can be much larger than the measurement number M. Fig. 2. Flowchart of proposed image retrieval method.
J.-P. Liu, K.-C. Tsai and B.-B. Luo et al.
The computational complexity (one numerical addition or one multiplication) of GI [Eq. (2)] is 2P for each measurement calculation. By contrast, ITEC [Eq. (6)] is roughly 7P for each iteration, which is acceptable, provided ITEC can achieve much better image quality. On the other hand, the computational complexity of compressive-sensing-based algorithm is several orders of magnitude larger than that of GI as well as ITEC [19]. Therefore, ITEC is considered one of the most efficient algorithms for single-pixel imaging of random illuminations. 3. Simulation We have performed simulation to prove the ability of ITEC. The illumination patterns contains 80 × 60 pixels, and thus P is 4800. For the convenience of comparison, we adopt peak signal-to-noise ratio (PSNR) as the metrics of image quality here. PSNR is defined as [7] 𝑀𝐴𝑋 2 , (7) 𝑀𝑆𝐸 where MAX is the maximum value in the target image, and 𝑀𝑆𝐸 = 2 1 ∑ [𝐼(𝑟⃗) − 𝑂(𝑟⃗)] is the mean square error of the retrieved image. The 𝑃 results of a set of simulation are shown in Fig. 3. Fig. 3(a) is the image obtained by GI [Eq. (2)]. Even though the measurement number is as high as 10P (4.8 × 104 ), the PSNR is only 14.8 dB. In comparison, we have applied ITEC with (𝑀 , 𝑁 ) = (3600, 1.152 × 105 )to retrieve an image, as shown in Fig. 3(b). Note in this case the measurement number 𝑀 = 3600 is only 0.75P, that is the data is slightly compressed. The SNR of retrieved image is 17.4 dB. The image quality can be further improved by using a larger M and N. Two images obtained by ITEC at (𝑀 , 𝑁 ) = (4800, 1.152 × 105 ) and (𝑀 , 𝑁 ) = (4800, 4.8 × 106 ) are shown in Fig. 3(c) and (d), respectively. It can be found that the image quality can be significantly improved by applying a larger N. Fig. 3(e) shows the image obtained by ITEC at (𝑀 , 𝑁 ) = (9600, 1.152 × 105 ). Now the retrieved image is nearly the same as the original object, which is shown in Fig. 3(f). The simulation indicates that the image quality is more sensitive to the measurement number than the compensation number. To see the dependence of M and N, the PSNR at different combinations of (M, N) is illustrated in Fig. 4. Apparently, one can get a retrieved image with acceptable PSNR by using a greater number of measurements and less number of compensations, or by using less number of measurements and a lot of number of compensations. In ITEC, there is no limitation on the initial retrieved image 𝐼0 (𝑟⃗). That is, even a full black image can be an initial image to start ITEC. 𝑃 𝑆𝑁𝑅 = 10 log10
Optics and Lasers in Engineering 124 (2020) 105852
We also found that there is a threshold value of the measurement number in ITEC. If the measurement number is less than the threshold value, the image can be still retrieved. However, the SNR will approach a limited value even though more compensations are conducted. If the measurement number is larger than the threshold value, the PSNR of the retrieved image can be continuously improved by more compensations. In our simulations, the threshold value is about P/4 for binary object and P for grayscale object. The reason of the threshold value of measurement number is explained as follows. In ITEC, every calculation [Eq. (6)] compensates only the average value of all pixels, but not any individual pixel. Therefore, in a single compensation, the error of some pixels is reduced. but that of other pixels remains or increases. Because each mask 𝑇𝑚 (𝑟⃗) selects different pixels, each measurement can be applied to compensate different group of pixels. Nevertheless, if two pixels are always selected together or unselected together in a set of 𝑇𝑚 (𝑟⃗), then only the average value of the two pixels are compensated. There is no way to correct each of the two pixels. This effect is called pixel degeneracy. Although the probability of always selecting two pixels together is very low, it is possible that multiple pixels are frequently selected together in a set of 𝑇𝑚 (𝑟⃗), i.e. partial pixel degeneracy. Partial pixel degeneracy should be the reason that the measurement number must be larger than a threshold value. Finally, we have also investigated the influence of noise in ITEC. Various amount of white Gaussian noise was added to the raw data Sm to simulate a real noisy imaging environment, and the images were retrieved by ITEC. The resulting PSNR curves are shown in Fig. 5. The PSNR for noise-free data continuously grows, as what we have seen in Fig. 4. For the raw data with signal-to-noise ratio (SNR) of 20 dB, the PSNR of retrieved image still grows with increasing compensation number. However, the slope is not as large as that of noise-free case. For the raw data with SNR 15 dB, the PSNR reaches 28 dB at 2 × 106 compensation number but decreases afterward. Similarly, for the raw data with SNR 10 dB, the maximum PSNR is about 27 dB at 9 × 105 compensation number. These simulations indicate that ITEC can still work in practical noisy environment. However, if the SNR of raw data is less than 20 dB, the PSNR of retrieved image will quickly reach a maximum value because of the influence of noise. The downward trend of the PSNR after the maximum should result from the over compensation of the retrieved image. That is, when the image is good enough, the process of more compensation cannot reduce the noise in the image; instead, more noise from Sm is added to the image. 4. Experiment
Fig. 3. Simulation results of single-pixel imaging. (a) Image retrieved by GI. (b)–(e), images retrieved by ITEC at (M, N) of (3600, 1.152 × 105 ), (4800, 1.152 × 105 ), (4800, 4.8 × 106 ), and (9600, 1.152 × 105 ), respectively. (f) Original image.
ITEC has also been tested by using optically obtained data. A LED made by CREE (XM-L) with central wavelength 530 nm is applied as the light source. The SLM is a liquid crystal (LC) SLM made by Holoeye (LC-2002), and it contains 800 × 600 pixels with 32 𝜇m pixel pitch. The sampling pixel of measurement mask is represented by 10 × 10 SLM pixels. Therefore, the mask size is still 80 × 60 pixels. The detector is made by Thorlabs (PDA-100A) with an active area of 100 mm2 . The object is a transparency film, on which a photo is half-toning printed. Fig. 6(a) shows the image of the prepared transparency film obtained by a photo scanner. The actual size of the printed photo is 19.2 × 25.6 mm2 . Therefore, the printing dot size is much smaller than the sampling pixel size of our system, and thus the object can be regarded as a grayscale image. Fig. 6(b) shows the image retrieved by GI while the measurement number is 9600. It should be noted that the PSNR in experiment is not calculated using Fig. 6(a) as the target image because of the misalignment problem and the nonlinearity of photo scanner. Instead, a highquality image retrieved by ITEC at (𝑀 , 𝑁 ) = (2 × 104 , 2 × 105 ) is first calculated and applied as the target image. By using ITEC at (𝑀 , 𝑁 ) = (4800, 9600) [Fig. 6(c)], the image quality is apparently better than that obtained by GI. The quality of the retrieved image can be continuously improved by increasing N. Fig. 6(d) shows the image retrieved by ITEC at (𝑀 , 𝑁 ) = (4800, 1.152 × 105 ). By using a larger M, says 9600, the im-
J.-P. Liu, K.-C. Tsai and B.-B. Luo et al.
Optics and Lasers in Engineering 124 (2020) 105852
Fig. 4. PSNR as a function of measurement number (M) and compensation number (N). Fig. 5. PSNR as a function of compensation number (N) for raw data with different noise level.
age retrieved at the same N can be better. The images retrieved by ITEC at (𝑀 , 𝑁 ) = (9600, 9600)and (𝑀 , 𝑁 ) = (9600, 5.76 × 104 ) are shown in Fig. 6(e) and (f), respectively. The quality of image retrieved by ITEC is very good in comparison with that obtained by GI, but cannot be continuously improved. This result agrees with our simulation of raw data containing Gaussian noise. In addition, systematic error may degrade the image further. In our experiment, we used a LED as the source to alleviate the speckle noise. However, its spatially and temporally low coherence may result in nonuni-
form illumination and light leakage of the LC-SLM. These problems can be solved by using a digital micromirror device (DMD) and careful calibration. Finally, ITEC is sensitive to the measurement value of S0 , which is taken into account in Eq. (5). Unfortunately, it is hard to measure S0 at the same level as that of Sm . The high-spatial-frequency light and high-order diffractive light cannot be completely measured by the photodetector because of the limited size of active area of the single-pixel detector. Therefore, the measured Sm will be always smaller than the theoretical value. On the other hand, there is less high-spatial-frequency
J.-P. Liu, K.-C. Tsai and B.-B. Luo et al.
Optics and Lasers in Engineering 124 (2020) 105852
method is accurate and faster. The advantage of ITEC is that it applies random illuminations, which can be generated by scattering or other means without a SLM or an arrayed light source. Therefore, it provides more application flexibility of single-pixel imaging. In addition, ITEC also makes high quality single pixel imaging at any spectral range of electromagnetic wave possible. Funding Ministry of Science and Technology of Taiwan (106-2628-E-035002-MY3) References
Fig. 6. Results of optical experiments. (a) Object photo, (b) image retrieved by GI. (c)-(f) are images retrieved by ITEC at (M, N) of (4800, 9600), (4800, 1.152 × 105 ), (9600, 9600), and (9600, 5.76 × 104 ) , respectively.
component in S0 , and thus most of the transmitted light can be collected by the detector. To solve this problem, instead of directly measurement, we have assumed 𝑆0 ≡ 2𝑆̄ in our experiment. By this way, the influence of high-spatial-frequency light and high-order diffractive light is similar in both S0 and Sm , which promised the success of ITEC. 5. Conclusions In conclusion, in this paper we have proposed iterative total-error compensation (ITEC) for image retrieval of random-illumination-based single-pixel imaging. ITEC can be easily implemented and demands much less computational complexity than compressive sensing. In principle, the quality of image retrieved by ITEC can be limitless improved if the measurement number is larger than a threshold value and the SNR of raw data is larger than 20 dB. To our knowledge, no other method possesses this unique property. Therefore, either the measurement time or the computing time can be easily optimized in ITEC. Indeed, Hadamardpattern-based illumination patterns and Fourier-coefficient-based illumination patterns can realize high-quality imaging if all Hadamard components or Fourier components are measured. Although ITEC can be also applied to the image retrieval of single pixel imaging by Hadamardpattern-based illumination patterns or Fourier-coefficient-based illumination patterns, it is not recommended because the standard retrieval
[1] Sun B, Edgar MP, Bowman R, Vittert LE, Welsh S, Bowman A, Padgett MJ. 3D Computational imaging with single-pixel detectors. Science 2013;340:844–7. [2] Ota S, Horisaki R, Kawamura Y, Ugawa M, Sato I, Hashimoto K, Kamesawa R, Setoyama K, Yamaguchi S, Fujiu K, Waki K, Noji H. Ghost cytometry. Science 2018;360:1246–51. [3] Shapiro JH. Computational ghost imaging. Phys. Rev. A 2008;78:061802. [4] Ferri F, Magatti D, Lugiato LA, Gatti A. Differential ghost imaging. Phys. Rev. Lett. 2010;104:253603. [5] Sun B, Welsh SS, Edgar MP, Shapiro JH, Padgett MJ. Normalized ghost imaging. Opt. Express 2012;20:16892–901. [6] Martínez-León L, Clemente P, Mori Y, Climent V, Lancis J, Tajahuerce E. Single-pixel digital holography with phase-encoded illumination. Opt. Express 2017;25:4975– 4984. [7] Zhang Z, Wang X, Zheng G, Zhong J. Hadamard single-pixel imaging versus fourier single-pixel imaging. Opt. Express 2017;25:19619–39. [8] Zhang Z, Ma X, Zhong J. Single-pixel imaging by means of fourier spectrum acquisition. Nat. Commun. 2015;6:6225. [9] Zhang Z, Liu S, Peng J, Yao M, Zheng G, Zhong J. Simultaneous spatial, spectral, and 3D compressive imaging via efficient fourier single-pixel measurements. Optica 2018;5:315–19. [10] Salvador-Balaguer E, Latorre-Carmona P, Chabert C, Pla F, Lancis J, Tajahuerce E. Low-cost single-pixel 3D imaging by using an led array. Opt. Express 2018;26:15623–31. [11] Yao X-R, Yu W-K, Liu X-F, Li L-Z, Li M-F, Wu L-A, Zhai G-J. Iterative denoising of ghost imaging. Opt. Express 2014;22:24268–75. [12] Wang W, Hu X, Liu J, Zhang S, Suo J, Situ G. Gerchberg-Saxton-like ghost imaging. Opt. Express 2015;23:28416–22. [13] Katz O, Bromberg Y, Silberberg Y. Compressive ghost imaging. Appl. Phys. Lett. 2009;95:131110. [14] Huang H, Zhou C, Tian T, Liu D, Song L. High-quality compressive ghost imaging. Opt. Commun. 2018;412:60–5. [15] Liu X-F, Chen X-H, Yao X-R, Yu W-K, Zhai G-J, Wu L-A. Lensless ghost imaging with sunlight. Opt. Lett 2014;39:2314–17. [16] Zhang D-J, Tang Q, Wu T-F, Qiu H-C, Xu D-Q, Li H-G, Wang H-B, Xiong J, Wang K. Lensless ghost imaging of a phase object with pseudo-thermal light. Appl. Phys. Lett. 2014;104:121113. [17] Gong W. High-resolution pseudo-inverse ghost imaging. Photon. Res. 2015;3:234–7. [18] Choi Y, Yoon C, Kim M, Yang TD, Fang-Yen C, Dasari RR, Lee KJ, Choi W. Scanner-Free and wide-field endoscopic imaging by using a single multimode optical fiber. Phys. Rev. Lett. 2012;109:203901. [19] Bian L, Suo J, Dai Q, Chen F. Experimental comparison of single-pixel imaging algorithms. J. Opt. Soc. Am. A 2018;35:78–87.