- Email: [email protected]
Resolution enhancement of confocal fluorescence microscopy via two illumination beams
Optics and Lasers in Engineering 122 (2019) 8–13
Contents lists available at ScienceDirect
Optics and Lasers in Engineering journal homepage: www.elsevier.com/locate/optlaseng
Resolution enhancement of confocal fluorescence microscopy via two illumination beams Vannhu Le a,c,+, Xiaona Wang a,+, Cuifang Kuang a,b,∗, Xu Liu a,b a
State Key Laboratory of Modern Optical Instrumentation, College of Optical Science and Engineering, Zhejiang University, Hangzhou 310027, China Collaborative Innovation Center of Extreme Optics, Shanxi University, Taiyuan 030006, China c Department of Optical Engineering, Le Quy Don Technical University, Hanoi, Vietnam b
a r t i c l e
i n f o
Keywords: Confocal fluorescence microscopy Superresolution Digital processing
a b s t r a c t Confocal fluorescence microscopy is an effective imaging technique, but its resolution is limited by the diffractionlimit. Fluorescence emission difference (FED) method is a useful way to improve the resolution of confocal fluorescence microscopy, but the negative values generated during subtraction process might cause loss of valid information. In this paper, we propose one effective method to enhance the resolution of confocal fluorescence microscopy without generating significant negative values. The proposed method combines digital processing and FED method, obtaining the final images with higher resolution and less information loss.
1. Introduction Confocal scanning microscopy is a routine tool in the life sciences. This imaging system can be used to improve resolution of conventional √ microscopy by a factor of 2 [1, 2], and could acquire high-resolution optical images with a certain depth [3]. As a result, confocal scanning microscopy has been used widely in three-dimensional specimen analysis. However, its spatial resolution is restricted to ∼200 nm due to the diffraction limit under common experimental conditions [4]. The demand for higher spatial resolution in optical microscopy has promoted the development of novel super-resolution microscopy, which is capable of breaking the diffraction limit. Multi-beam combination is one of the most common ways to achieve subcellular images and these techniques have developed rapidly, such as stimulated emission depletion (STED) microscopy [5, 6], ground-state depletion (GSD) microscopy, [7–9], reversible saturable/switchable optical transitions (RESOLFT) [10] and so on. STED is a two-beam microscopy that can improve the resolution of conventional fluorescence microscopy significantly, with a Gaussian beam exciting fluorescence, and a donut beam depleting surrounding fluorescence. The spot size can be reduced by matching the Gaussian pumping and donut depletion beams, so that the spatial resolution can be improved. It is a relatively fast way to achieve super-resolution and requires no data post processing. GSD is another two-beam super-resolution microscopy with a similar principle but different time sequences. This method excites the sample with relatively low-power
continuous wave, so that the photo-bleaching and photo-damage are slightly reduced. RESOLFT is a three-beam super-resolution microscopy. In RESOLFT, a Gaussian beam is used to activate the sample, a donut beam turns off the activation of the sample, and a second Gaussian beam excites the fluorophore that is still active. Although these super-resolution microscopy systems mentioned above have been commercially available and they are attractive for biological imaging, the application of these systems is still limited, because of the intricate optical system, special specimen preparation, expensive instrument, higher power source, time-consuming data processing, and high photo-damage. Fluorescence emission difference (FED) method [11–13] provides a new possibility to improve spatial resolution of confocal microscopy. In FED, the sample is illuminated by a solid beam and donut beam, respectively, to obtain two images, called solid image and donut image. The final image is obtained by subtracting the acquired solid image and the donut image. FED method has the following advantages. First, its calculation is simple since we only need to perform subtraction. Second, the cost of FED system is low, and the required beam paths can be assembled easily in the laboratory. Third, the required laser power of FED method is lower than that of STED and GSD, which minimizes photo-damage during the imaging process. However, negative intensity values are inevitably generated during subtraction processing [14]. Although these negative values are zeroed in the final result, they would still lead to information loss. There is a trade-off between the negative values and resolution: the higher the resolution, the more serious the
∗ Corresponding author at: State Key Laboratory of Modern Optical Instrumentation, College of Optical Science and Engineering, Zhejiang University, Room 418, Academic Building #3, #38Zheda Road,Xihu District, Hangzhou 310027, China. E-mail address: [email protected] (C. Kuang). + Authors have contributed equally to this work.
https://doi.org/10.1016/j.optlaseng.2019.05.018 Received 11 January 2019; Received in revised form 19 April 2019; Accepted 17 May 2019 0143-8166/© 2019 Published by Elsevier Ltd.
V. Le, X. Wang and C. Kuang et al.
Optics and Lasers in Engineering 122 (2019) 8–13
Fig. 1. Optical schemes via two illumination beams in confocal fluorescence microscopy.
effect of negative values [15]. To suppress the generation of negative values without reducing the resolution, Researchers have made great efforts and achieved some results [16, 17]. However, the negative values on the final image is still a problem in these methods. In this article, we proposed a new method combining the digital processing and FED methods, to obtain images with high resolution and lower information loss. The proposed method includes two steps. First, we obtain the restored image with higher resolution by digital processing. Then, remove the background of the restored image by introducing FED method. After these two steps, we can achieve high quality images.
Fig. 2. Model of the proposed method.
The procedure of the proposed method is shown in Fig. 2. This method includes two main parts: digital processing and FED method. Firstly, the digital processing is applied to restore the high-resolution image from two original images excited by solid and donut PSFs. Then, FED method is introduced to remove the background in restored image. This is the proposed method to get a high quality final image.
2. Method 2.1. Raw image data The raw image data acquired by the proposed method is the same as that of the FED method: a solid image excited by solid beam and a donut image excited by donut beam. Similar to the FED method, the donut beam is obtained by phase modulation of 0–2𝜋 vortex pattern. In Fig. 1, we indicate two feasible imaging schemes for the proposed method. In Fig. 1(a), there are two illumination paths: one path is solid beam, the other path is donut one. The donut PSF is modulated by 0–2𝜋 vortex phase mask. The two illumination paths should be adjust carefully, to ensure that the solid and donut spots are in the same position on the sample. In Fig. 1(b), a spatial light modulator (SLM) is added to the illumination path to control phase modulation. The phase pattern on the SLM is switched between 0 and 0–2𝜋 vortex phase mask. When the pattern on the SLM is set to 0, the illumination beam is solid; when the pattern on the SLM is set to 0–2𝜋 vortex phase mask, the illumination beam is donut. In the detection path, there is one pinhole, working as the spatial filter. The illumination part shown in Fig. 1(b) is simpler than that in Fig. 1(a). Besides, there is no need to adjust the solid and donut beam to ensure the spot overlap on the sample. So, we use the SLM to modulate the illumination beam. The two images excited by solid and donut PSFs can be presented by, 𝐼𝑠 (𝑥, 𝑦) = 𝑜(𝑥, 𝑦) ⊗ 𝑃 𝑆 𝐹𝑠 (𝑥, 𝑦)
(1)
𝐼𝑑 (𝑥, 𝑦) = 𝑜(𝑥, 𝑦) ⊗ 𝑃 𝑆 𝐹𝑑 (𝑥, 𝑦)
(2)
2.2. Digital processing The two raw images excited by different two PSFs are digitally processed to reconstruct a high-resolution image. In this article, we use the blind post processing to achieve the high-resolution image. As shown in Ref. [18], the blind post processing can correct the effect of the optical aberrations. In this article, we use the Richardson-Lucy deconvolution to achieve the blind post processing, and the iterative formula can be presented by, ( ) 𝐼 𝑡 𝑜𝑡+1 = 𝑜𝑡 × ⊗ 𝑃 𝑆𝐹 (3) 𝑟 𝑜𝑡 ⊗ 𝑃 𝑆 𝐹 𝑡 ( ) 𝐼 𝑃 𝑆 𝐹 𝑡+1 = 𝑃 𝑆 𝐹 𝑡 × ⊗ 𝑜𝑡𝑟 (4) 𝑜𝑡+1 ⊗ 𝑃 𝑆 𝐹 𝑡 where o is the object; or is the flipped object; I is the input image; PSF is the point spread function of illumination beam; PSFr is the flipped point spread function; t is the number of iterations; ⊗ is the convolution operator. Since there are two raw images, the iterative process can be presented by, [ )] ( 𝐼𝑠 𝑡 𝑜𝑡𝑠+1 = 𝑛𝑜𝑟𝑚 𝑜𝑡 × ⊗ 𝑃 𝑆𝐹 (5) 𝑠𝑟 𝑜𝑡 ⊗ 𝑃 𝑆𝐹𝑠𝑡 [ )] ( 𝐼𝑠 𝑃 𝑆𝐹𝑠𝑡+1 = 𝑛𝑜𝑟𝑚 𝑃 𝑆𝐹𝑠𝑡 × ⊗ 𝑜𝑡𝑟 (6) 𝑜𝑡+1 ⊗ 𝑃 𝑆𝐹𝑠𝑡 ( [ )] 𝐼𝑑 𝑡 𝑜𝑡𝑑+1 = 𝑛𝑜𝑟𝑚 𝑜𝑡 × ⊗ 𝑃 𝑆𝐹 (7) 𝑑𝑟 𝑜𝑡 ⊗ 𝑃 𝑆𝐹𝑑𝑡
where o is the object; PSFs , PSFd are the point spread functions of solid beam and donut beams in the cross-section perpendicular to the optical axis; Is , Id are the images excited by solid and donut PSFs, respectively; ⊗ is the convolution operator. 9
V. Le, X. Wang and C. Kuang et al.
Optics and Lasers in Engineering 122 (2019) 8–13
Fig. 3. The simulation results of spoke-like sample with the size of 10𝜆 × 10𝜆. (a) The spoke-like sample. (b) The imaging result of conventional confocal fluorescence microscopy. (c) The imaging result imaged by donut PSF. (d) The values of evaluation function C(𝛼) of the spoke-like sample depending on different 𝛼 values. (e) The imaging result of proposed method in this article, and the subtraction factor is set to 0.06. The size of the images is 10𝜆 × 10𝜆.
[ 𝑃 𝑆𝐹𝑑𝑡+1 = 𝑛𝑜𝑟𝑚 𝑃 𝑆𝐹𝑑𝑡 ×
(
𝐼𝑑
𝑜𝑡+1 ⊗ 𝑃 𝑆𝐹𝑑𝑡
( ) 𝑜𝑡+1 = 𝑛𝑜𝑟𝑚 𝑜𝑡𝑠+1 + 𝑜𝑡𝑑+1
)] ⊗ 𝑜𝑡𝑟
}|( ) ∑| { |𝐹 𝑇 𝑃 𝑆 𝐹𝑟𝑒𝑠 − 𝛼 ⋅ 𝑃 𝑆 𝐹𝑑 | 𝑟∕𝑟𝑚𝑎𝑥 | | 𝐶 (𝛼) = }| ∑| { |𝐹 𝑇 𝑃 𝑆 𝐹𝑟𝑒𝑠 − 𝛼 ⋅ 𝑃 𝑆 𝐹𝑑 | | |
(8)
(9)
(11)
There is also background in the restored image. To remove the background, we can subtract the restored image and donut image, which can be presented by,
where Ʃ represents the summation operation; FT represents the Fourier transform; PSFres is the restored solid point spread function (PSF) after the digital processing; PSFd is the donut PSF; 𝛼 is the subtraction 2 2 )1∕2 = factor; r is the polar radius of frequency space; 𝑟𝑚𝑎𝑥 = (𝜉𝑚𝑎𝑥 + 𝜂𝑚𝑎𝑥 √ 2 2 1∕(2𝛿𝜉 ) + 1∕(2𝜂𝜉 ) , and 𝛿 𝜉 , 𝜂 𝜉 represent the length and width of the pixel, respectively. In the following simulation and experiment, the subtraction factor is calculated by this function. As the calculation results show in the following simulation and experiments, the subtraction factor in our proposed method is greatly reduced. Therefore, the negative value generated in the image is also reduced in the subtraction process. Then, the final image is obtained by zeroing the negative value of the subtraction result. Because the negative values in the subtraction result are reduced, the zeroing process would not cause as much information loss as the conventional FED method.
𝐼 (𝑥, 𝑦) = 𝐼𝑟𝑒𝑠 (𝑥, 𝑦) − 𝛼 ⋅ 𝐼𝑑 (𝑥, 𝑦)
3. Simulation results
where o is the object restored from the two raw images; or is the flipped o; os is the object restored from the raw image excited by solid PSF; od is the object restored from the raw image excited by donut PSF; Is is the raw image excited by solid PSF; Id is the raw image excited by donut PSF; PSFs is the point spread function of solid beam; PSFsr is the flipped PSFs ; PSFd is the point spread function of donut beam; PSFdr is the flipped PSFd ; norm() represents normalized calculation; t is the number of iterations; ⊗ is the convolution operator. 2.3. FED method
(10)
where Ires is the restored image, after the digital processing; Id is the image excited by donut PSF; 𝛼 is the subtraction factor. The subtraction factor in above formula is much smaller than that in conventional FED method, so the negative values in the subtraction result are greatly reduced. To obtain an optimal subtraction factor, we analysis the imaging results of different subtraction factors with an evaluation function [19]:
In order to show the effectiveness of the proposed method, the simulation results are presented in this section. In the simulation, the numerical aperture (NA) of objective was set to 1.49, and wavelength was set to 640 nm, which were the parameters used in experiments. As is shown in Fig. 3(a), the sample is a spoke-like sample with the size of 10𝜆 × 10𝜆. The values of C(𝛼) are shown in Fig. 3(d), and the peak value of C(𝛼) is 10
V. Le, X. Wang and C. Kuang et al.
Optics and Lasers in Engineering 122 (2019) 8–13
Fig. 4. The simulation results of cell microtubules. (a) The original cell microtubules intensity simulated by computer. (b) The imaging results of cell microtubules with conventional confocal fluorescence microscopy. (c) The imaging results of cell microtubules imaged by donut PSF. (d) The values of evaluation function C(𝛼) of cell microtubules depending on different 𝛼 values. (e) The imaging results of cell microtubules with conventional FED method. The subtraction factor is set to 0.6. (f) The imaging results of cell microtubules with the proposed method in this article, and the subtraction factor is set to 0.08. The size of the images is 6𝜆 × 6𝜆.
taken when 𝛼 is equal to 0.06. Therefore, the subtraction factor of the proposed method is set to 0.06 in the simulation of spoke-like sample. The imaging results of conventional confocal fluorescence microscopy and proposed method in this article are shown in Fig. 3(b) and (e), respectively. By comparing the unresolved area in central part of the imaging results, we can notice that the proposed method can improve resolution of the conventional confocal fluorescence microscopy. Furthermore, we also simulate cell microtubules by computer to demonstrate that the proposed method can produce better imaging result. The parameters of the microtubules simulation are the same as those of spoke-like simulation. The simulated cell microtubules are shown in Fig. 4(a). The width of single microtubule is 13 nm, and the size of the whole sample is 6𝜆 × 6𝜆. As is shown in Fig. 4(d), the peak value of the evaluation function C(𝛼) is taken at 𝛼 equal to 0.08, so the subtraction factor is set to 0.08 in the simulation of cell microtubules. Fig. 4(b, e and f) show the imaging results of conventional confocal fluorescence microscopy, conventional FED method, and the proposed method, respectively. As is shown in Fig. 4(b and f), there are more details in the imaging result of proposed method, proving that the proposed method can enhance the resolution of conventional confocal microscopy. Besides, the information loss of conventional FED method has also been alleviated in the proposed method due to the smaller subtraction factor in the proposed. Comparing Fig. 4(e) and (f), it is obvious that the imaging result of proposed method is better than that of conventional FED method. As is shown in Fig. 4(e), the width of microtubules is not uniform in the imaging result of conventional FED method. At the position indicated by the arrows of 1 and 2 in Fig. 4(e), there is no intensity information where microtubules actually exist. It is obvious that the information loss is very serious in conventional FED method.
However, because of the reduction of subtraction factor, the information loss is significantly suppressed through the proposed method in this article. We can see from Fig. 4(f) that the microtubules are continuous when they are imaged by the proposed method, and the intensity information exists at the position indicated by the arrows of 1 and 2. 4. Experiments In experiment, we use 200 nm spherical fluorescence particles to perform the experiment. The laser wavelength is 640 nm, the NA of the objective is set to 1.49, 100X. The experimental results of conventional confocal fluorescence microscopy, conventional FED method and the proposed method are shown below. As is shown in Fig. 5(b), the peak value of the evaluation function C(𝛼) is taken at 𝛼 equal to 0.19, so the subtraction factor in this experiment is set to 0.19. The imaging result of conventional confocal fluorescence microscopy is shown in Fig. 5(a), and the image of the proposed method is depicted in Fig. 5(c). As Fig. 5 shows, the resolution of the proposed method is higher than that of conventional confocal fluorescence microscopy. At the positions indicated by the white arrows in Fig. 5(a and c), the proposed method can resolve two fluorescence particles, while the conventional confocal fluorescence microscopy cannot resolve them. This means that the proposed method can achieve higher resolution than conventional confocal fluorescence microscopy. In order to compare the effectiveness of the proposed method and conventional confocal fluorescence microscopy clearly, we draw the intensity profiles along the green line in Fig. 5(a and c). The intensity profiles of the conventional confocal fluorescence microscopy and the proposed method are shown in Fig. 5(d). In the intensity profiles, there 11
V. Le, X. Wang and C. Kuang et al.
Optics and Lasers in Engineering 122 (2019) 8–13
Fig. 5. The imaging results of 200 nm spherical fluorescence particles. (a) The fluorescence particles imaged by conventional confocal fluorescence microscopy. (b) The value of evaluation function C(𝛼) depending on different 𝛼 values. (c) The fluorescence particles imaged by the proposed method in the article, with subtraction factor set to 0.19. (d) The intensity profile of the two methods along the green line in (a) and (c). (The blue dotted line represents the result of conventional confocal fluorescence microscopy, and the black line represents the result of the proposed method. The distance between two peak intensities is 210 nm. The size of (a, c) is 5𝜇m × 5𝜇m.
we can infer that the proposed method with the subtraction factor equal to 0.19 would produce less information loss after zeroing negative. Furthermore, the resolution of the proposed method is higher than that of conventional FED method. At the position indicated by the arrows of 1 and 2 in Fig. 6(a), the conventional FED method can’t resolve two fluorescence particles, but these two fluorescence particles are resolved by the proposed method in Fig. 6(b). At the positions in Fig. 6 indicated by the arrow of 3, the conventional FED method can resolve the two fluorescence particles, but the resolution is not as high as the proposed method. 5. Conclusion Fig. 6. The imaging results of fluorescence particles for (a) the conventional FED method (α = 0.6) and (b) the proposed method (α = 0.19). The negative values still exist on the subtraction results. The size of the image is 5𝜇m × 5𝜇m.
In this article, we proposed a new method based on combination of digital processing and FED method to obtain images with higher resolution. Because of the digital processing, the subtraction factor of proposed method is much smaller than that of conventional FED method. Therefore, the information loss is also alleviated in the proposed method. The simulation and experimental results are presented. Compared with conventional confocal fluorescence microscopy and conventional FED method, the proposed method shows higher resolution and reduces the loss of valid information in conventional FED method significantly.
are two intensity peaks in curve of the proposed method (black line in Fig. 5(d)), while there is only one intensity peak in conventional confocal microscopy (blue dotted line in Fig. 5(d)). As is shown in Fig. 5(d), the lateral resolution of the proposed method is as high as 210 nm, which is much higher than that of conventional confocal microscopy. Next, we compare the imaging results of conventional FED method and proposed method. The subtraction factor of the conventional FED method is set to 0.6, this is the commonly used value [12]. The subtraction factor of the proposed method is much smaller than that of the conventional FED method, which is set to 0.19 according to the peak position of C(𝛼) in Fig. 5(b). The subtraction results of these two methods are shown in Fig. 6. The result of the conventional FED method is shown in Fig. 6(a), while the result of the proposed method is depicted in Fig. 6(b). It is obvious that the negative values have worse effect on the conventional FED method than that on proposed method. Therefore,
Acknowledgments This work is supported by the National Key Research and Development Program of China (2016YFF0101400); National Basic Research Program of China (973Program) (2015CB352003); National Natural Science Foundation of China (NSFC) (61851110762, 61427818, 61827825, 61735017); Natural Science Foundation of Zhejiang Province (LR16F050001), and the Fundamental Research Funds for the Central Universities(2018FZA5005); Vietnam National Foundation for Science and Technology Development (NAFOSTED) under Grant number (103.03-2018.08). 12
V. Le, X. Wang and C. Kuang et al.
Optics and Lasers in Engineering 122 (2019) 8–13
Supplementary materials
[9] Dirk BS, Heit B, Dikeakos JDJARHR. Visualizing interactions between HIV-1 Nef and host cellular proteins using ground-state depletion microscopy. AIDS Res Hum Retroviruses 2015;31(7):671–2. [10] Hofmann M, Eggeling C, Jakobs S, Hell SW. Breaking the diffraction barrier in fluorescence microscopy at low light intensities by using reversibly photoswitchable proteins. Proc Natl Acad Sci USA 2005;102(49):17565–9. [11] Kuang C. Breaking the diffraction barrier using fluorescence emission difference microscopy. Sci Rep 2013;3(3):1441. [12] Le V, Wang X, Kuang C, Liu X. Axial resolution enhancement for light sheet fluorescence microscopy via using the subtraction method. Opt Eng 2018;57(10):103107. [13] Le V, Wang X, Kuang C, Liu X. Background suppression in confocal scanning fluorescence microscopy with superoscillation. Optics Commun 2018;426:541. [14] You S. Eliminating deformations in fluorescence emission difference microscopy. Opt Express 2014;22(21):26375–85. [15] Nan W, Takayoshi KJOE. Numerical study of the subtraction threshold for fluorescence difference microscopy. Opt Express 2015;22(23):28819–30. [16] Ma Y. Virtual fluorescence emission difference microscopy based on photon reassignment. Opt Lett 2015;40(20):4627–30. [17] Zhao G. Resolution enhancement of saturated fluorescence emission difference microscopy. Opt. Express 2016;24(20):23596–609. [18] Brinicombe AM, et al. Blind deconvolution by means of the Richardson–Lucy algorithm. J Opt Soc Am A 1995;12(1):58–65. [19] Gao P, Ulrich Nienhaus G. Precise background subtraction in stimulated emission double depletion nanoscopy. Opt Lett 2017;42(4):831–4.
Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.optlaseng.2019.05.018. References [1] Wilson T. Confocal microscopy, 426. London: Academic Press; 1990. p. 1–64. [2] Gu M. Principles of three-dimensional imaging in confocal microscopies. Singapore: World Scientific; 1996. [3] Pawley J. Handbook of biological of confocal microscopy. 3rd ed. springer; 2006. [4] Segawa S, Kozawa Y, Sato S. Resolution enhancement of confocal microscopy by subtraction method with vector beams. Opt Lett 2014;39(11):3118–21. [5] Hell SW, Wichmann J. Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy. Opt Lett 1994;19(11):780–2. [6] Gao P, Prunsche B, Zhou L, Nienhaus K, Nienhaus GU. Background suppression in fluorescence nanoscopy with stimulated emission double depletion. Nat Photonic 2017;11(3):163–9. [7] Hell SW, Kroug MJAPB. Ground-state-depletion fluorscence microscopy: a concept for breaking the diffraction resolution limit. Appl Phys B. 1995;60(5):495–7. [8] Rittweger E, Wildanger D, Epl SWHJ. Far-field fluorescence nanoscopy of diamond color centers by ground state depletion. Euro Phys Lett 2009;86(1):14001–6 .
13
Contents lists available at ScienceDirect
Optics and Lasers in Engineering journal homepage: www.elsevier.com/locate/optlaseng
Resolution enhancement of confocal fluorescence microscopy via two illumination beams Vannhu Le a,c,+, Xiaona Wang a,+, Cuifang Kuang a,b,∗, Xu Liu a,b a
State Key Laboratory of Modern Optical Instrumentation, College of Optical Science and Engineering, Zhejiang University, Hangzhou 310027, China Collaborative Innovation Center of Extreme Optics, Shanxi University, Taiyuan 030006, China c Department of Optical Engineering, Le Quy Don Technical University, Hanoi, Vietnam b
a r t i c l e
i n f o
Keywords: Confocal fluorescence microscopy Superresolution Digital processing
a b s t r a c t Confocal fluorescence microscopy is an effective imaging technique, but its resolution is limited by the diffractionlimit. Fluorescence emission difference (FED) method is a useful way to improve the resolution of confocal fluorescence microscopy, but the negative values generated during subtraction process might cause loss of valid information. In this paper, we propose one effective method to enhance the resolution of confocal fluorescence microscopy without generating significant negative values. The proposed method combines digital processing and FED method, obtaining the final images with higher resolution and less information loss.
1. Introduction Confocal scanning microscopy is a routine tool in the life sciences. This imaging system can be used to improve resolution of conventional √ microscopy by a factor of 2 [1, 2], and could acquire high-resolution optical images with a certain depth [3]. As a result, confocal scanning microscopy has been used widely in three-dimensional specimen analysis. However, its spatial resolution is restricted to ∼200 nm due to the diffraction limit under common experimental conditions [4]. The demand for higher spatial resolution in optical microscopy has promoted the development of novel super-resolution microscopy, which is capable of breaking the diffraction limit. Multi-beam combination is one of the most common ways to achieve subcellular images and these techniques have developed rapidly, such as stimulated emission depletion (STED) microscopy [5, 6], ground-state depletion (GSD) microscopy, [7–9], reversible saturable/switchable optical transitions (RESOLFT) [10] and so on. STED is a two-beam microscopy that can improve the resolution of conventional fluorescence microscopy significantly, with a Gaussian beam exciting fluorescence, and a donut beam depleting surrounding fluorescence. The spot size can be reduced by matching the Gaussian pumping and donut depletion beams, so that the spatial resolution can be improved. It is a relatively fast way to achieve super-resolution and requires no data post processing. GSD is another two-beam super-resolution microscopy with a similar principle but different time sequences. This method excites the sample with relatively low-power
continuous wave, so that the photo-bleaching and photo-damage are slightly reduced. RESOLFT is a three-beam super-resolution microscopy. In RESOLFT, a Gaussian beam is used to activate the sample, a donut beam turns off the activation of the sample, and a second Gaussian beam excites the fluorophore that is still active. Although these super-resolution microscopy systems mentioned above have been commercially available and they are attractive for biological imaging, the application of these systems is still limited, because of the intricate optical system, special specimen preparation, expensive instrument, higher power source, time-consuming data processing, and high photo-damage. Fluorescence emission difference (FED) method [11–13] provides a new possibility to improve spatial resolution of confocal microscopy. In FED, the sample is illuminated by a solid beam and donut beam, respectively, to obtain two images, called solid image and donut image. The final image is obtained by subtracting the acquired solid image and the donut image. FED method has the following advantages. First, its calculation is simple since we only need to perform subtraction. Second, the cost of FED system is low, and the required beam paths can be assembled easily in the laboratory. Third, the required laser power of FED method is lower than that of STED and GSD, which minimizes photo-damage during the imaging process. However, negative intensity values are inevitably generated during subtraction processing [14]. Although these negative values are zeroed in the final result, they would still lead to information loss. There is a trade-off between the negative values and resolution: the higher the resolution, the more serious the
∗ Corresponding author at: State Key Laboratory of Modern Optical Instrumentation, College of Optical Science and Engineering, Zhejiang University, Room 418, Academic Building #3, #38Zheda Road,Xihu District, Hangzhou 310027, China. E-mail address: [email protected] (C. Kuang). + Authors have contributed equally to this work.
https://doi.org/10.1016/j.optlaseng.2019.05.018 Received 11 January 2019; Received in revised form 19 April 2019; Accepted 17 May 2019 0143-8166/© 2019 Published by Elsevier Ltd.
V. Le, X. Wang and C. Kuang et al.
Optics and Lasers in Engineering 122 (2019) 8–13
Fig. 1. Optical schemes via two illumination beams in confocal fluorescence microscopy.
effect of negative values [15]. To suppress the generation of negative values without reducing the resolution, Researchers have made great efforts and achieved some results [16, 17]. However, the negative values on the final image is still a problem in these methods. In this article, we proposed a new method combining the digital processing and FED methods, to obtain images with high resolution and lower information loss. The proposed method includes two steps. First, we obtain the restored image with higher resolution by digital processing. Then, remove the background of the restored image by introducing FED method. After these two steps, we can achieve high quality images.
Fig. 2. Model of the proposed method.
The procedure of the proposed method is shown in Fig. 2. This method includes two main parts: digital processing and FED method. Firstly, the digital processing is applied to restore the high-resolution image from two original images excited by solid and donut PSFs. Then, FED method is introduced to remove the background in restored image. This is the proposed method to get a high quality final image.
2. Method 2.1. Raw image data The raw image data acquired by the proposed method is the same as that of the FED method: a solid image excited by solid beam and a donut image excited by donut beam. Similar to the FED method, the donut beam is obtained by phase modulation of 0–2𝜋 vortex pattern. In Fig. 1, we indicate two feasible imaging schemes for the proposed method. In Fig. 1(a), there are two illumination paths: one path is solid beam, the other path is donut one. The donut PSF is modulated by 0–2𝜋 vortex phase mask. The two illumination paths should be adjust carefully, to ensure that the solid and donut spots are in the same position on the sample. In Fig. 1(b), a spatial light modulator (SLM) is added to the illumination path to control phase modulation. The phase pattern on the SLM is switched between 0 and 0–2𝜋 vortex phase mask. When the pattern on the SLM is set to 0, the illumination beam is solid; when the pattern on the SLM is set to 0–2𝜋 vortex phase mask, the illumination beam is donut. In the detection path, there is one pinhole, working as the spatial filter. The illumination part shown in Fig. 1(b) is simpler than that in Fig. 1(a). Besides, there is no need to adjust the solid and donut beam to ensure the spot overlap on the sample. So, we use the SLM to modulate the illumination beam. The two images excited by solid and donut PSFs can be presented by, 𝐼𝑠 (𝑥, 𝑦) = 𝑜(𝑥, 𝑦) ⊗ 𝑃 𝑆 𝐹𝑠 (𝑥, 𝑦)
(1)
𝐼𝑑 (𝑥, 𝑦) = 𝑜(𝑥, 𝑦) ⊗ 𝑃 𝑆 𝐹𝑑 (𝑥, 𝑦)
(2)
2.2. Digital processing The two raw images excited by different two PSFs are digitally processed to reconstruct a high-resolution image. In this article, we use the blind post processing to achieve the high-resolution image. As shown in Ref. [18], the blind post processing can correct the effect of the optical aberrations. In this article, we use the Richardson-Lucy deconvolution to achieve the blind post processing, and the iterative formula can be presented by, ( ) 𝐼 𝑡 𝑜𝑡+1 = 𝑜𝑡 × ⊗ 𝑃 𝑆𝐹 (3) 𝑟 𝑜𝑡 ⊗ 𝑃 𝑆 𝐹 𝑡 ( ) 𝐼 𝑃 𝑆 𝐹 𝑡+1 = 𝑃 𝑆 𝐹 𝑡 × ⊗ 𝑜𝑡𝑟 (4) 𝑜𝑡+1 ⊗ 𝑃 𝑆 𝐹 𝑡 where o is the object; or is the flipped object; I is the input image; PSF is the point spread function of illumination beam; PSFr is the flipped point spread function; t is the number of iterations; ⊗ is the convolution operator. Since there are two raw images, the iterative process can be presented by, [ )] ( 𝐼𝑠 𝑡 𝑜𝑡𝑠+1 = 𝑛𝑜𝑟𝑚 𝑜𝑡 × ⊗ 𝑃 𝑆𝐹 (5) 𝑠𝑟 𝑜𝑡 ⊗ 𝑃 𝑆𝐹𝑠𝑡 [ )] ( 𝐼𝑠 𝑃 𝑆𝐹𝑠𝑡+1 = 𝑛𝑜𝑟𝑚 𝑃 𝑆𝐹𝑠𝑡 × ⊗ 𝑜𝑡𝑟 (6) 𝑜𝑡+1 ⊗ 𝑃 𝑆𝐹𝑠𝑡 ( [ )] 𝐼𝑑 𝑡 𝑜𝑡𝑑+1 = 𝑛𝑜𝑟𝑚 𝑜𝑡 × ⊗ 𝑃 𝑆𝐹 (7) 𝑑𝑟 𝑜𝑡 ⊗ 𝑃 𝑆𝐹𝑑𝑡
where o is the object; PSFs , PSFd are the point spread functions of solid beam and donut beams in the cross-section perpendicular to the optical axis; Is , Id are the images excited by solid and donut PSFs, respectively; ⊗ is the convolution operator. 9
V. Le, X. Wang and C. Kuang et al.
Optics and Lasers in Engineering 122 (2019) 8–13
Fig. 3. The simulation results of spoke-like sample with the size of 10𝜆 × 10𝜆. (a) The spoke-like sample. (b) The imaging result of conventional confocal fluorescence microscopy. (c) The imaging result imaged by donut PSF. (d) The values of evaluation function C(𝛼) of the spoke-like sample depending on different 𝛼 values. (e) The imaging result of proposed method in this article, and the subtraction factor is set to 0.06. The size of the images is 10𝜆 × 10𝜆.
[ 𝑃 𝑆𝐹𝑑𝑡+1 = 𝑛𝑜𝑟𝑚 𝑃 𝑆𝐹𝑑𝑡 ×
(
𝐼𝑑
𝑜𝑡+1 ⊗ 𝑃 𝑆𝐹𝑑𝑡
( ) 𝑜𝑡+1 = 𝑛𝑜𝑟𝑚 𝑜𝑡𝑠+1 + 𝑜𝑡𝑑+1
)] ⊗ 𝑜𝑡𝑟
}|( ) ∑| { |𝐹 𝑇 𝑃 𝑆 𝐹𝑟𝑒𝑠 − 𝛼 ⋅ 𝑃 𝑆 𝐹𝑑 | 𝑟∕𝑟𝑚𝑎𝑥 | | 𝐶 (𝛼) = }| ∑| { |𝐹 𝑇 𝑃 𝑆 𝐹𝑟𝑒𝑠 − 𝛼 ⋅ 𝑃 𝑆 𝐹𝑑 | | |
(8)
(9)
(11)
There is also background in the restored image. To remove the background, we can subtract the restored image and donut image, which can be presented by,
where Ʃ represents the summation operation; FT represents the Fourier transform; PSFres is the restored solid point spread function (PSF) after the digital processing; PSFd is the donut PSF; 𝛼 is the subtraction 2 2 )1∕2 = factor; r is the polar radius of frequency space; 𝑟𝑚𝑎𝑥 = (𝜉𝑚𝑎𝑥 + 𝜂𝑚𝑎𝑥 √ 2 2 1∕(2𝛿𝜉 ) + 1∕(2𝜂𝜉 ) , and 𝛿 𝜉 , 𝜂 𝜉 represent the length and width of the pixel, respectively. In the following simulation and experiment, the subtraction factor is calculated by this function. As the calculation results show in the following simulation and experiments, the subtraction factor in our proposed method is greatly reduced. Therefore, the negative value generated in the image is also reduced in the subtraction process. Then, the final image is obtained by zeroing the negative value of the subtraction result. Because the negative values in the subtraction result are reduced, the zeroing process would not cause as much information loss as the conventional FED method.
𝐼 (𝑥, 𝑦) = 𝐼𝑟𝑒𝑠 (𝑥, 𝑦) − 𝛼 ⋅ 𝐼𝑑 (𝑥, 𝑦)
3. Simulation results
where o is the object restored from the two raw images; or is the flipped o; os is the object restored from the raw image excited by solid PSF; od is the object restored from the raw image excited by donut PSF; Is is the raw image excited by solid PSF; Id is the raw image excited by donut PSF; PSFs is the point spread function of solid beam; PSFsr is the flipped PSFs ; PSFd is the point spread function of donut beam; PSFdr is the flipped PSFd ; norm() represents normalized calculation; t is the number of iterations; ⊗ is the convolution operator. 2.3. FED method
(10)
where Ires is the restored image, after the digital processing; Id is the image excited by donut PSF; 𝛼 is the subtraction factor. The subtraction factor in above formula is much smaller than that in conventional FED method, so the negative values in the subtraction result are greatly reduced. To obtain an optimal subtraction factor, we analysis the imaging results of different subtraction factors with an evaluation function [19]:
In order to show the effectiveness of the proposed method, the simulation results are presented in this section. In the simulation, the numerical aperture (NA) of objective was set to 1.49, and wavelength was set to 640 nm, which were the parameters used in experiments. As is shown in Fig. 3(a), the sample is a spoke-like sample with the size of 10𝜆 × 10𝜆. The values of C(𝛼) are shown in Fig. 3(d), and the peak value of C(𝛼) is 10
V. Le, X. Wang and C. Kuang et al.
Optics and Lasers in Engineering 122 (2019) 8–13
Fig. 4. The simulation results of cell microtubules. (a) The original cell microtubules intensity simulated by computer. (b) The imaging results of cell microtubules with conventional confocal fluorescence microscopy. (c) The imaging results of cell microtubules imaged by donut PSF. (d) The values of evaluation function C(𝛼) of cell microtubules depending on different 𝛼 values. (e) The imaging results of cell microtubules with conventional FED method. The subtraction factor is set to 0.6. (f) The imaging results of cell microtubules with the proposed method in this article, and the subtraction factor is set to 0.08. The size of the images is 6𝜆 × 6𝜆.
taken when 𝛼 is equal to 0.06. Therefore, the subtraction factor of the proposed method is set to 0.06 in the simulation of spoke-like sample. The imaging results of conventional confocal fluorescence microscopy and proposed method in this article are shown in Fig. 3(b) and (e), respectively. By comparing the unresolved area in central part of the imaging results, we can notice that the proposed method can improve resolution of the conventional confocal fluorescence microscopy. Furthermore, we also simulate cell microtubules by computer to demonstrate that the proposed method can produce better imaging result. The parameters of the microtubules simulation are the same as those of spoke-like simulation. The simulated cell microtubules are shown in Fig. 4(a). The width of single microtubule is 13 nm, and the size of the whole sample is 6𝜆 × 6𝜆. As is shown in Fig. 4(d), the peak value of the evaluation function C(𝛼) is taken at 𝛼 equal to 0.08, so the subtraction factor is set to 0.08 in the simulation of cell microtubules. Fig. 4(b, e and f) show the imaging results of conventional confocal fluorescence microscopy, conventional FED method, and the proposed method, respectively. As is shown in Fig. 4(b and f), there are more details in the imaging result of proposed method, proving that the proposed method can enhance the resolution of conventional confocal microscopy. Besides, the information loss of conventional FED method has also been alleviated in the proposed method due to the smaller subtraction factor in the proposed. Comparing Fig. 4(e) and (f), it is obvious that the imaging result of proposed method is better than that of conventional FED method. As is shown in Fig. 4(e), the width of microtubules is not uniform in the imaging result of conventional FED method. At the position indicated by the arrows of 1 and 2 in Fig. 4(e), there is no intensity information where microtubules actually exist. It is obvious that the information loss is very serious in conventional FED method.
However, because of the reduction of subtraction factor, the information loss is significantly suppressed through the proposed method in this article. We can see from Fig. 4(f) that the microtubules are continuous when they are imaged by the proposed method, and the intensity information exists at the position indicated by the arrows of 1 and 2. 4. Experiments In experiment, we use 200 nm spherical fluorescence particles to perform the experiment. The laser wavelength is 640 nm, the NA of the objective is set to 1.49, 100X. The experimental results of conventional confocal fluorescence microscopy, conventional FED method and the proposed method are shown below. As is shown in Fig. 5(b), the peak value of the evaluation function C(𝛼) is taken at 𝛼 equal to 0.19, so the subtraction factor in this experiment is set to 0.19. The imaging result of conventional confocal fluorescence microscopy is shown in Fig. 5(a), and the image of the proposed method is depicted in Fig. 5(c). As Fig. 5 shows, the resolution of the proposed method is higher than that of conventional confocal fluorescence microscopy. At the positions indicated by the white arrows in Fig. 5(a and c), the proposed method can resolve two fluorescence particles, while the conventional confocal fluorescence microscopy cannot resolve them. This means that the proposed method can achieve higher resolution than conventional confocal fluorescence microscopy. In order to compare the effectiveness of the proposed method and conventional confocal fluorescence microscopy clearly, we draw the intensity profiles along the green line in Fig. 5(a and c). The intensity profiles of the conventional confocal fluorescence microscopy and the proposed method are shown in Fig. 5(d). In the intensity profiles, there 11
V. Le, X. Wang and C. Kuang et al.
Optics and Lasers in Engineering 122 (2019) 8–13
Fig. 5. The imaging results of 200 nm spherical fluorescence particles. (a) The fluorescence particles imaged by conventional confocal fluorescence microscopy. (b) The value of evaluation function C(𝛼) depending on different 𝛼 values. (c) The fluorescence particles imaged by the proposed method in the article, with subtraction factor set to 0.19. (d) The intensity profile of the two methods along the green line in (a) and (c). (The blue dotted line represents the result of conventional confocal fluorescence microscopy, and the black line represents the result of the proposed method. The distance between two peak intensities is 210 nm. The size of (a, c) is 5𝜇m × 5𝜇m.
we can infer that the proposed method with the subtraction factor equal to 0.19 would produce less information loss after zeroing negative. Furthermore, the resolution of the proposed method is higher than that of conventional FED method. At the position indicated by the arrows of 1 and 2 in Fig. 6(a), the conventional FED method can’t resolve two fluorescence particles, but these two fluorescence particles are resolved by the proposed method in Fig. 6(b). At the positions in Fig. 6 indicated by the arrow of 3, the conventional FED method can resolve the two fluorescence particles, but the resolution is not as high as the proposed method. 5. Conclusion Fig. 6. The imaging results of fluorescence particles for (a) the conventional FED method (α = 0.6) and (b) the proposed method (α = 0.19). The negative values still exist on the subtraction results. The size of the image is 5𝜇m × 5𝜇m.
In this article, we proposed a new method based on combination of digital processing and FED method to obtain images with higher resolution. Because of the digital processing, the subtraction factor of proposed method is much smaller than that of conventional FED method. Therefore, the information loss is also alleviated in the proposed method. The simulation and experimental results are presented. Compared with conventional confocal fluorescence microscopy and conventional FED method, the proposed method shows higher resolution and reduces the loss of valid information in conventional FED method significantly.
are two intensity peaks in curve of the proposed method (black line in Fig. 5(d)), while there is only one intensity peak in conventional confocal microscopy (blue dotted line in Fig. 5(d)). As is shown in Fig. 5(d), the lateral resolution of the proposed method is as high as 210 nm, which is much higher than that of conventional confocal microscopy. Next, we compare the imaging results of conventional FED method and proposed method. The subtraction factor of the conventional FED method is set to 0.6, this is the commonly used value [12]. The subtraction factor of the proposed method is much smaller than that of the conventional FED method, which is set to 0.19 according to the peak position of C(𝛼) in Fig. 5(b). The subtraction results of these two methods are shown in Fig. 6. The result of the conventional FED method is shown in Fig. 6(a), while the result of the proposed method is depicted in Fig. 6(b). It is obvious that the negative values have worse effect on the conventional FED method than that on proposed method. Therefore,
Acknowledgments This work is supported by the National Key Research and Development Program of China (2016YFF0101400); National Basic Research Program of China (973Program) (2015CB352003); National Natural Science Foundation of China (NSFC) (61851110762, 61427818, 61827825, 61735017); Natural Science Foundation of Zhejiang Province (LR16F050001), and the Fundamental Research Funds for the Central Universities(2018FZA5005); Vietnam National Foundation for Science and Technology Development (NAFOSTED) under Grant number (103.03-2018.08). 12
V. Le, X. Wang and C. Kuang et al.
Optics and Lasers in Engineering 122 (2019) 8–13
Supplementary materials
[9] Dirk BS, Heit B, Dikeakos JDJARHR. Visualizing interactions between HIV-1 Nef and host cellular proteins using ground-state depletion microscopy. AIDS Res Hum Retroviruses 2015;31(7):671–2. [10] Hofmann M, Eggeling C, Jakobs S, Hell SW. Breaking the diffraction barrier in fluorescence microscopy at low light intensities by using reversibly photoswitchable proteins. Proc Natl Acad Sci USA 2005;102(49):17565–9. [11] Kuang C. Breaking the diffraction barrier using fluorescence emission difference microscopy. Sci Rep 2013;3(3):1441. [12] Le V, Wang X, Kuang C, Liu X. Axial resolution enhancement for light sheet fluorescence microscopy via using the subtraction method. Opt Eng 2018;57(10):103107. [13] Le V, Wang X, Kuang C, Liu X. Background suppression in confocal scanning fluorescence microscopy with superoscillation. Optics Commun 2018;426:541. [14] You S. Eliminating deformations in fluorescence emission difference microscopy. Opt Express 2014;22(21):26375–85. [15] Nan W, Takayoshi KJOE. Numerical study of the subtraction threshold for fluorescence difference microscopy. Opt Express 2015;22(23):28819–30. [16] Ma Y. Virtual fluorescence emission difference microscopy based on photon reassignment. Opt Lett 2015;40(20):4627–30. [17] Zhao G. Resolution enhancement of saturated fluorescence emission difference microscopy. Opt. Express 2016;24(20):23596–609. [18] Brinicombe AM, et al. Blind deconvolution by means of the Richardson–Lucy algorithm. J Opt Soc Am A 1995;12(1):58–65. [19] Gao P, Ulrich Nienhaus G. Precise background subtraction in stimulated emission double depletion nanoscopy. Opt Lett 2017;42(4):831–4.
Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.optlaseng.2019.05.018. References [1] Wilson T. Confocal microscopy, 426. London: Academic Press; 1990. p. 1–64. [2] Gu M. Principles of three-dimensional imaging in confocal microscopies. Singapore: World Scientific; 1996. [3] Pawley J. Handbook of biological of confocal microscopy. 3rd ed. springer; 2006. [4] Segawa S, Kozawa Y, Sato S. Resolution enhancement of confocal microscopy by subtraction method with vector beams. Opt Lett 2014;39(11):3118–21. [5] Hell SW, Wichmann J. Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy. Opt Lett 1994;19(11):780–2. [6] Gao P, Prunsche B, Zhou L, Nienhaus K, Nienhaus GU. Background suppression in fluorescence nanoscopy with stimulated emission double depletion. Nat Photonic 2017;11(3):163–9. [7] Hell SW, Kroug MJAPB. Ground-state-depletion fluorscence microscopy: a concept for breaking the diffraction resolution limit. Appl Phys B. 1995;60(5):495–7. [8] Rittweger E, Wildanger D, Epl SWHJ. Far-field fluorescence nanoscopy of diamond color centers by ground state depletion. Euro Phys Lett 2009;86(1):14001–6 .
13