Optics and Lasers in Engineering 123 (2019) 45–52
Contents lists available at ScienceDirect
Optics and Lasers in Engineering journal homepage: www.elsevier.com/locate/optlaseng
Total internal reflection fluorescence pattern-illuminated Fourier ptychographic microscopy Qiulan Liu a,†, Youhua Chen a,c,†, Wenjie Liu a, Yubing Han a, Ruizhi Cao a, Zhimin Zhang a, Cuifang Kuang a,b,∗, Xu Liu a,b a
State Key Laboratory of Modern Optical Instrumentation, College of Optical Science and Engineering, Zhejiang University, Hangzhou, Zhejiang 310027, China Collaborative Innovation Center of Extreme Optics, Shanxi University, Taiyuan 030006, China c Key Laboratory of Instrumentation Science & Dynamic Measurement of Ministry of Education, North University of China, Taiyuan, Shanxi 030051, China b
a b s t r a c t We report a widefield super-resolution (SR) fluorescence microscopy technique termed total internal reflection fluorescence pattern-illuminated Fourier ptychographic microscopy (TIRF-piFPM). It employs pattern-illuminated Fourier ptychography (piFP) under the total internal reflection fluorescence (TIRF) mode, providing a lateral resolution of ∼100 nm, reducing the background level, and correcting unknown optical aberrations. Like the total internal reflection fluorescence structure illuminated microscopy (TIRF-SIM), the illumination field of TIRF-piFPM is modulated by sinusoidal patterns generated by evanescent wave interference. It differs from TIRF-SIM in that TIRF-piFPM reconstructs the SR image by FP iteration. To demonstrate the performance of TIRF-piFPM, we compare it with TIRF-SIM by conducting simulations and experiments and prove that TIRF-piFPM can provide a better result than TIRF-SIM in terms of its robustness to noise and aberration correction ability. In addition, dynamic changes of mitochondria in U2OS cells are captured with a temporal resolution of 2 s, demonstrating its live-cell imaging capability. The addvantages may enable TIRF-piFPM to serve as an alternative to SR fluorescence microscopes in the TIRF family, with promising applications in biological and biomedical imaging.
1. Introduction Fluorescence microscopy is widely employed in biological research and clinical diagnosis owing to its noninvasive characteristic. However, because of the diffraction limit, the typical resolution of a light microscope cannot surpass 200 nm. Over the past several decades, many techniques have been developed to surpass this limit. Among these techniques, structured illumination microscopy (SIM) [1–4] is a powerful method used for biomedical imaging as it provides both high temporal and spatial resolution, enabling video-rate super-resolution (SR) imaging speed. Although only two-fold resolution improvement can be achieved using SIM, the required light intensity is significantly lower compared with stimulated emission depletion microscopy (STED) [5,6], which requires several gigawatts per square centimeter to achieve nanometer-level resolution. Other methods, such as the photoactivated localization microscopy (PALM) [7,8] and stochastic optical reconstruction microscopy (STORM) [9,10], are significantly slower than SIM, as they require thousands of images to reconstruct an SR image, making them difficult for live cell imaging. In recent years, SIM has become a popular method for imaging live cells in a whole life event owing to its high speed and low phototoxicity and photobleaching.
A total internal reflection fluorescence (TIRF) mode-based SIM, i.e., TIRF-SIM [11–15], has a better resolution than conventional SIM because of the finer interference illumination patterns generated. Accurately determining the modulation frequency and phase of the illumination pattern from raw images captured in the TIRF mode is crucial to the separation of the superimposed information components. To date, various algorithms have been proposed to improve the quality of the reconstructed image with an optimized pattern phase obtained from the raw data of TIRF-SIM [16–18]. Although the performance of SIM is excellent in terms of resolution and speed, it is sensitive to system aberrations and signal-to-noise ratio (SNR) of the image [19]. The distortion of the images due to aberrations or system errors and low SNR may lead to an unsuccessful reconstruction process or a poor reconstruction result [18]. In this paper, we report a pattern-illuminated Fourier ptychography [20] (piFP) technique under the TIRF mode, termed the total internal reflection fluorescence pattern-illuminated Fourier ptychographic microscopy (TIRF-piFPM). It can provide a lateral resolution of ∼100 nm, reduce the background level, and correct unknown optical aberrations. Like TIRF-SIM, two inversely propagating evanescent waves interfere and generate fine sinusoidal illumination patterns. The non-uniform
∗ Corresponding author at: State Key Laboratory of Modern Optical Instrumentation, College of Optical Science and Engineering, Zhejiang University, Room 418, Academic Building #3, #38 Zheda Road, Xihu District, Hangzhou, Zhejiang 310027, China. E-mail address:
[email protected] (C. Kuang). † These authors contributed equally to this work.
https://doi.org/10.1016/j.optlaseng.2019.06.023 Received 21 December 2018; Received in revised form 10 June 2019; Accepted 26 June 2019 0143-8166/© 2019 Elsevier Ltd. All rights reserved.
Q. Liu, Y. Chen and W. Liu et al.
Optics and Lasers in Engineering 123 (2019) 45–52
illumination introduced by the evanescent wave interference enables it to down-modulate even higher spatial frequency information of an object into the passband of the optical transfer function (OTF) and realize a higher imaging resolution. The evanescent waves induced in the TIRF mode provide an extremely thin excited layer, thus improving the detection contrast of TIRF-piFPM. In the image reconstruction process, the piFP algorithm is employed to reconstruct an SR object image based on the captured low-resolution images with superimposed information components. The piFP is an iterative approach that sequentially updates the images via switching between the spatial and Fourier domains, similar to the conventional FP [21–25]. The iteration is done until the solution converges, enabling to reconstruct results under certain statistic requirement. As an optimization algorithm, piFP with OTF correction is relatively unsusceptible to noise and system errors compared with SIM. The mechanical or optical changes in the microscope can induce aberrations in the pupil function, thus leading to an inaccurate estimation of the OTF. An inaccurately estimated OTF will result in a poor recovery. TIRF-piFPM can reconstruct a high-quality image without a priori knowledge of the aberration in the system by updating and recovering the correct OTF in an iterative manner. In the presence of system errors, we show that TIRF-piFPM outperforms TIRF-SIM while maintaining the aberration correction ability of the optimization algorithm [26–30]. Finally, to show its live cell imaging capability, we present a biological dynamic application, wherein the fission process of mitochondria in living U2OS cells is captured with high spatial and temporal resolution.
2.1. Illumination pattern estimation As shown in the area enclosed by the green dashed lines in Fig. 1, the illumination patterns 𝑃𝑛 (𝑛 = 1, 2, 3, … , 𝑁 ) are extracted from the recorded raw images 𝐼𝑛 (𝑛 = 1, 2, 3, … , 𝑁 ) prior to the piFP iteration. The primary step in the illumination pattern estimation is to obtain the correct modulation frequency and phase. Several methods have been proposed to determine the illumination pattern [16–18]. Here, we extract the illumination pattern according to the following method. For a three-step phase shifting strategy, the image sequences 𝐼 𝑗 (𝑗 = 1, 2, 3) taken under a sinusoidal pattern illumination with a certain orientation can be expressed as [ ] 𝐼 𝑗 (𝐫) = 𝑃 𝑗 ⋅ 𝑂 ∗ 𝑃 𝑆𝐹 = 1 + 𝑚 cos(𝐤0 𝐫 + 𝜑𝑗 ) ⋅ 𝑂(𝐫) ∗ 𝐻(𝐫), (1) where 𝑃 𝑗 (𝑗 = 1, 2, 3) is the illumination pattern under the jth phase step, k0 is the modulation frequency vector of the pattern and 𝜑j is its corresponding phase, m is the modulation depth determining the pattern contrast, O denotes the object function, H(r) is the detection point spread function (PSF), and ∗ denotes the convolution operation. In the Fourier domain, Eq. (1) can be written in the matrix form as ⎡1 ⎡ ⎤ ⎢ ⎢𝐼̃2 (𝐤)⎥=⎢⎢1 ⎢ ̃3 ⎥ ⎢ ⎣𝐼 (𝐤)⎦ ⎢1 ⎣ 𝐼̃1 (𝐤)
𝑚 𝑖𝜑1 ⋅𝑒 2 𝑚 𝑖𝜑2 ⋅𝑒 2 𝑚 𝑖𝜑3 ⋅𝑒 2
𝑚 −𝑖𝜑1 ⎤ ⋅𝑒 2 ⎥ 𝑂̃ (𝐤) ⋅ 𝐻̃ (𝐤) ⎤ ) 𝑚 −𝑖𝜑2 ⎥⎡⎢ ( ⋅𝑒 𝑂̃ 𝐤 − 𝐤0 ⋅ 𝐻̃ (𝐤)⎥, ⎥ ( ) 2 ⎢ ⎥ 𝑂̃ 𝐤 + 𝐤0 ⋅ 𝐻̃ (𝐤)⎥⎦ 𝑚 − 𝑖 𝜑 3 ⎥⎣ ⋅𝑒 ⎦ 2
(2)
where ∼ over I and O denotes the corresponding Fourier transform, and 𝐻̃ (𝐤) is the detection OTF. It is important to accurately estimate the modulation frequency vector k0 and phase 𝜑j from the recorded images. When |k0 | is smaller than the cutoff frequency kc of the OTF and the SNR is relatively high, the peak searching method [16] that estimates k0 and retrieves the phase 𝜑j from the Fourier transform of a certain captured image is a good strategy. However, this method will give a poor outcome when |k0 | is close to or greater than kc , like the case in the TIRF mode. To improve the estimation accuracy of the illumination pattern in the TIRF mode, an auto-correlation method [17] is applied as follows.
2. Principle Non-uniform illumination enables intensity modulation and shifts the high-frequency information of the object to the passband of the detection OTF. Each recorded low-resolution image contributes additional information to the recovery of the final high-resolution object profile. piFP utilizes pattern illumination and iteratively reconstructs a highresolution image of the object using the captured low-resolution images. In the recovery process, piFP switches between the spatial and Fourier domains. In the spatial domain, the acquired images are used to constrain the intensity of the sample estimate, similar to the phase retrieval technique [31,32]. In the Fourier domain, the OTF of the objective is used as a support constraint for the solution. In this study, the proposed TIRF-piFPM achieves SR imaging using the piFP algorithm under the TIRF mode. Although the piFP method does not require the phase shift of the illumination pattern, the proposed TIRF-piFPM still utilizes a framework that shares similarity with SIM. To acquire an isotropic resolution improvement, different orientations of the sinusoidal patterns generated by the evanescent waves interference are projected onto the object plane. We assume N/3 orientations of sinusoidal patterns with the azimuths in the range [0, 𝜋) are employed to illuminate the object, and three phase steps strategy is implemented to laterally shift the pattern across the object. N modulated images 𝐼𝑛 (𝑛 = 1, 2, 3, … , 𝑁 ) are then captured in the acquisition procedure as the raw data stack for SR image reconstruction. To ensure the convergence, an adequate data redundancy is required as the constraint in the iterative process. In the original FPM, a spectrum overlap of ∼60% is used in the implementation to facilitate image convergence [21,33]. In our TIRF-piFPM, more than five orientations of the illumination patterns (i.e., N > 15; raw images) are utilized to recover the complex SR image. We discuss this redundancy issue in the discussion and conclusion section. piFP can handle situations wherein the illumination pattern is unknown, in which case an unknown pattern can be used to generate multiple low-resolution images by projecting the pattern at different spatial positions or rotating it at different orientations [20]. In TIRF-piFPM, the piFP algorithm is implemented with the prior knowledge of the illumination pattern. Fig. 1 shows the recovery procedure of our TIRF-piFPM in acquiring an SR image.
( ) [ ] 𝐴̃ 𝐤′ = 𝐼̃𝑗 (𝐤) ⊗ 𝐼̃𝑗 (𝐤) =
∫
[ 𝑗 ]∗ 𝑗 ( ) 𝐼̃ (𝐤) ⋅ 𝐼̃ 𝐤 + 𝐤′ 𝑑𝐤,
(3)
where ⊗ denotes the correlation operation, and the superscript ∗ denotes the complex conjugation. 𝐴̃ (𝐤′ ) will output the peak value in the Fourier space when 𝐤′ = ±𝐤0 . When 𝐤′ = 𝐤0 , Eq. (3) will satisfy the following expression [ ( ) 𝑚 𝑗 ( )] 𝑗 𝑚 𝑂̃ ∗ (𝐤) + 𝑒−𝑖𝜑 𝑂̃ ∗ 𝐤 − 𝐤0 + 𝑒𝑖𝜑 𝑂̃ ∗ 𝐤 + 𝐤0 𝐻̃ ∗ (𝐤) ∫ 2 2 [ ( ) 𝑚 𝑖𝜑𝑗 )] ( ) 𝑚 −𝑖𝜑𝑗 ̃ ( ̃ ̃ 𝑂 𝐤 + 2𝐤0 𝐻̃ 𝐤 + 𝐤0 𝑑𝐤 × 𝑂 𝐤 + 𝐤 0 + 𝑒 𝑂 (𝐤 ) + 𝑒 2 2 ( ) 𝑚 𝑗 ≈ 𝑂̃ ∗ (𝐤) 𝑒𝑖𝜑 𝑂̃ (𝐤)𝐻̃ ∗ (𝐤)𝐻̃ 𝐤 + 𝐤0 𝑑𝐤 ∫ 2 ) ( ) ( ) 𝑚 𝑖𝜑𝑗 ̃ ∗ ( + 𝑒 𝑂 𝐤 + 𝐤0 𝑂̃ 𝐤 + 𝐤0 𝐻̃ ∗ (𝐤)𝐻̃ 𝐤 + 𝐤0 𝑑𝐤 ∫ 2
( ) 𝐴̃ 𝐤0 =
𝑗
𝑗
𝑗
= 𝑎1 𝑒𝑖𝜑 + 𝑎2 𝑒𝑖𝜑 = 𝛾𝑒𝑖𝜑 .
(4)
Eq. (4) can be approximated, as only the terms 𝑂̃ ∗ (𝐤) ⋅ 𝑂̃ (𝐤) and 𝑂̃ ∗ (𝐤 + 𝐤0 ) ⋅ 𝑂̃ (𝐤 + 𝐤0 ) produce the peak value in the integral expression. The other terms can be ignored. Thus, the phase 𝜑j of the illumination pattern can be obtained as [ ( )] 𝜑𝑗 = arg 𝐴̃ 𝐤0 , (5) where arg( · ) is the argument of the complex number. By determining the peak in the autocorrelation result, we can obtain the modulation frequency k0 and the phase 𝜑 of a certain illumination pattern. The operation is repeated for the modulated 46
Q. Liu, Y. Chen and W. Liu et al.
Optics and Lasers in Engineering 123 (2019) 45–52
Fig. 1. Procedure to recover an SR image in TIRF-piFPM. The captured raw images In are first employed to extract the illumination patterns Pn , as indicated in the box enclosed by green dashed lines. The raw images In and extracted patterns Pn are then applied to the piFP iteration procedure (indicated by red and blue colors) to generate an SR image of the object.
images with other illumination patterns. Finally, all the illumination patterns, 𝑃𝑛 (𝑛 = 1, 2, 3, … , 𝑁 ) can be extracted from the volume of raw data 𝐼𝑛 (𝑛 = 1, 2, 3, … , 𝑁 ).
herent imaging [26] to improve the image result. A theoretical detection OTF is used when the iteration algorithm begins, and it will be updated using Eq. (8) as the reconstruction proceeds [30].
2.2. Pattern illumination Fourier ptychographic (piFP) algorithm
∗ ⋅ (𝐼̃ − 𝐻 ̃ ⋅ 𝐼̃𝑡𝑛 ) 𝑎||𝐼̃𝑡𝑛 || ⋅ 𝐼̃𝑡𝑛 𝑛 𝐻̃ ′ = 𝐻̃ + , 2 | | | | ̃ ̃ max(|𝐼𝑡𝑛 |) ⋅ (|𝐼𝑡𝑛 | + 𝜀)
Once the illumination patterns have been acquired, the piFP algorithm can be applied. In Fig. 1, the piFP algorithm flow, indicated by the red and blue colors, illustrates the iteration process and the resolution enhancement via intensity modulation with different sinusoidal patterns. This process can be described as follows: First, the recorded images In in the data stack are summed to obtain a widefield image of the object, and this image is considered as an initial guess of the original object profile O (the initial guess with random values can also work) (Step 1). Second, this initial guess is sequentially updated using the intensity measurements modulated with different illumination patterns (Step 2). For each update sub-step, the object estimation is first multiplied by a known illumination Pn extracted using the aforementioned method to obtain a target image 𝐼𝑡𝑛 = 𝑂 ∗ 𝑃𝑛 (Step 2.1). Then, the Fourier transform of the target image Itn is updated using the corresponding lowresolution measurement In via the following equation in the Fourier domain. ′ 𝐼̃𝑡𝑛 = 𝐼̃𝑡𝑛 + 𝐻̃ ∗ ⋅ (𝐼̃𝑛 − 𝐻̃ ⋅ 𝐼̃𝑡𝑛 ), ′ 𝐼̃𝑡𝑛
where 𝐻̃ ′ is the updated OTF, a is a constant to adjust the update step for OTF and 𝑎 = 1 is used in this manuscript, and ɛ is a positive constant introduced to avoid the zero denominator (Step 2.4). Steps 2.1–2.4 are applied to all the other low-resolution measurements modulated with the other illumination patterns. Third, Step 2 is repeated several times until the solution converges, and an SR image of the object is ultimately acquired (Step 3). 3. Experimental setup The TIRF-piFPM imaging system, shown in Fig. 2(a), is designed based on a Nikon Ti2-E microscope [34]. The linearly polarized laser beam coupled into a polarization-maintaining single-mode fiber (PMSF, Oz Optics, NA = 0.11) is collimated using a collimator (CL, Thorlabs, ZC618FC) and split into two paths via a polarizing beam splitter (PBS, Thorlabs, CCM1-PBS251/M). The beams respectively enter a twodimensional galvanometer (2D-GM, Cambridge Technology). The two 2D-GMs are used to control the focus position of the beams at the back focal plane (BFP) of an objective lens (OL, Nikon, 1.49/100 × ) after they are respectively focused by a scanning lens (SL, Thorlabs, SL50-CLS2), combined by a beam splitter (BS, Thorlabs, CCM1-BS013) and relayed by two lenses (L1 and L2, Thorlabs). The beams exit the OL with an incident angle greater than the critical angle and excite two counterpropagating evanescent waves. They interfere with each other and form a sinusoidal pattern at the object plane. A piezoelectric transducer (PZT, PI, P-753) is used in one of the paths to change the corresponding
(6) ′ 𝐼𝑡𝑛
where is the updated target image spectrum (Step 2.2). Next, is utilized to update the SR object estimation in the spatial domain using Eq. (7). 𝑃𝑛 ′ 𝑂′ = 𝑂 + [ ]2 (𝐼𝑡𝑛 − 𝐼𝑡𝑛 ), max(𝑃𝑛 )
(8)
(7)
where O′ is the updated object estimation (Step 2.3). Finally, a digital correction method is introduced to correct the aberration as in the co47
Q. Liu, Y. Chen and W. Liu et al.
Optics and Lasers in Engineering 123 (2019) 45–52
Fig. 2. Schematic of the TIRF-piFPM. (a) Experimental setup. PMSF, polarization maintaining single-mode fiber; CL, collimator; HWP, half-wave plate; PBS, polarizing beam splitter; BS, beam splitter; M, mirror; 2D-GM, two-dimensional galvanometer; SL, scanning lens; PC, polarization converter; L, lens; DM, dichroic mirror; BFP, back focal plane; OL, objective lens; O, object; F, color filter; TL, tube lens. (b) Evanescent wave will be generated when the incident angle is greater than the critical angle, and only a very thin layer of the object is excited to produce fluorescent emission.
optical path length and realize the three phase steps of the illumination pattern. To improve the contrast of the illumination pattern, which is particularly important for a TIRF-mode system, a polarization converter (PC, Thorlabs, WPV10L) is installed at the conjugated BFP to produce an s-polarized light illuminating the object. A thin layer of the object (right part of Fig. 2(b)) is excited, and the fluorescent signals are collected using the same OL and detected using a camera (EMCCD, Andor, iXon Ultra 888) after passing through a dichroic mirror (DM, Chroma, C174298) and a tube lens (TL, Thorlabs, TTL200-A). 4. Simulation and experimental results To investigate the imaging performance of the proposed technique, we conduct simulations and experiments. 4.1. Simulation results In Fig. 3, a spoke-like sample is used to estimate the resolution improvement ability of TIRF-piFPM. Ten orientations of the sinusoidal patterns with an incident angle (of the illumination beam) of approximately 68° (critical angle θc = 61.2° for water-immersion sample) are employed for data acquisition (excitation wavelength: 561 nm, emission wavelength: 599 nm). The same data stack is applied to TIRF-SIM and TIRF-piFPM. As shown in Fig. 3(a1)–(c1), both TIRF-piFPM and TIRFSIM achieve about a two-fold resolution enhancement. In addition, to analyze the robustness against noise, Gaussian noise with standard deviations of 10% and 25% is added to the raw data stack. The SNR is quantified by the ratio of the noise-free image average intensity to the mean square error between the noise-free result and its noise-corrupted counterpart. TIRF-piFPM outperforms TIRF-SIM in the presence of the Gaussian noise (Fig. 3(a2)–(c3)). In Fig. 3(a3), although the raw data contain 25% Gaussian noise, TIRF-piFPM could still recover the sample with high fidelity (Fig. 3(c3)), thus demonstrating its reliability against noise. To verify the aberration correction capability of the proposed TIRFpiFPM, we introduce a spherical aberration, a field curvature aberration, an astigmatism, and a superposition of the three aberrations into
Fig. 3. Simulation of a spoke-like sample with different Gaussian noise levels. (a1)–(a3) Widefield images without noise and with 10% and 25% Gaussian noise levels, respectively. (b1)–(b3) The corresponding reconstructed images using TIRF-SIM. (c1)–(c3) The corresponding reconstructed images using TIRF-piFPM.
the detection pupil function. TIRF-piFPM without aberration correction (Fig. 4(b1)–(b4)) and TIRF-SIM (Fig. 4(d1)–(d4)) both result in degraded image quality. This is because the aberrated wavefront of the pupil function influences the frequency components of the sample spectrum. As an aberration correction step (Eq. (8)) is added, TIRF-piFPM shows high resistance toward different types of aberrations and can give an optimized object estimation (Fig. 4(c1)–(c4)) with updated and corrected OTF via the iterative process. 48
Q. Liu, Y. Chen and W. Liu et al.
Optics and Lasers in Engineering 123 (2019) 45–52
Fig. 4. Simulation of a spoke-like sample with different aberration types. (a1)–(a4) Phase (wavefront) distribution of the pupil function under spherical aberration, field curvature aberration, astigmatism, and superposition of the above three aberrations, respectively. The corresponding reconstructed image using TIRF-piFPM without aberration correction (b1)–(b4) and with aberration correction (c1)–(c4). (d1)–(d4) The corresponding TIRFSIM reconstructed image.
4.2. Experimental results
ergy to the cell. Fig. 6(a1)–(c1) show the images acquired from the infocus raw images. Unlike the widefield image (Fig. 6(a1)), we can observe the hollow globular mitochondrial morphology, the intermediate mitochondrial structures, and the interconnected tubular network morphology with more details from Fig. 6(b1) and (c1). It is worthwhile to note that the number of raw images for TIRF-SIM reconstruction is the same as that for TIRF-piFPM to ensure an identical expansion of the spectra in the 2D Fourier space. To further verify the performance of TIRF-piFPM, we record the raw images under a defocused condition by moving up the sample stage by approximately 300 nm relative to the original position. The depth of field of the OL and the thickness of mitochondria (generally a few hundred nanometers) are considered in determining the defocus distance. As shown in Fig. 6(a2), the out-of-focus widefield deconvolution image of the mitochondrial structures has a blurry outline with a bright background. As shown in Fig. 6(b2), the bright background is removed when using TIRF-piFPM, giving clear mitochondrial details. However, in Fig. 6(c2), the TIRF-SIM outputs a result with a low SNR. This is particularly obvious in the zoomed insets shown in Fig. 6(a2)–(c2). This poor reconstruction generally appears when raw data are captured with low SNR. Fig. 6(d) shows the comparison in terms of the intensity graphs, in which the TIRF-piFPM gives the highest contrast. Moreover, TIRF-piFPM can be used to capture the dynamic changes of organelles in live cells. We demonstrate this ability by imaging live human bone osteosarcoma epithelial cells (U2OS) labeled with ATTO 647 N NHS ester (Sigma, excitation wavelength: 640 nm, emission wavelength: 670 nm). ATTO 647 N has excellent photostability and high fluorescence intensity as a mitochondrial marker in live-cell imaging. A high fluorescent quantum yield of the dye is crucial for live-cell imaging; however, the captured images sometimes contain a bright background signal. The deconvolved widefield image, shown in Fig. 7(a), is blurred with a strong fluorescent background. In the TIRF-piFPM
In the experiment, we first image the 100 nm red (excitation wavelength: 640 nm, emission wavelength: 670 nm) fluorescent beads using the proposed TIRF-piFPM system. Twenty orientations of the fine illumination patterns are generated, and 60 raw images are recorded with a camera exposure time of 50 ms. Four iterations are performed to reconstruct an SR image of the beads. Fig. 5(b) shows the reconstructed results obtained using our method, with clearly distinguished beads. The TIRF-SIM recovery (Fig. 5(c)) reconstructed with the same data stack is shown for comparison. As shown in Fig. 5(d)–(f), TIRF-piFPM provides more separated beads relative to TIRF-SIM. Fig. 5(j) and (k) show the comparison plots corresponding to the green and yellow lines shown in Fig. 5(d)–(f), respectively. The full-width at half-maximum (FWHM) of TIRF-piFPM, shown in Fig. 5(j), decreases from 228 nm to 108 nm, which is approximately a two-fold resolution enhancement compared with the widefield image. Moreover, the image obtained using TIRFpiFPM exhibits a higher contrast, as shown in Fig. 5(k). In Fig. 5(i), i.e., the margin field of view of the image shown in Fig. 5(c), the reconstructed TIRF-SIM beads appear to be deformed because of system aberration. In the first row of Fig. 5(i), the beads are stretched along the direction indicated by the gold arrow. The beads in the lower left of Fig. 5(i) exhibit a trailing phenomenon with the tails oriented along the direction indicated by the light green arrow. In contrast, in Fig. 5(h), TIRF-piFPM gives round beads owing to its aberration correction ability. We then apply TIRF-piFPM to image the mitochondrial network in fixed bovine pulmonary artery endothelial cells (BPAEC) dyed with Mito Tracker ® Red CMXRos (ThermoFisher, excitation wavelength: 561 nm, emission wavelength: 599 nm) to demonstrate its cell imaging capability. We reconstruct the images from a volume of 60 raw images. Mitochondria usually distribute around the nucleus (Fig. 6), providing en49
Q. Liu, Y. Chen and W. Liu et al.
Optics and Lasers in Engineering 123 (2019) 45–52
Fig. 5. Experimental results of 100 nm fluorescent beads. (a) Deconvolved widefield image. (b) and (c) Images reconstructed using TIRFpiFPM and TIRF-SIM, respectively. (d)–(f) The corresponding zoomed-in images in the central field of view of the images shown in (a)–(c) enclosed by green and yellow dashed lines, respectively. (g)–(i) The corresponding zoomedin images in the margin field of view of the images shown in (a)–(c) enclosed by gold, light green, and blue lines, respectively. (j) and (k) Normalized intensity profiles along the green and yellow lines shown in (d)–(f).
image (Fig. 7(c)), the background removal and resolution enhancement help view the mitochondrion more clearly. In Fig. 7(b), the background subtraction (BS) of the widefield image is done to compare with the image shown in Fig. 7(c). The results demonstrate the resolution improvement achieved using TIRF-piFPM. Fig. 7(d) shows the resolution improvement in a quantitative manner. In the recording process of the live-cell images, we capture a sequence of images lasting approximately 60 s. The exposure time is set to 50 ms per image, corresponding to an acquisition time of 2 s for 30 TIRF-piFPM images (10 illumination orientations) considering the response time of the camera. Fig. 7(e) shows the sequential process of mitochondrial fission, where the organelles, marked by a light red triangle, split into two mitochondria.
tional lateral information with a clear background. With the OTF correction scheme integrated into the iterative process, TIRF-piFPM achieves improved reconstruction results. To demonstrate its biological imaging performance, TIRF-piFPM is also applied to the dynamic imaging of the fission and morphological changes of mitochondria in live U2OS cells with an acquisition time of 2 s for each frame. Compared with TIRF-SIM, the advantages of TIRF-piFPM are its stronger robustness to system errors and noise owing to the optimization ability via iteration. The disadvantage is that the iterative optimization process requires more low-resolution images to recover the estimated object. A certain number of raw images is required to constrain the solution and promote convergence. In theory, three orientations of the illumination patterns (i.e., nine images) is sufficient to expand the support of the OTF in the Fourier space. However, more than three orientations of the illumination patterns are required for TIRF-piFPM to attain a better result than TIRF-SIM when recovered with the same image stack. Fig. 8 shows the reconstructed results of TIRF-SIM and TIRF-piFPM illuminated by patterns with a different number of orientations. From the simulation and experimental results of TIRF-SIM (Fig. 8(a) and 8(c)), we find that the reconstruction results are largely the same with the increase in the number of raw images. However, the images recovered using TIRF-piFPM (Fig. 8(b) and 8(d)) exhibit improved quality because a good convergence is achieved when the data are sufficient. Sixty raw
5. Discussion and conclusion In summary, we have developed a technique based on TIRF, referred to as TIRF-piFPM, to determine the 2D structure of objects with a lateral resolution of approximately 100 nm. With the concept of non-uniform illumination, the iterative algorithm piFP was proposed to obtain highfrequency information beyond the passband of the OTF for incoherent imaging. Inspired by SIM exploiting TIRF to improve contrast and resolution, piFP under the TIRF mode is proven effective in providing addi50
Q. Liu, Y. Chen and W. Liu et al.
Optics and Lasers in Engineering 123 (2019) 45–52
Fig. 6. Experimental results of mitochondria in fixed BPAEC. (a1)–(c1) In-focus images acquired by deconvoluting the data in the widefield, TIRF-piFPM reconstruction, and TIRF-SIM reconstruction, respectively. (a2)–(c2) The corresponding images obtained from the raw data stack when captured under 300 nm out of focus. The insets at the upper right corner of the images shown in (a1)–(c2) are the magnified views. (d) Normalized intensity profiles along the white lines in the insets of the images shown in (a2)–(c2).
Fig. 7. Mitochondria dynamics in live U2OS cells labeled with ATTO 647 N. (a) Deconvolved widefield image. (b) Widefield image with background subtraction. (c) Recovered TIRF-piFPM image. (d) Normalized intensity profiles along the green lines shown in (a)–(c). (e) Dynamic events occurring in the yellow box in the image shown in (c). The scale bars in (a) and (e) are 5 𝜇m and 2 𝜇m, respectively.
51
Q. Liu, Y. Chen and W. Liu et al.
Optics and Lasers in Engineering 123 (2019) 45–52 [4] Kai W. Super-resolution fluorescence microscopy using structured illumination. Super-Resolution Microscopy Techniques in the Neurosciences. Totowa, NJ: Humana Press; 2014. p. 133–65. [5] Hell SW, Wichmann J. Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy. Opt Lett 1994;19(11):780–2. [6] Klar TA, Jakobs S, Dyba M, Egner A, Hell SW. Fluorescence microscopy with di raction resolution barrier broken by stimulated emission. Proc. Natl Acad Sci U S A 2000;97:8206–10. [7] Betzig E, Patterson GH, Sougrat R, Lindwasser OW, Olenych S, Bonifacino JS, Davidson MW, Lippincottschwartz J, Hess HF. Imaging intracellular fluorescent proteins at nanometer resolution. Science 2006;313(5793):1642–5. [8] Manley S, Gillette JM, Patterson GH, Shroff H, Hess HF, Betzig E, Lippincott-Schwartz J. High-density mapping of single-molecule trajectories with photoactivated localization microscopy. Nat Methods 2008;5(2):155. [9] Rust MJ, Bates M, Zhuang X. Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM). Nat Methods 2010;3(10):793. [10] Dudok B, Barna L, Ledri M, Szabó SI, Szabadits E, Pintér B, Woodhams SG, Henstridge CM, Balla GY, Nyilas R. Cell-specific STORM super-resolution imaging reveals nanoscale organization of cannabinoid signaling. Nat. Neurosci. 2015;18(1):75–86. [11] Cragg GE, So PTC. Lateral resolution enhancement with standing evanescent waves. Opt Lett 2000;25(1):46–8. [12] Chung E, Kim D, Cui Y, Kim YH, So PT. Two-dimensional standing wave total internal reflection fluorescence microscopy: superresolution imaging of single molecular and biological specimens. Biophys J 2007;93(5):1747–57. [13] Fiolka R, Beck M, Stemmer A. Structured illumination in total internal reflection fluorescence microscopy using a spatial light modulator. Opt Lett 2008;33(14):1629–31. [14] Chen Y, Cao R, Liu W, Zhu D, Zhang Z, Kuang C, Liu X. Widefield and total internal reflection fluorescent structured illumination microscopy with scanning galvo mirrors. J Biomed Opt 2018;23(04):1. [15] Huang X, Fan J, Li L, Liu H, Wu R, Wu Y, Wei L, Mao H, Lal A, Xi P. Fast, long-term, super-resolution imaging with hessian structured illumination microscopy. Nat Biotechnol 2018;36(5):451. [16] Shroff SA, Fienup JR, Williams DR. Phase-shift estimation in sinusoidally illuminated images for lateral superresolution. J Opt Soc Am Opt Image Sci Vision 2009;26(2):413–24. [17] Wicker K. Non-iterative determination of pattern phase in structured illumination microscopy using auto-correlations in Fourier space. Opt Express 2013;21(21):24692–701. [18] Wicker K, Mandula O, Best G, Fiolka R, Heintzmann R. Phase optimisation for structured illumination microscopy. Opt Express 2013;21(2):2032–49. [19] Sahl SJ, Balzarotti F, Keller-Findeisen J, Leutenegger M, Westphal V, Egner A, Lavoie-Cardinal F, Chmyrov A, Grotjohann T, Jakobs S. comment on Extended-resolution structured illumination imaging of endocytic and cytoskeletal dynamics. Science 2016;352:527. [20] Dong S, Nanda P, Shiradkar R, Guo K, Zheng G. High-resolution fluorescence imaging via pattern-illuminated fourier ptychography. Opt Express 2014;22(17):20856–70. [21] Zheng G, Horstmeyer R, Yang C. Wide-field, high-resolution Fourier ptychographic microscopy. Nat Photonics 2013;7(9):739–45. [22] Ou X, Horstmeyer R, Yang C, Zheng G. Quantitative phase imaging via Fourier ptychographic microscopy. Opt Lett 2013;38(22):4845. [23] Yang C, Zheng G, Horstmeyer R, Ou X. High numerical aperture fourier ptychography: principle, implementation and characterization. Opt Express 2015;23(3):3472–91. [24] Williams A, Chung J, Ou X, Zheng G, Rawal S, Ao Z, Datar R, Yang C, Cote R. Fourier ptychographic microscopy for filtration-based circulating tumor cell enumeration and analysis. J Biomed Opt 2014;19(6):066007. [25] Tian L, Li X, Ramchandran K, Waller L. Multiplexed coded illumination for Fourier ptychography with an LED array microscope. Biomed Opt Express 2014;5(7):2376. [26] Ou X, Zheng G, Yang C. Embedded pupil function recovery for Fourier ptychographic microscopy. Opt Express 2014;22(26):4960–72. [27] Yeh LH, Dong J, Zhong J, Tian L, Chen M, Tang G, Soltanolkotabi M, Waller L. Experimental robustness of Fourier ptychography phase retrieval algorithms. Opt Express 2015;23(26):33214. [28] Sun J, Chen Q, Zhang Y, Zuo C. Efficient positional misalignment correction method for Fourier ptychographic microscopy. Biomed Opt Express 2016;7(4):1336. [29] Bian SDZ, Zheng G. Adaptive system correction for robust Fourier ptychographic imaging. Opt Express 2013;21(26):32400. [30] Kuang C, Ma Y, Zhou R, Zheng G, Fang Y, Xu Y, Liu X, So PT. Virtual k-space modulation optical microscopy. Phys Rev Lett 2016;117:028102. [31] Manuel GS, Fienup JR. Phase retrieval with transverse translation diversity: a nonlinear optimization approach. Opt Express 2008;16(10):7264–78. [32] Chien-Hung L, Christopher B, Matthew O W, Nathan KJ, Jason W F. Phase retrieval using nonlinear diversity. Appl Opt 2013;52(10):D92–6. [33] Liu Q, Kuang C, Fang Y, Xiu P, Li Y, Wen R, Liu X. Effect of spatial spectrum overlap on fourier ptychographic microscopy. J Innov Opt Health Sci 2017;10(02):1641004. [34] Chen Y, Liu W, Zhang Z, Zheng C, Huang Y, Cao R, Zhu D, Xu L, Zhang M, Zhang Y-H. Multi-color live-cell super-resolution volume imaging with multi-angle interference microscopy. Nat Commun 2018;9(1):4818.
Fig. 8. Data redundancy analysis of TIRF-piFPM by simulation and experiment. Images reconstructed using (a) TIRF-SIM and (b) TIRF-piFPM under pattern illumination with a different number of orientations by simulation. Upper left: 3 orientations; upper right: 5 orientations; lower right: 10 orientations; lower left: 20 orientations. (c)–(d) The corresponding experimental results of beads.
images (illumination patterns with 20 orientations) are sufficient to attain a convergence solution. The data redundancy requirement of TIRF-piFPM may slow down its imaging speed for living cells. There is always a trade-off between the image quality and the imaging speed. Techniques in signal processing and optimization can be leveraged to facilitate the convergence and improve the imaging speed in the future. With the good performance and potential of increasing the imaging speed, we believe the proposed method will be an excellent addition to the TIRF-family microscopes, enabling wide applicability across diverse specimens in capturing live events in cells. Acknowledgments National Natural Science Foundation of China (NSFC) (61827825, 61735017); the Fundamental Research Funds for the Central Universities (2019XZZX003-06); Natural Science Foundation of Zhejiang province (LR16F050001); and Zhejiang Lab (No. 2018EB0ZX01). Disclosures The authors declare that there are no conflicts of interest related to this article. References [1] Gustafsson MGL. Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy. J Microsc 2010;198(2):82–7. [2] Kner P, Chhun B, Griffis E, Winoto L, Gustafsson M. Super-resolution video microscopy of live cells by structured illumination. Nat Methods 2009;6(5):339. [3] Cogswell CJ. Doubling the lateral resolution of wide-field fluorescence microscopy using structured illumination. In: Proceedings of SPIE - The International Society for Optical Engineering, 3919; 2000. p. 141–50.
52