- Email: [email protected]
A nonlinear filtering algorithm for denoising HR(S)TEM micrographs
Ultramicroscopy 151 (2015) 62–67
Contents lists available at ScienceDirect
Ultramicroscopy journal homepage: www.elsevier.com/locate/ultramic
A nonlinear filtering algorithm for denoising HR(S)TEM micrographs$ Hongchu Du a,b,c,n a b c
Ernst Ruska-Centre for Microscopy and Spectroscopy with Electrons, Jülich Research Centre, Jülich, 52425, Germany Central Facility for Electron Microscopy (GFE), RWTH Aachen University, Aachen 52074, Germany Peter Grünberg Institute, Jülich Research Centre, Jülich 52425, Germany
art ic l e i nf o
a b s t r a c t
Article history: Received 25 August 2014 Received in revised form 1 November 2014 Accepted 6 November 2014 Available online 17 November 2014
Noise reduction of micrographs is often an essential task in high resolution (scanning) transmission electron microscopy (HR(S)TEM) either for a higher visual quality or for a more accurate quantification. Since HR(S)TEM studies are often aimed at resolving periodic atomistic columns and their non-periodic deviation at defects, it is important to develop a noise reduction algorithm that can simultaneously handle both periodic and non-periodic features properly. In this work, a nonlinear filtering algorithm is developed based on widely used techniques of low-pass filter and Wiener filter, which can efficiently reduce noise without noticeable artifacts even in HR(S)TEM micrographs with contrast of variation of background and defects. The developed nonlinear filtering algorithm is particularly suitable for quantitative electron microscopy, and is also of great interest for beam sensitive samples, in situ analyses, and atomic resolution EFTEM. & 2014 Elsevier B.V. All rights reserved.
Keywords: HR(S)TEM Noise reduction Denoising Filtering Nonlinear filter Image processing
1. Introduction Aberration-corrected high resolution (scanning) transmission electron microscopy (HR(S)TEM) enables quantitatively imaging atomic structures of condensed matters at sub-angstrom resolution [1,2]. Nowadays, electron microscopes are widely equipped with CCD cameras. HR(S)TEM micrographs recorded from a CCD camera tend to be degraded by noise. A low signal-to-noise ratio (SNR) often makes the accurate quantification difficult. The SNR can be improved by either increasing the signal intensity or decreasing the noise level. When it is possible, the increase of signal intensity is of higher priority than the decrease of noise level in achieving high SNR. However, for the majority of beam sensitive samples or in situ analyses, the increase of signal intensity may not be practical. Under these circumstances, any improvement of the SNR through noise reduction by filtering is incredibly valuable. There are several important sources of noise in a micrograph [3]: (i) quantum noise (shot-noise) of electron beam; (ii) dark current noise from thermally generated electrons; (iii) the socalled Fano noise from electron–photon and photon–electron conversions; (iv) read-out noise from electronic devices to read ☆ It is a great pleasure to dedicate this paper to Prof. Harald Rose in celebration of his 80th birthday. n Correspondence address: Ernst Ruska-Centre for Microscopy and Spectroscopy with Electrons, Jülich Research Centre, Jülich 52425, Germany. fax: þ49 2461 61 6444. E-mail address: [email protected]
http://dx.doi.org/10.1016/j.ultramic.2014.11.012 0304-3991/& 2014 Elsevier B.V. All rights reserved.
the image from a CCD. The Poisson statistics would apply for the first three sources of noise, whereas Gaussian for the read-out noise. The dark current noise and the read-out noise are independent of beam intensity. The Fano noise is empirically described as a function of photon energy, while the shot noise is proportional to the square root of the number of recorded electrons per pixel. At low to moderate electron dose, the shot-noise tends to dominate the noise in electron micrographs because of its dependence on the beam intensity. For more than 10 electrons detected, the Poisson distributed shot-noise appears to approach a Gaussian distribution with its standard deviation proportional to the square root of the number of detected electrons in each pixel. Therefore it usually is valid to assume an experimental electron micrograph (Iexp ) as a certain theoretically predicted image (Ith ) from a specific model plus uncorrelated additive Poisson or Gaussian (Inoise , σ 2 = Ith ) noise for simplification, so that
Iexp = Ith + Inoise
(1)
Noise reduction can be considered as an inverse process to obtain an estimated theoretical micrograph (Iest ) by applying a filter to the Iexp :
Iest = Iexp ⊗ F
(2)
The noise reduction algorithms in general can be categorized into spatial and temporal filtering [4]. Frequently used spatial filters for denoising HR(S)TEM images include Bragg filter [5], Wiener filter
H. Du / Ultramicroscopy 151 (2015) 62–67
[6–9], and Gaussian filter [10,11]. Whereas the simplest method of temporal filtering is frame averaging [4]. Registration of frames to one another is often essential before frame averaging to account for drift of the sample between frames. Both rigid [12] and nonrigid [13,14] registration methods have been reported for frame averaging to obtain high SNR images. Because HR(S)TEM studies are often aimed at resolving periodic atomistic columns and their non-periodic deviation at defect areas, it is important to develop a noise reduction algorithm that can simultaneously handle both periodic and non-periodic features properly. It is possible, often justified, to find out how complete the noise is removed and how well the periodic and non-periodic atomistic information is preserved by inspecting the difference (Idiff ) between the recorded Iexp and the denoised Iest images:
Idiff = Iexp − Iest
63
and its estimation Iest (Fig. 2b). An approximate solution was given in the literature [7] as
F wiener ≈
∣Ith ∣2 − ∣Inoise ∣2 ∣Iexp ∣2
(5)
However, artifacts are found in the Wiener filtered image when the intensity of background varies and non-periodic feature is present (Fig. 2d). To solve these problems, a nonlinear filtering algorithm has been developed, which can efficiently reduce noise in HR(S)TEM micrographs without noticeable artifacts even for contrast of variation of background and defects. Moreover, peak position and intensity can be more accurately determined from the nonlinear filtered images.
(3)
The Idiff actually is estimated noise. In this work, simulated instead of experimental images were used as the first testing ground in order to justify the performance of filters by exactly knowing the true signal, Ith , so that an error image (Ierr ) can be obtained:
Ierr = Ith − Iest
(4)
2. Tests of Gaussian and Wiener filters A super cell of MgO with terrace and an edge dislocation with ¯ 〉 was constructed (Fig. 1a). A HRTEM Burgers vector of a/2 〈110 image of the super cell with size of 512 512 pixels (0.01 nm/ pixel) was simulated using the optimized FEI Titan 80–300 parameters at 300 kV under a negative spherical aberration (Cs) imaging (NCSI) condition (Fig. 1b), which is considered to be Ith . The maximum intensity of the noise-free image was normalized to 100. Poisson noise was included so that at each pixel the standard deviation of noise is proportional to the square root of the intensity, which results in the maximum SNR that is about 10. The image with Poisson noise included (Fig. 1c) is used as experimental image (Iexp ). Gaussian low-pass filtering is one of the simplest ways to reduce high spatial frequency noise in HR(S)TEM micrographs [15,16]. Fig. 2a shows a filtered image of the testing HRTEM image of the MgO super cell (Fi. 1c) by convoluting a Gaussian kernel (s ¼2 pixels and kernel size of 5 5 pixels). The peak attenuation is evidently seen in the error image (Fig. 2c), which hinders faithful quantification of the peak intensity. Wiener filter can effectively reduce peak attenuation by minimizing the summed square differences between the true signal Ith
a
b
3. Nonlinear filtering algorithm Reduction of peak attenuation of 1D spectra has been reported by adding low-pass filtered residuals to the original low-pass filter output [17]. The nonlinear filtering algorithm described in this paper is by adding residuals of Wiener filtering (Fw ) to the output of low-pass filtering (Flp ), or vice versa through an iterative process. Fig. 3 shows the flowchart of the nonlinear filtering algorithm. At each iteration, the output is used as input for the next iteration. The mathematic form of the nonlinear filter (Fnl ) can be described as N
Fnl =
∏ (Flp, i + F w, i
− Flp, i ·F w, i )
i=1
(6)
Provided that all Flp, i and Fw, i are known, the algorithm can be time-efficiently designed and implemented so that the Fnl is directly calculated without an iterative process. The cutoff frequency of the nonlinear filter is defined by the cutoff frequencies of lowpass filters.
4. Performance of the nonlinear filter 4.1. Simulated images Tests were made to investigate the performance of the nonlinear filter in noise reduction and quantification of the simulated HR(S)TEM micrographs. Fig. 4a shows the nonlinear filtered image of the testing HRTEM image of the MgO super cell (Fig. 1c). No noticeable peak attenuation and artifacts are seen in the error
c
c a
b
¯ 〉 (Mg: orange, O: blue), (b) simulated noise-free HRTEM image under a Fig. 1. (a) A super cell of MgO with terrace and an edge dislocation with Burgers vector of a/2 〈110 negative-Cs-imaging condition with the maximum of intensity normalized to 100 (512 512 pixels), and (c) with Poisson noise included. (For interpretation of the references to color in this figure caption, the reader is referred to the web version of this paper.)
64
H. Du / Ultramicroscopy 151 (2015) 62–67
a
b
c
d
Fig. 2. (a) Gaussian (s ¼2 pixels and kernel size of 5 5 pixels) and (b) Wiener filtered images of the simulated HRTEM image with Poisson noise included (Fig. 1c), (c) and (d) are error images of (a) and (b), respectively.
Start
XIN i=i+1 i
No
XOUT = XLPF + XWF
Yes Low-Pass Filter XIN XLPF
XOUT End
XDIFF = XIN − XLPF Wiener Filter XDIFF XWF XIN = XLPF + XWF Fig. 3. Flowchart of the iterative nonlinear filtering algorithm for denoising HR(S) TEM images.
image (Fig. 4b). For comparison, Fig. 4c shows the details at the dislocation core from the Poisson noise included (Sþ N), noise-free (S), nonlinear (NL), Wiener (W), and Gaussian low-pass (G) filtered images. All images are in the same scale, except for the image with noise (SþN) the scale is set for optimal contrast. The Wiener filtered image exhibits obvious artifacts around the dislocation core (Fig. 4c), whereas profile analysis reveals profound peak intensity attenuation in the Gaussian low-pass filtered image (Fig. 4d). In contrast, the nonlinear filtered image is more accurately representing the noise-free image even at the dislocation core. Noise reduction is aimed not only at improving the visual quality of HR(S)TEM micrographs, but also at improving the accuracy of image quantification. Images with 100 Gaussian peaks with s ¼3 pixels were calculated for testing the performance of the nonlinear algorithm in quantitative sense. Poisson noise was added based on the intensity of each pixel. The 13 13 pixels were used for each peak to perform least square fitting by a home-made package supplied within the Gatan GMS2 software based on the Levenberg-Marquardt technique from the MPFIT C code [20], a MINPACK-1 Least Squares Fitting Library in C [21]. The fitted peak position and intensity were compared to the numerically predetermined values for calculating the uncertainty. Table 1 presents a comparison of the 95% confidence level uncertainty of peak position (Ux and Uy ) and intensity (Uz ) at different SNRs. It clearly shows that both peak position and intensity have been more accurately determined after the nonlinear filtering. Moreover, it is worth mentioning that least square fitting often fails to reach convergent validity at low SNR (SNR ≤ 4 ). Under these conditions, noise reduction by filtering is necessary in order to quantify the peak position and intensity. Therefore, the developed nonlinear
H. Du / Ultramicroscopy 151 (2015) 62–67
b
c
d Counts (a.u.)
a
65
S+N S NL W G ErrNL ErrW ErrG
100 80 60 40 20 0
S+N
S
NL
W
G
0
20
Pixel
40
Fig. 4. (a) Nonlinear filtered image of the testing image (Fig. 1c); (b) error image; (c) colored images of the details of the dislocation core, Sþ N: Poisson noise included (Fig. 1c, white dot line: line profile marker), S: noise-free (Fig. 1b), NL: nonlinear, W: Wiener (Fig. 2b), and G: Gaussian low-pass (Fig. 2a) filtered images, respectively; (d) corresponding line profiles drawn from top to bottom along lines with 2 pixels in width as indicated in (c), the Err-data show the differences between the profiles from the simulated noise-free S and the respective denoised images. (For interpretation of the references to color in this figure caption, the reader is referred to the web version of this paper.)
Table 1 A comparison of the uncertainty (95% confidence) of peak position (Ux and Uy) and intensity (Uz) determined by least square fitting from the noisy and nonlinear filtered images at different SNR (defined by the standard deviation of Poisson noise at the peak maximum, approximately the square root of the peak maximum). SNR
10 8 6 4
Ux (pixel)
Uy (pixel)
Uz (%)
Noisy
Denoised
Noisy
Denoised
Noisy
Denoised
0.116 0.154 0.253 –
0.110 0.145 0.231 0.306
0.137 0.165 0.231 –
0.130 0.157 0.215 0.325
5.44 6.10 10.59 –
4.58 5.06 8.09 11.05
filtering algorithm is very suitable for quantitative electron microscopy. 4.2. Experimental images The following compares the performance of the nonlinear and Wiener filters in noise reduction of experimental HR(S)TEM micrographs. Fig. 5 shows the results of filtering a HAAD-STEM micrograph of SrTiO3 in [001] zone axis, which was taken from the edge of a lamella sample and shows a variation of background. The removed noise by the nonlinear filter and the Wiener filter is quite similar in the range of high spatial frequency as seen from the line
profiles of the residuals (Fig. 5e and f). However, an advantage of the nonlinear filter over the Wiener filter is that the low spatial frequency background is preserved. This is because that the low spatial frequency around zero is less touched in the nonlinear filter than the Wiener filter (Fig. 5g and h). The nonlinear filter also shows better performance in noise reduction of HRTEM micrographs than the Wiener filter. Fig. 6 compares the filtering results between the Wiener and the nonlinear filters on a NCSI HRTEM micrograph of SrTiO3 in [001] zone axis. Though there are no evident contrast variation of low spatial frequency background and no contrast of defects in the micrograph (Fig. 6a), the noise image from the Wiener filter (Fig. 6e) shows more features than that from the nonlinear filter (Fig. 6f). Since autocorrelation of uncorrelated Poisson and Gaussian additive noise ideally gives a delta-function, the noise from the nonlinear filter is evidently more close to ideal uncorrelated additive noise than that from the Wiener filter (Fig. 6g and h). The absence of artifacts is clearly an advantage of the nonlinear over the Wiener filter in denoising HRTEM micrographs. Moreover, the nonlinear filter can efficiently denoise images with a significant amount of noise. Fig. 7a shows an atomic-resolution EFTEM image formed using electrons that have undergone a Ti L 2,3 (450–470 eV) core–shell energy loss from SrTiO3 in [001] zone-axis using the Cc-corrected FEI Titan 60–300 PICO at the Ernst Ruska Centre, Jülich, Germany. The image is from an EFTEM image series recorded at 80 kV acceleration voltage with an
66
H. Du / Ultramicroscopy 151 (2015) 62–67
a
c
e
g
b
d
f
h
Fig. 5. (a) Experimental HAADF-STEM image of SrTiO3 in [001] zone-axis and (b) its FFT; (c) and (d) Wiener and nonlinear filtered images; (e, f) respective residual images overlaid with line profiles; (g, h) FFTs of the filtered images cropped from the center as indicated in (b).
Fig. 6. (a) Experimental HRTEM image of SrTiO3 in [001] zone-axis and (b) its FFT; (c) and (d) Wiener and iterative nonlinear filtered images, respectively; (e) and (f) the respective residual images; (g) and (h) autocorrelation images of the respective residual images.
energy window of 20 eV in steps of 20 eV for center-window energies between 390 eV and 610 eV using 20 s per frame and a current of 3 nA. Because of the delocalized nature of the transition potentials [18,19], the lattice fringe contrast in the elastic wave function generated by elastic, coherent scattering is preserved in the inelastic waves. However, the lattice contrast is barely visible in the raw experimental data (Fig. 7a) due to low electron count rates and high noise levels. Surprisingly, the lattice fringes are significantly enhanced and evidently seen in the denoised image
using the nonlinear filter (Fig. 7b). In addition, no evident features are found in the difference image (Fig. 7c). The autocorrelation of the difference image (Fig. 7d) appears to be very similar to a delta function as for that from the ideal uncorrelated noise. The present example clearly shows that the nonlinear filter can reliably and efficiently denoise images with a significant amount of noise, therefore, will be of great interest for beam sensitive samples, in situ analyses, and atomic resolution EFTEM.
H. Du / Ultramicroscopy 151 (2015) 62–67
Deutsche Forschungsgemeinschaft acknowledged.
67
(SFB
917)
is
kindly
References
Fig. 7. (a) Experimental atomic-resolution Ti L 2,3 (450–470 eV) EFTEM image of SrTiO3 in [001] zone-axis; (b) iterative nonlinear filtered image; (c) and (d) the residual image and its autocorrelation, respectively.
5. Conclusions A nonlinear filtering algorithm has been developed by iteratively adding Wiener filtered residuals to the low-pass filtered output or vice versa. The developed nonlinear algorithm overcomes peak intensity attenuation and artifacts of low-pass and Wiener filters so that it can simultaneously handle both periodic and non-periodic features properly. Moreover, the present results have shown that both peak position and intensity can be more accurately determined by the nonlinear filtering, which makes the nonlinear filter very suitable for quantitative electron microscopy. Furthermore, the nonlinear filter can reliably and efficiently denoise images even with a significant amount of noise, therefore, will be of particularly interest for beam sensitive samples, in situ analyses, and atomic resolution EFTEM. Though the presented examples are experimental HRTEM, HAADF-STEM, and EFTEM images, the nonlinear filtering algorithm appears to perform well in denoising other kinds of highresolution electron micrographs, such as BF-STEM and ABF-STEM micrographs, even when non-periodic deviations are present.
Acknowledgments The author greatly appreciates C.L. Jia for fruitful discussion and L. Houben for help in the EFTEM imaging with the PICO microscope. The author also thanks the reviewers for their extremely valuable comments and suggestions. Financial support from the
[1] C.L. Jia, K. Urban, Atomic-resolution measurement of oxygen concentration in oxide materials, Science 303 (5666) (2004) 2001–2004. http://dx.doi.org/ 10.1126/science.1093617. [2] P.E. Batson, N. Dellby, O.L. Krivanek, Sub-angstrom resolution using aberration corrected electron optics, Nature 418 (6898) (2002) 617–620. http://dx.doi. org/10.1038/nature00972. [3] J.M. Zuo, Electron detection characteristics of a slow-scan CCD camera, imaging plates and film, and electron image restoration, Microsc. Res. Tech. 49 (3) (2000) 245–268, URL 〈http://people.iom.cnr.it/carlino/Gladich/detector_zuo_ ccd.pdf〉. [4] J.S. Lim, Two-Dimensional Signal and Image Processing, Prentice-Hall, Inc., Upper Saddle River, NJ, USA, 1990. [5] A. De Jong, W. Coene, D. Van Dyck, Image processing of HRTEM images with non-periodic features, Ultramicroscopy 27 (1) (1989) 53–65. http://dx.doi.org/ 10.1016/0304-3991(89)90200-3. [6] L. Marks, Wiener-filter enhancement of noisy HREM images, Ultramicroscopy 62 (12) (1996) 43–52. http://dx.doi.org/10.1016/0304-3991(95)00085-2. [7] R. Kilaas, Optimal and near-optimal filters in high-resolution electron microscopy, J. Microsc. 190 (1–2) (1998) 45–51. http://dx.doi.org/10.1046/j.13652818.1998.3070861.x. [8] K. Ishizuka, P. Eilers, T. Kogure, Optimal noise filters in high-resolution electron microscopy, Microsc. Microanal. 13 (Supplement S02) (2007) 902–903. http://dx.doi.org/10.1017/S1431927607073060. [9] F. Lin, C. Jin, An improved wiener deconvolution filter for high-resolution electron microscopy images, Micron 50 (2013) 1–6. http://dx.doi.org/10.1016/j. micron.2013.03.005. [10] O.L. Krivanek, M.F. Chisholm, V. Nicolosi, T.J. Pennycook, G.J. Corbin, N. Dellby, M.F. Murfitt, C.S. Own, Z.S. Szilagyi, M.P. Oxley, S.T. Pantelides, S.J. Pennycook, Atom-by-atom structural and chemical analysis by annular dark-field electron microscopy, Nature 464 (7288) (2010) 571–574. http://dx.doi.org/10.1038/ nature08879. [11] O.L. Krivanek, M.F. Chisholm, M.F. Murfitt, N. Dellby, Scanning transmission electron microscopy: Albert crewe's vision and beyond, Ultramicroscopy 123 (2012) 90–98. http://dx.doi.org/10.1016/j.ultramic.2012.04.004. [12] K. Kimoto, T. Asaka, X. Yu, T. Nagai, Y. Matsui, K. Ishizuka, Local crystal structure analysis with several picometer precision using scanning transmission electron microscopy, Ultramicroscopy 110 (7) (2010) 778–782. http://dx. doi.org/10.1016/j.ultramic.2009.11.014. [13] B. Berkels, P. Binev, D.A. Blom, W. Dahmen, R.C. Sharpley, T. Vogt, Optimized imaging using non-rigid registration, Ultramicroscopy 138 (2014) 46–56. http: //dx.doi.org/10.1016/j.ultramic.2013.11.007. [14] A.B. Yankovich, B. Berkels, W. Dahmen, P. Binev, S. I. Sanchez, S.A. Bradley, A. Li, I. Szlufarska, P.M. Voyles, Picometre-precision analysis of scanning transmission electron microscopy images of platinum nanocatalysts, Nat. Commun. 5. http://dx.doi.org/10.1038/ncomms5155. [15] P.L. Galindo, S. Kret, A.M. Sanchez, J.-Y. Laval, A. Yez, J. Pizarro, E. Guerrero, T. Ben, S.I. Molina, The peak pairs algorithm for strain mapping from HRTEM images, Ultramicroscopy 107 (12) (2007) 1186–1193. http://dx.doi.org/10.1016/ j.ultramic.2007.01.019. [16] R. Erni, M. Rossell, C. Kisielowski, U. Dahmen, Atomic-resolution imaging with a sub-50-pm electron probe, Phys. Rev. Lett. 102(9). http://dx.doi.org/10.1103/ PhysRevLett.102.096101. [17] J. McNames, B. Goldstein, A nonlinear lowpass filter that eliminates peak attenuation, in: IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2002, vol. 2, IEEE, 2002, pp. II-1529. URL 〈http://ieeex plore.ieee.org/xpls/abs_all.jsp?arnumber ¼ 5744905〉. [18] K.W. Urban, J. Mayer, J.R. Jinschek, M.J. Neish, N.R. Lugg, L.J. Allen, Achromatic elemental mapping beyond the nanoscale in the transmission electron microscope, Phys. Rev. Lett. 110 (2013) 185507. http://dx.doi.org/10.1103/ PhysRevLett.110.185507. [19] B. Forbes, L. Houben, J. Mayer, R. Dunin-Borkowski, L. Allen, Elemental mapping in achromatic atomic-resolution energy-filtered transmission electron microscopy, Ultramicroscopy 147 (2014) 98–105. http://dx.doi.org/10.1016/j. ultramic.2014.07.002. [20] C.B. Markwardt, Non-linear least squares fitting in IDL with MPFIT, in: D.A. Bohlender, D. Durand, P. Dowler (Eds.), Astronomical Data Analysis Software and Systems XVIII, Vol. 411, ASP Conference Series, Astronomical Society of the Pacific, 2009, pp. 251-254, http://aspbooks.org/custom/publications/paper/ 411-0251.html. [21] J. Moré, The Levenberg-Marquardt algorithm: Implementation and theory, in: G.A. Watson (Ed.), Numerical Analysis, Vol. 630, Lecture Notes in Mathematics, Springer Berlin Heidelberg, 1978, pp. 105-116, isbn: 978-3-540-08538-6, http://dx.doi.org/10.1007/BFb0067700.
Contents lists available at ScienceDirect
Ultramicroscopy journal homepage: www.elsevier.com/locate/ultramic
A nonlinear filtering algorithm for denoising HR(S)TEM micrographs$ Hongchu Du a,b,c,n a b c
Ernst Ruska-Centre for Microscopy and Spectroscopy with Electrons, Jülich Research Centre, Jülich, 52425, Germany Central Facility for Electron Microscopy (GFE), RWTH Aachen University, Aachen 52074, Germany Peter Grünberg Institute, Jülich Research Centre, Jülich 52425, Germany
art ic l e i nf o
a b s t r a c t
Article history: Received 25 August 2014 Received in revised form 1 November 2014 Accepted 6 November 2014 Available online 17 November 2014
Noise reduction of micrographs is often an essential task in high resolution (scanning) transmission electron microscopy (HR(S)TEM) either for a higher visual quality or for a more accurate quantification. Since HR(S)TEM studies are often aimed at resolving periodic atomistic columns and their non-periodic deviation at defects, it is important to develop a noise reduction algorithm that can simultaneously handle both periodic and non-periodic features properly. In this work, a nonlinear filtering algorithm is developed based on widely used techniques of low-pass filter and Wiener filter, which can efficiently reduce noise without noticeable artifacts even in HR(S)TEM micrographs with contrast of variation of background and defects. The developed nonlinear filtering algorithm is particularly suitable for quantitative electron microscopy, and is also of great interest for beam sensitive samples, in situ analyses, and atomic resolution EFTEM. & 2014 Elsevier B.V. All rights reserved.
Keywords: HR(S)TEM Noise reduction Denoising Filtering Nonlinear filter Image processing
1. Introduction Aberration-corrected high resolution (scanning) transmission electron microscopy (HR(S)TEM) enables quantitatively imaging atomic structures of condensed matters at sub-angstrom resolution [1,2]. Nowadays, electron microscopes are widely equipped with CCD cameras. HR(S)TEM micrographs recorded from a CCD camera tend to be degraded by noise. A low signal-to-noise ratio (SNR) often makes the accurate quantification difficult. The SNR can be improved by either increasing the signal intensity or decreasing the noise level. When it is possible, the increase of signal intensity is of higher priority than the decrease of noise level in achieving high SNR. However, for the majority of beam sensitive samples or in situ analyses, the increase of signal intensity may not be practical. Under these circumstances, any improvement of the SNR through noise reduction by filtering is incredibly valuable. There are several important sources of noise in a micrograph [3]: (i) quantum noise (shot-noise) of electron beam; (ii) dark current noise from thermally generated electrons; (iii) the socalled Fano noise from electron–photon and photon–electron conversions; (iv) read-out noise from electronic devices to read ☆ It is a great pleasure to dedicate this paper to Prof. Harald Rose in celebration of his 80th birthday. n Correspondence address: Ernst Ruska-Centre for Microscopy and Spectroscopy with Electrons, Jülich Research Centre, Jülich 52425, Germany. fax: þ49 2461 61 6444. E-mail address: [email protected]
http://dx.doi.org/10.1016/j.ultramic.2014.11.012 0304-3991/& 2014 Elsevier B.V. All rights reserved.
the image from a CCD. The Poisson statistics would apply for the first three sources of noise, whereas Gaussian for the read-out noise. The dark current noise and the read-out noise are independent of beam intensity. The Fano noise is empirically described as a function of photon energy, while the shot noise is proportional to the square root of the number of recorded electrons per pixel. At low to moderate electron dose, the shot-noise tends to dominate the noise in electron micrographs because of its dependence on the beam intensity. For more than 10 electrons detected, the Poisson distributed shot-noise appears to approach a Gaussian distribution with its standard deviation proportional to the square root of the number of detected electrons in each pixel. Therefore it usually is valid to assume an experimental electron micrograph (Iexp ) as a certain theoretically predicted image (Ith ) from a specific model plus uncorrelated additive Poisson or Gaussian (Inoise , σ 2 = Ith ) noise for simplification, so that
Iexp = Ith + Inoise
(1)
Noise reduction can be considered as an inverse process to obtain an estimated theoretical micrograph (Iest ) by applying a filter to the Iexp :
Iest = Iexp ⊗ F
(2)
The noise reduction algorithms in general can be categorized into spatial and temporal filtering [4]. Frequently used spatial filters for denoising HR(S)TEM images include Bragg filter [5], Wiener filter
H. Du / Ultramicroscopy 151 (2015) 62–67
[6–9], and Gaussian filter [10,11]. Whereas the simplest method of temporal filtering is frame averaging [4]. Registration of frames to one another is often essential before frame averaging to account for drift of the sample between frames. Both rigid [12] and nonrigid [13,14] registration methods have been reported for frame averaging to obtain high SNR images. Because HR(S)TEM studies are often aimed at resolving periodic atomistic columns and their non-periodic deviation at defect areas, it is important to develop a noise reduction algorithm that can simultaneously handle both periodic and non-periodic features properly. It is possible, often justified, to find out how complete the noise is removed and how well the periodic and non-periodic atomistic information is preserved by inspecting the difference (Idiff ) between the recorded Iexp and the denoised Iest images:
Idiff = Iexp − Iest
63
and its estimation Iest (Fig. 2b). An approximate solution was given in the literature [7] as
F wiener ≈
∣Ith ∣2 − ∣Inoise ∣2 ∣Iexp ∣2
(5)
However, artifacts are found in the Wiener filtered image when the intensity of background varies and non-periodic feature is present (Fig. 2d). To solve these problems, a nonlinear filtering algorithm has been developed, which can efficiently reduce noise in HR(S)TEM micrographs without noticeable artifacts even for contrast of variation of background and defects. Moreover, peak position and intensity can be more accurately determined from the nonlinear filtered images.
(3)
The Idiff actually is estimated noise. In this work, simulated instead of experimental images were used as the first testing ground in order to justify the performance of filters by exactly knowing the true signal, Ith , so that an error image (Ierr ) can be obtained:
Ierr = Ith − Iest
(4)
2. Tests of Gaussian and Wiener filters A super cell of MgO with terrace and an edge dislocation with ¯ 〉 was constructed (Fig. 1a). A HRTEM Burgers vector of a/2 〈110 image of the super cell with size of 512 512 pixels (0.01 nm/ pixel) was simulated using the optimized FEI Titan 80–300 parameters at 300 kV under a negative spherical aberration (Cs) imaging (NCSI) condition (Fig. 1b), which is considered to be Ith . The maximum intensity of the noise-free image was normalized to 100. Poisson noise was included so that at each pixel the standard deviation of noise is proportional to the square root of the intensity, which results in the maximum SNR that is about 10. The image with Poisson noise included (Fig. 1c) is used as experimental image (Iexp ). Gaussian low-pass filtering is one of the simplest ways to reduce high spatial frequency noise in HR(S)TEM micrographs [15,16]. Fig. 2a shows a filtered image of the testing HRTEM image of the MgO super cell (Fi. 1c) by convoluting a Gaussian kernel (s ¼2 pixels and kernel size of 5 5 pixels). The peak attenuation is evidently seen in the error image (Fig. 2c), which hinders faithful quantification of the peak intensity. Wiener filter can effectively reduce peak attenuation by minimizing the summed square differences between the true signal Ith
a
b
3. Nonlinear filtering algorithm Reduction of peak attenuation of 1D spectra has been reported by adding low-pass filtered residuals to the original low-pass filter output [17]. The nonlinear filtering algorithm described in this paper is by adding residuals of Wiener filtering (Fw ) to the output of low-pass filtering (Flp ), or vice versa through an iterative process. Fig. 3 shows the flowchart of the nonlinear filtering algorithm. At each iteration, the output is used as input for the next iteration. The mathematic form of the nonlinear filter (Fnl ) can be described as N
Fnl =
∏ (Flp, i + F w, i
− Flp, i ·F w, i )
i=1
(6)
Provided that all Flp, i and Fw, i are known, the algorithm can be time-efficiently designed and implemented so that the Fnl is directly calculated without an iterative process. The cutoff frequency of the nonlinear filter is defined by the cutoff frequencies of lowpass filters.
4. Performance of the nonlinear filter 4.1. Simulated images Tests were made to investigate the performance of the nonlinear filter in noise reduction and quantification of the simulated HR(S)TEM micrographs. Fig. 4a shows the nonlinear filtered image of the testing HRTEM image of the MgO super cell (Fig. 1c). No noticeable peak attenuation and artifacts are seen in the error
c
c a
b
¯ 〉 (Mg: orange, O: blue), (b) simulated noise-free HRTEM image under a Fig. 1. (a) A super cell of MgO with terrace and an edge dislocation with Burgers vector of a/2 〈110 negative-Cs-imaging condition with the maximum of intensity normalized to 100 (512 512 pixels), and (c) with Poisson noise included. (For interpretation of the references to color in this figure caption, the reader is referred to the web version of this paper.)
64
H. Du / Ultramicroscopy 151 (2015) 62–67
a
b
c
d
Fig. 2. (a) Gaussian (s ¼2 pixels and kernel size of 5 5 pixels) and (b) Wiener filtered images of the simulated HRTEM image with Poisson noise included (Fig. 1c), (c) and (d) are error images of (a) and (b), respectively.
Start
XIN i=i+1 i
No
XOUT = XLPF + XWF
Yes Low-Pass Filter XIN XLPF
XOUT End
XDIFF = XIN − XLPF Wiener Filter XDIFF XWF XIN = XLPF + XWF Fig. 3. Flowchart of the iterative nonlinear filtering algorithm for denoising HR(S) TEM images.
image (Fig. 4b). For comparison, Fig. 4c shows the details at the dislocation core from the Poisson noise included (Sþ N), noise-free (S), nonlinear (NL), Wiener (W), and Gaussian low-pass (G) filtered images. All images are in the same scale, except for the image with noise (SþN) the scale is set for optimal contrast. The Wiener filtered image exhibits obvious artifacts around the dislocation core (Fig. 4c), whereas profile analysis reveals profound peak intensity attenuation in the Gaussian low-pass filtered image (Fig. 4d). In contrast, the nonlinear filtered image is more accurately representing the noise-free image even at the dislocation core. Noise reduction is aimed not only at improving the visual quality of HR(S)TEM micrographs, but also at improving the accuracy of image quantification. Images with 100 Gaussian peaks with s ¼3 pixels were calculated for testing the performance of the nonlinear algorithm in quantitative sense. Poisson noise was added based on the intensity of each pixel. The 13 13 pixels were used for each peak to perform least square fitting by a home-made package supplied within the Gatan GMS2 software based on the Levenberg-Marquardt technique from the MPFIT C code [20], a MINPACK-1 Least Squares Fitting Library in C [21]. The fitted peak position and intensity were compared to the numerically predetermined values for calculating the uncertainty. Table 1 presents a comparison of the 95% confidence level uncertainty of peak position (Ux and Uy ) and intensity (Uz ) at different SNRs. It clearly shows that both peak position and intensity have been more accurately determined after the nonlinear filtering. Moreover, it is worth mentioning that least square fitting often fails to reach convergent validity at low SNR (SNR ≤ 4 ). Under these conditions, noise reduction by filtering is necessary in order to quantify the peak position and intensity. Therefore, the developed nonlinear
H. Du / Ultramicroscopy 151 (2015) 62–67
b
c
d Counts (a.u.)
a
65
S+N S NL W G ErrNL ErrW ErrG
100 80 60 40 20 0
S+N
S
NL
W
G
0
20
Pixel
40
Fig. 4. (a) Nonlinear filtered image of the testing image (Fig. 1c); (b) error image; (c) colored images of the details of the dislocation core, Sþ N: Poisson noise included (Fig. 1c, white dot line: line profile marker), S: noise-free (Fig. 1b), NL: nonlinear, W: Wiener (Fig. 2b), and G: Gaussian low-pass (Fig. 2a) filtered images, respectively; (d) corresponding line profiles drawn from top to bottom along lines with 2 pixels in width as indicated in (c), the Err-data show the differences between the profiles from the simulated noise-free S and the respective denoised images. (For interpretation of the references to color in this figure caption, the reader is referred to the web version of this paper.)
Table 1 A comparison of the uncertainty (95% confidence) of peak position (Ux and Uy) and intensity (Uz) determined by least square fitting from the noisy and nonlinear filtered images at different SNR (defined by the standard deviation of Poisson noise at the peak maximum, approximately the square root of the peak maximum). SNR
10 8 6 4
Ux (pixel)
Uy (pixel)
Uz (%)
Noisy
Denoised
Noisy
Denoised
Noisy
Denoised
0.116 0.154 0.253 –
0.110 0.145 0.231 0.306
0.137 0.165 0.231 –
0.130 0.157 0.215 0.325
5.44 6.10 10.59 –
4.58 5.06 8.09 11.05
filtering algorithm is very suitable for quantitative electron microscopy. 4.2. Experimental images The following compares the performance of the nonlinear and Wiener filters in noise reduction of experimental HR(S)TEM micrographs. Fig. 5 shows the results of filtering a HAAD-STEM micrograph of SrTiO3 in [001] zone axis, which was taken from the edge of a lamella sample and shows a variation of background. The removed noise by the nonlinear filter and the Wiener filter is quite similar in the range of high spatial frequency as seen from the line
profiles of the residuals (Fig. 5e and f). However, an advantage of the nonlinear filter over the Wiener filter is that the low spatial frequency background is preserved. This is because that the low spatial frequency around zero is less touched in the nonlinear filter than the Wiener filter (Fig. 5g and h). The nonlinear filter also shows better performance in noise reduction of HRTEM micrographs than the Wiener filter. Fig. 6 compares the filtering results between the Wiener and the nonlinear filters on a NCSI HRTEM micrograph of SrTiO3 in [001] zone axis. Though there are no evident contrast variation of low spatial frequency background and no contrast of defects in the micrograph (Fig. 6a), the noise image from the Wiener filter (Fig. 6e) shows more features than that from the nonlinear filter (Fig. 6f). Since autocorrelation of uncorrelated Poisson and Gaussian additive noise ideally gives a delta-function, the noise from the nonlinear filter is evidently more close to ideal uncorrelated additive noise than that from the Wiener filter (Fig. 6g and h). The absence of artifacts is clearly an advantage of the nonlinear over the Wiener filter in denoising HRTEM micrographs. Moreover, the nonlinear filter can efficiently denoise images with a significant amount of noise. Fig. 7a shows an atomic-resolution EFTEM image formed using electrons that have undergone a Ti L 2,3 (450–470 eV) core–shell energy loss from SrTiO3 in [001] zone-axis using the Cc-corrected FEI Titan 60–300 PICO at the Ernst Ruska Centre, Jülich, Germany. The image is from an EFTEM image series recorded at 80 kV acceleration voltage with an
66
H. Du / Ultramicroscopy 151 (2015) 62–67
a
c
e
g
b
d
f
h
Fig. 5. (a) Experimental HAADF-STEM image of SrTiO3 in [001] zone-axis and (b) its FFT; (c) and (d) Wiener and nonlinear filtered images; (e, f) respective residual images overlaid with line profiles; (g, h) FFTs of the filtered images cropped from the center as indicated in (b).
Fig. 6. (a) Experimental HRTEM image of SrTiO3 in [001] zone-axis and (b) its FFT; (c) and (d) Wiener and iterative nonlinear filtered images, respectively; (e) and (f) the respective residual images; (g) and (h) autocorrelation images of the respective residual images.
energy window of 20 eV in steps of 20 eV for center-window energies between 390 eV and 610 eV using 20 s per frame and a current of 3 nA. Because of the delocalized nature of the transition potentials [18,19], the lattice fringe contrast in the elastic wave function generated by elastic, coherent scattering is preserved in the inelastic waves. However, the lattice contrast is barely visible in the raw experimental data (Fig. 7a) due to low electron count rates and high noise levels. Surprisingly, the lattice fringes are significantly enhanced and evidently seen in the denoised image
using the nonlinear filter (Fig. 7b). In addition, no evident features are found in the difference image (Fig. 7c). The autocorrelation of the difference image (Fig. 7d) appears to be very similar to a delta function as for that from the ideal uncorrelated noise. The present example clearly shows that the nonlinear filter can reliably and efficiently denoise images with a significant amount of noise, therefore, will be of great interest for beam sensitive samples, in situ analyses, and atomic resolution EFTEM.
H. Du / Ultramicroscopy 151 (2015) 62–67
Deutsche Forschungsgemeinschaft acknowledged.
67
(SFB
917)
is
kindly
References
Fig. 7. (a) Experimental atomic-resolution Ti L 2,3 (450–470 eV) EFTEM image of SrTiO3 in [001] zone-axis; (b) iterative nonlinear filtered image; (c) and (d) the residual image and its autocorrelation, respectively.
5. Conclusions A nonlinear filtering algorithm has been developed by iteratively adding Wiener filtered residuals to the low-pass filtered output or vice versa. The developed nonlinear algorithm overcomes peak intensity attenuation and artifacts of low-pass and Wiener filters so that it can simultaneously handle both periodic and non-periodic features properly. Moreover, the present results have shown that both peak position and intensity can be more accurately determined by the nonlinear filtering, which makes the nonlinear filter very suitable for quantitative electron microscopy. Furthermore, the nonlinear filter can reliably and efficiently denoise images even with a significant amount of noise, therefore, will be of particularly interest for beam sensitive samples, in situ analyses, and atomic resolution EFTEM. Though the presented examples are experimental HRTEM, HAADF-STEM, and EFTEM images, the nonlinear filtering algorithm appears to perform well in denoising other kinds of highresolution electron micrographs, such as BF-STEM and ABF-STEM micrographs, even when non-periodic deviations are present.
Acknowledgments The author greatly appreciates C.L. Jia for fruitful discussion and L. Houben for help in the EFTEM imaging with the PICO microscope. The author also thanks the reviewers for their extremely valuable comments and suggestions. Financial support from the
[1] C.L. Jia, K. Urban, Atomic-resolution measurement of oxygen concentration in oxide materials, Science 303 (5666) (2004) 2001–2004. http://dx.doi.org/ 10.1126/science.1093617. [2] P.E. Batson, N. Dellby, O.L. Krivanek, Sub-angstrom resolution using aberration corrected electron optics, Nature 418 (6898) (2002) 617–620. http://dx.doi. org/10.1038/nature00972. [3] J.M. Zuo, Electron detection characteristics of a slow-scan CCD camera, imaging plates and film, and electron image restoration, Microsc. Res. Tech. 49 (3) (2000) 245–268, URL 〈http://people.iom.cnr.it/carlino/Gladich/detector_zuo_ ccd.pdf〉. [4] J.S. Lim, Two-Dimensional Signal and Image Processing, Prentice-Hall, Inc., Upper Saddle River, NJ, USA, 1990. [5] A. De Jong, W. Coene, D. Van Dyck, Image processing of HRTEM images with non-periodic features, Ultramicroscopy 27 (1) (1989) 53–65. http://dx.doi.org/ 10.1016/0304-3991(89)90200-3. [6] L. Marks, Wiener-filter enhancement of noisy HREM images, Ultramicroscopy 62 (12) (1996) 43–52. http://dx.doi.org/10.1016/0304-3991(95)00085-2. [7] R. Kilaas, Optimal and near-optimal filters in high-resolution electron microscopy, J. Microsc. 190 (1–2) (1998) 45–51. http://dx.doi.org/10.1046/j.13652818.1998.3070861.x. [8] K. Ishizuka, P. Eilers, T. Kogure, Optimal noise filters in high-resolution electron microscopy, Microsc. Microanal. 13 (Supplement S02) (2007) 902–903. http://dx.doi.org/10.1017/S1431927607073060. [9] F. Lin, C. Jin, An improved wiener deconvolution filter for high-resolution electron microscopy images, Micron 50 (2013) 1–6. http://dx.doi.org/10.1016/j. micron.2013.03.005. [10] O.L. Krivanek, M.F. Chisholm, V. Nicolosi, T.J. Pennycook, G.J. Corbin, N. Dellby, M.F. Murfitt, C.S. Own, Z.S. Szilagyi, M.P. Oxley, S.T. Pantelides, S.J. Pennycook, Atom-by-atom structural and chemical analysis by annular dark-field electron microscopy, Nature 464 (7288) (2010) 571–574. http://dx.doi.org/10.1038/ nature08879. [11] O.L. Krivanek, M.F. Chisholm, M.F. Murfitt, N. Dellby, Scanning transmission electron microscopy: Albert crewe's vision and beyond, Ultramicroscopy 123 (2012) 90–98. http://dx.doi.org/10.1016/j.ultramic.2012.04.004. [12] K. Kimoto, T. Asaka, X. Yu, T. Nagai, Y. Matsui, K. Ishizuka, Local crystal structure analysis with several picometer precision using scanning transmission electron microscopy, Ultramicroscopy 110 (7) (2010) 778–782. http://dx. doi.org/10.1016/j.ultramic.2009.11.014. [13] B. Berkels, P. Binev, D.A. Blom, W. Dahmen, R.C. Sharpley, T. Vogt, Optimized imaging using non-rigid registration, Ultramicroscopy 138 (2014) 46–56. http: //dx.doi.org/10.1016/j.ultramic.2013.11.007. [14] A.B. Yankovich, B. Berkels, W. Dahmen, P. Binev, S. I. Sanchez, S.A. Bradley, A. Li, I. Szlufarska, P.M. Voyles, Picometre-precision analysis of scanning transmission electron microscopy images of platinum nanocatalysts, Nat. Commun. 5. http://dx.doi.org/10.1038/ncomms5155. [15] P.L. Galindo, S. Kret, A.M. Sanchez, J.-Y. Laval, A. Yez, J. Pizarro, E. Guerrero, T. Ben, S.I. Molina, The peak pairs algorithm for strain mapping from HRTEM images, Ultramicroscopy 107 (12) (2007) 1186–1193. http://dx.doi.org/10.1016/ j.ultramic.2007.01.019. [16] R. Erni, M. Rossell, C. Kisielowski, U. Dahmen, Atomic-resolution imaging with a sub-50-pm electron probe, Phys. Rev. Lett. 102(9). http://dx.doi.org/10.1103/ PhysRevLett.102.096101. [17] J. McNames, B. Goldstein, A nonlinear lowpass filter that eliminates peak attenuation, in: IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2002, vol. 2, IEEE, 2002, pp. II-1529. URL 〈http://ieeex plore.ieee.org/xpls/abs_all.jsp?arnumber ¼ 5744905〉. [18] K.W. Urban, J. Mayer, J.R. Jinschek, M.J. Neish, N.R. Lugg, L.J. Allen, Achromatic elemental mapping beyond the nanoscale in the transmission electron microscope, Phys. Rev. Lett. 110 (2013) 185507. http://dx.doi.org/10.1103/ PhysRevLett.110.185507. [19] B. Forbes, L. Houben, J. Mayer, R. Dunin-Borkowski, L. Allen, Elemental mapping in achromatic atomic-resolution energy-filtered transmission electron microscopy, Ultramicroscopy 147 (2014) 98–105. http://dx.doi.org/10.1016/j. ultramic.2014.07.002. [20] C.B. Markwardt, Non-linear least squares fitting in IDL with MPFIT, in: D.A. Bohlender, D. Durand, P. Dowler (Eds.), Astronomical Data Analysis Software and Systems XVIII, Vol. 411, ASP Conference Series, Astronomical Society of the Pacific, 2009, pp. 251-254, http://aspbooks.org/custom/publications/paper/ 411-0251.html. [21] J. Moré, The Levenberg-Marquardt algorithm: Implementation and theory, in: G.A. Watson (Ed.), Numerical Analysis, Vol. 630, Lecture Notes in Mathematics, Springer Berlin Heidelberg, 1978, pp. 105-116, isbn: 978-3-540-08538-6, http://dx.doi.org/10.1007/BFb0067700.