Software for automated acquisition of electron tomography tilt series

Software for automated acquisition of electron tomography tilt series

CHAPTER Software for automated acquisition of electron tomography tilt series 8 Guenter P. Resch* Nexperion e.U.—Solutions for Electron Microscopy,...

5MB Sizes 33 Downloads 227 Views

CHAPTER

Software for automated acquisition of electron tomography tilt series

8 Guenter P. Resch*

Nexperion e.U.—Solutions for Electron Microscopy, Vienna, Austria *Corresponding author: e-mail address: [email protected]

Chapter outline 1 Introduction......................................................................................................137 1.1 General challenges in acquisition of tilt series.......................................138 1.2 Special functions...............................................................................140 1.2.1 Montage tomography......................................................................140 1.2.2 Cryo-electron tomography................................................................140 1.2.3 Special tilt schemes........................................................................141 1.2.4 Batch tomography..........................................................................141 1.2.5 Continuous-tilting tomography.........................................................142 1.2.6 STEM tomography..........................................................................142 2 Academic software............................................................................................142 2.1 Leginon MSI-tomography....................................................................144 2.1.1 Hardware support...........................................................................144 2.1.2 Software architecture......................................................................145 2.1.3 Tilt series acquisition workflow.........................................................146 2.1.4 Output and processing....................................................................147 2.1.5 Conclusions....................................................................................148 2.2 SerialEM...........................................................................................148 2.2.1 Hardware support...........................................................................148 2.2.2 Software architecture......................................................................149 2.2.3 Installation and calibration...............................................................149 2.2.4 Tilt series acquisition workflow.........................................................151 2.2.5 Data output and processing.............................................................153 2.2.6 Conclusions....................................................................................154 2.3 UCSF Tomography..............................................................................154 2.3.1 Hardware support...........................................................................155 2.3.2 Software architecture......................................................................155 2.3.3 Installation and calibration...............................................................155 2.3.4 Tilt series acquisition workflow.........................................................156 2.3.5 Output and processing....................................................................157 2.3.6 Conclusion.....................................................................................158 Methods in Cell Biology, Volume 152, ISSN 0091-679X, https://doi.org/10.1016/bs.mcb.2019.05.002 © 2019 Elsevier Inc. All rights reserved.

135

136

CHAPTER 8 Automated acquisition of tilt series

3 Commercial software.........................................................................................158 3.1 Hitachi 3D tilt image acquisition.........................................................158 3.1.1 Hardware support...........................................................................158 3.1.2 Installation and calibration...............................................................158 3.1.3 Tilt series acquisition workflow.........................................................159 3.1.4 Output and processing....................................................................160 3.1.5 Conclusion.....................................................................................160 3.2 TEMography Recorder.........................................................................160 3.2.1 Hardware support...........................................................................160 3.2.2 Software architecture......................................................................161 3.2.3 Installation and calibration...............................................................161 3.2.4 Tilt series acquisition workflow.........................................................161 3.2.5 Output and processing....................................................................163 3.2.6 Conclusion.....................................................................................164 3.3 Thermo Scientific Tomography............................................................164 3.3.1 Hardware support...........................................................................164 3.3.2 Software architecture......................................................................165 3.3.3 Installation and calibration...............................................................165 3.3.4 Tilt series acquisition workflow.........................................................166 3.3.5 Output and processing....................................................................168 3.3.6 Conclusion.....................................................................................168 3.4 TVIPS EM-Tomo.................................................................................168 3.4.1 Hardware support...........................................................................168 3.4.2 Installation and calibration...............................................................169 3.4.3 Tilt series acquisition workflow.........................................................169 3.4.4 Output and processing....................................................................171 3.4.5 Conclusion.....................................................................................171 4 Considerations for choosing the software............................................................173 Acknowledgments..................................................................................................173 Conflict of interest.................................................................................................173 References............................................................................................................173

Abstract For automated acquisition of tilt series for electron tomography, software needs to handle complications such as movements of the sample in x/y and z, increased projected thickness at high tilt, specimen drift, etc. In addition, many applications require special functionality such as low dose acquisition, automated sequential (batch) tomography, or montage tomography. After reviewing how these difficulties can be addressed and a closer look at what advanced acquisition strategies are employed in biosciences, this chapter introduces acquisition software both developed in academia as well as by hardware vendors. It covers the hardware

1 Introduction

requirements and compatibility, the functional principle and workflow implemented, as well as what advanced functions are supported by the individual programs.

1 Introduction Electron tomography (ET) allows to obtain the 3D structure of a pleomorphic sample from a series of 2D transmission electron microscopy (TEM) images. It involves the acquisition of a series of projection images of a specimen with a TEM at numerous different tilt angles (Koning, Koster, & Sharp, 2018; M€uller-Reichert, Kiewisz, & Redemann, 2018; Weber, Wojtynek, & Medalia, 2019). Following the acquisition of a “tilt series,” different reconstruction algorithms (reviewed in Penczek, 2010 and Sorzano et al., 2017) are employed to calculate a three-dimensional reconstruction (tomogram) which in turn can be virtually sectioned, manually modeled, surface rendered, or processed further. Besides the lack of image detectors directly recording large areas in digital format and limitations in computer power to process the large number of micrographs, the need to manually acquire tilt series was a limiting factor in the early days of ET (Baumeister, Grimm, & Walz, 1999). The manual collection of tilt series is feasible—as demonstrated in Hoppe, Gassmann, Hunsmann, Schramm, and Sturm (1974), Knauer, Hegerl, and Hoppe (1983), and McEwen, Radermacher, Rieder, and Frank (1986)— but cumbersome and can suffer from the operator’s inaccuracy. It is also very expensive in terms of electron dose due to the slow nature of manual operation and hence unsuitable for dose sensitive samples (Koster, Chen, Sedat, & Agard, 1992). The advent of microprocessor-controlled TEM and the development of software able to control microscope and detectors to drive automated acquisition of tilt series under welldefined conditions was one key factor for opening the way for ET as a routine application. The Max-Planck-Institute of Biochemistry in Martinsried, Germany, (Dierksen, Typke, Hegerl, Koster, & Baumeister, 1992) and the University of California at San Francisco (Koster et al., 1992) were key centers of these early developments. This chapter discusses the complications software is facing in the acquisition process and outline how they can be addressed. Special applications in electron tomography will be introduced and discussed in the context of automated data acquisition. Regardless of the difficulties documenting something as fast moving as software in a book, current software solutions for automated acquisition of tilt series both developed in academia as well as by hardware manufacturers will be introduced. The focus here will be the underlying concepts, the hardware supported, and what special applications are implemented in each software package. This text largely covers the software versions available in early 2019. What is beyond the scope of this chapter is the hardware (column and detectors) required for acquisition of tilt series or any image processing downstream of the collection of the tilt series, such as reprojection, filtering, segmentation or other modeling approaches.

137

138

CHAPTER 8 Automated acquisition of tilt series

1.1 General challenges in acquisition of tilt series The automated acquisition of a series of projection images at discrete tilt angles comes with a number of challenges the software driving acquisition has to be capable of handling. It is important to be aware that good microscope alignments— mechanically and optically—are a critical prerequisite for automated data collection. First, sound alignments help to minimize the impact of some of the issues outlined below; second, the acquisition software needs to rely on commands sent to the microscope producing predictable results. •



Due to a lateral displacement of the goniometer’s (stage’s) tilt axis vs. the optical axis, deviations of the sample’s z height from the height of the tilt axis of the goniometer (eucentric height), and mechanical imperfections of the goniometer, the specimen will be shifted in x and y after each tilt (for details, see Liu et al., 2016; Zheng, Braunfeld, Sedat, & Agard, 2004). To what extent this displacement becomes an issue in the tilt series depends on the magnification being used, with higher magnifications being more critical: without compensation, only a small portion or none of the field of view might be covered throughout the entire tilt series, rendering it unusable for processing. Displacements in x/y can be measured by cross-correlating “tracking” images before and after tilting and/or calculated based on a mathematical model of the specimen moving in space, and/or predicted based on previously collected data. An offset can be corrected either with goniometer motor driven shift (maximum range, inaccurate), by using the beam shift coils above the sample in combination with the image shift (IS) or projector alignment (PLA; JEOL) coils below the sample to shift the beam out of the optical axis when passing through the specimen (fast, reproducible, but very limited in range), or a piezo drive. Ideally, all available mechanisms are combined to achieve the biggest range possible while maintaining high accuracy and favorable optical conditions. The same factors as described above also contribute to a displacement of the sample in z, causing differences in focus from one projection to the next. These deviations have a particularly strong impact in imaging techniques that rely on defocus dependent phase contrast, e.g., cryo-TEM, or high resolution studies. Displacements in z can also be calculated based on the modeled trajectory of the sample in space or predicted based on calibrations, but autofocusing is considered an essential step in most automated acquisition schemes for tilt series. First, the TEM defocus is typically calculated accurately and quickly from the known (calibrated) relationship between a beam-tilt induced image shift and the defocus (Koster, Van den Bos, & van der Mast, 1987). Subsequently, the defocus can be adjusted to the target value either by changing the objective lens current and/or by a goniometer movement in z. Working with the objective lens current is faster, more accurate and more reproducible than using the goniometer, but the larger the amount of defocus change, the more magnification changes, rotations, and beam shifts induced by the changed objective lens current will become an issue (Koster et al., 1992), especially under non-parallel illumination conditions. In

1 Introduction









scanning transmission electron microscopy (STEM; see Section 1.2.6), the beamtilt correlation method used in TEM is not applicable; the defocus is only measured by analyzing images from a through-focus series, e.g., via power spectra. “Backlash” is a property of the mechanical drive of all axes in a goniometer, affecting the motion in x, y and z, as well as tilting. It is caused by gaps between the gears within the goniometer, leading to lost motion and motion-direction dependent differences in positioning of the stage. This error can be significantly reduced by continually approaching coordinates from one defined direction of motion. If the opposite direction is required to approach a coordinate, backlash correction adds a minor overshooting and a subsequent counter-motion in the “correct” direction. In automated acquisition routines, backlash correction can be implemented for all axes of motion. Specimen drift is induced by multiple factors, including the residual movement of mechanical components of the stage, deflectors in the column, or the onset of irradiation with electrons. Besides obeying appropriate relaxation times for mechanics and electron optics, beam induced drift can be addressed by preexposing the sample to bring most of the drift to a halt before the actual exposure starts. On direct electron detectors and some CMOS cameras, drift regardless of its origin can be reduced by splitting up the exposure into sub-frames (dose fractionation, “movies”), subsequent alignment of the individual sub-frames (e.g., McLeod, Kowal, Ringler, & Stahlberg, 2017), and summing into one final image instead of acquiring a single integrated image blurred by drift. By tilting the specimen, the thickness projected along the optical axis increases significantly, leading to a loss of intensity and an increase of multiple scattering events at higher tilts: at 60° tilt, the effective thickness is increased by a factor of 2.0, at 70° by 2.9. The intensity loss can be compensated by either increasing exposure times or alternatively by working with a more focused beam. The noise added by multiple scattering events in thick samples/projections can be reduced by removing inelastically scattered electrons by zero loss filtering (Grimm et al., 1997), i.e., only electrons that suffered no energy loss when passing the sample can enter the slit of an in-column or post-column energy filter. A perfect tomogram would require a tilting range of 180°. However, shadowing of the beam by the grid bars and/or the specimen holder will occur on all routine instruments and/or with samples placed on standard EM grids at higher tilts, typically limiting the tilt range to 120–140°. This leads to an incomplete coverage of the frequency domain with information (missing wedge), affecting the quality of the final reconstruction predominantly into one spatial direction (for an example, see Mastronarde, 1997). This artifact can be reduced by tilting the specimen around two orthogonal axes, combining the resulting data sets, and reducing the missing wedge to a missing pyramid (double tilt/dual axis tomography; (Penczek, Marko, Buttle, & Frank, 1995). Typically, the second (beta) axis is not available or does not offer the precision or range required. Hence, double tilt tomography is usually implemented by rotating the specimen

139

140

CHAPTER 8 Automated acquisition of tilt series



by 90°, either using more less accurate rotation mechanisms or manual rotation, and tilting around the primary (alpha) axis once again. The challenge for the operator and software here is the precise relocation of the site of the first tilt series, especially on dose sensitive and/or low contrast specimens, after rotating the grid by an ill-defined amount and translation. A problem closely related to automated data acquisition and recent advances in hardware, such as detectors with many pixels and dose fractionation, or rapid acquisition schemes (Chreifi, Chen, Metskas, Kaplan, & Jensen) are the transport, the storage, and the processing of dramatically increasing amounts of raw data. In parallel with introducing these new technologies, the IT infrastructure handling the data has to be prepared accordingly (see also Baldwin et al., 2018).

There are further complications in automated tilt series acquisition that are sample specific. This includes the shrinkage of resin embedded material in the electron beam calling for pre-irradiation, to prevent acquisition of a tilt series of a specimen undergoing deformation. Another prominent example is the dose sensitivity and the low contrast of unstained frozen hydrated specimens and, that requires the acquisition software to handle low dose imaging (see Section 1.2.2) and to deal with generally noisy images, e.g., for tracking or autofocusing.

1.2 Special functions 1.2.1 Montage tomography To cover larger areas while maintaining resolution, some of the programs introduced in this chapter implement “montage tomography” (O’Toole, van der Heide, Richard McIntosh, & Mastronarde, 2018): instead of acquiring a single image per tilt, multiple tiles in x and y are acquired and merged at a later stage, yielding a larger field of view. Typically, this approach is only used for dose tolerant specimens as the inevitable overlap of the illuminated areas of the individual exposures would lead to a rapid accumulation of dose in parts of the montaged area.

1.2.2 Cryo-electron tomography Dose sensitive specimens such as frozen hydrated samples for cryo-TEM require special strategies to minimize the dose inflicted to the region of interest (ROI) except for the final exposure (Baumeister et al., 1999). To eliminate the dose arising from manually switching and optimizing microscope settings, focusing, etc., all modern TEM platforms include a “low dose” (Thermo Scientific/FEI, Hitachi) or “minimal dose” (JEOL) interface. This software typically implements different optics states for searching the sample, focusing, and the acquisition: by automatically switching between these modes with the beam blanked above the specimen, the exposure overhead of manual operation can be omitted. Furthermore, the mode for dose expensive operations like focusing is shifted by typically one to a few micrometers away from the region of interest, providing a result of acceptable accuracy but not inflicting any dose to the ROI at all if the illuminated area for focusing was sufficiently small. For acquisition of cryo-tilt series of flat specimens, the same strategy

1 Introduction

can be applied; the image shift for the focus area should be parallel to the tilt axis, however, to focus and acquire at the same physical height of the specimen. Another issue of importance when dealing with dose sensitive specimens is prediction of the dose over vacuum that will accumulate over the course of a tilt series at current settings and a dose calculation at the end of the acquisition. Typically, a cumulative ˚ 2) is aimed for (Koning et al., 2018), that dose of less than 10,000 e /nm2 (100 e /A has to be distributed over the individual projection images. When working with unstained cryo-specimens, the use of direct electron detectors, phase plates (e.g., Danev, Buijsse, Khoshouei, Plitzko, & Baumeister, 2014), and zero loss energy filtering (Grimm et al., 1997) can be beneficial to get the most out of the limited incident dose, requiring the integration of the respective hardware into the acquisition software.

1.2.3 Special tilt schemes The order in which individual projection images of dose tolerant samples are acquired is not of significance, as the information in these samples hardly degrades. A continuous/unidirectional tilt scheme starting at one extreme tilt angle and collecting projection images in one sweep to the other extreme angle in typically used. Acquiring neighboring projections consecutively also minimizes complications in tilt series alignment from the mass loss of plastic specimens. In contrast to that, the high resolution information in frozen hydrated or other dose sensitive samples is lost very quickly upon exposure to the beam; hence, the earliest projections should be taken under the most favorable conditions at low tilt, where the projected thickness of the sample is minimal. Different schemes for data collection can be employed: “bidirectional tilt schemes” divide the tilt series into two branches, both starting at 0° and tilting up consecutively in both directions. Optimal high resolution information at low tilt/minimal projected thickness is only collected in the first branch, however, and the dose difference between the starting images of the first and the second branch can cause alignment problems. This is addressed by dedicated tilt schemes for cryo-ET, e.g., the dose symmetric tilt scheme by Hagen, Wan, and Briggs (2017), that typically starts at 0° tilt and then alternates between increasingly high positive and negative tilts.

1.2.4 Batch tomography For many studies, the acquisition of a just a few tilt series is not sufficient, numerous tomograms are required: examples include biological questions that require the comparison of many tilt series for unambiguous interpretation (Briegel et al., 2008), serial section tomography where the same structure is imaged on consecutive serial sections (O’Toole et al., 2018), and the acquisition of sufficient raw data for subtomogram averaging (Schur et al., 2015). Localizing a feature of interest, manually starting a tilt series, and resuming the search for interesting positions once the tilt series has completed is uneconomical due to the extended waiting times. Batch tomography (automated sequential acquisition) as implemented in a number of the programs introduced in this chapter helps to make optimal use of both operator as well as instrument time: numerous points of interest (including stage positions, reference

141

142

CHAPTER 8 Automated acquisition of tilt series

images, and acquisition parameters) are defined during working hours, the acquisition is started subsequently and can continue unattended into the night or over several days.

1.2.5 Continuous-tilting tomography The time required to acquire a single tilt series, typically 20–60 min, is a major bottleneck in many applications and binds significant instrument time even if staff time is freed up by using an unattended acquisition approach (see Section 1.2.4). Advances in hardware—a more eucentric stage on the Thermo Scientific Krios and detectors with a very fast readout rate—allowed Chreifi et al. (2019) recently to implement continuous-tilting cryo-tomography. In their experiment, the camera was recording a long movie while the target was exposed to the electron beam and the goniometer tilted continuously. Tracking was omitted completely, and even at high magnifications covering a field of view of  400  400 nm2, only 1/3 of the field of view was lost with this stage. Presumably due to vibrations originating from the continuous operation of the goniometer’s tilt motor, the resolution of the resulting tomograms was limited, however.

1.2.6 STEM tomography In STEM, the sample is scanned by a focused electron beam (probe); for each point, signals of unscattered or scattered electrons can be picked up by detectors below the specimen, covering different angles around the optical axis. This approach has a number of advantages over bright-field TEM, including improved contrast without defocusing and, with an optimized probe setup, a depth of field that is significantly increased over TEM (Biskupek, Leschner, Walther, & Kaiser, 2010). Furthermore, STEM allows for dynamic focusing if required: the STEM focus is progressively changed during acquisition of one image to keep a tilted specimen in its entirety in focus (a 1 μm field of view tilted to 60° exhibits a 860 nm difference in z). The potential for STEM in tomography of biological specimens was first demonstrated in stained, plastic embedded sections (Hohmann-Marriott et al., 2009; see also Walther et al., 2018) and later also in frozen hydrated material (Wolf, Houben, & Elbaum, 2014) with a theoretical analysis in Rez, Larsen, and Elbaum (2016).

2 Academic software Numerous examples (Chreifi et al., 2019; Hagen et al., 2017; Koster et al., 1992; Mastronarde, 2005; Nickell et al., 2005; Suloway et al., 2005; Zheng, Keszthelyi, et al., 2006) show that academia always was and is the driving force behind automating the acquisition of EM data: the inherent interest in developing at the forefront of technology and short decision making processes accelerate the incorporation of the newest hardware and novel, innovative software features. All software packages introduced in this section are developed in academic institutions and available for download for free, license restrictions for use of these software outside academia can apply. While the total cost of ownership is typically

2 Academic software

significantly lower in case of free software products, it is not necessarily zero due to investments in hardware or services related to the software. Open source software (as opposed to closed source software) is software with source code made available by the programmer for inspection or modification. This allows experts to understand in detail how a piece of software is performing a task and also enables them to build on the existing code. Different licenses governing how the code can be used exist, e.g., enforcing that derivatives are also open source. While open source software is often associated with free software, one does not necessarily imply the other: free software is not necessarily open source, and open source software is not necessarily free in distributed form or as a service. Most of the programs introduced here are open source, but currently, the opportunity to work with the source code seems less accepted in case of EM/ET data acquisition software than in related fields. Another benefit of acquisition software developed in academia can be the support of a wide range of microscope platforms and detectors, providing a manufacturerindependent uniform user experience and workflow. Crucial factors for EM software development in academia are the conditions and the extent to which hardware manufacturers are providing application programming interfaces (APIs) allowing third party software to read from and control their hardware. In the case of software for automated acquisition of TEM tilt series, this concerns both microscopes and any kind of detector (cameras, analytical detectors). For all Thermo Scientific/FEI instruments since the Tecnai platform, instrument owners can purchase a license for the “TEM Scripting Adapter” which unlocks this programming interface. The newest versions of the Thermo Scientific microscope software feature “Advanced TEM Scripting,” adding functionality for Thermo Scientific direct electron detectors, phase plates, and the automated aperture system. The software using either API has to be run on the microscope computer. With detailed documentation and sample applications in several programming languages provided, the standard TEM Scripting is quite easily accessible. To access a JEOL microscope (models with the TEMCON or TEMCENTER user interface) from third party software, the “TEM External” component is used. It is installed on a computer with a network connection to the microscope PC (typically, the camera computer) and provides an API there. The files required to develop software are not available as a product but for free upon request, somewhat at the cost of good documentation. Communication with newer Hitachi instruments can be established via a socket interface employing an unpublished communications protocol. For Zeiss Libra microscopes, there was also an extremely well documented API available, but none of the academic software solutions described in this chapter provides support for this obsolete platform. The commercial TVIPS EM-Tomo (Section 3.4) and the program by Liu et al. (2016) (in combination with Gatan Digital Micrograph), however, do. The situation for Thermo Scientific/FEI detectors (cameras and STEM) is largely the same as for microscope access—the TEM Scripting Adapter is required.

143

144

CHAPTER 8 Automated acquisition of tilt series

Originally, basic camera access was provided in the standard scripting adapter. In order to support newer cameras, an improved interface is provided in the “Advanced Scripting” interface described earlier, including third party access to acquire counting mode images or dose fractionation movies from the Falcon detectors. The “CamC” interface of TVIPS cameras can be accessed directly via a COM object, with the limitation that only one program can bind to it (e.g., EM-Menu and SerialEM cannot access a TVIPS camera concurrently). This interface is available for all TVIPS cameras, the documentation is available upon request from the manufacturer (Andreas Wisnet, personal communication). In addition, EM-Menu “Extended” provides a scripting option that is also implemented as a COM object. The files and the documentation required to access the interfaces most other camera manufacturers provide can only be obtained only under a non-disclosure agreement. Programs that are not covered in detail due to a shortage of space are the TOM Toolbox (Nickell et al., 2005), developed at the Max-Planck-Institute for Biochemistry in Martinsried, Germany, but no longer actively maintained,a and the “fully mechanically controlled” approach by Liu et al. (2016).

2.1 Leginon MSI-tomography Leginon (http://www.leginon.org; Leginon, n.d.; Anchi Cheng, personal communication) is a system designed for automated image data collection from cryo-TEM, developed at the laboratory of Bridget Carragher and Clint Potter, until 2015 at Scripps Research Institute (La Jolla, CA, USA) and now at the New York Structural Biology Center (New York City, NY, USA). Initially, the “new” Leginon software (Suloway et al., 2005) was designed for data collection for single particle analysis and 2D crystallography with the goal of improving the throughput for macromolecular microscopy. It was extended in 2009 with a new application for ET (Suloway et al., 2009). Leginon is an open source software package released under the Apache License 2.0. It is available from the developer’s website in a stable, a beta, and a development version.

2.1.1 Hardware support The whole Leginon system can interact with TEM from Thermo Scientific/FEI, namely, the Tecnai, Titan and Talos platforms with TEM Scripting, a number of JEOL microscopes with the TEM External program, and the JEOL JEM-1230 via serial communication (Hu et al., 2010). Cameras from TVIPS, Gatan, Direct Electron, and Thermo Scientific/FEI are supported; for direct detectors by latter, the “Advanced Scripting” interface is required to use counting mode and dose fractionation. Gatan post-column filters can by fully operated via Leginon, the in-column filter of a

The source code of is available at https://sourceforge.net/projects/tom2/.

2 Academic software

the JEM-3200FSC partially. The pyScope Python extension developed by the Leginon group serves as a wrapper between the Leginon system and the different APIs of microscope and camera manufacturers.

2.1.2 Software architecture Leginon is designed to be a single installation interacting with multiple microscopes and providing long-term bookkeeping. To allow for an efficient and scalable handling of the large amounts of data that can be produced by Leginon’s applications, and to enable integration into image processing pipelines such as Appion (Lander, Stagg, Voss, & Cheng, 2009) a database forms the core of the Leginon system. Hence, the whole Leginon system is comprised of components providing the following functions: 1. TEM and camera control (Windows); implemented in Python and C 2. Image processing operations (Linux or Windows) 3. Storage of image metadata, calibrations, users, user settings, experiment settings, and applications is implemented in a MySQL database 4. Storage of image data; implemented as files, not database blobs 5. Leginon web interface providing image and data viewers and functioning as Appion GUI; implemented in PHP and JavaScript 6. (Direct electron detector frame alignment server) In a minimal configuration, all functions except microscope and camera control are combined on one (Linux) workstation; for installations with multiple microscopes and/or many users, it is recommended for performance reasons to distribute theses different tasks onto dedicated servers. In this setup, the computer assigned to image processing is typically also running the Leginon user interface. Leginon’s user interface implemented in wxPython is compatible with all major operating systems. It provides data display, allows the modification of settings, and the execution of commands. Leginon’s web interface (“myamiweb” webviewer) on the other hand is retrieving information from the database and the file storage system to display both raw information and organized reports, both on-site as well as to remote collaborators. A specific page of the web viewer, the “Leginon Observer Interface” provides an automatically refreshed dashboard of the queue processing status. For different types of data collection, Leginon provides different applications, that in turn are built from smaller pieces called “nodes,” serving for specific tasks within the applications. Examples include the “Calibration” application, several applications for targeted acquisition of untilted projection images through holes of perforated carbon film for SPA, and an application for random conical tilt and orthogonal tilt reconstruction (Yoshioka et al., 2007). Most Leginon applications are based on multi-scale imaging (MSI)b: this basically describes the concept that positions of interest are identified (and later relocated) in micrographs at b

Originally, the term “MSI” was used as name of a single application designed for imaging cryo-TEM samples in the holes of perforated carbon film (Suloway et al., 2005). Now, the term refers to a class off applications, including the original one now known as MSI-Edge.

145

146

CHAPTER 8 Automated acquisition of tilt series

progressively higher magnification, e.g., first at the grid level, then at level of an individual grid square, and finally at the level of the holes of the perforated carbon film. MSI involves the use of “presets” (microscope and camera settings) matching to the imaging sequence and calibrations to support navigating on and switching between them. To implement tilt series data collection in Leginon, a new application called “MSI-Tomography” was developed by Suloway et al. (2009). It features the same sequential targeting abilities as the previous Leginon MSI applications, but introduces new user interface elements for ET and records entire tilt series at the target positions instead of single projection images. Sample tracking after tilting is based on the predictive algorithm introduced in Zheng et al. (2004) and code directly ported from UCSF Tomography (Section 2.3). In addition to the usual MSI presets (“grid,” “square,” “hole,” “focus” and “focus-auto”), a “tomo” preset is defined in this application and used to collect the final TS. Furthermore, a highly binned and defocused “preview” uses acquisitions at minimal dose to assess the quality of the region before the final acquisition. MSI-Tomography is sharing calibrations with other MSI applications. While these calibrations require a significant time investment, they are stored in the database and can be used over long periods of time.

2.1.3 Tilt series acquisition workflow As Leginon MSI-Tomography was dedicatedly designed for batch tomography, the main goal was to bundle all user tasks in a step prior to the theoretically day-long, unattended data collection session. In the first user task, the presets introduced earlier are typically not generated from scratch, but imported from a previous experiment via the “Presets Manager” and optimized for the application at hand. In a next, typically longer step that requires user input, the targets for tilt series are selected (see also workflow in Fig. 1): first, an atlas is prepared at LM to survey the grid, e.g., for areas of optimal ice thickness. Higher magnification images covering selected grid squares are subsequently recorded with the “square” preset. Based on these images, carbon film holes of interest along with points to set the eucentric height automatically can be selected and added to the acquisition queue. In “Tomography targeting” the resulting images are reviewed, and the best targets can be identified using “preview” at minimal dose, yet high defocus and binning. The final step before starting the queue of selected targets allows one to set up data collection parameters such as dose, tilt range, and tilt interval (Fig. 1). Dedicated tilt schemes include bidirectional tilt and the dose symmetric tilt scheme (Hagen et al., 2017) since version 3.3; unidirectional tilting can be implemented via tweaking bidirectional parameters. Before imaging the first target, one or two housekeeping functions are executed at a user-defined grid position providing a blank beam (reference position): a measurement of the beam intensity to compensate for small fluctuations between and within sessions, and centering of the zero loss peak. If required, these nodes can be run at periodic intervals during the batch acquisition.

2 Academic software

FIG. 1 The Leginon user interface with the MSI-Tomography application loaded, the node selectors of the workflow are visible on the left, the settings for individual tilt series acquisition are on display. Screenshot courtesy of Anchi Cheng.

Once the target has been approached by goniometer movement, the eucentric height is refined, focus and objective stigmation are optimized, the target is recentered based on reference images and the goniometer backlash in x, y, and alpha is released. The actual tilt series begins with acquiring an image of the untilted sample and another image of the sample tilted by one increment. Based on the shifts measured here and fixed, pre-calibrated tilt axis parameters, the movement of the specimen in x, y, and z is modeled. This model (see also Section 2.3) is used for the further tilts to predict focus changes. The tracking in x/y does not involve the use of such model, but is done by smooth curve fitting usually from four preceding tilts to the current tilt angle. Tracking information from preceding tilt series in the current session and information from previous sessions loaded from a database can be compared and used as “seeds” for the model in future runs. The web interface allows users to monitor tilt-series acquisition and analyze patterns based on data acquired during the acquisition.

2.1.4 Output and processing Tilt series from Leginon can be processed directly in Appion’s protomo2 pipeline (Noble & Stagg, 2015). Alternatively, an MRC stack can be dynamically generated in the web interface from the individual MRC images stored by Leginon and downloaded to the users’ workstation for processing.

147

148

CHAPTER 8 Automated acquisition of tilt series

2.1.5 Conclusions While Leginon is one of the most, if not the most, complex software solutions in this field to set up, the developers provide very detailed documentation on installation and use of the system on their website. This also includes dedicated troubleshooting section, a public online forum to discuss application and development, and a bug tracking system. Notable research papers featuring data acquired with Leginon MSI Tomography include (Beeby et al., 2016; Dobro et al., 2017; Noble et al., 2018; Pilhofer, Ladinsky, McDowall, Petroni, & Jensen, 2011).

2.2 SerialEM SerialEM is a software package developed for automated data acquisition from TEM by David Mastronarde at the Department of Molecular, Cellular and Developmental Biology, University of Colorado Boulder while in the former Boulder Laboratory for 3D Electron Microscopy of Cells. It started as a program to automate tilt series acquisition on a JEOL 1 MV TEM, was greatly modified to control FEI Tecnai microscopes, and later adapted to JEOL microscopes by Tobin Fricke in Ken Downing’s laboratory at Lawrence Berkeley National Labs (David Mastronarde, personal communication; Mastronarde, 2005, 2018). It has continually grown in functionality and hardware support ever since. Consequently, SerialEM has matured into a universal tool for semiautomatic and automated transmission electron microscopy (e.g., Bukov, Cohen, Gabrielli, & Briggs, 2019; Jason de la Cruz, Martynowycz, Hattne, & Gonen, 2018; Schorb, Haberbosch, Hagen, Schwab, & Mastronarde, 2019), well established in bio-sciences and receiving increasing attention from materials research. In contrast to many other software introduced in this chapter, SerialEM does not aim to define step-by-step workflows, but leaves it to the operator to combine the wealth of functions in a sensible way, imposing a steep learning curve but also providing great flexibility. Different versions of SerialEM (stable and beta) can be downloaded for free for non-commercial purposes from http://bio3d.colorado.edu/SerialEM/; since the discontinuation of NIH support in 2014, contributions by the EM community for extended support services are encouraged. The code of the SerialEM core (OpenSerialEM) was made open source in 2017 under an MIT license; with exception of the open source “SerialEMCCD” plugin for Gatan cameras, the plugins that serve as a hardware abstraction layer for communication with microscopes and detectors remain closed source to avoid disclosing the vendors’ interfaces.

2.2.1 Hardware support Among all products developed in academia, SerialEM has the widest support of hardware: it runs on Thermo Scientific/FEI microscopes with the TEM Scripting Adapter, JEOL TEM with the TEMCON or TEMCENTER user interfaces, and two 120 kV instruments from Hitachi HT (HT7700, HT7800). CCD/CMOS cameras as well as direct electron detectors from Gatan, Thermo Scientific/FEI, JEOL, Direct Electron, EMSIS (formerly, Olympus Soft Imaging Solutions), TVIPS, and

2 Academic software

AMT are supported as well as Gatan DigiScan, Thermo Scientific/FEI, including support of dose fractionation and counting mode with the “Advanced Scripting” interface, and some JEOL STEM units and in- and post-column energy filters.

2.2.2 Software architecture Current versions of the SerialEM executable (Fig. 2) run on operating systems as old as Windows 2000 and as new as Windows 10, with both 32bit and 64bit versions available where supported by the operating system. Plugins (DLLs) are used to interface with the microscope and the detectors. For most configurations, the SerialEM software can be installed on any Windows computer with a network connection to the microscope PC and the camera PC. Remote hardware is accessed via communication servers provided either by SerialEM (e.g., FEI SerialEM Server) or the instrument manufacturer (e.g., JEOL TEM External). Only a few camera manufacturers require a direct installation on the camera PC due to the architecture of their API/plugin. To use the graphics processor (GPU) infrastructure of NVIDIA graphics adapters via CUDA for frame alignment of data acquired on direct electron detectors, additional packages have to be installed.

2.2.3 Installation and calibration The installation of the software is straightforward, particularly in a single-PC setup, but the configuration and calibration of SerialEM can be a challenging task. It is recommended to obtain initial configuration files tailored to a specific hardware configuration from the developer. While these configuration files are designed to help establishing an initial connectivity between the hardware components, many calibrations are required before the software can be used productively for automated acquisition. This includes calibrations of magnification, image rotation, image shift, stage shift, beam intensity vs. condenser 2/3 setting, beam intensity vs. spot size, beam shift, etc., many of which are required to be repeated under multiple settings to cover all working conditions. The calibration procedures are described in detail in the SerialEM manual, commercial services for installation and calibration of the software are available (see website). The configuration of SerialEM and the calibrations are saved in text files; ideally, all users of SerialEM on one instrument share the same “properties” (configuration) and calibrations, unless different high tensions are used. To maintain workflows reproducible and the user experience consistent, application- or user-specific settings can be saved to and read from an unlimited number of “settings files.” These settings are independent from Windows logins, different desktop shortcuts can be used to pre-load specific settings by default. The frequency calibrations need to be redone depends highly on the TEM platform, the extent of interventions in the microscope hardware, or changes in calibrations. The electron dose calibration is enforced to be redone frequently; a few other important calibrations are recommended to be checked briefly under the relevant microscope conditions before starting data acquisition.

149

FIG. 2 SerialEM’s user interface features a (1) main menu, (2) floatable and collapsable function panels, (3) an image area displaying image buffers, annotations, and an optional FFT, additional image display windows, ancillary windows like the (4) log window, (5) the Navigator or (6) script editors, and a status bar.

2 Academic software

2.2.4 Tilt series acquisition workflow After starting SerialEM, setting up camera parameter sets for different tasks (“Focus,” “Trial,” “Record,” etc.), and testing key calibrations including autofocus and image shift, regions of interest can be located either by moving the stage manually or assisted by the SerialEM “Navigator.” This tool allows one to explore a grid based on montages of the full grid or selected areas of increasingly high magnification, points and regions (polygons) of interest based on these images, and the automatic execution of tasks at these positions. For a routine tilt series, data acquisition is set up in the “Tilt Series Setup” dialog (Fig. 3) once a structure of interest has been identified. This dialog offers a plethora of options, including: tilt range and scheme (linear or Saxton, starting at 0° or high tilt, or dose symmetric tilt scheme), control of specimen exposure via changing the illumination or the exposure time, autofocus target and beam tilt, many tracking options, initial tasks such as refining the eucentric height or checking the autofocus functionality, housekeeping tasks like centering the beam or the zero loss peak, and final tasks like closing the column valves. If possible, this dialog also provides an initial estimate of the electron dose the sample will be exposed to in the course of

FIG. 3 SerialEM’s “Tilt Series Setup” dialog (version 3.7.4) is the central point to control tilt series parameters. Among many other values, all parameters set here are saved to “settings” files, allowing to separate and easily recall settings used by different users or for different applications.

151

152

CHAPTER 8 Automated acquisition of tilt series

the whole tilt series. For routine situations, the defaults work fairly well; for specific applications, protocols are available on the SerialEM website. SerialEM measures the specimen movement in x, y, and z during tilt series using tracking images and autofocusing. To speed up data acquisition, this is not done at each tilt or in necessarily regular intervals, but in an adaptive way based on a prediction from previous measurements: the prediction is relied upon (i.e., tracking or focusing is skipped) when the statistical error is low enough. According to Mastronarde (2005), this approach is more robust than a holder precalibration (Ziese et al., 2002) in unfavorable conditions such as an imprecisely set eucentric height, a drifting sample, or non-reproducible holder movements. Primarily, output from the tilt series controller is printed in the log window, displaying information such as regular and enforced focusing and tracking steps, the deviation of the sample movement form the model, information about saved images, repeated “Record” images if an initial image is too far from centered, etc. It is recommended to save these log files along with the actual tilt series for later reference. In addition, a preview window displaying the previously acquired images in a tilt series can be enabled. Tilt series can be stopped either manually to interfere with problems or will stop automatically if SerialEM detects issues, e.g., an intensity drop that might indicate the end of the useful tilt range, or a repeated failure in autofocusing. Different options allow the tilt series to be resumed by proceeding to the next step, repeating the last acquisition, or even going back several tilts in the series. Once a tilt series is finished, it needs to be “terminated” to end restrictions in the user interface imposed during a tilt series and to close the file. For dose sensitive samples, first and foremost frozen hydrated specimens, SerialEM also implements its own low dose mode with five different parameter sets (areas) for different purposes. Two (“Focus” for autofocus and other tuning tasks; “Trial” for tracking) can be image shifted away from the acquisition position, per default along the tilt axis to maintain the same z height; two others (“Search” and “View” as low magnification views) can be acquired with additional defocus for enhanced contrast and an image shift offset to keep the feature of interest centered. Once the SerialEM low dose mode is enabled, the behavior of the high level functions in SerialEM such as autofocus and eucentricity changes accordingly. The same applies to the tilt series controller and scripts (see below), that do not need be set up specifically for normal dose or low dose acquisition. SerialEM can also be set up to combine montaging (tiling) and tilt series acquisition (H€ o€ og et al., 2007; O’Toole et al., 2018): instead of a single acquisition at each tilt, a montage using image shift to cover a larger area in x/y is acquired. SerialEM saves the individual tiles for later stitching into a seamless image; this can be done in the reconstruction interface in IMOD (Kremer, Mastronarde, & McIntosh, 1996) or by other software that can deal with data in this form. STEM tomography on room temperature or cryo-samples is available through SerialEM as well: STEM is defined as a separate camera with possibly multiple channels, one per detector. SerialEM also supports dynamic focusing for STEM. This requires a setup in which the horizontal (fast) scan direction is parallel to the tilt axis.

2 Academic software

Acquisition strategies for tilt series that are not part of SerialEM’s user interface can be implemented in its own, easily accessible scripting language (formerly referred to as “macros”). This scripting language comprising more than 400 commands allows one to automate most of the functionality accessible via the user interface. It can control the microscope’s optics, stage, acquire from cameras, change acquisition parameters, call high level functions like autofocusing or beam centering, manipulate files, change samples in robotic loaders, perform calculations, etc. Prominent examples for using scripts for tilt series acquisition is the dose symmetric tilt scheme by Hagen et al. (2017) (now also available in the GUI recent SerialEM versions) or the continuous tilting/fast incremental methods by Chreifi et al. (2019). A public repository of SerialEM scripts including the dose symmetric tilt scheme script and many others for applications such as single particle data collection, random sampling, calibrations, etc. is available at https://serialemscripts.nexperion.net—for examples of scripts, see there. Finally, SerialEM’s Navigator also allows sequential automated collection of tilt series in SerialEM (batch tomography; O’Toole et al., 2018): for each position of interest, the stage coordinates, a map image at the exact magnification and field of view of the tilt series, and an additional lower magnification image are saved as references for later realignment (anchor images). The tilt series parameters are set up for the first position and cloned to all further positions, but all parameters can be edited individually. During unattended data acquisition, a general strategy has to be in place how to deal with errors, whether the whole acquisition should stop with a message or whether the Navigator should terminate the offending series and go on to the next. Batch acquisition saves a separate log file for each series.

2.2.5 Data output and processing SerialEM’s native data format are 8 or 16 bit MRC stacks. They include a standard and an extended header for metadata, an optional metadata text file (mdoc), and 16 bit files use the MRC2014 standard (Cheng et al., 2015). Depending on context, file name extensions other than “mrc” might be used for files in MRC format, e.g., “st” for tilt series or “map” for maps. In addition, SerialEM can also write single TIF files with optional compression, series of single-image TIF files referenced in a metadata text file, and single JPEG files. Data from SerialEM are best processed in IMOD (Kremer et al., 1996, Mastronarde & Held, 2017), a package for tomography reconstruction, other 3D reconstruction methods, and modeling also developed in Dr. Mastronarde’s group. MRC metadata written by SerialEM can be used seamlessly in IMOD: special data formats such as MRC stacks from montage tomography data can be processed (O’Toole et al., 2018) and the dose information in the metadata file can be used for dose-weighting. Another software product from the same group is PEET (Particle Estimation for Electron Tomography; Heumann, Hoenger, & Mastronarde, 2011; Nicastro et al., 2006), a package for aligning and averaging particles in subvolumes extracted from tomograms.

153

154

CHAPTER 8 Automated acquisition of tilt series

2.2.6 Conclusions SerialEM’s manual (available on- and offline in separate versions for stable and beta) describes the individual functions of the software in great detail, but does not cover many workflows from the start to the finish. As without step-by-step instructions, these procedures are often not exactly intuitive owing to SerialEM’s complexity, many of them are covered in protocols available on the SerialEM website or on the Boulder lab’s YouTube channel.c In addition, there is also great community support in the form of a mailing list and SerialEM is regularly taught at workshops worldwide. Due to the wide range of hardware SerialEM supports, it also offers the possibility to serve as a “unified user interface” in facilities with instruments from multiple manufacturers. Once a sample is loaded and the most basic column parameters are set up, both manual as well as automated acquisition can be carried out even by users completely unfamiliar with the software provided by the microscope or camera manufacturer. SerialEM undoubtedly demands a significant initial effort, both from the instrument owner for installation and calibration, and from the user regarding training and establishing workflows. What they receive is, however, an extremely powerful and flexible tool—no less than a “Swiss Army Knife” for automated electron microscopy. The software’s popularity is also emphasized by a number of developments from the EM community. Examples are Zuul, a SerialEM toolbox designed to edit and manage SerialEM scripts and to automatically integrate them into SerialEM settings files,d the integration into the FOCUS system interfacing cryo-TEM data collection with image processing (Biyani et al., 2017), the combination of SerialEM with image analysis tools for automated guided acquisition (Schorb et al., 2019), and the script repository already introduced above. Examples for research papers including tilt series that were acquired with SerialEM are among many (de Marco et al., 2010; Engel et al., 2015; Englmeier, Pfeffer, & F€ orster, 2017; Ladinsky et al., 2014; Nicastro et al., 2011).

2.3 UCSF Tomography UCSF Tomography (UCSF Tomo) was developed from 2004 to 2016 by Shawn (Qingxiong) Zheng and David Agard at Howard Hughes Medical Institute (HHMI) and the University of California at San Francisco (Shawn Zheng, personal communication; Zheng, 2012, 2013; Zheng, Keszthelyi, et al., 2006). The software is free for academic and/or non-profit users and available from http://msg.ucsf.edu/em/EMNEW2/ tomography_page.html. It is written mainly in C++ and JavaScript; the complete source code and commercial licenses are available upon request from Dr. Agard. UCSF Tomography is based on the assumption that the sample movement in x, y, and z due to goniometer tilt can be predicted with sufficient accuracy if a few geometric parameters regarding the stage and the optical axis are known, allowing to completely omit focusing and tracking during data collection (Zheng et al., 2004). c

https://youtube.com/bl3demc https://c-cina.unibas.ch/tools/soft/zuul/

d

2 Academic software

While two of those parameters (the angle of the tilt axis relative to the camera and the lateral offset between the optical axis and the tilt axis) are very stable and can be pre-calibrated, the third parameter, the z offset between the object and the tilt axis, i.e., the residual non-eucentricity, can be estimated using collected tilted images and refined on the fly during the tilt series. The estimated z offset along with the two pre-calibrated parameters can then be used to predict displacement due to stage tilting to the next angular step. As a result, tracking and focus exposures can be skipped, and no dedicated low/minimal dose mode is required in UCSF Tomography as all dose after the localization of the structure of interest is invested in data collection. The same predictive algorithm, based on UCSF Tomography code, is also implemented in Leginon MSI-Tomography (see Section 2.1; Suloway et al., 2009).

2.3.1 Hardware support The software supports Thermo Scientific/FEI microscopes with the TEM Scripting adapter and is available in a “CCD” version from 2013 that supports cameras by Gatan and TVIPS and a “K2” version released in 2016 for support of the Gatan K2 and K3 direct electron detectors and their special features like counting mode or dose fractionation. Post-column energy filters from Gatan are supported as well.

2.3.2 Software architecture UCSF Tomography is installed either directly on the microscope PC or a dedicated camera PC (Zheng, 2013). In both cases, the GUI communicates via TCP/IP with a server process binding to the microscope’s API, hence an Ethernet connection is required if USCF Tomography is to be installed on the camera PC. UCSF Tomography’s graphical user interface (Fig. 4) guides the user though the various steps of setting up the software, localizing targets, and data collection.

2.3.3 Installation and calibration An installation guide (Zheng, 2013) is available for the two versions of UCSF Tomography, describing in detail where files are to be extracted, how the installer is run, and how the configuration of the instrument is to be specified in the configuration text file. Typically, after setting up five different modes (camera and optics settings for different tasks) named “Atlas,” “Search,” “Track,” “Focus,” and “Collect,” and a flat field correction, calibrations are performed. Calibrations in UCSF Tomography are as lightweight and straightforward as the installation: from one pixel size per camera, all others are extrapolated. Stage shift and tilt axis, image shift, focus, eucentricity (the conversion factor between goniometer z change and defocus change), and optical axis (lateral distance to tilt axis) calibrations are limited to the modes (magnifications and camera settings) that will be employed during localizing points of interest and the tilt series itself. An experienced user can perform a complete calibration of the software in approx. 40 min. Calibrations are saved, hence there is no need to re-calibrate UCSF Tomography each time the software is restarted.

155

156

CHAPTER 8 Automated acquisition of tilt series

FIG. 4 UCSF Tomography’s user interface with the server process communicating with the microscope visible in the background. A typical workflow progresses through the GUI’s tabs from the left to the right, starting at the “Image” tab as shown here for setting up and testing the camera and optics settings used for different tasks. Screenshot courtesy of Wen Yang, Leiden University.

2.3.4 Tilt series acquisition workflow An “Atlas” map of the whole or a part of the grid is acquired for navigating to points of interest. For the tightly integrated batch tomography workflow (sequential data collection) that can be run over several days, the atlas map also allows one to acquire “target” images which in turn serve to identify up to five target points per image for acquisition. In the tomography setup window, parameters such as exposure control, (optional) defocus measurement, data type, etc. for a single or sequential tilt series can be selected. In case of sequential tilt series, the settings will be applied to all targets selected during one batch run. After the start of a tilt series via the “Tomography” tab, UCSF Tomography prints progress information into the programs “log window.” This includes the geometric parameters calculated from the first two tilts (Fig. 5). Semiautomatic dual-axis sequential data collection (Zheng, Matsuda, Braunfeld, Sedat, & Agard, 2009) of samples rotated by approx. 90° is also integrated into the GUI: a pre- and a post-rotation low magnification map (covering typically 1000  1000 μm2) are used to determine the rotation angle, magnification change,

2 Academic software

FIG. 5 UCSF Tomography in the process of acquiring a tilt series. Progress information is printed on the log window on the top left, the three images represent the previous and the current “Collect” image, and their cross-correlation with the selected correlation peak highlighted. Screenshot courtesy of Wen Yang, Leiden University.

and translation. Based on pre- and post-rotation medium magnification maps (nonLM mode, smaller than 200  200 μm2), the new stage coordinates of the target images are determined. Correlating the pre- and post-rotation target images results in refined rotation and translation values that allow to precisely localize the target points in the post-rotation stage coordinate system, ready for acquisition.

2.3.5 Output and processing Data in UCSF Tomography are saved in MRC files where microscope information and imaging conditions, such as tilt angle, stage position, pixel size, etc., are stored in the extended header. Dose fractionation frames from direct electron detectors are saved as 4bit MRC files. Unlike other software that save each stack of dose fractionated exposures into a single file, UCSF Tomography writes—for convenience—one single MRC file containing all frames sorted by the tilting angles with the camera gain reference saved at the bottom of the extended header. A notable feature of earlier versions of UCSF Tomography was the interface of the acquisition software to real-time reconstruction. The intention of Zheng, Keszthelyi, et al. (2006) was not to provide a perfect reconstruction, but to improve the acquisition workflow by providing immediate feedback on the target and the settings used. Technically, a dedicated reconstruction software was running on a Linux (GPU) workstation/cluster, that was controlled by UCSF Tomography via ssh.

157

158

CHAPTER 8 Automated acquisition of tilt series

Using nearest neighbor image alignment and the weighted backprojection (WBP) algorithm, a first reconstruction of the imaged volume could be provided in the moment the tilt series finished. When the UCSF Tomography GUI is closed, the settings are saved per user in an XML file that will be reloaded once the same user logs back in.

2.3.6 Conclusion The software comes with a detailed installation and application manual, and further support is available directly from the developer. Structures reconstructed based on tilt series acquired with UCSF Tomography are shown in Henderson, Gan and Jensen (2007), Briegel et al. (2008), Ingerson-Mahar, Briegel, Werner, Jensen, and Gitai (2010), Pilhofer et al. (2011), and Kollman et al. (2015). Besides the GUI for the atlas and tomography, UCSF Tomography supports random conical tilt/orthogonal tilt (Zheng, Kollman, Braunfeld, Sedat, & Agard, 2006) and single particle data collection.

3 Commercial software The software products introduced in this section are developed directly by or in close connection to a microscope or camera manufacturer. All are closed source and distributed under a commercial license.

3.1 Hitachi 3D tilt image acquisition “3D tilt image acquisition” is an optional function for the Hitachi High-Technologies (HHT, Tokyo, Japan) TEM System Control software (Hitachi, 2017; Thomas Schmidt, personal communication).

3.1.1 Hardware support It is available for the TEM models HT7700, HT7800 (both 120 kV W/LaB6), and the HF5000 (200 kV FEG) with a dedicated holder. This section refers to the functionality on the HT7800, an instrument that includes this function by default for most configurations. Image acquisition is limited to CCD and CMOS cameras that are fully integrated in the TEM System Control Software; as of 11/2018 these are the AMT XR41, AMT XR81, EMSIS Xarosa and Gatan Rio. An extension to more cameras will be following the demands of the market.

3.1.2 Installation and calibration As a prerequisite to collecting tilt series, only magnification-specific image shift calibrations and routine axial alignments are required in addition to standard automated microscope alignments.

3 Commercial software

3.1.3 Tilt series acquisition workflow “3D tilt image acquisition” is started from within the TEM System Control software; the main window (Fig. 6) and its menus allow one to configure basic parameters. The user can chose between an automatic and a manual sequence; the latter would allow to manually adjust the position of the field of view during the tilt series (instead of automatic tracking), but is of little importance in practice. All imaging conditions are set up on the microscope beforehand. Besides an instructive depiction of the workflow in the style of a programming flowchart (Fig. 6) and information on the currently active step in this workflow, the “Customize capturing sequence” window also offers a possibility to tweak a few more parameters. This includes the length of waiting times, the tracking process, and autofocusing.

FIG. 6 (Left) The main window of the Hitachi “3D Tilt Image Acquisition function” and its sub-menus allow to set up basic parameters such as start/end angles (automatically limited according to specimen holder and x/y position), angular increment, stage tilt speed, or the area used for tracking. (Right) The “customized capturing sequence” displays the acquisition as a programming-style workflow and allows one to fine-tune further parameters. Screenshots courtesy of Thomas Schmidt, Hitachi High Technologies Germany.

159

160

CHAPTER 8 Automated acquisition of tilt series

Tilt series acquisition can be combined with the Hitachi TEM System Control software’s low dose feature that supports an image-shifted “Focus” area, optionally at a magnification higher than the working magnification. Combining the acquisition of tilt series with other advanced features of the software such as “auto multiple frame” (AMF; montaging) is currently not supported. There is also no support for batch tomography or tilt schemes other than unidirectional. During acquisition, the software is tracking the region of interest and correcting via image and stage shift. Autofocusing can be called before and/or after shifting to the correct field of view. The focus measurement is based on beam tilted image pairs and can aim for exact focus or one of three underfocus values defined by the system’s “main” user.

3.1.4 Output and processing Tilt series are saved a series of single layer/page 16bit TIF files with metadata in the description field of the file header plus a text file that offers easy access to the same metadata (date/time, magnification, HT, spot, stage position, pixel size, etc.). In Hitachi’s workflow, tilt series are processed in the optional 3D reconstruction packages of the EMIP-EX software (Hitachi, 2016), but a combined workflow with other reconstruction or modeling software such as IMOD (Kremer et al., 1996) is possible as well.

3.1.5 Conclusion While the functionality of “3D tilt image acquisition” is limited, the software’s simplicity, the seamless integration into the microscope’s own control software, and the lack of complicated additional calibrations make the program very accessible and compelling for new users and routine data collection at room temperature.

3.2 TEMography Recorder “Recorder” is the data acquisition component for electron tomography from the “TEMography” software suite, initially developed by JEOL System Technology and now by System in Frontier Inc. (Tokyo, Japan; Hiromitsu Furukawa and Emanuel Katzmann, personal communication).

3.2.1 Hardware support It is available for all current models of JEOL TEM for TEM and STEM tomography. In TEM mode, it supports CCD and CMOS cameras from AMT, EMSIS, Gatan, JEOL, and TVIPS as well as direct electron detectors from Gatan; STEM is limited to JEOL’s own STEM system. The software also supports analytical electron tomography via EDS mapping accessing signals from Oxford Instruments, Thermo Scientific, and JEOL JED detectors. JEOL in-column energy filters are fully supported, filtered tilt series with post-column filters can be collected in a semiautomatic fashion: after tilting, tracking and focus adjustment, Recorder will suspend for manual acquisition in DigitalMicrograph, subsequently the user resumes the tilt series.

3 Commercial software

3.2.2 Software architecture Recorder and STEM Recorder are installed on the same Windows 7 or Windows 10 computer driving the camera. They are standalone programs operated independently of the JEOL microscope user interface (TEMCENTER). The TEM and the STEM versions have very similar user interfaces (Fig. 7) and implement largely identical workflows; for the STEM Recorder, STEM (ASID) mode has to be enabled before starting the software.

3.2.3 Installation and calibration A basic configuration and calibration of the software is performed by either the manufacturer or JEOL upon installation. In addition, per-session calibrations (including checks of images shift, projector shift, STEM image shift, beam tilt, STEM autofocus position) are required before starting data acquisition. These are run largely automatically and are limited to the optics settings and the camera that will be used for data acquisition, hence can be completed in as little as a minute.

3.2.4 Tilt series acquisition workflow Tilt series recording parameters can be set up either in a “basic” or an “advanced” dialog (Fig. 8). The basic dialog features routine parameters like angular step, binning, continuous vs. 0°-start-tilt scheme, a drift check, autofocus beam tilt, etc.

FIG. 7 The main window of the “TEMography Recorder” TEM version during acquisition of a tilt series. Key parameters such as the tilt range, the settings for focus and position tracking, the anti-creep function (an automatic offset in x and y), and the acquisition progress are displayed and easily accessible. Screenshot courtesy of Emanuel Katzmann, JEOL (Germany) GmbH.

161

162

CHAPTER 8 Automated acquisition of tilt series

FIG. 8 The basic (top) vs. the advanced (bottom) version of TEMography Recorder’s “Recording Settings” dialogs, demonstrating the significant difference in flexibility and complexity of both options. Screenshots courtesy of Emanuel Katzmann, JEOL (Germany) GmbH.

3 Commercial software

In addition, the advanced version allows one to define several tilt angle ranges employing different parameters, several recording modes in parallel (e.g., for a focus series [“defocus bracketing”], multi-detector STEM series, etc.), and provides an interface to the ShotMeister/STEMMeister software (see below). Previously used recording settings from either dialog can be easily recalled from files selectable the Recorder main window. The relationship of different areas (acquisition, tracking, focusing) relative to each other and the tilt axis is visualized graphically during setup. Right after starting the tilt series, Recorder preforms an interactive check of the reference image (used for tracking) and the sample position after tilting. During acquisition of the tilt series, the current task, the remaining time, and the last acquired image with offsets annotated are displayed. Additionally, the data collected so far can be reviewed in both a thumbnail gallery (Viewer for Recorder) as well as in an animated window. Interrupting and resuming the acquisition is possible if required. The x/y displacement of the sample during tilting can be compensated by an amount defined in “anti-creep function” (a fixed amount in pixel or percent field of view of displacement per tilt step parallel and perpendicular to the tilt axis) and/or by an amount based on a prediction of the stage movement plus calculated from tracking images. To apply this shift, Recorder can access image shift, projector shift, stage motor, or the stage piezo drive. Focus can be tracked either automatically, manually, or not at all. The STEM Recorder uses maximization of contrast for focusing; it supports dynamic focusing using several user defined focus areas to calculate the change of the focus level. Low dose tilt series can be collected either by combining Recorder with JEOL’s MDS (minimal dose system) or with System in Frontier’s “ShotMeister” software (resp. “STEMMeister” to implement cryo-STEM tomography). The MDS is integrated into the microscopes UI, but limited to three different settings (Search/Focus/Photo) with limited saving capabilities. ShotMeister has it’s own user interface and sets of calibrations an offers great flexibility, e.g., numerous different imaging states that can modify almost all of the microscope’s settings and can be assigned to focusing/tracking/recording (also referred to as “Ultimate Dose Control,” UDC).

3.2.5 Output and processing Tilt series can be saved as a series of TIF files (that apparently include no/very little meta information), MRC files (with the pixel size in the header), or in the proprietary TEMography “TMG” format. In addition, Recorder writes two text files including all recording parameters and a detailed log of microscope settings (defocus, stage position, lens values, etc.) for each tilt, but no meta-information file for the images. Downstream software packages in the TEMography software suite that also support the TMG format are Composer for alignment and reconstruction and Visualizer-evo for the visualization of the volume. Via support of other file formats, Composer and Visualizer-evo can be used for processing tilt series from any TEM platform.

163

164

CHAPTER 8 Automated acquisition of tilt series

3.2.6 Conclusion TEMography Recorder is a software with an easily accessible user interface, requiring only minimal calibrations, but offering substantial flexibility in advanced mode. Up-to-date documentation is lacking as of 01/2019 but support can be obtained by System in Frontier Inc. and the JEOL application team. Oda Kikkawa (2013) include examples of tomography data of biological samples collected using this software, (Hata et al., 2017) of materials samples.

3.3 Thermo Scientific Tomography “Tomography 4” is Thermo Scientific’s (formerly FEI’s) software for automated acquisition of tomography tilt series (Thermo Scientific, 2018, Sonja Welsch and Andreas Voigt, personal communication) in biomedical and material sciences. It is intended to be used in conjunction with Inspect3D for offline reconstruction and Amira and Avizo for visualization, but the output format allows full flexibility with regard to image processing software. In comparison to previous versions (Xplore3D), Tomography 4 received a new user interface technology and a new codebase. The software is optimized for the use in cryo-electron tomography; accordingly, a lot of code and workflow is shared with EPU, a Thermo Scientific software for the automated acquisition of large data sets for single particle analysis used in structural biology.

3.3.1 Hardware support The current version of the software is available for the Titan and Talos platforms of Thermo Scientific/FEI with the Windows 7 operating system, the latest version of the software supporting the legacy Tecnai platform is 4.9. For support in Tomography, cameras have to be integrated in the TEM server; on Windows 7 based systems, this applies to the CMOS cameras Ceta (Thermo Scientific) and OneView (Gatan), and the direct electron detectors Falcon (Thermo Scientific) and K2 in combination with a GIF. Support for the K3/GIF will be expected to be added in a later version. With a GIF, energy filtered images at any user defined energy loss can be acquired. Tomography provides fully integrated support of the beam-generated Volta phase plate Danev et al., 2014), including the activation of a new phase plate position by irradiation in certain intervals (e.g., after each 3rd tilt series) and conditioning of the present position. STEM Tomography can access FEI STEM hardware. If the instrument is equipped with a SuperX G2 or any other EDS detector integrated in the Velox software, spectrum images for elemental analysis can be acquired in addition to the BF/DF STEM images during tilt series acquisition. STEM-EDS Tomography is licensed as a separate product.

3 Commercial software

3.3.2 Software architecture TEM and STEM Tomography are two separate, standalone programs that are run on the same PC as the microscope user interface and the microscope server itself (Fig. 9).

3.3.3 Installation and calibration The software is installed by the microscope manufacturer, who also provides initial calibrations. Some calibrations such as the system-wide calibrations for magnification and focus are shared with EPU, MAPS, Velox, and other software. Calibrations to be carried out by the user in the workflow described below include an automatic routine to calculate the “optimized position” that brings the optical axis to coincidence with the tilt axis by applying a (lateral) image shift of up to 5 μm. They also include a holder calibration routine: even at eucentric height and with the “optimized position” set, the sample will shift in x, y, and z, resulting from the imperfect movement of the goniometer and holders. These residual shifts are reproducible and can be measured by this routine. They are subsequently applied as offsets during tilt series acquisition and help to minimize the displacement/defocus of the sample in the first place (Ziese et al., 2002). This calibration is specimen holder

FIG. 9 The user interface of Tomography 4 features (1) tabs that guide the user through the workflow, (2) a ribbon with context sensitive controls, (3) a lists of tasks related to the progress of the workflow, (4) a “workflow” area with one or more images and display tools, (5) a status/ progress display and (6) an image information panel.

165

166

CHAPTER 8 Automated acquisition of tilt series

and obviously goniometer specific and can be saved to and imported from files to cover (a) different tilt schemes and (b) configurations with multiple holders in use for tomography (Fig. 10). In addition, “Tomography Calibrations” for STEM includes a routine assisting to manually set a correction factor for the slope in dynamic focusing.

3.3.4 Tilt series acquisition workflow The user interface provides a series of tabs that guide the user step by step through the workflow. Each of these tabs is organized as tasks designed to be done in sequence. The most relevant steps are: On the “Preparation”/“Preparation (STEM)” tab, different presets for optics and acquisition settings for different functions such as Atlas, Tracking, Focus, Drift, or Exposure can be defined. These are either activated manually from the “Preparation” tab for testing, or automatically from within the respective function. A whole set of presets can be imported from and exported to XML files to implement different applications. Residual image shifts in TEM due to magnification changes between the presets can be eliminated using a semiautomatic image shift calibration routine. Finally, Tomography allows fine grained control of the filters used for crosscorrelating images in this step. In the next step, an “Atlas” overview of the grid in LM (Fig. 9), starting at the current stage position or the center and progressing outwards, can be prepared for navigating to positions of interest. In addition to the interactive (zoom/pan/move stage to) Atlas display, the data is also saved as merged JPEG and TIF files, as well as in the form of individual tiles. “TEM Tomography Calibrations” covers the tomography-specific “optimized position” and holder calibration described earlier. The actual setup of a tilt series is performed in the “Tomography” tab. The software makes a number of assumptions based on selecting either a slab-like or a rodlike sample geometry. Options such as low dose tomography (TEM and STEM), batch tomography, or the use of the Volta phase plate can be enabled individually or in combination; dual axis tomography is not available in the current version any longer. For low dose and batch tomography options, an additional step for graphically defining areas of interest and/or areas for focusing and tracking for each tilt series is required. If the “FEI Email Component” is set up, email notifications of the operator at the end of a run can be enabled. Angular parameters include the maximum tilt, the arbitrary starting angle (allowing uni- or bidirectional tilt series), the tilt increment (linear or Saxton), optionally with a switch point allowing smaller increments at high tilt, and partial tilt series. Further settings include the optional application of holder calibrations, the optional autofocus and tracking functions, the focus target (allowing a different value per position in batch mode), the focus mode (objective or stage z), objective stigmator correction, zero loss peak centering, STEM channels, or dynamic focusing for STEM. Once started, a tilt series can be interrupted to manually optimize tracking and focusing. If a tilt series fails in batch mode (e.g., autofocus fails repeatedly), the series is canceled and the software proceeds to the next position.

0.05

Image shift X [um]

0.00 –0.05 –0.10 –0.15 –0.20 –0.25 –100

–80

–60

–40

–20

0

20

40

60

80

100

20

40

60

80

100

40

60

80

100

Tilt angle [°] 1.0

Image shift Y [um]

0.8 0.6 0.4 0.2 0.0 –0.2 –0.4 –0.6 –100

–80

–60

–40

–20

0

Tilt angle [°] 2

Defocus [um]

1 0 –1 –2 –3 –4 –100

–80

–60

–40

–20

0

20

Tilt angle [°]

FIG. 10 Exemplary holder calibration curves for a bidirectional tilt series from the Thermo Scientific Tomography software showing the image shifts in x and y and the defocus as a function of the tilt angle. The significant shifts at 0° arise from the changed tilt direction of the goniometer and illustrate the necessity of tilt scheme dependent holder calibrations. (Colors adjusted for reproduction.)

168

CHAPTER 8 Automated acquisition of tilt series

3.3.5 Output and processing Data from “Tomography” are saved in the MRC2014 format (Cheng et al., 2015), writing either all images into a single stack or each tilt into its own file. For the Falcon 3EC direct detector, on-the-fly drift correction can be enabled. The extended MRC header used for storing application specific metadata is well documented in Thermo Scientific (2018). In addition, text files containing further meta-information (textual summary of parameters, list of tilt angles, list of x/y shifts and defocus) and an XML file with structured parameters are written. Finally, the built-in “Visualizer” allows a quick inspection of the acquired data but is not meant for reconstruction of the tilt series.

3.3.6 Conclusion In conclusion, Thermo Scientific Tomography is a powerful software with a clear aim on automating data collection with batch tomography. It provides an easily accessible user interface and a well-defined workflow guiding the operator—as any software with a pre-defined workflow, that, however, somewhat comes at the cost of flexibility. While the seamless integration of Thermo Scientific/FEI hardware is very beneficial, delays with integration of third party hardware (e.g., direct electron detectors from other manufacturers) were perceived as very troublesome by the community.

3.4 TVIPS EM-Tomo Tietz Video and Image Processing Systems (TVIPS GmbH, Gauting, Germany) is a manufacturer of CMOS cameras, accompanying software, and accessories for TEM. As such, and due to a close connection to the Max-Planck-Institute for Biochemistry in Martinsried, Germany, they were an important player in automation of electron tomography from early on (Dierksen et al., 1992). EM-Menu is TVIPS’ current software package for image acquisition; “EMTools,” a software developed by Ingo Daberkow based on the scripting possibilities of EM-Menu, provides additional components with specialized functionality. This includes automatic TEM tuning (“EM-Align”; coma-free alignment and astigmatism correction using Zemlin tableaus), sample navigation (EM-Navi), data collection for single particle work (EM-SPC), and automated collection of tilt series (EM-Tomo). The latter package will be covered in this section (Andreas Wisnet & Hans Tietz, personal communication; TVIPS, 2014).

3.4.1 Hardware support TVIPS’ software products are compatible with TEM of all current manufacturers and run on the camera control computer separate of the microscope control PC. Notably, also legacy instruments like the Zeiss Libra or the Philips CM platform can be controlled. EM-Tools/EM-Tomo supports cameras manufactured by TVIPS and Direct Electron; for STEM tomography data acquisition, TVIPS’ own Universal Scan Generator (USG) is required. In-column filters can be used for zero loss filtering and

3 Commercial software

spectroscopic imaging, while modern post-column filters that are available exclusively with Gatan cameras are not.

3.4.2 Installation and calibration The initial installation and configuration of EM-Tools is provided by TVIPS upon installation of the camera. Further calibrations, including stage shift, image shift, autofocus, beam intensity, beam shift, illuminated area, etc., are performed in EM-Tools and shared between all EM-Tools applications. They are strictly done on a per-magnification basis, interpolation is not provided. Hence, exclusively the few magnifications used for different acquisition presets need to be calibrated and per magnification, a full calibration is only a matter of minutes.

3.4.3 Tilt series acquisition workflow A typical workflow for tomography starts in the “Navigation” component with a grid map, acquired in a spiral from the goniometer center. For this purpose a dedicated “preset” (camera and microscope parameters) is used; further presets with increasingly high magnification are used for further mapping the grid. The image shift between presets/magnifications can be compensated for automatically by crosscorrelation, or by entering the x/y offset manually. A “scan” of areas of interest, typically at the lowest non-LM magnification, can reveal further detail and can also serve as a reference image for latter identification of the area of interest in batch tomography. As an overlay to the “scan” images, features like the orientation of the tilt axis, the image shift range, or the position of other presets such as the higher magnification “search” are visualized (Fig. 11). For use of EM-Tomo under low dose conditions, the tracking and focus positions can be easily graphically shifted on the “search” images along the tilt axis to minimize exposure of the ROI. The actual and calibrated size of the illuminating beam for the individual presets is also indicated in order to avoid accidental exposure of the feature of interest by tracking or focusing. EM-Tomo allows the user to record multiple tilt series under both normal and low dose conditions at different positions, including re-centering of the position in x and y via multi-scale reference images and automatic correction of the z height to eucentricity. To enable this, further “scan” positions are added on the grid map and all parameters (defocus, position of tracking and focusing area, tilt range, etc.) are set up from scratch. Alternatively, a single tilt series can be started at the current goniometer position completely skipping the Navigation. A typical tilt series is carried out in three consecutive sub-series: the first subseries is used to find the maximum tilt angle in the negative tilt range via tracking images (exposures are being skipped), the second sub-series is the actual unidirectional data acquisition, and the third sub-series (with skipped exposures) is used to bring the goniometer back to 0°, a step essential for batch tomography. The tracking images recorded in previous sub-series can be used as “anchor images” for acquiring the following sub-series. For special applications such as bidirectional tilt series, up to 12 sub-series can be created. The tilt angles for each sub-series can be linear,

169

FIG. 11 TVIPS EM-Tomo: Graphical definition of the individual areas, indicating the tilt axis as a yellow line, the theoretical image shift range as a cyan box, and the illuminated area of the “Search” preset as a green circle. If enabled, illuminated areas of other presets, such as “Focus,” are also indicated to avoid inadvertent pre-exposure of the region of interest.

3 Commercial software

follow the Saxton scheme, or can be read from a file; the exposure time can be kept constant or increased at higher tilt to compensate for the decrease in signal. The software also estimates the incident dose for each sub-series, allowing to tweak the parameters accordingly. Preferences for the individual positions (exposure/tracking/focus) allow a high degree of flexibility beyond the standard parameters, including a drift check, multiple abort (break) conditions, beam centering, aperture centering, etc. Special options can be applied for the Exposure, like tiling for an increased image area, focus and/or dose series, or “dark field orientation mapping” of polycrystalline samples via multiple beam tilted images (as shown in Liu et al., 2011). Tracking and autofocus after tilting is based solely on measurements, a predictive algorithm or a holder calibration are not being used. The tilt-induced specimen displacement and drift are corrected using image shift and goniometer (motor and piezo) correction. During acquisition of the series, the lateral shift (x and y) and the axial shift (defocus) are being displayed graphically (Fig. 12), allowing a first assessment of the goniometer performance, the microscope alignment and the software calibrations, and prompt intervention while acquisition is still ongoing. Another option for an instant identification of problems is the “tilt player,” where the previously acquired images can be shown.

3.4.4 Output and processing EM-Tomo writes TIF files (one file with header data per angle, plus one text file with angles), 16 bit MRC stacks, or 16 bit EM format files as used by TOM Toolbox (Nickell et al., 2005). If the dose fractionation mode (burst mode) of the TVIPS XF416 is used, the individual fractions are saved sideways next to each other in a single layer TIF file or as separate z levels in an MRC stack. An alignment of the fractions has to be performed in external software. The generated tilt series can be viewed in EM-Menu and EM-Tools, for reconstruction third party products like IMOD (Kremer et al., 1996) or TomoJ (Messaoudii, Boudier, Sorzano, & Marco, 2007) are compatible.

3.4.5 Conclusion Instructions for this very versatile, yet also complex software can be obtained from the manual (TVIPS, 2014), a step-by-step tutorial, and an online help text that is displayed automatically for the currently selected function. As of 2019, “EM Tools” and its components are not under active development any longer, a completely new tool “TVIPS EMplified” is awaiting release. TVIPS EMplified will be independent of EM-Menu and feature a completely new, modernized user interface optimized for Windows 10 and high resolution screens. Support for GPU processing will, for example, allow on-line alignment of burst mode (dose fractionation) acquisitions from the XF416 CMOS camera. Early versions already demonstrate astounding multi-scale navigation possibilities. The ultimate goal is to move all applications—including tomography—from EM-Tools to TVIPS EMplified and to re-implement and add to the functionality described above.

171

FIG. 12 EM-Tomo’s user interface for tilt series acquisition, displaying current focus-, tracking- and exposure images along with results from cross correlation to the previous tilt. Information about timings, lateral, and axial (focus) shifts is provided as well.

References

4 Considerations for choosing the software In most situations, the definition or acquisition of TEM hardware will precede the decision for software for automated acquisition. Hence, a primary criterion for choosing the right program is compatibility with the present hardware (column and detectors), usually eliminating a considerable number of programs. For the remaining options, the samples and questions at hand will determine what functionality the acquisition software needs to support—are montage tomography, batch tomography, etc. required? Finally, the type of user base (numerous short-term users vs. dedicated specialists) and their requirements need to be taken into consideration: is a clearly outlined, step-by-step workflow preferable over great flexibility? The cost of commercial software is apparent. All software packages introduced in this chapter developed in academia are available for free, however, costs might also occur here in the form of additional hardware required, the time invested in installation and calibration or installation services, trainings, or support contracts. The choice of the image processing software, is usually not a limiting factor, as most packages use MRC or TIF files and there is a number of converters, e.g., as a part of IMOD or em2em from Image Science,e available.

Acknowledgments The author appreciates the opportunity to discuss with and the information and documentation provided by Anchi Cheng (NYSBC/Simons Electron Microscopy Center), Hiromitsu Furukawa (System In Frontier), Emanuel Katzmann (JEOL Germany), David Mastronarde (UC Boulder), Thomas Schmidt (HHT Germany), Ingo Daberkow, Andreas Wisnet, and Hans Tietz (TVIPS), Andreas Voigt and Sonja Welsch (Thermo Scientific), and Shawn Zheng (UCSF). He also thanks Thomas Heuser (VBCF EM Facility) and Thomas-M€ uller Reichert (TU Dresden) for their feedback, the VBCF Electron Microscopy Facility (Vienna, Austria) for use of their infrastructure, and Wen Yang (University of Leiden) for providing screenshots.

Conflict of interest The author’s company “Nexperion—Solutions for Electron Microscopy” is providing commercial services Related to the SerialEM software discussed in this chapter.

References Baldwin, P. R., Tan, Y. Z., Eng, E. T., Rice, W. J., Noble, A. J., Negro, C. J., et al. (2018). Big data in cryoEM: Automated collection, processing and accessibility of EM data. Current Opinion in Microbiology, 43, 1–8. https://doi.org/10.1016/j.mib.2017.10.005. e

https://www.imagescience.de/em2em.html

173

174

CHAPTER 8 Automated acquisition of tilt series

Baumeister, W., Grimm, R., & Walz, J. (1999). Electron tomography of molecules and cells. Trends in Cell Biology, 9, 81–85. Beeby, M., Ribardo, D. A., Brennan, C. A., Ruby, E. G., Jensen, G. J., & Hendrixson, D. R. (2016). Diverse high-torque bacterial flagellar motors assemble wider stator rings using a conserved protein scaffold. Proceedings of the National Academy of Sciences of the United States of America, 113(13), E1917–E1926. Biskupek, J., Leschner, J., Walther, P., & Kaiser, U. (2010). Optimization of STEM tomography acquisition—A comparison of convergent beam and parallel beam STEM tomography. Ultramicroscopy, 110, 1231–1237. https://doi.org/10.1016/j.ultramic.2010.05.008. Biyani, N., Righetto, R. D., Mcleod, R., Caujolle-Bert, D., Castano-Diez, D., Goldie, K. N., et al. (2017). Focus: The interface between data collection and data processing in cryo-EM. Journal of Structural Biology, 198, 124–133. https://doi.org/10.1016/j.jsb.2017.03.007. Briegel, A., Ding, H. J., Li, Z., Werner, J., Gitai, Z., Dias, D. P., et al. (2008). Location and architecture of the Caulobacter crescentus chemoreceptor array. Molecular Microbiology, 69(1), 30–41. Bykov, Y., Cohen, N., Gabrielli, N., & Briggs, J. A. G. (2019). Multiplexed electron microscopy by fluorescent barcoding allows screening for ultrastructural phenotype. bioRxiv. https://doi.org/10.1101/515841. Cheng, A., Henderson, R., Mastronarde, D., Ludtke, S. J., Schoenmakers, R. H., Short, J., et al. (2015). MRC2014: Extensions to the MRC format header for electron cryo-microscopy and tomography. Journal of Structural Biology, 192(2), 146–150. Chreifi, G., Chen, S., Metskas, L. A., Kaplan, M., & Jensen, G. J. (2019). Rapid tilt-series acquisition for electron cryotomography. Journal of Structural Biology, 205, 163–169. Available at https://doi.org/10.1016/j.jsb.2018.12.008. Danev, R., Buijsse, B., Khoshouei, M., Plitzko, J. M., & Baumeister, W. (2014). Volta potential phase plate for in-focus phase contrast transmission electron microscopy. Proceedings of the National Academy of Sciences of the United States of America, 111(44), 15635–15640. de Marco, A., M€uller, B., Glass, B., Riches, J. D., Kr€ausslich, H. G., & Briggs, J. A. (2010). Structural analysis of HIV-1 maturation using cryo-electron tomography. PLoS Pathogens, 6, e1001215. https://doi.org/10.1371/journal.ppat.1001215. Dierksen, K., Typke, D., Hegerl, R., Koster, A. J., & Baumeister, W. (1992). Towards automatic electron tomography. Ultramicroscopy, 40(1), 71–87. Dobro, M. J., Oikonomou, C. M., Piper, A., Cohen, J., Guo, K., & Jensen, T. (2017). Uncharacterized bacterial structures revealed by electron cryotomography. Journal of Bacteriology, 199(17), e00100–e00117. https://doi.org/10.1128/JB.00100-17. Engel, B. D., Schaffer, M., Kuhn Cuellar, L., Villa, E., Plitzko, J. M., & Baumeister, W. (2015). Native architecture of the Chlamydomonas chloroplast revealed by in situ cryoelectron tomography. eLife, 4, e04889. https://doi.org/10.7554/eLife.04889. Englmeier, R., Pfeffer, S., & F€orster, F. (2017). Structure of the human mitochondrial ribosome studied in situ by cryoelectron tomography. Structure, 25, 1574–1581. https://doi. org/10.1016/j.str.2017.07.011. Grimm, R., B€armann, M., H€ackl, W., Typke, D., Sackmann, E., & Baumeister, W. (1997). Energy filtered electron tomography of ice-embedded actin and vesicles. Biophysical Journal, 72, 482–489. https://doi.org/10.1016/S0006-3495(97)78689-2. Hagen, W. J. H., Wan, W., Briggs, J. A. G., et al. (2017). Implementation of a cryo-electron tomography tilt-scheme optimized for high resolution subtomogram averaging. Journal of Structural Biology, 197(2), 191–198.

References

Hata, S., Miyazaki, S., Gondo, T., Kawamoto, K., Horii, N., Sato, K., et al. (2017). In-situ straining and time-resolved electron tomography data acquisition in a transmission electron microscope. Microscopy, 66, 143–153. https://doi.org/10.1093/jmicro/dfw109. Henderson, G. P., Gan, L., & Jensen, G. J. (2007). 3-D ultrastructure of O. tauri: Electron cryotomography of an entire eukaryotic cell. PLoS One, 2(8), e749. Published 2007 Aug 15. https://doi.org/10.1371/journal.pone.0000749. Heumann, J. M., Hoenger, A., & Mastronarde, D. N. (2011). Clustering and variance maps for cryo-electron tomography using wedge-masked differences. Journal of Structural Biology, 175(3), 288–299. Hitachi (2016). 3d reconstruction instruction manual (Software operation edition) (1st ed.). Tokyo, 105-8717, Japan: Hitachi High-Technologies Corporation. June 2016. Hitachi (2017). Transmission electron microscope HT7800 instruction manual (Software operation edition) (1st ed.). Tokyo 105-8717, Japan: Hitachi high-technologies corporation. May 2017. Hohmann-Marriott, M. F., Sousa, A. A., Azari, A. A., Glushakova, S., Zhang, G., Zimmerberg, J., et al. (2009). Nanoscale 3D cellular imaging by axial scanning transmission electron tomography. Nature Methods, 6(10), 729. H€o€og, J. L., Schwartz, C., Noon, A. T., O’Toole, E. T., Mastronarde, D. N., McIntosh, J. R., et al. (2007). Organization of interphase microtubules in fission yeast analyzed by electron tomography. Developmental Cell, 12, P349–P361. https://doi.org/10.1016/j. devcel.2007.01.020. Hoppe, W., Gassmann, J., Hunsmann, N., Schramm, H. J., & Sturm, M. (1974). Threedimensional reconstruction of individual negatively stained yeast fatty-acid synthetase € Phymolecules from tilt series in the electron microscope. Hoppe-Seyler’s Zeitschrift fur siologische Chemie, 355, 1483–1487. PMID 4142629. Hu, M., Vink, M., Kim, C., Derr, K., Koss, J., D’Amico, K., et al. (2010). Automated electron microscopy for evaluating two-dimensional crystallization of membrane proteins. Journal of Structural Biology, 171(1), 102–110. Ingerson-Mahar, M., Briegel, A., Werner, J. N., Jensen, G. J., & Gitai, Z. (2010). The metabolic enzyme CTP synthase forms cytoskeletal filaments. Nature Cell Biology, 12(8), 739–746. Jason de la Cruz, M., Martynowycz, M. W., Hattne, J., & Gonen, T. (2018). MicroED data collection with SerialEM. Ultramicroscopy, 201, 77–88. https://doi.org/10.1016/j. ultramic.2019.03.009. Knauer, V., Hegerl, R., & Hoppe, W. (1983). Three-dimensional reconstruction and averaging of 30 S ribosomal subunits of Escherichia coli from electron micrographs. Journal of Molecular Biology, 163(3), 409–430. Kollman, J. M., Greenberg, C. H., Li, S., Moritz, M., Zelter, A., Fong, K. K., et al. (2015). Ring closure activates yeast γTuRC for species-specific microtubule nucleation. Nature Structural & Molecular Biology, 22(2), 132–137. Koning, R. I., Koster, A. J., & Sharp, T. H. (2018). Advances in cryo-electron tomography for biology and medicine. Annals of Anatomy, 217, 82–96. PMID 29526767. Koster, A. J., Chen, H., Sedat, J. W., & Agard, D. A. (1992). Automated microscopy for electron tomography. Ultramicroscopy, 46(1-4), 207–227. Koster, A. J., Van den Bos, A., & van der Mast, K. D. (1987). An autofocus method for a TEM. Ultramicroscopy, 21, 209–222. https://doi.org/10.1016/0304-3991(87)90146-X. Kremer, J. R., Mastronarde, D. N., & McIntosh, J. R. (1996). Computer visualization of threedimensional image data using IMOD. Journal of Structural Biology, 116, 71–76.

175

176

CHAPTER 8 Automated acquisition of tilt series

Ladinsky, M. S., Kieffer, C., Olson, G., Deruaz, M., Vrbanac, V., Tager, A. M., et al. (2014). Electron tomography of HIV-1 infection in gut-associated lymphoid tissue. PLoS Pathogens, 10, e1003899. https://doi.org/10.1371/journal.ppat. 1003899. Lander, G. C., Stagg, S. M., Voss, N. R., & Cheng, A. (2009). Appion: An integrated, databasedriven pipeline to facilitate EM image processing. Journal of Structural Biology, 166(1), 95–102. Leginon, n.d.: http://emg.nysbc.org/redmine/projects/leginon/wiki/Leginon_Manual. Liu, H. H., Schmidt, S., Poulsen, H. F., Godfrey, A., Liu, Z. Q., Sharon, J. A., et al. (2011). Three-dimensional orientation mapping in the transmission electron microscope. Science, 332(6031), 833–834. Liu, J., Li, H., Zhang, L., Rames, M., Zhang, M., Yu, Y., et al. (2016). Fully mechanically controlled automated electron microscopic tomography. Scientific Reports, 6, Article No. 29231. https://doi.org/10.1038/srep29231. Mastronarde, D. N. (1997). https://doi.org/10.1006/jsbi.1997.3919. Mastronarde, D. N. (2005). Automated electron microscope tomography using robust prediction of specimen movements. Journal of Structural Biology, 152(1), 36–51. Mastronarde, D. N. (2018). Advanced data acquisition from electron microscopes with SerialEM. Microscopy and Microanalysis, 24(Suppl. 1), 864–865. https://doi.org/10.1017/ S1431927618004816. Mastronarde, D. N., & Held, S. R. (2017). Automated tilt series alignment and tomographic reconstruction in IMOD. Journal of Structural Biology, 197, 102–113. McEwen, B. F., Radermacher, M., Rieder, C. L., & Frank, J. (1986). Tomographic three-dimensional reconstruction of cilia ultrastructure from thick sections. Proceedings of the National Academy of Sciences of the United States of America, 83(23), 9040–9044. McLeod, R. A., Kowal, J., Ringler, P., & Stahlberg, H. (2017). Robust image alignment for cryogenic transmission electron microscopy. Journal of Structural Biology, 197(3), 279–293. Messaoudii, C., Boudier, T., Sorzano, C. O. S., & Marco, S. (2007). TomoJ: Tomography software for three-dimensional reconstruction in transmission electron microscopy. BMC Bioinformatics, 8, 288. https://doi.org/10.1186/1471-2105-8-288. M€uller-Reichert, T., Kiewisz, R., & Redemann, S. (2018). Mitotic spindles revisited—New insights from 3D electron microscopy. Journal of Cell Science, 131(3), 1–8. Nicastro, D., Fu, X., Heuser, T., Tso, A., Porter, M. E., & Linck, R. W. (2011). Cryo-electron tomography reveals conserved features of doublet microtubules in flagella. PNAS, 108, E845–E853. https://doi.org/10.1073/pnas.1106178108. Nicastro, D., Schwartz, C., Pierson, J., Gaudette, R., Porter, M. E., & McIntosh, J. R. (2006). The molecular architecture of axonemes revealed by cryoelectron tomography. Science, 313(5789), 944–948. Nickell, S., F€orster, F., Linaroudis, A., Net, W. D., Beck, F., Hegerl, R., et al. (2005). TOM software toolbox: Acquisition and analysis for electron tomography. Journal of Structural Biology, 149(3), 227–234. Noble, A. J., & Stagg, S. M. (2015). Automated batch fiducial-less tilt-series alignment in Appion using Protomo. Journal of Structural Biology, 192, 270–278. https://doi.org/ 10.1016/j.jsb.2015.10.003. Noble, A. J., Dandey, V. P., Wei, H., Brasch, J., Chase, J., Acharya, P., et al. (2018). Routine single particle CryoEM sample and grid characterization by tomography. Elife, 7, e34257. Published 2018 May 29. https://doi.org/10.7554/eLife.34257.

References

O’Toole, E., van der Heide, P., Richard McIntosh, J., & Mastronarde, D. (2018). Large-scale electron tomography of cells using SerialEM and IMOD. In E. Hanssen (Ed.), Cellular Imaging. Biological and Medical Physics, Biomedical Engineering. Cham: Springer. https://doi.org/10.1007/978-3-319-68997-5_4. Oda, T., & Kikkawa, M. (2013). Novel structural labeling method using cryo-electron tomography and biotin–streptavidin system. Journal of Structural Biology, 183, 305–311. https:// doi.org/10.1016/j.jsb.2013.07.003. Penczek, P., Marko, M., Buttle, K., & Frank, J. (1995). Double-tilt electron tomography. Ultramicroscopy, 60, 393–410. https://doi.org/10.1016/0304-3991(95)00078-X. Penczek, P. A. (2010). Fundamentals of three-dimensional reconstruction from projections. Methods in Enzymology, 482, 1–33. Pilhofer, M., Ladinsky, M. S., McDowall, A. W., Petroni, G., Jensen, G. J., et al. (2011). Microtubules in bacteria: Ancient tubulins build a five-protofilament homolog of the eukaryotic cytoskeleton. PLoS Biology, 9(12), e1001213. Rez, P., Larsen, T., & Elbaum, M. (2016). Exploring the theoretical basis and limitations of cryo-STEM tomography for thick biological specimens. Journal of Structural Biology, 196, 466–478. https://doi.org/10.1016/j.jsb.2016.09.014. Schorb, M., Haberbosch, I., Hagen, W. J. H., Schwab, Y., & Mastronarde, D. N. (2019). Software tools for automated transmission electron microscopy. Nature Methods, 16(6), 471–477. Schur, F. K., Hagen, W. J., Rumlova´, M., Ruml, T., Mu¨ller, B., Kra¨usslich, H. G., et al. (2015). ˚ resolution. Nature, Structure of the immature HIV-1 capsid in intact virus particles at 8.8 A 517, 505–508. https://doi.org/10.1038/nature13838. Sorzano, C., Vargas, J., Oto´n, J., de la Rosa-Trevı´n, J. M., Vilas, J. L., Kazemi, M., et al. (2017). A survey of the use of iterative reconstruction algorithms in electron microscopy. BioMed Research International, 2017, Article ID: 6482567. Suloway, C., Pulokas, J., Fellmann, D., Cheng, A., Guerra, F., Quispe, J., et al. (2005). Automated molecular microscopy: The new Leginon system. Journal of Structural Biology, 151(1), 41–60. Suloway, C., Shi, J., Cheng, A., Pulokasc, J., Carragher, B., Potter, C. S., et al. (2009). Fully automated, sequential tilt-series acquisition with Leginon. Journal of Structural Biology, 167(1), 11–18. Thermo Scientific (2018). Tomography user manual, PN 1062977, revision L • 13-Apr-18. Thermo Fisher Scientific. TVIPS (2014). Tutorial of EM-TOOLS (25.09.2014). Gauting, Germany: TVIPS GmbH. Walther, P., Bauer, A., Wenske, N., Catanese, A., Garrido, D., Schneider, M., et al. (2018). STEM tomography of high-pressure frozen and freeze-substituted cells: A comparison of image stacks obtained at 200 kV or 300 kV. Histochemistry and Cell Biology, 150(5), 545–556. https://doi.org/10.1007/s00418-018-1727-0. Epub 2018 Sep 18. Weber, M. S., Wojtynek, M., & Medalia, O. (2019). Cellular and structural studies of eukaryotic cells by cryo-electron tomography. Cells, 8(1), 57. Published 2019 Jan 16. https://doi. org/10.3390/cells8010057. Wolf, S. G., Houben, L., & Elbaum, M. (2014). Cryo-scanning transmission electron tomography of vitrified cells. Nature Methods, 11, 423–428. https://doi.org/10.1038/ nmeth.2842. Yoshioka, C., Pulokas, J., Fellmann, D., Potter, C. S., Milligan, R. A., & Carragher, B. (2007). Automation of random conical tilt and orthogonal tilt data collection using feature-based correlation. Journal of Structural Biology, 159(3), 335–346.

177

178

CHAPTER 8 Automated acquisition of tilt series

Zheng, S. Q. (2012). UCSF tomography user manual. Howard Hughes Medical Institute and the Department of Biochemistry and Biophysics University of California at San Francisco. Zheng, S. Q. (2013). UCSF TOMO installation manual. K2 Version. Howard Hughes Medical Institute and the Department of Biochemistry and Biophysics University of California at San Francisco. Zheng, Q. S., Braunfeld, M. B., Sedat, J. W., & Agard, D. A. (2004). An improved strategy for automated electron microscopic tomography. Journal of Structural Biology, 147, 91–101. Zheng, S. Q., Keszthelyi, B., Branlund, E., Lyle, J. M., Braunfeld, M. B., Sedat, J. W., et al. (2006). UCSF tomography: An integrated software suite for real-time electron microscopic tomographic data collection, alignment, and reconstruction. Journal of Structural Biology, 157(1), 138–147. Zheng, S. Q., Kollman, J. M., Braunfeld, M. B., Sedat, J. W., & Agard, D. A. (2006). Automated acquisition of electron microscopic random conical tilt data sets. Journal of Structural Biology, 157, 148–155. Zheng, S. Q., Matsuda, A., Braunfeld, M. B., Sedat, J. W., & Agard, D. A. (2009). Dual axis target mapping and automated sequential acquisition of dual axis EM tomographic data. Journal of Structural Biology, 168, 323–331. Ziese, U., Janssen, A. H., Murk, J.-L., Geerts, W. J. C., Van der Krift, T., Verkleij, A. J., et al. (2002). Automated high-throughput electron tomography by pre-calibration of image shifts. Journal of Microscopy, 205, 187–200.