- Email: [email protected]
Double-arm optical tweezer system for precise and dexterous handling of micro-objects in 3D workspace
Optics and Lasers in Engineering 111 (2018) 65–70
Contents lists available at ScienceDirect
Optics and Lasers in Engineering journal homepage: www.elsevier.com/locate/optlaseng
Double-arm optical tweezer system for precise and dexterous handling of micro-objects in 3D workspace Yoshio Tanaka National Institute of Advanced Industrial Science and Technology (AIST), AIST Shikoku, 2217-14 Hayashi-cho, Takamatsu 761-0395, Japan
a r t i c l e
i n f o
Keywords: Dexterous handling Double-arm manipulator Dynamic microarray Three-dimensional micromanipulation Optical tweezers
a b s t r a c t Double-arm manipulators are unfamiliar as equipment used in microscopic work in biomedical laboratories, whereas they are prevalent in factory automation and humanoids. For non-contact micromanipulation in threedimensional (3D) workspaces, we propose and design a double-arm optical tweezer system that can easily exchange two types of end-effectors (i.e., optical landscapes for laser trapping) with a focus tunable lens and a microlens array. With a time-shared scanning approach under interactive personal computer (PC) mouse controls, the system can perform the precise and dexterous handling of micro-objects in a 3D workspace. As a proof of concept, we demonstrate the two-dimensional (2D) and 3D dexterous handling of microbeads in the motions of solving puzzle rings. We also demonstrate the precise and periodic patterning of microbeads for massive dynamic arrays. This double-arm system can be applied with versatile tools used for various non-contact micromanipulations in the biomedical field and for dynamic arrays in single cell and 3D biology.
1. Introduction Double-arm manipulators are often employed in intracytoplasmic sperm injection (ICSI) procedures in medical biology, in which a glass needle attached to one arm is used to deliver sperm into an egg cell and a glass pipette attached to a second arm is used to hold the egg cell [1]. However, mechanical double-arm manipulators are unfamiliar as equipment used in microscopic work in biomedical laboratories, except ICSI, whereas they are prevalent in factory automation [2] and humanoids [3]. One reason for this is that micromanipulation with mechanical arms and hands causes inevitable physical contact with other objects, and these contacts lead to undesired adhesions between the end-effectors and objects because the adhesion force is dominant in a microscopic environment [4]; consequently, these adhesions prevent mechanical manipulators from performing precise or automated positioning tasks, such as the pickup-and-release of objects into desired positions, with a high efficiency. Conversely, optical tweezers—first demonstrated by Ashkin et al. [5] and extended to holographic [6], generalized phase contrast (GPC) [7], and time-shared scanning (TSS) tweezers [8] to trap multiple objects simultaneously—are well-established in non-contact micromanipulation techniques with high accuracy. Because the objective lens of a microscope used for optical tweezers functions as both a generator of manipulating forces and an observer of a workspace, optical tweezers combined with image processing techniques are particularly suitable for the automated and simultaneous micromanipulation of multiple objects [9–17]. Moreover, because one linear polarized beam emanated from
a single laser source can be divided into two (i.e., p- and s-polarized) beams that never interfere with each other, double-arm (i.e., dual-trap) optical tweezers can be easily constructed without the introduction of an additional laser source [18–21] when compared with mechanical micromanipulators. For various micromanipulation tasks in a true three-dimensional (3D) workspace, we propose and design a double-arm optical tweezer system that can easily exchange end-effectors (i.e., optical landscapes for laser trapping) with a focus tunable lens and a microlens array, which are inexpensive key optical components used to form various optical landscapes. With the TSS approach under interactive control by personal computer (PC) mice, this double-arm system can perform the precise and dexterous handling of micro-objects in a 3D workspace. As a proof of concept, we demonstrate the two-dimensional (2D) and the 3D dexterous handlings of multiple microbeads in the motions of solving puzzle rings. We also demonstrate the precise and periodic patterning of microbeads for massive dynamic arrays. 2. Double-arm optical tweezer system For the manipulation of two spheres or the orientation control of a non-spherical object, dual-trap optical tweezers [18,19] have been developed using two divided (i.e., p- and s-polarized) beams since the early 1990s when multi-beam techniques including holograms, GPC, and TSS were not yet invented. However, if the intention is to manipulate a nonspherical object with six degrees of freedom (DOF), early dual-trap opti-
E-mail address: [email protected] https://doi.org/10.1016/j.optlaseng.2018.07.019 Received 22 May 2018; Received in revised form 20 July 2018; Accepted 29 July 2018 0143-8166/© 2018 The Author. Published by Elsevier Ltd. This is an open access article under the CC BY license. (http://creativecommons.org/licenses/by/4.0/)
Y. Tanaka
Optics and Lasers in Engineering 111 (2018) 65–70
cal tweezers do not have a sufficient control performance, and they must be extended to a 3D three-trap system by introducing an additional laser source [22]. Conversely, although 3D multi-trap optical tweezers using a single laser source have been demonstrated by multi-beam techniques (e.g., holograms [23] and counterpropagating-beam GPC [24]) since the middle 2000s, their achievements are still limited to the simple motions of trapped objects in a 3D workspace—the trajectories of trapped objects do not interfere with each other—and the 3D orientation control of an uncomplicated object. Here, to perform precise and dexterous handling of multiple micro-objects in the complex motions of solving puzzle rings, we design a double-arm optical tweezer system that can easily exchange two types of end-effectors (namely, optical landscapes for laser trapping). Compared with conventional multi-trap optical tweezers including our previous reported systems, the easy exchangeable end-effectors, which are the great advantage of this double-arm system, can generate the flexible and complex true 3D (or massive) optical traps based on TSS approach, without expensive spatial light modulators (SLMs). Fig. 1 shows the optical and control system configurations of doublearm optical tweezers with replaceable optical elements including an electrical focus tunable lens (LZ : Optotune EL-10-30-NIR-LD) for zcoordinate steering and a microlens array (LA : Newport MALS14) for massive periodic patterning. This double-arm system is a multi-trap and two-beam optical tweezer in which two beams divided by a polarized beam splitter (PBS1 ) individually configure the multi-trap optical tweezers (i.e., a true 3D-TSS with LZ or periodic 2D with LA ) based on the TSS technique. These two types of TSS-based multi-trap optical tweezers that compose the double-arm system employ the same 2f relay optical system (not a 4f afocal relay system) under a common relay lens (LR ) used in the TSS part with a 2-axis scanning gimbal-mirror (GM1 or GM2 ). The detailed layouts for these two types of TSS tweezers based on the 2f relay system are illustrated in Fig. 1(b) and (c). Although 4f afocal relay systems are often employed in holographic or GPC optical tweezers that need an SLM, optical tweezers based on the 2f relay systems require a fewer number of optical elements resulting in an improved light transmission, less potential optical aberrations, and a simpler alignment. Moreover, for the replacement of end-effectors with a reduced cost, we can also design these 2f systems to use the same beam shaping lens (L1 ) that directs a converging beam (or beamlets) through the GM1 to a focus at a distance fR prior to the relay lens (LR ), resulting in a collimated beam (or beamlets) incident on the objective lens (LO ). The desired focal length (fZ ) of LZ and the optimal distance (dA1 ) between LA and L1 , which satisfy the necessary conditions for generating invariant trapping powers for all traps in a 3D workspace [25] and the objective imaging plane that equals the origin of the traps in the z-coordinate, are derived from the Gaussian lens equation [21,25,26] as follows: 𝑓𝑧 =
𝑓1 𝑓R , 𝑓1 + 𝑓R
𝑑𝐴1 =
𝑓12 𝑓𝑅
+ 𝑓𝐴 + 𝑓1 ,
double-arm system. Here, we demonstrate the 2D and 3D dexterous handlings of microbeads (Duke Scientific, borosilicate glass microsphere, 2.5 𝜇m) that dynamically form one set of puzzle rings by two true 3DTSS multi-trap optical tweezers (i.e., two 3D optical hands). Namely, we interactively manipulate two puzzle rings by two PC mice, in the sequential motions of solving puzzle rings, while the individual microbeads forming each puzzle ring maintain their relative distances to each other. For each 3D-TSS hand, we adjusted the laser power at the entrance aperture of the objective lens to the equivalent value (50 mW) and set the dwell time of TSS between 12 ms and 15 ms. Fig. 2 shows the video frame sequence of the 2D dexterous handling of microbeads, where the 24 microbeads indicated by colored circles dynamically form and maintain two question mark-shaped (‘?’-shaped) clusters (i.e., one set of ‘?’-shaped puzzle rings) in optical landscapes generated by two true 3D-TSS optical tweezers. First, the initial loading procedure for generating two ‘?’-shaped clusters of microbeads was performed under a similar algorithm to that in our previous paper used for dynamical microbead arrays [13]. The recognition algorithm based on the circular Hough transform and the collisionless path planning algorithm were executed to automatically gather microbeads up to the 12th nearest neighborhood of each origin (OL/R ) and to form two ‘?’-shaped clusters (as shown in Fig. 2(a)–(c)). Second, under bimanual control using two PC mice, these ‘?’-shaped clusters indicated by white and yellow circles were able to independently rotate around each origin in which one microbead indicated by a red circle was trapped. Each origin was also able to move toward an arbitrary position using the drag motion of a PC mouse; therefore, two ‘?’-shaped clusters were able to interactively manipulate to gradually link up with each other at their clasper parts (Fig. 2(c)–(f)), while the individual microbeads forming one set of puzzle rings maintain ‘?’ shapes. Finally, the two clusters that were entwined with each other were manipulated again to unbind their claspers, while they were rotated around each origin (Fig. 2(f)–(h)). Thus, we can interactively perform the 2D dexterous handling of two micro-objects that are trapped by two optical hands (i.e., optical multiple-force clamps [27]) using two PC mice. In another demonstration shown in Fig. 3, fourteen microbeads forming one set of puzzle rings (i.e., two broken-octagons) are dexterously manipulated for entwining and disentwining with themselves in a 2D and 3D workspace. Fig. 3(a) shows a video frame sequence for the interactive crossing of the two broken-octagons in the same z-coordinates and their morphing into a 3D workspace, in which the red and green circles indicate the microbeads that form the two broken-octagons that are controlled by the left-handed and right-handed mice, respectively. After the interactive rotation and translation of the right broken-octagon (Fig. 3(a1) and (a2)), the two broken-octagons separating from each other were entwined in the same z-coordinates (Fig. 3(a3)). Subsequently, these entwined broken-octagons in the same z-coordinates were morphed into a 3D workspace by z-coordinate control (Fig. 3(a4)), while maintaining their xy-coordinates before and after the morphing. The 3D rendered views of the relative positions of the microbeads for this morphing are illustrated in Fig. 3 (ca3) and (ca4). Fig. 3(b) shows the 2nd video frame sequence for the interactive crossing of the two broken-octagons that are initially trapped in a 3D workspace. First, the two broken-octagons separating from each other in the Cartesian coordinate system (as illustrated in Fig. 3(cb1)) were bimanually controlled to approach each other while facing their broken parts (Fig. 3(b1)). Subsequently, in Fig. 3(b2)–(b4), the 3D brokenoctagon indicated by green circles was interactively entwined with another broken-octagon indicated by red circles while revolving on its own z-axis. The 3D rendered views of the relative positions of the microbeads for this entwining are illustrated in Fig. 3(cb2)–(cb4). Thus, we can perform the 3D dexterous handling of two micro-objects by the TSS approach combined with 3D interactive pointing and control devices (i.e., two PC mice).
(1)
(2)
where f1 , fR , and fA are the focal lengths of L1 , LR , and LA , respectively. In the case of our design (f1 = 120 mm, fR = 170 mm), the calculated fz is 70.3 mm, which is almost centered in the tunable focal range. Thus, we can easily exchange the two types of end-effectors for the double-arm system by adjusting the position of L1 to the designed distance (i.e., 2f1 or f1 ) and by the replacement of an optical element (i.e., LZ or LA ) equipped with a simple alignment mechanism. 3. Demonstrations 3.1. Two 3D optical hands In this section, to demonstrate the advanced performance against conventional dual-trap and 3D multi-trap optical tweezers with an SLM, we select two true 3D-TSS optical tweezers for the abovementioned 66
Y. Tanaka
Optics and Lasers in Engineering 111 (2018) 65–70
Fig. 1. Double-arm optical tweezers with replaceable end-effectors. (a) Schematic of the optical and control system configurations. Detailed layout of the endeffectors: (b) the electrical focus tunable lens (LZ ) for z-coordinate steering and (c) the microlens array (LA ) for massive periodic patterning.
67
Y. Tanaka
Optics and Lasers in Engineering 111 (2018) 65–70
Fig. 2. (Video1, MOV, 9.1 MB) Video frame sequence of the 2D dexterous handling of microbeads. The microbeads indicated by white and yellow circles form two puzzle rings, while the microbeads indicated by red circles are trapped in the left and right origins for interactive mouse control. The dwell time of TSS is 12 ms for each trap, and the lengths of the red scale bars are 10 𝜇m.
(a)
z
z
z
z
Z3 2 2
(c)
OR
x
1
2
OL 1 2
(a3)
1 7 7 OR
y
x
1 OL
Z2
Z2
1
Z1 Z0
7 OL
4 1
y
(a4)
z
Z3
7 OR
x
4
Z1 Z0 7
y
1
7
x (b1)
7
6
y
6
1 7
y
x (b2to3)
(b4)
Fig. 3. (Video2, MOV, 6.5 MB) Video frame sequences of the 3D dexterous handling of microbeads, where fourteen microbeads indicated by red and green circles form two broken-octagons. (a) Interactive crossing of two broken-octagons in the same z-coordinates and its morphing into a 3D workspace. (b) Interactive crossing of two broken-octagons in a 3D workspace. The dwell time of TSS is 15 ms for each trap, and the lengths of the white scale bars are 10 𝜇m each. (c) 3D rendered views of the relative positions of microbeads for the snapshots from video frame sequences, (a) and (b).
68
Y. Tanaka
Optics and Lasers in Engineering 111 (2018) 65–70
Fig. 4. (Video3, MOV, 3.9 MB) Periodic arrangements of microbeads (2.0 𝜇m). (a) Tilted roman alphabets ‘T’, (b) roman alphabets ‘U’, (c) squares by fluorescent microbeads, and (d) striped pattern by fluorescent microbeads. The total laser powers of beamlets are 430 mW for (a, b) and 840 mW for (c, d). The lengths of the white scale bars are 10 𝜇m each.
3.2. Optical hands with a microlens array
4. Conclusion
In this section, to demonstrate the precise and periodic patterning of many microbeads into various optical landscapes using a double-arm optical tweezer system, we select a true 3D tweezers with LZ and a periodic 2D tweezers with LA for the abovementioned double-arm system. Because we have already described the patterning principle and the features for this type of double-arm system in our previous papers [21,28], the results of typical massive arrays are only shown here to demonstrate the effectiveness of the double-arm system. We set the dwell time of TSS between 10 ms and 12 ms for periodic patterning, and we also set the laser power of the true 3D tweezers for initial loading at 60 mW. The optical landscape for laser traps forming a 2D periodical pattern can be simply generated using the TSS multi-trap optical tweezers with LA , and the loading of microbeads into the pattern can be easily performed by the pickup-and-release procedure using a PC mouse equipped on true 3D optical tweezers. In Fig. 4(a) and (b), the microbeads (Polysciences, Polybead® Polystyrene, 2.0 𝜇m) are arranged regularly in the periodic roman alphabetic pattern: tilted ‘T’ and ‘U’, respectively; these patterns can also be geometrically transformed (i.e., their primitive patterns can be rotated, expanded, and shrunk around their latticed focal points with a microlens array [28]), if necessary (Video3, MOV, 3.9 MB). In Fig. 4(c) and (d), the fluorescent (Molecular Probes, Fluo Spheres® , 2.0 𝜇m) and non-fluorescent (Polysciences, Polybead® Polystyrene, 2.0 𝜇m) microbeads are arranged alternately in an 8 × 8 array to form squares and stripes, respectively. In general, undesired stacks of microbeads often arise in a massive array composed of over 50 microbeads [29]. However, under the removal procedure using 3D tweezers, we can construct massive arrays without the undesired stacks of microbeads.
We have designed a double-arm optical tweezer system that can easily exchange end-effectors to form various optical landscapes for laser trapping, and we demonstrated three typical examples of dexterous handling and precise patterning in a 2D and 3D workspace. Although we have dealt with the relative motion of the two sets of unconnected microspheres (i.e., multiple microbeads that were simultaneously trapped by two 3D TSS-based optical tweezers), this double-arm system enables us to precisely control the relative motion of the two sets of 3D complex (or flexible such as DNAs [30,31]) structures as well as 3D probes [32,33] in a 3D workspace. Therefore, the double-arm 3D optical tweezers will open up new possibilities in the biomedical field, particularly in single cell and 3D biology as well as in ICSI. Furthermore, double-arm 3D optical tweezers combined with a natural user interface to control multiple cursors in a 3D workspace (e.g., a Leap Motion controller [34]) may provide us a user-friendly micromanipulation platform to perform more complex tasks in a 3D workspace, although two PC mice are insufficient user interface tools for bimanual micromanipulation. Funding This work was partly supported by Japan Society for Promotion of Science (JSPS) KAKENHI [Grant No. JP15K05921]. Conflict of interest None.
69
Y. Tanaka
Optics and Lasers in Engineering 111 (2018) 65–70
Supplementary materials
[15] Banerjee AG, Chowdhury S, Gupta SK. Optical tweezers: Autonomous robots for the manipulation of biological cells. IEEE Robot Autom Mag 2014;21:81–8. [16] Xie M, Wang Y, Feng G, Sun D. Automated pairing manipulation of biological cells with a robot-tweezers manipulation system. IEEE/ASME Trans Mechatron 2015;20:2241–51. [17] Hu S, Chen S, Chen S, Xu G, Sun D. Automated transportation of multiple cell types using a robot-aided cell manipulation system with holographic optical tweezers. IEEE/ASME Trans Mechatron 2017;22:804–14. [18] Misawa H, Sasaki K, Koshioka M, Kitamura N, Masuhara H. Laser manipulation and assembling of polymer latex particles in solution. Macromolecules 1993;26:282–6. [19] Fällman E, Axner O. Design for fully steerable dual-trap optical tweezers. Appl Opt 1997;36:2107–13. [20] Tanaka Y, Tsutsui S, Ishikawa M, Kitajima H. Hybrid optical tweezers for dynamic micro-bead arrays. Opt Express 2011;19:15445–51. [21] Tanaka Y, Wakida S. Time-shared optical tweezers with a microlens array for dynamic microbead arrays. Biomed Opt Express 2015;6:3670–7. [22] Tanaka Y, Hirano K, Nagata H, Ishikawa M. Real-time three-dimensional orientation control of non-spherical micro-objects using laser trapping. Electron Lett 2007;43:412–14. [23] Sinclair S, Jordan P, Courtial J, Padgett M, Cooper J, Laczik ZJ. Assembly of 3-dimensional structures using programmable holographic optical tweezers. Opt Express 2004;12:5475–80. [24] Rodrigo PJ, Gammelgaard L, Bøggild P, Perch-Nielsen IR, Glückstad J. Actuation of microfabricated tools using multiple GPC-based counterpropagating-beam traps. Opt Express 2005;13:6899–904. [25] Tanaka Y. 3D multiple optical tweezers based on time-shared scanning with a fast focus tunable lens. J Opt 2013;15:025708. [26] Nussbaum A. Optical system design. Upper Saddle River, NJ: Prentice-Hall; 1998. [27] Tanaka Y, Wakida S. Controlled 3D rotation of biological cells using optical multiple-force clamps. Biomed Opt Express 2014;5:2341–8. [28] Tanaka Y. Kaleidoscopic patterning of micro-objects based on software-oriented approach using dual optical tweezers with a microlens array. Micro Nano Lett 2017;12:689–92. [29] Shaw LA, Panas RM, Spadaccini CM, Hopkins JB. Scanning holographic optical tweezers. Opt Lett 2017;42:2862–5. [30] Arai Y, Yasuda R, Akashi K, Harada Y, Miyata H, Kinosita Jr K, et al. Tying a molecular knot with optical tweezers. Nature 2003;399:446–8. [31] Brouwer I, Sitters G, Candelli A, Heerema SJ, Heller I, de Melo AJ, et al. Sliding sleeves of XRCC4–XLF bridge DNA and connect fragments of broken DNA. Nature 2016;535:566–9. [32] Palima D, Bañas AR, Vizsnyiczai G, Kelemen L, Ormos P, Glückstad J. Wave-guided optical waveguides. Opt Express 2012;20:2004–14. [33] Phillips DB, Gibson GM, Bowman R, Padgett MJ, Hanna S, Carberry DM, et al. An optically actuated surface scanning probe. Opt Express 2012;20:29679–93. [34] Tomori Z, Keša P, Nikorovič M, Kaňka J, Jákl P, Šerý M, et al. Holographic Raman tweezers controlled by multi-modal natural user interface. J Opt 2016;18:015602.
Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.optlaseng.2018.07.019. References [1] Zareinejad M, Rezaei SM, Abdullah A, Ghidary SS. Development of a piezo-actuated micro-teleoperation system for cell manipulation. Int J Med Robotics Comput Assist Surg 2009;5:66–76. [2] Yang HZ, Yamafuji K, Tanaka T, Moromugi S. Development of a robotic system which assists unmanned production based on cooperation between off-line robots and on-line robots. Part 3. Development of an off-line robot, autonomous navigation, and detection of faulty workpieces in a vibrating parts feeder. Int J Adv Manufac Technol 2000;16:582–90. [3] Inoue K, Nishihama Y, Arai T, Mae Y. Mobile manipulation of humanoid robots – Body and leg control for dual arm manipulation. Proc IEEE Int Conf Robot Autom 2002:2259–64. [4] Horade M, Kojima M, Kamiyama K, Kurata T, Mae Y, Arai T. Development of an optimum end-effector with a nano-scale uneven surface for non-adhesion cell manipulation using a micro-manipulator. J Micromech Microeng 2015;25:115002. [5] Ashkin A, Dziedzic JM, Bjorkholm JE, Chu S. Observation of a single-beam gradient force optical trap for dielectric particles. Opt Lett 1986;11:288–90. [6] Grier DG. A revolution in optical manipulation. Nature 2003;424:810–16. [7] Rodrigo PJ, Eriksen RL, Daria VR, Glückstad J. Interactive light-driven and parallel manipulation of inhomogeneous particles. Opt Express 2002;26:1550–6. [8] Arai F, Yoshikawa K, Sakami T, Fukuda T. Synchronization laser micromanipulation of multiple targets along each trajectory by single laser. Appl Phys Lett 2004;85:4301–3. [9] Chapin SC, Germain V, Dufresue ER. Automated trapping, assembly, and sorting with holographic optical tweezers. Opt Express 2006;14:13095–100. [10] Rodrigo PJ, Kelemen L, Alonzo CA, Nielsen IRP, Dam JS, Ormos P, et al. 2D optical manipulation and assembly of shape-complementary planar microstructures. Opt Express 2007;15:9009–14. [11] Tanaka Y, Kawada H, Hirano K, Ishikawa M, Kitajima H. Automated manipulation of non-spherical micro-objects using optical tweezers combined with image processing techniques. Opt Express 2008;16:15115–22. [12] Rodrigo PJ, Kelemen L, Palima D, Alonzo CA, Ormos P, Glückstad J. Optical microassembly platform for constructing reconfigurable microenvironments for biomedical studies. Opt Express 2009;17:6578–83. [13] Tanaka Y, Kawada H, Tsutsui S, Ishikawa M, Kitajima H. Dynamic micro-bead arrays using optical tweezers combined with intelligent control techniques. Opt Express 2009;17:24102–11. [14] Banerjee AG, Chowdhury S, Losert W, Gupta SK. Real-time path planning for coordinated transport of multiple particles using optical tweezers. IEEE Trans Autom Sci Eng 2012;9:669–78.
70
Contents lists available at ScienceDirect
Optics and Lasers in Engineering journal homepage: www.elsevier.com/locate/optlaseng
Double-arm optical tweezer system for precise and dexterous handling of micro-objects in 3D workspace Yoshio Tanaka National Institute of Advanced Industrial Science and Technology (AIST), AIST Shikoku, 2217-14 Hayashi-cho, Takamatsu 761-0395, Japan
a r t i c l e
i n f o
Keywords: Dexterous handling Double-arm manipulator Dynamic microarray Three-dimensional micromanipulation Optical tweezers
a b s t r a c t Double-arm manipulators are unfamiliar as equipment used in microscopic work in biomedical laboratories, whereas they are prevalent in factory automation and humanoids. For non-contact micromanipulation in threedimensional (3D) workspaces, we propose and design a double-arm optical tweezer system that can easily exchange two types of end-effectors (i.e., optical landscapes for laser trapping) with a focus tunable lens and a microlens array. With a time-shared scanning approach under interactive personal computer (PC) mouse controls, the system can perform the precise and dexterous handling of micro-objects in a 3D workspace. As a proof of concept, we demonstrate the two-dimensional (2D) and 3D dexterous handling of microbeads in the motions of solving puzzle rings. We also demonstrate the precise and periodic patterning of microbeads for massive dynamic arrays. This double-arm system can be applied with versatile tools used for various non-contact micromanipulations in the biomedical field and for dynamic arrays in single cell and 3D biology.
1. Introduction Double-arm manipulators are often employed in intracytoplasmic sperm injection (ICSI) procedures in medical biology, in which a glass needle attached to one arm is used to deliver sperm into an egg cell and a glass pipette attached to a second arm is used to hold the egg cell [1]. However, mechanical double-arm manipulators are unfamiliar as equipment used in microscopic work in biomedical laboratories, except ICSI, whereas they are prevalent in factory automation [2] and humanoids [3]. One reason for this is that micromanipulation with mechanical arms and hands causes inevitable physical contact with other objects, and these contacts lead to undesired adhesions between the end-effectors and objects because the adhesion force is dominant in a microscopic environment [4]; consequently, these adhesions prevent mechanical manipulators from performing precise or automated positioning tasks, such as the pickup-and-release of objects into desired positions, with a high efficiency. Conversely, optical tweezers—first demonstrated by Ashkin et al. [5] and extended to holographic [6], generalized phase contrast (GPC) [7], and time-shared scanning (TSS) tweezers [8] to trap multiple objects simultaneously—are well-established in non-contact micromanipulation techniques with high accuracy. Because the objective lens of a microscope used for optical tweezers functions as both a generator of manipulating forces and an observer of a workspace, optical tweezers combined with image processing techniques are particularly suitable for the automated and simultaneous micromanipulation of multiple objects [9–17]. Moreover, because one linear polarized beam emanated from
a single laser source can be divided into two (i.e., p- and s-polarized) beams that never interfere with each other, double-arm (i.e., dual-trap) optical tweezers can be easily constructed without the introduction of an additional laser source [18–21] when compared with mechanical micromanipulators. For various micromanipulation tasks in a true three-dimensional (3D) workspace, we propose and design a double-arm optical tweezer system that can easily exchange end-effectors (i.e., optical landscapes for laser trapping) with a focus tunable lens and a microlens array, which are inexpensive key optical components used to form various optical landscapes. With the TSS approach under interactive control by personal computer (PC) mice, this double-arm system can perform the precise and dexterous handling of micro-objects in a 3D workspace. As a proof of concept, we demonstrate the two-dimensional (2D) and the 3D dexterous handlings of multiple microbeads in the motions of solving puzzle rings. We also demonstrate the precise and periodic patterning of microbeads for massive dynamic arrays. 2. Double-arm optical tweezer system For the manipulation of two spheres or the orientation control of a non-spherical object, dual-trap optical tweezers [18,19] have been developed using two divided (i.e., p- and s-polarized) beams since the early 1990s when multi-beam techniques including holograms, GPC, and TSS were not yet invented. However, if the intention is to manipulate a nonspherical object with six degrees of freedom (DOF), early dual-trap opti-
E-mail address: [email protected] https://doi.org/10.1016/j.optlaseng.2018.07.019 Received 22 May 2018; Received in revised form 20 July 2018; Accepted 29 July 2018 0143-8166/© 2018 The Author. Published by Elsevier Ltd. This is an open access article under the CC BY license. (http://creativecommons.org/licenses/by/4.0/)
Y. Tanaka
Optics and Lasers in Engineering 111 (2018) 65–70
cal tweezers do not have a sufficient control performance, and they must be extended to a 3D three-trap system by introducing an additional laser source [22]. Conversely, although 3D multi-trap optical tweezers using a single laser source have been demonstrated by multi-beam techniques (e.g., holograms [23] and counterpropagating-beam GPC [24]) since the middle 2000s, their achievements are still limited to the simple motions of trapped objects in a 3D workspace—the trajectories of trapped objects do not interfere with each other—and the 3D orientation control of an uncomplicated object. Here, to perform precise and dexterous handling of multiple micro-objects in the complex motions of solving puzzle rings, we design a double-arm optical tweezer system that can easily exchange two types of end-effectors (namely, optical landscapes for laser trapping). Compared with conventional multi-trap optical tweezers including our previous reported systems, the easy exchangeable end-effectors, which are the great advantage of this double-arm system, can generate the flexible and complex true 3D (or massive) optical traps based on TSS approach, without expensive spatial light modulators (SLMs). Fig. 1 shows the optical and control system configurations of doublearm optical tweezers with replaceable optical elements including an electrical focus tunable lens (LZ : Optotune EL-10-30-NIR-LD) for zcoordinate steering and a microlens array (LA : Newport MALS14) for massive periodic patterning. This double-arm system is a multi-trap and two-beam optical tweezer in which two beams divided by a polarized beam splitter (PBS1 ) individually configure the multi-trap optical tweezers (i.e., a true 3D-TSS with LZ or periodic 2D with LA ) based on the TSS technique. These two types of TSS-based multi-trap optical tweezers that compose the double-arm system employ the same 2f relay optical system (not a 4f afocal relay system) under a common relay lens (LR ) used in the TSS part with a 2-axis scanning gimbal-mirror (GM1 or GM2 ). The detailed layouts for these two types of TSS tweezers based on the 2f relay system are illustrated in Fig. 1(b) and (c). Although 4f afocal relay systems are often employed in holographic or GPC optical tweezers that need an SLM, optical tweezers based on the 2f relay systems require a fewer number of optical elements resulting in an improved light transmission, less potential optical aberrations, and a simpler alignment. Moreover, for the replacement of end-effectors with a reduced cost, we can also design these 2f systems to use the same beam shaping lens (L1 ) that directs a converging beam (or beamlets) through the GM1 to a focus at a distance fR prior to the relay lens (LR ), resulting in a collimated beam (or beamlets) incident on the objective lens (LO ). The desired focal length (fZ ) of LZ and the optimal distance (dA1 ) between LA and L1 , which satisfy the necessary conditions for generating invariant trapping powers for all traps in a 3D workspace [25] and the objective imaging plane that equals the origin of the traps in the z-coordinate, are derived from the Gaussian lens equation [21,25,26] as follows: 𝑓𝑧 =
𝑓1 𝑓R , 𝑓1 + 𝑓R
𝑑𝐴1 =
𝑓12 𝑓𝑅
+ 𝑓𝐴 + 𝑓1 ,
double-arm system. Here, we demonstrate the 2D and 3D dexterous handlings of microbeads (Duke Scientific, borosilicate glass microsphere, 2.5 𝜇m) that dynamically form one set of puzzle rings by two true 3DTSS multi-trap optical tweezers (i.e., two 3D optical hands). Namely, we interactively manipulate two puzzle rings by two PC mice, in the sequential motions of solving puzzle rings, while the individual microbeads forming each puzzle ring maintain their relative distances to each other. For each 3D-TSS hand, we adjusted the laser power at the entrance aperture of the objective lens to the equivalent value (50 mW) and set the dwell time of TSS between 12 ms and 15 ms. Fig. 2 shows the video frame sequence of the 2D dexterous handling of microbeads, where the 24 microbeads indicated by colored circles dynamically form and maintain two question mark-shaped (‘?’-shaped) clusters (i.e., one set of ‘?’-shaped puzzle rings) in optical landscapes generated by two true 3D-TSS optical tweezers. First, the initial loading procedure for generating two ‘?’-shaped clusters of microbeads was performed under a similar algorithm to that in our previous paper used for dynamical microbead arrays [13]. The recognition algorithm based on the circular Hough transform and the collisionless path planning algorithm were executed to automatically gather microbeads up to the 12th nearest neighborhood of each origin (OL/R ) and to form two ‘?’-shaped clusters (as shown in Fig. 2(a)–(c)). Second, under bimanual control using two PC mice, these ‘?’-shaped clusters indicated by white and yellow circles were able to independently rotate around each origin in which one microbead indicated by a red circle was trapped. Each origin was also able to move toward an arbitrary position using the drag motion of a PC mouse; therefore, two ‘?’-shaped clusters were able to interactively manipulate to gradually link up with each other at their clasper parts (Fig. 2(c)–(f)), while the individual microbeads forming one set of puzzle rings maintain ‘?’ shapes. Finally, the two clusters that were entwined with each other were manipulated again to unbind their claspers, while they were rotated around each origin (Fig. 2(f)–(h)). Thus, we can interactively perform the 2D dexterous handling of two micro-objects that are trapped by two optical hands (i.e., optical multiple-force clamps [27]) using two PC mice. In another demonstration shown in Fig. 3, fourteen microbeads forming one set of puzzle rings (i.e., two broken-octagons) are dexterously manipulated for entwining and disentwining with themselves in a 2D and 3D workspace. Fig. 3(a) shows a video frame sequence for the interactive crossing of the two broken-octagons in the same z-coordinates and their morphing into a 3D workspace, in which the red and green circles indicate the microbeads that form the two broken-octagons that are controlled by the left-handed and right-handed mice, respectively. After the interactive rotation and translation of the right broken-octagon (Fig. 3(a1) and (a2)), the two broken-octagons separating from each other were entwined in the same z-coordinates (Fig. 3(a3)). Subsequently, these entwined broken-octagons in the same z-coordinates were morphed into a 3D workspace by z-coordinate control (Fig. 3(a4)), while maintaining their xy-coordinates before and after the morphing. The 3D rendered views of the relative positions of the microbeads for this morphing are illustrated in Fig. 3 (ca3) and (ca4). Fig. 3(b) shows the 2nd video frame sequence for the interactive crossing of the two broken-octagons that are initially trapped in a 3D workspace. First, the two broken-octagons separating from each other in the Cartesian coordinate system (as illustrated in Fig. 3(cb1)) were bimanually controlled to approach each other while facing their broken parts (Fig. 3(b1)). Subsequently, in Fig. 3(b2)–(b4), the 3D brokenoctagon indicated by green circles was interactively entwined with another broken-octagon indicated by red circles while revolving on its own z-axis. The 3D rendered views of the relative positions of the microbeads for this entwining are illustrated in Fig. 3(cb2)–(cb4). Thus, we can perform the 3D dexterous handling of two micro-objects by the TSS approach combined with 3D interactive pointing and control devices (i.e., two PC mice).
(1)
(2)
where f1 , fR , and fA are the focal lengths of L1 , LR , and LA , respectively. In the case of our design (f1 = 120 mm, fR = 170 mm), the calculated fz is 70.3 mm, which is almost centered in the tunable focal range. Thus, we can easily exchange the two types of end-effectors for the double-arm system by adjusting the position of L1 to the designed distance (i.e., 2f1 or f1 ) and by the replacement of an optical element (i.e., LZ or LA ) equipped with a simple alignment mechanism. 3. Demonstrations 3.1. Two 3D optical hands In this section, to demonstrate the advanced performance against conventional dual-trap and 3D multi-trap optical tweezers with an SLM, we select two true 3D-TSS optical tweezers for the abovementioned 66
Y. Tanaka
Optics and Lasers in Engineering 111 (2018) 65–70
Fig. 1. Double-arm optical tweezers with replaceable end-effectors. (a) Schematic of the optical and control system configurations. Detailed layout of the endeffectors: (b) the electrical focus tunable lens (LZ ) for z-coordinate steering and (c) the microlens array (LA ) for massive periodic patterning.
67
Y. Tanaka
Optics and Lasers in Engineering 111 (2018) 65–70
Fig. 2. (Video1, MOV, 9.1 MB) Video frame sequence of the 2D dexterous handling of microbeads. The microbeads indicated by white and yellow circles form two puzzle rings, while the microbeads indicated by red circles are trapped in the left and right origins for interactive mouse control. The dwell time of TSS is 12 ms for each trap, and the lengths of the red scale bars are 10 𝜇m.
(a)
z
z
z
z
Z3 2 2
(c)
OR
x
1
2
OL 1 2
(a3)
1 7 7 OR
y
x
1 OL
Z2
Z2
1
Z1 Z0
7 OL
4 1
y
(a4)
z
Z3
7 OR
x
4
Z1 Z0 7
y
1
7
x (b1)
7
6
y
6
1 7
y
x (b2to3)
(b4)
Fig. 3. (Video2, MOV, 6.5 MB) Video frame sequences of the 3D dexterous handling of microbeads, where fourteen microbeads indicated by red and green circles form two broken-octagons. (a) Interactive crossing of two broken-octagons in the same z-coordinates and its morphing into a 3D workspace. (b) Interactive crossing of two broken-octagons in a 3D workspace. The dwell time of TSS is 15 ms for each trap, and the lengths of the white scale bars are 10 𝜇m each. (c) 3D rendered views of the relative positions of microbeads for the snapshots from video frame sequences, (a) and (b).
68
Y. Tanaka
Optics and Lasers in Engineering 111 (2018) 65–70
Fig. 4. (Video3, MOV, 3.9 MB) Periodic arrangements of microbeads (2.0 𝜇m). (a) Tilted roman alphabets ‘T’, (b) roman alphabets ‘U’, (c) squares by fluorescent microbeads, and (d) striped pattern by fluorescent microbeads. The total laser powers of beamlets are 430 mW for (a, b) and 840 mW for (c, d). The lengths of the white scale bars are 10 𝜇m each.
3.2. Optical hands with a microlens array
4. Conclusion
In this section, to demonstrate the precise and periodic patterning of many microbeads into various optical landscapes using a double-arm optical tweezer system, we select a true 3D tweezers with LZ and a periodic 2D tweezers with LA for the abovementioned double-arm system. Because we have already described the patterning principle and the features for this type of double-arm system in our previous papers [21,28], the results of typical massive arrays are only shown here to demonstrate the effectiveness of the double-arm system. We set the dwell time of TSS between 10 ms and 12 ms for periodic patterning, and we also set the laser power of the true 3D tweezers for initial loading at 60 mW. The optical landscape for laser traps forming a 2D periodical pattern can be simply generated using the TSS multi-trap optical tweezers with LA , and the loading of microbeads into the pattern can be easily performed by the pickup-and-release procedure using a PC mouse equipped on true 3D optical tweezers. In Fig. 4(a) and (b), the microbeads (Polysciences, Polybead® Polystyrene, 2.0 𝜇m) are arranged regularly in the periodic roman alphabetic pattern: tilted ‘T’ and ‘U’, respectively; these patterns can also be geometrically transformed (i.e., their primitive patterns can be rotated, expanded, and shrunk around their latticed focal points with a microlens array [28]), if necessary (Video3, MOV, 3.9 MB). In Fig. 4(c) and (d), the fluorescent (Molecular Probes, Fluo Spheres® , 2.0 𝜇m) and non-fluorescent (Polysciences, Polybead® Polystyrene, 2.0 𝜇m) microbeads are arranged alternately in an 8 × 8 array to form squares and stripes, respectively. In general, undesired stacks of microbeads often arise in a massive array composed of over 50 microbeads [29]. However, under the removal procedure using 3D tweezers, we can construct massive arrays without the undesired stacks of microbeads.
We have designed a double-arm optical tweezer system that can easily exchange end-effectors to form various optical landscapes for laser trapping, and we demonstrated three typical examples of dexterous handling and precise patterning in a 2D and 3D workspace. Although we have dealt with the relative motion of the two sets of unconnected microspheres (i.e., multiple microbeads that were simultaneously trapped by two 3D TSS-based optical tweezers), this double-arm system enables us to precisely control the relative motion of the two sets of 3D complex (or flexible such as DNAs [30,31]) structures as well as 3D probes [32,33] in a 3D workspace. Therefore, the double-arm 3D optical tweezers will open up new possibilities in the biomedical field, particularly in single cell and 3D biology as well as in ICSI. Furthermore, double-arm 3D optical tweezers combined with a natural user interface to control multiple cursors in a 3D workspace (e.g., a Leap Motion controller [34]) may provide us a user-friendly micromanipulation platform to perform more complex tasks in a 3D workspace, although two PC mice are insufficient user interface tools for bimanual micromanipulation. Funding This work was partly supported by Japan Society for Promotion of Science (JSPS) KAKENHI [Grant No. JP15K05921]. Conflict of interest None.
69
Y. Tanaka
Optics and Lasers in Engineering 111 (2018) 65–70
Supplementary materials
[15] Banerjee AG, Chowdhury S, Gupta SK. Optical tweezers: Autonomous robots for the manipulation of biological cells. IEEE Robot Autom Mag 2014;21:81–8. [16] Xie M, Wang Y, Feng G, Sun D. Automated pairing manipulation of biological cells with a robot-tweezers manipulation system. IEEE/ASME Trans Mechatron 2015;20:2241–51. [17] Hu S, Chen S, Chen S, Xu G, Sun D. Automated transportation of multiple cell types using a robot-aided cell manipulation system with holographic optical tweezers. IEEE/ASME Trans Mechatron 2017;22:804–14. [18] Misawa H, Sasaki K, Koshioka M, Kitamura N, Masuhara H. Laser manipulation and assembling of polymer latex particles in solution. Macromolecules 1993;26:282–6. [19] Fällman E, Axner O. Design for fully steerable dual-trap optical tweezers. Appl Opt 1997;36:2107–13. [20] Tanaka Y, Tsutsui S, Ishikawa M, Kitajima H. Hybrid optical tweezers for dynamic micro-bead arrays. Opt Express 2011;19:15445–51. [21] Tanaka Y, Wakida S. Time-shared optical tweezers with a microlens array for dynamic microbead arrays. Biomed Opt Express 2015;6:3670–7. [22] Tanaka Y, Hirano K, Nagata H, Ishikawa M. Real-time three-dimensional orientation control of non-spherical micro-objects using laser trapping. Electron Lett 2007;43:412–14. [23] Sinclair S, Jordan P, Courtial J, Padgett M, Cooper J, Laczik ZJ. Assembly of 3-dimensional structures using programmable holographic optical tweezers. Opt Express 2004;12:5475–80. [24] Rodrigo PJ, Gammelgaard L, Bøggild P, Perch-Nielsen IR, Glückstad J. Actuation of microfabricated tools using multiple GPC-based counterpropagating-beam traps. Opt Express 2005;13:6899–904. [25] Tanaka Y. 3D multiple optical tweezers based on time-shared scanning with a fast focus tunable lens. J Opt 2013;15:025708. [26] Nussbaum A. Optical system design. Upper Saddle River, NJ: Prentice-Hall; 1998. [27] Tanaka Y, Wakida S. Controlled 3D rotation of biological cells using optical multiple-force clamps. Biomed Opt Express 2014;5:2341–8. [28] Tanaka Y. Kaleidoscopic patterning of micro-objects based on software-oriented approach using dual optical tweezers with a microlens array. Micro Nano Lett 2017;12:689–92. [29] Shaw LA, Panas RM, Spadaccini CM, Hopkins JB. Scanning holographic optical tweezers. Opt Lett 2017;42:2862–5. [30] Arai Y, Yasuda R, Akashi K, Harada Y, Miyata H, Kinosita Jr K, et al. Tying a molecular knot with optical tweezers. Nature 2003;399:446–8. [31] Brouwer I, Sitters G, Candelli A, Heerema SJ, Heller I, de Melo AJ, et al. Sliding sleeves of XRCC4–XLF bridge DNA and connect fragments of broken DNA. Nature 2016;535:566–9. [32] Palima D, Bañas AR, Vizsnyiczai G, Kelemen L, Ormos P, Glückstad J. Wave-guided optical waveguides. Opt Express 2012;20:2004–14. [33] Phillips DB, Gibson GM, Bowman R, Padgett MJ, Hanna S, Carberry DM, et al. An optically actuated surface scanning probe. Opt Express 2012;20:29679–93. [34] Tomori Z, Keša P, Nikorovič M, Kaňka J, Jákl P, Šerý M, et al. Holographic Raman tweezers controlled by multi-modal natural user interface. J Opt 2016;18:015602.
Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.optlaseng.2018.07.019. References [1] Zareinejad M, Rezaei SM, Abdullah A, Ghidary SS. Development of a piezo-actuated micro-teleoperation system for cell manipulation. Int J Med Robotics Comput Assist Surg 2009;5:66–76. [2] Yang HZ, Yamafuji K, Tanaka T, Moromugi S. Development of a robotic system which assists unmanned production based on cooperation between off-line robots and on-line robots. Part 3. Development of an off-line robot, autonomous navigation, and detection of faulty workpieces in a vibrating parts feeder. Int J Adv Manufac Technol 2000;16:582–90. [3] Inoue K, Nishihama Y, Arai T, Mae Y. Mobile manipulation of humanoid robots – Body and leg control for dual arm manipulation. Proc IEEE Int Conf Robot Autom 2002:2259–64. [4] Horade M, Kojima M, Kamiyama K, Kurata T, Mae Y, Arai T. Development of an optimum end-effector with a nano-scale uneven surface for non-adhesion cell manipulation using a micro-manipulator. J Micromech Microeng 2015;25:115002. [5] Ashkin A, Dziedzic JM, Bjorkholm JE, Chu S. Observation of a single-beam gradient force optical trap for dielectric particles. Opt Lett 1986;11:288–90. [6] Grier DG. A revolution in optical manipulation. Nature 2003;424:810–16. [7] Rodrigo PJ, Eriksen RL, Daria VR, Glückstad J. Interactive light-driven and parallel manipulation of inhomogeneous particles. Opt Express 2002;26:1550–6. [8] Arai F, Yoshikawa K, Sakami T, Fukuda T. Synchronization laser micromanipulation of multiple targets along each trajectory by single laser. Appl Phys Lett 2004;85:4301–3. [9] Chapin SC, Germain V, Dufresue ER. Automated trapping, assembly, and sorting with holographic optical tweezers. Opt Express 2006;14:13095–100. [10] Rodrigo PJ, Kelemen L, Alonzo CA, Nielsen IRP, Dam JS, Ormos P, et al. 2D optical manipulation and assembly of shape-complementary planar microstructures. Opt Express 2007;15:9009–14. [11] Tanaka Y, Kawada H, Hirano K, Ishikawa M, Kitajima H. Automated manipulation of non-spherical micro-objects using optical tweezers combined with image processing techniques. Opt Express 2008;16:15115–22. [12] Rodrigo PJ, Kelemen L, Palima D, Alonzo CA, Ormos P, Glückstad J. Optical microassembly platform for constructing reconfigurable microenvironments for biomedical studies. Opt Express 2009;17:6578–83. [13] Tanaka Y, Kawada H, Tsutsui S, Ishikawa M, Kitajima H. Dynamic micro-bead arrays using optical tweezers combined with intelligent control techniques. Opt Express 2009;17:24102–11. [14] Banerjee AG, Chowdhury S, Losert W, Gupta SK. Real-time path planning for coordinated transport of multiple particles using optical tweezers. IEEE Trans Autom Sci Eng 2012;9:669–78.
70