- Email: [email protected]
360-degree viewable tabletop 3D display system based on integral imaging by using perspective-oriented layer
Optics Communications 438 (2019) 54–60
Contents lists available at ScienceDirect
Optics Communications journal homepage: www.elsevier.com/locate/optcom
360-degree viewable tabletop 3D display system based on integral imaging by using perspective-oriented layer Ling Luo a , Qiong-Hua Wang b ,∗, Yan Xing a , Huan Deng a , Hui Ren a , Sai Li a a b
School of Electronics and Information Engineering, Sichuan University, Chengdu 610065, China School of Instrumentation and Optoelectronic Engineering, Beihang University, Beijing 100191, China
ARTICLE
INFO
Keywords: Tabletop display 3D display Integral imaging Perspective-oriented layer
ABSTRACT We propose a 360-degree viewable tabletop three-dimensional (3D) display system based on integral imaging by using a perspective-oriented layer (POL). The proposed system is composed of a two-dimensional (2D) display device, a POL, a lens array with large pitch and a light shaping diffuser screen. The 2D display device with high refresh rate is utilized to present the adaptive element image array (AEIA) sequentially. The AEIA is generated based on partially-overlapping algorithm to ensure that the 3D image corresponding to adjacent viewing zone is reconstructed without flipping. The 2D display device, the POL and the lens array rotate simultaneously to create a 360-degree continuous viewing zone by time division multiplexing. In our proposed tabletop 3D display system, the best viewing zone can be modulated to the lateral and the crosstalk caused by the narrow viewing angle can be eliminated. In addition, the proposed tabletop 3D display system is capable of reconstructing 3D images with both vertical and horizontal parallaxes. The experimental results agree well with the basic principle of the proposed method.
1. Introduction The tabletop display can present information more intuitively and naturally compared with the conventional display when people sit around a table [1]. An ideal tabletop three-dimensional (3D) display should reconstruct 360-degree viewable 3D images, which can be shared with multiple viewers simultaneously in different directions. The reconstructed 3D images should also have correct field of view and spatial occlusion effect for the improved visual effect. Recently the tabletop 3D display has attracted much attention. Many studies combining the tabletop display with the 3D display have been researched [2–6]. Common 3D display technologies include integral imaging (II) display, volumetric display and holographic display, and these displays can reproduce the real-world 3D scene almost naturally [7–9]. The holographic display can be applied into the tabletop 3D display [10]. A time division method or an observer tracking unit is employed to achieve a 360◦ viewable display [11,12]. However, too much data and complex devices are required to generate the correct 3D images. A glasses-free 3D images of 360-degree can be observed based on volumetric display, but the showcase-like mechanical components are usually employed, which invades the tabletop space [13,14]. The II 3D display has the advantages of providing full-parallax and full-color 3D images, so it is one of the most promising 3D displays for achieving the ideal tabletop 3D display [15–20]. Current tabletop display systems based on II 3D display can be classified into two typical
systems. One is the system based on time division multiplexing and space division multiplexing [21,22]. The system usually only offers horizontal parallax, so the observers at different heights watch the same 3D images [23]. The other is a conventional wall display based on II lay down to be as a table display to achieve a 360-degree viewable tabletop 3D display [24]. With this arrangement, the viewing area of the reconstructed 3D images is turned from the front to the top. However, the best viewing direction is almost directly above the display due to the limited viewing angle of the II. When people sit around the table, the reconstructed 3D images of the system have serious crosstalk, which degrades the viewing experience. In this paper, we proposed a 360-degree viewable tabletop 3D display system based on II by using a perspective-oriented layer (POL). It is composed of a two-dimensional (2D) display device, a POL, a lens array with large pitch and a light shaping diffuser screen (LSDS). Our proposed system can provide both vertical and horizontal parallaxes with continuous viewpoints. The viewing angle of the reconstructed 3D images in the vertical direction can reach 40◦ . The viewing zone is modulated to the lateral viewing zone of the tabletop 3D display for the observers around the table. Crosstalk caused by the limitation of the narrow viewing angle is eliminated in our system, so multi-viewers can observe the correct reconstructed 3D images at the same time. LSDS is adopted to improve the quality of the reconstructed 3D images. We present an adaptive element image array (AEIA) generation method
∗ Corresponding author. E-mail address: [email protected] (Q.-H. Wang).
https://doi.org/10.1016/j.optcom.2019.01.013 Received 31 October 2018; Received in revised form 14 December 2018; Accepted 5 January 2019 Available online 9 January 2019 0030-4018/© 2019 Elsevier B.V. All rights reserved.
L. Luo, Q.-H. Wang, Y. Xing et al.
Optics Communications 438 (2019) 54–60
Fig. 1. Structure of the proposed system.
Fig. 2. Schematic diagram of the display process . (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
based on partially-overlapping algorithm to eliminate the flipping of adjacent viewing zone. The proposed display verification system is developed.
scene can be observed by multiple observers in 360-degree horizontal directions. Any adjacent viewing zone has an overlapped angle, which is large enough to ensure that 3D image corresponding to adjacent viewing zone is reconstructed without flipping.
2. Proposed tabletop 3D display system 2.2. Time division multiplexing scheme of the proposed system 2.1. Configuration and principle of the proposed system To realize 360-degree viewable tabletop 3D display, a time division multiplexing scheme is adopted to reconstruct 3D images successively. In our system, the 360-degree viewing space is divided into several viewing zones in the horizontal direction. Each viewing zone is equivalent to that of the conventional II system and it is denoted as 𝑧𝑖 , where i is within the range from 1 to n. We define the original viewing zone of our tabletop system as 𝑧1 . The viewing angle of each viewing zone is identical and it is denoted as 𝜃0 . Any adjacent viewing zones 𝑧𝑖 and 𝑧𝑖+1 are overlapped, and the interval angle is denoted as Δ𝜃. At a given moment 𝑡𝑖 , 3D image with viewing angle of [−𝜃0 ∕2 + (𝑖 − 1) × Δ𝜃, 𝜃0 ∕2 + (𝑖 − 1) × Δ𝜃], which corresponds to viewing zone 𝑧𝑖 , can be reconstructed by rotating the display device, the POL and the lens array. When the screen turns around a circle, the Δ𝜃 and the number of the refreshed image are mutually restrictive by the following equation:
The proposed 360-degree viewable tabletop 3D display system is mainly comprised of a 2D display device, a POL a lens array with large pitch and a LSDS, as shown in Fig. 1. The 2D display device, which can be achieved by an ultra-high-definition (UHD) panel with high refreshing rate or a screen projected by high-speed projector, is located at the bottom of the system to display the AEIA sequentially. The 2D display device, the POL and the lens array are assembled into a whole device on a rotating mechanism. The system realizes a 360-degree continuous viewable II tabletop display based on the time division multiplexing technology. The LSDS, which is made of special microstructure and can expand the input light with the expanding angle, is placed at the center depth plane to eliminate gaps of the reconstructed 3D image and improve the visual performance for the reconstructed 3D images. In the display process, light rays emitted from the pixels of the AEIA are modulated by the POL and propagate along the POL, so the subviewing zone in the traditional II display becomes the main viewing zone in the proposed system. And the 3D image of the viewing zone marked in red will be reconstructed, as shown in Fig. 2. Then we adopt a time division multiplexing scheme, the 3D image corresponding to the viewing zone marked in blue, green, brown et al. will be quickly reconstructed in succession. Depending on the visual persistence effect, a complete 3D
Δ𝜃 × 𝑁 = 360◦ ,
(1)
where N is the number of the refreshed image when the system rotates 360◦ . In our paper, we set Δ𝜃 = 1◦ and 𝑁 = 360. At moment 𝑡1 , the 3D image, which contains [−𝜃0 ∕2, 𝜃0 ∕2] information of the target 3D scene and is marked with red line fanshaped column in Fig. 3, will be reconstructed by light rays emitted from its corresponding AEIA. At moment 𝑡2 , the 3D image containing 55
L. Luo, Q.-H. Wang, Y. Xing et al.
Optics Communications 438 (2019) 54–60
Fig. 3. Principle of the time division multiplexing scheme adopted in the proposed tabletop 3D display system.
Fig. 4. (a) Structure of the POL, (b) vertical section of the special unit, and (c) installation diagram of the POL and the lens.
[−𝜃0 ∕2+Δ𝜃, 𝜃0 ∕2+Δ𝜃] information of target 3D scene, marked with blue line and fan-shaped column, will be reconstructed. At the same way, 3D image containing [360−𝜃0 ∕2, 360+𝜃0 ∕2] information of target 3D scene, marked with brown line and fan-shaped column, will be reconstructed at moment 𝑡𝑛 . Each reconstructed 3D image of the corresponding viewing zone will be obtained in turn, and a whole 360-degree viewable 3D image will be presented to the viewers based on the visual persistence effect.
a hole tilted along a certain direction with a tilted angle of 𝛼 off the 2D display device. The combinatorial cavity, which is composed of an aperture and a cavity, is used to make the lens install more firmly, as shown in Fig. 4(c). The lateral viewing zone, which faces the side of the tabletop display, is the best viewing zone for the observers around the tabletop display. In order to make the lateral viewing zone provide for the observers, the light rays only propagate along the tiling-mode modulating hole by the POL, as shown in Fig. 5(a). Thus, the observers who locate within the red-marked viewing zone can watch the corresponding 3D images, while the observers located at the other regions cannot observe these 3D images, as shown in Fig. 5(b). The viewing angles 𝛼ℎ in the horizontal direction and 𝛼𝑣 in the vertical direction can be deducted as
2.3. POL in the proposed system The viewing zone of the reconstructed 3D images is turned from the front to the top in the system where a conventional wall display based on II lies down to be as a table display. However, the best viewing direction, which is almost directly above the display due to the limited viewing angle of the II, cannot match the requirement of the tabletop 3D display. When people sit around the table, the reconstructed 3D images of these systems have serious crosstalk, which degrades the viewing experience. To solve these problems, we introduce the POL into our proposed system. As shown in Fig. 4(a), the POL is made up of special unit array. Each special unit is composed of a modulating hole and a combinatorial cavity, as shown in Fig. 4(b). The modulating hole is
𝑙 sin 𝛼 𝛼ℎ = 2 arctan , ( 2ℎ ) ( ) ℎ ℎ 𝛼𝑣 = arctan − arctan , ℎ∕ tan 𝛼 + 𝑤∕2 (ℎ∕ tan 𝛼) − 𝑤∕2
(2) (3)
where h, w and l are the height, width and length of the modulating hole, respectively. The vertical viewing angle is [𝛼−𝛼v ∕2, 𝛼+𝛼v ∕2]. In addition, the POL can also block light rays emitted from the adjacent lens. Thus, the crosstalk which is caused by the rays emitted from the 56
L. Luo, Q.-H. Wang, Y. Xing et al.
Optics Communications 438 (2019) 54–60
Fig. 5. Diagram on the modulation of the POL: (a) the modulation of the POL in the vertical direction, (b) the modulation of the POL in the horizontal directions . (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
Fig. 6. Relationship between the pickup parameters and the parameters of the display system.
Fig. 7. Diagram of the parallax image rectification.
pixels of neighboring elemental images also can be removed in the proposed tabletop display system.
the spacing 𝑉𝑑 is determined by the following equation: (( ( ))/ ( )) (𝐿 + 𝐺) 2𝑑 + 𝑑0 𝑝 𝑉𝑑 = 𝑟𝑜𝑢𝑛𝑑 , 𝐺 𝑝0
2.4. Generation of AEIA in the proposed system
(4)
where G is the distance between the lens array and the 2D display device, L is the distance between the camera array plane and the lens array, d is the diameter of the lens, 𝑑0 is the spacing between adjacent lens, p is the pitch of the lens array, and 𝑝0 is the size of the pixel. The number of the camera in the camera array for each viewing zone is determined by the parameters of the tabletop 3D display system, and it can be expressed as M × M, where ( ) 𝑀 = 𝑟𝑜𝑢𝑛𝑑 𝑝∕𝑝0 . (5)
In the proposed tabletop 3D display system, the AEIA corresponding to each viewing zone is generated efficiently. The generation process of the AEIA is divided into the following three steps. Firstly, in the pickup step, we set all pickup parameters and obtain the original parallax images by using a virtual camera array. Secondly, in the rectifying step, the homography matrices for each parallax image are calculated, and then the obtained parallax images are projected to the same plane. Finally, in the synthesizing step, the rectified parallax images are synthesized to create the AEIA for the corresponding viewing zone.
In the camera array, the cameras are arranged in a toed-in arrangement to ensure that the overlapped shooting area is maximized. However, such an arrangement causes a problem that the projection plane of each camera is not exactly coplanar with each other, as shown
In the pickup step, as shown in Fig. 6, the spacing of the adjacent cameras in both horizontal and vertical directions are denoted as 𝑉𝑑 ×𝑉𝑑 , 57
L. Luo, Q.-H. Wang, Y. Xing et al.
Optics Communications 438 (2019) 54–60 Table 1 Configuration parameters of the proposed tabletop 3D display system. Parameters of the tabletop 3D display system
Values
Tilted angle (𝜃) Number of the lens (M × N ) Pitch of the lens array (p) Focal length of the lens (f ) Length of the modulating hole (l) Width of the modulating hole (w) Pitch of the POL
60◦ 21 × 21 12.700 mm 12.700 mm 13.100 mm 13.100 mm 13.700 mm
3. Experiments and discussion In the verification experiment, the 360-degree viewable tabletop 3D display system is developed to verify the basic principle. As shown in Fig. 8, the proposed tabletop 3D display system is composed of a rotation device, a 23.5-inch Dell 4K monitor, a POL, a lens array and a LSDS. We use a rotating mechanism (Daheng Optics GCD-011200, with a speed of 1 r.p.m) to rotate the display device, the POL and the lens array simultaneously. The whole height of the POL is 15.800 mm. The height of the aperture is 1.200 mm, and the curvature radius of the aperture is 6.468 mm. The cavity for mounting lens is 1.100 mm in height, and 12.900 mm in inner diameter. The modulating hole and the cavity are made of black rigid resin, and the aperture is made of transparent resin. The resolution of the monitor is 3840 × 2160 pixels and the pixel density is 185 pixels per inch. The remaining parameters of the experiment setup are given in Table 1. Plano-convex lens is used in our experiment, and the curvature radius of the convex surface is 6.468 mm. The pitch of the special unit of the POL is 13.700 mm, and each lens covers 100 × 100 pixels. Three sets of experiments are performed to evaluate the display effect of the proposed tabletop 3D display system. The first experiment is designed to evaluate the effect of removing the crosstalk in our proposed system. The 3D scene contains a teapot and a ball model. AEIA corresponding to one viewing zone is generated. As shown in Fig. 9(a), in order to show the image shooting position in Fig. 9(b) and (c) clearly, the viewing space is divided into 360 segments and the center of the single viewing zone is assumed to be 0◦ . Figure 9(b) and (c) show the reconstructed 3D images from
Fig. 8. Experimental setup of the 360-degree viewable tabletop 3D display system.
in Fig. 7. The projection plane of each camera and the display plane are also not aligned as well, which will cause serious trapezoid distortion. Hence, the rectifying step is needed. Homography matrices of each camera are calculated to project the corresponding parallax images to the display plane. In this way, the projection plane of each camera can be coplanar by implementing the homography transformation, where the display plane corresponds to the tabletop plane. 𝑖 (𝑥, 𝑦) are interweaved to Finally, the rectified parallax images 𝐼𝑚,𝑛 generate the AEIA 𝐸𝑖 (x, y) for a certain viewing zone 𝑧𝑖 . Adjacent AEIAs 𝐸𝑖 (x, y) and 𝐸𝑖+1 (x, y) have the overlapped 3D information. The 𝐸𝑖 (x, y) can be derived by: ∑∑ ( ) 𝑖 𝐸 (𝑥, 𝑦) = 𝑀 2 𝐼𝑚,𝑛 𝑀 ⋅ 𝑖′ + 𝑚, 𝑀 ⋅ 𝑗 ′ + 𝑛 𝑚,𝑛 𝑖′ ,𝑗 ′
( ) × 𝛿 𝑥 − 𝑀 ⋅ 𝑖′ − 𝑚, 𝑦 − 𝑀 ⋅ 𝑗 ′ − 𝑛 , 𝑖′
(6)
𝑗′
where, and represent the index of the lens, m and n represent the index of the rectified parallax images, and the resolution of the AEIA is x × y pixels [25]. Based on the above-mentioned steps, the AEIA for all viewing zone can be generated successively, and continuous perspectives of the 3D scene in 360-degree viewing zone can be observed without flipping.
Fig. 9. (a) Schematic diagram of the partition of the 3D display space, (b) pictures taken from different directions with the POL, and (c) pictures taken from different directions without the POL. 58
L. Luo, Q.-H. Wang, Y. Xing et al.
Optics Communications 438 (2019) 54–60
Fig. 10. Different perspectives of the reconstructed 3D image in a certain viewing zone.
Fig. 11. Pictures taken from different positions of the teapot 3D images.
different directions in the case of adding the POL and removing the POL. Obviously, the proposed tabletop 3D display system can effectively eliminate the crosstalk and improve the quality of the reconstructed 3D images. Furthermore, the reconstructed images taken at 0◦ in both display systems almost show indistinguishableness, which indicates that the reconstructed 3D images by our proposed system hardly has any additional optical aberrations introduced. The second experiment is designed to verify the horizontal and vertical parallaxes of our proposed system. The 3D scene also consists of a teapot model and a ball model. We generate an AEIA for a certain viewing zone. As shown in Fig. 10, in the corresponding viewing zone, the pictures of the reconstructed 3D images are taken from the top view, the left view, the bottom view, the right view and the middle view,
respectively. The results show that the proposed system has obvious parallaxes in both the horizontal and vertical directions. The viewing angle in both the horizontal and vertical directions can achieve 40◦ in a single viewing zone. The third experiment is designed to prove the 360-degree viewable display effect of the proposed tabletop 3D display system. The low speed rotating device is used to simulate the effect of the high speed rotating device. The 3D scene is a teapot model. AEIAs corresponding to the different viewing zone are generated. The interval angle of adjacent viewing zone is set as 1◦ . As shown in Fig. 11, we show the teapot 3D images in different positions. The interval angle of the pictures between adjacent positions is 24◦ . Obviously, the teapot 3D images show no flipping effect within all of the viewing positions. 59
L. Luo, Q.-H. Wang, Y. Xing et al.
Optics Communications 438 (2019) 54–60
Consequently, a tabletop 3D information with 360-degree parallax in the horizontal direction and 40-degree parallax in the vertical direction can be presented to the observers by moving around the tabletop continuously.
[6] D. Zhao, B.Q. Su, G.W. Chen, H.E. Liao, 360-degree viewable floating autostereoscopic display using integral photography and multiple semitransparent mirrors, Opt. Express 23 (8) (2015) 9812–9823. [7] Q.H. Wang, 3D Display Technologies and Devices, Science Press, Beijing, 2011. [8] J. Geng, Three-dimensional display technologies, Adv. Opt. Photon. 5 (4) (2013) 456–535. [9] J.H. Park, K. Hong, B. Lee, Recent progress in three-dimensional information processing based on integral imaging, Appl. Opt. 48 (34) (2009) H77–H94. [10] Y. Lim, K. Hong, H. Kim, H.E. Kim, E.Y. Chang, S. Lee, T. Kim, J. Nam, H.G. Choo, J. Kim, J. Hahn, 360-degree tabletop electronic holographic display, Opt. Express 24 (22) (2016) 24999–25009. [11] Y. Sando, B. Daisuke, Y. Toyohiko, Holographic 3D display observable for multiple simultaneous viewers from all horizontal directions by using a time division method, Opt. Lett. 39 (19) (2014) 5555–5557. [12] D.D. Teng, L.L. Liu, Z.X. Wang, B.X. Sun, B.W. Wang, All-around holographic threedimensional light field display, Opt. Commun. 285 (21–22) (2012) 4235–4240. [13] G.E. Favalore, Volumetric 3D displays and application infrastructure, IEEE Comput. 38 (8) (2005) 37–44. [14] T. Yendo, T. Fujii, M. Tanimoto, M.P. Tehrani, The seelinder: cylindrical 3D display viewable from 360 degrees, J. Vis. Commun. Image Represent. 21 (5–6) (2010) 586–594. [15] G. Lippmann, La photographie integrale, C. R. Acad. Sci. 146 (1908) 446–451. [16] Y. Kim, K. Hong, B. Lee, Recent researches based on integral imaging display method, 3D Res. 1 (1) (2010) 17–27. [17] S.H. Hong, J.S. Jang, B. Javidi, Three-dimensional volumetric object reconstruction using computational integral imaging, Opt. Express 12 (3) (2004) 483–491. [18] Z.L. Xiong, Q.H. Wang, S.L. Li, H. Deng, C.C. Ji, Partially-overlapped viewing zone based integral imaging system with super wide viewing angle, Opt. Express 22 (19) (2014) 22268–22277. [19] H.J. Choi, S.W. Min, S.Y. Jung, J.H. Park, B. Lee, Multiple-viewing zone integral imaging using a dynamic barrier array for three-dimensional displays, Opt. Express 11 (8) (2003) 927–932. [20] C.X. Yu, J.H. Yuan, F.C. Fan, C.C. Jiang, S. Choi, X.Z. Sang, C. Lin, D.X. Xu, The modulation function and realizing method of holographic functional screen, Opt. Express 18 (26) (2010) 27820–27826. [21] D. Miyazaki, N. Akasaka, K. Okoda, Y. Maeda, T. Mukai, Floating three-dimensional display viewable from 360-degrees, in: Proc. SPIE, 2012, p. 8288, 1H. [22] Y. Shunsuke, fVisiOn: 360-degree viewable glasses-free tabletop 3D display composed of conical screen and modular projector arrays, Opt. Express 24 (12) (2016) 13194–13203. [23] X.X. Xia, X. Liu, H.F. Li, Z.R. Zheng, H. Wang, Y.F. Peng, W.D. Shen, A 360-degree floating 3D display based on light field regeneration, Opt. Express 21 (9) (2013) 11237–11247. [24] X. Gao, X.Z. Sang, X.B. Yu, W.L. Zhang, B.B. Yan, Ch. X. Yu, 360◦ light field 3D display system based on a triplet lenses array and holographic functional screen, Chin. Opt. Lett. 15 (12) (2017) 121201. [25] Z.L. Xiong, Q.H. Wang, Y. Xing, H. Deng, D.H. Li, Active integral imaging system based on multiple structured light method, Opt. Express 23 (21) (2015) 27094– 27104.
4. Conclusion In this paper, a 360-degree viewable tabletop 3D display system based on II by using a POL is proposed. Firstly, the proposed system does not have the crosstalk caused by the limitation of the narrow viewing angle. Secondly, viewing zone of II can be modulated to the lateral viewing zone of the display in our system. In addition, 3D images with both vertical and horizontal parallaxes are shown to observers in the proposed system. The AEIA generation method based on partiallyoverlapping algorithm is adopted and flipping is eliminated in our system. The proposed system’s lateral view angle is 360◦ , and the vertical view angle is 40◦ , which is enough for a tabletop display. A more compact and simpler 360-degree viewable tabletop 3D display without rotating mechanism is promising after improving the structure of the POL in the future. Acknowledgments This work is supported by the National Key R&D Program of China under Grant No. 2017YFB1002900 and the National Natural Science Foundation of China under Grant Nos. 61535007 and 61775151. References [1] Y. Kakehi, M. Iida, T. Naemura, Y. Shirai, M. Matsushita, T. Ohguro, Lumisight table: an interactive view-dependent tabletop display, IEEE CGA 25 (1) (2005) 48– 53. [2] T. Yasuhiro, S. Uchida, Table screen 360-degree three-dimensional display using a small array of high-speed projectors, Opt. Express 20 (8) (2012) 8848–8861. [3] Y. Shunsuke, S. Yano, H. Ando, Prototyping of glassesfree tablestyle 3D display for tabletop tasks, SID Sym. Dig. Tech. 41 (1) (2010) 211–214. [4] S. Yoshida, M. Kawakita, H. Ando, Light-field generation by several screen types for glasses-free tabletop 3D display, in: Proceedings of 3DTV Conference, Vol. 49, 2011, pp. 1–4. [5] J.Y. Hong, J. Yeom, Y. Jeong, J.H. Kim, S.G. Park, K.H. Hong, B. Lee, Table-top display using integral floating display, in: International Conference on 3D Imaging (IC3D), IEEE, 2013, pp. 1–5.
60
Contents lists available at ScienceDirect
Optics Communications journal homepage: www.elsevier.com/locate/optcom
360-degree viewable tabletop 3D display system based on integral imaging by using perspective-oriented layer Ling Luo a , Qiong-Hua Wang b ,∗, Yan Xing a , Huan Deng a , Hui Ren a , Sai Li a a b
School of Electronics and Information Engineering, Sichuan University, Chengdu 610065, China School of Instrumentation and Optoelectronic Engineering, Beihang University, Beijing 100191, China
ARTICLE
INFO
Keywords: Tabletop display 3D display Integral imaging Perspective-oriented layer
ABSTRACT We propose a 360-degree viewable tabletop three-dimensional (3D) display system based on integral imaging by using a perspective-oriented layer (POL). The proposed system is composed of a two-dimensional (2D) display device, a POL, a lens array with large pitch and a light shaping diffuser screen. The 2D display device with high refresh rate is utilized to present the adaptive element image array (AEIA) sequentially. The AEIA is generated based on partially-overlapping algorithm to ensure that the 3D image corresponding to adjacent viewing zone is reconstructed without flipping. The 2D display device, the POL and the lens array rotate simultaneously to create a 360-degree continuous viewing zone by time division multiplexing. In our proposed tabletop 3D display system, the best viewing zone can be modulated to the lateral and the crosstalk caused by the narrow viewing angle can be eliminated. In addition, the proposed tabletop 3D display system is capable of reconstructing 3D images with both vertical and horizontal parallaxes. The experimental results agree well with the basic principle of the proposed method.
1. Introduction The tabletop display can present information more intuitively and naturally compared with the conventional display when people sit around a table [1]. An ideal tabletop three-dimensional (3D) display should reconstruct 360-degree viewable 3D images, which can be shared with multiple viewers simultaneously in different directions. The reconstructed 3D images should also have correct field of view and spatial occlusion effect for the improved visual effect. Recently the tabletop 3D display has attracted much attention. Many studies combining the tabletop display with the 3D display have been researched [2–6]. Common 3D display technologies include integral imaging (II) display, volumetric display and holographic display, and these displays can reproduce the real-world 3D scene almost naturally [7–9]. The holographic display can be applied into the tabletop 3D display [10]. A time division method or an observer tracking unit is employed to achieve a 360◦ viewable display [11,12]. However, too much data and complex devices are required to generate the correct 3D images. A glasses-free 3D images of 360-degree can be observed based on volumetric display, but the showcase-like mechanical components are usually employed, which invades the tabletop space [13,14]. The II 3D display has the advantages of providing full-parallax and full-color 3D images, so it is one of the most promising 3D displays for achieving the ideal tabletop 3D display [15–20]. Current tabletop display systems based on II 3D display can be classified into two typical
systems. One is the system based on time division multiplexing and space division multiplexing [21,22]. The system usually only offers horizontal parallax, so the observers at different heights watch the same 3D images [23]. The other is a conventional wall display based on II lay down to be as a table display to achieve a 360-degree viewable tabletop 3D display [24]. With this arrangement, the viewing area of the reconstructed 3D images is turned from the front to the top. However, the best viewing direction is almost directly above the display due to the limited viewing angle of the II. When people sit around the table, the reconstructed 3D images of the system have serious crosstalk, which degrades the viewing experience. In this paper, we proposed a 360-degree viewable tabletop 3D display system based on II by using a perspective-oriented layer (POL). It is composed of a two-dimensional (2D) display device, a POL, a lens array with large pitch and a light shaping diffuser screen (LSDS). Our proposed system can provide both vertical and horizontal parallaxes with continuous viewpoints. The viewing angle of the reconstructed 3D images in the vertical direction can reach 40◦ . The viewing zone is modulated to the lateral viewing zone of the tabletop 3D display for the observers around the table. Crosstalk caused by the limitation of the narrow viewing angle is eliminated in our system, so multi-viewers can observe the correct reconstructed 3D images at the same time. LSDS is adopted to improve the quality of the reconstructed 3D images. We present an adaptive element image array (AEIA) generation method
∗ Corresponding author. E-mail address: [email protected] (Q.-H. Wang).
https://doi.org/10.1016/j.optcom.2019.01.013 Received 31 October 2018; Received in revised form 14 December 2018; Accepted 5 January 2019 Available online 9 January 2019 0030-4018/© 2019 Elsevier B.V. All rights reserved.
L. Luo, Q.-H. Wang, Y. Xing et al.
Optics Communications 438 (2019) 54–60
Fig. 1. Structure of the proposed system.
Fig. 2. Schematic diagram of the display process . (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
based on partially-overlapping algorithm to eliminate the flipping of adjacent viewing zone. The proposed display verification system is developed.
scene can be observed by multiple observers in 360-degree horizontal directions. Any adjacent viewing zone has an overlapped angle, which is large enough to ensure that 3D image corresponding to adjacent viewing zone is reconstructed without flipping.
2. Proposed tabletop 3D display system 2.2. Time division multiplexing scheme of the proposed system 2.1. Configuration and principle of the proposed system To realize 360-degree viewable tabletop 3D display, a time division multiplexing scheme is adopted to reconstruct 3D images successively. In our system, the 360-degree viewing space is divided into several viewing zones in the horizontal direction. Each viewing zone is equivalent to that of the conventional II system and it is denoted as 𝑧𝑖 , where i is within the range from 1 to n. We define the original viewing zone of our tabletop system as 𝑧1 . The viewing angle of each viewing zone is identical and it is denoted as 𝜃0 . Any adjacent viewing zones 𝑧𝑖 and 𝑧𝑖+1 are overlapped, and the interval angle is denoted as Δ𝜃. At a given moment 𝑡𝑖 , 3D image with viewing angle of [−𝜃0 ∕2 + (𝑖 − 1) × Δ𝜃, 𝜃0 ∕2 + (𝑖 − 1) × Δ𝜃], which corresponds to viewing zone 𝑧𝑖 , can be reconstructed by rotating the display device, the POL and the lens array. When the screen turns around a circle, the Δ𝜃 and the number of the refreshed image are mutually restrictive by the following equation:
The proposed 360-degree viewable tabletop 3D display system is mainly comprised of a 2D display device, a POL a lens array with large pitch and a LSDS, as shown in Fig. 1. The 2D display device, which can be achieved by an ultra-high-definition (UHD) panel with high refreshing rate or a screen projected by high-speed projector, is located at the bottom of the system to display the AEIA sequentially. The 2D display device, the POL and the lens array are assembled into a whole device on a rotating mechanism. The system realizes a 360-degree continuous viewable II tabletop display based on the time division multiplexing technology. The LSDS, which is made of special microstructure and can expand the input light with the expanding angle, is placed at the center depth plane to eliminate gaps of the reconstructed 3D image and improve the visual performance for the reconstructed 3D images. In the display process, light rays emitted from the pixels of the AEIA are modulated by the POL and propagate along the POL, so the subviewing zone in the traditional II display becomes the main viewing zone in the proposed system. And the 3D image of the viewing zone marked in red will be reconstructed, as shown in Fig. 2. Then we adopt a time division multiplexing scheme, the 3D image corresponding to the viewing zone marked in blue, green, brown et al. will be quickly reconstructed in succession. Depending on the visual persistence effect, a complete 3D
Δ𝜃 × 𝑁 = 360◦ ,
(1)
where N is the number of the refreshed image when the system rotates 360◦ . In our paper, we set Δ𝜃 = 1◦ and 𝑁 = 360. At moment 𝑡1 , the 3D image, which contains [−𝜃0 ∕2, 𝜃0 ∕2] information of the target 3D scene and is marked with red line fanshaped column in Fig. 3, will be reconstructed by light rays emitted from its corresponding AEIA. At moment 𝑡2 , the 3D image containing 55
L. Luo, Q.-H. Wang, Y. Xing et al.
Optics Communications 438 (2019) 54–60
Fig. 3. Principle of the time division multiplexing scheme adopted in the proposed tabletop 3D display system.
Fig. 4. (a) Structure of the POL, (b) vertical section of the special unit, and (c) installation diagram of the POL and the lens.
[−𝜃0 ∕2+Δ𝜃, 𝜃0 ∕2+Δ𝜃] information of target 3D scene, marked with blue line and fan-shaped column, will be reconstructed. At the same way, 3D image containing [360−𝜃0 ∕2, 360+𝜃0 ∕2] information of target 3D scene, marked with brown line and fan-shaped column, will be reconstructed at moment 𝑡𝑛 . Each reconstructed 3D image of the corresponding viewing zone will be obtained in turn, and a whole 360-degree viewable 3D image will be presented to the viewers based on the visual persistence effect.
a hole tilted along a certain direction with a tilted angle of 𝛼 off the 2D display device. The combinatorial cavity, which is composed of an aperture and a cavity, is used to make the lens install more firmly, as shown in Fig. 4(c). The lateral viewing zone, which faces the side of the tabletop display, is the best viewing zone for the observers around the tabletop display. In order to make the lateral viewing zone provide for the observers, the light rays only propagate along the tiling-mode modulating hole by the POL, as shown in Fig. 5(a). Thus, the observers who locate within the red-marked viewing zone can watch the corresponding 3D images, while the observers located at the other regions cannot observe these 3D images, as shown in Fig. 5(b). The viewing angles 𝛼ℎ in the horizontal direction and 𝛼𝑣 in the vertical direction can be deducted as
2.3. POL in the proposed system The viewing zone of the reconstructed 3D images is turned from the front to the top in the system where a conventional wall display based on II lies down to be as a table display. However, the best viewing direction, which is almost directly above the display due to the limited viewing angle of the II, cannot match the requirement of the tabletop 3D display. When people sit around the table, the reconstructed 3D images of these systems have serious crosstalk, which degrades the viewing experience. To solve these problems, we introduce the POL into our proposed system. As shown in Fig. 4(a), the POL is made up of special unit array. Each special unit is composed of a modulating hole and a combinatorial cavity, as shown in Fig. 4(b). The modulating hole is
𝑙 sin 𝛼 𝛼ℎ = 2 arctan , ( 2ℎ ) ( ) ℎ ℎ 𝛼𝑣 = arctan − arctan , ℎ∕ tan 𝛼 + 𝑤∕2 (ℎ∕ tan 𝛼) − 𝑤∕2
(2) (3)
where h, w and l are the height, width and length of the modulating hole, respectively. The vertical viewing angle is [𝛼−𝛼v ∕2, 𝛼+𝛼v ∕2]. In addition, the POL can also block light rays emitted from the adjacent lens. Thus, the crosstalk which is caused by the rays emitted from the 56
L. Luo, Q.-H. Wang, Y. Xing et al.
Optics Communications 438 (2019) 54–60
Fig. 5. Diagram on the modulation of the POL: (a) the modulation of the POL in the vertical direction, (b) the modulation of the POL in the horizontal directions . (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
Fig. 6. Relationship between the pickup parameters and the parameters of the display system.
Fig. 7. Diagram of the parallax image rectification.
pixels of neighboring elemental images also can be removed in the proposed tabletop display system.
the spacing 𝑉𝑑 is determined by the following equation: (( ( ))/ ( )) (𝐿 + 𝐺) 2𝑑 + 𝑑0 𝑝 𝑉𝑑 = 𝑟𝑜𝑢𝑛𝑑 , 𝐺 𝑝0
2.4. Generation of AEIA in the proposed system
(4)
where G is the distance between the lens array and the 2D display device, L is the distance between the camera array plane and the lens array, d is the diameter of the lens, 𝑑0 is the spacing between adjacent lens, p is the pitch of the lens array, and 𝑝0 is the size of the pixel. The number of the camera in the camera array for each viewing zone is determined by the parameters of the tabletop 3D display system, and it can be expressed as M × M, where ( ) 𝑀 = 𝑟𝑜𝑢𝑛𝑑 𝑝∕𝑝0 . (5)
In the proposed tabletop 3D display system, the AEIA corresponding to each viewing zone is generated efficiently. The generation process of the AEIA is divided into the following three steps. Firstly, in the pickup step, we set all pickup parameters and obtain the original parallax images by using a virtual camera array. Secondly, in the rectifying step, the homography matrices for each parallax image are calculated, and then the obtained parallax images are projected to the same plane. Finally, in the synthesizing step, the rectified parallax images are synthesized to create the AEIA for the corresponding viewing zone.
In the camera array, the cameras are arranged in a toed-in arrangement to ensure that the overlapped shooting area is maximized. However, such an arrangement causes a problem that the projection plane of each camera is not exactly coplanar with each other, as shown
In the pickup step, as shown in Fig. 6, the spacing of the adjacent cameras in both horizontal and vertical directions are denoted as 𝑉𝑑 ×𝑉𝑑 , 57
L. Luo, Q.-H. Wang, Y. Xing et al.
Optics Communications 438 (2019) 54–60 Table 1 Configuration parameters of the proposed tabletop 3D display system. Parameters of the tabletop 3D display system
Values
Tilted angle (𝜃) Number of the lens (M × N ) Pitch of the lens array (p) Focal length of the lens (f ) Length of the modulating hole (l) Width of the modulating hole (w) Pitch of the POL
60◦ 21 × 21 12.700 mm 12.700 mm 13.100 mm 13.100 mm 13.700 mm
3. Experiments and discussion In the verification experiment, the 360-degree viewable tabletop 3D display system is developed to verify the basic principle. As shown in Fig. 8, the proposed tabletop 3D display system is composed of a rotation device, a 23.5-inch Dell 4K monitor, a POL, a lens array and a LSDS. We use a rotating mechanism (Daheng Optics GCD-011200, with a speed of 1 r.p.m) to rotate the display device, the POL and the lens array simultaneously. The whole height of the POL is 15.800 mm. The height of the aperture is 1.200 mm, and the curvature radius of the aperture is 6.468 mm. The cavity for mounting lens is 1.100 mm in height, and 12.900 mm in inner diameter. The modulating hole and the cavity are made of black rigid resin, and the aperture is made of transparent resin. The resolution of the monitor is 3840 × 2160 pixels and the pixel density is 185 pixels per inch. The remaining parameters of the experiment setup are given in Table 1. Plano-convex lens is used in our experiment, and the curvature radius of the convex surface is 6.468 mm. The pitch of the special unit of the POL is 13.700 mm, and each lens covers 100 × 100 pixels. Three sets of experiments are performed to evaluate the display effect of the proposed tabletop 3D display system. The first experiment is designed to evaluate the effect of removing the crosstalk in our proposed system. The 3D scene contains a teapot and a ball model. AEIA corresponding to one viewing zone is generated. As shown in Fig. 9(a), in order to show the image shooting position in Fig. 9(b) and (c) clearly, the viewing space is divided into 360 segments and the center of the single viewing zone is assumed to be 0◦ . Figure 9(b) and (c) show the reconstructed 3D images from
Fig. 8. Experimental setup of the 360-degree viewable tabletop 3D display system.
in Fig. 7. The projection plane of each camera and the display plane are also not aligned as well, which will cause serious trapezoid distortion. Hence, the rectifying step is needed. Homography matrices of each camera are calculated to project the corresponding parallax images to the display plane. In this way, the projection plane of each camera can be coplanar by implementing the homography transformation, where the display plane corresponds to the tabletop plane. 𝑖 (𝑥, 𝑦) are interweaved to Finally, the rectified parallax images 𝐼𝑚,𝑛 generate the AEIA 𝐸𝑖 (x, y) for a certain viewing zone 𝑧𝑖 . Adjacent AEIAs 𝐸𝑖 (x, y) and 𝐸𝑖+1 (x, y) have the overlapped 3D information. The 𝐸𝑖 (x, y) can be derived by: ∑∑ ( ) 𝑖 𝐸 (𝑥, 𝑦) = 𝑀 2 𝐼𝑚,𝑛 𝑀 ⋅ 𝑖′ + 𝑚, 𝑀 ⋅ 𝑗 ′ + 𝑛 𝑚,𝑛 𝑖′ ,𝑗 ′
( ) × 𝛿 𝑥 − 𝑀 ⋅ 𝑖′ − 𝑚, 𝑦 − 𝑀 ⋅ 𝑗 ′ − 𝑛 , 𝑖′
(6)
𝑗′
where, and represent the index of the lens, m and n represent the index of the rectified parallax images, and the resolution of the AEIA is x × y pixels [25]. Based on the above-mentioned steps, the AEIA for all viewing zone can be generated successively, and continuous perspectives of the 3D scene in 360-degree viewing zone can be observed without flipping.
Fig. 9. (a) Schematic diagram of the partition of the 3D display space, (b) pictures taken from different directions with the POL, and (c) pictures taken from different directions without the POL. 58
L. Luo, Q.-H. Wang, Y. Xing et al.
Optics Communications 438 (2019) 54–60
Fig. 10. Different perspectives of the reconstructed 3D image in a certain viewing zone.
Fig. 11. Pictures taken from different positions of the teapot 3D images.
different directions in the case of adding the POL and removing the POL. Obviously, the proposed tabletop 3D display system can effectively eliminate the crosstalk and improve the quality of the reconstructed 3D images. Furthermore, the reconstructed images taken at 0◦ in both display systems almost show indistinguishableness, which indicates that the reconstructed 3D images by our proposed system hardly has any additional optical aberrations introduced. The second experiment is designed to verify the horizontal and vertical parallaxes of our proposed system. The 3D scene also consists of a teapot model and a ball model. We generate an AEIA for a certain viewing zone. As shown in Fig. 10, in the corresponding viewing zone, the pictures of the reconstructed 3D images are taken from the top view, the left view, the bottom view, the right view and the middle view,
respectively. The results show that the proposed system has obvious parallaxes in both the horizontal and vertical directions. The viewing angle in both the horizontal and vertical directions can achieve 40◦ in a single viewing zone. The third experiment is designed to prove the 360-degree viewable display effect of the proposed tabletop 3D display system. The low speed rotating device is used to simulate the effect of the high speed rotating device. The 3D scene is a teapot model. AEIAs corresponding to the different viewing zone are generated. The interval angle of adjacent viewing zone is set as 1◦ . As shown in Fig. 11, we show the teapot 3D images in different positions. The interval angle of the pictures between adjacent positions is 24◦ . Obviously, the teapot 3D images show no flipping effect within all of the viewing positions. 59
L. Luo, Q.-H. Wang, Y. Xing et al.
Optics Communications 438 (2019) 54–60
Consequently, a tabletop 3D information with 360-degree parallax in the horizontal direction and 40-degree parallax in the vertical direction can be presented to the observers by moving around the tabletop continuously.
[6] D. Zhao, B.Q. Su, G.W. Chen, H.E. Liao, 360-degree viewable floating autostereoscopic display using integral photography and multiple semitransparent mirrors, Opt. Express 23 (8) (2015) 9812–9823. [7] Q.H. Wang, 3D Display Technologies and Devices, Science Press, Beijing, 2011. [8] J. Geng, Three-dimensional display technologies, Adv. Opt. Photon. 5 (4) (2013) 456–535. [9] J.H. Park, K. Hong, B. Lee, Recent progress in three-dimensional information processing based on integral imaging, Appl. Opt. 48 (34) (2009) H77–H94. [10] Y. Lim, K. Hong, H. Kim, H.E. Kim, E.Y. Chang, S. Lee, T. Kim, J. Nam, H.G. Choo, J. Kim, J. Hahn, 360-degree tabletop electronic holographic display, Opt. Express 24 (22) (2016) 24999–25009. [11] Y. Sando, B. Daisuke, Y. Toyohiko, Holographic 3D display observable for multiple simultaneous viewers from all horizontal directions by using a time division method, Opt. Lett. 39 (19) (2014) 5555–5557. [12] D.D. Teng, L.L. Liu, Z.X. Wang, B.X. Sun, B.W. Wang, All-around holographic threedimensional light field display, Opt. Commun. 285 (21–22) (2012) 4235–4240. [13] G.E. Favalore, Volumetric 3D displays and application infrastructure, IEEE Comput. 38 (8) (2005) 37–44. [14] T. Yendo, T. Fujii, M. Tanimoto, M.P. Tehrani, The seelinder: cylindrical 3D display viewable from 360 degrees, J. Vis. Commun. Image Represent. 21 (5–6) (2010) 586–594. [15] G. Lippmann, La photographie integrale, C. R. Acad. Sci. 146 (1908) 446–451. [16] Y. Kim, K. Hong, B. Lee, Recent researches based on integral imaging display method, 3D Res. 1 (1) (2010) 17–27. [17] S.H. Hong, J.S. Jang, B. Javidi, Three-dimensional volumetric object reconstruction using computational integral imaging, Opt. Express 12 (3) (2004) 483–491. [18] Z.L. Xiong, Q.H. Wang, S.L. Li, H. Deng, C.C. Ji, Partially-overlapped viewing zone based integral imaging system with super wide viewing angle, Opt. Express 22 (19) (2014) 22268–22277. [19] H.J. Choi, S.W. Min, S.Y. Jung, J.H. Park, B. Lee, Multiple-viewing zone integral imaging using a dynamic barrier array for three-dimensional displays, Opt. Express 11 (8) (2003) 927–932. [20] C.X. Yu, J.H. Yuan, F.C. Fan, C.C. Jiang, S. Choi, X.Z. Sang, C. Lin, D.X. Xu, The modulation function and realizing method of holographic functional screen, Opt. Express 18 (26) (2010) 27820–27826. [21] D. Miyazaki, N. Akasaka, K. Okoda, Y. Maeda, T. Mukai, Floating three-dimensional display viewable from 360-degrees, in: Proc. SPIE, 2012, p. 8288, 1H. [22] Y. Shunsuke, fVisiOn: 360-degree viewable glasses-free tabletop 3D display composed of conical screen and modular projector arrays, Opt. Express 24 (12) (2016) 13194–13203. [23] X.X. Xia, X. Liu, H.F. Li, Z.R. Zheng, H. Wang, Y.F. Peng, W.D. Shen, A 360-degree floating 3D display based on light field regeneration, Opt. Express 21 (9) (2013) 11237–11247. [24] X. Gao, X.Z. Sang, X.B. Yu, W.L. Zhang, B.B. Yan, Ch. X. Yu, 360◦ light field 3D display system based on a triplet lenses array and holographic functional screen, Chin. Opt. Lett. 15 (12) (2017) 121201. [25] Z.L. Xiong, Q.H. Wang, Y. Xing, H. Deng, D.H. Li, Active integral imaging system based on multiple structured light method, Opt. Express 23 (21) (2015) 27094– 27104.
4. Conclusion In this paper, a 360-degree viewable tabletop 3D display system based on II by using a POL is proposed. Firstly, the proposed system does not have the crosstalk caused by the limitation of the narrow viewing angle. Secondly, viewing zone of II can be modulated to the lateral viewing zone of the display in our system. In addition, 3D images with both vertical and horizontal parallaxes are shown to observers in the proposed system. The AEIA generation method based on partiallyoverlapping algorithm is adopted and flipping is eliminated in our system. The proposed system’s lateral view angle is 360◦ , and the vertical view angle is 40◦ , which is enough for a tabletop display. A more compact and simpler 360-degree viewable tabletop 3D display without rotating mechanism is promising after improving the structure of the POL in the future. Acknowledgments This work is supported by the National Key R&D Program of China under Grant No. 2017YFB1002900 and the National Natural Science Foundation of China under Grant Nos. 61535007 and 61775151. References [1] Y. Kakehi, M. Iida, T. Naemura, Y. Shirai, M. Matsushita, T. Ohguro, Lumisight table: an interactive view-dependent tabletop display, IEEE CGA 25 (1) (2005) 48– 53. [2] T. Yasuhiro, S. Uchida, Table screen 360-degree three-dimensional display using a small array of high-speed projectors, Opt. Express 20 (8) (2012) 8848–8861. [3] Y. Shunsuke, S. Yano, H. Ando, Prototyping of glassesfree tablestyle 3D display for tabletop tasks, SID Sym. Dig. Tech. 41 (1) (2010) 211–214. [4] S. Yoshida, M. Kawakita, H. Ando, Light-field generation by several screen types for glasses-free tabletop 3D display, in: Proceedings of 3DTV Conference, Vol. 49, 2011, pp. 1–4. [5] J.Y. Hong, J. Yeom, Y. Jeong, J.H. Kim, S.G. Park, K.H. Hong, B. Lee, Table-top display using integral floating display, in: International Conference on 3D Imaging (IC3D), IEEE, 2013, pp. 1–5.
60