- Email: [email protected]
Integral imaging based light field display with enhanced viewing resolution using holographic diffuser
Optics Communications 402 (2017) 437–441
Contents lists available at ScienceDirect
Optics Communications journal homepage: www.elsevier.com/locate/optcom
Integral imaging based light field display with enhanced viewing resolution using holographic diffuser Zhiqiang Yan, Xingpeng Yan *, Xiaoyu Jiang, Hui Gao, Jun Wen Department of Information Engineering, Academy of Armored Forces Engineering, Beijing 100072, China
a r t i c l e
i n f o
Keywords: Light field display Integral imaging Viewing resolution
a b s t r a c t An integral imaging based light field display method is proposed by use of holographic diffuser, and enhanced viewing resolution is gained over conventional integral imaging systems. The holographic diffuser is fabricated with controlled diffusion characteristics, which interpolates the discrete light field of the reconstructed points to approximate the original light field. The viewing resolution can thus be improved and independent of the limitation imposed by Nyquist sampling frequency. An integral imaging system with low Nyquist sampling frequency is constructed, and reconstructed scenes of high viewing resolution using holographic diffuser are demonstrated, verifying the feasibility of the method. © 2017 Elsevier B.V. All rights reserved.
1. Introduction Integral imaging (II), which records and reproduces various perspectives of three-dimensional (3D) scene using a lenslet array (LA) or aperture array, is a novel 3D display technique, and provides human eyes with full and continuous motion parallax and true color presentation under incoherent illumination, without the necessity of wearing special glasses [1]. II has attracted enormous research interests in recent years for its potential of realizing 3D television in future. Despite the superiorities against other 3D techniques, II is still far from practical application for the limited field of view (FOV), depth of field (DOF) and viewing resolution [2]. In terms of viewing resolution, it is influenced by many factors, such as the lenslet pitch and the pixel pitch of display panel, and many fundamental and novel works have been done to improve it. For example, moving array lenslet technique (MALT) is proposed to enhance the spatial density of light rays by mechanical movement of lenslet array and synchronously displayed elemental image array (EIA) [3], without reducing the size of lenslet. To avoid vibration and noise caused by mechanical movement, electrical movement of liquid crystal lens array [4] and mask array [5] are proposed. Another attempt of MALT is using rotated prims which shift light rays around the original positions of lenslets [6]. Moving pinhole array on liquid crystal is proposed to improve the density of point light source, which leads to the improvement of resolution of point light source based II [7]. The essence of above methods is to increase the spatial density of exit pupils within unit time by time multiplexing, and * Corresponding author.
E-mail address: [email protected] (X. Yan). http://dx.doi.org/10.1016/j.optcom.2017.06.061 Received 22 March 2017; Received in revised form 12 June 2017; Accepted 16 June 2017 0030-4018/© 2017 Elsevier B.V. All rights reserved.
the viewing resolution limit imposed by Nyquist sampling frequency is overcome. Meanwhile, projection characteristics (including the maximum projectable spatial frequency, diffraction and aberration, etc.) of individual exit pupil act importantly on the viewing resolution of II [8]. High-resolution video system [9] and multi-projector system [10] are utilized to increase the pixel density of elemental images, and lenslet of long focal length [11] is adopted to increase the gap between LA and display panel. These approaches can enhance the projectable spatial frequency of exit pupil. Theoretical analysis of II are also made [8,12,13], in which resolution limitations and optimum designs are discussed. Nevertheless, the low viewing resolution of II still remains a severe problem. While light field displays which use projector arrays [14] or high speed projector combined with rotated mirror [15] can achieve high-resolution viewing images, but the systems are bulky and expensive to implement, moreover, only horizontal parallax can be observed if no tracking devices are adopted. In this paper, we propose a light field display method based on integral imaging by placing a holographic diffuser at fixed while proper depth planes. The holographic diffuser is holographically made with specified diffusion characteristics. It works as a light shaping component and expands a light ray of any incident angle to a cone-shape bundle along the original incident direction, and the expanded ray bundles form a continuous light field. The viewing resolution can thus be enhanced compared with conventional II systems, and more importantly, the upper limit of viewing resolution restricted by Nyquist sampling frequency can be overcome. The principles of the method, the characteristics of
Z. Yan et al.
Optics Communications 402 (2017) 437–441
the recording process of II. The continuous light field of the original scene is sampled by the lenslet array, and the sampling interval, denoted as Δ𝜃 (Fig. 1), can be expressed as 𝑑 (3) 𝑙 where 𝑧 = 𝑙 is the depth of the sampled point. In the display process, the light field of the sampled point is reconstructed with an angular interval of Δ𝜃, which makes the point visible only along certain directions. Therefore, the viewing resolution is limited even if the scene is reconstructed with high point density. The lenslet pitch determines how precisely the light field of a point is sampled or reconstructed. Hence, a small lenslet pitch is preferable, because the light field of points can be more precisely reconstructed, and the points can be observed in a more continuous way (small angular interval), and the final viewing quality is thus improved. But the lenslet pitch cannot be decreased limitlessly because of diffraction, the optimal lenslet pitch is thought to be 1–2 mm [3]. For conventional II, the viewing resolution limitation 𝛽nyq always exits. However, if we adopt a light shaping component to expand each light ray to connect or overlap mutually, as illustrated in Fig. 1(b), thus forming a continuous light field, the viewing resolution limitation can be overcome, and the viewing resolution is 𝛽proj if other factors (diffraction and focusing error of lenslet etc.) are not considered. This would be fascinating, because 𝛽proj is proportional to 𝛼, and 𝛼 can be enhanced by adopting high resolution display panels, or by increasing the gap between LA and display panel. Δ𝜃 ≈
Fig. 1. Viewing resolution of II (a) with Nyquist sampling frequency limitation, and (b) independent of Nyquist sampling frequency limitation after using a light shaping component.
2.2. Principle of II based light field display
holographic diffuser, the validity and the deficiencies of the method are presented as follows.
The principle of using holographic diffuser to enhance viewing resolution of II is illustrated in Fig. 2. Consider a point in natural scene, the light field 𝐼(𝜙) is continuous and varies as directional angle within range [𝜙1 , 𝜙2 ], as depicted in Fig. 2(a). The light field is sampled (reconstructed) in the recording (display) process of II with an interval of Δ𝜃. Suppose the point is imaged by 𝑁 lenslets in one dimension, then FOV ≈ 𝑁Δ𝜃. For simplicity, the sampled light field can be modeled as an impulse sequence 𝐼1 (𝜙), as shown in Fig. 2(b). In conventional II, the impulse sequence is directly presented to human eyes, which is not suitable to the human visual system (HVS). Inspired by audio communication systems, in which the audio impulse sequence should be low-pass filtered to recover the original signal, we adopt holographic diffuser which will work on every impulse. The overall effect is the impulse sequence 𝐼1 (𝜙) is interpolated to approximate the original light field 𝐼(𝜙). The process can be expressed as
2. Theory 2.1. Viewing resolution of II In II based 3D display, light rays from different elemental lenslets intersect to form a point in space, and all the reconstructed points constitute the surface of object, as depicted in Fig. 1. The minimal distance 𝑟 between two points of the same depth (surrounded by dashed rectangle in Fig. 1(a)), is 𝑟 = 𝑝𝑙∕𝑔, where 𝑝 denotes the pixel pitch of display panel, 𝑔 denotes the gap between display panel and LA, 𝑧 = 𝑙 is the plane where the reconstructed points are located. Circles per rad (cpr) is used as the unit of spatial frequency, and the maximum spatial frequency on the observing plane, owning to the projection of lenslets, is [8] 𝛼(𝐿 − 𝑙) 𝐿−𝑙 = (1) 2𝑟 𝑙 where 𝑧 = 𝐿 is the observing plane, 𝛼 = 𝑔∕2𝑝 is the maximum projectable spatial frequency from the exit pupil of lenslet. The density of the reconstructed points is proportional to the maximum projectable frequency 𝛼 from the exit pupil of lenslet. However, the light rays that enters the eye pupil are from lenslets, not all the reconstructed points can contribute a directional light ray to the eye pupil, so the observer at a fixed viewpoint may see a much lower point density than the system really presents. Therefore, the maximum spatial frequency that the observer can perceive, i.e., the viewing resolution 𝛽 of II, is restricted and can be expressed as [8] [ ] 𝛼(𝐿 − 𝑙) 𝐿 𝛽 = min(𝛽proj , 𝛽nyq ) = min , . (2) 𝑙 2𝑑
(4)
𝛽proj =
𝐼2 (𝜙) = 𝐼1 (𝜙) ⊗ 𝐷(𝜙)
where 𝛽nyq = 𝐿∕2𝑑 cpr denotes the Nyquist sampling frequency, and 𝑑 denotes the lenslet pitch. Although it is easy to enhance the maximum projectable frequency 𝛼 of the exit pupil, and thus enhancing 𝛽proj , the viewing resolution cannot be higher than 𝛽nyq . We believe this limitation is due to the discrete nature of the light field (i.e. light intensity distribution varying as directions) of reconstructed points. This can be known by considering
where ⊗ is convolution operator, 𝐷(𝜙) is the diffusion characteristics of holographic diffuser (Fig. 2(c)), 𝐼2 (𝜙) is the reconstructed light field using holographic diffuser (Fig. 2(d)). Once the light field of a point is well recovered, the light field of the 3D scene is also well recovered, and the viewing quality can be improved. The holographic diffuser is transmissive and wavelength independent, with random surface relief structure, and provides controlled directional diffusion characteristics of light rays [16]. The 1D diffusion characteristics of holographic diffuser is illustrated in Fig. 2(c). Expanding angle 𝜑, measured by full width at half maximum (FWHM), is used to characterize holographic diffusers. By using holographic diffuser, the light ray is expanded and the light energy is spread, which may attenuate the intensity of the integrated 3D image. This can be solved by designing holographic diffuser of proper diffusion specifications, such as a small expanding angle and a cliffy profile of 𝐷(𝜙), to restrain the light energy within a small angle. Two holographic diffusers with expanding angle of 3.5◦ and 5◦ are adopted in the experiments. The diffusion characteristics of them is tested as depicted in Fig. 3. A collimated laser beam is incident on the holographic diffuser, and a screen is placed 1m distant from the 438
Z. Yan et al.
Optics Communications 402 (2017) 437–441
Fig. 2. The principle of using holographic diffuser to enhance viewing resolution.
Fig. 3. 1D and 2D light intensity distribution of the expanded laser beam, when holographic diffusers with expanding angle of 3.5◦ and 5◦ are used.
holographic diffuser and perpendicular to the direction of the light ray. The 2D light intensity distribution of the expanded laser beam are formed on the screen. We use a digital camera to capture the patterns, as the circular spot shown in Fig. 3. And the 1D distribution of light intensity can be deduced from the 2D light intensity distribution by choosing an axial distribution. The normalized 1D light intensity distribution can be regarded as 𝐷(𝜙). 3. Experiments Experiments are conducted to verify the proposed method. The experimental setup depicted in Fig. 4 is composed of a liquid crystal display (LCD), a LA, an aperture array and a holographic diffuser. The resolution of LCD is 4k (3840 × 2160) and the pixel pitch 𝑝 is 72 μm. The LA is hexagonally arranged with lenslet pitch 𝑑 of 11 mm. Lenslet of small 𝐹 number is chosen to gain a large viewing angle, and the lenslet is circular with diameter of 10 mm and focal length of 10 mm. The gap 𝑔 is 10.7 mm. The aperture array with 6 mm-diameter circular elemental aperture is used to thin the ray bundles composed of directional light rays that emit from pixels on LCD and pass the lenslet, so the ray bundles more approach impulses (Fig. 2(b)), and thus enhancing the depth of field (DOF) of the system. The plane on which the holographic diffuser is positioned must be determined. When a holographic diffuser is placed on a certain depth plane, the light field of points on that plane can be estimated according to Fig. 2(d), which is closely related to the diffusion characteristics of
Fig. 4. Experimental setup for II based light field display with enhanced viewing resolution.
holographic diffuser, the sampling interval of light field, etc. The light field should be visually continuous, because it is perceived by HVS. Hence, the plane is determined experimentally. As shown in Fig. 5, a 2D image is integrated on different depth planes, and the image quality is assessed by HVS. The image to be displayed is shown in Fig. 5(a), with a resolution of 357 × 531 and physical size of 2.6 cm × 3.8 cm. The EIA is computer-generated using ray-tracing algorithm. Fig. 5(b) is the image integrated on 𝑧 = 9 cm without holographic diffuser, and the 439
Z. Yan et al.
Optics Communications 402 (2017) 437–441
Fig. 5. (a) The original image, (b) image integrated on 𝑧 = 9 cm without holographic diffuser, (c) image integrated and with holographic diffuser placed both on 𝑧 = 9 cm, while in (d) 𝑧 = 16 cm.
Fig. 6. (a) Three images are integrated on 𝑧 = 14 cm, 𝑧 = 16 cm and 𝑧 = 18 cm respectively, with holographic diffusers of (a) 5◦ and (b) 3.5◦ fixed on 𝑧 = 16 cm. Supplementary data 1 is a video of perspective images.
viewing resolution is very low. The camera used to capture the images is focused on the image plane. The aperture size of the camera lens is set to be 3 mm, which approaches the size of eye pupil. Fig. 5(c) is image integrated on 𝑧 = 9 cm with holographic diffuser of 3.5◦ placed on 𝑧 = 9 cm, and much higher viewing resolution is observed compared with Fig. 5(b), but the structure of LA can be observed, which deteriorates the viewing quality. Fig. 5(d) is the image integrated on 𝑧 = 16 cm with holographic diffuser placed on 𝑧 = 16 cm, which is the nearest plane that the structure of LA cannot be seen. Hence, the viewing resolution can be enhanced by making the light field continuous, and the holographic diffuser should be placed far enough from LA to eliminate the structure of LA. The nearest plane for holographic diffuser with expanding angle of 5◦ to be placed on is approximately 𝑧 = 6 cm. The above discussion focuses on the reconstructed points that are on the plane of holographic diffuser, and the spot size of them is not influenced by holographic diffuser. While for points that are not on the plane of holographic diffuser, the spot size of them increases, because the light rays that intersect to form them are expanded. Hence, the DOF of II is decreased by adopting holographic diffuser. The degradation of DOF can be eased by using holographic diffuser with smaller expanding angle and more concentrated diffusion characteristics. As shown in Fig. 6, three images are integrated on 𝑧 = 14 cm, 𝑧 = 16 cm and 𝑧 = 18 cm respectively. The holographic diffuser is fixed on 𝑧 = 16 cm. Fig. 6(a) and (b) shows the captured images when using holographic diffusers with expanding angle of 5◦ and 3.5◦ (Fig. 3) respectively. Images on 𝑧 = 14 cm and 𝑧 = 18 cm are clearer in Fig. 6(b) than those in Fig. 6(a), which means the DOF is larger when using holographic diffusers with smaller expanding angle and more concentrated diffusion characteristics. Supplementary data 1 shows various perspectives of the reconstructed scene, indicating the 3D property of the reconstructed scene. Though holographic diffuser with small expanding angle and concentrated diffusion characteristics is better to maintain a large DOF, but it takes a longer distance to eliminate the structure of LA, which in turn makes 𝛽proj decreased. Therefore, there is a trade-off between viewing resolution and DOF. This can be solved by adopting LA of smaller
lenslet pitch, then holographic diffuser of small expanding angle and concentrated diffusion characteristics can be placed on a nearer plane from LA to eliminate the structure of LA, and high viewing resolution can be achieved. In practice, the specifications of holographic diffuser should be in accordance with the lenslet pitch of LA and the depth range where the 3D scene is reconstructed, in order to achieve balanced viewing resolution and DOF. To demonstrate the overall display quality of II based light field display system using holographic diffuser, a magic cube of side length of about 9 cm is presented. The farthest point (vertex front) on the cube is approximately located on 𝑧 = 20 cm and the nearest point (vertex bottom right) is approximately located on 𝑧 = 11 cm. The EIA is computer-generated from a virtual model, and each elemental image contains 147 pixels horizontally and 127 pixels vertically. The digital camera is located on 𝑧 = 120 cm and focused on 𝑧 = 16 cm, and the iris of the camera is about 3 mm. The holographic diffuser with expanding angle of 3.5◦ is chosen and fixed on 𝑧 = 16 cm, which means part of the scene is before it and the other is behind (or on) it. Fig. 7(a) shows three perspectives of the magic cube without holographic diffuser. The crude shape (low spatial frequency) of the magic cube can be descried, but details (high spatial frequency) such as the borders of the small colorful squares is indistinguishable. Fig. 7(b) are three perspectives captured at the same positions as those in Fig. 7(a). The observed spatial frequency is much higher than that without holographic diffuser. The Nyquist sampling frequency is 𝛽nyq = 55 cpr (𝐿 = 120 cm), and the maximum spatial frequency owing to the projection of LA is 𝛽proj = 483 cpr (𝑙 = 16 cm). The perceived spatial frequency is smaller than 𝛽proj when observing the magic cube reconstructed in our system, but larger than 𝛽nyq , which means the Nyquist sampling frequency restriction on viewing resolution is overcome in the light field display system. The actual viewing resolution of the system is smaller than 𝛽proj with diffraction, focusing error of LA and other factors considered. Supplementary data 2 is a video captured horizontally along a straight track. The magic cube can be seen without flipping within 1.1 m, therefore, the horizontal FOV is about 49◦ . 440
Z. Yan et al.
Optics Communications 402 (2017) 437–441
Appendix A. Supplementary data Supplementary material related to this article can be found online at http://dx.doi.org/10.1016/j.optcom.2017.06.061. References [1] X. Xiao, B. Javidi, M. Martinezcorral, A. Stern, Advances in three-dimensional integral imaging: sensing, display, and applications [invited], Appl. Opt. 52 (4) (2013) 546–560. [2] J.H. Park, K. Hong, B. Lee, Recent progress in three-dimensional information processing based on integral imaging., Appl. Opt. 48 (34) (2009) 77–94. [3] J.S. Jang, B. Javidi, Improved viewing resolution of three-dimensional integral imaging by use of nonstationary micro-optics, Opt. Lett. 27 (5) (2002) 324–326. [4] T.H. Jen, X. Shen, G. Yao, Y.P. Huang, H.P. Shieh, B. Javidi, Dynamic integral imaging display with electrically moving array lenslet technique using liquid crystal lens, Opt. Express 23 (14) (2015) 18415–18421. [5] Y. Oh, D. Shin, B.G. Lee, S.I. Jeong, H.J. Choi, Resolution-enhanced integral imaging in focal mode with a time-multiplexed electrical mask array, Opt. Express 22 (15) (2014) 17620–17629. [6] H. Liao, T. Dohi, M. Iwahara, Improved viewing resolution of integral videography by use of rotated prism sheets, Opt. Express 15 (8) (2007) 4814–4822. [7] Y. Kim, J. Kim, J.-M. Kang, J.-H. Jung, H. Choi, B. Lee, Point light source integral imaging with improved resolution and viewing angle by the use of electrically movable pinhole array, Opt. Express 15 (26) (2007) 18253–18267. [8] H. Hoshino, F. Okano, H. Isono, I. Yuyama, Analysis of resolution limitation of integral photography, J. Opt. Soc. Am. A 15 (8) (1998) 2059–2065. [9] F. Okano, J. Arai, K. Mitani, M. Okui, Real-time integral imaging based on extremely high resolution video system, Proc. IEEE 94 (3) (2006) 490–501. [10] H. Liao, M. Iwahara, N. Hata, T. Dohi, High-quality integral videography using a multiprojector, Opt. Express 12 (6) (2004) 1067–1076. [11] Z. Wang, A. Wang, S. Wang, X. Ma, H. Ming, Resolution-enhanced integral imaging using two micro-lens arrays with different focal lengths for capturing and display, Opt. Express 23 (22) (2015) 28970–28977. [12] T. Okoshi, Optimum design and depth resolution of lens-sheet and projection-type three-dimensional displays, Appl. Opt. 10 (10) (1971) 2284–2291. [13] C. Burckhardt, Optimum parameters and resolution limitation of integral photography, J. Opt. Soc. Amer. 58 (1) (1968) 71–74. [14] J.-H. Lee, J. Park, D. Nam, S.Y. Choi, D.-S. Park, C.Y. Kim, Optimal projector configuration design for 300-mpixel multi-projection 3d display, Opt. Express 21 (22) (2013) 26820–26835. [15] A. Jones, I. McDowall, H. Yamada, M. Bolas, P. Debevec, Rendering for an interactive 360 light field display, in: ACM Transactions on Graphics (TOG), Vol. 26, ACM, 2007, p. 40 3. [16] S. Santoro, M. Crenshaw, I. Ashdown, Light control devices and methods implemented with kinoform diffusers having controllable diffusion characteristics Google Patents, US Patent 7,660,039, (Feb. 9 2010), 2010.
Fig. 7. Perspective images integrated (a) without holographic diffuser, and (b) with holographic diffuser fixed on 𝑧 = 16 cm. Supplementary data 2 is a video of perspective images.
4. Conclusion In conclusion, a method to realize II based light field display with improved viewing resolution is proposed. The discrete light field of the reconstructed points is interpolated to be continuous by use of holographic diffuser, and the viewing resolution is proportional to the maximum projectable spatial frequency of exit pupil, meanwhile, the viewing resolution limit imposed by Nyquist sampling frequency is overcome. Therefore, the viewing resolution can be improved by enlarging the gap between LA and display panel, or by using display panels with smaller pixel pitch. As display panel of large size and high dpi (dots per inch) is a trend, our method could be an effective way of realizing high-resolution and large-scale II based light field display. The defects of using holographic diffuser, such as degradation of the DOF and light intensity, should never be neglected, and we think they can be alleviated by optimal design of the diffusion characteristics of holographic diffuser and specifications of II systems. The diffusion characteristics of holographic diffuser have great impact on the reconstruction quality, and how it affects the reconstruction quality and its optimal design need further investigation. The proposed light field display method based on II is of good 3D effect and economical to mass production, and we expect it to be a solution for 3D display applications in future. Acknowledgments This work is partly supported by National Excellent Doctoral Dissertation of PR China (FANEDD) (201432), the Beijing Natural Science Foundation (4152049), and the Beijing NOVA program (Z1511000003150119).
441
Contents lists available at ScienceDirect
Optics Communications journal homepage: www.elsevier.com/locate/optcom
Integral imaging based light field display with enhanced viewing resolution using holographic diffuser Zhiqiang Yan, Xingpeng Yan *, Xiaoyu Jiang, Hui Gao, Jun Wen Department of Information Engineering, Academy of Armored Forces Engineering, Beijing 100072, China
a r t i c l e
i n f o
Keywords: Light field display Integral imaging Viewing resolution
a b s t r a c t An integral imaging based light field display method is proposed by use of holographic diffuser, and enhanced viewing resolution is gained over conventional integral imaging systems. The holographic diffuser is fabricated with controlled diffusion characteristics, which interpolates the discrete light field of the reconstructed points to approximate the original light field. The viewing resolution can thus be improved and independent of the limitation imposed by Nyquist sampling frequency. An integral imaging system with low Nyquist sampling frequency is constructed, and reconstructed scenes of high viewing resolution using holographic diffuser are demonstrated, verifying the feasibility of the method. © 2017 Elsevier B.V. All rights reserved.
1. Introduction Integral imaging (II), which records and reproduces various perspectives of three-dimensional (3D) scene using a lenslet array (LA) or aperture array, is a novel 3D display technique, and provides human eyes with full and continuous motion parallax and true color presentation under incoherent illumination, without the necessity of wearing special glasses [1]. II has attracted enormous research interests in recent years for its potential of realizing 3D television in future. Despite the superiorities against other 3D techniques, II is still far from practical application for the limited field of view (FOV), depth of field (DOF) and viewing resolution [2]. In terms of viewing resolution, it is influenced by many factors, such as the lenslet pitch and the pixel pitch of display panel, and many fundamental and novel works have been done to improve it. For example, moving array lenslet technique (MALT) is proposed to enhance the spatial density of light rays by mechanical movement of lenslet array and synchronously displayed elemental image array (EIA) [3], without reducing the size of lenslet. To avoid vibration and noise caused by mechanical movement, electrical movement of liquid crystal lens array [4] and mask array [5] are proposed. Another attempt of MALT is using rotated prims which shift light rays around the original positions of lenslets [6]. Moving pinhole array on liquid crystal is proposed to improve the density of point light source, which leads to the improvement of resolution of point light source based II [7]. The essence of above methods is to increase the spatial density of exit pupils within unit time by time multiplexing, and * Corresponding author.
E-mail address: [email protected] (X. Yan). http://dx.doi.org/10.1016/j.optcom.2017.06.061 Received 22 March 2017; Received in revised form 12 June 2017; Accepted 16 June 2017 0030-4018/© 2017 Elsevier B.V. All rights reserved.
the viewing resolution limit imposed by Nyquist sampling frequency is overcome. Meanwhile, projection characteristics (including the maximum projectable spatial frequency, diffraction and aberration, etc.) of individual exit pupil act importantly on the viewing resolution of II [8]. High-resolution video system [9] and multi-projector system [10] are utilized to increase the pixel density of elemental images, and lenslet of long focal length [11] is adopted to increase the gap between LA and display panel. These approaches can enhance the projectable spatial frequency of exit pupil. Theoretical analysis of II are also made [8,12,13], in which resolution limitations and optimum designs are discussed. Nevertheless, the low viewing resolution of II still remains a severe problem. While light field displays which use projector arrays [14] or high speed projector combined with rotated mirror [15] can achieve high-resolution viewing images, but the systems are bulky and expensive to implement, moreover, only horizontal parallax can be observed if no tracking devices are adopted. In this paper, we propose a light field display method based on integral imaging by placing a holographic diffuser at fixed while proper depth planes. The holographic diffuser is holographically made with specified diffusion characteristics. It works as a light shaping component and expands a light ray of any incident angle to a cone-shape bundle along the original incident direction, and the expanded ray bundles form a continuous light field. The viewing resolution can thus be enhanced compared with conventional II systems, and more importantly, the upper limit of viewing resolution restricted by Nyquist sampling frequency can be overcome. The principles of the method, the characteristics of
Z. Yan et al.
Optics Communications 402 (2017) 437–441
the recording process of II. The continuous light field of the original scene is sampled by the lenslet array, and the sampling interval, denoted as Δ𝜃 (Fig. 1), can be expressed as 𝑑 (3) 𝑙 where 𝑧 = 𝑙 is the depth of the sampled point. In the display process, the light field of the sampled point is reconstructed with an angular interval of Δ𝜃, which makes the point visible only along certain directions. Therefore, the viewing resolution is limited even if the scene is reconstructed with high point density. The lenslet pitch determines how precisely the light field of a point is sampled or reconstructed. Hence, a small lenslet pitch is preferable, because the light field of points can be more precisely reconstructed, and the points can be observed in a more continuous way (small angular interval), and the final viewing quality is thus improved. But the lenslet pitch cannot be decreased limitlessly because of diffraction, the optimal lenslet pitch is thought to be 1–2 mm [3]. For conventional II, the viewing resolution limitation 𝛽nyq always exits. However, if we adopt a light shaping component to expand each light ray to connect or overlap mutually, as illustrated in Fig. 1(b), thus forming a continuous light field, the viewing resolution limitation can be overcome, and the viewing resolution is 𝛽proj if other factors (diffraction and focusing error of lenslet etc.) are not considered. This would be fascinating, because 𝛽proj is proportional to 𝛼, and 𝛼 can be enhanced by adopting high resolution display panels, or by increasing the gap between LA and display panel. Δ𝜃 ≈
Fig. 1. Viewing resolution of II (a) with Nyquist sampling frequency limitation, and (b) independent of Nyquist sampling frequency limitation after using a light shaping component.
2.2. Principle of II based light field display
holographic diffuser, the validity and the deficiencies of the method are presented as follows.
The principle of using holographic diffuser to enhance viewing resolution of II is illustrated in Fig. 2. Consider a point in natural scene, the light field 𝐼(𝜙) is continuous and varies as directional angle within range [𝜙1 , 𝜙2 ], as depicted in Fig. 2(a). The light field is sampled (reconstructed) in the recording (display) process of II with an interval of Δ𝜃. Suppose the point is imaged by 𝑁 lenslets in one dimension, then FOV ≈ 𝑁Δ𝜃. For simplicity, the sampled light field can be modeled as an impulse sequence 𝐼1 (𝜙), as shown in Fig. 2(b). In conventional II, the impulse sequence is directly presented to human eyes, which is not suitable to the human visual system (HVS). Inspired by audio communication systems, in which the audio impulse sequence should be low-pass filtered to recover the original signal, we adopt holographic diffuser which will work on every impulse. The overall effect is the impulse sequence 𝐼1 (𝜙) is interpolated to approximate the original light field 𝐼(𝜙). The process can be expressed as
2. Theory 2.1. Viewing resolution of II In II based 3D display, light rays from different elemental lenslets intersect to form a point in space, and all the reconstructed points constitute the surface of object, as depicted in Fig. 1. The minimal distance 𝑟 between two points of the same depth (surrounded by dashed rectangle in Fig. 1(a)), is 𝑟 = 𝑝𝑙∕𝑔, where 𝑝 denotes the pixel pitch of display panel, 𝑔 denotes the gap between display panel and LA, 𝑧 = 𝑙 is the plane where the reconstructed points are located. Circles per rad (cpr) is used as the unit of spatial frequency, and the maximum spatial frequency on the observing plane, owning to the projection of lenslets, is [8] 𝛼(𝐿 − 𝑙) 𝐿−𝑙 = (1) 2𝑟 𝑙 where 𝑧 = 𝐿 is the observing plane, 𝛼 = 𝑔∕2𝑝 is the maximum projectable spatial frequency from the exit pupil of lenslet. The density of the reconstructed points is proportional to the maximum projectable frequency 𝛼 from the exit pupil of lenslet. However, the light rays that enters the eye pupil are from lenslets, not all the reconstructed points can contribute a directional light ray to the eye pupil, so the observer at a fixed viewpoint may see a much lower point density than the system really presents. Therefore, the maximum spatial frequency that the observer can perceive, i.e., the viewing resolution 𝛽 of II, is restricted and can be expressed as [8] [ ] 𝛼(𝐿 − 𝑙) 𝐿 𝛽 = min(𝛽proj , 𝛽nyq ) = min , . (2) 𝑙 2𝑑
(4)
𝛽proj =
𝐼2 (𝜙) = 𝐼1 (𝜙) ⊗ 𝐷(𝜙)
where 𝛽nyq = 𝐿∕2𝑑 cpr denotes the Nyquist sampling frequency, and 𝑑 denotes the lenslet pitch. Although it is easy to enhance the maximum projectable frequency 𝛼 of the exit pupil, and thus enhancing 𝛽proj , the viewing resolution cannot be higher than 𝛽nyq . We believe this limitation is due to the discrete nature of the light field (i.e. light intensity distribution varying as directions) of reconstructed points. This can be known by considering
where ⊗ is convolution operator, 𝐷(𝜙) is the diffusion characteristics of holographic diffuser (Fig. 2(c)), 𝐼2 (𝜙) is the reconstructed light field using holographic diffuser (Fig. 2(d)). Once the light field of a point is well recovered, the light field of the 3D scene is also well recovered, and the viewing quality can be improved. The holographic diffuser is transmissive and wavelength independent, with random surface relief structure, and provides controlled directional diffusion characteristics of light rays [16]. The 1D diffusion characteristics of holographic diffuser is illustrated in Fig. 2(c). Expanding angle 𝜑, measured by full width at half maximum (FWHM), is used to characterize holographic diffusers. By using holographic diffuser, the light ray is expanded and the light energy is spread, which may attenuate the intensity of the integrated 3D image. This can be solved by designing holographic diffuser of proper diffusion specifications, such as a small expanding angle and a cliffy profile of 𝐷(𝜙), to restrain the light energy within a small angle. Two holographic diffusers with expanding angle of 3.5◦ and 5◦ are adopted in the experiments. The diffusion characteristics of them is tested as depicted in Fig. 3. A collimated laser beam is incident on the holographic diffuser, and a screen is placed 1m distant from the 438
Z. Yan et al.
Optics Communications 402 (2017) 437–441
Fig. 2. The principle of using holographic diffuser to enhance viewing resolution.
Fig. 3. 1D and 2D light intensity distribution of the expanded laser beam, when holographic diffusers with expanding angle of 3.5◦ and 5◦ are used.
holographic diffuser and perpendicular to the direction of the light ray. The 2D light intensity distribution of the expanded laser beam are formed on the screen. We use a digital camera to capture the patterns, as the circular spot shown in Fig. 3. And the 1D distribution of light intensity can be deduced from the 2D light intensity distribution by choosing an axial distribution. The normalized 1D light intensity distribution can be regarded as 𝐷(𝜙). 3. Experiments Experiments are conducted to verify the proposed method. The experimental setup depicted in Fig. 4 is composed of a liquid crystal display (LCD), a LA, an aperture array and a holographic diffuser. The resolution of LCD is 4k (3840 × 2160) and the pixel pitch 𝑝 is 72 μm. The LA is hexagonally arranged with lenslet pitch 𝑑 of 11 mm. Lenslet of small 𝐹 number is chosen to gain a large viewing angle, and the lenslet is circular with diameter of 10 mm and focal length of 10 mm. The gap 𝑔 is 10.7 mm. The aperture array with 6 mm-diameter circular elemental aperture is used to thin the ray bundles composed of directional light rays that emit from pixels on LCD and pass the lenslet, so the ray bundles more approach impulses (Fig. 2(b)), and thus enhancing the depth of field (DOF) of the system. The plane on which the holographic diffuser is positioned must be determined. When a holographic diffuser is placed on a certain depth plane, the light field of points on that plane can be estimated according to Fig. 2(d), which is closely related to the diffusion characteristics of
Fig. 4. Experimental setup for II based light field display with enhanced viewing resolution.
holographic diffuser, the sampling interval of light field, etc. The light field should be visually continuous, because it is perceived by HVS. Hence, the plane is determined experimentally. As shown in Fig. 5, a 2D image is integrated on different depth planes, and the image quality is assessed by HVS. The image to be displayed is shown in Fig. 5(a), with a resolution of 357 × 531 and physical size of 2.6 cm × 3.8 cm. The EIA is computer-generated using ray-tracing algorithm. Fig. 5(b) is the image integrated on 𝑧 = 9 cm without holographic diffuser, and the 439
Z. Yan et al.
Optics Communications 402 (2017) 437–441
Fig. 5. (a) The original image, (b) image integrated on 𝑧 = 9 cm without holographic diffuser, (c) image integrated and with holographic diffuser placed both on 𝑧 = 9 cm, while in (d) 𝑧 = 16 cm.
Fig. 6. (a) Three images are integrated on 𝑧 = 14 cm, 𝑧 = 16 cm and 𝑧 = 18 cm respectively, with holographic diffusers of (a) 5◦ and (b) 3.5◦ fixed on 𝑧 = 16 cm. Supplementary data 1 is a video of perspective images.
viewing resolution is very low. The camera used to capture the images is focused on the image plane. The aperture size of the camera lens is set to be 3 mm, which approaches the size of eye pupil. Fig. 5(c) is image integrated on 𝑧 = 9 cm with holographic diffuser of 3.5◦ placed on 𝑧 = 9 cm, and much higher viewing resolution is observed compared with Fig. 5(b), but the structure of LA can be observed, which deteriorates the viewing quality. Fig. 5(d) is the image integrated on 𝑧 = 16 cm with holographic diffuser placed on 𝑧 = 16 cm, which is the nearest plane that the structure of LA cannot be seen. Hence, the viewing resolution can be enhanced by making the light field continuous, and the holographic diffuser should be placed far enough from LA to eliminate the structure of LA. The nearest plane for holographic diffuser with expanding angle of 5◦ to be placed on is approximately 𝑧 = 6 cm. The above discussion focuses on the reconstructed points that are on the plane of holographic diffuser, and the spot size of them is not influenced by holographic diffuser. While for points that are not on the plane of holographic diffuser, the spot size of them increases, because the light rays that intersect to form them are expanded. Hence, the DOF of II is decreased by adopting holographic diffuser. The degradation of DOF can be eased by using holographic diffuser with smaller expanding angle and more concentrated diffusion characteristics. As shown in Fig. 6, three images are integrated on 𝑧 = 14 cm, 𝑧 = 16 cm and 𝑧 = 18 cm respectively. The holographic diffuser is fixed on 𝑧 = 16 cm. Fig. 6(a) and (b) shows the captured images when using holographic diffusers with expanding angle of 5◦ and 3.5◦ (Fig. 3) respectively. Images on 𝑧 = 14 cm and 𝑧 = 18 cm are clearer in Fig. 6(b) than those in Fig. 6(a), which means the DOF is larger when using holographic diffusers with smaller expanding angle and more concentrated diffusion characteristics. Supplementary data 1 shows various perspectives of the reconstructed scene, indicating the 3D property of the reconstructed scene. Though holographic diffuser with small expanding angle and concentrated diffusion characteristics is better to maintain a large DOF, but it takes a longer distance to eliminate the structure of LA, which in turn makes 𝛽proj decreased. Therefore, there is a trade-off between viewing resolution and DOF. This can be solved by adopting LA of smaller
lenslet pitch, then holographic diffuser of small expanding angle and concentrated diffusion characteristics can be placed on a nearer plane from LA to eliminate the structure of LA, and high viewing resolution can be achieved. In practice, the specifications of holographic diffuser should be in accordance with the lenslet pitch of LA and the depth range where the 3D scene is reconstructed, in order to achieve balanced viewing resolution and DOF. To demonstrate the overall display quality of II based light field display system using holographic diffuser, a magic cube of side length of about 9 cm is presented. The farthest point (vertex front) on the cube is approximately located on 𝑧 = 20 cm and the nearest point (vertex bottom right) is approximately located on 𝑧 = 11 cm. The EIA is computer-generated from a virtual model, and each elemental image contains 147 pixels horizontally and 127 pixels vertically. The digital camera is located on 𝑧 = 120 cm and focused on 𝑧 = 16 cm, and the iris of the camera is about 3 mm. The holographic diffuser with expanding angle of 3.5◦ is chosen and fixed on 𝑧 = 16 cm, which means part of the scene is before it and the other is behind (or on) it. Fig. 7(a) shows three perspectives of the magic cube without holographic diffuser. The crude shape (low spatial frequency) of the magic cube can be descried, but details (high spatial frequency) such as the borders of the small colorful squares is indistinguishable. Fig. 7(b) are three perspectives captured at the same positions as those in Fig. 7(a). The observed spatial frequency is much higher than that without holographic diffuser. The Nyquist sampling frequency is 𝛽nyq = 55 cpr (𝐿 = 120 cm), and the maximum spatial frequency owing to the projection of LA is 𝛽proj = 483 cpr (𝑙 = 16 cm). The perceived spatial frequency is smaller than 𝛽proj when observing the magic cube reconstructed in our system, but larger than 𝛽nyq , which means the Nyquist sampling frequency restriction on viewing resolution is overcome in the light field display system. The actual viewing resolution of the system is smaller than 𝛽proj with diffraction, focusing error of LA and other factors considered. Supplementary data 2 is a video captured horizontally along a straight track. The magic cube can be seen without flipping within 1.1 m, therefore, the horizontal FOV is about 49◦ . 440
Z. Yan et al.
Optics Communications 402 (2017) 437–441
Appendix A. Supplementary data Supplementary material related to this article can be found online at http://dx.doi.org/10.1016/j.optcom.2017.06.061. References [1] X. Xiao, B. Javidi, M. Martinezcorral, A. Stern, Advances in three-dimensional integral imaging: sensing, display, and applications [invited], Appl. Opt. 52 (4) (2013) 546–560. [2] J.H. Park, K. Hong, B. Lee, Recent progress in three-dimensional information processing based on integral imaging., Appl. Opt. 48 (34) (2009) 77–94. [3] J.S. Jang, B. Javidi, Improved viewing resolution of three-dimensional integral imaging by use of nonstationary micro-optics, Opt. Lett. 27 (5) (2002) 324–326. [4] T.H. Jen, X. Shen, G. Yao, Y.P. Huang, H.P. Shieh, B. Javidi, Dynamic integral imaging display with electrically moving array lenslet technique using liquid crystal lens, Opt. Express 23 (14) (2015) 18415–18421. [5] Y. Oh, D. Shin, B.G. Lee, S.I. Jeong, H.J. Choi, Resolution-enhanced integral imaging in focal mode with a time-multiplexed electrical mask array, Opt. Express 22 (15) (2014) 17620–17629. [6] H. Liao, T. Dohi, M. Iwahara, Improved viewing resolution of integral videography by use of rotated prism sheets, Opt. Express 15 (8) (2007) 4814–4822. [7] Y. Kim, J. Kim, J.-M. Kang, J.-H. Jung, H. Choi, B. Lee, Point light source integral imaging with improved resolution and viewing angle by the use of electrically movable pinhole array, Opt. Express 15 (26) (2007) 18253–18267. [8] H. Hoshino, F. Okano, H. Isono, I. Yuyama, Analysis of resolution limitation of integral photography, J. Opt. Soc. Am. A 15 (8) (1998) 2059–2065. [9] F. Okano, J. Arai, K. Mitani, M. Okui, Real-time integral imaging based on extremely high resolution video system, Proc. IEEE 94 (3) (2006) 490–501. [10] H. Liao, M. Iwahara, N. Hata, T. Dohi, High-quality integral videography using a multiprojector, Opt. Express 12 (6) (2004) 1067–1076. [11] Z. Wang, A. Wang, S. Wang, X. Ma, H. Ming, Resolution-enhanced integral imaging using two micro-lens arrays with different focal lengths for capturing and display, Opt. Express 23 (22) (2015) 28970–28977. [12] T. Okoshi, Optimum design and depth resolution of lens-sheet and projection-type three-dimensional displays, Appl. Opt. 10 (10) (1971) 2284–2291. [13] C. Burckhardt, Optimum parameters and resolution limitation of integral photography, J. Opt. Soc. Amer. 58 (1) (1968) 71–74. [14] J.-H. Lee, J. Park, D. Nam, S.Y. Choi, D.-S. Park, C.Y. Kim, Optimal projector configuration design for 300-mpixel multi-projection 3d display, Opt. Express 21 (22) (2013) 26820–26835. [15] A. Jones, I. McDowall, H. Yamada, M. Bolas, P. Debevec, Rendering for an interactive 360 light field display, in: ACM Transactions on Graphics (TOG), Vol. 26, ACM, 2007, p. 40 3. [16] S. Santoro, M. Crenshaw, I. Ashdown, Light control devices and methods implemented with kinoform diffusers having controllable diffusion characteristics Google Patents, US Patent 7,660,039, (Feb. 9 2010), 2010.
Fig. 7. Perspective images integrated (a) without holographic diffuser, and (b) with holographic diffuser fixed on 𝑧 = 16 cm. Supplementary data 2 is a video of perspective images.
4. Conclusion In conclusion, a method to realize II based light field display with improved viewing resolution is proposed. The discrete light field of the reconstructed points is interpolated to be continuous by use of holographic diffuser, and the viewing resolution is proportional to the maximum projectable spatial frequency of exit pupil, meanwhile, the viewing resolution limit imposed by Nyquist sampling frequency is overcome. Therefore, the viewing resolution can be improved by enlarging the gap between LA and display panel, or by using display panels with smaller pixel pitch. As display panel of large size and high dpi (dots per inch) is a trend, our method could be an effective way of realizing high-resolution and large-scale II based light field display. The defects of using holographic diffuser, such as degradation of the DOF and light intensity, should never be neglected, and we think they can be alleviated by optimal design of the diffusion characteristics of holographic diffuser and specifications of II systems. The diffusion characteristics of holographic diffuser have great impact on the reconstruction quality, and how it affects the reconstruction quality and its optimal design need further investigation. The proposed light field display method based on II is of good 3D effect and economical to mass production, and we expect it to be a solution for 3D display applications in future. Acknowledgments This work is partly supported by National Excellent Doctoral Dissertation of PR China (FANEDD) (201432), the Beijing Natural Science Foundation (4152049), and the Beijing NOVA program (Z1511000003150119).
441