- Email: [email protected]
Backward ray tracing based rectification for real-time integral imaging display system
Optics Communications 458 (2020) 124752
Contents lists available at ScienceDirect
Optics Communications journal homepage: www.elsevier.com/locate/optcom
Backward ray tracing based rectification for real-time integral imaging display system Weiping Huo, Xinzhu Sang ∗, Shujun Xing, Yanxin Guan, Yuanhang Li State Key Laboratory of Information Photonics and Optical Communications, Beijing University of Posts and Telecommunications (BUPT), 100876 Beijing, China
ARTICLE
INFO
Keywords: Three-dimensional display Ray tracing Integral imaging Elemental image rectification
ABSTRACT In an integral imaging (InI) display system, the deviation of the lens position results in blur and distortion of the reconstructed three-dimensional (3D) image. To achieve undistorted and real-time 3D images, an image correction method based on the backward ray tracing (BRT) technology is presented. By finding the correct pixel-to-ray correspondence, the origin and the direction of the light ray for every pixel in elemental image array (EIA) are reset, and the corrected 3D image is obtained based on the BRT technique. Experimental results demonstrate the feasibility and effectiveness of the proposed method, and the structural similarity (SSIM) index of the reconstructed 3D image increased by 20∼30% compared with that without calibrating. The frame rate of EIA generation is over 40 fps, which can realize real-time generation.
1. Introduction Recently, 3D displays have attracted much attention [1–4]. There is growing evidence that 3D imaging techniques have future massmarket potential in the fields of entertainment and communications. As a method of autostereoscopic displays, Integral imaging display consist of an LCD panel, a lens array and a holographic functional screen (HFS), which can display full-color 3D images with continuous horizontal and vertical motion parallax [5,6]. However, some lenses in the lens array may deviate from the expected position due to insufficient processing techniques of equipment manufacture, which will degrade the reconstructed 3D image quality. To achieve a further high-quality 3D display, calibration is necessary to correct optical misalignment and optical aberrations. In the 3D display, several useful correction methods were proposed. Arai et al. [7] presented the effects of misarrangement of elements (elemental lenses and elemental images) that constructed 3D images in InI in terms of local and global positional errors, which were a foundation for many later works. Raúl Martínez-Cuenca et al. [8] corrected image distortions by improving the structure of the optical system. Joon-Jae Lee et al. [9] proposed a simple correction method of distorted elemental images with surface markers on lens array for computational InI reconstruction. In Keehoon Hong’s paper, a new method was applied to correct EIA and extract lens lattice by projection image transformation in InI [10]. Kawakita et al. [11] analyzed the relationships between the geometric distortion in elemental images caused by an elemental lens and the spatial distortion in the reconstructed 3D image. Recently, a hybrid camera array-based calibration for computer-generated InI with
flexible setup was proposed [12], the method could get high precision at a reasonable time cost. Xingpeng Yan et al. [13] proposed a postcalibration compensation method for the InI with microlens array, and the inter-lens position misalignment was corrected by forcing it to the image in a regular ideal reference grid, which effectively solved the distortion caused by lens position error. However, none of the methods mentioned above can solve the problem of lens misalignment in real-time 3D content generation. The above methods pay more attention to image correction, and therefore, the generation of images is time-consuming. Recently, a high-speed BRT technique with a general processor unit (GPU) [14–16] has been developed very quickly, which was also demonstrated in experiments [17,18]. Based on the real-time BRT technique, a new method to correct image distortion in the EIA is proposed. In our method, a virtual lens array whose parameters are determined by the real lens array of the InI display, so the lens position information needs to be required in the InI display system. The view ray in every pixel can be modified according to the lens position, and EIA can be generated real-timely by using BRT. Several 3D optically reconstructed images are used to demonstrate the validity of the proposed correction method. The SSIM index of the reconstructed 3D image increased by 20 ∼ 30%. The frame rate of EIA generation is over 40 fps, which can realize real-time generation. 2. Correction method The typical structure of the InI system is shown in Fig. 1(a), including an LCD panel, a lens array and an HFS [19]. Ideally, the
∗ Corresponding author. E-mail address: [email protected] (X. Sang).
https://doi.org/10.1016/j.optcom.2019.124752 Received 16 July 2019; Received in revised form 8 October 2019; Accepted 12 October 2019 Available online 16 October 2019 0030-4018/© 2019 Elsevier B.V. All rights reserved.
W. Huo, X. Sang, S. Xing et al.
Optics Communications 458 (2020) 124752
Fig. 1. (a) The sketch map of the InI display system, (b) the actual lens array with inter-lens position deviations, (c) the blurred image with inter-lens position deviations.
lens array in the InI system is arranged regularly, and the distance between adjacent lenses is fixed. However, for actual assembling, there are often positional errors of the lenses in Fig. 1(b). A blurred image due to deviations of the lens position is shown in Fig. 1(c). As a result, the reconstructed image is blurred and broken for the mismatch of pixel-to-ray correspondence. To reduce the impact of lens position error on image quality, measures must be taken. We remap the pixels and viewpoints according to the relative offset of the lens. Firstly, the positions of lenses are collected. Then, the range of the elemental image is redivided. Finally, the origin and the direction of the light ray for every pixel in EIA are reset, and the correct 3D image is obtained based on BRT. The detailed steps of the calibration are introduced as follows. The structure of the lens array is shown in Fig. 2. The lens is fixed on a perforated metal plate. The diameter of the hole is 10.5 ± 0.30 mm, the diameter of the lens is 10 ± 0.01 mm, and the center distance of the hole is 11 ± 0.10 mm. The total deviation during assembly is about ±0.401 mm (±3 pixels) from the expected position. The lens array includes 46 × 26 lens, and each lens covers 83 × 83 pixels. 2.1. Collecting lens position Fig. 2. (a) The arrangement of the lens array. (b) The assembling parameters of the lens array.
There is a one-to-one mapping between the lens and the elemental image in space. As shown in Fig. 3(a), the center position of the lens is aligned with the elemental image center position. The position of the lens’s center is measured. The pixel index corresponding to the center of the lens is recorded. As shown in Fig. 3(b), a red moving symbol ‘‘+’’ displayed on LCD is used to measure the position of lens center. The steps of the measuring process are as followed. Firstly, the position of the ‘‘+’’ is modified, and the picture of light field image at different ‘‘+’’ position is captured by cameras, as shown in Fig. 4(a). The size of ‘‘+’’ is 7 × 7 pixels (0.952 mm × 0.952 mm). To
reduce labor costs, cameras shoot videos automatically. The resolution of each picture is 3840 × 2160. The ‘‘+’’ iterates over all the pixels in an elemental image, so 83 × 83 pictures need to be captured, as shown in Fig. 4(b). Secondly, the clearest image of ‘‘+’’ is found in these pictures, and the corresponding pixel index is recorded. Each picture is divided into 2
W. Huo, X. Sang, S. Xing et al.
Optics Communications 458 (2020) 124752
Fig. 3. (a) The correspondence between lens array and EIA. (b) The lens position and its elemental image. Table 1 Related parameters of InI display system. LCD panel
Lens array
Resolution Pixel pitch
3840 × 2160 136 μm
Array size Radius Len focal distance Distance between lens array sheet and LCD panel
46 (H) × 26(V) 0.5 cm 16 mm 18 mm
to the Voronoi cells. The pixel P (x, y) on the LCD belongs to the elemental image of the nearest Lens (x, y) from this pixel. In other words, the dividing line between the elemental images is the pixels on the vertical bisector of the adjacent lens center as shown in Fig. 5(a). As shown in Fig. 5(b), the lens position Lensij is the nearest pixel from pixel 𝑷𝒙𝒚 , so the pixel P (x, y) belongs to the elemental image (i, j). 2.3. EIA generation BRT technique works by tracing a path from the starting point 𝑶𝒙𝒚 through the virtual lens 𝑳𝒙𝒚 as shown in Fig. 6, and the color of the object visible through it is calculated. A basic backward ray tracer includes three parts [16]. The first part is ray generation, where the origin and the direction of each pixel’s view light ray based on the camera geometry are calculated. The second part is a ray intersection, where the closest object intersecting the view ray is found. Shading processing is the final part, which computes the pixel color based on the results of the ray intersection. The starting point 𝑶𝒙𝒚 and the direction 𝑫𝒙𝒚 of the view Ray(x, y) is calculated based on the position of each pixel and the center of the corresponding lens. The view Ray(x, y) collides with all 3D objects in the virtual space, and the color of the nearest collision point can be used as the color of the pixel. The coordinate of the starting point 𝑷𝒙𝒚 of the view Ray(x, y) in the virtual world can be expressed as
46 × 26 small pictures, the SSIM index between small pictures and ideal ‘‘+’’ image is calculated. The maximum of these 83 × 83 SSIM indices is found, and the corresponding pixel indices of ‘‘+’’ is recorded. Thirdly, the lens position according to the pixel indices of ‘‘+’’ is calculated and the similar principle of triangle is used, as shown in Fig. 4(c). The lens’s position of the ith row and the jth column is recorded as Lensij = (m, n). The mean of the 5 repetitions of the above process is taken. To reduce the acquisition time, multiple cameras are used to collect data at the same time. The whole collecting process takes about 3 h. It takes 2 h for data acquisition, and the computing time is 1 h. The pixel deviation between the results and measurements with a ruler is ±1. If the lens diameter is smaller and the amount of lens is bigger, pictures of different lenses’ ‘‘+’’ image can be captured multiple times instead of one time. For example, we can capture pictures of lens on odd row firstly, and then pictures of lens on even row are captured. The size of symbols should be increased, appropriately.
𝑷 xy = 𝑥 ⋅ 𝑑 ⋅ 𝒖 + 𝑦 ⋅ 𝑑 ⋅ 𝒗 + 𝑧 ⋅ 𝒘
(1)
where (x, y, 0) is the pixel index, z is the depth of virtual screen panel, u, v, w represents unit vectors in the 3D coordinate system respectively, and d is the actual size of pixel.
2.2. Pixel allocation scheme of the elemental image
𝑳𝑥𝑦 = 𝑓 𝑖𝑛𝑑_𝑚𝑖𝑛_𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒_𝑝𝑜𝑖𝑛𝑡(𝑷 𝑥𝑦 , 𝑳𝒆𝒏𝒔𝑖𝑗 )
To obtain the highest possible 3D viewing and maximum field-ofview (FOV) of the InI display, it is essential that the structure lens array overlaid over the LCD panel matches the pixel structure of the EIA displayed on the LCD. The arrangement of lenses is irregular, consequently, the range of the elemental image of EIA needs to be redivided. In mathematics, a Voronoi diagram is a partitioning algorithm of a plane into regions based on the distance to points in a specific subset of the plane. That set of points called sites is specified beforehand, and for each seed, there is a corresponding region consisting of all points closer to the seed than to any other. These regions are called Voronoi cells [20]. The above method is used to allocate pixels of adjacent elemental images. Each lens center corresponds to a site in the above-described Voronoi diagram, and the pixel area after the distribution corresponds
(2)
0≤𝑖≤𝑀,0≤𝑗≤𝑀
find_min_distance_point (𝑷𝒙𝒚 , Lensij ) is a function defined to find the nearest point to 𝑷𝒙𝒚 in all Lens𝒙𝒚 points. The coordinate of direction unit vector 𝑶𝒙𝒚 of the view Ray(x, y) in the virtual world can be expressed as 𝑫 𝑥𝑦 = −𝑛𝑜𝑟𝑚𝑎𝑙𝑖𝑧𝑒(𝑳𝑥𝑦 − 𝑷 𝑥𝑦 )
(3)
𝑶𝑥𝑦 = 𝑳𝑥𝑦 − 𝑙𝑛𝑒𝑎𝑟 ⋅ 𝑫 𝑥𝑦
(4)
where normalize is a function normalize vector. Then, the parallel computing method is adapted to generate the view rays. Each Ray(x, y) from the starting point 𝑶𝒙𝒚 is projected into space along the direction 𝑫𝒙𝒚 , and intersects with all objects in virtual space. Finally, all the pixels on the EIA are shaded, and a high-quality 3D image is obtained as shown in Fig. 7. 3
W. Huo, X. Sang, S. Xing et al.
Optics Communications 458 (2020) 124752
Fig. 4. (a) The picture of light field image is captured at different ‘‘+’’ position with cameras. (b) The clearest image of ‘‘+’’ is found in these pictures, and corresponding pixel index is recorded. (c) The pixel index corresponding to the optical axis of the lens is calculated based on the similar principle of triangle.
Table 2 Pixel index corresponding to the central position of the lens. (row, col)
1
1 2 ... 25 26
(67, (72, ... (67, (68,
2 55) 141) 2052) 2132)
(153, (156, ... (157, (149,
3 59) 143) 2050) 2131)
(241, (240, ... (240, (234,
44 56) 140) 2046) 2134)
4
... ... ... ... ...
(3643, (3642, ... (3643, (3635,
45 142) 220) 2046) 2133)
(3719, (3720, ... (3723, (3726,
46 139) 221) 2047) 2131)
(3804, (3806, ... (3804, (3801,
141) 222) 204) 213)
W. Huo, X. Sang, S. Xing et al.
Optics Communications 458 (2020) 124752
Fig. 5. (a) The pixel distribution rule of the adjacent elemental image corresponding to the lens. (b) The shape and arrangement of the calibrated elemental image.
Fig. 6. The pixel color calculation process in the elemental image by BRT technology.
Fig. 7. (a) The EIA and (b) the constructed of 3D text ‘‘P’’ after calibration.
3. Result and discussion 3.1. Display settings basic settings A prototype is fabricated to evaluate the feasibility of the proposed method. The PC hardware is composed of an Intel(R) Core (TM) i74790 CPU @ 3.6 GHZ with 8 Gb RAM and NVidia GeForce GTX 970 (4 GB/NVidia) graphic card. The typical structure of the InI system is shown in Fig. 8, including an LCD panel, a lens array and an HFS. The parameters of the InI display device in the experiment are summarized in Table 1. The position data of 26 × 46 lenses are measured and recorded. The partial data of the lens position are shown in Table 2. The data show that the center of the lens deviates from the expected position. Each coordinate represents the center of a lens, ideally, the horizontal and vertical spacing between the coordinates is the same (83 pixels wide). The standard deviation is 0.14 mm in horizontal, and 0.21 mm in vertical.
Fig. 8. InI light field display system.
5
W. Huo, X. Sang, S. Xing et al.
Optics Communications 458 (2020) 124752
Fig. 9. The EIA of 3D text ‘‘P’’, (a) before calibration, (b) after calibration.
Fig. 10. (a) The picture of 3D object captured by a virtual camera. The display result for the reconstructed images locate in 200 cm away from the display, (b) before calibration and (c) after calibration.
Fig. 11. The SSIM index changes in different viewing angle (a) in horizontal and (b) vertical directions.
6
W. Huo, X. Sang, S. Xing et al.
Optics Communications 458 (2020) 124752
in Table 2, so the SSIM index is decreased with the absolute value of angle more remarkably in the vertical direction. The algorithm is accelerated by GPU, so it has a higher computation speed. As shown in Fig. 12, the frame rate decreases with increasing of the vertex number and EIA’s resolution. 4. Conclusion In summary, a correction method is presented to correct the distortion of the InI image caused by lens misalignment. Real-time BRT technology is adopted to regenerate a coded image based on the position of the lens in the actual display. Experimental results verify that the correction method can effectively decrease image distortion caused by lens position deviation. The SSIM index of the reconstructed 3D image is increased by 20 ∼ 30%, and the frame rate is over 40 fps. We believe that the proposed method is useful for the 3D content generation of InI with complex lens arrangement.
Fig. 12. The rendering performance changes with the resolution of EIA and vertex number.
Declaration of competing interest 3.2. Correction effect and analysis The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
As shown in Fig. 9, 3D character ‘‘P’’ is used to validate the proposed correction method. The green line is the boundary of the elemental images. Ideally, the boundary is a regular square. After calibrating, although the EIA becomes an irregular arrangement, each pixel is assigned to the correct viewpoint and lens. The pixel allocation rule is implemented by Section 2.2. Due to the correct correspondence between the viewpoint and the pixel, the quality of the 3D image is greatly improved. As shown in Fig. 10, the Tiger, Text and Forklift 3D models are used to verify the feasibility of the proposed method. As shown in Fig. 10(a), the pictures of 3D objects are captured with the virtual camera in a 3D rendering software, and their resolution is 3840 × 2160. The uncorrected image and corrected image are captured with the camera in the fact scene as shown in Fig. 10(b) and (c), and the resolution of pictures is also 3840 × 2160. The reconstructed 3D images with the original EIA show spatial deforms and blurs. On the contrary, the reconstructed 3D images with the EIA corrected with the proposed method do not show noticeable spatial distortions and deforms, and quite clear images of the three objects are perceived. Significant improvement in image display quality compared with 3D images before correction, and the frame rate of the rendering is over 40 fps. The resolution of EIA displayed on the LCD panel is 3840 × 2160. The resolution of a single viewpoint is 46(H) × 26(V), which is equal to the number of lenses. The viewpoint number is around 83 × 83 (3840/46 ≈ 83, 2160/26 ≈ 83). Because the spatial resolution can be improved with the HFS [4,6,21], the spatial resolution is higher than 46(H) × 26(V). The SSIM values of 3D reconstructed images from different viewing angle before and after correction are shown in Fig. 11. The SSIM index is based on the computation of three terms (luminance term, contrast term, and structural term) and the ratio is 1:1:1. Theoretically, the FOV of the InI display system is ±20o both in horizontal and vertical directions. The SSIM values of the reconstructed 3D image are increased by 20% ∼ 30% in the FOV. The SSIM values in both horizontal and vertical directions are improved, and the improvement is more obvious when it closer the middle point of view. In terms of reconstructed geometry accuracy and image quality, the proposed method is effective in interlens position error calibration. For the uncorrected results, whether within or outside the viewing area, the 3D image is distorted. Therefore, the SSIM indices are relatively low and do not change significantly. For the corrected results, the SSIM index is decreased with increasing the absolute value of the viewing angle, because the viewpoints in the large viewing angle are susceptible to be interfered from pixels in adjacent elemental images. The standard deviation of position deviation in the vertical direction is more than that in the horizontal direction, as shown
Funding National Key Research and Development Program (2017YFB100 2900); Fundamental Research Funds for the Cental Universities (2019RC13, 2019PTB-018); National Natural Science Foundation of China (61705014); State Key Laboratory of Information Photonics and Optical Communications. References [1] Y. Son, V.V. Saveljev, B. Javidi, D.S. Kim, M.C. Park, Pixel patterns for voxels in a contact-type three-dimensional imaging system for full-parallax image display, Appl. Opt. 45 (2006) 4325–4333. [2] E. Ives, Optical properties of a Lippmann lenticulated sheet, J. Opt. Soc. Amer. 21 (1931) 171–176. [3] H. Park, K. Hong, B. Lee, Recent progress in three-dimensional information processing based on integral imaging, Appl. Opt. 48 (2009) H77–H94. [4] X. Sang, X. Gao, X. Yu, S. Xing, Y. Li, Y. Wu, Interactive floating full-parallax digital three-dimensional light-field display based on wavefront recomposing, Opt. Express 26 (2018) 8883–8889. [5] M. Cho, M. Daneshpanah, I. Moon, B. Javidi, Three-dimensional optical sensing and visualization using integral imaging, in: Proceedings of the IEEE, IEEE, 2011, pp. 556–575. [6] X. Sang, F.C. Fan, C.C. Jiang, S. Choi, W. Dou, C. Yu, D. Xu, Demonstration of a large-size real-time full-color three-dimensional display, Opt. Lett. 34 (24) (2009) 3803–3805. [7] J. Arai, M. Okui, M. Kobayashi, F. Okano, Geometrical effects of positional errors in integral photography, J. Opt. Soc. Amer. A 21 (6) (2004) 951–958. [8] R. Martínez-Cuenca, A. Pons, G. Saavedra, M. Martínez-Corral, B. Javidi, Optically-corrected elemental images for undistorted integral image display, Opt. Express (2006). [9] J.-J. Lee, D.-H. Shin, B.-G. Lee, Simple correction method of distorted elemental images using surface markers on lenslet array for computational integral imaging reconstruction, Opt. Express (2009). [10] K. Hong, J. Hong, J.-H. Jung, J.-H. Park, B. Lee, Rectification of elemental image set and extraction of lens lattice by projective image transformation in integral imaging, Opt. Express (2010). [11] M. Kawakita, H. Sasaki, J. Arai, F. Okano, K. Suehiro, Y. Haino, M. Yoshimura, M. Sato, Geometric analysis of spatial distortion in projection-type integral imaging, Opt. Lett. (2008). [12] G. Chen, H. Wang, M. Liu, H. Liao, Hybrid camera array based calibration for computer-generated integral photography display, J. Opt. Soc. Amer. A (2018); Photography display, J. Opt. Soc. Am. A 35 (9) (2018) 1567–1574. [13] X. Yan, J. Wen, Z. Yan, T. Zhang, X. Jiang, Post-calibration compensation method for integral imaging system with macrolens array, Opt. Express (2019). [14] D.C. Anderson, J. Cychosz, An introduction to ray tracing, Image Vis. Comput. (2003). [15] J. Arvo, Backward ray tracing, Dev. Ray Tracing (1986). 7
W. Huo, X. Sang, S. Xing et al.
Optics Communications 458 (2020) 124752 [19] Z. Yan, X. Yan, X. Jiang, H. Gao, J. Wen, Integral imaging based light field display with enhanced viewing resolution using holographic diffuser, Opt. Commun. 402 (2017) 437–441. [20] F. Aurenhammer, Voronoi diagrams—a survey of a fundamental geometric data structure, ACM Comput. Surv. (1991). [21] S. Li, Haifeng Li, Z. Zheng, Y. Peng, S. Wang, X. Liu, Full-parallax three-dimensional display using new directional diffuser, Chin. Opt. Lett. (2011).
[16] G. Pratx, L. Xing, GPU computing in medical physics: a review, Med. Phys. (2011). [17] S. Xing, X. Sang, X. Yu, C. Duo, B. Pang, X. Gao, S. Yang, Y. Guan, B. Yan, J. Yuan, K. Wang, High-efficient computer-generated integral imaging based on the backward ray-tracing technique and optical reconstruction, Opt. Express (2017). [18] B. Pang, X. Sang, S. Xing, X. Yu, D. Chen, B. Yan, K. Wang, C. Yu, B. Liu, C. Cui, Y. Guan, W. Xiang, L. Ge, High-efficient rendering of the multi-view image for the three-dimensional display based on the backward ray-tracing technique, Opt. Commun. 405 (2017).
8
Contents lists available at ScienceDirect
Optics Communications journal homepage: www.elsevier.com/locate/optcom
Backward ray tracing based rectification for real-time integral imaging display system Weiping Huo, Xinzhu Sang ∗, Shujun Xing, Yanxin Guan, Yuanhang Li State Key Laboratory of Information Photonics and Optical Communications, Beijing University of Posts and Telecommunications (BUPT), 100876 Beijing, China
ARTICLE
INFO
Keywords: Three-dimensional display Ray tracing Integral imaging Elemental image rectification
ABSTRACT In an integral imaging (InI) display system, the deviation of the lens position results in blur and distortion of the reconstructed three-dimensional (3D) image. To achieve undistorted and real-time 3D images, an image correction method based on the backward ray tracing (BRT) technology is presented. By finding the correct pixel-to-ray correspondence, the origin and the direction of the light ray for every pixel in elemental image array (EIA) are reset, and the corrected 3D image is obtained based on the BRT technique. Experimental results demonstrate the feasibility and effectiveness of the proposed method, and the structural similarity (SSIM) index of the reconstructed 3D image increased by 20∼30% compared with that without calibrating. The frame rate of EIA generation is over 40 fps, which can realize real-time generation.
1. Introduction Recently, 3D displays have attracted much attention [1–4]. There is growing evidence that 3D imaging techniques have future massmarket potential in the fields of entertainment and communications. As a method of autostereoscopic displays, Integral imaging display consist of an LCD panel, a lens array and a holographic functional screen (HFS), which can display full-color 3D images with continuous horizontal and vertical motion parallax [5,6]. However, some lenses in the lens array may deviate from the expected position due to insufficient processing techniques of equipment manufacture, which will degrade the reconstructed 3D image quality. To achieve a further high-quality 3D display, calibration is necessary to correct optical misalignment and optical aberrations. In the 3D display, several useful correction methods were proposed. Arai et al. [7] presented the effects of misarrangement of elements (elemental lenses and elemental images) that constructed 3D images in InI in terms of local and global positional errors, which were a foundation for many later works. Raúl Martínez-Cuenca et al. [8] corrected image distortions by improving the structure of the optical system. Joon-Jae Lee et al. [9] proposed a simple correction method of distorted elemental images with surface markers on lens array for computational InI reconstruction. In Keehoon Hong’s paper, a new method was applied to correct EIA and extract lens lattice by projection image transformation in InI [10]. Kawakita et al. [11] analyzed the relationships between the geometric distortion in elemental images caused by an elemental lens and the spatial distortion in the reconstructed 3D image. Recently, a hybrid camera array-based calibration for computer-generated InI with
flexible setup was proposed [12], the method could get high precision at a reasonable time cost. Xingpeng Yan et al. [13] proposed a postcalibration compensation method for the InI with microlens array, and the inter-lens position misalignment was corrected by forcing it to the image in a regular ideal reference grid, which effectively solved the distortion caused by lens position error. However, none of the methods mentioned above can solve the problem of lens misalignment in real-time 3D content generation. The above methods pay more attention to image correction, and therefore, the generation of images is time-consuming. Recently, a high-speed BRT technique with a general processor unit (GPU) [14–16] has been developed very quickly, which was also demonstrated in experiments [17,18]. Based on the real-time BRT technique, a new method to correct image distortion in the EIA is proposed. In our method, a virtual lens array whose parameters are determined by the real lens array of the InI display, so the lens position information needs to be required in the InI display system. The view ray in every pixel can be modified according to the lens position, and EIA can be generated real-timely by using BRT. Several 3D optically reconstructed images are used to demonstrate the validity of the proposed correction method. The SSIM index of the reconstructed 3D image increased by 20 ∼ 30%. The frame rate of EIA generation is over 40 fps, which can realize real-time generation. 2. Correction method The typical structure of the InI system is shown in Fig. 1(a), including an LCD panel, a lens array and an HFS [19]. Ideally, the
∗ Corresponding author. E-mail address: [email protected] (X. Sang).
https://doi.org/10.1016/j.optcom.2019.124752 Received 16 July 2019; Received in revised form 8 October 2019; Accepted 12 October 2019 Available online 16 October 2019 0030-4018/© 2019 Elsevier B.V. All rights reserved.
W. Huo, X. Sang, S. Xing et al.
Optics Communications 458 (2020) 124752
Fig. 1. (a) The sketch map of the InI display system, (b) the actual lens array with inter-lens position deviations, (c) the blurred image with inter-lens position deviations.
lens array in the InI system is arranged regularly, and the distance between adjacent lenses is fixed. However, for actual assembling, there are often positional errors of the lenses in Fig. 1(b). A blurred image due to deviations of the lens position is shown in Fig. 1(c). As a result, the reconstructed image is blurred and broken for the mismatch of pixel-to-ray correspondence. To reduce the impact of lens position error on image quality, measures must be taken. We remap the pixels and viewpoints according to the relative offset of the lens. Firstly, the positions of lenses are collected. Then, the range of the elemental image is redivided. Finally, the origin and the direction of the light ray for every pixel in EIA are reset, and the correct 3D image is obtained based on BRT. The detailed steps of the calibration are introduced as follows. The structure of the lens array is shown in Fig. 2. The lens is fixed on a perforated metal plate. The diameter of the hole is 10.5 ± 0.30 mm, the diameter of the lens is 10 ± 0.01 mm, and the center distance of the hole is 11 ± 0.10 mm. The total deviation during assembly is about ±0.401 mm (±3 pixels) from the expected position. The lens array includes 46 × 26 lens, and each lens covers 83 × 83 pixels. 2.1. Collecting lens position Fig. 2. (a) The arrangement of the lens array. (b) The assembling parameters of the lens array.
There is a one-to-one mapping between the lens and the elemental image in space. As shown in Fig. 3(a), the center position of the lens is aligned with the elemental image center position. The position of the lens’s center is measured. The pixel index corresponding to the center of the lens is recorded. As shown in Fig. 3(b), a red moving symbol ‘‘+’’ displayed on LCD is used to measure the position of lens center. The steps of the measuring process are as followed. Firstly, the position of the ‘‘+’’ is modified, and the picture of light field image at different ‘‘+’’ position is captured by cameras, as shown in Fig. 4(a). The size of ‘‘+’’ is 7 × 7 pixels (0.952 mm × 0.952 mm). To
reduce labor costs, cameras shoot videos automatically. The resolution of each picture is 3840 × 2160. The ‘‘+’’ iterates over all the pixels in an elemental image, so 83 × 83 pictures need to be captured, as shown in Fig. 4(b). Secondly, the clearest image of ‘‘+’’ is found in these pictures, and the corresponding pixel index is recorded. Each picture is divided into 2
W. Huo, X. Sang, S. Xing et al.
Optics Communications 458 (2020) 124752
Fig. 3. (a) The correspondence between lens array and EIA. (b) The lens position and its elemental image. Table 1 Related parameters of InI display system. LCD panel
Lens array
Resolution Pixel pitch
3840 × 2160 136 μm
Array size Radius Len focal distance Distance between lens array sheet and LCD panel
46 (H) × 26(V) 0.5 cm 16 mm 18 mm
to the Voronoi cells. The pixel P (x, y) on the LCD belongs to the elemental image of the nearest Lens (x, y) from this pixel. In other words, the dividing line between the elemental images is the pixels on the vertical bisector of the adjacent lens center as shown in Fig. 5(a). As shown in Fig. 5(b), the lens position Lensij is the nearest pixel from pixel 𝑷𝒙𝒚 , so the pixel P (x, y) belongs to the elemental image (i, j). 2.3. EIA generation BRT technique works by tracing a path from the starting point 𝑶𝒙𝒚 through the virtual lens 𝑳𝒙𝒚 as shown in Fig. 6, and the color of the object visible through it is calculated. A basic backward ray tracer includes three parts [16]. The first part is ray generation, where the origin and the direction of each pixel’s view light ray based on the camera geometry are calculated. The second part is a ray intersection, where the closest object intersecting the view ray is found. Shading processing is the final part, which computes the pixel color based on the results of the ray intersection. The starting point 𝑶𝒙𝒚 and the direction 𝑫𝒙𝒚 of the view Ray(x, y) is calculated based on the position of each pixel and the center of the corresponding lens. The view Ray(x, y) collides with all 3D objects in the virtual space, and the color of the nearest collision point can be used as the color of the pixel. The coordinate of the starting point 𝑷𝒙𝒚 of the view Ray(x, y) in the virtual world can be expressed as
46 × 26 small pictures, the SSIM index between small pictures and ideal ‘‘+’’ image is calculated. The maximum of these 83 × 83 SSIM indices is found, and the corresponding pixel indices of ‘‘+’’ is recorded. Thirdly, the lens position according to the pixel indices of ‘‘+’’ is calculated and the similar principle of triangle is used, as shown in Fig. 4(c). The lens’s position of the ith row and the jth column is recorded as Lensij = (m, n). The mean of the 5 repetitions of the above process is taken. To reduce the acquisition time, multiple cameras are used to collect data at the same time. The whole collecting process takes about 3 h. It takes 2 h for data acquisition, and the computing time is 1 h. The pixel deviation between the results and measurements with a ruler is ±1. If the lens diameter is smaller and the amount of lens is bigger, pictures of different lenses’ ‘‘+’’ image can be captured multiple times instead of one time. For example, we can capture pictures of lens on odd row firstly, and then pictures of lens on even row are captured. The size of symbols should be increased, appropriately.
𝑷 xy = 𝑥 ⋅ 𝑑 ⋅ 𝒖 + 𝑦 ⋅ 𝑑 ⋅ 𝒗 + 𝑧 ⋅ 𝒘
(1)
where (x, y, 0) is the pixel index, z is the depth of virtual screen panel, u, v, w represents unit vectors in the 3D coordinate system respectively, and d is the actual size of pixel.
2.2. Pixel allocation scheme of the elemental image
𝑳𝑥𝑦 = 𝑓 𝑖𝑛𝑑_𝑚𝑖𝑛_𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒_𝑝𝑜𝑖𝑛𝑡(𝑷 𝑥𝑦 , 𝑳𝒆𝒏𝒔𝑖𝑗 )
To obtain the highest possible 3D viewing and maximum field-ofview (FOV) of the InI display, it is essential that the structure lens array overlaid over the LCD panel matches the pixel structure of the EIA displayed on the LCD. The arrangement of lenses is irregular, consequently, the range of the elemental image of EIA needs to be redivided. In mathematics, a Voronoi diagram is a partitioning algorithm of a plane into regions based on the distance to points in a specific subset of the plane. That set of points called sites is specified beforehand, and for each seed, there is a corresponding region consisting of all points closer to the seed than to any other. These regions are called Voronoi cells [20]. The above method is used to allocate pixels of adjacent elemental images. Each lens center corresponds to a site in the above-described Voronoi diagram, and the pixel area after the distribution corresponds
(2)
0≤𝑖≤𝑀,0≤𝑗≤𝑀
find_min_distance_point (𝑷𝒙𝒚 , Lensij ) is a function defined to find the nearest point to 𝑷𝒙𝒚 in all Lens𝒙𝒚 points. The coordinate of direction unit vector 𝑶𝒙𝒚 of the view Ray(x, y) in the virtual world can be expressed as 𝑫 𝑥𝑦 = −𝑛𝑜𝑟𝑚𝑎𝑙𝑖𝑧𝑒(𝑳𝑥𝑦 − 𝑷 𝑥𝑦 )
(3)
𝑶𝑥𝑦 = 𝑳𝑥𝑦 − 𝑙𝑛𝑒𝑎𝑟 ⋅ 𝑫 𝑥𝑦
(4)
where normalize is a function normalize vector. Then, the parallel computing method is adapted to generate the view rays. Each Ray(x, y) from the starting point 𝑶𝒙𝒚 is projected into space along the direction 𝑫𝒙𝒚 , and intersects with all objects in virtual space. Finally, all the pixels on the EIA are shaded, and a high-quality 3D image is obtained as shown in Fig. 7. 3
W. Huo, X. Sang, S. Xing et al.
Optics Communications 458 (2020) 124752
Fig. 4. (a) The picture of light field image is captured at different ‘‘+’’ position with cameras. (b) The clearest image of ‘‘+’’ is found in these pictures, and corresponding pixel index is recorded. (c) The pixel index corresponding to the optical axis of the lens is calculated based on the similar principle of triangle.
Table 2 Pixel index corresponding to the central position of the lens. (row, col)
1
1 2 ... 25 26
(67, (72, ... (67, (68,
2 55) 141) 2052) 2132)
(153, (156, ... (157, (149,
3 59) 143) 2050) 2131)
(241, (240, ... (240, (234,
44 56) 140) 2046) 2134)
4
... ... ... ... ...
(3643, (3642, ... (3643, (3635,
45 142) 220) 2046) 2133)
(3719, (3720, ... (3723, (3726,
46 139) 221) 2047) 2131)
(3804, (3806, ... (3804, (3801,
141) 222) 204) 213)
W. Huo, X. Sang, S. Xing et al.
Optics Communications 458 (2020) 124752
Fig. 5. (a) The pixel distribution rule of the adjacent elemental image corresponding to the lens. (b) The shape and arrangement of the calibrated elemental image.
Fig. 6. The pixel color calculation process in the elemental image by BRT technology.
Fig. 7. (a) The EIA and (b) the constructed of 3D text ‘‘P’’ after calibration.
3. Result and discussion 3.1. Display settings basic settings A prototype is fabricated to evaluate the feasibility of the proposed method. The PC hardware is composed of an Intel(R) Core (TM) i74790 CPU @ 3.6 GHZ with 8 Gb RAM and NVidia GeForce GTX 970 (4 GB/NVidia) graphic card. The typical structure of the InI system is shown in Fig. 8, including an LCD panel, a lens array and an HFS. The parameters of the InI display device in the experiment are summarized in Table 1. The position data of 26 × 46 lenses are measured and recorded. The partial data of the lens position are shown in Table 2. The data show that the center of the lens deviates from the expected position. Each coordinate represents the center of a lens, ideally, the horizontal and vertical spacing between the coordinates is the same (83 pixels wide). The standard deviation is 0.14 mm in horizontal, and 0.21 mm in vertical.
Fig. 8. InI light field display system.
5
W. Huo, X. Sang, S. Xing et al.
Optics Communications 458 (2020) 124752
Fig. 9. The EIA of 3D text ‘‘P’’, (a) before calibration, (b) after calibration.
Fig. 10. (a) The picture of 3D object captured by a virtual camera. The display result for the reconstructed images locate in 200 cm away from the display, (b) before calibration and (c) after calibration.
Fig. 11. The SSIM index changes in different viewing angle (a) in horizontal and (b) vertical directions.
6
W. Huo, X. Sang, S. Xing et al.
Optics Communications 458 (2020) 124752
in Table 2, so the SSIM index is decreased with the absolute value of angle more remarkably in the vertical direction. The algorithm is accelerated by GPU, so it has a higher computation speed. As shown in Fig. 12, the frame rate decreases with increasing of the vertex number and EIA’s resolution. 4. Conclusion In summary, a correction method is presented to correct the distortion of the InI image caused by lens misalignment. Real-time BRT technology is adopted to regenerate a coded image based on the position of the lens in the actual display. Experimental results verify that the correction method can effectively decrease image distortion caused by lens position deviation. The SSIM index of the reconstructed 3D image is increased by 20 ∼ 30%, and the frame rate is over 40 fps. We believe that the proposed method is useful for the 3D content generation of InI with complex lens arrangement.
Fig. 12. The rendering performance changes with the resolution of EIA and vertex number.
Declaration of competing interest 3.2. Correction effect and analysis The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
As shown in Fig. 9, 3D character ‘‘P’’ is used to validate the proposed correction method. The green line is the boundary of the elemental images. Ideally, the boundary is a regular square. After calibrating, although the EIA becomes an irregular arrangement, each pixel is assigned to the correct viewpoint and lens. The pixel allocation rule is implemented by Section 2.2. Due to the correct correspondence between the viewpoint and the pixel, the quality of the 3D image is greatly improved. As shown in Fig. 10, the Tiger, Text and Forklift 3D models are used to verify the feasibility of the proposed method. As shown in Fig. 10(a), the pictures of 3D objects are captured with the virtual camera in a 3D rendering software, and their resolution is 3840 × 2160. The uncorrected image and corrected image are captured with the camera in the fact scene as shown in Fig. 10(b) and (c), and the resolution of pictures is also 3840 × 2160. The reconstructed 3D images with the original EIA show spatial deforms and blurs. On the contrary, the reconstructed 3D images with the EIA corrected with the proposed method do not show noticeable spatial distortions and deforms, and quite clear images of the three objects are perceived. Significant improvement in image display quality compared with 3D images before correction, and the frame rate of the rendering is over 40 fps. The resolution of EIA displayed on the LCD panel is 3840 × 2160. The resolution of a single viewpoint is 46(H) × 26(V), which is equal to the number of lenses. The viewpoint number is around 83 × 83 (3840/46 ≈ 83, 2160/26 ≈ 83). Because the spatial resolution can be improved with the HFS [4,6,21], the spatial resolution is higher than 46(H) × 26(V). The SSIM values of 3D reconstructed images from different viewing angle before and after correction are shown in Fig. 11. The SSIM index is based on the computation of three terms (luminance term, contrast term, and structural term) and the ratio is 1:1:1. Theoretically, the FOV of the InI display system is ±20o both in horizontal and vertical directions. The SSIM values of the reconstructed 3D image are increased by 20% ∼ 30% in the FOV. The SSIM values in both horizontal and vertical directions are improved, and the improvement is more obvious when it closer the middle point of view. In terms of reconstructed geometry accuracy and image quality, the proposed method is effective in interlens position error calibration. For the uncorrected results, whether within or outside the viewing area, the 3D image is distorted. Therefore, the SSIM indices are relatively low and do not change significantly. For the corrected results, the SSIM index is decreased with increasing the absolute value of the viewing angle, because the viewpoints in the large viewing angle are susceptible to be interfered from pixels in adjacent elemental images. The standard deviation of position deviation in the vertical direction is more than that in the horizontal direction, as shown
Funding National Key Research and Development Program (2017YFB100 2900); Fundamental Research Funds for the Cental Universities (2019RC13, 2019PTB-018); National Natural Science Foundation of China (61705014); State Key Laboratory of Information Photonics and Optical Communications. References [1] Y. Son, V.V. Saveljev, B. Javidi, D.S. Kim, M.C. Park, Pixel patterns for voxels in a contact-type three-dimensional imaging system for full-parallax image display, Appl. Opt. 45 (2006) 4325–4333. [2] E. Ives, Optical properties of a Lippmann lenticulated sheet, J. Opt. Soc. Amer. 21 (1931) 171–176. [3] H. Park, K. Hong, B. Lee, Recent progress in three-dimensional information processing based on integral imaging, Appl. Opt. 48 (2009) H77–H94. [4] X. Sang, X. Gao, X. Yu, S. Xing, Y. Li, Y. Wu, Interactive floating full-parallax digital three-dimensional light-field display based on wavefront recomposing, Opt. Express 26 (2018) 8883–8889. [5] M. Cho, M. Daneshpanah, I. Moon, B. Javidi, Three-dimensional optical sensing and visualization using integral imaging, in: Proceedings of the IEEE, IEEE, 2011, pp. 556–575. [6] X. Sang, F.C. Fan, C.C. Jiang, S. Choi, W. Dou, C. Yu, D. Xu, Demonstration of a large-size real-time full-color three-dimensional display, Opt. Lett. 34 (24) (2009) 3803–3805. [7] J. Arai, M. Okui, M. Kobayashi, F. Okano, Geometrical effects of positional errors in integral photography, J. Opt. Soc. Amer. A 21 (6) (2004) 951–958. [8] R. Martínez-Cuenca, A. Pons, G. Saavedra, M. Martínez-Corral, B. Javidi, Optically-corrected elemental images for undistorted integral image display, Opt. Express (2006). [9] J.-J. Lee, D.-H. Shin, B.-G. Lee, Simple correction method of distorted elemental images using surface markers on lenslet array for computational integral imaging reconstruction, Opt. Express (2009). [10] K. Hong, J. Hong, J.-H. Jung, J.-H. Park, B. Lee, Rectification of elemental image set and extraction of lens lattice by projective image transformation in integral imaging, Opt. Express (2010). [11] M. Kawakita, H. Sasaki, J. Arai, F. Okano, K. Suehiro, Y. Haino, M. Yoshimura, M. Sato, Geometric analysis of spatial distortion in projection-type integral imaging, Opt. Lett. (2008). [12] G. Chen, H. Wang, M. Liu, H. Liao, Hybrid camera array based calibration for computer-generated integral photography display, J. Opt. Soc. Amer. A (2018); Photography display, J. Opt. Soc. Am. A 35 (9) (2018) 1567–1574. [13] X. Yan, J. Wen, Z. Yan, T. Zhang, X. Jiang, Post-calibration compensation method for integral imaging system with macrolens array, Opt. Express (2019). [14] D.C. Anderson, J. Cychosz, An introduction to ray tracing, Image Vis. Comput. (2003). [15] J. Arvo, Backward ray tracing, Dev. Ray Tracing (1986). 7
W. Huo, X. Sang, S. Xing et al.
Optics Communications 458 (2020) 124752 [19] Z. Yan, X. Yan, X. Jiang, H. Gao, J. Wen, Integral imaging based light field display with enhanced viewing resolution using holographic diffuser, Opt. Commun. 402 (2017) 437–441. [20] F. Aurenhammer, Voronoi diagrams—a survey of a fundamental geometric data structure, ACM Comput. Surv. (1991). [21] S. Li, Haifeng Li, Z. Zheng, Y. Peng, S. Wang, X. Liu, Full-parallax three-dimensional display using new directional diffuser, Chin. Opt. Lett. (2011).
[16] G. Pratx, L. Xing, GPU computing in medical physics: a review, Med. Phys. (2011). [17] S. Xing, X. Sang, X. Yu, C. Duo, B. Pang, X. Gao, S. Yang, Y. Guan, B. Yan, J. Yuan, K. Wang, High-efficient computer-generated integral imaging based on the backward ray-tracing technique and optical reconstruction, Opt. Express (2017). [18] B. Pang, X. Sang, S. Xing, X. Yu, D. Chen, B. Yan, K. Wang, C. Yu, B. Liu, C. Cui, Y. Guan, W. Xiang, L. Ge, High-efficient rendering of the multi-view image for the three-dimensional display based on the backward ray-tracing technique, Opt. Commun. 405 (2017).
8