- Email: [email protected]
Detecting and Removing Specular Reflectance Components Based on Image Linearization
Available online at www.sciencedirect.com Available online at www.sciencedirect.com
Available online at www.sciencedirect.com
ScienceDirect
Procedia Computer Science 00 (2019) 000–000 Procedia Computer Science 00 (2019) 000–000 Procedia Computer Science 159 (2019) 1576–1583
www.elsevier.com/locate/procedia www.elsevier.com/locate/procedia
23rd International Conference on Knowledge-Based and Intelligent Information & Engineering 23rd International Conference on Knowledge-Based Systems and Intelligent Information & Engineering Systems
Detecting Detecting and and Removing Removing Specular Specular Reflectance Reflectance Components Components Based Based on Image Linearization on Image Linearization Ryosuke Nakaoaa , Yuji Iwahoria,∗ , Yoshinori Adachia , Aili Wangb , M. K. Bhuyanc , Ryosuke Nakao , Yuji Iwahoria,∗, Yoshinori Adachida , Aili Wangb , M. K. Bhuyanc , Boonserm Kijsirikuld Boonserm Kijsirikul a Chubu University, Kasugai, 487-8501 Japan a Chubu University, Kasugai, 487-8501 Japan University of Science and Technology, Harbin 150080, China Science and Technology, Harbin781039, 150080,India China IndianUniversity Institute ofofTechnology Guwahati, Guwahati c Indian Institute of Technology Guwahati, 781039, India d Chulalongkorn University, BangkokGuwahati 20330, Thailand d Chulalongkorn University, Bangkok 20330, Thailand
b Harbin b Harbin c
Abstract Abstract Shape from Shading and Photometric Stereo are famous approaches to recover the 3D shape from image(s). These approaches can Shape Shading Photometric Stereoimage(s) are famous to recover the 3D shape from image(s).for These approaches can obtain from 3D shape fromand observed gray scale but approaches it is necessary to estimate reflectance parameters objects when some obtain 3D shape from observed gray scale image(s) but it is necessary to estimate reflectance parameters for objects when some specular reflectance components are observed. Specular reflectance is difficult to be handled as diffuse reflectance in general. It is specular components are observed. Specular reflectance is difficult to of be reflectance handled as function diffuse reflectance in model). general. This It is expected reflectance to remove specular reflectance component without assuming any kind (reflectance expected to remove specular reflectance component without assuming any kind of reflectance function (reflectance model). This paper proposes a method to remove specular reflection components using 4 observed images taken under 4 different light source paper proposes method toapproach remove specular reflection components observed images takenareunder 4 differentvia light source directions usingarelighting of diffused reflectance based onusing image4 linearization. Results demonstrated computer directions relighting approach of diffused reflectance based on image linearization. Results are demonstrated via computer simulationusing and experiments. simulation and experiments. c 2019 ⃝ 2019 The The Authors. Author(s). Published ElsevierB.V. B.V. © Published byby Elsevier c 2019an ⃝ The Author(s). Published bythe Elsevier B.V. This This is is an open open access access article article under under the CC CC BY-NC-ND BY-NC-ND license license (https://creativecommons.org/licenses/by-nc-nd/4.0/) (https://creativecommons.org/licenses/by-nc-nd/4.0/) This is an open access article under the CC BY-NC-ND Peer-review under responsibility of KES International. under responsibility of KES International. license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Peer-review under responsibility of KES International. Keywords: Photometric Stereo, Lambert Reflectance, Relighting, Specular Reflectance, Image Linearization Keywords: Photometric Stereo, Lambert Reflectance, Relighting, Specular Reflectance, Image Linearization
1. Introduction 1. Introduction 3D shape information has widely been used in the various fields such as robots, CG, VR, games, movies and medshape information has widely been used in the recovery various fields suchstudied as robots, VR, games, movies and medical3D diagnosis. In the meanwhile, methods for shape has been for CG, the purpose of obtaining accurate ical diagnosis. In the meanwhile, methods for shape recovery has been studied for the purpose of obtaining accurate 3D shape from the observed images such as Shape from Shading [1] and Photometric Stereo [2]. These approaches 3D the observed imagesgray suchscale as Shape frombut Shading [1] and Photometric Stereo [2]. parameters These approaches can shape obtainfrom 3D shape from observed image(s) it is necessary to estimate reflectance for obcan obtain 3D shape from observed gray scale image(s) but it is necessary to estimate reflectance parameters for objects when some specular reflectance components are observed. Ref.[3] proposed further improvement of photometric jects when some specular reflectance components are observed. Ref.[3] proposed further improvement of photometric stereo with many images for estimating reflectance properties and shape simultaneously. While Ref.[4] proposed to stereo with many images for estimating reflectance properties and shape simultaneously. While Ref.[4] proposed to ∗ ∗
Yuji Iwahori. Tel.: +81-568-51-9378 ; fax: +81-568-51-1540. Yuji Iwahori. Tel.: +81-568-51-9378 ; fax: +81-568-51-1540. E-mail address: [email protected] E-mail address: [email protected] c 2019 The Author(s). Published by Elsevier B.V. 1877-0509 ⃝ c 2019 1877-0509 ⃝ Thearticle Author(s). Published by Elsevierlicense B.V. (https://creativecommons.org/licenses/by-nc-nd/4.0/) This is an open access under the CC BY-NC-ND 1877-0509 © 2019 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Peer-review under responsibility of KES International. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Peer-review of KES International. Peer-reviewunder underresponsibility responsibility of KES International. 10.1016/j.procs.2019.09.328
2
Ryosuke Nakao et al. / Procedia Computer Science 159 (2019) 1576–1583 Nakao and Iwahori et al. / Procedia Computer Science 00 (2019) 000–000
1577
obtain 3D shape using neural network based photometric stereo and neural network rendering based on the obtained 3D shape itself. Ref.[5] proposed an extended method by neural network rendering of an object by rotating the object. Ref.[6] has been proposed for the estimation of Self-Localization and constructing surrounding 3D information using the geometrical feature maps. Relighting approach was proposed first in Shashua [7], then relighting approaches [8] [9] [10] have been proposed in addition to the shape recovery. It is quite difficult to generate the relighting image without knowing 3D shape and illuminating direction in general. It is not enough to combine together the existing images to generate the relighting image. Relighting is treated as a difficult problem with adjusting the color of the image, and it is treated as one of techniques in the virtual environment created by CG and the video contents to compose a scene naturally. It is shown that relighting process can be performed using linear combination of each image among three shading images in the case of Lambertian reflectance [7]. This approach is called image linearization. When the relighting process is applied to the real image, a portion of specular reflectance component or shadow which does not satisfy the Lambertian reflectance cannot be reproduced correctly when image linearization is applied based on the ref.[7]. In the case when such non-Lambertian reflectance component exists in the actual image, relighting process can not be correctly performed at the corresponding portion with specular reflectance component. This paper uses these properties using linear combination of images and proposes a new approach to detect and remove the specular reflectance components from the actual images. When four images are taken under four different light source directions, three images are selected among four images and the remained image is used to detect and to remove the specular reflectance region with the relighting image generated from three selected images. Although the ideal relighting process can be performed for Lambertian image, relighting image cannot be generated correctly if linearization process is applied to the specular reflection component. Comparing the generated image by relighting processing and the original input image which was not used for the relighting processing can remove the specular reflectance component and generate the corresponding Lambertian image by taking the difference between two images. Results are demonstrated via computer simulation and experiments. 2. Background Image fusion technology that combines an image of an object with the background is used for video production or movie etc., but when image fusion is applied and if the original image is just combined with another image, problem is that lighting conditions and taken time of images are not the same. In this case, image fusion fails and gives the discomfort. Research has been conducted to obtain images that have been synthesized naturally and there is a research called Relighting [7] which generates another image under another illuminating direction from the original input images taken under different illuminating directions. There are some methods to achieve the relighting, one of them performs Relighting by using linear combination of images in Lambertian reflectance model, and another approach achieves Relighting from one image for only the face of a person. In the relighting process, it is assumed that image linearization can be performed under the assumption of Lambertian reflectance, and three images for Lambertian reflectance model under three illuminating directions are used, and linear combination are used. Under this assumption, image intensity E4 generated from the observed three image intensities E1 to E3 satisfies a1 E1 + a2 E2 + a3 E3 = E4 as
(1)
Here no assumption is used for the surface gradient parameters (p, q) in Eq.(1). The left term of Eq.(1) is expanded a1 E 1 + a 2 E 2 + a 3 E 3 = a1 (s1x n x + s1y ny + s1z nz ) + a2 (s2x n x + s2y ny + s1z nz ) + a3 (s3x n x + s3y ny + s3z nz ) = (a1 s1x + a2 s2x + a3 s3x )n x + (a1 s1y + a2 s2y + a3 s3y )ny + (a1 s1z + a2 s2z + a3 s3z )nz
(2)
= s4x n x + s4y ny + s4z nz
(3)
Ryosuke Nakao et al. / Procedia Computer Science 159 (2019) 1576–1583 Nakao and Iwahori et al. / Procedia Computer Science 00 (2019) 000–000
1578
3
Eq.(4) is derived from the relations of Eq.(2) and Eq.(3). a1 s1x + a2 s2x + a3 s3x = s4x a1 s1y + a2 s2y + a3 s3y = s4y a1 s1z + a2 s2z + a3 s3z = s4z
(4)
Eq.(4) is rewritten in a matrix form as s1x s2x s3x a1 s4x (5) s1y s2y s3y a2 = s4y s4z s1z s2z s3z a3 s1x s2x s3x a1 s4x here, A = s1y s2y s3y ， x = a2 ， b = s4y Since the above equation can be represented in the form Ax = b, x s1z s2z s3z a3 s4z can be obtained using Eq.(6). x = A−1 b
Eq.(6) is represented as −1 a1 s1x s2x s3x s4x a2 = s1y s2y s3y s4y a3 s1z s2z s3z s4z
(6)
(7)
s4x a1 s When the fourth light source image (to be generated) 4y is given as known information, linear coefficients a2 s4z a3 a1 can be obtained. After a2 is obtained, E4 can be calculated using Eq.(1) at all pixels of the object. The process to a3 a1 obtain these linear coefficients a2 in the image linearization becomes the key in the Relighting process. a3 3. Proposed method
The proposed method removes the specular reflectance components using image linearization from 4 observed images. Procedure is shown as follows. Step 0. Step 1. Step 2. Step 3. Step 4.
Prepare four images of at the same viewpoint with known different light source directions. Identify the specular reflection region of the prepared image. Apply relighting processing using three among four images prepared in Step 1. Part of the diffuse reflectance is remained and the rest is interpolated from observed images. Generate whole diffuse reflectance images.
3.1. Relighting process for images with specular reflectance components Relighting process is performed using image linearization which was first proposed by ref.[7]. Since the linear combination is used, object image with the specular reflectance components can not be applied properly. Specular reflection region of the object is affected by the relighting process, and the region with the specular reflectance component becomes the region with the non-diffused (unnatural) representation. When four observed images are given, the combination of three images can produce the fourth image by relighting process, however there should be some difference between the fourth generated image and the fourth original observed image. This is based on the fact that
4
Ryosuke Nakao et al. / Procedia Computer Science 159 (2019) 1576–1583 Nakao and Iwahori et al. / Procedia Computer Science 00 (2019) 000–000
1579
relighting process is effective for the region with the diffused reflectance components. This method used this relighting process for the specular reflectance components which differs between the generated relighting image and the originally input image. Not only the specular reflectance region but also shadow region also gives the larger error between two images. 3.2. Identification of specular reflection region When Step 1 is applied, important factor is to find the region with the specular reflection component in generating the fourth image from three observed images. This processing becomes the key to identify the specular reflectance region for four input images. Removal of specular reflectance region is done by comparing the generated image and the corresponding input image. Taking difference by comparison finds the candidate region of specular reflectance components included in the input images. When the specular reflectance component is included in the image, image intensity E is extremely large in the specular reflectance component compared to the diffuse reflectance component. Specular reflectance region can be segmented by using the threshold if the strength of specular reflectance component is strong and specular peak is observed. Here, it is assumed that observed image has strong specular reflection. While if specular reflectance component is wide and dull, it is difficult to identify the specular reflectance region. So proposed method assumes the specular reflectance components which do not overlap each other in the observed images. 3.3. Seperation of specular reflectance region for generated image Since the image is generated by linear combination using three observed image including specular reflection, a strong peak occurs and specular reflectance region is selected. Specular reflectance region is seperated up to one surrounding pixel of specular reflectance region and the remained diffused region is extracted. Interpolation is applied to the removed specular reflectance region in the generated image with diffused reflectance. 3.4. Generation of diffuse reflectance image In the interpolation, the intensity of original input image is used to interpolate the corresponding point in the generated image. At this time, important factor is not to overlap the specular reflectance in the original images so that non-diffused reflectance region does not overlap in the generated image when diffused reflectance image is generated. 4. Computer Simulation and Experiments Experiment was performed to confirm that image under the fourth light source can be correctly generated by using three images under the first, second and the third light source among the four images taken under different light source directions but at the same viewpoint. First, experiments for a sphere image with Phong reflectance model are tested by computer simulation. 4.1. Phong reflectance model Synthesized images with Phong model with specular reflectance are used as the input images, where the width parameter n of the highlight is 100. Applying the proposed method to the synthesized sphere, it is confirmed that image under the fourth light source is generated by removing the specular reflectance component correctly. A total of four synthesized images were generated for each light source using the remained original images and the relighting process. The light source direction is represented using the zenith angle α and the azimuth angle β. Fig. 1 shows the input images taken under the corresponding α and β, respectively. Each image used different light source direction, where (a) α=30, β= 0, (b) α= 30, β=90, (c) α=30,β=180, (d) α=30, β=270, respectively. Here, input images (a), (b) and (c) are used to generate the corresponding relighting image of input image (d) under four light sources. The generated image and the corresponding intensity contour images of the generated image are shown in Fig.2. The specular reflectance region was further removed from the generated image and the result image was normalized between 0 and 1 (actually 255 as an 8-bit image).
1580
Ryosuke Nakao et al. / Procedia Computer Science 159 (2019) 1576–1583 Nakao and Iwahori et al. / Procedia Computer Science 00 (2019) 000–000
(a)Primary light source
(b)Second light source
(c)Third light source
(c)Fourth light source
5
Fig. 1. Four Phong Model Images under Different Light Source Directions
(e) Generated Image
(f) Corresponding Contour image
Fig. 2. Generated Image by Relighting Process
Table 1 shows the results of accuracy evaluation using Lambertian reflectance images and generated image under the same light source direction. Table 1. Luminance error
error
Minimum error 0.0008
Maximum error 0.4762
Average error 0.0059
From these results in Table 1, it is confirmed that the average error is within 0.01 and image under the fourth light source without specular reflectance is generated with high accuracy. The reason with the maximum error 0.4762 is considered to be that the result image is not generated correctly in the shadow part of each light source when the relighting process is applied.
6
Ryosuke Nakao et al. / Procedia Computer Science 159 (2019) 1576–1583 Nakao and Iwahori et al. / Procedia Computer Science 00 (2019) 000–000
(g) Generated Image with Interpolation
1581
(h) Normalized Image
Fig. 3. Generated Results
4.2. Real Sphere Image Subsequently, the similar experiment was performed on real images. Images taken are shown in Fig. 4. In addition, the angle of the light source direction of each image is (i) α = 19.50, β = 44.17, (j) α = 20.40, β = 114.41, (k) α = 34.07, β = 218.93, (l) α = 27.19, β = 320.25. These parameters of light source direction were estimated using the method of linear least squares under the assumption that target object is sphere and it is treated as known shape object.
(i) First light source
(k) Third light source
(j) Second light source
(l) Fourth light source
Fig. 4. Four real images with different light source directions
Generated normalized image under the fourth light source by removing specular reflectance region and the corresponding brightness contour image are shown in Fig.7. Table 2 shows the results of accuracy evaluation for the generated image using corresponding Lambertian images under the same light source direction. From the evaluation in Table 2, it is confirmed that the average error is within 0.06, and a generated image under the fourth light source with removal of specular reflectance is accurately performed although the real sphere object was used. As for the maximum error, it is considered that the relighting process is not applied in the shadow region of each light source similarly as the synthesized image.
1582
Ryosuke Nakao et al. / Procedia Computer Science 159 (2019) 1576–1583 Nakao and Iwahori et al. / Procedia Computer Science 00 (2019) 000–000
(n) Generated normalized image
7
(m) Contour image
Fig. 5. Generated Results
Table 2. Luminance error
error
Minimum error 0.0050
Maximum error 0.8767
Average error 0.0531
4.3. Real Object Image Subsequently, the similar experiment was performed on an actual different object image except a sphere. Input images are shown in Fig. 7. The angle of each light source direction is the same as that of real sphere image (o) α = 19.50, β = 44.17, (p) α = 20.40, β = 114.41, (q) α = 34.07, β = 218.93, (r) α = 27.19, β = 320.25.
(o) First light source
(q) Third light source
(p) Second light source
(r) Fourth light source
Fig. 6. Four real images with different light source directions
Generated normalized image with removal of the specular reflectance region under the fourth light source direction and the corresponding intensity contour image are shown in Fig.7. Table 3 shows the results of accuracy evaluation using the generated image and Lambertian images under the same light source direction. From Table 3, it is confirmed that the average error is within 0.09, and a generated image under the fourth light source with removal of specular reflectance region gives high accuracy. As for the maximum error, situation is the same that relighting processing has not been correctly in the shadow region of each light source. Another reason of
8
Ryosuke Nakao et al. / Procedia Computer Science 159 (2019) 1576–1583 Nakao and Iwahori et al. / Procedia Computer Science 00 (2019) 000–000
(s) Generated normalized image
1583
(t) Contour image
Fig. 7. Generated Results
Table 3. Intensity Error
error
Minimum error 0.0087
Maximum error 0.9348
Average error 0.0891
the worse evaluation in comparison with a sphere object is that shape is more complicated. It is shown that intensity changes large depending on the edge and error becomes large in the corresponding part of object. 5. Conclusion This paper proposed a new method to generate diffused reflectance images using relighting process and removing specular reflectance components using four input images. Relighting process is applied using three images taken under different light source directions and new image under the fourth light source is generated with removal of specular reflectance region and normalization. It is confirmed that generated image almost satisfies Lambertian reflectance model under the condition that there is no overlap specular region in input images. The effectiveness was confirmed via simulation and experiments. Extending this method to the object with nonuniform reflectance or overlapping problem of specular reflectance region are remained as the future subjects. ACKNOWLEDGMENT Iwahori’s research is supported by JSPS Grant-in-Aid for Scientific Research (C)(17K00252) and Chubu University Grant. References [1] B. K. P. Horn, M. J. Brooks ”Obtaining shape from shading information”, The MIT Press, pp. 123-171, 1989. [2] R. J. Woodham ”Photometric Method for Determining Surface Orientation from Multiple Images”, Optical Engineering, Vol.19, pp. 139-144, 1980. [3] A. Hertzmann, S. M. Seitz ”Shape and Materials by Example: A Photometric Stereo Approach”, IEEE CVPR 2003. Vol.1. pp.533-540, 2003. [4] H. Kawanaka, Y. Iwabori, R. J. Woodham, K. Funahashi, ”Color Photometric Stereo and Virtual Image Generation by Neural Network”, Transactions of the Institute of Electronics, Information and Communication Engineers, Vol.J89-D-II, No.2, pp.381-392, Feb. 2006. [5] Yi Ding, Yuji Iwahori, Tsuyoshi Nakamura, Lifeng He, Robert J.Woodham, Hidenori Itoh, ”Neural Network Implementation of Image Rendering via Self-Calibration”, Journal of Advanced Computational Intelligence and Intelligent Informatics (JACIII), Vol.14, No.4 pp. 344-352, Jan. 2010. [6] A. J. Davison, ”Real-time Simultaneous Localisation and Mapping with a Single Camera”, Vol.2, pp.1403-1410, IEEE ICCV 2003. [7] A. Shashua, ”Geometry and Photometry in 3D Visual Recognition”, Ph.D. Thesis, MIT, AITR-1401. Nov, 1992. [8] Z. Wen, Z. Liu, T. S. Huang, ”Face relighting with Rediance Environment Maps”, IEEE CVPR 2013, pp.158-165, 2003. [9] Y. Mukaigawa, H. Miyaki, S. Mihashi, T. Shakunaga, ”Photometric Image-Based Rendering for Image Generation in Arbitrary Illumination”, IEEE ICCV 2001, Vol.II, pp.652-659, Jul.2001. [10] T. Ikeda, F. Sorbiere, H. Saito, ”Real-time Relighting of Arbitrary Objects with Shape Change”, The 18th Annual Meeting of the Virtual Reality Society of Japan (VRSJ 2013), 32D-3, 2013.
Available online at www.sciencedirect.com
ScienceDirect
Procedia Computer Science 00 (2019) 000–000 Procedia Computer Science 00 (2019) 000–000 Procedia Computer Science 159 (2019) 1576–1583
www.elsevier.com/locate/procedia www.elsevier.com/locate/procedia
23rd International Conference on Knowledge-Based and Intelligent Information & Engineering 23rd International Conference on Knowledge-Based Systems and Intelligent Information & Engineering Systems
Detecting Detecting and and Removing Removing Specular Specular Reflectance Reflectance Components Components Based Based on Image Linearization on Image Linearization Ryosuke Nakaoaa , Yuji Iwahoria,∗ , Yoshinori Adachia , Aili Wangb , M. K. Bhuyanc , Ryosuke Nakao , Yuji Iwahoria,∗, Yoshinori Adachida , Aili Wangb , M. K. Bhuyanc , Boonserm Kijsirikuld Boonserm Kijsirikul a Chubu University, Kasugai, 487-8501 Japan a Chubu University, Kasugai, 487-8501 Japan University of Science and Technology, Harbin 150080, China Science and Technology, Harbin781039, 150080,India China IndianUniversity Institute ofofTechnology Guwahati, Guwahati c Indian Institute of Technology Guwahati, 781039, India d Chulalongkorn University, BangkokGuwahati 20330, Thailand d Chulalongkorn University, Bangkok 20330, Thailand
b Harbin b Harbin c
Abstract Abstract Shape from Shading and Photometric Stereo are famous approaches to recover the 3D shape from image(s). These approaches can Shape Shading Photometric Stereoimage(s) are famous to recover the 3D shape from image(s).for These approaches can obtain from 3D shape fromand observed gray scale but approaches it is necessary to estimate reflectance parameters objects when some obtain 3D shape from observed gray scale image(s) but it is necessary to estimate reflectance parameters for objects when some specular reflectance components are observed. Specular reflectance is difficult to be handled as diffuse reflectance in general. It is specular components are observed. Specular reflectance is difficult to of be reflectance handled as function diffuse reflectance in model). general. This It is expected reflectance to remove specular reflectance component without assuming any kind (reflectance expected to remove specular reflectance component without assuming any kind of reflectance function (reflectance model). This paper proposes a method to remove specular reflection components using 4 observed images taken under 4 different light source paper proposes method toapproach remove specular reflection components observed images takenareunder 4 differentvia light source directions usingarelighting of diffused reflectance based onusing image4 linearization. Results demonstrated computer directions relighting approach of diffused reflectance based on image linearization. Results are demonstrated via computer simulationusing and experiments. simulation and experiments. c 2019 ⃝ 2019 The The Authors. Author(s). Published ElsevierB.V. B.V. © Published byby Elsevier c 2019an ⃝ The Author(s). Published bythe Elsevier B.V. This This is is an open open access access article article under under the CC CC BY-NC-ND BY-NC-ND license license (https://creativecommons.org/licenses/by-nc-nd/4.0/) (https://creativecommons.org/licenses/by-nc-nd/4.0/) This is an open access article under the CC BY-NC-ND Peer-review under responsibility of KES International. under responsibility of KES International. license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Peer-review under responsibility of KES International. Keywords: Photometric Stereo, Lambert Reflectance, Relighting, Specular Reflectance, Image Linearization Keywords: Photometric Stereo, Lambert Reflectance, Relighting, Specular Reflectance, Image Linearization
1. Introduction 1. Introduction 3D shape information has widely been used in the various fields such as robots, CG, VR, games, movies and medshape information has widely been used in the recovery various fields suchstudied as robots, VR, games, movies and medical3D diagnosis. In the meanwhile, methods for shape has been for CG, the purpose of obtaining accurate ical diagnosis. In the meanwhile, methods for shape recovery has been studied for the purpose of obtaining accurate 3D shape from the observed images such as Shape from Shading [1] and Photometric Stereo [2]. These approaches 3D the observed imagesgray suchscale as Shape frombut Shading [1] and Photometric Stereo [2]. parameters These approaches can shape obtainfrom 3D shape from observed image(s) it is necessary to estimate reflectance for obcan obtain 3D shape from observed gray scale image(s) but it is necessary to estimate reflectance parameters for objects when some specular reflectance components are observed. Ref.[3] proposed further improvement of photometric jects when some specular reflectance components are observed. Ref.[3] proposed further improvement of photometric stereo with many images for estimating reflectance properties and shape simultaneously. While Ref.[4] proposed to stereo with many images for estimating reflectance properties and shape simultaneously. While Ref.[4] proposed to ∗ ∗
Yuji Iwahori. Tel.: +81-568-51-9378 ; fax: +81-568-51-1540. Yuji Iwahori. Tel.: +81-568-51-9378 ; fax: +81-568-51-1540. E-mail address: [email protected] E-mail address: [email protected] c 2019 The Author(s). Published by Elsevier B.V. 1877-0509 ⃝ c 2019 1877-0509 ⃝ Thearticle Author(s). Published by Elsevierlicense B.V. (https://creativecommons.org/licenses/by-nc-nd/4.0/) This is an open access under the CC BY-NC-ND 1877-0509 © 2019 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Peer-review under responsibility of KES International. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Peer-review of KES International. Peer-reviewunder underresponsibility responsibility of KES International. 10.1016/j.procs.2019.09.328
2
Ryosuke Nakao et al. / Procedia Computer Science 159 (2019) 1576–1583 Nakao and Iwahori et al. / Procedia Computer Science 00 (2019) 000–000
1577
obtain 3D shape using neural network based photometric stereo and neural network rendering based on the obtained 3D shape itself. Ref.[5] proposed an extended method by neural network rendering of an object by rotating the object. Ref.[6] has been proposed for the estimation of Self-Localization and constructing surrounding 3D information using the geometrical feature maps. Relighting approach was proposed first in Shashua [7], then relighting approaches [8] [9] [10] have been proposed in addition to the shape recovery. It is quite difficult to generate the relighting image without knowing 3D shape and illuminating direction in general. It is not enough to combine together the existing images to generate the relighting image. Relighting is treated as a difficult problem with adjusting the color of the image, and it is treated as one of techniques in the virtual environment created by CG and the video contents to compose a scene naturally. It is shown that relighting process can be performed using linear combination of each image among three shading images in the case of Lambertian reflectance [7]. This approach is called image linearization. When the relighting process is applied to the real image, a portion of specular reflectance component or shadow which does not satisfy the Lambertian reflectance cannot be reproduced correctly when image linearization is applied based on the ref.[7]. In the case when such non-Lambertian reflectance component exists in the actual image, relighting process can not be correctly performed at the corresponding portion with specular reflectance component. This paper uses these properties using linear combination of images and proposes a new approach to detect and remove the specular reflectance components from the actual images. When four images are taken under four different light source directions, three images are selected among four images and the remained image is used to detect and to remove the specular reflectance region with the relighting image generated from three selected images. Although the ideal relighting process can be performed for Lambertian image, relighting image cannot be generated correctly if linearization process is applied to the specular reflection component. Comparing the generated image by relighting processing and the original input image which was not used for the relighting processing can remove the specular reflectance component and generate the corresponding Lambertian image by taking the difference between two images. Results are demonstrated via computer simulation and experiments. 2. Background Image fusion technology that combines an image of an object with the background is used for video production or movie etc., but when image fusion is applied and if the original image is just combined with another image, problem is that lighting conditions and taken time of images are not the same. In this case, image fusion fails and gives the discomfort. Research has been conducted to obtain images that have been synthesized naturally and there is a research called Relighting [7] which generates another image under another illuminating direction from the original input images taken under different illuminating directions. There are some methods to achieve the relighting, one of them performs Relighting by using linear combination of images in Lambertian reflectance model, and another approach achieves Relighting from one image for only the face of a person. In the relighting process, it is assumed that image linearization can be performed under the assumption of Lambertian reflectance, and three images for Lambertian reflectance model under three illuminating directions are used, and linear combination are used. Under this assumption, image intensity E4 generated from the observed three image intensities E1 to E3 satisfies a1 E1 + a2 E2 + a3 E3 = E4 as
(1)
Here no assumption is used for the surface gradient parameters (p, q) in Eq.(1). The left term of Eq.(1) is expanded a1 E 1 + a 2 E 2 + a 3 E 3 = a1 (s1x n x + s1y ny + s1z nz ) + a2 (s2x n x + s2y ny + s1z nz ) + a3 (s3x n x + s3y ny + s3z nz ) = (a1 s1x + a2 s2x + a3 s3x )n x + (a1 s1y + a2 s2y + a3 s3y )ny + (a1 s1z + a2 s2z + a3 s3z )nz
(2)
= s4x n x + s4y ny + s4z nz
(3)
Ryosuke Nakao et al. / Procedia Computer Science 159 (2019) 1576–1583 Nakao and Iwahori et al. / Procedia Computer Science 00 (2019) 000–000
1578
3
Eq.(4) is derived from the relations of Eq.(2) and Eq.(3). a1 s1x + a2 s2x + a3 s3x = s4x a1 s1y + a2 s2y + a3 s3y = s4y a1 s1z + a2 s2z + a3 s3z = s4z
(4)
Eq.(4) is rewritten in a matrix form as s1x s2x s3x a1 s4x (5) s1y s2y s3y a2 = s4y s4z s1z s2z s3z a3 s1x s2x s3x a1 s4x here, A = s1y s2y s3y ， x = a2 ， b = s4y Since the above equation can be represented in the form Ax = b, x s1z s2z s3z a3 s4z can be obtained using Eq.(6). x = A−1 b
Eq.(6) is represented as −1 a1 s1x s2x s3x s4x a2 = s1y s2y s3y s4y a3 s1z s2z s3z s4z
(6)
(7)
s4x a1 s When the fourth light source image (to be generated) 4y is given as known information, linear coefficients a2 s4z a3 a1 can be obtained. After a2 is obtained, E4 can be calculated using Eq.(1) at all pixels of the object. The process to a3 a1 obtain these linear coefficients a2 in the image linearization becomes the key in the Relighting process. a3 3. Proposed method
The proposed method removes the specular reflectance components using image linearization from 4 observed images. Procedure is shown as follows. Step 0. Step 1. Step 2. Step 3. Step 4.
Prepare four images of at the same viewpoint with known different light source directions. Identify the specular reflection region of the prepared image. Apply relighting processing using three among four images prepared in Step 1. Part of the diffuse reflectance is remained and the rest is interpolated from observed images. Generate whole diffuse reflectance images.
3.1. Relighting process for images with specular reflectance components Relighting process is performed using image linearization which was first proposed by ref.[7]. Since the linear combination is used, object image with the specular reflectance components can not be applied properly. Specular reflection region of the object is affected by the relighting process, and the region with the specular reflectance component becomes the region with the non-diffused (unnatural) representation. When four observed images are given, the combination of three images can produce the fourth image by relighting process, however there should be some difference between the fourth generated image and the fourth original observed image. This is based on the fact that
4
Ryosuke Nakao et al. / Procedia Computer Science 159 (2019) 1576–1583 Nakao and Iwahori et al. / Procedia Computer Science 00 (2019) 000–000
1579
relighting process is effective for the region with the diffused reflectance components. This method used this relighting process for the specular reflectance components which differs between the generated relighting image and the originally input image. Not only the specular reflectance region but also shadow region also gives the larger error between two images. 3.2. Identification of specular reflection region When Step 1 is applied, important factor is to find the region with the specular reflection component in generating the fourth image from three observed images. This processing becomes the key to identify the specular reflectance region for four input images. Removal of specular reflectance region is done by comparing the generated image and the corresponding input image. Taking difference by comparison finds the candidate region of specular reflectance components included in the input images. When the specular reflectance component is included in the image, image intensity E is extremely large in the specular reflectance component compared to the diffuse reflectance component. Specular reflectance region can be segmented by using the threshold if the strength of specular reflectance component is strong and specular peak is observed. Here, it is assumed that observed image has strong specular reflection. While if specular reflectance component is wide and dull, it is difficult to identify the specular reflectance region. So proposed method assumes the specular reflectance components which do not overlap each other in the observed images. 3.3. Seperation of specular reflectance region for generated image Since the image is generated by linear combination using three observed image including specular reflection, a strong peak occurs and specular reflectance region is selected. Specular reflectance region is seperated up to one surrounding pixel of specular reflectance region and the remained diffused region is extracted. Interpolation is applied to the removed specular reflectance region in the generated image with diffused reflectance. 3.4. Generation of diffuse reflectance image In the interpolation, the intensity of original input image is used to interpolate the corresponding point in the generated image. At this time, important factor is not to overlap the specular reflectance in the original images so that non-diffused reflectance region does not overlap in the generated image when diffused reflectance image is generated. 4. Computer Simulation and Experiments Experiment was performed to confirm that image under the fourth light source can be correctly generated by using three images under the first, second and the third light source among the four images taken under different light source directions but at the same viewpoint. First, experiments for a sphere image with Phong reflectance model are tested by computer simulation. 4.1. Phong reflectance model Synthesized images with Phong model with specular reflectance are used as the input images, where the width parameter n of the highlight is 100. Applying the proposed method to the synthesized sphere, it is confirmed that image under the fourth light source is generated by removing the specular reflectance component correctly. A total of four synthesized images were generated for each light source using the remained original images and the relighting process. The light source direction is represented using the zenith angle α and the azimuth angle β. Fig. 1 shows the input images taken under the corresponding α and β, respectively. Each image used different light source direction, where (a) α=30, β= 0, (b) α= 30, β=90, (c) α=30,β=180, (d) α=30, β=270, respectively. Here, input images (a), (b) and (c) are used to generate the corresponding relighting image of input image (d) under four light sources. The generated image and the corresponding intensity contour images of the generated image are shown in Fig.2. The specular reflectance region was further removed from the generated image and the result image was normalized between 0 and 1 (actually 255 as an 8-bit image).
1580
Ryosuke Nakao et al. / Procedia Computer Science 159 (2019) 1576–1583 Nakao and Iwahori et al. / Procedia Computer Science 00 (2019) 000–000
(a)Primary light source
(b)Second light source
(c)Third light source
(c)Fourth light source
5
Fig. 1. Four Phong Model Images under Different Light Source Directions
(e) Generated Image
(f) Corresponding Contour image
Fig. 2. Generated Image by Relighting Process
Table 1 shows the results of accuracy evaluation using Lambertian reflectance images and generated image under the same light source direction. Table 1. Luminance error
error
Minimum error 0.0008
Maximum error 0.4762
Average error 0.0059
From these results in Table 1, it is confirmed that the average error is within 0.01 and image under the fourth light source without specular reflectance is generated with high accuracy. The reason with the maximum error 0.4762 is considered to be that the result image is not generated correctly in the shadow part of each light source when the relighting process is applied.
6
Ryosuke Nakao et al. / Procedia Computer Science 159 (2019) 1576–1583 Nakao and Iwahori et al. / Procedia Computer Science 00 (2019) 000–000
(g) Generated Image with Interpolation
1581
(h) Normalized Image
Fig. 3. Generated Results
4.2. Real Sphere Image Subsequently, the similar experiment was performed on real images. Images taken are shown in Fig. 4. In addition, the angle of the light source direction of each image is (i) α = 19.50, β = 44.17, (j) α = 20.40, β = 114.41, (k) α = 34.07, β = 218.93, (l) α = 27.19, β = 320.25. These parameters of light source direction were estimated using the method of linear least squares under the assumption that target object is sphere and it is treated as known shape object.
(i) First light source
(k) Third light source
(j) Second light source
(l) Fourth light source
Fig. 4. Four real images with different light source directions
Generated normalized image under the fourth light source by removing specular reflectance region and the corresponding brightness contour image are shown in Fig.7. Table 2 shows the results of accuracy evaluation for the generated image using corresponding Lambertian images under the same light source direction. From the evaluation in Table 2, it is confirmed that the average error is within 0.06, and a generated image under the fourth light source with removal of specular reflectance is accurately performed although the real sphere object was used. As for the maximum error, it is considered that the relighting process is not applied in the shadow region of each light source similarly as the synthesized image.
1582
Ryosuke Nakao et al. / Procedia Computer Science 159 (2019) 1576–1583 Nakao and Iwahori et al. / Procedia Computer Science 00 (2019) 000–000
(n) Generated normalized image
7
(m) Contour image
Fig. 5. Generated Results
Table 2. Luminance error
error
Minimum error 0.0050
Maximum error 0.8767
Average error 0.0531
4.3. Real Object Image Subsequently, the similar experiment was performed on an actual different object image except a sphere. Input images are shown in Fig. 7. The angle of each light source direction is the same as that of real sphere image (o) α = 19.50, β = 44.17, (p) α = 20.40, β = 114.41, (q) α = 34.07, β = 218.93, (r) α = 27.19, β = 320.25.
(o) First light source
(q) Third light source
(p) Second light source
(r) Fourth light source
Fig. 6. Four real images with different light source directions
Generated normalized image with removal of the specular reflectance region under the fourth light source direction and the corresponding intensity contour image are shown in Fig.7. Table 3 shows the results of accuracy evaluation using the generated image and Lambertian images under the same light source direction. From Table 3, it is confirmed that the average error is within 0.09, and a generated image under the fourth light source with removal of specular reflectance region gives high accuracy. As for the maximum error, situation is the same that relighting processing has not been correctly in the shadow region of each light source. Another reason of
8
Ryosuke Nakao et al. / Procedia Computer Science 159 (2019) 1576–1583 Nakao and Iwahori et al. / Procedia Computer Science 00 (2019) 000–000
(s) Generated normalized image
1583
(t) Contour image
Fig. 7. Generated Results
Table 3. Intensity Error
error
Minimum error 0.0087
Maximum error 0.9348
Average error 0.0891
the worse evaluation in comparison with a sphere object is that shape is more complicated. It is shown that intensity changes large depending on the edge and error becomes large in the corresponding part of object. 5. Conclusion This paper proposed a new method to generate diffused reflectance images using relighting process and removing specular reflectance components using four input images. Relighting process is applied using three images taken under different light source directions and new image under the fourth light source is generated with removal of specular reflectance region and normalization. It is confirmed that generated image almost satisfies Lambertian reflectance model under the condition that there is no overlap specular region in input images. The effectiveness was confirmed via simulation and experiments. Extending this method to the object with nonuniform reflectance or overlapping problem of specular reflectance region are remained as the future subjects. ACKNOWLEDGMENT Iwahori’s research is supported by JSPS Grant-in-Aid for Scientific Research (C)(17K00252) and Chubu University Grant. References [1] B. K. P. Horn, M. J. Brooks ”Obtaining shape from shading information”, The MIT Press, pp. 123-171, 1989. [2] R. J. Woodham ”Photometric Method for Determining Surface Orientation from Multiple Images”, Optical Engineering, Vol.19, pp. 139-144, 1980. [3] A. Hertzmann, S. M. Seitz ”Shape and Materials by Example: A Photometric Stereo Approach”, IEEE CVPR 2003. Vol.1. pp.533-540, 2003. [4] H. Kawanaka, Y. Iwabori, R. J. Woodham, K. Funahashi, ”Color Photometric Stereo and Virtual Image Generation by Neural Network”, Transactions of the Institute of Electronics, Information and Communication Engineers, Vol.J89-D-II, No.2, pp.381-392, Feb. 2006. [5] Yi Ding, Yuji Iwahori, Tsuyoshi Nakamura, Lifeng He, Robert J.Woodham, Hidenori Itoh, ”Neural Network Implementation of Image Rendering via Self-Calibration”, Journal of Advanced Computational Intelligence and Intelligent Informatics (JACIII), Vol.14, No.4 pp. 344-352, Jan. 2010. [6] A. J. Davison, ”Real-time Simultaneous Localisation and Mapping with a Single Camera”, Vol.2, pp.1403-1410, IEEE ICCV 2003. [7] A. Shashua, ”Geometry and Photometry in 3D Visual Recognition”, Ph.D. Thesis, MIT, AITR-1401. Nov, 1992. [8] Z. Wen, Z. Liu, T. S. Huang, ”Face relighting with Rediance Environment Maps”, IEEE CVPR 2013, pp.158-165, 2003. [9] Y. Mukaigawa, H. Miyaki, S. Mihashi, T. Shakunaga, ”Photometric Image-Based Rendering for Image Generation in Arbitrary Illumination”, IEEE ICCV 2001, Vol.II, pp.652-659, Jul.2001. [10] T. Ikeda, F. Sorbiere, H. Saito, ”Real-time Relighting of Arbitrary Objects with Shape Change”, The 18th Annual Meeting of the Virtual Reality Society of Japan (VRSJ 2013), 32D-3, 2013.