- Email: [email protected]
Photometric Stereo on Large Flattened Surfaces Using Image Stitching
Available online at www.sciencedirect.com Available online at www.sciencedirect.com
Available online at www.sciencedirect.com
ScienceDirect
Procedia Computer Science 00 (2019) 000–000 Procedia Computer Science (2019) 000–000 Procedia Computer Science 14700 (2019) 247–253
www.elsevier.com/locate/procedia www.elsevier.com/locate/procedia
2018 International Conference on Identification, Information and Knowledge 2018 International Conference on Identification, Information and Knowledge in the Internet of Things, IIKI 2018 in the Internet of Things, IIKI 2018
Photometric Photometric Stereo Stereo on on Large Large Flattened Flattened Surfaces Surfaces Using Using Image Image Stitching Stitching Tingting Zhuaa , Lin Qia,∗ , Hao Fana , Junyu Donga Tingting Zhu , Lin Qia,∗, Hao Fana , Junyu Donga
a Department a Department
of Information Science and Engineering Ocean University of China Qingdao, China of Information Science and Engineering Ocean University of China Qingdao, China
Abstract Abstract In this paper, we proposed a method to enlarge the recovered surface normal from photometric stereo method using image stitching. In paper, we proposed a method to enlarge recovered normal from photometric stereo method using image stitching. Wethis used the laser triangulation to provide the the actual height surface as the reference to reduce the low-frequency noise from photometric We usedThe theGrid-based laser triangulation to provide(GMS) the actual height was as the reference to the reduce the low-frequency noise from photometric stereo. Motion Statistics algorithm used to obtain Homography matrix. We employed and tested stereo. The Grid-based Motion Statistics (GMS) to schemes obtain thewere Homography matrix. Weand employed and tested two algorithms for stitching: graph cut and imagealgorithm quilting. was Twoused splice used: post-stitching pre-stitching. The two algorithms forand stitching: graphofcut imagewere quilting. Two splice schemes were used: post-stitching and built pre-stitching. proposed method comparison twoand schemes tested by experiments using captured surfaces with our devices. The proposed method and comparison of two schemes were tested by experiments using captured surfaces with our built devices. c 2019 2019 The The Authors. Authors. Published Published by by Elsevier Elsevier B.V. B.V. © c 2019 The Authors. Published by Elsevier B.V. This is This is an an open open access access article article under under the the CC CC BY-NC-ND BY-NC-ND license license (https://creativecommons.org/licenses/by-nc-nd/4.0/) (https://creativecommons.org/licenses/by-nc-nd/4.0/) This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Peer-review under responsibility of the scientific committee the 2018 International Conference on Identification, Information under responsibility of the scientific committee ofof the 2018 International Conference on Identification, Information and Peer-review responsibility the scientific committee of the 2018 International Conference on Identification, Information and Knowledge the Internet of of Things. Knowledge inunder theinInternet of Things. and Knowledge in the Internet of Things. Keywords: Photometric stereo; Laser triangulation; Graph cut; Image quilting; Texture synthesis. Keywords: Photometric stereo; Laser triangulation; Graph cut; Image quilting; Texture synthesis.
1. Introduction 1. Introduction 3D reconstruction is an important and challenging problem in computer vision. There are existing works include 3D vision, reconstruction an important and(SFM), challenging problem in computer There are stereo existing include stereo structureis from motion [1] photometric method and sovision. on. Photometric is works a typical phostereo vision, structure from motion [1] (SFM), photometric method and so on. Photometric stereo is a typical tometric method that uses images of a scene taken from the same viewpoint and different illumination directionspho[2]. tometric method thatrecovers uses images of a scene taken from the pixel same of viewpoint different Photometric stereo gradients (or normal) at each capturedand images, andillumination the surface directions height can[2]. be Photometric stereo recovers field gradients (or normal) at each pixel of captured images, and the surface height can be integrated from the gradient [3][4][5]. integrated the gradient [3][4][5]. Classis from photometric stereofield requires uniform and collimate illumination [6]. However, the light sources employed Classis photometric stereo requires uniform andnonuniform. collimate illumination the lightby sources employed in practical photometric stereo setups are usually The images[6]. areHowever, usually captured near point light in practical photometric stereo setups are usually nonuniform. The images are usually captured by near point light sources equipment, which will introduce errors caused by the nonuniform illumination. We used two methods to sources equipment, which will introduce errors caused by the nonuniform illumination. We used two methods to correct the surface shape bias based on our previous work [7]. The first method is by fitting low-frequency error correct the surface shape bias based on our previous work [7]. The first method is by fitting low-frequency error without any reference points. The second method employs reference points with known 3D information, such as those without any reference points. The second method employs reference points with known 3D information, such as those ∗ ∗
Corresponding author. Tel.: +86-532-66781729. Corresponding Tel.: +86-532-66781729. E-mail address:author. [email protected] E-mail address: [email protected] c 2019 The Authors. Published by Elsevier B.V. 1877-0509 1877-0509 © 2019 The Authors. Published by B.V. c 2019 1877-0509 Thearticle Authors. Published by Elsevier Elsevier B.V. This isisan under the CC licenselicense (https://creativecommons.org/licenses/by-nc-nd/4.0/) This anopen openaccess access article under the BY-NC-ND CC BY-NC-ND (https://creativecommons.org/licenses/by-nc-nd/4.0/) This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Peer-review under of the committee of the 2018 Conference on Identification, Information and Knowledge in Peer-review underresponsibility responsibility of scientific the scientific committee of theInternational 2018 International Conference on Identification, Information and Peer-review under responsibility of the scientific committee of the 2018 International Conference on Identification, Information and Knowledge in the Internet of Knowledge inThings. the Internet of Things. the Internet of Things. 10.1016/j.procs.2019.01.245
Tingting Zhu et al. / Procedia Computer Science 147 (2019) 247–253 Tinting Zhu / Procedia Computer Science 00 (2019) 000–000
248 2
from laser triangulation or binocular stereo. The experiments prove that the result obtained after calibration is more accurate than traditional methods under near-point lighting. Photometric stereo can recover highly detailed surface geometry, but this also limits the reconstruction field in the view of the camera. Therefore, in this paper, we proposed a stitching method to reconstruct large-scale surface. Different from multiview photometric stereo, We employ texture synthesis technology to do the splice, which is image quilting and graph cut [9][10]. Texture synthesis is a fundamental problem in computer graphics. Given piece of small sample texture, the objective is to generate larger, visually similar texture image. In this paper, we proposed two methods for splicing. The first is to stitch the normal maps obtained by photometric stereo from different view point, and then the corresponding height can be integrated from the normal map. Another is to stitch the original images under the same illumination angle, and then do the photometric stereo to mosaic images to get the large normal map. The relative movement of the target object and the camera is not strictly translation, so we calibrate the small rotation by GMS algorithm [11]. The main contributions of this paper can be summarized as follows: • The method of combining texture synthesis with photometric stereo is proposed, which implements fine reconstruction of large-scale surfaces. • Two methods of stitching using texture synthesis are tested. One is to stitch the normal maps from photometric stereo. The other is to stitch the original images and then feed to photometric stereo to calculate normal map. • The low-frequency deformation caused by the nonuniform illumination is corrected for better synthesis results. 2. Related work 2.1. Photometric Stereo In recent years, many researchers studied how to convert photometric stereo from single view to multi-view. Based on traditional photometric stereo, Jong woo Lim [14] introduced an iterative algorithm for shape reconstruction from multiple images of a moving object illuminated at a distance. However, they need to manually mark feature points to recover the object surface by detecting and tracking the feature points in the image sequence. G.Vogiatzis and C.Hernanez [15][16][17] presented an algorithm which uses silhouettes of object. Starting from the silhouettes, the algorithm recovers camera motion and constructs the objects visual hull which is then used to recover the illumination and initialize a multi-view photometric stereo scheme to obtain a closed surface reconstruction. Although it can achieve high accuracy 3D reconstruction even for textureless shiny objects, but it is not applicable to large-scale surfaces such as the seabed. Tomoai Higot [18] attached a point light source to a hand-held camera to add a photometric constraint to the multi-view stereo (MVS) problem, and then simultaneously solve for shape, surface normal, and reflectance. Jeasik Park and Yuwing Tai [19] proposed a new photometric stereo based mesh refinement technique that can efficiently reconstruct meshes with fine geometric details by directly estimating a displacement texture map in the 2D parameter domain. SFM is used to calibrate the cameras and MVS is used to recover a coarse mesh. They required lots of images from different viewpoints and in our experiment, such conditions are almost impossible to achieve. Boxin Shi [20] focused on Internet images of large outdoor sculptures which are difficult to capture by Lidar or flying drone due to their size or security reason. These 3D reconstruction methods can achieve fine result on specific occasions, and in this paper, we studied the problem of recovering large-scale surface geometry using texture synthesis algorithms. 2.2. Texture Synthesis In this paper, we use two texture synthesis algorithms, i.e. the image quilting and the graph cut. Efros [21][22] proposed a texture synthesis algorithm named image quilting, which completes texture synthesis by calculating the cumulative error and minimum error path of the input texture block and the synthesized texture block in the overlapped region. Its main task is to deal with the boundary of the texture block and calculate the Euclidean distance of the RGB space in the overlapping region, thus finding the best seam and further finding the minimum error path. The algorithm is fast, but there may be discontinuity on 3D surface textures. Kwatra [23] proposed a texture synthesis algorithm
Tingting Zhu et al. / Procedia Computer Science 147 (2019) 247–253 Tinting Zhu / Procedia Computer Science 00 (2019) 000–000
249 3
Fig. 1. (a) Points with known 3D information from laser triangulation; (b) The original gradients and height produced by classic photometric stereo and the corrected results after calibration.
based on graph cut. Its principle is the maximum flow minimum cut problem. The maximum flow minimum cut theorem is proposed by Ford and Fulkerson [24] in 1956, that is, in any network, the maximum flow value is equal to the minimum cut capacity. Firstly we need to build a graph, and after solving the capacity between adjacent nodes in the graph, using the Ford-Fulkerson labeling method, the maximum flow minimum cut C can be obtained. 3. ALGORITHM 3.1. Gradient Deviation in Classic Photometric Stereo Classic photometric stereo required uniform and collimated illumination [25], but the light sources employed in this paper are near point light sources, so the reconstruction errors [26][12][13]] caused by the nonuniform lighting cannot be ignored. We plotted slate surface in Fig. 1(b), from which it is obvious that the gradients show linear deviation, and the surface shape is contaminated by quadratic low-frequency noise. The second line shows the corrected gradients and surface shape after calibration. Gradient Self-Correction: From the previous experimental observations, we found that a binary quadratic function f (x, y) = Ax2 + By2 + Cxy + Dx + Ey + F approximates the height deviation well. The gradient deviation and the quadratic center position will be pe (x, y) = 2Ax + Cy + D, qe (x, y) = 2By + Cx + E (xc , yc ) = (
CE − 2BD CD − 2AE , ) 4AB − c2 4AB − C 2
(1) (2)
where (pe , qe ) denotes the gradient deviation and (p0 , q0 ) denotes the original gradient obtained by classical photometric stereo. The corrected gradient (pc , qc ) will be (pc , qc ) = (p0 − pe , q0 − qe )
(3)
Optimized Gradient Self-Correction: If some surface points with known 3-D information are available, as shown in Fig.1(a), these points can be used as constraints to improve the reconstruction accuracy. The gradient deviation will be corrected by minimizing the difference between the height and the known height of these points. The 3-D information of some surface points can be obtained by methods such as laser triangulation. The optimization equation is = min(|Hi − H ps (pc , qc )|)
(4)
Where H1 denotes the correct height from laser triangulation, H ps denotes the height produced by photometric stereo method in real size. P − c(x, y) = p0 (x, y) − pe (x, y) = p0 (x, y) − (2Ax0 + Cy0 + D), qc (x, y) = q0 (x, y) − qe (x, y) = q0 (x, y) − (2By0 + Cx0 + E).The final surface height H ps is obtained through the best parameters. 3.2. Grid-based Motion Statistics Image stitching requires two images with the same size and has overlapping area. The object has horizontal displacement in the ideal situation, however the object motion in our experiment will cause little deviation, and so the seam will obvious. Therefore, we must calibrate the deviation before stitching the image.
Tingting Zhu et al. / Procedia Computer Science 147 (2019) 247–253 Tinting Zhu / Procedia Computer Science 00 (2019) 000–000
250 4
Fig. 2. left: Cloth and slate1; right: The experimental setup.
Grid-based Motion Statistics provides a real-time and ultra-robust correspondence system. It is a simple means of encapsulating motion smoothness as the statistical likelihood of a certain number of matches in a region. GMS enables translation of high match numbers into high match quality and develop an efficient Grid-based score estimator that can be incorporated into a real-time feature matcher. In this paper, we use GMS algorithm to obtain the relative position of camera from two view points, as shown in the left of Fig. 2. 3.3. Proposed Methods In this paper, we proposed two methods to stitch the images. The first method is obtained the normal map produced by classical photometric stereo, and then calibrated the height deformation caused by near-range light sources. Next, stitch the calibrated image. The traditional texture synthesis methods image quilting and graph cut are used to image stitching. Another method is to splice the original images with different angles of viewpoints but with the same illumination first, then obtain the normal map, next calibrate the deformation use the methods mentioned above. 1)Stitching After Photometric Stereo We use photometric metric to obtain the normal in two angels of view, and then stitching the normal map. Our experiment objects are cloth, paperboard, slate1 and slate2, which photograph in the laboratory. Each object is shot from two angles, the left side and the right side, there are overlapping parts in the middle. The normal map of four objects from different angels of view are obtained by photometric stereo and calibrated by the methods mentioned above. After the normal map is obtained, the next step is stitching the image. In this paper, image quilting and graph cut algorithm are used to stitching. Image Quilting Stitching: Input the size of texture1, texture2 and overlap respectively, and then calculate the size of the output image. Next the spliced image is copied to the specified position of the output image, and the middle overlap area can be regarded as texture2 covered on texture1. Calculated the Euclidean distance of the RGB space in the overlapping region, and reverse tracing to find a minimum error line which we required minimum path. Graph Cut Stitching: The first half of the algorithm is consistent with the image quilting algorithm. When finding the optimal seam, graph cut defines an energy function E= e(p, q) (5) p,q∈N
where N is the neighborhood of a point, (p, q) denoted the adjacent two points, and its corresponding points in image B is p , q . e(p, q) is a weight function. Using the weights of adjacent points as the capacity to build a network graph, to get a path of minimum energy, that is, the best seam we need. And then keep the content of the image A on the left side and preserve the content of the image B on the right side, so as to complete the splicing. 2)Photometric Stereo After Stitching Firstly, we stitch the original images through image quilting and graph cut algorithm. The experiment objects are cloth, paperboard, slate1 and the illumination angles are 0, 60, 120, 180, 240, 300 degree.
Tingting Zhu et al. / Procedia Computer Science 147 (2019) 247–253 Tinting Zhu / Procedia Computer Science 00 (2019) 000–000
251 5
Fig. 3. Pictures from left to right are the result of graph cut stitching and image quilting stitching respectively.
Fig. 4. Pictures from left to right are the result of graph cut stitching and image quilting stitching respectively.
After stitching the original images, we use photometric stereo to obtain the normal for synthetic image. Then the normal map is calibrated use the methods metioned above. 4. EXPERIMENTS 4.1. Experimental Setup Our experimental setup is shown in the right of Fig. 2, including the photometric stereo system and the laser triangulation system. A camera (IDS UI-358xCP-C) and six light sources constituted the photometric stereo system. The camera was placed at the center of the system, surrounded by six light-emitting diode (LED) point light sources. The light sources with adjustable lighting direction were placed around a rotating circular orbit. The line-structured laser was placed alongside the camera. The laser and the camera constituted a line laser 3-D imaging system. The objects were placed below the camera. In our experiments, the distance between the camera and objects along the optical axis was about 574 mm. The slant angle of each light source was set to 45 degree, and the distance between the light sources and objects along the optical axis was about 375 mm, which is similar to the radius of the circular orbit. The cameras resolution was 2592 × 2048 pixels. In addition, the camera calibration is carried out by the method of Zhang [8]. The images were captured in a dark room, and the LED lights are the only light sources. Photometric stereo was computed under six light directions, which had the same interval along the circular orbit. The object is moved at a fixed distance, and then take photos in the same way, there are overlaps between two images so that they can be stitching. 4.2. Experiment Result In this paper, we have two methods to stitch. The first method, obtained the normal map using classical photometric stereo first, and then calibrated the height deformation caused by near-range light sources. Next stitch the calibrated normal map. Another method is to splice the original images with different angles of view but with the same illumination first, then obtain the normal map, next calibrate the deformation. Stitching after Photometric Stereo: We use photometric metric to obtain the normal maps in two angles of view, and then stitching the normal map. Image quilting and graph cut algorithms are used to splice, the algorithm contrast is shown as Fig. 3. The first column indicated the normal map after stitching. The second column indicated heights produced by photometric stereo. The third column is the 3D surface of the object and the last line is the surface detail
Tingting Zhu et al. / Procedia Computer Science 147 (2019) 247–253 Tinting Zhu / Procedia Computer Science 00 (2019) 000–000
252 6
Fig. 5. Left: the left side of ball and the right side of ball. Right: the result of original image stitching.
Fig. 6. Left: stitching after photometric stereo. Right: photometric stereo after stitching.
map. In experiment, the overlap area of cloth and paperboard was 1944, and the overlap area of slate1 and slate2 was 1280. Photometric Stereo after stitching: After stitching the original images, we use photometric metric to obtain the normal maps for synthetic images, the result as shown in Fig. 4, they express the results of photometric stereo after splicing with graph cut and image quilting algorithm respectively. 4.3. Quantitative Evaluation In order to quantitatively evaluate the stitching result, we tested the standard sphere. The original images are shown in Fig. 5. Table 1. Method Comparison. Error(degree)
post-stitching
pre-stitching
PS Graph cut Image quilting
12.2382 12.6295 12.9145
12.2382 12.1205 12.2052
Stitching after Photometric Stereo: Using two methods to stitch normal map, the results are shown in the left of Fig.6. The top line and the bottom line is the result of graph cut and image quilting algorithm respectively. Each column from left to right is the ground truth, normal, gradient, height and 3D surface produced by photometric stereo, and the last column is an error graph between the true normal and the obtained normal. From the error analysis (TABLE I)on the ball, it can be seen that the precision of our methods is below 1 degree. Photometric Stereo after Stitching: The other method is stitched original images first, and then implement photometric stereo. The results are shown in the right of Fig. 6. From the error analysis (TABLE I) on the ball, it can be seen that the precision of our methods is below 1 degree. Compared two methods, we found that photometric stereo after stitching is slightly better than stitching after photometric stereo, and in these two methods, the result of graph cut is a little better than image stitching. 5. CONCLUSION This paper extends the photometric stereo from the fixed angle to multi-view angles. Two methods are adopted. First, the original images we collected are spliced according to the corresponding illumination angle, and then the photometric stereo is used to obtain the normal direction from two angles. The other method is to get the normal map from each view first and then merge the normal map. The first method uses self-calibration to calibrate the height error caused by uneven illumination. The second method uses laser triangulation to get the true height of the surface of the
Tingting Zhu et al. / Procedia Computer Science 147 (2019) 247–253 Tinting Zhu / Procedia Computer Science 00 (2019) 000–000
253 7
object, and uses it as a reference to calibrate the error. Although the method can achieve multi-view photometric stereo, there are still many defects. For example, the method needs to stop to capture images at every angel, because different illumination angles needed for photometric stereo. For splicing results, there is no accurate verification method to determine whether the splicing is good or bad. Therefore, in the future research, we will focus on how to speed up the reconstruction process to achieve real-time results. Acknowledgements This work was supported by the National Natural Science Foundation of China (NSFC) (No.61501417) and the Ph. D. Program Foundation of Ministry of Education of China (No.20120132110018). References [1] S. Agarwal et al. (2011) “Building Rome in a day.” Commun. ACM 54 (10): 105112. [2] S. Agarwal et al. Woodham R J. (1980)“Photometric method for determining surface orientation from multiple images[J].” Optical engineering 19 (1): 191139-191139. [3] R. T. Frankot and R. Chellappa. (1988)“A method for enforcing integrability in shape from shading algorithms.” IEEE Trans. Pattern Anal. Mach. Intel. 10 (4): 439-451. [4] A. Agrawal, R. Raskar, and R. Chellapp. (2006)“What is the range of surface reconstructions from a gradient field?” in European Conf. on Computer Vision pp.191139-191139. [5] M. Harker and P. OLeary. (2015)“Regularized reconstruction of a surface from its measured gradient field.” J. Math. Imaging Vision 51 (1): 46-70. [6] N. Kolagani, J. S. Fox, and D. R. Blidberg. (1992)“Photometric stereo using point light sources.” in Proc. 1992 IEEE Int. Conf. on Robotics and Automation pp.1759-1764. [7] Fan, H. Qi, L. Wang, N. Dong, J. Chen, Y. Yu, H. (2017)“Deviation correction method for close-range photometric stereo with nonuniform illumination.” Opt. Eng 56 (1): 1-13. [8] Z. Zhang. (2000)“A flexible new technique for camera calibration.” IEEE Trans. Pattern Anal. Mach. Intell 22 (11): 1330-1334. [9] Kwatra V, Schoodl A, Essa I, et al. (2003)“Graph cut Textures: image and video synthesis using graph cuts[J].” ACM Transactions on Graphics 22 (3): 277-286. [10] Boykov. Y, Veksler. O, and Zabih. R. (1999)“Fast approximate energy minimization via graph cuts.” In International Conference on Computer Vision pp.377-384. [11] Bian, J.W., Lin, W.Y., Matsushita, Y., et al. (2017)“GMS: Grid-Based Motion Statistics for Fast, Ultra-Robust Feature Correspondence.” IEEE Conference on Computer Vision and Pattern Recognition pp.2828-2837. [12] I. Horovitz and N. Kiryati. (2004)“Depth from gradient fields and control points: bias correction in photometric stereo.” Image Vision Compute 22 (9): 681-694. [13] D. Cho et al. (2016)“Photometric stereo under non-uniform light intensities and exposures.” in European Conf. on Computer Vision pp.170-186 [14] J. Lim, J. Ho, M. Yang, and D. Kriegman. (2005)“Passive Photometric stereo from Motion.” in Proc. IEEE International Conference on Computer Vision pp.383-391 [15] C. Hernandez, G. Vogiatzis, and R. Cipolla. (2008)“multiview photometric stereo.” IEEE Trans. Pattern Anal. Mach. Intell pp.548-554 [16] C. Hernandez, F. Schmitt, and R. Cipoll. (2007)“Silhouette coherence for camera calibration under circular motion.” IEEE Trans. Pattern Anal. Mach. Intell 29 (2): 343-349. [17] G. Vogiatzis, C. Hernandez, and R. Cipolla. (2006)“Reconstruction in the round using photometric normals and silhouettes.” in Proc. IEEE Conf. on Computer Vision and Pattern Recognition (2): 343-349. [18] T. Higo et al. (2009)“A hand-held photometric stereo camera for 3-D modeling.” in IEEE 12th Int. Conf. on Computer Vision pp.1234-1241 [19] J. Park, S. N. Sinha, Y. Matsushita, Y. Tai and I. S. Kweon. (2013)“Multiview Photometric Stereo Using Planar Mesh Parameterization.” 2013 IEEE International Conference on Computer Vision (ICCV) pp.1161-1168 [20] B. Shi, K. Inose, Y. Matsushita, and P. Tan. (2014)“Photometric stereo using internet images.” In 3DV pp.361-368 [21] Efros A A, Freeman W T. (2001)“Image Quilting for Texture Synthesis and Transfer[C].” Proceeding of SIGGRAPH pp.341-347 [22] Liang L, Liu C. (2001)“Real-time Texture Synthesis by Patch-based Sampling[R].” Proceeding of SIGGRAPH [23] Kwatra V, Schoodl A, Essa I, et a. (2003) “Graph cut Textures: image and video synthesis using graph cuts[J].” ACM Transactions on Graphics 22 (3): 277–286. [24] FordLRand DRFulkson. (1956)“Maximal flow throeIgh a network.” Candian Journal of Mathematics pp.399-404 [25] N. Kolagani, J. S. Fox, and D. R. Blidberg. (1992)“Photometric stereo using point light sources.” in Proc. 1992 IEEE Int. Conf. on Robotics and Automation pp.1759-1764 [26] B. Shi et al. (2016)“A benchmark dataset and evaluation for non-Lambertian and uncalibrated photometric stereo.” in Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition pp.3707-3716
Available online at www.sciencedirect.com
ScienceDirect
Procedia Computer Science 00 (2019) 000–000 Procedia Computer Science (2019) 000–000 Procedia Computer Science 14700 (2019) 247–253
www.elsevier.com/locate/procedia www.elsevier.com/locate/procedia
2018 International Conference on Identification, Information and Knowledge 2018 International Conference on Identification, Information and Knowledge in the Internet of Things, IIKI 2018 in the Internet of Things, IIKI 2018
Photometric Photometric Stereo Stereo on on Large Large Flattened Flattened Surfaces Surfaces Using Using Image Image Stitching Stitching Tingting Zhuaa , Lin Qia,∗ , Hao Fana , Junyu Donga Tingting Zhu , Lin Qia,∗, Hao Fana , Junyu Donga
a Department a Department
of Information Science and Engineering Ocean University of China Qingdao, China of Information Science and Engineering Ocean University of China Qingdao, China
Abstract Abstract In this paper, we proposed a method to enlarge the recovered surface normal from photometric stereo method using image stitching. In paper, we proposed a method to enlarge recovered normal from photometric stereo method using image stitching. Wethis used the laser triangulation to provide the the actual height surface as the reference to reduce the low-frequency noise from photometric We usedThe theGrid-based laser triangulation to provide(GMS) the actual height was as the reference to the reduce the low-frequency noise from photometric stereo. Motion Statistics algorithm used to obtain Homography matrix. We employed and tested stereo. The Grid-based Motion Statistics (GMS) to schemes obtain thewere Homography matrix. Weand employed and tested two algorithms for stitching: graph cut and imagealgorithm quilting. was Twoused splice used: post-stitching pre-stitching. The two algorithms forand stitching: graphofcut imagewere quilting. Two splice schemes were used: post-stitching and built pre-stitching. proposed method comparison twoand schemes tested by experiments using captured surfaces with our devices. The proposed method and comparison of two schemes were tested by experiments using captured surfaces with our built devices. c 2019 2019 The The Authors. Authors. Published Published by by Elsevier Elsevier B.V. B.V. © c 2019 The Authors. Published by Elsevier B.V. This is This is an an open open access access article article under under the the CC CC BY-NC-ND BY-NC-ND license license (https://creativecommons.org/licenses/by-nc-nd/4.0/) (https://creativecommons.org/licenses/by-nc-nd/4.0/) This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Peer-review under responsibility of the scientific committee the 2018 International Conference on Identification, Information under responsibility of the scientific committee ofof the 2018 International Conference on Identification, Information and Peer-review responsibility the scientific committee of the 2018 International Conference on Identification, Information and Knowledge the Internet of of Things. Knowledge inunder theinInternet of Things. and Knowledge in the Internet of Things. Keywords: Photometric stereo; Laser triangulation; Graph cut; Image quilting; Texture synthesis. Keywords: Photometric stereo; Laser triangulation; Graph cut; Image quilting; Texture synthesis.
1. Introduction 1. Introduction 3D reconstruction is an important and challenging problem in computer vision. There are existing works include 3D vision, reconstruction an important and(SFM), challenging problem in computer There are stereo existing include stereo structureis from motion [1] photometric method and sovision. on. Photometric is works a typical phostereo vision, structure from motion [1] (SFM), photometric method and so on. Photometric stereo is a typical tometric method that uses images of a scene taken from the same viewpoint and different illumination directionspho[2]. tometric method thatrecovers uses images of a scene taken from the pixel same of viewpoint different Photometric stereo gradients (or normal) at each capturedand images, andillumination the surface directions height can[2]. be Photometric stereo recovers field gradients (or normal) at each pixel of captured images, and the surface height can be integrated from the gradient [3][4][5]. integrated the gradient [3][4][5]. Classis from photometric stereofield requires uniform and collimate illumination [6]. However, the light sources employed Classis photometric stereo requires uniform andnonuniform. collimate illumination the lightby sources employed in practical photometric stereo setups are usually The images[6]. areHowever, usually captured near point light in practical photometric stereo setups are usually nonuniform. The images are usually captured by near point light sources equipment, which will introduce errors caused by the nonuniform illumination. We used two methods to sources equipment, which will introduce errors caused by the nonuniform illumination. We used two methods to correct the surface shape bias based on our previous work [7]. The first method is by fitting low-frequency error correct the surface shape bias based on our previous work [7]. The first method is by fitting low-frequency error without any reference points. The second method employs reference points with known 3D information, such as those without any reference points. The second method employs reference points with known 3D information, such as those ∗ ∗
Corresponding author. Tel.: +86-532-66781729. Corresponding Tel.: +86-532-66781729. E-mail address:author. [email protected] E-mail address: [email protected] c 2019 The Authors. Published by Elsevier B.V. 1877-0509 1877-0509 © 2019 The Authors. Published by B.V. c 2019 1877-0509 Thearticle Authors. Published by Elsevier Elsevier B.V. This isisan under the CC licenselicense (https://creativecommons.org/licenses/by-nc-nd/4.0/) This anopen openaccess access article under the BY-NC-ND CC BY-NC-ND (https://creativecommons.org/licenses/by-nc-nd/4.0/) This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Peer-review under of the committee of the 2018 Conference on Identification, Information and Knowledge in Peer-review underresponsibility responsibility of scientific the scientific committee of theInternational 2018 International Conference on Identification, Information and Peer-review under responsibility of the scientific committee of the 2018 International Conference on Identification, Information and Knowledge in the Internet of Knowledge inThings. the Internet of Things. the Internet of Things. 10.1016/j.procs.2019.01.245
Tingting Zhu et al. / Procedia Computer Science 147 (2019) 247–253 Tinting Zhu / Procedia Computer Science 00 (2019) 000–000
248 2
from laser triangulation or binocular stereo. The experiments prove that the result obtained after calibration is more accurate than traditional methods under near-point lighting. Photometric stereo can recover highly detailed surface geometry, but this also limits the reconstruction field in the view of the camera. Therefore, in this paper, we proposed a stitching method to reconstruct large-scale surface. Different from multiview photometric stereo, We employ texture synthesis technology to do the splice, which is image quilting and graph cut [9][10]. Texture synthesis is a fundamental problem in computer graphics. Given piece of small sample texture, the objective is to generate larger, visually similar texture image. In this paper, we proposed two methods for splicing. The first is to stitch the normal maps obtained by photometric stereo from different view point, and then the corresponding height can be integrated from the normal map. Another is to stitch the original images under the same illumination angle, and then do the photometric stereo to mosaic images to get the large normal map. The relative movement of the target object and the camera is not strictly translation, so we calibrate the small rotation by GMS algorithm [11]. The main contributions of this paper can be summarized as follows: • The method of combining texture synthesis with photometric stereo is proposed, which implements fine reconstruction of large-scale surfaces. • Two methods of stitching using texture synthesis are tested. One is to stitch the normal maps from photometric stereo. The other is to stitch the original images and then feed to photometric stereo to calculate normal map. • The low-frequency deformation caused by the nonuniform illumination is corrected for better synthesis results. 2. Related work 2.1. Photometric Stereo In recent years, many researchers studied how to convert photometric stereo from single view to multi-view. Based on traditional photometric stereo, Jong woo Lim [14] introduced an iterative algorithm for shape reconstruction from multiple images of a moving object illuminated at a distance. However, they need to manually mark feature points to recover the object surface by detecting and tracking the feature points in the image sequence. G.Vogiatzis and C.Hernanez [15][16][17] presented an algorithm which uses silhouettes of object. Starting from the silhouettes, the algorithm recovers camera motion and constructs the objects visual hull which is then used to recover the illumination and initialize a multi-view photometric stereo scheme to obtain a closed surface reconstruction. Although it can achieve high accuracy 3D reconstruction even for textureless shiny objects, but it is not applicable to large-scale surfaces such as the seabed. Tomoai Higot [18] attached a point light source to a hand-held camera to add a photometric constraint to the multi-view stereo (MVS) problem, and then simultaneously solve for shape, surface normal, and reflectance. Jeasik Park and Yuwing Tai [19] proposed a new photometric stereo based mesh refinement technique that can efficiently reconstruct meshes with fine geometric details by directly estimating a displacement texture map in the 2D parameter domain. SFM is used to calibrate the cameras and MVS is used to recover a coarse mesh. They required lots of images from different viewpoints and in our experiment, such conditions are almost impossible to achieve. Boxin Shi [20] focused on Internet images of large outdoor sculptures which are difficult to capture by Lidar or flying drone due to their size or security reason. These 3D reconstruction methods can achieve fine result on specific occasions, and in this paper, we studied the problem of recovering large-scale surface geometry using texture synthesis algorithms. 2.2. Texture Synthesis In this paper, we use two texture synthesis algorithms, i.e. the image quilting and the graph cut. Efros [21][22] proposed a texture synthesis algorithm named image quilting, which completes texture synthesis by calculating the cumulative error and minimum error path of the input texture block and the synthesized texture block in the overlapped region. Its main task is to deal with the boundary of the texture block and calculate the Euclidean distance of the RGB space in the overlapping region, thus finding the best seam and further finding the minimum error path. The algorithm is fast, but there may be discontinuity on 3D surface textures. Kwatra [23] proposed a texture synthesis algorithm
Tingting Zhu et al. / Procedia Computer Science 147 (2019) 247–253 Tinting Zhu / Procedia Computer Science 00 (2019) 000–000
249 3
Fig. 1. (a) Points with known 3D information from laser triangulation; (b) The original gradients and height produced by classic photometric stereo and the corrected results after calibration.
based on graph cut. Its principle is the maximum flow minimum cut problem. The maximum flow minimum cut theorem is proposed by Ford and Fulkerson [24] in 1956, that is, in any network, the maximum flow value is equal to the minimum cut capacity. Firstly we need to build a graph, and after solving the capacity between adjacent nodes in the graph, using the Ford-Fulkerson labeling method, the maximum flow minimum cut C can be obtained. 3. ALGORITHM 3.1. Gradient Deviation in Classic Photometric Stereo Classic photometric stereo required uniform and collimated illumination [25], but the light sources employed in this paper are near point light sources, so the reconstruction errors [26][12][13]] caused by the nonuniform lighting cannot be ignored. We plotted slate surface in Fig. 1(b), from which it is obvious that the gradients show linear deviation, and the surface shape is contaminated by quadratic low-frequency noise. The second line shows the corrected gradients and surface shape after calibration. Gradient Self-Correction: From the previous experimental observations, we found that a binary quadratic function f (x, y) = Ax2 + By2 + Cxy + Dx + Ey + F approximates the height deviation well. The gradient deviation and the quadratic center position will be pe (x, y) = 2Ax + Cy + D, qe (x, y) = 2By + Cx + E (xc , yc ) = (
CE − 2BD CD − 2AE , ) 4AB − c2 4AB − C 2
(1) (2)
where (pe , qe ) denotes the gradient deviation and (p0 , q0 ) denotes the original gradient obtained by classical photometric stereo. The corrected gradient (pc , qc ) will be (pc , qc ) = (p0 − pe , q0 − qe )
(3)
Optimized Gradient Self-Correction: If some surface points with known 3-D information are available, as shown in Fig.1(a), these points can be used as constraints to improve the reconstruction accuracy. The gradient deviation will be corrected by minimizing the difference between the height and the known height of these points. The 3-D information of some surface points can be obtained by methods such as laser triangulation. The optimization equation is = min(|Hi − H ps (pc , qc )|)
(4)
Where H1 denotes the correct height from laser triangulation, H ps denotes the height produced by photometric stereo method in real size. P − c(x, y) = p0 (x, y) − pe (x, y) = p0 (x, y) − (2Ax0 + Cy0 + D), qc (x, y) = q0 (x, y) − qe (x, y) = q0 (x, y) − (2By0 + Cx0 + E).The final surface height H ps is obtained through the best parameters. 3.2. Grid-based Motion Statistics Image stitching requires two images with the same size and has overlapping area. The object has horizontal displacement in the ideal situation, however the object motion in our experiment will cause little deviation, and so the seam will obvious. Therefore, we must calibrate the deviation before stitching the image.
Tingting Zhu et al. / Procedia Computer Science 147 (2019) 247–253 Tinting Zhu / Procedia Computer Science 00 (2019) 000–000
250 4
Fig. 2. left: Cloth and slate1; right: The experimental setup.
Grid-based Motion Statistics provides a real-time and ultra-robust correspondence system. It is a simple means of encapsulating motion smoothness as the statistical likelihood of a certain number of matches in a region. GMS enables translation of high match numbers into high match quality and develop an efficient Grid-based score estimator that can be incorporated into a real-time feature matcher. In this paper, we use GMS algorithm to obtain the relative position of camera from two view points, as shown in the left of Fig. 2. 3.3. Proposed Methods In this paper, we proposed two methods to stitch the images. The first method is obtained the normal map produced by classical photometric stereo, and then calibrated the height deformation caused by near-range light sources. Next, stitch the calibrated image. The traditional texture synthesis methods image quilting and graph cut are used to image stitching. Another method is to splice the original images with different angles of viewpoints but with the same illumination first, then obtain the normal map, next calibrate the deformation use the methods mentioned above. 1)Stitching After Photometric Stereo We use photometric metric to obtain the normal in two angels of view, and then stitching the normal map. Our experiment objects are cloth, paperboard, slate1 and slate2, which photograph in the laboratory. Each object is shot from two angles, the left side and the right side, there are overlapping parts in the middle. The normal map of four objects from different angels of view are obtained by photometric stereo and calibrated by the methods mentioned above. After the normal map is obtained, the next step is stitching the image. In this paper, image quilting and graph cut algorithm are used to stitching. Image Quilting Stitching: Input the size of texture1, texture2 and overlap respectively, and then calculate the size of the output image. Next the spliced image is copied to the specified position of the output image, and the middle overlap area can be regarded as texture2 covered on texture1. Calculated the Euclidean distance of the RGB space in the overlapping region, and reverse tracing to find a minimum error line which we required minimum path. Graph Cut Stitching: The first half of the algorithm is consistent with the image quilting algorithm. When finding the optimal seam, graph cut defines an energy function E= e(p, q) (5) p,q∈N
where N is the neighborhood of a point, (p, q) denoted the adjacent two points, and its corresponding points in image B is p , q . e(p, q) is a weight function. Using the weights of adjacent points as the capacity to build a network graph, to get a path of minimum energy, that is, the best seam we need. And then keep the content of the image A on the left side and preserve the content of the image B on the right side, so as to complete the splicing. 2)Photometric Stereo After Stitching Firstly, we stitch the original images through image quilting and graph cut algorithm. The experiment objects are cloth, paperboard, slate1 and the illumination angles are 0, 60, 120, 180, 240, 300 degree.
Tingting Zhu et al. / Procedia Computer Science 147 (2019) 247–253 Tinting Zhu / Procedia Computer Science 00 (2019) 000–000
251 5
Fig. 3. Pictures from left to right are the result of graph cut stitching and image quilting stitching respectively.
Fig. 4. Pictures from left to right are the result of graph cut stitching and image quilting stitching respectively.
After stitching the original images, we use photometric stereo to obtain the normal for synthetic image. Then the normal map is calibrated use the methods metioned above. 4. EXPERIMENTS 4.1. Experimental Setup Our experimental setup is shown in the right of Fig. 2, including the photometric stereo system and the laser triangulation system. A camera (IDS UI-358xCP-C) and six light sources constituted the photometric stereo system. The camera was placed at the center of the system, surrounded by six light-emitting diode (LED) point light sources. The light sources with adjustable lighting direction were placed around a rotating circular orbit. The line-structured laser was placed alongside the camera. The laser and the camera constituted a line laser 3-D imaging system. The objects were placed below the camera. In our experiments, the distance between the camera and objects along the optical axis was about 574 mm. The slant angle of each light source was set to 45 degree, and the distance between the light sources and objects along the optical axis was about 375 mm, which is similar to the radius of the circular orbit. The cameras resolution was 2592 × 2048 pixels. In addition, the camera calibration is carried out by the method of Zhang [8]. The images were captured in a dark room, and the LED lights are the only light sources. Photometric stereo was computed under six light directions, which had the same interval along the circular orbit. The object is moved at a fixed distance, and then take photos in the same way, there are overlaps between two images so that they can be stitching. 4.2. Experiment Result In this paper, we have two methods to stitch. The first method, obtained the normal map using classical photometric stereo first, and then calibrated the height deformation caused by near-range light sources. Next stitch the calibrated normal map. Another method is to splice the original images with different angles of view but with the same illumination first, then obtain the normal map, next calibrate the deformation. Stitching after Photometric Stereo: We use photometric metric to obtain the normal maps in two angles of view, and then stitching the normal map. Image quilting and graph cut algorithms are used to splice, the algorithm contrast is shown as Fig. 3. The first column indicated the normal map after stitching. The second column indicated heights produced by photometric stereo. The third column is the 3D surface of the object and the last line is the surface detail
Tingting Zhu et al. / Procedia Computer Science 147 (2019) 247–253 Tinting Zhu / Procedia Computer Science 00 (2019) 000–000
252 6
Fig. 5. Left: the left side of ball and the right side of ball. Right: the result of original image stitching.
Fig. 6. Left: stitching after photometric stereo. Right: photometric stereo after stitching.
map. In experiment, the overlap area of cloth and paperboard was 1944, and the overlap area of slate1 and slate2 was 1280. Photometric Stereo after stitching: After stitching the original images, we use photometric metric to obtain the normal maps for synthetic images, the result as shown in Fig. 4, they express the results of photometric stereo after splicing with graph cut and image quilting algorithm respectively. 4.3. Quantitative Evaluation In order to quantitatively evaluate the stitching result, we tested the standard sphere. The original images are shown in Fig. 5. Table 1. Method Comparison. Error(degree)
post-stitching
pre-stitching
PS Graph cut Image quilting
12.2382 12.6295 12.9145
12.2382 12.1205 12.2052
Stitching after Photometric Stereo: Using two methods to stitch normal map, the results are shown in the left of Fig.6. The top line and the bottom line is the result of graph cut and image quilting algorithm respectively. Each column from left to right is the ground truth, normal, gradient, height and 3D surface produced by photometric stereo, and the last column is an error graph between the true normal and the obtained normal. From the error analysis (TABLE I)on the ball, it can be seen that the precision of our methods is below 1 degree. Photometric Stereo after Stitching: The other method is stitched original images first, and then implement photometric stereo. The results are shown in the right of Fig. 6. From the error analysis (TABLE I) on the ball, it can be seen that the precision of our methods is below 1 degree. Compared two methods, we found that photometric stereo after stitching is slightly better than stitching after photometric stereo, and in these two methods, the result of graph cut is a little better than image stitching. 5. CONCLUSION This paper extends the photometric stereo from the fixed angle to multi-view angles. Two methods are adopted. First, the original images we collected are spliced according to the corresponding illumination angle, and then the photometric stereo is used to obtain the normal direction from two angles. The other method is to get the normal map from each view first and then merge the normal map. The first method uses self-calibration to calibrate the height error caused by uneven illumination. The second method uses laser triangulation to get the true height of the surface of the
Tingting Zhu et al. / Procedia Computer Science 147 (2019) 247–253 Tinting Zhu / Procedia Computer Science 00 (2019) 000–000
253 7
object, and uses it as a reference to calibrate the error. Although the method can achieve multi-view photometric stereo, there are still many defects. For example, the method needs to stop to capture images at every angel, because different illumination angles needed for photometric stereo. For splicing results, there is no accurate verification method to determine whether the splicing is good or bad. Therefore, in the future research, we will focus on how to speed up the reconstruction process to achieve real-time results. Acknowledgements This work was supported by the National Natural Science Foundation of China (NSFC) (No.61501417) and the Ph. D. Program Foundation of Ministry of Education of China (No.20120132110018). References [1] S. Agarwal et al. (2011) “Building Rome in a day.” Commun. ACM 54 (10): 105112. [2] S. Agarwal et al. Woodham R J. (1980)“Photometric method for determining surface orientation from multiple images[J].” Optical engineering 19 (1): 191139-191139. [3] R. T. Frankot and R. Chellappa. (1988)“A method for enforcing integrability in shape from shading algorithms.” IEEE Trans. Pattern Anal. Mach. Intel. 10 (4): 439-451. [4] A. Agrawal, R. Raskar, and R. Chellapp. (2006)“What is the range of surface reconstructions from a gradient field?” in European Conf. on Computer Vision pp.191139-191139. [5] M. Harker and P. OLeary. (2015)“Regularized reconstruction of a surface from its measured gradient field.” J. Math. Imaging Vision 51 (1): 46-70. [6] N. Kolagani, J. S. Fox, and D. R. Blidberg. (1992)“Photometric stereo using point light sources.” in Proc. 1992 IEEE Int. Conf. on Robotics and Automation pp.1759-1764. [7] Fan, H. Qi, L. Wang, N. Dong, J. Chen, Y. Yu, H. (2017)“Deviation correction method for close-range photometric stereo with nonuniform illumination.” Opt. Eng 56 (1): 1-13. [8] Z. Zhang. (2000)“A flexible new technique for camera calibration.” IEEE Trans. Pattern Anal. Mach. Intell 22 (11): 1330-1334. [9] Kwatra V, Schoodl A, Essa I, et al. (2003)“Graph cut Textures: image and video synthesis using graph cuts[J].” ACM Transactions on Graphics 22 (3): 277-286. [10] Boykov. Y, Veksler. O, and Zabih. R. (1999)“Fast approximate energy minimization via graph cuts.” In International Conference on Computer Vision pp.377-384. [11] Bian, J.W., Lin, W.Y., Matsushita, Y., et al. (2017)“GMS: Grid-Based Motion Statistics for Fast, Ultra-Robust Feature Correspondence.” IEEE Conference on Computer Vision and Pattern Recognition pp.2828-2837. [12] I. Horovitz and N. Kiryati. (2004)“Depth from gradient fields and control points: bias correction in photometric stereo.” Image Vision Compute 22 (9): 681-694. [13] D. Cho et al. (2016)“Photometric stereo under non-uniform light intensities and exposures.” in European Conf. on Computer Vision pp.170-186 [14] J. Lim, J. Ho, M. Yang, and D. Kriegman. (2005)“Passive Photometric stereo from Motion.” in Proc. IEEE International Conference on Computer Vision pp.383-391 [15] C. Hernandez, G. Vogiatzis, and R. Cipolla. (2008)“multiview photometric stereo.” IEEE Trans. Pattern Anal. Mach. Intell pp.548-554 [16] C. Hernandez, F. Schmitt, and R. Cipoll. (2007)“Silhouette coherence for camera calibration under circular motion.” IEEE Trans. Pattern Anal. Mach. Intell 29 (2): 343-349. [17] G. Vogiatzis, C. Hernandez, and R. Cipolla. (2006)“Reconstruction in the round using photometric normals and silhouettes.” in Proc. IEEE Conf. on Computer Vision and Pattern Recognition (2): 343-349. [18] T. Higo et al. (2009)“A hand-held photometric stereo camera for 3-D modeling.” in IEEE 12th Int. Conf. on Computer Vision pp.1234-1241 [19] J. Park, S. N. Sinha, Y. Matsushita, Y. Tai and I. S. Kweon. (2013)“Multiview Photometric Stereo Using Planar Mesh Parameterization.” 2013 IEEE International Conference on Computer Vision (ICCV) pp.1161-1168 [20] B. Shi, K. Inose, Y. Matsushita, and P. Tan. (2014)“Photometric stereo using internet images.” In 3DV pp.361-368 [21] Efros A A, Freeman W T. (2001)“Image Quilting for Texture Synthesis and Transfer[C].” Proceeding of SIGGRAPH pp.341-347 [22] Liang L, Liu C. (2001)“Real-time Texture Synthesis by Patch-based Sampling[R].” Proceeding of SIGGRAPH [23] Kwatra V, Schoodl A, Essa I, et a. (2003) “Graph cut Textures: image and video synthesis using graph cuts[J].” ACM Transactions on Graphics 22 (3): 277–286. [24] FordLRand DRFulkson. (1956)“Maximal flow throeIgh a network.” Candian Journal of Mathematics pp.399-404 [25] N. Kolagani, J. S. Fox, and D. R. Blidberg. (1992)“Photometric stereo using point light sources.” in Proc. 1992 IEEE Int. Conf. on Robotics and Automation pp.1759-1764 [26] B. Shi et al. (2016)“A benchmark dataset and evaluation for non-Lambertian and uncalibrated photometric stereo.” in Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition pp.3707-3716