- Email: [email protected]
An automatic 3D reconstruction system for texture-less objects
Robotics and Autonomous Systems 117 (2019) 29–39
Contents lists available at ScienceDirect
Robotics and Autonomous Systems journal homepage: www.elsevier.com/locate/robot
An automatic 3D reconstruction system for texture-less objects ∗
Ali Hosseininaveh Ahmadabadian , Ali Karami, Rouhallah Yazdan The Faculty of Geodesy and Geomatics Engineering, K. N. Toosi University of Technology, Tehran, Iran
highlights • • • •
A novel pattern projection system for 3D reconstruction of texture-less objects. A 3D reconstruction system controllable using a cell phone through Raspberry pi. A specific method for the system calibration. Performance evaluation of the system with several tests and comparing with GOM ATOS Compact laser scanner.
article
info
Article history: Received 17 October 2017 Received in revised form 2 October 2018 Accepted 1 April 2019 Available online 3 April 2019 Keywords: Structure from motion (SfM) Dense multi-view stereo (DMVS) 3D reconstruction system Texture-less objects Autonomous systems
a b s t r a c t Structure from Motion (SfM) is an image based method for 3D reconstruction of objects. This method coupled with Dense Multi-View Stereo (DMVS) can be used to generate an accurate point cloud of texture-full objects. Although this process is fully automatic, capturing images in proper locations is hard especially for texture-less objects which need a pattern projection procedure to have a successful matching step in SfM. This study aims to propose an automatic and portable system which can provide a pattern on objects and capture a set of high quality images in a way that a complete and accurate 3D model can be generated by the captured images using SFM and DMVS method. The system consists of three parts including a glassy turntable with a novel pattern projection system, a digital camera located on a mono-pod mounted on a length adjustable bar attached to the box of turntable and a controller system to control two other parts. Given the speed and step parameters for the system in a smart phone as the controller system, the digital camera automatically captures an image after every rotation step of the table. To evaluate the system, five different objects were tested under four criteria including plane fitting, structural resolution test, scale resolving test and comparing with a reference 3D model obtained with a commercial accurate laser scanner known as GOM ATOS Compact laser scanner. The average standard deviation for all the cited criteria was around 0.2 mm which illustrates the ability of the proposed system to capture high quality images for 3D reconstruction of texture-less objects. © 2019 Elsevier B.V. All rights reserved.
1. Introduction Non-contact measuring techniques which are classified into triangulation, time-of-flight and interferometry have got great interest by researchers in recent years [1]. Structure from Motion coupled with Dense Multi-View Stereo (DMVS) techniques are the-state-of-the-art techniques among triangulation techniques which reach the highest attention in the recent decade [2]. Information such as 3D coordinates of the object and the intrinsic and extrinsic parameters of the captured images can be achieved from a set of 2D overlapping images from different scenes in SfM procedure. Given sufficiently many corresponding key points in ∗ Corresponding author. E-mail addresses: [email protected] (A. Hosseininaveh Ahmadabadian), [email protected] (A. Karami), [email protected] (R. Yazdan). https://doi.org/10.1016/j.robot.2019.04.001 0921-8890/© 2019 Elsevier B.V. All rights reserved.
two images using key point matching algorithms such as Scale Invariant Feature Transform (SIFT) [3] and removing outliers, a fundamental matrix is computed. This matrix can be decomposed to obtain the relative poses (rotation and translation) of the second image with respect to the first one up to an unknown scale factor for the translation. Next, 3D point coordinates can be estimated by triangulation. For more than two views sequential and factorization-based structure from motion algorithms. The final stage of SfM algorithms is bundle adjustment, which is exploited iteratively to refine the intrinsic and extrinsic parameters of the camera as well as the 3D coordinates of the corresponding points to achieve a maximum likelihood estimate by the minimization of an appropriate cost function [4]. Given the extrinsic and intrinsic parameters of the images in SfM, stereo dense matching algorithms such as Semi-Global Matching (SGM) [5], Patch based Multi-View Stereo (PMVS) [6] or many other algorithms [7,8] followed by a space triangulation in a process known as Dense
30
A. Hosseininaveh Ahmadabadian, A. Karami and R. Yazdan / Robotics and Autonomous Systems 117 (2019) 29–39
Multi-View Stereo (DMVS) can be used to derive a dense point cloud [9]. Simplicity and affordability in terms of costs for non-experts makes SfM and DMVS techniques interesting for researchers in different fields. For example, in medicine for accurate modelling of body parts and their change detection for diagnosis [10–12]. Also, it is exploited in architectural engineering for detailed viewing of buildings from different scenes [13]. The simplicity of the technique makes it applicable in cultural heritage for producing digital 3D archives which can be used for both virtual museums and monument conservation [10,11]. The accuracy of the technique makes it an alternative for laser scanners in reverse engineering [10,14] and industrial inspection [14,15]. It has also applications in other areas such as geology, movie, game, entertainment industries [15–17], criminal investigation [10] and fashion [18–20]. There are a number of free software for SfM and DMVS processing (e.g., Bundler [21], VisualSFM [22,23], and Autodesk Recap [24], Agisoft PhotoScan [25], and web based software (e.g. Arc3D [26])). Compared with other techniques such as FARO Laser Scanners, Primesense [27] and EinScan [28], SfM and DMVS methods are relatively low-cost and straightforward to process. The accuracy of these techniques depends on several parameters including: the resolution of the captured images; the focal length of the camera; the aperture diameter of the lens; the type and quality of the lens; the quality of texture on the object; the size and complexity of the object shape; the light conditions; the distances and directions of the images with respect to the object; and the number of images. Based on these parameters, the relative accuracy varies between 1:10 and 1:300000 [29, 30]. For instance, the accuracy of Agisoft PhotoScan software reported by researchers varies between less than 0.1 mm and several centimetres for close range laboratory [31] and aerial photogrammetry [32] evaluations. Although SfM and DMVS methods applied on a set of images captured with common medium priced sensors can be a competitor for high resolution expensive sensors [31], these methods face real challenges for 3D modelling of texture-less or single coloured objects. For 3D reconstruction of such objects by using these techniques, researchers usefully project random patterns on the object from relatively fixed range to the object using projectors [3,7,33,34]. This makes the process of image capturing difficult and expensive due to the need for at least three projectors with relatively high lumens. Moreover, capturing enough images in suitable positions is a time consuming procedure, which needs an expert in the field of photogrammetry and computer vision. Furthermore, As it was mentioned, the scale is unknown in SfM method and the scale of the final 3D model is not correct due to defining an arbitrary scale factor in collinearity equations used in SfM [1]. These three challenges have been addressed by some researchers and several systems have been proposed for resolving one or all of them [14,35–38]. For such systems, many patterns such as grid-based [39,40], point-based [41], random dots [42] and multi coloured patterns [43] can be used for resolving the first challenge. However, all of them need projectors to provide the pattern on the objects. A single projector cannot cover all the faces of the object and subsequently more projectors are needed. To resolve the second challenge, although some of the proposed systems can automatically capture images in suitable poses [31,44], they either suffer from non-portability with expensive instruments or are not suitable for non-experts due to the need of setting many parameters by the operator. The third challenge, resolving the scale, has been resolved by researchers by using one of the following strategies. (1) using a scale bar placed on the object space and exploiting its length to resolve
the scale [36] with a drawback due to the uncertainty in three directions. (2) using stereo camera rigs and the length of the base line in collinearity equations as external constraints [31] with a drawback of using extra cameras. (3) Using control points on the object space with known 3D coordinates estimated in a calibration step. To automatically identifying control points in the images, coded markers are placed on the control points before capturing images. Different coded markers have been designed and can be generally classified into circular and rectangular shapes [45]. Based on the shape of the coded marker, different coded marker detection algorithms have been developed by researchers and implemented in software such as Agisoft Photoscan [46,47]. However, accurately estimating the coordinates of these markers needs accurate and expensive scale bars placed around the scene to define the datum and resolve the scale [7]. Considering the mentioned challenges, several systems have been proposed for 3D reconstruction of texture-less objects using SfM and DMVS methods. One of the low-cost systems was designed by Hosseininaveh et al. [36] for 3D modelling of small and texture-less objects. The system consisted of a computer workstation, a turntable, one digital camera and eight dot laser pointers with random pattern lenses placed around the turntable to provide patterns on the object as artificial markers. A set of images were captured by the digital camera from the object placed on a turntable for every 30 degree rotation [36]. Although, the system was low cost, the portability of the system was restricted due to the need of the computer workstation. Using the dot laser patterns could not provide a homogeneous pattern on the surface of the object due to the low lumens of the laser and a limited depth of field with sharp pattern on the object. Moreover, the process of capturing images was completely independent from the process of the turntable. This issue has reduced the time of capturing images considerably. Hosseininaveh et al. [14] also suggested a strategy to cluster and select vantage images in a their system to decrease the processing time for 3D reconstruction and showed that their system can provide a 3D model of a texture-less object with an accuracy around 0.1 mm [14] while there were some gaps in the final model due to the limitations of the projection system. [35], exploited 28 low-cost, 5 megapixels raspberry pi cameras with two small mobile projectors to develop a low-cost and automatic imaging system for 3D modelling of small texture-less objects. They also evaluated different patterns such as checkerboard, noise, tiled lines and squares to investigate which one could provide the best pattern on object and concluded that tiled lines and noise patterns can be the best patterns for their system [35]. Although the system could provide a relatively complete 3D model, the accuracy of the system was limited due to the use of low-cost cameras with only 5 megapixels. Moreover, the portability of the system was reduced with the scaffold holding the cameras and projectors. Although all the recent proposed systems can provide a complete 3D model from the object, they suffer from non-portability and non-user-friendly for non-experts. Thus, a portable, automatic and user-friendly system is required to generate an accurate 3D model from small and texture-less objects. This article aims to propose a user-friendly, low-cost, and portable system for accurate 3D reconstruction of small textureless objects. The system has a novelty in the pattern projection technique, which needs no projectors. Moreover, wisely exploiting both high-tech and off-the-shelf instruments in a compact form makes the system more user-friendly and portable than the recent proposed systems. Furthermore, a strategy is provided to resolve accurately the scale in a calibration step without the need for scale bars. The system can be used in many applications by non-experts in museums, industry, and many other fields. The remainder of this paper is as follows: Section 2 discusses the steps and details of the proposed system. Implementation is done in Section 3 and evaluations are performed in Section 4. The conclusion is reported in Section 5.
A. Hosseininaveh Ahmadabadian, A. Karami and R. Yazdan / Robotics and Autonomous Systems 117 (2019) 29–39
31
Fig. 1. An overview of the system.
Fig. 2. The pattern provided on a white paper cone.
2. The proposed system As it is illustrated in Fig. 1, the proposed system consists of three parts including: (1) a glassy turntable with a novel pattern projection system; (2) a digital camera located on a monopod mounted on a length adjustable bar attached to the box of turntable; (3) a controller system including a raspberry pi 2 model B and a mobile phone such as iPhone to control two other parts. An interesting object can be placed on the turntable while the projection system is working. Then, the height of the camera as well as its photography parameters such as range to the object, F-Stop, ISO and focal length should be adjusted. Afterwards, the speed and step of the rotation are set in a program in raspberry pi through VNC viewer on the mobile phone. Having run the program, the digital camera captures the first image and it is saved on a SD card on the camera. The turntable rotates the object based on the step and speed was set in the program and after a second another image is captured by the camera. This procedure is continued until a whole round completed. For complex object, this process should be repeated two or more times with different heights set for the camera. 2.1. The pattern projection system In the first part of the system, a pattern is projected on the object placed on the turntable by passing lights through a random pattern printed on the glassy circular plate and three glassy pieces mounted on bars located around the turntable. The lights are emitted from 48 coloured LEDs located under the glassy table and 12 other LEDs located above the three bars (see Fig. 2). The turntable itself includes several parts including, a hybrid stepper motor with half a degree step resolution with possible rotation in both clockwise and anti-clockwise directions, the driver
of the steeper motor, an anti-shocker to reduce the vibration of the system during the rotation, a 1:120 gearbox to increase the payload and step resolution of the stepper motor by 120 and a circular Plexiglass plate with 20 cm radius located below the glassy circular plate and is attached to the anti-shocker, the gearbox and stepper motor respectively. To resolve the scale problem, a set of coded markers are placed on the glassy turntable in different height (Fig. 3). The local coordinates of these markers are accurately estimated using a digital calliper and a photogrammetric procedure done in Agisoft software. 2.2. The imaging sensor As it is illustrated in Fig. 4, the imaging sensor of the system is a Nikon D5500 digital SLR camera with 24 Megapixels image resolution. This camera is attached to a raspberry pi 2 microcontroller [48] by a cable and its driver installed on the raspberry pi. The range and orientation of the camera to the object can be adjusted using an adjustable height monopod with a rotation switch placed on an adjustable length bar. The camera can also be placed on the box without monopod in case images from the bottom part the object should be captured. 2.3. The system controller The system controller includes a raspberry pi 2 model B microcontroller and a smart phone. The driver of stepper motor and the digital camera are connected to the raspberry pi through GPIO pins and USB cable respectively. A raspbian Jessie SD card image as a Linux operating system optimized for raspberry pi is cloned on the 16 Gigabytes SD card of the raspberry pi. The raspberry pi is remotely controlled with an iPhone 6 smart phone through a
32
A. Hosseininaveh Ahmadabadian, A. Karami and R. Yazdan / Robotics and Autonomous Systems 117 (2019) 29–39
Fig. 3. The pattern projection system (left) and its components: SMD LEDs (1) placed on aluminium plate (2). Stepper motor and its driver (3). Flexible shaft coupling (4) for connecting the gearbox (5) to the stepper motor and anti-shocker (6). The circular Plexiglas (7) holding 48 SMD LEDs. The glassy circular plate (8). The coded markers (9). Three 30 cm rods (10) holding the small glassy plates (11) and 12 other SMD LEDs (12).
Fig. 4. The digital camera mounted on the monopod placed on the adjustable length bar (left), the height of the camera for different rings (right).
three steps procedure including (1) attaching a Wi-Fi dongle to a raspberry pi USB port, (2) defining a static IP address in raspbian and (3) using tightvnc software (a Virtual Network Computing (VNC) viewer package) on both the iPhone and raspberry pi. An app is developed in QT programming language in raspberry pi to control both the stepper motor and the digital camera. Given the speed and step parameters for the turntable in the GUI of the App in raspbian and remotely controlled on a smart phone, the commands are sent to the driver of the stepper motor and the digital camera through GPIOs and a USB port respectively.
When a rotation is done, an image is captured while the pattern is projected on the object. This is repeated until a 360 degrees rotation is completed. In case of complexity of the object shape, second and third (or more) rings of images are captured by increasing the camera height and adjusting the orientation and range with the rotation switch and the adjustable length bar and repeating the image capturing procedure (Fig. 4, right). The whole procedure is done in a dark room.
A. Hosseininaveh Ahmadabadian, A. Karami and R. Yazdan / Robotics and Autonomous Systems 117 (2019) 29–39
33
Fig. 5. The system calibration procedure.
Fig. 6. The objects used for calibrating the system, hexagon metallic object (1) and evaluating its performance, the Cyrus tomb statue (2), the paper cone (3), the paper cube (4) and dude statue.
3. Methodology Having developed the proposed system, in the following sections the system calibration and 3D modelling with the proposed system are explained.
coordinates. This process can be done two or three times to make sure the reliability of the coordinates. The final coordinates can be exploited for every other objects without the need for repeating the calibration step. 3.2. 3D reconstruction with the proposed system
3.1. System calibration Resolving the scale in the proposed system is performed using the coded markers placed on the turntable. In order to estimate the accurate coordinates of these markers, a calibration step should be accomplished. As it is shown in Fig. 5, having captured a set of images from the coded markers and a reference object, the coded markers are automatically detected and SfM procedure is performed in Agisoft Photoscan software. The scale is then resolved in this software using the initial coordinates of the markers roughly estimated using a digital calliper. Next, in order to estimate the accurate scale factor, a dense point cloud is generated using a dense matching algorithm similar to SemiGlobal Matching algorithm (SGM-like) implemented in Agisoft Photoscan from the reference object. The obtained point cloud is then compared with a 3D point cloud of the object generated with a high-resolution expensive sensor such as GOM ATOS Compact laser scanner. The comparison can be done using either an extended Iterative Closest Point (ICP) algorithm [49] or a rigid registration algorithm with seven parameters (1 scale, 3 rotations and 3 translations) given several corresponding points in both point clouds [50]. Having estimated the scale factor using these algorithms implemented in CloudCompare software, it is applied on the initial coordinates of the markers, which can then be used for another iteration as initial values for the coded marker
Having estimated the accurate coordinates of the markers, the process of accurate 3D reconstruction using the proposed system and agisoft PhotoScan software is as follows: (1) Capturing images from the interesting object placed on the turntable while the patterns are projecting, (2) Coded Marker detection in the images, (3) Structure from Motion, (4) Resolving the scale with the valid coordinates of the coded markers estimated in calibration step, (5) Dense point cloud generation using DMVS and (6) Surface 3D reconstruction and texture mapping. In the capturing image step, the parameters of the camera should be set correctly to have sharp and full-textured patterns on the objects. This can be done by setting ISO, F-stop and shutter speed 100, 8 and 1:30 respectively. Due to the feature matching restriction algorithms such as Scale Invariant Feature Transform (SIFT) [3], the angles between two optical axes coming from two adjacent cameras should be less than 30 degrees. In order to satisfy this constraint, the step of the turntable should be set a value below 30 degrees. For instance with 15 degrees, 24 images can be captured in each camera height (ring). Moreover, due to occlusion area, having captured images in a single ring cannot guarantee a complete 3D model, and the images should be captured in two, three or more rings due to the complexity of the object. Having estimated the intrinsic (focal length, principal point coordinates, radial and tangential lens distortion, affinity and orthogonality
34
A. Hosseininaveh Ahmadabadian, A. Karami and R. Yazdan / Robotics and Autonomous Systems 117 (2019) 29–39
Fig. 7. The accurate models of the Cyrus tomb statue (middle), the hexagon (left) and the dude statue (right) generated with GOM ATOS Compact laser scanner.
Fig. 8. Defining the coordinate system for initial coordinates of the coded markers (Left), Imaging network and 3D model of the hexagon object obtained with the proposed system (Right).
parameters) and extrinsic parameters of the images (positions and orientations) in SfM procedure, scale should be resolved using the 2D coordinates (obtained by coded marker detection algorithm implemented in Agisoft PhotoScan) and 3D coordinates of the coded markers (Estimated in system calibration step) in a bundle adjustment procedure with additional constraints implemented in Agisoft PhotoScan [45]. A dense point cloud can be achieved using DMVS algorithm with running SGM-like algorithm implemented in the software [45]. Surface reconstruction and texture mapping can finally be applied on the point clouds to deliver a fine, complete 3D model from the object [51,52]. The object surface characteristics play a crucial role in the quality and the accuracy of the final 3D model in the proposed system. Objects with shiny, glassy and non-lambertian surface are the most challenging objects for the proposed system and all structured light 3D reconstruction techniques. To overcome this problem, a spry of titanium dioxide powder and alcohol can be used to provide a lambertian surface for such objects. This powder is usually used for structured light laser scanner such as GOM ATOS Compact laser scanner [53]. 4. Experiments and results In this section, the experiments accomplished for both the system calibration and 3D reconstruction with the proposed system are explained. For the first goal, system calibration, a small black metallic hexagon object was used. While, for evaluating the system in terms of the ability in resolving scale and overcoming texture-less issues, four different objects were chosen including a statue of the tomb of Cyrus used for evaluating the scale, a paper cone and a paper cube and a wooden dude statue as texture-less objects.
In order to calibrate the proposed system and evaluate the ability of the system in resolving the scale, the hexagon object, the dude statue and the statue of the tomb of Cyrus were scanned with GOM ATOS Compact laser scanner with 0.02 mm spatial resolution. Fig. 7 shows the generated 3D model for the mentioned objects. Given these models, system calibration and evaluating the system were performed. 4.1. System calibration In order to resolve the scale of the coordinates of the coded markers placed around the turntable, the procedure explained in Section 3.1 was followed. Based on this procedure, to estimate the rough coordinates of the markers, a sheet of paper including 6 coded markers were placed in the centre of the turntable and using a calliper, a coordinate system was defined for these markers (Fig. 8). Next, 48 images in two levels for every 15 degrees rotation of the turntable were captured using the digital camera. Having completed the steps of coded marker detection, SfM and resolving the scale using the rough coordinates of 6 coded markers in Agisoft PhotoScan software [54], the coordinate of all other markers placed around the turntable were estimated. Afterwards the paper sheet was replaced with the hexagon object and the procedure was repeated. Next, a dense point cloud of the object was achieved using the DMVS algorithm (Semi-Global Matching (SGM) like) implemented in this software. The point cloud was then compared with the accurate point cloud of this object provided with GOM ATOS Compact laser scanner [55]. A scale factor (0.09924 ± 0.0186) was computed using the method presented in [49] and implemented in cloudcompare software [56].
A. Hosseininaveh Ahmadabadian, A. Karami and R. Yazdan / Robotics and Autonomous Systems 117 (2019) 29–39
Fig. 9. The imaging network for the dude statue.
4.2. System evaluation Having calibrated the proposed system, in order to evaluate its performance, it was used for 3D reconstruction of a set of objects. The objects shown in Fig. 6 were 3D reconstructed by following the strategy described in Section 3.2. As it is shown in Fig. 9, for every 15 degrees rotation of the turntable, an image was captured and this procedure was repeated for complete rounds in several levels of height (four levels for the dude statue) with a suitable range to the object by adjusting the length of the bar attached to the monopod. Based on imaging network design principles [14], the range to the object can mathematically be estimated based on the focal length of the camera, the F-stop, the resolution of the image, the size of the object and the needed spatial resolution of the final model. Practically, having set the F-stop (8) and the focal length of the camera (24 mm), the range should be adjusted where the best quality of the image can be obtained. Given the images and known three coordinates of coded markers, a complete 3D model was obtained for each object by running the SFM, DMVS and surface reconstruction steps implemented in Agisoft Photoscan. Similar procedure was followed to reconstruct the object without using the proposed system by manually capturing the images in different height from the object placed on floor without using any projectors. Fig. 10 illustrates the 3D models of the objects obtained with and without the proposed system. As it is clear, for a complex and texture-less object, the system can provide a complete 3D model which could not be provided without the system. These results visually confirmed that the proposed system can be used for 3D reconstruction of texture-less objects. However, in order to accurately evaluate the performance of the system, four tests including plane fitting, structure resolution test, scale evaluation and comparing with a true data obtained with a GOM laser scanner were performed based on the principles presented in [7,31]. In the following sections, the tests are described. The plane fitting criterion is defined as the root mean square distances (standard deviation) between the points on a flat surface and a plane fitted on this surface. In this test, all the points on each surface are used for estimating the four parameters of a plane equation in a least square procedure. The standard deviation can be derived from the variance and co-variance matrix in the least square procedure [57]. This procedure has been implemented in point cloud processing software such as CloudCompare
35
[56], GOM inspect [58] and Geomagic Design X [59] which are software packages for the analysis of 3D measuring data for quality control, product development and production. The GOM inspect software, for instance, is exploited to assess 3D measuring data generated by 3D scanners, laser scanners produced by GOM Company. This test only can be used for paper cube and hexagon object. for objects with planar surfaces [14,36]. This criterion shows the level of noise produced by a method or a system. The complex object cannot be completely reconstructed in most of 3D reconstruction systems due to the limitation in the spatial resolution, occlusions and self-occlusion area which could be different depending on the technique used in the system [31]. In this test, having generated a section on both the point cloud generated with the system and the reference point cloud, the root mean square of distances between the corresponding points on both sections used as the structure resolution criterion. A lower value for this criterion shows the ability of the system in 3D reconstruction of complex objects [1,7]. Another criterion, scale resolving test, has been used by some researchers by comparing the distance between two parallel planes on the generated and the reference models. The planes are fitted to each face of the model as well as the reference 3D model and the distance between them are computed by selecting several sample points on each plane and estimating the average distances to the corresponding points on another plane in each model. The average distance values for both 3D model are then compared to check the ability of the system in resolving the scale [1,7]. The final criterion, comparing with a reference 3D model, is used for overall comparison including the noise and scale resolving and structural resolution errors. The generated 3D model is registered to the reference using ICP registration and the maximum, minimum and average standard deviations are calculated [14,36]. Having explained the criteria for evaluating the proposed system, they are applied on the two reference 3D models generated with the GOM ATOS Compact laser scanner (the hexagon and the statue of the tomb of Cyrus) and the 3D models obtained with the proposed system. All the tests were performed with the GOM inspect and CloudCompare software. Fig. 11 illustrates the results of the plan fitting test on the 3D model of the paper cube provided with the proposed system and the hexagon object for both the 3D models obtained with the proposed system and GOM laser scanner. The average of the standard deviations of plane fitting for the paper cube was 0.07 mm. A bigger value, 0.1 mm, was estimated for this test applied on the 3D model of the hexagon object obtained with the proposed system. While running this test on the reference 3D model of this object showed that the object is considerably flat with the average of the standard deviations around 0.01 mm. It should be considered that the hexagon object is much darker than the paper cube and absorbed some of the patterns projected by the system. This issue results in an uncertainty in dense matching algorithms and subsequently less accurate 3D model. The structural resolution tests were performed on the reference point cloud and the point cloud generated with the proposed system for the statue of Cyrus tomb and the hexagon object. The standard deviation of the distances between corresponding points on the sections generated on the reference point cloud and the point cloud generated with the proposed system for both the objects was 0.3 mm. As it is illustrated in Fig. 12, some sort of smoothness can be seen on the steps available on the statue in the point cloud generated with the proposed system, while they sharply reconstructed in the reference point cloud. Similar results were achieved for the hexagon object with smoothness at corners.
36
A. Hosseininaveh Ahmadabadian, A. Karami and R. Yazdan / Robotics and Autonomous Systems 117 (2019) 29–39
Fig. 10. The 3D model generated with the proposed system (top), the 3D model generated with SfM and DMVS without using the proposed system (bottom).
Fig. 11. The results of plane fitting for the hexagon object and the paper cube. From left to right: plane fitting for the paper cube — the proposed system (the average Root Mean Square (RMS): 0.07 mm), the plane fitting for the hexagon object — the proposed system (the average RMS: 0.1 mm) and the GOM laser scanner (the average RMS: 0.01 mm).
Fig. 12. The result of profiling and comparing the generated model with and the reference model for the hexagon object.
The scale resolving test was performed on the point clouds generated with the proposed system of the hexagon object and the paper cube. The average distances between sample points on
each face of the objects and their corresponding points on their parallel faces were estimated. Similar process was performed for the reference point cloud of the hexagon object to estimate the
A. Hosseininaveh Ahmadabadian, A. Karami and R. Yazdan / Robotics and Autonomous Systems 117 (2019) 29–39
37
Table 1 The result of scale resolving test for the hexagon object and the paper cube. The resolving scale error (mm)
Estimated distance (mm)
Reference distance (mm)
0.06
45.85
45.91
All the parallel planes
The resolving scale error (mm)
Estimated distance (mm)
Reference distance (mm)
0.16
99.62
99.78
Plane 1, 2
0.2
99.41
99.61
Plane 3, 4
Fig. 13. The result of model comparison for the dude statue (right), the statue of Cyrus tomb (left) and the hexagon object (middle) with 0.24, 0.25 mm and 0.2 mm standard deviation respectively. The 3D model generated with GOM ATOS Compact laser scanner (top right) is compared with the model obtained with the proposed system (top middle) and showed in colour.
reference distance for this object. While a digital calliper was used to estimate the reference distance for the paper cube. As can be seen in Table 1, the resolving scale error for the hexagon object, 0.06 mm is much better than this value for the paper cube, 0.16 and 0.2 mm for two directions, due to using the hexagon object for the system calibration and an uncertainty for the measuring distance with the calliper. The last but not least evaluation test, comparing with the reference 3D model, was accomplished on the dude statue, the hexagon object and the statue of the Cyrus tomb. As it was mentioned, the value obtained with this test is a cumulative error of the resolving scale error, structural resolution and flatness errors and as can be seen in Fig. 13, the values for this test are bigger than the previous test. The standard deviation for dude statue and the statue of the Cyrus tomb were 0.24 mm and 0.25 mm respectively. These values in comparison with the hexagon object with standard deviation of 0.2 mm were bigger due to the complexity of the objects. 5. Conclusion In this paper, a novel projection system implemented in a portable 3D reconstruction system was described. Moreover, the 3D reconstruction system was calibrated based on a new resolving scale technique in the process of SfM and DMVS. The projection system consists of four transparent sheets with random dot patterns in front of SMD lights in a suitable distance. This configuration is a low cost alternative for at least four projectors for providing similar pattern on object. The portable 3D reconstruction system is wisely designed using a software package written in raspberry pi 2 model B to send commands from a cell phone (e.g. an iPhone) to both a digital camera for capturing images and the driver of a stepper motor coupled with a gearbox for rotating the projection system. Using this configuration, a user just sets the speed and the step of the rotation in the software package using a VNC viewer app on the iPhone and the whole process of capturing images from the object in every rotation step is done automatically. These process should be repeated for
different heights of the camera to have two, three or more rings of images depending on the complexity of the object. The aim of the calibration step in the proposed system is to estimate the 3D coordinates of coded markers using a reference 3D model. Having calibrated the system, the coded markers can be detected automatically in the images captured from the interesting object and their estimated 3D coordinates can be imported in the bundle adjustment procedure in SfM and DMVS procedure implemented in Agisoft PhotoScan software. The ability of easily providing a complete 3D model of texture-less objects was justified using three completely texture-less objects including a paper cube, a dude statue and a paper cone. The system performance was also evaluated using three objects (consists of the paper cube, a hexagon metallic object and a statue of the Cyrus tomb) and based on four criteria including plane fitting, structural resolution test, scale resolving test and comparing with a reference 3D model obtained with a GOM ATOS Compact laser scanner. Each of these criteria demonstrates the strength and weakness of the proposed system in respect to noises, outliers, the level of details and scale. The best results for these criteria were obtained in plane fitting test for the paper cube and the hexagon object with 0.07 mm and 0.1 mm error respectively. The most challenging issue for the proposed system was clarified in the structural resolution test of the complex objects with 0.3 mm error. The overall accuracy of the system was depicted in comparing with the reference point clouds after ICP registration with 0.2 mm, 0.24 mm and 0.25 mm standard deviation errors for the hexagon, the dude statue and the statue of the Cyrus tomb respectively. Although the proposed system is a portable and suitable for 3D reconstruction of texture-less objects, to make the system more automatically, the mono pod can be replaced with an arm robot with a drawback of increasing cost and weight of the system. Appendix A. Supplementary data Supplementary material related to this article can be found online at https://doi.org/10.1016/j.robot.2019.04.001.
38
A. Hosseininaveh Ahmadabadian, A. Karami and R. Yazdan / Robotics and Autonomous Systems 117 (2019) 29–39
References [1] T. Luhmann, S. Robson, S. Kyle, I. Harley, Close Range Photogrammetry: Principles, Techniques and Applications, Whittles, 2006. [2] D. Forsyth, J. Ponce, Computer Vision: A Modern Approach, Prentice Hall, Upper Saddle River, NJ, London, 2011. [3] D.G. Lowe, Distinctive image features from scale-invariant keypoints, Int. J. Comput. Vis. 60 (2004) 91–110. [4] R. Hartley, A. Zisserman, Multiple View Geometry in Computer Vision, Cambridge university press, 2003. [5] M. Rothermel, K. Wenzel, D. Fritsch, N. Haala, Sure: photogrammetric surface reconstruction from imagery, in: Proceedings LC3D Workshop, Berlin, 2012. [6] Y. Furukawa, J. Ponce, Accurate camera calibration from multi-view stereo and bundle adjustment, Int. J. Comput. Vis. 84 (2009) 257–268. [7] A.H. Ahmadabadian, S. Robson, J. Boehm, M. Shortis, K. Wenzel, D. Fritsch, A comparison of dense matching algorithms for scaled surface reconstruction using stereo camera rigs, ISPRS J. Photogramm. Remote Sens. 78 (2013) 157–167. [8] F. Remondino, M.G. Spera, E. Nocerino, F. Menna, F. Nex, State of the art in high density image matching, Photogramm. Rec. 29 (2014) 144–166. [9] T. Schöps, J.L. Schönberger, S. Galliani, T. Sattler, K. Schindler, M. Pollefeys, A. Geiger, A multi-view stereo benchmark with high-resolution images and multi-camera videos, in: CVPR, 2017. [10] G. Sansoni, M. Trebeschi, F. Docchio, State-of-the-art and applications of 3d imaging sensors in industry, cultural heritage, medicine, and criminal investigation, Sensors 9 (2009) 568–601. [11] Y.-C. Chang, I. Detchev, A. Habib, A photogrammetric system for 3D reconstruction of a scoliotic torso, in: Proceedings of the ASPRS 2009 Annual Conference, 2008. [12] L. Barazzetti, M. Gianinetto, M. Scaioni, Automatic image-based 3D modeling for medical applications, in: Biomedical Engineering and Informatics (BMEI), 2012 5th International Conference on, IEEE, 2012, pp. 228–232. [13] P. Arias, C. Ordóñez, H. Lorenzo, J. Herraez, J. Armesto, Low-cost documentation of traditional agro-industrial buildings by close-range photogrammetry, Build. Environ. 42 (2007) 1817–1827. [14] A. Hosseininaveh, R. Yazdan, A. Karami, M. Moradi, F. Ghorbani, Clustering and selecting vantage images in a low-cost system for 3D reconstruction of texture-less objects, Measurement (2016). [15] F. Remondino, Heritage recording and 3D modeling with photogrammetry and 3D scanning, Remote Sens. 3 (2011) 1104–1138. [16] J.E. Chadwick, D.R. Haumann, R.E. Parent, Layered construction for deformable animated characters, in: ACM Siggraph Computer Graphics, ACM, 1989, pp. 243–252. [17] P. Volino, N.M. Thalmann, S. Jianhua, D. Thalmann, 1998. The Evolution of a 3D System for Simulating Deformable Clothes on Virtual Actors. MIRALab Copyright Information.. [18] N. D’apuzzo, A. Gruen, Recent advances in 3D full body scanning with applications to fashion and apparel, in: Optical 3-D Measurement Techniques IX, 2009. [19] J. Gu, T. Chang, I. Mak, S. Gopalsamy, H.C. Shen, M.M.-F. Yuen, A 3D reconstruction system for human body modeling, in: Modelling and Motion Capture Techniques for Virtual Environments, Springer, 1998. [20] S. Gray, In virtual fashion, IEEE Spectr. 35 (1998) 18–25. [21] N. Snavely, S. Seitz, R. Szeliski, Photo tourism: exploring image collections in 3d, acm transactions on graphics. ACM Transactions on Graphics 2 (2006). [22] C. Wu, Towards linear-time incremental structure from motion, in: 3D Vision-3DV 2013, 2013 International Conference on, IEEE, 2013, pp. 127–134. [23] C. Wu, Agarwal. S., B. Curless, S.M. Seitz, Multicore bundle adjustment, in: Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, IEEE, 2011, pp. 3057–3064. [24] Autodesk, RECAP: Create accurate 3D models with reality capture [Online]. Available: https://www.autodesk.co.uk/products/recap/overview [Accessed 09/05/2018 2018], 2018. [25] Agisoft, Agisoft PhotoScan [Online]. Available: http://www.agisoft.com/ [Accessed 09/05/2018 2018], 2018. [26] D. Tingdahl, L. Van Gool, A public system for image based 3d model generation, in: International conference on computer vision/computer graphics collaboration techniques and applications, Springer, 2011, pp. 262–273. [27] 2018. PRIMESENSE CARMINE 1.09 [Online]. Available: http://xtionprolive. com/primesense-carmine-1.09 [Accessed 09/05/2018 2018]. [28] Einscan, EinScan 3D Scanners [Online]. Available: https://www.einscan. com/ [Accessed 09/05/2018 2018], 2018.
[29] Sapirstein, Accurate measurement with photogrammetry at large sites, J. Archaeol. Sci. 66 (2016) 137–145. [30] L. Zheng, Q. Yuan, F. Sheng-Hong, L. Chang-Ru, W. Hai-Ren, Z. Ying-Xi, C. Jin-Quan, Y. Ji, High-precision photogrammetric surface figure measurements under cryogenic environment, Chinese Astronomy and Astrophysics 41 (2017) 136–148. [31] M. Hess, S. Robson, A.H. Ahmadabadian, A contest of sensors in close range 3D imaging: performance evaluation with a new metric test object, Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 40 (2014) 277. [32] V. Barrile, G. Bilotta, A. Nunnari, 3D modeling with photogrammetry by UAVs and model quality verification, ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci. 4 (2017) 129. [33] K. Wenzel, M. Rothermel, N. Haala, D. Fritsch, Sure–the ifp software for dense image matching, Photogramm. week (2013) 59–70. [34] M.Z. Brown, D. Burschka, G.D. Hager, Advances in computational stereo, IEEE Trans. Pattern Anal. Mach. Intell. 25 (2003) 993–1008. [35] J. Hafeez, S.-C. Kwon, S.-H. Lee, A. Hamacher, 3D surface reconstruction of smooth and textureless objects, in: Emerging Trends & Innovation in ICT (ICEI), 2017 International Conference on, IEEE, 2017, pp. 145–149. [36] A. Hosseininaveh, R. Yazdan, A. Karami, M. Moradi, F. Ghorbani, A low-cost and portable system for 3D reconstruction of texture-less objects, Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 40 (2015) 327. [37] A. Hosseininaveh, B. Sargeant, T. Erfani, S. Robson, M. Shortis, M. Hess, J. Boehm, Towards fully automatic reliable 3D acquisition: From designing imaging network to a complete and accurate point cloud, Robot. Auton. Syst. 62 (2014) 1197–1207. [38] M. Levoy, The Stanford Spherical Gantry, 2002. [39] J. Le Moigne, A.M. Waxman, Projected light grids for short range navigation of autonomous robots, in: Proc. 7th Int. Conf. Pattern Recognition, 1984, pp. 203–206. [40] J. LeMoigne, A.M. Waxman, Multi-Resolution Grid Patterns for Building Range Maps, Society of Manufacturing Engineers, 1985. [41] H. Morita, K. Yajima, S. Sakata, Reconstruction of surfaces of 3-d objects by m-array pattern projection method, in: Computer Vision. Second International Conference on, IEEE, 1988, pp. 468–473. [42] M. Maruyama, S. Abe, Range sensing by projecting multiple slits with random cuts, IEEE Trans. Pattern Anal. Mach. Intell. 15 (1993) 647–651. [43] K.L. Boyer, A.C. Kak, Color-encoded structured light for rapid active ranging, IEEE Trans. Pattern Anal. Mach. Intell. (1987) 14–28. [44] T.T. Nguyen, D.C. Slaughter, N. Max, J.N. Maloof, N. Sinha, Structured light-based 3D reconstruction system for plants, Sensors 15 (2015) 18587–18612. [45] T. Luhmann, S. Robson, S. Kyle, J. Boehm, Close-Range Photogrammetry and 3D Imaging, Walter de Gruyter, 2014. [46] J. Xu, Z.P. Fang, A.A. Malcolm, H. Wang, A robust close-range photogrammetric system for industrial metrology, in: Control, Automation, Robotics and Vision, 2002. ICARCV 2002. 7th International Conference on, IEEE, 2002, pp. 114–119. [47] U. Wijenayake, S.-I. Choi, S.-Y. Park, Automatic detection and decoding of photogrammetric coded targets, in: Electronics, Information and Communications (ICEIC), 2014 International Conference on, IEEE, 2014, pp. 1–2. [48] E. Upton, G. Halfacree, Raspberry Pi user guide, John Wiley & Sons, 2014. [49] T. Zinßer, J. Schmidt, H. Niemann, Point set registration with integrated scale estimation, in: International Conference on Pattern Recognition and Image Processing, 2005, pp. 116–119. [50] B. Bellekens, V. Spruyt, R. Berkvens, M. Weyn, A survey of rigid 3D pointcloud registration algorithms, in: AMBIENT 2014: the Fourth International Conference on Ambient Computing, Applications, Services and Technologies, August 24–28, 2014, Rome, Italy, 2014, pp. 8–13. [51] S.P. Lim, H. Haron, Surface reconstruction techniques: a review, Artif. Intell. Rev. 42 (2014) 59–78. [52] L. Gomes, O.R.P. Bellon, L. Silva, 3D reconstruction methods for digital preservation of cultural heritage: A survey, Pattern Recognit. Lett. 50 (2014) 3–14. [53] V. Mousavi, M. Khosravi, M. Ahmadi, N. Noori, S. Haghshenas, A. Hosseininaveh, M. Varshosaz, The performance evaluation of multi-image 3D reconstruction software with different sensors, Measurement 120 (2018) 1–10. [54] L. Agisoft, R. St Petersburg, Agisoft Photoscan, Professional Edition, 2014. [55] GOM. GOM ATOS Compact Scan [Online]. Available: https://www.gom.com/ [Accessed 09/05/2018 2018], 2018. [56] D. Girardeau-Montaut, Cloudcompare-open source project, in: OpenSource Project, 2011. [57] D. Eberly, Least Squares Fitting of Data, Magic Software, Chapel Hill, NC, 2000.
A. Hosseininaveh Ahmadabadian, A. Karami and R. Yazdan / Robotics and Autonomous Systems 117 (2019) 29–39 [58] GOM, GOM Inspect Software [Online]. Available: https://www.gom.com/ 3d-software/gom-inspect.html [Accessed 09/09/2018 2018], 2018. [59] Europac3d, Geomagic Design X [Online]. Available: https://europac3d.com/products/geomagic-design-x/?gclid= EAIaIQobChMI66Ke8ret3QIVjYWRCh2viQiDEAAYASAAEgLBoPD_BwE [Accessed 09/09/2018 2018], 2018.
Ali Hosseininaveh Ahmadabadian received his B. Sc. in Geodesy and Geomatics engineering from Islamic Azad University in 2003, an MSc in Photogrammetry from the K. N. Toosi University in 2007, and a PhD in Photogrammetry from University College London (UCL) in 2014. Since 2014, he has been an Assistant Professor working with the photogrammetry and remote sensing group, Geomatics Department, K. N. Toosi University. He is the author of more than 10 articles. His research interests include robotics and machine vision, especially next best view planning, photogrammetry, accurate 3D measurement, 3D modelling, and point cloud processing
39
Ali Karami received his B. Sc. degree in Geodesy and Geomatics engineering from Tabriz University, Iran in 2014. He received M. Sc. degree in Photogrammetry in 2016 from K. N. Toosi University of Technology. His research interests include computer vision, 3D modelling and image processing.
Rouhallah Yazdan received his B. Sc. degree in Geodesy and Geomatics engineering from K. N. Toosi University in 2005, Tehran, Iran and the MSc. degree in Photogrammetry from this university in 2009 and has been continuing his academic efforts as a Ph. D. student since 2014. His research interests include image processing and remote sensing, photogrammetry and 3D reconstruction.
Contents lists available at ScienceDirect
Robotics and Autonomous Systems journal homepage: www.elsevier.com/locate/robot
An automatic 3D reconstruction system for texture-less objects ∗
Ali Hosseininaveh Ahmadabadian , Ali Karami, Rouhallah Yazdan The Faculty of Geodesy and Geomatics Engineering, K. N. Toosi University of Technology, Tehran, Iran
highlights • • • •
A novel pattern projection system for 3D reconstruction of texture-less objects. A 3D reconstruction system controllable using a cell phone through Raspberry pi. A specific method for the system calibration. Performance evaluation of the system with several tests and comparing with GOM ATOS Compact laser scanner.
article
info
Article history: Received 17 October 2017 Received in revised form 2 October 2018 Accepted 1 April 2019 Available online 3 April 2019 Keywords: Structure from motion (SfM) Dense multi-view stereo (DMVS) 3D reconstruction system Texture-less objects Autonomous systems
a b s t r a c t Structure from Motion (SfM) is an image based method for 3D reconstruction of objects. This method coupled with Dense Multi-View Stereo (DMVS) can be used to generate an accurate point cloud of texture-full objects. Although this process is fully automatic, capturing images in proper locations is hard especially for texture-less objects which need a pattern projection procedure to have a successful matching step in SfM. This study aims to propose an automatic and portable system which can provide a pattern on objects and capture a set of high quality images in a way that a complete and accurate 3D model can be generated by the captured images using SFM and DMVS method. The system consists of three parts including a glassy turntable with a novel pattern projection system, a digital camera located on a mono-pod mounted on a length adjustable bar attached to the box of turntable and a controller system to control two other parts. Given the speed and step parameters for the system in a smart phone as the controller system, the digital camera automatically captures an image after every rotation step of the table. To evaluate the system, five different objects were tested under four criteria including plane fitting, structural resolution test, scale resolving test and comparing with a reference 3D model obtained with a commercial accurate laser scanner known as GOM ATOS Compact laser scanner. The average standard deviation for all the cited criteria was around 0.2 mm which illustrates the ability of the proposed system to capture high quality images for 3D reconstruction of texture-less objects. © 2019 Elsevier B.V. All rights reserved.
1. Introduction Non-contact measuring techniques which are classified into triangulation, time-of-flight and interferometry have got great interest by researchers in recent years [1]. Structure from Motion coupled with Dense Multi-View Stereo (DMVS) techniques are the-state-of-the-art techniques among triangulation techniques which reach the highest attention in the recent decade [2]. Information such as 3D coordinates of the object and the intrinsic and extrinsic parameters of the captured images can be achieved from a set of 2D overlapping images from different scenes in SfM procedure. Given sufficiently many corresponding key points in ∗ Corresponding author. E-mail addresses: [email protected] (A. Hosseininaveh Ahmadabadian), [email protected] (A. Karami), [email protected] (R. Yazdan). https://doi.org/10.1016/j.robot.2019.04.001 0921-8890/© 2019 Elsevier B.V. All rights reserved.
two images using key point matching algorithms such as Scale Invariant Feature Transform (SIFT) [3] and removing outliers, a fundamental matrix is computed. This matrix can be decomposed to obtain the relative poses (rotation and translation) of the second image with respect to the first one up to an unknown scale factor for the translation. Next, 3D point coordinates can be estimated by triangulation. For more than two views sequential and factorization-based structure from motion algorithms. The final stage of SfM algorithms is bundle adjustment, which is exploited iteratively to refine the intrinsic and extrinsic parameters of the camera as well as the 3D coordinates of the corresponding points to achieve a maximum likelihood estimate by the minimization of an appropriate cost function [4]. Given the extrinsic and intrinsic parameters of the images in SfM, stereo dense matching algorithms such as Semi-Global Matching (SGM) [5], Patch based Multi-View Stereo (PMVS) [6] or many other algorithms [7,8] followed by a space triangulation in a process known as Dense
30
A. Hosseininaveh Ahmadabadian, A. Karami and R. Yazdan / Robotics and Autonomous Systems 117 (2019) 29–39
Multi-View Stereo (DMVS) can be used to derive a dense point cloud [9]. Simplicity and affordability in terms of costs for non-experts makes SfM and DMVS techniques interesting for researchers in different fields. For example, in medicine for accurate modelling of body parts and their change detection for diagnosis [10–12]. Also, it is exploited in architectural engineering for detailed viewing of buildings from different scenes [13]. The simplicity of the technique makes it applicable in cultural heritage for producing digital 3D archives which can be used for both virtual museums and monument conservation [10,11]. The accuracy of the technique makes it an alternative for laser scanners in reverse engineering [10,14] and industrial inspection [14,15]. It has also applications in other areas such as geology, movie, game, entertainment industries [15–17], criminal investigation [10] and fashion [18–20]. There are a number of free software for SfM and DMVS processing (e.g., Bundler [21], VisualSFM [22,23], and Autodesk Recap [24], Agisoft PhotoScan [25], and web based software (e.g. Arc3D [26])). Compared with other techniques such as FARO Laser Scanners, Primesense [27] and EinScan [28], SfM and DMVS methods are relatively low-cost and straightforward to process. The accuracy of these techniques depends on several parameters including: the resolution of the captured images; the focal length of the camera; the aperture diameter of the lens; the type and quality of the lens; the quality of texture on the object; the size and complexity of the object shape; the light conditions; the distances and directions of the images with respect to the object; and the number of images. Based on these parameters, the relative accuracy varies between 1:10 and 1:300000 [29, 30]. For instance, the accuracy of Agisoft PhotoScan software reported by researchers varies between less than 0.1 mm and several centimetres for close range laboratory [31] and aerial photogrammetry [32] evaluations. Although SfM and DMVS methods applied on a set of images captured with common medium priced sensors can be a competitor for high resolution expensive sensors [31], these methods face real challenges for 3D modelling of texture-less or single coloured objects. For 3D reconstruction of such objects by using these techniques, researchers usefully project random patterns on the object from relatively fixed range to the object using projectors [3,7,33,34]. This makes the process of image capturing difficult and expensive due to the need for at least three projectors with relatively high lumens. Moreover, capturing enough images in suitable positions is a time consuming procedure, which needs an expert in the field of photogrammetry and computer vision. Furthermore, As it was mentioned, the scale is unknown in SfM method and the scale of the final 3D model is not correct due to defining an arbitrary scale factor in collinearity equations used in SfM [1]. These three challenges have been addressed by some researchers and several systems have been proposed for resolving one or all of them [14,35–38]. For such systems, many patterns such as grid-based [39,40], point-based [41], random dots [42] and multi coloured patterns [43] can be used for resolving the first challenge. However, all of them need projectors to provide the pattern on the objects. A single projector cannot cover all the faces of the object and subsequently more projectors are needed. To resolve the second challenge, although some of the proposed systems can automatically capture images in suitable poses [31,44], they either suffer from non-portability with expensive instruments or are not suitable for non-experts due to the need of setting many parameters by the operator. The third challenge, resolving the scale, has been resolved by researchers by using one of the following strategies. (1) using a scale bar placed on the object space and exploiting its length to resolve
the scale [36] with a drawback due to the uncertainty in three directions. (2) using stereo camera rigs and the length of the base line in collinearity equations as external constraints [31] with a drawback of using extra cameras. (3) Using control points on the object space with known 3D coordinates estimated in a calibration step. To automatically identifying control points in the images, coded markers are placed on the control points before capturing images. Different coded markers have been designed and can be generally classified into circular and rectangular shapes [45]. Based on the shape of the coded marker, different coded marker detection algorithms have been developed by researchers and implemented in software such as Agisoft Photoscan [46,47]. However, accurately estimating the coordinates of these markers needs accurate and expensive scale bars placed around the scene to define the datum and resolve the scale [7]. Considering the mentioned challenges, several systems have been proposed for 3D reconstruction of texture-less objects using SfM and DMVS methods. One of the low-cost systems was designed by Hosseininaveh et al. [36] for 3D modelling of small and texture-less objects. The system consisted of a computer workstation, a turntable, one digital camera and eight dot laser pointers with random pattern lenses placed around the turntable to provide patterns on the object as artificial markers. A set of images were captured by the digital camera from the object placed on a turntable for every 30 degree rotation [36]. Although, the system was low cost, the portability of the system was restricted due to the need of the computer workstation. Using the dot laser patterns could not provide a homogeneous pattern on the surface of the object due to the low lumens of the laser and a limited depth of field with sharp pattern on the object. Moreover, the process of capturing images was completely independent from the process of the turntable. This issue has reduced the time of capturing images considerably. Hosseininaveh et al. [14] also suggested a strategy to cluster and select vantage images in a their system to decrease the processing time for 3D reconstruction and showed that their system can provide a 3D model of a texture-less object with an accuracy around 0.1 mm [14] while there were some gaps in the final model due to the limitations of the projection system. [35], exploited 28 low-cost, 5 megapixels raspberry pi cameras with two small mobile projectors to develop a low-cost and automatic imaging system for 3D modelling of small texture-less objects. They also evaluated different patterns such as checkerboard, noise, tiled lines and squares to investigate which one could provide the best pattern on object and concluded that tiled lines and noise patterns can be the best patterns for their system [35]. Although the system could provide a relatively complete 3D model, the accuracy of the system was limited due to the use of low-cost cameras with only 5 megapixels. Moreover, the portability of the system was reduced with the scaffold holding the cameras and projectors. Although all the recent proposed systems can provide a complete 3D model from the object, they suffer from non-portability and non-user-friendly for non-experts. Thus, a portable, automatic and user-friendly system is required to generate an accurate 3D model from small and texture-less objects. This article aims to propose a user-friendly, low-cost, and portable system for accurate 3D reconstruction of small textureless objects. The system has a novelty in the pattern projection technique, which needs no projectors. Moreover, wisely exploiting both high-tech and off-the-shelf instruments in a compact form makes the system more user-friendly and portable than the recent proposed systems. Furthermore, a strategy is provided to resolve accurately the scale in a calibration step without the need for scale bars. The system can be used in many applications by non-experts in museums, industry, and many other fields. The remainder of this paper is as follows: Section 2 discusses the steps and details of the proposed system. Implementation is done in Section 3 and evaluations are performed in Section 4. The conclusion is reported in Section 5.
A. Hosseininaveh Ahmadabadian, A. Karami and R. Yazdan / Robotics and Autonomous Systems 117 (2019) 29–39
31
Fig. 1. An overview of the system.
Fig. 2. The pattern provided on a white paper cone.
2. The proposed system As it is illustrated in Fig. 1, the proposed system consists of three parts including: (1) a glassy turntable with a novel pattern projection system; (2) a digital camera located on a monopod mounted on a length adjustable bar attached to the box of turntable; (3) a controller system including a raspberry pi 2 model B and a mobile phone such as iPhone to control two other parts. An interesting object can be placed on the turntable while the projection system is working. Then, the height of the camera as well as its photography parameters such as range to the object, F-Stop, ISO and focal length should be adjusted. Afterwards, the speed and step of the rotation are set in a program in raspberry pi through VNC viewer on the mobile phone. Having run the program, the digital camera captures the first image and it is saved on a SD card on the camera. The turntable rotates the object based on the step and speed was set in the program and after a second another image is captured by the camera. This procedure is continued until a whole round completed. For complex object, this process should be repeated two or more times with different heights set for the camera. 2.1. The pattern projection system In the first part of the system, a pattern is projected on the object placed on the turntable by passing lights through a random pattern printed on the glassy circular plate and three glassy pieces mounted on bars located around the turntable. The lights are emitted from 48 coloured LEDs located under the glassy table and 12 other LEDs located above the three bars (see Fig. 2). The turntable itself includes several parts including, a hybrid stepper motor with half a degree step resolution with possible rotation in both clockwise and anti-clockwise directions, the driver
of the steeper motor, an anti-shocker to reduce the vibration of the system during the rotation, a 1:120 gearbox to increase the payload and step resolution of the stepper motor by 120 and a circular Plexiglass plate with 20 cm radius located below the glassy circular plate and is attached to the anti-shocker, the gearbox and stepper motor respectively. To resolve the scale problem, a set of coded markers are placed on the glassy turntable in different height (Fig. 3). The local coordinates of these markers are accurately estimated using a digital calliper and a photogrammetric procedure done in Agisoft software. 2.2. The imaging sensor As it is illustrated in Fig. 4, the imaging sensor of the system is a Nikon D5500 digital SLR camera with 24 Megapixels image resolution. This camera is attached to a raspberry pi 2 microcontroller [48] by a cable and its driver installed on the raspberry pi. The range and orientation of the camera to the object can be adjusted using an adjustable height monopod with a rotation switch placed on an adjustable length bar. The camera can also be placed on the box without monopod in case images from the bottom part the object should be captured. 2.3. The system controller The system controller includes a raspberry pi 2 model B microcontroller and a smart phone. The driver of stepper motor and the digital camera are connected to the raspberry pi through GPIO pins and USB cable respectively. A raspbian Jessie SD card image as a Linux operating system optimized for raspberry pi is cloned on the 16 Gigabytes SD card of the raspberry pi. The raspberry pi is remotely controlled with an iPhone 6 smart phone through a
32
A. Hosseininaveh Ahmadabadian, A. Karami and R. Yazdan / Robotics and Autonomous Systems 117 (2019) 29–39
Fig. 3. The pattern projection system (left) and its components: SMD LEDs (1) placed on aluminium plate (2). Stepper motor and its driver (3). Flexible shaft coupling (4) for connecting the gearbox (5) to the stepper motor and anti-shocker (6). The circular Plexiglas (7) holding 48 SMD LEDs. The glassy circular plate (8). The coded markers (9). Three 30 cm rods (10) holding the small glassy plates (11) and 12 other SMD LEDs (12).
Fig. 4. The digital camera mounted on the monopod placed on the adjustable length bar (left), the height of the camera for different rings (right).
three steps procedure including (1) attaching a Wi-Fi dongle to a raspberry pi USB port, (2) defining a static IP address in raspbian and (3) using tightvnc software (a Virtual Network Computing (VNC) viewer package) on both the iPhone and raspberry pi. An app is developed in QT programming language in raspberry pi to control both the stepper motor and the digital camera. Given the speed and step parameters for the turntable in the GUI of the App in raspbian and remotely controlled on a smart phone, the commands are sent to the driver of the stepper motor and the digital camera through GPIOs and a USB port respectively.
When a rotation is done, an image is captured while the pattern is projected on the object. This is repeated until a 360 degrees rotation is completed. In case of complexity of the object shape, second and third (or more) rings of images are captured by increasing the camera height and adjusting the orientation and range with the rotation switch and the adjustable length bar and repeating the image capturing procedure (Fig. 4, right). The whole procedure is done in a dark room.
A. Hosseininaveh Ahmadabadian, A. Karami and R. Yazdan / Robotics and Autonomous Systems 117 (2019) 29–39
33
Fig. 5. The system calibration procedure.
Fig. 6. The objects used for calibrating the system, hexagon metallic object (1) and evaluating its performance, the Cyrus tomb statue (2), the paper cone (3), the paper cube (4) and dude statue.
3. Methodology Having developed the proposed system, in the following sections the system calibration and 3D modelling with the proposed system are explained.
coordinates. This process can be done two or three times to make sure the reliability of the coordinates. The final coordinates can be exploited for every other objects without the need for repeating the calibration step. 3.2. 3D reconstruction with the proposed system
3.1. System calibration Resolving the scale in the proposed system is performed using the coded markers placed on the turntable. In order to estimate the accurate coordinates of these markers, a calibration step should be accomplished. As it is shown in Fig. 5, having captured a set of images from the coded markers and a reference object, the coded markers are automatically detected and SfM procedure is performed in Agisoft Photoscan software. The scale is then resolved in this software using the initial coordinates of the markers roughly estimated using a digital calliper. Next, in order to estimate the accurate scale factor, a dense point cloud is generated using a dense matching algorithm similar to SemiGlobal Matching algorithm (SGM-like) implemented in Agisoft Photoscan from the reference object. The obtained point cloud is then compared with a 3D point cloud of the object generated with a high-resolution expensive sensor such as GOM ATOS Compact laser scanner. The comparison can be done using either an extended Iterative Closest Point (ICP) algorithm [49] or a rigid registration algorithm with seven parameters (1 scale, 3 rotations and 3 translations) given several corresponding points in both point clouds [50]. Having estimated the scale factor using these algorithms implemented in CloudCompare software, it is applied on the initial coordinates of the markers, which can then be used for another iteration as initial values for the coded marker
Having estimated the accurate coordinates of the markers, the process of accurate 3D reconstruction using the proposed system and agisoft PhotoScan software is as follows: (1) Capturing images from the interesting object placed on the turntable while the patterns are projecting, (2) Coded Marker detection in the images, (3) Structure from Motion, (4) Resolving the scale with the valid coordinates of the coded markers estimated in calibration step, (5) Dense point cloud generation using DMVS and (6) Surface 3D reconstruction and texture mapping. In the capturing image step, the parameters of the camera should be set correctly to have sharp and full-textured patterns on the objects. This can be done by setting ISO, F-stop and shutter speed 100, 8 and 1:30 respectively. Due to the feature matching restriction algorithms such as Scale Invariant Feature Transform (SIFT) [3], the angles between two optical axes coming from two adjacent cameras should be less than 30 degrees. In order to satisfy this constraint, the step of the turntable should be set a value below 30 degrees. For instance with 15 degrees, 24 images can be captured in each camera height (ring). Moreover, due to occlusion area, having captured images in a single ring cannot guarantee a complete 3D model, and the images should be captured in two, three or more rings due to the complexity of the object. Having estimated the intrinsic (focal length, principal point coordinates, radial and tangential lens distortion, affinity and orthogonality
34
A. Hosseininaveh Ahmadabadian, A. Karami and R. Yazdan / Robotics and Autonomous Systems 117 (2019) 29–39
Fig. 7. The accurate models of the Cyrus tomb statue (middle), the hexagon (left) and the dude statue (right) generated with GOM ATOS Compact laser scanner.
Fig. 8. Defining the coordinate system for initial coordinates of the coded markers (Left), Imaging network and 3D model of the hexagon object obtained with the proposed system (Right).
parameters) and extrinsic parameters of the images (positions and orientations) in SfM procedure, scale should be resolved using the 2D coordinates (obtained by coded marker detection algorithm implemented in Agisoft PhotoScan) and 3D coordinates of the coded markers (Estimated in system calibration step) in a bundle adjustment procedure with additional constraints implemented in Agisoft PhotoScan [45]. A dense point cloud can be achieved using DMVS algorithm with running SGM-like algorithm implemented in the software [45]. Surface reconstruction and texture mapping can finally be applied on the point clouds to deliver a fine, complete 3D model from the object [51,52]. The object surface characteristics play a crucial role in the quality and the accuracy of the final 3D model in the proposed system. Objects with shiny, glassy and non-lambertian surface are the most challenging objects for the proposed system and all structured light 3D reconstruction techniques. To overcome this problem, a spry of titanium dioxide powder and alcohol can be used to provide a lambertian surface for such objects. This powder is usually used for structured light laser scanner such as GOM ATOS Compact laser scanner [53]. 4. Experiments and results In this section, the experiments accomplished for both the system calibration and 3D reconstruction with the proposed system are explained. For the first goal, system calibration, a small black metallic hexagon object was used. While, for evaluating the system in terms of the ability in resolving scale and overcoming texture-less issues, four different objects were chosen including a statue of the tomb of Cyrus used for evaluating the scale, a paper cone and a paper cube and a wooden dude statue as texture-less objects.
In order to calibrate the proposed system and evaluate the ability of the system in resolving the scale, the hexagon object, the dude statue and the statue of the tomb of Cyrus were scanned with GOM ATOS Compact laser scanner with 0.02 mm spatial resolution. Fig. 7 shows the generated 3D model for the mentioned objects. Given these models, system calibration and evaluating the system were performed. 4.1. System calibration In order to resolve the scale of the coordinates of the coded markers placed around the turntable, the procedure explained in Section 3.1 was followed. Based on this procedure, to estimate the rough coordinates of the markers, a sheet of paper including 6 coded markers were placed in the centre of the turntable and using a calliper, a coordinate system was defined for these markers (Fig. 8). Next, 48 images in two levels for every 15 degrees rotation of the turntable were captured using the digital camera. Having completed the steps of coded marker detection, SfM and resolving the scale using the rough coordinates of 6 coded markers in Agisoft PhotoScan software [54], the coordinate of all other markers placed around the turntable were estimated. Afterwards the paper sheet was replaced with the hexagon object and the procedure was repeated. Next, a dense point cloud of the object was achieved using the DMVS algorithm (Semi-Global Matching (SGM) like) implemented in this software. The point cloud was then compared with the accurate point cloud of this object provided with GOM ATOS Compact laser scanner [55]. A scale factor (0.09924 ± 0.0186) was computed using the method presented in [49] and implemented in cloudcompare software [56].
A. Hosseininaveh Ahmadabadian, A. Karami and R. Yazdan / Robotics and Autonomous Systems 117 (2019) 29–39
Fig. 9. The imaging network for the dude statue.
4.2. System evaluation Having calibrated the proposed system, in order to evaluate its performance, it was used for 3D reconstruction of a set of objects. The objects shown in Fig. 6 were 3D reconstructed by following the strategy described in Section 3.2. As it is shown in Fig. 9, for every 15 degrees rotation of the turntable, an image was captured and this procedure was repeated for complete rounds in several levels of height (four levels for the dude statue) with a suitable range to the object by adjusting the length of the bar attached to the monopod. Based on imaging network design principles [14], the range to the object can mathematically be estimated based on the focal length of the camera, the F-stop, the resolution of the image, the size of the object and the needed spatial resolution of the final model. Practically, having set the F-stop (8) and the focal length of the camera (24 mm), the range should be adjusted where the best quality of the image can be obtained. Given the images and known three coordinates of coded markers, a complete 3D model was obtained for each object by running the SFM, DMVS and surface reconstruction steps implemented in Agisoft Photoscan. Similar procedure was followed to reconstruct the object without using the proposed system by manually capturing the images in different height from the object placed on floor without using any projectors. Fig. 10 illustrates the 3D models of the objects obtained with and without the proposed system. As it is clear, for a complex and texture-less object, the system can provide a complete 3D model which could not be provided without the system. These results visually confirmed that the proposed system can be used for 3D reconstruction of texture-less objects. However, in order to accurately evaluate the performance of the system, four tests including plane fitting, structure resolution test, scale evaluation and comparing with a true data obtained with a GOM laser scanner were performed based on the principles presented in [7,31]. In the following sections, the tests are described. The plane fitting criterion is defined as the root mean square distances (standard deviation) between the points on a flat surface and a plane fitted on this surface. In this test, all the points on each surface are used for estimating the four parameters of a plane equation in a least square procedure. The standard deviation can be derived from the variance and co-variance matrix in the least square procedure [57]. This procedure has been implemented in point cloud processing software such as CloudCompare
35
[56], GOM inspect [58] and Geomagic Design X [59] which are software packages for the analysis of 3D measuring data for quality control, product development and production. The GOM inspect software, for instance, is exploited to assess 3D measuring data generated by 3D scanners, laser scanners produced by GOM Company. This test only can be used for paper cube and hexagon object. for objects with planar surfaces [14,36]. This criterion shows the level of noise produced by a method or a system. The complex object cannot be completely reconstructed in most of 3D reconstruction systems due to the limitation in the spatial resolution, occlusions and self-occlusion area which could be different depending on the technique used in the system [31]. In this test, having generated a section on both the point cloud generated with the system and the reference point cloud, the root mean square of distances between the corresponding points on both sections used as the structure resolution criterion. A lower value for this criterion shows the ability of the system in 3D reconstruction of complex objects [1,7]. Another criterion, scale resolving test, has been used by some researchers by comparing the distance between two parallel planes on the generated and the reference models. The planes are fitted to each face of the model as well as the reference 3D model and the distance between them are computed by selecting several sample points on each plane and estimating the average distances to the corresponding points on another plane in each model. The average distance values for both 3D model are then compared to check the ability of the system in resolving the scale [1,7]. The final criterion, comparing with a reference 3D model, is used for overall comparison including the noise and scale resolving and structural resolution errors. The generated 3D model is registered to the reference using ICP registration and the maximum, minimum and average standard deviations are calculated [14,36]. Having explained the criteria for evaluating the proposed system, they are applied on the two reference 3D models generated with the GOM ATOS Compact laser scanner (the hexagon and the statue of the tomb of Cyrus) and the 3D models obtained with the proposed system. All the tests were performed with the GOM inspect and CloudCompare software. Fig. 11 illustrates the results of the plan fitting test on the 3D model of the paper cube provided with the proposed system and the hexagon object for both the 3D models obtained with the proposed system and GOM laser scanner. The average of the standard deviations of plane fitting for the paper cube was 0.07 mm. A bigger value, 0.1 mm, was estimated for this test applied on the 3D model of the hexagon object obtained with the proposed system. While running this test on the reference 3D model of this object showed that the object is considerably flat with the average of the standard deviations around 0.01 mm. It should be considered that the hexagon object is much darker than the paper cube and absorbed some of the patterns projected by the system. This issue results in an uncertainty in dense matching algorithms and subsequently less accurate 3D model. The structural resolution tests were performed on the reference point cloud and the point cloud generated with the proposed system for the statue of Cyrus tomb and the hexagon object. The standard deviation of the distances between corresponding points on the sections generated on the reference point cloud and the point cloud generated with the proposed system for both the objects was 0.3 mm. As it is illustrated in Fig. 12, some sort of smoothness can be seen on the steps available on the statue in the point cloud generated with the proposed system, while they sharply reconstructed in the reference point cloud. Similar results were achieved for the hexagon object with smoothness at corners.
36
A. Hosseininaveh Ahmadabadian, A. Karami and R. Yazdan / Robotics and Autonomous Systems 117 (2019) 29–39
Fig. 10. The 3D model generated with the proposed system (top), the 3D model generated with SfM and DMVS without using the proposed system (bottom).
Fig. 11. The results of plane fitting for the hexagon object and the paper cube. From left to right: plane fitting for the paper cube — the proposed system (the average Root Mean Square (RMS): 0.07 mm), the plane fitting for the hexagon object — the proposed system (the average RMS: 0.1 mm) and the GOM laser scanner (the average RMS: 0.01 mm).
Fig. 12. The result of profiling and comparing the generated model with and the reference model for the hexagon object.
The scale resolving test was performed on the point clouds generated with the proposed system of the hexagon object and the paper cube. The average distances between sample points on
each face of the objects and their corresponding points on their parallel faces were estimated. Similar process was performed for the reference point cloud of the hexagon object to estimate the
A. Hosseininaveh Ahmadabadian, A. Karami and R. Yazdan / Robotics and Autonomous Systems 117 (2019) 29–39
37
Table 1 The result of scale resolving test for the hexagon object and the paper cube. The resolving scale error (mm)
Estimated distance (mm)
Reference distance (mm)
0.06
45.85
45.91
All the parallel planes
The resolving scale error (mm)
Estimated distance (mm)
Reference distance (mm)
0.16
99.62
99.78
Plane 1, 2
0.2
99.41
99.61
Plane 3, 4
Fig. 13. The result of model comparison for the dude statue (right), the statue of Cyrus tomb (left) and the hexagon object (middle) with 0.24, 0.25 mm and 0.2 mm standard deviation respectively. The 3D model generated with GOM ATOS Compact laser scanner (top right) is compared with the model obtained with the proposed system (top middle) and showed in colour.
reference distance for this object. While a digital calliper was used to estimate the reference distance for the paper cube. As can be seen in Table 1, the resolving scale error for the hexagon object, 0.06 mm is much better than this value for the paper cube, 0.16 and 0.2 mm for two directions, due to using the hexagon object for the system calibration and an uncertainty for the measuring distance with the calliper. The last but not least evaluation test, comparing with the reference 3D model, was accomplished on the dude statue, the hexagon object and the statue of the Cyrus tomb. As it was mentioned, the value obtained with this test is a cumulative error of the resolving scale error, structural resolution and flatness errors and as can be seen in Fig. 13, the values for this test are bigger than the previous test. The standard deviation for dude statue and the statue of the Cyrus tomb were 0.24 mm and 0.25 mm respectively. These values in comparison with the hexagon object with standard deviation of 0.2 mm were bigger due to the complexity of the objects. 5. Conclusion In this paper, a novel projection system implemented in a portable 3D reconstruction system was described. Moreover, the 3D reconstruction system was calibrated based on a new resolving scale technique in the process of SfM and DMVS. The projection system consists of four transparent sheets with random dot patterns in front of SMD lights in a suitable distance. This configuration is a low cost alternative for at least four projectors for providing similar pattern on object. The portable 3D reconstruction system is wisely designed using a software package written in raspberry pi 2 model B to send commands from a cell phone (e.g. an iPhone) to both a digital camera for capturing images and the driver of a stepper motor coupled with a gearbox for rotating the projection system. Using this configuration, a user just sets the speed and the step of the rotation in the software package using a VNC viewer app on the iPhone and the whole process of capturing images from the object in every rotation step is done automatically. These process should be repeated for
different heights of the camera to have two, three or more rings of images depending on the complexity of the object. The aim of the calibration step in the proposed system is to estimate the 3D coordinates of coded markers using a reference 3D model. Having calibrated the system, the coded markers can be detected automatically in the images captured from the interesting object and their estimated 3D coordinates can be imported in the bundle adjustment procedure in SfM and DMVS procedure implemented in Agisoft PhotoScan software. The ability of easily providing a complete 3D model of texture-less objects was justified using three completely texture-less objects including a paper cube, a dude statue and a paper cone. The system performance was also evaluated using three objects (consists of the paper cube, a hexagon metallic object and a statue of the Cyrus tomb) and based on four criteria including plane fitting, structural resolution test, scale resolving test and comparing with a reference 3D model obtained with a GOM ATOS Compact laser scanner. Each of these criteria demonstrates the strength and weakness of the proposed system in respect to noises, outliers, the level of details and scale. The best results for these criteria were obtained in plane fitting test for the paper cube and the hexagon object with 0.07 mm and 0.1 mm error respectively. The most challenging issue for the proposed system was clarified in the structural resolution test of the complex objects with 0.3 mm error. The overall accuracy of the system was depicted in comparing with the reference point clouds after ICP registration with 0.2 mm, 0.24 mm and 0.25 mm standard deviation errors for the hexagon, the dude statue and the statue of the Cyrus tomb respectively. Although the proposed system is a portable and suitable for 3D reconstruction of texture-less objects, to make the system more automatically, the mono pod can be replaced with an arm robot with a drawback of increasing cost and weight of the system. Appendix A. Supplementary data Supplementary material related to this article can be found online at https://doi.org/10.1016/j.robot.2019.04.001.
38
A. Hosseininaveh Ahmadabadian, A. Karami and R. Yazdan / Robotics and Autonomous Systems 117 (2019) 29–39
References [1] T. Luhmann, S. Robson, S. Kyle, I. Harley, Close Range Photogrammetry: Principles, Techniques and Applications, Whittles, 2006. [2] D. Forsyth, J. Ponce, Computer Vision: A Modern Approach, Prentice Hall, Upper Saddle River, NJ, London, 2011. [3] D.G. Lowe, Distinctive image features from scale-invariant keypoints, Int. J. Comput. Vis. 60 (2004) 91–110. [4] R. Hartley, A. Zisserman, Multiple View Geometry in Computer Vision, Cambridge university press, 2003. [5] M. Rothermel, K. Wenzel, D. Fritsch, N. Haala, Sure: photogrammetric surface reconstruction from imagery, in: Proceedings LC3D Workshop, Berlin, 2012. [6] Y. Furukawa, J. Ponce, Accurate camera calibration from multi-view stereo and bundle adjustment, Int. J. Comput. Vis. 84 (2009) 257–268. [7] A.H. Ahmadabadian, S. Robson, J. Boehm, M. Shortis, K. Wenzel, D. Fritsch, A comparison of dense matching algorithms for scaled surface reconstruction using stereo camera rigs, ISPRS J. Photogramm. Remote Sens. 78 (2013) 157–167. [8] F. Remondino, M.G. Spera, E. Nocerino, F. Menna, F. Nex, State of the art in high density image matching, Photogramm. Rec. 29 (2014) 144–166. [9] T. Schöps, J.L. Schönberger, S. Galliani, T. Sattler, K. Schindler, M. Pollefeys, A. Geiger, A multi-view stereo benchmark with high-resolution images and multi-camera videos, in: CVPR, 2017. [10] G. Sansoni, M. Trebeschi, F. Docchio, State-of-the-art and applications of 3d imaging sensors in industry, cultural heritage, medicine, and criminal investigation, Sensors 9 (2009) 568–601. [11] Y.-C. Chang, I. Detchev, A. Habib, A photogrammetric system for 3D reconstruction of a scoliotic torso, in: Proceedings of the ASPRS 2009 Annual Conference, 2008. [12] L. Barazzetti, M. Gianinetto, M. Scaioni, Automatic image-based 3D modeling for medical applications, in: Biomedical Engineering and Informatics (BMEI), 2012 5th International Conference on, IEEE, 2012, pp. 228–232. [13] P. Arias, C. Ordóñez, H. Lorenzo, J. Herraez, J. Armesto, Low-cost documentation of traditional agro-industrial buildings by close-range photogrammetry, Build. Environ. 42 (2007) 1817–1827. [14] A. Hosseininaveh, R. Yazdan, A. Karami, M. Moradi, F. Ghorbani, Clustering and selecting vantage images in a low-cost system for 3D reconstruction of texture-less objects, Measurement (2016). [15] F. Remondino, Heritage recording and 3D modeling with photogrammetry and 3D scanning, Remote Sens. 3 (2011) 1104–1138. [16] J.E. Chadwick, D.R. Haumann, R.E. Parent, Layered construction for deformable animated characters, in: ACM Siggraph Computer Graphics, ACM, 1989, pp. 243–252. [17] P. Volino, N.M. Thalmann, S. Jianhua, D. Thalmann, 1998. The Evolution of a 3D System for Simulating Deformable Clothes on Virtual Actors. MIRALab Copyright Information.. [18] N. D’apuzzo, A. Gruen, Recent advances in 3D full body scanning with applications to fashion and apparel, in: Optical 3-D Measurement Techniques IX, 2009. [19] J. Gu, T. Chang, I. Mak, S. Gopalsamy, H.C. Shen, M.M.-F. Yuen, A 3D reconstruction system for human body modeling, in: Modelling and Motion Capture Techniques for Virtual Environments, Springer, 1998. [20] S. Gray, In virtual fashion, IEEE Spectr. 35 (1998) 18–25. [21] N. Snavely, S. Seitz, R. Szeliski, Photo tourism: exploring image collections in 3d, acm transactions on graphics. ACM Transactions on Graphics 2 (2006). [22] C. Wu, Towards linear-time incremental structure from motion, in: 3D Vision-3DV 2013, 2013 International Conference on, IEEE, 2013, pp. 127–134. [23] C. Wu, Agarwal. S., B. Curless, S.M. Seitz, Multicore bundle adjustment, in: Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, IEEE, 2011, pp. 3057–3064. [24] Autodesk, RECAP: Create accurate 3D models with reality capture [Online]. Available: https://www.autodesk.co.uk/products/recap/overview [Accessed 09/05/2018 2018], 2018. [25] Agisoft, Agisoft PhotoScan [Online]. Available: http://www.agisoft.com/ [Accessed 09/05/2018 2018], 2018. [26] D. Tingdahl, L. Van Gool, A public system for image based 3d model generation, in: International conference on computer vision/computer graphics collaboration techniques and applications, Springer, 2011, pp. 262–273. [27] 2018. PRIMESENSE CARMINE 1.09 [Online]. Available: http://xtionprolive. com/primesense-carmine-1.09 [Accessed 09/05/2018 2018]. [28] Einscan, EinScan 3D Scanners [Online]. Available: https://www.einscan. com/ [Accessed 09/05/2018 2018], 2018.
[29] Sapirstein, Accurate measurement with photogrammetry at large sites, J. Archaeol. Sci. 66 (2016) 137–145. [30] L. Zheng, Q. Yuan, F. Sheng-Hong, L. Chang-Ru, W. Hai-Ren, Z. Ying-Xi, C. Jin-Quan, Y. Ji, High-precision photogrammetric surface figure measurements under cryogenic environment, Chinese Astronomy and Astrophysics 41 (2017) 136–148. [31] M. Hess, S. Robson, A.H. Ahmadabadian, A contest of sensors in close range 3D imaging: performance evaluation with a new metric test object, Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 40 (2014) 277. [32] V. Barrile, G. Bilotta, A. Nunnari, 3D modeling with photogrammetry by UAVs and model quality verification, ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci. 4 (2017) 129. [33] K. Wenzel, M. Rothermel, N. Haala, D. Fritsch, Sure–the ifp software for dense image matching, Photogramm. week (2013) 59–70. [34] M.Z. Brown, D. Burschka, G.D. Hager, Advances in computational stereo, IEEE Trans. Pattern Anal. Mach. Intell. 25 (2003) 993–1008. [35] J. Hafeez, S.-C. Kwon, S.-H. Lee, A. Hamacher, 3D surface reconstruction of smooth and textureless objects, in: Emerging Trends & Innovation in ICT (ICEI), 2017 International Conference on, IEEE, 2017, pp. 145–149. [36] A. Hosseininaveh, R. Yazdan, A. Karami, M. Moradi, F. Ghorbani, A low-cost and portable system for 3D reconstruction of texture-less objects, Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 40 (2015) 327. [37] A. Hosseininaveh, B. Sargeant, T. Erfani, S. Robson, M. Shortis, M. Hess, J. Boehm, Towards fully automatic reliable 3D acquisition: From designing imaging network to a complete and accurate point cloud, Robot. Auton. Syst. 62 (2014) 1197–1207. [38] M. Levoy, The Stanford Spherical Gantry, 2002. [39] J. Le Moigne, A.M. Waxman, Projected light grids for short range navigation of autonomous robots, in: Proc. 7th Int. Conf. Pattern Recognition, 1984, pp. 203–206. [40] J. LeMoigne, A.M. Waxman, Multi-Resolution Grid Patterns for Building Range Maps, Society of Manufacturing Engineers, 1985. [41] H. Morita, K. Yajima, S. Sakata, Reconstruction of surfaces of 3-d objects by m-array pattern projection method, in: Computer Vision. Second International Conference on, IEEE, 1988, pp. 468–473. [42] M. Maruyama, S. Abe, Range sensing by projecting multiple slits with random cuts, IEEE Trans. Pattern Anal. Mach. Intell. 15 (1993) 647–651. [43] K.L. Boyer, A.C. Kak, Color-encoded structured light for rapid active ranging, IEEE Trans. Pattern Anal. Mach. Intell. (1987) 14–28. [44] T.T. Nguyen, D.C. Slaughter, N. Max, J.N. Maloof, N. Sinha, Structured light-based 3D reconstruction system for plants, Sensors 15 (2015) 18587–18612. [45] T. Luhmann, S. Robson, S. Kyle, J. Boehm, Close-Range Photogrammetry and 3D Imaging, Walter de Gruyter, 2014. [46] J. Xu, Z.P. Fang, A.A. Malcolm, H. Wang, A robust close-range photogrammetric system for industrial metrology, in: Control, Automation, Robotics and Vision, 2002. ICARCV 2002. 7th International Conference on, IEEE, 2002, pp. 114–119. [47] U. Wijenayake, S.-I. Choi, S.-Y. Park, Automatic detection and decoding of photogrammetric coded targets, in: Electronics, Information and Communications (ICEIC), 2014 International Conference on, IEEE, 2014, pp. 1–2. [48] E. Upton, G. Halfacree, Raspberry Pi user guide, John Wiley & Sons, 2014. [49] T. Zinßer, J. Schmidt, H. Niemann, Point set registration with integrated scale estimation, in: International Conference on Pattern Recognition and Image Processing, 2005, pp. 116–119. [50] B. Bellekens, V. Spruyt, R. Berkvens, M. Weyn, A survey of rigid 3D pointcloud registration algorithms, in: AMBIENT 2014: the Fourth International Conference on Ambient Computing, Applications, Services and Technologies, August 24–28, 2014, Rome, Italy, 2014, pp. 8–13. [51] S.P. Lim, H. Haron, Surface reconstruction techniques: a review, Artif. Intell. Rev. 42 (2014) 59–78. [52] L. Gomes, O.R.P. Bellon, L. Silva, 3D reconstruction methods for digital preservation of cultural heritage: A survey, Pattern Recognit. Lett. 50 (2014) 3–14. [53] V. Mousavi, M. Khosravi, M. Ahmadi, N. Noori, S. Haghshenas, A. Hosseininaveh, M. Varshosaz, The performance evaluation of multi-image 3D reconstruction software with different sensors, Measurement 120 (2018) 1–10. [54] L. Agisoft, R. St Petersburg, Agisoft Photoscan, Professional Edition, 2014. [55] GOM. GOM ATOS Compact Scan [Online]. Available: https://www.gom.com/ [Accessed 09/05/2018 2018], 2018. [56] D. Girardeau-Montaut, Cloudcompare-open source project, in: OpenSource Project, 2011. [57] D. Eberly, Least Squares Fitting of Data, Magic Software, Chapel Hill, NC, 2000.
A. Hosseininaveh Ahmadabadian, A. Karami and R. Yazdan / Robotics and Autonomous Systems 117 (2019) 29–39 [58] GOM, GOM Inspect Software [Online]. Available: https://www.gom.com/ 3d-software/gom-inspect.html [Accessed 09/09/2018 2018], 2018. [59] Europac3d, Geomagic Design X [Online]. Available: https://europac3d.com/products/geomagic-design-x/?gclid= EAIaIQobChMI66Ke8ret3QIVjYWRCh2viQiDEAAYASAAEgLBoPD_BwE [Accessed 09/09/2018 2018], 2018.
Ali Hosseininaveh Ahmadabadian received his B. Sc. in Geodesy and Geomatics engineering from Islamic Azad University in 2003, an MSc in Photogrammetry from the K. N. Toosi University in 2007, and a PhD in Photogrammetry from University College London (UCL) in 2014. Since 2014, he has been an Assistant Professor working with the photogrammetry and remote sensing group, Geomatics Department, K. N. Toosi University. He is the author of more than 10 articles. His research interests include robotics and machine vision, especially next best view planning, photogrammetry, accurate 3D measurement, 3D modelling, and point cloud processing
39
Ali Karami received his B. Sc. degree in Geodesy and Geomatics engineering from Tabriz University, Iran in 2014. He received M. Sc. degree in Photogrammetry in 2016 from K. N. Toosi University of Technology. His research interests include computer vision, 3D modelling and image processing.
Rouhallah Yazdan received his B. Sc. degree in Geodesy and Geomatics engineering from K. N. Toosi University in 2005, Tehran, Iran and the MSc. degree in Photogrammetry from this university in 2009 and has been continuing his academic efforts as a Ph. D. student since 2014. His research interests include image processing and remote sensing, photogrammetry and 3D reconstruction.