- Email: [email protected]
4D modeling of soil surface during excavation using a solid-state 2D profilometer mounted on the arm of an excavator
Automation in Construction 112 (2020) 103112
Contents lists available at ScienceDirect
Automation in Construction journal homepage: www.elsevier.com/locate/autcon
4D modeling of soil surface during excavation using a solid-state 2D profilometer mounted on the arm of an excavator
T
⁎
Ilpo Niskanena,b, , Matti Immonena, Tomi Makkonena, Pekka Keränenc, Pekka Tynia, Lauri Hallmanc, Mikko Hiltunena, Tanja Kollia, Yrjö Louhisalmid, Juha Kostamovaarac, Rauno Heikkiläa a
Faculty of Technology, Structures and Construction Technology, University of Oulu, P.O. Box 4000, FI-90014 Oulu, Finland Public Works Research Institute, 1-6 Minamihara, Tsukuba, Ibaraki 300-2621, Japan c Faculty of Information Technology and Electrical Engineering, Circuits and Systems Research Unit, University of Oulu, P.O. Box 7300, FI-90014 Oulu, Finland d Faculty of Technology, Intelligent Machines and Systems, University of Oulu, P.O. Box 4200, FI-90014 Oulu, Finland b
A R T I C LE I N FO
A B S T R A C T
Keywords: Time-of-flight (TOF) Profilometer LIDAR Excavator 4D measurement system Construction
The aim of this research is to create a 4D point cloud map from a trench through a solid-state 2D profilometer. The profilometer is integrated with an 8.5-ton medium-size excavator's machine control system with pose calculation. Accordingly, the point cloud was transformed to a work coordinate system. We used a recently developed pulsed time-of-flight laser light detection and ranging profilometer, which makes possible simultaneous depth measurements to 256 directions in an angle range of ± 20° and measurement range from 1 m to 8 m with a frame rate of 25 frames per second under sunlight conditions (≈50 Klux). The analysis is based on a 4D map, which consists of 3D data (XYZ) and intensity information. The XYZ coordinates give the position of an object, and the intensity data can be used to roughly identify materials and recognise surface markings, such as texts. An analysis of the results shows that the detection accuracy of the profilometer is better than ± 10 mm. The main advantages of our method are accuracy, high update rate, compact size, real-time measurement and a construction without moving parts. Our technique has a great potential in construction applications, where accurate measurements of a surface shape are needed.
1. Introduction Over the past few years, construction machinery, mining and forest industry have experienced a vigorous development in the field of work machines, directly integrated digital processes and automation [1]. The control of earthmoving machinery is based on information models that provide the necessary information for the control systems of machines [2]. The control systems of machines offer the opportunity to improve machine utilisation, savings in fuel costs, improvements in machine maintenance, improvements in health and construction site safely [3–5]. Nowadays, the application of 3D technology is widely spread in different fields of the industry, including those deployed for robotic guidance and product profiling [6,7]. This achievement is due to lowcost and reliable optoelectronic technology and mass-production techniques, such as semiconductor lasers, miniature optics, precision electronics, signal processing and signal processing technologies [8]. The goal of the measurement of 3D surface profiles is to realise work quality, reduce material waste and ensure more efficient and productive ⁎
construction sites by connecting information. Traditionally, a surveyor prepares data for measuring an equipment and performs surveying in a construction site. In the new method, all measurements are performed by an automated machine guidance that uses positioning devices, such as the Global Positioning Systems (GPS). The use of 3D applications in the worksite can increase the performance of an individual task by typically 15% [9]. Currently, the most commonly used 3D navigation system of excavators is the inertial measurement unit (IMU) sensors embedded in the GPS or global navigation satellite system (GNSS). In the system, the position (XYZ) of the bucket of an excavator can be used to estimate the profile of a surface [10]. Bradley and Seward reported the first implementation of a measurement system that was equipped with GPS and pressure and displacement sensors in 1998 [3]. Trimble Inc. developed 3D visualize tools (Trimple Siteworks Positioning Systems) in the worksite, which is based on the GNSS antenna. Novatron Ltd. offers solutions for digitalizing earthmoving job sites and robotizing earthmoving machinery and also guides the XSitePro system. The XSitePro system is integrated into the open building information
Corresponding author at: Faculty of Technology, Structures and Construction Technology, University of Oulu, P.O. Box 4000, FI-90014 Oulu, Finland. E-mail address: Ilpo.Niskanen@oulu.fi (I. Niskanen).
https://doi.org/10.1016/j.autcon.2020.103112 Received 18 June 2019; Received in revised form 17 January 2020; Accepted 31 January 2020 0926-5805/ © 2020 Elsevier B.V. All rights reserved.
Automation in Construction 112 (2020) 103112
I. Niskanen, et al.
x ⎡ ⎤ P = ⎢ y ⎥, ⎣z⎦
modeling (BIM) process (Open Infra. Unfortunately, this method gives coordinates of the individual points from the surface. Only a few papers in the literature report other techniques to measure the profile of a surface from an excavator. Komatsu introduced a measurement system based on stereo cameras, GPS and drones. Large worksites can be quickly surveyed in 3D with the use of a drone equipped with a stereo camera. For example, 3D data help estimate stockpile volumes of materials and trench depths for real-time decision making. Fukui et al. developed a stereo camera-based method for autonomous rock pile excavation from an excavator [11]. The goal of stereo matching is to find out the corresponding matching points in the left and right images after calibration. However, matching these images with high accuracy presents challenges and needs an implementation of specific procedures [12]. Stereo matching offers good target separation, high resolution, density, repeatability, accuracy with no moving parts and simple setups at a very low cost. The measurement accuracy of the method varies depending upon the object to be measured and if the object does not present a rich surface texture [13,14]. Other limiting factors include low reliability in bad light conditions and calibration difficulties. Yamamoto et al. developed a 3D visualisation method in a worksite based on stereo cameras and light detection and ranging (LIDAR) [15]. The 3D image was obtained by combining the vertical plane of the 2D LIDAR to the rotation movement of the excavator. LIDAR systems base their operation on the measurement of the time of flight (TOF) of a pulsed light emitted from a laser diode until it is received by an emitter. Therefore, the measurement requires high-precision electronics to measure at the speed of light. The emission lines are in infrared ranges (905 or 1550 nm). The LIDAR can be classified according to their construction as rotary or solid state [16]. The advantages of LIDAR are real-time distance estimation, quick data collection, use in day and night time and good measurement accuracy [17]. The common limitations for all-optical techniques are extreme weather conditions, such as heavy rain, snow and fog, or risk of mechanical durability. Moreover, the signal quality and maximum range can be reduced due to the contamination of lenses, and the camera methods are sensitive to low or high illumination. Our goal is to develop an accurate surface profile measurement for an excavator. Method can enable us to prevent overly digging and eliminating excess material use and additional transportation costs at an earth construction site. A new kind of pulsed TOF 2D LIDAR profilometer to the 4D visualisation of the ground surface was used in this work. The profilometer enables simultaneous measurements to 256 directions in an angle range of ± 20° and a frame rate of 25 frames per second. Our flash 3D LIDAR sensor with no moving parts provides high precision with a sufficient frame rate and resolution especially in sunny conditions [18,19]. The profilometer was mounted in a plastic case on the Bobcat E85 excavator, which was equipped with Novatron's control system. The solutions suggested in this work are to use a solid-state 2D profilometer to measure the profile of a surface in the horizontal view and to use the movement of the excavator boom to inherently scan the view vertically to provide a 3D surface profile. The data analysis of a terrain is based on a 4D map consisting of 3D (XYZ) and intensity data. The XYZ coordinates determine the location of an object, and intensity information can be used to roughly identify material and surface markings.
(1)
where x, y and z are the individual components of the vector [20]. The transform matrix consists of a series of rotations (alignment of the x, y and z axes) and translations (movement of the origin). The position of the vector can be represented by the transformation matrix:
P= T1 × T2 × Pm = ⎡ RORI ⎢ ⎣ 000
⎡ 0 ⎤ P ⎤ ⎡ RScanner 0 ⎤ ⎢ 0 ⎥, 1 ⎦ ⎢ me ⎥ 1⎥ ⎦ ⎣ 000 ⎢ ⎣ 0 ⎥ ⎦
A
(2)
where T2 and T1 are the transformation matrices. Vector Pm is a measurement result of the scanner, vector AP presents the scanner position in the workspace coordinate system and RORI presents the orientation of the scanners, which is obtained through a socket network from the excavator in a quaternion form (q0, q1, q2, q3), where q0 is the quaternion w component, q1 is the quaternion x component, q2 is the quaternion y component and q3 is the quaternion z component. Transformation matrix P can be used to unambiguously describe the scanner/profilometer coordinate system relative to the base coordinate system. Rotation matrix RORI is a quaternion (q0, q1, q2, q3) expressed in a rotational matrix. 2 2 2 2 2(q1 q2 − q0 q3 ) 2(q1 q3 + q0 q2 ) ⎤ ⎡ q0 + q1 − q2 − q3 ⎥ ⎢ 2 2 2 2 q0 − q1 + q2 − q3 2(q2 q3 − q0 q1 ) ⎥, = ⎢ 2(q1 q2 + q0 q3 ) 2 2 2 2⎥ ⎢ 2(q q − q q ) 2(q2 q3 + q0 q1 ) q0 − q1 − q2 + q3 1 3 0 2 ⎦ ⎣ (3)
RORI
where q0, q1, q2, q3 are the quaternion components [21]. Thus, transformation matrix T1 can result from the following:
T1 = ⎡ RORI ⎢ ⎣ 000
⎡ q02 + q12 − q22 2(q1 q2 − q0 q3 ) 2(q1 q3 + q0 q2 ) ⎢ 2 ⎢ − q3 ⎢ 2 2 2 ⎢ 2(q1 q2 + q0 q3 ) q0 − q1 + q2 2(q2 q3 − q0 q1 ) A P⎤ = ⎢ − q32 1⎥ ⎦ ⎢ ⎢ 2 2 2 ⎢ 2(q1 q3 − q0 q2 ) 2(q2 q3 + q0 q1 ) q0 − q1 − q2 ⎢ + q32 ⎢ ⎢ 0 0 0 ⎣
,
px ⎤ ⎥ ⎥ ⎥ py ⎥ ⎥ ⎥ ⎥ pz ⎥ ⎥ ⎥ 1⎥ ⎦ (4)
where px, py and pz are the positions of the scanner in the workspace coordinate in excavator AP. The transformation matrix (T2) consists of the following parts in our case: 1
π T2 = RScanner = Roty ⎛ ⎞ × Rotx (ρ) × Roty (φoffset ), ⎝2⎠ 0
(5)
where ρ is the rotation around the scanner's x-axis and the present scanner's inner beam angle, which is ± 20°, and φoffset is the angle between the scanner and excavator's boom. The work uses a direct kinematics solution, which has the following transformation matrices:
2. Theory (excavator kinematics) In our system, a profilometer is attached to the arm of excavators. The arm is rotated to obtain 3D scans of the object. Coordinates for each frame of the profilometer is obtained by including its orientation and coordinates and distances it measured to the transformation matrix introduced in this section. Our approach is quaternion algebra. Fig. 1 shows the coordinate transformation frames in our profilometer system. The position of vector P with respect to the world coordinate system can be expressed in the standard form as follows: 2
⎡ cos ⎛ π ⎞ 0 sin ⎛ π ⎞ ⎤ ⎢ ⎝2⎠ ⎝2 ⎠⎥ π ⎥. 0 1 0 Roty ⎛ ⎞ = ⎢ ⎢ ⎥ ⎝2⎠ π π ⎥ ⎢ ⎛ ⎞ ⎛ ⎞ ⎢ − sin 2 0 cos 2 ⎥ ⎝ ⎠ ⎝ ⎠⎦ ⎣
(6)
⎡ cos(φoffset ) 0 sin(φoffset ) ⎤ ⎥. 0 1 0 Roty (φoffset ) = ⎢ ⎥ ⎢ ⎢ −sin(φoffset ) 0 cos(φoffset ) ⎥ ⎦ ⎣
(7)
Automation in Construction 112 (2020) 103112
I. Niskanen, et al.
Fig. 1. Coordinate transformation frames (translation and rotation).
3.2. KUKA industrial robot
0 0 ⎡1 ⎤ Rotx (ρ) = ⎢ 0 cos(ρ) −sin(ρ) ⎥. ⎢ ⎥ ⎣ 0 sin(ρ) cos(ρ) ⎦
The KUKA KR150 R2700 Extra robot was found to be an ideal motion generator for the first studies of the sensor system [22]. It is accurate due to factory calibration, a unidirectional absolute accuracy of 0.7 mm and repeatability of 0.06 mm. The absolute path accuracy of the linear motion (velocity: 1 m/s) is 1.2 mm and repeatability is 0.2 mm. The maximum payload is 150 kg and reach is 2696 mm. The robot was mounted on the floor and was connected to Beckhoff C6015 Industrial PC via the EL6695-1001 bridge. KUKA mXautomation software and Beckhoff EL6751 CANopen terminal were used Novatron's sensor system. The C6015 PC reads robot sensor data via EL6695 and Novatron's sensor data via EL6751. This allowed simultaneous measurements of robot motions with the robot and Novatron's sensors.
(8)
Multiplying the transform matrices by Eqs. (6)–(8) obtains the following:
⎡− cos (ρ) sin (φoffset ) sin (ρ) cos (ρ) cos (φoffset ) ⎢ sin (ρ) sin (φ ) cos (ρ) − sin (ρ) cos (φ ) offset offset T2 (φoffset , ρ) = ⎢ ⎢ − cos (φoffset ) − sin (φoffset ) 0 ⎢ 0 0 0 ⎣
0⎤ 0⎥ ⎥. 0⎥ ⎥ 1⎦
(9)
3. Materials and methods 3.1. Excavator with a control system
3.3. 2D pulsed TOF profilometer
The commercial excavator selected for the research was Bobcat E85 (8.5 tons), which has been modified for automation purposes. Fig. 2 shows a view of the system. An excavator is comprised of three planar implements connected through revolute joints, known as the boom, tipper and bucket. The boom, arm and bucket are controlled via Novatron's control system. The hydraulic system of the machine valves was retrofitted for a precise electrical control. The excavator was outfitted with electrohydraulic controls, a suite of sensors and on-board computer. The basis of the sensor system is the newly developed Novatron's IMU G2 sensors, which can work up to 200 Hz frequency. They provide accurate and precise positioning of the excavator, good enough for automatic movements. Novatron's G2-type sensors have a CAN bus interface. The use of CAN bus decreases the need for wiring, and all sensors can be set at the same bus. A laser profilometer was attached to the excavator of the boom, and all information were collected by the on-board computer.
The developed solid-state 2D pulsed TOF line profiling laser radar is based on transmitting short (~150 ps) laser pulses and measuring the TOF of the optical pulses reflected from the targets. Based on the measured TOFs and speed of light, the distances to the target points can be accurately resolved. The designed line profiler uses a custom quantum well laser diode [23] specifically designed for generating short optical pulses below a few hundred picoseconds at a relatively high pulse energy level of 1 nJ. Cylindrical transmitter optics are used to illuminate a fan-like profile below the excavator dipper with a field of illumination of approximately 38° × 0.3°. On the receiver side, a custom CMOS integrated circuit, with a single-photon avalanche diode (SPAD) detector array of 256 × 8 pixels and 256 TOF measurement channels [24,25], was utilised to detect backscattered photons with an angular resolution of approximately 0.15°. The receiver uses a 16 mm/ F1.4 objective to match the field of view of the pixel array with the field of illumination of the transmitter. Of note, the developed 2D profiler is
Fig. 2. Components of the University of Oulu's smart Bobcat E85 excavator. 3
Automation in Construction 112 (2020) 103112
I. Niskanen, et al.
Fig. 3. Complete line profiler with optomechanics attached.
fully solid state, i.e. a line scanner without any mechanically moving part, which paves the way for low power and cheap realisation in small size. A SPAD is basically a p-n junction reverse biased above the breakdown voltage, thus allowing individual absorbed photons to trigger a breakdown. The breakdown is then sensed and time-stamped with respect to the transmitted laser pulse using a time-to-digital converter. Although the time-stamping allows the detection of the time instances of individual breakdowns, a histogram of the time-stamps should be built by repeating the measurement several hundred times. From the histogram, the TOF can be determined with less uncertainty. Moreover, from the envelope of the histogram, the relative intensity of the optical echo can be resolved with high accuracy. The measurement range of the laser radar varies depending on the used frame rate, level of ambient light and reflectivity of the target. For example, in a typical office with an ambient light level and target reflectivity ranging from 0.1 to 1, the laser radar is capable of capturing line profiles reliably from a distance of approximately 20 m with a frame rate of 28 fps. Figs. 3 and 4 show photographs of the solid-state line profiler with and without optomechanics. The dimensions of the device are approximately 75 mm × 50 mm, and the device is powered and operated through a USB. The device was encased in an additional 3D-printed waterproof case for the measurements at the construction site.
Fig. 5. Original rectangular box for imaging (a) and laser profilometer installed on a robot (b).
4D measurement system was demonstrated that is able to map the terrain from the excavator. 4.1. Accuracy evaluation For this study, the measurement accuracy of the profilometer was estimated with a rectangular box (425 (W) × 569 (H) × 625 (L) mm3) with a license plate on it (Fig. 5a). The rectangular box was easy to make as a reference point cloud based on its dimensions. To determine dimensions for comparison, the sides of the box were measured by a mechanical caliper with a reading accuracy of 1 mm. The profilometer itself offers scans in two dimensions. The 3D point cloud was obtained by moving the profilometer in a vertical direction of 18 mm/s from the KUKA industrial robot, as shown in Fig. 5b). The profilometer was configured to send data at a frequency of 25 frames/s and an angle range of ± 20°. The distance between the profilometer and the floor was 2090 mm. The profilometer gave the distance values (hypotenuses) for each laser point (256) from a scan. By combining these values with the angular data of the profilometer, the x and y coordinates of each point can be calculated by trichromatic formulas. The configuring settings of the profilometer and calculation were implemented by the C++ program. The data of the robot (Z coordinate) and profilometer (X and Y coordinates) were combined into a 3D point cloud based on time labels using MATLAB software. MATLAB is standard industry software for numerical computing and offers compatibility with numerous other programming languages and hardware. It is widely used in applied mathematics and engineering. The number of points in the 3D cloud was approximately 263,000 when the total measurement time was 52 s. The experiment results are presented in Fig. 6. The profilometer could get not only the 2D data but also the intensity data. In Fig. 6a), the intensity map indicates the variability of the reflection from an object, which is coloured by using the intensity data. Fig. 6a) also shows that light objects reflected more light than the dark objects. The details of the rectangular box were formed from the 3D point cloud (Fig. 6b)). The sides of the box became oblique due to the sharp corner effects because the profilometer cannot scan behind corners. Finally, Fig. 6a) and b) were connected to Fig. 6c), where the colour describes the intensity of the reflection. In Fig. 6c), the dimensions of the rectangular box were determined using MATLAB software's image analysis tool. The mean and standard deviation of 10 observations of W, H and L were calculated as 417 ± 2, 623 ± 4 and 570 ± 2 mm, respectively. The image analysis results indicate that the profilometer has an accuracy better than 10 mm compared with the real dimensions of the box. The width and length of the box were slightly
4. Experiments and results In this work, first, the accuracy of the system was tested by measuring the dimensions of the rectangular box using the robot. Second, a
Fig. 4. Main internal components of the custom-designed line profiler showing the CMOS SPAD array-based receiver circuit and the laser diode for illumination. 4
Automation in Construction 112 (2020) 103112
I. Niskanen, et al.
Fig. 6. Experiment of rectangular box. (a) intensity map, (b) 3D point cloud and (c) 4D imaging.
Before the actual field tests, the profilometer location and orientation information were recorded by a tachometer (Leica TS12) to the control system of the excavator. The excavator dug a trench about 900 mm deep, 1600 mm wide and 1800 mm long. The text ‘XSite’ was written in paper and placed at the bottom of the trench (Fig. 8). The experiment was executed by moving the boom position from 110° to 70° above the trench. The location and intensity data were recorded along
underestimated, and the height has almost the same value. 4.2. Demonstration of the 4D system in the field tests Field tests were carried out on the earthmoving site with the help of Haukipudas vocational college unit. The profilometer was mounted with a plastic case to the excavator boom, as shown in Fig. 7.
Fig. 7. Profilometer mounted on the excavator.
Fig. 8. Trench dug with the excavator. 5
Automation in Construction 112 (2020) 103112
I. Niskanen, et al.
results are demonstrated in Fig. 9c), where the colour change describes a change of intensity. Moreover, it is observed in Fig. 9c) due to the white area found on the front of the trench. The reason is that the front edge of the trench blocked the views of the profilometer. The problem can be solved by the measurement of the trench from different angles. The method also provides information on the true shape of the trench and roughly identifies the materials of the earth, such as stones and sands. In addition, it can identify underground structures, such as power lines and pipes. Furthermore, the 4D map helps determine the amount of materials removed and the surface profile and aids in eliminating excess material use and transportation cost. 5. Discussion In this study, we implemented a low-cost 2D laser profilometer for capturing the 4D point cloud information from a terrain. The profilometer was integrated into the control of the excavator system, which enabled automatic data collection. In the construction site, the profilometer quickly gives results, which could be checked on the spot right away. In this study, the detection accuracy of the profilometer was estimated to be better than 10 mm in an indoor condition. The surface reflectivity has an effect on the measurement time since with lower reflectivity more laser shots needs to be collected for the needed signalto-noise ratio. The noise arises mostly from the random hits due to the background illumination. With a higher illumination level, the measurement time is longer since more results (laser shots) need to be collected for a valid SNR. In general, the level of background noise is minimized by a proper optical band-pass filter at the input of the receiver optics. In practice, this means that in high background illumination conditions (e.g. bright sunlight) the maximum range of the profiler is reduced at a given measurement rate (e.g. from 35 m to 5 m). In addition, materials, conditions and excavator censoring (GPS and IMU sensors) are important factors that impact the uncertainty in a 3D imaging system. The absolute error of the profilometer is not distance dependent as long as the SNR is high enough. It was measured in a calibrated track test and found to be within the limits of ± 2 mm (including the possible errors in the track calibration) in a range of 1–35 m [25]. The precision linearly decreased as a function of distance at a given measurement rate and is typically at the level of 1…10 mm within a range of 1–35 m at a measurement rate of 28 fpsec [25]. Ambient bright light, temperature, humidity and dust are factors in field conditions that cause uncertainty to measurements [26]. In addition to the geometric shape of the material, colour, moisture and reflectivity affect the measurement accuracy [27]. One source of error is related to the excavator GPS/GNSS positioning, which can reach a centimetre level of accuracy [28]. The error of the IMU sensor is less than 1.0°. The overall accuracy of the measurement system is approximately 20–30 mm. However, the measurement method achieves a sufficient precision for surface profiling at an earth construction site. Few studies in the literature have reported the measurement of the surface profile from an excavator. Yamamoto et al. [15] used LIDAR technique with a precision of 50 mm and measurement accuracy of 18 Hz. In addition, the applicability of the method for the excavator use is limited by the method based on a moving mirror. As a summary, the applicability of a 2D profilometer in an excavator is based on the following considerations: (a) The profilometer can be operated in a vertical measuring range between 1 and 8 m in sunlight conditions. It is a comprehensive tool for measuring the soil profile from an excavator. Moreover, the profilometer is located 3 m in height on the boom of excavator, and the profile line is 2 m long horizontally. (b) The results show that the high-density point cloud (approximately 6500 points/s) accurately describes the shapes of a trench. (c) The accuracy of our profilometer is better than 10 mm and the accuracy of the whole measurement system was estimated as 20–30 mm. The accuracy fulfils the need for many construction applications. (d) The 4D
Fig. 9. Experiment of the trench (a) 3D point cloud, (b) intensity map and (c) 4D imaging.
with time labels. This method allowed the connection of the values between the scanner and excavator. The profilometer was configured to send data at a frequency of 25 frames/s and an angle range of ± 20°. About 21 s was spent to scan to collect 131,600 laser points from the trench. The result was created as a 3D point cloud, where every point has an X, Y and Z coordinates. The Z coordinates were determined with the use of the IMU sensors and GPS. Moreover, the intensity value for each point was recorded. The 3D point cloud was transformed into a common coordinate system for the convenience of the result comparison using Eqs. (2)–(9). The results are shown in Fig. 9. Fig. 9a) presents the 3D point cloud of the trench computed by the excavator measurement system. A visual inspection of the 3D map indicated that the shape of the trench can be accurately visualised in Fig. 9a). The point cloud of the density is higher near the boom than far away from the boom. The intensity value of the trench changes depending on the material, as shown in Fig. 9b). White paper and snow give a good reflection intensity value compared with sand and soil. The profilometer installation place on the boom is not ideal because the bucket cylinder partly prevents the view of the profilometer, which is shown as the black line in Fig. 9b). The 4D map can be created by combining Fig. 9a) and b). The
6
Automation in Construction 112 (2020) 103112
I. Niskanen, et al.
map consists of 3D (XYZ) data, which gives the location of objects and intensity map and can be used as the fourth dimension to roughly identify material and surface markings. (e) The 2D profilometer has a compact size, with no moving parts, and apply real-time measurement. (f) The profilometer 3D point cloud (global coordinate system X,Y, and Z) can be easily integrated with other data sources, such as BIM, GIS or 3D machine guidance software. (g) A profilometer can also be used day and night time, but limitations are extreme weather conditions, such as heavy rain, snow and fog. Moreover, information can be lost in blind spots where objects in the front cut the sight.
doi.org/10.22260/ISARC2019/0134 (Accessed 22nd October 2019). [3] D.A. Bradley, D.W. Seward, The development, control and operation of an autonomous robotic excavator, J. Intell. Robot. Syst. 21 (1998) 73–97, https://doi.org/ 10.1023/A:1007932011161. [4] D. Schmidt, K. Berns, An autonomous excavator for landscaping tasks, ISR 2010 (41st International Symposium on Robotics) and ROBOTIK 2010 (6th German Conference on Robotics) Print ISBN: 978-3-8007-3273-9 https://ieeexplore.ieee. org/document/5756882, Accessed 22nd October 2019. [5] L. Pérez, Í. Rodríguez, N. Rodríguez, R. Usamentiaga, D.F. García, Robot guidance using machine vision techniques in industrial environments: a comparative review, Sensors 16 (3) (2016) 335, https://doi.org/10.3390/s16030335. [6] C. Emmer, K.-H. Glaesner, A. Pfouga, J. Stjepandić, Advances in 3D measurement data management for industry 4.0, Procedia Manufacturing 11 (2017) 1335–1342, https://doi.org/10.1016/j.promfg.2017.07.262. [7] W.R. Jamroz, R. Kruzelecky, E.I. Haddad, Applied Microphotonics, CRC Press Taylor & Francis Group, Boca Ratom, 2006, pp. 197–199, https://doi.org/10.1201/ 9781315221502. [8] J. Guehring, Dense 3-D surface acquisition by structured light using off-the-shelf components, Proceeding Volume SPIE 4309, Videometrics and Optical Methods for 3D Shape Measurement, 2000, https://doi.org/10.1117/12.410877 (Accessed 22nd October 2019). [9] V. Hokkanen, 3D excavator application in a future jobsite, 1st international Mobile Machine Control (MMC) conference, CAN in Automation (2013) 15–21. https:// www.can-cia.org/fileadmin/resources/documents/proceedings/2013_mmc_ hokkanen.pdf, Accessed 22nd October 2019. [10] S. Lee, M.-S. Kang, D.-S. Shin, C.-S. Han, Estimation with applications to dynamic status of an excavator without renovation, Conference: 29th International Symposium on Automation and Robotics in Construction, 2012, https://doi.org/10. 22260/ISARC2012/0093 Netherlands. (Accessed 22nd October 2019). [11] R. Fukui, T. Niho, M. Nakao, M. Uetake, Imitation-based control of automated ore excavator: improvement of autonomous excavation database quality using clustering and association analysis processes, Journal Advanced Robotics 31 (11) (2017) 595–606, https://doi.org/10.1080/01691864.2017.1297735. [12] A. O’Riordan, T. Newe, G. Dooly, D. Toal, Stereo vision sensing: review of existing systems, 12th International Conference on Sensing Technology (ICST), IEEE (2018) 178–184, https://doi.org/10.1109/ICSensT.2018.8603605. [13] C.S. Pan, Y. Kim, 3D terrain reconstruction of construction sites using a stereo camera, Autom. Constr. 64 (2016) 65–77, https://doi.org/10.1016/j.autcon.2015. 12.022. [14] M. Zhang, T. Cao, X. Zhao, Applying sensor-based technology to improve construction safety management, Sensors 17 (2017) 1841, https://doi.org/10.3390/ s17081841. [15] H. Yamamoto, M. Motek, H. Shao, K. Ootuki, Y. Yanagisawa, Y. Sakaida, A. Nozue, T. Yamaguchi, S. Yuta, Development of the autonomous hydraulic excavator prototype using 3-D information for motion planning and control, 2010 IEEE/SICE International Symposium on System Integration, IEEE (2010) 49–54, https://doi. org/10.1109/SII.2010.5708300. [16] F. Rosique, P.J. Navarro, C. Fernández, A. Padilla, A systematic review of perception system and simulators for autonomous vehicles research, Sensors 19 (3) (2019) 1–29, https://doi.org/10.3390/s19030648. [17] Y. Yang, X. Meng, M. Gao, Vision system of mobile robot combining binocular and depth cameras, Journal of Sensors 2017 (2017) 1–11, https://doi.org/10.1155/ 2017/4562934. [18] D. Bronzi, Y. Zou, F. Villa, S. Tisa, A. Tosi, F. Zappa, Automotive three-dimensional vision through a single-photon counting SPAD camera, IEEE Trans. Intell. Transp. Syst. 17 (3) (2006) 782–795, https://doi.org/10.1109/TITS.2015.2482601. [19] L.W. Hallman, S. Jahromi, J.-P. Jansson, J. Kostamovaara, On 2-D rangefinding using a ~1 nJ/~100 ps laser diode transmitter and a CMOS SPAD matrix, IEEE Photonics Journal 10 (4) (2018) 6802712, https://doi.org/10.1109/JPHOT.2018. 2819242. [20] A.J. Koivu, Fundamentals for Control of Robotic Manipulators, John Wiley & Sons, INC. New York, 1989, pp. 468. ISBN-13: 978-047185714. [21] H. Andrew, Visualizing quaternions, Elsevier, San Francisco, 2006, pp. 530. ISBN: 9780120884001. [22] KR Quantec, Kuka AG (2019). https://www.kuka.com/en-my/products/roboticssystems/industrial-robots/kr-quantec. Accessed 22nd October 2019. [23] J.M.T. Huikari, E.A. Avrutin, B.S. Ryvkin, J.J. Nissinen, J.T. Kostamovaara, Highenergy picosecond pulse generation by gain switching in asymmetric waveguide structure multiple quantum well lasers, IEEE Journal of Selected Topics Quantum Electron 21 (6) (2015) 189–194, https://doi.org/10.1109/JSTQE.2015.2416342. [24] P. Keränen, J. Kostamovaara, 256 x TDC array with cyclic interpolators based on calibration free 2x time amplifier, IEEE Transactions on Circuits and Systems I: Regular Papers 66 (2) (2019) 524–533, https://doi.org/10.1109/tcsi.2018. 2868242. [25] P. Keränen, J. Kostamovaara, 256x8 SPAD array with 256 column TDCs for a line profiling laser radar, IEEE Transactions on Circuits and Systems I: Regular Papers, 99 2019, pp. 1–12, , https://doi.org/10.1109/TCSI.2019.2923263. [26] J.-A. Beraldin, Basic theory on surface measurement uncertainty of 3D imaging systems, Proceedings of SPIE The International Society for Optical Engineering, San Jose, United State 2009, 72239-01. doi:https://doi.org/10.1117/12.804700. [27] J. Janßen, T. Medic, H. Kuhlmann, C. Holst, Decreasing the uncertainty of the target center estimation at terrestrial laser scanning by choosing best target design, Remote Sens. 11 (2019) 845, https://doi.org/10.3390/rs11070845. [28] C. Wenzhong, Principles of Modeling Uncertainties in Spatial Data and Spatial Analyses, CRC Press, Boca Raton, USA, 2009, pp. 432. ISBN 9781420059274.
6. Conclusions The obtained preliminary field test results show that the profilometer allows the inspection and 4D visualisation of the surface from the excavator. The working profilometer proved its feasibility and robustness, and the method offers enough measurement accuracy in relation to the measurement range in the worksite. Two main advantages of the method are identified to be robustness followed from the fact that no moving parts are used in the solid scanner and the relatively low price of the solid-state laser scanner. Further, the solid-state laser scanner used here has an accuracy of about 10 mm that is enough for most earthmoving works. Recorded 4D data can be documented from the trenching phase before the next work step fills the trench. The method saves measurement costs by collecting accurate actual data from the excavator itself. The simultaneous map can be distributed to other machines and management software on construction site. This 4D method is useful for applications where additional objects might be identified, such as texts, contours or materials, to some extent. The real-time surface profile measurements of the excavator would help the project manager to visualize the construction site at any time, understanding of project phases better and enabling good decision making. By linking the profile data to the BIM model, it would be possible to analyse the energy consumption of a construction site project and then find better solutions such as materials management. The using model would reduce costs and boosting efficiency at the construction site. In the future, our goal is to develop a fully autonomous excavator operating on a worksite according to the parameters given by the operator. In addition, excavator algorithms may include a safety layer preventing the excavator from undesired and possibly destructive movements in a site. A self-moving machine must always know control tasks. The most important facilitator of an autonomous control and machine learning, i.e. ‘machine intelligence’, is integrated into the information modeling process used. In addition, we will study more accurately the classification of earth materials based on intensity measurement in future works. Declaration of competing interest There are no conflicts to declare. Acknowledgment We gratefully acknowledge funding from the Business Finland, (4294/31/2018). References [1] P. Saeedi, P.D. Lawrence, D.G. Lowe, P. Jacobsen, D. Kusalovic, K. Ardron, P.H. Sorensen, An autonomous excavator with vision-based track-slippage control, IEEE Trans. Control Syst. Technol. 13 (1) (2005) 67–84, https://doi.org/10.1109/ TCST.2004.838551. [2] R. Heikkilä, T. Makkonen, I. Niskanen, M. Immonen, M. Hiltunen, T. Kolli, P. Tyni, Development of an earthmoving machinery autonomous excavator platform, 2019 Proceedings of the 36th International Symposium on Automation and Robotics in Construction (ISARC 2019), Banff, Alberta, Canada, 2019, pp. 1005–1010, , https://
7
Contents lists available at ScienceDirect
Automation in Construction journal homepage: www.elsevier.com/locate/autcon
4D modeling of soil surface during excavation using a solid-state 2D profilometer mounted on the arm of an excavator
T
⁎
Ilpo Niskanena,b, , Matti Immonena, Tomi Makkonena, Pekka Keränenc, Pekka Tynia, Lauri Hallmanc, Mikko Hiltunena, Tanja Kollia, Yrjö Louhisalmid, Juha Kostamovaarac, Rauno Heikkiläa a
Faculty of Technology, Structures and Construction Technology, University of Oulu, P.O. Box 4000, FI-90014 Oulu, Finland Public Works Research Institute, 1-6 Minamihara, Tsukuba, Ibaraki 300-2621, Japan c Faculty of Information Technology and Electrical Engineering, Circuits and Systems Research Unit, University of Oulu, P.O. Box 7300, FI-90014 Oulu, Finland d Faculty of Technology, Intelligent Machines and Systems, University of Oulu, P.O. Box 4200, FI-90014 Oulu, Finland b
A R T I C LE I N FO
A B S T R A C T
Keywords: Time-of-flight (TOF) Profilometer LIDAR Excavator 4D measurement system Construction
The aim of this research is to create a 4D point cloud map from a trench through a solid-state 2D profilometer. The profilometer is integrated with an 8.5-ton medium-size excavator's machine control system with pose calculation. Accordingly, the point cloud was transformed to a work coordinate system. We used a recently developed pulsed time-of-flight laser light detection and ranging profilometer, which makes possible simultaneous depth measurements to 256 directions in an angle range of ± 20° and measurement range from 1 m to 8 m with a frame rate of 25 frames per second under sunlight conditions (≈50 Klux). The analysis is based on a 4D map, which consists of 3D data (XYZ) and intensity information. The XYZ coordinates give the position of an object, and the intensity data can be used to roughly identify materials and recognise surface markings, such as texts. An analysis of the results shows that the detection accuracy of the profilometer is better than ± 10 mm. The main advantages of our method are accuracy, high update rate, compact size, real-time measurement and a construction without moving parts. Our technique has a great potential in construction applications, where accurate measurements of a surface shape are needed.
1. Introduction Over the past few years, construction machinery, mining and forest industry have experienced a vigorous development in the field of work machines, directly integrated digital processes and automation [1]. The control of earthmoving machinery is based on information models that provide the necessary information for the control systems of machines [2]. The control systems of machines offer the opportunity to improve machine utilisation, savings in fuel costs, improvements in machine maintenance, improvements in health and construction site safely [3–5]. Nowadays, the application of 3D technology is widely spread in different fields of the industry, including those deployed for robotic guidance and product profiling [6,7]. This achievement is due to lowcost and reliable optoelectronic technology and mass-production techniques, such as semiconductor lasers, miniature optics, precision electronics, signal processing and signal processing technologies [8]. The goal of the measurement of 3D surface profiles is to realise work quality, reduce material waste and ensure more efficient and productive ⁎
construction sites by connecting information. Traditionally, a surveyor prepares data for measuring an equipment and performs surveying in a construction site. In the new method, all measurements are performed by an automated machine guidance that uses positioning devices, such as the Global Positioning Systems (GPS). The use of 3D applications in the worksite can increase the performance of an individual task by typically 15% [9]. Currently, the most commonly used 3D navigation system of excavators is the inertial measurement unit (IMU) sensors embedded in the GPS or global navigation satellite system (GNSS). In the system, the position (XYZ) of the bucket of an excavator can be used to estimate the profile of a surface [10]. Bradley and Seward reported the first implementation of a measurement system that was equipped with GPS and pressure and displacement sensors in 1998 [3]. Trimble Inc. developed 3D visualize tools (Trimple Siteworks Positioning Systems) in the worksite, which is based on the GNSS antenna. Novatron Ltd. offers solutions for digitalizing earthmoving job sites and robotizing earthmoving machinery and also guides the XSitePro system. The XSitePro system is integrated into the open building information
Corresponding author at: Faculty of Technology, Structures and Construction Technology, University of Oulu, P.O. Box 4000, FI-90014 Oulu, Finland. E-mail address: Ilpo.Niskanen@oulu.fi (I. Niskanen).
https://doi.org/10.1016/j.autcon.2020.103112 Received 18 June 2019; Received in revised form 17 January 2020; Accepted 31 January 2020 0926-5805/ © 2020 Elsevier B.V. All rights reserved.
Automation in Construction 112 (2020) 103112
I. Niskanen, et al.
x ⎡ ⎤ P = ⎢ y ⎥, ⎣z⎦
modeling (BIM) process (Open Infra. Unfortunately, this method gives coordinates of the individual points from the surface. Only a few papers in the literature report other techniques to measure the profile of a surface from an excavator. Komatsu introduced a measurement system based on stereo cameras, GPS and drones. Large worksites can be quickly surveyed in 3D with the use of a drone equipped with a stereo camera. For example, 3D data help estimate stockpile volumes of materials and trench depths for real-time decision making. Fukui et al. developed a stereo camera-based method for autonomous rock pile excavation from an excavator [11]. The goal of stereo matching is to find out the corresponding matching points in the left and right images after calibration. However, matching these images with high accuracy presents challenges and needs an implementation of specific procedures [12]. Stereo matching offers good target separation, high resolution, density, repeatability, accuracy with no moving parts and simple setups at a very low cost. The measurement accuracy of the method varies depending upon the object to be measured and if the object does not present a rich surface texture [13,14]. Other limiting factors include low reliability in bad light conditions and calibration difficulties. Yamamoto et al. developed a 3D visualisation method in a worksite based on stereo cameras and light detection and ranging (LIDAR) [15]. The 3D image was obtained by combining the vertical plane of the 2D LIDAR to the rotation movement of the excavator. LIDAR systems base their operation on the measurement of the time of flight (TOF) of a pulsed light emitted from a laser diode until it is received by an emitter. Therefore, the measurement requires high-precision electronics to measure at the speed of light. The emission lines are in infrared ranges (905 or 1550 nm). The LIDAR can be classified according to their construction as rotary or solid state [16]. The advantages of LIDAR are real-time distance estimation, quick data collection, use in day and night time and good measurement accuracy [17]. The common limitations for all-optical techniques are extreme weather conditions, such as heavy rain, snow and fog, or risk of mechanical durability. Moreover, the signal quality and maximum range can be reduced due to the contamination of lenses, and the camera methods are sensitive to low or high illumination. Our goal is to develop an accurate surface profile measurement for an excavator. Method can enable us to prevent overly digging and eliminating excess material use and additional transportation costs at an earth construction site. A new kind of pulsed TOF 2D LIDAR profilometer to the 4D visualisation of the ground surface was used in this work. The profilometer enables simultaneous measurements to 256 directions in an angle range of ± 20° and a frame rate of 25 frames per second. Our flash 3D LIDAR sensor with no moving parts provides high precision with a sufficient frame rate and resolution especially in sunny conditions [18,19]. The profilometer was mounted in a plastic case on the Bobcat E85 excavator, which was equipped with Novatron's control system. The solutions suggested in this work are to use a solid-state 2D profilometer to measure the profile of a surface in the horizontal view and to use the movement of the excavator boom to inherently scan the view vertically to provide a 3D surface profile. The data analysis of a terrain is based on a 4D map consisting of 3D (XYZ) and intensity data. The XYZ coordinates determine the location of an object, and intensity information can be used to roughly identify material and surface markings.
(1)
where x, y and z are the individual components of the vector [20]. The transform matrix consists of a series of rotations (alignment of the x, y and z axes) and translations (movement of the origin). The position of the vector can be represented by the transformation matrix:
P= T1 × T2 × Pm = ⎡ RORI ⎢ ⎣ 000
⎡ 0 ⎤ P ⎤ ⎡ RScanner 0 ⎤ ⎢ 0 ⎥, 1 ⎦ ⎢ me ⎥ 1⎥ ⎦ ⎣ 000 ⎢ ⎣ 0 ⎥ ⎦
A
(2)
where T2 and T1 are the transformation matrices. Vector Pm is a measurement result of the scanner, vector AP presents the scanner position in the workspace coordinate system and RORI presents the orientation of the scanners, which is obtained through a socket network from the excavator in a quaternion form (q0, q1, q2, q3), where q0 is the quaternion w component, q1 is the quaternion x component, q2 is the quaternion y component and q3 is the quaternion z component. Transformation matrix P can be used to unambiguously describe the scanner/profilometer coordinate system relative to the base coordinate system. Rotation matrix RORI is a quaternion (q0, q1, q2, q3) expressed in a rotational matrix. 2 2 2 2 2(q1 q2 − q0 q3 ) 2(q1 q3 + q0 q2 ) ⎤ ⎡ q0 + q1 − q2 − q3 ⎥ ⎢ 2 2 2 2 q0 − q1 + q2 − q3 2(q2 q3 − q0 q1 ) ⎥, = ⎢ 2(q1 q2 + q0 q3 ) 2 2 2 2⎥ ⎢ 2(q q − q q ) 2(q2 q3 + q0 q1 ) q0 − q1 − q2 + q3 1 3 0 2 ⎦ ⎣ (3)
RORI
where q0, q1, q2, q3 are the quaternion components [21]. Thus, transformation matrix T1 can result from the following:
T1 = ⎡ RORI ⎢ ⎣ 000
⎡ q02 + q12 − q22 2(q1 q2 − q0 q3 ) 2(q1 q3 + q0 q2 ) ⎢ 2 ⎢ − q3 ⎢ 2 2 2 ⎢ 2(q1 q2 + q0 q3 ) q0 − q1 + q2 2(q2 q3 − q0 q1 ) A P⎤ = ⎢ − q32 1⎥ ⎦ ⎢ ⎢ 2 2 2 ⎢ 2(q1 q3 − q0 q2 ) 2(q2 q3 + q0 q1 ) q0 − q1 − q2 ⎢ + q32 ⎢ ⎢ 0 0 0 ⎣
,
px ⎤ ⎥ ⎥ ⎥ py ⎥ ⎥ ⎥ ⎥ pz ⎥ ⎥ ⎥ 1⎥ ⎦ (4)
where px, py and pz are the positions of the scanner in the workspace coordinate in excavator AP. The transformation matrix (T2) consists of the following parts in our case: 1
π T2 = RScanner = Roty ⎛ ⎞ × Rotx (ρ) × Roty (φoffset ), ⎝2⎠ 0
(5)
where ρ is the rotation around the scanner's x-axis and the present scanner's inner beam angle, which is ± 20°, and φoffset is the angle between the scanner and excavator's boom. The work uses a direct kinematics solution, which has the following transformation matrices:
2. Theory (excavator kinematics) In our system, a profilometer is attached to the arm of excavators. The arm is rotated to obtain 3D scans of the object. Coordinates for each frame of the profilometer is obtained by including its orientation and coordinates and distances it measured to the transformation matrix introduced in this section. Our approach is quaternion algebra. Fig. 1 shows the coordinate transformation frames in our profilometer system. The position of vector P with respect to the world coordinate system can be expressed in the standard form as follows: 2
⎡ cos ⎛ π ⎞ 0 sin ⎛ π ⎞ ⎤ ⎢ ⎝2⎠ ⎝2 ⎠⎥ π ⎥. 0 1 0 Roty ⎛ ⎞ = ⎢ ⎢ ⎥ ⎝2⎠ π π ⎥ ⎢ ⎛ ⎞ ⎛ ⎞ ⎢ − sin 2 0 cos 2 ⎥ ⎝ ⎠ ⎝ ⎠⎦ ⎣
(6)
⎡ cos(φoffset ) 0 sin(φoffset ) ⎤ ⎥. 0 1 0 Roty (φoffset ) = ⎢ ⎥ ⎢ ⎢ −sin(φoffset ) 0 cos(φoffset ) ⎥ ⎦ ⎣
(7)
Automation in Construction 112 (2020) 103112
I. Niskanen, et al.
Fig. 1. Coordinate transformation frames (translation and rotation).
3.2. KUKA industrial robot
0 0 ⎡1 ⎤ Rotx (ρ) = ⎢ 0 cos(ρ) −sin(ρ) ⎥. ⎢ ⎥ ⎣ 0 sin(ρ) cos(ρ) ⎦
The KUKA KR150 R2700 Extra robot was found to be an ideal motion generator for the first studies of the sensor system [22]. It is accurate due to factory calibration, a unidirectional absolute accuracy of 0.7 mm and repeatability of 0.06 mm. The absolute path accuracy of the linear motion (velocity: 1 m/s) is 1.2 mm and repeatability is 0.2 mm. The maximum payload is 150 kg and reach is 2696 mm. The robot was mounted on the floor and was connected to Beckhoff C6015 Industrial PC via the EL6695-1001 bridge. KUKA mXautomation software and Beckhoff EL6751 CANopen terminal were used Novatron's sensor system. The C6015 PC reads robot sensor data via EL6695 and Novatron's sensor data via EL6751. This allowed simultaneous measurements of robot motions with the robot and Novatron's sensors.
(8)
Multiplying the transform matrices by Eqs. (6)–(8) obtains the following:
⎡− cos (ρ) sin (φoffset ) sin (ρ) cos (ρ) cos (φoffset ) ⎢ sin (ρ) sin (φ ) cos (ρ) − sin (ρ) cos (φ ) offset offset T2 (φoffset , ρ) = ⎢ ⎢ − cos (φoffset ) − sin (φoffset ) 0 ⎢ 0 0 0 ⎣
0⎤ 0⎥ ⎥. 0⎥ ⎥ 1⎦
(9)
3. Materials and methods 3.1. Excavator with a control system
3.3. 2D pulsed TOF profilometer
The commercial excavator selected for the research was Bobcat E85 (8.5 tons), which has been modified for automation purposes. Fig. 2 shows a view of the system. An excavator is comprised of three planar implements connected through revolute joints, known as the boom, tipper and bucket. The boom, arm and bucket are controlled via Novatron's control system. The hydraulic system of the machine valves was retrofitted for a precise electrical control. The excavator was outfitted with electrohydraulic controls, a suite of sensors and on-board computer. The basis of the sensor system is the newly developed Novatron's IMU G2 sensors, which can work up to 200 Hz frequency. They provide accurate and precise positioning of the excavator, good enough for automatic movements. Novatron's G2-type sensors have a CAN bus interface. The use of CAN bus decreases the need for wiring, and all sensors can be set at the same bus. A laser profilometer was attached to the excavator of the boom, and all information were collected by the on-board computer.
The developed solid-state 2D pulsed TOF line profiling laser radar is based on transmitting short (~150 ps) laser pulses and measuring the TOF of the optical pulses reflected from the targets. Based on the measured TOFs and speed of light, the distances to the target points can be accurately resolved. The designed line profiler uses a custom quantum well laser diode [23] specifically designed for generating short optical pulses below a few hundred picoseconds at a relatively high pulse energy level of 1 nJ. Cylindrical transmitter optics are used to illuminate a fan-like profile below the excavator dipper with a field of illumination of approximately 38° × 0.3°. On the receiver side, a custom CMOS integrated circuit, with a single-photon avalanche diode (SPAD) detector array of 256 × 8 pixels and 256 TOF measurement channels [24,25], was utilised to detect backscattered photons with an angular resolution of approximately 0.15°. The receiver uses a 16 mm/ F1.4 objective to match the field of view of the pixel array with the field of illumination of the transmitter. Of note, the developed 2D profiler is
Fig. 2. Components of the University of Oulu's smart Bobcat E85 excavator. 3
Automation in Construction 112 (2020) 103112
I. Niskanen, et al.
Fig. 3. Complete line profiler with optomechanics attached.
fully solid state, i.e. a line scanner without any mechanically moving part, which paves the way for low power and cheap realisation in small size. A SPAD is basically a p-n junction reverse biased above the breakdown voltage, thus allowing individual absorbed photons to trigger a breakdown. The breakdown is then sensed and time-stamped with respect to the transmitted laser pulse using a time-to-digital converter. Although the time-stamping allows the detection of the time instances of individual breakdowns, a histogram of the time-stamps should be built by repeating the measurement several hundred times. From the histogram, the TOF can be determined with less uncertainty. Moreover, from the envelope of the histogram, the relative intensity of the optical echo can be resolved with high accuracy. The measurement range of the laser radar varies depending on the used frame rate, level of ambient light and reflectivity of the target. For example, in a typical office with an ambient light level and target reflectivity ranging from 0.1 to 1, the laser radar is capable of capturing line profiles reliably from a distance of approximately 20 m with a frame rate of 28 fps. Figs. 3 and 4 show photographs of the solid-state line profiler with and without optomechanics. The dimensions of the device are approximately 75 mm × 50 mm, and the device is powered and operated through a USB. The device was encased in an additional 3D-printed waterproof case for the measurements at the construction site.
Fig. 5. Original rectangular box for imaging (a) and laser profilometer installed on a robot (b).
4D measurement system was demonstrated that is able to map the terrain from the excavator. 4.1. Accuracy evaluation For this study, the measurement accuracy of the profilometer was estimated with a rectangular box (425 (W) × 569 (H) × 625 (L) mm3) with a license plate on it (Fig. 5a). The rectangular box was easy to make as a reference point cloud based on its dimensions. To determine dimensions for comparison, the sides of the box were measured by a mechanical caliper with a reading accuracy of 1 mm. The profilometer itself offers scans in two dimensions. The 3D point cloud was obtained by moving the profilometer in a vertical direction of 18 mm/s from the KUKA industrial robot, as shown in Fig. 5b). The profilometer was configured to send data at a frequency of 25 frames/s and an angle range of ± 20°. The distance between the profilometer and the floor was 2090 mm. The profilometer gave the distance values (hypotenuses) for each laser point (256) from a scan. By combining these values with the angular data of the profilometer, the x and y coordinates of each point can be calculated by trichromatic formulas. The configuring settings of the profilometer and calculation were implemented by the C++ program. The data of the robot (Z coordinate) and profilometer (X and Y coordinates) were combined into a 3D point cloud based on time labels using MATLAB software. MATLAB is standard industry software for numerical computing and offers compatibility with numerous other programming languages and hardware. It is widely used in applied mathematics and engineering. The number of points in the 3D cloud was approximately 263,000 when the total measurement time was 52 s. The experiment results are presented in Fig. 6. The profilometer could get not only the 2D data but also the intensity data. In Fig. 6a), the intensity map indicates the variability of the reflection from an object, which is coloured by using the intensity data. Fig. 6a) also shows that light objects reflected more light than the dark objects. The details of the rectangular box were formed from the 3D point cloud (Fig. 6b)). The sides of the box became oblique due to the sharp corner effects because the profilometer cannot scan behind corners. Finally, Fig. 6a) and b) were connected to Fig. 6c), where the colour describes the intensity of the reflection. In Fig. 6c), the dimensions of the rectangular box were determined using MATLAB software's image analysis tool. The mean and standard deviation of 10 observations of W, H and L were calculated as 417 ± 2, 623 ± 4 and 570 ± 2 mm, respectively. The image analysis results indicate that the profilometer has an accuracy better than 10 mm compared with the real dimensions of the box. The width and length of the box were slightly
4. Experiments and results In this work, first, the accuracy of the system was tested by measuring the dimensions of the rectangular box using the robot. Second, a
Fig. 4. Main internal components of the custom-designed line profiler showing the CMOS SPAD array-based receiver circuit and the laser diode for illumination. 4
Automation in Construction 112 (2020) 103112
I. Niskanen, et al.
Fig. 6. Experiment of rectangular box. (a) intensity map, (b) 3D point cloud and (c) 4D imaging.
Before the actual field tests, the profilometer location and orientation information were recorded by a tachometer (Leica TS12) to the control system of the excavator. The excavator dug a trench about 900 mm deep, 1600 mm wide and 1800 mm long. The text ‘XSite’ was written in paper and placed at the bottom of the trench (Fig. 8). The experiment was executed by moving the boom position from 110° to 70° above the trench. The location and intensity data were recorded along
underestimated, and the height has almost the same value. 4.2. Demonstration of the 4D system in the field tests Field tests were carried out on the earthmoving site with the help of Haukipudas vocational college unit. The profilometer was mounted with a plastic case to the excavator boom, as shown in Fig. 7.
Fig. 7. Profilometer mounted on the excavator.
Fig. 8. Trench dug with the excavator. 5
Automation in Construction 112 (2020) 103112
I. Niskanen, et al.
results are demonstrated in Fig. 9c), where the colour change describes a change of intensity. Moreover, it is observed in Fig. 9c) due to the white area found on the front of the trench. The reason is that the front edge of the trench blocked the views of the profilometer. The problem can be solved by the measurement of the trench from different angles. The method also provides information on the true shape of the trench and roughly identifies the materials of the earth, such as stones and sands. In addition, it can identify underground structures, such as power lines and pipes. Furthermore, the 4D map helps determine the amount of materials removed and the surface profile and aids in eliminating excess material use and transportation cost. 5. Discussion In this study, we implemented a low-cost 2D laser profilometer for capturing the 4D point cloud information from a terrain. The profilometer was integrated into the control of the excavator system, which enabled automatic data collection. In the construction site, the profilometer quickly gives results, which could be checked on the spot right away. In this study, the detection accuracy of the profilometer was estimated to be better than 10 mm in an indoor condition. The surface reflectivity has an effect on the measurement time since with lower reflectivity more laser shots needs to be collected for the needed signalto-noise ratio. The noise arises mostly from the random hits due to the background illumination. With a higher illumination level, the measurement time is longer since more results (laser shots) need to be collected for a valid SNR. In general, the level of background noise is minimized by a proper optical band-pass filter at the input of the receiver optics. In practice, this means that in high background illumination conditions (e.g. bright sunlight) the maximum range of the profiler is reduced at a given measurement rate (e.g. from 35 m to 5 m). In addition, materials, conditions and excavator censoring (GPS and IMU sensors) are important factors that impact the uncertainty in a 3D imaging system. The absolute error of the profilometer is not distance dependent as long as the SNR is high enough. It was measured in a calibrated track test and found to be within the limits of ± 2 mm (including the possible errors in the track calibration) in a range of 1–35 m [25]. The precision linearly decreased as a function of distance at a given measurement rate and is typically at the level of 1…10 mm within a range of 1–35 m at a measurement rate of 28 fpsec [25]. Ambient bright light, temperature, humidity and dust are factors in field conditions that cause uncertainty to measurements [26]. In addition to the geometric shape of the material, colour, moisture and reflectivity affect the measurement accuracy [27]. One source of error is related to the excavator GPS/GNSS positioning, which can reach a centimetre level of accuracy [28]. The error of the IMU sensor is less than 1.0°. The overall accuracy of the measurement system is approximately 20–30 mm. However, the measurement method achieves a sufficient precision for surface profiling at an earth construction site. Few studies in the literature have reported the measurement of the surface profile from an excavator. Yamamoto et al. [15] used LIDAR technique with a precision of 50 mm and measurement accuracy of 18 Hz. In addition, the applicability of the method for the excavator use is limited by the method based on a moving mirror. As a summary, the applicability of a 2D profilometer in an excavator is based on the following considerations: (a) The profilometer can be operated in a vertical measuring range between 1 and 8 m in sunlight conditions. It is a comprehensive tool for measuring the soil profile from an excavator. Moreover, the profilometer is located 3 m in height on the boom of excavator, and the profile line is 2 m long horizontally. (b) The results show that the high-density point cloud (approximately 6500 points/s) accurately describes the shapes of a trench. (c) The accuracy of our profilometer is better than 10 mm and the accuracy of the whole measurement system was estimated as 20–30 mm. The accuracy fulfils the need for many construction applications. (d) The 4D
Fig. 9. Experiment of the trench (a) 3D point cloud, (b) intensity map and (c) 4D imaging.
with time labels. This method allowed the connection of the values between the scanner and excavator. The profilometer was configured to send data at a frequency of 25 frames/s and an angle range of ± 20°. About 21 s was spent to scan to collect 131,600 laser points from the trench. The result was created as a 3D point cloud, where every point has an X, Y and Z coordinates. The Z coordinates were determined with the use of the IMU sensors and GPS. Moreover, the intensity value for each point was recorded. The 3D point cloud was transformed into a common coordinate system for the convenience of the result comparison using Eqs. (2)–(9). The results are shown in Fig. 9. Fig. 9a) presents the 3D point cloud of the trench computed by the excavator measurement system. A visual inspection of the 3D map indicated that the shape of the trench can be accurately visualised in Fig. 9a). The point cloud of the density is higher near the boom than far away from the boom. The intensity value of the trench changes depending on the material, as shown in Fig. 9b). White paper and snow give a good reflection intensity value compared with sand and soil. The profilometer installation place on the boom is not ideal because the bucket cylinder partly prevents the view of the profilometer, which is shown as the black line in Fig. 9b). The 4D map can be created by combining Fig. 9a) and b). The
6
Automation in Construction 112 (2020) 103112
I. Niskanen, et al.
map consists of 3D (XYZ) data, which gives the location of objects and intensity map and can be used as the fourth dimension to roughly identify material and surface markings. (e) The 2D profilometer has a compact size, with no moving parts, and apply real-time measurement. (f) The profilometer 3D point cloud (global coordinate system X,Y, and Z) can be easily integrated with other data sources, such as BIM, GIS or 3D machine guidance software. (g) A profilometer can also be used day and night time, but limitations are extreme weather conditions, such as heavy rain, snow and fog. Moreover, information can be lost in blind spots where objects in the front cut the sight.
doi.org/10.22260/ISARC2019/0134 (Accessed 22nd October 2019). [3] D.A. Bradley, D.W. Seward, The development, control and operation of an autonomous robotic excavator, J. Intell. Robot. Syst. 21 (1998) 73–97, https://doi.org/ 10.1023/A:1007932011161. [4] D. Schmidt, K. Berns, An autonomous excavator for landscaping tasks, ISR 2010 (41st International Symposium on Robotics) and ROBOTIK 2010 (6th German Conference on Robotics) Print ISBN: 978-3-8007-3273-9 https://ieeexplore.ieee. org/document/5756882, Accessed 22nd October 2019. [5] L. Pérez, Í. Rodríguez, N. Rodríguez, R. Usamentiaga, D.F. García, Robot guidance using machine vision techniques in industrial environments: a comparative review, Sensors 16 (3) (2016) 335, https://doi.org/10.3390/s16030335. [6] C. Emmer, K.-H. Glaesner, A. Pfouga, J. Stjepandić, Advances in 3D measurement data management for industry 4.0, Procedia Manufacturing 11 (2017) 1335–1342, https://doi.org/10.1016/j.promfg.2017.07.262. [7] W.R. Jamroz, R. Kruzelecky, E.I. Haddad, Applied Microphotonics, CRC Press Taylor & Francis Group, Boca Ratom, 2006, pp. 197–199, https://doi.org/10.1201/ 9781315221502. [8] J. Guehring, Dense 3-D surface acquisition by structured light using off-the-shelf components, Proceeding Volume SPIE 4309, Videometrics and Optical Methods for 3D Shape Measurement, 2000, https://doi.org/10.1117/12.410877 (Accessed 22nd October 2019). [9] V. Hokkanen, 3D excavator application in a future jobsite, 1st international Mobile Machine Control (MMC) conference, CAN in Automation (2013) 15–21. https:// www.can-cia.org/fileadmin/resources/documents/proceedings/2013_mmc_ hokkanen.pdf, Accessed 22nd October 2019. [10] S. Lee, M.-S. Kang, D.-S. Shin, C.-S. Han, Estimation with applications to dynamic status of an excavator without renovation, Conference: 29th International Symposium on Automation and Robotics in Construction, 2012, https://doi.org/10. 22260/ISARC2012/0093 Netherlands. (Accessed 22nd October 2019). [11] R. Fukui, T. Niho, M. Nakao, M. Uetake, Imitation-based control of automated ore excavator: improvement of autonomous excavation database quality using clustering and association analysis processes, Journal Advanced Robotics 31 (11) (2017) 595–606, https://doi.org/10.1080/01691864.2017.1297735. [12] A. O’Riordan, T. Newe, G. Dooly, D. Toal, Stereo vision sensing: review of existing systems, 12th International Conference on Sensing Technology (ICST), IEEE (2018) 178–184, https://doi.org/10.1109/ICSensT.2018.8603605. [13] C.S. Pan, Y. Kim, 3D terrain reconstruction of construction sites using a stereo camera, Autom. Constr. 64 (2016) 65–77, https://doi.org/10.1016/j.autcon.2015. 12.022. [14] M. Zhang, T. Cao, X. Zhao, Applying sensor-based technology to improve construction safety management, Sensors 17 (2017) 1841, https://doi.org/10.3390/ s17081841. [15] H. Yamamoto, M. Motek, H. Shao, K. Ootuki, Y. Yanagisawa, Y. Sakaida, A. Nozue, T. Yamaguchi, S. Yuta, Development of the autonomous hydraulic excavator prototype using 3-D information for motion planning and control, 2010 IEEE/SICE International Symposium on System Integration, IEEE (2010) 49–54, https://doi. org/10.1109/SII.2010.5708300. [16] F. Rosique, P.J. Navarro, C. Fernández, A. Padilla, A systematic review of perception system and simulators for autonomous vehicles research, Sensors 19 (3) (2019) 1–29, https://doi.org/10.3390/s19030648. [17] Y. Yang, X. Meng, M. Gao, Vision system of mobile robot combining binocular and depth cameras, Journal of Sensors 2017 (2017) 1–11, https://doi.org/10.1155/ 2017/4562934. [18] D. Bronzi, Y. Zou, F. Villa, S. Tisa, A. Tosi, F. Zappa, Automotive three-dimensional vision through a single-photon counting SPAD camera, IEEE Trans. Intell. Transp. Syst. 17 (3) (2006) 782–795, https://doi.org/10.1109/TITS.2015.2482601. [19] L.W. Hallman, S. Jahromi, J.-P. Jansson, J. Kostamovaara, On 2-D rangefinding using a ~1 nJ/~100 ps laser diode transmitter and a CMOS SPAD matrix, IEEE Photonics Journal 10 (4) (2018) 6802712, https://doi.org/10.1109/JPHOT.2018. 2819242. [20] A.J. Koivu, Fundamentals for Control of Robotic Manipulators, John Wiley & Sons, INC. New York, 1989, pp. 468. ISBN-13: 978-047185714. [21] H. Andrew, Visualizing quaternions, Elsevier, San Francisco, 2006, pp. 530. ISBN: 9780120884001. [22] KR Quantec, Kuka AG (2019). https://www.kuka.com/en-my/products/roboticssystems/industrial-robots/kr-quantec. Accessed 22nd October 2019. [23] J.M.T. Huikari, E.A. Avrutin, B.S. Ryvkin, J.J. Nissinen, J.T. Kostamovaara, Highenergy picosecond pulse generation by gain switching in asymmetric waveguide structure multiple quantum well lasers, IEEE Journal of Selected Topics Quantum Electron 21 (6) (2015) 189–194, https://doi.org/10.1109/JSTQE.2015.2416342. [24] P. Keränen, J. Kostamovaara, 256 x TDC array with cyclic interpolators based on calibration free 2x time amplifier, IEEE Transactions on Circuits and Systems I: Regular Papers 66 (2) (2019) 524–533, https://doi.org/10.1109/tcsi.2018. 2868242. [25] P. Keränen, J. Kostamovaara, 256x8 SPAD array with 256 column TDCs for a line profiling laser radar, IEEE Transactions on Circuits and Systems I: Regular Papers, 99 2019, pp. 1–12, , https://doi.org/10.1109/TCSI.2019.2923263. [26] J.-A. Beraldin, Basic theory on surface measurement uncertainty of 3D imaging systems, Proceedings of SPIE The International Society for Optical Engineering, San Jose, United State 2009, 72239-01. doi:https://doi.org/10.1117/12.804700. [27] J. Janßen, T. Medic, H. Kuhlmann, C. Holst, Decreasing the uncertainty of the target center estimation at terrestrial laser scanning by choosing best target design, Remote Sens. 11 (2019) 845, https://doi.org/10.3390/rs11070845. [28] C. Wenzhong, Principles of Modeling Uncertainties in Spatial Data and Spatial Analyses, CRC Press, Boca Raton, USA, 2009, pp. 432. ISBN 9781420059274.
6. Conclusions The obtained preliminary field test results show that the profilometer allows the inspection and 4D visualisation of the surface from the excavator. The working profilometer proved its feasibility and robustness, and the method offers enough measurement accuracy in relation to the measurement range in the worksite. Two main advantages of the method are identified to be robustness followed from the fact that no moving parts are used in the solid scanner and the relatively low price of the solid-state laser scanner. Further, the solid-state laser scanner used here has an accuracy of about 10 mm that is enough for most earthmoving works. Recorded 4D data can be documented from the trenching phase before the next work step fills the trench. The method saves measurement costs by collecting accurate actual data from the excavator itself. The simultaneous map can be distributed to other machines and management software on construction site. This 4D method is useful for applications where additional objects might be identified, such as texts, contours or materials, to some extent. The real-time surface profile measurements of the excavator would help the project manager to visualize the construction site at any time, understanding of project phases better and enabling good decision making. By linking the profile data to the BIM model, it would be possible to analyse the energy consumption of a construction site project and then find better solutions such as materials management. The using model would reduce costs and boosting efficiency at the construction site. In the future, our goal is to develop a fully autonomous excavator operating on a worksite according to the parameters given by the operator. In addition, excavator algorithms may include a safety layer preventing the excavator from undesired and possibly destructive movements in a site. A self-moving machine must always know control tasks. The most important facilitator of an autonomous control and machine learning, i.e. ‘machine intelligence’, is integrated into the information modeling process used. In addition, we will study more accurately the classification of earth materials based on intensity measurement in future works. Declaration of competing interest There are no conflicts to declare. Acknowledgment We gratefully acknowledge funding from the Business Finland, (4294/31/2018). References [1] P. Saeedi, P.D. Lawrence, D.G. Lowe, P. Jacobsen, D. Kusalovic, K. Ardron, P.H. Sorensen, An autonomous excavator with vision-based track-slippage control, IEEE Trans. Control Syst. Technol. 13 (1) (2005) 67–84, https://doi.org/10.1109/ TCST.2004.838551. [2] R. Heikkilä, T. Makkonen, I. Niskanen, M. Immonen, M. Hiltunen, T. Kolli, P. Tyni, Development of an earthmoving machinery autonomous excavator platform, 2019 Proceedings of the 36th International Symposium on Automation and Robotics in Construction (ISARC 2019), Banff, Alberta, Canada, 2019, pp. 1005–1010, , https://
7