Deploying MAVs for autonomous navigation in dark underground mine environments

Journal Pre-proof Deploying MAVs for autonomous navigation in dark underground mine environments Sina Sharif Mansouri, Christoforos Kanellakis, Dariusz Kominiak, George Nikolakopoulos

PII: DOI: Reference:

S0921-8890(19)30625-6 https://doi.org/10.1016/j.robot.2020.103472 ROBOT 103472

To appear in:

Robotics and Autonomous Systems

Received date : 2 August 2019 Revised date : 10 January 2020 Accepted date : 10 February 2020 Please cite this article as: S.S. Mansouri, C. Kanellakis, D. Kominiak et al., Deploying MAVs for autonomous navigation in dark underground mine environments, Robotics and Autonomous Systems (2020), doi: https://doi.org/10.1016/j.robot.2020.103472. This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

© 2020 Published by Elsevier B.V.

lP repro of

Journal Pre-proof

Deploying MAVs for Autonomous Navigation in Dark Underground Mine Environments I

Sina Sharif Mansouri1,2 , Christoforos Kanellakis1 , Dariusz Kominiak, George Nikolakopoulos

Robotics and AI Team, Department of Computer, Electrical and Space Engineering, Lule˚ a University of Technology, Lule˚ a SE-97187, Sweden

Abstract

Operating Micro Aerial Vehicles (MAVs) in subterranean environments is be-

coming more and more relevant in the field of aerial robotics. Despite the large spectrum of technological advances in the field, flying in such challenging en-

vironments is still an ongoing quest that requires the combination of multiple

sensor modalities like visual/thermal cameras as well as 3D and 2D lidars. Nevertheless, there exist cases in subterranean environments where the aim is to

deploy fast and lightweight aerial robots for area reckoning purposes after an event (e.g. blasting in production areas). This work proposes a novel base-

line approach for the navigation of resource constrained robots, introducing the

rna

aerial underground scout, with the main goal to rapidly explore unknown areas

and provide a feedback to the operator. The main proposed framework focuses on the navigation, control and vision capabilities of the aerial platforms with low-cost sensor suites, contributing significantly towards real-life applications. The merit of the proposed control architecture is that it considers the flying platform as a floating object, composing a velocity controller on the x, y axes

Jou

and altitude control to navigate along the tunnel. Two novel approaches make I This work has been partially funded by the European Unions Horizon 2020 Research and Innovation Programme under the Grant Agreement No. 730302 SIMS. Email addresses: [email protected] (Sina Sharif Mansouri), [email protected] (Christoforos Kanellakis), [email protected] (Dariusz Kominiak), [email protected] (George Nikolakopoulos) 1 Authors contributed equally 2 Corresponding author

Preprint submitted to Elsevier

January 10, 2020

lP repro of

Journal Pre-proof

up the cornerstone of the proposed contributions for the task of navigation: 1) a vector geometry method based on 2D lidar, and 2) a Deep Learning (DL)

method through a classification process based on an on-board image stream, where both methods correct the heading towards the center of the mine tun-

nel. Finally, the framework has been evaluated in multiple field trials in an underground mine in Sweden.

Keywords: MAVs Navigation, Autonomous Tunnel Inspection, Mining Aerial Robotics

1. Introduction

In recent years, deployment of Micro Aerial Vehicles (MAVs) has experienced a rapid growth in different applications. These platforms have the potential to provide leading solutions in a wide range of tasks, especially in harsh, 5

unreachable and dangerous environments, such as underground mine inspection [1], subterranean exploration [2], large-scale infrastructure inspection [3],

and Mars canyon exploration [4]. The MAVs can provide access to unreachable,

complex, dark and dangerous locations for the monitoring personnel, exploring the area and minimizing service and inspection procedures, while minimizing down-times.

rna

10

The mining sector envisions that a further mine digitization, as well as robotics and automation, will contribute to safer mines, fewer emissions to air and water, and higher efficiency mining operations [5]. These factors contribute to sustainable mining, the unlocking of potential mineral reserves, and a higher 15

acceptance of mining activities and a social licence to operate. Overall, the deployment of MAVs can have a high impact on decreasing the time needed to

Jou

inspect shafts, openings, and areas after blasting, since the autonomous MAVs could directly enter in the areas and perform their instructed tasks without the need of pilots having line of sight for their operation. In this manner, human

20

exposure to dangerous environments (e.g. blind openings, areas after blasting, etc.) will be reduced, while zero emissions and safer workplaces will increase

2

lP repro of

Journal Pre-proof

the attractiveness of the mining operations.

Aerial robots, operating in underground mine tunnels, should be able to cope with harsh environmental conditions to successfully complete their mis25

sion. Usually, in this type of environments there is a lack of any natural illumi-

nation, there exist unconditioned narrow passages and crossing paths, dirt, high

moisture and dust. Figure 1 depicts a photo from one underground mine in Sweden to indicate the low illumination levels, as well as the uneven wall surfaces.

These dark and featureless environments challenge the state estimation schemes 30

since cameras do not yield sufficient information [6]. It becomes a necessity to

incorporate various kinds of on-board sensors for the aerial vehicles, including thermal cameras, 3D lidars and radars (e.g. Ulta-WideBand), to reach reliable

navigation capabilities underground [7] [6]. The common approach in the literature address the autonomous navigation of MAVs underground, by equipping 35

them with high-end 3D lidars and thermal cameras [8]. In these approaches, a

global localization scheme is employed, where the aerial vehicle should follow the designed exploration trajectories. Inspired from these challenges, this work addresses the autonomous navigation from another novel perspective trying to remove the dependency on high end and highly expensive sensors, replacing 40

them with lower cost solutions, while accomplishing the navigation task. The

rna

lightweight aerial scout is described as a robot capable of navigating along the

tunnels, utilizing relative, rather than global information, resembling the behavior of a floating object. Although, such an approach has both advantages and disadvantages, it fits more for applications where there is a need for fast 45

deployment of lightweight aerial vehicles for area reconnaissance, before deploying the high-end aerial robots. Nevertheless, the proposed system fits in various

Jou

application scenarios of underground mine inspection routines. This work presents the baseline system for a low cost and resource-constrained,

for the first time to the best of the authors knowledge, aerial scout to experi-

50

mentally establish autonomous navigation in underground mines. To this end, the proposed architecture addresses the problems of control, heading estimation, obstacle avoidance, and perception in a minimalist way, combining both novel 3

lP repro of

Journal Pre-proof

and well established components that contribute to this goal. One of the main

challenges in robotics is to answer the question of “where am I?”, in this case 55

focusing on the area of mine production tunnels. To address this challenge, this work introduces the concept of a floating object for the MAVs navigation. In-

stead of the commonly used full pose estimation of the platform, this is done by utilizing x, y velocities, height and attitude commands, based on optical flow

and Inertial Measurement Unit (IMU) measurements respectively. Generally, 60

position estimation using visual sensors has the disadvantage of uncertainties

and drift over time, due to the lack of prominent visual and geometric features in the general absence of illumination, thus making these approaches not reliable. Additionally, employing 3D lidars for pose estimation is out of the scope of this work to keep the cost of the platform’s sensing modules low. Furthermore, 65

the control commands are generated using a Nonlinear Model Predictive Control (NMPC), specifically designed and tuned to track the desired velocities on x and

y axis and desired altitude on z, while considering the dynamics of the platform. Another main challenge in the concept of floating MAVs in underground mine

navigation, is the definition of “where to go?”. Thus, to successfully navigate in 70

a tunnel, the MAV has to identify the direction of the flight in unknown areas, while avoiding collisions to the surrounding walls, without depending on accu-

rna

rate state estimation schemes. In this article, two different novel approaches

are proposed to define proper heading (yaw) for the MAV (Figure 1), which is commanded to keep a constant forward linear velocity. 75

1.1. Background & Motivation

In the related literature, there have been many works that addressed the

Jou

navigation problem in 2D and 3D environments. The three main exploration methods are the entropy based [9], the frontier based [10], and the information gain based exploration [11]. The entropy gain and information gain based meth-

80

ods compute regions that reduce the map uncertainty, based on current information on the map. Furthermore, in the frontier based exploration approaches, the exploration frontiers are computed as the discrete boundary between the 4

lP repro of

Journal Pre-proof

Figure 1: Photo of an underground mine in Sweden for indicating uneven surfaces and lighting

condition in a mine tunnel, while the MAV is equipped with artificial light source. The concept of the two proposed methods for defining heading angle is shown in white blocks.

unknown regions of the current map. These methods have been successfully applied in 2D environments [12]. Towards a 3D exploration, in [13], it has 85

been proposed a stochastic differential equation-based algorithm to enable exploration in indoor environments. This method resulted from the evaluation

of existing methods of frontier based exploration, however, there were no re-

quirements of dense representation of the free space. In [14], random trees were

90

rna

generated to find the best branch, the method was evaluated in indoor envi-

ronments and the paths were calculated on-line, while the occupancy map of the perceived environment was conducted. These exploration methods generally require a high computation power to process the images, to calculate the best next point and to accurately localize and store the previous information of the map in order to avoid revisiting the area. This fact limits the usage of

95

these methods in large-scale structures, while at the same time there have been

Jou

very few works that considered the navigation problem in dark tunnels. In [6], the authors addressed the problem of estimation, control, navigation and mapping problems for autonomous inspection of tunnels using aerial vehicles with the overall approach to be validated through field trials, however, the high-end

100

sensor suit that was utilized has limited the applicability of the overall method.

5

lP repro of

Journal Pre-proof

Furthermore, there are few works using machine learning techniques for the

problem of navigation in in-door and out-door environments, mainly due to the fact that these methods require a large amount of data and in most cases a high

computation power for training a Convolutional Neural Network (CNN), which 105

is an off-line procedure. However, after the training, the CNN can be used for enabling an autonomous navigation with much lower computation power,

especially when compared to the training phase. The works using CNN for nav-

igation, such as [15], [16], [17], [18], [19], utilized the image frame of on-board

cameras to feed the CNN for providing heading commands to the platform. 110

These works have been evaluated and tuned in out-door environments with

good illumination for the camera thus providing rich data about the surroundings of the platforms. Furthermore, preliminary and limited studies of MAV

navigation in an underground mine using CNN was presented in [20], however the method was evaluated in off-line collected data-sets from two underground 115

tunnels, without the MAV in the loop.

Within the related literature of MAVs in underground mine operations, few

research efforts have been reported trying to address challenging tasks within the mine. In [21] a visual inertial navigation framework has been proposed,

to implement position tracking control of the platform. In this case, the MAV

was controlled to follow obstacle-free paths, and the system was experimentally

rna

120

evaluated in a real scale tunnel environment, simulating a coal mine, where the illumination challenge was assumed to be solved. In [22] a more realistic approach, compared to [21] regarding underground localization, has been performed. More specifically, a hexacopter equipped with a Visual Inertia (VI) 125

sensor and a laser scanner was manually guided across a vertical mine shaft to

Jou

collect data for post-processing. The extracted information from the measurements has been utilized to create a 3D mesh of the environment and localize the vehicle. Finally, in [6] the estimation, navigation, mapping and control capabilities for autonomous inspection of penstocks and tunnels using aerial vehicles

130

has been studied, using IMUs, cameras and lidar sensors.

6

1.2. Contributions

lP repro of

Journal Pre-proof

Based on the aforementioned state of the art, the main contributions of this article are provided in this Section, describing all the components that are combined to establish the envisioned baseline solution for an autonomous aerial 135

scout.

The first contribution is to introduce the MAV as a floating object in the x and y axis, removing the high dependency on accurate localization systems.

Furthermore, the control commands are executed using a NMPC, specifically designed and tuned to track the desired velocities on x and y axes and desired 140

altitude on z, while considering the dynamics of the platform. Another merit of the system is that the state estimation is assigned to different sensors making

it modular. The velocities are acquired using an optical flow sensor looking downwards and assisted with additional led lights. On top of the state esti-

mation and control, the navigation component is introduced in the system to 145

accomplish the autonomous flight.

The second contribution highlights specifically the navigation modules de-

veloped to handle the heading command generation. More specifically, tunnel navigation is challenging since the MAV has to identify the direction of the flight

in unknown areas, while avoiding the surrounding walls. To address this issue, this work proposes two different approaches, where the first one depends on 2D

rna

150

lidar measurements and the second one on camera image streams, which depicts the flexibility of the system. In the first approach, the range measurements are geometrically processed to find the open spaces in the mine tunnel. Generally, larger range measurements indicate longer distances that are treated as open 155

space for the robot to move towards. For this purpose different range measure-

Jou

ments are added together, and the MAV moves its heading towards the range with the largest value. In the second approach, the on-board camera stream is used to correct the heading of the MAV in the dark tunnel, for the first time ever, based on the existing literature. The image stream is fed to a CNN to clas-

160

sify the image into three categories: left, center and right. This categorization results into preventing potential collisions with the walls. 7

lP repro of

Journal Pre-proof

Finally, the third major contribution stems from evaluating the performance of the proposed method at a 790 m underground mine without external illumina-

tion, located in Sweden. As it will be presented, the obtained experimental re165

sults have a significant novelty and impact on enabling the real-world exploration

in underground mines. The following link https://youtu.be/sW35Q3wVpI0 provides a video summary of the overall developed and proposed system. 1.3. Outline

The rest of the article is structured as follows. Initially, Section 2 presents 170

detailed information of the developed components for the MAV and the open challenges. The proposed method is evaluated in an underground mine and the

results are presented in Section 3, followed up with a corresponding comparison and discussion in Section 3.3. Finally, Section 4 conclude the article by summarizing the findings and offering some directions for future research.

175

2. Problem Statement and Open Challenges

The mining industry targets the integration of the MAVs in application

scenarios related to the inspection of production areas, like underground tunnels.

rna

In these cases, the inspection task includes the deployment of an aerial platform,

endowed with a sensor suite, to autonomously navigate along the tunnel and 180

collect mine oriented valuable information (images, gas levels, dust levels, 3D models, etc). In the sequel, the captured data will be used from the mine operators for further analysis to determine the status of the inspected area. This technology is aligned with the envisioned mine of Zero Entry Production Areas (ZEPA) [23], to create a safer working environment with increased production, while reducing the overall operation costs.

Jou

185

Deployment of MAVs in harsh underground mine environments poses mul-

tiple challenges [1], and in general, to increase the level of autonomy and to provide stability and reliability in their operation, MAVs are equipped with high-end and expensive components. Nonetheless, the main aim for these plat-

8

190

lP repro of

Journal Pre-proof

forms is long-term operation underground, where their performance and in-

tegrity degrade over time [24] and they need to be maintained/replaced. This can be a bottleneck for expensive solutions, since the maintenance/repairing

costs accumulate to high levels. Therefore, the overall vision is to consider the aerial vehicles as consumables task-oriented scout platforms, designed to ac195

complish specific needs in subterranean environments. These platforms could be deployed for various application scenarios, such as navigating quickly and

providing a rough topological map or visual feedback of the area to the operator, or navigate and collect data from harsh areas.

The core part to enable autonomous navigation is considered to be the state 200

estimation, which provides information of the local “positioning” of the MAV,

e.g. position, orientation, velocities, etc. Due to limitations posed from both the mine environment and the platform, separate sensors have been assigned to specific states: 1) attitude [φ, θ]> obtained from the on-board IMU, 2) lin-

ear velocities [vx , vy ]> from optical flow, and 3) height z estimation from one 205

beam lidar. This architecture is characterized by low computational power sav-

ing computational resources for other components. Once state information is available, navigation and control components use it to generate and execute

the navigation commands respectively. The main challenge in the navigation

210

rna

commands is to extract proper heading to follow an obstacle free path along the

tunnels. In this article heading commands are generated either from a vector geometry method based on 2D lidar for finding open spaces or a CNN approach that classifies images from a looking forward camera and finds the center axis of the tunnel. The controller has been designed to consider the MAV as a floating object on x and y axes, mainly selected to remove high dependencies on accurate localization schemes, while the platform is still able to perform the desired

Jou

215

exploration task. An overview of the proposed concept for navigation is shown in Figure 2.

The navigation component incorporates a potential field approach as a fail-

safe system that is always enabled and acts when any of the other methods

220

fail. Compared to [1], this work address the challenge of the navigation in real 9

lP repro of

Journal Pre-proof

Figure 2: Top-view concept image of a mine tunnel with a MAV, which shows the necessity

of heading correction in autonomous navigation. Body frame xb and yb are shown by red and green arrows respectively, while it is assumed that the heading of the MAV is towards the x-axis.

underground mine environments, adopting the concept of floating object, where in [1] the simulations considered known odometry measurements provided from

the simulation environment. Moreover, dust is another challenge when flying

in actual tunnels, but in our case we have been visiting the production areas 225

of mines where the ground was wet and only small dry parts were generating

negligible amount of dust. Therefore in the presented field test, although dust is a major issue it was not considered since it heavily depends on the area of

rna

the flight. The overall, system diagram is presented in Figure 3. 2.1. MAV Dynamics 230

The MAV is considered as a six Degree of Freedom (DoF) object with a Body-Fixed Frame B attached and the inertial frame E as depicted in Figure 4. The dynamic model of the MAV is described by the position of the center of the mass in x, y, and z axes in the inertia frame and the orientation of the body around each axes with respect to the inertial frame [25, 26]. This modelling requires full state estimation for the position and orientation of the MAV. How-

Jou

235

ever, one of the main challenges in the autonomous navigation of the MAV is the estimating and maintaining of accurate values for the MAV position and orientation. While the IMU measurements are affected from the accumulated error, underground areas are Global Positioning System (GPS) denied environments

10

lP repro of

Journal Pre-proof

Figure 3: Overall proposed architecture for navigating in the mine; it should be highlighted that each components have few sub components and for simplicity it is not shown hereby.

240

and the lack of illumination limits the vision based localization methods, such as

Visual Odometry (VO) [27]. The drifts and uncertainties in x, y positions may

result to collisions for the MAV, thus the proposed method does not depend on position estimation on the x and y-axes and only the estimation of velocities,

attitudes and altitude are considered for defining the MAV as a floating object.

Jou

rna

The MAV is modelled by (1) in body frame.

Figure 4: Illustration of the quad-copter with attached body fixed frame B and inertial frame

E, while the safety distance is shown by ds and desired velocities by vd,x , and vd,y .

11

lP repro of

Journal Pre-proof

z(t) ˙ = vz (t)

     A 0 0      x      ˙ V (t) = Rx,y (θ, φ)  0  +  0  −  0      0 −g T ˙ = 1/τφ (Kφ φd (t) − φ(t)) φ(t)

0

Ay 0

  0  V (t)  Az

˙ = 1/τθ (Kθ θd (t) − θ(t)) θ(t) 245

0



(1a)

(1b)

(1c)

(1d)

Where V ∈ R3 indicates the velocities along each axis [vx , vy , vz ]> , φ ∈ R and θ ∈ R are roll and pitch, Rx,y is rotation matrix without rotation around zaxis since coordinates are in the body frame. T ∈ [0, 1] is the mass normalized

thrust, g is the gravitational acceleration, Ax , Ay , and Az are normalized mass

drag coefficients, τφ and τθ are time constants, and Kφ , Kθ are roll angle and 250

pitch angle gains which depends on behaviour of flight controller for tracking attitudes and φd , θd are desired roll angle and pitch angle. 2.2. Controller

The NMPC is developed in a cascaded fashion and solved by Proximal Av-

eraged Newton-type method for Optimal Control (PANOC) [28], while guaran255

teeing real-time performance. The NMPC objective is to generate attitude φ, θ

rna

and thrust commands T for the low-level controller u = [T, φ, θ]> , while the reference velocities [vd,x , vd,y ]> are provided from the navigation module, the constant altitude zd and vz,d = 0 from the operator. The low level controller is integrated to the flight controller and Proportional Integral Derivative (PID) 260

controllers are generating the motor commands [n1 , . . . , n4 ]> for the MAV. The low-level attitude controller [29] is able to track the desired roll, pitch and yaw

Jou

rate and to calculate the corresponding rotor speeds for the MAV. A block diagram representation of the proposed NMPC and the corresponding low-level controller is shown in Figure 5, where R = {ri |ri ∈ R+ , i ∈ Z ∩ [−π, π]} are

265

ranges from a 2D lidar, I is image stream from looking forward camera, ax , ay , az , wx , wy , and wz are linear and angular accelerations along each axis

12

lP repro of

Journal Pre-proof

from IMU measurements, and vx and vy are linear velocities from a down-ward optical-flow sensor and z is altitude from the single beam lidar.

Figure 5: Control scheme of the proposed navigation module with potential field for desired velocities generation, while the heading commands can be provided either from the CNN or

the vector geometry approaches. The NMPC generates thrust and attitude commands, while the low level controller generates motor commands [n1 , . . . , n4 ]> . The velocity estimation is based on IMU measurements, optical flow and single beam lidar.

While considering the non-linear dynamics of the MAV (1), the state of the 270

ˆ θ] ˆ > is the estimated state ˆ = [ˆ system is X = [z, vx , vy , vz , φ, θ]> , X z , vˆx , vˆy , vˆz , φ,

rna

and F is a discretization of (1), which is obtained by the Euler method [30].

Moreover, zk+j|k , Vk+j|k , and uk+j|k are the altitude, velocities and control action ahead of k + j steps from the current time k. For the proposed NMPC,

Jou

the following finite horizon cost function is introduced.

13

J=

N −1 X j=0

lP repro of

Journal Pre-proof

|

zk+j|k − zk+j|ref


>


Qz zk+j|k − zk+j|ref {z }

(2a)

tracking the altitude

+ Vk+j|k − Vk+j|ref |


>


QV Vk+j|k − Vk+j|ref {z }

(2b)

tracking velocity commands

+


>
φk+j|k − φk+j|ref Qφ φk+j|k − φk+j|ref +
>
θk+j|k − θk+j|ref Qθ θk+j|k − θk+j|ref {z } |

(2c)

tracking the attitude

+ uk+j|k − uref |


>


Qu uk+j|k − uref {z }

(2d)

hovering term

275

The cost function of the optimization problem involves three terms, the first

term minimizes the error between the estimated altitude zˆ and desired altitude

zd . This term is responsible to keep the constant altitude of the MAV. The second term tracks the velocity profile. In the proposed method, the potential

fields provides a reference value for the velocities to align in the x and y-axes as 280

[vd,x , vd,y ]> for avoiding collision to the obstacles, while the vd,z is zero based on the constant altitude of the flight. The third term tracks the desired attitude

commands of the controller as from (1) the attitude control system is modeled

rna

by simple first-order dynamics with time constants τθ , τφ and gains of Kφ , Kθ .

Thus, the φd , θd are the desired attitudes and φ, θ are states of the system and 285

the NMPC accounts for the time constants of the attitude terms. The forth term is the hovering term where uref is [g, 0, 0]> , which is the hover thrust with horizontal angles. Additionally, Qz ∈ R1×1 , QV ∈ R3×3 , Qφ ∈ R1×1 ,

Qθ ∈ R1×1 , and Qu ∈ R3×3 are the weights for each term of the objective

Jou

function, which reflects the relative importance of each term.

14

lP repro of

Journal Pre-proof

The following optimization problem is defined: minimize J

(3a)

N −1 {uk+j|k }j=0

s.t.

290

Xk+j+1|k = F(Xk+j|k , uk+j|k ), for j ∈ N[0,N −1]

(3b)

uk+j|k ∈ [umin , umax ], for j ∈ N[0,N −1]

(3c)

ˆk Xk|k = X

(3d)

At every time instant k, a finite-horizon optimal path with prediction horizon N is solved and a corresponding optimal sequence of control actions u?k|k , . . . ,

u?k+N −1|k is generated. The first control action u?k|k is applied to the flight controller by a zero-order hold element, which is, u(t) = u?k|k for t ∈ [kTs , (k + 1)Ts ], where Ts is the sampling time. 295

2.3. State Estimation

Successful autonomous navigation of MAVs requires the combination of accuracy and low latency in the state estimation. Generally, for localization pur-

poses different sensor suites are used. The most common sensors are a com-

bination of GPS with IMU units, however underground mines are GPS-denied

environments. Moreover, motion capture systems provide localization with high

rna

300

precision and high sampling rate in GPS-denied areas, nonetheless these systems require multiple cameras in fixed positions to be installed beforehand, their performance is affected by dust and high humidity and their cost is high, which makes them unsuitable for the case of the underground mine environments. 305

Another group of localization systems are forward looking monocular or stereo

Jou

cameras for visual/visual-inertial pose estimation. These methods are widely used in indoor and outdoor environments, however, they rely on features from the environment which can be limited in case of low-illumination environment. Additionally, due to payload and power constraints of MAVs, these algorithms

310

should perform under computational limitations of embedded hardware [31].

15

lP repro of

Journal Pre-proof

In this article the autonomous MAV is considered a floating object that explores 3D space. The measurement updates of the state vector X = [z, vx , vy , vz , φ, θ]> is accomplished assigning sensors to specific states. The altitude state z

is provided from a downward pointing single beam lidar, while the linear ve315

locity vz is estimated from z using Kalman Filtering [32]. Moreover, φ and θ

are provided from the flight controller through the filtering of IMU measurements from an Extended Kalman Filter (EKF). Finally, the states vx and vy

are calculated from a down-ward optical-flow sensor. The optical flow sensor

is equipped with additional illumination source to avoid drifts in the measure320

ments, while providing high update rates. To robustify the performance of the state estimation, the optical flow has been calibrated at a specific height and

the MAV is commanded to fly at fixed altitude. More details about the software and hardware of the employed optical-flow [33]. The optical flow based velocity estimation and EKF are shown as state estimator block in Figure 5. 325

2.4. Navigation and Collision Avoidance

For collision free navigation of the MAV, the platform should be able to follow obstacle free paths to complete the assigned task, identifying the direction

that it should move in such constrained and unknown environments. The core

330

rna

issue that should be addressed refers to heading command generation for the

MAV. This section describes two different methods that provide heading commands to the aerial platform 1) geometry approach using lidar measurements and 2) Deep Learning (DL) approach using camera stream, both trying to steer the platform to move forward into the open areas. Moreover, on top of these methods a potential field approach is always running in the background as a fail-safe system to avoid collisions.

Jou

335

2.4.1. Vector Geometry Based The purpose of the vector geometry approach is to generate the heading

rate commands based on range measurements of the 2D lidar. The heading rate commands is adjusted to guide the MAV towards the direction that has

16

340

lP repro of

Journal Pre-proof

the largest amount of navigable space. The schematic of the proposed method

is shown in Figure 6, while the three range measurements from the 2D lidar is

shown with arrows. The colors of the arrows correspond to the magnitude of the

vectors, where green, yellow and red are largest to smallest vectors respectively.

yb

~r d xb

~ d l

~c d

Figure 6: Schematic of the proposed vector geometry based method for heading correcting. The color of the vectors represent their distance to the obstacles, while green, yellow and red

are large, medium and small distances to obstacles. The heading rate is generated based on Alg. 1 and the MAV heading rate is generated based on open areas.

+

Initially, three range measurements ri from 2D lidar ranges R = {ri |ri ∈

R , i ∈ Z∩[−π, π]} are measured for the right dr , center dc and left dl distances. The d~c has a direction towards the x-axis of the body frame (θgeom,c = 0), d~r

and d~l are right and left vectors with a constant value of θgeom,r ∈ [0, π/2] and

rna

θgeom,l ∈ [−π/2, 0] from the x-axis of the body frame respectively. The values of the θgeom,r and θgeom,l are symmetric and depend on the design choices and for clarity it is called θgeom . The resulting distance vector of dr , dc , and dl is calculated as:

d~sum = d~r + d~c + d~l

(4)

Based on the direction of the d~sum the heading can be obtained, however achiev-

Jou

ing accurate heading especially for low-cost navigation systems is not always possible. Furthermore, most of the methods rely on information from a magnetometer and gyrocompass, multi-antenna Global Navigation Satellite Systems (GNSSs) or position information of the platform [34]. Nonetheless, magnetometers are not reliable especially in underground mines and GNSS is not available for underground areas. Thus, in the proposed method, the obtained heading is 17

lP repro of

Journal Pre-proof

converted to a heading rate by dividing the ∠d~sum to θgeom . The ψ˙ is bounded between −1 and 1, e.g. dr = 0, dl = 0, and dc > 0 that results to ψ˙ = 0 rad/sec, dc = 0, dl = 0, dr > 0 results to ψ˙ = −1 rad/sec, and dr = 0, dc = 0, and

dl > 0 result to ψ˙ = 1 rad/sec. However, a larger value of θgeom can be selected to have slower heading rate commands.

∠d~sum ψ˙ = θgeom

345

(5)

It should be highlighted that in order to avoid disturbances in the range measurements, the array of ranges can be used for d~c , d~r , and d~l . The algorithm 1 pro-

vides the overview of the vector geometry based approach for generating heading

rate commands, while the θgeom = π/3, the array of ranges from [−π/4, π/4], [π/4, π/2], and [−π/2, π/4] are considered for d~c , d~r , and d~l respectively.

Algorithm 1 Calculate heading rate command based on 2D lidar measurements. Input: R

Output: ψ˙ > ~c = Pπ/4 1: d i=−π/4 ri [cos θi , sin θi ] Pπ/2 d~r = i=π/4 ri [cos θi , sin θi ]> P−π/4 > d~l = //obtaining d~c , d~r , and d~l for arrays of i=−π/2 ri [cos θi , sin θi ]

350

rna

ranges

2:

d~sum = d~r + d~c + d~l

3:

ψ˙ =

∠d~sum θgeom , θgeom

= π/3

2.4.2. Convolutional Neural Network

Jou

A CNN [35, 36] is composed of a series of non-linear processing layers stacked on top of each other. The typical layers presented in a CNN are: the convolutional, the pooling, the fully connected, and the non-linear activation layers. The main component of the CNN is the convolutional layer which operates on

355

the local volumes of data through convolutional kernels, also called filters, that extract feature representations (feature maps). The pooling layer reduces the 18

lP repro of

Journal Pre-proof

size of the original image and the subsequent feature maps and thus provid-

ing translation invariance. The nonlinear activation layers (usually consisted of Rectified Linear Units (ReLUs) that have almost completely substituted the tra360

ditional sigmoid activation functions), as in the case of the conventional Nueral

Networks (NNs) allows to the CNNs to learn non-linear mappings between the input and the output. The final layers of a CNN are “flat” fully connected layers

identical to the ones used in conventional MultiLayer Perceptrons (MLPs). A

CNN is trained “end to end” to learn to map the inputs to their corresponding 365

targets by using gradient-descent based learning rules.

In this work the same CNN structure as presented in [20] is used, while

the layers and implementation details are discussed comprehensively. The CNN receives a fixed-size image as an input and provides three categories for each image. Similar to most neural networks, a CNN has an input layer, an output 370

layer and many hidden layers in between. These layers learn features specific

to the task by altering the input data. The main difference of these novel architectures is that the features are learned during the training process, instead of relying on tedious feature engineering processes. In the case of CNNs this

is basically achieved via the convolution filters. Each convolution filters learns 375

to be activated by certain features of the image. One advantage of convolution

rna

connections especially compared to a fully connected architecture is the dra-

matic reduction in number of parameters due to weight sharing. ReLUs has been used as an additional operation after each convolution operation. ReLUs is an element-wise operation and allows for faster and more effective training 380

by mapping negative values to zero and maintaining positive values. ReLUs introduces non-linearity to the CNN, since most of the real-world data would

Jou

be nonlinear and convolution is a linear operation. Another layer which is not part of NNs is the pooling layer. Pooling performs nonlinear down-sampling, while at the same time reducing the number of parameters that the network

385

needs to learn while retaining the most important information. These operations are repeated over a large number of layers, with each layer learning to identify different features and extracts the useful information from the input 19

lP repro of

Journal Pre-proof

image, introduce nonlinearity and reduce feature dimensions. The outcome of

convolution and pooling layers represent high-level features of the input image, 390

thus the fully connected layer uses these information for classifying the images.

The input layer of the CNN is a matrix of 128 × 128, followed by 2D convolution layer with size of 2 × 2 and output of 126 × 126 × 32, followed by 2D max

pooling with output of 63 × 63 × 32 and size of 2 × 2, then next 2D convolution

layer with size of 3 × 3 to extract features with output of 61 × 61 × 32, after 395

that the 2D pooling layer of size 3 × 3 with output of 30 × 30 × 32, followed by flatten layer and ending with fully connected layer with softmax activation for classifying images to three classes. Depending on on-board camera the images

can have different resolutions, however to reduce computation time and size of data the input image should be resized to 128 × 128 pixels. Moreover, object 400

recognition based on gray-scale images can outperform RGB image based recognition [37], and the RGB sensors do not provide any extra information about the

dark mine environments. Thus, for reducing the computation time and noise,

the images from the cameras are converted to gray-scale. Moreover, CNNs with large numbers of parameters and multiple nonlinear hidden layers are power405

ful techniques, however the large networks require high computation power and suffer from the problem of overfitting. In the proposed method, the dropout [38]

rna

layer is added to reduce the overfitting during the training of a model and is

not used when evaluating the trained model. The dropout layer removes units with all incoming and outgoing connections in the CNN architecture. Random 410

dropout of neurons during training phase, results to adaptation of other neurons to make prediction for the missing neurons. In this way, multiple independent internal representations are learned by the network. This results to a network

Jou

with less size and less sensitivity to the specific weights of neurons. Moreover, during the training, the model cross-validation [39] method is used to reduce

415

overfitting and to provide models with less biased estimation. The training data-set is split to subsets and the model is trained in all subsets except one, which is used to evaluate the performance of the trained model. The process is continued until all subsets are evaluated as a validation set and the performance 20

lP repro of

Journal Pre-proof

measure is averaged across all models.

For the proposed method, a loss function is categorical crossentropy, an

420

optimization is adam optimizer [40], and the network is trained on a workstation

equipped with an Nvidia GTX 1070 GPU which is located outside of the mine

with 25 epochs and 200 steps per epoch. For each image, the CNNs provides the class of the image that can be Left, Center, Right. Each label represents 425

the direction of the platform’s heading, e.g. in case of Center the heading

rate should be zero ψ˙ = 0, in case of Right the heading rate should have a positive value, in case of Left the heading rate should have a negative value

to avoid the left wall. In the proposed CNN approach the constant value of ψ˙ = {−0.2, 0, 0.2}rad/sec is selected for each label, 0.2 rad/sec is selected to 430

avoid sudden heading rate commands in case of wrong classification from the CNN approach. The algorithm 2 provides an overview of the proposed method for generating heading rate commands based on the CNN classification method. Algorithm 2 Calculate heading rate command based on the CNN approach. Input: I Output: ψ˙ d 1:

I m×n×3 → I 128×128×1 //converting the RGB image with m × n pixels to gray scale and resizing to 128 × 128 pixels

3:

CNN(I 128×128×1 ) → {left,center,right} //Output of the CNN

rna

2:

if CNN(I)==left then ψ˙ d = −0.2 rad/sec

4:

else if CNN(I)==center then ψ˙ d = 0.0 rad/sec

5:

else ψ˙ d = 0.2 rad/sec

6:

end if

Jou

2.4.3. Potential Fields

The potential field’s objective is to generate linear x-axis and y-axis velocity

435

commands in order to avoid collisions to the walls or any other obstacle standing in the way of the MAV. The heading commands are provided from the vector geometry method (Section 2.4.1) or CNN method (Section 2.4.2). In this work

21

lP repro of

Journal Pre-proof

the potential field uses range measurements from 2D lidar placed on top of

the vehicle to obtain repulsive velocity commands for both x-axis and y-axis 440

when flying close to obstacles, while the attractive velocity command is given a constant value on the x-axis. The potential field approach generates desired velocities vd,x , vd,y for the platform and are fed to the NMPC as depicted in

Figure 5. Figure 4 depicts the 2D lidar position on top of the MAV, while indicating the generated potential field velocities direction and the defined safety 445

distance ds , based on the geometry and size of the aerial platform.

In classical potential field methods [1], the relative vector between the agent

position and destination is required for obtaining the attractive force. However,

in this article the position estimation is not available thus it is assumed that the desired vector is constant and it is towards the x-axis. The desired velocity vector is defined as:

~v = [vx , 0]> , vx ∈ R>0

(6)

As mentioned above the repulsive potential is structured around the 2D lidar information and is defined in the body frame of the aerial vehicle. Before

describing the algorithm it is essential to introduce the notation used in this case. Initially, the measurements are transformed from ranges into a 2D pointcloud Λ, removing invalid values. The MAV is considered always to lie at the origin of Λ,

rna

450

expressed always through p~s = [0, 0]> . For every time instance k the pointcloud is updated according to the new range measurements. For simplicity reasons the time instance indexes are omitted in the following Equations. Every point registered in Λ is considered an obstacle and is described by its 2D position 455

p~o = [x, y]> . Based on the p~s and p~o , the repulsive potential is calculated and

Jou

desired velocities are obtained similar to the work in [1].

3. Results

This section describes the experimental setup and the experimental trials

performed in the underground environment.

22

460

lP repro of

Journal Pre-proof

Link: https://youtu.be/sW35Q3wVpI0 provides a video summary of the system.

3.1. Experimental Setup

In this work two separate platforms are used to evaluate the proposed methods. The first platform is a quad-copter which is developed at Lule˚ a University 465

of Technology based on the ROSflight [41] flight controller and the second one is the commercially available quad-copter Parrot Bebop 2 [42]. A detailed presentation of the platforms is provided in the sequel. 3.1.1. ROSflight Based Quad-copter

The vehicle’s weight is 1.5 kg and provides 8 mins of flight time with 4470

cell 1.5 hA LiPo battery. The flight controller is ROSflight and the Aaeon UPBoard3 is the main processing unit, incorporating an Intel Atom x5-Z8350 pro-

cessor and 4 GB RAM. The operating system running on the board is Ubuntu Desktop 16.04, to which Robot Operating System (ROS) Kinetic has been also

included. Regarding the sensor configuration, a 2D rotating Rplidar is placed 475

on top of the vehicle, providing range measurements at 5Hz. The velocity esti-

mation is based on the PX4Flow optical flow sensor at 20 Hz, while the height

rna

measurements are provided from the single beam Lidar-lite v3 at 100 Hz, both

installed on the bottom of the vehicle pointing down. Furthermore, the aerial platform is equipped with two 10 W LED light bars in both front arms for 480

providing additional illumination for the forward-looking camera and four lowpower LEDs looking down to provide additional illumination for the optical flow sensor. Figure 7 presents the platform at different viewpoints, highlighting it’s

Jou

dimensions and the sensor configuration. 3.1.2. Parrot Bebop 2

485

Parrot Bebop 2 weighs 0.5 kg, offering 25 mins of flight time and it is equipped

with a forward-looking camera, optical flow and sonar sensor looking down as 3 https://www.aaeon.com/en/p/up-board-computer-board-for-professional-makers

23

lP repro of

Journal Pre-proof

Figure 7: The developed quad-copter equipped with a forward looking camera, a LED lights, optical flow, 2D lidar and single beam lidar.

depicted in Figure 8. The platform provides WiFi link and all computation is

done on the ground station computer with an Intel Core i7-6600U CPU, 2.6GHz

and 8GB RAM. The Bebop-Autonomy 4 is used for the estimation of the states 490

and controlling the platform, while the heading of the MAV is corrected based

rna

on the proposed CNN method.

Figure 8: The commercial quad-copter Parrot Bebop 2.

3.2. Navigation Modules Results

This work demonstrates the performance of the proposed methods in three

Jou

different test cases, highlighting the adaptability and feasibility of the meth495

ods under various vehicle configurations. The location of the field trials was 790 m deep in an underground mine in Sweden without any natural illumination sources. Furthermore, the underground tunnels did not have strong corrupting 4 https://bebop-autonomy.readthedocs.io/

24

lP repro of

Journal Pre-proof

magnetic fields, while their morphology resembled an S shape environment with small inclination. Overall, the field trial area had width, height and length di500

mensions of 6 m, 4 m, and 150 m respectively. Figure 9 depicts one part of the underground mine with uneven surfaces. The controller has been tuned to

Figure 9: Photo of 790 m underground mine in Sweden.

compensate light wind-gusts that were present in the tunnel, however no wind

measurement device was available to measure the speed of the wind. The tuning parameters of the NMPC and parameters of the MAV model are presented in

the Tables 1 and 2 correspondingly. The NMPC prediction horizon N is 40 and

rna

505

control sampling frequency Ts is 20 hz. Table 1: The tuning parameters of the NMPC.

Qz

QV

Qφ

Qθ

Qu

Tmin

Tmax

φmin

φmax

θmin

θmax

[2, 2, 3]>

3

3

[2, 10, 10]>

0

1

−0.4 rad/s

0.4 rad/s

−0.4 rad/s

0.4 rad/s

Jou

6

Table 2: The parameters of the MAV model.

g

Ax

Ay

Az

Kφ

Kθ

τφ

τθ

9.8 m/s2

0.1

0.1

0.2

1

1

0.5 s

0.5 s

25

lP repro of

Journal Pre-proof

In general for evaluating the geometry based approach, nine flights with

an overall flight time of 1804 sec have been performed. Moreover, nine more 510

flights with a duration of 2019 sec have been executed for evaluating the CNN approach. In both approaches none of the flights resulted to a collision and the

flights were terminated only due to battery drainage and not based on another external or internal faulty operation.

3.2.1. Case 1: Potential Field and Vector Geometry Based Approaches 515

The main purpose of case 1 is to evaluate the floating object concept by potential field and vector geometry navigation methods on the ROSflight based

platform.In this case the GoPro7 camera is used for collecting images, the 2D lidar measurements are used for both potential fields and heading corrections,

the optical-flow sensor provides velocity state estimation and the single beam 520

lidar provides altitude estimations. In order to reduce 2D lidar measurement noise and uncertainties for calculating the heading rate, instead of relying on

only one beam, an array of beams are selected and passed through a median filter. Moreover, the potential field module is relying on 360 beams of 2D lidar measurement. The maximum ranges of 4 m and the safety distance ds

for potential field is 1.5 m which enables the repulsive forces, while the vector

rna

525

geometry module is always active. Moreover, the desired constant altitude of zd = 1 m is considered for the MAV. Figure 10 depicts the generated vd,x , vd,y commands from the potential fields. The desired vd,x velocity is constant at 0.1 m/s, while the changes in vd,x and vd,y are due to repulsive forces from 530

the tunnel walls. Figure 11 depicts the ψd and ψ˙ d from the vector geometry modules. It can be seen that when the vd,y has negative values, the ψ˙ d has

Jou

positive values which means the platform is close to the left wall, trying to move away from the left wall with the heading towards the other side of the tunnel. Additionally, it can be seen that in some cases the value of vd,x is changing,

535

this is due to the fact that the MAV is looking towards walls and the potential field provides different velocities. Moreover, due to ds = 1.5 m in the potential 26

lP repro of

Journal Pre-proof

field, the commands are mainly constant until the MAV is close to the obstacles,

while the heading commands are always correcting the MAV to look towards open areas. The mean absolute error (MAE) between vx , vy and vd,x , vd,y are 540

0.25 m/s and 0.13 m/s respectively. The velocity measurements provided from the optical flow sensor are noisy, but it does not affect the overall navigation mission. Moreover, the sensor placement and parameter tuning significantly influence the performance of the measurements. 2

vx [m/s]

Estimated velocity Reference

1

0

−1

0

1

0

rna

vy [m/s]

0.5

50

100

150

200

100 150 time [sec]

200

−0.5 −1

0

50

Figure 10: The velocity commands from the potential field in the first case, while the Rosflight

Jou

platform navigates based on potential field and vector geometry modules.

Moreover, the altitude of the MAV is shown in Figure 12. It can be seen that

545

during the take off the platform reaches to 1.4 m and then stabilizes around 1 m. The initial high thrust command during the take-off phase has been selected to boost the take-off and faster reach the desired height of 1 m, assisting the 27

10

ψd [◦ ]

5 0 −5 −10

0

0.1 0 −0.1

50

rna

ψ˙ d [rad/sec]

0.2

lP repro of

Journal Pre-proof

−0.2

0

50

100 150 time [sec]

200

100 150 time [sec]

200

Figure 11: The heading rate command and heading angle from the vector geometry based approach in the first case, while the Rosflight platform navigates based on potential field and

Jou

vector geometry modules.

28

lP repro of

Journal Pre-proof

stability of the optical-flow measurements, since the sensor has been tuned to operate around 1 m altitude. Usually, slower take off resulted in drifting velocity 550

state estimation, leading to undesirable behaviour of the platform during the take off procedure. 1.5

Altitude Reference

z[m]

1

0.5

0

0

50

100 150 time [sec]

200

Figure 12: The desired altitude and the altitude of the MAV during the navigation in the

first case, while the Rosflight platform navigates based on potential field and vector geometry modules.

During the field trials, it was observed that the platform heading was cor-

rected towards the open area. Nevertheless, due to the combination of the

shape.

rna

tunnel width with the limitations of the 2D lidar, the MAV path followed an S 555

3.2.2. Case 2: Potential Field and CNN Approaches In the second case, the same platform as in case 1 is evaluated by potential field and CNN for heading correction. The MAV desired altitude is 1 m with constant desired vd,x = 0.1 m/s. Additionally, the platform is equipped with PlayStation Eye Camera to provide image data for the CNN module as depicted

Jou

560

in Figure 13. The PlayStation Eye camera was attached to the camera mount, facing forward with a weight of 0.150 kg, this camera was operated at 20 fps and with a resolution of 640 × 480 pixels. Furthermore, the LED light bars provide

460 lux illumination in 1 m distance.

565

The CNN requires data sets for training, thus the setup in Figure 14 has 29

lP repro of

Journal Pre-proof

PlayStation Eye Camera

Figure 13: The ROSflight based platform equipped with PlayStation Eye camera.

been used. The setup consists of three mounted cameras with separated LED light bars pointing towards the field of view of each camera. Different cameras

including Gopro Hero 7, Gopro Hero 4, and GoPro Hero 3 are used to reduce the dependency of training data sets to specific cameras. Additionally, the light 570

bars were calibrated to provide an equal illumination power, while data sets are collected with different illumination levels of 460 lux and 2200 lux from 1 m

distance to consider uncertainties in illumination. During the data collection each camera has a different resolution and Frame Per Second (fps) of 3840×2160 pixels with 60 fps, 3840 × 2160 pixels with 30 fps, and 1920 × 1080 pixels with 575

30 fps for Gopro Hero 7, Gopro Hero 4, and GoPro Hero 3 respectively. To reduce the redundancy of the images, the videos are down-sampled to 5 fps and

rna

converted to a sequel of images, however, without loss of generality faster fps

could be selected. The images are re-sized to 128×128 pixels, converted to grayscale mode and labeled based on the direction of the camera. During the data 580

set collection, the triple camera setup is carried by a person, while guaranteeing that the middle camera is always looking toward the tunnel axis, and the camera is held 1 m above the ground.

Jou

Furthermore, the collected data-sets are carefully selected to be not from the same tunnel area which the MAV should be evaluated in for autonomous

585

navigation. There has not been applied any camera/rig calibration process during the training process of the CNN. The captured images from the triple camera set have been resized and provided as input to the training phase of the

30

lP repro of

Journal Pre-proof

Figure 14: The triple camera setup for obtaining training data sets for CNN.

learning workflow without any rectification, thus with their fisheye distortions. Figure 15 depicts the sample images from an underground mine collected by 590

triple camera setup for training the CNN.

After training the CNN uses the obtained data-sets on the workstation out-

side of the mine. The online video stream from the PlayStation Eye camera is

rna

re-sized, converted to gray-scale mode and fed to the CNN for generating the

heading rate commands for the MAV to navigate autonomously in the mine. 595

The output of the CNN is the class of the image, while for each label constant heading rate commands are generated for the MAV. The −0.2 rad/s, 0.0 rad/s, and 0.2 rad/s are the heading rate commands that correspond to the left, center, and right labels respectively. Figure 16 shows sample images from the PlayStation Eye camera, where the class is written above each image. A merit of the proposed CNN navigation method is the applicability of the

Jou

600

network in unexplored areas of similar structure. More specifically, the trained network can provide accurate classification of tunnel areas that have not been included in the training phase or have different surfaces with different level of illumination. Therefore, the CNN can be deployed in new tunnels that has not

31

GoPro Hero 4

lP repro of

Journal Pre-proof

GoPro Hero 7

GoPro Hero 3

Figure 15: Examples of collected images from underground mine used for training the CNN.

The left, center and right images are from cameras looking toward left, center and right correspondingly.

Center

Right

Jou

rna

Left

Figure 16: Examples of images from PlayStation Eye camera, while the MAV performs autonomous navigation in underground mine. The class of images from the CNN is written above the images.

32

605

lP repro of

Journal Pre-proof

been trained for, reducing the complexity and time of the inspection task.

In this case, similar to the first case, the potential field is active when ds is

less than 1.5 m. The velocity commands from the potential field and estimated velocities are depicted in Figure 17. The MAE of vx and vy are 0.1 m/s and 0.08 m/s respectively. 2

vx [m/s]

Estimated velocity Reference

1

0

−1

0

0

100

150

200

100 150 time [sec]

200

rna

vy [m/s]

1

50

−1

0

50

Figure 17: The velocity commands from potential field, while CNN is providing heading commands in the second case with Rosflight platform.

Furthermore, the altitude of the MAV for this case is shown in Figure 18.

Jou

610

Similar to the first case, the initial high thrust command during the take-off phase has been selected for assisting the stability of the optical-flow measurements.

In the summary video, it can be seen that the platform heading is corrected

615

towards the center of the tunnel and the platform follows almost a straight line 33

1.5

lP repro of

Journal Pre-proof

Altitude Reference

z[m]

1

0.5

0

0

50

100 150 time [sec]

200

Figure 18: The desired altitude and the altitude of the MAV during the navigation in the second case, while the Rosflight platform navigates based on CNN and potential field modules.

as the image streams provide sufficient information about the walls and center of tunnel. However the heading of the MAV changes frequently with 2[hz], this is due to the width of the tunnel and the corresponding camera field of view,

small heading angle rotations results to different label of the images e.g. from 620

right label to left label.

3.2.3. Case 3: CNN Approach with Parrot Bebop 2

In this case, the Parrot Bebop 2 is used for autonomous navigation in the

rna

mine and only the CNN module is used for correcting the heading with velocity vd,x = 0.1 m/s and constant altitude of zd = 1 m is tested. The illumination of 625

310 lux from 1 m distance from the source is provided for this platform. The area is different from the collected training data-sets and the same trained network as case 2 is used. The main purpose of this case is to evaluate the performance of the stand-alone CNN module, while the on-board camera is different, the

Jou

light source is provided by a person following the platform, and the potential 630

fields component is not available. Figure 19 shows the examples of classification images, while flying autonomously in the mine. It can be seen that some classes of images are not identical but the CNN provides correct heading and avoid collision to the mine walls. It can be observed that the proposed CNN obtained the necessary information for correct image classification, without any 34

lP repro of

Journal Pre-proof

brightness adjustment of the images.

Brightness adjustment

On-board image

Brightness adjustment

On-board image

Left

Center

Right

Figure 19: Examples of images from forward-looking camera of the Parrot Bebop 2, while the MAV performs autonomous navigation in underground mine. The class of images from the

CNN is written above the images.The brightness adjustment with Adobe Photoshop [43] is

635

rna

applied offline for better visibility of images.

Figure 20 shows the estimation of velocities and reference values. The MAE for vx and vy is 0.09 m/s and 0.015 m/s respectively. Furthermore, Figure 21 depicts the altitude trajectory of the platform, while the MAE is 0.23 m. The frequency of heading rate commands generated from the CNN module is 4 hz. The heading rate command changes frequently with

Jou

640

same reasons mentioned in the second case. 3.3. Lessons Learned Throughout the experimental trials for the underground tunnel inspection

scenario, many different experiences were gained that assisted in the devel-

35

Estimated velocity Reference

vx [m/s]

0.4 0.2 0 −0.2 −0.4

0

50

0

50

0.1 vy [m/s]

lP repro of

Journal Pre-proof

0

−0.1

100

150

200

250

300

350

100

150 200 time [sec]

250

300

350

Figure 20: The desired velocities and estimated velocities of the MAV during the navigation

rna

in the third case with vx = 0.1 m/s.

z[m]

1

0.5

Jou

Altitude Reference

0

0

50

100

150 200 time [sec]

250

300

350

Figure 21: The desired altitude and the altitude of the MAV during the navigation in the third case with vx = 0.1 m/s and Bebop platform.

36

645

lP repro of

Journal Pre-proof

opment and tuning of the algorithms utilized. Based on this experience, an overview of the lessons learned is provided in the sequel with connections to the different utilized field algorithms. 3.3.1. MAV Control

Due to the nature of a floating object approach, identifying the explicit 650

problem is challenging. However there are multiple parameters that can affect the performance of the control scheme. The weights in the objective function

have a direct impact on the controller’s tracking performance, e.g. the higher

weight in altitude terms results in a smaller altitude error. Nonetheless, due to fluctuations in velocity estimation, eliminating the error in tracking cannot 655

be obtained, while too large values on the weights results in oscillation of the

platform. Moreover, the physical constraints of the MAV such as maximum

velocity, maximum acceleration, bounds on angles, etc should be taken into consideration based on the field trial conditions. As an example due to high

wind gusts in the underground tunnels the NMPC should have larger bounds 660

on angles for better wind gust compensation. Nonetheless, it is observed that high wind gusts cannot be compensated even with maximum bounds on the

rna

angles due to larger wind velocity than the maximum velocity of the platform. 3.3.2. Localization

Overall, the performance was substantial but alternative methods should 665

be studied to improve the performance and stability of this particular component. More specifically, sensor fusion should be part of the localization module, combining the merits and redundancy of various sensors into a solid solution.

Jou

3.3.3. Navigation

Potential Field: This components relies on 2D lidar and the update rate is

670

limited to 5 hz. Additionally, large angles for roll and pitch results in considering the ground as an obstacle. Vector Geometry Based: This module similarly to the potential field

method relies on 2D lidar, thus the same limitations are observed. Moreover, 37

lP repro of

Journal Pre-proof

due to the limited range of the 2D lidar and low sampling rate, the generated 675

heading rate commands result in an S shape path. Finally, it is observed that obtaining heading rate commands based on only one beam measurement for each

direction is not reliable. Instead an array of beams are selected for representing each direction, which results in smoother heading rate commands.

CNN: In contrary to the two other modules, CNN relies on input images 680

from a camera, thus the update rate is 30 hz. However, the CNN requires

training data-sets, LED light bar for the camera and there is no guarantee that the proposed solution will work in completely different mine environments, thus

there is a need of further investigation towards this direction. Additionally, during the field trials, it has been observed that the stand alone CNN, in some 685

cases, resulted in the MAV navigating close to the wall, as depicted in Figure 22,

due to the fact that the CNN does not provide y-axis reference velocity and only generates heading rate commands towards the tunnel axis. This is caused due to the lack of a 2D lidar sensor on the platform, which results to unavailability

of the potential field that is responsible for providing velocity commands on x 690

and y axes towards the open area, while respecting the specified safety distance. However, this case is avoided by utilizing a potential field module on top of the

CNN, while for such an operation, the CNN approach should be fused with a

rna

lidar, a fact that will increase the performance and the overall safety but it will

Jou

decrease the payload as well.

Figure 22: The MAV navigates close to the wall due to the stand-alone CNN module and the corresponding absence of a potential field module for obstacle avoidance in the Bebop case.

695

Finally, the method is providing discrete heading command rates compared 38

lP repro of

Journal Pre-proof

to the geometry vector based method, which results to bang-bang heading movements. However, both methods provide correct heading rate commands and avoid collisions.

3.3.4. MAV Developments 700

The proposed system has been developed in a cost-efficient manner, while carrying the minimum amount of sensors to enable autonomous flight under-

ground. As presented in the Sections above the high level obstacle avoidance scheme, as well as part of the navigation scheme, were based on a low cost 2D

lidar that was fixed on the top of the platform. Nevertheless, the particular 705

sensor selection caused issues during the experimental trials due to its limitations. Initially, the ranges above 4 m were corrupted with severe noise and were

filtered out, limiting the sensing range of the platform. Additionally, the update rate of the sensor was around 5 hz, which affected the performance of the platform navigation, since the lidar feedback was not reacting fast on the shape 710

variations of the environment.

Another on-board sensor of the proposed system is the optical flow sensor which was providing velocity measurements of the MAV. Initially, the sensor has been configured to operate at 1 m and all experimental trials have been

715

rna

performed at such height. Additionally, the placement of the sensor, close to

the center of mass, plays a major role in the velocity measurements and therefore it should be taken under consideration when designing the aerial platform. Another important factor that was tested during the experiments was the illumination of the ground. Generally, the optical flow sensor, and as a consequence the flights, were more stable when increasing the illumination levels towards the

720

ground. The ground of the tunnel was mainly planar with rocky parts on the

Jou

sides and small rock particles scattered along. The optical flow sensor, overall, was able to handle the rock particles along the tunnel but was unstable when the MAV was flying close to the rocky part on the sides. Furthermore, the shortcomings of the developed system observed during

725

the experimental trials that should be addressed were the payload and flight 39

lP repro of

Journal Pre-proof

time limitations. More specifically, the platforms were flying on the limits of

their payload capacity shortening the flight time. Additionally, in the proposed sensor suite there was not any sensor feedback from the ceiling to avoid collisions

in case of failure of the height controller. Finally, an issue faced while flying 730

was the communication between the operator and the MAV. Moreover, the communication was based on wi-fi and the connection was dropping when the platform was flying further away from the access point.

4. Conclusions

This article presents a novel framework to enable the deployment of MAVs 735

in dark underground mine environments. The concept of consumable aerial

tunnel inspectors is proposed, implemented as a floating object with low-cost sensor suite, relying on velocity commands to explore the mine tunnel. The

MAV controller is based on NMPC, while the estimation of the altitude, attitude and velocities are obtained from IMU, optical-flow measurements and one 740

beam lidar. In order to avoid collision with the wall surfaces the potential field

method based on 2D lidar measurement is implemented. Moreover, to correct the heading of the MAV toward tunnel axis, the heading rate command is gen-

rna

erated based on two methods: 1) a vector geometry method based on 2D lidar,

and 2) CNN classification based on a forward-looking camera stream. Further745

more, the on-board sensor data are used for generating maps of the visited area, supporting the remote operations. Finally, the framework has been evaluated in field trials in an actual dark underground mine in Sweden demonstrating the capability of low-cost and resource-constrained aerial vehicles to fly through tunnel confined spaces. Based on the obtained results it is observed that due to limited range of 2D lidar in the case of navigation with vector geometry

Jou

750

method the MAV follows the S shape path, and in the case of navigation with

the CNN method the platform almost follows the straight line. However due to the camera field of the view and width of the tunnel in the CNN approach the heading changes more frequently. Moreover, it is seen that the platform

40

755

lP repro of

Journal Pre-proof

without the potential field module navigates close to wall surfaces. In general, in all cases the platform has no collision and it flies until the end of the flight

time is reached. The article includes discussions about the lessons learned for the proposed method to provide guidance for the community for limitations of each method and challenges in dark underground mine tunnels.

760

5. Acknowledgement

We would like to thank, Boliden AB for providing the access to underground

mine and gave the opportunity to evaluate the proposed methods and Wolfit AB for the valuable guidance and support inside the mine.

References 765

[1] C. Kanellakis, S. S. Mansouri, G. Georgoulas, G. Nikolakopoulos, Towards Autonomous Surveying of Underground Mine Using MAVs, in: Interna-

tional Conference on Robotics in Alpe-Adria Danube Region, Springer, 2018, pp. 173–180.

[2] J. G. Rogers, R. E. Sherrill, A. Schang, S. L. Meadows, E. P. Cox, B. Byrne,

D. G. Baran, J. W. Curtis, K. M. Brink, Distributed subterranean explo-

rna

770

ration and mapping with teams of uavs, in: Ground/Air Multisensor Interoperability, Integration, and Networking for Persistent ISR VIII, Vol. 10190, International Society for Optics and Photonics, 2017, p. 1019017. [3] S. S. Mansouri, C. Kanellakis, E. Fresk, D. Kominiak, G. Nikolakopoulos, 775

Cooperative uavs as a tool for aerial inspection of the aging infrastructure,

Jou

in: Field and Service Robotics, Springer, 2018, pp. 177–189. [4] J. Matthaei, T. Kr¨ uger, S. Nowak, U. Bestmann, Swarm exploration of unknown areas on mars using slam, in: International Micro Air Vehicle Conference and Flight Competition (IMAV), 2013.

41

780

[5] SPARC,

lP repro of

Journal Pre-proof

Robotics

2020

multi-annual

roadmap,

https://www.

eu-robotics.net/cms/upload/topic_groups/H2020_Robotics_ Multi-Annual_Roadmap_ICT-2017B.pdf, (2016).

accessed:

November 2018

¨ [6] T. Ozaslan, G. Loianno, J. Keller, C. J. Taylor, V. Kumar, J. M. Wozen785

craft, T. Hood, Autonomous navigation and mapping for inspection of pen-

stocks and tunnels with MAVs, IEEE Robotics and Automation Letters 2 (3) (2017) 1740–1747.

[7] F. Cunha, K. Youcef-Toumi, Ultra-wideband radar for robust inspection drone in underground coal mines, in: 2018 IEEE International Conference 790

on Robotics and Automation (ICRA), IEEE, 2018, pp. 86–92.

[8] C. Papachristos, S. Khattak, F. Mascarich, K. Alexis, Autonomous navigation and mapping in underground mines using aerial robots, in: 2019 IEEE Aerospace Conference, IEEE, 2019, pp. 1–8.

[9] W. Burgard, M. Moors, C. Stachniss, F. E. Schneider, Coordinated multi795

robot exploration, IEEE Transactions on Robotics 21 (3) (2005) 376–386.

[10] J. A. P. Fentanes, R. F. Alonso, E. Zalama, J. G. Garc´ıa-Bermejo, A new

rna

method for efficient three-dimensional reconstruction of outdoor environ-

ments using mobile robots, Journal of Field Robotics 28 (6) (2011) 832–853. [11] S. Bhattacharya, N. Michael, V. Kumar, Distributed coverage and explo800

ration in unknown non-convex environments, in: Distributed Autonomous Robotic Systems, Springer, 2013, pp. 61–75.

Jou

[12] A. Cowley, C. J. Taylor, B. Southall, Rapid multi-robot exploration with topometric maps., in: IEEE International Conference on Robotics and Automation (ICRA), 2011, pp. 1044–1049.

805

[13] S. Shen, N. Michael, V. Kumar, Stochastic differential equation-based exploration algorithm for autonomous indoor 3d exploration with a micro-

42

lP repro of

Journal Pre-proof

aerial vehicle, The International Journal of Robotics Research 31 (12) (2012) 1431–1444.

[14] A. Bircher, M. Kamel, K. Alexis, H. Oleynikova, R. Siegwart, Receding 810

horizon next-best-view planner for 3d exploration, in: IEEE International Conference on Robotics and Automation (ICRA), 2016, pp. 1462–1468.

[15] S. P. Adhikari, C. Yang, K. Slot, H. Kim, Accurate natural trail detection using a combination of a deep neural network and dynamic programming, Sensors 18 (1) (2018) 178. 815

[16] A. Giusti, J. Guzzi, D. C. Ciresan, F.-L. He, J. P. Rodr´ıguez, F. Fontana, M. Faessler, C. Forster, J. Schmidhuber, G. Di Caro, et al., A machine learning approach to visual perception of forest trails for mobile robots, IEEE Robotics and Automation Letters 1 (2) (2016) 661–667.

[17] L. Ran, Y. Zhang, Q. Zhang, T. Yang, Convolutional neural network-based 820

robot navigation using uncalibrated spherical images, Sensors 17 (6) (2017) 1341.

[18] N. Smolyanskiy, A. Kamenev, J. Smith, S. Birchfield, Toward low-flying

autonomous MAV trail navigation using deep neural networks for environ-

825

rna

mental awareness, arXiv preprint arXiv:1705.02550.

[19] A. Kouris, C.-S. Bouganis, Learning to fly by myself: A self-supervised cnn-based approach for autonomous navigation, in: Intelligent Robots and Systems (IROS), 2018 IEEE/RSJ International Conference on, 2017. [20] S. S. Mansouri, C. Kanellakis, G. Georgoulas, G. Nikolakopoulos, Towards mav navigation in underground mine using deep learning, in: 2018 IEEE International Conference on Robotics and Biomimetics (ROBIO), IEEE,

Jou 830

2018, pp. 880–885.

[21] K. Schmid, P. Lutz, T. Tomi´c, E. Mair, H. Hirschm¨ uller, Autonomous vision-based micro air vehicle for indoor and outdoor navigation, Journal of Field Robotics 31 (4) (2014) 537–570. 43

835

lP repro of

Journal Pre-proof

[22] P. Gohl, M. Burri, S. Omari, J. Rehder, J. Nikolic, M. Achtelik, R. Siegwart, Towards autonomous mine inspection, in: Applied Robotics for the Power

Industry (CARPI), 2014 3rd International Conference on, IEEE, 2014, pp. 1–6.

[23] G. Nikolakopoulos, T. Gustafsson, P. Martinsson, U. Andersson, A 840

vision of zero entry production areas in minesthis work has been par-

tially funded by the sustainable mining and innovation for the future

research program., IFAC-PapersOnLine 48 (17) (2015) 66 – 68, 4th IFAC Workshop on Mining, Mineral and Metal Processing MMM 2015. doi:https://doi.org/10.1016/j.ifacol.2015.10.079. 845

URL

http://www.sciencedirect.com/science/article/pii/

S2405896315019552

[24] A. Manukyan, M. A. Olivares-Mendez, T. F. Bissyand´e, H. Voos, Y. Le Traon, Uav degradation identification for pilot notification using ma-

chine learning techniques, in: Emerging Technologies and Factory Automa850

tion (ETFA), 2016 IEEE 21st International Conference on, IEEE, 2016, pp. 1–8.

[25] M. Kamel, T. Stastny, K. Alexis, R. Siegwart, Model predictive control

rna

for trajectory tracking of unmanned aerial vehicles using robot operating system, in: Robot Operating System (ROS), Springer, 2017, pp. 3–39. 855

[26] K. Alexis, G. Nikolakopoulos, A. Tzes, Switching model predictive attitude control for a quadrotor helicopter subject to atmospheric disturbances, Control Engineering Practice 19 (10) (2011) 1195–1207. [27] G. Balamurugan, J. Valarmathi, V. Naidu, Survey on uav navigation in

Jou

gps denied environments, in: 2016 International Conference on Signal Pro-

860

cessing, Communication, Power and Embedded System (SCOPES), IEEE, 2016, pp. 198–204.

[28] A. S. Sathya, P. Sopasakis, R. Van Parys, A. Themelis, G. Pipeleers, P. Patrinos, Embedded nonlinear model predictive control for obstacle avoidance 44

lP repro of

Journal Pre-proof

using panoc, in: Proceedings of the 2018 European Control Conference, 865

2018.

[29] J.-J. Xiong, E.-H. Zheng, Position and attitude tracking control for a

quadrotor uav, ISA Transactions 53 (3) (2014) 725 – 731. doi:https: //doi.org/10.1016/j.isatra.2014.01.004.

[30] H. J. Stetter, Analysis of discretization methods for ordinary differential 870

equations, Vol. 23 of Springer Tracts in Natural Philosoph, Springer, 1973.

[31] J. Delmerico, D. Scaramuzza, A benchmark comparison of monocular visual-inertial odometry algorithms for flying robots, Memory 10 (2018) 20.

[32] S. Haykin, Kalman filtering and neural networks, Vol. 47, John Wiley & 875

Sons, 2004.

[33] D. Honegger, L. Meier, P. Tanskanen, M. Pollefeys, An open source and open hardware embedded metric optical flow cmos camera for indoor and

outdoor applications, in: Robotics and Automation (ICRA), 2013 IEEE International Conference on, IEEE, 2013, pp. 1736–1741.

[34] K. Gade, The seven ways to find heading, The Journal of Navigation 69 (5)

rna

880

(2016) 955–970.

[35] A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep convolutional neural networks, in: Advances in neural information processing systems, 2012, pp. 1097–1105. 885

[36] D. Cire¸san, U. Meier, J. Schmidhuber, Multi-column deep neural networks

Jou

for image classification, arXiv preprint arXiv:1202.2745.

[37] H. M. Bui, M. Lech, E. Cheng, K. Neville, I. S. Burnett, Using grayscale images for object recognition with convolutional-recursive neural network, in: IEEE Sixth International Conference on Communications and Electronics

890

(ICCE), 2016, pp. 321–325. 45

lP repro of

Journal Pre-proof

[38] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov, Dropout: a simple way to prevent neural networks from overfitting, The Journal of Machine Learning Research 15 (1) (2014) 1929–1958.

[39] Y. Bengio, Y. Grandvalet, No unbiased estimator of the variance of k-fold 895

cross-validation, Journal of machine learning research 5 (Sep) (2004) 1089– 1105.

[40] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980.

[41] J. Jackson, G. Ellingson, T. McLain, ROSflight: A lightweight, inexpensive 900

MAV research and development tool, in: 2016 International Conference on

Unmanned Aircraft Systems (ICUAS), 2016, pp. 758–762. doi:10.1109/ ICUAS.2016.7502584.

[42] S. Parrot, Parrot bebop 2, Retrieved from Parrot. com: http://www. parrot. com/products/bebop2. 905

[43] Adobe Photoshop. [link].

Jou

rna

URL https://www.adobe.com/products/photoshop.html

46

Journal Pre-proof

• •

•

The framework has been evaluated in multiple field trials in an underground mine in Sweden

Jou

rna

lP repro of

•

The navigation of resource constrained robots in unknown underground environments The framework focuses on the navigation, control and vision capabilities of the aerial platforms with low-cost sensor suites The navigation module for deducing the proper heading of the aerial platform along the tunnel is implemented in two different approaches based a) on a vector geometry method using 2D lidar and b) a deep learning method through a classification process using the on-board image stream

Journal Pre-proof Sina Sharif Mansouri PhD Candidate Robotics Group Division of Signals and Systems Luleå University of Technology

1st of August 2019 Article submission to Robotics and Autonomous Systems Editor in Chief

Dear Editor-in-Chief,

lP repro of

DATE: SUBJECT: TO:

This work has been partially funded by the European Unions Horizon 2020 Research and Innovation Programme under the Grant Agreement No. 730302 SIMS. Boliden AB provides the access to underground mine and gave the opportunity to evaluate the proposed methods and Wolfit AB provides guidance and support inside the mine.

Sincerely,

Jou

rna

Sina Sharif Mansouri

Journal Pre-proof

lP repro of

Author Bios Sina Sharif Mansouri is currently pursuing his Ph.D degree within the Control Engineering Group, Department of Computer Science, Electrical and Space Engineering, Luleå University of Technology (LTU), Luleå, Sweden. He received his master of science and bachelor of science from Technical University of Dortmund, Germany in 2014 and University of Tehran, Iran in 2012 respectively. He currently works in the field of robotics, focusing on control, navigation and exploration with multiple agents. Christoforos Kanellakis is currently pursuing his Ph.D degree within the Control Engineering Group, Department of Computer Science, Electrical and Space Engineering, Luleå University of Technology (LTU), Luleå, Sweden. He received his Diploma from the Electrical & Computer Engineering Department of the University of Patras (UPAT), Greece in 2015. He currently works in the field of robotics, focusing on the combination of control and vision to enable robots perceive and interact with the environment.

rna

Dariusz Kominiak is senior research engineer at the control engineering group at the division of Signals and Systems. He graduated from Technical University of Lodz in Poland in the field of automatic control and robotics with major control of industrial processes and applied computer science. His main interests are robotics and mechatronics.

Jou

George Nikolakopoulos is Professor on Robotics and Automation at the Department of Computer Science, Electrical and Space Engineering at Luleå University of Technology, Luleå, Sweden. His work is focusing in the area of Robotics, Control Applications, while he has a significantly large experience in Creating and Managing European and National Research Projects. In the past he has been working as project manager and principal investigator in Several R&D&I projects funded by the EU, ESA, Swedish and the Greek National Ministry of Research. George is the coordinator of H2020-ICT AEROWORKS project in the field of Aerial Collaborative UAVs and H2020-SPIRE project DISIRE in the field of Integrated Process Control. In 2013 he has established the bigger outdoors motion capture

Journal Pre-proof

Jou

rna

lP repro of

system in Sweden and most probably in Europe as part of the FROST Field Robotics Lab at Luleå University of Technology. In 2014, he has been nominated as LTU’s Wallenberg candidate, one out of three nominations from the university and 16 in total engineering nominations in Sweden. In year 2003, George has received the Information Societies Technologies (IST) Prize Award for the best paper that Promotes the scopes of the European IST (currently known as ICT) sector. His publications in the field of UAVs have received top recognition from the related scientific community, while have been several times listed in the TOP 25 most popular publications in Control Engineering Practice from Elsevier. In 2014 he has received the 2014 Premium Award for Best Paper in IET Control Theory and Applications, Elsevier for the research work in the area of UAVs, His published scientific work includes more than 150 published International Journals and Conferences in the fields of his interest.

lP repro of

Journal Pre-proof

Jou

rna

Sina Sharif Mansouri

Christoforos Kanellakis

lP repro of

Journal Pre-proof

Jou

rna

Dariusz Kominiak

George Nikolakopoulos