A vision-based landing system for small unmanned aerial vehicles using an airbag

ARTICLE IN PRESS Control Engineering Practice 18 (2010) 812–823

Contents lists available at ScienceDirect

Control Engineering Practice journal homepage: www.elsevier.com/locate/conengprac

A vision-based landing system for small unmanned aerial vehicles using an airbag Sungsik Huh, David Hyunchul Shim n Department of Aerospace Engineering, KAIST, 335 Gwahakro, Yuseong-gu, Daejeon 305-701, South Korea

a r t i c l e in f o

a b s t r a c t

Article history: Received 2 April 2009 Accepted 10 May 2010

Statistics show that the landing accounts for the largest portion of all mishaps of unmanned aerial vehicles (UAVs) due to many difficulties including limited situational awareness of the external pilot and the limited maneuverability during the low speed flight before touchdown. In this paper, a visionbased automatic landing system using a dome-shaped airbag is proposed for small UAVs. Its isotropic shape allows airplanes to approach in any direction to avoid crosswind unlike net-assisted landing. The dome’s distinctive color improves the detection owing to its strong visual cue. Color- and shape-based detection vision algorithms are applied for robust detection under varying lighting conditions. Due to the insufficient accuracy of navigation sensors, a direct visual servoing is used for terminal guidance. The proposed algorithm is validated in a series of flight tests. & 2010 Elsevier Ltd. All rights reserved.

Keywords: Unmanned aerial vehicles Automatic landing Direct visual servoing Image moments Geolocation

1. Introduction Landing is the most accident-prone stage in the entire flight envelope of both manned and unmanned airplanes. For commercial and military airplanes, the instrument landing system (ILS) has been developed and now widely used at major airports. It has dramatically improved the overall safety of landing even when the visibility is as poor as in Category III-A/B (Templeman & Parker, 1968). However, since the ILS is only available around certain airports and the onboard equipment needed for ILS is too heavy and complicated for typical UAVs, they are usually landed manually by pilots or portable external aiding systems. For manual landing, the pilot obtains visual cue by naked eyes or through the relayed video taken by the onboard camera. Piloting outside the vehicle needs a lot of training due to the limited situation awareness. As a consequence, a large portion of mishaps happen during the landing phase. Many fixed-wing military UAVs are known to suffer a significant portion of accidents due to human factors during landing and as for Pioneer, almost 70% of mishaps occur during landing (Manning, Rash, LeDuc, Noback, & McKeon., 2004; Williams, 2004). Therefore, it has been very much desired to automate the landing of UAVs, preferably without using expensive aiding systems. Automatic landing of airplanes on runway has been well investigated using various techniques for both manned and unmanned aircraft (Duranti & Malmfors, 2005; Looey & Joos, 2006; Malaek, Izadi, & Pakmehr, 2006). Global Hawk relies on a

n

Corresponding author. Tel.: + 82 42 350 3724; fax: +82 42 350 3710. E-mail address: [email protected] (D.H. Shim).

0967-0661/$ - see front matter & 2010 Elsevier Ltd. All rights reserved. doi:10.1016/j.conengprac.2010.05.003

high-precision differential GPS and a radar altimeter for landing (Loegering, 2002). For tactical UAVs, external aiding systems are favored. Sierra Nevada Corporation’s UCARS and TALS1 are externally located aiding systems consisting of tracking radars and onboard transponders, which measure the relative position and altitude of the inbound aircraft and relays back to its onboard flight controller for automatic landing. They have been successfully used for landing many military fixed-wing and helicopter UAVs such as Hunter or Fire Scout on a runway or even on a ship deck. Some UAVs can be retrieved in a confined space using nets or other special arresting devices. Scan Eagle is retrieved by a special arresting cable attached to a tall boom, to which the vehicle is precisely guided by a differential GPS.2 These external aids listed above rely on special equipment, which are not always available or applicable to smaller UAVs due to complexity, cost, or other limits from the operating environment. Therefore, automatic landing systems that are inexpensive, passive, and reliable are highly desirable. Vision-based landing has been found attractive since it is passive and does not require any special equipment other than a camera and a vision processing unit. A vision-enabled landing system will detect the runway or other visual markers and guide the vehicle to the touchdown point. There are a number of previous works, theoretical or experimental, for fixed-wing and helicopter UAVs (Bourquardez & Chaumette, 2007; Saripalli, Montgomery, & Sukhatme, 2002; Trisiripisal, Parks, Abbott, Liu, & Fleming, 2006). Notably, Barber, McLain, and Edwards (2007)

1 2

http://www.sncorp.com. http://www.insitu.com/scaneagle.

ARTICLE IN PRESS S. Huh, D.H. Shim / Control Engineering Practice 18 (2010) 812–823

Nomenclature (a,b,c) tuning parameters for color-based detection (IR,IG,IB) RGB values of a pixel (x,y) image coordinates fi Hu’s moments B, C, S body, camera, and local Cartesian (spatial) coordinate system, respectively

proposed a vision-based landing for small fixed-wing UAVs, where a visual marker is used to generate roll and pitch commands to the flight controller. In this paper, for the landing of small fixed-wing UAVs, a visionbased landing system using an air dome is proposed (Fig. 1). An airfilled dome secured on the ground serves as a shock-absorbing arrestor as well as a visual marker. It can be reliably detected by relatively simple but reliable vision algorithms, which is suitable for visual servoing of fast-moving airplanes under various lighting conditions. Since the position error of typical navigation sensors available for small UAVs are quite large compared with the size of the dome, the vehicle would not be able to land on it even if the flight control is capable of perfectly tracking the desired path based on its poor navigation solution only. Therefore, instead of using the dome’s coordinates estimated by fusing the vision with the navigation solution, it is proposed that the feature detection results are directly applied to steer the vehicle into the air dome, in a manner referred to as direct visual servoing. This paper is organized as follows. In Section 2, the overall approach and its component technologies are explained in detail. In Section 3, the experiment results of the proposed method are presented and discussed. In Section 4, the conclusion and closing remarks are given.

2. System description The proposed landing system consists of three major components: an air-filled dome, a vision processing system, and a flight control system ready for visual servoing. Before commencing the descent, the UAV looks for the dome using the images obtained by the onboard camera. When the onboard camera detects the dome, its location is estimated by combining the vehicle’s location

(fc, yc) RA/B k f (f,y,c) wCT

813

camera installation angle transformation matrix form B to A frame relative distance from camera to object focal length Euler angles (roll, pitch, and yaw, respectively) unit vector in the direction of line of sight in the camera coordinate system

calculated by the navigation system and the image coordinates of the dome. Once its location is computed, the vehicle starts descending on the glide slope that leads to the dome. As the vehicle approaches and the vision system locks on to the dome, the flight control switches from glide slope tracking to direct visual servoing, where the offset of the dome from the center of the image taken from the onboard front-looking camera is used as the error signals for the pitch and yaw control loops. Unlike the conventional landing, the vehicle does not flare but continues to fly along the glide slope into the dome with an incident angle that would not cause the vehicle to bounce off. Before the final impact, the vehicle maintains a speed slightly higher than its stall speed in order to maintain minimal maneuverability. In Fig. 1, the proposed landing procedure using the dome is illustrated. 2.1. Air dome The dome is constructed with sturdy nylon dyed with a strong red color, which would have a high contrast to typical outdoor environment. The prototype dome is a hemisphere of 4-m diameter. The dome can be easily transported in a compact package, inflated/deflated using a portable air blower or a chemical gas generator and secured to the ground using pegs. The dome can be used as a stand-alone air cushion for landing of small UAVs or as a backup air cushion when used with a net. Since the air-filled dome can absorb quite a large shock during landing, the vehicle would not need to make any special maneuver but simply fly into it at a low speed with a reasonable incident angle. The dome allows landing from any direction unlike the net-based recovery, which has to be installed to face the wind to avoid crosswind landing. The dome also functions as a visual marker. It provides a distinctive feature both in color and shape. Its vivid color can be

Geolocation

Beginning of glide slope Incident Angle Terminal visual servoing

Air Dome

Fig. 1. Proposed dome-assisted landing procedure of a small fixed-wing UAV using the dome.

ARTICLE IN PRESS 814

S. Huh, D.H. Shim / Control Engineering Practice 18 (2010) 812–823

detected from a long distance and the shape provides a relatively uniform projection when seen from any direction or altitude. Since the proposed vision-based approach is passive, the dome is not equipped with any sensors, signal lights, or communication devices. Therefore it does not consume any power or require special setup procedure.

2.2. Vision algorithms In order to achieve reliable landing on the dome using visionbased algorithms, the distinctive features of dome should be fully

RGB Color Space

exploited. For reliable detection, two detection algorithms based on color and shapes are proposed as explained in the following.

2.2.1. Color-based dome detection The dome’s color is a strong visual cue that distinguishes itself from other objects in the background. In RGB coding, a pixel at image coordinate (x,y) has three integers, (IR,IG,IB), each varying from 0 to 255. As the pixels that belong to the red dome vary a lot depending on lighting condition, shadow, color balance, or noise, a filtering rule is needed to determine if a pixel belongs to the dome or not. Based on many images of the dome taken under various conditions, the following filtering rule is proposed: aIB ðx,yÞ oIR ðx,yÞ r255 bIG ðx,yÞ o IB ðx,yÞ r 255 0 rc o IR ðx,yÞ

ð2:1Þ

250 where (a,b,c)¼ (1.5,1.5,20). The threshold levels are chosen rather inclusively for detection of wider color range. Fig. 2 visualizes the colors that are qualified as a dome using Eq. (2.1).

BLUE

200 a

150 100

c b

50 0

0 100 N

200 100 RED

200 0

E RE

G

Fig. 2. RGB region specified by Eq. (2.1) for color-based dome detection.

2.2.2. Moment-based dome detection When multiple red objects are found by the color-based detection method described above, another classification method is needed to tell whether an object is indeed the air dome or not. For this, a shape-based filtering is proposed. Among many algorithms such as template matching or image moment comparison, Hu’s method is used (Flusser, 2006; Hu, 1962) as it is computationally efficient for high-rate detection needed for vision-based landing of an airplane. Hu’s moment consists of seven moment invariants derived from the first three normalized central moments. Using the second- and third-order moments, the following Hu’s image moments, which are invariant to translation, rotation, and scale,

Fig. 3. Application of Hu’s method to various objects of similar red colors. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

ARTICLE IN PRESS S. Huh, D.H. Shim / Control Engineering Practice 18 (2010) 812–823

are derived:

f1 ¼ Z20 þ Z02 f2 ¼ ðZ20 Z02 Þ2 þ ð2Z11 Þ2 f3 ¼ ðZ30 3Z12 Þ2 þ ð3Z21 Z03 Þ2 f4 ¼ ðZ30 Z12 Þ2 þ ðZ21 þ Z03 Þ2

ð2:2Þ

815

in this case, are discarded because the original shape can be overly distorted due to discretization or it can be an artifact created by noise. Next, image moments are computed from its pixel distribution using Eq. (2.2). These moments can be used for determining the pixel area and the geometric profile by comparing them with the reference values in the constructed database and, as a result, the right side blob is identified as the dome. It is possible that additional information on the pose of the vehicle can be extracted from the moments for visual servoing (Tahri and Chaumette, 2004). However, it has been discovered that, when the image is corrupted with noise and interlace lines by the video transmitter/receiver and the frame grabber, the Hu’s moments in Table 2 tend to vary too much for pose estimation. The pose estimation can be further disrupted by dropped frames due to video link problems or frames without a dome when the camera points away from the dome. Therefore, the image moments are used only for dome detection in this paper.

where Zij are the ith and jth order normalized moments with regard to horizontal axis and vertical axis, respectively. Since the dome is projected to a range of shapes between a semicircle to a full circle, its Hu’s moments do not vary much regardless of the distance and the viewing angle. Therefore, Hu’s moment can be a suitable invariant characterization of the target shape. The first four image moments are considered enough for detection. Hu’s moment-based approach is applied to various red objects in Fig. 3 and their moment values are listed in Table 1. The rooftop and the automobile in Fig. 3 have relatively larger Hu’s moments while the stop sign and the air dome have comparable first moments due their round shapes. Higher-order moments of the road sign are much smaller than those of the air dome since the road sign has a smaller area for given number of pixels. Although the dome seen directly above would look much like the road sign, such cases are excluded because the camera is mounted to see only the side of the dome during landing. In Fig. 4, the image processing results of air domes seen from various viewpoints are presented. Once a database of Hu’s moments of these domes is built, the average and standard deviation values of the dome can be determined as shown in Table 2. Based on these values, the contour with the largest area in a given image is finally declared as the air dome. On the other hand, when none of candidates has Hu’s moments in correct ranges, the algorithm concludes that the air dome does not exist in that image. In order to validate if the proposed algorithms can detect the dome reliably in the presence of other similar color objects nearby, a number of sample images taken during flight are tested. In Fig. 5, an in-flight image with two nearby red objects, a dome seen from the side and a red cross sign, are shown. The colorbased algorithm detects two red blobs. Prior to applying the moment method, the candidate blobs are filtered based on their pixel area, i.e., the number of pixels. Blobs with pixel areas smaller than minimum, 10  10 from the original image size of 640  480

where k 40 and wCT is the unit vector in the bore sight direction of the camera and its value is [1 0 0] (Fig. 6). The matrix RS/B is a standard ZYX Euler angle transformation characterized by yaw, pitch, and roll angles, where S and B denote the local Cartesian coordinate and the body coordinate systems, respectively. In Eq. (2.3), XSB is estimated by GPS-aided inertial navigation system (INS) and XSC=B , the relative distance between the camera and the C.G. of the airplane, is directly measured. The relative

Table 1 Hu’s moments values for samples in Fig. 3.

Table 2 The mean and standard deviation of Hu’s moment values for the domes in Fig. 4.

f1 f2 f3 f4

2.3. Geolocation algorithm Once the image coordinates of the dome is found by the detection algorithms introduced above, its location in a local Cartesian frame can be computed by Eq. (2.3). This process is referred to as geolocation, and Fig. 6 shows the coordinate systems and geometric relationship of the UAV and the dome. The vector from the origin of the local Cartesian coordinate system (an arbitrary point in the landing area) to the dome can be written as XST ¼ XST=C þ XSC=B þ XSB ¼ RS=B RB=C XCT=C þ RS=B XBC=B þXSB ¼ kRS=B RB=C wCT þ RS=B XBC=B þ XSB

ð2:3Þ

Rooftop

Stop sign

Automobile

Air dome

Hu’s moment

Mean

Standard deviation

0.518446 0.226489 0.000618 0.000323

0.204633 0.000712 0.000099 0.000282

0.244057 0.029964 0.000066 0.000001

0.213385 0.018256 0.000771 0.000053

f1 f2 f3 f4

0.197248 0.011668 0.000386 0.000028

0.012811 0.004468 0.000210 0.000025

Fig. 4. Application of Hu’s method to domes seen from various viewpoints.

ARTICLE IN PRESS 816

S. Huh, D.H. Shim / Control Engineering Practice 18 (2010) 812–823

Fig. 5. Dome detection results using color- and moment-based methods.

Fig. 6. Coordinate systems and geometric relationship of an airplane in flight.

distance k from the camera to the object is unknown unless a separate measurement using a ranging devise such as a laser range finder is available. If the altitude of the target is known, however, the last equation of Eq. (2.3) can be solved for the horizontal coordinates (x,y) of the dome and the relative distance k. along the unit vector wCT The camera installation angles fc and yc that are rotation in terms of XC and YC, respectively, should be compensated in the transformation matrix from body to camera coordinate. Therefore, the transformation RB/C is found by RB=C ðfc , yc ,xT ,yT Þ 3 2 1 0   0     7 6 y yT 7 6 0 cos fc tan1 T sin fc tan1 7 6 f f ¼6       7 7 6 y y 5 4 T T 0 sin fc tan1 cos fc tan1 f f       3 2 xT 1 xT 0 sin yc þ tan1 7 6 cos yc þ tan f f 7 6 7 6 0   1 0 6 7      7 6 5 4 1 xT 1 xT sin yc þ tan 0 cos yc þtan f f

(triangles) where the image was taken. The dome’s position measured by a separate differential GPS is shown as a larger red circle for reference. Except for one outlier at (62,  165), the geolocation results show some biased error in the range of 30 and 40 m in x- and y-directions, respectively. Obviously from this result, the geolocation alone will not be sufficient for precision guidance to the dome. The relatively large error of the geolocation result is mainly attributed to the errors in the attitude and pose of the vehicle and the camera.

2.4. Direct visual servoing

ð2:4Þ where tan  1(yT/f) and tan  1(xT/f) are the terms to convert the coordinates in the image plane to the local Cartesian coordinates. The forward looking camera used in this paper was installed with (fc,yc)¼(0,851). Fig. 7 shows a geolocation result obtained from a test flight. The image coordinates of the dome at each image is applied to Eq. (2.3). The estimated coordinates of the dome (smaller red dots) are plotted with the corresponding vehicle’s location

The image features detected by the algorithms in Section 2.2 is used to estimate the location of the dome as described in Section 2.3. However, the estimated position by the geolocation method is not accurate enough for terminal guidance due to the large error of the low-cost GPS-aided INS and the imperfectly calibrated optic system. The accuracy of the navigation solution can be improved by vision-based recursive estimation algorithms (Kim and Sukkarieh, 2007) but it is not suitable for the scenario considered in this paper since the vehicle makes a single pass over a scene without many distinctive features except for the dome. Even if the dome’s position is accurately known and the flight control system can track any given trajectory perfectly, the navigation error will be still larger than the size of the dome, making it very difficult to achieve consistent landing on a small target. Therefore, a direct visual servoing is proposed, where the image processing result is directly fed into the flight controller of the vehicle. According to Hutchinson, Hager, and Corke (1996), this method is classified as image-based (IB) as opposed to position-based (PB) method, which is in fact similar to the

ARTICLE IN PRESS S. Huh, D.H. Shim / Control Engineering Practice 18 (2010) 812–823

817

In-flight Geolocation -260 UAV

-250

LCC Y [m]

lineofsight

-240

estimateddome

-230

truedome

-220 -210 -200 -190 -180 -170 -40

-20

0

20 LCC X [m]

40

60

Fig. 7. Geolocation of air dome during flight.

geolocation presented in Section 2.3. Unlike typical applications where the vision sensor is mounted on a secure platform such as robot arms, position-based method has a significant disadvantage over IB method if the platform’s position estimates contains significant amount of error. Suppose the camera is mounted to look forward in the x-axis of the body frame of the vehicle. As illustrated in Fig. 6, the offset coordinates of the dome from the center of the image can be directly associated with the heading and pitch angle deviations such that

Dc ¼ arctanðyT =f Þ Dy ¼ arctanðxT =f Þ:

ð2:5Þ

These two equations are a substitute for an interaction matrix used for robot visual servoing since fixed-wing airplanes have different dynamical characteristics. The camera can be installed so that the dome appears at the center of the image, i.e. Dc ¼ Dy ¼0 when the vehicle flies directly into the dome. Therefore, the

control in heading and pitch axes should be made to minimize these errors in such a way that the heading and pitch angle deviation is directly fed to the controllers in the corresponding loop, unlike the conventional flight control that computes the error between the reference and the actual navigation parameters. Since the pitch and the yaw deviation angles computed from images are directly sent to the pitch and the yaw controllers, respectively, the heading and pitch output from INS are not needed. This approach is simple and robust against the navigation errors in the low-cost GPS-aided INS. As the dome-assisted landing can be viewed as a ‘‘controlled collision’’, the airplane simply flies into the dome with an incident angle so that the vehicle would not bounce off. Also, taking advantage of the dome’s isotropic shape, an airplane can approach the dome from any direction to avoid crosswind landing unlike the case with arresting net, which must be approached from only one direction. If the wind direction is known or estimated by wind estimator (Kumon, Mizumoto, Iwai, & Nagata, 2005), the path planner can generate a glide slope that always leads the vehicle to

ARTICLE IN PRESS 818

S. Huh, D.H. Shim / Control Engineering Practice 18 (2010) 812–823

Fig. 8. Direct visual servoing scheme.

Fig. 9. Heading and pitch angle error controller for direct visual servoing.

Fig. 10. KAIST BWB UAV testbed.

the dome in head wind condition. In Figs. 8 and 9, the flight control system with the proposed direct visual servoing is presented. During the normal waypoint navigation and the glide slope tracking, a conventional multi-loop controller that tracks the desired yaw angle and the altitude is engaged (Huh, 2009). When the vehicle is in the direct visual servoing mode, the pitch and the yaw angle errors defined in Eq. (2.5) are sent to the flight controller shown in Fig. 9 so that Dc-0 and Dy-Dy0. Note that the offset in pitch angle is introduced by the camera mounting angle. The heading error is fed into the inner roll angle control

loop, which commands the elevon to achieve a bank-to-turn maneuver for heading control.

3. Experiment result 3.1. Experiment setup The automatic landing experiment is performed using a blended wingbody (BWB)-based UAV (Fig. 10). BWBs are known

ARTICLE IN PRESS S. Huh, D.H. Shim / Control Engineering Practice 18 (2010) 812–823

819

Speed Controller

Motor

Battery

Receiver

Servos

Servo Reader/ Driver/ Sensor Board

PWM

Pitot Tube

RS-232

Flight Management Computer QNX, PC104 Celeron 400MHz

Video TX Camera

Airframe

Video Overlay

RS-232

IMU

RS-232

GPS

RS-232

Wireless Modem

Video RX

Ground Station

Fig. 11. Avionics and hardware architecture.

to offer a substantially larger payload capability per airframe weight due to the extra lift generated by the airfoil-shaped fuselage. It is constructed with sturdy expanded polypropylene (EPP), which is reinforced with carbon fiber spars embedded at strategic locations. The vehicle is resilient to shock and crash, a highly welcomed trait for landing experiments. The airplane is launched by a catapult and lands on its belly so a runway is not necessary. This BWB design has only two control surfaces at the trailing edge of wings, known as elevons, which function as both aileron and elevator. At the wingtips, vertical winglets are installed to improve static directional stability. The vehicle has a DC brushless motor mounted at the trailing edge of the airframe powered by lithium-polymer cells. The same battery powers the avionics to reduce the overall weight. The large payload compartment (30 cm  15 cm  7.5 cm) in the middle of the fuselage houses the flight computer, IMU, battery, and radio receiver (Fig. 11). The flight control system is built around a PC104-complient industrial computer with a Pentium processor running at 400 MHz. The ample processing power is suitable for computationally heavy flight control algorithms. The navigation system consists of low-cost IMU and a single GPS. U-blox Supersense GPS offers outstanding satellite tracking even when the vehicle makes very dynamic maneuvers. Inertial Science’s MEMS-IMU is used for INS. The flight computer performs loosely coupled GPS-aided INS algorithm. The vehicle communicates with the ground station through a telemetry link at 900 MHz. A relatively high-bandwidth communication device (  1 Mbps) is desirable to reduce the latency (Table 3). The flight control loop consists of multiple proportionalintegral-differential (PID) gains around roll, pitch, yaw, altitude, and cruise speed dynamics (Huh, 2009). The BWB platform does

Table 3 Specification of KAIST BWB UAV. Base platform Dimensions

Weight Powerplant Operation time Avionics

Vision system

Operation Autonomy

StingRay 60 by Haak Works Blended wingbody Wing span: 1.52 m(W) Wing area: 0.52 m2 Aspect ratio: 4.43 2.6 kg (fully instrumented) Axi 2217/16 DC Brushless Motor Lithium-ion-polymer (11.1 V 5300 mAH) 410 min Navigation: single GPS-aided INS GPS: U-Blox Supersense 5 IMU: Inertial science MEMS IMU Differential/absolute pressure gauges Flight Computer: PC104 Pentium-M 400 MHz Communication: 900 MHz Wireless Ethernet Color CCD Camera : KPC-S226C Image resolution : 640  480 pixels Focal length : 670 , FOV : 701 Frame grabber : USB 2.0 Analog video capture kit 2.4 GHz analog transmitter Catapult-launch, body landing Speed, altitude, heading, altitude hold/command Waypoint navigation/automatic landing

not have a rudder so the heading control is achieved by bank-toturn only. The absence of rudder causes strong adverse yaw behavior (Nickel & Wohlfahrt, 1994), which can be quite detrimental to precision heading control as shown in the experiment result in Fig. 13. The BWB does not have clearly defined wings and fuselages and many existing aerodynamic modeling tools are known to be less effective. Therefore, the control gains are found from a model identified using the vehicle response to sine

ARTICLE IN PRESS 820

S. Huh, D.H. Shim / Control Engineering Practice 18 (2010) 812–823

UAV

Flight data

Modem 900MHz

Flight Computer

Sensors

Ground Control System

Visual servoing command

Video

Video TX 2.4GHz

Camera

Modem 900MHz

Ground Computer USB

Video RX 2.4GHz

Frame Grabber

Fig. 12. Experiment setup for vision-based landing.

80

alt.: Kalman filter alt.: GPS alt.: Ref. trajectory

70

Switches to Direct Visual Servoing at 5.8 sec

50

40

LCC altitude [m]

altitude [m]

60

landing at t = 9.1sec

30

80

60

Dome

40

-50

-350

0

1

2

3

4

5

6

7

8

x C LC

LCC y [m]

vision-based landing stage

automatic landing stage 10

0

-250

20

-200 -150

9

[m

]

-300

50

10

Roll [deg]

50

Ref. Actual

0

body vel [m/s]

time [sec] 30

vx

20 10

vy

0

vz

-10

0

1

2

3

4

5

6

7

8

9

Ref. Actual

20

0

1

2

3

4

5

6

7

8

9

Ref. Actual

120

Yaw [deg]

10

100 80 60 40 0

1

2

3

4

5

6

7

8

9

10

2

3

4

5

6

7

8

9

10

ax ay

-20

az

-40 0

-20

1

0

0

control input [PWM] angular rate [deg/s]

Pitch [deg]

10

body acc [m/s2]

0 -50

1

2

3

4

5

6

7

8

9

10

100

p q r

0 -100 0

1

2

3

4

5

6

7

8

9

10

2

left elevon right elevon

1.5

0

1

2

3

4

time [sec]

5

6

7

8

9

10

time [sec]

Fig. 13. Flight test results of vision-based landing using the air dome.

sweep-like input entered by the pilot during actual flight (Huh, 2009). The onboard vision system consists of a camera mounted at the nose of the airframe, a video overlay board, and a 2.4 GHz video transmitter with an antenna. The camera is mounted to look slightly downward so that the vehicle can see the ground in trimmed flight. Due to the payload limit, the image processing is currently done on the ground station computer. Image processing

and vision algorithm are processed in a laptop computer with Pentium Core 2 Duo CPU (2 GHz) and 2 GB memory. The vision algorithms introduced in Sections 2.2–2.4 are coded in Microsoft Visual C++ using OpenCV library.3 The estimated location of the dome and the angle deviations are transmitted back to the

3

Intel Open Source Computer Vision Library.

ARTICLE IN PRESS S. Huh, D.H. Shim / Control Engineering Practice 18 (2010) 812–823

821

-100 trial 1 trial 2 trial 3 trial 4

trial 1 trial 2 trial 3 trial 4

-50 altitude [m]

LCC x [m]

90 80 70 60 50

0 -400

Dome

50 -350

-300

-250 LCC y [m]

-200

-100 -350

-150

-80 -60

-300 LCC y [m

-40 ]

-250

-20 0

-200

CC

m]

x[

L

Dome

Fig. 14. Trajectories of test UAV from four vision-based landing experiments.

airplane over the telemetry link in real time. It is noted that the onboard vision processing is more desirable since the transmitted image can be degraded by noise and flickering. The time delay caused by video transmission and frame capture can also adversely affect the direct visual servoing. As of now, the image processing runs at every 240 ms. The experiment setup is illustrated in Fig. 12.

3.2. Flight test results Using the experiment setup described so far, the proposed vision-based algorithm is validated in a series of test flights. The dome is secured on the ground and its local Cartesian coordinates are measured by a differential GPS for reference. After the INS is initialized, the vehicle is launched using a catapult at a speed excess of 25 m/s. When the landing is commanded, the vehicle starts looking for the dome using its onboard camera. When the dome is in the view, the vision system geolocates the dome’s position using the algorithm in Section 2.3. As mentioned above, the estimated location of the dome may not be accurate enough for precision landing. Therefore, the vehicle initially flies following the glide slope using the navigation solution only. When the vision algorithm locks on to the dome during the descent, it switches to the direct visual servoing. Therefore the vehicle is commanded to maintain the constant airspeed of 20 m/s, the minimal safe speed of the test plane for landing maneuver (the stall speed is about 15 m/s). This landing procedure contrasts with the conventional landing on a runway, where the airplane needs to flare in order to reduce the vertical descent speed and allow the main landing gear to touch the ground first. Since the BWB-type test plane without any vertical and horizontal tail planes has very limited stall recovery capability and severely limited maneuverability in slow, high-alpha flight, flare during the final descent was found not suitable. In Fig. 13, a direct visual servoing test result is presented. At t ¼0, the vehicle is commanded to land. The glide slope starts at about 40 m above ground level (AGL). Until the dome is detected by the vision system, the vehicle is controlled by the waypoint navigator, which tracks the reference heading and the altitude commands shown in Fig. 13. About 100 m away horizontally and 20 m vertically from the dome, the vision system detects the dome and the direct visual servoing begins. In this mode, the flight controller is fed with the pitch and the heading angle errors computed by Eq. (2.5). The large fluctuation in the yaw angle is caused by the yaw controller that tries to track the heading of 701 from the initial heading angle of 1001. As mentioned above, since

the BWB test plane does not have a rudder, the yaw control performance is somewhat limited due to the large Dutch roll and adverse yaw. In less than 4 s, the vehicle travels about 80 m horizontally until impact on the dome. At the time of impact (t ¼9.1 s), the vehicle experiences about 1g in x-direction and 3g acceleration in z-direction, where the overall magnitude of 3.1g is well under the structural load limit of typical airframes. At around t¼ 6.5 s in Fig. 13, the flight controller momentarily switches back to the waypoint mode from the visual servoing mode because of a failure of the dome detection algorithm due to video transmission noise. When the vision algorithm fails to detect the dome in the frame, the flight controller is programmed to switch back to the waypoint navigation as a safety feature. Due to the sudden mode change, the vehicle experiences sudden jolt especially in the pitch direction. The altitude of the dome was initially estimated to be 51 m in absolute altitude as shown with the reference trajectory (red line), but it turned out to be 41 m according to the altitude measurements from GPS and Kalman filter at the time of impact. This observation advocates that the proposed image-based approach, i.e., the direct visual servoing achieves a precision landing that would have been impossible with position-based approach. The photographs taken from this flight test are given in Fig. 15, where one can see that dome was reliably detected. In Fig. 14, the paths from four consecutive flight tests are drawn together for comparison, where the flight test result in Fig. 13 is labeled as trial 2. In these trials, while all the parameters of the vision and the flight controllers were fixed, the vehicle entered the glide slope with all different initial conditions including altitude, heading, and speed. One can see that the initial offset of the heading affects the horizontal maneuver of the vehicle. In trial 4, when the vehicle initially flying directly towards the dome, the vehicle does not make large corrective maneuvers unlike trial 2, where the initial heading offset causes a fluctuating flight path both in horizontal and vertical directions. It is caused by the inherent weak coupling between the horizontal and vertical dynamics of a fixed-wing airplane. In all cases, the vision system was able to successfully recognize the dome and command the flight controller to guide the vehicle into the dome accurately and reliably.

4. Conclusion In this paper, a vision-based landing system for fixed-wing small unmanned aerial vehicles (UAVs) using an inflated air dome is presented. Its distinctive color provides strong yet passive

ARTICLE IN PRESS 822

S. Huh, D.H. Shim / Control Engineering Practice 18 (2010) 812–823

Fig. 15. Images of test UAV landing on the dome (right column) and captured video image taken from the onboard camera (left column).

visual cue for the onboard vision system, which runs a fast and robust color tracking algorithm for geolocation and visual servoing. The air-filled dome allows airplanes to approach from any direction, unlike the net-based approach. Since the onboard navigation sensors based on low-cost sensors cannot provide accurate enough navigation solutions, a direct visual servoing is implemented for precision guidance to the dome. The proposed idea has been validated in a series of experiments using a blended

wingbody (BWB) airplane to show that the proposed system is viable approach for landing of small fixed-wing UAVs.

Acknowledgments The authors gratefully acknowledge the financial support by Korea Ministry of Knowledge and Economy.

ARTICLE IN PRESS S. Huh, D.H. Shim / Control Engineering Practice 18 (2010) 812–823

Appendix A. Supplementary material Supplementary data associated with this article can be found in the online version at doi:10.1016/j.conengprac.2010.05.003.

References Barber, B., McLain, T., Edwards, & B. (2007). Vision-based landing of fixed-wing miniature air vehicles. AIAA infotech@ aerospace conference and exhibit. Bourquardez, O., Chaumette, & F. (2007). Visual servoing of an airplane for autolanding. In Proceeding of the 2007 IEEE/RSJ international conference on intelligent robots and systems. Duranti, S., & Malmfors, V. (2005). Flight test of the autonomous take off and landing functions of the SHARC technology demonstrator. In Flight Test – Sharing Knowledge and Experience (pp. 18-1–18-18). Meeting Proceedings RTO-SCI-162, Paper 18. Flusser, J. (2006). Moment invariants in image analysis. Proceedings of World Academy of Science, Engineering, and Technology Hu, M.-K. (1962). Visual pattern recognition by moment invariants. IRE Transactions on Information Theory Huh, S. (2009). A vision-based automatic landing of fixed-wing UAVs. Master’s Thesis, KAIST. Hutchinson, S., Hager, G. D., & Corke, P. I. (1996). A tutorial on visual servo control. IEEE Transactions on Robotics and Automation, 12(5), 651–670. Kim, J., & Sukkarieh, S. (2007). Real-time implementation of airborne inertialSLAM. Robotics and Autonomous Systems, 55, 62–71.

823

Kumon, M., Mizumoto, I., Iwai, Z., Nagata, M. (2005). Wind estimation by unmanned air vehicle with delta wing. In Proceedings of the 2005 IEEE international conference on robotics and automation, Barcelona, Spain, April 2005. Loegering, G. (2002). Landing dispersion results—global Hawk auto-land system. AIAA’s first technical conference and workshop on unmanned aerial vehicles. Looey, G., & Joos, H.-D. (2006). Design of autoland controller functions with multiobjective optimization. Journal of Guidance, Control, and Dynamics, 29(2), 475–484. Malaek, S. M. B., Izadi, H. A., & Pakmehr, M. (2006). Flight envelope expansion in landing phase using classic, intelligent, and adaptive controllers. Journal of Aircraft, 43(1), 91–101. Manning, S. D., Rash, C. E., LeDuc, P. A., Noback, R. K., McKeon, & J. (2004). The role of human causal factors in U. S. army unmanned aerial vehicle accidents. USAARL Report No. 200411. Nickel, K., & Wohlfahrt, M. (1994). Tailless aircraft in theory and practice. Butterworth-Heinemann. Saripalli, S., Montgomery J. F., Sukhatme, & G. S. (2002). Vision-based autonomous landing of an unmanned aerial vehicle. In Proceedings of the IEEE international conference on robotics & automation. Tahri, O., Chaumette, & F. (2004). Image moments: generic descriptors for decoupled image-based visual servo. In Proceedings of the IEEE International conference on robotics & automation. Templeman, J. E., & Parker, R. H. (1968). Boeing/Sperry automatic landing System 727 airplane. Journal of Aircraft, 5(3), 285–291. Trisiripisal, P., Parks, M. R., Abbott, A. L., Liu, T., & Fleming, G. A. (2006). Stereo analysis for vision-based guidance and control of aircraft landing. In 44th AIAA aerospace sciences meeting and exhibit. Williams, K. W. (2004). A summary of unmanned aircraft accident/incident data: human factors implications. U.S. Department of Transportation Report, No. DOT/ FAA/AM04/24.