Recognition of a landing platform for unmanned aerial vehicles by using computer vision-based techniques

Expert Systems With Applications 76 (2017) 152–165

Contents lists available at ScienceDirect

Expert Systems With Applications journal homepage: www.elsevier.com/locate/eswa

Recognition of a landing platform for unmanned aerial vehicles by using computer vision-based techniques J.A. García-Pulido a,∗, G. Pajares b, S. Dormido a, J.M. de la Cruz c a

Dept. Computer Science and Automatic Control, National University of Distance Education, 28040 Madrid, Spain Dept. Software Engineering and Artificial Intelligence, Faculty of Computer Science, Complutense University, 28040 Madrid, Spain c Dept. of Computer Architecture and Automatic, Faculty of Physics, University Complutense of Madrid 28040 Madrid, Spain b

a r t i c l e

i n f o

Article history: Received 5 September 2016 Revised 8 January 2017 Accepted 19 January 2017 Available online 20 January 2017 Keywords: Automatic expert system Image segmentation Landing platform UAV Pattern recognition

a b s t r a c t The use of Unmanned Aerial Vehicles (UAVs) is growing significantly for many and varied purposes. During the mission, an outdoor UAV is guided by following the planned path using GPS signals. However, the GPS capability may become defective or the environment may be GPS-denied, and an additional safety aid is therefore required for the automatic landing phase that is independent of GPS data. Most UAVs are equipped with machine vision systems which, together with onboard analysis, can be used for safe, automatic landing. This contributes greatly to the overall success of autonomous flight. This paper proposes an automatic expert system, based on image segmentation procedures, that assists safe landing through recognition and relative orientation of the UAV and platform. The proposed expert system exploits the human experience that has been incorporated into the machine vision system, which is mapped into the proposed image processing modules. The result is an improved reliability capability that could be incorporated into any UAV, and is especially robust for rotary wing UAVs. This is clearly a desirable fail-safe capability. © 2017 Elsevier Ltd. All rights reserved.

1. Introduction 1.1. Problem statement The use of UAVs (Unmanned Aerial Vehicles) is growing significantly for a wide range of purposes and missions (Pajares, 2015). Once the mission ends, the UAV must land onto a safe place, such as a secure platform. The autonomous navigation is based on GPS, allowing the UAV to find the platform location. However, when GPS data are not available, the final phase of landing can be achieved using a machine vision system that recognizes a patented geometric figure (Cruz, Sánchez, & Pajares, 2012). Fig. 2. shows this figure in place on the landing platform. Additionally, the platform could be used as reference for stabilization and control (Lungu, 2012; Lungu & Lungu, 2013) for safety considerations during landing. The geometric pattern used for this purpose is unique and could not occur naturally. The figure is made up of a collection of geometric shapes combined in such a way as to try to ensure that if a part is blinded, the expert system can still work effectively,

∗

Corresponding author. E-mail addresses: [email protected] (J.A. García-Pulido), [email protected] (G. Pajares), [email protected] (S. Dormido), [email protected] (J.M. de la Cruz). http://dx.doi.org/10.1016/j.eswa.2017.01.017 0957-4174/© 2017 Elsevier Ltd. All rights reserved.

and the UAV landed. Based on these assumptions, the problem is then to identify the constituent geometric shapes on the platform through the images captured with the onboard vision system. This uses a technique based on structural geometric relations between these shapes which are mapped as a degree of confidence or probability according to the number of relations correctly identified. For our evaluation and testing we use a quadrotor md-400, Fig. 1, along with two different Samsung Galaxy mobile devices that capture and process the images. 1.2. Summary of available methods Several strategies have been proposed that combine computer vision with geometric shapes for autonomous flight: (1) Guili, Yong, Shengyu, Yuehua, and Yupeng (2009): a Tshaped figure was proposed to determine the angle that provides the relative orientation of the figure and the UAV. They used infrared radiation to achieve a high temperature contrast between the figure used to identify the landing place and the environment. This method uses the actual shape of the target on the ground and the start of the disappearance of the line, generating an error that is smallest when the UAV is near the runway. Compliance with the basic requirements of autonomous landing of UAV is achieved.

J.A. García-Pulido et al. / Expert Systems With Applications 76 (2017) 152–165

Fig. 1. Real UAVs flying during the testing process.

Computer vision techniques employed are pattern recognition using invariant moments related, segmentation and extraction of edges. Although this technique uses infrared radiation instead of a conventional visible light photographic picture, the simplicity of the figure pattern may cause difficulties in outdoor environments. These can be easily confused with natural objects because they do not consider design elements such as texture and color. (2) Lange, Sünderhauf, and Protzel (2008): a figure was designed that consists of several concentric white rings with specific diameters on a black background to enable identification and differentiation of each ring. The difference between outer and inner ring size ratios increases moving away from the centre of the figure. Hence, the visible part of the platform is identifiable even when not all the rings are within the field of view – which can occur when the air vehicle is at a low altitude. The relative height is determined by processing the camera image. This method assumes that the UAV is always parallel to the ground, ignoring the angles of inclination / orientation from the figure. However, this can be a significant limitation of this method, especially if used for landing UAVs that cannot fly vertically, and rather at an angle of inclination to the horizontal. Determining the orientation and these angles is often very important for most vehicles landing in outdoor environments where the platform is not horizontal, such as on a boat. (3) Saripalli, Montgomery, and Sukhatme (2002): like both the preceding techniques, this method is based on the use of a platform that also has a geometric figure silk-printed on its surface. It uses a drawing shape of a white “H”, typical of heliports. The position of the centre-of-gravity of the image is extracted from the figure captured by the camera in a two-dimensional Cartesian plane. The image also allows determination of the angle of orientation of the figure with respect to the UAV. To extract this information from the image, the following computer vision techniques are applied: thresholding, filtering, segmentation and labelling of related components. However, the symmetry and morphological simplicity of the figure results in limitations. If part of it is blind – which frequently occurs in outdoor environments, where you cannot control the brightness and other lighting parameters - it may be impossible to identify the figure or determine the orientation angle. Therefore, if the system cannot access the part of the half of the figure containing the differentiating features relative to the other half of the figure, it becomes impossible to obtain the orientation of an H-shaped figure at an angle of 0–360°. Outdoor envi-

153

ronments frequently require the use of more complex figures with subfigures so that recognition is still possible even if a portion is blind. (4) Sharp, Shakernia, and Sastry (2001): this method is based on the use of a figure that is built from six white squares, five having the same size. These squares are included in a larger black square which, in turn, is included in another white square. This figure is acquired from the UAV using a builtin digital camera, to then be processed using the following computer vision-based techniques, including: the thresholding, segmentation, connected components labeling and detection of points of interest (corners). It introduces a design and implementation of a real-time vision system at 30 Hz for a rotor-craft UAV which estimates its pose and speed relative to a known landing target. It applies both linear- and nonlinear optimization algorithms to perform the pose estimation. Although the figure is more complex than those described previously, the use of a simple square may cause those parts of the figure to be confused by other items that may occur in an external environment, such as typical straw cubes cut by a mower. For this reason, it is more convenient to use geometrical shapes that are more difficult to replicate in outdoor environments, such as ellipses or circles. (5) In addition to the methods described above, there are other techniques where an artificial geometric figure is not needed to carry out the approximation process. For example, Cesetti, Frontoni, Mancini, Zingaretti, and Longhi (2010) developed a technique which tries to find natural markings on the outdoor scene by using algorithms based on an analysis of optical flow, from which the image is divided into smaller frames. Finally, a technique known as SIFT (Scale invariant Feature Transform) is applied. Once marks have been identified, the UAV is guided by these to a safe landing. 1.3. Motivational research of the proposed strategy All the methods, above, apply image thresholding, segmentation and labeling. The main contribution of this work is to exploit the performance of these proven techniques, however with a novel approach that increases efficiency and reliability. This allows the identification of the figure with its use of non-repetitive shapes (circles and ellipses strategically located), allowing the application of a reasoning process, which is translated by expert systems. Our proposed features, from which the reasoning process is derived, are based on: (a) Using the physical platform (paper) as an extra region, that includes all others. (b) Use of metrical and topological descriptors (Euler number) as prior filtering. (c) Recognition degree of confidence function based on Comparative Partial Analysis (CPA). (d) Use of combined geometric descriptors (area, eccentricity and Euclidean distance) as a unique value. (e) Use of several metric thresholds based on ratios between subfigures. (f) A complex figure comprising different geometrical subfigures in different sizes. (g) Support for different perspectives and distances. By using these new features, the proposed expert system can overcome some issues existing in other proposed methods: the complex morphology of the proposed figure can be easily recognized in an outdoor environment, the inclination/orientation can be determined, and it still works when part of the figure is blind due to the reflection of sun or brightness.

154

J.A. García-Pulido et al. / Expert Systems With Applications 76 (2017) 152–165

2. Expert system design

2.2. Geometrical figure analysis

2.1. Preliminary analysis of the landing platform

Although we use the following perspective as a basis for performing the identification process, the thresholds and parameters described in Fig. 2 are used to provide sufficient versatility to allow recognition from many perspectives and angles of inclination of the platform with respect to the vision system, i.e. the image. These parameters are described, below. They were obtained from the original platform assuming the proportions and relations remain on the images. This image, along with tests that were carried out, formed the basis to calibrate the system, i.e. to define the values for the parameters. The constants and error thresholds are considered for recognition. The constants arise from an exhaustive geometrical analysis of this figure by using zenithal perspective, whilst the error threshold gives the required flexibility to allow the recognition in many different perspectives. All the error thresholds and values, and the proposed strategy section, are estimated using hundreds of tests and a trial and error method. This refining process is important because determining the error thresholds is essential to perform the recognition process using different and heterogeneous types of perspectives. The algorithm description has an example of their usage.

The landing platform is identified by a set of related features that are basic for recognition: four circles (C1, C2, C3 and C4) and three ellipses (E1, E2 and E3) form part of the platform, Fig. 2. The aim is to identify these regions by using CPA, an identification criterion that either works with single- or many regions simultaneously based on specific geometric descriptors such as area, angles and Euclidean distance. The region and descriptors depend on the type of CPA that was applied, and the final computation of the degree of confidence is determined by the total CPA that was correctly identified. The pose in Fig. 2 determines the orientation and the name of the regions used in the different CPA that were carried out. In our proposed system, Fig. 2 defines the reference point for orientation, i.e. 0°, and the regions that are used during the identification process. The reader should refer to this image and description throughout the document when specific details are discussed. The proposed image segmentation procedure is designed to manage both recognition and orientation. Once the platform has been identified and oriented, as described below, it guides the UAV during the entire landing process.

2.3. Proposed strategy The full image recognition procedure, defining the proposed expert system, is summarized in Fig. 3.

Fig. 2. Geometric characteristics and identification thresholds of the proposed image.

J.A. García-Pulido et al. / Expert Systems With Applications 76 (2017) 152–165

155

Fig. 3. Architecture and design of the proposed image recognition strategy.

2.3.1. Image acquisition 1. Once the mission of the UAV is close to completion, an image of the landing platform with its figure is acquired using the digital camera in the mobile phone attached to the UAV. 2. Adapt the image to the distance using a scaling image process. Given that the UAV knows the distance to the platform from its altitude sensor, we lower the resolution when closer. The lower the distance is the less resolution is needed. 2.3.2. Image segmentation 3. Transform the image from RGB color space to HSV. Then, for each image we first apply the direct thresholding method (Asaad, Al-Salih, & Ahson, 2009; Mathworks, 2016). If this recognition fails, the system automatically applies Otsu’s method (Otsu, 1979) to obtain a binary image. Following hundreds of tests, we observed that the initial direct thresholding is faster and gives better results (see below) than automatic thresholding. However, both thresholding methods can work together. 4. Component-connected labeling (Dillencourt M., Samet, & Tamminen, 1992) is then applied by using the bwlabeln function in Matlab (Mathworks, 2016), based on (Sedgewick, 1998). This process measures properties of image regions for each component. 2.3.3. Image figures detection 5. We identify each subfigure using the following measured properties of the image regions by using the regionprops function in Matlab achieved in a previous step: area (number of pixels), Euler number, eccentricity, orientation and centroid. Mathematical theory and symbols are shown in Fig. 2: We iterate through all regions to determine the region that satisfies the equations, below. Positive regions are considered as candidate platform:

Rel ∈ {−1, 0, 1} Rma ≥ A∗ Rmi Rma ≤ A∗ Rmi





et1 1− ; 100   et1 1+ ; 100

(1) (2) (3)

where Rel is the Euler number, Rma is the major axis length, and Rmi is the minor axis length. Next, a further filtering process is applied to regions from step 4, to determine candidate ellipses. Eq. (1) must be fulfilled again, and we must account for the fact that these values depend of the size of the platform (A3 paper in our case) and the type of camera used. In our instance, based on experimental analysis, we determined:

Ra ∈ [165, 80 0 0 0]; Re ∈ [0.55, 0.95]; where Ra is the area and Re is the eccentricity.

Using the regions provided by the previous step, all the ellipse candidates are mathematically combined considering groups of three elements of a larger set of three objects, so that the total number of iterations is given by the following expression: C (n, r ) =

(nr ), where n is the number or regions, and r is the number of ellipses we need to identify in a single analysis - 3 in this case. For each combination, another filtering process is performed using the area and eccentricity. However, this time the aim is to only keep regions that are very similar in area and eccentricity. To achieve this goal, we compare all area and eccentricity values of each group of 3 ellipses using the following thresholds: Area: each 3 combination of regions must meet the following equations:

R1a ≥ R3a ∗ R1a ≥ R2a ∗ R2a ≥ R3a ∗ R2a ≥ R1a ∗ R3a ≥ R2a ∗ R3a ≥ R1a ∗

     

1− 1− 1− 1− 1− 1−



ast ; 100  ast ; 100  ast ; 100  ast ; 100  ast ; 100  ast ; 100

R1a ≤ R3a ∗ R1a ≤ R2a ∗ R2a ≤ R3a ∗ R2a ≤ R1a ∗ R3a ≤ R2a ∗ R3a ≤ R1a ∗

     

1+ 1+ 1+ 1+ 1+ 1+



ast ; 100  ast ; 100  ast ; 100  ast ; 100  ast ; 100  ast ; 100

(4)

where R1a , R2a and R3a is the area of each region Eccentricity: each 3 combination of regions must satisfy these equations:

R1e ≥ R3e ∗ R1e ≥ R2e ∗ R2e ≥ R3e ∗ R2e ≥ R1e ∗ R3e ≥ R2e ∗ R3e ≥ R1e ∗

     

1− 1− 1− 1− 1− 1−



est ; 100  est ; 100  est ; 100  est ; 100  est ; 100  est ; 100

R1e ≤ R3e ∗ R1e ≤ R2e ∗ R2e ≤ R3e ∗ R2e ≤ R1e ∗ R3e ≤ R2e ∗ R3e ≤ R1e ∗

     

1+ 1+ 1+ 1+ 1+ 1+



est ; 100  est ; 100  est ; 100  est ; 100  est ; 100  est ; 100

(5)

where R1e , R2e and R3e is the eccentricity of each region. If all groups of 3-regions are rejected when using these error thresholds, an auto-threshold is used. The ast and est thresholds are increased from 7 and 35 respectively, to a maximum of 70 in unit steps. For the groups of three regions which have not been discarded we perform the following: By using a centroid property, we determine the Euclidean distance between the centers of each three regions groups. We then

156

J.A. García-Pulido et al. / Expert Systems With Applications 76 (2017) 152–165

get the ratio of the distances between regions (l1, l2 and l3) and subtract the estimated ratio, which is determined in the geometrical analysis shown above. Beginning with this estimation and the absolute value of the subtraction, we identify the E1 and E3 ellipses for each group of 3 regions. By using the orientation property of regions E1 and E3, we determine the absolute value of the subtraction of their orientation values. Obviously, the result of this subtraction is ideally zero because the orientation of ellipses E1 and E3 is the same. By using area property, we calculate the arithmetic mean of the 3-group of candidate ellipses to determine the area ratio with each candidate detected platform (A3 paper). Then, we use the area ratio estimation of the platform and ellipses (ar1 = 2.87) and error threshold (et1 = ± 25%). If the area of the candidate platform meets the condition, we calculate the Euclidean distance between the center of the candidate platform and the middle of the distance between E1-E3 (l2), which must ideally be zero. Then, for each group of three regions we sum the result obtained in the previous three steps. The group that has the value nearest to zero matches ellipses E1, E2, E3, and we also determine the region that matches the platform. If, for any reason, the physical platform (paper) is not detected, the system can work perfectly using only ellipses. 6. Once ellipses E1, E2 and E3 have been identified, we calculate the Euclidean distances between ellipses to get the values of ratios 11/l2, l3/l2 and l3/l1. E2 can be identified because the ratio distance between E1-E3 (l3/l1 = 72%) is greater than the distance of E2 to any of the other two ellipses: E1-E2 (l1/l2 = 72%) and E2E3 (l3/l2 = 69%). By using the orientation of E2 and the constraint that E1, E2, E3 is 90°, 0°, 90° respectively, the orientation of the platform is determined. Note that orientation property is the angle between the x-axis and the major axis of the ellipse that has the same second-moments as the region, which takes values of −90° to 90°. However, our proposed system returns the orientation in the interval [0°, 360°). 7. By using these measured properties of the image regions returned by the regionprops function from Matlab - area (number of pixels), eccentricity, orientation, centroid, and Euler number - we can identify the circles C1, C2, C3 and C4 in this way: To consider an image region as a circle candidate, it must satisfy these criteria:

Rel ∈ {−2, −1, 0, 1}; Ra ∈ [5, 450 0 0]; Re ∈ [0, 0.8]

(6)

where Rel is the Euler number, Ra is the area and Re is the eccentricity. Using the regions from the previous step, we now identify the circles in two steps: First, we identify circles C1 and C2 using the following algorithm: C1 is identified, as previously, by the ratio of the area of each region using the arithmetic mean ellipses’ areas. Ideally, this is 50%. To give more flexibility to other, different perspectives, as show in Fig. 2, we again use a percent auto-threshold as applied to the ellipses identification. However, in this case the allowable error threshold is ± 45% because the maximum threshold allowed in the auto-threshold procedure is 70%. We know the center of the circles C1 and C2 coincides with the center of the platform, and we also that this point occurs in the middle of the Euclidean distance between the centers of ellipses E1 and E3 (l2). Using the orientation and centroid properties of E1 and E3 we now obtain X and Y Cartesian coordinates of this point. For all regions passing the filter, above, we calculate the Euclidean distance between the centroid of each region and this point. The region having a value nearest zero is identified as C1. C2 is determined by ratio of the area of each region to that of C1, whose value is 4.6%. The threshold allows a value difference

of ± 45%. We again use an auto-threshold if a region does not meet the condition that 70% is the maximum allowed threshold. We calculate the Euclidean distance between the centroid of each region and the middle of the imaginary straight line, which joins E1 and E3 centers (l2). The region having a value nearest to zero is identified as C2. We now identify C3 and C4: We use as a filter the area of each region relative to the area of C1, which has a theoretical value of 3.34% for the both circles. In this case, the threshold also allows a difference of ± 45% of the value. If these conditions do not lead to identification of the two regions, the auto-threshold is increased unit by unit to a maximum of ± 70%. For each region that was not discarded using the filter above, we calculate the Euclidean distance between the centroid of each region and that of C1, identified previously. Then, we obtain the ratio between this distance (l4 and l5) and the Euclidean distance between the centers of E1 and E3 (l2). Finally, we subtract from this ratio another two ratios, whose values are both 51% (l4/l2 and l5/l2). The first two regions that have the value nearest to zero are C3 and C4, respectively. Finally, given that at this point we have determined the orientation of the platform and we also know the angle between the center of C3 and C4 with C1, we can easily discriminate, between regions C3 and C4 by using the orientation property. 2.3.4. Image recognition 8. Once the physical Platform and figures E1, E2, E3, C1, C2, C3, and C4 have been identified, we map on the image the centre and their perimeters by using the general equation for an ellipse and circle. 9. We carry out each CPA described in Table 1 using both the figure orientation (rotation regarding with the perspective of Fig. 2) and regions obtained in the step above (Image figures detection). 10. The error thresholds and constant values were defined in Fig. 2. However, certain values such as Platform.width, Platform.height, Platform.area, Platform.orientation, Ellipses.averageArea, CX.area and CX.eccentricity were obtained only after the image figures are detection. The underscore symbol is used to distinguish regions, e.g. E1_E3_Distance refers to the distance between the centre of the ellipse1 and ellipse3. 11. From the results obtained in each CPA, the final degree of confidence is determined by using an arithmetic mean with the same weight that was given to each CPA. A total of 10 criteria are used to determine the average, and all the above implementation conditions must be met to be considered a hit. 12. If the degree of confidence ≥ 0.6 the recognition is successful. If not, a new cycle of recognition is started using different spatial resolutions and new thresholds. As implemented, this option can be enabled or disabled by the user via the system configuration section. Determination of the final degree of confidence, using several CPAs, relies on segmentation of the figure and analysis of each geometrical subfigure individually through several ratios between subfigures. In this way, we can determine a degree of confidence of recognition even when parts of the figure are blind either because of acquisition picture issues or an incorrect binarization thresholding selection. 2.4. Visual debugging process A chronological sequence of the identification process is shown in Fig. 4, to illustrate the several stages undertaken by the algorithm. The image has been identified with a score of 100% of recognition probability. Each stage is explained in detail in this section.

J.A. García-Pulido et al. / Expert Systems With Applications 76 (2017) 152–165

Table 1 Comparative partial analysis (CPA) implementation. All error thresholds and constants are described in Fig. 2. ID

Regions

Description

Hit

Implementation

CPA1

Platform

Analysis of the rectangle (A3 paper in our case) where the landing figure is printed (platform).

10

CPA2

E1 E2 E3

Analysis of angles formed by the straight lines connecting the centres of the ellipses by subtracting the orientation of the landing platform.

10

CPA3

E1 E2 E3

Analysis of the ratios of the Euclidean distances between ellipses. Additional flexibility is provided by an available threshold.

10

CPA4

E1 E2 E3

Analysis of the area of the three ellipses. The area of each ellipse is compared with the other two ellipses which are part of the group by carrying out an analysis of their area similarity. Additional flexibility is provided by an available threshold.

10

CPA5

E1 E2 E3

Analysis of the eccentricity of the three ellipses. The eccentricity of each ellipse is compared with the other two ellipses which are part of the group by carrying out an analysis of their eccentricity similarity. Additional flexibility is provided by an available threshold.

10

CPA6

C1 E1 E2 E3

Circle1 analysis regarding with the ellipses by using the following metric descriptors: area, angles and distances.

10

CPA7

C2 E1 E2 E3

Circle2 analysis regarding with the ellipses by using the following metric descriptors: area, angles and distances.

10

CPA8

C3 E1 E2 E3

Circle3 analysis regarding with the ellipses by using the following metric descriptors: area, angles and distances.

10

CPA9

C4 E1 E2 E3

Circle4 analysis regarding with the ellipses by using the following metric descriptors: area, angles and distances.

10

CPA10

C1 C2 C3 C4

Analysis circles together for determining coherence by using the following metric descriptors: angles and distances.

10

(Platform.width / Platform.height) ≥ A ∗ (1 – (et1 / 100)) (Platform.width / Platform.height) ≤ A ∗ (1 + (et1 / 100)) (Platform.area / Ellipses.averageArea) ∗ 100 ≥ ar1 ∗ (1 – (et2 / 100)) (Platform.area / Ellipses.averageArea) ∗ 100 ≥ ar1 ∗ (1 – (et2 / 100)) Platform.orientation – E1_E3_Angle ≤ a3 + et3 Platform.orientation – E1_E3_Angle ≥ a3 – et3 Platform.orientation – E2_E3_Angle ≥ a1 ∗ (1 – (et3 / 100)) Platform.orientation – E2_E3_Angle ≤ a1 ∗ (1 + (et3 / 100)) Platform.orientation – E1_E2_Angle ≥ a2 ∗ (1 – (et3 / 100)) Platform.orientation – E1_E2_Angle ≤ a2 ∗ (1 + (et3 / 100)) (E1_E3_Distance / E2_E3_Distance) ∗ 100) ≥ l2/l3 ∗ (1 – (et4 / 100)) (E1_E3_Distance / E2_E3_Distance) ∗ 100) ≤ l2/l3 ∗ (1 + (et4 / 100)) (E1_E2_Distance / E1_E3_Distance) ∗ 100) ≥ l1/l2 ∗ (1 – (et4 / 100)) (E1_E2_Distance / E1_E3_Distance) ∗ 100) ≤ l1/l2 ∗ (1 + (et4 / 100)) (E2_E3_Distance / E1_E2_Distance) ∗ 100) ≥ l3/l1 ∗ (1 – (et4 / 100)) (E2_E3_Distance / E1_E2_Distance) ∗ 100) ≤ l3/l1 ∗ (1 + (et4 / 100)) E1.area ≥ E3.area ∗ (1 – (et5 / 100)) E1.area ≥ E2.area ∗ (1 – (et5 / 100)) E2.area ≥ E3.area ∗ (1 – (et5 / 100)) E2.area ≥ E1.area ∗ (1 – (et5 / 100)) E3.area ≥ E2.area ∗ (1 – (et5 / 100)) E3.area ≥ E1.area ∗ (1 – (et5 / 100)) E1.area ≤ E3.area ∗ (1 + (et5 / 100)) E1.area ≤ E2.area ∗ (1 + (et5 / 100)) E2.area ≤ E3.area ∗ (1 + (et5 / 100)) E2.area ≤ E1.area ∗ (1 + (et5 / 100)) E3.area ≤ E2.area ∗ (1 + (et5 / 100)) E3.area ≤ E1.area ∗ (1 + (et5 / 100)) E1.eccentricity ≥ E3.eccentricity ∗ (1 – (et6 / 100)) E1.eccentricity ≥ E2.eccentricity ∗ (1 – (et6 / 100)) E2.eccentricity ≥ E3.eccentricity ∗ (1 – (et6 / 100)) E2.eccentricity ≥ E1.eccentricity ∗ (1 – (et6 / 100)) E3.eccentricity ≥ E2.eccentricity ∗ (1 – (et6 / 100)) E3.eccentricity ≥ E1.eccentricity ∗ (1 – (et6 / 100)) E1.eccentricity ≤ E3.eccentricity ∗ (1 + (et6 / 100)) E1.eccentricity ≤ E2.eccentricity ∗ (1 + (et6 / 100)) E2.eccentricity ≤ E3.eccentricity ∗ (1 + (et6 / 100)) E2.eccentricity ≤ E1.eccentricity ∗ (1 + (et6 / 100)) E3.eccentricity ≤ E2.eccentricity ∗ (1 + (et6 / 100)) E3.eccentricity ≤ E1.eccentricity ∗ (1 + (et6 / 100)) Platform.orientation – C1_E3_Angle ≤ a3 + et10 Platform.orientation – C1_E3_Angle ≥ a3 – et10 C1_E3_Distance ≥ (E1_E3_Distance / 2) ∗ (1 – (et9 / 100)) C1_E3_Distance ≤ (E1_E3_Distance / 2) ∗ (1 + (et9 / 100)) (C1.area / Ellipses.averageArea) ∗ 100 ≥ ar2 ∗ (1 – (et8 / 100)) (C1.area / Ellipses.averageArea) ∗ 100 ≤ ar2 ∗ (1 + (et8 / 100)) Platform.orientation – C2_E3_Angle ≤ a3 + et10 Platform.orientation – C2_E3_Angle ≥ a3 – et10 C2_E3_Distance ≥ (E1_E3_Distance / 2) ∗ (1 – (et9 / 100)) C2_E3_Distance ≤ (E1_E3_Distance / 2) ∗ (1 + (et9 / 100)) (C2.rea / Ellipses.averageArea) ∗ 100 ≥ ar3 ∗ (1 – (et8 / 100)) (C2.area / Ellipses.averageArea) ∗ 100 ≤ ar3 ∗ (1 + (et8 / 100)) Platform.orientation – C3_Platform.centre_Angle ≤ a6 ∗ (1 + (et10 / 100)) Platform.orientation – C3_Platform.centre_Angle ≥ a6 ∗ (1 – (et10 / 100)) C3_Platform.centre_Distance ≥ l4/l2 ∗ (1 – (et9 / 100)) C3_Platform.centre_Distance ≤ l4/l2 ∗ (1 + (et9 / 100)) (C3.area / Ellipses.averageArea) ∗ 100 ≥ ar4 ∗ (1 – (et8 / 100)) (C3.area / Ellipses.averageArea) ∗ 100 ≤ ar4 ∗ (1 + (et8 / 100)) Platform.orientation – C4_Platform.centre_Angle ≤ a5 ∗ (1 + (et10 / 100)) Platform.orientation – C4_Platform.centre_Angle ≥ a5 ∗ (1 – (et10 / 100)) C4_Platform.centre_Distance ≥ l5/l2 ∗ (1 – (et9 / 100)) C4_Platform.centre_Distance ≤ l5/l2 ∗ (1 + (et9 / 100)) (C4.area / Ellipses.averageArea) ∗ 100 ≥ ar5 ∗ (1 – (et8 / 100)) (C4.area / Ellipses.averageArea) ∗ 100 ≤ ar5 ∗ (1 + (et8 / 100)) Platform.orientation – C1_C3_Angle ≥ a6 ∗ (1 – (et10 / 100)) Platform.orientation – C1_C3_Angle ≤ a6 ∗ (1 + (et10 / 100)) Platform.orientation – C1_C4_Angle ≥ a6 ∗ (1 – (et10 / 100)) Platform.orientation – C1_C4_Angle ≤ a6 ∗ (1 + (et10 / 100)) Platform.orientation – C2_C3_Angle ≥ a5 ∗ (1 – (et10 / 100)) Platform.orientation – C2_C3_Angle ≤ a5 ∗ (1 + (et10 / 100)) Platform.orientation – C2_C4_Angle ≥i a5 ∗ (1 – (et10 / 100)) Platform.orientation – C2_C4_Angle ≤ a5 ∗ (1 + (et10 / 100)) C1_C3_Distance ≥ l4/l2 ∗ (1 – (et9 / 100)) C1_C3_Distance ≤ l4/l2 ∗ (1 + (et9 / 100)) C1_C4_Distance ≥ l5/l2 ∗ (1 – (et9 / 100)) C1_C4_Distance ≤ l5/l2 ∗ (1 + (et9 / 100)) C2_C3_Distance ≥ l4/l2 ∗ (1 – (et9 / 100)) C2_C3_Distance ≤ l4/l2 ∗ (1 + (et9 / 100)) C2_C4_Distance ≥ l5/l2 ∗ (1 – (et9 / 100)) C2_C4_Distance ≤ l5/l2 ∗ (1 + (et9 / 100))

157

158

J.A. García-Pulido et al. / Expert Systems With Applications 76 (2017) 152–165

Fig. 4. (a) Original picture; (b) Binary picture; (c) Component-connected labeling picture. (d) Recognized image.

3. Results

3.2. Distance test

3.1. Images acquisitions conditions

One-hundred-and-sixty images were used when distances from the UAV to the platform were 0.6 cm to 10 m. Despite distance generally having a significant impact on the recognition effectiveness in computer vision - and especially in our case - the effect of distance can be best described using the ratio of area in pixels of the platform / area in pixels of the full image. The higher the ratio value, the more effective and potentially useful the image - and vice versa. The results obtained are summarized in Fig. 5. The left and right graphs display the effectiveness, defined as the averaged percentage of success in recognizing the platform, for different distances from the camera system to the platform acquired by each device, and using direct and automatic thresholding. Graphics were less effective at distances ranging from 4 m to 8 m than at 8 m and above, which may seem inconsistent: however, this is solely due to the angle of inclination, because for pictures taken at 8 m, this angle was noticeably smaller. The quality of the camera significantly influences the effectiveness, as seen by the Galaxy S6 consistently showing better results than the Galaxy Ace at all distances. In fact, 100% success was seen for distances up to 10 m and with an inclination angle lower than 35% for the pictures taken using the Galaxy S6. However, 79.75% success was observed when this angle was < 60°. If the angle of inclination is < 40°, the system behaves perfectly up to 8 m. Beyond that the system cannot recognize the landing platform properly. For distances of 0 - 4 m, the system performs well when low spatial resolutions (1024 × 768) are used. However, as the distance between the camera lens and the landing platform increases, and therefore the area in pixels of the platform / are in pixel of the image decreases, the best results are obtained using high resolution (2592 × 1944). However, if we use this high resolution with distances < 4 m, the number of regions obtained through the segmentation process increases inordinately. This increases the number of candidate regions with similar attributes, negatively affecting the identification process because the discrimination between regions becomes overcomplicated. To find the maximum distance, one of the key results is that the resolution adaption of the image based on distance is an important consideration, as seen in Fig. 5b.

All images used in the experiments were acquired in a real environment and using a UAV. The device is a quadrotor of the ISCAR Group (2016) with four 10 0 0 rpm/V brushless engines mounted with a 10-inch propeller, fed by its own battery. To determine differences relating to camera quality, we mounted two different mobile phones onto the UAV and used these in the testing period. The mobile phones not only obtain images, but act as a control unit managing the engines, sensors and other UAV components. Once the orientation of camera and platform has been determined, we then ensure:

(1) The landing figure that is under study has been printed on a blank sheet of white A3 paper (297 mm × 420 mm). Although the original patented platform (Cruz et al., 2012) is 1 m x 1 m, this disparity is not significant. In fact, since a proportionally smaller figure emulates conditions of greater distance, and therefore thoroughly tests the determination of limits and recognition weakness. (2) The color depth was 3 bytes (24 bits), translating to 8 bits per RGB channel. Standard JPEG compression was used. (3) To determine differences that are attributed to the quality of the camera, the input images used integrated camera mobile phones on (a) a Samsung Galaxy Ace GT-S5830 (5MP, F/2.6), and (b) a Samsung Galaxy S6 (16MP, F/1.9). (4) The images were taken with a resolution of 4 MP, with dimensions of 2560 × 1944 pixels. However, the procedure applies a scaling factor, which is fully configurable. (5) 160 pictures were acquired in varying conditions (see below). These images were processed using their mobile phone systems and their results are shown by these test type: distance, inclination, lighting conditions. (6) The test scenario used the UAV flying in the courtyard of a building within a residential area. All images were taken without use of zoom or flash.

We tested the effect of camera quality on perceived distance, inclination angle and lighting conditions. Four internal configurations were varied with each attempt, varying the thresholding technique and resolution: 1024 × 768 with direct threshold, 1024 × 768 with the Otsu threshold, 2592 × 1944 with direct threshold, and 2592 × 1944 with Otsu threshold. If any picture configuration is successful, then the others are not tested. Figs. 5a/6a/7a are result summaries (overall results) where a test is considered a hit if any of these four configurations is successful. Hence, its output matches with the final degree of confidence (recognition probability). Charts 5b/6b/7b show the configuration results for each image.

3.2.1. Representative examples of the distance In the tables, below, we show the interrelation between acquisition conditions and distance. Both images were obtained under similar lighting conditions, however, the distance is clearly very different (6.3 m and 0.66 m). The IMG_Distance_1 was tested in perspective because it has a low ratio of area occupied by the figure to total picture area. However, it has been successfully recognized by the system as well as the image IMG_Distance_2. We see the versatility of the algorithm in terms of distance in Table 2.

J.A. García-Pulido et al. / Expert Systems With Applications 76 (2017) 152–165

159

Fig. 5. The effectiveness of the two cameras as the distance to the target is varied (see legend) as described, above. (a) Overall success for the two mobile phones. (b) Success of each configuration type for the Samsung Galaxy S6.

Table 2 Meaningful sample of long and short distance cases. In both cases, we achieved recognition with a score of 100%.

3.3. Inclination angle test One-hundred-and-sixty images at inclination angles of 3° - 61° were analyzed, where this angle is between the vertical axis (OY) and the straight line connecting the center of the lens with the center point of the landing platform. The results are summarized in Fig. 6. For the Galaxy S6 camera, the success average is perfect in the inclination angle range of 0° - 20°. Although it is not possible from Fig. 6 to determine effectiveness for these data ranges, at higher angles the average begins to decrease, thus: 0° - 30° (94%), 0° 35° (89.66%), 0° - 40° (86.15%), 0° - 45° (82.86%), 0° - 50° (81.33%), 0° - 55° (81.57%), and 0° - 60° (79.74%). We see that a large angle compromises the overall recognition effectiveness much more than the distance. This is because, depending on the perspective, and therefore the inclination an-

gle, a circle begins to be perceived as an ellipse. It significantly affects the eccentricity and, to a lesser extent, the area of the regions. The use of error thresholds is essential to mitigate this limitation. 3.3.1. Representative examples of the inclination angle The two images below are shown to display the versatility in cases of different perspectives. The proposed method was effective at very high inclination angles, provided certain limits are not exceeded: factors that cause the eccentricity of a circle to increase and be perceived as an ellipse. The IMG_Inclination_2 shows this exact problem: in the case of a high inclination, the system confuses ellipses with C3 and C4 circles, and therefore the recognition process fails (40% score). However, with IMG_Inclination_1, despite having a very high degree of inclination, the system correctly identifies circles and ellipses as denoted by the red lines generated by the system (90% score), as shown in Table 3.

160

J.A. García-Pulido et al. / Expert Systems With Applications 76 (2017) 152–165

Fig. 6. We show the effectiveness of the two cameras as the inclination angle to the target is varied (see legend) as described, above. (a) Overall success for the two mobile phones. (b) Success of each configuration type for the Samsung Galaxy S6.

Table 3. Meaningful sample test cases of high inclination angle. The system was successful (upper images) and incorrect recognition (lower images) with scores of 90% and 40%, respectively.

3.4. Lighting condition test One-hundred-and-sixty images were acquired, but in this case the conditions of the images were manually determined. That is, under these manual conditions the images were recorded using the operator’s criteria while the tests were being carrying out. Hence, lighting conditions are of relevance affecting the performance, results are given in Fig. 7. As seen previously, the quality of the camera significantly influences the success of recognition. This is especially the case in sunny conditions, when there is a greater likelihood of blind regions over the platform. The longest bar always belongs to the first two bars (blue and red) which stand for the direct thresholding method. There-

fore, if the resolution parameter is not taken into account, the direct thresholding method (Asaad et al., 2009) outperformed Otsu’s method (Otsu, 1979) in all conditions. In sunny conditions the inclination angle is particularly significant because direct sunlight on the platform causes reflections which may cause blind regions during binarization. In tests where artificial light was used we also observed that better image resolution improves results because the pictures taken under these conditions were at a shorter average distance. The daytime tests generally showed worse effectiveness than those carried out at night under artificial light. At night, because the lighting conditions can be controlled, the camera quality has less influence than seen previously.

J.A. García-Pulido et al. / Expert Systems With Applications 76 (2017) 152–165

161

Fig. 7. The effectiveness of the two cameras as the lighting condition is varied (see legend) as described, above. (a) Overall success for the two mobile phones. (b) Success of each configuration type for the Samsung Galaxy S6. Table 4 Meaningful sample of lighting conditions- both successful recognition with a score of 100% and 70%.

3.4.1. Representative examples of the lighting conditions The IMG_Lighting_1 was obtained in sunny conditions, but with a shadow thrown over the middle of the figure. This lighting conditions had no impact on the perfect recognition probability of 100%. However, IMG_Lighting_2 image is challenged by a reflexion issue problem due to the intensity of the sun and the inclination angle. Even though some parts of the region are blind after the segmentation process, the recognition is performed with a score of 70% because some of the regions have been properly identified despite reflections, as shown in Table 4. 3.5. System assessment It is important to determine the interaction between the function value, and the estimated final degree of confidence that the image contains the figure of the landing platform. To evaluate the reliability of the system, and therefore decide what likelihood of

success is applicable, we performed a manual, visual inspection (performed by a human) - formally termed supervision in the field of artificial intelligence. This monitoring process requires a visual inspection of the result, to determine whether the recognition system has detected at least three regions marked red, and to correctly identify one of the seven figures {E1, E2, E3, C1, C2, C3, C4} that form part of the landing platform. Although several criteria can be used to decide what is really a successful outcome, this allows an objective evaluation of system behavior. That is, to check that for low probabilities at least three of the regions corresponding to one of the figures {E1, E2, E3, C1, C2, C3, C4} are detected, and vice versa. Finally, the reliability of the degree of confidence value shown as a recognition probability is high, as show in Fig. 8. The correct identification of visual features is confirmed by manual inspection, thereby giving credence to the algorithmic result. Fig. 8 shows the reliability of the degree of confidence value shown as a recogni-

162

J.A. García-Pulido et al. / Expert Systems With Applications 76 (2017) 152–165

Fig. 8. Actual reliability of the returned degree of confidence (probability recognition).

tion probability, which appears to be coherent with the outcome provided by the CPA feature. 3.6. Summary results

Table 6 A summary of success rate (%) as a function of lighting conditions.

Camera

Finally, the table shows the data used to build the graphs. Table 5, correlates the distance between the camera lens and the center of the figure that forms part of the platform. It also shows the inclination angle formed by a straight line joining center of camera lens and middle of the landing platform, with respect to the axis OY. The inclination, height, distance, speed, acceleration, etc., were determined using on-board sensors. The value is the percentage derived from the ratio of successful image/total images (for the camera field) and performed / total tests (for the other fields). In both cases, we judge that there is a successful outcome when the visual supervision of the user determines that there are at least 3 regions marked in red on the original image, and they match some of the figures of our platform. However, in the case of cells relating to the camera field, a success

Threshold Resolution

Galaxy S6 Galaxy Ace Otsu Direct 1024 × 768 2560 × 1920

TOTAL

Sunny

Foggy

Sun and shadow

Artificial Light

66.7 37.5 26.0 13.5 8.3 31.3 30.6

84.4 65.6 46.9 12.5 20.3 39.1 44.8

66.7 66.7 35.4 16.7 8.3 43.8 39.6

100.0 91.7 85.4 4.2 50.0 39.6 61.8

is recorded if at least one test (of the four that are made by image) is successful by the criteria, above. Table 6 is similar to those, above, but we now consider lighting conditions and environment. Each cell value in Table 6 was obtained in the same way as for Table 5. Finally, the following tables are extracted and filtered from the raw data in Table A1 in the Appendix. In Table 7 we correlate the

Table 5 A summary of success rate (%) as a function of distance and inclination. Distance (metres)

Camera Threshold Resolution TOTAL

Galaxy S6 Galaxy Ace Otsu Direct 1024 × 768 2560 × 1920

Inclination (degrees)

0–2

2–4

4–6

6–8

>8

0–10

10–20

20–30

30–40

> 40

100 91.7 81.3 33.3 64.6 50.0 70.1

83.3 66.7 59.7 19.4 40.3 38.9 51.4

66.7 41.7 25.0 6.3 0.0 31.3 28.5

58.8 41.2 26.5 2.9 1.5 27.9 26.5

85.7 66.7 36.9 4.8 1.2 40.5 39.3

100 92.9 57.1 14.3 19.6 51.8 56.0

100 81.3 64.1 25.0 34.4 54.7 59.9

85.0 55.0 40.0 7.5 11.3 36.3 39.2

60.0 46.7 38.3 3.3 13.3 28.3 31.7

46.7 33.3 25.0 11.7 20.0 16.7 25.6

J.A. García-Pulido et al. / Expert Systems With Applications 76 (2017) 152–165 Table 7 A summary of success rate (%) for distance and inclination angle in a way where the intervals exclude each other. “NA” indicates data not available. Samsung Galaxy S6/

Distance (metres)

All lightning conditions Inclination (degrees)

0–2

2–4

4–6

6–8

>8

0−10 10−20 20−30 30−40 > 40

100 100 100 NA 100

NA 100 100 72.7 100

100 100 NA NA 42.9

NA NA 75 100 0

100 100 100 0 NA

Table 8 A summary of success rate (%) relating to distance and inclination angle. The aggregate for each interval is used. Samsung Galaxy S6/

Distance (metres)

All lightning conditions Inclination (degrees)

0–2

0–4

0–6

0–8

0–>8

0−10 0−20 0−30 0−40 0 - > 40

100 100 100 100 100

100 100 100 88.5 90

100 100 100 90.3 83.3

100 100 90,6 86.4 76.3

100 100 94 86.2 78.8

distance and the inclination angle in a way where the intervals exclude each other. In all cases the Samsung Galaxy S6 camera images were used to perform the intervals analysis: Table 8 is similar to Table 7, but uses the aggregate for the intervals: From Tables 6–8, it is possible to filter and perform the analysis for different parameters excluding each other, like distance, inclination angles. If we filter using reasonable ranges of distance and inclination angle the results are generally good. However, when the inclination angle increases significantly, the success rate decreases. Large angles compromise the overall recognition effectiveness much more than the distance. 4. Conclusions Unlike many of the previously proposed methods that rely on simpler figures to assist platform identification and UAV landing, our proposal is based on a more complex figure composed of various types of strategically placed figures. We show that in (quite likely, real world) cases where a part of the figure is blind there are still sufficient singular regions to allow for their identification. Given that some figures are inside the others, and we initially try to detect the rectangle where the geometric figure is placed,

163

discarding those regions whose centroid is not in the main figure is simplified. However, although the system takes advantage of these benefits, it has been designed to work equally well if the rectangle is undetected. After having carried out the recognition by using figures printed in several colors, the use of black and white was shown to be the most effective because it facilitates the process of image binarization. Furthermore, we advise using anti-reflective materials, and avoid sunny conditions with the inclination angle exceeding 30°. The environment where the platform is placed may have a considerable influence on the recognition process, especially if the A3 paper rectangle has not been detected. In some instances, plant leaves having elliptical shapes were incorrectly detected as the E1, E2, E3 ellipses: the image should be placed where it is clear of interfering structures. Although the system behaves appropriately when images are obtained at a useful range of distances and inclination angles, we show that best results are observed when the acquisition process occurs with constant height, zoom and inclination. The results also show that the inclination angle should, wherever possible, not exceed 35%. The average recognition time – ca. 1.5 s for our system - largely depends on the number of regions and ellipse candidates for the identification process, the binarization threshold, and the image resolution. Because it will lessen recognition times, we recommend using the onboard UAV sensors to adapt the image resolution to the distance. Although the sensitivity to the inclination angle may be the limiting factor in the practical exploitation of our technology, this is not the case for rotary wing devices such as the quadrotor UAV used for this project. We therefore believe that our proposed design and analysis algorithm will at least be very effective for rotary wing UAVs under a wide range of real-world conditions, and fixedwing UAV under somewhat restricted circumstances of approach angle. Acknowledgments The Spanish Government has provided support of this research work through project DPI2013-46665-C1 (SALACOM). The authors thank the anonymous referees for their very valuable comments and suggestions. The author would also give a special thanks to the Dr Mike Bernstein, an NMR (Nuclear Magnetic Resonance) scientist, for linguistic assistance. Appendix Table A1.

164

J.A. García-Pulido et al. / Expert Systems With Applications 76 (2017) 152–165

Table A1 Eighty images captured under different parameters (inclination, height, length, distance), lighting conditions and success according to devices Samsung Galaxy S6 and Samsung Galaxy Ace. Thus, the total images used is one hundred and sixty. Name

IMG_001 IMG_002 IMG_003 IMG_004 IMG_005 IMG_006 IMG_007 IMG_008 IMG_009 IMG_010 IMG_011 IMG_012 IMG_013 IMG_014 IMG_015 IMG_016 IMG_017 IMG_018 IMG_019 IMG_020 IMG_021 IMG_022 IMG_023 IMG_024 IMG_025 IMG_026 IMG_027 IMG_028 IMG_029 IMG_030 IMG_031 IMG_032 IMG_033 IMG_034 IMG_035 IMG_036 IMG_037 IMG_038 IMG_039 IMG_040 IMG_041 IMG_042 IMG_043 IMG_044 IMG_045 IMG_046 IMG_047 IMG_048 IMG_049 IMG_050 IMG_051 IMG_052 IMG_053 IMG_054 IMG_055 IMG_056 IMG_057 IMG_058 IMG_059 IMG_060 IMG_061 IMG_062 IMG_063 IMG_064 IMG_065 IMG_066 IMG_067 IMG_068 IMG_069 IMG_070 IMG_071

Inclination(°)

10,39 5,42 3,66 20,97 11,21 7,62 46,4 28,51 20,12 59,78 41,6 30,91 14,57 7,66 5,18 48,99 30,75 21,86 61,17 43,22 32,36 10,39 5,42 3,66 20,97 11,21 7,62 46,4 28,51 20,12 59,78 41,6 30,91 46,4 28,51 20,12 46,4 28,51 20,12 39,24 22,9 15,9 39,24 22,9 15,9 39,24 22,9 15,9 39,24 22,9 15,9 39,24 22,9 15,9 39,24 22,9 15,9 39,24 22,9 15,9 22,9 15,9 12,34 16,39 8,75 50,71 20,14 56,31 9,73 12,88 3,27

Height(OY)

3 5,8 8,6 3 5,8 8,6 3 5,8 8,6 3 5,8 8,6 3 5,8 8,6 3 5,8 8,6 3 5,8 8,6 3 5,8 8,6 3 5,8 8,6 3 5,8 8,6 3 5,8 8,6 3 5,8 8,6 3 5,8 8,6 3 5,8 8,6 3 5,8 8,6 3 5,8 8,6 3 5,8 8,6 3 5,8 8,6 3 5,8 8,6 3 5,8 8,6 5,8 8,6 1,6 1,7 0,65 0,45 1,5 0,4 1,75 1,75 1,75

Length(OX)

0,55 0,55 0,55 1,15 1,15 1,15 3,15 3,15 3,15 5,15 5,15 5,15 0,78 0,78 0,78 3,45 3,45 3,45 5,45 5,45 5,45 0,55 0,55 0,55 1,15 1,15 1,15 3,15 3,15 3,15 5,15 5,15 5,15 3,15 3,15 3,15 3,15 3,15 3,15 2,45 2,45 2,45 2,45 2,45 2,45 2,45 2,45 2,45 2,45 2,45 2,45 2,45 2,45 2,45 2,45 2,45 2,45 2,45 2,45 2,45 2,45 2,45 0,35 0,5 0,1 0,55 0,55 0,6 0,3 0,4 0,1

Distance(m)

3,05 5,83 8,62 3,21 5,91 8,68 4,35 6,6 9,16 5,96 7,76 10,02 3,1 5,85 8,64 4,57 6,75 9,27 6,22 7,96 10,18 3,05 5,83 8,62 3,21 5,91 8,68 4,35 6,6 9,16 5,96 7,76 10,02 4,35 6,6 9,16 4,35 6,6 9,16 3,87 6,3 8,94 3,87 6,3 8,94 3,87 6,3 8,94 3,87 6,3 8,94 3,87 6,3 8,94 3,87 6,3 8,94 3,87 6,3 8,94 6,3 8,94 1,64 1,77 0,66 0,71 1,6 0,72 1,78 1,8 1,75

Lighting Conditions

Cloudy Cloudy Cloudy Cloudy Cloudy Cloudy Cloudy Cloudy Cloudy Cloudy Cloudy Cloudy Sun and shadow Sun and shadow Sun and shadow Sun and shadow Sun and shadow Sun and shadow Sun and shadow Sun and shadow Sun and shadow Sunny Sunny Sunny Sunny Sunny Sunny Sunny Sunny Sunny Sunny Sunny Sunny Sun and shadow Sun and shadow Sun and shadow Sunny Sunny Sunny Cloudy Cloudy Cloudy Cloudy Cloudy Cloudy Sunny Sunny Sunny Cloudy Cloudy Cloudy Sunny Sunny Sunny Cloudy Cloudy Cloudy Sunny Sunny Sunny Cloudy Cloudy Cloudy Cloudy Cloudy Cloudy Cloudy Cloudy Artificial Light Artificial Light Artificial Light

Success per Camera Galaxy S6

Galaxy Ace

true true true true true true true true true false false false true true true false true true false false false true true true true true true true true true false false false true true true false true true true false true true true true false true true true true true false false true true false true false true true true true true true true true true true true true true

true true true false false true false true true false false false true true true false true true false false false true true true true true true false false false false false false true true true false false false false false true false true true false false true true true false false false true true true false false false true true true true true true true true true true true true (continued on next page)

J.A. García-Pulido et al. / Expert Systems With Applications 76 (2017) 152–165

165

Table A1 (continued) Name

IMG_072 IMG_073 IMG_074 IMG_075 IMG_076 IMG_077 IMG_078 IMG_079 IMG_080

Inclination(°)

3,27 3,27 3,27 34,44 34,44 34,44 34,44 45 45

Height(OY)

1,75 1,75 1,75 1,75 1,75 1,75 1,75 1,75 1,75

Length(OX)

0,1 0,1 0,1 1,2 1,2 1,2 1,2 1,75 1,75

References Asaad, A., Al-Salih, M., & Ahson, S. I. (2009). Object detection and features extraction in video frames using direct thresholding. In Multimedia, signal processing and communication technologies. IMPACT ’09. International (pp. 221–224). doi:10.1109/ MSPCT.2009.5164215. Cesetti, A., Frontoni, E., Mancini, A., Zingaretti, P., & Longhi, S. (2010). A vision-based guidance system for UAV navigation. Journal of Intelligent and Robotic Systems, 57(1–4), 233–257. Cruz, J. M., Sánchez, B., & Pajares, G. (2012). System for guiding an unmanned vehicle towards a platform using visual analysis. Patent nº 201001592, 2013. Dillencourt, M. B., Samet, H., & Tamminen, M. (1992). A general approach to connected-component labeling for arbitrary image representations. Journal of the Association for Computing Machinery, 39(2), 252–280. Guili, X., Yong, Z., Shengyu, J., Yuehua, Ch., & Yupeng, T. (2009). Research on computer vision-based for UAV autonomous landing on a ship. Pattern Recognition Letters, 30(6), 600–605. ISCAR Group (2016). Systems engineering, control. Automation and robotics laboratory. Computer Architecture and Automatic Control Department at the Complutense University of Madrid http:// www.dacya.ucm.es/ isalab/ Accessed 04.08.2016. Lange, S., Sünderhauf, N., & Protzel, P. (2008). Autonomous landing for a multirotor UAV using vision. In Workshop proc. of SIMPAR 2008 Intl. conf. on simulation, modelling and programming for autonomous robots (pp. 482–491).

Distance(m)

1,75 1,75 1,75 2,12 2,12 2,12 2,12 2,47 2,47

Lighting Conditions

Artificial Artificial Artificial Artificial Artificial Artificial Artificial Artificial Artificial

Light Light Light Light Light Light Light Light Light

Success per Camera Galaxy S6

Galaxy Ace

true true true true true true true true true

true true false true true true true true true

Lungu, M. (2012). Stabilization and control of a UAV flight attitude angles using the backstepping method. In International conference on modelling and simulation (ICMS 2012), January 15-17 (pp. 299–306). World Academy of Science, Engineering and Technology. Lungu, M., & Lungu, R. (2013). Adaptive backstepping flight control for a miniUAV. International Journal of Adaptive Control and Signal Processing, 27, 635–650. doi:10.1002/acs.2330. MathWorks (2016). The Matlab https:// es.mathworks.com/ products/ matlab/ Accessed 18.11.2016. Otsu, N. (1979). A threshold selection method from gray-level histogram. IEEE Transactions on Systems Man and Cybernetics, 9, 62–66. Pajares, G. (2015). Overview and current status of remote sensing applications based on unmanned aerial vehicles (UAVs). Photogrammetric Engineering & Remote Sensing, 81(4), 281–329. Saripalli, S., Montgomery, J. F., & Sukhatme, G. S. (2002). Vision-based autonomous landing of an unmanned aerial vehicle. In IEEE international conference on robotics and automation: 11-15 (pp. 2799–2804). Sedgewick, R. (1998). Algorithms in c (pp. 11–20). Addison-Wesley. Sharp, C. S., Shakernia, O., & Sastry, S. S. (2001). A vision system for landing an unmanned aerial vehicle. In IEEE international conference on robotics and automation (pp. 1720–1727).