- Email: [email protected]
Precision Forestry: Trees Counting in Urban Areas Using Visible Imagery based on an Unmanned Aerial Vehicle
5th IFAC Conference on Sensing, Control and Automation for 5th IFAC Conference on Sensing, Control and Automation for Agriculture 5th IFAC IFAC Conference Conference on on Sensing, Sensing, Control Control and and Automation Automation for for 5th Agriculture August 14-17, 2016. Seattle, Washington, USA online at www.sciencedirect.com Available Agriculture Agriculture August 14-17, 2016. Seattle, Washington, USA August August 14-17, 14-17, 2016. 2016. Seattle, Seattle, Washington, Washington, USA USA
ScienceDirect
IFAC-PapersOnLine 49-16 (2016) 016–021
Precision Forestry: Trees Counting in Precision Forestry: Trees Counting in Precision Forestry: Trees Counting in Urban Areas Using Visible Imagery based Urban Areas Using Visible Imagery based Urban Areas Using Visible Imagery based on an Unmanned Aerial Vehicle on an Unmanned Aerial Vehicle on an Unmanned Aerial Vehicle Omair Hassaan, Ahmad Kamal Nasir, Hubert Roth*, M.Fakhir Khan Omair Hassaan, Ahmad Kamal Nasir, Hubert Roth*, M.Fakhir Khan Omair Omair Hassaan, Hassaan, Ahmad Ahmad Kamal Kamal Nasir, Nasir, Hubert Hubert Roth*, Roth*, M.Fakhir M.Fakhir Khan Khan Lahore University of Management Sciences , Pakistan (e-mail: Lahore University of Management Sciences , Pakistan (e-mail: Lahore University of Management Management Sciences ,, Pakistan Pakistan (e-mail: (e-mail: (14030008, ahmad.kamal, fakhir.khan)@lums.edu.pk). Lahore University of Sciences (14030008, ahmad.kamal, fakhir.khan)@lums.edu.pk). (14030008, ahmad.kamal, fakhir.khan)@lums.edu.pk). *Lehrstuhl fr Regelungsund Steuerungstechnik (RST), Department (14030008, ahmad.kamal, fakhir.khan)@lums.edu.pk). *Lehrstuhl fr Regelungsund Steuerungstechnik (RST), Department *Lehrstuhl und (RST), Department Elektrotechnik und Informatik, Fakultt IV, Universitt Siegen, 57068, *Lehrstuhl fr fr RegelungsRegelungsund Steuerungstechnik Steuerungstechnik (RST), Department Elektrotechnik und Informatik, Fakultt IV, Universitt Siegen, Elektrotechnik und Informatik, Fakultt IV, Universitt Universitt Siegen, Siegen, 57068, 57068, Siegen und Germany (e-mail: [email protected] Elektrotechnik Informatik, Fakultt IV, Siegen Germany (e-mail: [email protected] 57068, Siegen Siegen Germany Germany (e-mail: (e-mail: [email protected] [email protected] Abstract: This research work describes an approach to count trees in an urban environment. Abstract: This research work describes an approach to count trees in an urban environment. Abstract: research work describes an to trees in an environment. Furthermore it addresses problems involved in detection of trees aerial imagery. This work Abstract: This This researchthe work describes an approach approach to count count treesin in an urban urban environment. Furthermore it addresses the problems involved in detection of trees in aerial imagery. This work Furthermore it addresses the problems involved in detection of trees in aerial imagery. This can be used to solve the problem of forest degradation and deforestation. Right now forest man Furthermore it addresses the problems involved in detection of trees in aerial imagery. This work work can be used to solve the problem of forest degradation and deforestation. Right now forest man can be used to solve the problem of forest degradation and deforestation. Right now forest man labor isn’t efficient enough to detect or prevent this problem. A multi-rotor UAV equipped with can beisn’t usedefficient to solveenough the problem of forest degradation and deforestation. Right now forest with man labor to detect or prevent this problem. A multi-rotor UAV equipped labor resolution isn’t efficient efficient enough to detect detect or prevent prevent thisaerial problem. A multi-rotor multi-rotor UAV equipped with high RGB camera was used to acquire images and to count number of trees labor isn’t enough to or this problem. A UAV equipped with high resolution RGB camera was used to acquire aerial images and to count number of trees high resolution RGB camera was used acquire aerial images number of in surveyed area. Various issues the robust of proposed algorithm high resolution RGB camera wasinvolved used to to in acquire aerialimplementation images and and to to count count number of trees trees in surveyed area. Various issues involved in the robust implementation of proposed proposed algorithm in surveyed area. Various issues involved in the robust implementation of algorithm are discussed. The result of successful implementation of the proposed algorithm on multiple in surveyed area. Various issues involved in the robust implementation of proposed algorithm are discussed. The result of successful implementation of the proposed algorithm on multiple are The result implementation of proposed algorithm on scenarios are also and we show that our naive approach is able to achieve ≈ 0.72 are discussed. discussed. The presented result of of successful successful implementation of the the proposed algorithm on multiple multiple scenarios are also presented and we show that our naive approach is able to achieve scenarios within are also also presentedamount and we weofshow show that our our naive naive approach approach is is able able to to achieve achieve ≈ ≈ 0.72 0.72 accuracy reasonable time.that scenarios are presented and ≈ 0.72 accuracy within reasonable amount of time. accuracy within reasonable amount of time. accuracy within reasonable amount of time. © 2016, IFAC (International Federation of Automatic Control) Hosting by Elsevier Ltd. All rights reserved. Keywords: Precision Forestry, Robotics, Vision, UAV. Keywords: Precision Forestry, Forestry, Robotics, Vision, Vision, UAV. Keywords: Keywords: Precision Precision Forestry, Robotics, Robotics, Vision, UAV. UAV. 1. INTRODUCTION Object detection and counting has been widely used in 1. INTRODUCTION INTRODUCTION Object detection detection and and counting counting has has been been widely widely used in in 1. Object computer vision community overhas the been past decades. People 1. INTRODUCTION Object detection and counting widely used used in computer vision community over the past decades. People vision community the decades. have worked on automatic carover counting, Moranduzzo et al. computer vision community the past past decades. People People In underdeveloped countries, lack of resources causes the computer have worked on automatic automatic carover counting, Moranduzzo et al. al. In underdeveloped underdeveloped countries, countries, lack lack of of resources resources causes causes the the have worked car counting, Moranduzzo et (2014), and on people counting methods, Velipasalar et have worked on automatic car counting, Moranduzzo et al. al. In major trouble for forest department to monitor a countrys (2014), and people counting methods, Velipasalar et In underdeveloped countries, lack of resources causes the major trouble trouble for for forest forest department department to to monitor monitor aa countrys countrys (2014), and people methods, Velipasalar et al. (2006). Research intocounting automatic tree detection and delin(2014), and people counting methods, Velipasalar et al. major forest area. Expensive sensor networks dont allow teams to (2006). Research Research into into automatic automatic tree tree detection detection and and delindelinmajor area. trouble for forestsensor department to dont monitor a countrys forest Expensive networks allow teams to to (2006). eation digitalinto imagery was started back in and mid-1980. (2006).from Research automatic tree detection delinforest area. Expensive sensor networks dont allow teams get real time data to monitor forest degradation and man eation from digital imagery was started back in mid-1980. forest area. Expensive sensor networks dont allow teams to get real real time time data data to to monitor monitor forest forest degradation degradation and and man man eation from imagery was back mid-1980. number of detection algorithms been eation then from aadigital digital imagery was started started back in in have mid-1980. get labor is time expensive reallyforest slow. degradation In recent years, the Since Since then number of detection detection algorithms have been get real data toand monitor and man labor is expensive and really slow. In recent years, the Since then aa number algorithms have been proposed. Pinz (1991) of proposed a Vision Expert System Since then number of detection algorithms have been labor is expensive and really slow. In recent years, the use of Unmanned Aerial Vehicles (UAVs) has increased proposed. Pinz (1991) proposed a Vision Expert System labor is expensive and really slow. In recent years, the use of of Unmanned Unmanned Aerial Aerial Vehicles Vehicles (UAVs) (UAVs) has has increased increased proposed. (1991) Vision System using aerialPinz imagery. Heproposed was ableaa to locateExpert the center of proposed. Pinz (1991) proposed Vision Expert System use rapidly, mostly being adopted for many civilian applicausing aerial aerial imagery. imagery. He He was was able able to to locate locate the the center center of of use of Unmanned Aerial Vehicles (UAVs) has increased rapidly, mostly being adopted for many civilian applicausing trees crown and estimate its radius using local brightness using aerial imagery. He was able to locate the center of rapidly, mostly for civilian applications. Inexpensiveness of UAV systems enabled them trees crown and estimate its radius using local brightness rapidly, mostly being being adopted adopted for many manyhas civilian applications. Inexpensiveness of UAV systems has enabled them trees crown and estimate its radius using local brightness maxima. Valley following its and rule using basedlocal algorithm was trees crown and estimate radius brightness tions. Inexpensiveness of UAV systems has enabled them to be used for various research applications e.g. agriculmaxima. Valley Valley following following and and rule rule based based algorithm algorithm was was tions. UAV systems has enabled them maxima. to be Inexpensiveness used for for various various ofresearch research applications e.g. agriculagriculby Gougeon (1995), in mid-1990s in whichwas he maxima. Valley following and rule based algorithm to be used applications e.g. ture, mapping and survey. Recently people have used UAV presented presented by Gougeon (1995), in mid-1990s in which he to be used for various research applications e.g. agriculture, mapping and survey. Recently people have used UAV presented by Gougeon (1995), in mid-1990s in which he follows the valleys of shadows between tree crowns using presented by Gougeon (1995), between in mid-1990s in which he ture, and Recently have images for vegetation monitoring,Pollock andUAV car follows the valleys of shadows tree crowns using ture, mapping mapping and survey. survey. Recently people people (1996), have used used UAV images for vegetation monitoring,Pollock (1996), and car follows valleys of between tree using ground sampled distance digital aerial imagery. Machine follows the the valleys of shadows shadows between tree crowns crowns using images for vegetation monitoring,Pollock (1996), and car counting, Moranduzzo et al. (2014) Daigavane and Bajaj ground sampled distance digital aerial imagery. Machine images for vegetation monitoring,Pollock (1996), and car counting, Moranduzzo Moranduzzo et et al. al. (2014) (2014) Daigavane Daigavane and and Bajaj Bajaj ground digital Machine has alsodistance been used to aerial detect imagery. individual trees. ground sampled sampled digital Machine counting, (2010). In Moranduzzo fact, UAVs allow and monitoring small learning learning has also alsodistance been used used to aerial detect imagery. individual trees. counting, et al.mapping (2014) Daigavane and Bajaj (2010). In fact, UAVs allow mapping and monitoring small learning has been to detect individual trees. Pollock (1996) used model-based template matching techlearning has also been used to detect individual trees. (2010). In fact, UAVs allow mapping and monitoring small areas at extremely fine scales, quick survey of target area Pollock (1996) used model-based template matching tech(2010). In fact, UAVs allow mapping and monitoring small areas at extremely fine scales, quick survey of target area Pollock (1996) used model-based template matching techniques to recognize individual trees. Pollockto(1996) used individual model-based template matching techareas at scales, survey of and acquisitions over the same areaarea at niques recognize trees. areasmulti-temporal at extremely extremely fine fine scales, quick quick survey of target target area and multi-temporal acquisitions over the same area at niques to recognize individual trees. niques recognizeUAV’s individual trees. and acquisitions predefined times. and multi-temporal multi-temporal acquisitions over over the the same same area area at at We havetopreferred high resolution images over satelpredefined times. We have preferred UAV’s high resolution images over over satelsatelpredefined times. We have preferred UAV’s high resolution images predefined times. lite images because satellite images are readily by We have preferred UAV’s high resolution imagesaffected over satelIn this research work, we have implemented a technique to lite images because satellite images are readily affected by In this research work, we have implemented a technique to lite images because satellite images are readily affected by cloudy environments. Also, freely available satellite images lite images because satellite images are readily affected by In this research work, we have implemented a technique to count trees usingwork, UAVwe and computer vision aalgorithms. In cloudy environments. Also, freely available satellite images In this research have implemented technique to count trees trees using using UAV UAV and and computer computer vision vision algorithms. algorithms. In In cloudy environments. Also, freely available are of low resolution as compared to UAV satellite images. images cloudy environments. Also, freely available satellite images count developing countries millions of people depend on forests of low low resolution resolution as as compared compared to to UAV UAV images. images. count trees countries using UAVmillions and computer vision algorithms. In are of developing of people people depend on forests forests are ofrest lowofresolution images. developing of depend on fao (2015), countries directly ormillions indirectly, for their livelihood due are developing countries millions of people depend on forests The the paperasis compared organized to as UAV follows, in section II fao (2015), directly or indirectly, for their livelihood due The rest rest of of the the paper paper is is organized organized as as follows, follows, in in section section II II fao (2015), indirectly, for their livelihood due to which a directly notable or amount of forest is destroyed every The fao (2015), directly or indirectly, for their livelihood due we have presented some related work by use of in satellite and The rest of the paper is organized as follows, section II to which a notable amount of forest is destroyed every we have presented some related work by use of satellite and to which a notable amount of forest is destroyed every year in tropics either in the ofform of is forest degradation have presented some related by use satellite and to which a notable amount forest destroyed every we UAV imagery. In section III wework present ourof methodology we have presented some related work by use of satellite and year in tropics either in the form of forest degradation UAV imagery. imagery. In In section section III III we we present present our our methodology methodology year in in of or deforestation. Deforestation and degradation has socio- UAV year in tropics tropics either either in the the form form of forest forest degradation degradation usedour tomethodology count trees UAV discuss imagery.an Inapproach section IIIwe wehave present or deforestation. Deforestation and degradation degradation has sociosocio- and and discuss an approach we have used to count trees trees or deforestation. Deforestation and has economic impact on small communities. These communidiscuss an approach we have used to count or deforestation. Deforestation and degradation has socio- and from aerial imagery. Section IV presents results our and discuss an approach we have used to countof trees economic impact on small communities. These communifrom aerial imagery. Section IV presents results of our economic impact small communities. These communities use forests as on their living hood. Forests provide major from aerial imagery. Section IV presents results of our economic impact on small communities. These communiimplemented algorithm and Section V discusses some from aerial imagery. Section IV presents results of our ties use forests as their living hood. Forests provide major implemented algorithm and Section V discusses some ties use forests as their living hood. Forests provide major environmental benefit by reducing global warming. But implemented and Section V discusses some ties use forests as their by living hood. Forests provide major conclusion andalgorithm future work. implemented algorithm and Section V discusses some environmental benefit reducing global warming. But conclusion and and future work. work. environmental benefit by reducing warming. But excessive degradation canglobal damage atmosphere environmental benefit of byforests reducing warming. But conclusion conclusion and future future work. excessive degradation of forests canglobal damage atmosphere excessive degradation of forests can damage atmosphere and release greenhouse gases in air which leads to global 2. RELATED WORK excessive degradation forests canwhich damage atmosphere and release greenhouseofgases gases in air air leads to global global 2. RELATED RELATED WORK WORK and release greenhouse in which leads to warming. Which eventually leads to climate change. In 2. and release greenhouse gases in air which leads to global 2. RELATED WORK warming. Which Which eventually eventually leads leads to to climate climate change. change. In In warming. Vibha et al. (2009), produced promising results using a most of the cases, people clear tropical forest to cultivate warming. Which eventually leads to climate change. In Vibha et et al. al. (2009), (2009), produced produced promising promising results results using using aa most of of the the cases, cases, people people clear clear tropical tropical forest forest to to cultivate cultivate Vibha most robust technique for counting of trees from remotely senseda land. Vibha technique et al. (2009), produced promising results using most of the cases, people clear tropical forest to cultivate robust for counting of trees from remotely sensed land. robust technique for counting of trees from remotely land. robust technique for counting of trees from remotely sensed sensed land. Copyright © 2016, 2016 IFAC 16 Hosting by Elsevier Ltd. All rights reserved. 2405-8963 © IFAC (International Federation of Automatic Control) Copyright © 2016 IFAC 16 Copyright © 2016 IFAC 16 Peer review under responsibility of International Federation of Automatic Copyright © 2016 IFAC 16 Control. 10.1016/j.ifacol.2016.10.004
IFAC AGRICONTROL 2016 August 14-17, 2016. Seattle, Washington, USA Omair Hassaan et al. / IFAC-PapersOnLine 49-16 (2016) 016–021
data. They have used high resolution satellite imagery and pre-processed the data using median filter and intensity based processing. For vegetation they used feature extraction and template generation techniques where a masking function is generated and scanned from bottom over the whole image. This algorithm has its limitations due to the use of satellite imagery, since high resolution images are not freely available, and specifically targets only the species of palm trees which are structurally different from the rest of tree species.
17
Fig. 1. DJI Phantom 2.0 perform object extraction, image filtering operations and image segmentation operations. Making the assumption that tree crowns are located at the center or close to the center of tree top, Brandtberg and Walter (1998) detected trees using local maxima values. In this algorithm, image is first smoothed to reduce the noise effect and then tree crowns are filtered using local maximum values. Trees crown and background is delimited using edge detection methods. This method is simple and produces promising results in very less time, but light variations and un-wanted background objects in image badly affects the results.
Kattenborn et al. (2014), proposed a technique of automatic single palm tree detection using photogrammetric point clouds. Single camera images were processed with a structure from motion tool chain using Visual SFM. Each image was classified into 3 classes, i) palm ii) shrubs/trees iii) ground. For classification, a multi-scale dimensionality criterion was used in which the classifier is trained on different scale factors for train and test dataset. Local dimensionality characteristics of point cloud are used to classify palm trees and ground soil. Training a classifier for dataset posts limitations for such algorithms. Since training classifier is time consuming and requires more computational power. For each type of tree species one has to train the classifier before detecting trees.
Wang and Gong (2004) further improved the work of Brandtberg and Walter (1998) by first detecting boundaries of tree crowns using edge detection methods and then intersected the results of local non-maximum suppression on gray-level image & local maximum values of morphological transformed distance between pixels. Intersecting both methods give a good estimate of the tree tops which are then counted using contour based methods. This algorithm also suffers the presence of background objects, for example, buildings and roads.
Bazi et al. (2014), used Scale Invariant Feature Transform (SIFT) to extract key points and used a pre-trained classifier based on Extreme Learning Machine (ELM) to classify the extracted key points. As output the ELM classifier marked each detected palm tree as key points. In order to capture the shape, the extracted key points are merged using active contour based method based on Level-Set (LS). Since some green plants can be confused with palm trees so to classify palm and non-palm regions texture analysis is performed. Kernel based learning machines pose a limitation of high training cost which includes time and computational power. This also restricts the detection/classification approach to single species, in case, if trees from different species are required to be detected, ELM has to be pre-trained before using.
3. METHODOLOGY In this research work we implement a technique of image segmentation. To interpret an image, it is usually necessary to extract objects from background or separate image into several meaningful regions. Generally image segmentation clusters pixels of an image based on a given rule or criteria. Color textures along with correlation among color bands form a good feature extraction.
Hung et al. (2006) proposed vision based shadow aided tree crown tree detection and classification algorithm using imagery from UAV. Algorithm comprises of 2 parts i) detection ii) classification. The algorithm uses color and texture information to segment regions of interest (tree crown existence). A model object matching step is done which uses the shape, scale and context information for classification to differentiate the tree crown species. Image segmentation step is done using Support Vector Machines (SVM). Although SVMs have good generalization performance, they can by really slow in test phase. SVMs depend on the choice of kernel and have high algorithmic complexity along with extensive memory requirements in large-scale tasks.
We have used a quad-copter equipped with 14.4 Mega pixel camera pha (2014). Aerial imagery is acquired by first surveying the target area. Afterward we created a mosaic using acquired images so that we can easily detect and count trees using our algorithm. 3.1 Preprocessing Data Acquisition The data acquisition for this thesis problem was done using quadrotor DJI-Phantom 2. This quadrotor is equipped with 14.4 Megapixel camera and has a flight time of almost 15-20 minutes. Figure 1 shows quadrotor. Aerial imagery is acquired by first surveying the target area and then those images are used for further processing. Different data sets were taken at different sites and at different heights.
The use of Ground control points for UAV data acquisition and spatial filtering is also being used for forest tree counting. Mansur et al. (2014) used UAV to collect data. They have used the concept of crown geometry and vegetation response to radiation. The detection of tree crown can be possible by applying spatial convolution processing technique such low pass filter to enhanced image. After spatial filter, Morphological analysis is applied on the dataset to
LUMS dataset: Dataset at Lahore University of Management Sciences Pakistan was taken at a height of 100 meters from ground. GPS co-ordinates were given as input to the quad-rotor for trajectory to follow. A total of 150 images covering 610 meters of ground area were taken. 17
IFAC AGRICONTROL 2016 18 Omair Hassaan et al. / IFAC-PapersOnLine 49-16 (2016) 016–021 August 14-17, 2016. Seattle, Washington, USA
Fig. 2. UAV’s trajectory in LUMS
Fig. 5. Sample Image from ChangaManga Data set from optical axis. The apparent effect looks like an image which has been mapped around a sphere or barrel. To produce meaningful results, images must be pre-processed in order to remove lens distortions. Image rectification is applied to remove such distortions and images are straightened. We first get the camera’s intrinsic parameters using checkerboard images. These checkerboard images are given to camera calibrator application in MATLAB to determine intrinsic parameters. These intrinsic parameters include camera matrix, distortion model, rectification matrix and projection matrix.Figure 6 shows the calibration process using MATLAB’s calibrator application and Figure 7 show the intrinsic parameters determined through calibration process. Once we get calibration matrix, this along with distorted image are given to the image rectification tool of MATLAB in order to remove barrel distortion. Figure 10 shows the image after rectification process.
Fig. 3. UAV’s trajectory-1 at ChangaManga
Fig. 4. UAV’s trajectory-2 at ChangaManga Figure 2 shows the trajectory followed by quad rotor and Figure 9 shows a sample image from this data set. Changa Manga dataset: Changa Manga is a handplanted forest located approximately 80 kilometeres from Lahore, the provincial capital of Punjab, Pakistan. It was once considered as the largest man-made forest in the world, but regular forest degradation and deforestation at massive scale has reduced its size. It covers almost 12,423 acres of land. Two different data sets were taken at Changa Manga so that we can cover maximum portion of this forest. For data set 1, quad rotor was given GPS co-ordinates as input covering a total of 1612 meters. Quadrotor was flown at a height of 150 meters from ground and a total of 170 images were acquired in this process. Figure 3 shows the trajectory followed by quadrotor and Figure 5 shows a sample image from this data set. For data set 2, quad rotor covered an distance of 1973 meters at a height of 150 meters from ground. A total of 220 images were taken in this process. Figure 4 shows the trajectory map followed by quad rotor.
Fig. 6. Camera Calibration using MATLAB
Fig. 7. Camera intrinsic parameters Image Enhancement: After image acquisition and passing through rectification process, it is very difficult to segment out all green portions from the image due to the variance in shades of green color. As shown in Figure 9, due to weather conditions and acquisition time that is dependent on azimuthal angle of sun we have yellowish tints and also some bright & dark portions in trees. In order
Image Rectification: Images acquired by quad rotor’s camera have barrel distortion, as shown in Figure 9. In barrel distortion, image magnification decreases as we go away 18
IFAC AGRICONTROL 2016 August 14-17, 2016. Seattle, Washington, USA Omair Hassaan et al. / IFAC-PapersOnLine 49-16 (2016) 016–021
19
Fig. 8. Methodology Flow Chart Fig. 11. After Applying K-means clustering on L*a*b image 3.3 Vegetation Segmentation Color segmented image has some portions of trees, grass, other vegetation and some noise. In order to successfully count trees we have to remove grass, other vegetation and the noise from image. Here we use texture analysis to accomplish our task. We can quantify the intuitive qualities such as roughness, smoothness etc. in an image using the spatial variation in pixel intensities or gray values. We can use the 2 dimensional dependence matrix known as co-occurrence matrix to get insight of texture. To define a gray-level cooccurrence matrix P[i,j], we first specify a displacement vector d = (dx,dy) and counting all pairs of pixels separated by d having gray levels i & j. Consider an 8x8 binary image of checkerboard shown in Figure 12. Since there are only 2 gray-levels, P [i,j] is a 2x2 matrix and if we define d = (1,1) we get the normalized P [i,j] as shown in Figure 13. If black pixels in above image are randomly placed, the matrix is expected to be uniformly populated. To measure the randomness of gray-level distribution we used entropy, Jain et al. (1995), as defined in (1),
Fig. 9. Sample RGB input image
Fig. 10. Rectified and Enhanced RGB Image to overcome these artifacts we tune the color saturation of the acquired images to give a level playing field for our algorithm. We enhance the color saturation so that different colors are easily distinguished during color based segmentation. Figure 10 shows the image after rectification and enhancement process.
Entropy = −Σi Σj P [i, j]logP [i, j]
(1)
Entropy value is highest when all entries in P [i,j] are equal which corresponds to an image in which there are no preferred gray-level pairs a distance d.
3.2 Color Based Clustering In urban areas, where tree density is low and sparsely located in a region, we can count trees using color threshold techniques. Here we first convert the RGB input image, as shown in Figure 9, to L*a*b color space. The L*a*b* color space is derived from the CIE XYZ tristimulus values. The L*a*b* space consists of a luminosity layer ’L*’, chromaticity-layer ’a*’ indicating where color falls along the red-green axis, and chromaticity-layer ’b*’ indicating where the color falls along the blue-yellow axis. All of the color information is in the ’a*’ and ’b*’ layers. One can measure the difference between two colors using the Euclidean distance metric. Here we apply k-means clustering algorithm on both layers i.e. a* and b*. By applying threshold values on green clusters centers, we segment out the green portion out of the image without any noise. Figure 11 shows the resultant image after applying Kmeans clustering.
Fig. 12. 8x8 Checkerboard We have used the same technique to segment the vegetation out of green-masked image. As we can see in Figure 11, trees have more roughness in structure as compared to grass. So after applying entropy filter on our segmented image and normalizing the output between 0-1, we can clearly see that rough portion (trees) is more prominent as compared to grass, as shown in Figure 14. Here, in Figure 14, we can treat grass region as a part of noise in our image. In order to remove noise from image, we apply dilation and erosion filters to remove it. Figure 15 19
IFAC AGRICONTROL 2016 20 Omair Hassaan et al. / IFAC-PapersOnLine 49-16 (2016) 016–021 August 14-17, 2016. Seattle, Washington, USA
Fig. 13. P[i,j] for d = (1,1) shows the output after applying morphological operations. Here we can see that the only portions left in our image are trees.
Fig. 16. Circle fitting approximation technique Table 1. Results (count)
3.4 Tree Counting
Fig# 11 12 13
At this point, we have successfully segmented out trees from all the other information present in the image. Now we want to use this information to help us count the quantity of trees present in the scanned area. To get this count, we use a circle fitting approximation technique. To use this technique, we need to know the distance occupied by a pixel in the image. Since we have taken all the data set from a fixed altitude of 100 meters above ground and we know the original image resolution of our UAVs camera, we can calculate the per pixel distance in centimeters using basic mathematics. In our case, we have calculated it to be 3.84cm/pixel. Next, we calculated the measured the span of trees on ground by using conventional measurement tools. After getting the minimum and maximum values of radii in centimeters, we provided this information to our circle fitting algorithm. Which tries to fit circles based on the range of radius we have provided it. We use the binary image as shown in Figure 15 as input and try to fit circles where pixels value is 1. Figure 16 shows the result of circle fitting approximation.
GT 37 42 47
CC 35 49 51
TP 25 31 35
TN 0 0 0
FP 8 18 16
FN 9 11 14
Time(sec) 10.78 7.86 8.4
Table 2. Accuracy Fig# 11 12 13
TPR 0.735 0.738 0.714
FPR 1 1 1
TNR 0 0 0
FNR 1 1 1
Accuracy % 67.5 73.8 74.46
4. RESULTS In this section the results of the experiment described in chapter 3 are discussed. Test was performed using MATLAB on Intels core-i3 processor with 4GB RAM. Figure 17 to ?? show the sample input images tested on the above discussed algorithm and Figure 19 to ?? show trees detected in images respectively. Table 1 shows the numerical figures of Ground Truth (GT) Circle Count (CC), True Positives (TP), True Negatives (TN), False Positives(FP) and False Negatives(FN) for all the 3 test images (Figure 11 to 13). Table 2 shows the True Positive Rate (TPR), False Positive Rate (FPR), False Negative Rate (FNR) and accuracy of algorithm. True Positive Rate is given by T P R = T P/(T P + F N ), False Positive Rate: F P R = F P/(F P + T N ), True Negative Rate: T N R = T N/(F P + T N ) and Accuracy is given by (T P + T N )/T otal
5. DISCUSSION & FUTURE WORK This paper presents a novel work on trees detection and counting in urban areas which can significantly reduce man labor. Tree detection and counting helps us in solving the problem of forest degradation and deforestation. Generally this problem is addressed using man labor and in case of large forests, man labor isn’t effecient. This paper presents an algorithm which uses some basic image processing techniques for vegetation segmentation and also for tree counting. Here we haven’t used machine learning algorithms for tree counting method. Since we have used an approximation technique for tree counting, the output results of this algorithm has some false positives and false negative numbers. Trees which appear much smaller in acquired images don’t have significant change in texture as compared to their surrounding or grass. Such kind of trees are either missed during color segmentation part or during
Fig. 14. Result of Entropy Filter
Fig. 15. After applying Morphological Operations (RGB) 20
IFAC AGRICONTROL 2016 August 14-17, 2016. Seattle, Washington, USA Omair Hassaan et al. / IFAC-PapersOnLine 49-16 (2016) 016–021
21
REFERENCES
texture analysis. Also in case of large trees, circle fitting algorithm often tries to fit more than 1 circle on its canopy. Such kind of errors can be reduced and accuracy can be increased by taking into account for machine learning algorithms. Learning based on tree’s texture can help us in detecting and counting tree canopies. We can also follow the work done by Yakub Bazi et al. (2014) to create a SIFT feature based extended learning machine.
(2014). Dji phantom. URL http://store.dji.com/product/phantom-2/. (2015). Fao statistics on forestry. URL http://www.fao.org/forestry/46203/en/. Bazi, Y., Malek, S., Alajlan, N., and AlHichri, H. (2014). An automatic approach for palm tree counting in uav images. IGARSS, IEEE. Brandtberg and Walter (1998). Automated delineation of individual tree crowns in high spatial resolution aerial images by multiple-scale analysis. Machine Vision and Applications, 64–73. Daigavane and Bajaj (2010). Real time vehicle detection and counting method for unsupervised traffic video on highways. IJCSNS International Journal of Computer Science and Network Security, 10. Gougeon (1995). A crown-following approach to the automatic delineation of individual tree crowns in high spatial resolution aerial images. Canadian Journal of Remote Sensing, 21, 274–284. Hung, Bryson, and Sukkarieh (2006). Vision based shadow aided tree crown detection and classification algorithm using imagery from an unmanned airborne vehicle. International Symposium for Remote Sensing of the Environment. Jain, R., Kasturi, R., and Schunck, B. (1995). Individual tree-crown delineation and treetop detection in highspatial-resolution aerial imagery. McGraw-Hill. Kattenborn, Sperlich, Bataua, and Koch (2014). Automatic single palm tree detection in plantations using uav-based photogrrammteric point clouds. The international Archives of the Photogrammetry, Remote sensing and spatial information sciences, XL-3. Mansur, Mukhtar, and Al-Doksi, J. (2014). The usefullness of unmanned airborne vehicle (uav) imagery for automated palm oil tree counting. Research Journali’s journal of Forestry, 1. Moranduzzo, Thomas, and Melgani, F. (2014). Automatic car counting method for unmanned aerial vehicle images. Geoscience and Remote Sensing, IEEE Transactions on 52, 1635–1647. Pinz (1991). A computer vision system for recognition of trees in aerial photographs. International Association of Pattern Recognition Workshop, 111–124. Pollock (1996). the automatic recognition of individual trees in aerial images of forests based on a synthetic tree crown model. Doctoral dissertation, Concordia University. Velipasalar, S., Tian, Y.L., and Hampapu, A. (2006). Automatic counting of interacting people by using a single uncalibrated camera. Multimedia and Expo, IEEE International Conference. Vibha, Shenoy, D., Venugopal, and Patnaik (2009). Robust technique for segmentation and counting of trees from remotely sensed data. IEEE International Advance Computing Conference. Wang, L. and Gong, P. (2004). Individual tree-crown delineation and treetop detection in high-spatial-resolution aerial imagery. Photogrammetric Engineering and Remote Sensing, 70, 351–357.
Fig. 17. Test Image 1
Fig. 18. Test Image 2
Fig. 19. Circle fitting result of Test Image 1
Fig. 20. Circle fitting result of Test Image 2
21
ScienceDirect
IFAC-PapersOnLine 49-16 (2016) 016–021
Precision Forestry: Trees Counting in Precision Forestry: Trees Counting in Precision Forestry: Trees Counting in Urban Areas Using Visible Imagery based Urban Areas Using Visible Imagery based Urban Areas Using Visible Imagery based on an Unmanned Aerial Vehicle on an Unmanned Aerial Vehicle on an Unmanned Aerial Vehicle Omair Hassaan, Ahmad Kamal Nasir, Hubert Roth*, M.Fakhir Khan Omair Hassaan, Ahmad Kamal Nasir, Hubert Roth*, M.Fakhir Khan Omair Omair Hassaan, Hassaan, Ahmad Ahmad Kamal Kamal Nasir, Nasir, Hubert Hubert Roth*, Roth*, M.Fakhir M.Fakhir Khan Khan Lahore University of Management Sciences , Pakistan (e-mail: Lahore University of Management Sciences , Pakistan (e-mail: Lahore University of Management Management Sciences ,, Pakistan Pakistan (e-mail: (e-mail: (14030008, ahmad.kamal, fakhir.khan)@lums.edu.pk). Lahore University of Sciences (14030008, ahmad.kamal, fakhir.khan)@lums.edu.pk). (14030008, ahmad.kamal, fakhir.khan)@lums.edu.pk). *Lehrstuhl fr Regelungsund Steuerungstechnik (RST), Department (14030008, ahmad.kamal, fakhir.khan)@lums.edu.pk). *Lehrstuhl fr Regelungsund Steuerungstechnik (RST), Department *Lehrstuhl und (RST), Department Elektrotechnik und Informatik, Fakultt IV, Universitt Siegen, 57068, *Lehrstuhl fr fr RegelungsRegelungsund Steuerungstechnik Steuerungstechnik (RST), Department Elektrotechnik und Informatik, Fakultt IV, Universitt Siegen, Elektrotechnik und Informatik, Fakultt IV, Universitt Universitt Siegen, Siegen, 57068, 57068, Siegen und Germany (e-mail: [email protected] Elektrotechnik Informatik, Fakultt IV, Siegen Germany (e-mail: [email protected] 57068, Siegen Siegen Germany Germany (e-mail: (e-mail: [email protected] [email protected] Abstract: This research work describes an approach to count trees in an urban environment. Abstract: This research work describes an approach to count trees in an urban environment. Abstract: research work describes an to trees in an environment. Furthermore it addresses problems involved in detection of trees aerial imagery. This work Abstract: This This researchthe work describes an approach approach to count count treesin in an urban urban environment. Furthermore it addresses the problems involved in detection of trees in aerial imagery. This work Furthermore it addresses the problems involved in detection of trees in aerial imagery. This can be used to solve the problem of forest degradation and deforestation. Right now forest man Furthermore it addresses the problems involved in detection of trees in aerial imagery. This work work can be used to solve the problem of forest degradation and deforestation. Right now forest man can be used to solve the problem of forest degradation and deforestation. Right now forest man labor isn’t efficient enough to detect or prevent this problem. A multi-rotor UAV equipped with can beisn’t usedefficient to solveenough the problem of forest degradation and deforestation. Right now forest with man labor to detect or prevent this problem. A multi-rotor UAV equipped labor resolution isn’t efficient efficient enough to detect detect or prevent prevent thisaerial problem. A multi-rotor multi-rotor UAV equipped with high RGB camera was used to acquire images and to count number of trees labor isn’t enough to or this problem. A UAV equipped with high resolution RGB camera was used to acquire aerial images and to count number of trees high resolution RGB camera was used acquire aerial images number of in surveyed area. Various issues the robust of proposed algorithm high resolution RGB camera wasinvolved used to to in acquire aerialimplementation images and and to to count count number of trees trees in surveyed area. Various issues involved in the robust implementation of proposed proposed algorithm in surveyed area. Various issues involved in the robust implementation of algorithm are discussed. The result of successful implementation of the proposed algorithm on multiple in surveyed area. Various issues involved in the robust implementation of proposed algorithm are discussed. The result of successful implementation of the proposed algorithm on multiple are The result implementation of proposed algorithm on scenarios are also and we show that our naive approach is able to achieve ≈ 0.72 are discussed. discussed. The presented result of of successful successful implementation of the the proposed algorithm on multiple multiple scenarios are also presented and we show that our naive approach is able to achieve scenarios within are also also presentedamount and we weofshow show that our our naive naive approach approach is is able able to to achieve achieve ≈ ≈ 0.72 0.72 accuracy reasonable time.that scenarios are presented and ≈ 0.72 accuracy within reasonable amount of time. accuracy within reasonable amount of time. accuracy within reasonable amount of time. © 2016, IFAC (International Federation of Automatic Control) Hosting by Elsevier Ltd. All rights reserved. Keywords: Precision Forestry, Robotics, Vision, UAV. Keywords: Precision Forestry, Forestry, Robotics, Vision, Vision, UAV. Keywords: Keywords: Precision Precision Forestry, Robotics, Robotics, Vision, UAV. UAV. 1. INTRODUCTION Object detection and counting has been widely used in 1. INTRODUCTION INTRODUCTION Object detection detection and and counting counting has has been been widely widely used in in 1. Object computer vision community overhas the been past decades. People 1. INTRODUCTION Object detection and counting widely used used in computer vision community over the past decades. People vision community the decades. have worked on automatic carover counting, Moranduzzo et al. computer vision community the past past decades. People People In underdeveloped countries, lack of resources causes the computer have worked on automatic automatic carover counting, Moranduzzo et al. al. In underdeveloped underdeveloped countries, countries, lack lack of of resources resources causes causes the the have worked car counting, Moranduzzo et (2014), and on people counting methods, Velipasalar et have worked on automatic car counting, Moranduzzo et al. al. In major trouble for forest department to monitor a countrys (2014), and people counting methods, Velipasalar et In underdeveloped countries, lack of resources causes the major trouble trouble for for forest forest department department to to monitor monitor aa countrys countrys (2014), and people methods, Velipasalar et al. (2006). Research intocounting automatic tree detection and delin(2014), and people counting methods, Velipasalar et al. major forest area. Expensive sensor networks dont allow teams to (2006). Research Research into into automatic automatic tree tree detection detection and and delindelinmajor area. trouble for forestsensor department to dont monitor a countrys forest Expensive networks allow teams to to (2006). eation digitalinto imagery was started back in and mid-1980. (2006).from Research automatic tree detection delinforest area. Expensive sensor networks dont allow teams get real time data to monitor forest degradation and man eation from digital imagery was started back in mid-1980. forest area. Expensive sensor networks dont allow teams to get real real time time data data to to monitor monitor forest forest degradation degradation and and man man eation from imagery was back mid-1980. number of detection algorithms been eation then from aadigital digital imagery was started started back in in have mid-1980. get labor is time expensive reallyforest slow. degradation In recent years, the Since Since then number of detection detection algorithms have been get real data toand monitor and man labor is expensive and really slow. In recent years, the Since then aa number algorithms have been proposed. Pinz (1991) of proposed a Vision Expert System Since then number of detection algorithms have been labor is expensive and really slow. In recent years, the use of Unmanned Aerial Vehicles (UAVs) has increased proposed. Pinz (1991) proposed a Vision Expert System labor is expensive and really slow. In recent years, the use of of Unmanned Unmanned Aerial Aerial Vehicles Vehicles (UAVs) (UAVs) has has increased increased proposed. (1991) Vision System using aerialPinz imagery. Heproposed was ableaa to locateExpert the center of proposed. Pinz (1991) proposed Vision Expert System use rapidly, mostly being adopted for many civilian applicausing aerial aerial imagery. imagery. He He was was able able to to locate locate the the center center of of use of Unmanned Aerial Vehicles (UAVs) has increased rapidly, mostly being adopted for many civilian applicausing trees crown and estimate its radius using local brightness using aerial imagery. He was able to locate the center of rapidly, mostly for civilian applications. Inexpensiveness of UAV systems enabled them trees crown and estimate its radius using local brightness rapidly, mostly being being adopted adopted for many manyhas civilian applications. Inexpensiveness of UAV systems has enabled them trees crown and estimate its radius using local brightness maxima. Valley following its and rule using basedlocal algorithm was trees crown and estimate radius brightness tions. Inexpensiveness of UAV systems has enabled them to be used for various research applications e.g. agriculmaxima. Valley Valley following following and and rule rule based based algorithm algorithm was was tions. UAV systems has enabled them maxima. to be Inexpensiveness used for for various various ofresearch research applications e.g. agriculagriculby Gougeon (1995), in mid-1990s in whichwas he maxima. Valley following and rule based algorithm to be used applications e.g. ture, mapping and survey. Recently people have used UAV presented presented by Gougeon (1995), in mid-1990s in which he to be used for various research applications e.g. agriculture, mapping and survey. Recently people have used UAV presented by Gougeon (1995), in mid-1990s in which he follows the valleys of shadows between tree crowns using presented by Gougeon (1995), between in mid-1990s in which he ture, and Recently have images for vegetation monitoring,Pollock andUAV car follows the valleys of shadows tree crowns using ture, mapping mapping and survey. survey. Recently people people (1996), have used used UAV images for vegetation monitoring,Pollock (1996), and car follows valleys of between tree using ground sampled distance digital aerial imagery. Machine follows the the valleys of shadows shadows between tree crowns crowns using images for vegetation monitoring,Pollock (1996), and car counting, Moranduzzo et al. (2014) Daigavane and Bajaj ground sampled distance digital aerial imagery. Machine images for vegetation monitoring,Pollock (1996), and car counting, Moranduzzo Moranduzzo et et al. al. (2014) (2014) Daigavane Daigavane and and Bajaj Bajaj ground digital Machine has alsodistance been used to aerial detect imagery. individual trees. ground sampled sampled digital Machine counting, (2010). In Moranduzzo fact, UAVs allow and monitoring small learning learning has also alsodistance been used used to aerial detect imagery. individual trees. counting, et al.mapping (2014) Daigavane and Bajaj (2010). In fact, UAVs allow mapping and monitoring small learning has been to detect individual trees. Pollock (1996) used model-based template matching techlearning has also been used to detect individual trees. (2010). In fact, UAVs allow mapping and monitoring small areas at extremely fine scales, quick survey of target area Pollock (1996) used model-based template matching tech(2010). In fact, UAVs allow mapping and monitoring small areas at extremely fine scales, quick survey of target area Pollock (1996) used model-based template matching techniques to recognize individual trees. Pollockto(1996) used individual model-based template matching techareas at scales, survey of and acquisitions over the same areaarea at niques recognize trees. areasmulti-temporal at extremely extremely fine fine scales, quick quick survey of target target area and multi-temporal acquisitions over the same area at niques to recognize individual trees. niques recognizeUAV’s individual trees. and acquisitions predefined times. and multi-temporal multi-temporal acquisitions over over the the same same area area at at We havetopreferred high resolution images over satelpredefined times. We have preferred UAV’s high resolution images over over satelsatelpredefined times. We have preferred UAV’s high resolution images predefined times. lite images because satellite images are readily by We have preferred UAV’s high resolution imagesaffected over satelIn this research work, we have implemented a technique to lite images because satellite images are readily affected by In this research work, we have implemented a technique to lite images because satellite images are readily affected by cloudy environments. Also, freely available satellite images lite images because satellite images are readily affected by In this research work, we have implemented a technique to count trees usingwork, UAVwe and computer vision aalgorithms. In cloudy environments. Also, freely available satellite images In this research have implemented technique to count trees trees using using UAV UAV and and computer computer vision vision algorithms. algorithms. In In cloudy environments. Also, freely available are of low resolution as compared to UAV satellite images. images cloudy environments. Also, freely available satellite images count developing countries millions of people depend on forests of low low resolution resolution as as compared compared to to UAV UAV images. images. count trees countries using UAVmillions and computer vision algorithms. In are of developing of people people depend on forests forests are ofrest lowofresolution images. developing of depend on fao (2015), countries directly ormillions indirectly, for their livelihood due are developing countries millions of people depend on forests The the paperasis compared organized to as UAV follows, in section II fao (2015), directly or indirectly, for their livelihood due The rest rest of of the the paper paper is is organized organized as as follows, follows, in in section section II II fao (2015), indirectly, for their livelihood due to which a directly notable or amount of forest is destroyed every The fao (2015), directly or indirectly, for their livelihood due we have presented some related work by use of in satellite and The rest of the paper is organized as follows, section II to which a notable amount of forest is destroyed every we have presented some related work by use of satellite and to which a notable amount of forest is destroyed every year in tropics either in the ofform of is forest degradation have presented some related by use satellite and to which a notable amount forest destroyed every we UAV imagery. In section III wework present ourof methodology we have presented some related work by use of satellite and year in tropics either in the form of forest degradation UAV imagery. imagery. In In section section III III we we present present our our methodology methodology year in in of or deforestation. Deforestation and degradation has socio- UAV year in tropics tropics either either in the the form form of forest forest degradation degradation usedour tomethodology count trees UAV discuss imagery.an Inapproach section IIIwe wehave present or deforestation. Deforestation and degradation degradation has sociosocio- and and discuss an approach we have used to count trees trees or deforestation. Deforestation and has economic impact on small communities. These communidiscuss an approach we have used to count or deforestation. Deforestation and degradation has socio- and from aerial imagery. Section IV presents results our and discuss an approach we have used to countof trees economic impact on small communities. These communifrom aerial imagery. Section IV presents results of our economic impact small communities. These communities use forests as on their living hood. Forests provide major from aerial imagery. Section IV presents results of our economic impact on small communities. These communiimplemented algorithm and Section V discusses some from aerial imagery. Section IV presents results of our ties use forests as their living hood. Forests provide major implemented algorithm and Section V discusses some ties use forests as their living hood. Forests provide major environmental benefit by reducing global warming. But implemented and Section V discusses some ties use forests as their by living hood. Forests provide major conclusion andalgorithm future work. implemented algorithm and Section V discusses some environmental benefit reducing global warming. But conclusion and and future work. work. environmental benefit by reducing warming. But excessive degradation canglobal damage atmosphere environmental benefit of byforests reducing warming. But conclusion conclusion and future future work. excessive degradation of forests canglobal damage atmosphere excessive degradation of forests can damage atmosphere and release greenhouse gases in air which leads to global 2. RELATED WORK excessive degradation forests canwhich damage atmosphere and release greenhouseofgases gases in air air leads to global global 2. RELATED RELATED WORK WORK and release greenhouse in which leads to warming. Which eventually leads to climate change. In 2. and release greenhouse gases in air which leads to global 2. RELATED WORK warming. Which Which eventually eventually leads leads to to climate climate change. change. In In warming. Vibha et al. (2009), produced promising results using a most of the cases, people clear tropical forest to cultivate warming. Which eventually leads to climate change. In Vibha et et al. al. (2009), (2009), produced produced promising promising results results using using aa most of of the the cases, cases, people people clear clear tropical tropical forest forest to to cultivate cultivate Vibha most robust technique for counting of trees from remotely senseda land. Vibha technique et al. (2009), produced promising results using most of the cases, people clear tropical forest to cultivate robust for counting of trees from remotely sensed land. robust technique for counting of trees from remotely land. robust technique for counting of trees from remotely sensed sensed land. Copyright © 2016, 2016 IFAC 16 Hosting by Elsevier Ltd. All rights reserved. 2405-8963 © IFAC (International Federation of Automatic Control) Copyright © 2016 IFAC 16 Copyright © 2016 IFAC 16 Peer review under responsibility of International Federation of Automatic Copyright © 2016 IFAC 16 Control. 10.1016/j.ifacol.2016.10.004
IFAC AGRICONTROL 2016 August 14-17, 2016. Seattle, Washington, USA Omair Hassaan et al. / IFAC-PapersOnLine 49-16 (2016) 016–021
data. They have used high resolution satellite imagery and pre-processed the data using median filter and intensity based processing. For vegetation they used feature extraction and template generation techniques where a masking function is generated and scanned from bottom over the whole image. This algorithm has its limitations due to the use of satellite imagery, since high resolution images are not freely available, and specifically targets only the species of palm trees which are structurally different from the rest of tree species.
17
Fig. 1. DJI Phantom 2.0 perform object extraction, image filtering operations and image segmentation operations. Making the assumption that tree crowns are located at the center or close to the center of tree top, Brandtberg and Walter (1998) detected trees using local maxima values. In this algorithm, image is first smoothed to reduce the noise effect and then tree crowns are filtered using local maximum values. Trees crown and background is delimited using edge detection methods. This method is simple and produces promising results in very less time, but light variations and un-wanted background objects in image badly affects the results.
Kattenborn et al. (2014), proposed a technique of automatic single palm tree detection using photogrammetric point clouds. Single camera images were processed with a structure from motion tool chain using Visual SFM. Each image was classified into 3 classes, i) palm ii) shrubs/trees iii) ground. For classification, a multi-scale dimensionality criterion was used in which the classifier is trained on different scale factors for train and test dataset. Local dimensionality characteristics of point cloud are used to classify palm trees and ground soil. Training a classifier for dataset posts limitations for such algorithms. Since training classifier is time consuming and requires more computational power. For each type of tree species one has to train the classifier before detecting trees.
Wang and Gong (2004) further improved the work of Brandtberg and Walter (1998) by first detecting boundaries of tree crowns using edge detection methods and then intersected the results of local non-maximum suppression on gray-level image & local maximum values of morphological transformed distance between pixels. Intersecting both methods give a good estimate of the tree tops which are then counted using contour based methods. This algorithm also suffers the presence of background objects, for example, buildings and roads.
Bazi et al. (2014), used Scale Invariant Feature Transform (SIFT) to extract key points and used a pre-trained classifier based on Extreme Learning Machine (ELM) to classify the extracted key points. As output the ELM classifier marked each detected palm tree as key points. In order to capture the shape, the extracted key points are merged using active contour based method based on Level-Set (LS). Since some green plants can be confused with palm trees so to classify palm and non-palm regions texture analysis is performed. Kernel based learning machines pose a limitation of high training cost which includes time and computational power. This also restricts the detection/classification approach to single species, in case, if trees from different species are required to be detected, ELM has to be pre-trained before using.
3. METHODOLOGY In this research work we implement a technique of image segmentation. To interpret an image, it is usually necessary to extract objects from background or separate image into several meaningful regions. Generally image segmentation clusters pixels of an image based on a given rule or criteria. Color textures along with correlation among color bands form a good feature extraction.
Hung et al. (2006) proposed vision based shadow aided tree crown tree detection and classification algorithm using imagery from UAV. Algorithm comprises of 2 parts i) detection ii) classification. The algorithm uses color and texture information to segment regions of interest (tree crown existence). A model object matching step is done which uses the shape, scale and context information for classification to differentiate the tree crown species. Image segmentation step is done using Support Vector Machines (SVM). Although SVMs have good generalization performance, they can by really slow in test phase. SVMs depend on the choice of kernel and have high algorithmic complexity along with extensive memory requirements in large-scale tasks.
We have used a quad-copter equipped with 14.4 Mega pixel camera pha (2014). Aerial imagery is acquired by first surveying the target area. Afterward we created a mosaic using acquired images so that we can easily detect and count trees using our algorithm. 3.1 Preprocessing Data Acquisition The data acquisition for this thesis problem was done using quadrotor DJI-Phantom 2. This quadrotor is equipped with 14.4 Megapixel camera and has a flight time of almost 15-20 minutes. Figure 1 shows quadrotor. Aerial imagery is acquired by first surveying the target area and then those images are used for further processing. Different data sets were taken at different sites and at different heights.
The use of Ground control points for UAV data acquisition and spatial filtering is also being used for forest tree counting. Mansur et al. (2014) used UAV to collect data. They have used the concept of crown geometry and vegetation response to radiation. The detection of tree crown can be possible by applying spatial convolution processing technique such low pass filter to enhanced image. After spatial filter, Morphological analysis is applied on the dataset to
LUMS dataset: Dataset at Lahore University of Management Sciences Pakistan was taken at a height of 100 meters from ground. GPS co-ordinates were given as input to the quad-rotor for trajectory to follow. A total of 150 images covering 610 meters of ground area were taken. 17
IFAC AGRICONTROL 2016 18 Omair Hassaan et al. / IFAC-PapersOnLine 49-16 (2016) 016–021 August 14-17, 2016. Seattle, Washington, USA
Fig. 2. UAV’s trajectory in LUMS
Fig. 5. Sample Image from ChangaManga Data set from optical axis. The apparent effect looks like an image which has been mapped around a sphere or barrel. To produce meaningful results, images must be pre-processed in order to remove lens distortions. Image rectification is applied to remove such distortions and images are straightened. We first get the camera’s intrinsic parameters using checkerboard images. These checkerboard images are given to camera calibrator application in MATLAB to determine intrinsic parameters. These intrinsic parameters include camera matrix, distortion model, rectification matrix and projection matrix.Figure 6 shows the calibration process using MATLAB’s calibrator application and Figure 7 show the intrinsic parameters determined through calibration process. Once we get calibration matrix, this along with distorted image are given to the image rectification tool of MATLAB in order to remove barrel distortion. Figure 10 shows the image after rectification process.
Fig. 3. UAV’s trajectory-1 at ChangaManga
Fig. 4. UAV’s trajectory-2 at ChangaManga Figure 2 shows the trajectory followed by quad rotor and Figure 9 shows a sample image from this data set. Changa Manga dataset: Changa Manga is a handplanted forest located approximately 80 kilometeres from Lahore, the provincial capital of Punjab, Pakistan. It was once considered as the largest man-made forest in the world, but regular forest degradation and deforestation at massive scale has reduced its size. It covers almost 12,423 acres of land. Two different data sets were taken at Changa Manga so that we can cover maximum portion of this forest. For data set 1, quad rotor was given GPS co-ordinates as input covering a total of 1612 meters. Quadrotor was flown at a height of 150 meters from ground and a total of 170 images were acquired in this process. Figure 3 shows the trajectory followed by quadrotor and Figure 5 shows a sample image from this data set. For data set 2, quad rotor covered an distance of 1973 meters at a height of 150 meters from ground. A total of 220 images were taken in this process. Figure 4 shows the trajectory map followed by quad rotor.
Fig. 6. Camera Calibration using MATLAB
Fig. 7. Camera intrinsic parameters Image Enhancement: After image acquisition and passing through rectification process, it is very difficult to segment out all green portions from the image due to the variance in shades of green color. As shown in Figure 9, due to weather conditions and acquisition time that is dependent on azimuthal angle of sun we have yellowish tints and also some bright & dark portions in trees. In order
Image Rectification: Images acquired by quad rotor’s camera have barrel distortion, as shown in Figure 9. In barrel distortion, image magnification decreases as we go away 18
IFAC AGRICONTROL 2016 August 14-17, 2016. Seattle, Washington, USA Omair Hassaan et al. / IFAC-PapersOnLine 49-16 (2016) 016–021
19
Fig. 8. Methodology Flow Chart Fig. 11. After Applying K-means clustering on L*a*b image 3.3 Vegetation Segmentation Color segmented image has some portions of trees, grass, other vegetation and some noise. In order to successfully count trees we have to remove grass, other vegetation and the noise from image. Here we use texture analysis to accomplish our task. We can quantify the intuitive qualities such as roughness, smoothness etc. in an image using the spatial variation in pixel intensities or gray values. We can use the 2 dimensional dependence matrix known as co-occurrence matrix to get insight of texture. To define a gray-level cooccurrence matrix P[i,j], we first specify a displacement vector d = (dx,dy) and counting all pairs of pixels separated by d having gray levels i & j. Consider an 8x8 binary image of checkerboard shown in Figure 12. Since there are only 2 gray-levels, P [i,j] is a 2x2 matrix and if we define d = (1,1) we get the normalized P [i,j] as shown in Figure 13. If black pixels in above image are randomly placed, the matrix is expected to be uniformly populated. To measure the randomness of gray-level distribution we used entropy, Jain et al. (1995), as defined in (1),
Fig. 9. Sample RGB input image
Fig. 10. Rectified and Enhanced RGB Image to overcome these artifacts we tune the color saturation of the acquired images to give a level playing field for our algorithm. We enhance the color saturation so that different colors are easily distinguished during color based segmentation. Figure 10 shows the image after rectification and enhancement process.
Entropy = −Σi Σj P [i, j]logP [i, j]
(1)
Entropy value is highest when all entries in P [i,j] are equal which corresponds to an image in which there are no preferred gray-level pairs a distance d.
3.2 Color Based Clustering In urban areas, where tree density is low and sparsely located in a region, we can count trees using color threshold techniques. Here we first convert the RGB input image, as shown in Figure 9, to L*a*b color space. The L*a*b* color space is derived from the CIE XYZ tristimulus values. The L*a*b* space consists of a luminosity layer ’L*’, chromaticity-layer ’a*’ indicating where color falls along the red-green axis, and chromaticity-layer ’b*’ indicating where the color falls along the blue-yellow axis. All of the color information is in the ’a*’ and ’b*’ layers. One can measure the difference between two colors using the Euclidean distance metric. Here we apply k-means clustering algorithm on both layers i.e. a* and b*. By applying threshold values on green clusters centers, we segment out the green portion out of the image without any noise. Figure 11 shows the resultant image after applying Kmeans clustering.
Fig. 12. 8x8 Checkerboard We have used the same technique to segment the vegetation out of green-masked image. As we can see in Figure 11, trees have more roughness in structure as compared to grass. So after applying entropy filter on our segmented image and normalizing the output between 0-1, we can clearly see that rough portion (trees) is more prominent as compared to grass, as shown in Figure 14. Here, in Figure 14, we can treat grass region as a part of noise in our image. In order to remove noise from image, we apply dilation and erosion filters to remove it. Figure 15 19
IFAC AGRICONTROL 2016 20 Omair Hassaan et al. / IFAC-PapersOnLine 49-16 (2016) 016–021 August 14-17, 2016. Seattle, Washington, USA
Fig. 13. P[i,j] for d = (1,1) shows the output after applying morphological operations. Here we can see that the only portions left in our image are trees.
Fig. 16. Circle fitting approximation technique Table 1. Results (count)
3.4 Tree Counting
Fig# 11 12 13
At this point, we have successfully segmented out trees from all the other information present in the image. Now we want to use this information to help us count the quantity of trees present in the scanned area. To get this count, we use a circle fitting approximation technique. To use this technique, we need to know the distance occupied by a pixel in the image. Since we have taken all the data set from a fixed altitude of 100 meters above ground and we know the original image resolution of our UAVs camera, we can calculate the per pixel distance in centimeters using basic mathematics. In our case, we have calculated it to be 3.84cm/pixel. Next, we calculated the measured the span of trees on ground by using conventional measurement tools. After getting the minimum and maximum values of radii in centimeters, we provided this information to our circle fitting algorithm. Which tries to fit circles based on the range of radius we have provided it. We use the binary image as shown in Figure 15 as input and try to fit circles where pixels value is 1. Figure 16 shows the result of circle fitting approximation.
GT 37 42 47
CC 35 49 51
TP 25 31 35
TN 0 0 0
FP 8 18 16
FN 9 11 14
Time(sec) 10.78 7.86 8.4
Table 2. Accuracy Fig# 11 12 13
TPR 0.735 0.738 0.714
FPR 1 1 1
TNR 0 0 0
FNR 1 1 1
Accuracy % 67.5 73.8 74.46
4. RESULTS In this section the results of the experiment described in chapter 3 are discussed. Test was performed using MATLAB on Intels core-i3 processor with 4GB RAM. Figure 17 to ?? show the sample input images tested on the above discussed algorithm and Figure 19 to ?? show trees detected in images respectively. Table 1 shows the numerical figures of Ground Truth (GT) Circle Count (CC), True Positives (TP), True Negatives (TN), False Positives(FP) and False Negatives(FN) for all the 3 test images (Figure 11 to 13). Table 2 shows the True Positive Rate (TPR), False Positive Rate (FPR), False Negative Rate (FNR) and accuracy of algorithm. True Positive Rate is given by T P R = T P/(T P + F N ), False Positive Rate: F P R = F P/(F P + T N ), True Negative Rate: T N R = T N/(F P + T N ) and Accuracy is given by (T P + T N )/T otal
5. DISCUSSION & FUTURE WORK This paper presents a novel work on trees detection and counting in urban areas which can significantly reduce man labor. Tree detection and counting helps us in solving the problem of forest degradation and deforestation. Generally this problem is addressed using man labor and in case of large forests, man labor isn’t effecient. This paper presents an algorithm which uses some basic image processing techniques for vegetation segmentation and also for tree counting. Here we haven’t used machine learning algorithms for tree counting method. Since we have used an approximation technique for tree counting, the output results of this algorithm has some false positives and false negative numbers. Trees which appear much smaller in acquired images don’t have significant change in texture as compared to their surrounding or grass. Such kind of trees are either missed during color segmentation part or during
Fig. 14. Result of Entropy Filter
Fig. 15. After applying Morphological Operations (RGB) 20
IFAC AGRICONTROL 2016 August 14-17, 2016. Seattle, Washington, USA Omair Hassaan et al. / IFAC-PapersOnLine 49-16 (2016) 016–021
21
REFERENCES
texture analysis. Also in case of large trees, circle fitting algorithm often tries to fit more than 1 circle on its canopy. Such kind of errors can be reduced and accuracy can be increased by taking into account for machine learning algorithms. Learning based on tree’s texture can help us in detecting and counting tree canopies. We can also follow the work done by Yakub Bazi et al. (2014) to create a SIFT feature based extended learning machine.
(2014). Dji phantom. URL http://store.dji.com/product/phantom-2/. (2015). Fao statistics on forestry. URL http://www.fao.org/forestry/46203/en/. Bazi, Y., Malek, S., Alajlan, N., and AlHichri, H. (2014). An automatic approach for palm tree counting in uav images. IGARSS, IEEE. Brandtberg and Walter (1998). Automated delineation of individual tree crowns in high spatial resolution aerial images by multiple-scale analysis. Machine Vision and Applications, 64–73. Daigavane and Bajaj (2010). Real time vehicle detection and counting method for unsupervised traffic video on highways. IJCSNS International Journal of Computer Science and Network Security, 10. Gougeon (1995). A crown-following approach to the automatic delineation of individual tree crowns in high spatial resolution aerial images. Canadian Journal of Remote Sensing, 21, 274–284. Hung, Bryson, and Sukkarieh (2006). Vision based shadow aided tree crown detection and classification algorithm using imagery from an unmanned airborne vehicle. International Symposium for Remote Sensing of the Environment. Jain, R., Kasturi, R., and Schunck, B. (1995). Individual tree-crown delineation and treetop detection in highspatial-resolution aerial imagery. McGraw-Hill. Kattenborn, Sperlich, Bataua, and Koch (2014). Automatic single palm tree detection in plantations using uav-based photogrrammteric point clouds. The international Archives of the Photogrammetry, Remote sensing and spatial information sciences, XL-3. Mansur, Mukhtar, and Al-Doksi, J. (2014). The usefullness of unmanned airborne vehicle (uav) imagery for automated palm oil tree counting. Research Journali’s journal of Forestry, 1. Moranduzzo, Thomas, and Melgani, F. (2014). Automatic car counting method for unmanned aerial vehicle images. Geoscience and Remote Sensing, IEEE Transactions on 52, 1635–1647. Pinz (1991). A computer vision system for recognition of trees in aerial photographs. International Association of Pattern Recognition Workshop, 111–124. Pollock (1996). the automatic recognition of individual trees in aerial images of forests based on a synthetic tree crown model. Doctoral dissertation, Concordia University. Velipasalar, S., Tian, Y.L., and Hampapu, A. (2006). Automatic counting of interacting people by using a single uncalibrated camera. Multimedia and Expo, IEEE International Conference. Vibha, Shenoy, D., Venugopal, and Patnaik (2009). Robust technique for segmentation and counting of trees from remotely sensed data. IEEE International Advance Computing Conference. Wang, L. and Gong, P. (2004). Individual tree-crown delineation and treetop detection in high-spatial-resolution aerial imagery. Photogrammetric Engineering and Remote Sensing, 70, 351–357.
Fig. 17. Test Image 1
Fig. 18. Test Image 2
Fig. 19. Circle fitting result of Test Image 1
Fig. 20. Circle fitting result of Test Image 2
21