- Email: [email protected]
Developing an autonomous imaging and localisation capability for planetary aerobots
IAV2004 - PREPRINTS 5th IFAC/EURON Symposium on Intelligent Autonomous Vehicles Instituto Superior Técnico, Lisboa, Portugal July 5-7, 2004
DEVELOPING AN AUTONOMOUS IMAGING AND LOCALISATION CAPABILITY FOR PLANETARY AEROBOTS
Mark Woods 1 , Malcolm Evans 1 , Roger War d1 Dave Barnes 2 , Andrew Shaw2 , Phil Summers 2 Gerhard Paar 3 , Mark Sims 4
1
2
SciSys (Space & Defence) Ltd., Clothier Rd., Bristol, BS4 5SS, UK Dept. of Computer Science, University of Wales, Aberystwyth, SY23 3DB, UK 3 Joanneum Research, Graz, A-8010, Austria 4 Space Research Centre, Dept. University of Leicester, LE1 7RH, UK * Correspondence: [email protected] or [email protected]
Abstract: Balloon based planetary aerobots can be used for a variety of applications such as high resolution imaging and rover guidance. However short to medium term missions of this type will be constrained in terms of power, communications, data storage and processing capability. To be of use, they must be able to localise and manage image data in an autonomous manner, including intelligent prioritisation of images. This paper discusses the development of an intelligent imaging and localisation software package and demonstrator which will help to provide such an autonomous capability.
Keywords: Space and Aerial Robotics, Aerobot, Autonomous Systems, Localisation, Imaging, Planetary Surface Exploration, Vision Based Navigation.
1. INTRODUCTION
regions. This is required to bridge gaps in the datasets available currently and to enable more detailed scientific analysis , in effect offering nearground truth (Neck, et al., 1997).
The continued drive to explore our Solar System through initiatives such as the European Space Agency’s (ESA) Aurora programme has increased demand for a variety of intelligent, autonomous vehicles. Previous missions and those planned for the near term such as ExoMars have focused on the deployment of orbiters and rovers or landers on or around Mars. Orbiters such as NASA’s Mars Global Surveyor (MGS) or Odyssey and now ESA’s Mars Express have provided invaluable data which has vastly improved our understanding of Martian geological development through the acquisition of global imagery at increasingly fine resolutions. Orbital data has been substantially complemented by the success of in-situ missions such as the twin MER rovers which at the time of writing are returning highly significant results on the role of water in the formation of Martian geology. Such vehicles provide a unique opportunity to interact directly with remote planetary bodies and gather fine detail and highly localised data. However there is a need for an intermediary surveillance capability which can gather high resolution data in small to medium scale
1.1 Balloon Based Planetary Aerobot Technology and Autonomy Challenges Lighter Than Air (LTA) balloon aerobots are ideally suited to this type of mission and in a wider sense are also represent an extremely versatile exploration platform. In addition to high resolution imaging they can be used for a wide range of applications such as, rover guidance, data relay, sample site selection, payload delivery, atmospheric measurement and analysis and compositional mapping. It is possible to consider the advantages of tethering an aerobot to a rover thus providing a ‘birds-eye’ view of the surrounding terrain which would simplify the navigation and sample site selection tasks. A significant feature of LTA balloon aerobots is that many of the technology building blocks exist compared with alternative Heavier Than Air (HTA) schemes. Indeed simple LTA balloons have been flown on Venus as early as 1985 as part of the
926
Franco-Russo VEGA mission which lasted for two days. As such they represent a relatively simple, lowcost way of achieving complex scientific goals. There are however a number of technical challenges that must be addressed (Cutts, et al., 1985, Balaram, et al., 1996). ALTAIR-1, (Barnes, et al., 2000, Shaw, et al., 2000), has been constructed as part of the ALTAIR (Aberystwyth Lighter Than Air Intelligent Robot) programme to study the complex problem of autonomous flight control and navigation.
and of an intelligent assessment of the information content of the images at which resolution/compression they need to be transmitted to Earth In order to test the ILP software functionality the study will also develop a bi-modal demonstrator framework consisting of a balloon simulator, actual prototype balloon with wireless communication capability. The demonstrator will be used in either all software mode where the balloon simulator will be used to provide synthetic images under a variety of conditions to the ILP or in hardware mode where images will originate from a balloon mounted camera. In both cases a PC-based GUI application will be used to configure and control the demonstrator and allow interactive display of the ILP output. There are two components to this work therefore, the development of the ILP itself and the development of a hardware/software demonstrator.
In the immediate future it is likely that the first Martian aerobot mission will take the form of an uncontrolled, free flying high resolution imaging mission. Such an aerobot, drifting in the vagaries of a Martian wind across its often rugged terrain, would need to be autonomous, communicating with an orbiter as chance permits. With limited resources of memory and power, the main problem will be economic storage and use of images acquired. All unnecessary imagery will need to be suppressed. Given the communications and memory constraints it will be unlikely that the aerobot will be able to store and transmit all of the images it can acquire. It will therefore require sufficient intelligence to autonomously prioritise images according to predefined scientific goals.
2
ILP SYSTEM OVERVIEW
Fig. 1 outlines the basic components and processing flow of our proposed ILP system. The ILP system is designed to control and acquire images from a single camera suspended from the balloon gondola. It will subsequently process the images as described in the previous section.
Positional data would need to be provided to any scientific package carried. Although other means of navigation could be provided, using already available imagery would be the most economic. Thus there is a requirement to investigate localisation to the required accuracy, economic use of memory to store the vast quantity of images, definition of an intelligent image prioritisation scheme and the problem of predicting the next communication window with an orbiter, so that the best use of storage can be made to meet the mission targets without loss of any important information.
The technical challenges posed by individual tasks such as assessing scientific content or richness and predicting and uplink windows are now discussed. In addition there is the more general challenge of performing these tasks in a constrained real-time environment e.g. a cycle time of 4secs at a height of 100m at speed of 10m/sec assuming 70° Field of View (FOV) and image overlap of 70%. Significant algorithm optimisation is therefore required to meet these real-time demands.
1.2 Objectives The primary objective of our current work is to study, design and prototype an Imaging and Localisation Package (ILP) for a free-flying Martian balloon on a high-resolution imaging mission. The work is being carried out for ESA/ESTEC in order to investigate and demonstrate the viability of such a mission. An ILP package will allow optimal acquisition of images to reconstruct accurate models of the surface of the explored planet and accurate localisation of the balloon with respect to the Martian surface. The ILP , by means of aerobot mounted camera and computer vision techniques will:
acquire and store images at various resolutions construct and update a 3D model (DEM) of the surface constantly estimate the position (latitude, longitude and altitude) of the aerobot as well as its motion with respect to the surface and decide on the basis of the communications budget, of the morphology of the surface
Fig. 1. ILP System Overview
927
2.1 Image Acquisition and Richness Assessment
2.2 DEM Generation
The image acquisition process has been designed to reduce the computational overhead of the DEM generation process. An initial pre-processing algorithm determines aerobot displacement from successive images and uses this measure to control image acquisition. The aim of the control scheme is to ensure almost ideal overlap between successive images in order to reduce the time taken to generate the DEM and prevent redundant processing. DEMs require the image pairs to be similar but well contrasted. Therefore, with changing planetary conditions, camera exposure parameters may need to be controlled automatically.
A Digital Elevation Model (DEM) is a standard data structure for digital representation of a planetary surface. For each x/y coordinate within the represented area a height can be directly derived from the DEM. In this particular aerobot application it is used in the ILP to support the global localisation by comp arison with a global DEM and image richness assessment. By displaying received DEMs on the ground segment GUI, scientists can complement knowledge obtained from image mosaics on local geology. In the aerobot case, DEMs are built using images taken at different positions of the aerobot’s single camera system. For the photogrammetric determination of a distance to a certain scene point (without loss of generality in the coordinate system of the temporally first image) the relative orientation between the camera positions (pointing and displacement vector) must be known, the respective scene point must be visible in both images. Consequently the images involved need to overlap since only within the overlap area a DEM can be generated. Any missing information leads to ambiguities in the resulting DEM data, such as unknown scaling or rotation and displacement with respect to the global coordinate system. Scaling problems can be resolved by one independent distance measurement to a point identified in both involved images (e.g. using an altimeter to provide distance to a point whose direction is known in the camera coordinate system), or the knowledge about the displacement between the stereo images. The real-time constraints identified in 2 mean that DEM generation must be carried out as efficiently as possible. A number of techniques have been developed to speed up this process and reduce computational load including the provision of ideal image overlap through controlled acquisition thus removing the need to estimate displacement. Advance knowledge of displacement itself also shortens the DEM generation process. Integration into the global coordinate system is possible when local DEM landmarks are identified in a global DEM. DEM generation is also highly connected to relative localisation since tracking of landmarks in principle is the same process as the stereo matching needed for photogrammetric DEM reconstruction.
An Image Richness Index (IRI) is also needed to determine the priority that should be given to the acquired images to ensure that those of high interest are not lost should the amount of data that could be transmitted to the orbiter become restrictive. This is a non-trivial problem as Fig. 2 shows. Although Image A in the figure has many features, the most scientifically interesting feature (for a particular science mode) appears in B. So although the quantity of features in B is low it should be assigned a higher IRI. The system must therefore incorporate feature extraction and configurable classification algorithms to model the objectives of mission scientists.
Fig. 2: Illustration of the Image richness classification problem. Image A has multiple features of low scientific value in this mode while B has fewer features but they have higher scientific value indicating possible fluvial activity. This capability is vital in ensuring aerobot autonomy. Without it the mission will be of much less scientific value. Initial work has focused on the survey and selection of reference data sets and consultation with planetary scientists in order to determine the target feature types and richness rankings. This is essential to define classification models. Reference areas should contain a wide variety of interesting geological features including volcanics, channel material, basin material, mountainous terrain and plains material (West, 1991). Limited MGS Mars Orbiter Camera (MOC) image sets are available at resolutions of 1.5m-12/pixel showing target features. These images may be complemented with terrestrial analogues to enhance reference sets.
2.3 Localisation Global and Relative Autonomous localisation is another key requirement for a free-flying aerobot mission. It is required to provide a reference for the scientific image analysis and to enable prediction of forthcoming uplinks which in turn are required to manage on-board memory and image compression. However this is not trivial as Mars does not currently possess a Global Positioning (GPS) system of its own and systems such as rate gyros, accelerometers, flux gate, compasses and sun/star sensors are subject to error and drift and are therefore inadequate in themselves. It is possible to derive positional information when in
928
contact with an orbiter whose position is known, however such contacts will be infrequent and cannot be used as the sole means of localising the aerobot.
location which meets the feature/gradient relationship is at the bottom of the image. Through experimentation it was found that the largest influencing factor to obtain a match was the size of the local DEM. A local DEM that covered an area of 10x10, global resolution (or more) produced 95% correct matching and areas that covered 5x5 global resolution, produced over 70% correct positions.
Fig. 3: Relative localisation using landmark tracking An alternative scheme has been adopted for this work which will use a combination of computer vision based global and relative localisation in conjunction with orbiter contacts to provide continuous aerobot localisation. The proposed scheme uses landmark tracking shown in Fig. 3 which is a by-product of the DEM generation process to provide a relative location fix. First tests on a small-scale mockup have shown a repeatability within 1% of travelled distance for the relative localisation. These relative measurements must be corrected with an absolute reference. To make up for infrequent orbiter contacts a novel landmark comparison scheme (Shaw and Barnes 2003) will be used to provide an absolute position.
Fig. 5: shows the points which were selected for the gradient profile, the white lines show the actual gradients measured
The landmark comparison method consists of two individual elements; the first is the analysis of the terrain for geological features, providing a density profile and the second, a gradient profile of the surface. Fig. 4 shows the region selected for the feature density profile (area within the orange rectangle, the various colours represent different features), the feature density within this area is used during the search of the global DEM . Fig. 5 shows the points which were used in the gradient profile (the four corners); the gradient between these points is then used when searching the global DEM. An actual location for the local DEM within the global DEM can be generated by using the positional relationship of the feature and gradient profile areas (form the local DEM) as these have to be the same in the global DEM.
Fig. 6: Matched DEM locations indicated by overlapping orange rectangle (matching feature density profile) and white square (matching gradient profile). There are a number of possible sources of reference topographical information. Global Martian surface data obtained to date has been from orbiter’s, the best of which was gathered from two instruments on board the Mars Global Surveyor (MGS), the Mars Orbiter Laser Altimeter (MOLA) and the Mars Orbiter Camera (MOC). Absolute localisation can therefore be achieved by extracting naturally occurring features (peaks, ridges, channels etc.) from the topographical maps. By categorising the surface by its features, then by matching these features in a high resolution DEM generated by the aerobot, with the same features in the low resolution global map (e.g. MOLA) a position estimate is found. The obvious limitation of the scheme that it is better suited to areas which are reasonably feature rich and diverse such as the Southern Highlands. However this is constraint is fully compatible with scientific goals which seek to explore such geologically rich areas.
Fig. 4: shows the area selected for the feature density profile, which is within the orange rectangle.
2.4 Image Store and Management
The matching results from the profiles of Fig. 4 and Fig. 5 can be seen in Fig. 6 where the matched
Since the time between communications with an orbiter will be variable and doubtlessly long, and the 929
amount of imagery is dependent upon the profile of the wind velocity, assuming no area of the surface beneath the aerobot which is scientifically interesting is to be lost, image storage management is an important aspect of the work. The image richness index is to be used to determine the priority to be given to the image, and possibly the image’s most interesting areas.
using a simulator and a hardware mode (see Fig. 8) that encompasses a real RC balloon plus camera with RF interface. Both modes being interfaced to a demonstrator shell GUI and associated ILP software. The use of a simulator as one of the core demonstrator components affords a number of advantages over a hardware-only solution:
Because the amount of mass memory is unlikely to be sufficient to hold all images produced before they can be transmitted and then deleted, all unnecessary imagery must be removed. This means that overlap between final stored images should be kept to a minimum, while ensuring that no pixels are lost due to changes in yaw between consecutive images. Reconstruction of the continuous mosaic of images and DEMs must eventually be possible on the ground. This defines the minimum uncompressed imagery required to be stored. Compression provides a means to greatly reduce and control the actual amount of storage required. Progressive compression techniques (ECBOT wavelet JPEG 2000) are being used to compress the images and DEM data. Such a technique is resilient against drop out and will also allow for example, the entropy coding to be tailored to meet any data length against image richness profile required by controlling the scaling against signal to nois e, resolution, visual quality, or of regions of lesser interest etc. Since the window of communication will vary as much as the vagaries of the wind, continuous update of the capabilities of the up-link is required and the compression ratio is therefore adjusted by layered deletion of the compressed image to roughly meet the link capability, and the memory storage requirements.
Fig. 7: Demonstrator - software mode Simulators are able to re-construct environments and terrain to a very accurate level whilst still allowing the user to maintain control over the specific parameters. Terrain, weather, atmosphere and hardware devices, such as cameras, can be simulated in a realistic way and noise can be modelled to allow for random fluctuations in the environment or manufacturing tolerances for instruments. A simulator is well suited for modelling the flight of a balloon over terrain, experiments cannot always be carried out due to numerous factors and the use of simulation will overcome any of these potential problems.
2.5 Uplink Scheduling Communications links with surface or low-altitude planetary exploration units are usually constrained in terms of frequency, duration and uplink downlink transmission rates. In addition, real-time commanding is not possible given lengthy RTLT durations. Given that imaging is data intensive process, the communications link access needs to be optimised by employing intelligent use of available bandwidth and transmission opportunities. Thus the transmissions must be scheduled by predicting the timing and duration of the next up-link of data based on the aerobot’s trajectory, the orbit of the relay satellite and the characteristics of the two antenna.
The trajectory of the balloon is initially supplied to the simulator, with periodic updates being sent. During periods where the trajectory of the balloon is unknown, interpolation is used to provide an estimate of the current trajectory. This provides a simplistic, but sufficient method, for controlling and modelling the trajectory of the balloon. A simulator also affords the ability to evaluate algorithms for localisation and control that may be difficult to examine on real hardware in an Earth-based environment. A major factor in the deployment of planetary balloons is the effect the environment and atmospheric conditions have upon the vehicle. The simulator therefore models haze, clouds and also wind given its bearing on ILP performance
During the transmission phase, the link must be acquired, any commands received and confirmed by the aerobot, before its image data can be transferred up to the orbiter, and their correct receipt confirmed. Only then can any images be deleted to make space for the new collection of images. 3
Repeatability of experiments; Simulation of environments that cannot be re-constructed within the laboratory.
DEMONSTRATOR SYSTEM OVERVIEW
An ILP demonstrator system proposed requires two distinct modes. An all software mode (see Fig. 7)
930
Additionally, functionality is provided to perform measurements on the DEMs such as the co-ordinates of individual points and the distance between selected. CONCLUSION We are currently developing an Imaging and Localisation Package and hardware/software demonstrator for a Martian balloon. An overview of our proposed system has been presented. For localisation we have proposed a novel combination of image based relative and global localisation techniques. We propose to demonstrate our ILP using both real and simulated aerobot and thereby provide a comprehensive test and evaluation of the ILP functionality. It is hoped that this work will contribute to definition of a feasible robotic balloon mission to Mars within the current decade. ACKNOWLEDGEMENTS This work is funded under European Space Agency (ESA) Contract No.17400/03/NL/CH. The authors would like to thank Gianfranco Visentin, Head of the Automation and Robotics Section at ESTEC (ESA D/TOS) for his collaboration and technical guidance on this project.
Fig. 8: Demonstrator - hardware mode Despite the availability of the simulator there is a clear need to test the ILP software in a more realistic environment. To this end a model balloon complete with propulsion, downward looking camera, processing and wireless control and communication capability is a key requirement for our current work and a variety of configurations are being prototyped. In addition a realistic test facility is necessary and this is provided in the first instance by the ESTEC PTB with its mock Martian terrain and high ceiling. To complement the PTB and support system tests a second facility is being designed for installation at our main team HQ. Outputs from the richness system development are being used to guide this design to ensure that the terrain is representative of actual flight areas.
REFERENCES Balaram, J., R.E. Scheid, P.T. Salomon, (1996). OnBoard Perception System For Planetary Aerobot Balloon Navigation, 23rd AUVSI, Orlando FL. Barnes, D.P., P. Summers, A. Shaw, (2000). An Investigation into Aerobot Technologies for Planetary Exploration, 6th ESA Workshop on Advanced Space Technologies for Robotics and Automation, ASTRA 2000, pp. 3.6-5. Cutts, J. A., K.T. Neck, J.A. Jones, G. Rodriguez, J. Balaram, G.E. Powell, S.P. Synnott, (1985) Aerovehicles for Planetary Exploration, IEEE International Conference on Robotics & Automation. Neck, K.T., J. Balaram, M.K. Heun, S. Smith and T. Gamber (1997). Mars 2001 Aerobot/Balloon System Overview, http://lheawww.gsfc.nasa.gov/docs/balloon/MAB S_27sept1996/mabs_27sept96.html Shaw, A., P. Summers, E. Geneste. and D.P. Barnes, (2000). ALTAIR-1: A laboratory based aerobot for planetary exploration experiments., Proc. EUREL European Advanced Robotics Systems, Masteclass and Conference, Salford UK. Shaw, A.J. and D.P. Barnes, (2003). Landmark recognition for localisation and navigation of aerial vehicles. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas. West F.R., Dao Vallis – Hadriaca Patera, http://cmexwww.arc.nasa.gov/MarsTools/Mars_Cat/Part_3/ Sites/site080.html
The demonstrator shell communicates with both the ILP and balloon simulator in order to receive uplinked DEMs and images and to download camera models and balloon trajectory information. The shell will allow selection of the demonstration mode, hardware or software, provide an interface to allow selection and upload of camera, antenna and terrain models and enable the user to define the communication window function. The user is also able to start, stop and pause the simulation at will. Another function afforded by the demonstrator shell is the display of demonstrator software status. This consists of the data provided to the demonstrator shell by the balloon simulator and the data produced by the ILP itself. A graphical interface is used to display this data in a clear and concise form. The presentation of this ILP data incorporates the display of image mosaics. This requires the demonstrator shell to reconstruct the sequence and determine the placing of the individual images within these mosaics. It must also display the DEM data received by the demonstrator shell. This DEM data is presented in a format suitable for easy visualisation.
931
DEVELOPING AN AUTONOMOUS IMAGING AND LOCALISATION CAPABILITY FOR PLANETARY AEROBOTS
Mark Woods 1 , Malcolm Evans 1 , Roger War d1 Dave Barnes 2 , Andrew Shaw2 , Phil Summers 2 Gerhard Paar 3 , Mark Sims 4
1
2
SciSys (Space & Defence) Ltd., Clothier Rd., Bristol, BS4 5SS, UK Dept. of Computer Science, University of Wales, Aberystwyth, SY23 3DB, UK 3 Joanneum Research, Graz, A-8010, Austria 4 Space Research Centre, Dept. University of Leicester, LE1 7RH, UK * Correspondence: [email protected] or [email protected]
Abstract: Balloon based planetary aerobots can be used for a variety of applications such as high resolution imaging and rover guidance. However short to medium term missions of this type will be constrained in terms of power, communications, data storage and processing capability. To be of use, they must be able to localise and manage image data in an autonomous manner, including intelligent prioritisation of images. This paper discusses the development of an intelligent imaging and localisation software package and demonstrator which will help to provide such an autonomous capability.
Keywords: Space and Aerial Robotics, Aerobot, Autonomous Systems, Localisation, Imaging, Planetary Surface Exploration, Vision Based Navigation.
1. INTRODUCTION
regions. This is required to bridge gaps in the datasets available currently and to enable more detailed scientific analysis , in effect offering nearground truth (Neck, et al., 1997).
The continued drive to explore our Solar System through initiatives such as the European Space Agency’s (ESA) Aurora programme has increased demand for a variety of intelligent, autonomous vehicles. Previous missions and those planned for the near term such as ExoMars have focused on the deployment of orbiters and rovers or landers on or around Mars. Orbiters such as NASA’s Mars Global Surveyor (MGS) or Odyssey and now ESA’s Mars Express have provided invaluable data which has vastly improved our understanding of Martian geological development through the acquisition of global imagery at increasingly fine resolutions. Orbital data has been substantially complemented by the success of in-situ missions such as the twin MER rovers which at the time of writing are returning highly significant results on the role of water in the formation of Martian geology. Such vehicles provide a unique opportunity to interact directly with remote planetary bodies and gather fine detail and highly localised data. However there is a need for an intermediary surveillance capability which can gather high resolution data in small to medium scale
1.1 Balloon Based Planetary Aerobot Technology and Autonomy Challenges Lighter Than Air (LTA) balloon aerobots are ideally suited to this type of mission and in a wider sense are also represent an extremely versatile exploration platform. In addition to high resolution imaging they can be used for a wide range of applications such as, rover guidance, data relay, sample site selection, payload delivery, atmospheric measurement and analysis and compositional mapping. It is possible to consider the advantages of tethering an aerobot to a rover thus providing a ‘birds-eye’ view of the surrounding terrain which would simplify the navigation and sample site selection tasks. A significant feature of LTA balloon aerobots is that many of the technology building blocks exist compared with alternative Heavier Than Air (HTA) schemes. Indeed simple LTA balloons have been flown on Venus as early as 1985 as part of the
926
Franco-Russo VEGA mission which lasted for two days. As such they represent a relatively simple, lowcost way of achieving complex scientific goals. There are however a number of technical challenges that must be addressed (Cutts, et al., 1985, Balaram, et al., 1996). ALTAIR-1, (Barnes, et al., 2000, Shaw, et al., 2000), has been constructed as part of the ALTAIR (Aberystwyth Lighter Than Air Intelligent Robot) programme to study the complex problem of autonomous flight control and navigation.
and of an intelligent assessment of the information content of the images at which resolution/compression they need to be transmitted to Earth In order to test the ILP software functionality the study will also develop a bi-modal demonstrator framework consisting of a balloon simulator, actual prototype balloon with wireless communication capability. The demonstrator will be used in either all software mode where the balloon simulator will be used to provide synthetic images under a variety of conditions to the ILP or in hardware mode where images will originate from a balloon mounted camera. In both cases a PC-based GUI application will be used to configure and control the demonstrator and allow interactive display of the ILP output. There are two components to this work therefore, the development of the ILP itself and the development of a hardware/software demonstrator.
In the immediate future it is likely that the first Martian aerobot mission will take the form of an uncontrolled, free flying high resolution imaging mission. Such an aerobot, drifting in the vagaries of a Martian wind across its often rugged terrain, would need to be autonomous, communicating with an orbiter as chance permits. With limited resources of memory and power, the main problem will be economic storage and use of images acquired. All unnecessary imagery will need to be suppressed. Given the communications and memory constraints it will be unlikely that the aerobot will be able to store and transmit all of the images it can acquire. It will therefore require sufficient intelligence to autonomously prioritise images according to predefined scientific goals.
2
ILP SYSTEM OVERVIEW
Fig. 1 outlines the basic components and processing flow of our proposed ILP system. The ILP system is designed to control and acquire images from a single camera suspended from the balloon gondola. It will subsequently process the images as described in the previous section.
Positional data would need to be provided to any scientific package carried. Although other means of navigation could be provided, using already available imagery would be the most economic. Thus there is a requirement to investigate localisation to the required accuracy, economic use of memory to store the vast quantity of images, definition of an intelligent image prioritisation scheme and the problem of predicting the next communication window with an orbiter, so that the best use of storage can be made to meet the mission targets without loss of any important information.
The technical challenges posed by individual tasks such as assessing scientific content or richness and predicting and uplink windows are now discussed. In addition there is the more general challenge of performing these tasks in a constrained real-time environment e.g. a cycle time of 4secs at a height of 100m at speed of 10m/sec assuming 70° Field of View (FOV) and image overlap of 70%. Significant algorithm optimisation is therefore required to meet these real-time demands.
1.2 Objectives The primary objective of our current work is to study, design and prototype an Imaging and Localisation Package (ILP) for a free-flying Martian balloon on a high-resolution imaging mission. The work is being carried out for ESA/ESTEC in order to investigate and demonstrate the viability of such a mission. An ILP package will allow optimal acquisition of images to reconstruct accurate models of the surface of the explored planet and accurate localisation of the balloon with respect to the Martian surface. The ILP , by means of aerobot mounted camera and computer vision techniques will:
acquire and store images at various resolutions construct and update a 3D model (DEM) of the surface constantly estimate the position (latitude, longitude and altitude) of the aerobot as well as its motion with respect to the surface and decide on the basis of the communications budget, of the morphology of the surface
Fig. 1. ILP System Overview
927
2.1 Image Acquisition and Richness Assessment
2.2 DEM Generation
The image acquisition process has been designed to reduce the computational overhead of the DEM generation process. An initial pre-processing algorithm determines aerobot displacement from successive images and uses this measure to control image acquisition. The aim of the control scheme is to ensure almost ideal overlap between successive images in order to reduce the time taken to generate the DEM and prevent redundant processing. DEMs require the image pairs to be similar but well contrasted. Therefore, with changing planetary conditions, camera exposure parameters may need to be controlled automatically.
A Digital Elevation Model (DEM) is a standard data structure for digital representation of a planetary surface. For each x/y coordinate within the represented area a height can be directly derived from the DEM. In this particular aerobot application it is used in the ILP to support the global localisation by comp arison with a global DEM and image richness assessment. By displaying received DEMs on the ground segment GUI, scientists can complement knowledge obtained from image mosaics on local geology. In the aerobot case, DEMs are built using images taken at different positions of the aerobot’s single camera system. For the photogrammetric determination of a distance to a certain scene point (without loss of generality in the coordinate system of the temporally first image) the relative orientation between the camera positions (pointing and displacement vector) must be known, the respective scene point must be visible in both images. Consequently the images involved need to overlap since only within the overlap area a DEM can be generated. Any missing information leads to ambiguities in the resulting DEM data, such as unknown scaling or rotation and displacement with respect to the global coordinate system. Scaling problems can be resolved by one independent distance measurement to a point identified in both involved images (e.g. using an altimeter to provide distance to a point whose direction is known in the camera coordinate system), or the knowledge about the displacement between the stereo images. The real-time constraints identified in 2 mean that DEM generation must be carried out as efficiently as possible. A number of techniques have been developed to speed up this process and reduce computational load including the provision of ideal image overlap through controlled acquisition thus removing the need to estimate displacement. Advance knowledge of displacement itself also shortens the DEM generation process. Integration into the global coordinate system is possible when local DEM landmarks are identified in a global DEM. DEM generation is also highly connected to relative localisation since tracking of landmarks in principle is the same process as the stereo matching needed for photogrammetric DEM reconstruction.
An Image Richness Index (IRI) is also needed to determine the priority that should be given to the acquired images to ensure that those of high interest are not lost should the amount of data that could be transmitted to the orbiter become restrictive. This is a non-trivial problem as Fig. 2 shows. Although Image A in the figure has many features, the most scientifically interesting feature (for a particular science mode) appears in B. So although the quantity of features in B is low it should be assigned a higher IRI. The system must therefore incorporate feature extraction and configurable classification algorithms to model the objectives of mission scientists.
Fig. 2: Illustration of the Image richness classification problem. Image A has multiple features of low scientific value in this mode while B has fewer features but they have higher scientific value indicating possible fluvial activity. This capability is vital in ensuring aerobot autonomy. Without it the mission will be of much less scientific value. Initial work has focused on the survey and selection of reference data sets and consultation with planetary scientists in order to determine the target feature types and richness rankings. This is essential to define classification models. Reference areas should contain a wide variety of interesting geological features including volcanics, channel material, basin material, mountainous terrain and plains material (West, 1991). Limited MGS Mars Orbiter Camera (MOC) image sets are available at resolutions of 1.5m-12/pixel showing target features. These images may be complemented with terrestrial analogues to enhance reference sets.
2.3 Localisation Global and Relative Autonomous localisation is another key requirement for a free-flying aerobot mission. It is required to provide a reference for the scientific image analysis and to enable prediction of forthcoming uplinks which in turn are required to manage on-board memory and image compression. However this is not trivial as Mars does not currently possess a Global Positioning (GPS) system of its own and systems such as rate gyros, accelerometers, flux gate, compasses and sun/star sensors are subject to error and drift and are therefore inadequate in themselves. It is possible to derive positional information when in
928
contact with an orbiter whose position is known, however such contacts will be infrequent and cannot be used as the sole means of localising the aerobot.
location which meets the feature/gradient relationship is at the bottom of the image. Through experimentation it was found that the largest influencing factor to obtain a match was the size of the local DEM. A local DEM that covered an area of 10x10, global resolution (or more) produced 95% correct matching and areas that covered 5x5 global resolution, produced over 70% correct positions.
Fig. 3: Relative localisation using landmark tracking An alternative scheme has been adopted for this work which will use a combination of computer vision based global and relative localisation in conjunction with orbiter contacts to provide continuous aerobot localisation. The proposed scheme uses landmark tracking shown in Fig. 3 which is a by-product of the DEM generation process to provide a relative location fix. First tests on a small-scale mockup have shown a repeatability within 1% of travelled distance for the relative localisation. These relative measurements must be corrected with an absolute reference. To make up for infrequent orbiter contacts a novel landmark comparison scheme (Shaw and Barnes 2003) will be used to provide an absolute position.
Fig. 5: shows the points which were selected for the gradient profile, the white lines show the actual gradients measured
The landmark comparison method consists of two individual elements; the first is the analysis of the terrain for geological features, providing a density profile and the second, a gradient profile of the surface. Fig. 4 shows the region selected for the feature density profile (area within the orange rectangle, the various colours represent different features), the feature density within this area is used during the search of the global DEM . Fig. 5 shows the points which were used in the gradient profile (the four corners); the gradient between these points is then used when searching the global DEM. An actual location for the local DEM within the global DEM can be generated by using the positional relationship of the feature and gradient profile areas (form the local DEM) as these have to be the same in the global DEM.
Fig. 6: Matched DEM locations indicated by overlapping orange rectangle (matching feature density profile) and white square (matching gradient profile). There are a number of possible sources of reference topographical information. Global Martian surface data obtained to date has been from orbiter’s, the best of which was gathered from two instruments on board the Mars Global Surveyor (MGS), the Mars Orbiter Laser Altimeter (MOLA) and the Mars Orbiter Camera (MOC). Absolute localisation can therefore be achieved by extracting naturally occurring features (peaks, ridges, channels etc.) from the topographical maps. By categorising the surface by its features, then by matching these features in a high resolution DEM generated by the aerobot, with the same features in the low resolution global map (e.g. MOLA) a position estimate is found. The obvious limitation of the scheme that it is better suited to areas which are reasonably feature rich and diverse such as the Southern Highlands. However this is constraint is fully compatible with scientific goals which seek to explore such geologically rich areas.
Fig. 4: shows the area selected for the feature density profile, which is within the orange rectangle.
2.4 Image Store and Management
The matching results from the profiles of Fig. 4 and Fig. 5 can be seen in Fig. 6 where the matched
Since the time between communications with an orbiter will be variable and doubtlessly long, and the 929
amount of imagery is dependent upon the profile of the wind velocity, assuming no area of the surface beneath the aerobot which is scientifically interesting is to be lost, image storage management is an important aspect of the work. The image richness index is to be used to determine the priority to be given to the image, and possibly the image’s most interesting areas.
using a simulator and a hardware mode (see Fig. 8) that encompasses a real RC balloon plus camera with RF interface. Both modes being interfaced to a demonstrator shell GUI and associated ILP software. The use of a simulator as one of the core demonstrator components affords a number of advantages over a hardware-only solution:
Because the amount of mass memory is unlikely to be sufficient to hold all images produced before they can be transmitted and then deleted, all unnecessary imagery must be removed. This means that overlap between final stored images should be kept to a minimum, while ensuring that no pixels are lost due to changes in yaw between consecutive images. Reconstruction of the continuous mosaic of images and DEMs must eventually be possible on the ground. This defines the minimum uncompressed imagery required to be stored. Compression provides a means to greatly reduce and control the actual amount of storage required. Progressive compression techniques (ECBOT wavelet JPEG 2000) are being used to compress the images and DEM data. Such a technique is resilient against drop out and will also allow for example, the entropy coding to be tailored to meet any data length against image richness profile required by controlling the scaling against signal to nois e, resolution, visual quality, or of regions of lesser interest etc. Since the window of communication will vary as much as the vagaries of the wind, continuous update of the capabilities of the up-link is required and the compression ratio is therefore adjusted by layered deletion of the compressed image to roughly meet the link capability, and the memory storage requirements.
Fig. 7: Demonstrator - software mode Simulators are able to re-construct environments and terrain to a very accurate level whilst still allowing the user to maintain control over the specific parameters. Terrain, weather, atmosphere and hardware devices, such as cameras, can be simulated in a realistic way and noise can be modelled to allow for random fluctuations in the environment or manufacturing tolerances for instruments. A simulator is well suited for modelling the flight of a balloon over terrain, experiments cannot always be carried out due to numerous factors and the use of simulation will overcome any of these potential problems.
2.5 Uplink Scheduling Communications links with surface or low-altitude planetary exploration units are usually constrained in terms of frequency, duration and uplink downlink transmission rates. In addition, real-time commanding is not possible given lengthy RTLT durations. Given that imaging is data intensive process, the communications link access needs to be optimised by employing intelligent use of available bandwidth and transmission opportunities. Thus the transmissions must be scheduled by predicting the timing and duration of the next up-link of data based on the aerobot’s trajectory, the orbit of the relay satellite and the characteristics of the two antenna.
The trajectory of the balloon is initially supplied to the simulator, with periodic updates being sent. During periods where the trajectory of the balloon is unknown, interpolation is used to provide an estimate of the current trajectory. This provides a simplistic, but sufficient method, for controlling and modelling the trajectory of the balloon. A simulator also affords the ability to evaluate algorithms for localisation and control that may be difficult to examine on real hardware in an Earth-based environment. A major factor in the deployment of planetary balloons is the effect the environment and atmospheric conditions have upon the vehicle. The simulator therefore models haze, clouds and also wind given its bearing on ILP performance
During the transmission phase, the link must be acquired, any commands received and confirmed by the aerobot, before its image data can be transferred up to the orbiter, and their correct receipt confirmed. Only then can any images be deleted to make space for the new collection of images. 3
Repeatability of experiments; Simulation of environments that cannot be re-constructed within the laboratory.
DEMONSTRATOR SYSTEM OVERVIEW
An ILP demonstrator system proposed requires two distinct modes. An all software mode (see Fig. 7)
930
Additionally, functionality is provided to perform measurements on the DEMs such as the co-ordinates of individual points and the distance between selected. CONCLUSION We are currently developing an Imaging and Localisation Package and hardware/software demonstrator for a Martian balloon. An overview of our proposed system has been presented. For localisation we have proposed a novel combination of image based relative and global localisation techniques. We propose to demonstrate our ILP using both real and simulated aerobot and thereby provide a comprehensive test and evaluation of the ILP functionality. It is hoped that this work will contribute to definition of a feasible robotic balloon mission to Mars within the current decade. ACKNOWLEDGEMENTS This work is funded under European Space Agency (ESA) Contract No.17400/03/NL/CH. The authors would like to thank Gianfranco Visentin, Head of the Automation and Robotics Section at ESTEC (ESA D/TOS) for his collaboration and technical guidance on this project.
Fig. 8: Demonstrator - hardware mode Despite the availability of the simulator there is a clear need to test the ILP software in a more realistic environment. To this end a model balloon complete with propulsion, downward looking camera, processing and wireless control and communication capability is a key requirement for our current work and a variety of configurations are being prototyped. In addition a realistic test facility is necessary and this is provided in the first instance by the ESTEC PTB with its mock Martian terrain and high ceiling. To complement the PTB and support system tests a second facility is being designed for installation at our main team HQ. Outputs from the richness system development are being used to guide this design to ensure that the terrain is representative of actual flight areas.
REFERENCES Balaram, J., R.E. Scheid, P.T. Salomon, (1996). OnBoard Perception System For Planetary Aerobot Balloon Navigation, 23rd AUVSI, Orlando FL. Barnes, D.P., P. Summers, A. Shaw, (2000). An Investigation into Aerobot Technologies for Planetary Exploration, 6th ESA Workshop on Advanced Space Technologies for Robotics and Automation, ASTRA 2000, pp. 3.6-5. Cutts, J. A., K.T. Neck, J.A. Jones, G. Rodriguez, J. Balaram, G.E. Powell, S.P. Synnott, (1985) Aerovehicles for Planetary Exploration, IEEE International Conference on Robotics & Automation. Neck, K.T., J. Balaram, M.K. Heun, S. Smith and T. Gamber (1997). Mars 2001 Aerobot/Balloon System Overview, http://lheawww.gsfc.nasa.gov/docs/balloon/MAB S_27sept1996/mabs_27sept96.html Shaw, A., P. Summers, E. Geneste. and D.P. Barnes, (2000). ALTAIR-1: A laboratory based aerobot for planetary exploration experiments., Proc. EUREL European Advanced Robotics Systems, Masteclass and Conference, Salford UK. Shaw, A.J. and D.P. Barnes, (2003). Landmark recognition for localisation and navigation of aerial vehicles. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas. West F.R., Dao Vallis – Hadriaca Patera, http://cmexwww.arc.nasa.gov/MarsTools/Mars_Cat/Part_3/ Sites/site080.html
The demonstrator shell communicates with both the ILP and balloon simulator in order to receive uplinked DEMs and images and to download camera models and balloon trajectory information. The shell will allow selection of the demonstration mode, hardware or software, provide an interface to allow selection and upload of camera, antenna and terrain models and enable the user to define the communication window function. The user is also able to start, stop and pause the simulation at will. Another function afforded by the demonstrator shell is the display of demonstrator software status. This consists of the data provided to the demonstrator shell by the balloon simulator and the data produced by the ILP itself. A graphical interface is used to display this data in a clear and concise form. The presentation of this ILP data incorporates the display of image mosaics. This requires the demonstrator shell to reconstruct the sequence and determine the placing of the individual images within these mosaics. It must also display the DEM data received by the demonstrator shell. This DEM data is presented in a format suitable for easy visualisation.
931