- Email: [email protected]
Benchmarking water turbidity effect on tracking algorithms *
Proceedings of the 20th World Congress Proceedings of the 20th World The International Federation of Congress Automatic Control Proceedings of the 20th World The International of Congress Automatic Control Toulouse, France,Federation July 9-14, 2017 Available online at www.sciencedirect.com The International of Automatic Control Toulouse, France,Federation July 9-14, 2017 Toulouse, France, July 9-14, 2017
ScienceDirect
IFAC PapersOnLine 50-1 (2017) 11191–11196
J. J. J.
Benchmarking water turbidity effect on Benchmarking water turbidity effect ⋆ Benchmarking water turbidity effect on on tracking algorithms ⋆ tracking algorithms ⋆ tracking algorithms ∗ ∗ ∗ ∗
P´ erez ∗∗ P´ erez ∗ P´ erez
J. J. J.
Sales ∗∗ A. Pe˜ nalver ∗ J. J. Fern´ andez ∗∗ D. Fornas ∗∗∗ Sales A. Pe˜ J. aSanz ndez∗∗ D. Fornas ∗ ∗nalver ∗ ∗ Fern´ ∗Garc´ ∗ J. J. C. ıa R. Mar´ ın P. J. Sales A. Pe˜ J. Fern´ aSanz ndez∗∗ D. Fornas ∗ ∗ P. J. C. Garc´ ıa ∗∗∗nalver R. Mar´ ınJ. J. J. C. Garc´ıa R. Mar´ın ∗ P. J. Sanz ∗ ∗ Computer Science and Engineering Department, ∗ ∗ Computer Science and Engineering Department, ∗ of Jaume-I, Castell´ University on,Engineering Spain. (e-mail: [email protected]) Computer Science and Department, University of Jaume-I, Castell´ on, Spain. (e-mail: [email protected]) University of Jaume-I, Castell´ on, Spain. (e-mail: [email protected]) Abstract: Field experiments in underwater robotics research require a big amount of resources Abstract: experiments underwater robotics research require a big amount of resources in order toField be able to test inthe system in sea conditions. Moreover, sea conditions are Abstract: Field experiments underwater robotics research require a big amount of resources in order tochanging be able to test in the systemto in sea conditions. Moreover, sea conditions are constantly making impossible reproduce specific situations. For these reasons, in order to be able to test the system in sea conditions. Moreover, sea conditions are constantly changing and making impossible to reproduce specific situations. For these testing, comparing evaluating different algorithms in similar conditions is anreasons, utopic constantly changing making impossible to reproduce specific situations. For these reasons, testing, comparing and evaluating different algorithms in similar conditions is an utopic situation. In order to deal with this, a framework that mixes real experiments and a simulated testing, comparing evaluating algorithms in real similar conditions an utopic situation. In order toand dealto with this, adifferent framework that mixes experiments andisaas simulated environment is proposed allow objective comparison of algorithms in an scenario close as situation. In order to dealto with this, a framework that mixes real experiments and aas simulated environment is proposed allow objective comparison of algorithms in an scenario close as possible to field experiments. This objective is possiblecomparison using real sensors in a controllable environment, environment is proposed to allow of algorithms in an scenario as close for as possible to field experiments. This is possible using real sensors in a controllable environment, for instance to a water tank, addingThis simulated hostile conditions difficult to reproduce in a controlled possible field experiments. is possible using real sensors in a controllable environment, for instance a water adding simulated hostile conditions to reproduce in a controlled environment suchtank, as water turbidity, composing a Hardwaredifficult In the Loop (HIL) framework. This instance a water tank, adding simulated hostile conditions to reproduce in a controlled environment such as water turbidity, composing asimulator, Hardwaredifficult In the Loop (HIL) framework. This framework is formed by UWSim, an underwater and a benchmarking module able to environment such as water turbidity, composing a Hardware In the Loop (HIL) framework. This framework is performance formed by UWSim, an underwater simulator, and a benchmarking module ableuse to measure the of external software. This setup is used in a search and recovery framework is formed by UWSim, an underwater simulator, and a benchmarking module able to measure the performance of external software. predicting This setup is used a searchturbidity and recovery use case to compare different tracking algorithms, effectin in them. measure the performance of external software. This setupthe is used inofa water search and recovery use case to compare different tracking algorithms, predicting the effect of water turbidity in them. The results allow to choose the best option without the need of dealing with field experiments. case to compare tracking predicting the of effect of water in them. The results allowdifferent to choose the bestalgorithms, option without the need dealing with turbidity field experiments. The results to choose the bestofoption without the Hosting need ofby dealing experiments. © 2017, IFACallow (International Federation Automatic Control) Elsevierwith Ltd. field All rights reserved. Keywords: Underwater simulator; Benchmarking; Underwater intervention; Robotics; Keywords: Dehazing. Underwater simulator; Benchmarking; Underwater intervention; Robotics; Keywords: Dehazing. Underwater simulator; Benchmarking; Underwater intervention; Robotics; Dehazing. 1. INTRODUCTION Regarding benchmarking in robotics, a big effort has been 1. INTRODUCTION Regarding a big effortEuropean has been made over benchmarking the last years.in Inrobotics, fact, some recent 1. INTRODUCTION Regarding benchmarking robotics, a big effortEuropean has been made overlike the FP7-BRICS last years.inIn(Best fact, Practice some recent projects, in Robotics), During the last 8-years period (i.e. 2009-2016) the IRS- made over the last years. In fact, some recent European projects, like to FP7-BRICS (Best Practice Robotics), During the last 8-years (i.e.active 2009-2016) theinIRSdevoted this specific context as in in Nowak et al. Lab research group has period been very working the were projects, like to FP7-BRICS (Best Practice in Robotics), During the last 8-years period (i.e. 2009-2016) the IRSwere devoted this specific context as in Nowak etconal. Lab research group has been very active working in the (2010). Moreover, following previous research in this underwater robotics manipulation field under three in differdevoted to this specificprevious contextresearch as in Nowak etconal. Lab researchrobotics group has been veryfield active working the were (2010). Moreover, following in this underwater manipulation under three differtext, such as DEXMART it research is clear in that: ent research robotics projects:manipulation RAUVI (Sanzfield et al., 2010), TRITON (2010). Moreover, following(2009), previous thisComconunderwater under three differtext, such as DEXMART (2009), it is clear that: Coment research RAUVI (Sanz et MERBOTS, al., 2010), TRITON results from different approaches andthat: assessComthe (Sanz et al.,projects: 2013a) and the ongoing funded paring text, such as DEXMART (2009), it is clear ent research projects: RAUVI (Sanz et MERBOTS, al., 2010), TRITON paring results from different approaches and assess the (Sanz et al., 2013a) and the ongoing funded quality of the research is extremely difficult in the robotics by the et Spanish Ministry, and FP7-TRIDENT (Sanzfunded et al., paring results from different approaches and assess the (Sanz al., 2013a) and the ongoing MERBOTS, quality of the research is extremely difficult in the robotics by the Spanish and FP7-TRIDENT (Sanz al., research field. Furthermore, trying to do it when the robot 2013b), funded Ministry, by the European Commission. All et these quality of the research is extremely difficult in the robotics by the Spanish Ministry, and FP7-TRIDENT (Sanz et al., research field.with Furthermore, trying to domore it when the robot 2013b), by coordinated the European Commission. All these is interacting the real world is even complicated. projects funded have been with several partners and research field.with Furthermore, trying to domore it when the robot 2013b), funded by coordinated the European Commission. All these is interacting the real world is even complicated. projects have been with several partners and with a high complexity in both, hardware and software is interacting with the real world is even more complicated. projects havecomplexity been coordinated with several and partners and Several definitions of benchmarks have been proposed, but with a high in both, hardware software definitions of stated benchmarks have (2004) been proposed, but components. Moreover, these projects wereand targeted to Several in this paper the one at Dillman will be used. with a high complexity in both, hardware software definitions of stated benchmarks have (2004) been proposed, but components. Moreover, these projects were intervention targeted to Several in this paper the one at Dillman will be used. common objectives, dealing with underwater In it, benchmarks are defined as numerical evaluation of components. Moreover, these projects were targeted to in this paper the one stated at Dillman (2004) will be used. common objectives, dealing with underwater intervention systems to be validated in sea conditions at the end. In it, benchmarks are defined as numerical evaluation of results being repeatability, independency and unambiguity common objectives, dealing with underwater intervention In it, benchmarks are defined as numerical evaluation of systems to be validated in sea conditions at the end. results being repeatability, independency and unambiguity main aspects of these metrics. systems to be validated in sea conditions end. that the As a consequence, all the partners need at to the be sure results being repeatability, independency and unambiguity main aspects of these metrics. As consequence, all theand partners to be will sure work that the theira part of the system their need algorithms mainto aspects of these metrics. In order simulate the experiments, the UWSim simulaAs a consequence, all theand partners need to be will sure work that the their part of the system their algorithms In order to simulate the experiments, the UWSim simulaproperly when the system is completely assembled and tor (Prats et al., 2012) and a benchmarking tool, which is their part of the system and their algorithms will work In order toetsimulate theand experiments, the UWSim simulaproperly whenthis the aim, system is completely assembled and tor (Prats al., 2012) a benchmarking tool, which is tested. With a simulator that allows the rehighly integrated with the simulator, were developed (see properly when the system is completely assembled and tor (Prats et al., 2012) and a benchmarking tool, which is tested. this aim,the a model simulator that allows the reintegrated with the simulator, were developed (see searchersWith to introduce of the whole system, as highly figure 1). Moreover, a methodology that allows researchers tested. With this aim, a simulator that allows the rehighly integrated with the simulator, were developed (see searchers to introduce the model of the whole system, as well as a realistic scenario for testing their algorithms, was 1).inMoreover, a methodology that allows researchers to work different conditions and increasing gradually the searchers to introduce thefor model of the system,was as figure figure 1).inMoreover, a methodology that allows researchers well as a realistic scenario testing theirwhole algorithms, to work different conditions and increasing gradually the considered to be an extremely important tool. In addition level of difficulty has been designed. This methodology also well as a realistic scenario for testing their algorithms, was to work in different conditions and increasing graduallyalso the considered to be an extremely important tool.can In help addition level of difficulty has been designed. This methodology to the simulator, benchmarking capabilities the helps to improve the scenarios for the benchmarking, thus considered to be an extremely important tool.can In help addition level of difficulty has been designed. This methodology also to the simulator, benchmarking capabilities the helps to improve the scenarios for the benchmarking, thus researchers to compare different algorithms and better unincreasingly a more realistic one. to the simulator, benchmarking capabilitiesand canbetter help unthe obtaining helps to improve the scenarios for the benchmarking, thus researchers to compare different algorithms derstand their limitations and robustness, making possible obtaining increasingly a more realistic one. researchers to compare different algorithms and better unobtaining increasingly a more realistic one. derstand their limitations and robustness, making possible One of the main drawbacks of autonomous underwater their improvement. derstand their limitations and robustness, making possible One of the main drawbacks of autonomous underwater their improvement. interventions is thedrawbacks need to interpret the hazardous enviOne of the main of autonomous underwater their improvement. interventions is the need to interpret the hazardous environment. For instance, being able to detect and recognize interventions is the need to interpret the hazardous environment. For instance, being able to detect and recognize ⋆ This work was partly supported by Spanish Ministry of Economy objects in For degraded images to able grasptoand manipulate them. ronment. instance, being detect and recognize ⋆ This work was partly supported by Spanish Ministry of Economy objects in degraded images to grasp and manipulate them. This is the reason why manytoworks been presented in and Competitiveness under grant DPI2014-57746-C3 (MERBOTS ⋆ objects in degraded images grasphave and manipulate them. This work was partly supported by Spanish Ministry(MERBOTS of Economy This is the reason why many works have been presented in and Competitiveness under grant DPI2014-57746-C3 the underwater image processing context, Raimondo and Project), by Universitat Jaume I grant PID2010-12 and PhD grants This is the reason why many works have been presented in and Competitiveness under grant DPI2014-57746-C3 (MERBOTS the underwater image processing context, Raimondo and Project), by Universitat Jaume I grant PID2010-12 and PhD grants PREDOC/2012/47 and PREDOC/2013/46, by Generalitat ValenSilvia (2010) offers a review of them. Although many works the underwater image processing context, Raimondo and Project), by Universitat I grant PID2010-12 and PhD Valengrants PREDOC/2012/47 and Jaume PREDOC/2013/46, by Generalitat Silvia (2010) offers a review of them. Although many works ciana PhD grant ACIF/2014/298 and PROMETEO/2016/066 grant. PREDOC/2012/47 and PREDOC/2013/46, by Generalitat ValenSilvia (2010) offers a review of them. Although many works ciana PhD grant ACIF/2014/298 and PROMETEO/2016/066 grant.
ciana PhD grant ACIF/2014/298 and PROMETEO/2016/066 grant. Copyright © 2017, 2017 IFAC 11683Hosting by Elsevier Ltd. All rights reserved. 2405-8963 © IFAC (International Federation of Automatic Control) Copyright © 2017 IFAC 11683 Peer review under responsibility of International Federation of Automatic Control. Copyright © 2017 IFAC 11683 10.1016/j.ifacol.2017.08.1243
Proceedings of the 20th IFAC World Congress 11192 J. Pérez et al. / IFAC PapersOnLine 50-1 (2017) 11191–11196 Toulouse, France, July 9-14, 2017
UWSim UWSim Scene Config Configuration File Update
Sensors
Fig. 1. UWSim underwater simulation: panel manipulation scenario. Girona 500 I-AUV in water pool conditions manipulating an underwater panel using the ARM5E light-weight robotic arm. try to correct the images restoring original colors such as Bryson et al. (2016), Torres-M´endez and Dudek (2005) or Roser et al. (2014) none, to the best of the authors knowledge, try to objectively measure the effect of water turbidity on the performance of vision algorithms. The aim of this paper is to present a research framework that allows to measure the effect of water turbidity in vision algorithms in simulated and real conditions. Furthermore, experiments comparing different tracking algorithms in increasing water turbidity conditions are shown to illustrate the performance drop due to the hostile environment. The next section describes the software used in this work, detailing the reasons why UWSim was chosen and describing the benchmarking module. Then, in Section 3 the experimental setup is explained while results are discussed in 4. Finally, some conclusions and future work possibilities are provided in Section 5. 2. FRAMEWORK FOR BENCHMARKING IN UNDERWATER INTERVENTION SYSTEMS Field experiments in underwater robotics are a challenging experience due to the amount of requirements and the difficulties concerning the environment. Prior to field experiments, a water tank or pool deep enough for the system is required to develop and set up the system with all the significant space and maintenance costs it implies. Once the system is ready for the final location, such as lakes or the sea, special logistics are needed to translate and operate the system. Finally, supervising and interacting with the robot is hardly possible, making the whole experimental validation very laborious. For these reasons, facilitate the development of Autonomous Underwater Vehicles using specialized simulators is a necessary step. Additionally, there are extra features that increase the importance of this simulator: the possibility to benchmark the system and mission before the robot is deployed into the water, and supervise the task in case of the lack of direct view of the system. Usually, experimental validation takes place in the sea where there are many changing parameters such as underwater currents, bad visibility, etc. that can be hardly modelled, anticipated or replicated. These differences are the main issue when comparing and replicating results from different studies, as it is almost impossible to reproduce
Algorithm Results
GroundTruth
Scene Updater Benchmark Configuration File
Evaluated Algorithm
Benchmarking Module
Measure Tracker Evaluation Activate Trigger
Evaluation Results
Fig. 2. Benchmarking module flow diagram: a benchmark configuration is loaded into the benchmark module and a scene is loaded into the simulator. Then, the benchmark module produces some results that can be logged for posterior analysis. Green: Evaluated software, Orange: Measures the same real conditions in a ever-changing environment. The use of an automated comparison system in simulation, and controlled laboratory environments, helps to establish an objective benchmarking methodology. In this work UWSim, the underwater simulator, is used to simulate the visual sea conditions and benchmark tracking algorithms to decide which to use in different situations. However, different simulators have been taken into consideration, but most of them remain obsolete or are used for very specific purposes. In Matsebe et al. (2008) and Craighead et al. (2007), a review of simulators for autonomous underwater vehicles can be found. A more recent review (Cook et al., 2014) takes into consideration several underwater simulators and concludes about UWSim that it is an “excellent fit for the simulation of underwater vehicles.”. Nevertheless, the majority of the reviewed simulators have not been designed as open source, which makes it difficult to adapt them to other purposes. Other simulators, such as ROVSim (Marine Simulation), Vortex (LABS) or DeepWorks (Fugro General Robotics Ltd.), have been designed to train ROV pilots automatically discarding them for the objective of this research. About benchmarking, a specific module for UWSim has been developed in (Perez et al., 2015) allowing to evaluate and compare software from different sources. This module uses Robot Operating System (ROS) interfaces, presented in Quigley et al. (2009), as middleware, reading the necessary information from external software in order to evaluate it while consulting ground truth information from UWSim. The main goals of this module are: be transparent to the user, do not require major modifications to the algorithm to be evaluated and be adaptable to all kind of underwater intervention tasks. The general workflow of the framework can be seen in figure 2. Benchmarks are defined in XML (eXtensible Markup Language) files. Each file will define which measures are going to be used and how they will be evaluated. This allows the creation of standard benchmarks defined in a document to evaluate different aspects of underwater robotic algorithms, being able to compare algorithms from different origins. Each of these benchmarks will be associated with one or more UWSim scene configuration files, being the results of the benchmark dependent on the predefined scene.
11684
Proceedings of the 20th IFAC World Congress Toulouse, France, July 9-14, 2017 J. Pérez et al. / IFAC PapersOnLine 50-1 (2017) 11191–11196
Two main aspects define a benchmark in UWSim: measures and scene updaters. Measures are the different things to measure in a benchmark, for instance position error or elapsed time. Scene updaters create a controlled environment change through the benchmark execution, allowing multiple tests to be performed, such as visibility or water current force. This feature allows the possibility to not only testing algorithms in a fixed condition setup, but also in a range of possible scenarios, and checking the performance of the software depending on the parameters variation. Measures and scene updaters can be controlled through triggers to start and stop evaluation depending on events such as reaching a position or elapsed time.
Recorded Experiment ESM Tracker
Raw Image Fogged image
UWSim
Fog Updater
Using the previously presented framework, several experiments to determine the effect of water turbidity in vision algorithms have already been conducted. The first of them is P´erez et al. (2014a) where different trackers are compared in an increasing simulated underwater fog situation. Different trackers from the ViSP library, a visual tracking and visual servoing library useful in computer vision described in ViSP (2010), were compared proving the ZeroMean Normalized Cross Correlation (ZNCC) configuration as the most resistant to visibility changes. In P´erez et al. (2014b), the same experiments are performed adding a controller in charge of keeping the vehicle stable with respect to the target. The goal is decide if trackers are also affected by the controller movements, concluding that a smooth control is helps the tracking task. However, these experiments were performed in a simulated setup, thus additional experimentation is needed to validate the results for real situations. For this reason, a 3D reconstruction benchmark was performed in Perez et al. (2015), evaluating stereo and laser projector 3D reconstruction varying the light conditions. In this case, the experiments were performed in a water tank. In order to acquire the ground truth, a thorough calibration was performed allowing to configure a simulated scene with high fidelity models of the real scene. The main drawback of the last experiment is light conditions needed to be configured manually, thus requiring human intervention, which is prone to errors and slows the whole process. Furthermore, the experimental setup only allowed to change visibility increasing or decreasing illumination, thus even automatizing the light conditions would not allow to benchmark other parameters. In this case, a hybrid approach is used allowing real input to the system while mixing simulated sources to explore new situations known as Hardware In The Loop (HIL). A schema of the experimental setup used can be seen in figure 3. The first step is to acquire images of the target to evaluate, in this case the black box mock-up was recorded in a water tank with almost perfect visibility
Tracker
Groundtruth Object on Cam
Object on Cam
Fog
3. EXPERIMENTAL SETUP The use case proposed is the search and recovery problem in the context of TRIDENT project. In this case, the robot goal is to find a black box mock-up and grasp it. The purpose of this work is to compare different trackers able to find and follow the black box under different visibility conditions caused by water turbidity.
11193
Tracker Tracker Error
Benchmarking Module
Fig. 3. Schema used in the Hardware In the Loop (HIL) visibility experiment: Orange: benchmark outputs, Green: evaluated software
Fig. 4. Image acquisition setup. Left: image camera in a cylinder moving in the water surface. Right: black box mockup from the camera output conditions. Although, online images can be used executing the benchmark in real time, recorded images have been used to assure repeatability between different experiments. This will be used for two purposes: acquire the ground truth of the ideal tracking, and serve as input to UWSim which will add simulated turbidity. Image acquisition was made in a water tank using a camera in a cylinder moving in the water surface facing a black box mockup as can be seen in figure 4. In order to acquire the ground truth, a regular ESM tracker (Malis, 2004) has been used under human supervision. As previous simulation experiments proved in P´erez et al. (2014a), in good visibility conditions any of the trackers was able to achieve good performance, so the output of the tracker with the raw input is enough to obtain a good ground truth to compare with. Simultaneously, the raw image is also an input to UWSim which adds simulated fog to the image creating a new fogged image suitable for the proposed tracking evaluation. The model used to add haze to the real image introduces a fog factor value. The simulated fog factor is a value ranging from 0 to infinity and defines the visibility in the water also depending on the distance to the object. Visibility is a value between 0 and 1 where 1 represents a perfect visibility of the object and 0 represents no visibility at all. The visibility depends therefore on the water fog factor and distance to the object, as it is represented by equation 1.
11685
Proceedings of the 20th IFAC World Congress 11194 J. Pérez et al. / IFAC PapersOnLine 50-1 (2017) 11191–11196 Toulouse, France, July 9-14, 2017
have been repeated. The conclusions are similar to the simulation results presented in P´erez et al. (2014a) in relative terms, the algorithms that performed better in simulation still perform better in the real setup. Nevertheless, there are substantial differences in the absolute values of the results.
Fig. 5. From left to right top to bottom increasing simulated fog factors in the real captured image. Visibility vs fog factor at fixed 80 centimeters distance 1
All the evaluated trackers are included in ViSP library. The trackers are classified according to two different registration criterion: Sum of Squared Differences (SSD) and Zero-mean Normalized Cross Correlation (ZNCC). Both registration criterions can be used in different ways: ZNCC can be used in Forward Additional (FA) and Inverse Compositional (IC), while SSD allows Forward Compositional (FC) and ESM modes too.
0.9 0.8 0.7 0.6 Visibility
In other words, the real setup is a more challenging environment causing the trackers to loose target with less fog level than in the simulated setup. However, the trackers relative performance is the same, the ones able to work with a higher amount of fog in simulation are also the best in the real setup. For this reason, the use of real images is interesting as it is a more challenging benchmark than the simulated experiments.
0.5 0.4 0.3 0.2 0.1 0 0
0.5
1 Fog factor
1.5
2
Fig. 6. Visibility value depending on fog factor for a fixed 80 centimeters distance. visibility = e−(f ogf actor∗distance)
2
(1)
This equation models the attenuation of the light in the image formation model. Although it is a very simplified model, more complex equations can be used to test the influence of different phenomena such as backscattering or vignetting. In this work a simple initial model has been used to show the validity of the presented framework.
Besides these tracker options, different warps have been evaluated. A warp function encodes the transformation between camera and target, so it restricts the possible positions of it. For instance, translation warp only allows two-axis translations while Scale Rotation Translation (SRT) warp describes a scale factor, a rotation and twoaxis translation. In the previous case of study (P´erez et al., 2014a), the target was not moving, therefore most restrictive warps had an advantage. In this case, the camera is moving over the target and translation warp may not be the best option. Five different warps were tested: • Translation: This is the most simple transformation available for template trackers. It only considers translation on two-axis (x-axis and y-axis). • SRT (Scale Rotation Translation): The SRT warp considers a scale factor, a rotation on z-axis and a 2D translation as in Translation warp. • Affine: This warping function preserves points, straight lines, and planes. • Homography: Estimates the eight parameters of the homography matrix H. • Homography SL3: This warp works similar to the previous one, but the homography parameters are estimated in the SL3 reference frame.
The result of this visibility parameter is showed in figure 5. As can be seen, due to the exponential relation as the fog factor increases it is really difficult to distinguish the black box. As in this experiment distance is almost invariant because movements are restricted to planar displacements and rotations, the only variant is the fog factor. In order to simulate it, the previous formula has been used to know the correct amount of fog color to be added in each moment of the experiment. The relationship for the distance in this experiment can be seen in the figure 6. Finally, the benchmarking module is the software that controls the amount of fog added in the image sending this information to the simulator. It also evaluates the selected tracker using the output produced with the fogged image and the output from the tracker with the raw image as ground truth. 4. RESULTS Using this configuration, the visibility experiments where different trackers were used while changing underwater fog
Results on figure 7 show the performance of the trackers using an affine warp. The figure shows the error of the centroid of the tracker with respect to fog factor, computed as a mean of the error on each image of the video. The fog factor is the simulated haze introduced in the real video. The results show non monotonic behaviour, already shown in simulation, probably caused by the trackers nature that do not provide a valid tracking result when they are not capable of locating the target. As happened in the previous simulation study, ZNCC algorithms achieved better results than SSD based methods. There are no important differences between both ZNCC similarity functions. On the other hand, SSD techniques
11686
Proceedings of the 20th IFAC World Congress Toulouse, France, July 9-14, 2017 J. Pérez et al. / IFAC PapersOnLine 50-1 (2017) 11191–11196
Tracking similarity function comparison using affine warp
Tracking similarity function comparison using SRT warp 100
SSD-FA SSD-FC SSD-IC ZNCC-FA 80 ZNCC-IC SSD-ESM
centroid error (pixels)
centroid error (pixels)
100
60
40
20
SSD-FA SSD-FC SSD-IC ZNCC-FA 80 ZNCC-IC SSD-ESM 60
40
20
0
0 0
0.5
1 Fog factor
1.5
2
0
Fig. 7. Comparison of different tracking similarity functions using affine warp in a real video with added simulated haze.
AFFINE HOMOGRAPHY HOMOGRAPHY SL3 SRT 80 TRANSLATION
60
40
0 0.5
1 Fog factor
1.5
1 Fog factor
1.5
2
Fig. 9. Comparison of different tracking similarity functions using SRT warp in a real video with added simulated haze.
This conclusion is confirmed comparing the previous results of the affine warp with the results of the SRT warp in figure 9. This figure shows the results of the different tracking similarity functions using the Scale Rotation Translation warp.
20
0
0.5
thus the camera could still see the target. In the case of bigger movements, such as depth changes or pitch and roll rotations, these kind of warps may not be able to represent the transformation between camera and target, but this use case considers the scenario where the vehicle tries to keep a stable position in order to manipulate an object. This situation repeats with different similarity functions just showing small changes in the exact amount of fog where the tracker lose the target. In conclusion, translation and SRT warps perform better in this kind of situation.
SSD-ForwardCompositional tracker using different warps 100
centroid error (pixels)
11195
2
Fig. 8. Comparison of different warps for tracking objects in a real video with added simulated haze. show big differences being IC the least resistant to visibility changes followed by ESM, FA and finally FC. When using different warps this trend remains the same with slight changes in the moment when the algorithm starts to lose the target. Thus, it is possible to state ZNCC registration criterion is clearly better in visibility change situations, because it was able to maintain the target using every warp during the experiments. On the other hand, SSD algorithms demonstrated good capabilities maintaining the target under small amount of simulated fog.
As can be seen, all the tested algorithms are able to keep the track of the object with a higher value of the fog factor parameter than when using the affine warp. Furthermore, the results for the different similarity functions are also robust, ZNCC based methods are capable of tracking the black box for the whole experiment while SSD are not. Taking this into account, the best configuration option in order to resist visibility changes is a ZNCC based similarity function together with the most restrictive warp. 5. CONCLUSIONS
Regarding warps, results can be seen in figure 8, which shows the evaluation for the SSD-FC algorithm. This graph shows it exists a big difference in the performance depending on the warp used. In this use case, most restrictive warps, such as translation and SRT, perform better because the camera perspective does not suffer big changes during the experiment.
As it has been demonstrated along the aforementioned works, the proposed platform including simulator and benchmarking module provides crucial results and capabilities to achieve the expected objectives when complexity is high and integration of several resources is required. Concerning the benefits of this platform, it is noticeable that some EU projects like PANDORA and MORPH have successfully used this tool in their work plan.
Although some movement and rotation was added to the camera, as a difference with the previous simulated experiments, it seems using a restrictive warp is still a better option. The movements added were slow, few centimetres or radians per second in XY and yaw axis,
Moreover, the platform has been extended to use it not only in simulated environments but real scenarios and Hardware In the Loop (HIL) approaches. This fact allows further validation of results before the final experiments, making them easier to perform.
11687
Proceedings of the 20th IFAC World Congress 11196 J. Pérez et al. / IFAC PapersOnLine 50-1 (2017) 11191–11196 Toulouse, France, July 9-14, 2017
A comparison of different visual trackers in varying water turbidity conditions has been performed showing significant differences in the compared methods. This experiment shows the validity of the proposed architecture, allowing to decide the best possible configuration before field experiments happen. In this case, the setup proved ZNCC tracker is a solid option as registration criterion, and translation or STR warps are the best option among the tested algorithms. However, further validation at sea should be carried out to confirm the final results. As conclusion, the benchmarking tool has demonstrated to be an excellent software for integration and experimental validation, for both, virtual and real scenarios. It allows the definition of ordered datasets from real experiments, and their further use for comparing specific scientific algorithms such as visual trackers, pattern recognition and 3D object reconstruction for manipulation, taking into account specific visibility and current conditions. Further work will focus on enhancing the tool to provide more datasets and use cases that enable measuring and comparison of specific robotics algorithms. REFERENCES Bryson, M., Johnson-Roberson, M., Pizarro, O., and Williams, S.B. (2016). True color correction of autonomous underwater vehicle imagery. Journal of Field Robotics, 33(6), 853–874. doi:10.1002/rob.21638. URL http://dx.doi.org/10.1002/rob.21638. Cook, D., Vardy, A., and Lewis, R. (2014). A survey of auv and robot simulators for multi-vehicle operations. In 2014 IEEE/OES Autonomous Underwater Vehicles (AUV), 1–8. IEEE. Craighead, J., Murphy, R., Burke, J., and Goldiez, B. (2007). A survey of commercial open source unmanned vehicle simulators. In Robotics and Automation, 2007 IEEE International Conference on, 852 –857. doi:10. 1109/ROBOT.2007.363092. DEXMART (2009). Specification of benchmarks. In Deliverable D6.1 from FP7-DEXMART Project (DEXterous and autonomous dual-arm/hand robotic manipulation with sMART sensory-motor skills: A bridge from natural to artificial cognition. URL http://www.dexmart.eu. Dillman, R. (2004). KA 1.10 Benchmarks for Robotics Research. Technical report, University of Karlsruhe. Fugro General Robotics Ltd. (????). Deepworks. Available online: http://www.fugrogrl.com/software/. LABS, C. (????). Vortex. URL http://www.cm-labs. com/energy-offshore/products/simulators. Malis, E. (2004). Improving vision-based control using efficient second-order minimization techniques. In Robotics and Automation, 2004. Proceedings. ICRA ’04. 2004 IEEE International Conference on, volume 2, 1843 – 1848 Vol.2. doi:10.1109/ROBOT.2004.1308092. Marine Simulation (????). Marine Simulation ROVsim. Available online: http://marinesimulation.com. Matsebe, O., Kumile, C., and Tlale, N. (2008). A review of virtual simulators for autonomous underwater vehicles (AUVs). NGCUV, Killaloe, Ireland. Nowak, W., Zakharov, A., Blumenthal, S., and Prassler, E. (2010). Benchmarks for mobile manipulation and robust obstacle avoidance and navigation. In Deliverable D3.1
from FP7-BRICS Project (Best Practice in Robotics). URL http://www.best-of-robotics.org/home. Perez, J., Sales, J., Penalver, A., Fornas, D., Fernandez, J.J., Garcia, J.C., Sanz, P.J., Marin, R., and Prats, M. (2015). Exploring 3-d reconstruction techniques: A benchmarking tool for underwater robotics. IEEE Robotics & Automation Magazine, 22(3), 85–95. Prats, M., P´erez, J., Fern´ andez, J., and Sanz, P. (2012). An open source tool for simulation and supervision of underwater intervention missions. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, 2577–2582. doi:10.1109/IROS.2012.6385788. P´erez, J., Sales, J., Mar´ın, R., and Sanz, P.J. (2014a). Web-based configuration tool for benchmarking of simulated intervention autonomous underwater vehicles. In Autonomous Robot Systems and Competitions (ICARSC), 2014 IEEE International Conference on, 279–284. Espinho, Portugal. doi:10.1109/ICARSC.2014. 6849799. URL http://dx.doi.org/10.1109/ICARSC. 2014.6849799. P´erez, J., Sales, J., Mar´ın, R., and Sanz, P. (2014b). Online tool for benchmarking of simulated intervention autonomous underwater vehicles: Evaluating position controllers in changing underwater currents. In 2014 Second International Conference on Artificial Intelligence, Modelling and Simulation (AIMS2014), 246–251. Madrid, Spain. Quigley, M., Conley, K., Gerkey, B.P., Faust, J., Foote, T., Leibs, J., Wheeler, R., and Ng, A.Y. (2009). ROS: an open-source Robot Operating System. In ICRA Workshop on Open Source Software. Raimondo, S. and Silvia, C. (2010). Underwater image processing: state of the art of restoration and image enhancement methods. EURASIP Journal on Advances in Signal Processing, 2010. Roser, M., Dunbabin, M., and Geiger, A. (2014). Simultaneous underwater visibility assessment, enhancement and improved stereo. In Robotics and Automation (ICRA), 2014 IEEE International Conference on, 3840– 3847. IEEE. Sanz, P.J., Pe˜ nalver, A., Sales, J., Fornas, D., Fern´ andez, J.J., Perez, J., and Bernab´e, J.A. (2013a). GRASPER: A multisensory based manipulation system for underwater operations. In 2013 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, Manchester, UK. Sanz, P.J., Prats, M., Ridao, P., Ribas, D., Oliver, G., and Orti, A. (2010). Recent progress in the RAUVI project. a reconfigurable autonomous underwater vehicle for intervention. In 52-th International Symphosium ELMAR-2010, 471–474. Zadar, Croatia. Sanz, P.J., Ridao, P., Oliver, G., Casalino, G., Petillot, Y., Silvestre, C., Melchiorri, C., and Turetta, A. (2013b). TRIDENT: An european project targeted to increase the autonomy levels for underwater intervention missions. In OCEANS’13 MTS/IEEE conference. San Diego, CA. Torres-M´endez, L.A. and Dudek, G. (2005). Color correction of underwater images for aquatic robot inspection. In Energy Minimization Methods in Computer Vision and Pattern Recognition, 60–73. Springer. ViSP (2010). Visual Servoing Platform. Available online: http://www.irisa.fr/lagadic/visp/visp.html.
11688
ScienceDirect
IFAC PapersOnLine 50-1 (2017) 11191–11196
J. J. J.
Benchmarking water turbidity effect on Benchmarking water turbidity effect ⋆ Benchmarking water turbidity effect on on tracking algorithms ⋆ tracking algorithms ⋆ tracking algorithms ∗ ∗ ∗ ∗
P´ erez ∗∗ P´ erez ∗ P´ erez
J. J. J.
Sales ∗∗ A. Pe˜ nalver ∗ J. J. Fern´ andez ∗∗ D. Fornas ∗∗∗ Sales A. Pe˜ J. aSanz ndez∗∗ D. Fornas ∗ ∗nalver ∗ ∗ Fern´ ∗Garc´ ∗ J. J. C. ıa R. Mar´ ın P. J. Sales A. Pe˜ J. Fern´ aSanz ndez∗∗ D. Fornas ∗ ∗ P. J. C. Garc´ ıa ∗∗∗nalver R. Mar´ ınJ. J. J. C. Garc´ıa R. Mar´ın ∗ P. J. Sanz ∗ ∗ Computer Science and Engineering Department, ∗ ∗ Computer Science and Engineering Department, ∗ of Jaume-I, Castell´ University on,Engineering Spain. (e-mail: [email protected]) Computer Science and Department, University of Jaume-I, Castell´ on, Spain. (e-mail: [email protected]) University of Jaume-I, Castell´ on, Spain. (e-mail: [email protected]) Abstract: Field experiments in underwater robotics research require a big amount of resources Abstract: experiments underwater robotics research require a big amount of resources in order toField be able to test inthe system in sea conditions. Moreover, sea conditions are Abstract: Field experiments underwater robotics research require a big amount of resources in order tochanging be able to test in the systemto in sea conditions. Moreover, sea conditions are constantly making impossible reproduce specific situations. For these reasons, in order to be able to test the system in sea conditions. Moreover, sea conditions are constantly changing and making impossible to reproduce specific situations. For these testing, comparing evaluating different algorithms in similar conditions is anreasons, utopic constantly changing making impossible to reproduce specific situations. For these reasons, testing, comparing and evaluating different algorithms in similar conditions is an utopic situation. In order to deal with this, a framework that mixes real experiments and a simulated testing, comparing evaluating algorithms in real similar conditions an utopic situation. In order toand dealto with this, adifferent framework that mixes experiments andisaas simulated environment is proposed allow objective comparison of algorithms in an scenario close as situation. In order to dealto with this, a framework that mixes real experiments and aas simulated environment is proposed allow objective comparison of algorithms in an scenario close as possible to field experiments. This objective is possiblecomparison using real sensors in a controllable environment, environment is proposed to allow of algorithms in an scenario as close for as possible to field experiments. This is possible using real sensors in a controllable environment, for instance to a water tank, addingThis simulated hostile conditions difficult to reproduce in a controlled possible field experiments. is possible using real sensors in a controllable environment, for instance a water adding simulated hostile conditions to reproduce in a controlled environment suchtank, as water turbidity, composing a Hardwaredifficult In the Loop (HIL) framework. This instance a water tank, adding simulated hostile conditions to reproduce in a controlled environment such as water turbidity, composing asimulator, Hardwaredifficult In the Loop (HIL) framework. This framework is formed by UWSim, an underwater and a benchmarking module able to environment such as water turbidity, composing a Hardware In the Loop (HIL) framework. This framework is performance formed by UWSim, an underwater simulator, and a benchmarking module ableuse to measure the of external software. This setup is used in a search and recovery framework is formed by UWSim, an underwater simulator, and a benchmarking module able to measure the performance of external software. predicting This setup is used a searchturbidity and recovery use case to compare different tracking algorithms, effectin in them. measure the performance of external software. This setupthe is used inofa water search and recovery use case to compare different tracking algorithms, predicting the effect of water turbidity in them. The results allow to choose the best option without the need of dealing with field experiments. case to compare tracking predicting the of effect of water in them. The results allowdifferent to choose the bestalgorithms, option without the need dealing with turbidity field experiments. The results to choose the bestofoption without the Hosting need ofby dealing experiments. © 2017, IFACallow (International Federation Automatic Control) Elsevierwith Ltd. field All rights reserved. Keywords: Underwater simulator; Benchmarking; Underwater intervention; Robotics; Keywords: Dehazing. Underwater simulator; Benchmarking; Underwater intervention; Robotics; Keywords: Dehazing. Underwater simulator; Benchmarking; Underwater intervention; Robotics; Dehazing. 1. INTRODUCTION Regarding benchmarking in robotics, a big effort has been 1. INTRODUCTION Regarding a big effortEuropean has been made over benchmarking the last years.in Inrobotics, fact, some recent 1. INTRODUCTION Regarding benchmarking robotics, a big effortEuropean has been made overlike the FP7-BRICS last years.inIn(Best fact, Practice some recent projects, in Robotics), During the last 8-years period (i.e. 2009-2016) the IRS- made over the last years. In fact, some recent European projects, like to FP7-BRICS (Best Practice Robotics), During the last 8-years (i.e.active 2009-2016) theinIRSdevoted this specific context as in in Nowak et al. Lab research group has period been very working the were projects, like to FP7-BRICS (Best Practice in Robotics), During the last 8-years period (i.e. 2009-2016) the IRSwere devoted this specific context as in Nowak etconal. Lab research group has been very active working in the (2010). Moreover, following previous research in this underwater robotics manipulation field under three in differdevoted to this specificprevious contextresearch as in Nowak etconal. Lab researchrobotics group has been veryfield active working the were (2010). Moreover, following in this underwater manipulation under three differtext, such as DEXMART it research is clear in that: ent research robotics projects:manipulation RAUVI (Sanzfield et al., 2010), TRITON (2010). Moreover, following(2009), previous thisComconunderwater under three differtext, such as DEXMART (2009), it is clear that: Coment research RAUVI (Sanz et MERBOTS, al., 2010), TRITON results from different approaches andthat: assessComthe (Sanz et al.,projects: 2013a) and the ongoing funded paring text, such as DEXMART (2009), it is clear ent research projects: RAUVI (Sanz et MERBOTS, al., 2010), TRITON paring results from different approaches and assess the (Sanz et al., 2013a) and the ongoing funded quality of the research is extremely difficult in the robotics by the et Spanish Ministry, and FP7-TRIDENT (Sanzfunded et al., paring results from different approaches and assess the (Sanz al., 2013a) and the ongoing MERBOTS, quality of the research is extremely difficult in the robotics by the Spanish and FP7-TRIDENT (Sanz al., research field. Furthermore, trying to do it when the robot 2013b), funded Ministry, by the European Commission. All et these quality of the research is extremely difficult in the robotics by the Spanish Ministry, and FP7-TRIDENT (Sanz et al., research field.with Furthermore, trying to domore it when the robot 2013b), by coordinated the European Commission. All these is interacting the real world is even complicated. projects funded have been with several partners and research field.with Furthermore, trying to domore it when the robot 2013b), funded by coordinated the European Commission. All these is interacting the real world is even complicated. projects have been with several partners and with a high complexity in both, hardware and software is interacting with the real world is even more complicated. projects havecomplexity been coordinated with several and partners and Several definitions of benchmarks have been proposed, but with a high in both, hardware software definitions of stated benchmarks have (2004) been proposed, but components. Moreover, these projects wereand targeted to Several in this paper the one at Dillman will be used. with a high complexity in both, hardware software definitions of stated benchmarks have (2004) been proposed, but components. Moreover, these projects were intervention targeted to Several in this paper the one at Dillman will be used. common objectives, dealing with underwater In it, benchmarks are defined as numerical evaluation of components. Moreover, these projects were targeted to in this paper the one stated at Dillman (2004) will be used. common objectives, dealing with underwater intervention systems to be validated in sea conditions at the end. In it, benchmarks are defined as numerical evaluation of results being repeatability, independency and unambiguity common objectives, dealing with underwater intervention In it, benchmarks are defined as numerical evaluation of systems to be validated in sea conditions at the end. results being repeatability, independency and unambiguity main aspects of these metrics. systems to be validated in sea conditions end. that the As a consequence, all the partners need at to the be sure results being repeatability, independency and unambiguity main aspects of these metrics. As consequence, all theand partners to be will sure work that the theira part of the system their need algorithms mainto aspects of these metrics. In order simulate the experiments, the UWSim simulaAs a consequence, all theand partners need to be will sure work that the their part of the system their algorithms In order to simulate the experiments, the UWSim simulaproperly when the system is completely assembled and tor (Prats et al., 2012) and a benchmarking tool, which is their part of the system and their algorithms will work In order toetsimulate theand experiments, the UWSim simulaproperly whenthis the aim, system is completely assembled and tor (Prats al., 2012) a benchmarking tool, which is tested. With a simulator that allows the rehighly integrated with the simulator, were developed (see properly when the system is completely assembled and tor (Prats et al., 2012) and a benchmarking tool, which is tested. this aim,the a model simulator that allows the reintegrated with the simulator, were developed (see searchersWith to introduce of the whole system, as highly figure 1). Moreover, a methodology that allows researchers tested. With this aim, a simulator that allows the rehighly integrated with the simulator, were developed (see searchers to introduce the model of the whole system, as well as a realistic scenario for testing their algorithms, was 1).inMoreover, a methodology that allows researchers to work different conditions and increasing gradually the searchers to introduce thefor model of the system,was as figure figure 1).inMoreover, a methodology that allows researchers well as a realistic scenario testing theirwhole algorithms, to work different conditions and increasing gradually the considered to be an extremely important tool. In addition level of difficulty has been designed. This methodology also well as a realistic scenario for testing their algorithms, was to work in different conditions and increasing graduallyalso the considered to be an extremely important tool.can In help addition level of difficulty has been designed. This methodology to the simulator, benchmarking capabilities the helps to improve the scenarios for the benchmarking, thus considered to be an extremely important tool.can In help addition level of difficulty has been designed. This methodology also to the simulator, benchmarking capabilities the helps to improve the scenarios for the benchmarking, thus researchers to compare different algorithms and better unincreasingly a more realistic one. to the simulator, benchmarking capabilitiesand canbetter help unthe obtaining helps to improve the scenarios for the benchmarking, thus researchers to compare different algorithms derstand their limitations and robustness, making possible obtaining increasingly a more realistic one. researchers to compare different algorithms and better unobtaining increasingly a more realistic one. derstand their limitations and robustness, making possible One of the main drawbacks of autonomous underwater their improvement. derstand their limitations and robustness, making possible One of the main drawbacks of autonomous underwater their improvement. interventions is thedrawbacks need to interpret the hazardous enviOne of the main of autonomous underwater their improvement. interventions is the need to interpret the hazardous environment. For instance, being able to detect and recognize interventions is the need to interpret the hazardous environment. For instance, being able to detect and recognize ⋆ This work was partly supported by Spanish Ministry of Economy objects in For degraded images to able grasptoand manipulate them. ronment. instance, being detect and recognize ⋆ This work was partly supported by Spanish Ministry of Economy objects in degraded images to grasp and manipulate them. This is the reason why manytoworks been presented in and Competitiveness under grant DPI2014-57746-C3 (MERBOTS ⋆ objects in degraded images grasphave and manipulate them. This work was partly supported by Spanish Ministry(MERBOTS of Economy This is the reason why many works have been presented in and Competitiveness under grant DPI2014-57746-C3 the underwater image processing context, Raimondo and Project), by Universitat Jaume I grant PID2010-12 and PhD grants This is the reason why many works have been presented in and Competitiveness under grant DPI2014-57746-C3 (MERBOTS the underwater image processing context, Raimondo and Project), by Universitat Jaume I grant PID2010-12 and PhD grants PREDOC/2012/47 and PREDOC/2013/46, by Generalitat ValenSilvia (2010) offers a review of them. Although many works the underwater image processing context, Raimondo and Project), by Universitat I grant PID2010-12 and PhD Valengrants PREDOC/2012/47 and Jaume PREDOC/2013/46, by Generalitat Silvia (2010) offers a review of them. Although many works ciana PhD grant ACIF/2014/298 and PROMETEO/2016/066 grant. PREDOC/2012/47 and PREDOC/2013/46, by Generalitat ValenSilvia (2010) offers a review of them. Although many works ciana PhD grant ACIF/2014/298 and PROMETEO/2016/066 grant.
ciana PhD grant ACIF/2014/298 and PROMETEO/2016/066 grant. Copyright © 2017, 2017 IFAC 11683Hosting by Elsevier Ltd. All rights reserved. 2405-8963 © IFAC (International Federation of Automatic Control) Copyright © 2017 IFAC 11683 Peer review under responsibility of International Federation of Automatic Control. Copyright © 2017 IFAC 11683 10.1016/j.ifacol.2017.08.1243
Proceedings of the 20th IFAC World Congress 11192 J. Pérez et al. / IFAC PapersOnLine 50-1 (2017) 11191–11196 Toulouse, France, July 9-14, 2017
UWSim UWSim Scene Config Configuration File Update
Sensors
Fig. 1. UWSim underwater simulation: panel manipulation scenario. Girona 500 I-AUV in water pool conditions manipulating an underwater panel using the ARM5E light-weight robotic arm. try to correct the images restoring original colors such as Bryson et al. (2016), Torres-M´endez and Dudek (2005) or Roser et al. (2014) none, to the best of the authors knowledge, try to objectively measure the effect of water turbidity on the performance of vision algorithms. The aim of this paper is to present a research framework that allows to measure the effect of water turbidity in vision algorithms in simulated and real conditions. Furthermore, experiments comparing different tracking algorithms in increasing water turbidity conditions are shown to illustrate the performance drop due to the hostile environment. The next section describes the software used in this work, detailing the reasons why UWSim was chosen and describing the benchmarking module. Then, in Section 3 the experimental setup is explained while results are discussed in 4. Finally, some conclusions and future work possibilities are provided in Section 5. 2. FRAMEWORK FOR BENCHMARKING IN UNDERWATER INTERVENTION SYSTEMS Field experiments in underwater robotics are a challenging experience due to the amount of requirements and the difficulties concerning the environment. Prior to field experiments, a water tank or pool deep enough for the system is required to develop and set up the system with all the significant space and maintenance costs it implies. Once the system is ready for the final location, such as lakes or the sea, special logistics are needed to translate and operate the system. Finally, supervising and interacting with the robot is hardly possible, making the whole experimental validation very laborious. For these reasons, facilitate the development of Autonomous Underwater Vehicles using specialized simulators is a necessary step. Additionally, there are extra features that increase the importance of this simulator: the possibility to benchmark the system and mission before the robot is deployed into the water, and supervise the task in case of the lack of direct view of the system. Usually, experimental validation takes place in the sea where there are many changing parameters such as underwater currents, bad visibility, etc. that can be hardly modelled, anticipated or replicated. These differences are the main issue when comparing and replicating results from different studies, as it is almost impossible to reproduce
Algorithm Results
GroundTruth
Scene Updater Benchmark Configuration File
Evaluated Algorithm
Benchmarking Module
Measure Tracker Evaluation Activate Trigger
Evaluation Results
Fig. 2. Benchmarking module flow diagram: a benchmark configuration is loaded into the benchmark module and a scene is loaded into the simulator. Then, the benchmark module produces some results that can be logged for posterior analysis. Green: Evaluated software, Orange: Measures the same real conditions in a ever-changing environment. The use of an automated comparison system in simulation, and controlled laboratory environments, helps to establish an objective benchmarking methodology. In this work UWSim, the underwater simulator, is used to simulate the visual sea conditions and benchmark tracking algorithms to decide which to use in different situations. However, different simulators have been taken into consideration, but most of them remain obsolete or are used for very specific purposes. In Matsebe et al. (2008) and Craighead et al. (2007), a review of simulators for autonomous underwater vehicles can be found. A more recent review (Cook et al., 2014) takes into consideration several underwater simulators and concludes about UWSim that it is an “excellent fit for the simulation of underwater vehicles.”. Nevertheless, the majority of the reviewed simulators have not been designed as open source, which makes it difficult to adapt them to other purposes. Other simulators, such as ROVSim (Marine Simulation), Vortex (LABS) or DeepWorks (Fugro General Robotics Ltd.), have been designed to train ROV pilots automatically discarding them for the objective of this research. About benchmarking, a specific module for UWSim has been developed in (Perez et al., 2015) allowing to evaluate and compare software from different sources. This module uses Robot Operating System (ROS) interfaces, presented in Quigley et al. (2009), as middleware, reading the necessary information from external software in order to evaluate it while consulting ground truth information from UWSim. The main goals of this module are: be transparent to the user, do not require major modifications to the algorithm to be evaluated and be adaptable to all kind of underwater intervention tasks. The general workflow of the framework can be seen in figure 2. Benchmarks are defined in XML (eXtensible Markup Language) files. Each file will define which measures are going to be used and how they will be evaluated. This allows the creation of standard benchmarks defined in a document to evaluate different aspects of underwater robotic algorithms, being able to compare algorithms from different origins. Each of these benchmarks will be associated with one or more UWSim scene configuration files, being the results of the benchmark dependent on the predefined scene.
11684
Proceedings of the 20th IFAC World Congress Toulouse, France, July 9-14, 2017 J. Pérez et al. / IFAC PapersOnLine 50-1 (2017) 11191–11196
Two main aspects define a benchmark in UWSim: measures and scene updaters. Measures are the different things to measure in a benchmark, for instance position error or elapsed time. Scene updaters create a controlled environment change through the benchmark execution, allowing multiple tests to be performed, such as visibility or water current force. This feature allows the possibility to not only testing algorithms in a fixed condition setup, but also in a range of possible scenarios, and checking the performance of the software depending on the parameters variation. Measures and scene updaters can be controlled through triggers to start and stop evaluation depending on events such as reaching a position or elapsed time.
Recorded Experiment ESM Tracker
Raw Image Fogged image
UWSim
Fog Updater
Using the previously presented framework, several experiments to determine the effect of water turbidity in vision algorithms have already been conducted. The first of them is P´erez et al. (2014a) where different trackers are compared in an increasing simulated underwater fog situation. Different trackers from the ViSP library, a visual tracking and visual servoing library useful in computer vision described in ViSP (2010), were compared proving the ZeroMean Normalized Cross Correlation (ZNCC) configuration as the most resistant to visibility changes. In P´erez et al. (2014b), the same experiments are performed adding a controller in charge of keeping the vehicle stable with respect to the target. The goal is decide if trackers are also affected by the controller movements, concluding that a smooth control is helps the tracking task. However, these experiments were performed in a simulated setup, thus additional experimentation is needed to validate the results for real situations. For this reason, a 3D reconstruction benchmark was performed in Perez et al. (2015), evaluating stereo and laser projector 3D reconstruction varying the light conditions. In this case, the experiments were performed in a water tank. In order to acquire the ground truth, a thorough calibration was performed allowing to configure a simulated scene with high fidelity models of the real scene. The main drawback of the last experiment is light conditions needed to be configured manually, thus requiring human intervention, which is prone to errors and slows the whole process. Furthermore, the experimental setup only allowed to change visibility increasing or decreasing illumination, thus even automatizing the light conditions would not allow to benchmark other parameters. In this case, a hybrid approach is used allowing real input to the system while mixing simulated sources to explore new situations known as Hardware In The Loop (HIL). A schema of the experimental setup used can be seen in figure 3. The first step is to acquire images of the target to evaluate, in this case the black box mock-up was recorded in a water tank with almost perfect visibility
Tracker
Groundtruth Object on Cam
Object on Cam
Fog
3. EXPERIMENTAL SETUP The use case proposed is the search and recovery problem in the context of TRIDENT project. In this case, the robot goal is to find a black box mock-up and grasp it. The purpose of this work is to compare different trackers able to find and follow the black box under different visibility conditions caused by water turbidity.
11193
Tracker Tracker Error
Benchmarking Module
Fig. 3. Schema used in the Hardware In the Loop (HIL) visibility experiment: Orange: benchmark outputs, Green: evaluated software
Fig. 4. Image acquisition setup. Left: image camera in a cylinder moving in the water surface. Right: black box mockup from the camera output conditions. Although, online images can be used executing the benchmark in real time, recorded images have been used to assure repeatability between different experiments. This will be used for two purposes: acquire the ground truth of the ideal tracking, and serve as input to UWSim which will add simulated turbidity. Image acquisition was made in a water tank using a camera in a cylinder moving in the water surface facing a black box mockup as can be seen in figure 4. In order to acquire the ground truth, a regular ESM tracker (Malis, 2004) has been used under human supervision. As previous simulation experiments proved in P´erez et al. (2014a), in good visibility conditions any of the trackers was able to achieve good performance, so the output of the tracker with the raw input is enough to obtain a good ground truth to compare with. Simultaneously, the raw image is also an input to UWSim which adds simulated fog to the image creating a new fogged image suitable for the proposed tracking evaluation. The model used to add haze to the real image introduces a fog factor value. The simulated fog factor is a value ranging from 0 to infinity and defines the visibility in the water also depending on the distance to the object. Visibility is a value between 0 and 1 where 1 represents a perfect visibility of the object and 0 represents no visibility at all. The visibility depends therefore on the water fog factor and distance to the object, as it is represented by equation 1.
11685
Proceedings of the 20th IFAC World Congress 11194 J. Pérez et al. / IFAC PapersOnLine 50-1 (2017) 11191–11196 Toulouse, France, July 9-14, 2017
have been repeated. The conclusions are similar to the simulation results presented in P´erez et al. (2014a) in relative terms, the algorithms that performed better in simulation still perform better in the real setup. Nevertheless, there are substantial differences in the absolute values of the results.
Fig. 5. From left to right top to bottom increasing simulated fog factors in the real captured image. Visibility vs fog factor at fixed 80 centimeters distance 1
All the evaluated trackers are included in ViSP library. The trackers are classified according to two different registration criterion: Sum of Squared Differences (SSD) and Zero-mean Normalized Cross Correlation (ZNCC). Both registration criterions can be used in different ways: ZNCC can be used in Forward Additional (FA) and Inverse Compositional (IC), while SSD allows Forward Compositional (FC) and ESM modes too.
0.9 0.8 0.7 0.6 Visibility
In other words, the real setup is a more challenging environment causing the trackers to loose target with less fog level than in the simulated setup. However, the trackers relative performance is the same, the ones able to work with a higher amount of fog in simulation are also the best in the real setup. For this reason, the use of real images is interesting as it is a more challenging benchmark than the simulated experiments.
0.5 0.4 0.3 0.2 0.1 0 0
0.5
1 Fog factor
1.5
2
Fig. 6. Visibility value depending on fog factor for a fixed 80 centimeters distance. visibility = e−(f ogf actor∗distance)
2
(1)
This equation models the attenuation of the light in the image formation model. Although it is a very simplified model, more complex equations can be used to test the influence of different phenomena such as backscattering or vignetting. In this work a simple initial model has been used to show the validity of the presented framework.
Besides these tracker options, different warps have been evaluated. A warp function encodes the transformation between camera and target, so it restricts the possible positions of it. For instance, translation warp only allows two-axis translations while Scale Rotation Translation (SRT) warp describes a scale factor, a rotation and twoaxis translation. In the previous case of study (P´erez et al., 2014a), the target was not moving, therefore most restrictive warps had an advantage. In this case, the camera is moving over the target and translation warp may not be the best option. Five different warps were tested: • Translation: This is the most simple transformation available for template trackers. It only considers translation on two-axis (x-axis and y-axis). • SRT (Scale Rotation Translation): The SRT warp considers a scale factor, a rotation on z-axis and a 2D translation as in Translation warp. • Affine: This warping function preserves points, straight lines, and planes. • Homography: Estimates the eight parameters of the homography matrix H. • Homography SL3: This warp works similar to the previous one, but the homography parameters are estimated in the SL3 reference frame.
The result of this visibility parameter is showed in figure 5. As can be seen, due to the exponential relation as the fog factor increases it is really difficult to distinguish the black box. As in this experiment distance is almost invariant because movements are restricted to planar displacements and rotations, the only variant is the fog factor. In order to simulate it, the previous formula has been used to know the correct amount of fog color to be added in each moment of the experiment. The relationship for the distance in this experiment can be seen in the figure 6. Finally, the benchmarking module is the software that controls the amount of fog added in the image sending this information to the simulator. It also evaluates the selected tracker using the output produced with the fogged image and the output from the tracker with the raw image as ground truth. 4. RESULTS Using this configuration, the visibility experiments where different trackers were used while changing underwater fog
Results on figure 7 show the performance of the trackers using an affine warp. The figure shows the error of the centroid of the tracker with respect to fog factor, computed as a mean of the error on each image of the video. The fog factor is the simulated haze introduced in the real video. The results show non monotonic behaviour, already shown in simulation, probably caused by the trackers nature that do not provide a valid tracking result when they are not capable of locating the target. As happened in the previous simulation study, ZNCC algorithms achieved better results than SSD based methods. There are no important differences between both ZNCC similarity functions. On the other hand, SSD techniques
11686
Proceedings of the 20th IFAC World Congress Toulouse, France, July 9-14, 2017 J. Pérez et al. / IFAC PapersOnLine 50-1 (2017) 11191–11196
Tracking similarity function comparison using affine warp
Tracking similarity function comparison using SRT warp 100
SSD-FA SSD-FC SSD-IC ZNCC-FA 80 ZNCC-IC SSD-ESM
centroid error (pixels)
centroid error (pixels)
100
60
40
20
SSD-FA SSD-FC SSD-IC ZNCC-FA 80 ZNCC-IC SSD-ESM 60
40
20
0
0 0
0.5
1 Fog factor
1.5
2
0
Fig. 7. Comparison of different tracking similarity functions using affine warp in a real video with added simulated haze.
AFFINE HOMOGRAPHY HOMOGRAPHY SL3 SRT 80 TRANSLATION
60
40
0 0.5
1 Fog factor
1.5
1 Fog factor
1.5
2
Fig. 9. Comparison of different tracking similarity functions using SRT warp in a real video with added simulated haze.
This conclusion is confirmed comparing the previous results of the affine warp with the results of the SRT warp in figure 9. This figure shows the results of the different tracking similarity functions using the Scale Rotation Translation warp.
20
0
0.5
thus the camera could still see the target. In the case of bigger movements, such as depth changes or pitch and roll rotations, these kind of warps may not be able to represent the transformation between camera and target, but this use case considers the scenario where the vehicle tries to keep a stable position in order to manipulate an object. This situation repeats with different similarity functions just showing small changes in the exact amount of fog where the tracker lose the target. In conclusion, translation and SRT warps perform better in this kind of situation.
SSD-ForwardCompositional tracker using different warps 100
centroid error (pixels)
11195
2
Fig. 8. Comparison of different warps for tracking objects in a real video with added simulated haze. show big differences being IC the least resistant to visibility changes followed by ESM, FA and finally FC. When using different warps this trend remains the same with slight changes in the moment when the algorithm starts to lose the target. Thus, it is possible to state ZNCC registration criterion is clearly better in visibility change situations, because it was able to maintain the target using every warp during the experiments. On the other hand, SSD algorithms demonstrated good capabilities maintaining the target under small amount of simulated fog.
As can be seen, all the tested algorithms are able to keep the track of the object with a higher value of the fog factor parameter than when using the affine warp. Furthermore, the results for the different similarity functions are also robust, ZNCC based methods are capable of tracking the black box for the whole experiment while SSD are not. Taking this into account, the best configuration option in order to resist visibility changes is a ZNCC based similarity function together with the most restrictive warp. 5. CONCLUSIONS
Regarding warps, results can be seen in figure 8, which shows the evaluation for the SSD-FC algorithm. This graph shows it exists a big difference in the performance depending on the warp used. In this use case, most restrictive warps, such as translation and SRT, perform better because the camera perspective does not suffer big changes during the experiment.
As it has been demonstrated along the aforementioned works, the proposed platform including simulator and benchmarking module provides crucial results and capabilities to achieve the expected objectives when complexity is high and integration of several resources is required. Concerning the benefits of this platform, it is noticeable that some EU projects like PANDORA and MORPH have successfully used this tool in their work plan.
Although some movement and rotation was added to the camera, as a difference with the previous simulated experiments, it seems using a restrictive warp is still a better option. The movements added were slow, few centimetres or radians per second in XY and yaw axis,
Moreover, the platform has been extended to use it not only in simulated environments but real scenarios and Hardware In the Loop (HIL) approaches. This fact allows further validation of results before the final experiments, making them easier to perform.
11687
Proceedings of the 20th IFAC World Congress 11196 J. Pérez et al. / IFAC PapersOnLine 50-1 (2017) 11191–11196 Toulouse, France, July 9-14, 2017
A comparison of different visual trackers in varying water turbidity conditions has been performed showing significant differences in the compared methods. This experiment shows the validity of the proposed architecture, allowing to decide the best possible configuration before field experiments happen. In this case, the setup proved ZNCC tracker is a solid option as registration criterion, and translation or STR warps are the best option among the tested algorithms. However, further validation at sea should be carried out to confirm the final results. As conclusion, the benchmarking tool has demonstrated to be an excellent software for integration and experimental validation, for both, virtual and real scenarios. It allows the definition of ordered datasets from real experiments, and their further use for comparing specific scientific algorithms such as visual trackers, pattern recognition and 3D object reconstruction for manipulation, taking into account specific visibility and current conditions. Further work will focus on enhancing the tool to provide more datasets and use cases that enable measuring and comparison of specific robotics algorithms. REFERENCES Bryson, M., Johnson-Roberson, M., Pizarro, O., and Williams, S.B. (2016). True color correction of autonomous underwater vehicle imagery. Journal of Field Robotics, 33(6), 853–874. doi:10.1002/rob.21638. URL http://dx.doi.org/10.1002/rob.21638. Cook, D., Vardy, A., and Lewis, R. (2014). A survey of auv and robot simulators for multi-vehicle operations. In 2014 IEEE/OES Autonomous Underwater Vehicles (AUV), 1–8. IEEE. Craighead, J., Murphy, R., Burke, J., and Goldiez, B. (2007). A survey of commercial open source unmanned vehicle simulators. In Robotics and Automation, 2007 IEEE International Conference on, 852 –857. doi:10. 1109/ROBOT.2007.363092. DEXMART (2009). Specification of benchmarks. In Deliverable D6.1 from FP7-DEXMART Project (DEXterous and autonomous dual-arm/hand robotic manipulation with sMART sensory-motor skills: A bridge from natural to artificial cognition. URL http://www.dexmart.eu. Dillman, R. (2004). KA 1.10 Benchmarks for Robotics Research. Technical report, University of Karlsruhe. Fugro General Robotics Ltd. (????). Deepworks. Available online: http://www.fugrogrl.com/software/. LABS, C. (????). Vortex. URL http://www.cm-labs. com/energy-offshore/products/simulators. Malis, E. (2004). Improving vision-based control using efficient second-order minimization techniques. In Robotics and Automation, 2004. Proceedings. ICRA ’04. 2004 IEEE International Conference on, volume 2, 1843 – 1848 Vol.2. doi:10.1109/ROBOT.2004.1308092. Marine Simulation (????). Marine Simulation ROVsim. Available online: http://marinesimulation.com. Matsebe, O., Kumile, C., and Tlale, N. (2008). A review of virtual simulators for autonomous underwater vehicles (AUVs). NGCUV, Killaloe, Ireland. Nowak, W., Zakharov, A., Blumenthal, S., and Prassler, E. (2010). Benchmarks for mobile manipulation and robust obstacle avoidance and navigation. In Deliverable D3.1
from FP7-BRICS Project (Best Practice in Robotics). URL http://www.best-of-robotics.org/home. Perez, J., Sales, J., Penalver, A., Fornas, D., Fernandez, J.J., Garcia, J.C., Sanz, P.J., Marin, R., and Prats, M. (2015). Exploring 3-d reconstruction techniques: A benchmarking tool for underwater robotics. IEEE Robotics & Automation Magazine, 22(3), 85–95. Prats, M., P´erez, J., Fern´ andez, J., and Sanz, P. (2012). An open source tool for simulation and supervision of underwater intervention missions. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, 2577–2582. doi:10.1109/IROS.2012.6385788. P´erez, J., Sales, J., Mar´ın, R., and Sanz, P.J. (2014a). Web-based configuration tool for benchmarking of simulated intervention autonomous underwater vehicles. In Autonomous Robot Systems and Competitions (ICARSC), 2014 IEEE International Conference on, 279–284. Espinho, Portugal. doi:10.1109/ICARSC.2014. 6849799. URL http://dx.doi.org/10.1109/ICARSC. 2014.6849799. P´erez, J., Sales, J., Mar´ın, R., and Sanz, P. (2014b). Online tool for benchmarking of simulated intervention autonomous underwater vehicles: Evaluating position controllers in changing underwater currents. In 2014 Second International Conference on Artificial Intelligence, Modelling and Simulation (AIMS2014), 246–251. Madrid, Spain. Quigley, M., Conley, K., Gerkey, B.P., Faust, J., Foote, T., Leibs, J., Wheeler, R., and Ng, A.Y. (2009). ROS: an open-source Robot Operating System. In ICRA Workshop on Open Source Software. Raimondo, S. and Silvia, C. (2010). Underwater image processing: state of the art of restoration and image enhancement methods. EURASIP Journal on Advances in Signal Processing, 2010. Roser, M., Dunbabin, M., and Geiger, A. (2014). Simultaneous underwater visibility assessment, enhancement and improved stereo. In Robotics and Automation (ICRA), 2014 IEEE International Conference on, 3840– 3847. IEEE. Sanz, P.J., Pe˜ nalver, A., Sales, J., Fornas, D., Fern´ andez, J.J., Perez, J., and Bernab´e, J.A. (2013a). GRASPER: A multisensory based manipulation system for underwater operations. In 2013 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, Manchester, UK. Sanz, P.J., Prats, M., Ridao, P., Ribas, D., Oliver, G., and Orti, A. (2010). Recent progress in the RAUVI project. a reconfigurable autonomous underwater vehicle for intervention. In 52-th International Symphosium ELMAR-2010, 471–474. Zadar, Croatia. Sanz, P.J., Ridao, P., Oliver, G., Casalino, G., Petillot, Y., Silvestre, C., Melchiorri, C., and Turetta, A. (2013b). TRIDENT: An european project targeted to increase the autonomy levels for underwater intervention missions. In OCEANS’13 MTS/IEEE conference. San Diego, CA. Torres-M´endez, L.A. and Dudek, G. (2005). Color correction of underwater images for aquatic robot inspection. In Energy Minimization Methods in Computer Vision and Pattern Recognition, 60–73. Springer. ViSP (2010). Visual Servoing Platform. Available online: http://www.irisa.fr/lagadic/visp/visp.html.
11688