Integrated Development Environments and Visual Design Environments for ROV Mission Programming

Copyright @ IF AC Manoeuvring and Control of Marine Craft, Aalborg, Denmark, 2000

INfEGRATED DEVELOPMENT E.1WIRONMENTS AND VISUAL DESIGN ENVIRONMENTS FOR ROV MISSION PROGRAMMING

Daniele Maffei(l), Bruno Papalia(l), Giorgio Allasia(2), Fabio Bagnoli (2)

ENEA - sector of Robotics and Information Technology Casaccia, via Anguillarese, 301- 00060 Rome, Italy (2) D'APPOLONlA _ via S, Nazaro, 19 - 16145 Genova, Italy

(l)

C,R,

Abstract: Simplification of the remotely operated vehicles (ROVs) piloting in real time so as to carry out the scientific mission in an efficient, reliable and repeatable way, is an ambitious aim, With the presented ROV interface, adopting the integrated development environment (IDE), pilot and scientists have high-level facilities for their cooperation and synchronization besides the possibility of panning, debugging and executing mission, The capabilities are enhanced with a visual design environments (VDE) that allows the pilot to drive graphically the vehicle through the map, Such system as this, combining the language and action paradigms , integrates their advantages and forms the core of direct manipulation programming for ROV navigation, Copyright C 2000 IFAC Keywords: human machine interface, direct drive robots, robot programming,

I, INTRODUCTION

designing and the developing of a ROV interface for an EC MAST III project (see § 5), The presented HMI satisfy the above mentioned requirements and has the following features : operates with one, two or more workstations and users; plans the missions for the ROV and the scientific instruments (SIs) as well; permits the modification of the planned mission even during its execution; allows to repeat accurately missions; synchronizes the ROV and SI tasks; displays scientific data making easier to settle mission progression, The adopted approach needs a reliable positioning system and an obstacle avoidance system to preserve the ROV integrity, For the HMI prototype it is wilfully adopted "off the shelf' components and a client-server architecture, The main tasks of the server module are : to receive status messages from SS; to store status information in data base; to reply status information messages to client modules; to provide data requested by client modules, While the main tasks of the client module are: to show status information received from server module and to allow reading, editing, saving and running user programs,

There is an increasing demand by the scientific community for a mobile underwater platform capable of obtaining automatically samples of water and sediment, carrying out accurate quantitative photos and videos, The use of remotely operated vehicles (ROVs) helps in this type of scientific investigations, ROV piloting needs, unfortunately, great mastery and training, therefore scientific missions are frequently leaded by two kinds of professionals: scientists and pilots, Inside this staff, the synchronization of working phases and the cooperatio n on heterogeneous tasks are fundamental to achieve targets, A critical factor, in this situation, is the human machine interface (HMI) and its support to users with high-level facilities for cooperation and synchronization, The additional possibility of planning a mission is considered as extremely valuable, particularly when it may be repeated in further occasions, even if unexpected conditions are occurred, These themes have been faced during the

63

2. IDE APPROACH TO MISSION PLANNING syntactic level

semantic level

(programmable by pilol)

- - - - -= = = = = = =-

--car

can'

~

I I I

-r u.6r

I'

C4RII.

-

an.o.n

...

IOV

i

-.I

I I

IL ~J

. 1 ~yt~lQ~~

SI

IDE

command

.

SS

instructions

.......

ROV

Z IIW

. 3 FOllOW TliJICTOlY

-"Hm pOsmoil·- "

status

~~~.i

. }!!~Q2(i'1 " "

sensor signals

, WAllUAl..Pn.OT1llG

!~Q.TL~= ~-

!.IOTATI .!.iOIOO). ' UEP.PosmON IiJ"FOllOW nillCTOlY 12 SH.!.iEO.rn.OT1llG

Fig. 3. Macro system framework.

~~~..?

fi nE, pomiON"·-·

SIPr¥.... 3

normal

• •

executed executing next to execute

10'1 ........

~

.!.I ...,.• I:

normal

Fig. 4. Colour code for highlighting the program row status.

Fig. L Integrated development environment for ROY programming.

programs, at any moment during the miSSIOn progression, and executing it; at run time, during the mission progression, without editing program. This versatility is very useful to operate in under water environments, that are typically de structured and quite dynamic, but arouses indeterminacy for mission repeating because the commands executed might not be in the planned sequence. This problem has been solved with a simple shrewdness: by storing the mission log (data output) in the same format as the programs (data input). The commands, during execution, are sent to the Supervisory System (SS) that interprets and dynamically translates them in a sequence of low level instructions for the ROY . The specific sequence depends on the feedback from the ROY that is constantly monitored by the SS. This framework assures the required decoupling between semantic and syntactic level of tasks (Fig. 3), so that a ROY program may be repeated accurately in further missions.

Fig. 2. State diagram ofIDE. Nowadays the integrated development environments (lDE) are currently used to develop computer programs. This approach can be exported in order to develop ROY programs and debug them during mission progression. The definition of states that are object of programming is of primary importance for ROY control. Among these states, essential for piloting is the definition of the position related to a xyz reference system. Consequently the implementation of high level commands, such as "go to a xyz position with a specific speed" (GOTO), "keep the current position" (KEEP) and "rotate" (ROT ATE) it is possible only having a reliable positioning system. The core of IDE realized in this work is the program whose steps are formed by the mentioned commands, one each row. The user is provided with a first set of functions for program editing (left frame of the window in Fig. I). The parameters needed for each command are defined and edited in a specific window that is activated by clicking on program rows or command palette. With just a single environment, the user is able to carry out a mission using the commands in three different ways, at planning. mission and run time (Fig. 2): at planning time, by editing programs before the mission execution; at mission time, by editing the

To control the command interpretation carried out by the SS, IDE provides a second set of functions (bottom frame of window in Fig. I) as "execute one step of program" (STEP), "pause and resume the executing step" (PAUSE) and "abort the executing step" (STOP). This set is enhanced by the capability to change the order of pre-arranged sequence, even without editing the program, just by pointing to the next step. To support these capabilities and test the HMI functionality, each program row is highlighted using a colour code: white, is the normal status of program row; blue, the command is being edited; yellow, the command will be the next to be executed' red, the command is being executed; green, th~ command has been successfully executed. The pilot also checks the connection and the Supervisor System (SS) status by means of coloured rows that blink until acknowledgment messages are received from the SS.

64

Fig. 6. Map for pilot with ROV course and ahitude. Fig. 5. ROV status window for pilot.

(Shneiderman,1998). A rapid visual feedback provide evaluative information whose benefits, transferred in a ROV mission, allow to reduce route errors with significant energy saving.

Using the colour code, during the normal sequential execution, the program rows appear as shown in Fig. 4.

The aforesaid considerations related to pilot and his ROV IDE must be extended also to scientist and his SI IDE that allow to plan, edit and execute programs for each tele-controllable scientific tool.

The "a priori" knowledge of the seabed is not absolutely necessary for taking advantage by actionobject approach. The pilot may carry out an initial visual survey of an unknown environment using the IDE and the VDE in the "empty" georeferenced space. The graphic tools of the HMI allow the pilot to describe the meaningful morphological features of the sea bed and to enrich them with photo references. In this phase the obstacle detection and avoidance systems together with the auto-altitude c.apabi~ity give the necessary safety margin for ROV mtegnty. Scientific data as temperature, conductivity, ph, fluorescence, dissolved oxygen, etc. are contemporary gathered and shown in overlay to the ROV course. This preliminary set of information is enough to characterize the environment and to recognize the most interesting sites for scientific investigations. The planning of the remaining part of the mission and its repetition can go as if the map is available.

3. DATA DISPLAYING AND ROV PILOTING llIROUGH VDE. In order to show the ROV conditions it is possible to take inspiration from visual design environments (VD E): the result is a specific ROV status window that enhances the IDE performance (Fig. 5). The window does not perform a simple display process but it also allows the pilot to point out particular information using the map and colours, to add personal references using graphic tools and, in short, to more easily manage the mission . For this support, the most of the window area is assigned to a seabed map and the relative graphic functions (top in Fig. 5). In overlay to the map, the ROV course is plotted together with its altitude, which is coded in blue tone : dark or light according to low or high altitude (Fig. 6). Actual depth may be simply retrieved by pointing the mouse on the course. The pilot can also store remarks and symbols on the map in relation to relevant events. Besides, as an even more useful feature, he may also drag the target ROV position from the map to the parameter window of the selected command. With a simple actionobject interaction (Barfield, 1993) it is possible to drive the vehicle graphically with the map, among planning, mission and run time . navigation, is the embryo of direct-manipulation programming. The interactive use of course on map, as model-world metaphor for ROV (Shneiderman, 1983): "each action produces a comprehensible result in the task domain that is visible in the interface immediately"

The remaining area of status window (bottom of Fig. 5) is assigned to display the further information about navigation. Specific device status may be highlighted through a background colour that is assigned to the data fields by pilot: red for alert and green for normal condition.

4. COOPERATION AND SYNCHRONlZA nON BETWEEN INSTRUMENT AND ROV TASKS One of the most relevant problems during a multidisciplinary activity is to allow an effective cooperation and synchronization among the roles and the relati ve tasks .

65

HUMAN PROCESSES

ROVdomain ROV program

0.-

COMPUTERIZED PROCESSES

RaV program row

SI domain SI program

0 .. SI program row

0.-

0 .. -

0.. 1

0 .. 1

RaV command

SI command

Fig. 9. Entity-relation diagram for ROY-SI through soft synchronization (SSync).

link

HCI

Rav map (A) for pilot

SI map (8) for scientist

Fig. 7. Actor, roles and privileges for ROY control.

green back colour

Fig.

8. Task overlapping.

synchronization

and

authority Fig. 10. Maps relative to ROY and SI showing the enabled status instrument through hard synchronization (HSync).

Cooperation between pilot and scientist is supported by the presented HMI through a privilege system. It is implemented opening the windows (processes) in two ways: advisory mode, and interactive mode. This approach allows to resolve the conflicts of cooperation without ambiguity: each user can (only) command the processes related to his own role and that are reached through windows in interactive mode. Moreover all the processes related to other roles are (only) observed through advisory mode windows: in this way the users are enabled to have a good knowledge of whole system status. The current privileges are well recognizable by the background colour (Fig. 7): interactive mode window has a light blue background (or other selectable colour); advisory mode window has a standard grey background. The user privileges may be selected, for each process, during mission configuration. This system shows its limits when two tasks, pertaining to different authorities, must be carried out at the same time . Indeed the privilege mechanism may turn in crisis for common details and the inherent overlapping of authority fig. 8). For task synchronization problem, we have obtained a satisfying result improving the privilege system with two synchronization methods : soft (SSync) and hard (HSync).

The first is adopted for scientific tasks that are not dangerous for ROY safety, as water sampling, video shooting, etc. It is achieved with a link "by reference" between a ROY program row and the SI task, whose program name is shown in right column of ROY IDE fig. I) . With this approach (Fig. 9) the planned synchronization is under the pilot authority but it is not compulsory because each user may change his programs and related executing sequence. All users are informed of mutual activity through advisory IDE and its coloured program rows. HSync is adopted for the other scientific tasks that must be strictly controlled by the pilot. These kinds of activities, as generally complex, are entrusted to external interface whose messages are sent via TCP/IP network to HMI server. Here their messages are screened for type classification: if the pilot disable the instrument, messages will be rejected, otherwise ifhe enable it, they will be redirected. The scientists may monitor the system status using a HMI client process and the external interface in a multi-window environment on the same workstation. In particular, he may graphically observe the enabled or disabled status on SI map (Fig. 10).

66

5. THE ARAMIS PROJECT

Fig. 12 Victor of the French Institute of Research and Exploitation of the Sea (IFREMER), the large size ROV adopted for ARAMIS. Courtesy of lFREMER.

Fig. 11. Romeo of the Italian National Research Council - Institute for Naval Automation (CNRIAN), the medium size ROV adopted for ARAMIS. Courtesy of IAN.

Table 1 Romeo features

The above-described HMI will be developed for the ARAMIS project supported by the EC in the framework of MAST III programme (http J/europa.eu.intlconun/dgI2/marineI .html). The project intends to address the needs of the scientific investigations and to promote and support efficient and economic data collection with innovative system. ARAMIS (Advanced ROV package for Automatic Mobile hvestigation of Sediments) is able to carry out integrated multidisciplinary seabed research of physical, chemical and biological processes.

Operational Depth Vehicle dimensions

Propeller design Power system

500 meters Vehicle Weight: 300 kgs Size 1.30 x 0.90 x 0.90 m (\wh) 8 Thrusters: 4 vertical and 4 horizontal Mixed (Batteries+Tether)

The resulting system (ARAMIS+ROY) will allow the pilot/scientist to easily command in real time both the ROV and the scientific instruments/tools so as to carry out the scientific mission in an efficient, reliable and repeatable way. The scientific capabilities in terms of data acquisition are: automatic water physical and chemical profiles; automatic sea water sampling; automatic sediment sampling; physical and chemical sediment profiles (temperature, pH, H2 S, oxygen); micro-bathymetric mapping for geological and morphological analys~s of bottom; quantitative and dimensional s:lrvey vIa processing of stereo TV images; visual analysis of sea floor features through automatic image processing techniques . . . The technological capabilities, in terms of navIgatIOn functions are : automatic swim to a position; automatic station keeping; automatic navigation; automatic landing on the sea bottom to take samples; automatic obstacle detection and avoidance.

The ARAMIS system consists of two parts: a suitable skid to be installed under the underwater vehicle; a control wnsole to be interfaced with the vehicle surface controller and formed by the presented HMI and the SS. Both parts will include scientific and technological equipment. The ARAMIS skid on its turn consists: a core skid for missions with medium size class ROVs, as Romeo (Fig. 1 I and Table 1); a large skid, including the core skid and additional equipment, for mission with large size class ROVs, as Victor (Fig. 12 and Table 2); To allow the running of the presented HMI in the different configurations adopted by ARAMIS (Romeo or Victor), it is essential that HMI is independent from hardware and software platforms. Therefore its implementation (semantic level, see § 2) has been realized using the Java language, taking full advantage of its intrinsic c1ientlserver orientation and machine independent feature.

The project team is coordinated by Tecnomare and also includes: Heriot-Watt University (UK) , CNRIAN (Italy), IFREMER (France), ENEA (Italy), Challenger Oceanic (UK), Universitat de Barcelona (Spain), Institute of Marine Biology, Crete (Greece), National University oflreland, Galway (Ireland). The demonstration test campaigns of the systems are scheduled in the summer of the year 2000.

67

Table 2 Victor features Operational Depth Vehicle dimensions

Speed

Toolsled - Payload

Umbilical electromechanical cable

6000 metres Vehicle Weight: 3700 kgs Size 3.15 x 2.15 x 1.80 m (lwh) + tools led Forward: > 1.5 knots Lateral: > I knots Vertical: > I knots Payload/water: 150 kgs Reversible variable buoyancy system Capacity: 70 kgs Flow rate: 2 kgs/mm Length: 8500 m Diameter: 20 mm Power: 20 kW

6. CONCLUSION The synergy between the integrated development environment (!DE) and the visual design environment (VDE) allows to develop the core of the direct manipUlation programming for ROV missions. With this approach, a combination between language and action paradigms (Dix, et al., 1998), the pilot may concentrate his attention on learning the task domain, not learning the computer system (Hutchins,

et al., 1986) Of course, for ROV piloting, the aforesaid adjustments and shrewdness are needed to export profitably the !DE and VDE philosophies to the prototype. The main features of this style of interaction are mission programmability, syntax error reduction, task cooperation and synchronization, accurate repetitiveness of missions, high integration of mission managing phases and interactive data displaying on the map.

REFERENCES Barfield, L (1993) The user interface - Concepts & design, Addison Wesley Dix, A., 1. Finland, G. Abowd. R. Beale, (1998) Human-computer interaction, Prentice Hall Hutchins, E.L., 1.D. Hollan, and D.A. Norman, (1986) Direct manipulation interfaces. In User centered system design. (Norman, D. A. and Draper, S. W. (Eds». Lawrence Erlbaum Associates, Inc. Shneiderrnan, B. (1998) Designing the User Interface

- Strategies for Effective Human Computer Interaction, Addison Wesley Shneiderman, B. (1983) Direct Manipulation: A step beyond programming languages, IEEE computer, 16(8)

68