Interactive task planning in assistant robotics

IAV2004 - PREPRINTS 5th IFAC/EURON Symposium on Intelligent Autonomous Vehicles Instituto Superior Técnico, Lisboa, Portugal July 5-7, 2004

INTERACTIVE TASK PLANNING IN ASSISTANT ROBOTICS Cipriano Galindo, Javier Gonzalez, and Juan-Antonio Fernandez-Madrigal System Engineering and Automation Department University of Malaga, Campus Teatinos Complejo Tecnológico 29071, Malaga, Spain. {cipriano, jgonzalez, jafma}ctima.uma.es

Abstract: Close interaction with humans (non-expert people) is one of the main characteristic of assistant robots. Non-expert people must be able to know the robot intentions as well as to supervise its operation during the planning of tasks. Such kind of user-supervision requires that the robot manages the same world concepts that humans do, like rooms, corridors, objects, etc., at the same time that it reports the user on intermediate plans before they are executed. In this paper, an efficient hierarchical task planner that deals with human concepts is described. It has been implemented and tested in a robotic wheelchair for impaired people. Copyright © 2002 IFAC Keywords: Planning, Artificial Intelligence, Autonomous Mobile Robots, Interfaces, Cognitive Science.

1. INTRODUCTION1

interaction between assistant robots and humans reveals the necessity for the robot to manage the information of the environment in a human-like manner.

Situated agents are intelligent systems which can decide their own actions depending on the current environment situation (Russel and Norving, 1995). A special kind of situated agents are assistant robots that work in real situations interacting intelligently with humans. In assistant applications, humans become the most important element of the robot environment (Morioka et al., 2002). Some examples of assistant application are tour-guider robots, office delivery robots, clean and keeper robots, etc. In all these cases, the robot must plan and act to accomplish the task entrusted by a non-expert person which has been specified in a human-like manner, i.e. “bring my copies from the copier room”.

Humans have a particular way to structure the knowledge of the world. It is stated in literature that they use a mechanism called abstraction to store and manage efficiently the huge amount of information emerged from their environment (Fernandez and Gonzalez, 2002). Abstraction allows humans to work with concepts that group sets of other more concrete concepts. For instance, when somebody refers to “my office” she/he is talking about an abstract concept dropping unnecessary details of other ones, like door, wall, table, chair, cabinet, etc. When a person decides to go to her/his office, she/he focuses on the abstract concept, regardless of the other concepts that the office entails. In this example, the office concept may be further abstracted to the “rooms of the second floor” concept, which may be abstracted again to the “building” concept, etc.

The main characteristic of assistant applications is that they are designed to serve non-expert people within their environments. Therefore, the robot must possess and manage a model of its environment (world model) which allows it to friendly communicate with humans as well as to plan and perform tasks involving human concepts. In this paper these concepts refer to physical world elements, such as rooms, objects, buildings, etc. For example a delivery robot that performs tasks like “fetch my mail from the mail-room”, has to manage human concepts like “mail” and “mail-room”. The

In particular, in this paper we are interested in using abstraction for implementing task planning in assistant applications, which will permit us to include interaction with the human at the level of planning symbolically. Our approach covers some important issues such as reliable communication mechanisms between humans and robots, and user supervision of the robot plans, without forgetting the thorny topic of efficiency in task planning.

1

This work was supported by the Spanish Government under research project DPI2002-01319. 609

In section 2 we give some motivation for using abstractions in task planning for assistant robotics. Section 3 introduces a multihierarchical world model of the environment of the robot, called Multi-AHgraph. Section 4 outlines our hierarchical task planner which can deal with the abstract concepts represented in that model. Section 5 describes the implementation of the task planner for a particular assistant application: a robotic wheelchair for impaired people. Finally some conclusions and future work are outlined.

detailed plan, so they can be ignored when planning. This hierarchical planning approach improves planning efficiency. A more detailed description of this topic and comments on several hierarchical planning approaches can be found in (Galindo, et al. 2003a). The task planning approach presented in the next sections relies on a multihierarchical arrangement of the robot world model (Fernandez and Gonzalez 2002), which enhances assistant applications with 1) user-supervision of abstract plans, 2) a proper robotuser communication mechanism, and 3) task planning efficiency. This task planner has been integrated into a robotic architecture, called ACHRIN, which has been specifically designed for assistant applications (Galindo, et al. 2003b).

2. MOTIVATION An assistant robot working and interacting with humans must be capable to deal with world abstract concepts in order to better accommodate its behaviour to human requests. One of the most evident benefits of using abstract concepts is that of a proper humanrobot communication through concepts shared by both humans and the robot. But in addition, humans can use abstract concepts to specify abstract robot tasks. For instance, an user can ask a delivery robot to perform the task: “pass through the west-corridor, go to the library and ask for a book”. Although this task is not completely detailed since it involves abstract concepts (“library” and “south-corridor”), a robot managing such concepts could solve it.

3. THE MULTI-AH-GRAPH MODEL This section gives a non-formal description of a graph-based model of multiple abstractions, called Multi-AH-graph, which is derived from simpler representations such as graphs and hierarchical graphs, and has been successfully used in other mobile robot experiments (Fernandez and Gonzalez 1998; Remolina, et al. 1999). Complete formalizations of all these models can be found in (Fernandez and Gonzalez 2002). Section 2.1 focuses on a single-hierarchy model called AH-graph, which includes only one hierarchy of abstraction. The AHgraph model is the basis for the construction of the multihierarchical model called Multi-AH-graph.

Apart from the benefit of communicating to the user, the usage of abstract concepts brings the robot other advantages. On one hand, the process to obtain a sequence of actions to perform a task can take advantage of using abstract concepts. Thus, an assistant robot might plan a task at a certain intermediate level of abstraction and ask the user if such scheme for a plan (an abstract plan) meets her/his wishes. This feature permits humans to supervise the robot actions at the same time that they are reported on the robot intentions. For example, when the robot is dealing with abstract concepts to plan the fetch-my-mail task, the user can reject the resulting abstract plan if one of its actions involve a non desired abstract concept, i.e. navigation through a crowded area of the building. In other words, the user can modify abstract plans before a less intuitive and more detailed one (that demands an additional computational effort) is produced by the robot.

3.1 A Single-Hierarchy Graph Model (AH-graph) An AH-graph2 is a graph representation that includes hierarchical information, that is, the possibility of abstracting groups of elements to “super-elements”. Abstraction produces different layers, called hierarchical levels, that can represent the same environment with different amount of detail. These layers consist of flat graphs whose nodes represent elements of the environment (concepts), and whose arcs represent different relations between those concepts (for example, “navigability”). A simple example of AH-graph that models a typical scenario is shown in fig. 1.

On the other hand, the usage of a world model that includes abstractions can also improve the efficiency in robot task planning (Galindo, et al. 2003a). When the robot obtains an abstract plan, it can notice that some abstract world concepts are irrelevant for the plan. Detecting irrelevant concepts for a plan can be very useful when the environment of the robot becomes quite large since all concepts embraced by irrelevant concepts can be discarded when planning in a lower abstracted level. For example, if an abstract plan to bring the user mail does not contain the “North-corridor” concept, all concrete elements that this concept entails will not participate into the final

A group of nodes of a hierarchical level can be abstracted to a single node of the next higher hierarchical level, which becomes their supernode (the original nodes are called subnodes of that supernode). Analogously, a group of arcs of a hierarchical level can be represented by a single arc (their superarc) at the next higher level. The lowest hierarchical level of the AH-graph is called the ground level, and it represents the environment with the maximum amount of detail that is available. The 2

610

“AH-graph” stands for Annotated Hierarchical Graph.

highest hierarchical level is called the universal level, and it typically contains a single node representing the whole world. The AH-graph is not only its hierarchical structure: nodes and arcs can hold any piece of non-structural information (procedural, geometrical, etc.). This is achieved by means of annotations. Any number of annotations can be attached to a node or arc.

this paper. In that particular case, different paths in the multihierarchy between the ground level and the universal level account for hierarchies (AH-graphs) produced by different abstraction operations. Using a symbolic multihierarchical representation of the environment provides three important benefits (please refer to (Fernandez and Gonzalez 2001; Fernandez and Gonzalez 2002) for more detail): first, a multiple hierarchy permits us to choose the best hierarchy to solve each problem (i.e.: to adapt better to diverse problems, improving the overall efficiency); second, when several problems have to be solved, a multiple hierarchy provides the possibility of solving part of them simultaneously; and third, solutions to the problems can be expressed in terms of the concepts of any of the hierarchies, thus the information is given in the most suitable way for each specific purpose. In general, multiple hierarchies have proven to be a more adaptable and efficient model than single-hierarchy or non-hierarchical models.

Finally, the arcs of an AH-graph have associated costs that usually represent the strength of the relations they represent. Arc costs are not defined as numeric values like in typical flat graphs, but as finite numeric intervals. Numeric intervals enable the AHgraph to deal with a certain degree of uncertainty in geometrical measurements and can also be abstracted in a very efficient way. 6

9

7 8

10

14 4 2

15

3

5

17

16

1

18

11 Distinctive Place

a)

Laboratory 6

9

7 8

Office 1 10

4

Corridor

1

18 3

5 16

11

12

Northern Rooms

Northern Rooms

15

17

13

b) Laboratory Office 1

14

2

12

Corridor Rooms

Corridor Rooms

Corridor Southern Rooms

Office 2 13

Library Office 2

Library

Southern Rooms Environment

c)

Fig. 2. An example of multihierarchy (with three hierarchies). In this directed acyclic graph, hierarchical levels are represented by rectangular boxes. Thick arrows represent abstraction links, that is, bunches of abstractions of nodes and arcs of hierarchical levels. Level A is the only ground hierarchical level of this multihierarchy, and level H the only universal level.

Fig. 1. An example of an AH-graph. (a) A schematic plan of a real environment. Distinctive places for robot navigation are marked with small rhombuses. (b) Ground level: topology of distinctive places. This will be the lowest hierarchical level (ground level) of the AH-Graph. (c) Upper levels of the hierarchy.

In the real implementation of this work a twohierarchical Multi-AH-graph has been used to test the suitability of our approach. Upon the same ground level, a particular hierarchy, the cognitive-hierarchy, is aimed to establish a cognitive interface with the human (Fernandez, et al., 2003). Such hierarchy, similar to the one depicted in figure 1, enables the robot to manage user concepts (world elements), enhancing the human-robot communication. The other hierarchy, the planning-hierarchy, arranges properly the world elements in order to improve the common tasks of the assistant robot, in the case of this paper, task planning. Other hierarchies can be added, for example for localizing accurately the robot during navigation (Galindo, et al., 2003a).

3.2 Multihierarchical Graph Model (Multi-AHgraph) The AH-graph model explained in section 2.1 can be enhanced in order to deal with multiple hierarchies of abstraction, becoming a Multi-AH-graph. Broadly speaking, a Multi-AH-graph is a set of hierarchies interwoven in a directed acyclic graph structure, where each hierarchy is an AH-graph whose levels can be shared by others hierarchies (see fig. 2). It is common to use Multi-AH-graph models that contain just one ground and one universal hierarchical levels, such as the examples presented in the rest of

611

4. HIERARCHICAL, INTERACTIVE TASK PLANNER

environment (only the cognitive-hierarchy) which comprises several zones (a laboratory, four rooms, two corridors, and a hall), the mobile robot itself (which initially is at the entrance H1 of the hall), and a box to be handled (which is on the table L3 of the laboratory).

In this work, we face interactive task planning by adapting a hierarchical approach called HPWA, Hierarchical Planning through World Abstraction, which has been formalized elsewhere (Galindo, et al., 2003a). HPWA allows the planning process to successively obtain abstract plans. This feature enables human to monitor and control the task planning work. HPWA also improves the efficiency of strips-like planners when dealing with a world model arranged in a hierarchical way. Next, an example of HPWA planning is given. For clarity in the explanation, only the cognitive-hierarchy is used both to communicate with the user and to perform task planning3. at

The general idea of HPWA is to solve the task at a high level of world abstraction by running a certain embedded task planner, and then use the resulting abstract plan at the next lower level ignoring the irrelevant world elements that do not take part in the abstract plan. In this paper we have chosen the Metric-FF as our embedded planner (Hoffmann and Bernhard, 2001) which is a well-known and efficient task planner. In the following, let us consider a relatively simple environment to illustrate the HPWA method and the user-supervision feature. A delivery robot with a manipulator on board has to perform different tasks involving a given object of the world named “box”.

at

Box’’

Floor

Robot’’

L2

Na v

Room 1

Room 3

Hall Nav

Nav

West Corridor

i) Planning at the universal level L2

Robot’

at

Nav

Na

East Corridor

Laboratory Nav

Nav

Room 2

Room 4

FLOOR

L1

Box’

ROOM 3

at

ROOM 1 v Na

At the ground level the predicate that represents the goal state (the desired final state of the world when the plan is completely executed) is (holding Box) (operator definitions are as shown in figure fig. 4). This state is abstracted up to the universal level L2 as (holding Box’’). At the universal level, the predicates that model the initial state of the world are: (at Box’’ Floor), and (at Robot’’ Floor).

v

Entrance (H1)

Robot R3-3 av N

R1-2

HALL

H2

R1-1

Nav

R3-2

Metric-FF is run at L2 finding the following abstract plan: (PICKUP Box’’ Floor)

R3-1

R1-3

Nav

In front of Room 1 (WC3)

In front of Room 2 (WC4)

Nav WC2

Nav

Nav WC5

Nav

Nav

Nav WC6

EC1

In front of Lab (EC87 Na

Nav

v

Entrance (L1)

Nav

In front of Room 3 (EC3) Nav

EC5

EAST CORRIDOR Cabinet (L2)

N av

at Box

Table (L3)

Nav Chair (L4)

(operator PICKUP (params ( OBJECT) LOCATION )) (preconds (at-robot )(at-object )) (effects (holding )(del at-object )))

R4-1

v

ROOM 2

(operator GO (params ( LOCATION) ( LOCATION) ) (preconds (at-robot ) (nav ) ) (effects (del at-robot ) (at-robot )))

In front of Room 4 (EC4)

Na

R2-2

Nav EC6

Nav

R2-1

Nav EC2 Na v

In front of Lab (WC7) v Na

WEST CORRIDOR

Nav

Fork (H3)

WC1

Window (L5)

ROOM 4

LABORATORY

L0

Fig. 3. Example of the cognitive hierarchy of a robot environment. For clarity, the set of nodes (subnodes) abstracted to the same node (supernode) are contained within a grey-shaded region. Some nodes are labelled with the name of human concepts, i.e. “entrance”, “table”, etc.

Fig. 4. Definitions of the operators used in the example. Observe that they involve some preconditions and effects, as in a classical STRIPS syntax.

Fig. 3 shows the hierarchical model of the

Now, the resulting abstract plan at level L2 is moved down (refined) to level L1, where only the subnodes of nodes of L2 that take part in the plan are considered4. In this particular case no node is ignored

ii) Planning at level L1

3

When there is more than one hierarchy in the model of the world, a process to translate concepts from one hierarchy into the others is required. That process is out of the scope of this paper, but broadly, it consists of moving concepts down to the common ground hierarchical level and turning them up into concepts of the target hierarchy.

4

Although the Robot element does not appear in the plan, it is an implicit concept that cannot be removed.

612

since the abstract plan contains all elements of its hierarchical level. At level L1, the predicates that model the initial state are shown in table 1, while the goal state is (holding Box’). The abstract plan provided by Metric-FF to solve the task at level L1 is: (GO Hall WestCorridor), (GO WestCorridor Laboratory), and (PICKUP Box’ Laboratory).

task planner. The backtracking mechanism included in HPWA is used to obtain another abstract plan according to the human preferences. LOGICAL PREDICATES AT GROUND LEVEL (at Box Table-L3) (nav WC6 WC7) (at Robot Entrance-H1) (nav WC1 WC2) (nav Entrance-H1 H2) (nav WC2 WC3) (nav H2 Fork-H3) (nav WC7 Entrance-L1) (nav Fork-H3 WC1) (nav Entrance-L1 Chair-L4) (nav WC1 WC2) (nav Entrance-L1 Cabinet-L2) (nav WC2 WC3) (nav Chair-L4 Table-L3) (nav WC2 WC5) (nav Cabinet-L2 Window-L5) (nav WC5 WC4) (nav Chair-L4 Window-L5) (nav WC5 WC6)

Observe that several world elements (the all four rooms and the EastCorridor), do not appear in the plan: they are irrelevant for this abstract task, and therefore, at the next lower level, the ground level, their subnodes can be discarded.

Table 2. Logical predicates that represent the ground level when planning. Notice how irrelevant concepts are ignored to plan at this level. The significant reduction of world information improves the planning efficiency.

Also notice that, at this moment, the user knows the robot intentions to carry out the proposed task, since the abstract plan can be translated into human concepts of her/his environment (as commented before, this paper does not focus on that interhierarchical translation; we consider for simplifying the example that the task-planning hierarchy is the same as the cognitive hierarchy). User may disagree with the generated abstract plan, i.e. she/he does not want to pass through the WestCorridor because its roof is leaking. In that case, the robot must produce an alternatively plan by eliminating all references about the WestCorridor concept. The other possible plan is: (GO Hall EastCorridor), (GO EastCorridor Laboratory), and (PICKUP Box’ Laboratory), where the irrelevant concepts are now the four rooms and the WestCorridor. If this new abstract plan is not rejected by the user, the planning process can continue.

5. APPLICATION TO AN ASSISTANT ROBOT In this section we describe the implementation of our planning approach to a robotic wheelchair for elderly and handicapped people, called SENA (Sena homepage 2003). Our approach has been integrated in a specialized robot control architecture, called ACHRIN, which stands for “Architecture for Cognitive Human-Robot Integration”, and has been presented in (Galindo, et al., 2003b). Fig. 5 depicts the general scheme of ACHRIN. It is a hybrid robotic architecture that exhibits the classical three layers division: the deliberative layer produces plans, the execution and supervisor layer manages and controls the plan execution, and finally, the functional layer physically executes the plan. ACHRIN supports a new degree of human integration, called cognitive integration, which enables the assistant robot to share part of the human symbolic world model (Fernandez, et al., 2003). In this paper, we focus on the deliberative layer of ACHRIN, which has been implemented as follows:

1

LOGICAL PREDICATES AT LEVEL L (at Box’ Laboratory) (nav EastCorridor Laboratory) (at Robot’ Hall) (nav EastCorridor Room1) (nav Hall WestCorridor) (nav EastCorridor Room2) (nav Hall EastCorridor) (nav WestCorridor Room3) (nav WestCorridor Room4) (nav WestCorridor Laboratory)

Table 1. Logical predicates that represent the world information at level L1. iii) Planning at the ground level L0 Finally, at the ground level, the predicates that model the initial state of the world are shown in table 2. Notice that no node within the Room is included in this world state, hence reducing the computational cost of planning.

The Hierarchical World Model has been implemented by a Multi-AH-graph with two hierarchies upon the same ground information. As previously commented, the cognitive-hierarchy provides a proper human communication (it contains concepts understandable by the human) while the other one provides efficiency in task planning. The process to construct each hierarchy is quite different: the cognitive-hierarchy is constructed in a semi-autonomous way in which the user interacts with the robot to specify what and where ground concepts are, i.e. table, door, etc. and how such concepts are grouped to form abstract concepts like rooms, corridors, etc. On the other hand, the planning hierarchy is automatically constructed by

The goal state at the ground level is (holding Box) and the resulting plan becomes the real plan that achieves the requested task: (GO Entrance-H1 H2), (GO H2 Fork-H3), (GO Fork-H3 WC1), (GO WC1 WC2), (GO WC2 WC5), (GO WC5 WC6), (GO WC6 WC7), (GO WC7 Entrance-L1), (GO Entrance-L1 Chair-L4), (GO Chair-L4 Table-L3), (PICKUP Box Table-L3)

User can also reject the final plan produced by the

613

exploring hierarchies that solve the most common tasks efficiently. More details on the construction of the hierarchies are reported elsewhere (Fernandez, et al., 2003; Fernandez and Gonzalez, 2002). Hierarchical World Model

CONCLUSIONS AND FUTURE WORK

Task Manager

Deliberative Layer

Task Planner

PLEXAM

Functional Group #1

user through the Task Manager. If the user does not reject the proposed abstract plans, the planner will refine them to produce a plan involving concepts from the ground level, which is executed.

…...

Consciousnesssupervised Alert System

Functional Group #n

This paper has addressed some issues related to planning in assistant robots. The close relation with human that imposes this kind of applications makes relevant the user supervision of the robot plans before they are executed. The work presented in this paper has explored this topic through a task planner that makes use of a multihierarchical model that represents the world. Our hierarchical planning approach has been successfully tested in a real assistant application: a robotic wheelchair. Our goal aims to improve the robot-human interfaces of the robot, including voice and speech recognition, as well as in further improving of the user supervision of plan execution. REFERENCES

Execution and Supervisor Layer

Functional Layer

Fig. 5. A scheme of the Architecture for Cognitive Human-Robot Integration (ACHRIN). The Task Manager enables the user to command the robot, at the same time that it translates concepts from the cognitive-hierarchy into the planning-hierarchy and vice versa. When the planner solves an abstract task, the Task Manager translates its concepts from the planning-hierarchy into concepts of the cognitivehierarchy to be accepted or rejected by the user via a graphical interface. Once a plan involving concepts from the ground level is accepted by the user, other components of ACHRIN are in charge of executing the plan and react to any unforeseen circumstance.

Fernandez J.A., and J. Gonzalez. (1998). Hierarchical Graph Search for Mobile Robot Path Planning. ICRA'98, Leuven, Belgium. Fernandez J.A. and J. Gonzalez (2001). MultiHierarchical Representation of Large-Scale Space, Kluwer Academic Publishers. Fernandez J.A. and J. Gonzalez (2002). MultiHierarchical Graph Search. IEEE PAMI. Vol. 24. No.1, pp. 103-113. Fernandez J.A., C. Galindo and J. Gonzalez (2003). Assistive Navigation using a Hierarchical Model of the Environment. To appear in Integrated Computer-Aided Engineering. Galindo C., J.A. Fernandez and J. Gonzalez (2003a). Hierarchical task Plannig through World Abstraction. To appear in IEEE Trans. on Robotics and Automation. Galindo C., J.Gonzalez and J.A. Fernandez (2003b). An Architecture for Cognitive Human-Robot Integration. Application to Assistant Robotics. Submitted to IROS 2004. Hoffmann J. and Bernhard N. (2001). The FF Planning System: Fast Plan Generation through Heuristic Search. Journal of Artificial Intelligence Research, Vol. 14. Morioka K., Lee J-H, and Hashimoto H. Human Centered Robotics in Intelligent Space. IEEE International Conference on Robotics & Automation, Washington, DC, May 2002. Remolina E., J.A. Fernandez, B.J. Kuipers and J. Gonzalez (1999). Formalizing Regions in the Spatial Semantic Hierarchy: an AH-Graphs Implementation Approach. In Lecture Notes in Computer Science ,Vol. 1661, pp. 109-124. Russell, S. and P. Norvig, P. (1995). Artificial Intelligence a Modern Approach. Prentice Hall, Upper Saddle River. Sena homepage (2003). http://www.isa.uma.es/ent_inv.htm

Fig. 6. The SENA robotic wheelchair navigating along the east corridor of our building and passing through a door. Several navigation experiences have been performed within a real environment with the robotic wheelchair SENA and our task planning system (Sena homepage, 2003), (see fig. 6). These experiences are similar to the example described in section 3, and consist of asking the wheelchair to navigate to a certain destination, i.e. the laboratory. The user selects the destination place by its symbolic name stored in the cognitive hierarchy. The task planner produces abstract plans for navigation which are reported to the

614