- Email: [email protected]
Foundations for designing an ecological interface for mobile robot teleoperation
Robotics and Autonomous Systems 31 (2000) 193–207
Foundations for designing an ecological interface for mobile robot teleoperation Tetsuo Sawaragi∗ , Takayuki Shiose, Go Akashi Department of Precision Engineering, Graduate School of Engineering, Kyoto University, Yoshida Honmachi, Sakyo, Kyoto 606-8501, Japan Received 5 December 1998; received in revised form 2 July 1999
Abstract This paper presents foundations needed in designing an interface system for mobile robot teleoperation. For this purpose, Vicente’s idea of ecological interface design (EID) is firstly reviewed, and a preliminary design of our teleoperation system is presented. Our system enables the naturalistic collaboration between a human and a robot’s autonomy at the skill level by joining their respective perception–action cycles via virtually constructed visual 3D space, in which a human naturalistic reflex between stimulus and response is effectively utilized. Following the presentation of an overview of our system, discussions on the interface design for the teleoperation are done concentrating on its ecological aspects. © 2000 Elsevier Science B.V. All rights reserved. Keywords: Human–robot systems; Ecological interface; Shared autonomy; Teleoperation
1. Introduction Recent tremendous improvements of highly advanced automated mechatro-informatic facilities have increasingly eliminated the human users out of their control loops, where the collaboration between humans and machines has been getting more and more difficult [10,13]. To overcome these difficulties, various new concepts for interface design have been proposed. As a design principle of human-friendly interface, Vicente has proposed an idea of an “ecological interface” [17]. Wherein the key issues of the interface design must be coherent with the ways of human thinking and/or perception performed under ∗ Corresponding author. Tel.: +81-75-753-5266; fax: +81-75-771-7286. E-mail address: [email protected] (T. Sawaragi)
their bounded cognitive resources. Interface must be able to stimulate a human user to perceive patterns in time and space and across the senses [14] as well as his/her intuitive leap to find innovative ways to fulfill the responsibilities [9]. Based on the above ideas, in this paper we propose a new interface system for mobile robot teleoperation based on the idea of ecological interface design. Our system aims at realizing naturalistic collaboration between a human and a robot’s autonomy by joining their respective perception–action cycles via virtually constructed visual 3D space, in which a human bodily reflex between stimulus and response is effectively utilized. This property of “embodied” interfacing capability is original in this paper and enables a human operator to acquire coordinated perception and action associations. After presenting a prototype system of our proposed interface system, we discuss a novel,
0921-8890/00/$ – see front matter © 2000 Elsevier Science B.V. All rights reserved. PII: S 0 9 2 1 - 8 8 9 0 ( 9 9 ) 0 0 1 0 8 - 6
194
T. Sawaragi et al. / Robotics and Autonomous Systems 31 (2000) 193–207
interactive style of collaboration between human and machine autonomies that is enabled by our interface system. Concerning the above, we propose an idea of “shared autonomy” as a new style of human–machine collaboration [5], which stresses that a human and a machine should collaborate with each other as equivalent partners. However, this idea still remains at the conceptual level and mainly focuses onto the aspects of flexible role allocation between the human and the machine. In this paper, we provide more functional explanations on why shared autonomy is needed and what relations between them must be expected to emerge from a perspective of ecological psychology. Then, we discuss how our proposed interface design for a teleoperated mobile robot does contribute to such emergence. 2. Ecological interface As a design principle of human-friendly interface, Vicente has proposed an idea of an “ecological interface” [17]. In this section, we first review this idea and discuss its significance in terms of the design principles of an interface for teleoperation systems. It can be generally said that it is very difficult to design styles of interactions made between the humans and their work environment composed of complex artifacts. The current complex systems to be controlled or monitored by the human operator are causing difficulties in keeping them in the control loop while maintaining their intrinsic, innate capabilities of perception–action coordination. This is because the systems come to have complex automatic control loops and devices within them, almost all of which are invisible to human operator. Moreover, since the consequences of the operator’s actions are becoming more and more critical meaning that it may bring about very high risks especially in emergency, the operator has to deliberate very carefully before initiating his actions in reply to a provided stimulus. That is, he is forced to do a “delayed” reaction, which makes it difficult to acquire perception–action coordination. To overcome such a problem, we have to develop some “aided eye” for a human operator that can stimulate his naturalistic reactions to reduce the dimensionality of their description of the system with so many components and so many interactions with
the human. That is, supporting interaction via the intrinsic perception–action cycles can lead to reducing deliberative reasoning burden of the operator and also can provide the appropriate computer support for the comparatively more laborious process of his problem solving. Concerning this, Rasmussen has proposed that there exist different cognition levels concerned with the operator’s behavioral modeling: SBB (skill-based behavior), RBB (rule-based behavior) and KBB (knowledge-based behavior) [8]. SBB is a behavior in which the specific features are experienced together frequently and the response is more or less automatic, while RBB is a procedure-oriented task including monitoring and interpreting. KBB includes the full range of problem solving and planning activity with the manipulation of some kinds of “deep” models. This is illustrated in Fig. 1. As this figure shows, in an ideal teleoperated robot system, parameters that are involved in the control and fed back to the operator may be of variety. As for those parameters Kheddar et al. have listed stability regions (i.e., distance from a particular point to an enveloping safe surface or volume), inertial forces, applied torque, slip direction and amplitude, joint limit margins, contact point and forces between the mobile robot and its environment, related obstacles distance, robot attitude relative to the environment, etc. [6]. However, in the conventional teleoperated system those parameters cannot be perceived directly by the operator, but should be reconstructed from the transmitted raw data that are shown in the display. That is, he/she has to decide the appropriate action at the levels of either of RBB or KBB, not at that of SBB. The other important characteristic of the SBB distinguished from other RBB and KBB is a property of “continuity” of behavior, which takes a form of a chain consisting of a series of reactive “perception–action” units and also enables a proficient, skillful dynamic perception. Thus, it would be desirable to develop an interface system that can stimulate an operator’s SBB by providing him/her with an “aided eye”. Another design principle of an EID advocated by Vicente is to use the means–end hierarchy for representing the work domain as a hierarchy of “nested” affordances. Here, “affordance” means a relationship between properties of the environment and properties of an organism’s action capabilities. That is, this is
T. Sawaragi et al. / Robotics and Autonomous Systems 31 (2000) 193–207
195
Fig. 1. Rasmussen’s SRK model for a teleoperator.
a relational property that specifies the match between the environmental structure and the functional abilities of an actor. Difference between the present use of affordance and Gibson’s original affordance [4] is abandoning an assumption that detection of an environmental affordance is always “direct” and unmediated. Rather, we assume that relatively simple perceptual access to the artificial stimulus of the display information is sufficient to specify the existence of an environmental affordance, thus to guide the operator’s appropriate reactions. Since in the original Vicente’s paper he dealt with the interface for a process plant, his advocated means–end hierarchy was a functional decomposition of the monitoring task within the thermodynamics domain. For our purpose of the teleoperated mobile robot domain, this means–ends hierarchy turns out to be a different one, and a particular behavior can be represented as a hierarchically arranged set of action mechanisms as shown in Fig. 2 [7]. Taking account of this nested action structure, in this paper we attempt to define the following kinds of affordance for our teleoperated mobile robot domain:
1. sighting affordance of the obstacles, 2. obstacles-evading affordance, and 3. locomoting toward destination affordance. We attempt to establish a mapping of the semantics of the above affordances onto the display of 3D virtual environment in order to navigate the operator by his/her naturalistic reflex between visual stimuli and responses to them to support interactions via the innate perception–action cycles at the level of SBB, since the cognitive process ongoing at this level is the most naturalistic in a sense that it is driven by the intrinsic reflex of stimulus and response. In the following, we design an interface system based on an idea of EID to provide an operator with a display that is “transparent” enough and a process ongoing there should be directly “exertible” by an operator in a reactive way. We extract such reactivity from operators by letting them use their own “bodies” without letting them use any artifact devices such as joysticks and mouses. This is an original aspect of our proposed interface system that is distinguished from Vicente’s implementation of EID.
196
T. Sawaragi et al. / Robotics and Autonomous Systems 31 (2000) 193–207
Fig. 2. Hierarchical organization of action mechanisms of the operator.
3. System architecture In an architecture of our proposed teleoperated robot system, a robot is a commercial product (“Pioneer” provided by Applied AI), on which ultrasonic range sensors are mounted. Steering and moving speed are controllable either by programs or externally provided commands. We do not assume a human operator to use any pointing device such as a mouse and a joystick, since those artifact tools intervene his/her naturalistic stimulus–response behaviors. The task we consider here is restricted to a very simple one: to let a robot go to some destination without colliding with walls and obstacles. The world recognized by a robot with such restricted sensors is reconstructed as a virtual 3D visual world using VRML (Virtual Reality Modeling Language) and presented to a human operator. Getting this world as a visual stimulus, an operator makes some responses of inclining a head leftward or rightward as well as in the front and in the rear. These movements are monitored by the vision-based motion capturing device and are sent to a robot that behaves reflecting an operator’s unconscious responses. Accordingly, the display will change in reaction to such an operator’s movement, and an operator is able to see this changed display from his/her moved viewpoint. This point is essential, because operators are
able to acquire direct associations between perceptional changes and their bodily actions of movements. Generally speaking, it is well recognized that a bodily reaction is more prompt and reliable than any other reactions using some instrumental devices. Especially, in our system such naturalistic reactions are derived from their head movements that are equivalent to their movements of eyes’ positions. Thus, synchronized changes of viewers’ perceptions and their postures are essential for stimulating their naturalistic interactions with the external things. This would not be possible if operators use mouses or joysticks through their hand manipulations. This system architecture is illustrated in Fig. 3. 3.1. A robot’s autonomy In an ideal teleoperated robot system, parameters that are involved in the control and fed back to the operator may be shown in Fig. 1. In our current implementing system, however, we assume that the following parameters are sensible in the local robot control subsystem: • moving speed and moving direction of the robot, • distances to and the directions of the neighboring obstacles measurable by ultrasonic range sensors, • recognized walls and distances to them, and • direction towards the destination.
T. Sawaragi et al. / Robotics and Autonomous Systems 31 (2000) 193–207
197
Fig. 3. An architecture of a teleoperated mobile robot system.
The role of robot’s local control characterizing the machine autonomy is to realize reactive/reflexive navigation of a mobile robot. For this purpose we adopt a potential field method to produce the appropriate velocity and steering commands for the robot. Arkin has proposed a method of motor schema-based mobile robot navigation, in which a schema is defined as a unit to model the interaction between robot’s perception and action [1]. Potential field method is widely utilized not only in the control of mobile robots and manipulators but also in the modeling method in brain theory and psychology; Arbib and House have developed a model for a worm acquisition by the frog in an obstacle-cluttered environment and implemented in their machine analog of Rana Computatrix [2]. In their work, they describe primitive vector fields: a prey-attractant field, a barrier-repellent field, and a field for the animal itself. These fields, when combined, yield a model of behavior that is consistent with experimental observations of the frog’s behavior. Arkin has taken analogs of these fields in the mobile robot system and introduced “move-to-goal”
and “avoid-static-obstacle” fields with the addition of new fields to describe other motor tasks such as “stay-on-path” and “avoid-moving-obstacle”. He defined a collection of schemata as templates for generating behavior, and being parametrized those generic schemata are instantiated to provide the potential actions for the control of the robot as well as their sensory expectations, thus the ways of interaction with the environment. When obstacles are detected with sufficient certainty, the motor schema associated with a particular obstacle starts to produce a force tending to move the robot away from the object (i.e., the instantiation of “avoid-static-obstacle” schema). Wherein, the degree of certainty of obstacle recognition is parametrized in a formation of the potential, which determines the velocity vector at the robot’s current location relative to the environmental object and/or the destination. In the current version of our system, objects to be distinguished are restricted to the walls and the obstacles including moving and static ones, and are recognized from a series of readings of the ultra-
198
T. Sawaragi et al. / Robotics and Autonomous Systems 31 (2000) 193–207
Fig. 4. Readings of the sensors plotted in series.
sonic range sensors as shown in Fig. 4. 1 Destination (i.e., a goal) is currently assumed to be known both to the robot and the operator, but of course it must be detected by the machine autonomy. Multiple instantiations of schemata are possible and the influences from each of the instantiated schemata are combined through a simple addition of the individual velocity vector. Thus, the specific velocity and direction for the robot can be determined at any point in time, and are transmitted to the low-level robot control system. The field equations for both the “avoid-staticobstacle” and “move-to-goal” schemata are exponential. An example showing the velocity produced by an obstacle (O) is given below: 1
Here the certainty of recognition is calculated using a Bayesian network by gathering new evidences on the changes of readings of ultrasonic range sensors confirming the existence of walls or obstacles (i.e., a probabilistic reasoning).
0 S−d − 1) ∗ G Omagnitude = (exp S−R ∞
for d > S, for R < d 6 S, for d 6 R,
where S is the sphere of influence (radial extent of force from the center of the obstacle), R is the radius of obstacle, G is the gain, d is the distance of robot to center of obstacle, and Odirection is the velocity along a line from robot to center of obstacle, moving away from the obstacle. Wherein, the parameter of gain is adjusted according to the degree of beliefs to which that obstacle is recognized by the robot. Fig. 5 illustrates a potential field constructed using the above equation. The well-known limitation of the generic potential method concerns with problems with dead spot or plateaus where the robot can become stranded. These problems occur when the goal-seeking robot moves to
T. Sawaragi et al. / Robotics and Autonomous Systems 31 (2000) 193–207
Fig. 5. An example of a potential field (instantiation of a “avoid-obstacle” schema).
the space that is enclosed by the obstacles, or in more general, when the forces of conflicting actions are generated from simultaneously activating schemata. Fig. 6(a) illustrates such a situation and Fig. 6(b)
Fig. 6. An example of a dead spot or plateaus.
199
shows the corresponding potential field. Arkin has embedded a mechanism for reconciliation to overcome these problems by allowing communication between the schema instantiations being motivated by psychological and neuroscientific studies. He also suggested a way of a human navigator’s intervention to the robot control system, which was restricted only when the robot fails in reconciliation and in finding the possible paths any more. This style of human–robot collaboration is typical in the conventional human–machine systems, that is, a policy of “leave what can be automated to the machine and leave what cannot be automated (or what the automation fails to do) to the human”, which is sometimes accused as “ironies of automation” [3]. Instead of Arkin’s approach, in our system we seek for an alternative style of human–machine collaboration based on the principle of shared autonomy. That is, instead of restricting the human intervention to the time of the failure of the machine autonomy, the human operator is allowed to intervene anytime indifferent from the status of the machine autonomy. The human is always “loosely” and “redundantly” connected with the machine autonomy, although his/her intervention is not forced all the time. Wherein, to some degree an error of the operator’s response is accepted (i.e., which is compensated by the machine autonomy), thus the operator’s response variability is allowed. This kind of variability is essential for enhancing learning and skill acquisition [14]. To provide a physical work environment to the operator, we designed an interface system, which consists of the 3D display information showing the status of a robot’s perceiving potential field and a command-input device transmitting the movement of the operator’s head. This design is based on the careful considerations on the fitted, naturalistic coupling between the operator’s perception from the display information and his response to that. Visual images contain too much information for a human operator to pick up only the task-relevant information. This sometimes demands to the operator an additional inferential interpretation efforts. Therefore, in our system information displayed to the operator is reconstructed by deforming the real visual information to the abstracted 3D image that may contain minimally necessary information to afford the operator’s quick responses.
200
T. Sawaragi et al. / Robotics and Autonomous Systems 31 (2000) 193–207
Fig. 7. An example of virtual 3D world constructed by VRML.
3.2. Construction of 3D display information using VRML The before-mentioned information available to the robot onboard is transmitted to the interface and is transformed into the quasi 3D display as shown in Fig. 7. This virtual world consists of the following items: • a number of cones each of which corresponds to the individual repulsive potential fields of the robot’s sensible obstacles (walls are constructed as a series of cones), • a slope of the ground expressing the attractant field toward the robot’s destination, and • a change of textures of the ground expressing the velocity of the robot’s movement. The 3D world is parametrized by positions of the cones, a slope and a texture of the ground, each of which is mapped from the specifications of the constructed potential field as well as from the robot’s sensing parameters, and is constructed in real-time using VRML. In the current version of the system, this mapping between the robot’s sensible world and the con-
structed virtual world has not yet be optimized, but is only done experientially in a trial-and-error fashion. Ideally, this mapping must be designed so that it can reveal the isomorphic mapping between the two different worlds. Displayed information changes dynamically as the robot moves around based on its machine autonomy. Since this virtual world is abstracting only the information that is needed for the steering from the actual world, the operator does not need any inferential efforts. Moreover, and what is more important, this virtual world is designed so that this can afford the operator’s naturalistic responses of his head movement. That is, the operator is navigated in a naturalistic fashion by the evolving external world of display. The operator can make quick responses within a local loop of perception and action (i.e., a loop of stimulus and response) without any need of inferential efforts. For instance, when some obstacle in front occludes another one behind that and this obstacle is of concern for the current task, the operator would look into the back of the occluding obstacle pursuing that. And when some obstacle is approach-
T. Sawaragi et al. / Robotics and Autonomous Systems 31 (2000) 193–207
201
ing to the operator with a high speed, he/she would unconsciously turn his/her face away immediately. Since obstacles approaching from far away are also in his/her peripheral view, the operator could make preparation for avoiding the collision with them while he/she is managing the navigation concerning with another obstacle that lies immediately in front. What we would like to stress here is that a human is a very skillful controller as well as a planner even when facing with the situations under severely bounded resources; he/she could manage attention focusing to pick up goal- and/or task-relevant properties out of the messy world, be able to selectively respond to the events that may harm his/her survival, and be able to manage monitoring multiple things judging appropriately the priorities among them. All of these are done at the skill level, but we are not going to transfer those into the machine. Rather, our concern is how to let those abilities coexist with the machine autonomy and to realize a human–artifact symbiotic system in which both can co-evolve through the proactive interactions, not through controlled interactions, between them.
3.3. Measuring operator’s head movement by motion capture device The operator’s head movements along the left–right direction and the front–rear direction are monitored by a motion capture device of Quick Mag, which traces the movements of the color marker, a colored ball with one inch diameter, attached on the operator’s forehead by a CCD camera. This is illustrated in Fig. 8. The measured movement is transmitted to the mobile robot and is transformed into a velocity vector. Within the machine autonomy, this vector is dealt with as externally provided steering command, and adding this to the velocity vector that is generated from the current potential fields, the robot transmits this to the low-level robot control system. Then, as far as the added velocity vector does not exceed the stability region for the robot, the operator’s movement is reflected in the robot’s movement. On the other hand, when this vector forces the robot to collide with the obstacles or walls, warning is fedback to the operator by ringing a beap. In this way, there established a shared, bilateral
Fig. 8. Measuring an operator’s head movement by a visual motion capture device.
connection between a machine autonomy and a human autonomy. Since the displayed information dynamically changes in response to the operator’s conscious or unconscious movements, it is very easy for him/her to grasp and develop the association among the perception of the virtual world, his/her own actions and their virtual sensory expectations via the machine autonomy. In other words, this work environment may contribute much to the acquisition and development of the operator’s “motor schemata”.
202
T. Sawaragi et al. / Robotics and Autonomous Systems 31 (2000) 193–207
Fig. 9. A comparison of the ways of human intervention with the machine autonomy (a) with the command inputs from the keyboard, (b) with the proposing interface system.
4. Experiments We are now beginning to evaluate the usability of our proposed system. Here we show the preliminary result of comparing the performance of the human–robot collaborative task (Fig. 9(b)) with the one using a conventional input device (Fig. 9(a)). This graph shows how the human operator behaved as a teleoperator of the semi-autonomous mobile robot under a particular situation where he/she has to avoid encountering the deadlock caused by the potential field method of the machine autonomy (Fig. 10). In Fig. 9(a) the operator is allowed to use usual command streams from the keyboard to guide a semi-autonomous robot. The vertical axis represents
the amount of stimulus measured by the sum of the absolute values of the velocity vector produced within the machine autonomy as a robot moves around. The dotted lines show the timings of a human intervention through the head movement and the command inputs. Although the appearing performance is similar with other, the ways of human intervention into the machine autonomy is quite different and the resulted relationships between them seem to be distinct; intermittent intervention and continuous intervention. Such continuous intervention by the operator plays a role of adding noise to the machine autonomy. This produces a low-magnitude random direction velocity vector that changes at random time intervals but persists sufficiently long to produce a change in the
T. Sawaragi et al. / Robotics and Autonomous Systems 31 (2000) 193–207
203
ther of those alone. We are now doing experiments by varying the tasks to the more complicated one requiring the operator simultaneous subtasks demands, and by varying the level of automation installed within the machine autonomy. Through those experiments we will analyze the emergent properties of our system that are enabled by the skill level interactions between the human and the machine autonomies. In the following we summarize a number of issues that are particular to our system design and are expected to be realized using our system. 5.1. Navigating and navigated human operator: two-way flows within the action hierarchy
Fig. 10. A configuration of the obstacles used in the experiment.
robot’s position if the robot’s velocity was not zero. Its role is to perturb the velocity of the robot slightly, removing the robot from undesirable equilibrium points that arise when the active schema instantiations counterbalance each other. However, the current result is insufficient with respect to the experimental environment. Since in both environments, the information is displayed in the same way, it is not clear how much the VRML representation contributes to the operator’s picking up goal-relevant properties out of the messy world. We have to compare these results with the one using the conventional teleoperation in which the visual information is displayed, out of which the human operator is forced to pick up information, interpret it and infer about the real world. This is under investigation.
5. Discussions In the current system there exists no more data showing the superiority of our proposing system to the conventional interface, nor the superiority of collaboration between human–machine autonomy to ei-
Hierarchical organization of the simple action mechanism shown in Fig. 2 is very profound in a sense that the mechanism at the top of this is responsible for managing the flow of information to the lower-level mechanism that actually simulate observable control activity and also for coordinating physical activities that an operator performs. In the figure, the links between the mechanisms denote information flow, which is assumed to be always active in the downward direction. This corresponds to the operator’s visual navigating task. That is, he has to first recognize the destination, then determine some waypoints to be passed and operate an robot’s movement toward each of the waypoints avoiding the neighboring obstacles, which performed by the machine autonomy. At that time the highest level action mechanism is activated each time, and based on the need for action this mechanism sends information downward to the appropriate mechanism at the next lower level. Although information always flows downward in the figure, an implicit upward flow of information does and can exist through the operator’s skill level interactions with external environment; that is, the results of actions performed by the lower level mechanism can become known to the upper-level mechanisms through the perception of the environmental changes displayed in VRML caused by those actions. This aspect corresponds to a fact that the robot does navigate the operator through the constructed display, not always the operator navigates the robot. Using the terms from the ecological fields, it can be said that the display does afford the operator’s particular actions. For instance, let us imagine a situation where after avoiding the
204
T. Sawaragi et al. / Robotics and Autonomous Systems 31 (2000) 193–207
approaching obstacles or passing through the crowded space consisting of many stationary obstacles, some free space comes into the operator’s views opportunistically. At that time, apart from whether it is a planned behavior or not, and as far as he is aware of the existence of such free space, he will unconsciously look into this space and a head movement caused by this behavior is linked to the robot’s navigation. Then, observing that the robot passes through the crowds successfully, then he could notice that it was a good behavior. In the ecological psychology, it is proposed that there is a mode in which the behavioral intention does emerge after the actions are executed, rather than before the actions are initiated. This aspect is very important since it might be impossible to make up a complete plan for the robot’s behavior due to the unexpectedness of the environment, and some style of opportunistic planning of utilizing any opportunities available at a run-time will be needed. In this sense, the hierarchy of Fig. 2 is not an absolute, tightly fixed thing at all, but must be flexible enough to be reconstructed based on the bottom-up information flow. Wherein, the deformation and/or exaggeration of the actual reality by the perceptual display of VRML can efficiently stimulate the operator to behave coordinately within the uncertain environment. 5.2. Reciprocity of influence within the human Our fundamental departure from the conventional interface design principles is that we have to deal with a human as he/she is. We have to abandon regarding a human as some mechanical component that is assumed to operate correctly and appropriately as expected by the interface designers. Rather, we have to reconsider that he/she is a living organism. Wherein, the environment influences the living organism and the living organism influences the environment. The reciprocity of influence is made possible by the evolution of a complex interior of the organism. Historically, the modeling of the human as a living organism can be classified into the three schools. Skinner thought that human behavior might be “shaped” and believed that the observed correlation between stimulus and response is necessary and sufficient for a scientific account of behavior (i.e., Skinner’s box) [15]. On the contrary, Tolman had thought that hu-
man behavior might be “cognized” by postulating unobservable constructs of the complex interior of the organism mediating between stimulus and response (i.e., Tolman’s SOR (Stimulus–Organism–Response) approach [16]). Different from those, Gibson thought that human behavior might be “tuned” toward a goal state and regarded that the organism is merely a “throughout system” meaning that stimulation must be informative about the environment in ways that a stimulus could never be. Needless to say, an ecological approach is most similar to Gibson’s view [4]. Here we would like to explicate what Gibson has called animal–environment mutuality, and others have called animal–environment duality, or reciprocity. Two fundamental reciprocities compose this more general reciprocity: the reciprocity between the interior and exterior frames of reference and the reciprocity between force (or energy) and flow (or information). To explain about these, let us introduce an ecological psychologist Shaw’s diagram shown in Fig. 11, that represents reciprocal maps of the perceiving–acting cycle of the organism [12]. Let A = detection, B = intention, C = control, and D = goal. Here B represents the formulation of an intention to seek for a particular goal D, and C represents control the organism must execute given A, the detection of goal-specific information. Wherein, states A and C act on states in the exterior frame, while conversely the pair of B and D act on states in the interior frame. A → B and C → D are information and energy from exterior states into interior states, respectively, while B → C and D → A are energy and information from interior states to exterior states, respectively. Thus, the reciprocity between force and flow is between the control of goal-directed activities (generation of forces) and the information detected over the space-time path joining the actor to its goal (generation of flows). Behaviors proceeds depending on information inflow and the controlled manner of outflow of behavior, which produces changes of inflow. These reciprocal characteristics of a human behavior are very important to the interface design for the teleoperation. The most fundamental advantages of the skill level interactions between the human and the machine autonomy is its potential to support the map from C to A via a goal G. The conventional design of the interface such as a GUI has been concentrated
T. Sawaragi et al. / Robotics and Autonomous Systems 31 (2000) 193–207
205
Fig. 11. Shaw’s perceiving–acting cycle map representing two kinds of reciprocities.
into the flow from A to C, that is, on how to support the human to detect the situation, to motivate him the proper intention, thus to take the appropriate control action. This flow of A → B → C terminates at C and there has been no linkage from C to A supported. As a result of that, the behavior is always guided by the a priori intention, and the formation of intentions are never guided by the behavior. There has been little efforts to support him to guide his goal-oriented perception out of messy world resulted by his executing control actions, which may stimulate him to form an intention posteriorly (i.e., the linkage from C → D → A → B: C ⇒ B). This corresponds to the usage of hierarchically organized action hierarchy of Fig. 2 as mentioned in the previous subsection. That is, a top-down flow within the hierarchy corresponds to the control of goal-directed activities (generation of forces), while a bottom-up flow deals with the information detected over the space-time path joining the actor to its goal (generation of flows). Based on the ecological view regarding a human as a living organism, we can conclude that the fundamental issues of the interface design for the teleoperated robots is to explicitly relate the interior degrees of freedom of the human operator to the exterior degrees of freedom existing within the environment where the robot operates. That is, as overlaid in the figure, a machine autonomy and a human autonomy correspond to the exterior frame and the interior frames, respectively.
Thus, the expected functions of the interface system correspond to the transitions among A, B, C and D; to provide a human with a work environment that consists of information inflow and the controlled manner of outflow of behavior, which produces changes of inflow. Wherein, the interface must be designed so that it can absorb the degrees of freedom (or a variability) both from the actual world and from the human operator.
6. Conclusions The current highly automated systems by themselves are apt to be a black box, but in spite of that an expert human operator of some automated equipment gets to have some valid mental model of how his equipment works. We need to assist operators in having and retaining valid mental models, particularly when the computer or automation system is complex. In this sense an interface system must be a mental model support system and its function would be to keep the human operator “tuned in” to what the machine autonomy is now doing, feeling, thinking or planning. Based on such an idea, we presented a preliminary design of an ecological interface for the teleoperated mobile robot and discussed about the foundations of the ecological interface design.
206
T. Sawaragi et al. / Robotics and Autonomous Systems 31 (2000) 193–207
As for the system that is now being implemented based on such ecological aspects, the mapping between the sensor signals available on the robot and the visual design of the perceptual world using VRML is still done just in an ad hoc way. Moreover, the tuning of the parameters needed in building a potential field from schemata are merely done in a heuristic fashion. We think the automatization of those parametric tuning will be possible using current prevailing techniques such as a neural network by obtaining the feedback from an operator’s performance. Since even in a trivial task exemplified in this paper the operation policy is different among the individual operators (i.e., how close the robot is allowed to approach to the obstacles, and so on), the machine autonomy must be able to be customized by the individuals and such run-time autotuning would be essential. Our current concern is being extended towards the collaborative knowledge acquisition between the human and the machine. There exists no rule-based and knowledge-based level behaviors installed within the machine autonomy in the current system. On the contrary, the human operator can unconsciously behave using the know-how at these levels. Therefore, if the operator’s know-how can be transferred to the machine autonomy somehow mechanically, the machine autonomy can acquire the rule-based behavior with the collaboration with the human operator. This kind of collaborative relationships among the human and the machine autonomy for acquiring the higher-level, strategic knowledge would be very promising in the future design of general human–machine systems. It would be also useful if the computer could have a mental model of the human [11]. This is not because it gets to behave as a human intends, but because the human operator gets to feel or find himself within the machine autonomy from its apparent behaviors. This is close to an idea of virtual actor called “avatar” discussed in a field of cyberspace communication. Embedding such a self-referential opportunities for a human operator in the artifact work environment would be of importance, especially when the human–machine systems turn out to be mixed initiative systems in which both humans and machines assume the role of decision making agent and the chaos of distributed interactions must be “controlled” in some way.
References [1] R.C. Arkin, Motor schema-based mobile robot navigation, in: Proceedings of the IEEE International Conference on Robotics and Automation, Raleigh, NC, 1987, pp. 261– 271. [2] M. Arbib, D. House, Depth and detours: An essay on visually guided behavior, in: M. Arbib, A. Hanson (Eds.), Vision, Brain, and Cooperative Computation, MIT Press, Cambridge, MA, 1987, pp. 139–163. [3] L. Bainbridge, Ironies of automation, Automatica 19 (6) (1983) 775–779. [4] J.J. Gibson, The Ecological Approach to Visual Perception, Houghton Mifflin Company, Boston, MA, 1979. [5] S. Hirai, Theory of shared autonomy, Journal of the Robotics Research in Japan 11 (6) (1993) 20–25 (in Japanese). [6] A. Kheddar, J.G. Fontaine, P. Coiffet, Mobile robot teleoperation in virtual reality, in: Proceedings of IMACS and IEEE/SMC Multiconference on Computational Engineering in Systems Applications: Symposium on Signal Processing and Cybernetics, Vol. L, Tunisia, 1998, pp. 104–109. [7] A. Kirlik, R.A. Miller, R.J. Jagacinski, Supervisory control in a dynamic and uncertain environment: A process model of skilled human–environment interaction, IEEE Transactions on System, Man, and Cybernetics SMC-23 (4) (1993) 929– 951. [8] J. Rasmussen, Skills, rules, knowledge, signals, signs, and symbols, and other distinctions in human performance models, IEEE Transactions on System, Man and Cybernetics SMC-13 (3) (1983) 257—266. [9] H. Rosenbrock, The future of control, in: Proceedings of IFAC 6th World Congress, Pergamon Press, Oxford, 1976. [10] W.B. Rouse, The human role in advanced manufacturing systems, in: D. Compton (Ed.), Design and Analysis of Integrated Manufacturing Systems, National Academy Press, Washington, DC, 1988. [11] T. Sawaragi, O. Katai, Resource-bounded reasoning for interface agent for realizing flexible human–machine collaboration, in: Proceedings of the Sixth IEEE International Workshop on Robot and Human Communication, Sendai, 1997, pp. 484–489. [12] R. Shaw, P. Kugler, J. Kinsella-Shaw, Reciprocities of intentional systems, in: R. Warren, A. Wertheim (Eds.), Perception and Control of Self-Motion, Lawrence Erlbaum Assoc., Hillsdale, NJ, 1990, pp. 579–619. [13] T.B. Sheridan, Supervisory control, in: G. Salvendy (Ed.), Handbook of Human Factors, Wiley, New York, 1987, pp. 1243–1263. [14] T.B. Sheridan, Human-centered automation: oxymoron or common sense?, in: Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, Vancouver, Canada. [15] B.F. Skinner, The behavior of organisms, AppletonCentry-Crofts, New York, 1938. [16] E.C. Tolman, Cognitive maps in rats and men, Psychology Review 55 (1948) 189–208.
T. Sawaragi et al. / Robotics and Autonomous Systems 31 (2000) 193–207 [17] K. Vicente, J. Rasmussen, The ecology of human–machine systems II: mediating “direct perception” in complex work domain, Ecology and Psychology 2 (3) (1990) 207–249. Tetsuo Sawaragi is an Associate Professor at the Department of Precision Engineering, Graduate School of Engineering, Kyoto University, Japan. He received his B.S., M.S. and Ph.D. degrees in Systems Engineering from Kyoto University in 1981, 1983 and 1988, respectively. From 1986 to 1994, he was an instructor at the Department of Precision Mechanics, Faculty of Engineering, Kyoto University, and in 1994 he was with the current department as an Associate Professor. From 1991 to 1992, he was a visiting scholar at Department of Engineering-Economic Systems, Stanford University, USA. He has been engaged in researches on Systems Engineering, Cognitive Science and Artificial Intelligence, particularly in the development of human–machine collaborative systems, modeling the transfer of human cognitive skills into machines. He is a member of the Society of Instrument and Control Engineers, the Institute of Systems, Control and Information Engineers, Japanese Society for Artificial Intelligence, Japan Society for Fuzzy Theory and Systems, Human Interface Society, JASME, and IEEE.
207
Takayuki Shiose was born in Osaka Prefecture, Japan on January 11, 1973. He received the B.S. and M.S. degrees in Precision Engineering from Kyoto University, Kyoto, Japan, in 1996 and 1998, respectively. He is currently engaged in the Ph.D. program at the Department of Precision Engineering, Graduate School of Engineering, Kyoto University. His current interest is in joining Engineering Philosophy and Systems Engineering, especially on the design of human–system interactions. He is a member of the Society of Instrument and Control Engineers and Human Interface Society.
Go Akashi was born in Kanagawa Prefecture, Japan on April 20, 1974. He received the B.S. degrees in the Faculty of Engineering Science from Kyoto University, Kyoto, Japan, in 1998. He is currently engaged in the M.S. program at the Department of Precision Engineering, Graduate School of Engineering, Kyoto University. His current interest is in the development of teleoperated mobile robot systems.
Foundations for designing an ecological interface for mobile robot teleoperation Tetsuo Sawaragi∗ , Takayuki Shiose, Go Akashi Department of Precision Engineering, Graduate School of Engineering, Kyoto University, Yoshida Honmachi, Sakyo, Kyoto 606-8501, Japan Received 5 December 1998; received in revised form 2 July 1999
Abstract This paper presents foundations needed in designing an interface system for mobile robot teleoperation. For this purpose, Vicente’s idea of ecological interface design (EID) is firstly reviewed, and a preliminary design of our teleoperation system is presented. Our system enables the naturalistic collaboration between a human and a robot’s autonomy at the skill level by joining their respective perception–action cycles via virtually constructed visual 3D space, in which a human naturalistic reflex between stimulus and response is effectively utilized. Following the presentation of an overview of our system, discussions on the interface design for the teleoperation are done concentrating on its ecological aspects. © 2000 Elsevier Science B.V. All rights reserved. Keywords: Human–robot systems; Ecological interface; Shared autonomy; Teleoperation
1. Introduction Recent tremendous improvements of highly advanced automated mechatro-informatic facilities have increasingly eliminated the human users out of their control loops, where the collaboration between humans and machines has been getting more and more difficult [10,13]. To overcome these difficulties, various new concepts for interface design have been proposed. As a design principle of human-friendly interface, Vicente has proposed an idea of an “ecological interface” [17]. Wherein the key issues of the interface design must be coherent with the ways of human thinking and/or perception performed under ∗ Corresponding author. Tel.: +81-75-753-5266; fax: +81-75-771-7286. E-mail address: [email protected] (T. Sawaragi)
their bounded cognitive resources. Interface must be able to stimulate a human user to perceive patterns in time and space and across the senses [14] as well as his/her intuitive leap to find innovative ways to fulfill the responsibilities [9]. Based on the above ideas, in this paper we propose a new interface system for mobile robot teleoperation based on the idea of ecological interface design. Our system aims at realizing naturalistic collaboration between a human and a robot’s autonomy by joining their respective perception–action cycles via virtually constructed visual 3D space, in which a human bodily reflex between stimulus and response is effectively utilized. This property of “embodied” interfacing capability is original in this paper and enables a human operator to acquire coordinated perception and action associations. After presenting a prototype system of our proposed interface system, we discuss a novel,
0921-8890/00/$ – see front matter © 2000 Elsevier Science B.V. All rights reserved. PII: S 0 9 2 1 - 8 8 9 0 ( 9 9 ) 0 0 1 0 8 - 6
194
T. Sawaragi et al. / Robotics and Autonomous Systems 31 (2000) 193–207
interactive style of collaboration between human and machine autonomies that is enabled by our interface system. Concerning the above, we propose an idea of “shared autonomy” as a new style of human–machine collaboration [5], which stresses that a human and a machine should collaborate with each other as equivalent partners. However, this idea still remains at the conceptual level and mainly focuses onto the aspects of flexible role allocation between the human and the machine. In this paper, we provide more functional explanations on why shared autonomy is needed and what relations between them must be expected to emerge from a perspective of ecological psychology. Then, we discuss how our proposed interface design for a teleoperated mobile robot does contribute to such emergence. 2. Ecological interface As a design principle of human-friendly interface, Vicente has proposed an idea of an “ecological interface” [17]. In this section, we first review this idea and discuss its significance in terms of the design principles of an interface for teleoperation systems. It can be generally said that it is very difficult to design styles of interactions made between the humans and their work environment composed of complex artifacts. The current complex systems to be controlled or monitored by the human operator are causing difficulties in keeping them in the control loop while maintaining their intrinsic, innate capabilities of perception–action coordination. This is because the systems come to have complex automatic control loops and devices within them, almost all of which are invisible to human operator. Moreover, since the consequences of the operator’s actions are becoming more and more critical meaning that it may bring about very high risks especially in emergency, the operator has to deliberate very carefully before initiating his actions in reply to a provided stimulus. That is, he is forced to do a “delayed” reaction, which makes it difficult to acquire perception–action coordination. To overcome such a problem, we have to develop some “aided eye” for a human operator that can stimulate his naturalistic reactions to reduce the dimensionality of their description of the system with so many components and so many interactions with
the human. That is, supporting interaction via the intrinsic perception–action cycles can lead to reducing deliberative reasoning burden of the operator and also can provide the appropriate computer support for the comparatively more laborious process of his problem solving. Concerning this, Rasmussen has proposed that there exist different cognition levels concerned with the operator’s behavioral modeling: SBB (skill-based behavior), RBB (rule-based behavior) and KBB (knowledge-based behavior) [8]. SBB is a behavior in which the specific features are experienced together frequently and the response is more or less automatic, while RBB is a procedure-oriented task including monitoring and interpreting. KBB includes the full range of problem solving and planning activity with the manipulation of some kinds of “deep” models. This is illustrated in Fig. 1. As this figure shows, in an ideal teleoperated robot system, parameters that are involved in the control and fed back to the operator may be of variety. As for those parameters Kheddar et al. have listed stability regions (i.e., distance from a particular point to an enveloping safe surface or volume), inertial forces, applied torque, slip direction and amplitude, joint limit margins, contact point and forces between the mobile robot and its environment, related obstacles distance, robot attitude relative to the environment, etc. [6]. However, in the conventional teleoperated system those parameters cannot be perceived directly by the operator, but should be reconstructed from the transmitted raw data that are shown in the display. That is, he/she has to decide the appropriate action at the levels of either of RBB or KBB, not at that of SBB. The other important characteristic of the SBB distinguished from other RBB and KBB is a property of “continuity” of behavior, which takes a form of a chain consisting of a series of reactive “perception–action” units and also enables a proficient, skillful dynamic perception. Thus, it would be desirable to develop an interface system that can stimulate an operator’s SBB by providing him/her with an “aided eye”. Another design principle of an EID advocated by Vicente is to use the means–end hierarchy for representing the work domain as a hierarchy of “nested” affordances. Here, “affordance” means a relationship between properties of the environment and properties of an organism’s action capabilities. That is, this is
T. Sawaragi et al. / Robotics and Autonomous Systems 31 (2000) 193–207
195
Fig. 1. Rasmussen’s SRK model for a teleoperator.
a relational property that specifies the match between the environmental structure and the functional abilities of an actor. Difference between the present use of affordance and Gibson’s original affordance [4] is abandoning an assumption that detection of an environmental affordance is always “direct” and unmediated. Rather, we assume that relatively simple perceptual access to the artificial stimulus of the display information is sufficient to specify the existence of an environmental affordance, thus to guide the operator’s appropriate reactions. Since in the original Vicente’s paper he dealt with the interface for a process plant, his advocated means–end hierarchy was a functional decomposition of the monitoring task within the thermodynamics domain. For our purpose of the teleoperated mobile robot domain, this means–ends hierarchy turns out to be a different one, and a particular behavior can be represented as a hierarchically arranged set of action mechanisms as shown in Fig. 2 [7]. Taking account of this nested action structure, in this paper we attempt to define the following kinds of affordance for our teleoperated mobile robot domain:
1. sighting affordance of the obstacles, 2. obstacles-evading affordance, and 3. locomoting toward destination affordance. We attempt to establish a mapping of the semantics of the above affordances onto the display of 3D virtual environment in order to navigate the operator by his/her naturalistic reflex between visual stimuli and responses to them to support interactions via the innate perception–action cycles at the level of SBB, since the cognitive process ongoing at this level is the most naturalistic in a sense that it is driven by the intrinsic reflex of stimulus and response. In the following, we design an interface system based on an idea of EID to provide an operator with a display that is “transparent” enough and a process ongoing there should be directly “exertible” by an operator in a reactive way. We extract such reactivity from operators by letting them use their own “bodies” without letting them use any artifact devices such as joysticks and mouses. This is an original aspect of our proposed interface system that is distinguished from Vicente’s implementation of EID.
196
T. Sawaragi et al. / Robotics and Autonomous Systems 31 (2000) 193–207
Fig. 2. Hierarchical organization of action mechanisms of the operator.
3. System architecture In an architecture of our proposed teleoperated robot system, a robot is a commercial product (“Pioneer” provided by Applied AI), on which ultrasonic range sensors are mounted. Steering and moving speed are controllable either by programs or externally provided commands. We do not assume a human operator to use any pointing device such as a mouse and a joystick, since those artifact tools intervene his/her naturalistic stimulus–response behaviors. The task we consider here is restricted to a very simple one: to let a robot go to some destination without colliding with walls and obstacles. The world recognized by a robot with such restricted sensors is reconstructed as a virtual 3D visual world using VRML (Virtual Reality Modeling Language) and presented to a human operator. Getting this world as a visual stimulus, an operator makes some responses of inclining a head leftward or rightward as well as in the front and in the rear. These movements are monitored by the vision-based motion capturing device and are sent to a robot that behaves reflecting an operator’s unconscious responses. Accordingly, the display will change in reaction to such an operator’s movement, and an operator is able to see this changed display from his/her moved viewpoint. This point is essential, because operators are
able to acquire direct associations between perceptional changes and their bodily actions of movements. Generally speaking, it is well recognized that a bodily reaction is more prompt and reliable than any other reactions using some instrumental devices. Especially, in our system such naturalistic reactions are derived from their head movements that are equivalent to their movements of eyes’ positions. Thus, synchronized changes of viewers’ perceptions and their postures are essential for stimulating their naturalistic interactions with the external things. This would not be possible if operators use mouses or joysticks through their hand manipulations. This system architecture is illustrated in Fig. 3. 3.1. A robot’s autonomy In an ideal teleoperated robot system, parameters that are involved in the control and fed back to the operator may be shown in Fig. 1. In our current implementing system, however, we assume that the following parameters are sensible in the local robot control subsystem: • moving speed and moving direction of the robot, • distances to and the directions of the neighboring obstacles measurable by ultrasonic range sensors, • recognized walls and distances to them, and • direction towards the destination.
T. Sawaragi et al. / Robotics and Autonomous Systems 31 (2000) 193–207
197
Fig. 3. An architecture of a teleoperated mobile robot system.
The role of robot’s local control characterizing the machine autonomy is to realize reactive/reflexive navigation of a mobile robot. For this purpose we adopt a potential field method to produce the appropriate velocity and steering commands for the robot. Arkin has proposed a method of motor schema-based mobile robot navigation, in which a schema is defined as a unit to model the interaction between robot’s perception and action [1]. Potential field method is widely utilized not only in the control of mobile robots and manipulators but also in the modeling method in brain theory and psychology; Arbib and House have developed a model for a worm acquisition by the frog in an obstacle-cluttered environment and implemented in their machine analog of Rana Computatrix [2]. In their work, they describe primitive vector fields: a prey-attractant field, a barrier-repellent field, and a field for the animal itself. These fields, when combined, yield a model of behavior that is consistent with experimental observations of the frog’s behavior. Arkin has taken analogs of these fields in the mobile robot system and introduced “move-to-goal”
and “avoid-static-obstacle” fields with the addition of new fields to describe other motor tasks such as “stay-on-path” and “avoid-moving-obstacle”. He defined a collection of schemata as templates for generating behavior, and being parametrized those generic schemata are instantiated to provide the potential actions for the control of the robot as well as their sensory expectations, thus the ways of interaction with the environment. When obstacles are detected with sufficient certainty, the motor schema associated with a particular obstacle starts to produce a force tending to move the robot away from the object (i.e., the instantiation of “avoid-static-obstacle” schema). Wherein, the degree of certainty of obstacle recognition is parametrized in a formation of the potential, which determines the velocity vector at the robot’s current location relative to the environmental object and/or the destination. In the current version of our system, objects to be distinguished are restricted to the walls and the obstacles including moving and static ones, and are recognized from a series of readings of the ultra-
198
T. Sawaragi et al. / Robotics and Autonomous Systems 31 (2000) 193–207
Fig. 4. Readings of the sensors plotted in series.
sonic range sensors as shown in Fig. 4. 1 Destination (i.e., a goal) is currently assumed to be known both to the robot and the operator, but of course it must be detected by the machine autonomy. Multiple instantiations of schemata are possible and the influences from each of the instantiated schemata are combined through a simple addition of the individual velocity vector. Thus, the specific velocity and direction for the robot can be determined at any point in time, and are transmitted to the low-level robot control system. The field equations for both the “avoid-staticobstacle” and “move-to-goal” schemata are exponential. An example showing the velocity produced by an obstacle (O) is given below: 1
Here the certainty of recognition is calculated using a Bayesian network by gathering new evidences on the changes of readings of ultrasonic range sensors confirming the existence of walls or obstacles (i.e., a probabilistic reasoning).
0 S−d − 1) ∗ G Omagnitude = (exp S−R ∞
for d > S, for R < d 6 S, for d 6 R,
where S is the sphere of influence (radial extent of force from the center of the obstacle), R is the radius of obstacle, G is the gain, d is the distance of robot to center of obstacle, and Odirection is the velocity along a line from robot to center of obstacle, moving away from the obstacle. Wherein, the parameter of gain is adjusted according to the degree of beliefs to which that obstacle is recognized by the robot. Fig. 5 illustrates a potential field constructed using the above equation. The well-known limitation of the generic potential method concerns with problems with dead spot or plateaus where the robot can become stranded. These problems occur when the goal-seeking robot moves to
T. Sawaragi et al. / Robotics and Autonomous Systems 31 (2000) 193–207
Fig. 5. An example of a potential field (instantiation of a “avoid-obstacle” schema).
the space that is enclosed by the obstacles, or in more general, when the forces of conflicting actions are generated from simultaneously activating schemata. Fig. 6(a) illustrates such a situation and Fig. 6(b)
Fig. 6. An example of a dead spot or plateaus.
199
shows the corresponding potential field. Arkin has embedded a mechanism for reconciliation to overcome these problems by allowing communication between the schema instantiations being motivated by psychological and neuroscientific studies. He also suggested a way of a human navigator’s intervention to the robot control system, which was restricted only when the robot fails in reconciliation and in finding the possible paths any more. This style of human–robot collaboration is typical in the conventional human–machine systems, that is, a policy of “leave what can be automated to the machine and leave what cannot be automated (or what the automation fails to do) to the human”, which is sometimes accused as “ironies of automation” [3]. Instead of Arkin’s approach, in our system we seek for an alternative style of human–machine collaboration based on the principle of shared autonomy. That is, instead of restricting the human intervention to the time of the failure of the machine autonomy, the human operator is allowed to intervene anytime indifferent from the status of the machine autonomy. The human is always “loosely” and “redundantly” connected with the machine autonomy, although his/her intervention is not forced all the time. Wherein, to some degree an error of the operator’s response is accepted (i.e., which is compensated by the machine autonomy), thus the operator’s response variability is allowed. This kind of variability is essential for enhancing learning and skill acquisition [14]. To provide a physical work environment to the operator, we designed an interface system, which consists of the 3D display information showing the status of a robot’s perceiving potential field and a command-input device transmitting the movement of the operator’s head. This design is based on the careful considerations on the fitted, naturalistic coupling between the operator’s perception from the display information and his response to that. Visual images contain too much information for a human operator to pick up only the task-relevant information. This sometimes demands to the operator an additional inferential interpretation efforts. Therefore, in our system information displayed to the operator is reconstructed by deforming the real visual information to the abstracted 3D image that may contain minimally necessary information to afford the operator’s quick responses.
200
T. Sawaragi et al. / Robotics and Autonomous Systems 31 (2000) 193–207
Fig. 7. An example of virtual 3D world constructed by VRML.
3.2. Construction of 3D display information using VRML The before-mentioned information available to the robot onboard is transmitted to the interface and is transformed into the quasi 3D display as shown in Fig. 7. This virtual world consists of the following items: • a number of cones each of which corresponds to the individual repulsive potential fields of the robot’s sensible obstacles (walls are constructed as a series of cones), • a slope of the ground expressing the attractant field toward the robot’s destination, and • a change of textures of the ground expressing the velocity of the robot’s movement. The 3D world is parametrized by positions of the cones, a slope and a texture of the ground, each of which is mapped from the specifications of the constructed potential field as well as from the robot’s sensing parameters, and is constructed in real-time using VRML. In the current version of the system, this mapping between the robot’s sensible world and the con-
structed virtual world has not yet be optimized, but is only done experientially in a trial-and-error fashion. Ideally, this mapping must be designed so that it can reveal the isomorphic mapping between the two different worlds. Displayed information changes dynamically as the robot moves around based on its machine autonomy. Since this virtual world is abstracting only the information that is needed for the steering from the actual world, the operator does not need any inferential efforts. Moreover, and what is more important, this virtual world is designed so that this can afford the operator’s naturalistic responses of his head movement. That is, the operator is navigated in a naturalistic fashion by the evolving external world of display. The operator can make quick responses within a local loop of perception and action (i.e., a loop of stimulus and response) without any need of inferential efforts. For instance, when some obstacle in front occludes another one behind that and this obstacle is of concern for the current task, the operator would look into the back of the occluding obstacle pursuing that. And when some obstacle is approach-
T. Sawaragi et al. / Robotics and Autonomous Systems 31 (2000) 193–207
201
ing to the operator with a high speed, he/she would unconsciously turn his/her face away immediately. Since obstacles approaching from far away are also in his/her peripheral view, the operator could make preparation for avoiding the collision with them while he/she is managing the navigation concerning with another obstacle that lies immediately in front. What we would like to stress here is that a human is a very skillful controller as well as a planner even when facing with the situations under severely bounded resources; he/she could manage attention focusing to pick up goal- and/or task-relevant properties out of the messy world, be able to selectively respond to the events that may harm his/her survival, and be able to manage monitoring multiple things judging appropriately the priorities among them. All of these are done at the skill level, but we are not going to transfer those into the machine. Rather, our concern is how to let those abilities coexist with the machine autonomy and to realize a human–artifact symbiotic system in which both can co-evolve through the proactive interactions, not through controlled interactions, between them.
3.3. Measuring operator’s head movement by motion capture device The operator’s head movements along the left–right direction and the front–rear direction are monitored by a motion capture device of Quick Mag, which traces the movements of the color marker, a colored ball with one inch diameter, attached on the operator’s forehead by a CCD camera. This is illustrated in Fig. 8. The measured movement is transmitted to the mobile robot and is transformed into a velocity vector. Within the machine autonomy, this vector is dealt with as externally provided steering command, and adding this to the velocity vector that is generated from the current potential fields, the robot transmits this to the low-level robot control system. Then, as far as the added velocity vector does not exceed the stability region for the robot, the operator’s movement is reflected in the robot’s movement. On the other hand, when this vector forces the robot to collide with the obstacles or walls, warning is fedback to the operator by ringing a beap. In this way, there established a shared, bilateral
Fig. 8. Measuring an operator’s head movement by a visual motion capture device.
connection between a machine autonomy and a human autonomy. Since the displayed information dynamically changes in response to the operator’s conscious or unconscious movements, it is very easy for him/her to grasp and develop the association among the perception of the virtual world, his/her own actions and their virtual sensory expectations via the machine autonomy. In other words, this work environment may contribute much to the acquisition and development of the operator’s “motor schemata”.
202
T. Sawaragi et al. / Robotics and Autonomous Systems 31 (2000) 193–207
Fig. 9. A comparison of the ways of human intervention with the machine autonomy (a) with the command inputs from the keyboard, (b) with the proposing interface system.
4. Experiments We are now beginning to evaluate the usability of our proposed system. Here we show the preliminary result of comparing the performance of the human–robot collaborative task (Fig. 9(b)) with the one using a conventional input device (Fig. 9(a)). This graph shows how the human operator behaved as a teleoperator of the semi-autonomous mobile robot under a particular situation where he/she has to avoid encountering the deadlock caused by the potential field method of the machine autonomy (Fig. 10). In Fig. 9(a) the operator is allowed to use usual command streams from the keyboard to guide a semi-autonomous robot. The vertical axis represents
the amount of stimulus measured by the sum of the absolute values of the velocity vector produced within the machine autonomy as a robot moves around. The dotted lines show the timings of a human intervention through the head movement and the command inputs. Although the appearing performance is similar with other, the ways of human intervention into the machine autonomy is quite different and the resulted relationships between them seem to be distinct; intermittent intervention and continuous intervention. Such continuous intervention by the operator plays a role of adding noise to the machine autonomy. This produces a low-magnitude random direction velocity vector that changes at random time intervals but persists sufficiently long to produce a change in the
T. Sawaragi et al. / Robotics and Autonomous Systems 31 (2000) 193–207
203
ther of those alone. We are now doing experiments by varying the tasks to the more complicated one requiring the operator simultaneous subtasks demands, and by varying the level of automation installed within the machine autonomy. Through those experiments we will analyze the emergent properties of our system that are enabled by the skill level interactions between the human and the machine autonomies. In the following we summarize a number of issues that are particular to our system design and are expected to be realized using our system. 5.1. Navigating and navigated human operator: two-way flows within the action hierarchy
Fig. 10. A configuration of the obstacles used in the experiment.
robot’s position if the robot’s velocity was not zero. Its role is to perturb the velocity of the robot slightly, removing the robot from undesirable equilibrium points that arise when the active schema instantiations counterbalance each other. However, the current result is insufficient with respect to the experimental environment. Since in both environments, the information is displayed in the same way, it is not clear how much the VRML representation contributes to the operator’s picking up goal-relevant properties out of the messy world. We have to compare these results with the one using the conventional teleoperation in which the visual information is displayed, out of which the human operator is forced to pick up information, interpret it and infer about the real world. This is under investigation.
5. Discussions In the current system there exists no more data showing the superiority of our proposing system to the conventional interface, nor the superiority of collaboration between human–machine autonomy to ei-
Hierarchical organization of the simple action mechanism shown in Fig. 2 is very profound in a sense that the mechanism at the top of this is responsible for managing the flow of information to the lower-level mechanism that actually simulate observable control activity and also for coordinating physical activities that an operator performs. In the figure, the links between the mechanisms denote information flow, which is assumed to be always active in the downward direction. This corresponds to the operator’s visual navigating task. That is, he has to first recognize the destination, then determine some waypoints to be passed and operate an robot’s movement toward each of the waypoints avoiding the neighboring obstacles, which performed by the machine autonomy. At that time the highest level action mechanism is activated each time, and based on the need for action this mechanism sends information downward to the appropriate mechanism at the next lower level. Although information always flows downward in the figure, an implicit upward flow of information does and can exist through the operator’s skill level interactions with external environment; that is, the results of actions performed by the lower level mechanism can become known to the upper-level mechanisms through the perception of the environmental changes displayed in VRML caused by those actions. This aspect corresponds to a fact that the robot does navigate the operator through the constructed display, not always the operator navigates the robot. Using the terms from the ecological fields, it can be said that the display does afford the operator’s particular actions. For instance, let us imagine a situation where after avoiding the
204
T. Sawaragi et al. / Robotics and Autonomous Systems 31 (2000) 193–207
approaching obstacles or passing through the crowded space consisting of many stationary obstacles, some free space comes into the operator’s views opportunistically. At that time, apart from whether it is a planned behavior or not, and as far as he is aware of the existence of such free space, he will unconsciously look into this space and a head movement caused by this behavior is linked to the robot’s navigation. Then, observing that the robot passes through the crowds successfully, then he could notice that it was a good behavior. In the ecological psychology, it is proposed that there is a mode in which the behavioral intention does emerge after the actions are executed, rather than before the actions are initiated. This aspect is very important since it might be impossible to make up a complete plan for the robot’s behavior due to the unexpectedness of the environment, and some style of opportunistic planning of utilizing any opportunities available at a run-time will be needed. In this sense, the hierarchy of Fig. 2 is not an absolute, tightly fixed thing at all, but must be flexible enough to be reconstructed based on the bottom-up information flow. Wherein, the deformation and/or exaggeration of the actual reality by the perceptual display of VRML can efficiently stimulate the operator to behave coordinately within the uncertain environment. 5.2. Reciprocity of influence within the human Our fundamental departure from the conventional interface design principles is that we have to deal with a human as he/she is. We have to abandon regarding a human as some mechanical component that is assumed to operate correctly and appropriately as expected by the interface designers. Rather, we have to reconsider that he/she is a living organism. Wherein, the environment influences the living organism and the living organism influences the environment. The reciprocity of influence is made possible by the evolution of a complex interior of the organism. Historically, the modeling of the human as a living organism can be classified into the three schools. Skinner thought that human behavior might be “shaped” and believed that the observed correlation between stimulus and response is necessary and sufficient for a scientific account of behavior (i.e., Skinner’s box) [15]. On the contrary, Tolman had thought that hu-
man behavior might be “cognized” by postulating unobservable constructs of the complex interior of the organism mediating between stimulus and response (i.e., Tolman’s SOR (Stimulus–Organism–Response) approach [16]). Different from those, Gibson thought that human behavior might be “tuned” toward a goal state and regarded that the organism is merely a “throughout system” meaning that stimulation must be informative about the environment in ways that a stimulus could never be. Needless to say, an ecological approach is most similar to Gibson’s view [4]. Here we would like to explicate what Gibson has called animal–environment mutuality, and others have called animal–environment duality, or reciprocity. Two fundamental reciprocities compose this more general reciprocity: the reciprocity between the interior and exterior frames of reference and the reciprocity between force (or energy) and flow (or information). To explain about these, let us introduce an ecological psychologist Shaw’s diagram shown in Fig. 11, that represents reciprocal maps of the perceiving–acting cycle of the organism [12]. Let A = detection, B = intention, C = control, and D = goal. Here B represents the formulation of an intention to seek for a particular goal D, and C represents control the organism must execute given A, the detection of goal-specific information. Wherein, states A and C act on states in the exterior frame, while conversely the pair of B and D act on states in the interior frame. A → B and C → D are information and energy from exterior states into interior states, respectively, while B → C and D → A are energy and information from interior states to exterior states, respectively. Thus, the reciprocity between force and flow is between the control of goal-directed activities (generation of forces) and the information detected over the space-time path joining the actor to its goal (generation of flows). Behaviors proceeds depending on information inflow and the controlled manner of outflow of behavior, which produces changes of inflow. These reciprocal characteristics of a human behavior are very important to the interface design for the teleoperation. The most fundamental advantages of the skill level interactions between the human and the machine autonomy is its potential to support the map from C to A via a goal G. The conventional design of the interface such as a GUI has been concentrated
T. Sawaragi et al. / Robotics and Autonomous Systems 31 (2000) 193–207
205
Fig. 11. Shaw’s perceiving–acting cycle map representing two kinds of reciprocities.
into the flow from A to C, that is, on how to support the human to detect the situation, to motivate him the proper intention, thus to take the appropriate control action. This flow of A → B → C terminates at C and there has been no linkage from C to A supported. As a result of that, the behavior is always guided by the a priori intention, and the formation of intentions are never guided by the behavior. There has been little efforts to support him to guide his goal-oriented perception out of messy world resulted by his executing control actions, which may stimulate him to form an intention posteriorly (i.e., the linkage from C → D → A → B: C ⇒ B). This corresponds to the usage of hierarchically organized action hierarchy of Fig. 2 as mentioned in the previous subsection. That is, a top-down flow within the hierarchy corresponds to the control of goal-directed activities (generation of forces), while a bottom-up flow deals with the information detected over the space-time path joining the actor to its goal (generation of flows). Based on the ecological view regarding a human as a living organism, we can conclude that the fundamental issues of the interface design for the teleoperated robots is to explicitly relate the interior degrees of freedom of the human operator to the exterior degrees of freedom existing within the environment where the robot operates. That is, as overlaid in the figure, a machine autonomy and a human autonomy correspond to the exterior frame and the interior frames, respectively.
Thus, the expected functions of the interface system correspond to the transitions among A, B, C and D; to provide a human with a work environment that consists of information inflow and the controlled manner of outflow of behavior, which produces changes of inflow. Wherein, the interface must be designed so that it can absorb the degrees of freedom (or a variability) both from the actual world and from the human operator.
6. Conclusions The current highly automated systems by themselves are apt to be a black box, but in spite of that an expert human operator of some automated equipment gets to have some valid mental model of how his equipment works. We need to assist operators in having and retaining valid mental models, particularly when the computer or automation system is complex. In this sense an interface system must be a mental model support system and its function would be to keep the human operator “tuned in” to what the machine autonomy is now doing, feeling, thinking or planning. Based on such an idea, we presented a preliminary design of an ecological interface for the teleoperated mobile robot and discussed about the foundations of the ecological interface design.
206
T. Sawaragi et al. / Robotics and Autonomous Systems 31 (2000) 193–207
As for the system that is now being implemented based on such ecological aspects, the mapping between the sensor signals available on the robot and the visual design of the perceptual world using VRML is still done just in an ad hoc way. Moreover, the tuning of the parameters needed in building a potential field from schemata are merely done in a heuristic fashion. We think the automatization of those parametric tuning will be possible using current prevailing techniques such as a neural network by obtaining the feedback from an operator’s performance. Since even in a trivial task exemplified in this paper the operation policy is different among the individual operators (i.e., how close the robot is allowed to approach to the obstacles, and so on), the machine autonomy must be able to be customized by the individuals and such run-time autotuning would be essential. Our current concern is being extended towards the collaborative knowledge acquisition between the human and the machine. There exists no rule-based and knowledge-based level behaviors installed within the machine autonomy in the current system. On the contrary, the human operator can unconsciously behave using the know-how at these levels. Therefore, if the operator’s know-how can be transferred to the machine autonomy somehow mechanically, the machine autonomy can acquire the rule-based behavior with the collaboration with the human operator. This kind of collaborative relationships among the human and the machine autonomy for acquiring the higher-level, strategic knowledge would be very promising in the future design of general human–machine systems. It would be also useful if the computer could have a mental model of the human [11]. This is not because it gets to behave as a human intends, but because the human operator gets to feel or find himself within the machine autonomy from its apparent behaviors. This is close to an idea of virtual actor called “avatar” discussed in a field of cyberspace communication. Embedding such a self-referential opportunities for a human operator in the artifact work environment would be of importance, especially when the human–machine systems turn out to be mixed initiative systems in which both humans and machines assume the role of decision making agent and the chaos of distributed interactions must be “controlled” in some way.
References [1] R.C. Arkin, Motor schema-based mobile robot navigation, in: Proceedings of the IEEE International Conference on Robotics and Automation, Raleigh, NC, 1987, pp. 261– 271. [2] M. Arbib, D. House, Depth and detours: An essay on visually guided behavior, in: M. Arbib, A. Hanson (Eds.), Vision, Brain, and Cooperative Computation, MIT Press, Cambridge, MA, 1987, pp. 139–163. [3] L. Bainbridge, Ironies of automation, Automatica 19 (6) (1983) 775–779. [4] J.J. Gibson, The Ecological Approach to Visual Perception, Houghton Mifflin Company, Boston, MA, 1979. [5] S. Hirai, Theory of shared autonomy, Journal of the Robotics Research in Japan 11 (6) (1993) 20–25 (in Japanese). [6] A. Kheddar, J.G. Fontaine, P. Coiffet, Mobile robot teleoperation in virtual reality, in: Proceedings of IMACS and IEEE/SMC Multiconference on Computational Engineering in Systems Applications: Symposium on Signal Processing and Cybernetics, Vol. L, Tunisia, 1998, pp. 104–109. [7] A. Kirlik, R.A. Miller, R.J. Jagacinski, Supervisory control in a dynamic and uncertain environment: A process model of skilled human–environment interaction, IEEE Transactions on System, Man, and Cybernetics SMC-23 (4) (1993) 929– 951. [8] J. Rasmussen, Skills, rules, knowledge, signals, signs, and symbols, and other distinctions in human performance models, IEEE Transactions on System, Man and Cybernetics SMC-13 (3) (1983) 257—266. [9] H. Rosenbrock, The future of control, in: Proceedings of IFAC 6th World Congress, Pergamon Press, Oxford, 1976. [10] W.B. Rouse, The human role in advanced manufacturing systems, in: D. Compton (Ed.), Design and Analysis of Integrated Manufacturing Systems, National Academy Press, Washington, DC, 1988. [11] T. Sawaragi, O. Katai, Resource-bounded reasoning for interface agent for realizing flexible human–machine collaboration, in: Proceedings of the Sixth IEEE International Workshop on Robot and Human Communication, Sendai, 1997, pp. 484–489. [12] R. Shaw, P. Kugler, J. Kinsella-Shaw, Reciprocities of intentional systems, in: R. Warren, A. Wertheim (Eds.), Perception and Control of Self-Motion, Lawrence Erlbaum Assoc., Hillsdale, NJ, 1990, pp. 579–619. [13] T.B. Sheridan, Supervisory control, in: G. Salvendy (Ed.), Handbook of Human Factors, Wiley, New York, 1987, pp. 1243–1263. [14] T.B. Sheridan, Human-centered automation: oxymoron or common sense?, in: Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, Vancouver, Canada. [15] B.F. Skinner, The behavior of organisms, AppletonCentry-Crofts, New York, 1938. [16] E.C. Tolman, Cognitive maps in rats and men, Psychology Review 55 (1948) 189–208.
T. Sawaragi et al. / Robotics and Autonomous Systems 31 (2000) 193–207 [17] K. Vicente, J. Rasmussen, The ecology of human–machine systems II: mediating “direct perception” in complex work domain, Ecology and Psychology 2 (3) (1990) 207–249. Tetsuo Sawaragi is an Associate Professor at the Department of Precision Engineering, Graduate School of Engineering, Kyoto University, Japan. He received his B.S., M.S. and Ph.D. degrees in Systems Engineering from Kyoto University in 1981, 1983 and 1988, respectively. From 1986 to 1994, he was an instructor at the Department of Precision Mechanics, Faculty of Engineering, Kyoto University, and in 1994 he was with the current department as an Associate Professor. From 1991 to 1992, he was a visiting scholar at Department of Engineering-Economic Systems, Stanford University, USA. He has been engaged in researches on Systems Engineering, Cognitive Science and Artificial Intelligence, particularly in the development of human–machine collaborative systems, modeling the transfer of human cognitive skills into machines. He is a member of the Society of Instrument and Control Engineers, the Institute of Systems, Control and Information Engineers, Japanese Society for Artificial Intelligence, Japan Society for Fuzzy Theory and Systems, Human Interface Society, JASME, and IEEE.
207
Takayuki Shiose was born in Osaka Prefecture, Japan on January 11, 1973. He received the B.S. and M.S. degrees in Precision Engineering from Kyoto University, Kyoto, Japan, in 1996 and 1998, respectively. He is currently engaged in the Ph.D. program at the Department of Precision Engineering, Graduate School of Engineering, Kyoto University. His current interest is in joining Engineering Philosophy and Systems Engineering, especially on the design of human–system interactions. He is a member of the Society of Instrument and Control Engineers and Human Interface Society.
Go Akashi was born in Kanagawa Prefecture, Japan on April 20, 1974. He received the B.S. degrees in the Faculty of Engineering Science from Kyoto University, Kyoto, Japan, in 1998. He is currently engaged in the M.S. program at the Department of Precision Engineering, Graduate School of Engineering, Kyoto University. His current interest is in the development of teleoperated mobile robot systems.