A formal description of tactical plan recognition

Information Fusion 4 (2003) 47–61 www.elsevier.com/locate/inffus

A formal description of tactical plan recognition Frank Mulder a

a,b,*

, Frans Voorbraak

c

Section Communications Research & Semiotics, Faculty of General Sciences, University Maastricht, Grote Gracht 82, P.O. Box 616, 6200 MD Maastricht, The Netherlands b Thales Nederland B.V., Zuidelijke havenweg 40, P.O. Box 42-7550 GD Hengelo(Ov), The Netherlands c Academisch Medisch Centrum, University of Amsterdam, Meibergdreef 15, 1105 AZ Amsterdam, The Netherlands Received 26 July 2001; received in revised form 20 August 2002; accepted 21 August 2002

Abstract Plan recognition can roughly be described as the problem of finding the plan(s) underlying the observed behaviour of agent(s). Of course, usually, the observed behaviour and available background knowledge does not determine the underlying plan, and therefore one can typically at best generate (reasonable) plan hypotheses. Traditionally, plan recognition has been studied, formalized and implemented in areas like story understanding and user modelling. In this paper, we propose a formal definition of tactical plan recognition, i.e. the recognition of enemy plans. We will focus on military applications, where this task of tactical plan recognition is crucial, but this task is relevant for every application where one has to deal with intelligent adversial agents. Tactical plan recognition differs from traditional plan recognition in a number of ways. For example, an enemy will often try to avoid making his plans known. We will not pay much explicit attention to this feature. We will focus on another important characteristic feature of tactical plan recognition, namely that the identity of the observed enemy objects, for which plans are to be recognized, may be unknown. A consequence of this is that it is typically not known which observations originate from the same objects. Our formalization of plan recognition is based on classical abduction. The concepts of classical abduction can readily be applied to plan recognizers for identified observations, as has been done by Lin and Goebel [18] and Bauer and Paul [7]. However, for tactical plan recognition some adaptations have to be made. Here the plan recognizer will not only have to generate plan hypotheses, but also assignment hypotheses, which correspond to formal links of objects to observations. A choice for an assignment is essentially a decision concerning the question which observations originate from the same objects. For observations with stochastic variables the probability of an assignment hypothesis is calculated, rather than the probability of the plan hypotheses. For this, ReidÕs multiple hypothesis tracking formula can be adapted to calculate the assignment hypothesis probability.
2002 Elsevier Science B.V. All rights reserved. Keywords: Tactical plan recognition; Multiple hypothesis tracking; Abduction

1. Introduction Recognizing the plan(s) underlying the observed behavior of agent(s) is an important problem in many contexts. Traditionally, automatic plan recognition has been applied in areas like story understanding and user modelling. In this paper, we propose a formal definition of tactical plan recognition, i.e. the recognition of enemy *

Corresponding author. Address: Thales Nederland B.V., Zuidelijke havenweg 40, P.O. Box 42-7550 GD Hengelo(Ov), The Netherlands. Fax: +31-74-2484077. E-mail addresses: [email protected] (F. Mulder), [email protected] (F. Voorbraak).

plans. We will focus on military applications, where tactical plan recognition plays a crucial role, but obviously this task is relevant whenever one has to deal with adversial agents. Tactical plan recognition differs from traditional plan recognition in a number of ways. For example, an enemy will often try to avoid making his plans known. One could try to explicitly capture this adversial behavior in a game-theoretical setting, but we will assume that it is tackled implicitly. For example, the considered plan templates should take into account that the enemy tries to reduce the probability of detection. This paper concentrates on another important characteristic feature of tactical plan recognition, namely that the identity of

1566-2535/02/$ - see front matter
2002 Elsevier Science B.V. All rights reserved. PII: S 1 5 6 6 - 2 5 3 5 ( 0 2 ) 0 0 1 0 2 - 1

48

F. Mulder, F. Voorbraak / Information Fusion 4 (2003) 47–61

the observed enemy objects may be unknown. This feature implies that it is typically not known which observations originate from the same objects. Our formalization of plan recognition is based on classical abduction. The concepts of classical abduction can readily be applied to plan recognizers for identified observations, as has been done by [7,18]. The main theoretical contribution of this paper is our proposed adaptation of this approach to the case of plan recognition with observations of unidentified objects. We formally define plan recognition in this more general context, where in addition to plan hypotheses, one also has to generate assignment hypotheses. The assignment hypotheses correspond to formal links of objects to observations. We illustrate our proposal by working out a simple example of plan recognition in a tactical situation. Although the example is very simple, we hope it makes clear how having knowledge about possible plans of objects allows for much longer predictions of the (likely) future behavior of the objects than can be obtained by tracking approaches which only take the kinematic state of the objects into account. In the remainder of this introduction we give a brief overview of plan recognition applications, discuss plan recognition in the military context, introduce a simple example illustrating plan recognition in a tactical situation, and briefly discuss related work. The rest of the paper is built up as follows: In Sections 2 and 3, we make more precise which notions of plan and plan recognition are used in this paper. Section 4 discusses different types of uncertainty relevant for the planning problem. In Section 5, we give our definition of plan recognition with observations of unidentified objects, and we illustrate our approach using the simple example introduced in the introduction. Finally, the conclusions are given in Section 6. 1.1. An overview of plan recognition applications Plan recognition was first recognized as a problem as such, by Schmidt [27], in work on the believer system. The first attempt to find a generalized frame work was in KautzÕs generalized plan recognition paper, [16], that was based on set minimization and circumscription. Here plan templates consisted of an event hierarchy, that links observations of events to a hierarchy of plans. KautzÕ application was a pasta cooking problem. An important link with classical abduction (that applies classical predicate logic) was recognized by Lin and Goebel [18]. Consequently techniques from machine learning were applied by Bauer [6] and techniques from inductive concept learning were applied to plan recognition by Lesh and Edzioni [17]. A view that recognizing plans is much like parsing a text was given by Sidner [28] and further explored by Villain [31]. Villain established a

correspondence between a restricted KautzÕ framework and a grammatical formalism. But a purely logical framework gives a large number of hypotheses. Therefore several authors have applied general techniques for reasoning about uncertainty to plan recognition in order to rank the hypotheses and to select the best hypotheses. Bayesian nets were first applied to plan recognition by Calistri-Yeh [10] and Charniak and Goldman [13]. Charniak applied his earlier research in such nets. They where followed by Pynadath and Wellman [23] and several other authors. Dempster– Shafer theory was applied to plan recognition in the area of natural language understanding by Carberry [12] and by Bauer [5] in the area of intelligent user interfaces. Plan recognition has been around for some time now and a wealth of research results are available, posing solutions for a number of applications. The most important applications of plan recognition are: • Story understanding: In a story, written descriptions of the state of the world and of the storyÕs agentsÕ actions are described. In story understanding a program attempts to infer the plans that, when executed, may cause the descriptions. Once the plan is recognized the story understanding program can produce a paraphrase or a summary of a story and answer questions about the story. • Psychological modelling: See e.g. the publication of Schmidt, [27], that explores plan recognition as an intersection between psychology and artificial intelligence. • Automated driving: In a car driving environment, the driving agents have to coordinate their driving plans with each other, in order to prevent collisions. The coordination relies on observations of other drivers actions: car movements and signals. Based on these observations a driver recognizes driverÕs plans and modifies itÕs own driving plans to it. • Natural language understanding: Here verbal descriptions of actions and the world are available and the plan recognizer constitutes a plan that explains the utterances of the human. • Intelligent human–computer interfaces: Here the plan recognizer attempts to infer the userÕs plan(s) by his/ her interactions with the computer. When the interface program knows the userÕs plan(s), it can give advice to the user on how to attain the goals of the plan. • Program understanding: This is reverse engineering of computer program code. The plan behind a piece of code is recognized and thus its goals and effects are understood. This area differs from the previous applications in that all actions (program statements) are available to the plan recognizer. • Multi-agent coordination: Huber and Durfee [15], first applied plan recognition for collaborative planning.

F. Mulder, F. Voorbraak / Information Fusion 4 (2003) 47–61

• Tactical plan recognition: This is, off course, the subject of this article. Currently there is a fairly large body of research concerning plan recognition and plan recognition applications. However there is not yet any known application which that has found some widespread commercial or otherwise use.

Military World

49 C&C system

Military World

Model maintenance retrospective/ prospective/ current partial state

models

enemy plan recognition

models models own force plan

own force (reactive) planning

operators

best plan hyp(s) mission hyp best plan hyp

mission determine mission

1.2. Plan recognition in the military world Fig. 1. Planning and plan recognition in a military C&C System.

Command and control (C&C) systems support human operators in managing the operations of civil or military agencies. But these systems are mostly developed and applied for military applications (naval, army, air force applications). Nevertheless in some civil agencies, such as the police and the fire-department, C&C type systems are also applied. In a military C&C world own forcesÕ agents (an agent being an object that acts ÔintelligentlyÕ) cooperate to achieve some shared military goals under some constraints (the goals and constraints of a force are stated in a mission). They are opposed in their effort by enemy agents. Military research in the area of C&C systems concerns models of warfare, which directly model a military conflict, as well as methods of attaining an understanding or assessment of the tactical situation and finally it concerns management aids for military conflicts or logistics problems, given the understanding of the tactical situation. In the latter case one will find tactical plan recognition. For military commanders, it is of paramount importance to have a correct assessment or understanding of the state of the actual and future military world. This state of the military world is called a tactical situation and it describes: • own forcesÕ and enemy forcesÕ objects (vehicles, airplanes, military bases, sensors and weapons) and their position, state and relationships, • human agents (troops), their position and their relationships (military hierarchical organization, e.g. platoon, regiment and spatial and temporal relations) and • the plan behind their operations, or in military terms their operational plan or Ôcourse of action (COA)Õ, • the environment: relevant artifacts (e.g. bridges, roads, towns and cities) and natural objects (e.g. rivers, hills and mountains) and their relations in the military world, • own forceÕs reactions on enemy plans and an estimate of their successfulness. The planning and plan recognition tasks and their interaction in a C&C system is given in Fig. 1, from [20], and has two main components: enemy plan recognition and own forces planning. The C&C system attempts to

understand a tactical situation, which it perceives by its sensors and from intelligence messages about enemy forcesÕ objects, by deriving hypothetical enemy forcesÕ plans which explain their observed actions and observed states. Based on these hypothetical enemy forcesÕ plans, own forcesÕ plans are devised in-operation (during the execution of the own forcesÕ plan) or pre-operation (before execution of the own forcesÕ plan). Both tasks are supported by two generic tasks: model maintenance and determine mission. Model maintenance keeps the models of the outside world up-to-date. Determine mission discovers hypothetical enemy missions (goals and constraints) during plan recognition and aids in developing own forcesÕ missions during own forcesÕ planning. For each hypothetical plan one can, preoperation, compose own forcesÕ plans which counter the expected enemy forcesÕ plans and also achieve the own forcesÕ goals. But own forcesÕ planning can also be done in-operation, so during the execution of an own forcesÕ plan this plan is adapted to the actual situation. Thus own forcesÕ planning is an instance of the problem class reactive planning. In this context reactive planning stands for modifying an already being executed plan to the actual situation during plan execution. The military world is a professional world which is a part of a real world. Note that a professional world differs from a real world. A real world is fairly complex and may entail common sense reasoning, which is not well defined. In the case of a professional world, the objects and agents and their relations and states and state-changes are well defined. Therefore the professional knowledge can be made explicit and it is possible to develop computer applications for problems here. Real world planning problems and therefore also real world plan recognition problems differ significantly from toy world (e.g. the blocks world or a most notable toy world in the area of plan recognition is the pasta cooking example of Kautz [16]). First, the real world is inherently distributed. Some planning problems concern distributed resources and agents and/or are so complex (intricate, large scaled), that they have to be solved by a number of distributed, cooperating agents. Here coordination of the activities of the agents is required, but

50

F. Mulder, F. Voorbraak / Information Fusion 4 (2003) 47–61

allowing for sufficient autonomy for the agents, so they can operate efficiently. Second, the real world is dynamic: during plan execution unexpected events can change the world such that a plan part may no longer be applicable or sufficiently (near) optimal: it contains conflicts or it should be improved. Here reactive planning is required, where the development of a plan part is postponed to incorporate the most certain information about the world. To handle uncertainties, the world is monitored for relevant events or states during the plan execution. If a plan part is not applicable any more or might be improved significantly, it is adjusted to the new situation. The above paragraph also characterizes the world of tactical planning and plan recognition, but here also some additional problems occur. These are listed below: 1. Multiple agents and objects, independent or interacting. In the naval domain the number of objects and agents may be in the range of 1 till 100. In larger scaled army conflicts the number of objects and agents to be considered may be several thousands. 2. Multiple independent plans or related plans. At the same time several plans may be executed by the combatants of the military world. The executed plans may be part of a larger plan, but may also be independent of each other. 3. Covertness, the enemy forces will try to cover their operations as much as possible. Also the sensors and observers only have a limited view of the objects and agents in the military world. This results in various forms of uncertainty about the observations: (a) an uncertain identification of the observation. An identification concerns an unique identification of the observed objects/agents. (b) Uncertainty about the truth of an observation. The observation itself may only have a limited degree of belief: Positive, Probable, NotProbable, Confirmed. In the last case the observation is confirmed by another, reliable source. (c) Uncertainty in the parameters of the observation. The parameters concern attributes of the object(s): itÕs radar cross-section, shape, etc. and the state vector (position, speed) of the object or parameters of actions that are performed by the object. 4. The observations originate from various different observers and sensors. The plan recognizer has to fuse observations from all these sensors and sensor data fusion techniques have to be applied. 5. Feints: The enemy forces will perform feints in order to mislead the own forces plan recognizers. 6. Plans may be executed erroneously, due to bad communications or wrong identification of enemy objects. Even fratricide may occur in the military world.

7. Partial or no achievement of goals. During the execution of a plan, the goals of a plan may not be achieved at all or only partially. It is not yet feasible for us to design a system that accounts for all these items. In this report we formalize the uncertainties in the observations. However we will define a system that can handle observations with uncertain identification stemming from multiple observers. Feints and partial achievement of goals can be modelled by accounting for these contingencies in the enemy doctrines that are applied. Erroneous execution is not addressed here, and may not be handled correctly by our plan recognizer. 1.3. An example of tactical plan recognition In this section we will introduce a tactical situation and see how it develops in a number of stages. Sequentially some observations of enemy helicopters become available. The plan recognizer tries to state a number of assignment hypotheses. The assignment hypothesis is a unique relation between the set of the observations and the set of plan hypotheses. An assignment hypothesis is based upon previous assignment hypotheses and a new observations. We will see that the assignment hypotheses evolve through time and after the last observations the plan recognizer is quite certain about the enemy plan. Consider the following example of tactical plan recognition from tactical support of ground troops by armed helicopters. (1) In Fig. 2, a tactical situation is displayed. The symbols of this figure are given in the legenda, Fig. 3. In the middle of the figure there is the line representing the front of the battle field, a line dividing the combatants: the red enemy forces and the blue own forces. To the left of the front are the red forces with an air base (indicated by the rectangular icon) as unit 1, a forward air refuelling point (FARP) below the air base as unit 2 (indicated by the circular icon), and close to the front a communication unit, unit 3, (indicated by the triangular icon). To the right of the front are the blue forces consisting of

MUN 1

3

4 6

FARP

2 5

Fig. 2. The example tactical situation.

7

F. Mulder, F. Voorbraak / Information Fusion 4 (2003) 47–61

51 A(H1,MUN)

Air force base

Tank unit

MUN 7

1

T1

1 3

Observation at time T1

T1

6

FARP 2

FARP

4

T2

Munitions depot

MUN

FARP

A(H2,TK5)

Communications Unit 5

Fig. 3. Symbols of the example of a tactical situation. Fig. 5. The example: two observations.

three tank units (units 4, 5, 6) and a munitions depot (indicated by the circular icon) as unit 7. Ground support entails support of battle helicopters and aircraft for own forcesÕ ground troops. It concerns attacks on enemy ground troops near or at the front. A ground support plan consists of the following actions: (1) fly from an air base or a forward refuelling point towards a communication unit, (2) receive target information from the communication unit and (3) ingress, attack and egress to a forward refuelling point or an air base. The task now is to recognize red ground support operations. (2) In the following Fig. 4, there is a first observation (O1 ) of an red armed helicopter. The arrow indicates the direction of flight. In Fig. 8, the possible plan hypotheses are presented in a tree. For the first observation there are two plan hypotheses: (1) the helicopter of O1 , H1, attacks the munitions depot: A(H1,MUN), (2) H1 attacks the tank unit 5: A(H1,TK5). Other possible targets than TK5 and MUN are not considered for simplicity of the example. (3) In Fig. 5, a second observation (O2 ) enters the plan recognition system. Now there are various hypotheses with regard to the origin of the observation. It might be H1 of the first observation O1 or it might be a second helicopter H2. The helicopter of the second observation will only attack the tank unit 5: A(H2,TK5). Three hypotheses can occur: (1) in Fig. 5, the solid arrows represent the plan hypotheses: A(H1,MUN), A(H2,TK5), (2) in Fig. 6, the solid arrows represent the plan hypotheses: A(H1,TK5), A(H2,TK5), (3) finally when the observations both stem from H1: A(H1,TK5),

MUN 7

1

T1 4

3 T2 FARP

6

A(H1,TK5)

2 A(H2,TK5)

5

Fig. 6. The example: two observations.

the dashed arrow in Fig. 6. These hypotheses can also be found in the tree of Fig. 8 above O2 . (4) In Fig. 7, a final third observation is entered. The observation matches very well with the plan of H1 attacking tank unit 5. At this stage there is very little support for the other hypotheses that are possible. All possible hypotheses are illustrated in Fig. 8. In the Table 1 the assignment hypotheses, that can occur when observations are entered into the system, are given. 1.4. Related work The first publications on tactical plan recognition stem from Azarewicz et al., [3,4]. He developed goal/ plan templates which describe enemy doctrines. When

MUN

A(H1,MUN)

MUN

7

1

T1

7 1

T1 3

4

T2

3

4 FARP

FARP

6 2

A(H1,TK5)

6

T3

2 5

5

Fig. 4. The example: one observation.

Fig. 7. The example: three observations, assume observations from one helicopter.

52

F. Mulder, F. Voorbraak / Information Fusion 4 (2003) 47–61 3,1 A(H3,TK5) A(H1,MUN),A(H2,TK5),A(H3,TK5) 2,1 A(H2,TK5) 3,2

A(H1,MUN),A(H2,TK5)

A(H2,TK5) 1,1 3,3

A(H1,TK5),A(H2,TK5),A(H3,TK5)

A(H1,MUN) A(H3,TK5) 3,4 A(H1,TK5),A(H2,TK5) A(H2,TK5) 2,2

A(H1,TK5)

3,5 A(H1,TK5) 1,2

A(H1,TK5),A(H2,TK5)

A(H2,TK5) 3,6 A(H1,TK5),A(H3,TK5) A(H3,TK5) A(H1,TK5) 2,3

O1

O2

3,7 A(H1,TK5) A(H1,TK5) O3

Fig. 8. The example: the hypothesis tree.

Table 1 Assignments in the example Assignment

Elements of assignment

O1 W1;1 W1;2

O1 ! A(H1,MUN) O1 ! A(H1,TK5)

O2 W2;1 W2;2 W2;3

O1 ! A(H1,MUN), O2 ! A(H2,TK5) O1 ! A(H1,TK5), O2 ! A(H2,TK5) O1 ! A(H1,TK5)

O3 W3;1 W3;2 W3;3 W3;4 W3;5 W3;6 W3;7

O1 ! A(H1,MUN), O2 ! A(H2,TK5), O3 ! A(H3,TK5) O1 ! A(H1,MUN), O2 ! A(H2,TK5) O1 ! A(H1,TK5), O2 ! A(H2,TK5), O3 ! A(H3,TK5) O1 ! A(H1,TK5), O2 ! A(H2,TK5) O1 ! A(H1,TK5), O2 ! A(H2,TK5) O1 ! A(H1,TK5), O3 ! A(H3,TK5) O1 ! A(H1,TK5)

receiving the first observations, goal/plan templates are instantiated to a set of active goal/plan hierarchies. His hierarchy consists of top goals and plans. The plans are elaborated in multi-agent plans, consisting of a set of hypothetical agent-role matrices and a multi-agent temporal dependence matrix. At the lowest level are single agent plans positioned. The following observations are used bottom up to determine the support for observable conditions for various agent roles in the agent-role matrix. Like much work of that time it is based on psychological models and applies heuristic numerical support calculations. In our work there is a formal definition of input and the output of the plan recognizer. There are more plan

hypotheses generated and the ranking of assignment hypotheses is based on probability theory, rather than heuristics. In the NATO Data Fusion Demonstrator project (DFD) it was attempted to do army tactical plan recognition, though this was not realized by itÕs designers. The most interesting result from DFD is a concise set of army plan templates, that can be found in [32,33]. The DFD first attempts to correlate observations to recognize sets of observations that stem from the same object. Then it calculates the support of such a set for some plan hypotheses for the setÕs object. However the correlator has no knowledge of plan hypotheses nor does it account for a new plan hypothesis for each new observation. The correlator cannot use a hypothetical plan for prediction of a future state or future action of an object or a number of objects. Thus it has insufficient information to do medium term predictions (e.g. several minutes to several hours). In our example above the DFD system would only produce the one plan hypothesis of Fig. 7. It would not consider explicitly multiple targets causing the observations. In the US some research followed that of Azarewitz, e.g. [22]. But this research was only presented in US restricted conferences, such as the Tri-Service Data Fusion Symposium. The bayesian net plan recognition method of Charniak and Goldman is also applied to enemy plan recognition in the US, [8], but no unrestricted publications are available. But the work of Charniak employed bayesian networks to support a plan hypothesis and not an assignment hypothesis, as is done in this report. Canamero, [11], proposed a knowledge-level model (i.e. a generic task model) for keyhole plan recognition. It differs on the work of Azarewicz and the DFD project in that it gives a formal generic task model. But, again, it only generates plan hypotheses instead of assignment hypotheses. Ross in [25], used Hidden Markov Models to extract the Course of Action (COA) of a number of units from their spatial order. Again this concerns only one particular plan hypothesis. In work on military distributed simulation (DIS) such as that of Tambe and Rosenbloom, [29,30] applies plan recognition for a software agent. Here actions as well as events (non-intentional) of other agents are tracked. They focus on complex plans, but assume an input of tracks with 100% track continuity. A track continuity of 100% is not always achieved by a tracking function. In research on computer games, such as RoboCup, one is also faced with adverserial parties. However here the identity of the observed agents is clear. Sets of observations that concern one and the same agent are easily identified and from these sets the support for a plan hypothesis of one agent can be calculated directly.

F. Mulder, F. Voorbraak / Information Fusion 4 (2003) 47–61

Our work is similar to that of Mevassvik [19]. Here the input are intelligence reports and tracks and an actual set of flight/sail plans in an area. He then associates these reports to the plan, applying a statistical distance measure between observations and plans to associate them. In the association he also considers various constraints. However in the case of tactical plan recognition the enemy plans should be unknown and plan hypotheses are created and maintained for each observation. Our work integrates plan recognition with tracking algorithms. A traditional tracker, see e.g. Blackman [9], only attributes limited information to an observed object: kinematic state and friend–foe identification. It can only predict the state of an object over a few seconds, before the uncertainty of the prediction becomes very large. Our plan recognition follows traditional tracking naturally. ItÕs input are tracks (that often have a limited track continuity) and intelligence from human observers. The tracking information is augmented with an agentÕs plans and intensions. Longer term prediction of the agentÕs state and actions is now possible.

2. Plans and plan templates The plans to be recognized are described in plan templates. A simple example of a plan template is that of flight KL175 from Frankfurt to Heathrow: 8xðFlightðKL175; xÞ () FlyðFrankfurt; Amsterdam; xÞ ^ StopðAmsterdam; xÞ ^ FlyðAmsterdam; Heathrow; xÞ A more complex example of plan templates are STRIPS operators of partial order planning. Information about STRIPS planners can be found in textbooks like [26]. The STRIPS operator still contains some free variables and only a partial order of the steps of the plan may be specified. A partial plan entails multiple complete plans and therefore is a more efficient representation than an enumeration of itÕs complete plans. The characteristics of the planning world (a computer userÕs plans, military world, etc.) determine the characteristics of the plans that are executed there. The following plan characteristics also have a great impact on the plan recognizer: 1. Single plan/multiple plans: Whether the plan recognizer can only recognize one plan at hand or multiple independent plans. 2. Hierarchical/non-hierarchical: Whether one or more plan steps are composed of a number of lower level steps. In this report, hierarchical plans are not considered.

53

The background knowledge can be considered to be exhaustive. If a model of the actions is an exhaustive description of the ways actions can be executed, then two closed world assumption can be applied (from Azarewicz [3]). 1. The known ways of performing an action are the only ways to perform the action. 2. All actions are purposeful and the assumption that actions can be distinguished from events and that all possible reasons (plans) for performing an action are known. A plan template is always partial: it represents a wide range of complete plans. But also it encompasses reasonable variations in the execution of a plan and reasonable variations in the observation of actions resulting in probabilistic, uncertain or unidentified observations. Now one has to decide about using partial or complete plans in the plan hypotheses, and thus about whether or not to interleave planning and plan recognition. The sequence of actions of a plan hypothesis can be incomplete and itÕs variables may be partially instantiated. As new observations enter the plan recognition system, the operators are unified with the observations and thus substitutions for the variables become available. When a new observation is entered into the system and unification with a plan hypothesis is not possible or it thus not fit a partial order, the plan hypothesis is rejected. However there are some advantages to supply completely instantiated plans as plan hypotheses: • Correlation with new observations is only useful if the plan hypotheses are sufficiently refined. • If the plan is complete, constraints on the plan can be examined. For example: if one states the geographic start point of an objectÕs trajectory and, given the current position of the object, one can check whether the endurance of the object suffices to be able to execute the plan of a plan hypothesis. • By supplying complete plans all the planning is done before correlation and it is not necessary to interleave planning and correlation, which is rather complex. A disadvantage is that the complete set of plan hypotheses may become large in complex cases. However this is not yet the case here and so we choose to provide completely instantiated plans in the plan hypotheses. However, in the case of the partial order plan, we maintain an incomplete sequence of plan actions. In the following sections we only consider non-hierarchical plans. The hierarchical case is a topic for further research, and is not yet described here.

54

F. Mulder, F. Voorbraak / Information Fusion 4 (2003) 47–61

3. The formal definition of plan recognition In this section the properties of the plan recognition process and the data structures involved in them will be described. First by classification of various types of plan recognition and secondly the terms used in the definitions will be related to each other formally. Plan recognition is done from observations of the properties (position, speed) and actions of the agent(s). Two classes of plan recognition problems that are generally distinguished are: intended plan recognition and keyhole plan recognition. The definition of Kautz is as follows, [2]: Intended plan recognition/keyhole plan recognition, in intended plan recognition, but not keyhole, the recognizer can assume that the observed agent is deliberately structuring its activities in order to make its intentions clear. So in intended plan recognition the observed agents may even cooperate with the plan recognizer to make their plans clear. However in the case of keyhole plan recognition the observed agents generally do not assist the plan recognizer in understanding their plans. It is quite clear that in our military case we find keyhole plan recognition. Recognition problems were also classified as to whether the observer has complete knowledge of the domain, and whether the agent may try to perform erroneous plans. Again it is clear that in our case the observer only has incomplete knowledge of the world and that an enemy agent may perform erroneous plans. These aspects will not be considered in this article. The traditional plan recognition problem can then be defined as follows. See [7,18]. Definition 3.1. A plan recognition problem is a pair ðO; CÞ, where O is a set formulas representing observations and C is a set of formulas representing the background knowledge. It is assumed that the observations are not entailed by the background knowledge: for all o 2 O, C 2 o. A solution of the plan recognition problem ðO; CÞ is a plan hypothesis / which satisfies the following two conditions. 1. / is consistent with the observations and the background knowledge. Formally: C [ O 2 :/. 2. / explains the observations given the background knowledge. Formally: for all o 2 O, C [ f/g o. Obtaining a solution to a plan recognition problem is similar to drawing inferences using classical abduction, which can be described as the process of conjecturing explanatory hypotheses. For example, if you are told that all ravens are black (8xðRðxÞ ) BðxÞÞ), and you observe that a bird b is black (BðbÞ), then you can conjecture that b is a raven (RðbÞ). In this case, the

background knowledge C is the set f8xðRðxÞ ) BðxÞÞg, O is the set fBðbÞg, and the formula RðbÞ is consistent with the observations together with the background knowledge, and it explains the observation given the background knowledge. In general, there exist of course many explanatory hypotheses. In the above example, the formula BðbÞ ^ F ðbÞ, representing ‘‘b is black and can fly’’, could in principle also be conjectured. Which candidate explanatory hypotheses are appropriate depends on the application area of the abductive reasoning. This is one of the main reasons why in the literature many different formalizations of abduction can be found. See e.g. [1] for a discussion of the many interpretations of abduction. A popular method for reducing the size of the set of possible explanatory hypotheses is to restrict these possible hypotheses to a particular (syntactic) form. In the case of plan recognition, the considered explanatory hypotheses can be restricted to plan hypotheses. Therefore, the background knowledge has to contain a set of plan templates, i.e. generic descriptions of plans to be recognized, in addition to formulas describing knowledge of the world. It should be remarked that we allow a plan recognizer to generate partial plans, so we cannot assume that the generated plan hypotheses are mutually exclusive. Also, in general, we will not assume that the plan hypothesis generation algorithm generates all possible plan hypotheses. In the following sections, first the observations and the plan hypotheses are elaborated, then various types of plan recognition are defined.

4. Uncertainties in the observations and plan execution Planning is changing a planning world from an initial state to a required state. The required state is described in a number of goals. The planner executes actions that transform the planning world from the initial state to the goal state. An observation concerns the actual state of the world or actual changes of the world state by actions. A state is represented by a conjunction of grounded function free literals. Distinguish events from actions: events are non-intentional changes of the world state, contrary to actions being intentional changes of the world. An observation can refer to an observed action, e.g. Go(KL174, Heathrow), or an observed state, e.g. At(KL174, Heathrow). States as well as actions will be observed, therefore define: Definition 4.1. An action observation refers to an observed step in some plan to be recognized. A state observation refers to some observed state of agents and objects during the execution of some plan to be recognized.

F. Mulder, F. Voorbraak / Information Fusion 4 (2003) 47–61

A plan consists of actions, so subsequent state observations will be translated into action observations of actions which that will cause the state changes. However, as will be shown later, state observations are useful for assessing the probability of hypotheses, if the uncertainty in the state of an object can be modelled by stochastic variables. A plan is completely observable when all itÕs actions can be observed. Mostly one is interested in more significant actions that are often observable. E.g. one is interested in the observation that an airplane is flying from London to Amsterdam, but not in the individual steering actions of the pilot or auto-pilot during this flight. In control theory two types of uncertainty are taken into account when observing a physical process: measurement noise and process noise. When applied to plan recognition these concepts are defined as: • Measurement noise: Uncertainties in the observations, introduced by the observer. • Process noise: Uncertainties in the execution of a plan. Both items are elaborated in the following sections. 4.1. The observations LetÕs elaborate on the measurement noise. One can distinguish four types of observations, one certain observation, called an identified observation, and three types with some form of uncertainty introduced by the observer. The observations are elaborated below: • Identified observations: Here the object or objects involved in an action are identified. Each object is uniquely identified and the observer can tell what objects from different observations are identical. An example is: the airplane KL174 is flying to Heathrow. Here the observer has read the name of the airplane and knows that it executes the scheduled flight KL174 to Heathrow. All the relevant facts of the observation are known in the identified observation. It is formally a literal (function free, grounded). Formally the observation tells us: Fly(KL174, Heathrow). • Unidentified observations: Here the observer cannot read the name of the airplane, he only sees one aircraft flying to one destination. If the observer is near Heathrow, he might infer that the destination of the aircraft is Heathrow. Of course a classified object: Heli, Prop, Jet, . . . can be positioned between an identified and an unidentified object. In first order logic this is modelled by an extra predicate like HeliðxÞ. The observations of the example of a flight to Heathrow then have the form: there is an airplane flying

55

to Heathrow. When also the destination is unknown: there is an airplane flying to a destination. So in the case of an unidentified observation some objects of the observation are unknown. So formally the observations of the example are: 9xfFlyðx; HeathrowÞg 9x; yfFlyðx; yÞg • Probabilistic observations: Now the observer can estimate some variables of the observations. E.g. the observer can estimate the speed of flight KL174 to be 200 (km/h). So the observation looks like: there is one aircraft flying to Heathrow with an approximate speed of 200 (km/h). Formally we can model the uncertainty of the speed probabilistically, e.g. with a Gaussian probability density function: v  Nð200 ðkm=hÞ; 30 ðkm=hÞÞ. Formally the observation then is: FlyðKL174; Heathrow; vÞ; v  Nð200 ðkm=hÞ; 30 ðkm=hÞÞ • Uncertain observations: In this case one does not trust the observer to make a correct statement. Only a percentage of the statements of these observations are actually true. A degree of belief, in our case a probability, can be defined for such an observation: e.g. Confirmed (95%), Probable (70%) and NotProbable (25%). An uncertain observation is sometimes represented by adding a prime to the symbol of the observation: if the certain observation is o then itÕs uncertain observation is denoted as o0 . Formally the following statements go for an uncertain observation: P ðFlyðKL174; HeathrowÞjObsðFlyðKL174; HeathrowÞÞÞ ¼ Confirmed ¼ 95% or equivalent: P ðojo0 Þ ¼ Confirmed

These types of observations, that a tactical plan recognition system has to be able to handle, are summarized in Table 2. The type of observations that are entered in the plan recognition system, of course have great impact on its processing. In this report the uncertain observations are not handled yet. 4.2. The plan execution In the case of the execution of a tactical plan one is uncertain about: 1. erroneous execution: cannot be handled, 2. localization uncertainty: this can be modelled by a probability density function along the ideal paths of the plan, 3. partial achievement of goals: abort actions are explicitly modelled in the plan template, e.g. see single agent plan template.

56

F. Mulder, F. Voorbraak / Information Fusion 4 (2003) 47–61

Table 2 Different types of observations Observation type

Example

Formal representation of example

Identified observations

The airplane KL174 is going to Heathrow

Fly(KL174, Heathrow)

Unidentified observations

There is one airplane going to Heathrow There is one airplane going to one destination

9xfFlyðx; HeathrowÞg 9x; yfFlyðx; yÞg

Probabilistic observations

The airplane KL174 is going to Heathrow with an approximate speed of 200 (km/h)

FlyðKL174; Heathrow; vÞ, v  Nð200 ðkm=hÞ; 30 ðkm=hÞÞ

Uncertain observations

There is a Confirmed observation of airplane KL174 going to Heathrow

O1 ¼ ObsðFlyðKL174; HeathrowÞÞ P ðFlyðKL174; HeathrowÞjO1 Þ ¼ Confirmed ¼ 95%

Apart from the observations, also the types of plans that have to be recognized are very important. This is elaborated in the following section. 5. Plan recognition The input of a plan recognizer: the observations and the background knowledge uniquely characterizes a class of plan recognizers. In this chapter we will distinguish two classes of plan recognizers. The first are deterministic plan recognizers. These have identified or unidentified observations as input. As the background knowledge is assumed to be certain, such a recognizer will output a set of plan hypotheses. Note that these hypotheses are not ranked. The second class are probabilistic plan recognizers. These have as input: identified, unidentified and probabilistic observations. Their output is a number of observation––plan hypothesis assignment hypotheses, ranked by the probability of an assignment hypothesis. 5.1. Deterministic plan recognition The simplest type concerns the recognition of only one plan, executed by one agent. This coincides with the earlier given definition of ‘‘traditional’’ plan recognition. Definition 5.1. A (single) plan recognition problem is a pair ðO; CÞ, where O is a set formulas representing observations (concerning a single plan) and C is a set of formulas representing the background knowledge. It is assumed that the observations are not entailed by the background knowledge: for all o 2 O, C 2 o. A solution of the plan recognition problem ðO; CÞ is a plan hypothesis / which is consistent with the observations and the background knowledge, and explains the observations given the background knowledge. Of course, typically, one is not satisfied with finding a single solution to a plan recognition problem, but prefers to generate several solutions, from which the most

interesting hypotheses are selected. A plan hypothesis may be a partial plan or a complete plan. Whether a partial plan hypothesis suffices depends on the userÕs requirements. Sometimes the user will demand complete plans, but it is also possible that a certain level of abstraction is preferred. The observations can be either identified or unidentified, but in the above definition it is assumed that they all concern the same plan. Also, any plan hypothesis of U is required to explain all observations. This is adequate in the case of a single agent plan, but when there are multiple agents executing multiple independent plans, a plan hypothesis should only be required to explain the observations originating from the same plan. (The case of multiple agents cooperating in a single multi-agent plan or in multiple dependent plans can perhaps best be viewed as multiple agents executing multiple plans with some constraints between the plans.) For the case of multiple plans, we need to propose several hypothetical plans at once and add information about the subset of the observations that is supposed to be explained by each hypothetical plan. For this purpose, we define the following notion of an assignment of observations to hypothetical plans. Definition 5.2. Given a set of observations O and a set of generated plan hypotheses, U, an assignment Wh is a surjective function Wh : O ! U, with as domain the set of observations and as co-domain the set of generated hypothetical plans. An assignment incorporates hypotheses about which observations stem from the same object. The codomain of Wh only consists of the plan hypotheses generated for the observations, and not of all possible plan hypotheses. Therefore, we can assume Wh to be surjective. Moreover, we can use the notation Wh ðOÞ for the set of plan hypotheses generated for the observations O, i.e. the codomain. Given an assignment Wh , the set of observations O is partitioned into equivalence classes of observations relevant to the same plan. We write kokh for the equivalence class fo0 2 OjWh ðo0 Þ ¼ Wh ðoÞg of observation o.

F. Mulder, F. Voorbraak / Information Fusion 4 (2003) 47–61

Recognition of multiple independent plans can now essentially be defined as several single plan recognitions, where each single plan recognition is limited to an equivalence class of observations. Definition 5.3. A multiple (independent) plan recognition problem is a pair ðO; CÞ, where O is a set formulas representing observations (not necessarily concerning a single plan) and C is a set of formulas representing the background knowledge. It is assumed that the observations are not entailed by the background knowledge: for all o 2 O, C 2 o, and that the background knowledge does not contain constraints between possible plans. A solution of the plan recognition problem ðO; CÞ is a set of plan hypotheses U and an assignment Wh : O ! U such that, for each o 2 O, the following two conditions are satisfied. 1. Wh ðoÞ is consistent with the observations assigned to Wh ðoÞ and the background knowledge. Formally: C [ kokh 2 :Wh ðoÞ. 2. Wh ðoÞ explains the observations assigned to Wh ðoÞ given the background knowledge. Formally: for all o 2 kokh , C [ fWh ðoÞg o. If the plans are dependent, then the plans have to be consistent with the background knowledge and all the observations. However, each plan hypothesis only has to explain the observations assigned to it. Definition 5.4. A multiple dependent plan recognition problem is a pair ðO; CÞ, where O is a set formulas representing observations (not necessarily concerning a single plan) and C is a set of formulas representing the background knowledge (which may include constraints between possible plan). It is assumed that the observations are not entailed by the background knowledge: for all o 2 O, C 2 o. A solution of the plan recognition problem ðO; CÞ is a set of plan hypotheses U and an assignment Wh : O ! U such that, for each o 2 O, the following two conditions are satisfied. 1. The conjunction of all assigned plan hypotheses ^fWh ðoÞjo 2 Og is consistent with all the observations and the background knowledge. Formally: C [ O 2 : ^ fWh ðoÞjo 2 Og. 2. Wh ðoÞ explains the observations assigned to Wh ðoÞ given the background knowledge. Formally: for all o 2 kokh , C [ fWh ðoÞg o. Example 5.1. We consider two airplanes S (‘‘Swaen’’) and Z (‘‘Zwaluw’’) which can be used for two routes: KL174 from H (Heathrow) to F (Frankfurt), and KL175 from Frankfurt to Heathrow. Both flights stop in A (Amsterdam). This results in four plans: Flight(KL174, S), Flight(KL174, Z), Flight(KL175, S),

57

and Flight(KL175, Z). The background knowledge C contains the following formulas: 8xðFlightðKL174; xÞ () FlyðH; A; xÞ ^ StopðA; xÞ ^ FlyðA; F; xÞÞ and 8x ðFlightðKL175; xÞ () FlyðF; A; xÞ ^ StopðA; xÞ^ FlyðA; H; xÞÞ. For simplicity, we abstract from temporal considerations, and represent the flights as conjunctions of parts. (1) Suppose we have the observation o1 ¼ StopðA; SÞ. This observation o1 can be assigned to either Flight(KL174, S) or Flight(KL175, S). Both plan hypotheses are consistent with C [ fo1 g, and explain o1 given the background knowledge. In this case, the assignments are not really necessary, since it concerns a single plan. (2) Suppose the observations are o1 ¼ StopðA; SÞ, and o2 ¼ StopðA; ZÞ. Now both observations o1 and o2 can be assigned to either flight KL174 or flight KL175. This results in four multiple plans: (a) (b) (c) (d)

Flight(KL174, S), Flight(KL175, Z). Flight(KL175, S), Flight(KL174, Z). Flight(KL174, S), Flight(KL174, Z). Flight(KL175, S), Flight(KL175, Z).

The last two (multiple) plans can be excluded by adding some constraints between the possible plans. For example, 8x8yðFlightðx; yÞ ) 8zðFlightðx; zÞ ) y ¼ zÞÞ represents the constraint that each flight is performed by at most one airplane. (3) Suppose the observations are o1 ¼ StopðA; SÞ, o2 ¼ StopðA; ZÞ, and o3 ¼ FlyðA; F; SÞ. Now the only possible assignment for o1 and o3 is Flight(KL174, S). The remaining observation o2 can be assigned to either Flight(KL174, Z) or Flight(KL175, Z) if no constraints between the flights are present. With the constraints mentioned above only Flight(KL175, Z) remains possible. (4) Suppose now we have an unidentified observation o1 ¼ 9x StopðA; xÞ. This observation o1 can be assigned to all four plans Flight(KL174, S), Flight(KL175, S), Flight(KL174, Z), and Flight(KL175, Z). (5) Suppose we have two unidentified observation o1 ¼ 9x StopðA; xÞ and o2 ¼ 9x FlyðA; F; xÞ. Let us assume that the background knowledge contains the constraint that each flight is performed by at most one airplane. Then the following four multiple plans remain possible: (a) Flight(KL174, S), Flight(KL175, Z). Here observation o1 is assigned to Flight(KL174, S) and o2 to Flight(KL175, Z). (b) Flight(KL175, S), Flight(KL174, Z). Here observation o1 is assigned to Flight(KL175, S) and o2 to Flight(KL174, Z). (c) Flight(KL174, S). Here observations o1 and o2 are both assigned to Flight(KL174,S).

58

F. Mulder, F. Voorbraak / Information Fusion 4 (2003) 47–61

(d) Flight(KL174, Z). Here observations o1 and o2 are both assigned to Flight(KL174,Z). In practice the observations can be temporally ordered in a sequence ½o ¼ ½o1 ; . . . ; on , rather than only to be organized in a set. In the case of scanning sensors delivering the observations, they are organized in a sequence of mutually exclusive subsets of O, each subset of observations having arrived in a certain scan of the sensor k: O ¼ ½Ok  ¼ ½fo1v ; . . . ; onv g; . . . ; fo1k ; . . . ; omk g

ð1Þ

As the first case is a specialization to singletons of the latter case, only the latter case will be elaborated. In the scan case one wants intermediate results directly after receiving the set of observations of scan k. This rather than to wait until the complete set of observations is available. The latter will never be the case if the sequence is infinite! Note that multi-sensor data fusion can occur. The scans of different sensors can be combined into one sequence of scans. If two sensors have overlapping coverage, an adaptation of the scan organization is required, but is not mentioned here. The definition of the assignment now is: Definition 5.5. An assignment of the current scan k, is a surjective function Wh;k : Ok ! Uk1;h [ fNPk g, with as domain the set of observations of the current scan Ok and as codomain the set of hypothetical plans of the previous scan Uk1;h ¼

k[ 1

ðNPi  DPi Þ

ð2Þ

i¼0

augmented with the new plan set NPk of scan k. Here NPi indicates the new plan set of scan i and DPi represents the terminated plan set of scan i. The total assignment hypothesis consists of that of the current scan, Wh , concatenated with those of the previous scans, Xkg : ðWh : Xk1 g Þ. The set of hypothetical plans is split up in plans that already existed in the previous scans minus the terminated plans and new plans that are created in the current scan by observations of the current scan. Each scanning sensor only gives one observation for each observed object per scan. In this case we define a bijective assignment function for the current scan for each sensor. 5.2. Probabilistic plan recognition The previous sections concerned deterministic plan recognition with identified or unidentified observations. These are explained by a consistent set of plan hypotheses or assignment hypotheses. These plan hypotheses cannot be ranked directly, as they all explain the ob-

servations. In the case of probabilistic or uncertain observations the plan recognizer can calculate the probability of each plan hypothesis. This can appropriately be called probabilistic plan recognition. In the case of identified observations, a plan recognizer gives pairs: ð/; P ð/ÞÞ; where: / is the plan hypothesis and P ð/Þ is the probability of the plan hypothesis. However the set of hypotheses U must be exhaustive and its elements must be mutually exclusive in some sensor, so P the sum of the individual probabilities adds up to one: P ð/Þ ¼ 1. As noted before, in the case of partial plan hypotheses, the hypotheses are not mutually exclusive! In the case of unidentified (uncertain/probabilistic) observations the hypothesis becomes the pair: ðWh ; P ðWh ÞÞ where: Wh is the assignment hypothesis and P ðWh Þ its probability. Here the assignment hypotheses P must be exhaustive and mutually exclusive so: P ðWh Þ ¼ 1. Note that in the latter case only the probability of the assignment is calculated and not the direct probability of plan hypotheses in the assignment. In the literature attempts have been made to define some special semantics for abductive inference with evidence augmented with a numerical uncertainty. One has to redefine terms such as consistency and explanation, in a probabilistic setting. For this we refer to the work of Neapolitan, in [21] and NilssonÕs probabilistic logic, 1987, in [14]. Though in both plan recognition cases the number of assignment hypotheses is worst case exponential in the number of observations. The bijective assignment is more attractive than the surjective. For a bijective assignment function, various polynomial algorithms are readily available to select the k-best assignments. This is not the case for a surjective assignment function. Furthermore the bijective assignment is certainly an adequate model for the observations from a scanning sensor or a human observer. Though there is no consensus about probabilistic abduction, a technique to handle probabilistic, unidentified observations to generate a number of assignment hypotheses and their probabilities, is available from the field of tracking. 5.2.1. Reid’s formula Tracking algorithms organize unidentified observations of airplanes, ships and vehicles from one or more sensors, called plots, into tracks: sets of plots which are thought to originate from the same object. The most correct and successful model for tracking is the multiple hypothesis tracking (MHT) algorithm from Reid, [24]. In this algorithm plot - track assignment hypothesis are formulated for each new scan of data, based upon assignment hypotheses from the previous scans. The assignment hypotheses are ranked by ReidÕs formula (formula 3). This formula gives the probability of a set

F. Mulder, F. Voorbraak / Information Fusion 4 (2003) 47–61

of assignments consisting of Xk1 g , the assignments of the previous scans and Wh , the assignment of the current scan k, given the set of plots of all the scans, Z: P ðXk1 g ; Wh jZÞ. This is calculated by using BayesÕ formula and the chain rule for conditional probabilities. Below the assignment probability is given for the singlesensor case: 1 NDT P ð1  PD ÞNTGT NDT P ðXk1 g ; Wh jZÞ ¼ C D Y  NT gxy P ðXk1  bNFTFT bNNT ð3Þ g Þ where • C ¼ is normalization constant, • PD is the detection probability of the sensor when the target is in the range of the sensor, • NTGT ¼ the number of prior targets (tracks) that were recognized by the tracking system in the previous scans, • NDT ¼ the number of plots of the current scan associated with prior targets of previous scans, • NFT ¼ the number of plots of the current scan associated with false targets (e.g. birds and sea or land clutter), • NNT ¼ the number of plots of the current scan associated with new targets in the coverage, • bFT ¼ NFT =Vsensor is the false alarm density per scan of the sensor, and Vsensor is the volume of a 3D sensor or the area of a 2D sensor, • bNT ¼ NNT =Vsensor is the new target density per scan of the sensor, • Wh ¼ the assignment of plots of the current scan to priorStargets, false targets and new targets, • Z ¼ i¼k i¼0 Zk where Zk the set of Mk ¼ NDT þ NFT þ NNT plots of the current scan, • gxy ¼ the correlation likelihood between a plot x and a track y, x 2 ½1 . . . NDT ; y 2 ½1 . . . NTGT . This is based on the statistical distance between the observed state of the plot and the predicted estimated state of the track. In the case of Gaussian stochastic variables this is the Mahanalobis distance and thus a probability density function. This distance is also applied by a similar application of Mevassvik [19]. The multiple sensor case of tracking with non-overlapping sensors is a simple generalization of the formula above. In the case of an overlapping coverage of the sensors, extra constraints go on the assignment hypotheses. P ðXk1 g ; Wh jZÞ ¼

sensor n 1 NY Nse;DT N N Pse;D ð1  Pse;D Þ TGT se;DT C se¼1 Y o Nse;FT Nse;NT gse;xy P ðXk1  bse;FT bse;NT g Þ

where se 2 ½1 . . . Nsensor  is a sensorÕs index.

ð4Þ

59

5.2.2. Reid’s formula for tactical plan recognition In the case of plan recognition with probabilistic unidentified observations ReidÕs formula is redefined below for a single observer: P ðXk1 g ; Wh jOÞ ¼

1 NDT N N P ð1  PD Þ TGT DT C D Y  NT gxy P ðXk1  bNNT g Þ

ð5Þ

The main difference with the ReidÕs formula is the absence of false alarms and that the gxy now is the likelihood of a observation being associated to a prior plan hypothesis of the previous scan, k  1. As mentioned before we had two types of observations with respect to actions and states: state observations and action observations. Both are used to correlate with plan hypotheses. (1) Action observation: The correlation of an action observation with a plan hypothesis entails a check whether the observed actions ÔfitsÕ in the sequences of actions generated by the plan hypothesis. If it fits the correlation probability is one, otherwise it is zero. More elaborate schemaÕs utilizing Markov schemes can be devised. (2) State observation: The state observations are used to calculate the correlation likelihood gxy . It gives the statistical distance between the predicted state at the observation time of a plan hypothesis in the assignment and the observed state. In the following sub-section we utilize the adapted formula of Reid for the running example of Figs. 2, 4–8. Here the values of the possible assignment hypotheses are calculated. 5.2.3. A numerical example If the first observation at time T1, O1 , enters the system, it is assumed that the base of the observationÕs helicopter is air base (unit 1). Only two targets are considered by the plan recognizer for simplicity: the munitions depot, MUN, and tank unit 5, TK5. And so a plan hypothesis of an attack of a helicopter, H1, on the munitions depot (MUN): A(H1,MUN) and a plan hypothesis of an attack of a helicopter, H1, on unit 5 (TK5): A(H1,TK5) are generated. The plan A(heli,target) consists of the steps: ½Flyðheli; base; commÞ; CommTargetðheli; commÞ; Fly ðheli; comm; targetÞ; Flyðheli; target; baseÞ The three observations tell that a helicopter has been spotted at a certain position. So they concern a position: a state. The observations can match with one of the first two Fly steps: Fly(heli,base,comm), Fly(heli,comm,target). So after the first observation enters the system, there are two assignment hypotheses, one for each plan hypothesis. Their probabilities are determined by the bNT .

60

F. Mulder, F. Voorbraak / Information Fusion 4 (2003) 47–61

Table 3 Assignment hypothesis probabilities in the example Assignment

Plan hypotheses

Probability

Value

O1 W1;1 W1;2

A(H1,MUN) A(H1,TK5)

ð1=CÞbNT ð1=CÞbNT

0.5 0.5

O2 W2;1 W2;2 W2;3

A(H1,MUN), A(H2,TK5) A(H1,TK5), A(H2,TK5) A(H1,TK5)

ð1=CÞbNT ð1  PD ÞP ðW1;1 Þ ð1=CÞbNT ð1  PD ÞP ðW1;2 Þ ð1=CÞPD gxy21 P ðW1;2 Þ

0.24339 0.24339 0.51322

O3 W3;1 W3;2 W3;3 W3;4 W3;5 W3;6 W3;7

A(H1,MUN), A(H2,TK5), A(H3,TK5) A(H1,MUN), A(H2,TK5) A(H1,TK5), A(H2,TK5), A(H3,TK5) A(H1,TK5), A(H2,TK5) A(H1,TK5), A(H2,TK5) A(H1,TK5), A(H3,TK5) A(H1,TK5)

ð1=CÞbNT ð1  PD Þ2 P ðW2;1 Þ ð1=CÞPD gxy31 ð1  PD ÞP ðW2;1 Þ ð1=CÞbNT ð1  PD Þ2 P ðW2;2 Þ ð1=CÞPD gxy32 ð1  PD ÞP ðW2;2 Þ ð1=CÞPD gxy33 ð1  PD ÞP ðW2;2 Þ ð1=CÞbNT ð1  PD ÞP ðW2;3 Þ ð1=CÞPD gxy34 P ðW2;3 Þ

0.007380 0.015562 0.007380 0.015562 0.015562 0.155616 0.782939

• W1;1 ¼ fO1 $ AðH1; MUNÞg, P ðW1;1 Þ ¼ ð1=CÞbNT • W1;2 ¼ fO1 $ AðH1; TK5Þg, P ðW1;2 Þ ¼ ð1=CÞbNT Next the observation O2 at T2 generates a new plan hypothesis of helicopter H2 coming from the FARP and attacking TK5, but it also correlates with the plan of H1 attacking the TK5 unit. Now there are the following assignment hypotheses: • W2;1 ¼ fO2 $ AðH2; TK5Þg and the and H1 is still assumed to attack MUN. The probability of the hypothesis is P ðW2;1 Þ ¼ ð1=CÞbNT ð1  PD ÞP ðW1;1 Þ The plan hypotheses in this assignment are: A(H1,MUN), A(H2,TK5). • W2;2 ¼ fO2 $ AðH2; TK5Þg where H1 is also assumed to attack TK5. So now there are two helicopters, both attacking TK5. The probability here is P ðW2;2 Þ ¼ ð1=CÞbNT ð1  PD ÞP ðW1;2 Þ. The plan hypotheses in this assignment are again: A(H1,TK5), A(H2,TK5). • W2;3 ¼ fO2 $ AðH1; TK5Þg. So now both observations are assigned to the same helicopter H1 and the probability of the hypothesis is P ðW2;2 Þ ¼ ð1=CÞPD gxy21 P ðW1;2 Þ. The plan hypothesis in this assignment is: A(H1,TK5).

bNT ¼ NNT =Area ¼ 1=ð4 ðkm2 ÞÞ ¼ 2:5  107 PD ¼ 0:9 gxy ¼ Nðv; BÞ is normally distributed in ðv; BÞ, where v ¼ the residue between planned position and observed position v ¼ ½dx; dy and • B ¼ is the two dimensional cartesian covariance matrix of the residue • ðv; BÞ ¼ ð½1000; 1000, ½½1002 ; 102 ; 102 ; 1002 Þ for the following correlation probabilities: gxy21 ¼ gxy31 ¼ gxy32 ¼ gxy33 ¼ 5:86  108 • ðv; BÞ ¼ ð½500; 100; ½½1002 ; 102 ; 102 ; 1002 Þ for the following correlation probability: gxy34 ¼ 1:40  107 • • • •

After receiving the first observation, O1 , two assignment hypotheses are created, that cannot be discerned from each other. The second observation, O2 , correlates very well with the attack of unit 5 (TK5), and at this stage this hypothesis wins. The third observation, O3 , also correlates very well with the original attack of unit 5 and so after receiving the final observation still the optimal assignment contains only one plan hypothesis of the attack of unit 5.

6. Conclusion The last observation O3 at T3 gives the best match with the the attack of helicopter H1 of TK5 and the the W3;7 assignment hypothesis gets the most support: • W3;7 ¼ fO3 $ AðH1; TK5Þg, the probability of the hypothesis is P ðW3;1 Þ ¼ PD gxy P ðW2;3 Þ. Note that now the relative difference of the probability of this hypothesis and that of the others is larger than in the previous scan. In Table 3 all the assignment hypotheses are given. Now assume the following values for the variables of Table 3:

In this report we have shown that tactical plan recognition differs significantly from classical plan recognition. In classical plan recognition the plan recognizer has as input certain, identified observations of actions. Whereas in tactical plan recognition the observations are uncertain in various ways. The uncertainties of the observations have been listed here and formal representations of the uncertainties have been described. The formal definition for classical plan recognition has been extended to definitions of tactical plan recognition for the cases of unidentified, probabilistic obser-

F. Mulder, F. Voorbraak / Information Fusion 4 (2003) 47–61

vations and non-hierarchically single agent plans. In the case of probabilistic and unidentified observations one can calculate the probability of the assignment hypotheses, by adapting ReidÕs MHT formula. In future work the formal definitions will be applied to prove characteristics, such as soundness and completeness, of tactical plan recognition algorithms.

Acknowledgements The authors would like to thank the reviewers of the Information Fusion Journal for their useful comments on this paper. We also appreciate the support of Mr. H.M. Menninga, Dr. M.A.W. Houtsma and Dr. P.J. Braspenning in our plan recognition research. References [1] A. Aliseda-LLera, Seeking explanations: abduction in logic, philosophy of science and artificial intelligence, PhD thesis, Stanford University. ILLC Dissertation Series 1997-4. Institute for Logic, Language and Computation, Amsterdam, 1997. [2] J.F Allan, H.A. Pelavin, R.N. Tenenberg, Reasoning About Plans, Morgan-Kaufmann, San Mateo, CA, 1991. [3] J. Azarewicz, Plan recognition for airborne tactical decision making, in: Proceedings of the Fifth National Conference On Artificial Intelligence (AAAI-86), 1986, pp. 805–811. [4] J. Azarewicz, Template-based multi-agent plan recognition for tactical situation assessment, in: Proceedings on the Fifth Conference on Artificial Intelligence Applications, 1989. [5] M. Bauer, Integrating probabilistic reasoning into plan recognition, in: Proceedings of the Tenth European Conference on Artificial Intelligence (ECAI-94), 1994, pp. 620–624. [6] M. Bauer, Machine learning for user modelling and plan recognition, in: Working Notes on the ICML-96 Workshop: ML meets HCI, 1996. [7] M. Bauer, G. Paul, Logic-based plan recognition for intelligent help systems, in: E. Sandewal, C. Backstrom (Eds.), Current Trends in AI Planning, IOS Press, 1994, pp. 60–73. [8] A.N. Bernstein, Multi Sensor Data Fusion and Multi Target Tracking, H. Silver and Associates, 1998. [9] S.S. Blackman, Multiple Target Tracking with Radar Applications, Artech House, Norwood, MA, 1986. [10] R.J Calistri-Yeh, Classifying and detection of plan-based misconceptions for robust plan recognition, AI magazine 12 (3) (1991). [11] D. Canamero, Plan recognition for decision support, in: IEEE Expert, June 1996. [12] S. Carberry, Plan Recognition in Natural Language Dialogue, MIT Press, Cambridge MA, 1990. [13] E. Charniak, R.P. Goldman, A probabilistic model of plan recognition, in: Proceedings Ninth National Conference on Artificial Intelligence (AAAI-91), vol. 1, 1991, pp. 160–165.

61

[14] J. Genesereth, N. Nillson, Logical Foundations of Artificial Intelligence, Morgan Kaufmann, San Mateo, CA, 1987. [15] M.J. Huber, E.H. Durfee, Observational uncertainty in plan recognition among interacting robots, in: Working Notes of the IJCAIJ-93 Workshop on Dynamically Interacting Robots, Elsevier Science Publishers, 1993. [16] H.A. Kautz, J. Allan, Generalized plan recognition, in: Proceedings of the Fifth National Conference on Artificial Intelligence, AAAI-86, 1986, pp. 32–37. [17] N. Lesh, O. Edzioni, Insights from machine learning for plan recognition, in: IJCAI-95 Workshop on the Next Generation of Plan Recognition Systems, 1995. [18] D. Lin, R. Goebel, A message passing algorithm for plan recognition, in: Proceedings of the Twelfth International Conference on Artificial Intelligence (IJCAI-91), vol. 1, Morgan Kaufmann, San Mateo, CA, 1991, pp. 280–285. [19] O.M. Mevassvik, A. Lokka, Fusion of radar tracks, reports and plans, in: Proceedings of Third International Conference on Information Fusion, International Society of Information Fusion, 2000. [20] F.W. Mulder, A generic task model of tactical plan recognition, in: Proceedings of the 15th World Computer Congress, Vienna/ Budapest, International Federation of Information Processing (IFIPS), 1998. [21] R.E. Neapolitan, Probabilistic Reasoning in Expert Systems: Theory and Algorithms, Wiley, 1990. [22] D.F. Noble, Template-based data fusion for situation assessment, in: Proceedings of the Tri-Service Data Fusion Symposium, 1987, pp. 152–162. [23] D.V. Pynadath, M.P. Wellman, Accounting for context in plan recognition, with application to traffic monitoring, in: Proceedings of the Fourteenth National Conference on Artificial Intelligence (AAAI-95), Morgan Kaufman, 1995. [24] D.B. Reid, An algorithm for tracking multiple targets, IEEE Transactions on Automatic Control AC-24 (6) (1979) 843–854. [25] K.N. Ross, R.D. Chaney, Hidden markov models for threat prediction fusion, in: B.V. Dasarathy (Ed.), Proceedings of SPIE, Sensor Fusion: Architectures, Algorithms, and Applications IV, vol. 4051, SPIE, 2000. [26] S.J. Russel, P. Norvig, Artificial Intelligence: A Modern Approach, Prentice Hall, Englewood Cliffs, NY, 1995. [27] C.F. Schmidt, N.S. Sridharan, J.L. Goodson, The plan recognition problem: An intersection between psychology and artificial intelligence, Artificial Intelligence 11 (1978) 45–83. [28] C. Sidner, Plan parsing for intended response recognition in discourse, Computational Intelligence 1 (1985) 1–10. [29] M. Tambe, Resc: an approach to agent tracking in a real-time, dynamic environment. 1995. [30] M. Tambe, P.S. Rosenbloom, Event tracking in a dynamic multiagent environment, Computational Intelligence (1996). [31] M. Villain, Getting serious about parsing plans: a grammatical analysis of plan recognition, in: Proceedings of the Eleventh National Conference on Artificial Intelligence (AAAI-90), Morgan Kaufmann, San Mateo, CA, 1990, pp. 190–197. [32] R. Wols, Wp300, ntp-datafusie: Military expert knowledge base. Technical report, FEL-TNO, The Hague, 1995. [33] R. Wols, Wp3002, ntp-datafusie: The aggregation process. Technical report, FEL-TNO, The Hague, 1995.