Model Checking vs. Theorem Proving: A Manifesto

Model Checking vs. Theorem Proving: A Manifesto Joseph Y. Halpern and Moshe Y. Vardi I B M Almaden Research Center San Jose, California 9 5 1 2 0

1

Introduction

The standard approach in A I to knowledge representation, going back to [47], is to represent an agent's knowledge as a collection of formulas, which we c a n view as a knowledge base. An agent is then said t o know a fact if it is provable from the formulas in his knowledge base. This is called in [60] the "interpreted-symbolic-structures" approach. There are two problems in applying this approach. T h e first comes in the difficulty of representing agents' knowledge in terms of formulas in some appropriate language. T h e second lies in the difficulty of theorem proving. These problems are closely related; the need to use logic to represent agents' knowledge necessitates the use of very expressive logics, but the more expressive a logic, the harder it is to prove theorems in t h a t logic. In this chapter, we argue for a model-theoretic rather t h a n a proof-theoretic approach to the problem. E s sentially, the idea is to represent the agent's knowledge by a d a t a structure representing some semantic model (in the spirit of the "situated-automata" approach [60, 5 9 ] ) , and replace theorem proving by model checking, t h a t is, checking whether a given formula is true in the model. As an example of this approach, consider the context of relational database systems. L e t Β be a relational database and let φ be a firstorder query. T h e theorem-proving approach would view Β as representing some formula φ Β and would evaluate the query by trying to prove or disprove φ Β => ψ- Unfortunately, theorem proving for first-order logic is undecidable. T h e model-checking approach, on the other hand, would check whether φ holds in the database B. This can be evaluated in time polynomial in the size of the d a t a (cf. [66]). As another example, consider an agent Alice who knows t h a t , given her current epistemic s t a t e (i.e., the information she has obtained thus far),

151

Copyright © 1991 by Academic Press, Inc. AU rights of reproduction in any form reserved. ISBN 0-12-450010-2

Halpern and Vardi

152

the world could be in any one of three possible states.

In the possible-

world approach, this situation is modeled by a Kripke structure with three possible worlds. As usual, we say t h a t Alice knows a fact ρ if ρ is true in all three of the worlds t h a t Alice considers possible given her epistemic s t a t e . Since, in particular, all tautologies will be true at all three worlds Alice considers possible, it follows t h a t Alice knows all propositional tautologies. This m a y seem strange. T h e set of propositional tautologies is well-known t o be co-NP-complete. How c a n Alice, who after all does not seem to possess any extraordinary reasoning power, know all tautologies when she cannot prove t h e m ?

In the model-checking approach, there is nothing unusual

about this fact. Alice knows a propositional formula φ if φ is true in the three states t h a t Alice considers possible. This c a n be checked in time linear in the length of φ.

Notice t h a t even though Alice knows all tautologies

(among other facts she knows given her epistemic s t a t e ) , she does not know which of the facts she knows are tautologies (and probably does not care!). Thus, the well-known logical omniscience problem [32] does not present the same difficulties in the mo del-checking approach as it does in the theoremproving approach (although, as we shall see, other related difficulties do arise). T h e paradigm of model checking arose explicitly in the context of finites t a t e program verification. a

finite-state

(See [10] for an overview.)

Suppose we have

p r o g r a m Ρ (think of Ρ as, for example, a communications

protocol), and we want t o know whether it satisfies some specification ψ, which we assume c a n be expressed in temporal logic.

1

It is not hard to

completely characterize the p r o g r a m Ρ by a temporal logic formula

φρ.

(Essentially, φ ρ describes all the possible transitions of Ρ in each possible global state; this is possible since F is a finite-state protocol.) One way of checking whether Ρ satisfies the specification ψ is t o check Ίίφρ =ϊ ψ is valid [45].

Unfortunately, the validity problem for t e m p o r a l logic is extremely

difficult.

(Technically, it is exponential-time complete [14], which most

likely makes it much harder t h a n the validity problem for propositional logic.) L a t e r , researchers noticed t h a t another approach would work equally well [9, 13, 57]. R a t h e r t h a n having Ρ be represented by a formula, Ρ c a n be represented by a Kripke structure Mp:

the states in Μ ρ represent the

possible global states of P , and the edges represent the possible transitions of P.

Note t h a t the size (i.e., the number of states) of Mp

is essentially

the same as the length of φ ρ (viewed as a string of symbols).

Checking

whether Ρ satisfies the specification ψ now amounts t o checking if iß is true 1

O u r discussion here presumes the use of branching temporal logic. In the case of linear temporal logic, the situation is somewhat more complicated, see [14, 62, 44].

Model Checking vs. Theorem Proving

153

at t h e s t a t e in Mp t h a t corresponds t o t h e initial s t a t e of P. This c a n be done in time linear in the size of Mp and φ. Of course, it could be argued t h a t all t h a t this argument shows is t h a t for a special subclass of formulas (namely, those of the form φ ρ => ψ), the validity problem is significantly simpler t h a n it is in general. Indeed, perhaps this observation should encourage us t o find other subclasses for which the validity problem is solvable in polynomial time (cf. [15]). However, we would claim t h a t this argument misses the point. T h e reason t h a t formulas of the form φ ρ => ψ are easy t o deal with is because φ ρ characterizes a particular s t a t e in a particular structure, in t h a t any s t a t e where φ ρ is satisfied is isomorphic t o t h e initial s t a t e of t h e structure defined by P. Thus, for these formulas, the validity problem reduces t o a model-checking problem, and so is tractable. T h e model-theoretic approach t o finite-state p r o g r a m verification is quite practical, a n d several systems based on model checking have been implemented [10, 7]. Moreover, it has been extended t o deal with more complicated protocols and environments (including probabilistic protocols and assumptions of fairness [69]). In some cases, the assumptions t h a t are being dealt with (such as fairness) are not even expressible in t h e language being used for the specification formula. This is not a hindrance t o the model-theoretic approach. W e c a n often check whether the model satisfies assumptions t h a t are not expressible in the language. T h e core of this chapter (Section 2 ) is devoted t o a detailed description of the model-checking approach, a discussion of potential problems with t h e approach and how some might be dealt with by using current techniques, and a comparison of the model-checking approach and t h e theorem-proving approach. W e conclude in Section 3 with a general discussion of the appropriateness of the model-checking and theorem-proving approaches. T h e key observation here is t h a t t h e model-checking approach allows us t o decouple the description of t h e model from t h e description of the properties we want to prove about the model.

2 2.1

Using the Model-Theoretic Approach The Muddy Children Puzzle

Before getting into technical details, we motivate the model-theoretic approach by applying it t o t h e well-known "muddy children" puzzle. This seems particularly appropriate for this Festschrift, since M c C a r t h y was one of the pioneers of attempting t o formalize such puzzles using epistemic logics and, indeed, the muddy children puzzle is a generalization of the wise-men

154

Halpern and Vardi

puzzle considered in [48]. T h e version of the muddy children puzzle given here is taken from [5]: Imagine η children playing together. T h e mother of these children has told t h e m t h a t if they get dirty there will be severe consequences. So, of course, each child wants to keep clean, but each would love to see the others get dirty. Now it happens during their play t h a t some of the children, say k of them, get mud on their foreheads. E a c h c a n see the mud on others but not on his own forehead. So, of course, no one says a thing. Along comes the father, who says, "At least one of you has mud on your head," thus expressing a fact known to each of t h e m before he spoke (if fc > 1 ) . T h e father then asks the following question, over and over: "[Does any of you know whether] you have mud on your head?" Assuming t h a t all the children are perceptive, intelligent, truthful, and t h a t they answer simultaneously, what will happen? There is a "proof" t h a t the first k — 1 times he asks the question, they will all say "no," but then the fcth time the dirty children will answer "yes." T h e "proof" is by induction on k. For k = 1 the result is obvious: the dirty child sees t h a t no one else is muddy, so he must be the muddy one. Let us do k = 2. So there are just two dirty children, α and b. E a c h answers "no" the first time, because of the mud on the other. B u t , when b says "no," α realizes t h a t he must be muddy, for otherwise b would have known the mud was on his head and answered "yes" the first time. Thus α answers "yes" the second time. B u t b goes through the same reasoning. Now suppose k = 3; so there are three dirty children, a, 6, c. Child a argues as follows. Assume I don't have mud on my head. Then, by the k = 2 case, both 6 and c will answer "yes" the second time. W h e n they don't, he realizes that the assumption was false, t h a t he is muddy, and so will answer "yes" on the third question. Similarly for b and c. [The general case is similar.] There have been many a t t e m p t s to describe this type of reasoning within a logic, particularly in the context of the wise-men puzzle. Besides McCarthy's own formalization in [48], other formalizations (many carried out at M c C a r t h y ' s instigation) can be found in, for example, [3, 1, 2, 3 8 , 37, 52, 64]. T h e wise-men puzzle is essentially a special case of the muddy children puzzle, where there are three muddy children, and the wise men

Model Checking vs. Theorem Proving

155

are queried sequentially rather t h a n simultaneously. E v e n for this special case, formalizing the puzzle is nontrivial; and all the previous formalizations involve some mechanism (typically using higher-order logic or circumscription) for capturing the deduction capabilities of the agents within the logic. We show how t h e situation c a n be characterized model-theoretically, in a particularly elegant way. To explain our characterization, we need some preliminaries on modal logic and the possible-worlds paradigm. W e want t o describe a n agent's knowledge. T h e traditional way t o do this is t o say t h a t a n agent knows a fact φ if it is true in all worlds t h a t the agent considers possible. We capture this by means of a Kripke structure. A Kripke structure M for η agents is a tuple ( 5 , π , / C i , . . . , / C n ) , where 5 is a set of possible worlds or states, π associates with each world in 5 a t r u t h assignment (propositional or first-

order as the case may b e ) , and /Q is a binary relation on S. Intuitively, the t r u t h assignment π(ιυ) tells us what is true in a world w. T h e relation /Q is intended t o c a p t u r e the possibility relation according t o agent i: (u, v) G /Q if agent i considers world ν possible when in world u. W e then say agent i knows φ a t world u in structure M , and write ( M , u) f= Κΐψ, if φ is true in all t h e worlds ν t h a t agent i considers possible in world u. This captures the intuition t h a t a n agent knows a fact if it is true a t all t h e worlds he considers possible. Now we return t o the muddy children puzzle; much of our discussion is taken from a forthcoming book [22]. First consider t h e situation before the father speaks. Suppose there a r e η children altogether; we number t h e m l , . . . , n . Some of the children have muddy foreheads, while the rest do not. W e c a n describe a possible situation by a n η-tuple of O's and l's of the form ( x i , . . . , xn), where Xi = 1 if child i has a muddy forehead, and Xi = 0 otherwise. Thus, if η = 3 , then a tuple of the form ( 1 , 0 , 1 ) would say that there are exactly two children with muddy foreheads, namely, child 1 and child 3 . Suppose t h e a c t u a l situation is described by this tuple. W h a t situations does child 1 consider possible before the father speaks? Since child 1 c a n see the foreheads of all the children besides himself, his only doubt is about whether he has mud on his own forehead. Thus, child 1 considers two situations possible, namely, ( 1 , 0 , 1 ) ( t h e a c t u a l situation) and ( 0 , 0 , 1 ) . Similarly, child 2 considers two situations possible: ( 1 , 0 , 1 ) and ( 1 , 1 , 1 ) . Note t h a t in general, child i will have the same information in two possible worlds exactly if they agree in all components except possibly the i t h component. W e can capture the general situation by a Kripke structure M consisting n n of 2 states (one for each of the 2 η-tuples). Since child i considers a world possible if it agrees in all components except possibly the ith. component, we

156

Halpern and Vardi

take (s, t) € /C* exactly if s and t agree in all components except possibly the ith component. Notice t h a t this definition makes K% an equivalence relation. To complete the description of the Kripke s t r u c t u r e M , we have to define the t r u t h assignment π. To do this, we have to decide what the primitive propositions are in our language. W e take t h e m to be { p i , . . . , p n , p } , where, intuitively, pi stands for "child i has a muddy forehead," while ρ stands for "at least one child has a muddy forehead." Thus, we define π so t h a t (Af, ( # i , . . . , x n ) ) | = Pi if and only if Xi = 1, and ( M , (xly... , x n ) ) | = ρ if and only if Xj = 1 for some j. Of course, ρ is equivalent to p\ V . . . V p n , so its t r u t h value can be determined from the t r u t h value of the other primitive propositions. There is nothing to prevent us from choosing a language where the primitive propositions are not independent. Since it is convenient to add a primitive proposition describing the father's statement, we do so. This completes the description of M . While this Kripke structure may seem quite complicated, it actually has n an elegant graphical representation. W e c a n think of the 2 states in S as nodes in a graph, and place an edge labeled i between states s and t if i cannot tell apart states s and t (because they agree in all components other than the i component). Suppose we ignore self-loops and the labeling on n the edges for the moment. T h e n we have a structure with 2 nodes, each described by an η-tuple of O's and l's, such t h a t two nodes are joined by an edge exactly if they differ in one component. T h e reader with a good imagination will see t h a t this defines an η-dimensional cube. The case η = 3 is illustrated in Figure 1. Intuitively, each child knows which of the other children have muddy foreheads. This intuition is borne out in our formal definition of knowledge. For example, it is easy to see t h a t when the a c t u a l situation is ( 1 , 0 , 1 ) , we have ( M , ( 1 , 0 , 1 ) ) | = K\^p2, since in both worlds t h a t child 1 considers possible if the a c t u a l situation is ( 1 , 0 , 1 ) , child 2 does not have a muddy forehead. Similarly, we have ( M , ( 1 , 0 , 1 ) ) (= Kip3\ child 1 knows t h a t child 3's forehead is muddy. However, ( M , ( 1 , 0 , 1 ) ) | = -»ΑΊρι. Child 1 does not know t h a t his own forehead is muddy, since in the other world he considers possible—(0,0,1)—his forehead is not muddy. Returning to our analysis of the puzzle, consider what happens after the father speaks. T h e father says p, a fact t h a t is already known to all the children if there are two or more children with muddy foreheads. Nevertheless, the state of knowledge changes, even if all the children already know p. Going back to our example with η = 3, if the initial situation was ( 1 , 0 , 1 ) , although everyone knew before the father spoke t h a t at least one child had a muddy forehead, child 1 considered the situation ( 0 , 0 , 1 ) possible before the father spoke. In t h a t situation, child 3 would consider

Model Checking vs. Theorem Proving

157

(0,0,0)

Figure 1: T h e Kripke structure for the muddy children puzzle with η = 3.

( 0 , 0 , 0 ) possible. Thus, before the father spoke, child 1 thought it was possible t h a t child 3 thought it was possible t h a t none of the children had a muddy forehead. After the father speaks, it is common knowledge t h a t at least one child has a muddy forehead. T h e notion of c o m m o n knowledge, what everyone knows t h a t everyone knows t h a t everyone knows . . . , or, in M c C a r t h y ' s terminology [46], what "any fool" knows, plays a crucial role here, as we shall see. W e can represent the change in the group's state of knowledge graphically (in the general case) by simply removing the point ( 0 , 0 , . . . , 0 ) from the cube, getting a "truncated" cube. (More accurately, what happens is t h a t the node ( 0 , 0 , . . . , 0 ) remains, but all the edges between ( 0 , 0 , . . . , 0 ) and nodes with exactly one 1 disappear, since it is c o m m o n knowledge t h a t even if only one child has a muddy forehead, after the father speaks t h a t child will not consider it possible t h a t no one has a muddy forehead.) T h e situation is illustrated in Figure 2. We next show t h a t each time the children respond to the father's question with a "no," the group's s t a t e of knowledge changes and the cube is further truncated. Consider what happens after the children respond "no"

158

Halpern and Vardi

Figure 2: T h e Kripke structure after the father speaks.

to the father's first question. We claim t h a t now all the nodes with exactly one 1 can be eliminated. (Or, more accurately, the edges to these nodes from nodes with exactly two l's all disappear from the graph. Nodes with one or fewer l's are no longer reachable from nodes with two or more l's. If there are, in fact, two or more children with muddy foreheads, then it is common knowledge among all the children t h a t there are at least two children who have muddy foreheads.) T h e reasoning parallels t h a t done in the "proof" given in the story. If the actual situation were described by, say, the tuple ( 1 , 0 , . . . , 0 ) , then child 1 would initially have considered two situations possible: ( 1 , 0 , . . . , 0) and ( 0 , 0 , . . . , 0 ) . Since once the father speaks it is common knowledge t h a t ( 0 , 0 , . . . , 0 ) is not possible, he would then know t h a t the situation is described by ( 1 , 0 , . . . , 0 ) , and thus would know t h a t his own forehead is muddy. Once everyone answers "no" to the father's first question, it is common knowledge t h a t the situation cannot be ( 1 , 0 , . . . , 0 ) . (Note t h a t here we must use the assumption t h a t it is common knowledge t h a t everyone is intelligent and truthful, and so can do the reasoning required to show ( 1 , 0 , . . . , 0 ) is not possible.) Similar reasoning allows us to eliminate every situation with exactly one 1. Thus, after all the children have answered "no" to the father's first question, it is common knowledge t h a t at least two children have muddy foreheads.

Model Checking vs. Theorem Proving

159

Further arguments in the same spirit can be used to show t h a t after the children answer "no" k times, we can eliminate all the nodes with at most k l's (or, more accurately, disconnect these nodes from the rest of the graph). W e thus have a sequence of Kripke structures, describing the children's knowledge at every step in the process. Essentially, what is going on is t h a t if, in some node s, it becomes c o m m o n knowledge t h a t a node t is impossible, then all edges from every node u reachable from s to t are eliminated. (This situation is even easier t o describe once we add time to the picture. W e return to this point in the next section.) After k rounds of questioning, it is c o m m o n knowledge t h a t at least fc + 1 children have mud on their foreheads. If the true situation is described by a tuple with exactly k + 1 l's, then before the father asks the question for the (k + l ) s t time, those children with muddy foreheads will know the exact situation, and in particular know their foreheads are muddy, and consequently answer "yes." Note t h a t they could not answer "yes" any earlier, since up to this point each child with a muddy forehead considers it possible t h a t he did not have a muddy forehead. We contend t h a t this model-based argument gives a much clearer indication of what is going on with this puzzle t h a n any of the previous formalizations t h a t a t t e m p t e d to describe the theorem-proving abilities of the agents within a formal logic. T h e reader should note t h a t our reasoning about the puzzle applies to any number η of children and any number k < η of dirty children. In contrast, the theorem-proving approaches to the puzzle usually deal with some fixed k and η (typically η = k = 3 ) . T h e ability t o deal with a parametrized family of models, which is important in many instances of common-sense reasoning (cf. [36]), is an important feature of the model-checking approach; see [11, 63].

2.2

What is the Appropriate Semantic Model?

The construction of the semantic model for the muddy children puzzle was somewhat ad hoc. Clearly, if we are to apply the model-theoretic approach in general, we need techniques for describing and constructing the semantic model, and techniques for checking if a formula is true in t h a t model. We consider the first issue in this subsection, and leave the second to the next subsection. Many A I applications deal with agents interacting among themselves 2 a n d / o r with an external environment. In such a setting, we believe t h a t 2

T h e notion of "agent" should be taken rather loosely here. An agent can be a robot observing an environment, a knowledge base (or knowledge bases) being told information, or a processor in a parallel machine. Everything we say applies in all of these contexts.

Halpern and Vardi

160

the possible-worlds paradigm yields t h e appropriate semantic model. Thus, we would like t o model a system of interacting agents as a Kripke structure. To do so, we use the formal model of [21, 25] (which in t u r n is based on earlier models t h a t appeared in [27, 5 5 , 8, 5 9 ] ) . W e briefly describe the formal model here, since we shall make use of it later, referring t h e reader to [25] for more details. W e assume t h a t a t each point in time, each agent is in some local state. Informally, this local s t a t e encodes t h e information it has observed thus far. In addition, there is also a n environment s t a t e , which keeps track of everything relevant t o the system not recorded in t h e agents' states. A

global state is a sequence (se, s\,...,

sn) consisting of the environment state

se and t h e local s t a t e S{ of each agent i. A run of t h e system is a function from time (which, for ease of exposition, we assume ranges over the natural numbers, although we could easily take it t o range over t h e reals) t o global states. Thus, if r is a run, then r ( 0 ) , r ( l ) , . . . is a sequence of global states t h a t , roughly speaking, is a complete description of what happens over time in one possible execution of the system. W e take a system t o consist of a set of runs. Intuitively, these runs describe all the possible sequences of events t h a t could occur in a system. Given a system 1Z, we refer t o a pair (r, m ) consisting of a run r G TZ and a time m as a point. T h e points c a n be viewed as worlds in a Kripke structure. W e say two points ( r , m ) a n d (r',m') such t h a t r(m) =

(se, si,...,

f

sm) and r'(m') = (s'e, s x,...,

s'm) are indistinguishable t o agent

i, and write (r, m) ~i ( r ' , m ' ) , is Si = s[, i.e., if agent i has t h e same local state a t both points. W e take ~i t o play t h e role of the possibility relation /C»; note t h a t /Q is a n equivalence relation under this interpretation. A n interpreted system is a pair (11, π) consisting of a system 1Z together with a mapping π t h a t associates a t r u t h assignment with each point. T h e semantics of knowledge formulas in interpreted systems is identical t o t h a t in Kripke structures. In particular, given a point (r, m ) in a n interpreted system J = (R,, π ) , we have (I, r, m) f= Κχψ if ( J , r', τη') | = φ for all points (r',m') such t h a t (r',m') ~i (r,m). Notice t h a t under this interpretation, an agent knows φ if φ is true a t all t h e situations t h e system could be in, given the agent's current information (as encoded by its local s t a t e ) . For a given distributed protocol, it is often relatively straightforward t o construct t h e system corresponding t o the protocol. T h e local state of each process c a n typically be characterized by a number of internal variables (that, for example, describe t h e messages thus far received and the values of certain local variables), and there is a transition function t h a t describes how the system changes from one global s t a t e t o another. (See, for example, [28] for a detailed example of the modeling process and a knowledge-based

Model Checking vs. Theorem Proving

161

analysis of a distributed protocol.) In the case of the muddy children puzn zle, there are 2 runs, one corresponding t o each initial situation. T h e η-dimensional cube we s t a r t e d with describes the initial situation: t h a t is, the Kripke structure consisting just of the time 0 points. T h e "truncated cubes" are the Kripke structures consisting of time ra points, for m = 1 , 2 , 3 , . . . B y viewing the system as a set of runs, we bring time into the picture in a n a t u r a l way, and see t h a t , in fact, nodes do not disappear, only edges. More accurately, while two runs may be joined by an edge at time ra, there may not be an edge between t h e m at time m + 1 , because the agent t h a t could not distinguish t h e m at time ra acquired some information t h a t allowed him t o distinguish t h e m at time m +1. (See [25] for a detailed description of the set of runs corresponding to the muddy children puzzle.) For knowledge-based applications, the modeling problem may be harder because of the difficulty in describing the s t a t e space. Basically, it is not clear how to describe the states when they contain information about agents' knowledge. In this c o n t e x t , the knowledge structures approach suggested in [20] may prove useful (cf. [31, 3 9 ] ) . Suppose we have constructed a model of the system. Provided t h a t we can completely characterize an agent's local s t a t e s by a formula φ8, then we can reduce the problem of checking whether an agent in local s t a t e s knows φ to checking the validity of φ8 => φ, since agent i knows φ in local state s if φ is t r u e at all possible worlds where its local s t a t e is s. However, being able to characterize an agent's local s t a t e by means of a formula will almost certainly require the use of a very expressive logic, for which theorem proving is quite intractable. T h e model-checking approach, on the other hand, does not require t h a t local states be encodable as formulas. Thus, it does not require resorting to a logic t h a t is more expressive t h a n necessary to express the assertion φ. An important issue to consider when comparing the two approaches is that of representation. T h e theorem-proving approach requires us to represent the agent's knowledge by a collection of formulas in some language. T h e model-checking approach instead represents the agent's knowledge as a local s t a t e in some structure. W e have, however, complete freedom t o decide on the representation of the local s t a t e and the structure. There are applications t h a t arise frequently in A I where the logical choice for the representation is in terms of formulas. For example, the problem-solving system S T R I P S represents a s t a t e by a set of first-order formulas; operations on the s t a t e are represented by adding certain formulas and deleting others [23]. As another example, Levesque [41] studies knowledge bases ( K B s ) where we have T E L L and A S K operations. After the K B is told a sequence of facts, we should be able to represent the situation using the

Halpern and Vardi

162 set κ, of facts t h a t it has been t o l d .

3

4

W h a t Kripke structure does this

represent? The first thing we have t o decide is what the global states are going to be. There is only one agent in the system, namely, the K B itself, so a global state will be a pair consisting of the environment state and the K B ' s local state.

As we said above, we c a n identify the K B ' s local state with

the formulas t h a t it has been told. W h a t about the environment?

Since

we assume t h a t there is a real world external t o the K B , we c a n take the environment t o be a complete description of the relevant features of the world. W e model this by taking the environment s t a t e t o be a relational structure.

5

Thus, a global state is a pair (Α, κ), where A is a relational

structure (intuitively, describing the world) and κ is a formula (intuitively, the conjunction of the facts the K B has been told). Next we have t o decide what the allowable global states are. This turns out t o be quite sensitive t o t h e expressiveness of the K B . Suppose we assume t h a t the K B is assertion-based, i.e, it is told only facts in the assertion language and no facts about its knowledge. T h e n we c a n restrict attention to global states (^4, κ) where the formula κ is true in A.

(This captures

the assumption t h a t the K B is only told true facts about the world.) If we make no further restrictions, then, according t o our définition of knowledge, the knowledge base K B knows a fact φ in a state κ if φ holds in all the global states of the form (Α, κ). B u t t h a t means t h a t the K B knows φ in a state κ if φ holds in all the models of κ or, equivalently, if ψ is a logical consequence of κ. This example already shows how the model-checking approach c a n be 3

To model the evolution of the knowledge base, one has t o use some temporal structure where each point is represented by a set of formulas. See [50] and references therein for a discussion of such a model. 4 Although it may indeed be reasonable in many cases t o represent a situation by the set of facts an agent has been told, note that by doing so we are assuming (among other things) that the K B is being told facts about a static situation, so that when the information arrived is irrelevant. For example, if we consider propositions whose truth may change over time, it may be that at some point the K B is told φ and then later it is told -up. We do not necessarily want t o view the K B as being in an inconsistent state at this point. We can get around this difficulty by augmenting our representation to include time, so that "φ at time 3" would not be inconsistent with "-up at time 4." We can also imagine a more sophisticated K B that discards an earlier fact if it is inconsistent with later information. However, such an approach quickly leads t o all the difficulties one encounters in general with updating beliefs. (See, e.g., [19, 24].) For example, what do we do if we have three facts that are pairwise consistent, but whose conjunction is inconsistent? Which of the three facts do we discard? 5 A relational structure consists of a domain of individual elements, and an assignment of functions and relations to the function and relation symbols of the assertion language.

Model Checking vs. Theorem Proving

163

viewed as a generalization of the theorem-proving approach. B u t the modelchecking approach gives us added flexibility. F o r example, consider t h e closed-world assumption [58], where any fact not explicitly stored in the database is taken t o be false. T h e need for such an assumption about negative information stems from t h e fact t h a t in any complex application, the number of negative facts vastly outnumbers the number of positive ones, so that it is totally infeasible t o explicitly represent the negative information in the database. W e c a n c a p t u r e this assumption model-theoretically by restricting attention t o pairs (Α, κ) where not only is κ t r u e in A, but every atomic formula P ( d i , . . . , dn) not implied by κ is false in A. While the closed-world assumption can be formalized in first-order logic, which means that we c a n use t h e theorem-proving approach t o query closed-world knowledge bases [58], analyzing the problem from t h e model-checking perspective leads t o a complete characterization of the complexity of query answering, and furthermore, it enables one t o deal with higher-order queries, which are not usually amenable t o the theorem-proving approach [68]. T h e closed-world assumption is only one of many we might consider making t o restrict attention t o only certain (Α, κ). As Shoham [61] observed, most forms of nonmonotonic reasoning c a n be viewed as a t t e m p t s to restrict attention t o some collection of preferred pairs. T h e focus in many of the works on nonmonotonic reasoning is on using some logical formalism to describe the set of preferred models. This is necessary if one wishes to apply the theorem-proving approach. Our contention is t h a t theorem proving is just one way t o evaluate queries and not necessarily the optimal way. F r o m our point of view, the right question is not whether one c a n prove if a n assertion φ follows from a K B using some nonmonotonic logic, but rather if φ holds in the Kripke structure represented by κ (where the nonmonotonicity is captured by restricting attention t o a preferred set of possible worlds). Even without the complication arising from the notion of preference, the situation gets more complicated if we assume t h a t t h e K B is told facts that include information about its own knowledge, i.e., the specification is knowledge-based. A s pointed out in [41], this c a n be quite important in practice. F o r example, suppose a K B is told the following facts about a small group of people: "John is a teacher," "Mary is a teacher," and "you know about all t h e teachers." T h e n we expect t h e K B t o know t h a t Bill is not a teacher (assuming it knows t h a t Bill is distinct from M a r y and J o h n ) . Intuitively, this is because the three assertions together restrict the set of possible worlds t o ones where only J o h n a n d M a r y are teachers. How c a n we model this? T h a t is, what pairs (Α, κ) should we allow now (where κ is the conjunction of t h e three formulas mentioned above)?

Halpern and Vardi

164

Clearly, taking pairs (Α, κ) such t h a t κ is true in A will not work, since a relational structure A does not determine the t r u t h of formulas involving knowledge. T h e answer is t h a t when we have a knowledge-based specification, we cannot select the allowable states one at a time, even if we allow some notion of preference. R a t h e r , it is the collection of all possible states that has to be consistent with the K B . Thus, we cannot just focus on the global states; we have t o consider the whole resulting Kripke structure. Intuitively, we are trying t o describe a Kripke structure consisting of a collection of pairs (Α, κ) such t h a t κ is true in the resulting Kripke structure. (Notice t h a t the t r u t h of κ depends on the whole Kripke structure, not just on A.) J u s t as there are potentially m a n y states consistent with a given assertionbased specification, there are potentially many Kripke structures consistent with a given knowledge-based specification. W e c a n use some notion of preference t o select a unique Kripke structure. A popular notion of preference is one t h a t tries t o circumscribe the K B ' s knowledge. T h e idea is to view the knowledge specified in the K B as all that is known. Thus, this preference can be viewed as a closed-world assumption at the knowledge level. This notion has received a lot of attention in the past decade [20, 2 6 , 37, 4 0 , 4 1 , 4 3 , 54, 67]. So far we have considered only situations with a single agent. In such situations, Kripke structures degenerate t o essentially sets of worlds. Many AI applications, however, deal with multiple interacting agents. It is when we t r y to give formal semantics t o sentences such as "Dean doesn't know whether Nixon knows t h a t Dean knows t h a t Nixon knows about the Watergate break-in" t h a t the full power of Kripke structures comes into play. In such situations, circumscribing the agents' knowledge is highly nontrivial (cf. [20, 54, 6 7 ] ) . To summarize, in relatively simple settings it m a y not be too difficult to describe, implicitly or explicitly, the set of runs t h a t characterize the system, and thus we c a n get a Kripke structure, which is the appropriate semantic model. Preference criteria and defaults m a y make this task more difficult, but we expect t h a t even in this case, the task will become manageable as our understanding of these notions deepens.

2.3

Model Checking

Suppose we have somehow constructed what we consider t o be the appropriate semantic model. W e now want to check if a given formula φ is true in a particular s t a t e of t h a t model. Checking whether an arbitrary formula in the extended language holds

Model Checking vs. Theorem Proving

165

in a given world of a given Kripke structure has two components: we have t o be able t o check whether assertions (i.e., knowledge-free formulas) hold in a given s t a t e (we call this assertion checking), and check whether knowledge formulas (i.e., formulas of the form Κχψ) hold in a given state. If we can do both of these tasks, then we c a n handle arbitrary formulas by induction on the structure of the formula. Since checking whether a knowledge formula holds involves quantification over t h e possible worlds, t h e complexity of model checking depends on (and is typically polynomial in t h e product of) three quantities: the complexity of assertion checking, the size of the given Kripke structure, and t h e size of the formula. Thus, our ability t o do model checking efficiently depends crucially on our ability t o do assertion checking efficiently and our ability t o deal with "large" structures. For propositional languages, assertion checking is quite easy: it c a n be done in linear time. Adding modalities for knowledge does not significantly complicate things. Given a finite Kripke s t r u c t u r e M = (S, π, / C i , . . . , / C n ) , define | | M | | t o be t h e sum of the number of states in S and the number of pairs in / Q , i = 1 , . . . , n. Thus, | | M | | is a measure of the size of the Kripke structure M. L e t \φ\ be the length of φ, viewed as a string of symbols.

Proposition 2 . 1 . There is an algorithm that checks if a propositional formula φ is satisfied in a structure M. The algorithm runs in time 0(\\M\ \ χ

Μ)· Proof. L e t φι,...,

φ^ be the subformulas of φ, listed in order of length, with ties broken arbitrarily. Thus, we have φ*, = φ, and if φι is a subformula of ipj, then i < j. It is easy t o check t h a t there are a t most \φ\ subformulas of φ, so we must have k < \φ\. A n easy induction on k' shows t h a t we can label each s t a t e s in M with ipj or -«pj, for j = 1 , . . . , k', depending on whether or not φ$ is true a t s, in time 0 ( f c ' | | M | | ) . T h e only nontrivial case is if φ$ is of t h e form Κιφγ, where j' < j. W e label a s t a t e s with Kiipy iff each s t a t e t such t h a t (s,t) G Ki is labeled with φγ. Assuming inductively t h a t each s t a t e has already been labeled with φγ or - « y y , this step c a n clearly be carried out in time 0 ( | | M | | ) , as desired. I Things get more complicated once we move t o first-order languages. E v e n assertion checking m a y be difficult. If we consider structures with infinite domains, it is not always clear how t o represent them, let alone do 6 assertion checking. E v e n if we restrict attention t o structures with finite domains, assertion checking m a y be very difficult due t o the size of the domain. In general, t o check t h e t r u t h of a formula such as Μχφ(χ), we may have t o check t h e t r u t h of φ(ά) for each domain element d. 6

R e c e n t works [35, 34] address special cases of assertion checking for infinite structures.

166

Halpern and Vardi

Even if assertion cheeking is easy, we still have to cope with the multitude of possible worlds. For example, in the context of incomplete knowledge bases, the number of possible worlds c a n be exponential in the size of the knowledge base, which makes query evaluation intractable [68]. We note, however, t h a t the fact t h a t the number of possible worlds is quite large does not automatically mean t h a t it is hard t o check all knowledge formulas. T h e results of [12, 11] demonstrate t h a t some knowledge formulas of interest c a n be evaluated efficiently in certain contexts despite an exponential number of possible worlds. W h a t can we do to deal with situations where the number of possible worlds or the number of domain elements is too large to handle? There are a number of approaches one could pursue. For one thing, we might hope to be able to find restricted (but still interesting) subclasses of formulas for which we c a n do model checking efficiently in certain restricted (but still interesting) subclasses of structures, just as we now have techniques for doing inference efficiently with certain subclasses of formulas (such as Horn clauses). Indeed, IS-A formulas (which essentially correspond to Horn clauses, although see [6] for some caveats) c a n be checked very efficiently in IS-A hierarchies. A second approach is to consider heuristics and defaults. Since the difficulty in model checking arises when we have large structures a n d / o r large domains, we need good heuristics for handling such situations. We in fact use such heuristics all the time in our daily life. For example, if there are too many possibilities (i.e., too many possible worlds), we focus attention on only a few (the ones we deem t o be the "most relevant" according to some metric or "most likely" according t o some probability distribution), ignoring the rest. Consider a situation where Alice is asked if (she knows t h a t ) φ is the case. To answer this question, Alice would have to check all the worlds she considers possible t o see if φ holds in all of them. There may be 1 , 0 0 0 , 0 0 0 worlds t h a t Alice considers possible, which means t h a t she has a lot of checking t o do. However, if Alice has a probability space on these worlds, then there may be a subset of worlds of size 1 , 0 0 0 t h a t has probability .99. Alice could check these 1 , 0 0 0 worlds; if φ holds in all of them, then this might be taken as sufficient "evidence" for Alice to say t h a t φ indeed holds. E v e n without explicit probability, Alice might still have a notion of which of the 1 , 0 0 0 , 0 0 0 worlds are most likely or most relevant. Similarly, if we have a large domain and want t o check the t r u t h of a formula such as Vx
Model Checking vs. Theorem Proving

167

"relevant" t o the problem, and identify all worlds t h a t agree on irrelevant features, thus cutting down on the search space. Being able t o do this depends both on being able t o identify relevant notions and on being able to check relevant features rapidly. Going back t o our previous example, suppose φ is a propositional formula such as (pi V p2) Λ (->p2 V P 3 ) , where P i , p2, and p3 are primitive propositions. It is easy t o see t h a t φ is true unless pi and at least one of p2 or p3 are false. T h e t r u t h values of all other propositions are clearly irrelevant t o the t r u t h of φ. Depending on how Alice's possible worlds are represented, it may be t h a t Alice c a n check that Alice does not consider a world possible where pi and at least one of P2 and p3 are false, without checking all 1 , 0 0 0 , 0 0 0 possible worlds. We remark t h a t formalisms for expressing and reasoning with irrelevance have been developed, both probabilistic (for example, [56]) and non-probabilistic [65], but this is an area where much further work remains t o be done. As all these approaches show, much depends on precisely how the worlds are generated and represented. Finding appropriate representations is, of course, a major open problem. A great deal of effort in theorem proving has been expended on finding heuristics t h a t work well for the formulas t h a t arise in practice. Analogously, in model checking, it would be useful t o find heuristics t h a t work well for structures t h a t arise in practice. Although the work on model checking in this regard is still in its infancy, the early results appear quite promising [33, 7]. For example, when verifying circuits, we are not dealing with arbitrary Kripke structures. T h e regularity in the structure suggests 20 heuristics t h a t seem to work well in structures with up t o 1 0 states [7]! Of course, techniques need to be developed t o allow us to use these heuristics and defaults in a principled way. Much of the recent work in "vivifying" c a n be viewed as being in this spirit [16, 4 2 ] . T h e idea is t o cut down on the number of possibilities by filling in some (hopefully) inessential details. Work on a possible-worlds framework for probability [4, 18, 3 0 , 53] may provide insights into using probabilistic heuristics in a principled way to cut down the search space. Notice t h a t by using defaults in this way, we are led naturally t o nonmonotonicity. However, rather t h a n the logic being nonmonotonic, the model checking is nonmonotonic. W e might withdraw some of our conclusions about a formula being true in a given structure if we get further information t h a t leads us t o believe t h a t the default assumptions we used to simplify the model-checking problem are not correct.

Halpern and Vardi

168

3

Discussion and Conclusions

We have argued here for a model-theoretic rather t h a n a proof-theoretic approach to reasoning about knowledge. W e do not mean to suggest t h a t the model-theoretic approach is a panacea. There are difficulties to be overcome here, involving how to find the model and how t o do model checking on large structures, just as there are difficulties in the theorem-proving approach. Our hope is t h a t the model-theoretic perspective will suggest new heuristics and new approaches. W e feel t h a t many of the approaches t h a t are currently being tried, including the preferred models approach to nonmonotonic reasoning and the idea of "vivifying" knowledge bases, are also best understood in the context of the model-checking framework. Thus, this perspective may allow us to unify a number of current lines of research. T h e question still remains, for any particular application, whether to use the model-theoretic or proof-theoretic approach. At some level, it could be argued t h a t this is a non-question; at a sufficiently high level, the two approaches converge: W i t h a sufficiently rich logic, we can certainly characterize a model inside the logic, and there is nothing in the proof-theoretic approach t h a t prevents us from doing our theorem proving by using model checking. Similarly, as we have suggested in the previous sections, the model-checking approach c a n capture theorem proving: we simply look at the class of all models consistent with the axioms. Assuming our logic has a complete axiomatization, if a fact is true in all of them, then it is provable. However, this convergence obscures the fact t h a t the two approaches have rather different philosophies and domains of applicability. T h e theoremproving approach is more appropriate when we do not know what the models look like and the best way to describe t h e m is by means of axioms. In contrast, the model-checking approach is appropriate when we do know what the models look like and we c a n describe t h e m rather precisely, as in the muddy children puzzle. We would argue, however, t h a t the model-checking approach has one major advantage over the theorem-proving approach, which is perhaps not immediately obvious from the way we have described them. T h e theoremproving approach implicitly assumes t h a t the language for describing the situation (i.e., the system or the model) is the same as the language for describing the properties of the system in which we are interested. For example, the situation calculus has been used for both purposes since McCarthy first introduced it in 1 9 5 8 (when the first version of [47] appeared). T h a t we may need rather powerful languages for describing a situation (typically far more powerful t h a n those needed for describing the properties we want to prove about the system) is evident already in the muddy children

Model Checking vs. Theorem Proving

169

and the wise-men puzzles. T h e model-checking approach allows us to effectively decouple the description of the model from the description of the properties we want to prove about the model. This decoupling is useful in many cases besides those where circumscription is an issue. Consider the case of programming languages. Here we c a n view the p r o g r a m as describing the system. (There will be a particular set of runs corresponding to each p r o g r a m . ) W e may then want to prove properties of the p r o g r a m (such as termination, or some invariants). It seems awkward to require t h a t the properties of the program be expressed in the same language as the model of the program, but t h a t is essentially what would be required by a theorem-proving approach. These examples suggest t h a t when using the model-checking approach, it will probably be useful to have techniques for constructing more complicated models from simpler models, using various constructors. Work on such techniques has been a m a j o r focus in the study of programming languages for years. (A typical example is the work on C C S by Milner and his colleagues [49].) Indeed, in almost any discipline where complicated systems need to be analyzed, there are techniques for constructing models for a complicated system from simpler components. More emphasis on a model-checking approach might lead to more work along these lines. T h e hope would be t h a t if we have a better understanding of where the models are coming from, we might be able to devise better techniques for doing model checking. In summary, we do not expect the model-checking approach to supplant the theorem-proving approach. R a t h e r , we would hope t h a t techniques from each one can inspire advances in the other. In any case, we hope this manifesto inspires further research on model-checking techniques from the AI perspective.

Acknowledgments This is an expanded version of a paper t h a t appears in Principles of Knowl-

edge Representation and Reasoning: Procedings of the Second International Conference, 1 9 9 1 , J . A. Allen, R . Fikes, and E . Sandewall, editors. T h e ideas in this paper have been brewing for a while. (A brief paragraph discussing some of these issues appeared in [29].) Discussions with Hector Levesque and Y o r a m Moses helped us formulate these issues. W e would like to thank R o n Fagin, A d a m Grove, David Israel, Steve Kaufmann, Gerhard Lakemeyer, and Hector Levesque for useful comments on an earlier draft of this chapter.

Halpern and Vardi

170

References [I] L . C. Aiello, D. Nardi, and M. Schaerf. Y e t Another Solution to the

Three Wisemen Puzzle. In Proc. 3rd InVl. Symp. on Methodologies for Intelligent Systems, pages 3 9 8 - 4 0 7 , 1 9 8 8 . [2] L . C. Aiello, D. Nardi, and M. Schaerf.

Reasoning about

Knowledge

and Ignorance. In Proc. Int'I. Conf. on Fifth Generation Systems, pages 618-627, 1989. [3] G. A t t a r d i and M. Simi. Reasoning across Viewpoints. EC AI 81 pages 3 1 5 - 3 2 5 , 1984.

In Proc. of

[4] F . Bacchus. Representing and Reasoning with Probabilistic Knowledge. M I T Press, Cambridge, M A , 1 9 9 0 . [5] J . Barwise. Scenes and Other Situations. L X X V I I I ( 7 ) : 3 6 9 - 3 9 7 , 1981.

Journal

of Philosophy,

[6] R . B r a c h m a n . I Lied about the Trees (or, Defaults and Definitions in Knowledge Representation). The AI Magazine, 6 ( 3 ) : 8 0 - 9 3 , 1 9 8 5 . [7] J . R. Burch, Ε . M. Clarke, D. L . Dill, J . Hwang, and K . L . McMillan. 20 Symbolic Model Checking: 1 0 States and Beyond. In Proc. 5th IEEE

Symp. on Logic in Computer Science, pages 4 2 8 - 4 3 9 , 1990. [8] Κ. M. Chandy and J . Misra. How Processes Learn. Distributed Com-

puting, l ( l ) : 4 0 - 5 2 , 1986. [9] Ε . M. Clarke, E . A. Emerson, and A. P. Sistla. A u t o m a t i c Verification of Finite-State Concurrent Systems Using Temporal Logic Specifi-

cations. A CM Trans, on Programming Languages and Systems, 8 ( 2 ) : 2 4 4 263, 1986. A n early version appeared in Proceedings of the 10th

ACM

Symposium on Principles of Programming Languages, 1983. [10] Ε . M. Clarke and O. Grümberg. Research on A u t o m a t i c Verification and Finite-State Concurrent Systems. In J . F . T r a u b , B . J . Grosz, B . W .

Lampson, and N. J . Nilsson, editors, Annual Review of Computer Science, Vol. 2, pages 2 6 9 - 2 8 9 . Annual Reviews Inc., Palo Alto, C A , 1987. [II] Ε . M. Clarke, Ο. Grümberg, and M. Browne. Reasoning about Networks with Many Finite-State Processes. In Proc. 5th ACM Symp. on

Principles of Distributed Computing, pages 2 4 0 - 2 4 8 , 1986.

Model Checking vs. Theorem Proving

171

[12] C. Dwork and Y . Moses. Knowledge and C o m m o n Knowledge in a Byzantine Environment: Crash Failures. Information and Computation, 8 8 ( 2 ) : 1 5 6 - 1 8 6 , 1990. [13] E . A. E m e r s o n and Ε . M. Clarke. Using Branching T i m e Temporal Logic to Synthesize Synchronization Skeletons. Science of Computer

Programming, 2 : 2 4 1 - 2 6 6 , 1982. [14] E . A. E m e r s o n and J . Y . Halpern. Decision Procedures and E x p r e s siveness in the Temporal Logic of Branching Time. Journal of Computer

and System Sciences, 3 0 ( l ) : l - 2 4 , 1985. [15] E . A. Emerson, T . Sadler, and J . Srinivasan. Efficient Temporal Rea-

soning. In Proc. 16th ACM Symp. on Principles of Programming Languages, pages 1 6 6 - 1 7 8 , 1 9 8 9 . [16] D. Etherington, A. Borgida, R . J . B r a c h m a n , and H. K a u t z . Vivid Knowledge and Tractable Reasoning: Preliminary R e p o r t . In Eleventh

International

Joint Conference

on Artificial Intelligence

(IJCAI-89),

pages 1 1 4 6 - 1 1 5 2 , 1989. [17] R . Fagin and J . Y . Halpern. Belief, Awareness, and Limited Reasoning.

Artificial Intelligence, 3 4 : 3 9 - 7 6 , 1988. [18] R . Fagin and J . Y . Halpern.

Uncertainty, Belief, and Probability.

In Eleventh International Joint Conference

on Artificial Intelligence

(IJCAI-89), pages 1 1 6 1 - 1 1 6 7 , 1 9 8 9 . A n expanded version appears as I B M Research R e p o r t R J 6 1 9 1 , April 1 9 8 8 , and will appear in Compu-

tational Intelligence. [19] R . Fagin, J . D. Ullman, and M. Y . Vardi. On the Semantics of Up-

dates in Databases. In Proc. 2nd ACM Symp. on Principles of Database Systems, pages 3 5 2 - 3 6 5 , 1 9 8 3 . [20] R . Fagin, J . Y . Halpern, and M. Y . Vardi. A Model-Theoretic Anal-

ysis of Knowledge: Preliminary Report. In Proc. 25th IEEE Symp. on Foundations of Computer Science, pages 2 6 8 - 2 7 8 , 1984. A revised and expanded version appears as I B M Research R e p o r t R J 6 6 4 1 , 1 9 8 8 , and will appear in Journal of the ACM. [21] R . Fagin, J . Y . Halpern, and M. Y . Vardi. W h a t C a n Machines Know?

On T h e Epistemic Properties of Machines. In Proc. of National Confer-

ence on Artificial Intelligence (AAAI-86),

pages 4 2 8 - 4 3 4 , 1986. A re-

vised and expanded version appears as I B M Research R e p o r t R J 6 2 5 0 ,

Halpern and Vardi

172

1988, under the title "What C a n Machines Know? On the Properties of Knowledge in Distributed Systems," and will appear in Journal of the

ACM. [22] R . Fagin, J . Y . Halpern, Y . Moses, and M. Y . Vardi. Reasoning about

Knowledge.

1 9 9 1 . To appear.

[23] R . E . Fikes and N. J . Nilsson. S T R I P S : A New Approach to the Application of Theorem Proving to Problem Solving. Artificial Intelligence, 2 : 1 8 9 - 2 0 8 , 1972. [24] P. Gärdenfors. Knowledge in Flux. Cambridge University Press, 1988. [25] J . Y . Halpern and R . Fagin.

Modelling Knowledge and Action in

Distributed Systems. Distributed Computing, 3 ( 4 ) : 1 5 9 - 1 7 9 , 1 9 8 9 . [26] J . Y . Halpern and Y . Moses.

Towards a Theory of Knowledge

and

Ignorance. In Proc. AAAI Workshop on Non-monotonic Logic, pages 1 2 5 - 1 4 3 , 1984. Reprinted in Logics and Models of Concurrent Systems, (ed., Κ. A p t ) , Springer-Verlag, Berlin/New York, pp. 4 5 9 - 4 7 6 , 1 9 8 5 . [27] J . Y . Halpern and Y . Moses. Knowledge and C o m m o n Knowledge in a Distributed Environment. Journal of the ACM, 3 7 ( 3 ) : 5 4 9 - 5 8 7 , 1990.

An early version appeared in Proceedings of the 3rd ACM Symposium on

Principles of Distributed Computing, 1984. [28] J . Y . Halpern and L . D. Zuck. A Little Knowledge Goes a Long Way: Simple Knowledge-Based Derivations and Correctness Proofs for a F a m -

ily of Protocols. In Proc. 6th ACM Symp. on Principles of Distributed Computing, pages 2 6 9 - 2 8 0 , 1987. A revised and expanded version appears as I B M Research R e p o r t R J 5857, 1987, and will appear in Journal

of the ACM. [29] J . Y . Halpern. Using Reasoning about Knowledge to Analyze Distributed Systems. In J . F . Traub, B . J . Grosz, B . W . Lampson, and

N. J . Nilsson, editors, Annual Review of Computer Science, Vol. 2, pages 3 7 - 6 8 . Annual Reviews Inc., Palo Alto, C A , 1987. [30] J . Y . Halpern. An Analysis of First-Order Logics of Probability.

Ar-

tificial Intelligence, 4 6 : 3 1 1 - 3 5 0 , 1990. [31] S. J . Hamilton and J . P. Delgrande. An Investigation of Modal Structures as an Alternative Semantic Basis for Epistemic Logics. Computa-

tional Intelligence, 5:82-96, 1989.

Model Checking vs. Theorem Proving [32]

173

J . Hintikka. Impossible Possible Worlds Vindicated. Journal of Philo-

sophical Logic, 4:475-484, 1975. [33]

G. J . Holtzmann. A n Improved P r o t o c o l Reachability Analysis Tech-

nique. Software Practice and Experience, 1 8 ( 2 ) : 1 3 7 - 1 6 1 , 1988. [34]

F . K a b a n z a , J . - M . Stevenne, a n d P. Wolper. Handling Infinite Tempo-

ral Data. In Proc. 9th ACM Symp. on Principles of Database Systems, pages 3 9 2 - 4 0 3 , 1 9 9 0 . [35]

F . Kanellakis, G. M. Kuper, and P. Z. Revesz.

Constraint Query

Languages. In Proc. 9th ACM Symp. on Principles of Database Systems, pages 2 9 9 - 3 1 3 , 1 9 9 0 . [36]

S. G. Kaufmann.

A Formal T h e o r y of Spatial Reasoning.

Allen, R . Fike, and E . Sandewall, editors, Principles

In J . A.

of Knowledge

Rep-

resentation and Reasoning: Proceedings of the Second International Conference,

pages 3 4 7 - 3 5 6 , San Mateo, C A , 1 9 9 1 . Morgan Kaufmann.

[37] Κ . Konolige. Circumscriptive Ignorance. In Proc. of National Confer-

ence on Artificial Intelligence (AAAI-84),

pages 2 0 2 - 2 0 4 , 1984.

[38] V. K u o . A Formal N a t u r a l Deduction System about Knowledge. Unpublished manuscript, C o m p u t e r Science Dept., Stanford University, 1984. [39] P h . Lejoly and M. Minsoul. published manuscript, 1 9 9 0 . [40]

A Subjective Logic of Knowledge. Un-

H. J . Levesque. T h e Interaction with Incomplete Knowledge Bases: A

Formal Treatment. In Seventh International Joint Conference on Artificial Intelligence (IJCAI-81), pages 2 4 0 - 2 4 5 , 1981. [41] H. J . Levesque. Foundations of a Functional Approach t o Knowledge

Representation. Artificial Intelligence, 2 3 : 1 5 5 - 2 1 2 , 1984. [42] H. J . Levesque. Making Believers out of Computers. Artificial Intelli-

gence, 3 0 : 8 1 - 1 0 8 , 1986. [43]

H. J . Levesque. All I Know: A Study in Autoepistemic Logic. Artificial

Intelligence, 4 2 ( 3 ) : 2 6 3 - 3 0 9 , 1990. [44] O. Lichtenstein a n d A . Pnueli. Checking the Finite-State Concurrent P r o g r a m s Satisfy Their Linear Specifications. In Proc. 13th ACM Symp.

on Principles of Programming Languages, pages 9 7 - 1 0 7 , 1985.

Halpern and Vardi

174

[45] Z. M a n n a and A. Pnueli. Verification of Temporal Programs: T h e Temporal Framework. In R . S. Boyer and J . S. Moore, editors, The

Correctness Problem in Computer Science. Academic Press, New York, 1981. [46] J . McCarthy, M. Sato, T . Hayashi, and S. Igarishi. On the Model Theory of Knowledge. Technical R e p o r t S T A N - C S - 7 8 - 6 5 7 , Stanford University, 1 9 7 9 . [47]

J . McCarthy. P r o g r a m s with C o m m o n Sense. In M. Minsky, editor,

Semantic Information Processing, pages 4 0 3 - 4 1 8 . T h e M I T Press, Cambridge, M A , 1 9 6 8 . P a r t of this article is a reprint from an an article by

the same title, in Proc. Conf. on the Mechanization of Thought Processes, National Physical Laboratory, Teddington, England, Vol. 1, pp. 7 7 - 8 4 , 1958. [48] J . McCarthy. Formalization of T w o Puzzles Involving Knowledge. Unpublished manuscript, Computer Science Dept., Stanford University, 1978.

[49] R . Milner. A Calculus of Communicating Systems, volume 92 of Lecture Notes in Computer Science. Springer-Verlag, Berlin/New York, 1980. [50] P. H. Morris and R . A. Nado. Representing Actions with an Assumption-Based T r u t h Maintenance System. In Proc. of National

Conference on Artificial Intelligence (AAAI-86),

pages 13-17, 1986.

[51] Y . Moses and M. R . Tuttle. P r o g r a m m i n g Simultaneous Actions Using C o m m o n Knowledge. Algorithmica, 3 : 1 2 1 - 1 6 9 , 1 9 8 8 . [52]

M. A. Nait Abdallah. A Logico-Algebraic Approach t o the Model The-

ory of Knowledge. Theoretical Computer Science, 6 6 ( 2 ) : 2 0 5 - 2 3 2 , 1989. [53]

N. Nilsson. Probabilistic Logic. Artificial Intelligence, 2 8 : 7 1 - 8 7 , 1 9 8 6 .

[54] R . Parikh. Logics of Knowledge, Games, and Dynamic Logic. In FST-TCS, Lecture Notes in Computer Science, Vol. 1 8 1 , pages 2 0 2 - 2 2 2 . Springer-Verlag, Berlin/New York, 1 9 8 4 . [55]

R . Parikh and R . R a m a n u j a m . Distributed Processing and the Logic

of Knowledge. In R . Parikh, editor, Proc. of the Workshop on Logics of Programs, pages 2 5 6 - 2 6 8 , 1 9 8 5 .

Model Checking vs. Theorem Proving

175

[56] J. Pearl and T. Verma. The Logic of Representing Dependencies by Directed Graphs. In Proc. of National Conference on Artificial Intelligence (AAAI-87), pages 374-379, 1987. [57] J. P. Queille and J. Sifakis. Specification and Verification of Concurrent Systems in CESAR. In Proc. 5th Int'l Symp. on Programming, 1981. [58] R. Reiter. Towards a Logical Reconstruction of Relational Database Theory. In M. L. Brodie, J. Mylopoulos, and J. W. Schmidt, editors, On Conceptual Modelling, pages 191-233. Springer-Verlag, Berlin/New York, 1984. [59] S. J. Rosenschein and L. P. Kaelbling. The Synthesis of Digital Machines with Provable Epistemic Properties. In J. Y. Halpern, editor, Theoretical Aspects of Reasoning about Knowledge: Proceedings of the 1986 Conference, pages 83-97, San Mateo, CA, 1986. Morgan Kaufmann. [60] S. J. Rosenschein. Formal Theories of AI in Knowledge and Robotics. New Generation Computing, 3:345-357, 1985. [61] Y. Shoham. A Semantical Approach to Nonmonotonic Logics. In Proc. 2nd IEEE Symp. on Logic in Computer Science, pages 275-279, 1987. [62] A. P. Sistla and Ε. M. Clarke. The Complexity of Propositional Linear Temporal Logics. Journal of the ACM, 32(3):733-749, 1985. [63] A. P. Sistla and S. M. German. Reasoning with Many Processes. In Proc. 2nd IEEE Symp. on Logic in Computer Science, pages 138-152, 1987. [64] W. R. Stark. A Logic of Knowledge. Zeitschrift für Mathematische Logik und Grundlagen der Mathematik, 27:371-374, 1981. [65] D. Subramanian and M. R. Genesereth. The Relevance of Irrelevance. In Tenth International Joint Conference on Artificial Intelligence (IJCAI-87), pages 416-422, 1987. [66] M. Y. Vardi. The Complexity of Relational Query Languages. In Proc. 14th ACM Symp. on Theory of Computing, pages 137-146, 1982. [67] M. Y. Vardi. A Model-Theoretic Analysis of Monotonie Knowledge. In Ninth International Joint Conference on Artificial Intelligence (IJCAI85), pages 509-512, 1985.

Halpern and Vardi

176 [68] M. Y . Vardi. Querying Logical Databases.

Journal of Computer

and

System Sciences, 3 3 : 1 4 2 - 1 6 0 , 1986. [69] M. Y . Vardi and P. Wolper.

An A u t o m a t a - T h e o r e t i c Approach to

Automatic P r o g r a m Verification. In Proc. 1st IEEE Symp. on Logic in Computer Science, pages 3 3 2 - 3 4 4 , 1 9 8 6 .