Intention recognition in manufacturing applications

Robotics and Computer-Integrated Manufacturing 33 (2015) 29–41

Contents lists available at ScienceDirect

Robotics and Computer-Integrated Manufacturing journal homepage: www.elsevier.com/locate/rcim

Intention recognition in manufacturing applications Craig Schlenoff a,n, Zeid Kootbally b, Anthony Pietromartire a, Marek Franaszek d, Sebti Foufou a,c a

Le2i Lab, University of Burgundy, Dijon, France Department of Mechanical Engineering, University of Maryland, College Park, MD, USA c Computer Science and Engineering Department, Qatar University, Doha, Qatar d Intelligent Systems Division, NIST, Gaithersburg, MD, USA b

art ic l e i nf o

a b s t r a c t

Article history: Received 1 April 2014 Received in revised form 24 June 2014 Accepted 25 June 2014 Available online 18 August 2014

In this article, we present a novel approach to intention recognition, based on the recognition and representation of state information in a cooperative human–robot environment. States are represented by a combination of spatial relations along with cardinal direction information. The output of the Intention Recognition Algorithms will allow a robot to help a human perform a perceived operation or, minimally, not cause an unsafe situation to occur. We compare the results of the Intention Recognition Algorithms to those of an experiment involving human subjects attempting to recognize the same intentions in a manufacturing kitting domain. In almost every case, results show that the Intention Recognition Algorithms performed as well, if not better, than a human performing the same activity. Published by Elsevier Ltd.

Keywords: Intention recognition Manufacturing kitting Ontology State recognition Human–robot collaboration Robotics

1. Introduction Safe and seamless human–robot collaboration is a future goal of the robotics research community. When humans and robots can safely occupy the same area, many tasks become amenable to automation, including collaborative assembly and parts/material handling. Many areas of research are necessary to allow this, including (1) the ability for a robot to perceive, monitor, and understand its environment, (2) the ability for a robot to infer the intent of humans in the workspace, and (3) the ability for a robot recognize potential dangers in enough time to avoid them. Current research is helping to revolutionize robots to help in areas such as manufacturing by allowing robots and humans to work in close proximity to each other while performing collaborative tasks [1]. There are existing and proposed standards to address the issue of human–robot collaborative safety, but they primarily address limiting the robot’s approach distance and contact force when confronted with a human [2,3]. The standard specifies that the robot should attempt to minimize the likelihood and potential severity of collisions between the human and the robot. These approaches focus on reactive processes based only on current input from sensors, and do not consider what tasks the human is performing or where s/he is expected to be at points in the future. n

Corresponding author.

http://dx.doi.org/10.1016/j.rcim.2014.06.007 0736-5845/Published by Elsevier Ltd.

What is missing from all of these standards, and is a key difference maker in the field of human–robot cooperation and safety, is intention recognition. This traditionally involves the robot attempting to use sensor information to better interpret the human’s intention by being able to identify their actions [4] to aid in the prediction of their location in the future. Being able to accurately predict the human’s future location will give the robot the information it needs to take steps to either help the human accomplish their tasks or, for safety reasons, not put itself in a location where an unsafe condition can occur.

1.1. Intention recognition scenario With the addition of reliable intention recognition, one might imagine the following: A robot is placed in a manufacturing kitting environment with the goal of helping the humans in the factory complete their tasks. Factory orders change very frequently, and the humans often switch between activities to make sure that all orders are met. The robot is tasked with making sure that all necessary parts are available to the humans and that the kits and assemblies are produced correctly. This must all occur while ensuring safe interaction with the humans. In addition, the

30

C. Schlenoff et al. / Robotics and Computer-Integrated Manufacturing 33 (2015) 29–41

robot may be asked to finish the creation of a kit when the human is pulled to another task. There are many robot functionalities needed to allow this simple scenario to occur, including perception, manipulation, mobility, etc. In this paper, we will focus on the ability for the robot to recognize the intention of the human. By recognizing the human’s intention, the robot will be able to determine what overall goal the human is trying to accomplish (i.e., what kit is desired to be developed). Once the intention is determined, the robot can: (1) retrieve any additional parts necessary for the remainder of the kit-building process, but not immediately available to the human; (2) ensure that the remainder of the kit is

developed to specification; and/or (3) determine where the human left off and complete the development of the kit when the human is pulled to another task. The robot may also ensure that it interacts with the human safely by accurately predicting which actions will occur next (based on the identification of the intention and its known subsequent steps) and ensuring that it stays away from the path of those actions. 1.2. Paper contribution In [5], Heinze proposed six possible paths for intention recognition that start with the intention of the intending agent (in this case the human) and end with the recognition of the intention by the

Fig. 1. Six proposed paths for intention recognition.

C. Schlenoff et al. / Robotics and Computer-Integrated Manufacturing 33 (2015) 29–41

recognizing agent (in this case the robot). These possible paths are shown in Fig. 1 and are described below: 1. Top left: This case is most common in the literature. The intending agent executes his intentions, composed of actions, which causes states in the world to be true. The states are recognized by the recognizing agent which then infers the activities and finally infers the intention. 2. Middle left: This case requires that the intending agent provide direct access (and descriptions) of the actions that it is performing to the recognizing agent. The recognizing agent then uses these activity descriptions to infer the intending agent’s intention. This requires that the intending agent is part of the system design process as its internal state must be made available to the recognizing agent. 3. Bottom left: This case requires that the intention of the intending agent be made directly available to the recognizing agent, so no reasoning is needed at all. This can be done through direct communication between the intending agent and the recognizing agent. 4. Top right, middle right, and bottom right: These three cases all focus on the concept of “direct perception,” where perception not only involves the recognition of objects and actions, but also incorporates the inference that is associated with them. In the top right case, the recognized intention is provided directly from the perception of the state which innately infers the intention. Likewise in the middle right case, the perception system innately infers which actions are being performed based on the perceived state of the environment. Inference then occurs to determine the intention from the recognized actions. In the bottom right case, the recognized intention is provided directly from the perception of the activities which innately infers the intention.

In this paper, we propose a new structure for recognizing intentions, as shown in Fig. 2. In this approach, we focus on the relationship between states and intentions, while bypassing the activity level all together. Using states for intention recognition offers some interesting advantages over activity-based recognition, including:


States are often more easily recognizable by sensor systems than actions. As shown in [6], existing algorithms that attempt to recognize the state of the environment (the identification, location, and orientation of objects) perform approximately 20% better than algorithms that attempt to identify activities that are being performed in the environment. As such, state information can be provided with higher accuracy as input into

Fig. 2. Novel path for intention recognition.







31

the intention recognition algorithms, which in turn would allow for better algorithm output; Using activities, intention recognition is often limited to inferring the intention of a single person. [4] Using states for intention recognition eliminates this shortfall, in that the existence of the state is independent of who created it. In this approach, multiple people can work on the same intention. This is because the algorithms do not look at the activities that are being performed, they only focus on the state that is achieved once each activity is completed; State information tends to be more ubiquitous than activity information, thus allowing for reusability of the ontology. Spatial relations such as On-Top-Of, Partially-In, and InContact-With are generic across multiple domains, while “Place the Box in the Kit Tray” may only be specific to the manufacturing kitting domain.

An overview of the intention recognition approach can be found in [7,8] and is summarized in the beginning of this paper. The majority of this paper shows the performance of the Intention Recognition Algorithms compared to human performance. For the purpose of this work, we will use the following definitions:


State relation—a relationship between two objects consisting of







specific recognizable configuration(s) (e.g., Object 1 is on top of Object 2); State—a set of state relations corresponding to one or more objects in an area of interest at a point in time; Intention—the state that is intended to be achieved. In an experiment described later in this paper, the intention represents which kit is being created (e.g., Kit 1 Created). One advantage of defining intentions as states is that it allows us to define multiple levels of intentions (each composed of subintentions and bottoming out at states) without having to modify the approach presented in this paper; Intention recognition—the task of recognizing the intentions of agent(s) by analyzing the changes in the state resulting from their actions; Manufacturing kitting—a process in which several different, but related items are placed into a container and supplied together as a single unit (kit).

This intention recognition approach is novel as compared to other intention recognition approaches (as described in the next section), as they primarily use activity information as input into their intention recognition algorithms as opposed to using only state information. However, in both cases, a form of abduction is used to assign probabilities to the intentions based on the inputted information. We identify primitive state relations based on sensor observations and by applying Region Connection Calculus 8 (RCC-8) [9,10] formalisms and then combine these observations to infer the overall state of the environment. Once a sequence of state relations has been determined, we use a Bayesian probabilistic approach to correlate those states with possible predetermined intentions to predict the next probable. The state of objects in the real world can be determined by using different types of sensors, including cameras, laser range finders, and depth sensors. For the purpose of this work, we assume a Kinect-like sensor,1 which includes a 3-D depth sensor and a Red–Green–Blue (RGB) camera. In the experiments described in [6] showing the accuracy of state detection algorithms, a Kinect sensor was used. The placement of the Kinect sensor(s) to ensure maximum coverage of the work area is very 1

http://support.xbox.com/en-US/xbox-360/kinect/kinect-sensor-components.

32

C. Schlenoff et al. / Robotics and Computer-Integrated Manufacturing 33 (2015) 29–41

Fig. 3. Example Kit.

important, and this placement will be specific to the individual application. In the next section, we report on a literature review of intention recognition approaches and state representation approaches using ontologies. Then we explain our approach to state representation using RCC-8 [9,10]. Based on these state relations, we show how they are combined to form intentions and how likelihoods are assigned to the intentions. We continue by describing an experiment that was performed to compare the output of the algorithms to that of a set of humans trying to infer the same intentions of agents in the environment. We use the manufacturing kitting domain for the experiment. Kitting is the process in which various different items are put into a box (a kit as shown in Fig. 3) and provided to an assembly area to be used in final assembly. We conclude the paper by summarizing the contribution and describe some future work to extend the metrics used to assign likelihoods to perceived intentions.

2. Intention recognition and state representation related work Intention recognition has typically involved recognizing the overall goal of an agent by analyzing the actions they perform. Abduction has often been used as the underlying methodology to provide guesses as to what intention is being performed. In abduction, the system “guesses” that a certain intention could be true based on the existence of a series of observed actions. As more information is learned and activities are performed, probabilities of certain intentions can be refined to be consistent with the observations [4]. Intention recognition, and specifically the approach described in this paper, builds upon research in the area of causal theory used for planning. In this approach, a background theory is provided, which contains intentions that are expected to be performed in the environment along with the relationships between the states and the intentions. There are also observations of the states that exist over time, along with the hypotheses (and probabilities) of the agent’s intentions based on these states. Such an approach is closely related conceptually to causal theories used for planning [11], with the exception of states taking the place of activities. In the literature, there are a wide array or approaches applied to the field of intention recognition, including ontology-based approaches [12], multiple probabilistic frameworks such as Hidden Markov Models [13] and Dynamic Bayesian Networks [14], utilitybased approaches [15], and graph-based approaches [16]. Intention recognition is a challenging field. Very often, there are many equally-plausible hypotheses regarding an observed agent’s intentions. One of the main challenges is the ability to choose between these hypotheses. Another challenge is that an

agent may not want his/her intentions to be observed and may take actions to hide their intentions. In addition, the scene may be occluded by other objects so the actions cannot be seen in their entirety. There is a significant amount of literature in the area of intention recognition for human–robot collaboration, though all of these approaches use activity recognition as the primary input for the intention recognition system. Schrempf, Albrecht and Hanebeck [17] use a form of Bayesian networks with a reduced state space to allow for tractable on-line evaluation. They use a combination of telepresence techniques and virtual environments in a kitchen-like environment. In this environment, they were able to reduce the state space from 216 to 17. The approach described in this paper is similar to Schrempf’s approach in that it also applies a Bayesian technique but uses state information (as opposed to activity information) to infer intentions. Wang et al. [18] propose the Intention-Driven Dynamic Model (IDDM), which is a latent variable model for inferring unknown human intentions. They apply this model to two human–robot scenarios: robot table tennis and action recognition for interactive robots. They show that modeling the intention-driven dynamics can achieve better prediction than algorithms without modeling dynamics. The drawback of their approach is that it suffers from its computational complexity, which is not present in the approach presented in the paper. Goto et al. [19] explore human–robot collaboration in assembly tasks. Focusing on the assembly of a table, they use finite state machines and a set of visual recognition routines for state transition detection to recognize the human’s intention. They also incorporate verbal messages as cues during various stages of the process. The limitations that appear in this system, namely the fact that task models are hand-made and the scene recognition routines are slow, are not present in the approach described in this paper due to the selective nature of the states that are being considered. Tahboub [20] use Dynamic Bayesian Networks with a focus on eliminating cycles by time delay. In essence, they feed back sensed states from previous time slices instead of using the current states. This paper stresses the need to handle qualitative spatial data, though they do not go into detail on how to address this. This need will be focused on in this paper. In all of these cases, activity recognition is the primary input into the intention recognition system, with state information being of secondary importance. Approaches in the literature have primarily started with recognizing activities being performed in the environment as the primary input for intention recognition systems. However, activity recognition is a very difficult challenge and the accuracy of such systems is very low [4]. Some research efforts have attempted to use Radio Frequency Identification (RFID) readers and tags to address this problem by attaching these tags to relevant objects as an aid to follow their movement with the ultimate goal of correlating their motion to overall activities of interest [21]. Unfortunately, the RFID tags (and other supporting hardware) tend to be unnatural and inhibiting. By basing intention recognition on the identification and representation of states as opposed to actions, many of the challenges of activity recognition can be addressed. The field of state representation has been well researched, though it has not been extensively used for ontology-based intention recognition. There are a large set of detailed ontologies that offer representations of space and spatial semantics. It is beneficial to focus on the areas of qualitative spatial representation, which will be the focus of the state representation approach present in this paper. In qualitative representations, a situation is characterized by variables that can only take a small, predetermined number of values [22]. Qualitative descriptions can include concepts such as in_front_of, behind, on_top_of, next_to, etc. This

C. Schlenoff et al. / Robotics and Computer-Integrated Manufacturing 33 (2015) 29–41

33

provides descriptions at a higher level of abstraction, replacing specific measures with qualitatively described characteristics. It is believed that these types of qualitative descriptions are more representative of human perception, as this is how humans typically refer to the spatial relations between objects. Bateman and Farrar [23] describe the approaches to representing spatial information as found within foundational ontologies. Ontologies include the Descriptive Ontology for Linguistics and Cognitive Engineering (DOLCE) [24], Cyc [25], the Standard Upper Merged Ontology (SUMO)[26], and Basic Formal Ontology (BFO) [27]. Region Connection Calculus 8 (RCC-8) [10] is a welldocumented approach for capturing the possible relationship between two regions in Euclidean space. It describes all eight possible relations in two dimensions, including equal, tangential proper part, and overlapping. There have been other efforts in the literature that have attempted to extend this into three dimensions [28]. FlipFlop calculus [29] describes the position of one point (the “referent”) in a plane with respect to two other points (the “origin” and the “relatum”). Single Cross Calculus (SCC) [30] is a ternary calculus that describes the direction of a point (C, the referent) with respect to a second point (B, the relatum) as seen from a third point (A, the origin) in a plane. The focus on this paper will be based on an intention recognition approach using ontology-based state representations within the context of a manufacturing kitting scenario.

3. State representation approach This section summarizes the approach presented in [8] for representing states to be applied to the Intention Recognition Algorithms. This context is necessary to explain the results described later in the paper. When the term “state representation algorithms” is mentioned in this paper, it refers to the software code that implements the approach described in this section. As mentioned earlier, RCC-8 characterizes regions in Euclidean space via their spatial relationships with each other. RCC-8 captures eight basic possible relations between any two convex regions: Disconnected (DC), Externally Connected (EC), Tangential Proper Part (TPP), Non-Tangential Proper Part (NTPP), Partially Overlapping (PO), Equal (EQ), Tangential Proper Part Inverse (TPPi), and Non-Tangential Proper Part Inverse (NTPPi). These relationships are represented in Fig. 4. RCC-8 was developed to model the spatial relations between two regions in a two-dimensional space. In the manufacturing kitting domain, as with other real-world domains, we are working in three-dimensional space. To account for this, we prepend an xy-, yz- or xz-before each RCC-8 relation to denote the associated axial plane. The RCC-8 spatial relations in each axial plane can be determined by projecting the point cloud representing the threedimensional solid object onto each of the axial planes. The first step in this process is to determine the contour of each shape projected onto each of the three axial planes. We use as input into this process the set of 2D vertices that are projected on each plane

Fig. 4. RCC-8 relations.

Fig. 5. Projecting objects onto planes and determining overlap, (a) build projection contour, (b) determine contour overlap.

as well as the list of nearest neighbor vertices. Fig. 5a shows a schematic drawing explaining the first part of algorithm. Open circles represent the starting point (k0) and the last point (k2) of a closed boundary loop organized in the clockwise direction. Black dots correspond to vertices on a boundary while gray dots represent vertices from inside a 2D region limited by a loop. The vertex k0 is directly connected with four vertices (nearest neighbors): k1,…,k4. Vertex k1 is set as ktry, which is an initial candidate for the next point on the loop. Then, remaining nearest neighbors are consecutively checked. Once this process is complete, the complete contour of the object on the specified projection is determined. The second step of the process is to determine if two projections are touching or overlapping each other, as shown in Fig. 5b. Two contours (A and B) intersect each other only when there is at least one line segment in A and at least one line segment in B which intersect. Necessary but not sufficient condition for intersection is that bounding boxes of both loops overlap and each loop has at least one vertex in overlapping region. In Fig. 5b, the open circle indicates the vertex belonging to loop B which protrudes into 2D area limited by loop A. Using this information, we can determine the overlap between the projections of two objects, allowing us to determine the RCC relations. The algorithms presented in this section have an execution time proportional to the number of points in the object projection’s contour multiplied by the total number of vertices in the object’s projection. This is a lower bound for the best 2D convex hull algorithms.2

2

Chan’s algorithm, http://en.wikipedia.org/wiki/Convex_hull_algorithms

34

C. Schlenoff et al. / Robotics and Computer-Integrated Manufacturing 33 (2015) 29–41

The combination of all 24 RCC relations (8 per axis) starts to describe the spatial relations between any two objects in the scene, but is not yet a sufficient step to capture the spatial relations between any two objects in the area of interest. For example, to state that a robot vacuum end effector is holding a part, one could represent this by saying that there is a part under_and_in_contact_with the end effector and there is nothing under and in contact with the part. For this approach, we use the reference frame attached to a fixed point of reference (in this case the end effector) with the positive z-direction pointing away from the gravitational center. Saying that:

ððy ECðobject1; object2Þ 3 y  NTPPðobject1; object2Þ 3 y TPPðobject1; object2Þ 3 y  POðobject1; object2Þ 3 y  NTPPiðobject1; object2Þ 3 y  TPPiðobject1; object2ÞÞ

ð6Þ

and then domain specific state relations such as: Vacuum  effector  holding  partðvacuum ef f ector; partÞSolidObjectðobject1Þ 4 Underðpart; vacuum ef f ectorÞ 4 In  Contact  Withðpart; vacuum ef f ectorÞ 4 :Underðobject1; partÞ 4 :In  Contact  Withðobject1; partÞ

ð1Þ

ð7Þ

which intuitively means that the part is externally connected to the end_effector in the z-dimension is not sufficient because the part could be either above or beneath the end_effector. What is missing is the representation of directionality. This is accomplished by the following Boolean operators:

which intuitively means that there is a part (part) that is under_and_in_contact_with a vacuum_effector and there is no other object (object2) that is under and in contact with the part. This would imply the vacuum_effector is “holding” the part since no other object is supporting it. The intermediate state relations are meant to be domainindependent in nature and therefore applicable to a wide range of application areas. The domain specific state relations are tailored for the individual domain of interest. More information about the state representation approach can be found in [31].

z ECðend_effector; partÞ

greater  zðA; BÞ

ð2Þ

smaller  zðA; BÞ

ð3Þ

which intuitively means, in the latter equation, that the centroid of object A is larger than (in the z-dimension) the centroid of object B, measured in the same reference frame. For this work, we are dealing with relatively compact objects (i.e., objects that have few voids when represented by convex hulls), thus the centroid is a good indication of the location of the object in geometric space. This discussion is meant to represent a beginning in modeling the relationships needed to describe the spatial relations between objects in a manufacturing environment. Other relations that could be modeled in the future could include objects’ absolute locations and relative distances with respect to each other (e.g., closer or farther). For the experiment described in this paper, those additional relationships were not needed but could be included as this work progresses. Using these basic relations, we can create higher-level state relations: In  Contact  Withðobject1; object2Þx  ECðobject1; object2Þ 3 y  ECðobject1; object2Þ 3 z  ECðobject1; object2Þ

ð4Þ

Contained  Inðobject1; object2Þðx TPPðobject1; object2Þ 3 x  NTPPðobject1; object2ÞÞ 4 ðy  TPPðobject1; object2Þ 3 y  NTPPðobject1; object2ÞÞ 4 ðz  TPPðobject1; object2Þ 3 z  NTPPðobject1; object2ÞÞ Underðobject1; object2Þ- smaller zðobject1; object2Þ 4 ðx  ECðobject1; object2Þ 3 x  NTPPðobject1; object2Þ 3 x TPPðobject1; object2Þ 3 x  POðobject1; object2Þ 3 x  NTPPiðobject1; object2Þ 3 x  TPPiðobject1; object2ÞÞ 4

ð5Þ

4. How states are combined to form intentions In the previous section, we showed how state relations are represented. In this section, we show how they can be combined to form an intention. When the term “intention recognition algorithms” is mentioned in this paper, it refers to the software code that implements the approach described in this section and the next section. To do this, we leverage OWL-S (Web Ontology Language— Services) [32]. Table 1 shows the OWL-S control constructs in the first column, the adapted state representation ordering constructs in the second column, and a brief definition of those adapted ordering constructs in the third column. We use the following simple kitting scenario as an example of how to apply the ordering constructs. A person is constructing a set of kits. In this case, the type of kit that is being constructed constitutes the intention that is trying to be inferred. One of the kits contains two instances of Part A, two instances of Part B, one instance of Part C, and one instance of Part D. This intention can be described as follows: SR1 ¼ On Top  OfðKitTray; TableÞ

ð8Þ

OC1 ¼ ExistsðSR1Þ

ð9Þ

SR2 ¼ Contained  InðPartA; KitTrayÞ

ð10Þ

SR3 ¼ Contained  InðPartB; KitTrayÞ

ð11Þ

Table 1 Initial state representation ordering constructs. OWL-S control construct

Adapted state representation ordering construct

State representation ordering construct definition

Perform Sequence Any-Order Iterate

Exists Ordered list Any-order Count

n/a Choice Join

Not exists Choice Co-exist

A state relationship must exist A set of state relationships that must occur in a specific order A set of state relationships that must all occur in any order A state relationship that must be present exactly the number of times specified (greater than one) State relationship that cannot exist A set of possible state relationships that can occur after a given state relationship Two or more state relationships that must be true

C. Schlenoff et al. / Robotics and Computer-Integrated Manufacturing 33 (2015) 29–41

35

Table 2 Initial state representation ordering constructs.

State Relation

1

2

2

3

3

4

5

6

Kit 1

On_Top_Of (Kit Tray, Table)

ContainedIn (PartA, KitTray)

ContainedIn (PartA, KitTray)

ContainedIn (PartB, KitTray)

Contained -In (PartB, KitTray)

ContainedIn (PartC, KitTray)

Contained -In (PartD, KitTray)

Contained-In (KitTray, CompletedKitBox)

Previous State Relations

n/a

1

1

1

1

1

1

2, 3, 4, 5

SR4 ¼ Contained  InðPartC; KitTrayÞ

ð12Þ

SR5 ¼ Contained  InðPartD; KitTrayÞ

ð13Þ

OC2 ¼ CountðSR2; 2Þ

ð14Þ

OC3 ¼ CountðSR3; 2Þ

ð15Þ

OC4 ¼ CountðSR4; 1Þ

ð16Þ

OC5 ¼ CountðSR5; 1Þ

ð17Þ

OC6 ¼ AnyOrderðOC2; OC3; OC4; OC5Þ

ð18Þ

SR6 ¼ Contained  InðKitTray; CompletedKitBoxÞ

ð19Þ

OC7 ¼ ExistsðSR6Þ

ð20Þ

OC8 ¼ OrderedListðOC1; OC6; OC7Þ

ð21Þ

We can represent this information as a table, as shown in Table 2, State relations that must occur multiple times (e.g., Part A in the Kit Tray) have a corresponding number of columns in the table. Precondition state relations are shown in the lowest row of the table to indicate which state relations must occur before a state relation is available for matching. We match state relations as they occur in the environment to expected state relations that are associated with an intention. However, state relations can only match when precondition state relations are true. For example, in this intention, the state relations of any part being in the kit tray can only match if the kit tray is first placed on the table. If the kit is not placed on the table first, the subsequent state relations would not evaluate to true even if they occur in the environment. Table 2 shows an example of five state relations being evaluated as true (as indicated by the green coloring). More information about the intention representation approach can be found in [6]. In the next section, we describe how likelihoods can be associated with various intentions based on observations.

The formula describing the chance that Kit j is being made after a given set of observed parts is: Prðkit j jobserved SRsÞ ¼

Prðobserved SRsjkit j ÞPrðkit j Þ ∑5j ¼ 1 Prðobserved SRsjkit j ÞPrðkit j Þ

ð22Þ

here, Prðobserved SRsjkit j ÞðPrðkit j Þ describes the chances of seeing a particular combination of state relations given that the state relations are being used to create Kit j. If each time a new part is chosen it is done randomly from the remaining parts, then this is described by a multivariate hypergeometric distribution. For this work, there is the assumption that each part is randomly selected from the set of parts remaining and that each kit is equally likely to be built, but this can be easily changed. Prðkit j Þ denotes the proportion of all kits made that are Kit j. In the denominator, the upper limit of the summation is set to five because there are five possible kits used in the experiment. For a specific intention (kit), even though a state relation may be true in the environment, it may not increase the probability of the given intention because a predecessor state relations many not have occurred. If a state relation (SR2) has a different state relation (SR1) that must first occur in an intention, and that previous state relation (SR1) has not yet occurred, the subsequent state relation (SR2) will not increase the probability of that intention. As a simple example, imagine an intention that requires a person to first place a ring on a dowel and then place a cap on the dowel. If the cap is placed on the dowel and the ring has not, the state relation will not increase the probability of that intention because the predecessor state relation (ring on dowel) did not occur.

6. Experiment and results An experiment was run to assess the performance of the Intention Recognition Algorithms described above. This was done by comparing the output of our algorithms to the performance of several humans watching agents performing the same intentions. This section described the experimental procedures and the results.

5. Assigning likelihoods to intentions

6.1. Experimental procedures

For the experiment described in this section, a Bayesian approach was used to assign likelihoods to intentions. Bayesian probability is a type of evidential probability and is sometimes seen as an extension of propositional logic. To determine the probability that a hypothesis (in this case, an intention) is true, the approach requires that some prior probability is determined. With each new set of relevant data that is provided to the system, the probabilities are updated.

Because the author was not aware of any clear metrics to assess the performance of the Intention Recognition Algorithms in a general sense, an experiment was developed to compare the results of the Intention Recognition Algorithms to the performance of humans. Five kits containing ten blocks were used (as shown in Fig. 6), but each contained a unique combination of differently shaped blocks. In the figure, blocks of different sizes are represented by

36

C. Schlenoff et al. / Robotics and Computer-Integrated Manufacturing 33 (2015) 29–41

Fig. 6. Experiment user interface.

different colors. The combination of blocks was carefully chosen to provide an appropriate level of challenge to the humans and the algorithms. We used a distance metric (specifically the L1 distance) to characterize the difference between kits. In this case, the L1 distance represents the sum of the absolute differences between the number of each colored part between two kits). For example, Kit 4 has four red blocks, two blue blocks, three green blocks, zero yellow blocks, and one orange block. Kit 5 has two red blocks, three blue blocks, three green blocks, one yellow block, and one orange block. The distance metric between these two kits would be: j4  2j þ j2  3jþ j3  3j þ j0 1j þ j1  1j ¼ 4

ð23Þ

The smallest the distance measure could be is zero if both kits were identical and the largest is 20 if there were no common blocks between the two. For this work, we chose values between two and six to represent challenging cases where any two kits were very similar but there were enough differences for an intention recognition system to be able to distinguish between the two. For the kits used in the experiment that contained five types of parts and ten total instances of parts, there are between 2500 and 50,000 possible orders in which the parts can be placed in the kit tray. For this experiment, we chose five of those orders for each kit, resulting in 25 total runs. These five orders were randomly generated through a computer program and assigned in a different order to each participant. At each state, the State Recognition Algorithms described in Section 3 were run to identify the state relations in the environment. Based on these state relations, the algorithm in Section 5 was run to assign likelihoods to intentions. The resulting likelihoods were captured for each possible intention (kit) at each state. To determine how well these likelihoods compared to what a human would perceive in the same situation, we developed a data set based on these same five kits. To compile the data set, 15 students served as the human participants. They were presented with the interface shown in Fig. 6. Included in the interface were five images of kits as shown in the figure. The kit images could be enlarged by clicking on them. Each kit represents an intention. They were not told how many of each kit were being built in each

of the 25 runs. All kits were presented in random order and the order varied from participant to participant. One by one, the humans were presented with a textual description of something that happened in the environment, for example, “Object red added in the kit tray.” Based on this information, the humans assigned probabilities as to which kit was being developed (i.e., what was the intention based on the observed events). The probabilities that the humans assigned were based on personal preferences. In the example shown in Fig. 6, the humans assigned a 20% probability to all kits based on the first state because a red object exists in all kit trays. In some cases, the humans scaled the probability based upon how many red objects existed in each kit tray. The only rule was that all probabilities for each state must add up to 100%. An update of how many percentage points were available to assign to each row is shown at the bottom of the figure. Once a human finished with a given row, the person pressed the “Validate Row” button at the bottom. If the sum of the percentages added to 100%, a new environment observation is provided, a new row is added to the table, and the process starts again. If the percentages did not equal 100%, the human was alerted to this and asked to change their percentages until they did equal 100%. Once a human clicked the “Validate Row” button and it successfully validated, they could not go back and edit any of the previous rows. Since there are exactly ten objects in each kit, exactly ten states are presented per kit. After ten states were completed, the human moved on to the next intention until all 25 intentions were complete. As input to the algorithms, we only provided the location, orientation, and classification of objects in the environment at a frequency of two Hertz. The algorithms had to (1) infer the relevant spatial relations using the RCC8 relations, (2) match the spatial relations to the predefined intentions in the ontology, and (3) associate likelihoods with each intention based on the matched state relations. To compare to the output of the Intention Recognition Algorithms, we averaged the probabilities for all 15 humans for each plan, each state, and each kit. Based upon this, we identified two performance metrics for evaluation: (1) the average state in which the humans first correctly identified the kit that was being created (and consistently identified that correct kit for the remainder of the states). This was

C. Schlenoff et al. / Robotics and Computer-Integrated Manufacturing 33 (2015) 29–41

measured by averaging probabilities that all humans assigned to each kit within a state, and then seeing if the kit with the highest probability was the correct kit being created; (2) the average state in which the humans were over 20 percentage points more confident that the correct kit was being developed compared to the second most probable kit. The 20 percentage point value could be changed as necessary, but appeared to be a reasonable value to show that the humans were confident that their choice was correct. The next section will compare the results of human experiments with the output of the Intention Recognition Algorithms. 6.2. Experimental results Based on the experiments described in Section 6.1, we directly compared the results of the human experiments with the output of the Intention Recognition algorithms described in Section 5. The Intention Recognition Algorithms was scored based on how closely it matched the human-generated results. We judged closeness by determining the difference between the algorithms’ performance and the average human performance using the two performance metrics defined in Section 6.1 (both the state in which the intention was first identified and the state in which the intention was confidently identified as indicated by the 20% difference between the top selected intention and second rated intention). Table 3 shows the comparison of the output of the Intention Recognition Algorithms to that of humans’ performance. Column 1 (Kit) shows the kit that was being developed and Column 2 (Plan) shows the plan that was used to develop the kit. The third through fifth columns of the table (“Correct Intention First Chosen”) show the average state in which the humans first identified the correct kit that was being developed (“Humans”), the state in which the algorithms first identified the correct kit that was being developed (“Algorithms”), and the difference between the two (“Difference”). Similarly the sixth through eighth columns of the table (“Over 20% Confident of Intention”) show the average state in which the humans first identified the correct kit that was being developed with over a 20% confidence as compared to the second most probable intention (“Humans”), the Table 3 Comparison of algorithm output to human intention recognition with ties allowed. Kit Plan Correct intention first chosen

state in which the algorithms first identified the correct kit that was being developed with over a 20% confidence as compared to the second most probable intention (“Algorithms”), and the difference between the two (“Difference”). A positive value in the difference column means that the algorithms identified the correct kit that many states earlier than the humans. Conversely, a negative value in the difference column means the algorithms identified the kit correlating to that many states later than the humans. A zero means the humans and the algorithms identified the kit at the same state. In analyzing Table 3, we see some very promising results in the performance of the Intention Recognition Algorithms as compared to human performance. If we first examine Columns 3–5 that compare the state which the algorithms and the humans first chose the correct intention, we see that in over half of the runs (13/25), the Intention Recognition Algorithms determined the proper kit at the exact same state as the humans (as indicated by the zero in the difference column). In eight of the runs, the Intention Recognition Algorithms determined the proper kit earlier than the humans (as indicated by the positive number in the difference column). This ranged from one to four states earlier than the humans. In only four runs, the Intention Recognition Algorithms determined the proper kits later than the humans. In three cases, this was one state later and in the other it was three states later. When looking at the data for when the algorithms and humans were greater than 20% confident (as compared to the next most probable intention), the results are equally promising. In almost half of the runs (12/25), the Intention Recognition Algorithms determined the proper kit (with over 20% confidence compared to the next most probable kit) at the exact same state as the humans (as indicated by the zero in the difference column). In all of the remaining runs, the Intention Recognition Algorithms determined the proper kit earlier than the humans (as indicated by the positive number in the difference column). This ranged from one to five states earlier than the humans. We can also take one of the Kit/Plan pairs in Table 3 and look closely at the details. We chose the second pair (Kit 2, Plan 4), but any of the 25 pairs could be used and would show very similar results. The results are shown in Fig. 7, with the following explanations of the sub-figures:


Top left Fig. 7(a)—this shows results of the human experiment.

Over 20% confident of intention

Human Algorithm Difference Human Algorithm Difference 1 1 1 1 1 2 2 2 2 2 3 3 3 3 3 4 4 4 4 4 5 5 5 5 5

1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5

10 7 10 3 5 4 5 9 9 1 6 4 3 1 6 8 5 7 7 5 4 3 6 8 2

10 7 10 2 3 4 1 6 8 1 4 1 4 1 3 8 5 8 7 8 4 3 6 9 2

0 0 0 1 2 0 4 1 0 2 3 1 0 3 0 0 1 0 3 0 0 0 1 0

10 7 10 6 5 7 7 3 9 7 7 5 7 6 8 8 6 9 7 8 4 3 6 9 7

10 7 10 5 5 5 3 8 9 3 6 2 5 3 6 8 5 8 7 8 4 3 6 9 2

0 0 0 1 0 2 4 1 0 4 1 3 2 3 2 0 1 1 0 0 0 0 0 0 5

37







The x-axis represents the sequence of the 10 states and the yaxis represents the likelihood that the specified kit is being developed. In this figure are five lines, each representing one kit. Each line shows the perceived likelihood by the humans that the specific kit is being developed at each state. In this case, the blue line indicates the kit that was being developed; Top right Fig. 7(b)—this shows the results of the Intention Recognition Algorithms. The x-axis represents the sequence of the ten states and the y-axis represents the likelihood that the specified kit is being developed. In this figure are five lines, each representing one kit. Each line shows the perceived likelihood by the algorithms that the specific kit is being developed at each state. In this case, the blue line indicates the kit that was being developed; Bottom Fig. 7(c)—this shows the comparison of the result of the humans and the algorithms for the individual kit that was being created (i.e., the correct kit). In all kit/plan pairs, both the humans and the algorithms ultimately predicted the correct kit that was being created.

By looking at Fig. 7c, we can make some immediate subjective, qualitative observations:


Form of the lines—Both the line representing the humans’ likelihoods (the red line) and that representing the algorithms’

38

C. Schlenoff et al. / Robotics and Computer-Integrated Manufacturing 33 (2015) 29–41

Fig. 7. Human/algorithm comparison data for Kit 2 Plan 4, (a) human’s likelihood for all 5 Kits. (b) algorithms’ likelihood for all 5 Kits, (c) comparison of humans and algorithms’ data for Kit 2 (correct Kit).





likelihoods (blue line) at each state have very similar forms. However, the algorithms’ line has a more pronounced “bump” at States 6 and 8. This indicates that, at these states, the algorithms were able to detect an aspect of the sequence of states that implied that Kit 2 was the correct intention which was not evident to the humans; Likelihoods at every state—The algorithms’ likelihood at almost every state (apart from State 2) is greater than that of the humans, thus indicating better results by the algorithms; Dramatic increase in likelihood—For both the humans and the algorithms, the probability of the correct kit increase dramatically at State 9, where the percentages went up to 87% and 100%, respectively. This shows that the humans and the algorithms were very confident that the specific kit was being made at the same state.

This data above shows that the Intention Recognition Algorithms, in almost every case, performed as good, if not better, than a humans performing the same activity.

7. Exploring additional metrics The Bayesian metric is certainly not the only metric available, but it seems to be the logical choice to determine which intention is most likely. We have also begun to explore numerous other metrics and ways to combine them. It is our belief that the best way to match or exceed human performance in intention recognition will be to rely on an optimal combination or multiple metrics, as opposed to just one. These metrics include:


Metric 1: Number of observed state relations that are true in an










intention at a given state (compared to other intentions). Every intention is composed of a set of state relations. The number of state relations that are true in one intention, as compared to others, should give an indication of the relative likelihood of that intention; Metric 2: Percentage of an intention that is complete at a given state. As opposed to the number of true state relations, the percentage of state relations that are true in an intention could also provide a relative likelihood of that intention occurring; Metric 3: Number of productive states since the first productive state relation in an intention. The Bayesian approach makes the underlying assumption that the intention starts at the beginning of the observations. This may not be the case. There could be situations when one intention is completing during the first few steps in an observation and a new one starts after the first is completed. This metric begins to address that situation by focusing only on the state relations that occur after the first state relation is true in an intention; Metric 4: Number of productive states that have occurred recently (within the past X states, where X is determined by the user). This metric looks at what intention is most probable based on what has happened recently. The supposition is that the more positive outcomes that have happened in the past X states, the more likely that a given intention is occurring;

Initial work has identified a generalized equation to combine these metrics, as shown in Eq. 24. ! ∑nk ¼ 1 ðM i;k  W Mi;k Þ Li ¼  100 ð24Þ ∑nk ¼ 1 W Mi;k

C. Schlenoff et al. / Robotics and Computer-Integrated Manufacturing 33 (2015) 29–41

where Li is the numeric likelihood of an intention (i); M i;k is the numeric result of applying the kth additive metric for intention I; W Mi;k is the weight of the kth additive metric for intention I; n is the total number of additive metrics. Eq. (24) shows that every metric is assigned a weight (WM), which indicates its relative importance in the equation. The higher the weight, the more effect the metric will have on the overall Table 4 Impact of weights on first four metrics—coarse level. Weights assigned to metrics format: (Metric 1 Weight-Metric 2 WeightMetric 3 Weight-Metric 4 WeightBayesian Metric Weight)

Sum of differences (measured in likelihood points)

Standard deviation

0-0-0-0-100 10-10-10-10-100 20-20-20-20-100 30-30-30-30-100 40-40-40-40-100 50-50-50-50-100 60-60-60-60-100 70-70-70-70-100 80-80-80-80-100 90-90-90-90-100 100-100-100-100-100

83.03 42.97 56.88 66.41 79.09 85.58 94.33 103.86 111.28 114.99 118.12

6.16 3.71 7.83 11.36 13.58 15.06 16.23 17.32 18.04 18.37 19.01

Table 5 Impact of weights on first four metrics—fine grained-level. Weights assigned to metrics format: (Metric 1 Weight-Metric 2 WeightMetric 3 Weight-Metric 4 WeightBayesian Metric Weight)

Sum of differences (measured in likelihood points)

Standard deviation

1-1-1-1-100 2-2-2-2-100 3-3-3-3-100 4-4-4-4-100 5-5-5-5-100 6-6-6-6-100 7-7-7-7-100 8-8-8-8-100 9-9-9-9-100

77.14 65.01 55.84 48.03 37.42 20.48 38.13 41.21 41.90

5.40 5.32 4.74 4.85 4.57 2.50 3.94 3.75 3.88

39

equation. Weights can be any number greater than or equal to zero. The user can decide what weights are appropriate for the application. These metrics can be directly applied to Eq. (24) and easily evaluated by the automated system. Applying these additional four metrics to the experiment described earlier showed some interesting results. Each metric on its own did not provide results that were superior to that of the humans. However, the introduction of these metrics to Eq. (24) with relatively small weights combined with the Bayesian metric containing a relatively high weight provided results that better mimicked the detailed performance of the humans. In all cases, the quantitative metrics (average state in which the kit was first detected and state which the likelihood was over 20% greater) described in Section 6.1 did not change. However, by varying the weights assigned with the additional four metrics, we observed that the similarity of the line generated by Eq. (24)as compared to the line generated by the humans increased. In this case, we assigned equal weights to all four additional metrics for simplicity purposes, but these metrics could have easily been assigned differing weights. We analyzed the difference between the line generated by Eq. (24) and the line generated by the humans. We measured the difference by looking at the sum of the absolute distance between corresponding points on each of the two lines at each of the ten states. This is shown in Table 4. Based on Table 4, there appears to be a significant reduction in the difference between the two lines when applying weights from zero to ten to the first four metrics, with a gradual and continuous increase when weights greater than ten are applied. Exploring the weights between zero and ten provided the data in Table 5. Tables 4 and 5 combined are shown graphically in Fig. 8. Fig. 9 shows that there is a clear pattern in the weight assigned to these four additional metrics as compared to the “closeness” of the resulting Eq. (24) line with that of the humans’. At a weight of six, the two lines most closely align and then consistently deviate as the weight moves further away in both directions. The comparison of the humans’ performance, the Bayesian metric by itself, and the addition of the four metrics at a weight of six are shown in Fig. 9. As can be seen in the figure, the humans’ performance (represented by the blue line) and the addition of the four metrics (represented by the green line), while not exactly the same, align quite well.

Fig. 8. Individual metric weights effect on sum of differences.

40

C. Schlenoff et al. / Robotics and Computer-Integrated Manufacturing 33 (2015) 29–41

Fig. 9. Comparison of human, Bayesian and additional metrics for Kit 2, Plan 4.

The results are interesting for the following reasons. From the original experiment, it was clear that humans’ performance followed that of a traditional Bayesian probability algorithm in the sense that the forms of the lines were very similar, though the actual location of the lines on the graph did not align (the algorithm line was generally higher on the graph than the humans’ line). The addition of the four new metrics started to explain this difference. When these additional metrics were introduced at a relatively small weight as compared to the Bayesian metric, the resulting algorithm line greatly approached that of the humans. This may indicate that the humans were consciously or unconsciously considering the other four metrics when anticipating which kit was being developed, namely, the number of true observed states, the percentage of the intention complete, the number of productive states since the first productive state, and the number of productive states that have occurred recently. Based on the data, a relative weight of 6% for these additional four metrics (as compared to the Bayesian metric) appear to best represent the thought process of the humans. This was meant to be an initial exploration of these additional four metrics. More experiments would be necessary to determine if the 6% relative weight holds for other situations and for different sets of humans. Additional exploration would also be necessary to determine how independently varying the weights of these four additional metrics would change the results and which of the metrics, if any, contribute to the best correspondence with the humans’ performance. The goal of this analysis was to determine if there appears to be a benefit in introducing these additional four metrics, which appears to be the case.

8. Conclusion and future work In this paper, we described a novel approach for intention recognition based on using state information as the primary form of input. States are determined using a combination of spatial relations along with cardinal direction information. These state relations were then ordered to develop intentions and a set of metrics were proposed to assign likelihoods to the various intentions. The proposed metrics were compared to the performance of a set of humans performing the same intention recognition.

One of the limitations of this work is that they require relatively compact objects (i.e., objects that have few voids when represented by convex hulls). This will be address in future work. In addition, we will explore the additional four metrics to determine if the results are consistent when applied to different sets of users and tasks. We will also apply the overall approach described in this paper to domains outside of kitting, including manufacturing assembly and part fabrication.

References [1] Szabo S, Norcross R, Shackleford W. Safety of human–robot collaboration systems project. Available: 〈http://www.nist.gov/el/isd/ps/safhumrobcollsys. cfm〉; 2011. [2] Shneier M. Safety of human–robot collaboration in anufacturing. In: Presented at the eighth safety across high-consequence industries conference. Saint Louis, MO; 2013. [3] Chabrol J. Industrial robot standardization at ISO. Robotics 1987;3:229–33. [4] Sadri F. Logic-based approaches to intention recognition. In: Handbook of research on ambient intelligence and smart environments: trends and perspectives. In: Chong N-Y, Mastrogiovanni F, editors. . p. 346–75. [5] Heinze C. Modelling intention recognition for intelligent agent systems, DTIC Document; 2004. [6] Schlenoff C, Pietromartire A, Kootbally Z, Balakirsky S, Kramer T, Foufou S. Inferring intention through state representation in cooperative human–robot environments. In: Engineering creative design in robotics and mechatronics. Habib M and Davim P. editors; 2013. [7] Schlenoff C, Foufou S, Balakirsky S. An approach to ontology-based intention recognition using state representations. In: Presented at the submitted to the fourth international conference on knowledge engineering and ontology development. Barcelona, Spain; 2012. [8] Schlenoff C, Pietromartire A, Kootbally Z, Balakirsky S, Foufou S. Ontologybased state representation for intention recognition in cooperative human– robot environments. Rob Autom Syst 2013;61:1224–34. [9] Randell D, Cui Z,A C. A spatial logic based on regions and connection. In: Presented at the third international conference on representation and reasoning. San Mateo, CA; 1992. [10] Wolter F, Zakharyaschev M. Spatio-temporal representation and reasoning based on RCC-8. In: Presented at the seventh conference on principles of knowledge representation and reasoning (KR2000). Breckenridge, CO; 2000. [11] Charniak E, McDermott D. Introduction to artificial intelligence. Reading, MA: Addison Wesley; 1985. [12] Jeon H, Kim T, Choi J. Ontology-based user intention recognition for proactive planning of intelligent robot behavior. In: Presented at the international conference on multimedia and ubiquitous engineering. Busan, Korea; 2008. [13] Kelley R, Tavakkoli A, King C, Nicolescu M, Nicolescu M, Bebis G. Understanding human intentions Via Hidden Markov models in autonomous mobile robots. In: Presented at the third ACM/IEEE international conference on human robot interaction. Amsterdam; 2008. [14] Schrempf O, Hanebeck U. A generic model for estimating user-intentions in human–robot cooperation. In: Presented at the second international

C. Schlenoff et al. / Robotics and Computer-Integrated Manufacturing 33 (2015) 29–41

[15]

[16] [17]

[18]

[19]

[20]

[21] [22] [23]

conference on informatics in control. Automation, and Robotics ICINCO 05: Barcelona; 2005. Mao W, Gratch J. “A utility-based approach to intention recognition.In: resented at the AAMAS workshop on agent tracking: modeling other agents from observations. New York, NY; 2004. Youn S-J, Oh K-W. Intention recognition using a graph representation. World Acad Sci Eng Technol 2007;25. Schrempf OC, Albrecht D, Hanebeck ED. Tractable probabilistic models for intention recognition based on expert knowledge. In: Presented at the international conference on intelligent robots and systems; 2007. Wang Z, Deisenroth M, Amor HB, Vogt D, Scholkopf N, Peters J. Probabilistic modeling of human movements for intention inference. In: Robotics: science and systems Sydney, Australia, 2012. Goto H, Miura J, Sugiyama J. Human–robot collaborative assembly by on-line human action recognition based on an FSM Task model. In: Presented at the human–robot interaction 2013 workshop on collaborative manipulation. Tokyo; 2013. Tahboub KA. Intelligent human–machine interaction based on dynamic Bayesian networks providing probablistic intention recognition. J Intell Rob Syst 2006;45:31–52. Philipose M, Fishkin K, Perkowitz M, Patterson D, Hahnel D, Fox D, et al. Inferring ADLs from interactions with objects. IEEE Pervasive Comput 2005. Kleer Jd, Brown JS. A qualitative physics based on confluence. Formal theories of the commonsense world. Norwood, NJ: Ablex Publishing Corporation; 1985. Bateman J, Farrar S. Spatial ontology baseline. University of Bremen Germany: Collaborative Research Center for Spatial Cognition; 2004.

41

[24] Oberle D, Ankolekar A, Hitzler P, Cimiano P, Sintek M, Kiesel M, et al. DOLCE ergo SUMO: on foundational and domain models in the SmartWeb Integrated Ontology (SWIntO). J Web Semant 2007;vol. 5:156–74. [25] Lenat D, Guha R, Pittman K, Pratt D, Shephard M. CYC: toward programs with common sense. Commun ACM 1990;33:30–49. [26] Pease A, Niles I. IEEE standard upper ontology: a progress report. Knowl Eng Rev Spec Issue Ontol Agents 2002;vol. 17:65–70. [27] Smith B, Grenon P. The Cornucopia of formal ontological relations. Dialectica 2004;58:279–96. [28] Albath J, Leopold J, Sabharwal C, Maglia A. RCC-3D: qualitative spatial reasoning in 3D. In: Presented at the 23rd international conference on computer applications in industry and engineering (CAINE). Las Vegas, NV; 2010. [29] Ligozat G. Qualitative triangulation for spatial reasoning. In: CPSIT 1993. In: Campari I, Frank AU, editors. Heidelberg, 716. Springer; 1993. p. 54–68. [30] Freksa C. “Using orientation information for qualitative spatial reasoningIn: Theories and methods of spatio-temporal reasoning in geographic space. In: Frank U, Campari I, Formentini U, editors. Heidelberg. Springer; 1992. p. 162–78. [31] Schlenoff C, Pietromartire A, Foufou S, Balakirsky S. “Ontology-based state representation for robot intention recognition in ubiquitous environments. In: Presented at the UBICOMP 2012 workshop on smart gadgets meet ubiquitous and social robots on the web (UbiRobs)”. Pittsburgh, PA; 2012. [32] Martin D, Burstein M, Hobbs J, Lassila O, McDermott D, McIlrath S., et al. . OWL-S: Semantic markup of web services. Available: 〈http://www.w3.org/ Submission/OWL-S/〉; 2004.