Classification of human-robot team interaction paradigms*

December 7-9, 2016. Florianopolis, Brazil& Human-Systems 1st IFAC Conference on Cyber-Physical December 7-9, 2016. Florianopolis, Brazil 1st IFAC Conference on Cyber-Physical & Human-Systems Human-Systems 1st IFAC Conference on Cyber-Physical December 7-9, 2016. Florianopolis, Brazil& Available online at www.sciencedirect.com December December 7-9, 7-9, 2016. 2016. Florianopolis, Florianopolis, Brazil Brazil

ScienceDirect IFAC-PapersOnLine 49-32 042–047 Classification of human-robot Classification of(2016) human-robot  Classification of human-robot interaction paradigms Classification of interaction paradigms  Classification of human-robot human-robot interaction paradigms  interaction interaction∗ paradigms paradigms∗

team team team team team

Selma c Selma Musi´ Musi´ c ∗∗ Sandra Sandra Hirche Hirche ∗ Selma Musi´ c ∗ Sandra Hirche ∗∗ Selma c Hirche ∗ Selma Musi´ c ∗ Sandra Sandra Hirche ∗hirche}@tum.de). a tt Musi´ M¨ u nchen, ({selma.music, ∗ Technische Universit¨ Technische Universit¨ a M¨ u nchen, ({selma.music, hirche}@tum.de). ∗ at M¨ unchen, ({selma.music, hirche}@tum.de). ∗ Technische Universit¨ ∗ Technische Universit¨ a t M¨ u nchen, ({selma.music, Technische Universit¨ at M¨ unchen, ({selma.music, hirche}@tum.de). hirche}@tum.de). Abstract: Abstract: Human-robot Human-robot team team interaction interaction is is challenging challenging in in terms terms of of system system complexity complexity and and control synthesis. Classifying different interaction paradigms between a human and aa robot Abstract: Human-robot teamdifferent interaction is challenging in terms of system complexity and control synthesis. Classifying interaction paradigms between a human and robot Abstract: Human-robot team interaction is challenging in terms of complexity and team eases the formal analysis. The challenge to classify paradigms appropriately, w.r.t. Abstract: Human-robot teamdifferent interaction is is challenging inthe terms of system system complexity and control synthesis. Classifying interaction paradigms between a human and a robot team eases the formal analysis. The challenge is to classify the paradigms appropriately, w.r.t. control synthesis. Classifying different interaction paradigms between a human and a robot the setting and the task to be performed. In this paper three interaction paradigms are formally control synthesis. Classifying different interaction paradigms between a human and a robot team eases the formal analysis. The challenge is to classify the paradigms appropriately, w.r.t. the and the taskanalysis. to be performed. In this threethe interaction paradigms are formally teamsetting eases the formal The challenge challenge is paper to classify classify paradigms appropriately, w.r.t. defined and analyzed using is shown that aa straightforward classification of team eases the formal analysis. The to paradigms appropriately, w.r.t. the setting the task to becontrollability. performed. In It this three interaction paradigms are formally defined andand analyzed using controllability. It is paper shown thatthe straightforward classification of the setting and the task to be performed. In this paper three interaction paradigms are formally interaction paradigms, based on the mapping properties of the input space to the tangent of the the setting and the task to be performed. In this paper three interaction paradigms are formally defined andparadigms, analyzed using controllability. Itproperties is shown that a input straightforward classification of interaction based on the mapping of the space to the tangent of the defined and analyzed using controllability. It is shown a straightforward classification of state space is possible. Specific in interaction for cooperative defined andparadigms, analyzed using controllability. Itthe is human-robot shown that that straightforward of interaction based on examples the mapping properties of thea team input space to theclassification tangent of the state space is possible. Specific examples in the human-robot team interaction for cooperative interaction paradigms, based on the mapping properties of the input space to the tangent of the manipulation tasks validate the proposed classification methods. interaction paradigms, based on the mapping properties of the input space to the tangent of the state space is possible. Specific examples in the human-robot team interaction for cooperative manipulation tasks validate the examples proposed in classification methods. state space possible. Specific the team state space is is tasks possible. Specific the human-robot human-robot team interaction interaction for for cooperative cooperative manipulation validate the examples proposed in classification methods. manipulation tasks validate the proposed classification methods. manipulation tasks validate the proposed classification methods. © 2016,1.IFAC (International Federation of Automatic Control) Hosting by Elsevier Ltd. All rights reserved. 1. INTRODUCTION INTRODUCTION 1. INTRODUCTION 1. 1. INTRODUCTION INTRODUCTION Human-robot Human-robot interaction interaction is is aa wide wide area area of of research research that that benefits from a number of scientific fields: psychology, Human-robot interaction is a wide area of research that benefits from interaction a number is of ascientific fields: psychology, Human-robot wide area of research that control As humans are able to perHuman-robot is wide area of research that benefits theory, from interaction ahaptics, numberetc.. of ascientific fields: psychology, control theory, haptics, etc.. As humans are able to perbenefits from aa number of scientific fields: psychology, form tasks which require cognitive capabilities such as benefits from number of scientific fields: psychology, control theory, haptics, etc.. As humans are able to perform tasks which require cognitive capabilities such as control theory, haptics, etc.. As humans are able to perplanning and adapting to uncertainties, robots are able to control theory, haptics, etc.. As humans are able to perform tasks which require cognitive capabilities such as planning and adapting to uncertainties, robots are able to form tasks which require cognitive capabilities such as conduct tasks which require high precision. Therefore, it form tasks which require cognitive capabilities such as planning and adapting to uncertainties, robots are able to conduct tasks which require high precision. Therefore, it planning and adapting to uncertainties, robots are able to is reasonable to exploit their complementary abilities in aa planning and adapting to uncertainties, robots are able to conduct taskstowhich require high precision. Therefore, it is reasonable exploit their complementary abilities in it conduct tasks which require high precision. Therefore, way that human operator(s) conduct high-level sub-tasks conduct tasks which require high precision. Therefore, it is reasonable to exploit their complementary abilities in a way that human operator(s) conduct high-level sub-tasks is reasonable to exploit their complementary abilities in aa and robot(s) conduct low-level sub-tasks, working together is reasonable to exploit their complementary abilities in way that human operator(s) conduct high-level sub-tasks and robot(s) conduct low-levelconduct sub-tasks, workingsub-tasks together way that human operator(s) high-level in way to achieve aa final goal. way that human operator(s) high-level andthis robot(s) conduct low-level sub-tasks, workingsub-tasks together Fig. 1. Bilateral teleoperation with wearable haptic devices in this way toconduct achieve final conduct goal. and robot(s) low-level sub-tasks, working together In order perform multiple sub-tasks (constituting and robot(s) sub-tasks, working togetheraa Fig. 1. Bilateral teleoperation with wearable haptic devices in this wayto toconduct achievelow-level a final goal. In order to perform multiple sub-tasks (constituting Fig. 1. Bilateral teleoperation with wearable haptic devices in this way to achieve a final goal. complex task) simultaneously, interaction between in way achieve a final goal. 1. teleoperation with haptic devices In this order totoperform multiple sub-tasks (constituting Fig. 1. Bilateral Bilateral teleoperation with wearable wearable haptic devices complex task) simultaneously, interaction(constituting between mulmul-aa Fig. in Baker and Yanco (2004). However, the formal analysis of In order to perform multiple sub-tasks in Baker and Yanco (2004). However, the formal analysis of tiple humans and/or robots is necessary. As a In order to perform multiple sub-tasks (constituting complex task) simultaneously, interaction between multiple humans and/or multiple robots is necessary. As a the levels of autonomy is not provided. Novel forms of incomplex task) simultaneously, interaction between mulBaker Yanco (2004). However, theNovel formalforms analysis of the levelsand of autonomy is not provided. of inresult the interactive system (highly) The complex task) simultaneously, interaction between As multiple humans and/or multipleis robots is redundant. necessary. a in in Baker and Yanco (2004). However, the formal analysis of result the interactive system is (highly) redundant. The teraction between and robots are possible thanks Baker and Yancohumans (2004). However, the formal analysis of tiple humans and/or multiple robots is necessary. As aa in the levels of autonomy is not provided. Novel forms of interaction between humans and robots are possible thanks specific set of sub-tasks can be dynamically assigned eitiple humans and/or multiple robots is necessary. As result the interactive system is (highly) redundant. The the levels of autonomy is not provided. Novel forms of inspecific set of sub-tasks can be dynamically assigned eito the availability of wearable haptic devices (e.g. Chinello the levels of autonomy is not provided. Novel forms of inresult the interactive system is (highly) redundant. The teraction between humans and robots are possible thanks the availability of wearable haptic devices (e.g. Chinello ther to human(s), robot(s) Depending on the result interactive system is both. (highly) redundant. specific set of sub-tasks can or be dynamically assigned ei- to teraction between humans and robots are to possible thanks ther tothe human(s), robot(s) or both. Depending on The the et al. (2015)). For example, it is possible establish the teraction between humans and robots are possible thanks specific set of sub-tasks can be dynamically assigned eito the availability of wearable haptic devices (e.g. Chinello et al. (2015)). For example, it is possible to establish the distribution of sub-tasks among human(s) and robot(s) specific set of sub-tasks can be dynamically assigned either to human(s), robot(s) or both. Depending on the to the availability of wearable haptic devices (e.g. Chinello distribution of sub-tasks among human(s) and robot(s) direct teleoperation without using the master robot. The to the availability of wearable haptic devices (e.g. Chinello ther to human(s), robot(s) or both. Depending on the et al. (2015)). For example, it is possible to establish the direct teleoperation without using the master robot. The in specific stages of a general task, it is reasonable to ther to human(s), robot(s) or both. Depending on the distribution of sub-tasks amongtask, human(s) and robot(s) al. For example, it is establish the in specific stages of a general it is reasonable to et human moves freely, which allows for aa to transition from et al. (2015)). (2015)). For example, it is possible possible to establish the distribution of sub-tasks among human(s) and robot(s) direct teleoperation without using the master robot. The human moves freely, which allows for transition from define a number of interaction paradigms. In order to make distribution of sub-tasks among human(s) and robot(s) in specific stages of a general task, it is reasonable to direct teleoperation without using the master robot. The define a number of interaction paradigms. In order to make uncoupled (teleoperative) to coupled (physical) humandirect teleoperation without using the master robot. The in specific stages of a general task, it is reasonable to human moves freely, which allows for a transition from uncoupled (teleoperative) toallows coupled (physical) humanaadefine between paradigms, the suitable in distinction specific stages of ainteraction general task, it is to human a number of interaction paradigms. In reasonable order to make moves freely, which aa transition from distinction between interaction paradigms, the to suitable robot interaction. An example of aa for teleoperation experihuman moves freely, which for transition from define aa number of interaction paradigms. In order make uncoupled (teleoperative) toallows coupled (physical) humanrobot interaction. An example of teleoperation experitools for classification are the system properties such as define number of interaction paradigms. In order to make a distinction between interaction paradigms, the suitable uncoupled (teleoperative) to coupled (physical) humantools for classification are the system properties such as uncoupled mental scenario with the human, equipped with wearable (teleoperative) to coupled (physical) humana distinction between interaction paradigms, the suitable interaction. Anthe example of equipped a teleoperation experimental scenario with human, with wearable controllability and a distinction between interaction paradigms, the suitable tools for classification are the system properties such as robot robot interaction. An example of aa teleoperation expericontrollability and observability. observability. thimble devices (Chinello et al., 2015) is depicted in fig. robot interaction. Anthe example of teleoperation experitools for classification are the system properties such as mental scenario with human, equipped with wearable thimble devices (Chinello et al., 2015) is depicted in fig. 1. 1. Different types of human-robot interaction are summertools for classification are the system properties such as controllability and observability. mental scenario with the human, equipped with wearable Different types of human-robot interaction are summerAdditionally, brain-computer interface (BCI) also enables mental scenario with the human, equipped with wearable controllability and observability. thimble devices (Chinello et al., 2015) is depicted in fig. 1. Additionally, brain-computer interface (BCI) also enables ized in Yanco Drury (2004). However, the literature controllability and observability. Different types of human-robot interaction are summerthimble devices (Chinello et al., 2015) is depicted in fig. 1. ized in Yanco and Drury (2004).interaction However, the literature definition of novel ways in humans and robots thimble devices (Chinello etwhich al., 2015) is(BCI) depicted inenables fig.in1. Different types of human-robot are summerAdditionally, brain-computer interface also definition of novel ways in which humans and robots inon human-robot team interaction mainly analyzes physDifferent types of human-robot interaction are summerized in Yanco and Drury (2004). However, the literature Additionally, brain-computer interface (BCI) also enables on human-robot team interaction mainly analyzes phys- Additionally, teract (Tonin et al., 2010). brain-computer interface (BCI) also enables ized in Yanco and Drury (2004). However, the literature definition of novel in which humans and robots in(Tonin et al.,ways 2010). ical human-robot interaction scenarios et al. ized in Yanco andteam Drury (2004). However, the literature on human-robot interaction mainly(Lawitzky analyzes physdefinition of ways in which humans and robots ical human-robot interaction scenarios (Lawitzky et al. teract In this paper define and formally classify definition of novel novel ways in humans and interaction robots ininon human-robot team interaction mainly analyzes physteract (Tonin etwe al., 2010). In this paper we define andwhich formally classify interaction (2010)). Non-physical interaction, or more specifically, on human-robot team interaction mainly analyzes physical human-robot interaction scenarios (Lawitzky et al. teract (Tonin et al., 2010). (2010)). Non-physical interaction, or more specifically, paradigms between a single human and multiple robots teract (Tonin et al., 2010). ical human-robot interaction scenarios (Lawitzky et al. In this paper we define and formally classify interaction between a single human and multiple robots teleoperation of teams is in aa classic ical human-robot interaction (Lawitzky et al. paradigms (2010)). Non-physical interaction, or more specifically, In paper we and formally classify interaction teleoperation of robot robot teams scenarios is considered considered in classic that form aabetween team and use controllability property In this this paper we define define and the formally classify interaction (2010)). Non-physical interaction, or more specifically, paradigms a single human and multiple robots that form team and use the controllability property setting of coupling the human to the master robot (Lee (2010)). Non-physical interaction, or more specifically, teleoperation of robot teams is considered in a classic paradigms between a single human and multiple robots setting of coupling the teams human isto considered the masterinrobot (Lee paradigms to distinguish between the paradigms. defined inbetween a single human and The multiple robots teleoperation of robot aa classic that form a team and use the controllability property to distinguish between the paradigms. The defined inand Spong, 2005), (Lin et al., 2015). Different modes of teleoperation of robot teams is considered in classic setting of coupling the human to the Different master robot (Lee that form a team and use the controllability property and Spong, 2005), (Lin et al., 2015). modes of teraction paradigms differ depending on the distribution that form a team and use the controllability property setting of coupling the human to the master robot (Lee to distinguish between the paradigms. The defined interaction paradigms differ depending on the distribution interaction together with the concept of adaptable semisetting of coupling the human to the master robot (Lee and Spong, 2005), (Lin et al., 2015). Different modes of to distinguish between the paradigms. The defined ininteraction together withetthe concept of adaptable semiof sub-tasks and on the level of autonomy. Therefore, to distinguish between the paradigms. The defined inand Spong, 2005), (Lin al., 2015). Different modes of teraction paradigms differ depending on the distribution sub-tasks and ondiffer the level of autonomy. Therefore, autonomy introduced in (2001). Laschi and Spong,are 2005), (Lin al.,Laschi 2015).et Different modes of of interaction together withetthe concept ofal. adaptable semiteraction paradigms depending on the distribution autonomy are introduced in Laschi et al. (2001). Laschi the provided analysis enables the theoretical consideration teraction paradigms differ depending on the distribution interaction together with the concept of adaptable semiof sub-tasks and on the level of autonomy. Therefore, the provided analysis enables the theoretical consideration et al. claim that the involvement of human(s) in decision interaction together with the concept of adaptable semiautonomy are introduced in Laschi et al. (2001). Laschi of sub-tasks and the level of autonomy. Therefore, et al. claimare that the involvement of human(s) in decision extreme modes; teleoperation sub-tasks and on on enables theinteraction level autonomy. Therefore, autonomy introduced in et al. Laschi the providedhuman-robot analysis theof theoretical of extreme human-robot interaction modes; consideration teleoperation process and autonomous behavior robots for autonomy are introduced in Laschi Laschi et al. (2001). (2001). Laschi of et al. claim that the involvement ofof human(s) inrepetitive decision the provided analysis enables the theoretical consideration process and autonomous behavior of robots for repetitive and physical interaction. The classification of interaction the provided analysis enables the theoretical consideration et al. claim that the involvement of human(s) in decision of extreme human-robot interaction modes; teleoperation physical interaction. interaction The classification interaction tasks aa desirable levels of autonomy et al. is claim that thecombination. involvement ofofhuman(s) decision and process and autonomous behaviorThe robots for repetitive of extreme extreme human-robot modes; of teleoperation tasks is desirable combination. The levels for of in autonomy paradigms is motivated by the cooperative manipulation of human-robot interaction modes; teleoperation process and autonomous behavior of robots repetitive and physical interaction. The classification of interaction paradigms is motivated by the cooperative manipulation ranging from teleoperation to full autonomy are proposed process and autonomous behavior of robots for repetitive tasks is a desirable combination. The levels of autonomy and physical interaction. The classification of interaction ranging from teleoperation to full The autonomy are proposed and task in which aa team of multiple robots manipulates physical interaction. The classification of interactiona tasks is aa desirable combination. levels of autonomy paradigms is motivated by the cooperative manipulation task in which team of multiple robots manipulates a tasks is desirable combination. The levels of autonomy ranging from teleoperation to full autonomy are proposed paradigms is motivated by the cooperative manipulation single object over extended workspace (Erhart and Hirche, paradigms is motivated by the cooperative manipulation  ranging from teleoperation to full autonomy are proposed task in which a team of multiple robots manipulates a The research leading to these results has received funding from the single object over extended workspace (Erhart and Hirche,  ranging from teleoperation to full autonomy are proposed task in which aa team of multiple robots manipulates a The research leading to these results has received funding from the 2016). The contribution of this paper is the proposition task in which team of multiple robots manipulates a single object over extended workspace (Erhart and Hirche, European Union Seventh Framework Programme FP7/20072013  2016). The contribution of workspace this paper(Erhart is the and proposition The research leading to these results has received funding from2013 the European Union Seventh Framework Programme FP7/2007single object over extended Hirche,  The research leading to these results has received funding from the and the analysis of three interaction paradigms: direct, single object over extended workspace (Erhart and Hirche,  under grant agreement no. 601165 ofhas thereceived project: WEARHAP 2016). The contribution of this paper is the proposition The research leading to these results funding from the European Union Seventh Framework Programme FP7/20072013 and theThe analysis of threeof interaction paradigms: direct, under grant agreement no. 601165 of the project: WEARHAP 2016). contribution this is European Union Seventh Framework Programme FP7/2007- 2013 complementary and In the probWEARable HAPtics for humans and of robots. 2016). contribution this paper paper is the the 22proposition proposition and theThe analysis of overlapping. threeof interaction paradigms: European Union Seventh Framework Programme 2013under grant agreement no. 601165 the project:FP7/2007WEARHAP complementary and overlapping. In Section Section the direct, probWEARable HAPtics for humans and robots. and the analysis of three interaction paradigms: direct, under grant agreement no. 601165 of the project: WEARHAP and the analysisand of overlapping. three interaction paradigms: under grantHAPtics agreement no. 601165 the project: WEARHAP complementary In Section 2 the direct, probWEARable for humans and of robots. complementary and overlapping. In Section 2 the probWEARable for WEARable HAPtics HAPtics for humans humans and and robots. robots. Copyright@ 2016 IFAC 42 complementary and overlapping. In Section 2 the probCopyright@ 2016 IFAC 42 2405-8963 © 2016, IFAC (International Federation of Automatic Control) Copyright@ 2016 IFAC 42 Hosting by Elsevier Ltd. All rights reserved. Copyright@ 2016 IFAC 42 Peer review under of International Federation of Automatic Copyright@ 2016 responsibility IFAC 42 Control. 10.1016/j.ifacol.2016.12.187

2016 IFAC CPHS December 7-9, 2016. Florianopolis, Brazil

Selma Musić et al. / IFAC-PapersOnLine 49-32 (2016) 042–047

lem is formulated and necessary theoretical concepts are introduced. In Section 3 three interaction paradigms are defined and their properties are derived. The verification of the properties is conducted on the analytical examples of cooperative manipulation systems.

where the states are x = [pT1 pT2 p˙ T1 p˙ T2 ]T , M (x) = ˙ = [cT1 (p1 , p˙ 1 ) cT2 (p2 , p˙ 2 )]T , bldiag(M1 (p1 ) M2 (p2 )), c(x, x) gT gT hg (x) = [h1 (p1 ) h2 (p2 )]T and u = [hT1 hT2 ]T . 3. CLASSIFICATION OF INTERACTION PARADIGMS

2. PROBLEM FORMULATION

3.1 Direct interaction paradigm

Let us consider a multi-input multi-output (MIMO) nonlinear, affine control system of the form: x˙ = f (x) + G(x)u (1) where x ∈ M is the state vector defined on an ndimensional, smooth manifold M, u ∈ K is the input vector defined on an m-dimensional, smooth manifold K and G = [g1 , ..., gm ]. The real-valued mappings f , g1 , ..., gm are smooth vector fields defined on the manifold M. Let us define the input vector u as a stacked vector of the input commands provided by two control inputs; the human and the autonomous controller: u = [uh1 , ..., uhk , uak+1 , ..., uam ]T (2)       uh

Definition 1. The interaction paradigm is called direct if the complete vector space of the system (1), T M is accessible by the control inputs uh . According to the definition 1, the input vector (2) has the following structure: u = [uh1 , ..., uhk , 0m−k ]T . Due to the structure of the input vector u it is sufficient to focus only on the properties of the vector fields denoted as Gh . The input vector uh is mapped by the mapping matrix Gh (x) onto the tangent space Tx M.The distribution spanned by the vector fields Gh is: ∆h = span{g1h , ..., gkh }, ∀x ∈ M (7)

ua

The distribution (7) is non-singular, i.e. the following equality is fulfilled: dim{∆h } = const. = m (8) As a consequence of non-singularity, complete input vector uh is mapped onto the tangent space T M for all x. The distribution (7) is involutive. This means that the Lie bracket of any gih and gjh belongs to the distribution ∆h :

where superscripts h and a indicate the human and the autonomous control inputs, respectively. The input manifold K can be divided into two submanifolds: K = Kh ∪ Ka . Let us write G as: G = [Gh Ga ] where Gh = [g1 , ..., gk ] and Ga = [gk+1 , ..., gm ]. The general feedback control input is assumed to be state dependent and is represented as: uh,a = αh,a (x) + β h,a (x)v h,a (3) h,a h,a where α (x) and β (x) are defined on U0 around point x and β h,a (x) is nonsingular for all x while v h,a is the new reference input. The feedback control law (3) modifies the system dynamics given by (1) into the form: ˜ (4) x˙ = f˜(x) + G(x)v i ˜ ˜ where: f (x) = f (x) + G(x)α(x) and G(x) = G(x)β(x). We propose three interaction paradigms between a human operator and a robot team: direct, complementary and overlapping (further categorized into cooperative and competitive). An overview of the interaction paradigms is given in fig. 2. A general architecture of the analyzed Level of autonomy

gih ∈ ∆h , gjh ∈ ∆h ⇒ [gih , gjh ] ∈ ∆h ∂gjh h ∂x gi

Complementary

Cooperative

Gh ]. where ad0f Gh = Gh and adkf Gh = [f , adk−1 f

Proposition 1. If the distribution ∆h is non-singular and involutive and if: dim{R} = n (11) where n is the number of the states, then the system 1 is controllable from the human input. For the proof of the proposition (1) we refer to the (Isidori (1995)). Example 1. A classical example of the direct interaction paradigm in robotics is bilateral teleoperation. Let us consider a robotic system of 2 manipulators, given by (6). Furthermore, let us consider this robotic team is teleoperated by the human operator. The motion of the two human fingers is the desired motion of the manipulators in the task space, as depicted in the fig. 4. Let us consider the translational motions. The control inputs are position and translational velocity of the human fingers, ph1 ph2 ∈ R3 and p˙ h1 , p˙ h2 ∈ R3 , respectively. Jointly, the inputs are represented as a stacked vector: v h = [ph1 , ph2 , p˙ h1 , p˙ h2 ]T ∈ R12      

Competitive

Fig. 2. Overview of the interaction paradigms. system is depicted in fig. 3. The selection mechanism block is in charge of selecting the appropriate interaction paradigm. The system exemplarily analyzed in the paper is a cooperative manipulation system, with the dynamics of a single manipulator in the task space: Mi (pi )p¨i + ci (pi , p˙ i ) + hgi (pi ) = hi i = 1, 2 (5) with pi ∈ SE(3) being the pose of the i-th end-effector in the task space, Mi (pi ) ∈ R6×6 its inertial matrix, ci (pi , p˙ i ) ∈ R6 its Coriolis terms, hgi (pi ) ∈ R6 its gravitational forces and hi ∈ R6 its wrench input. Transformation of (5) into the form (1) for two manipualtors, gives:       p˙ 0 p˙ = + u (6) p¨ ˙ − hg (p)) M (p)−1 (−c(p, p) M (p)−1        x˙

f (x)

(9)

∂gih h ∂x gj

− is the Lie bracket. where: [gih , gjh ] = Controllability of non-linear systems can be analyzed locally using the controllability distribution. R(x) = [Gh (x), adf Gh (x), ..., adn−1 Gh (x)] (10) f

Overlapping Direct

43

ph

g(x)

Using the impedance feedback control strategies for each subsystem, the human and autonomous input commands are mapped to the input wrenches of the system (6): uh = [K D] x + [−K −D] v h (12)       α(x)

43

p˙ h

β

2016 IFAC CPHS 44 December 7-9, 2016. Florianopolis, Brazil Selma Musić et al. / IFAC-PapersOnLine 49-32 (2016) 042–047

hh Wearable haptics

Operator

uh ua

yh u Selection y mechanism ya

Autonomous controller

Robot team cooperation

Fig. 3. General architecture for the human-robot team interaction in a cooperative manipulation task. are ordered so that the first d states are controllable by the human and the remaining (n−d) states are controllable by the autonomous controller. In order to ensure the control inputs, uh and ua are appropriately mapped onto the distributions T Mh and T Ma , respectively, it is necessary to ensure the off-diagonal terms of (13), G1a and G2h i i , h vanish. The distribution spanned by G is given by (18) and the distribution spanned by Ga is: a a ∆a = span{gk+1 }, ∀x ∈ M (14) , ..., gm

ph2 , p˙ h2 ph1 , p˙ h1 p2 , p˙ 2 p1 , p˙ 1

where D = bldiag(D1 D2 ) is a block diagonal damping matrix with D1 , D2 ∈ R3×3 being the damping matrices for each manipulator and K = bldiag(K1 K2 ) is a block diagonal stiffness matrix with K1 , K2 ∈ R3×3 being the stiffness matrices for each manipulator. It is possible to show that the dimension of the controllability distribution (10) is dim{R} = dim{x} = 12 if the matrix M (x) is positive-definite and for any choice of positivedefinite damping and stiffness matrices.

Proposition 2. Let T Mh and T Ma be nonsingular, involutive distributions of dimensions d and n − d, respectively. Furthermore, assume T Mh is invariant under the vector fields f , g1 , ..., gk and T Ma is invariant under the vector fields f , gk+1 , ..., gm . Moreover, suppose the distribution span{g1 , ..., gk } ⊂ T Mh and the distributions span{gk+1 , ..., gm } ⊂ T Ma . Than, for each x0 it is possible to find a neighborhood U0 of x0 and transformations z1 = φ1 (x) and z2 = φ2 (x) defined on U0 such that in the new coordinates, the system is: ξ˙h = f h (ξ h , ξ a ) + Gh (ξ h , ξ a )uhi (15) ξ˙a = f a (ξ h , ξ a ) + Ga (ξ h , ξ a )uai

3.2 Complementary interaction paradigm

where ξ h = {z1 , ..., zd } and ξ a = {zd+1 , ..., zn }.

Fig. 4. Cooperative manipulation example of the direct interaction paradigm

Proof. It is possible to construct a candidate transformation z1 = φ1 (x) around x0 such that the last n − d elements of its Jacobian span a distribution (T Mh )T (Isidori, 1995). Since vector fields f , g1 , ..., gk are in T Mh by assumption, by transforming them to the new coordinates: ∂φ1 f (x)]x=φ−1 (z) , f¯(z) = [ ∂x ¯ h (z) = [ ∂φ1 Gh (x)]x=φ−1 (z) G ∂x the last n − d elements of f and g h vanish, yielding: f¯(z) = col(f¯1 (z), ..., f¯d (z), 0n−d )

Definition 2. The interaction paradigm is called complementary if it is possible to define a d-dimensional distribution, T Mh ⊂ T M and an (n−d)-dimensional distribution T Ma ⊂ T M such that: T Mh ∩ T Ma = ∅ and T Mh ∪ T Ma = T M and if the distribution T Mh is accessible to the control inputs uh while the distribution T Ma is accessible to the control inputs ua . According to the definition (2), a subspace reachable to the human control input is unreachable to the autonomous control input and vice versa. This means the influences from the human control input and the autonomous control input are mutually complementary. The mappings of the complementary interaction paradigm are depicted in the fig. 5. Let us write the mapping matrix G as:  1h 1a   h a G G G= G G = (13) G2h G2a 1h

[g11 , ..., gk1 ]

1a

x˙ h

uh Kh

x˙ a

ua K

1 1 = [gk+1 , ..., gm ] 2h 2 2 G = [g1 , ..., gk ]

= and G are where G and sets of d-dimensional mappings and 2 2 G2a = [gk+1 , ..., gm ] are sets of (n − d)-dimensional mappings. Let us assume, without loss of generality, the states

a

Tx M h Tx M a

Fig. 5. Geometrical representation of the complementary interaction paradigm 44

2016 IFAC CPHS December 7-9, 2016. Florianopolis, Brazil Selma Musić et al. / IFAC-PapersOnLine 49-32 (2016) 042–047

¯ h (z) = col(¯ g11 (z), ..., g¯d1 (z), 0n−d ) G Analogously, it is possible to construct a candidate transformation z2 = φ2 (x) around x0 such that the first d elements of its Jacobian span a distribution (T Ma )T . Since vector fields f , gk+1 , ..., gm are in T Ma by assumption, than by transformation to the new coordinates the first d elements of f and Ga vanish, yielding: f¯(z) = col(0d , f¯d+1 (z), ..., f¯n (z)) ¯ Ga (z) = col(0d , g¯2 (z), ..., g¯2 (z)) d+1

behavior of the manipulators it is possible to achieve object manipulation and with the relative behavior it is possible to approach to the object and maintain the grasp of the object. Let us assume the human commands the cooperative motion, while the autonomous controller commands the relative motion. The human control inputs are the position and the translational velocity of the human hand: v h = [ph , p˙ h ]T and the autonomous control inputs are the desired relative position and the relative translational velocity between the manipulators: v a = [pa , p˙ a ]T . Let us introduce coordinate transformations z1 = φ1 (x) and z2 = φ2 (x):         1 p1 + p 2 p1 − p2 pc pr h a ξ = = = , ξ = (20) p˙ c p˙ r p˙ 1 − p˙ 2 2 p˙ 1 + p˙ 2      

n

This proves the proposition 2. Let us define two controllability distributions: ¯ h (ξ h ), adf¯h G ¯ h (ξ h ), ..., adn−1 ¯ h (ξ h )], (16) Rh (ξ h ) = [G G f¯h ¯ a (ξ a ), adf¯a G ¯ a (ξ a ), ..., adn−1 ¯ a (ξ a )] Ra (ξ a ) = [G G f¯a

(17)

φ1 (x)

T (x)

The transformation of the system, T (x), complies with the proposition (2) and ensures decoupling of the system (6). Furthermore, the following relations hold:   0 Mc (p) −T −1 (22) T (p)M (p)T (p) = 0 Mr (p) where Mc (p) ∈ Rd×d is the inertial matrix of the cooperative subsystem and Mr (p) ∈ R(n−d)×(n−d) is the inertial matrix of the subsystem for the relative behavior. Furthermore:   d ˙ c (p, p) 0 T −T M T −1 + T −T cT −1 (p) = c (23) ˙ 0 cr (p, p) dt

i

Local controlled invariance of the T Mh is guaranteed iff: [f¯h , T Mh ] ⊂ T Mh + Gh h

φ2 (x)

where pc ∈ R3 and p˙ c ∈ R3 are a mean of positions of the end-effectors (position of the mid-point between the manipulators) and the mean velocity of the team, respectively. Relative position and relative velocity of the robots are pr ∈ R3 and p˙ r ∈ R3 , respectively. Time derivative of (20) gives:  1 1   I I 0 0 3 3 3 3 ∂φ1  2  h 2   ξ˙  ∂x   03 03 1 I3 1 I3  x˙ ˙ (21) = x =   a   ˙ ∂φ 2 ξ  I −I 20 20  3 3 3 3  ∂x   03 03 I3 −I3

Proposition 3. If the following holds: dim{Rh } = d (18) where d is the number of the states ξ h of the system 1, the states ξ h are controllable w.r.t. the human input and the corresponding subsystem is controllable. Proposition 4. If the following holds: dim{Ra } = n − d (19) where n − d is the number of the states ξ a of the system 1, the states ξa are controllable w.r.t. the autonomous input and the corresponding subsystem is controllable. If the control inputs, uh and ua , are given as feedback control inputs of the form (3), the property of controlled invariance needs to be imposed. Proposition 5. A distribution T Mh is controlled invariant on U if there exists a feedback pair (αh , β h ) defined on U such that T Mh is invariant under the new vector fields (modified by a control law) f˜h , g˜11 , ..., g˜k1 : [f˜h , T Mh ] ⊂ T Mh , [˜ g h , T Mh ] ⊂ T Mh i = 1, ...k

[¯ gih , T Mh ] ⊂ T Mh + Gh

45

i = 1, ...k

h

Since by assumption G ⊂ T M (analogously Ga ⊂ T Ma ), the controlled invariance for the complementary interaction paradigm is always guaranteed.

where cc ∈ Rd is a vector of Coriolis terms for the cooperative subsystem. Analogously, cr ∈ R(n−d) is a vector of Coriolis terms for the subsystem for the relative behavior. The decoupled dynamics given by (6) is represented by the subsystem for cooperative and relative behavior:       p˙c 0 p˙c = + uh p¨c ˙ − hgc (p)) Mc (p)−1 Mc (p)−1 (−cc (p, p)         

ph , p˙ h pr , p˙ r

ξ˙h

f h (ξh ,ξa )

g(ξh ,ξa )

    (24) ˙ prel 0 p˙r = + ua p¨r ˙ − hgr (p)) Mr (p)−1 Mr (p)−1 (−cr (p, p)          

pc , p˙ c Fig. 6. Cooperative manipulation example of the complementary interaction paradigm

ξ˙a

f2 (ξh ,ξa )

g(ξh ,ξa )

(25)

Example 2. Let us consider a cooperative manipulation system (6) in a setting depicted in fig. 6. Let us assume the two manipulators perform cooperative and relative behaviors. The cooperative behavior is a team behavior of the manipulators and the relative behavior is a relative motion between the manipulators. With the cooperative

Using the impedance feedback control strategies for each subsystem, the human and autonomous input commands are properly mapped to the input wrenches: uh,a = [Kc,r Dc,r ] ξ h,a + [−Kc,r −Dc,r ] v h,a (26)       αc,r (ξh,a )

45

βc,r

2016 IFAC CPHS 46 December 7-9, 2016. Florianopolis, Brazil Selma Musić et al. / IFAC-PapersOnLine 49-32 (2016) 042–047

It is possible to show that the dimension of the controllability distribution (16) is dim{Rh } = dim{ξh } = d if the matrix Mc (x) is positive-definite and for any choice of positive-definite matrices, Dc and Kc . It is also possible to show that the dimension of the controllability distribution (17) is dim{Ra } = dim{ξ a } = n − d if the matrix Mr (x) is positive-definite and for any choice of positivedefinite Dr and Kr .

Fig. 8. Cooperative manipulation example of the overlapping interaction paradigm Object inertial matrix, centrifugal and Coriolis forces and gravitational forces are given by Mo (po ), c(po , p˙ o ), g(po ), respectively. Grasp matrix is G = [I3 I3 ] For more information on the physical human-robot interaction in a cooperative manipulation task see e.g. (Lawitzky et al., 2010). It is possible to define control inputs as linear homotopy between the forces applied on the object by the human, f h , and the robot, f a :  h    α1 1 − α1 f h u u= = (28) 1 − α 2 α2 ua fa where the α1,2 ∈ [0, 1] represent constant or time dependent mappings. The controller thus defined is termed as homotopy-based (Evrard and Kheddar, 2009). In this way a cooperative model of the system is obtained. Its transformation into the state space gives:       0 p˙ o p˙ o + Gu (29) = p¨o Mo−1 −Mo−1 (co + hgo ) where the dependencies on the object position and velocities are omitted for brevity. The inertial matrix is positive-definite for robotic systems. The cooperative behavior is described by the control law with all non-negative terms (28) that are not dependent on the system states. Therefore, the system is cooperative.

3.3 Overlapping interaction paradigm Definition 3. The interaction paradigm is called overlapping if there exists an intersection of distributions T Mh and T Ma : ∆ha = T Mh ∩T Ma = ∅ and if the distribution ∆ha is accessible to the control inputs uh and ua . The overlapping interaction paradigm considers a coupled system in which the human input commands and the autonomous input commands jointly steer states of the system, as depicted in fig. (7). Controllability of the system is unaffected if additional inputs are added. The interconnections between subsystems are expressed by the non-zero off-diagonal blocks of the g Jacobian. It is important to distinguish two overlapping cases: cooperative and competitive. The cooperative systems are characterized by the overlapping of submanifolds onto which human and autonomous inputs act in the same direction. For a competitive system the influence from the human and the autonomous controller is opposite.

Tx M h

uh Kh ua Ka

3.4 Control design guidelines

x˙ Tx M a

If for a specific model the conditions are not satisfied they can, nevertheless, be met by introducing an appropriate feedback control strategy. For example, the controllability condition imposed by the proposition (1) can be met by state-space exact linearization which gives linearized and controllable system if the outputs of the system are chosen appropriately and the assumption of non-singularity is met. It is possible that the system (1) cannot be fully decoupled into subsystems (15). In this case the decoupling of the linearized system is achievable by designing the noninteracting control. Furthermore, it is necessary to ensure the reference inputs provided by the human, v h , and the autonomous controller, v a , stabilize the system (6). It is important to consider common cases in which the dimensionality of the input signal from the human, uh , is less than the number of states that need to be controlled. This can be due to the kinematic constraints of the human motion or because of the available undersensing measurement devices. In order to achieve the controllability of the system or a subsystem, the appropriate forward mapping of the human input commands is required, e.g. in the form of hand pose reconstruction, synergy based approach, etc.

Fig. 7. Geometrical representation of the overlapping interaction paradigm Definition 4. The vector field g is cooperative on a mani∂g (x) has all non-negative fold M if the Jacobian matrix ∂x off-diagonal elements for all x ∈ M. The system 1 is cooperative, if g is cooperative. Definition 5. The vector field g is competitive on a man∂g ifold M if the Jacobian matrix − (x) has all non∂x negative off-diagonal elements for all x ∈ M. The system 1 is competitive, if g is competitive. Example 3. A classical example of the cooperative overlapping interaction paradigm is load sharing in physical human-robot team interaction, depicted in the fig. 8. The human operator and the robot team cooperatively grasp the object. Let us consider only the translational motion. Furthermore, let us assume the object dynamics is known and is given with the following equation: Mo (po )p¨o + co (po , p˙ o ) + hg (po ) = Gu (27) where po , p˙ o , p¨o ∈ R3 are position, translational velocity and translational acceleration of the common object.

3.5 Simulation results Simulation results, depicted in fig. (9) and fig. (10), show the behavior of the cooperative system. Cooperative behavior is achieved in z direction, and the relative behavior 46

2016 IFAC CPHS December 7-9, 2016. Florianopolis, Brazil

Selma Musić et al. / IFAC-PapersOnLine 49-32 (2016) 042–047

0.1 0.05 0 −0.05 −0.1

0.05

20 15 10 5 0

p˙ x(r1,r2)

p˙ h,a 2

[m/s]

0.1

0

[N]

p˙ h,a 1

p˙ c

6

8

10 t[s]

12

14 fc

6

8

10 t[s]

0

2

4

6

8

10

t[s]

[N]

[m/s]

p˙ h(1,2,c)

47

12

14

20 10 0 −10 −20

f1x

0

2

4

6

8

f2x

10

t[s]

Fig. 9. Top: velocity of the object. Bottom: wrench on the object.

Fig. 10. Top: relative motion is blocked by the object. Bottom: internal forces applied by the manipulators.

in x direction. The results can represent all three interaction paradigms. We assume Coriolis, centrifugal and gravity terms to be compensated. In the case of direct interaction paradigm, a desired object manipulation is achieved if the motion of the human fingers is in coordination. However, relative motion between the fingers can occur due to inherent uncertainty of the human behavior. This can cause an internal loading on the object and, hence, its undesirable motion. Therefore, the direct interaction paradigm is suitable when the robots are not in contact with the object (e.g. the grasping stage). When the robots are in contact it is reasonable to to achieve a precise, desired, relative behavior with the autonomous controller. Hence, the complementary interaction paradigm is suitable. It can be observed that the robots share equal load of the object as the forces applied in x direction are acting opposite and of the same intensity. Relative forces do not affect the cooperative sub-task of the system. When the agents are heterogeneous (e.g. humans and robots), the load sharing depends on the capabilities of the agents. For example, one may wish to minimize the effort exerted by the human and assign the effort completely to the robot. In this case the overlapping interaction paradigm is suitable.

tems, Man and Cybernetics, 2004 IEEE International Conference on, volume 3, 2948–2953 vol.3. Chinello, F., Malvezzi, M., Pacchierotti, C., and Prattichizzo, D. (2015). Design and development of a 3rrs wearable fingertip cutaneous device. In Proc. IEEE/ASME International Conference on Advanced Intelligent Mechatronics, 293–298. Erhart, S. and Hirche, S. (2016). Model and analysis of the interaction dynamics in cooperative manipulation tasks. IEEE Transactions on Robotics (T-RO). Evrard, P. and Kheddar, A. (2009). Homotopy-based controller for physical human-robot interaction. In ROMAN 2009 - The 18th IEEE International Symposium on Robot and Human Interactive Communication, 1–6. Isidori, A. (1995). Nonlinear Control Systems. Laschi, C., Teti, G., Tamburrini, G., Datteri, E., and Dario, P. (2001). Adaptable semi-autonomy in personal robots. In Proceedings of the 10th IEEE International Workshop on Robot and Human Interactive Communication, 152–157. Lawitzky, M., Moertl, A., and Hirche, S. (2010). Load sharing in human-robot cooperative manipulation. In 19th International Symposium in Robot and Human Interactive Communication. Lee, D. and Spong, M. (2005). Bilateral teleoperation of multiple cooperative robots over delayed communication networks: Theory. In IEEE International Conference in Robotics and Automation. Lin, C.W., Khong, M.H., and Liu, Y.C. (2015). Experiments on human-in-the-loop coordination for multirobot system with task abstraction. IEEE Transactions on Automation Science and Engineering, 12(3), 981–989. Tonin, L., Leeb, R., Tavella, M., Perdikis, S., and d. R. Millan, J. (2010). The role of shared-control in bci-based telepresence. In Systems Man and Cybernetics (SMC), 2010 IEEE International Conference on, 1462–1466. Yanco, H.A. and Drury, J.L. (2004). Classifying humanrobot interaction: an updated taxonomy. In SMC (3), 2841–2846.

4. CONCLUSION In this paper three interaction paradigms between a single human and multiple robot systems are proposed: direct, complementary and overlapping interaction paradigms.It is shown that it is possible to distinguish the interaction paradigms using the controllability property and the structure of the Jacobian matrix of the mapping g. The identified properties ease the selection of the controller objectives and strategies.In future work the required properties of the observability will be considered. Additionally, the identified properties will be used to perform synthesis of controllers. REFERENCES Baker, M. and Yanco, H.A. (2004). Autonomy mode suggestions for improving human-robot interaction. In Sys47