A tour-guide robot: Moving towards interaction with humans

Engineering Applications of Artificial Intelligence 88 (2020) 103356

Contents lists available at ScienceDirect

Engineering Applications of Artificial Intelligence journal homepage: www.elsevier.com/locate/engappai

A tour-guide robot: Moving towards interaction with humans✩ Biel Piero E. Alvarado Vásquez ∗, Fernando Matía Centre for Automation and Robotics (CAR), UPM-CSIC 28006 Madrid, Spain

ARTICLE

INFO

Keywords: Fuzzy systems Emotions Indoor localization and navigation Task planning Social robotics Human–machine interaction

ABSTRACT The aim of this research project is to develop a smart social robot showing sufficient intelligence to work as a tour-guide in different environments. In doing so, both a software and a hardware architecture are proposed, the different modules of which, such as a laser, cameras, platform, face, and voice, among others, control the different components of the robot. Those components are in turn used by other modules designed for navigation and interaction. A sensor fusion for the purposes of localization is implemented by means of an Extended Kalman Filter, which is one of the navigation module components, together with the proposed fuzzy controllers needed for path following. A fuzzy emotion system that controls the face and the voice modules also forms part of this architecture for assisting interaction. Finally, all the modules are controlled with a customized programming language that is a mixture of C, Pascal, and JavaScript. The modules are optimized for immediate execution to achieve realistic human–machine interaction.

1. Introduction Robots that interact and assist the public conjure up futuristic scenarios from television, cinema, and other media. This research focuses its attention on robots that interact with people indoors, such as waiters, domestic assistants, and museum tour-guides, among others. These types of robots must, however, be sufficiently well programmed for realistic human–machine interaction. In this study, the mobile robot designed to work in indoor environments as a laboratory tour-guide is called Doris. Programming a robot is no easy task in view of the many issues that need to be solved, such as the localization of the robot, its movements, its ability to process information in complex environments, the definition of set conditions, the interpretation of an input and the appropriate response, and so forth. Once solved, these tasks will have to work in complete synchrony. The aim of this research is to program the robotic guide, Doris, with sufficient intelligence, to perform tours in different environments such as laboratories and museums, a goal which will require the design of an efficient software/hardware architecture. The movement of the robot around the premises of the Centre for Automation and Robotics (CAR) will require synchronicity between several subsystems of its architecture to achieve a successful tour. These days, with the rapid growth of Artificial Intelligence (AI) linked to machine operations, robotic advances can cause concern over employment issues, even though those concerns are misplaced. The

intention is to assist communities by helping people with every-day, repetitive tasks, so that employee time can be dedicated to creative productivity, so that employees can learn new skills at work, and so that they can be freed from tiresome chores. In that sense, the prospect of a tour-guide robot can only bring beneficial consequences for employees and communities. As regards the hardware, the robot is built upon a mobile platform with a skeletal structure attached to a head that gives Doris a human form, for user friendly interaction with the environment, as well as additional sensors such as RFID and an omnidirectional camera, which are used for localization, navigation, and interaction. The software architecture involves a client–server application with multiple threads that communicate with each other, in order to ensure fluid human–machine interaction and between internal processes. The other subsystems designed for this architecture, in addition to the localization system, include the path-planning system, the mapping system, the emotional response systems, the facial response system, the lip synchronization system, and the task planner system. The paper will be structured as follows: in the following section, Doris’ architecture will be briefly explained, detailing the modular interconnections for multiple task performance. The navigation system will be explained in Section 3 that will include the localization system, the path planning, and the semantic maps. The different controllers designed for the path planning will also be outlined. Then, the interaction system will be explained in Section 4, where the speech system, the emotional response system, the facial interaction and the

✩ No author associated with this paper has disclosed any potential or pertinent conflicts which may be perceived to have impending conflict with this work. For full disclosure statements refer to https://doi.org/10.1016/j.engappai.2019.103356. ∗ Corresponding author. E-mail addresses: [email protected] (B.P.E. Alvarado Vásquez), [email protected] (F. Matía).

https://doi.org/10.1016/j.engappai.2019.103356 Received 18 March 2019; Received in revised form 3 October 2019; Accepted 4 November 2019 Available online xxxx 0952-1976/© 2019 Elsevier Ltd. All rights reserved.

B.P.E. Alvarado Vásquez and F. Matía

Engineering Applications of Artificial Intelligence 88 (2020) 103356

focus on those languages for task planning and interaction with people. Some languages can be mentioned here, such as ROBOL/0 (Suzuki and Yuta, 1991) which describes a set of actions for navigation behavior. This language consists of two parts. One is the control structure where a set of events are written and the conditions for those events are concurrently evaluated. If the condition to execute the event is true then the robot proceeds to perform the actions assigned to that concrete event. The second part of ROBOL/0 consists of the robotic functions, which are sensor and actuator functions. Another programming language is Lapham (1999), which provides an easy tool to program a robotic arm for certain tasks. In Levesque et al. (1997), a new logic programming language called GOLOG is proposed, which maintains a representation of the dynamic world. This language was programmed in PROLOG and appears well suited for high-level control of robotic applications and industrial processes. A related study can also be found in Giacomo et al. (2000), an extension of GOLOG that includes concurrency, which facilities the prioritization of concurrent processes and their execution. Execution is interrupted when certain conditions become true and when responding to exogenous actions. A programming language for robot swarm named Buzz was presented in Pinciroli and Beltrame (2016) which is dedicated to heterogeneous robot swarms. Buzz syntax, inspired by dynamic languages such as JavaScript, Lua, and Python, mixes imperative and functional constructs. Stenmark and Nugues (2013) introduced a method to use written instructions using natural language to program assembly tasks for industrial robots, applying compiler development methodology to the design of semantic and syntactic parsers. Another contribution can be found in Diprose (2011) where a robot called RURU implements a visual programming language that is described for end users. With the extended use of XML for different purposes, Lutovac et al. (2012) created L-IRL, a procedural language for industrial robot programming based on XML code.

lip synchronization systems are all detailed as well as their operation in connection with their respective databases. The task planner employs a new programming language that will be briefly described in Section 5, detailing the development of an interpreter, and showing the grammar and the functionality of each command of the new language. Finally, the results of integrating the system and its respective demonstration will be shared and may be observed in videos. 2. Related work Social robots are very useful for interacting with people in different environments, from waiters to museum tour-guides accompanying people through different exhibition spaces while commenting on portraits and sculptures. Work has been underway in this area for many years. One example is MINERVA, a social robot for human–machine interaction designed by Thrun et al. (2000) that used to work in the Smithsonian museum. Its probabilistic software means that it relies on explicit representations of uncertainty through perception and control. Another example of a study on an interactive robot that functions as a museum guide can be found in Burgard et al. (1999), which describes a software architecture for RHINO that integrates localization, mapping, collision avoidance, planning, and various modules concerned with user interaction. Other work performed by the same author with the same robot can also be found in Burgard et al. (1998). Boboc et al. (2014) also studied the use of a robot as a tour-guide at the Industrial Informatics and Robotics Laboratory from the Research & Development Institute of Brasov. The robot from Aldebaran Robotics, called NAO, welcomed new visitors and gave them tours of the lab, providing explanations on the research that was conducted there. Vogiatzis et al. (2008) presented another work for human–robot interaction, in particular, robotic emotions, speech, and facial expressions, to determine whether humans feel comfortable in front of a machine that demonstrates basic rational and intelligent behavior. Another study on interactive robots, Tekerek (2009), presented an educational robot for teaching robotics to children, planning applications, and performing mechanical construction. The robots used for this application were RHINO XR3 and MARK 3 Controller. A further example of robotic architecture can be found in Stricker et al. (2012). It focuses on the constraints resulting from the challenging environment of a multifloor building as well as the integration aspects of various skills for navigation and human–robot interaction. Basically, according to Fong et al. (2002), all interactive robots have a common taxonomy and many functions, among which facial expressions, artificial emotions, and speech capabilities, and they are capable of performing certain tasks and achieving set goals. All these works are based on psychological studies of human interactions with machines. One of the best known studies, Mori et al. (1970), sets out the debate over the ’uncanny valley’: the unsettled and sometimes emotional reactions of human beings to humanoid robots. Social robots of varied morphology (human, dog, cat, penguin, rabbit, etc.) will have greater morphological acceptance and human rejection will be less likely. Borody (2013) was a further work on the concept of the uncanny valley that discussed similarity with humans. The greater the similarity, the higher the likelihood of rejection. Research into human acceptance of robots by ul Haque et al. (2007) identified design of increasingly human-like appearances, behavioral aspects, interaction, and learning capabilities in relation to their domestic applications. Besides appearance, there is also the key feature of speech capabilities, as a robot is expected to address and to respond to a user. Ramey et al. (2012) performed a voice-recognition study to control a robot by voice alone. An emotional model, such as the one used by Álvarez et al. (2010), is also a significant feature for social robotics. Concerning task planning, several research projects have pursued the sequential execution of tasks. Different types of programming languages can be found in Yang et al. (2015), showing the applications (general or robot-specific) of each one. The interest of this research is to

3. Doris, the social robot girl Doris is a mobile robot designed to interact with people in dynamic indoor environments such as public buildings, museums, theaters, conferences, trade fairs, etc. Doris represents the upgrade of Blacky and Urbano, the two previous robots that the group at CAR has been developing over recent years at the Universidad Politécnica de Madrid. 3.1. Hardware architecture The hardware consists of three parts: ∙ The platform, designed by Adept MobileRobots, is a differential steer mobile robot, which has built-in sensors (bumpers, sonars, and laser), all connected to the serial port of the internal computer. This computer is equipped with USB ports, Ethernet, and a WIFI connection. Fig. 1(a) shows the platform. ∙ The skeleton, attached to the mobile platform, made of methacrylate, holds up speakers, the RFID antennas, a USB HUB, a PoE switch and the robot’s head. The torso is designed to replicate the physical appearance of a human. In the future, an arm will be attached to develop pick and place tasks. ∙ The head, attached to the skeleton, with 20 DoF can show different expressions and can be also selected by the robot user. The robot is also speech capable and its mouth will synchronize with the sounds that are emitted; a combination known as viseme-based acoustic speech, that is shown in Fig. 1(c). 3.2. Software architecture The software works as a Client-Server application, enabling direct communication between Doris (server) and the (client) that controls 2

B.P.E. Alvarado Vásquez and F. Matía

Engineering Applications of Artificial Intelligence 88 (2020) 103356

Fig. 1. Doris hardware overview.

Fig. 2. Software architecture of Doris.

it. Nowadays, there are three different types of client applications for Doris: Web, Desktop, and Mobile devices. The server application is divided into four layers detailed in Fig. 2. Each block on the figure details a task that is asynchronously under execution on a separate thread, implying the existence of a task handler at the higher level of the architecture.

the server communications with a networked camera and an RF device. • Link Layer: instructions for robotic movements are generated at this level of Doris’ programming. The localization task is a thread at this layer that processes data from the logic layer, by using odometrical information and data from the R-Marker and the V-Markers threads. This thread, once data from sensors are processed, purges the buffer for new incoming data. The path controllers are also executed in this layer. These are the path follower and the hallway controller, developed using fuzzy logic, which switch each other on according to environmental conditions. Commands that move face features and/or the arm can also express emotional responses and expressive gestures. The TTS Manager is the process responsible for speech in Doris. It receives an output message and it modifies the phrase, so as to ensure synchronization with the lip movements. • Task Planner Layer: This layer links all the modules from the lower layers with a program developed by the robot user. Hence, this layer contains a lexer and a parser for program interpretations and the program event handler which connects with the events dispatched by the modules from the link layer. • User Layer: Client communications are performed at the user layer along with database loading. These databases correspond to mapping, viseme codes, texts to speech, robot configuration parameters, and so on. The data are read from the threads using shared memory. Doris can manage up to ten users at the same time, but only one can have full access to perform tasks.

• Hardware Layer: activates connections with different devices installed on the platform (encoders, bumpers, sonars, cameras, laser, etc.), as well as with USB and Ethernet devices, at a level with permissions for read/write data approved by superior levels. e.g. robot positioning, speed of robot movements, etc. • Logic Layer: at this level, data sent from the first layer via serial channel are processed for later use by the immediate superior layer, e.g. Data from encoders are translated into differential polar coordinates [𝛥𝜌 (𝑘) 𝛥𝜙 (𝑘)]𝑇 for odometry, and then converted into an oriented position [𝑥 (𝑘) 𝑦 (𝑘) 𝜃 (𝑘)]𝑇 . All of the modules in this layer are threads that are launched when the application starts up, for example: the R-Marker is a thread dedicated to detect the reflective landmarks with the laser. Once collected, the data are sent to other threads using shared memory. The V-Markers represent another thread detecting the visual markers using the omnidirectional camera. Shared memory is also used to transmit the detected markers to other threads and for sharing data in the context of RF-Tag detection, a thread for detecting RFID tags using the RFID System. The Face controller is another thread that is at all times ready to process information in messages on the servo positions of the face. It also turns on 3

B.P.E. Alvarado Vásquez and F. Matía

Engineering Applications of Artificial Intelligence 88 (2020) 103356

Fig. 3. Diagram of Doris sensor fusion. Fig. 5. Fiducial Marker used for localization.

Fig. 4. Reflective landmark used for localization.

4. The navigation system Doris is equipped with a navigation system based on three elements: mapping, localization, and path planning. A semantic map is defined for greater familiarity with the environment without using SLAM and to provide information on localization and path planning. Localization is based on an Extended Kalman Filter using sensor fusion, explained in Alvarado Vásquez et al. (2018b), for detecting landmarks and path planning is done by using the paths referenced in the same semantic map. Doris also has an RFID system that is used for identifying the semantic items placed throughout the environment, rather than for localization purposes, and for the identification of people. The task planning for the navigation system is executed by the programming language developed for Doris, as detailed in Section 6. Related works can be found at Castao et al. (2017), La Fe-Perdomo et al. (2019) and Beruvides et al. (2016, 2013).

Fig. 6. Sensor locations on Doris.

and 5 columns, to avoid orientation issues, are positioned at a height of 275 cm above the ground, in an attempt to avoid occlusions, and to place the maximum numbers of visual markers. The camera can visualize up to 5 markers at the same time, while the laser can react with at least 2. The presence of people and different objects can be located when both sensors are working at the same time, although Doris can also use the camera by itself for location purposes. However, there will be situations where both types of landmarks will not be easy to place, or the tour-guide site will not permit beacons in those positions, or the room might be too dark to place the visual markers; all of which were reasons for implementing the sensor fusion system. This methodology was developed to avoid using the SLAM technique. In Alvarado Vásquez et al. (2018a), research and information on RFID systems was also discussed. At first, RFID was the preferred localization option, but due to the large scale of error and deviation, it was decided to use the RFID system for semantic purposes inside the environment. In Fig. 6, all of the sensor locations built into Doris can be seen.

4.1. Localization The localization for Doris is solved by using sensor fusion explained in Alvarado Vásquez et al. (2018b), where detection of reflective beacons is performed using a laser range finder and visual marker detection is performed using an omnidirectional camera. The information provided along with the information contained in the semantic map is fused into an Extended Kalman Filter. Fig. 5 shows an explanation of how the sensor fusion works (see Fig. 3). The reflective landmarks are detected by using a LMS-200 laser range finder. These landmarks are circular shaped and were designed on an ad-hoc basis. The detection technique follows certain classification conditions and detects the center of the landmark by applying the Law of Sines, which is used as a model in an iterative minimum squared error estimator (see Fig. 4). The visual markers are detected using an omnidirectional Mobotix C25 camera. The fiducial markers, arranged in a matrix of 6 rows

Fig. 7 showing sensor fusion localization with the Extended Kalman Filter at the Centre for Automation and Robotics, Universidad Politecnica de Madrid. 4

B.P.E. Alvarado Vásquez and F. Matía

Engineering Applications of Artificial Intelligence 88 (2020) 103356

Fig. 7. Localization of Doris using sensor fusion in a laboratory.

Fig. 8. Inter-connections between each sector.

• features: contains information on objects placed in the environment such as doors, tables, closets, etc. also detailing, item by item, the location, the width, and the height. • sites: a set of waypoints around which Doris will be able to move. Each waypoint transmits its position in the form of an origin of coordinates, the semantic name of that site, and optionally some waypoints with the semantic label of doors, in case that site leads to another sector. This feature is optional, because there can be doors on which no information is needed, if for instance they connect to the outside of a building. • ways: contains the information on the interconnections between sites (adjacencies of each site). • tags: contains the information on tags spread over the sector, referencing semantic information contained in the sector.

Fig. 9. Representations of the paths in a sector where Doris is navigating.

4.2. Mapping Listing 1: Semantic map sample 1

The world in which Doris will perform has to be represented. An XML file is a common solution for world descriptions. This world is represented by some important features (tables, chairs, doors, chargers, objects...), sites where the robot needs to go to, the paths to arrive at those sites, and the landmarks located in each room, all of which will help Doris with localization.

2 3 4 5

In Listing 1, a sample is shown of one of the locations through which Doris will circulate, highlighting the set of landmarks, features, sites, and paths. This map is built with the a priori knowledge of the user.

6

7 8 9

• landmarks: contains information on the location of a set of landmarks and the type of each member of the set. The type of landmark can be sensitive to either lasers or to cameras.

10

5

< s e c t o r i d=" 1 " name=" Computer V i s i o n " width=" 1030 " h e i g h t=" 1146 " polygon="m 0 0 l 0 1146 l 1030 0 l 0 −1146 z " a dj a c e nc y=" 0 , 2 , 7 , 8 "> < f e a t u r e s>

B.P.E. Alvarado Vásquez and F. Matía

Engineering Applications of Artificial Intelligence 88 (2020) 103356

Fig. 10. Block diagram of the proposed fuzzy controller applied to Doris.

11 12 13

14 15 16 17

18 19

20

21 22 23 24 25 26 27 28 29 30 31 32 33

34

35 36

Inside each node depicted in Fig. 8, there are also nodes that represent the paths along which the robot can move towards other nodes that represent the places at which it will arrive. The second path planner takes place here, looking for the shortest path between two points of interest, using the paths specified in the semantic map. Those paths are marked inside the path subtree inside the sector where the attribute st corresponds to one site and the attribute adjacency indicates the nodes that are adjacent to that specific site. An A* algorithm is executed (Hart et al., 1968; Stentz, 1994; Koenig and Likhachev, 2005; Roditty and Zwick, 2011; Gudmundsson et al., 2002; Jan et al., 2008; Sudhakara and Ganapathy, 2016), to map the optimum path to the site where the robot is programming itself to move, where each weight between each node is dynamically assigned, depending on the proximity of the site. In Fig. 9, a representation of the paths that Doris will follow and the crossing points can be seen. In the figure, a door means that the crossing point is connected to another sector, which helps the external planner to map the best path between two sectors.

< s i t e s sequence=" 5 , 3 , 2 , 0 "> < s i t e i d=" 0 " name=" door " x=" 115 " y=" 1174 " l i n k e d − s e c t o r −i d=" 0 " xcoord=" 18 " ycoord=" −1146 " /> < s i t e i d=" 1 " name=" way " x=" 115 " y=" 713 " /> < s i t e i d=" 2 " name=" way " x=" 464 " y=" 713 " /> < s i t e i d=" 3 " name=" frame " x=" 115 " y=" 209 " /> < s i t e i d=" 4 " name=" door " x=" 0 " y=" 209 " l i n k e d −s e c t o r − i d=" 7 " xcoord=" −18 " ycoord=" 1180 " /> < s i t e i d=" 5 " name=" way " x=" 115 " y=" 80 " /> < s i t e i d=" 6 " name=" door " x=" 0 " y=" 80 " l i n k e d −s e c t o r −i d =" 8 " xcoord=" −18 " ycoord=" 1180 " /> < s i t e i d=" 7 " name=" door " x=" 115 " y=" 0 " l i n k e d −s e c t o r − i d=" 2 " xcoord=" 0 " ycoord=" 1059 " />

4.4. Path following Two fuzzy controllers were designed to complete the whole trajectory, one for speed and another for passing through hallways. Their structures are shown in Fig. 10. A fuzzy speed controller for Doris was proposed that uses the distance and the angle errors as inputs to the fuzzy system and that provides the linear and the angular speeds as outputs. A fuzzy controller was chosen as a regulator (similar to a Proportional–Derivative controller), in order to obtain the desired output during movement. The control sequence commands the robot to first turn towards the new direction and then to move forward to the desired point. The robot has a tendency to move towards the right, for reasons linked to both its wheel alignments and the ground. Fuzzy controllers are therefore a good choice to reduce this type of dynamic problem. Fuzzy sets were created for inputs (distance and angle error) and outputs(linear and angular speeds), in order to achieve that behavior. The inputs are represented as the membership functions shown in Figs. 11 and 12. As may be noted, the membership functions (MF) are triangular, except for the trapezoidal function Zero in the distance error fuzzy set. This arrangement will avoid continuous breaks during forward movement when the robot deviates excessively from a straight line, so that it will slightly correct its onward motion while continuing to move forward. The outputs of the fuzzy speed controller are depicted in the fuzzy sets with triangular membership functions shown in Figs. 13 and 14. As previously described, the robot must turn and then move forward to the desired point, and in cases where the robot deviates from a straight line, it will slightly turn again while it keeps moving forward. As well as the fuzzy sets, the rules of the fuzzy system must also express this behavior. The fuzzy rules are in the Listing 2.

4.3. Path planning As shown in Section 4.2, the semantic map is divided by sectors, so two path planners are needed for trajectories. One planner is used inside a specified sector by navigating to specific points inside the museum room, following a specific path detailed in the semantic map. The second planner is used for inter-sector navigation, for example, when the robot has ended the tour and has to return to the starting point. For external navigation a Branch and Bound algorithm is used (Nau et al., 1984; Ibaraki, 1978; Narendra and Fukunaga, 1977; Olson, 2000; Binney and Sukhatme, 2012; Nishi et al., 2005). A graph representing the external connection between sectors is built and used by the algorithm. The graph is also built by information provided in the semantic map, where each sector has an attribute named adjacency indicating the sectors linked to the current sector. In Fig. 8, a representation of the connections between the sectors of a specific map can be seen. Moreover, in this figure, the Hallway label represents a special condition that, as can be seen, has to activate a Hallway Controller. 6

B.P.E. Alvarado Vásquez and F. Matía

Engineering Applications of Artificial Intelligence 88 (2020) 103356

Fig. 11. Membership functions of the distance error input to the speed controller.

Fig. 12. Membership functions of the angle error input to the speed controller.

Fig. 13. Linear speed output membership function of the speed controller.

Fig. 14. Angular speed output membership functions of the speed controller.

Listing 2: Fuzzy speed controller rules 1

2

3

4

16

i f a n g l e E r r o r i s NVeryFar then l i n e a r V e l o c i t y i s Zero and a n g u l a r V e l o c i t y i s F a s t R i g h t i f a n g l e E r r o r i s NFar then l i n e a r V e l o c i t y i s Zero and a n g u l a r V e l o c i t y i s SlowRight i f a n g l e E r r o r i s PFar then l i n e a r V e l o c i t y i s Zero and angularVelocity i s SlowLeft i f a n g l e E r r o r i s PVeryFar then l i n e a r V e l o c i t y i s Zero and a n g u l a r V e l o c i t y i s F a s t L e f t

17

18

19 20

5 6

7

8

9

10

11

i f a n g l e E r r o r i s NNear and d i s t a n c e E r r o r i s NVeryFar then l i n e a r V e l o c i t y i s FastBackwards and a n g u l a r V e l o c i t y i s VSlowRight i f a n g l e E r r o r i s NNear and d i s t a n c e E r r o r i s NFar then l i n e a r V e l o c i t y i s Backwards and a n g u l a r V e l o c i t y i s VSlowRight i f a n g l e E r r o r i s NNear and d i s t a n c e E r r o r i s NNear then l i n e a r V e l o c i t y i s MediumBackwards and a n g u l a r V e l o c i t y i s VSlowRight i f a n g l e E r r o r i s NNear and d i s t a n c e E r r o r i s PNear then l i n e a r V e l o c i t y i s MediumForward and a n g u l a r V e l o c i t y i s VSlowRight i f a n g l e E r r o r i s NNear and d i s t a n c e E r r o r i s PFar then l i n e a r V e l o c i t y i s Forward and a n g u l a r V e l o c i t y i s VSlowRight i f a n g l e E r r o r i s NNear and d i s t a n c e E r r o r i s PVeryFar then l i n e a r V e l o c i t y i s FastForward and a n g u l a r V e l o c i t y i s VSlowRight

21

22

23

24

25

26

12 13

14

15

i f a n g l e E r r o r i s PNear and d i s t a n c e E r r o r i s PNear then l i n e a r V e l o c i t y i s MediumForward and a n g u l a r V e l o c i t y i s VSlowLeft i f a n g l e E r r o r i s PNear and d i s t a n c e E r r o r i s PFar then l i n e a r V e l o c i t y i s Forward and a n g u l a r V e l o c i t y i s VSlowLeft i f a n g l e E r r o r i s PNear and d i s t a n c e E r r o r i s PVeryFar then l i n e a r V e l o c i t y i s FastForward and a n g u l a r V e l o c i t y i s VSlowLeft

i f a n g l e E r r o r i s PNear and d i s t a n c e E r r o r i s NVeryFar then l i n e a r V e l o c i t y i s FastBackwards and a n g u l a r V e l o c i t y i s VSlowLeft i f a n g l e E r r o r i s PNear and d i s t a n c e E r r o r i s NFar then l i n e a r V e l o c i t y i s Backwards and a n g u l a r V e l o c i t y i s VSlowLeft i f a n g l e E r r o r i s PNear and d i s t a n c e E r r o r i s NNear then l i n e a r V e l o c i t y i s MediumBackwards and a n g u l a r V e l o c i t y i s VSlowLeft

i f a n g l e E r r o r i s Zero and d i s t a n c e E r r o r i s NVeryFar then l i n e a r V e l o c i t y i s FastBackwards and a n g u l a r V e l o c i t y i s Zero i f a n g l e E r r o r i s Zero and d i s t a n c e E r r o r i s NFar then l i n e a r V e l o c i t y i s Backwards and a n g u l a r V e l o c i t y i s Zero i f a n g l e E r r o r i s Zero and d i s t a n c e E r r o r i s NNear then l i n e a r V e l o c i t y i s MediumBackwards and a n g u l a r V e l o c i t y i s Zero i f a n g l e E r r o r i s Zero and d i s t a n c e E r r o r i s PNear then l i n e a r V e l o c i t y i s MediumForward and a n g u l a r V e l o c i t y i s Zero i f a n g l e E r r o r i s Zero and d i s t a n c e E r r o r i s PFar then l i n e a r V e l o c i t y i s Forward and a n g u l a r V e l o c i t y i s Zero i f a n g l e E r r o r i s Zero and d i s t a n c e E r r o r i s PVeryFar then l i n e a r V e l o c i t y i s FastForward and a n g u l a r V e l o c i t y i s Zero i f a n g l e E r r o r i s Zero and d i s t a n c e E r r o r i s Zero then l i n e a r V e l o c i t y i s Zero and a n g u l a r V e l o c i t y i s Zero

The other fuzzy controller proposed in this research is a controller that is activated when Doris is passing through a hallway. The main idea is that when entering a hallway, Doris will detect the entrance and align itself in the middle, except when a person passes in front or to one side, and the same applies to elements present in the hallway. The label hallway can be detected in two ways. One way is on the semantic map by identifying the corresponding sectors or sites using the hallway label. The second way is also on the semantic map, but in 7

B.P.E. Alvarado Vásquez and F. Matía

Engineering Applications of Artificial Intelligence 88 (2020) 103356

Fig. 15. Fuzzification of zones performed by the laser.

the Tags subtree, where the corresponding tags of the RFID system are used to identify the beginning and the end of a hallway. As stated, when detecting an approach to a hallway, Doris activates a controller specially designed for this type of environment. This controller overrides the speed controller designed for path following and keeps the robot aligned in the middle of the hallway. Centering the robot might be affected by dynamic or static objects not present in the semantic map, although they should pose no problem to Doris when crossing the hallway from beginning to end in a safe and secure mode. So, the first step in activating the hallway controller is the reception of a signal that will either be provided by the semantic map or by the RFID tags. Once inside the hallway, the controller will start to iterate until the ’end of hallway’ signal is received. The fuzzy system was designed with the FuzzyLite library (RadaVilela, 2018), the engine for which was formed of three inputs and two outputs. Doris’ laser emits a beam every 0.5 degrees in 180 degree scans resulting in 361 measurements. Those measurements are divided into three zones (left, front and right) each one of 120 measurements. Each zone is used as an input to the fuzzy systems as depicted in Fig. 15, while the angular and the linear speeds are used as outputs. Each zone is divided into five sub-zones, in order to avoid false measurement, due to dust or any other interference, which will represent every fuzzy membership of a fuzzy set (shown in Fig. 15), in order to reject all the meaningless measurements. Each input fuzzy set has triangular membership functions and looks like Fig. 16: The fuzzy outputs are defined as in Fig. 17 for linear speed and Fig. 18 for angular speed. Special cases apply to this controller. For example, when the controller finds an open door and the robot is commanded to move forward, the controller will keep Doris in the middle of the hallway, but if the width of the doorway or hallway widens or narrows, then the controller will modify the pathway so that Doris will move towards the middle. A sample of the fuzzy rule set is shown in the Listing 3.

5

6 7 8

9

10

11

12

14 15

16

17

18

19

20

2

3

4

/ ∗∗∗∗∗∗∗ / i f l a s e r L e f t Z o n e i s F and l a s e r F r o n t Z o n e i s VF and l a s e r R i g h t Z o n e i s VF then l i n e a r V e l o c i t y i s MediumForward and a n g u l a r V e l o c i t y i s Zero i f l a s e r L e f t Z o n e i s F and l a s e r F r o n t Z o n e i s VF and l a s e r R i g h t Z o n e i s F then l i n e a r V e l o c i t y i s MediumForward and a n g u l a r V e l o c i t y i s Zero i f l a s e r L e f t Z o n e i s F and l a s e r F r o n t Z o n e i s VF and l a s e r R i g h t Z o n e i s M then l i n e a r V e l o c i t y i s SlowForward and a n g u l a r V e l o c i t y i s S l o w L e f t i f l a s e r L e f t Z o n e i s F and l a s e r F r o n t Z o n e i s VF and l a s e r R i g h t Z o n e i s N then l i n e a r V e l o c i t y i s Zero and a n g u l a r V e l o c i t y i s S l o w L e f t i f l a s e r L e f t Z o n e i s F and l a s e r F r o n t Z o n e i s VF and l a s e r R i g h t Z o n e i s VN then l i n e a r V e l o c i t y i s Backwards and a n g u l a r V e l o c i t y i s F a s t L e f t

13

Listing 3: Fuzzy hallway controller rules sample 1

i f l a s e r L e f t Z o n e i s VF and l a s e r F r o n t Z o n e i s VF and l a s e r R i g h t Z o n e i s VN then l i n e a r V e l o c i t y i s Backwards and a n g u l a r V e l o c i t y i s F a s t L e f t

i f ( l a s e r L e f t Z o n e i s VF and l a s e r F r o n t Z o n e i s VF ) and l a s e r R i g h t Z o n e i s VF then l i n e a r V e l o c i t y i s FastForward and a n g u l a r V e l o c i t y i s Zero i f l a s e r L e f t Z o n e i s VF and l a s e r F r o n t Z o n e i s VF and l a s e r R i g h t Z o n e i s F then l i n e a r V e l o c i t y i s MediumForward and a n g u l a r V e l o c i t y i s Zero i f l a s e r L e f t Z o n e i s VF and l a s e r F r o n t Z o n e i s VF and l a s e r R i g h t Z o n e i s M then l i n e a r V e l o c i t y i s SlowForward and a n g u l a r V e l o c i t y i s S l o w L e f t i f l a s e r L e f t Z o n e i s VF and l a s e r F r o n t Z o n e i s VF and l a s e r R i g h t Z o n e i s N then l i n e a r V e l o c i t y i s Zero and a n g u l a r V e l o c i t y i s S l o w L e f t

/ ∗∗∗∗∗∗∗ / i f l a s e r L e f t Z o n e i s M and l a s e r F r o n t Z o n e i s VF and l a s e r R i g h t Z o n e i s VF then l i n e a r V e l o c i t y i s SlowForward and a n g u l a r V e l o c i t y i s SlowRight i f l a s e r L e f t Z o n e i s M and l a s e r F r o n t Z o n e i s VF and l a s e r R i g h t Z o n e i s F then l i n e a r V e l o c i t y i s SlowForward and a n g u l a r V e l o c i t y i s SlowRight i f l a s e r L e f t Z o n e i s M and l a s e r F r o n t Z o n e i s VF and l a s e r R i g h t Z o n e i s M then l i n e a r V e l o c i t y i s SlowForward and a n g u l a r V e l o c i t y i s Zero i f l a s e r L e f t Z o n e i s M and l a s e r F r o n t Z o n e i s VF and l a s e r R i g h t Z o n e i s N then l i n e a r V e l o c i t y i s Zero and a n g u l a r V e l o c i t y i s S l o w L e f t i f l a s e r L e f t Z o n e i s M and l a s e r F r o n t Z o n e i s VF and l a s e r R i g h t Z o n e i s VN then l i n e a r V e l o c i t y i s Backwards and a n g u l a r V e l o c i t y i s F a s t L e f t i f l a s e r L e f t Z o n e i s M and l a s e r F r o n t Z o n e i s F and l a s e r R i g h t Z o n e i s VF then l i n e a r V e l o c i t y i s SlowForward and a n g u l a r V e l o c i t y i s SlowRight

21 22 23

24

25

8

/ ∗∗∗∗∗∗∗ / i f l a s e r L e f t Z o n e i s N and l a s e r F r o n t Z o n e i s VF and l a s e r R i g h t Z o n e i s VF then l i n e a r V e l o c i t y i s Zero and a n g u l a r V e l o c i t y i s SlowRight i f l a s e r L e f t Z o n e i s N and l a s e r F r o n t Z o n e i s VF and l a s e r R i g h t Z o n e i s F then l i n e a r V e l o c i t y i s Zero and a n g u l a r V e l o c i t y i s SlowRight i f l a s e r L e f t Z o n e i s N and l a s e r F r o n t Z o n e i s VF and l a s e r R i g h t Z o n e i s M then l i n e a r V e l o c i t y i s Zero and a n g u l a r V e l o c i t y i s SlowRight

B.P.E. Alvarado Vásquez and F. Matía

Engineering Applications of Artificial Intelligence 88 (2020) 103356

Fig. 16. Each membership function of the input zones.

Fig. 17. Membership functions of linear speed.

Fig. 18. Membership functions of angular speed.

26

27

28

i f l a s e r L e f t Z o n e i s N and l a s e r F r o n t Z o n e i s VF and l a s e r R i g h t Z o n e i s N then l i n e a r V e l o c i t y i s Zero and a n g u l a r V e l o c i t y i s Zero i f l a s e r L e f t Z o n e i s N and l a s e r F r o n t Z o n e i s VF and l a s e r R i g h t Z o n e i s VN then l i n e a r V e l o c i t y i s Backwards and a n g u l a r V e l o c i t y i s S l o w L e f t i f l a s e r L e f t Z o n e i s N and l a s e r F r o n t Z o n e i s F and l a s e r R i g h t Z o n e i s VF then l i n e a r V e l o c i t y i s Zero and a n g u l a r V e l o c i t y i s SlowRight

29 30 31

32

33

34

35

36

/ ∗∗∗∗∗∗∗ / i f l a s e r L e f t Z o n e i s VN and l a s e r F r o n t Z o n e i s VF l a s e r R i g h t Z o n e i s VF then l i n e a r V e l o c i t y i s Backwards and a n g u l a r V e l o c i t y i s F a s t R i g h t i f l a s e r L e f t Z o n e i s VN and l a s e r F r o n t Z o n e i s VF l a s e r R i g h t Z o n e i s F then l i n e a r V e l o c i t y i s Backwards and a n g u l a r V e l o c i t y i s F a s t R i g h t i f l a s e r L e f t Z o n e i s VN and l a s e r F r o n t Z o n e i s VF l a s e r R i g h t Z o n e i s M then l i n e a r V e l o c i t y i s Backwards and a n g u l a r V e l o c i t y i s F a s t R i g h t i f l a s e r L e f t Z o n e i s VN and l a s e r F r o n t Z o n e i s VF l a s e r R i g h t Z o n e i s N then l i n e a r V e l o c i t y i s Backwards and a n g u l a r V e l o c i t y i s SlowRight i f l a s e r L e f t Z o n e i s VN and l a s e r F r o n t Z o n e i s VF l a s e r R i g h t Z o n e i s VN then l i n e a r V e l o c i t y i s Backwards and a n g u l a r V e l o c i t y i s Zero i f l a s e r L e f t Z o n e i s VN and l a s e r F r o n t Z o n e i s F l a s e r R i g h t Z o n e i s VF then l i n e a r V e l o c i t y i s Backwards and a n g u l a r V e l o c i t y i s F a s t R i g h t

and

and Fig. 19. Dialog task scheme.

and

5.1. The speech system

and

As Doris has a need to communicate with people, a task for task was created. This task receives a text as input provided by the user with whom Doris is interacting and according to the current emotion, the task will provide a reasonable answer, which is then relayed to the TTS module responsible for reproducing the answer. Matsusaka et al. (1999) performed a very similar task and was capable of multiple interactions with several people in such places as a laboratory or a museum. Wilcock et al. (2016) performed a similar task, so that the NAO robot could address users in both English and Sami. The idea proposed in this research is that once a user writes or repeats a phrase in one of the client applications designed for Doris, the phrase is converted into a module input. The input is looked up in the XML database built for this purpose, as shown in the Listing 4. Once the input has been approximated with a phrase, the id is taken and sent to the emotional response system module. The emotional response system provides the new updated emotion for Doris and with the previous id, a proper answer can be given to the user. Finally, the answer is sent to the TTS module where the text will be reproduced for the user. Fig. 19 shows a diagram that depicts the proposed scheme. As stated, the dialog task receives an input text from the user. The text is analyzed to find the same input in the database that consists of the following content in an XML file:

and

and

The final behavior of the fuzzy controllers are explained in the results Section 7.

5. The interaction system The system components through which Doris interacts are the face, the emotional response system, the speech system and the lip synchronization system. In the research of Kiesler (2005), effective communication between people and interactive robots is discussed, seeking to address the problem of establishing common grounds for understanding. Basically, the simpler the communication is, the better the robotic response to the user. Each robotic subsystem, its design, and operation, are detailed in its own section. As with the navigation system, the task planning concerning the face is coordinated by the language developed for Doris, as detailed in Section 6. 9

B.P.E. Alvarado Vásquez and F. Matía

Engineering Applications of Artificial Intelligence 88 (2020) 103356

Listing 4: Sample of XML file of input and output messages 1 2 3

4

5

6 7 8

9

10

11

5.2. The emotion system



In recent research, people have sought to take robotics a step further, by seeking to develop a system that is capable of providing robotic characteristics, so that users feel they are talking to more than a simple machine. One of the paths to achieve this characteristic can be by providing the robot with some kind of emotional response system. lai Lee et al. (2005) demonstrated that designers of human humanoid robots must react to several social cues to which most robots will not react, such as stimuli from different user inputs. iCat, a chess gamer robot that expresses emotions, represents an attempt by Leite et al. (2008) to configure an emotional robot. Designed on the basis of a heuristic evaluation of the state of the game, while iCat is winning its emotional response is happiness, but when loosing the emotions worsen and show sadness or even anger. Our aim here is not only limited to facial mannerisms, but also to a real feeling of awareness of the state of emotion, by using semantic self-representation, as described by Sanchez-Escribano and Sanz (2014) and Sanchez-Escribano (2018). As Doris is intended for human interaction, its system must be able to change its internal behavior, so that it can imitate human conversation as closely as possible. This emotional system consists of the user’s formulation of a question, relayed to Doris as a text string. This string is processed in the dialogue system, as explained in 5.1, by referencing a database where common questions and requests are stored. Depending on the result of this system, an id is provided to the emotional response system, which is then matched in the impulses database. This database is shown in Listing 5, which contains variations on each state of emotion and the effect on the variables of any response to the text supplied by the user.

As seen in XML sample file 4, there is an attribute named compl in the group of input messages, signaling whether the phrase is complete (0) or incomplete (1), and comparing the input text with the value in the request attribute. The wining request is the one with the highest number of word hits that is attributed the wining request id. This id is sent to the Emotion Task which will provide the corresponding state of emotion. The state of emotion is compared with the estattribute located in the output message group, which together with the previous id are factors considered in the selection of the emotional response. Finally, this response is sent to the corresponding TTS module. The task is detailed in the Algorithm 1.

Listing 5: Sample of XML emotions variations 1

Algorithm 1 Dialog Task principal function

2

function DialogTask(𝑖𝑛𝑝𝑢𝑡𝑇 𝑒𝑥𝑡) 𝑖𝑑 ← 𝑒𝑚𝑝𝑡𝑦 for each 𝑚𝑒𝑠𝑠𝑎𝑔𝑒 ∈ 𝑖𝑛𝑝𝑢𝑡𝑀𝑒𝑠𝑠𝑎𝑔𝑒𝑠 do if 𝑚𝑒𝑠𝑠𝑎𝑔𝑒.𝑐𝑜𝑚𝑝𝑙 = 0 and 𝑖𝑛𝑝𝑢𝑡𝑇 𝑒𝑥𝑡 = 𝑚𝑒𝑠𝑠𝑎𝑔𝑒.𝑟𝑒𝑞𝑢𝑒𝑠𝑡 then 𝑖𝑑 ← 𝑚𝑒𝑠𝑠𝑎𝑔𝑒.𝑖𝑑 else ℎ𝑖𝑡𝑠 ← 𝑐𝑜𝑖𝑛𝑐𝑖𝑑𝑒𝑛𝑐𝑒𝑠 𝑏𝑒𝑡𝑤𝑒𝑒𝑛 𝑖𝑛𝑝𝑢𝑡𝑇 𝑒𝑥𝑡 and 𝑚𝑒𝑠𝑠𝑎𝑔𝑒.𝑟𝑒𝑞𝑢𝑒𝑠𝑡 if ℎ𝑖𝑡𝑠 = 𝑛𝑢𝑚𝑏𝑒𝑟𝑂𝑓 𝑊 𝑜𝑟𝑑𝑠(𝑚𝑒𝑠𝑠𝑎𝑔𝑒.𝑟𝑒𝑞𝑢𝑒𝑠𝑡) then 𝑖𝑑 ← 𝑚𝑒𝑠𝑠𝑎𝑔𝑒.𝑖𝑑 end if end if end for return 𝑖𝑑 end function

3

4

5

6

7

8

9

10

11

Then, when the Emotion Task returns the state of emotion, the final response is selected randomly from the set of output messages. Algorithm 2.

12



The decision on the variations of each emotional variable will depend on the input that is empirically set, according to the previous work developed by Álvarez et al. (2010), and on the interaction between the membership functions of Doris. The set of emotions is based on these state variables ranging from 0 to 100:

Algorithm 2 Dialog response task function after emotion state is computed Require: 𝑖𝑑 function DialogResponseTask(𝑒𝑚𝑜𝑡𝑖𝑜𝑛𝑆𝑡𝑎𝑡𝑒) 𝑟𝑒𝑠𝑝𝑜𝑛𝑠𝑒𝑠 as 𝑒𝑚𝑝𝑡𝑦 Set for each 𝑚𝑒𝑠𝑠𝑎𝑔𝑒 ∈ 𝑜𝑢𝑡𝑝𝑢𝑡𝑀𝑒𝑠𝑠𝑎𝑔𝑒𝑠 do if 𝑚𝑒𝑠𝑠𝑎𝑔𝑒.𝑒𝑠𝑡 = 𝑒𝑚𝑜𝑡𝑖𝑜𝑛𝑆𝑡𝑎𝑡𝑒 and 𝑚𝑒𝑠𝑠𝑎𝑔𝑒.𝑖𝑑 = 𝑖𝑑 then add 𝑚𝑒𝑠𝑠𝑎𝑔𝑒.𝑟𝑒𝑠𝑝𝑜𝑛𝑠𝑒 to 𝑟𝑒𝑠𝑝𝑜𝑛𝑠𝑒𝑠 end if end for 𝑟𝑒𝑠𝑝𝑜𝑛𝑠𝑒 ← random(𝑟𝑒𝑠𝑝𝑜𝑛𝑠𝑒𝑠) TTS(𝑟𝑒𝑠𝑝𝑜𝑛𝑠𝑒) end function

• Approval • Joy • Fear Each state variable is bounded to some minimum and maximum values so that the emotional system will always be stable. Finally, to compute the final emotion, the state variables are used as inputs in a Fuzzy Engine. Each variable can be represented as a Fuzzy set shown in Fig. 20, where each membership function was calibrated on 10

B.P.E. Alvarado Vásquez and F. Matía

Engineering Applications of Artificial Intelligence 88 (2020) 103356

Fig. 20. Input emotion variables sample as a Fuzzy Set.

Fig. 21. Fuzzy output eyelids variable.

Fig. 22. Fuzzy output eyebrows variable.

Fig. 23. Fuzzy output Mouth variable.

Fig. 24. Fuzzy output voice rate variable.

Fig. 25. Fuzzy output response level variable.

a individual basis to each emotion, according to the desired behavior for Doris. The fuzzy system outputs are the different levels of facial expression on Doris, considering the eyelids, eyebrows, and mouth positions for all the emotional expressions. The tone and the final output message are also outputs of the fuzzy system:

As with every fuzzy system, there is a set of fuzzy rules, in order to decide upon the final output value. As the fuzzy system contains 3 input variables with 3 membership functions, by combination it will result in 27 rules. Some of those rules are shown in Listing 6. Listing 6: Emotion system fuzzy rules

• Eyelids: which semantic values are closed, half closed, normal, half opened and opened, with a range from 0 o 100. • Eyebrows: which semantic values are low, Normal and high, with a range from 0 o 100. • Mouth: which semantic values are so sad, sad, normal, smiley and big smiley, with a range from 0 o 100. • Voice Rate: which semantic values are low, normal and high, with a range from 0 o 100. • Response State: which semantic values are blue, hypocrite, normal, easy and enthusiastic, with a range from 0 to 5 (see Figs. 21–25).

1

i f JOY i s LOW and FEAR i s LOW and APPROVAL i s LOW then FACE_EYELIDS i s CLOSED and FACE_EYEBROWS i s LOW and FACE_MOUTH i s SO_SAD and VOICE_RATE i s LOW and VOICE_RESPONSE i s BLUE

2 3

i f JOY i s LOW and FEAR i s NORMAL and APPROVAL i s NORMAL then FACE_EYELIDS i s CLOSED and FACE_EYEBROWS i s NORMAL and FACE_MOUTH i s SAD and VOICE_RATE i s NORMAL and VOICE_RESPONSE i s HYPOCRITE

4 5

The fuzzy outputs are triangular membership functions: 11

i f JOY i s LOW and FEAR i s HIGH and APPROVAL i s HIGH then FACE_EYELIDS i s HALF_CLOSED and FACE_EYEBROWS i s NORMAL and FACE_MOUTH i s SAD and VOICE_RATE i s NORMAL and VOICE_RESPONSE i s HYPOCRITE

B.P.E. Alvarado Vásquez and F. Matía

Engineering Applications of Artificial Intelligence 88 (2020) 103356 Table 1 Doris head degrees of freedom distribution. Quantity

Head part description

Degrees of freedom

Total

2 2 2 1 1 1 1 1

Eye Eyelash Eyelid Left commissure Right commissure Upper lip Lower lip Neck

2 2 1 2 2 1 1 3

4 4 2 2 2 1 1 3

Total:

19

Table 2 Humans muscles involved in speech. Fig. 26. Doris different faces.

6 7

i f JOY i s NORMAL and FEAR i s LOW and APPROVAL i s LOW then FACE_EYELIDS i s NORMAL and FACE_EYEBROWS i s LOW and FACE_MOUTH i s SAD and VOICE_RATE i s HIGH and VOICE_RESPONSE i s HYPOCRITE

N◦

Muscle

Description

1 2

Upper lip elevator Orbicular

3 4

Lower lip depressor Mentalis

5 6 7

Risorius Zygomaticus major Zygomaticus minor

Raises the upper lip Open and close lips and helps with forward and backward movements Moves lower lip downwards Raises the chin skin and affects the raising of the lower lip Retracts the labial commissure producing a smile Raises the labial commissure abductor Acts as an elevator and abductor of the upper lip middle part

8 9

i f JOY i s NORMAL and FEAR i s NORMAL and APPROVAL i s NORMAL then FACE_EYELIDS i s NORMAL and FACE_EYEBROWS i s NORMAL and FACE_MOUTH i s NORMAL and VOICE_RATE i s NORMAL and VOICE_RESPONSE i s NORMAL

over control and when another user is waiting to take over control of the robot. The web interface must also be user friendly and well organized by showing the information received by the robot over a WebSocket dedicated exclusively to data reception on the different states of emotion of the robot. In addition, requests for actions that the robot must perform like moving, changing expressions, and loading a new map must be sent over another WebSocket. Figs. 27(a), 27(b), 27(c) show the web interface of Doris. As can be observed, it contains a menu bar with the buttons:

10 11

i f JOY i s NORMAL and FEAR i s HIGH and APPROVAL i s HIGH then FACE_EYELIDS i s NORMAL and FACE_EYEBROWS i s HIGH and FACE_MOUTH i s SMILEY and VOICE_RATE i s NORMAL and VOICE_RESPONSE i s EASY

12 13

i f JOY i s HIGH and FEAR i s NORMAL and APPROVAL i s NORMAL then FACE_EYELIDS i s HALF_OPENED and FACE_EYEBROWS i s HIGH and FACE_MOUTH i s SMILEY and VOICE_RATE i s NORMAL and VOICE_RESPONSE i s EASY

• Expressions: showing a new page that allows the user to modify the current facial expression. This window is shown in Fig. 27(a) • Feelings: showing a chart where changes to the states of emotion are displayed. Fig. 27(b) shows those changes. • Mapping: permits the user to know or to establish the current map where Doris is performing a tour. Fig. 27(c) • Arm, Chat, Settings: Future implementation of an interface to control a robotic arm for installation on Doris, a chat window, and configuration of the application.

14 15

i f JOY i s HIGH and FEAR i s LOW and APPROVAL i s HIGH then FACE_EYELIDS i s OPENED and FACE_EYEBROWS i s HIGH and FACE_MOUTH i s BIG_SMILEY and VOICE_RATE i s HIGH and VOICE_RESPONSE i s ENTHUSIASTIC

Hypothetical cases were included in this set of rules where maybe 2 or 3 emotions could be activated at the same time. For example, a case where a museum visitor is lost and when the person reappears Doris can simultaneously show happiness and displeasure, see the face depicted in Fig. 26(a). Alternatively, Doris might be in a good mood and might blink at her interlocutor, see Fig. 26(b).

There is also an icon that shows the connection status with Doris and another icon in a keypad shape that shows the token owner. 5.4. The expression system

5.3. The remote control system The importance of a robot displaying emotions has been explained in Section 5.2. Those emotions can determine the voice frequency and the facial gestures of Doris. A related work was completed in Kim et al. (2008), which examined the user gestures to produce other robot gestures. As Doris will interact with humans, its gestures also need to be human, that is the main reason why Doris has a face which can be very similar to a human, which is a key feature so that the user reaction will be closer to acceptance rather than rejection, as explained in Breazeal (2003), providing natural human–machine interaction (Fong and Nourbakhsh, 2003). Working along those lines, the face was constructed of a plastic faceplate, with aluminum supports for the eyes, the mouth, and 3D printed eyelids, eyelashes and lips. The design allows the face to have several degrees of freedom. In Table 1, the location of the degrees of freedom are shown.

As stated, Doris has been designed for pre-programmed human interaction, but also for interaction under remote control which can be monitored through a web application. The remote control can be configured so as to control Doris from a remote location next door, or even in a different city, or country, although the communications will work faster depending on the proximity of the robot supervisor. This web application will function with Android and iOS mobiles/tablets and with desktop navigators, such as Mozilla, Safari, Chrome, etc. Similar works can be found at del-Solar (2007) where the authors proposed an empathic web interface to configure personal robots. Moreover, in Schulz et al. (2011), a web interface was developed for controlling and configuring a tour-guide robot for use by several users. The web application must permit multiple user connections managing the user permissions and must notify the user when taking 12

B.P.E. Alvarado Vásquez and F. Matía

Engineering Applications of Artificial Intelligence 88 (2020) 103356

Fig. 27. Doris Webpage.

After the text is normalized, it is split into words without considering that there might be two consecutive words pronounced as one. Each word then is separated into syllables based on Spanish syllabication rules and those syllables are compared to the viseme database resulting in a viseme position. The transition time from the current viseme to the next is estimated and used to synchronize the speech rate. At the same time as the Text to Speech (TTS) device, which will previously have been initialized, is configured with a predetermined setup such as voice rate, pitch, and volume, the configuration will be modified by the Emotion Task explained in Section 5.2. After the configuration of the TTS device, it will keep waiting for some text to be emitted. With regard to the viseme database, the Spanish language has 27 letters which when combined in accordance with the syllabification rules results in almost 2500 syllables. Those syllables were added to an XML file, the contents of which were organized in 52 groups.

Fig. 28. Human-Doris mouth comparison.

5.4.1. Lip synchronization interaction Lip synchronization consists of parallel coordination between Doris’ mouth movements and the synthesized audio of a text predetermined by the user so that the robot can communicate with its human interlocutors in an almost natural way. As stated before, Doris has lips that have 6 degrees of freedom, commanded by each respective servomotors, which are controlled in order to perform synchronized movements with the sounds emitted by speakers, resulting in a accentuated speaking reality. The 6 degrees of freedom for the lips are similar to the human labial muscles. Table 2 shows the correspondence of each degree of freedom with those muscles. Figs. 28(a) and 28(b) show comparisons between human lips and Doris’ lips and their respective correspondence with Table 2. Some details such as teeth, tongue, jaw, palate were omitted in the design. Fig. 29 shows a flow chart of the procedure to achieve synchronization of the lip with speech. The first step is to receive the response text from the Dialog Task. This text is normalized by removing interrogation, exclamation, and any other symbols which are different to letters.

6. A programming language for Doris Understanding that a tour is a set of actions to be performed by Doris, the idea of creating a programming language was considered. The language would be similar to any other simple programming language, but would also include direct commands to control Doris’ movements, configuration of what, when, and how to say sentences, and direct actions, in response to different sensor data. Grammar, which will be used to program the sequence of actions for Doris, must contain common features of all programming languages such as input/output instructions, conditionals clauses, loops, arithmetic operations and Boolean operations. Besides those features, the new programming language for Doris will permit variable definitions, which can be either numeric or literal or defined as an array that can contain both numeric and literal definitions along the same vector. An 13

B.P.E. Alvarado Vásquez and F. Matía

Engineering Applications of Artificial Intelligence 88 (2020) 103356

Fig. 29. Lips synchronization flowchart.

example of the language that was customized for that purpose can be seen in the results Section 7.3, where a tour is performed that integrates all the components that Doris can employ (Alvarado Vásquez and Matia, 2019). 7. Results Test were performed at the Centre for Automation and Robotics that included individual tests of each subsystem explained in the previous sections and tests that integrated all of them. The integration was performed by using the task planner designed in the new programming language developed for Doris. The script in use is the one shown in the Listing 7, which covers an area of around 161 m2 containing immobile objects and people moving around. 7.1. Navigation system Besides localization with the sensor fusion system that has been successfully tested in Alvarado Vásquez et al. (2018b,a) the planning and the fuzzy controllers also had to be tested, which are the position controller and hallway controller, both dependent on the linear and angular speed inputs. With regard to the hallway controller, tests were performed in the main hallway of the building within the Central Laboratory of Electrical Engineering which is 175 cm wide and 24 m in length. The performance can be seen in Fig. 30 and a video, named Hallway Controller Test, can be found at the group’s webpage gallery Intelligent Control Group UPM-CSIC - Gallery.

Fig. 30. Hallway where controller was tested.

7.2. Interaction system Another experiment is to try to have a conversation with Doris. Table 4 shows a conversation with different topics that include flattery and insults The table shows that after an initial emotion where the robot appears quite, an input text is sent that appears to make her feel good. The emotion remains stable after the first insult, which appears not to be appreciated. The changes are based on the pre-established impulses for her behavior. Although it might appear strange to say that a robot can be angry or happy, states of emotion are within the remit of this research. Of course, as may be assumed, Doris was neither sufficiently well programmed to pass a Turing test, nor was that ever our intention. Some adjustments needed to be performed, in order to improve performance and to avoid drastic changes in the state of emotion. Those adjustments were made in the impulses xml file shown in Listing 5. A

Lip synchronization, facial interaction, and the emotional expressions were tested at the same time by providing some texts as inputs to observe how the state of emotion would vary and what the answers of Doris would be, randomly selected according to the emotion computed at that moment. One of the tests was to provoke sadness in Doris and to do so, a set of phrases was sent to the robot. The following Table 3 shows the changes of emotion in each phrase: The results shown in Table 3 were as expected. Negative input texts to the robot make the state of emotion change from normal to hypocritical and finally to sad. The changes to each emotional variable can be seen in the table: joy and approval decreased, as expected, although Fear almost remained the same. 14

B.P.E. Alvarado Vásquez and F. Matía

Engineering Applications of Artificial Intelligence 88 (2020) 103356

Table 3 Doris expressing different degrees of sadness. Input phrase You are a fool You are ugly You are stupid Jackanapes

Output phrase

Joy

Fear

Approval

Expression

Emotion value

Do not say that to me You are joking Do not say that You have gone crazy

50 37 32 27 22

50 47 50 55 58

50 40 30 20 10

Normal Hypocrite Sad So sad So sad

3 1 0 0 0

Table 4 Doris expressing different degrees of anger. Input phrase Hi What is your favorite song? You are so funny Nucklehead You are clumsy You are fool You are fool

Output phrase

Joy

Fear

Approval

Expression

Total

Hi Do not stop me now by queen Thank you Have you lost your mind? Why do you say that? Please, do not say that again Please, do not repeat yourself so much

50 60 72 92 87 74 68 57

50 55 62 72 75 72 66 60

50 50 50 55 45 35 25 14

Normal Normal Bit happy Happy Less happy Less happy Bit sad Bit angry

2 3 3 3 3 2 2 1

full video, named Emotion System Test: Setting Doris’ Sadness level, where the robot is performing a set of text inputs provided by a user can be found at the group’s webpage gallery Intelligent Control Group UPM-CSIC - Gallery.

26 27 28 29

7.3. Performing a tour

30 31

A program for touring the CAR premises was developed, by integrating the submodules into an execution file. The program is shown in Listing 7. It consists of going to the entrance of the laboratory where the robot starts to speak to the visitors. Then the program commands the robot to move to the first location where it will give its first explanation. As stated, the explanation is based on points of interest, which are previously defined and when the robot arrives at that point it will execute the following command. In this specific case the command to provide an explanation is the say instruction.

32

33 34 35

36

Listing 7: Program that performs a tour of the Centre for Automation and Robotics

37 38

1

var f i n i s h e d T o u r = #0

2 3 4

onRfidTagsDetected ( rfidEvent ) onSectorChanged ( newSectorEvent )

39

5 6 7 8 9 10

11 12 13 14 15

40

function r f i d E v e n t ( t a g s L i s t ) var i = #0 while ( i < s i z e o f ( t a g s L i s t ) ) i f ( t a g s L i s t [ i ] == " e280−1160−6000−0207−28b5−2a04 " ) say ( " That b i g box t h a t i s p l a c e d r i g h t t h e r e a t the bottom , i t i s where I am s t o r e d t o go out the u n i v e r s i t y " ) say [ f a c e=a f r a i d ] ( " That t h i n g s c a r e s me a l o t " ) endif i = i + #1 endw endfunction

41

42

43

44 45

16 17 18 19 20

21 22 23 24

function e x p l a i n M o i l e R o b o t s ( ) goto (#−1, #1.6 , #2.27) turn (#0) say [ f a c e=happy ] ( " I f you look a t the top o f t h a t l i g h t brown c l o s e t , you can f i n d VAD 1 , which was the f i r s t mobile r o b o t a t the l a b o r a t o r y " ) goto (#−1, #2.4 , #2.27) turn (#90) say [ f a c e=sad ] ( " Then we have here our l i t t l e f i s h Nemo . Some o f you may have seen the movie F i n d i n g Nemo . When he got l o s t he a r r i v e d here and became a t h i s l i t t l e r o b o t t o be used f o r exploration " ) goto (#−1, #4.8 , #2.27) turn (#90) say [ f a c e=s u r p r i s e ] ( " R i g h t t h e r e i s another mobile r o b o t c a l l e d Blacky . But as you can observe , i t does not have a gorgeous f a c e l i k e mine and people did not know how t o i n t e r a c t with i t " ) goto (#−1, #6.6 , #2.27) turn (#90) say [ f a c e=s u r p r i s e ] ( " And f i n a l l y , here i s my p r e d e c e s o r Urbano . He was t o u r i n g around museums performing almost the same work l i k e me, but he has turned old the poor guy . " ) turn (#180) say [ f a c e=s u r p r i s e ] ( " There i s a p o s t e r near Blacky and can be observed t h a t he and Urbano have a r r i v e d t o the V a l e n c i a S c i e n c e Museum " ) say ( " as I am a g i r l o f her house , I s t a y r i g h t here . " ) say [ f a c e=happy ] ( " Well people , There i s nothing l e f t t o add b e s i d e s s a y i n g t h a t behind me i s the desk o f my maker and where he has given me l i f e , l i k e Frankenstein to his creation . " ) say [ f a c e=happy ] ( " Thank you f o r e v e r y t h i n g and have a n i c e day . " ) endfunction

46

function newSectorEvent ( s e c t o r I d ) say ( "New s e c t o r achieved " ) i f ( s e c t o r I d == #0) say [ f a c e=happy ] ( "Now we a r e a t the p l a c e where I have been b u i l t . T h i s i s the Mobile Robots Room" ) explainMoileRobots ( ) f i n i s h e d T o u r = #1 endif endfunction

47 48

function main ( ) goto (#−1, #0.9 , #11.50) endfunction

A sample video, named Full System Test: Touring in the Laboratory, in which the robot performs the above-mentioned tour, is posted on the group’s webpage gallery Intelligent Control Group UPM-CSIC Gallery. Another tour was performed in the same laboratory, but this time the robot was explaining the lab to young visitors between 20 and 35 years old. They were asked to give their opinions on Doris, on the

25

15

B.P.E. Alvarado Vásquez and F. Matía

Engineering Applications of Artificial Intelligence 88 (2020) 103356

• Text To Speech: update the Emic2 card for another with more capabilities. • Emotion system: add more variables to the emotion system, facial recognition, previous interaction with people, and other elements that will equip Doris with the means to give the impression of intelligent communication in real life scenarios.

explanations she gave and her appearance. These are some of their replies: • Person 1 (22 years old): ‘‘it is quite funny how she explains things, because she has a robotic voice, but she could be a little faster. As for her looks, the face could be fixed a bit by adding some hair, but I definitely prefer Doris, because with a real person I’ll just fall sleep’’. • Person 2 (35 years old): ‘‘She is a bit slow, she could go a little bit faster, but everything else is awesome’’. • Person 3 (27 years old): ‘‘I was more excited by watching her speak and move around than by what she was saying exactly’’.

Acknowledgments The results in this paper report the research supported by the Spanish Ministry of Science, Innovation and Universities, Spain under DPI2017-86915-C3-3-R: Técnicas de Inteligencia Artificial y Ayuda a la Navegación Autónoma, and by Programas de Actividades I+D en la Comunidad de Madrid, cofunded with EU Structural Funds, through the EURoboCity2030-IV-CM project (Robótica aplicada a la mejora de la calidad de vida de los ciudadanos. fase IV; S2018/NMT-4331). We would also like to acknowledge the help of both Ángel Luis Martínez, the engineer responsible for designing the face and for the hardware installation on Doris, and Gema FB Martín, PhD in Psychology, for collaboration and guidance with the development of the emotional response system for Doris.

As can be seen from the previous statements, Doris was well accepted by the visitors, considering the speed with which she moves from one point to another, and her robotic appearance. The age range of the audience is an encouraging factor, in so far as they all fully accepted the interaction with Doris. The views of 40-year old adult, whose expectations are that the robot will tell the truth and interact, are not the same as those of a 20-year-old adult, whose concerns were related to its appearance. 8. Conclusions

References Doris, an interactive robot for guided tours around museums and laboratories, has been developed. Several modules have been integrated into a single architecture to achieve a degree of human–machine interaction. The use of the navigation previously developed using the EKF has been integrated with sensor fusion, although it needed some adjustment when interacting with people, in order to improve performance. The fuzzy sets of the speed and hallway controllers were modified, to avoid conflicts when the robot was surrounded by people, and the integration of the RFID system assisted sector identification using the semantic map, which optimized localization and navigation. Identification tags were placed on doors at a height of 1.0 m for sector identification and the RFID system was calibrated with the power and the sensitivity of the antennae, in order to detect the tags only when the robot was just passing through the door. An emotion system has been also developed, in order to achieve a more realistic interaction. The system modifies the face and the voice and provides an output message to the visitor according to the emotion at each instant. The use of a fuzzy engine for an emotional response in the face and in the voice was very useful to achieve a gradual evolution of each emotional input. Other emotions will be taken into account for future developments where the sensor inputs may affect the range of emotions that are expressed, or if the person is unknown, among other situations. A new programming language has been proposed for task planning, so that the interactive robot can work in laboratories and museums. This programming language integrates different components of the robot such as sensors, facial expressions and voice interaction, emotional responses, navigation, etc. The new language is a mix between C, Pascal, and JavaScript, which can be written in a simpler terms than other languages developed for robot task planning. The language also integrates the face and voice modules that have to be synchronized with the lip movements and the navigation of the robot when arriving at a specific place. Many tasks remain for future work on each module:

Alvarado Vásquez, B.P.E., Gonzalez, R., Matia, F., de la Puente, P., 2018a. Sensor fusion for tour-guide robot localization. IEEE Access 6, 78947–78964. Alvarado Vásquez, B.P.E., Matia, F., 2019. A social robot empowered with a new programming language and its performance in a laboratory. In: IEEE International Symposium on INnovations in Intelligent Systems and Applications, INISTA. Alvarado Vásquez, B.P.E., Matia, F., Galan, R., 2018b. Improving indoor robots localisation by fusing different sensors. In: IEEE International Conference on Intelligent Robots and Systems, IROS’18, pp. 2616–2623. Álvarez, M., Galán, R., Matía, F., Rodríguez-Losada, D., Jiménez, A., 2010. An emotional model for a guide robot. IEEE Trans. Syst. Sci. Cybern. 40 (5), 982–992. Beruvides, G., Quiza, R., Del Toro, R., Haber, R., 2013. Sensoring systems and signal analysis to monitor tool wear in microdrilling operations on a sintered tungstencopper composite material. Sensors Actuators A 199, 165–175. http://dx.doi.org/ 10.1016/j.sna.2013.05.021, cited By 20. URL https://www.scopus.com/inward/ record.uri?eid=2-s2.0-84879593202&doi=10.1016. Beruvides, G., Quiza, R., Haber, R., 2016. Multi-objective optimization based on an improved cross-entropy method. A case study of a micro-scale manufacturing process. Inform. Sci. 334–335, 161–173. http://dx.doi.org/10.1016/j.ins.2015.11.040, cited By 15. URL https://www.scopus.com/inward/record.uri?eid=2-s2.0-84959378486& doi=10.1016. Binney, J., Sukhatme, G.S., 2012. Branch and bound for informative path planning. In: IEEE International Conference on Robotics and Automation, pp. 2147–2154. Boboc, R.G., Horaiu, M., Talabă, D., 2014. An educational humanoid laboratory tour guide robot. Proced. Soc. Behav. Sci. 141, 424–430. Borody, W., 2013. The Japanese roboticist Masahiro Mori’s buddhist inspired concept of ‘‘The Uncanny Valley’’. J. Evol. Technol. 23 (1), 31–44. Breazeal, C., 2003. Toward sociable robots. Robot. Auton. Syst. 42. Burgard, W., Cremers, A.B., Fox, D., Hähnel, D., Lakemeyer, G., Schulz, D., Steiner, W., Thrun, S., 1999. Experiences with an interactive museum tour-guide robot. Artif. Intell. 114 (1–2), 3–55. Burgard, W., Cremers, A.B., Fox, D., Hanel, D., Lakemeyer, G., Schulz, D., Steiner, W., Thrun, S., 1998. The interactive museum tour-guide robot. In: AAAI ’98/IAAI ’98 Proceedings of the Fifteenth National/Tenth Conference on Artificial Intelligence/Innovative Applications of Artificial Intelligence, pp. 11–18. Castao, F., Beruvides, G., Haber, R., Artuedo, A., 2017. Obstacle recognition based on machine learning for on-chip lidar sensors in a cyber-physical system. Sensors (Switzerland) 17 (9), http://dx.doi.org/10.3390/s17092109, cited By 14. URL https://www.scopus.com/inward/record.uri?eid=2-s2.0-85029661618& doi=10.3390. del-Solar, J.R., 2007. Personal robots as ubiquitous-multimedial-mobile web interfaces. In: Latin American Web Conference, Vol. 120–127. Diprose, J.P., 2011. End user robot programming via visual languages. In: IEEE Symposium on Visual Languages and Human-Centric Computing: Graduate Consortium, pp. 229–230. Fong, T., Nourbakhsh, I., 2003. Socially interactive robots. Robot. Auton. Syst. Fong, T., Nourbakhsh, I., Dautenhahn, K., 2002. A Survey of Socially Interactive Robots: Concepts, Design, and Applications. Tech. Rep., Robotics Institute, Carnegie Mellon University. Giacomo, G.D., Lespérance, Y., Levesque, H.J., 2000. ConGolog, a concurrent programming language based on the situation calculus. Artif. Intell. 121 (1–2), 109–169.

• Localization: add more sensors, in order to create a network, so the robot can be prepared to localize itself in any environment where visual or reflective landmarks cannot be placed. • Mapping: at present, the maps are built by the user with the help of a text editor. An application would be useful to help build the maps or the robot could be programmed to build them using artificial vision. 16

B.P.E. Alvarado Vásquez and F. Matía

Engineering Applications of Artificial Intelligence 88 (2020) 103356 Olson, C.F., 2000. Probabilistic self-localization for mobile robots. IEEE Trans. Robot. Autom. 16 (1), 55–66. Pinciroli, C., Beltrame, G., 2016. A programming language for robot swarms. IEEE Softw. 33 (4), 97–100. Rada-Vilela, J., 2018. The Fuzzylite libraries for Fuzzy logic control. URL https:// fuzzylite.com/downloads/. Ramey, A., Gorostiza, J.F., Salichs, M.A., 2012. A social robot as an aloud reader: Putting together recognition and synthesis of voice and gestures for HRI experimentation. In: 7th ACM/IEEE International Conference on Human-Robot Interaction, HRI, pp. 213–214. Roditty, L., Zwick, U., 2011. On dynamic shortest paths problems. Algorithmica 61 (2), 389–401. Sanchez-Escribano, M.G., 2018. Engineering Computational Emotion. A Reference Model for Emotion in Artificial Systems, first ed. Springer. Sanchez-Escribano, M.G., Sanz, R., 2014. Emotions and the engineering of adaptiveness in complex systems. Proced. Comput. Sci. 28, 473–480. Schulz, D., Burgard, W., Fox, D., Thrun, S., Cremers, A.B., 2011. Robots as web services: Reproducible experimentation and application development using rosjs. In: IEEE International Conference on Robotics and Automation, pp. 6078–6083. Stenmark, M., Nugues, P., 2013. Natural language programming of industrial robots. In: IEEE ISR, pp. 1–5. Stentz, A., 1994. Optimal and efficient path planning for partially-known environments. In: Proceedings IEEE International Conference on Robotics and Automation, Vol. 4, pp. 3310–3317. Stricker, R., Muller, S., Einhor, E., Schroter, C., Volkhardt, M., Debes, K., Gross, H.-M., 2012. Konrad and Suse, two robots guiding visitors in a university building. In: Autonomous Mobile Systems. AMS 2012, pp. 49–58. Sudhakara, P., Ganapathy, V., 2016. Trajectory planning of a mobile robot using enhanced A-star algorithm. Indian J. Sci. Technol. 9 (41). Suzuki, S., Yuta, S., 1991. Analysis and description of sensor based behavior program of autonomous robot using action mode representation and Analysis and description of sensor based behavior program of autonomous robot using action mode representation and ROBOL/0 language. In: IEEE/RSJ International Workshop on Intelligent, Vol. 3, pp. 1497–1502. Tekerek, M., 2009. A human robot interaction application for robotic education. Proced. Soc. Behav. Sci. 1 (1), 2164–2169. Thrun, S., Beetz, M., Bennewitz, M., Burgard, W., Cremers, A., Dellaert, F., Fox, D., Hahnel, D., Rosenberg, C., Roy, N., Schulte, J., Schulz, D., 2000. Probabilistic algorithms and the interactive museum tour-guide minerva. J. Robot. Res. 19 (11), 972–999. Vogiatzis, D., Spyropoulos, C.D., Konstantopoulos, S., Karkaletsis, V., Kasap, Z., Matheson, C., Deroo, O., 2008. An affective robot guide to museums. In: 4th International Workshop on Human-Computer Conversation. Wilcock, G., Laxtröm, N., leinonen, J., Smit, P., Kurimo, M., Jokinen, K., 2016. Towards samitalk: A sami-speaking robot linked to sami wikipedia. In: 7th International Workshop on Spoken Dialog Systems, pp. 343–351. Yang, S., Mao, X., Ge, B., Yang, S., 2015. The roadmap and challenges of robot programming languages. In: IEEE International Conference on Systems, Man, and Cybernetics, pp. 328–333.

Gudmundsson, J., Levcopoulos, C., Narasimhan, G., 2002. Fast greedy algorithm for constructing sparse geometric spanners. SIAM J. Comput. 31 (5), 1479–1500. ul Haque, Q.S.M.Z., Wang, Z., Jadoon, N.R., 2007. Investigating the Uncanny Valley and human desires for interactive robots. In: IEEE International Conference on Robotics and Biomimetics, ROBIO, pp. 2228–2233. Hart, P., Nilsson, N., Raphael, B., 1968. A formal basis for the heuristic determination of minimum cost paths. IEEE Trans. Syst. Sci. Cybern. 4 (2), 100–107. Ibaraki, T., 1978. Branch-and-bound procedure and state - space representation of combinatorial optimization problems. Inf. Control 36 (1), 1–27. Jan, G.E., Chang, K.Y., Parberry, I., 2008. Optimal path planning for mobile robot navigation. IEEE/ASME Trans. Mechatronics 13 (4), 451–460. Kiesler, S., 2005. Fostering common ground in human-robot interaction. In: Robot and Human Interactive Communication. pp. 729–734. Kim, H., Kwak, S.S., Kim, M., 2008. Personality design of sociable robots by control of gesture design factors. In: 17th IEEE International Symposium on Robot and Human Interactive Communication, pp. 494–499. Koenig, S., Likhachev, M., 2005. Fast replanning for navigation in unknown terrain. IEEE Trans. Robot. 21 (3), 354–363. La Fe-Perdomo, I., Beruvides, G., Quiza, R., Haber, R., Rivas, M., 2019. Automatic selection of optimal parameters based on simple soft-computing methods: A case study of micromilling processes. IEEE Trans. Ind. Inf. 15 (2), 800–811. http:// dx.doi.org/10.1109/TII.2018.2816971, cited By 11. URL https://www.scopus.com/ inward/record.uri?eid=2-s2.0-85044368783&doi=10.1109. Lapham, J., 1999. RobotScript™: the introduction of a universal robot programming language. Ind. Robot: Int. J. 26 (1), 17–25. lai Lee, S., Kiesler, S., man Lau, I.Y., Chiu, C.-Y., 2005. Human mental models of humanoid robots. In: IEEE International Conference on Robotics and Automation, pp. 2767–2772. Leite, I., Pereira, A., Martinho, C., Paiva, A., 2008. Are emotional robots more fun to play with? In: IEEE International Symposium on Robot and Human Interactive Communication, pp. 77–82. Levesque, H., Reiter, R., Lespérance, Y., Lin, F., Scherl, R., 1997. GOLOG: A logic programming language for dynamic domains. J. Log. Program. 31 (1–3), 59–83. Lutovac, M., Ferenc, G., JelenaVidaković, Dimić, Z., Kvrgić, V., 2012. Usage of XML and P code for Robot Motion Control. In: Mediterranean Conference on Embedded Computing, pp. 162–165. Matsusaka, Y., Tojo, T., Kubota, S., Furukawa, K., Tamiya, D., Hayata, K., Nakano, Y., Kobayashi, T., 1999. Multi-person conversation via multi-modal interface. a robot who communicate with multi-user. In: Sixth European Conference on Speech Communication and Technology, EUROSPEECH. Mori, M., MacDorman, K.F., Minato, T., 1970. Bukimi no tani [The uncanny Valley]. Energy 7 (4), 33–35. Narendra, P., Fukunaga, K., 1977. A branch and bound algorithm for feature subset selection. IEEE Trans. Comput. C-26 (9), 917–922. Nau, D.S., Kumar, V., Kanal, L., 1984. General branch and bound, and its relation to a* and ao*. Artif. Intell. 23 (1), 29–58. Nishi, T., Ando, M., Konishi, M., 2005. Distributed route planning for multiple mobile robots using an augmented Lagrangian decomposition and coordination technique. IEEE Trans. Robot. 21 (6), 1191–1200.

17