- Email: [email protected]
Multisensor based security robot system for intelligent building
Robotics and Autonomous Systems 57 (2009) 330–338
Contents lists available at ScienceDirect
Robotics and Autonomous Systems journal homepage: www.elsevier.com/locate/robot
Multisensor based security robot system for intelligent building Ren C. Luo a,∗,1 , Tung Y. Lin b , Kuo L. Su c a
Intelligent Robotic & Automation Laboratory, Department of Electrical Engineering, National Taiwan University, 1, Sec. 4, Roosevelt Road, Taipei 106, Taiwan, ROC
b
Intelligent Automation Laboratory, Department of Electrical Engineering, National Chung Cheng University, 168 University Road, Ming-Hsiung, Chia-Yi, 621, Taiwan, ROC
c
Department of Electrical Engineering, National Yunlin University of Science & Technology, 123, Sec. 3, University Road, Touliu, Yunlin, 640, Taiwan, ROC
article
info
Article history: Available online 24 December 2008 Keywords: Intelligent building Intelligent security robot (ISR) Multisensor fusion algorithms Adaptive fusion method General user interface (GUI) GSM modern
a b s t r a c t Intelligent building can provide safety, convenience, efficiency and entertainment for life in the 21st century. The most importance role of the intelligent building is the security system. We develop a multi sensor-based intelligent security robot (ISR) that is widely employed in intelligent buildings. The intelligent security robot can detect abnormal and dangerous situations and notify users. The robot has the shape of cylinder and its diameter, height, and weight are 50 cm, 130 cm and 100 kg respectively. The function of the ISR contains six parts. There is the software development system; avoiding obstacle and motion planning system, image system, sensor system, remote supervise system and other systems. We develop a multi sensor-based sensor system in the ISR. We use multiple multisensor fusion algorithms to get an exact decision in the detection subsystem of the sensor system. There is an adaptive fusion method, a rule based method, and a statistical signal method. We demonstrate the remote supervisory system to control the ISR using a direct control mode and a behavior control mode. We think that the man–machine interface in a security robot system must have mobility and convenience. Therefore, we use a touch screen to display the system state, and design a general user interface (GUI) to service the user and visitors. The user can remotely control the appliance using a cell phone through a GSM modem, too. The appliance module can feedback reaction results to the user through a cell phone. Finally, we implement the fire detection system in the intelligent security robot (Chung-Cheng-I). If a fire occurs, the intelligent security robot can find out the fire source using the fire detection system. In intruder detection, we program the same scenario to detect the intruder using the intelligent security robot. The intelligent security robot transmits the message of the detection result to the user using a GSM modem for a fire event or intruder, and transmits the detection result to a client computer through the internet. © 2009 Published by Elsevier B.V.
1. Introduction With robotic technology development with each passing day, robot systems have been widely employed in many applications. Nowadays, robot systems have been applied in factory automation, dangerous environments, hospitals, surgery, entertainment, space exploration, farmland, military, security systems, and so on. Recently, more and more research takes an interest in robots which can help people in our daily life, such as service robots, office robots, security robots, and so on. We believe that robots will play an important role in our daily life in the future, especially security robots. The security of our home, building, laboratory or factory is important. The security system can identify hazards to
∗
Corresponding author. Tel.: +886 2 33669824; fax: +886 2 33669823. E-mail addresses: [email protected] (R.C. Luo), [email protected] (K.L. Su). URL: http://www.ntu.edu.tw (R.C. Luo).
1 Fellow, IEEE. 0921-8890/$ – see front matter © 2009 Published by Elsevier B.V. doi:10.1016/j.robot.2008.10.025
alarm and protect humans, and can detect intruders, fire, gas, and environment status. Meanwhile a greater variety of devices can be installed, such as service robots. In general, the security devices are relatively fixed and passive; the security robot is an active system. The security robot is more flexible than security devices. Fundamentally, the developed security robot has the following functions to perform such a security service: autonomous navigation, master-slave operated system, supervision through the Internet, a remotely operated camera vision system and danger detection and diagnosis system [1–6]. Recently, the wireless Internet technology is becoming more and more important. In the past literature, many experts have done research in security robots. Some research addressed in developing a target-tracking system of security robots [7,8], such as Hisato Kobayashi et al. proposed a method to detect human beings by an autonomous mobile guard robot [9]. Yoichi Shimosasa et al. developed an Autonomous Guard Robot [10] which integrates the security and service system to an autonomous guard robot; the robot can guide visitors in daytime and patrol at night.
R.C. Luo et al. / Robotics and Autonomous Systems 57 (2009) 330–338
331
Fig. 1. Architecture of intelligent building security system.
D.A. Ciccimaro developed an autonomous security robot—‘‘ROBART III’’ which is equipped with a non-lethal-response weapon [11,12]. Moreover, some research addressed in the robot has the capability of fire fighting [13]. There are some products that have been published for security robots. Such as SECON and SOC in Japanese and International Robotics in USA. The rest of the paper is organized as follows: Section 2 describes the system architecture of intelligent building and the intelligent security robot. Section 3 explains the system function of the intelligent security robot. Section 4 presents the detection method of fire detection and intruder detection, and environment detection for intelligent security robots. The experimental results of fire detection and intruder detection using the intelligent security robot are implemented in Section 5. Section 6 presents brief concluding comments.
Fig. 2. The contour and structure of intelligent security robot.
2. System architecture The system architecture of the intelligent security system is shown in Fig. 1. The system contains intelligent a security robot (ISR), remote supervisory computer, security module and appliance control module. The ISR contains the main controller (IPC), security detection, RF interface and GSM modem. The ISR can receive the status of the security module and appliance control module using a wireless RF interface. In the security module, it uses one-way communication with the security robot and cellular phone. But the appliance control module uses twoway communication with ISR and cellular phone. The ISR can communicate with GSM modem using a RS232 interface. The GSM modular (WMOD2) was made by Wavecom. The modular is a self-contained E-GSM900/GSM1800 (or E-GSM900/GSM1900) dual band module. The RF interface of the ISR can get information from the security module. The remote supervisory computer can interact with the ISR through the Internet. The Web server is usually set up in the local site in order to reduce the internet time delay. Therefore, the supervisory computer and remote supervisory computer communicate in the same local area network. The user can connect to the supervisory computer to get security information of the home or building, and control the mobile robot to patrol everywhere in the home or building, and supervise the appliance control module through the internet. The user can acquire the security information and supervise the appliance control module using a cellular phone through GSM. The security robot is constructed using an aluminum frame. The contour of the robot is cylindrical. The security robot contains an upper body and a lower body. The upper body of the security robot contains the main controller (Industrial Standard PC with a Pentium-III 933 CPU and 256M RAM), touch screen, CCD, sensors and sensory circuits and some hardware devices. The lower body contains the drive system, batteries and two DC servomotors.
Fig. 3. The hardware structure of the intelligent security robot.
There are six systems in the security robot, including a sensor system, remote supervisory system, software development system, an image system, obstacle avoidance and motion planning system and other systems. Fig. 2 shows the function of the intelligent security robot. Fig. 3 is the hierarchy structure of the security robot, and each system includes some subsystem. Each system has some function for implementation. For example, the other system contains auto-dialling, alarm, auto recharge, power schedule and structure. Every function can be finished by a dividable device. And it transmits the experimental results to the main controller. The hardware structure of the security robot is shown in Fig. 3. There are many devices to be designed by ourselves. Such as an auto-recharging switch, ultrasonic driver, fire detection. . . and so on. In the multifunction I/O card, it can acquire the sensor information and control driver status. In the motion control card, it can control the motion of the ISR, and acquire the IR sensor information. The main controller (Industry Personal Computer, IPC) can acquire the distance of obstacles by an ultrasonic driver using a series interface. The other device has a CCD, touch screen, wireless LAN, alarm device and GSM modem. In the autorecharging function and power detection, we design the autorecharging station for the intelligent security robot. The ‘‘Battery Detection’’ block contains recharging and discharging current measurement values, and measures the battery voltage value. The ‘‘Auto Recharging Switch’’ block contains a driver circuit and relay
332
R.C. Luo et al. / Robotics and Autonomous Systems 57 (2009) 330–338
Fig. 4. The sensory system architecture.
element, and controls the recharging current to prevent overload. It can turn on and cut off the recharging current. 3. System function The hierarchy structure of sensory system for the ISR is shown in Fig. 4. The sensory system has six variety subsystems. These are: fire detection subsystem, intruder detection subsystem, power detection subsystem, environment detection subsystem, motor control subsystem and obstacle detection subsystem. These subsystems can acquire sensory signals and processes these signals using an amplifier and calibration circuits, and transmits sensory data to the IPC using interface devices. In the fire detection subsystem and intruder detection subsystem and motor control subsystem, we use a digital input/output interface card to transmit sensory data. In the power detection subsystem and environment subsystem, we use an analogue input interface card to acquire measured values. We use sixteen ultrasonic sensors and eight IR sensors to detect obstacles in the obstacle detection subsystem. Then we use a microprocessor (MCS-5A series) to drive the ultrasonic sensors and get a distance value from obstacles and transmit the distance value to the IPC using a series interface (RS232). These sensors are listed in Table 1. The arrangement of intruder detection sensors and other hardware are shown in Fig. 5. In the fire detection sensors, we combine a smoke sensor, flame sensor and temperature sensor in one module. The module is fixed on the front side of the intelligent security robot. In the intruder sensors, we use three body sensors, sixteen ultrasonic sensors and eight IR sensors arranged outside of the intelligent security robot. The intelligent security robot can transmit messages to the user using GSM modem. The relationship of the GSM modem and the intelligent security robot is shown in Fig. 6. The intelligent security robot communicates with a cell phone using GSM (Global System for Mobile) module. The Remote Control and Monitor Interface are shown in Fig. 7; the interface includes four sub-windows: the top window is for environment detection; the top-left window shows the robot’s vision system. The bottom-left window shows the global map of the remote environment and the trajectory of mobile robot from dead-reckoning measurement; the bottom-right window shows the several control modes for the robot’s motion system. The top sub-window shows the environmental detection status from the mobile robot. The environmental sensors include flame sensors, smoke sensors, temperature sensors, body sensors, humidity sensors, gas sensors, noise sensor and luminance sensors. When a dangerous situation occurs, the security robot server sounds the alarm and notifies the remote site through the internet. The finished 3-in-1 fire detection sensor that we designed is shown in Fig. 8. The left image shows the original hardware, and the right image shows that we encapsulate the sensors with a shell to match the outside of ISR. It is integrated with a flame sensor, temperature sensor and smoke sensor to detect
Fig. 5. The sensor arrangement.
Fig. 6. The relation of GSM modem and main controller of the security robot.
Fig. 7. Remote control and monitor interface.
the flame temperature and smoke information. When fires occur, the most abnormal environmental changes are flames, increased temperature and smoke filling the air. So we employ a flame sensor, temperature sensor and smoke sensor to detect whether these three condition are happening or not. When the information of fire detection modules is collected, we use an adaptive multisensor fusion algorithm to judge whether a fire is occurring or not.
R.C. Luo et al. / Robotics and Autonomous Systems 57 (2009) 330–338
333
Table 1 Sensors in the security robot. Subsystem
Sensors
Examples
Fire detection subsystem
Temperature sensor Smoke sensor Flame sensor Body sensor Ultrasonic sensor IR sensor Light-operated Proximity sensor Ultrasonic sensor Gas sensor Humidity sensor Lux meter sensor Voice sensor Current sensor
AD590 TG135 R2686 Body Polaroid 6500 SMC-10R CDD-40N Polaroid 6500 TGS 822 C2 – M3 S1133 Microphone MEL-55P
Intruder detection subsystem
Obstacle detection subsystem Environment detection subsystem
Power detection subsystem
Fig. 8. 3-in-1 fire detection sensor.
In the hierarchy of the remote supervisory system, it contains the communication protocol, data base, Internet, user interface and touch screen. The hardware configuration of the remote surveillance and control system includes a robot server (main controller of the security robot, i.e. IPC) to be embedded in the intelligent security robot, a web server workstation, client computers (PC or PDA) and sensors, as shown in Fig. 9. The main controller of the ISR can get sensory data to detect fire, intruders, gas, etc. It can also control the color CCD, motors, etc. The main controller of the ISR can interact with the Web server. In order to reduce the Internet time delay, the Web server is usually set up at the local site. Therefore, the main controller of the ISR and the Web sever communicate in the same local area network. The web server can get the sensory data (sensor status, commands, robot status, environment conditions, etc) from the robot server through the Internet. The web server can also receive the client’s commands via the Internet and then transmit it to the robot server. The remote user can connect to the Web server or the robot server to get all information by computer or PDA through the Internet. There are three control modes in the GUI implemented with Visual C++, direct control mode, supervisory control mode and job scheduling mode. In the direct control mode, the dialog includes the control buttons such as go forward, go backward, turn left, turn right and stop. The control buttons relative to controlling the camera include up, down, left, right, zoom in and zoom out. The user can not only click the buttons by using a mouse but also push the button arrow up, arrow left, arrow right and arrow down on a keyboard. Furthermore, the user can push the button in the toolbar to select the current velocity of the robot’s motion (Fig. 10). In the supervisory control mode, if the user decides a goal position, the robot generates a moving path and moves toward the goal position autonomously. After the goal position is determined, the shortest moving path is generated by using the ‘‘via’’ points which were already given in the map. The ‘‘via’’ points can be considered as all the points in the path through which the robot passes.
Fig. 9. The hardware configuration of the remote surveillance and control architecture.
Fig. 10. The user can control the motion of a robot with different velocities.
334
R.C. Luo et al. / Robotics and Autonomous Systems 57 (2009) 330–338
Fig. 13. The detection and diagnostic rule of the intruder system. Fig. 11. Architecture of wireless appliance control system.
4. Detection method There are many algorithms applied in the sensor system. In the fire detection subsystem, we use an adaptive multisensor fusion algorithm. In the intruder detection subsystem, we use a rulebased method and combine body sensors IR sensors and ultrasonic sensors to implement. In the environment detection subsystem, we use a statistical signal method to decide the exact measurement value, and it contains humidity detection, illumination detection, temperature detection and gas detection. The adaptive decision fusion method was proposed by Ansari [14–16]. The parameters and details of the equation are discussed in another paper [17–19] and the reinforcement updating rules are:
∆ω ˆi =
Fig. 12. Architecture of wireless security detection system.
The architecture of wireless appliance control system is shown in Fig. 11. The module can control appliances using relay elements, and transmits the action status to the mobile robot and security device using a wireless RF interface. The basic function of the wireless appliance remote control is shown below: with UHF emission circuit, it can perform wireless transmission and control. 1. With encoded IC, it will not be interfered by outside world noise. 2. System is composed of four-groups-key wireless emitter and 89C2051 receiver control board. 3. The work frequency of high-frequency carrier of wave wireless emitter and receiver are 315 MHz and 433 MHz. 4. ON/OFF of appliance can be controlled by a switch or relay. In the home security system, it can execute the same function to receive the security module information, and transmits the detection information to the user through a GSM modem and the Internet. But it connects with the remote supervisory computer using the Internet. The architecture of wireless security detection System is shown in Fig. 12.
∆ω ˆ0 =
1 m1i 1
m0i 1
if ui = +1
ˆ1 and H
if ui = −1
ˆ0 and H
ˆ0 eωˆ i +ωˆ 0 if ui = +1 and H − m1i − 1 eωˆ i −ωˆ 0 if ui = −1 and Hˆ 1 m0i 1 ˆ 1 occurs when H m
1 − eωˆ 0 m
(1)
(2)
ˆ 0 occurs when H
ωˆ i+ = ωˆ i− + ∆ωˆ i , i = 0, 1, 2, where ωˆ i+ and ωˆ i− represent the weight value after and before each update. In the intruder sensors, we use three body sensors, sixteen ultrasonic sensors and eight IR sensors arranged outside of the ISR. The detection and diagnosis rule is shown in Fig. 13. The detection rule of the intruder system has three stages. The high stage uses a body sensor to detect intruders. If the body sensor detect an intruder, the output state is ‘‘Yes’’, otherwise, the output state is ‘‘No’’. We use an ultrasonic sensor to detect intruders in the second stage. In the third stage, we use IR sensors to detect intruders. Three varieties of sensor can decide five varieties of result. They are intruder (near), intruder (beside), fail, obstacle and normal, and can diagnose which sensor will be in error. It can transmit messages to the client’s computer through the Internet. The intelligent security robot communicates with a cell phone using GSM (Global System for Mobile) modem. In the environment subsystem, we use approaches based on statistical signal detection methods. The subsystem contains a
R.C. Luo et al. / Robotics and Autonomous Systems 57 (2009) 330–338
335
humidity sensor, luxmeter sensor, temperature sensor, and gas sensor and voice sensor. We modeled the observed environment detection system as the sum of 3 signal components, to be shown in Eq. (3). For a faultfree device F (t ) = 0; conversely, when any faults are occurring, F (t ) is not zero. The observed environment detection system of the security robot is [20]: Z (t ) = W (t ) + n(t ) + F (t )
(3)
where: Z (t ) The matrix of parameter by the environment detection system as measured or observed, W (t ) The matrix of parameter by the environment detection system under ideal conditions, F (t ) Any signal change as a consequence of faults, n(t ): Any signal change as a consequence of thermal effects, sample error, etc. If we are able to calculate a w( ˆ t ), either through simulation or repeated observation of a no-fault device then we can get :
w( ˆ t ) = W (t ) + e(t )
Fig. 14. Screen of sensors I/O state and GSM control buttons.
(4)
where e(t ) is estimate error. We combine Eqs. (3) and (4), and get the following result. Z (t ) − w( ˆ t ) = n(t ) − e(t ) + F (t ).
(5)
With a no-fault device, we use a maximum likelihood estimator (MLE) method to apply in the environment detection system. If the signal is deterministic and the noise is Gaussian with zero mean, then the MLE value is adequate:
w ˆ MLE (t ) =
X N 1 N
Z (t ) .
(a) Flame signal.
(b) Detect flame source.
Fig. 15. Use a lighter to mimic a flame occurred.
(6)
1
We can use w ˆ MLE (t ) to replace w ˆ (t ) of Eq. (5), and then get: Z (t ) − w ˆ (t ) = n (t ) + F (t ) .
(7)
Again, if the signal is deterministic and the noise is Gaussian with zero mean an adequate detector for F (t ) is the LRT detector [21]:
λ = φ(n)0 Rn−1 φ(n) φ(n) = Z (n) − W (n).
(a) Gas signal.
(b) Detect gas signal.
Fig. 16. State of smoke sensor shows gas have been detected.
(8)
The λ is over the threshold. We can say the measurement value to be faulty. Otherwise the measurement value w ˆ MLE (t ) is exact. 5. Experimental results Fig. 14 is the interface that shows the I/O states of the intelligent security robot. ‘‘Block 1’’ shows the states of fire sensors, ‘‘Block 2’’ shows the power states of battery. The panel display 36 V power status for driver system, and displays the 12 V power status for IPC and sensory detection system. ‘‘Block 3’’ shows the states of body sensors, ‘‘Block 4’’ shows the buttons of the fire and intruder alarm switches, and ‘‘Block 5’’ shows the GSM control buttons and the state of GSM massage sending. We open the switch of fire alarm, so its state is ON; but the intelligent security robot does not detect any flame or smoke, all states of fire sensors are 0 (means no flame or smoke is detected). In Fig. 15(a), we use a lighter to simulate a flame when the ISR is on its way executing a patrol task. Fig. 15(b) shows that the switch of fire alarm has been opened. ISR will detect the flame and then the state of flame sensor will become 1 (means flame has been detected). At the same time the fire alarm will sound and send a message to the previously assigned number. Then we use a lighter to simulate a gas leak (Fig. 16(a)), and the state of smoke sensor
(a) Intruder detection.
(b) Detect a intruder.
Fig. 17. Mimic an intruder detect.
shows gas has been detected (Fig. 16(b)). Fig. 16 shows that we open the switch of intruder alarm, so its state is ON; but intelligent security robot does not detect any intruders, all states of body sensors are 0 (means no intruders are detected). In Fig. 17(a) mimic intruder when ISR is on its way executing patrol task. Fig. 17(b) shows that the switch of intruder alarm has been opened. ISR will detect the intruder and then the state of intruder sensor will become 1 (means an intruder has been detected). At the same time the intruder alarm will sound and send a message to the previously set number. We implement the proposed method applying in the intelligent security robot. We design the fire sensory team (smoke sensor,
336
R.C. Luo et al. / Robotics and Autonomous Systems 57 (2009) 330–338
(a) The robot scenario.
(b) The remote site.
Fig. 18. The robot detects fire event.
(a) The robot scenario.
(b) The remote site.
Fig. 19. The robot change orientation.
(a) The robot scenario.
(b) The remote site.
Fig. 20. The robot transmits the fire status to client.
flame sensor and temperature sensor) to be fixed on the front side of the security robot. The security robot can move in free space. When a fire event happens, the security robot can detect the fire source using the fire sensory team. It is shown in Fig. 18(a). The CCD that is fixed on the intelligent security robot transmits an image to the remote supervisory computer. The real image is shown in Fig. 18(b). The intelligent security robot can move its orientation to face the fire source, and transmits the fire source to the remote supervisory computer. The scenario is shown in Fig. 19. Then it moves to the fire source, and use the fire sensory team to decide the exact result using an adaptive fusion algorithm. If the fire event is true, the security robot must alarm and send a message to the remote supervise computer. It can display the fire status in the monitor. The green point represents the normal status, and the red point represents hazards. The security robot moves to the fire source to decide an exact detection using an adaptive fusion algorithm. The scenario is shown in Fig. 20. At the remote site, the monitor can display the path of the intelligent security robot in the fire detection processing.
(a) Find intruder.
In the other condition, the intelligent security robot can detect intruders using body sensors, ultrasonic sensors and IR sensors. Because the security robot will execute its task during night, the people they meet will all be thought of as intruders. The process of intruder detection is described below. First, the security robot uses body sensors to detect intruders, and is shown in Fig. 21(a). The detection distance is about 6 m. Next, the security robot use ultrasonic sensors and IR sensors to decide an exact result, and the scenario is shown in Fig. 21(b). The ultrasonics can measure the varying distance from the intruder (obstacle). If the intruder comes to the security robot, the IR sensor can detect the intruder, and the detection distance is 30 cm. Finally, the security can display the intruder status, and alarm to alert the intruder. The scenario is shown in Fig. 21(c). If the fire event is true, the ISR must alarm and sent the message to the user using the GSM module. The user can receive the message using a cell phone. The experimental result is shown in Fig. 22(a). The ISR can transmit the fire event to the client computer using Internet. The client computer can display the fire status in the monitor and is shown in Fig. 22(b). The ISR can move to a safe distance from the fire source using ultrasonic sensors and temperature sensor. That is to say, if the ISR moves to the fire source, the detection distance must be smaller using ultrasonic sensors or the temperature measured value must be higher using the temperature sensor. In the intruder detection, the security robot must alarm and transmit the detection results to the cell phone (shown in Fig. 23(a)) and the client computer (shown in Fig. 23(b)). In the appliance control module, the user can control the appliance using a cellular phone through a GSM modem, or use a remote supervisory computer to control the appliance through the Internet. In the Fig. 24(a), the user transmits the message to the ISR to turn on an electric fan. If the appliance control module receives a wireless RF signal from the ISR, it must turn on the electric fan, and transmit the action result to the ISR. The panel of the ISR can display the electric fan’s status. The ISR can feed the message of the lamp’s status back to the user’s cellular phone through a GSM modem. The experimental scenario is shown in Fig. 24(b). 6. Conclusion We have developed a multisensor based intelligent security robot that includes six systems. There is a software development system; obstacle avoidance and motion planning system, image system, sensor system, remote supervisory system and other systems. The intelligent security robot has been applied in an intelligent building. The user can control the ISR via the client’s terminal through the Internet. In the sensor system, we have designed an intelligent fire detection module which is applied to fire events. The three sensors (smoke sensor, flame sensor and temperature sensor) of the fire detection module use an adaptive fusion method to arrive at an adequate decision result. The ISR can use the proposed method to detect fire events and intruders, and transmits the message to a cell phone using a GSM modem, and
(b) Move to the intruder.
(C) Alarm to alert the intruder.
Fig. 21. The intruder detection scenario for the security robot.
R.C. Luo et al. / Robotics and Autonomous Systems 57 (2009) 330–338
(a) Fire condition.
(b) Client terminal.
Fig. 22. Cell phone and client terminal display fire message.
(a) Intruder condition.
(b) Client terminal.
Fig. 23. Cell phone and client terminal display intruder massage.
Fig. 24. The appliance control module experimental result. (a) The lamp on (b) The cellular phone and the LCD panel of the mobile robot can display ‘‘ON’’ status.
transmits the detection results to the client’s computer through the Internet. The security robot can detect intruders, and transmits the message to a cell phone, too. Further research is to investigate multiple security robots cooperating on the security system for an intelligent building. Acknowledgment This work was supported by the project, ‘‘The Development of Integrated Robotics System’’, under DOIT TDPA of Taiwan, ROC., 95EC-17-A-04-SI-054. References [1] A. Birk, H. Kenn, An industrial application of behavior-oriented robotics, in: IEEE International Conference on Robotics and Automation, ICRA, vol. 1, 2001, pp. 749–754. [2] M. Saitoh, Y. Takahashi, A. Sankaranarayanan, H. Ohmachi, K. Marukawa, A mobile robot testbed with manipulator for security guard application, in: IEEE International Conference on Robotics and Automation, vol. 3, 1995, pp. 2518–2523. [3] Y. Takahashi, I. Masuda, A visual interface for security robots, in: IEEE International Workshop on Robot and Human Communication, 1992, pp. 123–128. [4] M. Saitoh, Y. Takahashi, A. Sankaranarayanan, H. Ohmachi, K. Marukawa, A mobile robot testbed with manipulator for security guard application, in: Proceedings of the IEEE International Conference on Robotics and Automation, 1995, vol. 3, pp. 2518–2523.
337
[5] H. Hara, Development of ROBOT KOBAN, IEEE European Convention on Security and Detection (1995) 193–197. [6] R.C. Luo, C.C. Lin, S.H. Pan, K.L. Su, The internet-based security robot for the home automation system, in: The 13th Automation Technology Conference in Taipei, 6/17-6/18, 2004, pp. 954–959. [7] S.O. Lee, Y.J. Cho, M.H. Bo, B.J. You, S.R. Oh, A stable target-tracking control for unicycle mobile robots, in: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2000, vol. 3, 2000, pp. 1822–1827. [8] L.E. Parker, B.A. Emmons, Cooperative multi-robot observation of multiple moving targets, in: Proceedings of the IEEE International Conference on Robotics and Automation, vol. 3, 1997, pp. 2082–2089. [9] H. Kobayashi, M. Yanagida, Moving object detection by an autonomous guard robot, in: Proceedings of the 4th IEEE International Workshop on Robot and Human Communication, TOKYO, 1995, pp. 323–326. [10] Y. Shimosasa, J. Kanemoto, K. Hakamada, H. Horii, T. Ariki, Y. Sugawara, F. Kojio, A. Kimura, S. Yuta, Some results of the test operation of a security service system with autonomous guard robot, in: 26th Annual Conference of the IEEE on Industrial Electronics Society, IECON 2000, vol. 1, 2000, pp. 405–409. [11] G.A. Gilbreath, D.A. Ciccimaro, H.R. Everett, An advanced telereflexive tactical response robot, in: Proceedings, Workshop 7: Vehicle Teleoperation Interfaces, IEEE International Conference on Robotics and Automation, ICRA2000, San Francisco, CA, 28 April, 2000. [12] D.A. Ciccimaro, H.R. Everett, M.H. Bruch, C.B. Phillips, A supervised autonomous security response robot, in: American Nuclear Society 8th International Topical Meeting on Robotics and Remote Systems, ANS’99, Pittsburgh, April, 1999, pp. 25–29. [13] A. Bradshaw, The UK security and fire fighting advanced robot project, IEE Colloquium on Advanced Robotic Initiatives in the UK (1991) 1/1–1/4. [14] N. Ansari, J.G. Chen, Y.Z. Zhang, Adaptive decision fusion for unequiprobable sources, IEE Proceedings Radar, Sonar and Navigation (1997) 105–111. [15] N. Ansari, E.S.H. Hou, B.O. Zhu, J.G. Chen, Adaptive fusion by reinforcement learning for distributed detection systems, IEEE Transactions on Aerospace and Electronic Systems 32 (2) (1996) 524–531. [16] G. Chen, N. Ansari, Adaptive fusion of correlated local decision, IEEE Transaction on System, Man and Cybernetics—PART C: Application and Review 28 (2) (1998). [17] R.C. Luo, K.L. Su, K.H. Tsai, Intelligent security robot fire detection system using adaptive sensory fusion method, in: IEEE International Conference on Industrial Electronic, Control, and Instrumentation, IECON’2002, pp. 2663–2668. [18] R.C. Luo, K.L. Su, K.H. Tsai, Fire detection and isolation for intelligent building system using adaptive sensory fusion method, in: IEEE International Conference on Robotics and Automation, ICRA’2002, pp. 1777–1781. [19] R.C. Luo, S.Y. Lin, K.L. Su, A multiagent multisensor based security system for intelligent building, IEEE/SICE/RSJ, in: International Conference on Multisensor Fusion and Intelligent for Intelligent System, MFI03, pp. 311–316. [20] J.F. Frenzel, Power-supply current diagnosis of VLSI circuit, IEEE Transactions on Reliability 43 (1) (1994) 30–38. [21] R.J. Bibbero, Microprocessors in Instruments and Control, Jolin Wiley & Sons. Inc., 1997.
Ren C. Luo (M’83–SM’88–F’92) received the Ph.D.degree in electrical engineering from the Technische Universitaet Berlin, Berlin, Germany. He is currently a Distinguished Professor in the Department of Electrical Engineering at National Taiwan University in Taiwan. He was a Professor of Department of Electrical and Computer Engineering at North Carolina State University, Raleigh, NC, USA and Toshiba Chair Professor in the University of Tokyo, Japan. His research interests include sensor-based intelligent robotic systems, multisensor fusion and integration, computer vision, micro/nano technologies, rapid prototyping, and advanced manufacturing systems. He has authored more than 300 papers on these topics, which have been published in refereed technical journals and conference proceedings. He also holds several patents. Dr. Luo received IEEE Eugean Mittlemann Outstanding Research Achievement Award, 1996; ALCOA Foundation Outstanding Engineering Research Award, NCSU, USA; National Science Council Outstanding Research Awards, 1998–1999, 2000–2001, 2002–2005; National Science Council Distinguished Research Awards, 2006–2008; TECO Outstanding Science and Technology Research Achievement Award, 2001. Dr. Luo is currently Editor-in-Chief of IEEE/ASME Transactions On Mechatronics. He served as President of IEEE Industrial Electronics Society (2000–2001). He also served as President of Chinese Institute of Automation Engineers, Convener of Automation Technology Division, National Science Council; Adviser of Ministry of Economics Affairs and Technical Adviser of Prime Minister’s Office in Taiwan. He contributes regularly to IEEE sponsored international conferences by serving as conference General Chairs (IEEE IROS 1992, MFI 1994, IECON 1996, MFI 1999, ICRA 2003, IECON 2007, IROS 2010), Program Chairs, program committees, and offers short courses or tutorials and plenary/keynote speeches in various countries and research communities. Dr. Luo is a Fellow of IEEE since 1992 and a Fellow of IEE.
338
R.C. Luo et al. / Robotics and Autonomous Systems 57 (2009) 330–338 Tung Y. Lin received the B.S. and M.S. degrees in electrical engineering in 2004 and 2006 respectively at National Chung-Cheng University, Taiwan. He is currently a Ph.D. student in North Carolina State University, USA. His research interests include multisensor fusion and intelligent robotics.
Kuo L. Su received the B.S. and M.S. degrees in automatic control engineering from Feng-Chia University, Seatwen, Taiwan, ROC, and received the PH.D. degree in electrical engineering at National Chung-Cheng University, Chia-Yi, Taiwan, ROC, He is currently a teacher in the Department of Electrical Engineering, National Yunlin University of Science & Technology. His research interests include multisensor fusion and robotics.
Contents lists available at ScienceDirect
Robotics and Autonomous Systems journal homepage: www.elsevier.com/locate/robot
Multisensor based security robot system for intelligent building Ren C. Luo a,∗,1 , Tung Y. Lin b , Kuo L. Su c a
Intelligent Robotic & Automation Laboratory, Department of Electrical Engineering, National Taiwan University, 1, Sec. 4, Roosevelt Road, Taipei 106, Taiwan, ROC
b
Intelligent Automation Laboratory, Department of Electrical Engineering, National Chung Cheng University, 168 University Road, Ming-Hsiung, Chia-Yi, 621, Taiwan, ROC
c
Department of Electrical Engineering, National Yunlin University of Science & Technology, 123, Sec. 3, University Road, Touliu, Yunlin, 640, Taiwan, ROC
article
info
Article history: Available online 24 December 2008 Keywords: Intelligent building Intelligent security robot (ISR) Multisensor fusion algorithms Adaptive fusion method General user interface (GUI) GSM modern
a b s t r a c t Intelligent building can provide safety, convenience, efficiency and entertainment for life in the 21st century. The most importance role of the intelligent building is the security system. We develop a multi sensor-based intelligent security robot (ISR) that is widely employed in intelligent buildings. The intelligent security robot can detect abnormal and dangerous situations and notify users. The robot has the shape of cylinder and its diameter, height, and weight are 50 cm, 130 cm and 100 kg respectively. The function of the ISR contains six parts. There is the software development system; avoiding obstacle and motion planning system, image system, sensor system, remote supervise system and other systems. We develop a multi sensor-based sensor system in the ISR. We use multiple multisensor fusion algorithms to get an exact decision in the detection subsystem of the sensor system. There is an adaptive fusion method, a rule based method, and a statistical signal method. We demonstrate the remote supervisory system to control the ISR using a direct control mode and a behavior control mode. We think that the man–machine interface in a security robot system must have mobility and convenience. Therefore, we use a touch screen to display the system state, and design a general user interface (GUI) to service the user and visitors. The user can remotely control the appliance using a cell phone through a GSM modem, too. The appliance module can feedback reaction results to the user through a cell phone. Finally, we implement the fire detection system in the intelligent security robot (Chung-Cheng-I). If a fire occurs, the intelligent security robot can find out the fire source using the fire detection system. In intruder detection, we program the same scenario to detect the intruder using the intelligent security robot. The intelligent security robot transmits the message of the detection result to the user using a GSM modem for a fire event or intruder, and transmits the detection result to a client computer through the internet. © 2009 Published by Elsevier B.V.
1. Introduction With robotic technology development with each passing day, robot systems have been widely employed in many applications. Nowadays, robot systems have been applied in factory automation, dangerous environments, hospitals, surgery, entertainment, space exploration, farmland, military, security systems, and so on. Recently, more and more research takes an interest in robots which can help people in our daily life, such as service robots, office robots, security robots, and so on. We believe that robots will play an important role in our daily life in the future, especially security robots. The security of our home, building, laboratory or factory is important. The security system can identify hazards to
∗
Corresponding author. Tel.: +886 2 33669824; fax: +886 2 33669823. E-mail addresses: [email protected] (R.C. Luo), [email protected] (K.L. Su). URL: http://www.ntu.edu.tw (R.C. Luo).
1 Fellow, IEEE. 0921-8890/$ – see front matter © 2009 Published by Elsevier B.V. doi:10.1016/j.robot.2008.10.025
alarm and protect humans, and can detect intruders, fire, gas, and environment status. Meanwhile a greater variety of devices can be installed, such as service robots. In general, the security devices are relatively fixed and passive; the security robot is an active system. The security robot is more flexible than security devices. Fundamentally, the developed security robot has the following functions to perform such a security service: autonomous navigation, master-slave operated system, supervision through the Internet, a remotely operated camera vision system and danger detection and diagnosis system [1–6]. Recently, the wireless Internet technology is becoming more and more important. In the past literature, many experts have done research in security robots. Some research addressed in developing a target-tracking system of security robots [7,8], such as Hisato Kobayashi et al. proposed a method to detect human beings by an autonomous mobile guard robot [9]. Yoichi Shimosasa et al. developed an Autonomous Guard Robot [10] which integrates the security and service system to an autonomous guard robot; the robot can guide visitors in daytime and patrol at night.
R.C. Luo et al. / Robotics and Autonomous Systems 57 (2009) 330–338
331
Fig. 1. Architecture of intelligent building security system.
D.A. Ciccimaro developed an autonomous security robot—‘‘ROBART III’’ which is equipped with a non-lethal-response weapon [11,12]. Moreover, some research addressed in the robot has the capability of fire fighting [13]. There are some products that have been published for security robots. Such as SECON and SOC in Japanese and International Robotics in USA. The rest of the paper is organized as follows: Section 2 describes the system architecture of intelligent building and the intelligent security robot. Section 3 explains the system function of the intelligent security robot. Section 4 presents the detection method of fire detection and intruder detection, and environment detection for intelligent security robots. The experimental results of fire detection and intruder detection using the intelligent security robot are implemented in Section 5. Section 6 presents brief concluding comments.
Fig. 2. The contour and structure of intelligent security robot.
2. System architecture The system architecture of the intelligent security system is shown in Fig. 1. The system contains intelligent a security robot (ISR), remote supervisory computer, security module and appliance control module. The ISR contains the main controller (IPC), security detection, RF interface and GSM modem. The ISR can receive the status of the security module and appliance control module using a wireless RF interface. In the security module, it uses one-way communication with the security robot and cellular phone. But the appliance control module uses twoway communication with ISR and cellular phone. The ISR can communicate with GSM modem using a RS232 interface. The GSM modular (WMOD2) was made by Wavecom. The modular is a self-contained E-GSM900/GSM1800 (or E-GSM900/GSM1900) dual band module. The RF interface of the ISR can get information from the security module. The remote supervisory computer can interact with the ISR through the Internet. The Web server is usually set up in the local site in order to reduce the internet time delay. Therefore, the supervisory computer and remote supervisory computer communicate in the same local area network. The user can connect to the supervisory computer to get security information of the home or building, and control the mobile robot to patrol everywhere in the home or building, and supervise the appliance control module through the internet. The user can acquire the security information and supervise the appliance control module using a cellular phone through GSM. The security robot is constructed using an aluminum frame. The contour of the robot is cylindrical. The security robot contains an upper body and a lower body. The upper body of the security robot contains the main controller (Industrial Standard PC with a Pentium-III 933 CPU and 256M RAM), touch screen, CCD, sensors and sensory circuits and some hardware devices. The lower body contains the drive system, batteries and two DC servomotors.
Fig. 3. The hardware structure of the intelligent security robot.
There are six systems in the security robot, including a sensor system, remote supervisory system, software development system, an image system, obstacle avoidance and motion planning system and other systems. Fig. 2 shows the function of the intelligent security robot. Fig. 3 is the hierarchy structure of the security robot, and each system includes some subsystem. Each system has some function for implementation. For example, the other system contains auto-dialling, alarm, auto recharge, power schedule and structure. Every function can be finished by a dividable device. And it transmits the experimental results to the main controller. The hardware structure of the security robot is shown in Fig. 3. There are many devices to be designed by ourselves. Such as an auto-recharging switch, ultrasonic driver, fire detection. . . and so on. In the multifunction I/O card, it can acquire the sensor information and control driver status. In the motion control card, it can control the motion of the ISR, and acquire the IR sensor information. The main controller (Industry Personal Computer, IPC) can acquire the distance of obstacles by an ultrasonic driver using a series interface. The other device has a CCD, touch screen, wireless LAN, alarm device and GSM modem. In the autorecharging function and power detection, we design the autorecharging station for the intelligent security robot. The ‘‘Battery Detection’’ block contains recharging and discharging current measurement values, and measures the battery voltage value. The ‘‘Auto Recharging Switch’’ block contains a driver circuit and relay
332
R.C. Luo et al. / Robotics and Autonomous Systems 57 (2009) 330–338
Fig. 4. The sensory system architecture.
element, and controls the recharging current to prevent overload. It can turn on and cut off the recharging current. 3. System function The hierarchy structure of sensory system for the ISR is shown in Fig. 4. The sensory system has six variety subsystems. These are: fire detection subsystem, intruder detection subsystem, power detection subsystem, environment detection subsystem, motor control subsystem and obstacle detection subsystem. These subsystems can acquire sensory signals and processes these signals using an amplifier and calibration circuits, and transmits sensory data to the IPC using interface devices. In the fire detection subsystem and intruder detection subsystem and motor control subsystem, we use a digital input/output interface card to transmit sensory data. In the power detection subsystem and environment subsystem, we use an analogue input interface card to acquire measured values. We use sixteen ultrasonic sensors and eight IR sensors to detect obstacles in the obstacle detection subsystem. Then we use a microprocessor (MCS-5A series) to drive the ultrasonic sensors and get a distance value from obstacles and transmit the distance value to the IPC using a series interface (RS232). These sensors are listed in Table 1. The arrangement of intruder detection sensors and other hardware are shown in Fig. 5. In the fire detection sensors, we combine a smoke sensor, flame sensor and temperature sensor in one module. The module is fixed on the front side of the intelligent security robot. In the intruder sensors, we use three body sensors, sixteen ultrasonic sensors and eight IR sensors arranged outside of the intelligent security robot. The intelligent security robot can transmit messages to the user using GSM modem. The relationship of the GSM modem and the intelligent security robot is shown in Fig. 6. The intelligent security robot communicates with a cell phone using GSM (Global System for Mobile) module. The Remote Control and Monitor Interface are shown in Fig. 7; the interface includes four sub-windows: the top window is for environment detection; the top-left window shows the robot’s vision system. The bottom-left window shows the global map of the remote environment and the trajectory of mobile robot from dead-reckoning measurement; the bottom-right window shows the several control modes for the robot’s motion system. The top sub-window shows the environmental detection status from the mobile robot. The environmental sensors include flame sensors, smoke sensors, temperature sensors, body sensors, humidity sensors, gas sensors, noise sensor and luminance sensors. When a dangerous situation occurs, the security robot server sounds the alarm and notifies the remote site through the internet. The finished 3-in-1 fire detection sensor that we designed is shown in Fig. 8. The left image shows the original hardware, and the right image shows that we encapsulate the sensors with a shell to match the outside of ISR. It is integrated with a flame sensor, temperature sensor and smoke sensor to detect
Fig. 5. The sensor arrangement.
Fig. 6. The relation of GSM modem and main controller of the security robot.
Fig. 7. Remote control and monitor interface.
the flame temperature and smoke information. When fires occur, the most abnormal environmental changes are flames, increased temperature and smoke filling the air. So we employ a flame sensor, temperature sensor and smoke sensor to detect whether these three condition are happening or not. When the information of fire detection modules is collected, we use an adaptive multisensor fusion algorithm to judge whether a fire is occurring or not.
R.C. Luo et al. / Robotics and Autonomous Systems 57 (2009) 330–338
333
Table 1 Sensors in the security robot. Subsystem
Sensors
Examples
Fire detection subsystem
Temperature sensor Smoke sensor Flame sensor Body sensor Ultrasonic sensor IR sensor Light-operated Proximity sensor Ultrasonic sensor Gas sensor Humidity sensor Lux meter sensor Voice sensor Current sensor
AD590 TG135 R2686 Body Polaroid 6500 SMC-10R CDD-40N Polaroid 6500 TGS 822 C2 – M3 S1133 Microphone MEL-55P
Intruder detection subsystem
Obstacle detection subsystem Environment detection subsystem
Power detection subsystem
Fig. 8. 3-in-1 fire detection sensor.
In the hierarchy of the remote supervisory system, it contains the communication protocol, data base, Internet, user interface and touch screen. The hardware configuration of the remote surveillance and control system includes a robot server (main controller of the security robot, i.e. IPC) to be embedded in the intelligent security robot, a web server workstation, client computers (PC or PDA) and sensors, as shown in Fig. 9. The main controller of the ISR can get sensory data to detect fire, intruders, gas, etc. It can also control the color CCD, motors, etc. The main controller of the ISR can interact with the Web server. In order to reduce the Internet time delay, the Web server is usually set up at the local site. Therefore, the main controller of the ISR and the Web sever communicate in the same local area network. The web server can get the sensory data (sensor status, commands, robot status, environment conditions, etc) from the robot server through the Internet. The web server can also receive the client’s commands via the Internet and then transmit it to the robot server. The remote user can connect to the Web server or the robot server to get all information by computer or PDA through the Internet. There are three control modes in the GUI implemented with Visual C++, direct control mode, supervisory control mode and job scheduling mode. In the direct control mode, the dialog includes the control buttons such as go forward, go backward, turn left, turn right and stop. The control buttons relative to controlling the camera include up, down, left, right, zoom in and zoom out. The user can not only click the buttons by using a mouse but also push the button arrow up, arrow left, arrow right and arrow down on a keyboard. Furthermore, the user can push the button in the toolbar to select the current velocity of the robot’s motion (Fig. 10). In the supervisory control mode, if the user decides a goal position, the robot generates a moving path and moves toward the goal position autonomously. After the goal position is determined, the shortest moving path is generated by using the ‘‘via’’ points which were already given in the map. The ‘‘via’’ points can be considered as all the points in the path through which the robot passes.
Fig. 9. The hardware configuration of the remote surveillance and control architecture.
Fig. 10. The user can control the motion of a robot with different velocities.
334
R.C. Luo et al. / Robotics and Autonomous Systems 57 (2009) 330–338
Fig. 13. The detection and diagnostic rule of the intruder system. Fig. 11. Architecture of wireless appliance control system.
4. Detection method There are many algorithms applied in the sensor system. In the fire detection subsystem, we use an adaptive multisensor fusion algorithm. In the intruder detection subsystem, we use a rulebased method and combine body sensors IR sensors and ultrasonic sensors to implement. In the environment detection subsystem, we use a statistical signal method to decide the exact measurement value, and it contains humidity detection, illumination detection, temperature detection and gas detection. The adaptive decision fusion method was proposed by Ansari [14–16]. The parameters and details of the equation are discussed in another paper [17–19] and the reinforcement updating rules are:
∆ω ˆi =
Fig. 12. Architecture of wireless security detection system.
The architecture of wireless appliance control system is shown in Fig. 11. The module can control appliances using relay elements, and transmits the action status to the mobile robot and security device using a wireless RF interface. The basic function of the wireless appliance remote control is shown below: with UHF emission circuit, it can perform wireless transmission and control. 1. With encoded IC, it will not be interfered by outside world noise. 2. System is composed of four-groups-key wireless emitter and 89C2051 receiver control board. 3. The work frequency of high-frequency carrier of wave wireless emitter and receiver are 315 MHz and 433 MHz. 4. ON/OFF of appliance can be controlled by a switch or relay. In the home security system, it can execute the same function to receive the security module information, and transmits the detection information to the user through a GSM modem and the Internet. But it connects with the remote supervisory computer using the Internet. The architecture of wireless security detection System is shown in Fig. 12.
∆ω ˆ0 =
1 m1i 1
m0i 1
if ui = +1
ˆ1 and H
if ui = −1
ˆ0 and H
ˆ0 eωˆ i +ωˆ 0 if ui = +1 and H − m1i − 1 eωˆ i −ωˆ 0 if ui = −1 and Hˆ 1 m0i 1 ˆ 1 occurs when H m
1 − eωˆ 0 m
(1)
(2)
ˆ 0 occurs when H
ωˆ i+ = ωˆ i− + ∆ωˆ i , i = 0, 1, 2, where ωˆ i+ and ωˆ i− represent the weight value after and before each update. In the intruder sensors, we use three body sensors, sixteen ultrasonic sensors and eight IR sensors arranged outside of the ISR. The detection and diagnosis rule is shown in Fig. 13. The detection rule of the intruder system has three stages. The high stage uses a body sensor to detect intruders. If the body sensor detect an intruder, the output state is ‘‘Yes’’, otherwise, the output state is ‘‘No’’. We use an ultrasonic sensor to detect intruders in the second stage. In the third stage, we use IR sensors to detect intruders. Three varieties of sensor can decide five varieties of result. They are intruder (near), intruder (beside), fail, obstacle and normal, and can diagnose which sensor will be in error. It can transmit messages to the client’s computer through the Internet. The intelligent security robot communicates with a cell phone using GSM (Global System for Mobile) modem. In the environment subsystem, we use approaches based on statistical signal detection methods. The subsystem contains a
R.C. Luo et al. / Robotics and Autonomous Systems 57 (2009) 330–338
335
humidity sensor, luxmeter sensor, temperature sensor, and gas sensor and voice sensor. We modeled the observed environment detection system as the sum of 3 signal components, to be shown in Eq. (3). For a faultfree device F (t ) = 0; conversely, when any faults are occurring, F (t ) is not zero. The observed environment detection system of the security robot is [20]: Z (t ) = W (t ) + n(t ) + F (t )
(3)
where: Z (t ) The matrix of parameter by the environment detection system as measured or observed, W (t ) The matrix of parameter by the environment detection system under ideal conditions, F (t ) Any signal change as a consequence of faults, n(t ): Any signal change as a consequence of thermal effects, sample error, etc. If we are able to calculate a w( ˆ t ), either through simulation or repeated observation of a no-fault device then we can get :
w( ˆ t ) = W (t ) + e(t )
Fig. 14. Screen of sensors I/O state and GSM control buttons.
(4)
where e(t ) is estimate error. We combine Eqs. (3) and (4), and get the following result. Z (t ) − w( ˆ t ) = n(t ) − e(t ) + F (t ).
(5)
With a no-fault device, we use a maximum likelihood estimator (MLE) method to apply in the environment detection system. If the signal is deterministic and the noise is Gaussian with zero mean, then the MLE value is adequate:
w ˆ MLE (t ) =
X N 1 N
Z (t ) .
(a) Flame signal.
(b) Detect flame source.
Fig. 15. Use a lighter to mimic a flame occurred.
(6)
1
We can use w ˆ MLE (t ) to replace w ˆ (t ) of Eq. (5), and then get: Z (t ) − w ˆ (t ) = n (t ) + F (t ) .
(7)
Again, if the signal is deterministic and the noise is Gaussian with zero mean an adequate detector for F (t ) is the LRT detector [21]:
λ = φ(n)0 Rn−1 φ(n) φ(n) = Z (n) − W (n).
(a) Gas signal.
(b) Detect gas signal.
Fig. 16. State of smoke sensor shows gas have been detected.
(8)
The λ is over the threshold. We can say the measurement value to be faulty. Otherwise the measurement value w ˆ MLE (t ) is exact. 5. Experimental results Fig. 14 is the interface that shows the I/O states of the intelligent security robot. ‘‘Block 1’’ shows the states of fire sensors, ‘‘Block 2’’ shows the power states of battery. The panel display 36 V power status for driver system, and displays the 12 V power status for IPC and sensory detection system. ‘‘Block 3’’ shows the states of body sensors, ‘‘Block 4’’ shows the buttons of the fire and intruder alarm switches, and ‘‘Block 5’’ shows the GSM control buttons and the state of GSM massage sending. We open the switch of fire alarm, so its state is ON; but the intelligent security robot does not detect any flame or smoke, all states of fire sensors are 0 (means no flame or smoke is detected). In Fig. 15(a), we use a lighter to simulate a flame when the ISR is on its way executing a patrol task. Fig. 15(b) shows that the switch of fire alarm has been opened. ISR will detect the flame and then the state of flame sensor will become 1 (means flame has been detected). At the same time the fire alarm will sound and send a message to the previously assigned number. Then we use a lighter to simulate a gas leak (Fig. 16(a)), and the state of smoke sensor
(a) Intruder detection.
(b) Detect a intruder.
Fig. 17. Mimic an intruder detect.
shows gas has been detected (Fig. 16(b)). Fig. 16 shows that we open the switch of intruder alarm, so its state is ON; but intelligent security robot does not detect any intruders, all states of body sensors are 0 (means no intruders are detected). In Fig. 17(a) mimic intruder when ISR is on its way executing patrol task. Fig. 17(b) shows that the switch of intruder alarm has been opened. ISR will detect the intruder and then the state of intruder sensor will become 1 (means an intruder has been detected). At the same time the intruder alarm will sound and send a message to the previously set number. We implement the proposed method applying in the intelligent security robot. We design the fire sensory team (smoke sensor,
336
R.C. Luo et al. / Robotics and Autonomous Systems 57 (2009) 330–338
(a) The robot scenario.
(b) The remote site.
Fig. 18. The robot detects fire event.
(a) The robot scenario.
(b) The remote site.
Fig. 19. The robot change orientation.
(a) The robot scenario.
(b) The remote site.
Fig. 20. The robot transmits the fire status to client.
flame sensor and temperature sensor) to be fixed on the front side of the security robot. The security robot can move in free space. When a fire event happens, the security robot can detect the fire source using the fire sensory team. It is shown in Fig. 18(a). The CCD that is fixed on the intelligent security robot transmits an image to the remote supervisory computer. The real image is shown in Fig. 18(b). The intelligent security robot can move its orientation to face the fire source, and transmits the fire source to the remote supervisory computer. The scenario is shown in Fig. 19. Then it moves to the fire source, and use the fire sensory team to decide the exact result using an adaptive fusion algorithm. If the fire event is true, the security robot must alarm and send a message to the remote supervise computer. It can display the fire status in the monitor. The green point represents the normal status, and the red point represents hazards. The security robot moves to the fire source to decide an exact detection using an adaptive fusion algorithm. The scenario is shown in Fig. 20. At the remote site, the monitor can display the path of the intelligent security robot in the fire detection processing.
(a) Find intruder.
In the other condition, the intelligent security robot can detect intruders using body sensors, ultrasonic sensors and IR sensors. Because the security robot will execute its task during night, the people they meet will all be thought of as intruders. The process of intruder detection is described below. First, the security robot uses body sensors to detect intruders, and is shown in Fig. 21(a). The detection distance is about 6 m. Next, the security robot use ultrasonic sensors and IR sensors to decide an exact result, and the scenario is shown in Fig. 21(b). The ultrasonics can measure the varying distance from the intruder (obstacle). If the intruder comes to the security robot, the IR sensor can detect the intruder, and the detection distance is 30 cm. Finally, the security can display the intruder status, and alarm to alert the intruder. The scenario is shown in Fig. 21(c). If the fire event is true, the ISR must alarm and sent the message to the user using the GSM module. The user can receive the message using a cell phone. The experimental result is shown in Fig. 22(a). The ISR can transmit the fire event to the client computer using Internet. The client computer can display the fire status in the monitor and is shown in Fig. 22(b). The ISR can move to a safe distance from the fire source using ultrasonic sensors and temperature sensor. That is to say, if the ISR moves to the fire source, the detection distance must be smaller using ultrasonic sensors or the temperature measured value must be higher using the temperature sensor. In the intruder detection, the security robot must alarm and transmit the detection results to the cell phone (shown in Fig. 23(a)) and the client computer (shown in Fig. 23(b)). In the appliance control module, the user can control the appliance using a cellular phone through a GSM modem, or use a remote supervisory computer to control the appliance through the Internet. In the Fig. 24(a), the user transmits the message to the ISR to turn on an electric fan. If the appliance control module receives a wireless RF signal from the ISR, it must turn on the electric fan, and transmit the action result to the ISR. The panel of the ISR can display the electric fan’s status. The ISR can feed the message of the lamp’s status back to the user’s cellular phone through a GSM modem. The experimental scenario is shown in Fig. 24(b). 6. Conclusion We have developed a multisensor based intelligent security robot that includes six systems. There is a software development system; obstacle avoidance and motion planning system, image system, sensor system, remote supervisory system and other systems. The intelligent security robot has been applied in an intelligent building. The user can control the ISR via the client’s terminal through the Internet. In the sensor system, we have designed an intelligent fire detection module which is applied to fire events. The three sensors (smoke sensor, flame sensor and temperature sensor) of the fire detection module use an adaptive fusion method to arrive at an adequate decision result. The ISR can use the proposed method to detect fire events and intruders, and transmits the message to a cell phone using a GSM modem, and
(b) Move to the intruder.
(C) Alarm to alert the intruder.
Fig. 21. The intruder detection scenario for the security robot.
R.C. Luo et al. / Robotics and Autonomous Systems 57 (2009) 330–338
(a) Fire condition.
(b) Client terminal.
Fig. 22. Cell phone and client terminal display fire message.
(a) Intruder condition.
(b) Client terminal.
Fig. 23. Cell phone and client terminal display intruder massage.
Fig. 24. The appliance control module experimental result. (a) The lamp on (b) The cellular phone and the LCD panel of the mobile robot can display ‘‘ON’’ status.
transmits the detection results to the client’s computer through the Internet. The security robot can detect intruders, and transmits the message to a cell phone, too. Further research is to investigate multiple security robots cooperating on the security system for an intelligent building. Acknowledgment This work was supported by the project, ‘‘The Development of Integrated Robotics System’’, under DOIT TDPA of Taiwan, ROC., 95EC-17-A-04-SI-054. References [1] A. Birk, H. Kenn, An industrial application of behavior-oriented robotics, in: IEEE International Conference on Robotics and Automation, ICRA, vol. 1, 2001, pp. 749–754. [2] M. Saitoh, Y. Takahashi, A. Sankaranarayanan, H. Ohmachi, K. Marukawa, A mobile robot testbed with manipulator for security guard application, in: IEEE International Conference on Robotics and Automation, vol. 3, 1995, pp. 2518–2523. [3] Y. Takahashi, I. Masuda, A visual interface for security robots, in: IEEE International Workshop on Robot and Human Communication, 1992, pp. 123–128. [4] M. Saitoh, Y. Takahashi, A. Sankaranarayanan, H. Ohmachi, K. Marukawa, A mobile robot testbed with manipulator for security guard application, in: Proceedings of the IEEE International Conference on Robotics and Automation, 1995, vol. 3, pp. 2518–2523.
337
[5] H. Hara, Development of ROBOT KOBAN, IEEE European Convention on Security and Detection (1995) 193–197. [6] R.C. Luo, C.C. Lin, S.H. Pan, K.L. Su, The internet-based security robot for the home automation system, in: The 13th Automation Technology Conference in Taipei, 6/17-6/18, 2004, pp. 954–959. [7] S.O. Lee, Y.J. Cho, M.H. Bo, B.J. You, S.R. Oh, A stable target-tracking control for unicycle mobile robots, in: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2000, vol. 3, 2000, pp. 1822–1827. [8] L.E. Parker, B.A. Emmons, Cooperative multi-robot observation of multiple moving targets, in: Proceedings of the IEEE International Conference on Robotics and Automation, vol. 3, 1997, pp. 2082–2089. [9] H. Kobayashi, M. Yanagida, Moving object detection by an autonomous guard robot, in: Proceedings of the 4th IEEE International Workshop on Robot and Human Communication, TOKYO, 1995, pp. 323–326. [10] Y. Shimosasa, J. Kanemoto, K. Hakamada, H. Horii, T. Ariki, Y. Sugawara, F. Kojio, A. Kimura, S. Yuta, Some results of the test operation of a security service system with autonomous guard robot, in: 26th Annual Conference of the IEEE on Industrial Electronics Society, IECON 2000, vol. 1, 2000, pp. 405–409. [11] G.A. Gilbreath, D.A. Ciccimaro, H.R. Everett, An advanced telereflexive tactical response robot, in: Proceedings, Workshop 7: Vehicle Teleoperation Interfaces, IEEE International Conference on Robotics and Automation, ICRA2000, San Francisco, CA, 28 April, 2000. [12] D.A. Ciccimaro, H.R. Everett, M.H. Bruch, C.B. Phillips, A supervised autonomous security response robot, in: American Nuclear Society 8th International Topical Meeting on Robotics and Remote Systems, ANS’99, Pittsburgh, April, 1999, pp. 25–29. [13] A. Bradshaw, The UK security and fire fighting advanced robot project, IEE Colloquium on Advanced Robotic Initiatives in the UK (1991) 1/1–1/4. [14] N. Ansari, J.G. Chen, Y.Z. Zhang, Adaptive decision fusion for unequiprobable sources, IEE Proceedings Radar, Sonar and Navigation (1997) 105–111. [15] N. Ansari, E.S.H. Hou, B.O. Zhu, J.G. Chen, Adaptive fusion by reinforcement learning for distributed detection systems, IEEE Transactions on Aerospace and Electronic Systems 32 (2) (1996) 524–531. [16] G. Chen, N. Ansari, Adaptive fusion of correlated local decision, IEEE Transaction on System, Man and Cybernetics—PART C: Application and Review 28 (2) (1998). [17] R.C. Luo, K.L. Su, K.H. Tsai, Intelligent security robot fire detection system using adaptive sensory fusion method, in: IEEE International Conference on Industrial Electronic, Control, and Instrumentation, IECON’2002, pp. 2663–2668. [18] R.C. Luo, K.L. Su, K.H. Tsai, Fire detection and isolation for intelligent building system using adaptive sensory fusion method, in: IEEE International Conference on Robotics and Automation, ICRA’2002, pp. 1777–1781. [19] R.C. Luo, S.Y. Lin, K.L. Su, A multiagent multisensor based security system for intelligent building, IEEE/SICE/RSJ, in: International Conference on Multisensor Fusion and Intelligent for Intelligent System, MFI03, pp. 311–316. [20] J.F. Frenzel, Power-supply current diagnosis of VLSI circuit, IEEE Transactions on Reliability 43 (1) (1994) 30–38. [21] R.J. Bibbero, Microprocessors in Instruments and Control, Jolin Wiley & Sons. Inc., 1997.
Ren C. Luo (M’83–SM’88–F’92) received the Ph.D.degree in electrical engineering from the Technische Universitaet Berlin, Berlin, Germany. He is currently a Distinguished Professor in the Department of Electrical Engineering at National Taiwan University in Taiwan. He was a Professor of Department of Electrical and Computer Engineering at North Carolina State University, Raleigh, NC, USA and Toshiba Chair Professor in the University of Tokyo, Japan. His research interests include sensor-based intelligent robotic systems, multisensor fusion and integration, computer vision, micro/nano technologies, rapid prototyping, and advanced manufacturing systems. He has authored more than 300 papers on these topics, which have been published in refereed technical journals and conference proceedings. He also holds several patents. Dr. Luo received IEEE Eugean Mittlemann Outstanding Research Achievement Award, 1996; ALCOA Foundation Outstanding Engineering Research Award, NCSU, USA; National Science Council Outstanding Research Awards, 1998–1999, 2000–2001, 2002–2005; National Science Council Distinguished Research Awards, 2006–2008; TECO Outstanding Science and Technology Research Achievement Award, 2001. Dr. Luo is currently Editor-in-Chief of IEEE/ASME Transactions On Mechatronics. He served as President of IEEE Industrial Electronics Society (2000–2001). He also served as President of Chinese Institute of Automation Engineers, Convener of Automation Technology Division, National Science Council; Adviser of Ministry of Economics Affairs and Technical Adviser of Prime Minister’s Office in Taiwan. He contributes regularly to IEEE sponsored international conferences by serving as conference General Chairs (IEEE IROS 1992, MFI 1994, IECON 1996, MFI 1999, ICRA 2003, IECON 2007, IROS 2010), Program Chairs, program committees, and offers short courses or tutorials and plenary/keynote speeches in various countries and research communities. Dr. Luo is a Fellow of IEEE since 1992 and a Fellow of IEE.
338
R.C. Luo et al. / Robotics and Autonomous Systems 57 (2009) 330–338 Tung Y. Lin received the B.S. and M.S. degrees in electrical engineering in 2004 and 2006 respectively at National Chung-Cheng University, Taiwan. He is currently a Ph.D. student in North Carolina State University, USA. His research interests include multisensor fusion and intelligent robotics.
Kuo L. Su received the B.S. and M.S. degrees in automatic control engineering from Feng-Chia University, Seatwen, Taiwan, ROC, and received the PH.D. degree in electrical engineering at National Chung-Cheng University, Chia-Yi, Taiwan, ROC, He is currently a teacher in the Department of Electrical Engineering, National Yunlin University of Science & Technology. His research interests include multisensor fusion and robotics.