- Email: [email protected]
Publish-Subscribe Architecture for Delivering Assistance to Visually Impaired People
4th IFAC Symposium on Telematics Applications 4th on Applications November 6-9, 2016. UFRGS, Porto Alegre, RS, Brazil 4th IFAC IFAC Symposium Symposium on Telematics Telematics Applications 4th IFAC Symposium on Telematics Applications November 6-9, 2016. UFRGS, Porto Alegre, RS, online Brazil at www.sciencedirect.com Available November 6-9, 2016. UFRGS, Porto Alegre, November 6-9, 2016. UFRGS, Porto Alegre, RS, RS, Brazil Brazil
ScienceDirect IFAC-PapersOnLine 49-30 (2016) 150–155
Publish-Subscribe Architecture for Publish-Subscribe Architecture for Publish-Subscribe Architecture for Delivering Assistance to Visually Impaired Impaired Delivering Assistance to Visually Visually Impaired People People People Ricardo N. Rodrigues ∗∗ Alexsander V. Canez ∗∗ ∗ Ricardo N. Rodrigues Alexsander V. Canez ∗ ∗ Ricardo N. Alexsander V. Gisele M. Simas ∗∗∗∗ Regina Barwaldt Ricardo N. Rodrigues Rodrigues Alexsander V. Canez Canez ∗ Regina Barwaldt ∗ ∗ Gisele M. Simas ∗ ∗ Gisele M. M. Simas Simas Regina Regina Barwaldt Barwaldt Gisele ∗ ∗ Universidade Federal do Rio Grande, Rio Grande, RS, Brasil ∗ Federal do Rio Grande, Rio Grande, RS, Brasil ∗ Universidade Universidade Federal Rio (e-mails: [email protected], [email protected], Universidade Federal do do Rio Rio Grande, Grande, Rio Grande, Grande, RS, RS, Brasil Brasil (e-mails: [email protected], [email protected], (e-mails: [email protected], [email protected], [email protected], [email protected]). (e-mails: [email protected], [email protected], [email protected], [email protected]). [email protected], [email protected], [email protected]). [email protected]).
Abstract: Abstract: Abstract: Visually impaired people face many difficulties in everyday tasks. In this paper we present a Abstract: Visually impaired people many in tasks. In this we present aa Visually impaired peopleforface face many difficulties difficulties in everyday everyday that tasks.can Inhelp this apaper paper wesuch present distributed architecture delivering Assistive Technologies user in tasks. Visually impaired people face many difficulties in everyday tasks. In this paper we present a distributed architecture for delivering Assistive Technologies that can help a user in such tasks. distributed architecture for delivering Assistive Technologies that can help a user in such tasks. The architecture allows for a user using aAssistive internet Technologies enabled device to can sendhelp images, andtasks. GPS distributed architecture delivering that a uservideo in such The architecture a userthat using internet enabled device send images, video The architecture allows a internet enabled device to send images, and GPS location to remoteallows “Helpers” canaaasend messages back with to helpful or and tips.GPS The The architecture allows a user user using using internet enabled device to send instructions images, video video and GPS location to remote “Helpers” that can send messages back with helpful instructions or tips. location to remote remote “Helpers” that can can send messages back withpeople helpfulinvited instructions oruser. tips. The The Helpers can be programs running on send a remote serverback or real by theor location to “Helpers” that messages with helpful instructions tips. The Helpers can be programs on server or people invited the The Helpers can is bebased programs running on aaa remote remote server using or real realMQTT peopleprotocol. invited by by the user. user.based The architecture on a running publish-subscribe paradigm A system, Helpers can be programs running on remote server or real people invited by the user. The architecture is based on a publish-subscribe paradigm using MQTT protocol. A system, based architecture is based on a publish-subscribe paradigm using MQTT protocol. A system, based on the proposed architecture, was implemented with a collision detector, a scene classifier and a architecture is based on a publish-subscribe paradigm using MQTT protocol. A system, based on the proposed architecture, was implemented with aa collision detector, aa scene classifier and aa on the proposed architecture, was implemented with collision detector, scene classifier and object recognized. A qualitative analysis of experiments have shown positive results and pointed on the proposed architecture, was implemented with a collision detector, a scene classifier and a object recognized. A object recognized. A qualitative qualitative analysis analysis of of experiments experiments have have shown shown positive positive results results and and pointed pointed out some improvements. object recognized. A qualitative analysis of experiments have shown positive results and pointed out some improvements. out some improvements. out some improvements. © 2016, IFAC (International Federation of Automatic Control) Hosting by Elsevier Ltd. All rights reserved. Keywords: System architectures; Computer vision; Distributed systems; Computer Keywords: architectures; vision; Distributed systems; Keywords: System System architectures; Computer vision; Distributed systems; Computer Computer applications; Obstacle detection; Computer Scene analysis; Object recognition. Keywords: System architectures; Computer vision; Distributed systems; Computer applications; Obstacle detection; Scene analysis; Object recognition. applications; Obstacle detection; Scene analysis; Object recognition. applications; Obstacle detection; Scene analysis; Object recognition. 1. INTRODUCTION • It enables the “Human-in-the-loop” paradigm by al1. •• It the “Human-in-the-loop” paradigm al1. INTRODUCTION INTRODUCTION It enables enables theinvited “Human-in-the-loop” paradigm bysend allowing a user person to receive data andby 1. INTRODUCTION • It enables the “Human-in-the-loop” paradigm by allowing a user invited person to receive data and send lowing a user invited person to receive data and send help when requested. This opens the door includlowing a user invited person to receive datafor and send The World Health Organization (WHO, 2014) estimated help when requested. opens the door for includThe World Health Organization (WHO, 2014) estimated help when requested. This opens the including in This assistive technology solutions. The World in Health Organization (WHO,285 2014) estimated helpsocial whennetworks requested. This opens the door door for for includin a survey 2012 that there are about million people The World Health Organization (WHO, 2014) estimated ing social networks in assistive technology solutions. in aathe survey in 2012 that there are about 285 million people ing social networks in assistive technology solutions. • It aims in providing an accessible and low-cost soin survey in 2012 that there are about 285 million people ing social networks in assistive technology solutions. world with that somethere kindare of about visual285 deficiency, being in a survey in 2012 million people •• It aims in providing an accessible and low-cost soin the world with some kind of visual deficiency, being It aims in providing an accessible and low-cost solution, since it does not require a specific device in the world with some kind of visual deficiency, being • It aims in providing an accessible and low-cost so246 million with partial sight and 39 million with total in the world with some kind of visual deficiency, being lution, since it does not require a specific device 246 million with partial sight and 39 million with total lution, since it does not require a specific device or sensor. In fact, our target is the use of common 246 million with partial sight and 39 million with total lution, since it does not require a specific device blindness. These people are constantly excluded because 246 million with partial sight and 39 million with total or sensor. In fact, our target and is the use of common blindness. These are constantly excluded because or In fact, our use devices, such smartphones tablets. blindness. These people people areAssistive constantly excludedsolutions because or sensor. sensor. In as fact, our target target is is the the use of of common common of their disability, but new Technology blindness. These people are constantly excluded because devices, such as smartphones and tablets. of their disability, but new Assistive Technology solutions devices, such as smartphones and tablets. • It is inspired in the ubiquitous computing of their disability, but new Assistive Technology solutions devices, such as smartphones and tablets. paradigm, aretheir beingdisability, researched and developed improve their social of but new AssistivetoTechnology solutions •• It in ubiquitous computing are being It is is inspired inspired in the thethat ubiquitous computing paradigm, which advocates technology shoudparadigm, be omare being researched researched and and developed developed to to improve improve their their social social • It is inspired in the ubiquitous computing paradigm, inclusion. are being researched and developed to improve their social which advocates that technology shoudorbe bespread ominclusion. which advocates that technology shoud nipresent (pervasive), invisible in devices inclusion. which advocates that technology shoud be omominclusion. nipresent (pervasive), invisible in devices or spread Recent advances in computer vision, electronic devices, nipresent (pervasive), invisible in devices or spread over the environment and its usage should be transnipresent (pervasive), invisible in devices or spread Recent advances in computer vision, electronic devices, Recent advancesand in wireless computer vision, electronic electronic devices, over thetoenvironment and its usage should be transmicroprocessors communication motivate the Recent advances in computer vision, devices, over and parent the user. over the the environment environment and its its usage usage should should be be transtransmicroprocessors and wireless communication motivate the microprocessors and wireless communication motivate the parent to the user. cientific comunity to search new solutions that combine microprocessors and wireless communication motivate the parent to to the the user. user. parent cientific comunity to search new solutions that combine We also present a system implementation based on the cientific comunitytoto toimprove search new new solutions thatof combine combine such technologies the quality of life visually cientific comunity search solutions that We also present aa system implementation based on the such technologies to improve the quality of life of visually We also present system implementation based on proposed architecture with three functionalities: collision such technologies to improve the quality of life of visually the impaired people. to Forimprove an assistive technology to visually have a We also present a system implementation based on the such technologies the quality of life of proposed architecture with three functionalities: collision impaired people. For an assistive technology to have a proposed architecture architecture with three three functionalities: collision detection, scene classification and functionalities: object recognition. Figimpaired people. For issues an assistive assistive technology to such haveas:aa proposed with collision positive affect, several need totechnology be explored, impaired people. For an to have scene classification and object Figpositive affect, issues explored, as: detection, scene classification and recognition. Figure 1 illustrate example where a blindrecognition. user is warned positive affect, several several issues need need to to be be explored, such such as: detection, detection, scene an classification and object object recognition. Figwhich paradigms and computational architectures should positive affect, several issues need to be explored, such as: ure 1 illustrate an example where a blind user is warned which paradigms and computational architectures should ure 1 illustrate an example where a blind user is warned on the presence of an obstacle and the object is properly which paradigms and computational computational architectures should be employed; which devices should architectures be used; how the ure 1 illustrate an example where a blind user is warned which paradigms and should of obstacle the properly be employed; which devices be on the the presence presence of an an obstacle and the object object isimproving properly recognized, allowing him a safer and locomotion andis be employed; which should devicesbeshould should be used; used; how how the the on on the presence of an obstacle and the object is properly available technologies combined. be employed; which devices should be used; how the recognized, allowing him a safer locomotion and improving available technologies should be combined. recognized, allowing him a safer locomotion and improving environment awareness. available technologies should be combined. recognized, allowing him a safer locomotion and improving available technologies shouldproposes be combined. environment awareness. awareness. In this context, our paper a distributed archi- environment environment awareness. In this context, our paper proposes aa distributed archiIn this context, our paper proposes distributed archi- This paper is organized as follows: section 2 describes some tecture based on publish/subscribe to support In this context, our paper proposesparadigm a distributed archiThis paper is organized organized as presents follows: section section describes some tecture on paradigm to paper is 222 describes some related works; section 3as the proposed architectecture based on publish/subscribe publish/subscribe paradigmpeople. to support support This paper is organized as follows: follows: section describes some assistivebased technologies for visually impaired The This tecture based on publish/subscribe paradigm to support related works; section 3 presents the proposed architecassistive technologies for visually impaired people. The related works; section 3 presents the proposed architecture; in section 4 the implemented system, based on the assistive technologies for visually visually impairedare: people. The The related works; section 3 presents the proposed architecmain features of the proposed architecture assistive technologies for impaired people. ture; in section 44 the implemented system, based on the main features of the proposed architecture are: ture; in section the implemented system, based on proposed architecture, is detailed; in section 5 we report main features of the proposed architecture are: in section 4 the implemented system, based on the the main of the architecture are:from other ture; proposed architecture, is detailed; in section 5 we report • Itfeatures is flexible andproposed customizable, differing proposed architecture, isa detailed; detailed; in section section we report some experiments and is qualitative analysis;55 we finally, in proposed architecture, in report •• It is flexible and customizable, differing from other It is flexible flexible and customizable, customizable, differing from other experiments and aaconclusion. qualitative analysis; finally, in works that target a specific typediffering of assistance. Our some • It is and from other some experiments and qualitative analysis; finally, in section 6 we present our some experiments and a qualitative analysis; finally, in works that target aa specific type Our works that targetthe specific type of of assistance. assistance. Our section 66 we present our conclusion. solution enables communication between users works that target a specific type of assistance. Our section we present our conclusion. section 6 we present our conclusion. solution enables the communication between users solution enables the communication communication between users and different “Helpers” by defining a between underlying ar2. RELATED WORK solution enables the users and different “Helpers” by defining aa underlying ar2. RELATED WORK and different “Helpers” by defining underlying ar2. chitecture for message exchange. and different “Helpers” by defining a underlying ar2. RELATED RELATED WORK WORK chitecture for message exchange. chitecture for for message message exchange. exchange. Researches related to Assistive Technology for the visually chitecture Researches related to Assistive Technology the visually The authors thank the financial support of CAPES, the Brazilian Researches related to Technology for the impaired are in constant development. Thefor are Researches related to Assistive Assistive Technology forfollowing the visually visually impaired are in constant development. The following are authors thank the financial support of CAPES, the Brazilian The impaired are in constant development. The following reported some examples of recent works in this area. Coordination of Higher Education Personnel Improvement. The authors thank the financial support of CAPES, the Brazilian impaired are in constant development. The following are are The authors thank the financial support of CAPES, the Brazilian reported some examples of recent works in this area. Coordination of Higher Education Personnel Improvement. reported some examples of recent works in this area. Coordination of Higher Education Personnel Improvement. reported some examples of recent works in this area. Coordination of Higher Education Personnel Improvement. Copyright © 2016, 2016 IFAC 150Hosting by Elsevier Ltd. All rights reserved. 2405-8963 © IFAC (International Federation of Automatic Control) Copyright 2016 IFAC 150 Peer review© of International Federation of Automatic Copyright © 2016 IFAC 150 Copyright ©under 2016 responsibility IFAC 150Control. 10.1016/j.ifacol.2016.11.145
2016 IFAC TA November 6-9, 2016. Porto Alegre, Brazil Ricardo N. Rodrigues et al. / IFAC-PapersOnLine 49-30 (2016) 150–155
Fig. 1. Illustration of an application based on the proposed architecture.
In (Cha et al., 2013) is proposed a navigation system for blinds using the Smartphone. The application uses the Google Map and Google TTS (Text-to-Speech) to lead the user to the destination. The system is basically divided into three stages. You must first inform the destination, then the voice recognition and Google TTS is applied, if the user is recognized by the system, then, in second step Google Map search and traces the route, the map is saved in the DB (database). Finally, the person is guided by the route through voice instructions to your destination. Varadarajan et al. (2014) system based at the principle Wireless Position System. Where the current position of the user is calculated point-to-point according to the level of the WiFi signal. The destination must be informed, in accordance with the current position is loaded a map where the person is guided by voice instructions, to the reach the destination a voice notification is issued stating that the destination has been reached. The Nakajima and Haruyama (2012) uses communication technology visible light and geomagnetic sensing integrated into Smartphone. This consists of LED lights, a Smartphone with integrated receiver, headphones and a Panasonic cloud environment (LBS Platform) which is referred to as position information base. The Receiver abstracts the identification of visible light by Bluetooth to find the user’s location, via WiFi receives the route the cloud and the position information for guidance is given through headphones. The geomagnetic sensor serves to guide the direction of travel.
151
Fig. 2. Architecture overview.
We observe that most works found in the literature focus in a single objective, as object recognition (Tapu et al., 2013), localization and navigation (Cha et al., 2013; Du Buf et al., 2011; Varadarajan et al., 2014; Nakajima and Haruyama, 2012) or object detection (Du Buf et al., 2011; Tapu et al., 2013). Few works consider issues related to the architecture of a computational Assistive technology (Ghorbel et al., 2015; Murua et al., 2011; Shojanoori et al., 2014). We also observe that most of the works propose the development of electronic gadgets for a specific target application (Jain, 2014; Tang and Li, 2014; Treuillet and Royer, 2010). This approach has the drawback that it usually result in a costly system with limited application scope. On the other hand, we defend the use of general purpose devices, largely accessible to a wide range of the population (e.g. smartphones, tablets), like in the works of (Cha et al., 2013; Tapu et al., 2013). Finally, we observe that few works works focus on web based technologies (Murua et al., 2011; Cha et al., 2013) which could expand the available computational resources. Most works are based in local processing. 3. ARCHITECTURE The proposed architecture can be divided in three main components as shown in Figure 2. The client device is responsible for the user interface as well as sensor data acquisition. The Publish-Subscribe middleware provides secure communication and the Helpers provide accessibility functionalities by receiving the data sent by the client, processing it, and sending back a text message to help the user. More details about each component is provided next.
Tapu et al. (2013) present a system that basically divides into obstacle detection (static and dynamic) and classification of objects to help visually impaired navigation through Smartphone. The detection is made based on the optical flow algorithm Lucas-Kanade, homographic transforms, the RANSAC (Random Sample Consensus) algorithm, agglomerative clustering technique, KNN (KNearest Neighbor). The classification is based on HOG (Histogram of Oriented Gradients) descriptor, K-means and BoVW (Bag of Visual Words) and has an image database on the device itself.
3.1 Client Device
Du Buf et al. (2011) developed a navigation system for outdoor and indoor, integrated by GPS and WiFi localization with a geographic information system (GIS) database, with RFID tags in sidewalks, still has the obstacles detection obstacle with computer vision.
The sensors available may vary among devices. Common sensors available in most smartphones and tablets include GPS, camera, microphone, accelerometer, light sensor, etc. Depending on the functionality, the data from these sensors may be requested continuously, generating a data
151
This component is responsible for providing an accessible interface to the user and for managing data acquisition from sensors attached to the device. The device will be composed of a hardware and a software part. The hardware can be, for example, a tablet (as implemented in our study-case), a smartphone or a dedicated device developed over Raspberry Pi board. Meanwhile, the software will be specifically developed to provide accessible user interface and other functionalities.
2016 IFAC TA 152 Ricardo N. Rodrigues et al. / IFAC-PapersOnLine 49-30 (2016) 150–155 November 6-9, 2016. Porto Alegre, Brazil
stream, or sporadically as needed, which will usually be triggered by a user request. For example, a collision detection functionality may have to use the camera continuously, while a image describer may rely on a single image, which is captured over user request.
Fig. 3. Example of MQTT topics with two users. Rectangles represent nodes and ellipses represent topics.
We assume the device is capable of performing Text-toSpeech (TTS). In other words, the device should be able to transform a given text string into a speech and deliver this speech to the user. The use of bone conduction earphones has been considered an interesting solution to deliver the sound to the user without disturbing the environment and with no reduction to auditory perception by the user. The accessible functionalities made available to the user are provided by Helpers that may run locally on the client device or in a remote device which is accessed via a MQTT client as detailed next. 3.3 Helper 3.2 Publish-Subscribe Middleware This component encompass the network infrastructure, including hardware and protocols, to deliver a publishsubscribe communication between network nodes. We use the Message Queue Telemetry Transport (MQTT) Locke (2010) protocol over internet. This protocol has been used in several applications of internet of things and robotics. When compared to other publish-subscribe architectures, such as ROS Quigley et al. (2009), the MQTT protocol has the advantage of being very efficient, both considering data overhead and computational resources. When the user activates some functionality, the client device uses MQTT protocol to publish the sensors data to topics which are subscribed by one or more Helpers. There is one topic for each sensor, and they are only published if a functionlity that uses they data is triggered by the user. Published messages are sent to a MQTT server, called the Brocker, which manages subscriptions and topics. An important feature implemented in the Brocker is Access Control List (ACL), which limits the access to topics by a list of users that are allowed to publish or subscribe on it Uzunov (2016). We use this feature to restrict the access to a user topic only to the user and the Helpers that the user has activated. Figure 3 shows an example of topics structure for two users and two Helpers. Note that the root topic is the username, which is set in the ACL as the only user allowed to publish on those topics. The Helpers are allowed access only to message and sensors topics when activated by the user. All users are authenticated via username/password when connecting to the MQTT brocker. Each sensor is published in one topic and the message is received in the message topic. There is also a config topic that is used to set user’s parameters that may be used by Helpers. Another MQTT security feature used in our proposed architecture is SSL encriptyion, which encrypt the messages sent among nodes. We consider that, with the use of ACL and SSL, the architecture can provide a good level of security and privacy. 152
This component receives the data sent by the client, process it, and send a text message to help the user. It may be an autonomous software, running an algorithm, or a person. In either case, the Helper should provide some aid to the user by analysing the received data and sending a helpful message. Some examples of functionalities that may be provided by Helpers are: • • • • • • •
Image description; Path navigation; Obstacle avoidance; Text recognition (OCR); Facial recognition; Object detection; Money recognition.
Software based Helpers may run locally on the client device or remotely in a server. Local Helpers have the advantage of not requiring network communication, but may be restricted by low computational capacities provided by the client device. For example, modern machine learning techniques such as deep neural networks require a significant amount of memory, so they are best suited to run remotely. Moreover, remote Helpers can be updated without the need of updating each client device. An important aspect is that the output text message provided by the Helper should be easy to understand, since it will be directly converted to speech in the client device without any kind of modification. 3.4 Data flow Figure 4 depicts the information flow, using the Business Process Model Notation, over the three components for an example when the user (Bob) captures an image. We assume two remote Helpers are active. The client device captures a frame from the camera, process it and publish to a topic using MQTT. The image will be received by the MQTT brocker, which will use the ACL to make sure the user is allowed to publish in the topic; the image is then sent to the remote Helpers, which receive the image, process it, generate a text message and publish to the user’s message topic.
2016 IFAC TA November 6-9, 2016. Porto Alegre, Brazil Ricardo N. Rodrigues et al. / IFAC-PapersOnLine 49-30 (2016) 150–155
153
Fig. 4. Data flow example.
4. IMPLEMENTATION We have implemented a system based on the proposed architecture with three functionalities: (1) Collision detection. We use a collision detection method as described in Canez (2016), that process the video stream and detects possible obstacles that may collide with the user during locomotion. The method generates a warning message when a possible collision is detected. (2) Scene classifier. This classifier analyses an image and returns the type of ambient the image was taken. Possible outputs include office, florest, parkway, kitchen, bedroom, etc. The method is based on Deep Neural Networks as described in Zhou et al. (2014) and was implemented using Caffe deep learning framework Jia et al. (2014). (3) Object recognition. We use the method described in Szegedy et al. (2015) to recognize specific objects, such as: computer, car, bed, t-shirt, cup, etc. As in the scene classifier, this method was also implemented using Caffe framework. Each of these functionalities is implemented as a Helper that may be activated or deactivated by the user. The collision detection runs locally on the client device, while the scene classifier and object recognition runs in a remote 153
server. When a remote Helper is deactivated it unsubscribe to the users topic and, consequently, do not receive or send messages anymore. For the Publish-Subscriber middleware, we have used the Mosquito MQTT as the broker and the Paho library in the client side implementation. A client was implemented in C++ using Qt Framework for user interface, OpenCv library for image handling and Microsoft .NET framework for Text to Speech. The client was deployed in a tablet with an Intel Atom processor with 1GB of RAM. Figure 5a shows the main user interface, which contains a button to capture an image and a button to configurations - which leads to screen shown in Figure 5b. The image capture command captures an image and publish it to the snapshot topic. The configuration menu contains buttons that allows the user to activate or deactivate the Helpers. The interface contains accessible features that enable a blind user to use it. For example, when a button is clicked once, a voice is synthesized givin the button description; when clicked twice, the button is triggered. 5. EXPERIMENTS In this section we present some usage experiments conducted with the system implemented in the previous
2016 IFAC TA 154 November 6-9, 2016. Porto Alegre, BrazilRicardo N. Rodrigues et al. / IFAC-PapersOnLine 49-30 (2016) 150–155
Fig. 5. User interface. a) Main screen; b) Configurations screen
section. These experiments were performed with seven healthy users, which were asked to close their eyes during the experiments. The objective is to get a qualitative feedback for future system improvements. Future experiments will be conducted with visually impaired persons in real world scenarios.
6. CONCLUSION In this paper we have proposed a new architecture for delivering Assistive Technologies to visually impaired people using a publish-subscribe architecture. The architecture objective is to provide a bridge between users and Helpers, which are algorithms or people that provide useful functionalities by processing the user’s data and sending back a text message that is synthesized using Text-To-Speech. We believe that the main advantage of this architecture is its flexibility to incorporate new sensors and functionalities. The initial results have shown good results, based on a qualitative assessment, and pointed out some improvements that should be corrected, specially regarding the efficiency of recognizers and detectors. As future works we intend to add social network functionalities in the system, improve the provided Helpers and add new ones. We are also planning real world experiments. REFERENCES
Experiment 1: Users were asked to randomly walk in a hallway using the collision detection Helper functionality.
Qualitative analysis: Overall the users reported that felt more confident with the aid of the collision detector. However, it was observed that false positives occur when turning or moving abruptly, which caused certain stress in the user. False negative errors occurred when the obstacle is homogeneous, such as a white wall, and in translucent glass. The users suggested the system could issue a progressive warning that increase its intensity according to the distance to the obstacle and give some hint to where is the obstacle located.
Experiment 2: Users were asked to randomly test the scene classifier and object recognizers in their everyday tasks.
Qualitative analysis: The scene classifier got positive evaluations, being able to give a general classification for the scene most of the time. However, the object recognizer presented more errors. We observed that objects where only recognized if they where fully framed as the only object in the image, which is not easily done without sight. We are currently investigating methods that can perform object detection, such as Ren et al. (2015), which will not only recognize the object but also find its location even in cluttered scenes. The Experiment 2 was performed over a wi-fi internet connection and over a 4G mobile network. It was observed that the latency between requesting help and receiving the message was around 1 (one) second in the wi-fi network and around 2 (two) seconds in the 4G mobile network, which was considered to be acceptable by the users. 154
Canez, A. (2016). Collision detection with a single camera. Technical report, Universidade Federal do Rio Grande, FURG. Cha, J.S., Lim, D.K., and Shin, Y.N. (2013). Design and implementation of a voice based navigation for visually impaired persons. International Journal of Bio-Science and Bio-Technology, 5(3), 61–68. Du Buf, J.H., Barroso, J., Rodrigues, J.M., Paredes, H., Farrajota, M., Fernandes, H., Jos´e, J., Teixeira, V., and Saleiro, M. (2011). The smartvision navigation prototype for blind users. Ghorbel, M., Mokhtari, M., and Renouard, S. (2015). A distributed approach for assistive service provision in pervasive environment. Proceeding WMASH 06 Proceedings of the 4th international workshop on Wireless mobile applications and services on WLAN hotspots. Jain, D. (2014). Path-guided indoor navigation for the visually impaired using minimal building retrofitting. ASSETS 14 Proceedings of the 16th international ACM SIGACCESS conference on Computers accessibility. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell, T. (2014). Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093. Locke, D. (2010). Mq telemetry transport (mqtt) v3. 1 protocol specification. IBM developerWorks Technical Library], available at http://www. ibm. com/developerworks/webservices/library/wsmqtt/index. html. Murua, A., Gonzalez, I., and Gomez-Martinez, E. (2011). Cloud-based assistive technology services. Computer Science and Information Systems (FedCSIS), 2011 Federated Conference on. Nakajima, M. and Haruyama, S. (2012). Indoor navigation system for visually impaired people using visible light communication and compensated geomagnetic sensing. In Communications in China (ICCC), 2012 1st IEEE International Conference on, 524–529. IEEE. Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., and Ng, A.Y. (2009). Ros: an
2016 IFAC TA November 6-9, 2016. Porto Alegre, Brazil Ricardo N. Rodrigues et al. / IFAC-PapersOnLine 49-30 (2016) 150–155
open-source robot operating system. In ICRA workshop on open source software, volume 3, 5. Kobe, Japan. Ren, S., He, K., Girshick, R., and Sun, J. (2015). Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems, 91–99. Shojanoori, R., Juric, R., Lohi, M., and Terstyanszky, G. (2014). Asecs: Assistive self-care software architectures for delivering service in care homes. 2014 47th Hawaii International Conference on System Sciences. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015). Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1–9. Tang, T.J.J. and Li, W.H. (2014). An assistive eyewear prototype that interactively converts 3d object locations into spatial audio. Proceedings of the 2014 ACM International Symposium on Wearable Computers. Tapu, R., Mocanu, B., Bursuc, A., and Zaharia, T. (2013). A smartphone-based obstacle detection and classification system for assisting visually impaired people. In Proceedings of the IEEE International Conference on Computer Vision Workshops, 444–451. Treuillet, S. and Royer, E. (2010). Outdoor/indoor vision based localization for blind pedestrian navigation assistance. Int. J. Image Grap. 10, 481. Uzunov, A.V. (2016). A survey of security solutions for distributed publish/subscribe systems. Computers Security, –. doi: http://dx.doi.org/10.1016/j.cose.2016.04.008. Varadarajan, V., Kolapkar, S., Saini, M., and Dange, S. (2014). Blindnav: A smartphone based navigation system for the visually impaired. Compusoft, 3(4), 758. WHO (2014). Visual impairment and blindness. fact sheet, n. 282, 2014. URL http://www.who.int/mediacentre/factsheets/fs282/en/. Zhou, B., Lapedriza, A., Xiao, J., Torralba, A., and Oliva, A. (2014). Learning deep features for scene recognition using places database. In Advances in neural information processing systems, 487–495.
155
155
ScienceDirect IFAC-PapersOnLine 49-30 (2016) 150–155
Publish-Subscribe Architecture for Publish-Subscribe Architecture for Publish-Subscribe Architecture for Delivering Assistance to Visually Impaired Impaired Delivering Assistance to Visually Visually Impaired People People People Ricardo N. Rodrigues ∗∗ Alexsander V. Canez ∗∗ ∗ Ricardo N. Rodrigues Alexsander V. Canez ∗ ∗ Ricardo N. Alexsander V. Gisele M. Simas ∗∗∗∗ Regina Barwaldt Ricardo N. Rodrigues Rodrigues Alexsander V. Canez Canez ∗ Regina Barwaldt ∗ ∗ Gisele M. Simas ∗ ∗ Gisele M. M. Simas Simas Regina Regina Barwaldt Barwaldt Gisele ∗ ∗ Universidade Federal do Rio Grande, Rio Grande, RS, Brasil ∗ Federal do Rio Grande, Rio Grande, RS, Brasil ∗ Universidade Universidade Federal Rio (e-mails: [email protected], [email protected], Universidade Federal do do Rio Rio Grande, Grande, Rio Grande, Grande, RS, RS, Brasil Brasil (e-mails: [email protected], [email protected], (e-mails: [email protected], [email protected], [email protected], [email protected]). (e-mails: [email protected], [email protected], [email protected], [email protected]). [email protected], [email protected], [email protected]). [email protected]).
Abstract: Abstract: Abstract: Visually impaired people face many difficulties in everyday tasks. In this paper we present a Abstract: Visually impaired people many in tasks. In this we present aa Visually impaired peopleforface face many difficulties difficulties in everyday everyday that tasks.can Inhelp this apaper paper wesuch present distributed architecture delivering Assistive Technologies user in tasks. Visually impaired people face many difficulties in everyday tasks. In this paper we present a distributed architecture for delivering Assistive Technologies that can help a user in such tasks. distributed architecture for delivering Assistive Technologies that can help a user in such tasks. The architecture allows for a user using aAssistive internet Technologies enabled device to can sendhelp images, andtasks. GPS distributed architecture delivering that a uservideo in such The architecture a userthat using internet enabled device send images, video The architecture allows a internet enabled device to send images, and GPS location to remoteallows “Helpers” canaaasend messages back with to helpful or and tips.GPS The The architecture allows a user user using using internet enabled device to send instructions images, video video and GPS location to remote “Helpers” that can send messages back with helpful instructions or tips. location to remote remote “Helpers” that can can send messages back withpeople helpfulinvited instructions oruser. tips. The The Helpers can be programs running on send a remote serverback or real by theor location to “Helpers” that messages with helpful instructions tips. The Helpers can be programs on server or people invited the The Helpers can is bebased programs running on aaa remote remote server using or real realMQTT peopleprotocol. invited by by the user. user.based The architecture on a running publish-subscribe paradigm A system, Helpers can be programs running on remote server or real people invited by the user. The architecture is based on a publish-subscribe paradigm using MQTT protocol. A system, based architecture is based on a publish-subscribe paradigm using MQTT protocol. A system, based on the proposed architecture, was implemented with a collision detector, a scene classifier and a architecture is based on a publish-subscribe paradigm using MQTT protocol. A system, based on the proposed architecture, was implemented with aa collision detector, aa scene classifier and aa on the proposed architecture, was implemented with collision detector, scene classifier and object recognized. A qualitative analysis of experiments have shown positive results and pointed on the proposed architecture, was implemented with a collision detector, a scene classifier and a object recognized. A object recognized. A qualitative qualitative analysis analysis of of experiments experiments have have shown shown positive positive results results and and pointed pointed out some improvements. object recognized. A qualitative analysis of experiments have shown positive results and pointed out some improvements. out some improvements. out some improvements. © 2016, IFAC (International Federation of Automatic Control) Hosting by Elsevier Ltd. All rights reserved. Keywords: System architectures; Computer vision; Distributed systems; Computer Keywords: architectures; vision; Distributed systems; Keywords: System System architectures; Computer vision; Distributed systems; Computer Computer applications; Obstacle detection; Computer Scene analysis; Object recognition. Keywords: System architectures; Computer vision; Distributed systems; Computer applications; Obstacle detection; Scene analysis; Object recognition. applications; Obstacle detection; Scene analysis; Object recognition. applications; Obstacle detection; Scene analysis; Object recognition. 1. INTRODUCTION • It enables the “Human-in-the-loop” paradigm by al1. •• It the “Human-in-the-loop” paradigm al1. INTRODUCTION INTRODUCTION It enables enables theinvited “Human-in-the-loop” paradigm bysend allowing a user person to receive data andby 1. INTRODUCTION • It enables the “Human-in-the-loop” paradigm by allowing a user invited person to receive data and send lowing a user invited person to receive data and send help when requested. This opens the door includlowing a user invited person to receive datafor and send The World Health Organization (WHO, 2014) estimated help when requested. opens the door for includThe World Health Organization (WHO, 2014) estimated help when requested. This opens the including in This assistive technology solutions. The World in Health Organization (WHO,285 2014) estimated helpsocial whennetworks requested. This opens the door door for for includin a survey 2012 that there are about million people The World Health Organization (WHO, 2014) estimated ing social networks in assistive technology solutions. in aathe survey in 2012 that there are about 285 million people ing social networks in assistive technology solutions. • It aims in providing an accessible and low-cost soin survey in 2012 that there are about 285 million people ing social networks in assistive technology solutions. world with that somethere kindare of about visual285 deficiency, being in a survey in 2012 million people •• It aims in providing an accessible and low-cost soin the world with some kind of visual deficiency, being It aims in providing an accessible and low-cost solution, since it does not require a specific device in the world with some kind of visual deficiency, being • It aims in providing an accessible and low-cost so246 million with partial sight and 39 million with total in the world with some kind of visual deficiency, being lution, since it does not require a specific device 246 million with partial sight and 39 million with total lution, since it does not require a specific device or sensor. In fact, our target is the use of common 246 million with partial sight and 39 million with total lution, since it does not require a specific device blindness. These people are constantly excluded because 246 million with partial sight and 39 million with total or sensor. In fact, our target and is the use of common blindness. These are constantly excluded because or In fact, our use devices, such smartphones tablets. blindness. These people people areAssistive constantly excludedsolutions because or sensor. sensor. In as fact, our target target is is the the use of of common common of their disability, but new Technology blindness. These people are constantly excluded because devices, such as smartphones and tablets. of their disability, but new Assistive Technology solutions devices, such as smartphones and tablets. • It is inspired in the ubiquitous computing of their disability, but new Assistive Technology solutions devices, such as smartphones and tablets. paradigm, aretheir beingdisability, researched and developed improve their social of but new AssistivetoTechnology solutions •• It in ubiquitous computing are being It is is inspired inspired in the thethat ubiquitous computing paradigm, which advocates technology shoudparadigm, be omare being researched researched and and developed developed to to improve improve their their social social • It is inspired in the ubiquitous computing paradigm, inclusion. are being researched and developed to improve their social which advocates that technology shoudorbe bespread ominclusion. which advocates that technology shoud nipresent (pervasive), invisible in devices inclusion. which advocates that technology shoud be omominclusion. nipresent (pervasive), invisible in devices or spread Recent advances in computer vision, electronic devices, nipresent (pervasive), invisible in devices or spread over the environment and its usage should be transnipresent (pervasive), invisible in devices or spread Recent advances in computer vision, electronic devices, Recent advancesand in wireless computer vision, electronic electronic devices, over thetoenvironment and its usage should be transmicroprocessors communication motivate the Recent advances in computer vision, devices, over and parent the user. over the the environment environment and its its usage usage should should be be transtransmicroprocessors and wireless communication motivate the microprocessors and wireless communication motivate the parent to the user. cientific comunity to search new solutions that combine microprocessors and wireless communication motivate the parent to to the the user. user. parent cientific comunity to search new solutions that combine We also present a system implementation based on the cientific comunitytoto toimprove search new new solutions thatof combine combine such technologies the quality of life visually cientific comunity search solutions that We also present aa system implementation based on the such technologies to improve the quality of life of visually We also present system implementation based on proposed architecture with three functionalities: collision such technologies to improve the quality of life of visually the impaired people. to Forimprove an assistive technology to visually have a We also present a system implementation based on the such technologies the quality of life of proposed architecture with three functionalities: collision impaired people. For an assistive technology to have a proposed architecture architecture with three three functionalities: collision detection, scene classification and functionalities: object recognition. Figimpaired people. For issues an assistive assistive technology to such haveas:aa proposed with collision positive affect, several need totechnology be explored, impaired people. For an to have scene classification and object Figpositive affect, issues explored, as: detection, scene classification and recognition. Figure 1 illustrate example where a blindrecognition. user is warned positive affect, several several issues need need to to be be explored, such such as: detection, detection, scene an classification and object object recognition. Figwhich paradigms and computational architectures should positive affect, several issues need to be explored, such as: ure 1 illustrate an example where a blind user is warned which paradigms and computational architectures should ure 1 illustrate an example where a blind user is warned on the presence of an obstacle and the object is properly which paradigms and computational computational architectures should be employed; which devices should architectures be used; how the ure 1 illustrate an example where a blind user is warned which paradigms and should of obstacle the properly be employed; which devices be on the the presence presence of an an obstacle and the object object isimproving properly recognized, allowing him a safer and locomotion andis be employed; which should devicesbeshould should be used; used; how how the the on on the presence of an obstacle and the object is properly available technologies combined. be employed; which devices should be used; how the recognized, allowing him a safer locomotion and improving available technologies should be combined. recognized, allowing him a safer locomotion and improving environment awareness. available technologies should be combined. recognized, allowing him a safer locomotion and improving available technologies shouldproposes be combined. environment awareness. awareness. In this context, our paper a distributed archi- environment environment awareness. In this context, our paper proposes aa distributed archiIn this context, our paper proposes distributed archi- This paper is organized as follows: section 2 describes some tecture based on publish/subscribe to support In this context, our paper proposesparadigm a distributed archiThis paper is organized organized as presents follows: section section describes some tecture on paradigm to paper is 222 describes some related works; section 3as the proposed architectecture based on publish/subscribe publish/subscribe paradigmpeople. to support support This paper is organized as follows: follows: section describes some assistivebased technologies for visually impaired The This tecture based on publish/subscribe paradigm to support related works; section 3 presents the proposed architecassistive technologies for visually impaired people. The related works; section 3 presents the proposed architecture; in section 4 the implemented system, based on the assistive technologies for visually visually impairedare: people. The The related works; section 3 presents the proposed architecmain features of the proposed architecture assistive technologies for impaired people. ture; in section 44 the implemented system, based on the main features of the proposed architecture are: ture; in section the implemented system, based on proposed architecture, is detailed; in section 5 we report main features of the proposed architecture are: in section 4 the implemented system, based on the the main of the architecture are:from other ture; proposed architecture, is detailed; in section 5 we report • Itfeatures is flexible andproposed customizable, differing proposed architecture, isa detailed; detailed; in section section we report some experiments and is qualitative analysis;55 we finally, in proposed architecture, in report •• It is flexible and customizable, differing from other It is flexible flexible and customizable, customizable, differing from other experiments and aaconclusion. qualitative analysis; finally, in works that target a specific typediffering of assistance. Our some • It is and from other some experiments and qualitative analysis; finally, in section 6 we present our some experiments and a qualitative analysis; finally, in works that target aa specific type Our works that targetthe specific type of of assistance. assistance. Our section 66 we present our conclusion. solution enables communication between users works that target a specific type of assistance. Our section we present our conclusion. section 6 we present our conclusion. solution enables the communication between users solution enables the communication communication between users and different “Helpers” by defining a between underlying ar2. RELATED WORK solution enables the users and different “Helpers” by defining aa underlying ar2. RELATED WORK and different “Helpers” by defining underlying ar2. chitecture for message exchange. and different “Helpers” by defining a underlying ar2. RELATED RELATED WORK WORK chitecture for message exchange. chitecture for for message message exchange. exchange. Researches related to Assistive Technology for the visually chitecture Researches related to Assistive Technology the visually The authors thank the financial support of CAPES, the Brazilian Researches related to Technology for the impaired are in constant development. Thefor are Researches related to Assistive Assistive Technology forfollowing the visually visually impaired are in constant development. The following are authors thank the financial support of CAPES, the Brazilian The impaired are in constant development. The following reported some examples of recent works in this area. Coordination of Higher Education Personnel Improvement. The authors thank the financial support of CAPES, the Brazilian impaired are in constant development. The following are are The authors thank the financial support of CAPES, the Brazilian reported some examples of recent works in this area. Coordination of Higher Education Personnel Improvement. reported some examples of recent works in this area. Coordination of Higher Education Personnel Improvement. reported some examples of recent works in this area. Coordination of Higher Education Personnel Improvement. Copyright © 2016, 2016 IFAC 150Hosting by Elsevier Ltd. All rights reserved. 2405-8963 © IFAC (International Federation of Automatic Control) Copyright 2016 IFAC 150 Peer review© of International Federation of Automatic Copyright © 2016 IFAC 150 Copyright ©under 2016 responsibility IFAC 150Control. 10.1016/j.ifacol.2016.11.145
2016 IFAC TA November 6-9, 2016. Porto Alegre, Brazil Ricardo N. Rodrigues et al. / IFAC-PapersOnLine 49-30 (2016) 150–155
Fig. 1. Illustration of an application based on the proposed architecture.
In (Cha et al., 2013) is proposed a navigation system for blinds using the Smartphone. The application uses the Google Map and Google TTS (Text-to-Speech) to lead the user to the destination. The system is basically divided into three stages. You must first inform the destination, then the voice recognition and Google TTS is applied, if the user is recognized by the system, then, in second step Google Map search and traces the route, the map is saved in the DB (database). Finally, the person is guided by the route through voice instructions to your destination. Varadarajan et al. (2014) system based at the principle Wireless Position System. Where the current position of the user is calculated point-to-point according to the level of the WiFi signal. The destination must be informed, in accordance with the current position is loaded a map where the person is guided by voice instructions, to the reach the destination a voice notification is issued stating that the destination has been reached. The Nakajima and Haruyama (2012) uses communication technology visible light and geomagnetic sensing integrated into Smartphone. This consists of LED lights, a Smartphone with integrated receiver, headphones and a Panasonic cloud environment (LBS Platform) which is referred to as position information base. The Receiver abstracts the identification of visible light by Bluetooth to find the user’s location, via WiFi receives the route the cloud and the position information for guidance is given through headphones. The geomagnetic sensor serves to guide the direction of travel.
151
Fig. 2. Architecture overview.
We observe that most works found in the literature focus in a single objective, as object recognition (Tapu et al., 2013), localization and navigation (Cha et al., 2013; Du Buf et al., 2011; Varadarajan et al., 2014; Nakajima and Haruyama, 2012) or object detection (Du Buf et al., 2011; Tapu et al., 2013). Few works consider issues related to the architecture of a computational Assistive technology (Ghorbel et al., 2015; Murua et al., 2011; Shojanoori et al., 2014). We also observe that most of the works propose the development of electronic gadgets for a specific target application (Jain, 2014; Tang and Li, 2014; Treuillet and Royer, 2010). This approach has the drawback that it usually result in a costly system with limited application scope. On the other hand, we defend the use of general purpose devices, largely accessible to a wide range of the population (e.g. smartphones, tablets), like in the works of (Cha et al., 2013; Tapu et al., 2013). Finally, we observe that few works works focus on web based technologies (Murua et al., 2011; Cha et al., 2013) which could expand the available computational resources. Most works are based in local processing. 3. ARCHITECTURE The proposed architecture can be divided in three main components as shown in Figure 2. The client device is responsible for the user interface as well as sensor data acquisition. The Publish-Subscribe middleware provides secure communication and the Helpers provide accessibility functionalities by receiving the data sent by the client, processing it, and sending back a text message to help the user. More details about each component is provided next.
Tapu et al. (2013) present a system that basically divides into obstacle detection (static and dynamic) and classification of objects to help visually impaired navigation through Smartphone. The detection is made based on the optical flow algorithm Lucas-Kanade, homographic transforms, the RANSAC (Random Sample Consensus) algorithm, agglomerative clustering technique, KNN (KNearest Neighbor). The classification is based on HOG (Histogram of Oriented Gradients) descriptor, K-means and BoVW (Bag of Visual Words) and has an image database on the device itself.
3.1 Client Device
Du Buf et al. (2011) developed a navigation system for outdoor and indoor, integrated by GPS and WiFi localization with a geographic information system (GIS) database, with RFID tags in sidewalks, still has the obstacles detection obstacle with computer vision.
The sensors available may vary among devices. Common sensors available in most smartphones and tablets include GPS, camera, microphone, accelerometer, light sensor, etc. Depending on the functionality, the data from these sensors may be requested continuously, generating a data
151
This component is responsible for providing an accessible interface to the user and for managing data acquisition from sensors attached to the device. The device will be composed of a hardware and a software part. The hardware can be, for example, a tablet (as implemented in our study-case), a smartphone or a dedicated device developed over Raspberry Pi board. Meanwhile, the software will be specifically developed to provide accessible user interface and other functionalities.
2016 IFAC TA 152 Ricardo N. Rodrigues et al. / IFAC-PapersOnLine 49-30 (2016) 150–155 November 6-9, 2016. Porto Alegre, Brazil
stream, or sporadically as needed, which will usually be triggered by a user request. For example, a collision detection functionality may have to use the camera continuously, while a image describer may rely on a single image, which is captured over user request.
Fig. 3. Example of MQTT topics with two users. Rectangles represent nodes and ellipses represent topics.
We assume the device is capable of performing Text-toSpeech (TTS). In other words, the device should be able to transform a given text string into a speech and deliver this speech to the user. The use of bone conduction earphones has been considered an interesting solution to deliver the sound to the user without disturbing the environment and with no reduction to auditory perception by the user. The accessible functionalities made available to the user are provided by Helpers that may run locally on the client device or in a remote device which is accessed via a MQTT client as detailed next. 3.3 Helper 3.2 Publish-Subscribe Middleware This component encompass the network infrastructure, including hardware and protocols, to deliver a publishsubscribe communication between network nodes. We use the Message Queue Telemetry Transport (MQTT) Locke (2010) protocol over internet. This protocol has been used in several applications of internet of things and robotics. When compared to other publish-subscribe architectures, such as ROS Quigley et al. (2009), the MQTT protocol has the advantage of being very efficient, both considering data overhead and computational resources. When the user activates some functionality, the client device uses MQTT protocol to publish the sensors data to topics which are subscribed by one or more Helpers. There is one topic for each sensor, and they are only published if a functionlity that uses they data is triggered by the user. Published messages are sent to a MQTT server, called the Brocker, which manages subscriptions and topics. An important feature implemented in the Brocker is Access Control List (ACL), which limits the access to topics by a list of users that are allowed to publish or subscribe on it Uzunov (2016). We use this feature to restrict the access to a user topic only to the user and the Helpers that the user has activated. Figure 3 shows an example of topics structure for two users and two Helpers. Note that the root topic is the username, which is set in the ACL as the only user allowed to publish on those topics. The Helpers are allowed access only to message and sensors topics when activated by the user. All users are authenticated via username/password when connecting to the MQTT brocker. Each sensor is published in one topic and the message is received in the message topic. There is also a config topic that is used to set user’s parameters that may be used by Helpers. Another MQTT security feature used in our proposed architecture is SSL encriptyion, which encrypt the messages sent among nodes. We consider that, with the use of ACL and SSL, the architecture can provide a good level of security and privacy. 152
This component receives the data sent by the client, process it, and send a text message to help the user. It may be an autonomous software, running an algorithm, or a person. In either case, the Helper should provide some aid to the user by analysing the received data and sending a helpful message. Some examples of functionalities that may be provided by Helpers are: • • • • • • •
Image description; Path navigation; Obstacle avoidance; Text recognition (OCR); Facial recognition; Object detection; Money recognition.
Software based Helpers may run locally on the client device or remotely in a server. Local Helpers have the advantage of not requiring network communication, but may be restricted by low computational capacities provided by the client device. For example, modern machine learning techniques such as deep neural networks require a significant amount of memory, so they are best suited to run remotely. Moreover, remote Helpers can be updated without the need of updating each client device. An important aspect is that the output text message provided by the Helper should be easy to understand, since it will be directly converted to speech in the client device without any kind of modification. 3.4 Data flow Figure 4 depicts the information flow, using the Business Process Model Notation, over the three components for an example when the user (Bob) captures an image. We assume two remote Helpers are active. The client device captures a frame from the camera, process it and publish to a topic using MQTT. The image will be received by the MQTT brocker, which will use the ACL to make sure the user is allowed to publish in the topic; the image is then sent to the remote Helpers, which receive the image, process it, generate a text message and publish to the user’s message topic.
2016 IFAC TA November 6-9, 2016. Porto Alegre, Brazil Ricardo N. Rodrigues et al. / IFAC-PapersOnLine 49-30 (2016) 150–155
153
Fig. 4. Data flow example.
4. IMPLEMENTATION We have implemented a system based on the proposed architecture with three functionalities: (1) Collision detection. We use a collision detection method as described in Canez (2016), that process the video stream and detects possible obstacles that may collide with the user during locomotion. The method generates a warning message when a possible collision is detected. (2) Scene classifier. This classifier analyses an image and returns the type of ambient the image was taken. Possible outputs include office, florest, parkway, kitchen, bedroom, etc. The method is based on Deep Neural Networks as described in Zhou et al. (2014) and was implemented using Caffe deep learning framework Jia et al. (2014). (3) Object recognition. We use the method described in Szegedy et al. (2015) to recognize specific objects, such as: computer, car, bed, t-shirt, cup, etc. As in the scene classifier, this method was also implemented using Caffe framework. Each of these functionalities is implemented as a Helper that may be activated or deactivated by the user. The collision detection runs locally on the client device, while the scene classifier and object recognition runs in a remote 153
server. When a remote Helper is deactivated it unsubscribe to the users topic and, consequently, do not receive or send messages anymore. For the Publish-Subscriber middleware, we have used the Mosquito MQTT as the broker and the Paho library in the client side implementation. A client was implemented in C++ using Qt Framework for user interface, OpenCv library for image handling and Microsoft .NET framework for Text to Speech. The client was deployed in a tablet with an Intel Atom processor with 1GB of RAM. Figure 5a shows the main user interface, which contains a button to capture an image and a button to configurations - which leads to screen shown in Figure 5b. The image capture command captures an image and publish it to the snapshot topic. The configuration menu contains buttons that allows the user to activate or deactivate the Helpers. The interface contains accessible features that enable a blind user to use it. For example, when a button is clicked once, a voice is synthesized givin the button description; when clicked twice, the button is triggered. 5. EXPERIMENTS In this section we present some usage experiments conducted with the system implemented in the previous
2016 IFAC TA 154 November 6-9, 2016. Porto Alegre, BrazilRicardo N. Rodrigues et al. / IFAC-PapersOnLine 49-30 (2016) 150–155
Fig. 5. User interface. a) Main screen; b) Configurations screen
section. These experiments were performed with seven healthy users, which were asked to close their eyes during the experiments. The objective is to get a qualitative feedback for future system improvements. Future experiments will be conducted with visually impaired persons in real world scenarios.
6. CONCLUSION In this paper we have proposed a new architecture for delivering Assistive Technologies to visually impaired people using a publish-subscribe architecture. The architecture objective is to provide a bridge between users and Helpers, which are algorithms or people that provide useful functionalities by processing the user’s data and sending back a text message that is synthesized using Text-To-Speech. We believe that the main advantage of this architecture is its flexibility to incorporate new sensors and functionalities. The initial results have shown good results, based on a qualitative assessment, and pointed out some improvements that should be corrected, specially regarding the efficiency of recognizers and detectors. As future works we intend to add social network functionalities in the system, improve the provided Helpers and add new ones. We are also planning real world experiments. REFERENCES
Experiment 1: Users were asked to randomly walk in a hallway using the collision detection Helper functionality.
Qualitative analysis: Overall the users reported that felt more confident with the aid of the collision detector. However, it was observed that false positives occur when turning or moving abruptly, which caused certain stress in the user. False negative errors occurred when the obstacle is homogeneous, such as a white wall, and in translucent glass. The users suggested the system could issue a progressive warning that increase its intensity according to the distance to the obstacle and give some hint to where is the obstacle located.
Experiment 2: Users were asked to randomly test the scene classifier and object recognizers in their everyday tasks.
Qualitative analysis: The scene classifier got positive evaluations, being able to give a general classification for the scene most of the time. However, the object recognizer presented more errors. We observed that objects where only recognized if they where fully framed as the only object in the image, which is not easily done without sight. We are currently investigating methods that can perform object detection, such as Ren et al. (2015), which will not only recognize the object but also find its location even in cluttered scenes. The Experiment 2 was performed over a wi-fi internet connection and over a 4G mobile network. It was observed that the latency between requesting help and receiving the message was around 1 (one) second in the wi-fi network and around 2 (two) seconds in the 4G mobile network, which was considered to be acceptable by the users. 154
Canez, A. (2016). Collision detection with a single camera. Technical report, Universidade Federal do Rio Grande, FURG. Cha, J.S., Lim, D.K., and Shin, Y.N. (2013). Design and implementation of a voice based navigation for visually impaired persons. International Journal of Bio-Science and Bio-Technology, 5(3), 61–68. Du Buf, J.H., Barroso, J., Rodrigues, J.M., Paredes, H., Farrajota, M., Fernandes, H., Jos´e, J., Teixeira, V., and Saleiro, M. (2011). The smartvision navigation prototype for blind users. Ghorbel, M., Mokhtari, M., and Renouard, S. (2015). A distributed approach for assistive service provision in pervasive environment. Proceeding WMASH 06 Proceedings of the 4th international workshop on Wireless mobile applications and services on WLAN hotspots. Jain, D. (2014). Path-guided indoor navigation for the visually impaired using minimal building retrofitting. ASSETS 14 Proceedings of the 16th international ACM SIGACCESS conference on Computers accessibility. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell, T. (2014). Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093. Locke, D. (2010). Mq telemetry transport (mqtt) v3. 1 protocol specification. IBM developerWorks Technical Library], available at http://www. ibm. com/developerworks/webservices/library/wsmqtt/index. html. Murua, A., Gonzalez, I., and Gomez-Martinez, E. (2011). Cloud-based assistive technology services. Computer Science and Information Systems (FedCSIS), 2011 Federated Conference on. Nakajima, M. and Haruyama, S. (2012). Indoor navigation system for visually impaired people using visible light communication and compensated geomagnetic sensing. In Communications in China (ICCC), 2012 1st IEEE International Conference on, 524–529. IEEE. Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., and Ng, A.Y. (2009). Ros: an
2016 IFAC TA November 6-9, 2016. Porto Alegre, Brazil Ricardo N. Rodrigues et al. / IFAC-PapersOnLine 49-30 (2016) 150–155
open-source robot operating system. In ICRA workshop on open source software, volume 3, 5. Kobe, Japan. Ren, S., He, K., Girshick, R., and Sun, J. (2015). Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems, 91–99. Shojanoori, R., Juric, R., Lohi, M., and Terstyanszky, G. (2014). Asecs: Assistive self-care software architectures for delivering service in care homes. 2014 47th Hawaii International Conference on System Sciences. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015). Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1–9. Tang, T.J.J. and Li, W.H. (2014). An assistive eyewear prototype that interactively converts 3d object locations into spatial audio. Proceedings of the 2014 ACM International Symposium on Wearable Computers. Tapu, R., Mocanu, B., Bursuc, A., and Zaharia, T. (2013). A smartphone-based obstacle detection and classification system for assisting visually impaired people. In Proceedings of the IEEE International Conference on Computer Vision Workshops, 444–451. Treuillet, S. and Royer, E. (2010). Outdoor/indoor vision based localization for blind pedestrian navigation assistance. Int. J. Image Grap. 10, 481. Uzunov, A.V. (2016). A survey of security solutions for distributed publish/subscribe systems. Computers Security, –. doi: http://dx.doi.org/10.1016/j.cose.2016.04.008. Varadarajan, V., Kolapkar, S., Saini, M., and Dange, S. (2014). Blindnav: A smartphone based navigation system for the visually impaired. Compusoft, 3(4), 758. WHO (2014). Visual impairment and blindness. fact sheet, n. 282, 2014. URL http://www.who.int/mediacentre/factsheets/fs282/en/. Zhou, B., Lapedriza, A., Xiao, J., Torralba, A., and Oliva, A. (2014). Learning deep features for scene recognition using places database. In Advances in neural information processing systems, 487–495.
155
155