A framework for collaborative computing and multi-sensor data fusion in body sensor networks

Information Fusion xxx (2014) xxx–xxx

Contents lists available at ScienceDirect

Information Fusion journal homepage: www.elsevier.com/locate/inffus

A framework for collaborative computing and multi-sensor data fusion in body sensor networks Giancarlo Fortino a,⇑, Stefano Galzarano a, Raffaele Gravina a, Wenfeng Li b a b

Department of Informatics, Modeling, Electronics and Systems (DIMES), University of Calabria, Via P. Bucci, 87036 Rende, CS, Italy School of Logistics Engineering, Wuhan University of Technology, 430063 Wuhan, China

a r t i c l e

i n f o

Article history: Available online xxxx Keywords: Body sensor networks SPINE Collaborative computing Multi-sensor data fusion Emotion detection Handshake detection

a b s t r a c t Body Sensor Networks (BSNs) have emerged as the most effective technology enabling not only new e-Health methods and systems but also novel applications in human-centered areas such as electronic health care, fitness/welness systems, sport performance monitoring, interactive games, factory workers monitoring, and social physical interaction. Despite their enormous potential, they are currently mostly used only to monitor single individuals. Indeed, BSNs can proactively interact and collaborate to foster novel BSN applications centered on collaborative groups of individuals. In this paper, C-SPINE, a framework for Collaborative BSNs (CBSNs), is proposed. CBSNs are BSNs able to collaborate with each other to fulfill a common goal. They can support the development of novel smart wearable systems for cyberphysical pervasive computing environments. Collaboration therefore relies on interaction and synchronization among the CBSNs and on collaborative distributed computing atop the collaborating CBSNs. Specifically, collaboration is triggered upon CBSN proximity and relies on service-specific protocols allowing for managing services among the collaborating CBSNs. C-SPINE also natively supports multi-sensor data fusion among CBSNs to enable joint data analysis such as filtering, time-dependent data integration and classification. To demonstrate its effectiveness, C-SPINE is used to implement e-Shake, a collaborative CBSN system for the detection of emotions. The system is based on a multisensor data fusion schema to perform automatic detection of handshakes between two individuals and capture of possible heart-rate-based emotion reactions due to the individuals’ meeting. Ó 2014 Elsevier B.V. All rights reserved.

1. Introduction In the last years, the progress of science and medicine allowed to considerably augment the average life expectancy. On the basis of recent studies, in 2050 life expectancy will be 80 years for men and 85 years for women whereas the population of the World having more than 65 years is projected to augment from 500 millions to one billions in 2030 [1]. The augmentation of elderly population will largely and specifically affect any health care system. At the same time, especially in the more developed countries, there is an always growing interest in maintaining, and improving the quality of life, and consequently health and wellness. ICT technologies and, in particular, domain-specific Wireless Sensor Networks [2] named Wireless Body Sensor Networks (BSNs) [3,4] have enormous possibilities for positively affecting the daily life of people. A BSN is constituted by a set of programmable wearable sensors that ⇑ Corresponding author. Tel.: +39 0984 494063; fax: +39 0984 494713. E-mail addresses: [email protected] (G. Fortino), [email protected] (S. Galzarano), [email protected] (R. Gravina), [email protected] (W. Li).

communicate with a local and/or personal coordinator device (or simply coordinator) to provide real-time, continuous and noninvasive monitoring of assisted livings. The sensor nodes include computation, memory, and wireless communication capabilities, a constrained power supply (usually a battery), and different on-board sensors based on specific physical transducer(s). Usual dimensions of physiological sensing, for strictly medical and non-medical purposes, include body motion, skin temperature, heartbeat rate, muscular activity, breathing volume and rate, skin conductivity, and brain activity. Wearable sensors can be positioned on the skin and/or in the garments. The coordinator is usually a tablet/smartphone or a personal computer, and can specifically enable monitoring in realtime, remote storage of long term, and analysis both online and offline. BSNs can facilitate and empower many human-centered application domains such as elderly assistance at home, early prevention or detection of diseases (e.g. heart attacks, Parkinson, diabetes), post trauma rehabilitation after a surgery, detection of gestures and motions, fitness, medical assistance in disaster events, cognitive and emotion recognition. Moreover, BSNs are effective

http://dx.doi.org/10.1016/j.inffus.2014.03.005 1566-2535/Ó 2014 Elsevier B.V. All rights reserved.

Please cite this article in press as: G. Fortino et al., A framework for collaborative computing and multi-sensor data fusion in body sensor networks, Informat. Fusion (2014), http://dx.doi.org/10.1016/j.inffus.2014.03.005

2

G. Fortino et al. / Information Fusion xxx (2014) xxx–xxx

enablers for a wide range of application areas, e.g. e-Wellness, eFitness, and e-Sport, in which the goal is not related to the disease detection and monitoring, but rather to support people maintaining physical and mental wellness. e-Factory is a novel application domain too where BSNs could provide many benefits; applications in such domain have the purpose of monitoring activities of workers, such as in the context of production chains, to support safety and drive the correct product manufacturing. BSNs can finally be key to facilitate social physical and virtual interactions as they could be used to detect, recognize and monitor emotion states of people during a meeting, so enabling specific services that rely on (mutual) emotions. The design and implementation of BSN-based applications and systems are complex and time-consuming tasks. Complexity mainly arises from programming intensive signal-processing algorithms for synchronized data interpretation and event/state classification on wireless sensors that have got very limited computation, storage and communication resources and need to address specific requirements concerning wearability and energy consumption. This is particularly stimulating as BSN systems commonly require high rates for sensor sampling that affect real-time signal processing and communication as available bandwidth and computation power have strong limitations. Nowadays, BSNs are commonly used for the monitoring of single assisted livings and the BSN systems currently developed are primarily centered on a single-hop wireless star network formed of a collection of wearable sensors managed by a coordinator device. In particular, BSN frameworks (e.g. SPINE [5], SPINE2 [6], CodeBlue [7], Titan [8], and RehabSpot [9]) aim at efficiently enabling the programming of wireless sensors and supporting the communications between the coordinator and the sensors to implement systems for remote, real-time monitoring of assisted livings. Nevertheless, in different areas (e.g. health care, sport, emergency, entertainment, factory, and social interaction), different types of BSN architectures can be defined where the single assisted livings’ monitoring is not sufficient to meet the application requirements. In fact, collaborative applications require that BSNs are able to interact and synchronize with each other for recognizing group activity, detecting events sensed by groups of people, monitoring single and multiple individuals, etc. We therefore define Collaborative Body Sensor Networks (CBSNs) as a novel kind of BSN architectures to allow BSNs interacting and collaborating among them to facilitate the implementation of collaborative systems in which both the monitoring of single assisted livings is enabled and the management of data communication and collaborative processing among BSNs is provided. To fully support CBSNs, novel frameworks are to be defined that, apart from typical mechanisms of the aforementioned conventional BSN frameworks, include specific programming abstractions and mechanisms: (i) inter-BSN data communication; (ii) BSN Proximity Detection; (iii) BSN mutual service discovery and activation; (iv) inter-BSN high-level protocols; and (v) cooperative multisensor data fusion. To this purpose we propose the Collaborative SPINE (C-SPINE) framework, an extension of the well-known SPINE middleware [5,10,11], for the development of applications based on CBSNs. In a previous paper [12], we have just introduced some basic concepts about CBSNs and sketched the architecture of C-SPINE along with the e-Shake system based on C-SPINE. Specifically, in this paper, we present a full-fledged description of C-SPINE by providing details about all components that concur to support basic and extensible services for enabling collaborative computing and multi-sensor data fusion across CBSNs. We also define a multi-sensor data fusion architecture that is effective in supporting collaborative real-time data processing and analysis among CBSNs for classification and detection of joint events.

To demonstrate the effectiveness of C-SPINE, a system, named e-Shake and based on C-SPINE, for emotion detection among meeting people is also described in detail and its accuracy evaluated in a controlled environment. In particular, the system integrates a collaborative and multi-sensor data fusion-based handshake detection process, which is a reverse engineered version of the system presented in [13], with an ECG-based heart rate monitoring component to analyze the variability of the heart rate during the handshakes between two people for detecting emotion reactions. This paper is organized as follows. In Section 2 reference architectures, programming frameworks and sensor data fusion methods for the design and implementation of BSN systems are discussed. In Section 3, first a taxonomy of collaborative services that can be provided by CBSNs is introduced to elicit a set of important and novel requirements for CBSNs; then, a reference architecture for collaborative computing and multi-sensor data fusion in CBSNs is defined to fulfill such novel requirements. CSPINE, which actually implements the reference architecture defined in Section 3, is described in detail in Section 4. Section 5 presents the e-Shake CBSN system along with its accuracy evaluation. Finally, conclusions summarize the main achievements of our research, and current and future research activities are briefly discussed.

2. Background and related work BSNs [3] represent a specific class of WSNs in which one or multiple wireless, non-invasive wearable sensors are applied to the human body to monitor several physiological signals. Thanks to their dimensions and wireless communication capabilities, BSNs can enable continuous monitoring of several vital signs and other physiological parameters (e.g. cardiac activity, respiratory rate, muscular tension, skin conductivity, blood oxygen and glucose level, body movements) without interfering with daily life activities of the assisted living. This emerging technology is offering new and very diversified human-centered application scenarios. A possible application domain taxonomy is the following:  e-Health, includes prevention or early detection of diseases, remote assistance at home, physical and mental rehabilitation, activity recognition and gait analysis [14,9];  e-Emergency, includes applications supporting disaster response teams in the event of earthquakes, landslides, wildfires [7];  e-Entertainment, is mainly associated to human-computer interaction (HCI) systems for real-time facial, posture, gesture and activity recognition [15–17];  e-Sport, is an application domain similar to e-Health, although it is focused on e-Fitness and e-Wellness applications and on enterprise systems for fitness clubs and sport teams that offer BSN-based services of training monitoring for their athletes [18];  e-Factory, includes management and monitoring of industrial processes, safety of workers and support to collaboration among workers [19];  e-Sociality, mainly includes emotion and stress recognition applications to realize new patterns of social interactions among individuals [13]. In the following subsections, we first introduce different kinds of BSN architectures able to support diversified application scenarios, then overview and compare available programming frameworks for the development of BSN applications, and finally present a reference multi-sensor data fusion architecture for BSNs along with interesting methods for sensor data fusion in BSN systems.

Please cite this article in press as: G. Fortino et al., A framework for collaborative computing and multi-sensor data fusion in body sensor networks, Informat. Fusion (2014), http://dx.doi.org/10.1016/j.inffus.2014.03.005

G. Fortino et al. / Information Fusion xxx (2014) xxx–xxx

2.1. Logical BSN architectures Due to the diverse BSN application scenario requirements, different logical network architectures have been to date proposed. In this case the word ‘‘logical’’ is used to stress the difference from the physical network topologies a certain scenario implementation may require (e.g. a logical one-to-one communication might be implemented with multi-hop routing through intermediate network devices). A categorization of the logical network architectures is the following (see Fig. 1): a. Single Body – Single Base station (SBSB) (Fig. 1(a)) – a single BSN, composed of one or more wearable wireless sensors (WS), is coordinated by one base station (BS), often the user’s smart-phone or tablet. This architecture is the choice of many e-Health and e-Sport applications, as personal vital signs are usually processed, stored, visualized locally, and occasionally forwarded to remote servers for long-term analysis. b. Single Body – Multiple BSs (SBMB) (Fig. 1(b)) – a single BSN is able to communicate with multiple BSs. Many in-home applications require a user BSN to interact with different BSs (at different times and/or concurrently) within the local environment (e.g. with the users tablet for health monitoring and with a videogame console for gesture recognition). c. Multiple Bodies – Single BS (MBSB) (Fig. 1(c)) – a single BS is able to discover and coordinate multiple BSNs. A possible scenario is represented by a medical monitoring system designed for family doctors that use their PC or tablet to collect health status information from their patients as they arrive to the office. A further example is an e-Entertainment accessory based on wearable sensor kits for game console or smart-TVs for enabling augmented gaming or social experiences among groups of people. d. Multiple Bodies – Multiple BSs (MBMB) (Fig. 1(d)) – multiple BSNs interact with multiple BSs. Future enterprise BSN-based systems will require multi-party interactions among multiple BSNs. For instance, rescue teams during emergency response to disaster events place wireless medical sensors to the victims and wear themselves BSNs to monitor vital signs. In this scenario, several members of the different teams (e.g. all the team leaders) carry a smart-phone acting as coordinator

Fig. 1. BSN architectures: (a) SBSB, (b) SBMB, (c) MBSB, and (d) MBMB.

3

device (i.e. a BS) which is able to communicate with the various BSNs within radio range e.g. to be immediately informed if a victim condition suddenly becomes life-critical. 2.2. Programming frameworks BSN system programming is very complex due to the issues of designing and implementing intensive and efficient data processing algorithms for data interpretation and event classification on wireless sensors which possess limited resources and need to fulfill stringent requirements concerning computation, memory, communication, wearability and energy consumption. Thus, in order to provide programmers with abstractions higher than the ones offered natively by currently available sensor platform APIs, several BSN software frameworks have been developed and proposed so far. Although all of them fully support the SBSB architecture type, none of them is able to straightforwardly support the MBMB architecture type (see Section 2.1). SPINE [5,10,11] is an open-source, domain-specific middleware for effective development of BSN systems. SPINE supports distributed signal-processing BSN systems through a variety of built-in sensor drivers, signal-processing utilities, and flexible data communication protocols. In addition, its architecture has been specifically designed to simply integrate new customized sensor drivers and processing functionalities; to fulfill specific designer and developer needs, it allows to tailor and customize very quickly what is natively provided. A primary key idea of SPINE is the reuse of software components to allow for run-time sensor nodes configuration according to application-specific requirements. This way the developed in-node code could be reused for multiple applications without re-installing the sensor nodes offline when moving from an application to another. SPINE currently supports the most popular programmable sensor node platforms. The TinyOS [20] version of SPINE executes atop TelosB/Tmote Sky, MicaZ, and Shimmer [21], which supports IEEE 802.15.4 and Bluetooth. Furthermore, there exist SPINE implementations for: (i) ZigBee devices based on the TI Z-Stack [22]; (ii) the Java Sun SPOTs sensors [23]. SPINE2 [6] is a full platform-independent reverse engineering of SPINE and is conceived to support a task-oriented programming approach. According to such a paradigm, a system is effectively designed as a network of tasks. Tasks represent specific activities (e.g. sensing, elaboration, data transfer). The design of an application by composing basic building blocks enables a more rapid system programming, run-time re-configuration and easier software maintenance. The software architecture of SPINE2 has been designed by following the software layering approach. It consists of several platform-independent components constituting the core of the framework and a set of platform-dependent components to access the specific platform resources and services. Such an architecture makes the porting of SPINE2 to different C-like sensor platforms a faster and easier operation, since only the platform-dependent components must be re-implemented according to the target architecture specifications. Currently, SPINE2 is available for TinyOS [20] and Z-Stack [22]. CodeBlue [7] was designed to support several medical scenarios, from monitoring patients in medical centers to assisting victims of disasters. CodeBlue supports flexible communications to address both patients/victims and doctors/rescuers mobility and the temporary loss of radio connectivity. CodeBlue is based on a set of wearable medical sensors (Telos and MicaZ) and a software platform, built atop TinyOS, that was designed for integrating these wearable devices with coordinators such as PDAs and laptop computers. The platform allows for device discovery, event notification, and data transmission. The CodeBlue architecture uses a data routing framework based on the publish/subscribe paradigm. In particular, sensors publish important data to given channels and coordinator devices subscribe to channels of interest.

Please cite this article in press as: G. Fortino et al., A framework for collaborative computing and multi-sensor data fusion in body sensor networks, Informat. Fusion (2014), http://dx.doi.org/10.1016/j.inffus.2014.03.005

4

G. Fortino et al. / Information Fusion xxx (2014) xxx–xxx

Titan (Tiny task network) [8] is a framework specifically designed to support context recognition on sensor networks. A processing application is defined by a set of interconnected tasks constituting a service task graph which is executed by the framework runtime as a whole over the network. Anyway, each single task is actually mapped and executed by a specific sensor node. Each task represents a specific operation, like a sensor reading, a mathematical function or a classifier and all defined tasks are the basic blocks for building a Titan application. Through these programming methods, building applications with Titan is a quicker and more intuitive job with respect to traditional programming approaches. In particular, Titan provides a set of predetermined tasks, from which users can choose needed blocks and connect them through connections that are in charge of passing data (enclosed in proper packets) from one task to another. In case of two tasks placed on different nodes, the data transfer take places trough messages exchanged via an adhoc communication protocol. Each task is defined to implement a certain algorithm for data processing, a mathematical function or an operation to access to node hardware like an available sensor. After having defined the whole application, the task network is split into a set of task sub-networks of which each is assigned and executed on a single node. RehabSPOT [9] is a customizable BSN platform for limbs rehabilitation built atop the Sun SPOT motes [23]. In RehabSPOT, every sensor device runs the same embedded code, although each of them may execute different operations at runtime. RehabSPOT is based on the client-server paradigm. The server application runs of the coordinator PC; the client runs on each wearable node. In particular, RehabSPOT is organized in a three-tier architecture. The first tier is represented by the sensors, organized as a stand-alone mesh network. The second tier is the coordinator PC that is connected with the managed sensor devices, forming a star-topology network. The PC allows for real-time display and on-line processing. Finally, the third tier relies on an Internet infrastructure. This tier is designed so that data stored inside the PC can be uploaded to remote servers for offline analysis. Table 1 reports a comparison among the discussed frameworks. Only C-SPINE, proposed in this paper and detailed in Section 4, is able to support each architecture type of BSN systems. 2.3. Multi-sensor data fusion for BSNs To fully support multi-sensor data fusion [24] in BSNs, the following three layers can be identified: 1. Sensing: data is gathered from the on-body sensors. 2. Analysis: decisions are derived from sensed signals. 3. Dissemination: high-level information is given to the end-user applications. Such layers can be realized to run both on the sensor nodes and on the coordinator (see Section 2.1). In particular, Fig. 2 shows a reference three-layered architecture for multi-sensor data fusion that is usually exploited in the BSN context [25]. At the sensing

Table 1 Comparison of BSN frameworks w.r.t. native support provided to logical BSN architectures. Frameworks

SBSB

SBMB

MBSB

SPINE SPINE2 Titan CodeBlue RehabSPOT C-SPINE

U U U U U U

U U

U U

U

U

U

U

MBMB

U

Fig. 2. Reference three-layered architecture for multi-sensor data fusion.

layer, the sensor sampling module offers sensory signals to the feature extraction component which in turn processes them to extract features, e.g. min, max, mean, standard deviation, variance, etc. At the analysis layer, the feature selection module selects the most significant features and the feature fusion module joins together them. Finally, the decision fusion module provides decision-support from the input feature set (e.g. body posture classification using features extracted from body-worn accelerometer signals). The feature selection component is only enabled during the construction of the decision algorithm (e.g. a classifier) and is usually based on methods such as information gain or the increase in classification accuracy. Once selected the features, this component can be removed from the processing chain. At the dissemination layer, the module of event propagation transmits the information to application modules (either locally or remotely). Major research efforts in BSN area have been devoted to human activity recognition by delivering different multi-sensor data fusion methods for BSNs. They all adhere to the reference threelayered architecture for multi-sensor data fusion shown in Fig. 2 and they are all based on accelerometer sensor data. In [26], authors define a physical activity monitor by data fusion in BSNs to provide real-time information on the assisted living’ postures and activities. Data gathered from multiple accelerometers located on different areas of the body are fused to recognize and monitor activities. The system uses Kalman Filters (KFs) and Hidden Markov Models (HMMs). In particular, on the basis of data sensed from two-axial accelerometer sensors, the flexion angle and other parameters (angular velocity, and horizontal and vertical acceleration) of wearable devices placed on different locations of the human body (e.g. right and left thigh, ankle, forearm, or waist) are estimated by using KFs. The flexion angles of body parts (identified by different sensor node locations) estimated with the KFs are real-time indicators of limbs and torso position which are in turn used to infer the whole body posture or activity. Physical activity is recognized by fusing the information on the body parts. In order to obtain accurate decisions, features of different types (statistical, temporal and spectral) are processed on the flexion angles. Specifically, the body part features are: static/dynamic status computed by a threshold-based energy measure, periodical/

Please cite this article in press as: G. Fortino et al., A framework for collaborative computing and multi-sensor data fusion in body sensor networks, Informat. Fusion (2014), http://dx.doi.org/10.1016/j.inffus.2014.03.005

G. Fortino et al. / Information Fusion xxx (2014) xxx–xxx

non-periodical dynamic status are differentiated using the DFT, and periodical movements (e.g. walking, jumping, cycling, ascending and descending stairs) are identified based on banks of trained HMM. Finally, since physical activities involves multiple body parts, information such as flexion angles, static/dynamic status, periodical/non periodical dynamic status, and identified HMM are fused together to infer the activity or posture. Such information fusion is performed by accessing to an activity description table (constructed for each specific activity to recognize). The table is composed of items (table rows), each of which represents a predefined tuple of status values related to a body part and characterizing a part of activity. Thus, for each of the body parts, the obtained parameters are fused together to be compared with the entries of different activity description tables. The best matching is selected as physical activity identity. In [5,10], a different multi-sensor data fusion method for human activity recognition was proposed. Only two 3-axial accelerometers are used and placed on the right thigh and on the waist. Data are collected according to a fixed-length window with a fixed shift equal to 50% of the window. Specific features are extracted on the data windows: max, min and average on the XYZ axes of the accelerometer of the waist sensor, and max on the X axis of the accelerometer of the thigh sensor. Such features are then merged and used as inputs of a classifier (constructed by using a KNNbased decision tree) to identify four different activities: standing still, sitting, lying down, walking. Moreover, to identify falls, the total energy feature is computed on the XYZ axes of the accelerometer of the waist sensor node. Then, based on a pre-defined threshold, the system is able to detect a fall and, by merging such event with aforementioned activity recognition, is also able to provide information about how long the fallen person was lying down before standing up; this could give an estimation of the extent of the fall. Authors in [27] define a multi-sensor data fusion method that relies on the D–S evidence theory that allows to detect different human postures: sitting, squatting, standing still and lying down. Here 3-axial accelerometers are used and positioned on calf, thigh, arm and waist. On the basis of empirical evidence, ranges of the gravity acceleration were defined for each axis (XYZ) with reference to each aforementioned activity. The theory of evidence is then used to define basic trust functions for each posture and for combining (or fusing) the basic trust functions on the basis of the real-time observations to obtain new trust functions providing more accurate recognition of the four postures. Currently this method is being extended to movements by using the Transferable Belief Model which aims at enhancing the recognition accuracy. In [28], the author proposes a contextual sensor data fusion framework for BSNs, including different sensor fusion methods, to exploit the context in which people wearing BSNs are, for optimizing resources and enhance recognition accuracy. Specifically, BFFS (Bayesian Framework for Feature Selection), a feature selection technique centered on filters, is used for optimal sensor placement in BSNs. Feature selection is a dimensionality reduction technique that allows elimination of irrelevant or redundant features. In the context of BSNs, it can be used for identifying (statically or dynamically) sensors that have direct influences and/or major influences on the decision process. This allows to remove redundant information, so minimizing exploited resources, by minimally affecting the system performance in terms of final recognition accuracy. A novel multi-objective BFFS algorithm and an efficient search method for optimal solutions are proposed. The proposed objective measures include feature redundancy and network complexity. This technique can be exploited to enhance the fault tolerance of the network whilst minimizing the hardware and network resources. Moreover, a contextual multi-sensor data fusion method is proposed to recognize people’s activities by using sensor data

5

coming from specific contexts where people is performing such activities. The method is based on model learning and inferencing algorithms, in which the model structure is defined by learning the dependencies and causalities among the user context and observed variables and successively inferencing information from the local model structure based on selected features. Finally, such method is based on instantaneous information or simple temporal features such as signal energy. In context recognition, temporal information is indeed important for further enhancing the performance of the network, thus the method is empowered by incorporating temporal information into the local inferencing model by using a novel technique named Spatio-Temporal Self-Organizing Map. [25] proposes the exploitation of self-healing techniques to detect faults in collected data from sensors and filter faulty data to enhance their quality. This is extremely important as faulty data could affect the recognition accuracy at analysis layer. Specifically, different synthetic faults are introduced into a BSN system for human activity recognition to show how classification accuracy is affected by such faults. Finally, some methods (still to be implemented) are proposed to improve sensor data quality so improving recognition accuracy. Finally in [29], authors define an algorithm for information fusion of sensor data coming from different BSNs to provide global application robustness against individual sensor failures. In particular, the authors present the formulation of a latent structure influence model that is able to capture the correlation among different dynamic (even noisy/faulty) sensing processes so to improve classification accuracy. A BSN case study for real-time context recognition is described; the influence model is exploited to mine hidden structures among speech, location, activity, and posture information. Such discovery activity enable more accurate and robust classification of the assisted living status. The BSN system consists of a PDA, an environment sound recorder, and 2 accelerometers placed on the hip and the wrist. Such system is able to recognize in realtime 8 locations, 6 speaking/non-speaking states, 6 postures, and 8 activities. The recognition process is composed of two phases: (i) a pre-classifier (e.g. a Gaussian, mixture of Gaussians, or a decision tree) is performed on the features of sound and motion to derive a preliminary pre-classification result of each of the aforementioned parameters; (ii) the pre-classification results are given to an influence model to learn inter-node structures which are in turn needed to obtain the final post-classification result. Thus, by merging evidence across different classes with the influence model, classification accuracy for locations, speaking, body postures, and activities can sensibly increase. The aim of our proposal is not to define a specific method for multi-sensor data fusion but to provide a flexible and customizable framework for implementing CBSN applications based on multisensor data fusion and collaborative computing. Our framework fully supports the reference three-layered architecture for multisensor data fusion atop which the aforementioned research efforts (and of course many others) can be effectively mapped on. Moreover, our framework goes beyond a single BSN, it notably supports CBSN systems composed of multiple interacting BSNs by providing specific mechanisms for enabling features and decision exchange among CBSNs (see Section 3.3). 3. Collaborative body sensor networks CBSNs (Collaborative Body Sensor Networks) [12] are BSNs aiming to collaborate among them to implement cooperative systems targeting co-located groups of people. Currently, BSNs are mainly used for monitoring individual assisted livings in real-time, therefore available BSN programming frameworks are not suitable to support CBSNs as they lack of a specific reference architecture

Please cite this article in press as: G. Fortino et al., A framework for collaborative computing and multi-sensor data fusion in body sensor networks, Informat. Fusion (2014), http://dx.doi.org/10.1016/j.inffus.2014.03.005

6

G. Fortino et al. / Information Fusion xxx (2014) xxx–xxx

and related programming mechanisms to deal with inter-BSN collaborative work. So they do need to be enhanced and/or reverse engineered. Specifically, novel system architectures, efficient middleware, and multi-sensor data fusion methods have to be developed for supporting communication and collaboration among CBSNs. To this purpose, in the next three subsections, we first propose a taxonomy of services enabled by CBSNs motivating the needs to address new requirements; then, we define a reference architecture for CBSNs supporting collaborative services, and finally we design an enhancement of the architecture for BSN-oriented multi-sensor data fusion (see Section 2.3) for CBSNs. The obtained designs can be notably exploited as references models to implement new CBSN frameworks or to enhance already existing ones [5–9]. 3.1. Service taxonomy A CBSN service can be of three types (see Fig. 3): Client/Server, Broadcast and Collaborative. Client/Server services are based on the pull model where a BSN having the server role makes it available a set of services that BSNs, having the client role, can request and use. Two main types of client/server services are Assisted Living Monitoring and Individual Querying. The Assisted Living Monitoring type allows client BSNs to continuously monitor status and information provided by server BSNs in real-time. In particular, upon the client BSN request, the server BSN continuously pushes information to the client BSN, until the client BSN stops monitoring. Examples of Assisted Living Monitoring are health-care monitoring (rehabilitation, cardiac monitoring, etc.), in which a doctor can monitor the status of a patient for a given duration, and physical performance monitoring (sport training, energy expenditure, etc.), in which trainers can control the performance of their athletes during the training period. Conversely, the Individual Querying type provides typical client/server services: upon a client BSN request, the server BSN provides a single informative reply. For example, spot information about the individual’s status (health, fitness, wellness, emotion, etc.) can be provided. Broadcast services are based on the push model according to which BSNs can broadcast information without being queried. Information can be of two types: Informative Messaging and Alarm Events. The former allows to broadcast information about the individual wearing the BSN, e.g. emotion states, health conditions, performed activities, etc. The latter allows sending alarm events triggered by an individual’s critical status, such as a fall, heartbreak, high-degree of stress.

Collaborative services aim at capturing or enabling implicit or explicit activities or events generated by interactions involving groups. In particular, collaborative services are designed around the peer-to-peer model according to which two or more BSNs can interact (exchanging messages containing on-body sensed values) when they are in proximity. Two types of collaborative services have been devised: Group Activity Recognition and Group Event Detection. The former type aims at recognizing group activities such as handshakes between two people, mutual waving, multiuser activity. The latter type allows detecting critical events deriving from implicit or explicit multi-user interaction or group context sensing, e.g. proximity of a group of known people, emergency based on collective emotions (e.g. terror deriving from disaster event), etc. The presented CBSN service types pose the following new specific requirements that need to be fulfilled by CBSN application frameworks:  Inter-BSN communication. Both client/server, multicast and peer-to-peer service models need to rely on a robust mechanism for application-level inter-BSN communication.  Proximity detection of BSNs. All the service models require a basic mechanism that allows BSNs to detect other BSNs that are in proximity.  Discovery of BSN services. To use CBSN services of proximous BSNs, BSNs have to discover each other by using dynamic methods.  Selection of BSN services. To select a discovered CBSN service, BSNs need to know the specific protocol of such service.  Activation of BSN services. To actually use a selected CBSN service, BSNs have to execute it according to the specific service protocol.  Collaborative multi-sensor data fusion. For the rapid prototyping of collaborative services for group classification/detection, a distributed architecture which allows to collect, merge, and analize sensor data (both raw and pre-processed) coming from multiple BSNs is needed. To fulfill such requirements and facilitate the design of CBSN applications, in the following subsections, we propose a reference high-level architecture for inter-BSN collaborative computing able to support the development of the presented CBSN service types and a collaborative multi-sensor data fusion architecture specifically addressing the design of inter-BSN sensor data processing algorithms. 3.2. Reference architecture for collaborative computing In order to support collaborative computing among CBSNs, a reference architecture for CBSN is defined [12]. The CBSN architecture includes:  the Network Architecture, encompassing the communication layer that is formed by basic and application-specific interaction protocols.  the Functional Architecture, defining types and activities of functional blocks performing system management and running application-specific tasks.

Fig. 3. Service taxonomy for CBSNs.

The CBSN Network Architecture is shown in Fig. 4. In particular, a CBSN consists of a collection of wireless wearable sensors (WSs) managed by a basestation (BS) through a single-hop protocol based on a star topology [5]. At application-level, the BS manages the WSs using an intra-BSN OTA (Over-The-Air) protocol and communicates with other BSs through inter-BSN OTA protocols. If a CBSN is not equipped with a BS but only a set of WSs, other CBSNs could

Please cite this article in press as: G. Fortino et al., A framework for collaborative computing and multi-sensor data fusion in body sensor networks, Informat. Fusion (2014), http://dx.doi.org/10.1016/j.inffus.2014.03.005

G. Fortino et al. / Information Fusion xxx (2014) xxx–xxx

7

 Configuration for configuring the WS services currently discovered.  Control for enabling (de)activation, monitoring, control of the WS services actually configured.  Data transmission for sending raw and/or elaborated data to the BS and/or to other WSs placed in the same CBSN.

Fig. 4. A reference architecture for CBSNs.

directly interact with these WSs through their BS that uses the intra-BSN OTA protocol. Hereafter, we assume that each CBSN is full-fledged, i.e. it is coordinated by a BS. The intra-BSN OTA protocol provides the following functions:  Discovery for discovering the WSs that belong to the CBSN and retrieving the services (i.e. computing, communication, storing, sensing, actuating) which each WS provides.

The inter-BSN OTA protocols include basic and application-specific protocols. The inclusion of the basic protocols is mandatory as they enable proximity detection, service description, and service activation. In particular, the proximity detection protocol allows detecting neighbor CBSNs and managing the detected CBSNs. The service description protocol provides the list of the available applications and/or services that CBSNs can share with each other. The service activation protocol enables the selection and activation of collaborative services. Finally, the application-specific protocols support interaction among specific applications and services that run on CBSNs. Fig. 5 reports (i) a sequence chart encompassing proximity detection, service description and general-purpose service activation (and execution) between two CBSNs, and (ii) an activity diagram describing the service activity flow of a CBSN. In particular, when a CBSN senses a near CBSN (during proximity detection), it broadcasts a service description request (during service discovery), upon reception of the service description one or more services are selected (in the phase of service selection), activated (during service activation), and finally run (in the phase of service execution). The CBSN Functional Architecture, which is shown in Fig. 6, is composed of different layers installed at the BS-side:

Fig. 5. CBSN interaction and activity: (a) sequence diagram of high-level interaction among CBSNs; (b) activity diagram of basic CBSN tasks.

Please cite this article in press as: G. Fortino et al., A framework for collaborative computing and multi-sensor data fusion in body sensor networks, Informat. Fusion (2014), http://dx.doi.org/10.1016/j.inffus.2014.03.005

8

G. Fortino et al. / Information Fusion xxx (2014) xxx–xxx

3.3. Collaborative multi-sensor data fusion architecture

Fig. 6. CBSN functional architecture layers.

 CBSN Manager, handles the phases of proximity detection, service discovery, service selection and activation by using PDP, SDP, and SSAP protocols (see below). The activation of a service is automatic (i.e. upon service discovery, the service is activated) or on-demand (i.e. upon service discovery, the owner decides whether or not to activate the service). Automatic activation usually relies on mutual knowledge relationships among CBSN owners and on service classification;  BSN Manager, manages the set of WSs belonging to the CBSN through IPB (see below);  Service-specific Manager, manages a specific service through ASP (see below);  Proximity Detection Protocol (PDP), implements the mechanisms for CBSN proximity detection;  Service Description Protocol (SDP), implements the format for service description and service description exchange;  Service Selection and Activation Protocol (SSAP), which implements the mechanisms for selecting and activating a service;  Intra-BSN Protocol (IBP), implements mechanisms for coordinating the interaction between BS and the set of managed WSs;  Application-specific Service Protocol (ASP), which implements the mechanism for execution and management of a specific application service.

The single-BSN architectural model for multi-sensor data fusion described in Section 2.3 has been extended for supporting collaborative multi-sensor data fusion across CBSNs (see Fig. 7). Specifically, the introduced distributed components are: Features Exchange and Joint Decision Coordination. The former allows for synchronized distribution of features among the involved CBSNs (at least two). The latter distributes local decisions and coordinates the fulfillment of a shared joint decision upon which CBSN will agree. Moreover, the components Feature Fusion and Decision Fusion can receive and use features and local decision coming from other CBSNs. As stated in Section 2.3, it is worth noting that the feature selection component is only enabled during the definition of the decision algorithm (e.g. a classifier). Service-specific Managers (see Section 3.2), managing specific collaborative applications, can exploit such architecture as basis to support data analysis and classification of multi-user activity. This architecture is concretely implemented in the C-SPINE framework (see Section 4) and and an example of its application is actually incorporated in the e-Shake system (see Section 5) to recognize handshakes between two people.

4. The C-SPINE framework The CBSN reference architecture introduced in Section 3 has been fully developed and integrated within the SPINE framework [5,10,30,11] to provide Collaborative SPINE (C-SPINE), a fullfledged SPINE-based CBSN middleware. The C-SPINE architecture (see Fig. 8) includes the SPINE sensor-side (WS), the SPINE basestation-side (BS), and the CBSN architectural components (see Section 3.2). In particular, the specific components of C-SPINE are: – Inter-BSN Communication, which relies on the C-SPINE Inter-BSN OTA Protocol (CIBOP) to offer a modular and efficient communication level to the basic and application-specific services. – BSN Proximity Detection, which allows to efficiently detect neighbor BSNs. – BSN Service Discovery, which enables the discovery of services among the neighbor BSNs.

Fig. 7. Reference three-layered architecture for multi-sensor data fusion in CBSNs.

Please cite this article in press as: G. Fortino et al., A framework for collaborative computing and multi-sensor data fusion in body sensor networks, Informat. Fusion (2014), http://dx.doi.org/10.1016/j.inffus.2014.03.005

G. Fortino et al. / Information Fusion xxx (2014) xxx–xxx

9

Fig. 8. The C-SPINE architecture.

– BSN Service Selection & Activation, which incorporates semiautomatic procedures and rules to select and activate services among neighbor BSNs. – Application-specific Protocols & Services, which specifically support collaborative computing among BSNs to implement collaborative applications. We first briefly describe the basic components of SPINE in Subsection 4.1, then detail the C-SPINE basic components in Subsections 4.2, 4.3, 4.4, 4.5, and finally present the reference architecture for collaborative multi sensor data fusion supported by C-SPINE in Subsection 4.6. 4.1. Basic SPINE components The basic SPINE components [5] are included into: (i) the SPINE Node (WS), which is programmed using the embedded programming language specific to the sensor platform and runs on the wearable sensors, and (ii) the SPINE Coordinator (BS), which is implemented in Java (or in Android on Android-enabled mobile devices) and is executed on the coordinator station. The SPINE Node components are: – Node Management, which manages the interactions among the Sensor Control, In-Node Processing, and Intra-BSN Communication components, and forwards requests coming from the BS to the appropriate component. – Intra-BSN Communication, which handles message transmission and reception according to the SPINE Intra-BSN OTA Protocol (SIBOP), and manages the radio duty cycling. – Sensor Control, which performs as an interface to the platform physical sensors, scheduling timers if periodic sensing is requested from the BS, or executing one-shot readings on the requested on-board sensors. It interacts with the Data Buffers

to store the sensor readings. It also manages a sensor registry in which compiled sensor drivers auto-register at boot-time for permitting the other node modules to get the list of sensors. – Data Buffers, which are formed by circular buffers specifically used to store data sampled from on-board sensors. It offers two basic mechanisms to retrieve the sensor data: (i) ondemand by using ‘‘get’’ functions, and (ii) by means of listeners that are notified when newly generated data from specific sensors are available. – In-Node Processing, which involves an efficient, customisable and extensible set of signal-processing functions such as filters, data aggregators, and threshold-based alarms which can be executed on sensor data streams. This component fetches the to-be-elaborated data from the Data Buffers, and returns the results to the BS. The SPINE Coordinator components reused in C-SPINE are: – Intra-BSN Communication, which includes functionalities similarly belonging to the corresponding module in the WS and was defined to dynamically load the appropriate adapter of the radio module. Currently two adapters are available: one for TinyOS motes and one for SunSPOT devices. Such a component is able to abstract the interaction between the BS and the WSs away from the low-level communication which relies on the specific platform technology that is employed. – WS Commands & Events, which is basically an interface offered to the developers of the end-user system to coordinate the BSN (e.g. WS sensing activation and signal processing) and to forward various events (e.g. new discovered nodes, alarms, and user data messages) to the set of registered listeners. – WS Discovery, which is based on the previous component, provides automatic and on-demand discovery of WS nodes.

Please cite this article in press as: G. Fortino et al., A framework for collaborative computing and multi-sensor data fusion in body sensor networks, Informat. Fusion (2014), http://dx.doi.org/10.1016/j.inffus.2014.03.005

10

G. Fortino et al. / Information Fusion xxx (2014) xxx–xxx

– High-Level Data Processing, which provides signal processing and pattern recognition at the BS. In particular, such a component allows for the development of new applications by offering flexible interfaces for data pre-processing, feature extraction and selection, signal processing, and pattern/data classification. The HighLevel Data Processing is defined to facilitate the integration of SPINE with signal processing and data mining frameworks. This allows to provide functionalities such as data aggregation, automatic network configuration, and graphical configuration. This component also provides an adaptation with the well-known WEKA Data Mining toolkit [31] to allow SPINE to use its rich set of algorithms for feature selection and pattern classification. 4.2. Inter-BSN Communication The inter-BSN communication component relies on the Message (Msg), Communication Provider (CP) and Message Handler (MH) components (see Fig. 9). In particular, the CP defines methods for the configuration of the BS to receive messages of specific types. In the configuration phase, MHs components are registered with the CP so as to notify the receipt of a message of the defined types. The CP provides the send method to transmit a message. This allows specifying the message targets and does not depend on the message type. Such effective mechanism enables high-level client components to manage incoming messages through different procedures. According to the Inter-BSN Communication component, an interaction protocol can be defined in four-step process: 1. Defining a new message type which is correlated to the interaction protocol (IP). 2. Creating a new set of messages that belong to the defined message type (i.e. the messages of the protocols). 3. Creating a new MH for the processing of the new message type (i.e. all messages of such type). 4. Registering the defined MH with the CP to manage the messages of the designed IP. The CIBOP protocol is developed according to such design schema and particularly supports the basic and application-specific services of C-SPINE. Moreover, the communication level is an abstraction of the specific lower-level communication protocol obtained through a set of adapters (see Fig. 8). Currently C-SPINE is based on the Bluetooth (BT) and the IEEE 802.15.4 protocols. The specific protocol is chosen depending on the base station technology. Specifically, the communication level on the top of the IEEE 802.15.4 relies on the Active Messages layer for TinyOS computers, whereas the one on top of BT relies on Android BT on Android-enabled devices and on BlueCove on PCs. 4.3. BSN Proximity Detection The proximity detection component incorporates an efficient beaconing mechanism [32,33], which is designed around two

strategies: network-driven beacon rate adaptation and networkdriven update of the neighbor cache. Specifically, the transmission rate of the beacon message (or beacon_rate) is adapted on the basis of the network conditions. At the same time, it is particularly significant the choice of the timeout value (called update_rate) after which a neighbor BSN is to be removed from the local cache, in case of no beacons coming from the neighbor BSN are received. To adapt the beacon_rate to the network conditions, each BSN handles a table containing information about its neighbor BSNs and adapts the transmission rate of the beacon messages in function of the turnover value of its neighbors BSNs. It is assumed that each BSN transmits the beacon with frequency fhello . The turnover rate (r t ) is computed as follows:

rt ¼

Nndn Ncn

ð1Þ

where Nndn is the number of new discovered neighbor BSNs and N cn is the total number of cached neighbor BSNs. If rt is less than the threshold r opt ; f hello is reduced as few changes occurred in the proximity. To this purpose, the interval between two successive beacons is incremented by Dt. Conversely, if r t is greater than the threshold ropt , the interval between to successive beacons is decremented by Dt, as the proximity network is rapidly evolving. Dt is usually set to 500 ms. The update_rate is adapted to the network conditions in function of the speed of neighbor BSNs and the beacon rate. In particular, each BSN stores the following history table for each neighbor BSN node v:

hbeacon timeðv Þ; T 1 ðv Þ; T 2 ðv Þ; Waitðv Þi

ð2Þ

where beacon timeðv Þ represents the timestamp of the last beacon received from node v ; T 1 ðv Þ and T 2 ðv Þ are the intervals referred to the last two received beacons from v, and Waitðv Þ indicates the time for which the entry of node v will be kept before it is removed if new beacons are not received from node v. Waitðv Þ, which therefore defines the update_rate of the neighbor BSN cache, is calculated as follows: 8 if T 1 ðv Þ ¼ T 2 ðv Þ > < k  T 1 ðv Þ 1 ðv Þ if jT 1 ðv Þ  T 2 ðv Þj P 1 Waitðv Þ ¼ T 1 ðv Þ þ jT 1 ðvTÞT 2 ðv Þj > : T 1 ðv Þ þ T 1 ðv Þ  ðjT 1 ðv Þ  T 2 ðv ÞjÞ if 0 < jT 1 ðv Þ  T 2 ðv Þj < 1 ð3Þ

4.4. BSN Service Discovery The BSN Service Discovery component supports the discovery of services provided by neighbor BSNs. This component is based on the BSN Proximity Detection component that provides the list of currently detected neighbor BSNs. Two mechanisms for service discovery are implemented: on-demand and advertisement-driven. The on-demand discovery allows to directly inquire one or more neighbor BSNs that, in turn, will reply with a list of available services. The advertisement-driven discovery is based on advertisement messages, containing the list of available services, which a BSN periodically broadcasts along with the beaconing messages. Services in C-SPINE are currently identified through a pre-defined numeric identifier. 4.5. BSN Service Selection & Activation

Fig. 9. Schema of the inter-BSN communication component.

After service discovery, this component supports selection of discovered services and activation of the selected services. Selection and activation rely on simple rules based on the selected service, the mutual acquaintance relation with the BSNs providing the

Please cite this article in press as: G. Fortino et al., A framework for collaborative computing and multi-sensor data fusion in body sensor networks, Informat. Fusion (2014), http://dx.doi.org/10.1016/j.inffus.2014.03.005

G. Fortino et al. / Information Fusion xxx (2014) xxx–xxx

selected service, and possible contextual information. In particular, a rule can be defined for each service as the following tuple:

hIDS ; RA ½; CTXi where

11

according to the defined rules, whereas the latter selects and activates services automatically on the basis of the defined rules. Finally, service execution is based on ‘‘a priori’’ knowledge of BSNs to run the activated service. 4.6. BSN collaborative multi-sensor data fusion

– IDS is a numeric identifier representing a given service; – RA # IDnBSN ; n P 2 is a relation associating the identifiers of n different interacting BSNs on the basis of their mutual acquaintance. If the interacting BSN identifier is a component of such relation, RA holds. A special IDBSN is used to refer to all BSNs to allow for public use of the service. Examples of relations are the following: (i) hIDx ; i, which indicates public services; (ii) hIDx ; IDy i, which enables a service only between a pair of BSNs; (iii) hID1 ; ID2 ; . . . ; IDn i, which enables a group of BSNs to use the service. – CXT (which is optional) is an attribute referring to a physical or logical context in which the interaction has to take place. If the interacting BSN has this attribute, the CXT holds. CXT ¼ home or CXT ¼ hospital are examples of physical contexts whereas CXT ¼ walking is an example of logical context. The rule holds if and only if both RA and CXT (if any) hold; this implies that the service can be selected and activated. Selection and activation can be configured as either manual or automatic. The former is driven by the user, who selects and activates services

The operational architecture (Fig. 10) for collaborative sensor data fusion of C-SPINE, which can be used by any application-specific service, specializes the reference three-layered architecture for multi-sensor data fusion in CBSNs shown in Fig. 7. In particular, the architectural components are:  Feature Extraction, which is based on the In-Node Processing basic SPINE component (see Section 4.1), extracts features from windows of sensed data coming from sensors.  Local Decision, which is also based on the In-Node Processing basic SPINE component, compute a local on-node decision on the basis of the extracted features. The local decision could activate the transmission of the features to the BS.  Feature Transmission, which transmits the computed features (and/or the local decision) to the BS by using the Inter-BSN Communication component (see Section 4.2).  Feature Snd/Rcv & Merging, which allows to send and receive the computed features to/from all partner CBSNs. Received features are then merged to feed the higher level decision maker.

Fig. 10. Architecture of the C-SPINE collaborative sensor data fusion component.

Please cite this article in press as: G. Fortino et al., A framework for collaborative computing and multi-sensor data fusion in body sensor networks, Informat. Fusion (2014), http://dx.doi.org/10.1016/j.inffus.2014.03.005

12

G. Fortino et al. / Information Fusion xxx (2014) xxx–xxx

 Merged Feature Decision, which provides decisions on the basis of the merged features. Decision makers are based on the High-Level Data Processing component (see Section 4.1).  Decision Snd/Rcv & Decision Making, which allows to send and receive the local high-level decisions to/from all partner CBSNs and to produce the final joint decision which all CBSNs agreed upon. 5. e-Shake: A CBSN system for collaborative detection of emotions The e-Shake is a CBSN system aiming at detecting emotion reactions between two people that are shaking their hands. Specifically, e-Shake, which is developed through C-SPINE, integrates the detection of handshake gestures with continuous heart rate calculation in a time synchronized fashion. Such integrated and synchronized information can be used to detect emotion states/ reactions of meeting people when their meeting starts with a handshake. At the sensor node side, the e-Shake system architecture relies on two sensor components (Fig. 11):  The HandShake Sensor (HS Sensor) component, which is executed on a Shimmer mote [21], is placed on the right arm wrist. The HS Sensor acquires data from the on-board three-axis accelerometer sensor according to a well-defined sampling time (ST), stores the data sampled from the XYZ channels of the accelerometer into the circular buffers, calculates selected features on the sampled data, and exploits a decision tree-based classifier for the handshaking detection (see Section 5.2 for more details).  The Heart Rate Sensor (HR Sensor) component, which is also executed on a Shimmer mote [21], is interfaced with an ECG sensor board to extract the heart rate [34,35]. To compute (and update) the HR, the original ECG signal is processed in real-time to detect the QRS complexes which occurs during ventricles contraction. Each detected beat is time-stamped, and a set of the latest consecutive inter-beat times is finally

used to compute an average HR value expressed in beat-perminute (BPM). Our QRS detection algorithm [36] is based on the method proposed in [37]. Specifically, it is a real-time adaptive peak detector algorithm composed of three main steps: i. A Linear High Pass Filter is applied to windows over the original ECG signal; ii. The filtered output is further processed with a Non-linear Low Pass Filter; iii. The output of the second step is used by a binary Decision Making algorithm, which detects the presence of peaks by using a threshold that is dynamically updated. The e-Shake coordinator (see Fig. 11) combines two components: (i) a handshake detection subsystem fully implemented through the C-SPINE mechanisms for the recognition of the handshake gesture between two individuals; (ii) a heart-rate subsystem measuring the heart beats in real-time. Specifically, the coordinator uses the Data & Event Synchronization Framework to temporally synchronize the heart rate data with the handshake detection events. Indeed, the framework is fully application-independent and allows to transparently synchronize different data streams/events coming from virtual and real sensors. Moreover the coordinator archives the heart rate data during time windows when the handshake events have occurred for enabling the detection of emotion reaction; the time windows are dinamically defined on the instant of the handshake detection and their lengths can be specifically set up as symmetric or asymmetric with respect to such instant. Collected data windows can also be remotely transmitted for offline analysis. In Fig. 12 the e-Shake GUI during the detection of a handshake is shown. In the next subsections, we first present a brief overview of the currently available system for handshake detection and specifically details about the multi-sensor data fusion process for handshake detection, which is based on the B-HanDS system [13] reverse engineered through C-SPINE, then we describe the method used to synchronize handshake detections and heart rate; finally, we analyze the emotion reactions that can be detected by using e-Shake and the related accuracy that can be achieved.

Fig. 11. The architecture of the e-Shake system.

Please cite this article in press as: G. Fortino et al., A framework for collaborative computing and multi-sensor data fusion in body sensor networks, Informat. Fusion (2014), http://dx.doi.org/10.1016/j.inffus.2014.03.005

G. Fortino et al. / Information Fusion xxx (2014) xxx–xxx

13

Fig. 12. The e-Shake GUI during a handshake detection event.

5.1. Related handshake detection systems In [38] the iBand system is proposed. It supports exchange of information between two individuals on the detection of their handshake. iBand relies on a wearable bracelet using infrared (IR) transceivers and accelerometer sensors for handshake detection. A handshake is specifically detected through the alignment of IR transceivers merged with the synchronized up-and-down motion sensed by the accelerometer sensor on the two bracelets which are under IR contact. The IR data transfer is just started when the wrists of the users are in a purposely calibrated handshake orientation. In order to enable exchange of information, iBand is based on a centralized database system to manage users’ credentials and to archive information about their associations. It is worthy to note that quantitative results related to the accuracy of the handshake detection are not described; only a qualitative evaluation is carried out on the basis of questionnaires and interviews, which only indicate that the prototype works according to its specifications. In [39] the authors propose a smart device-oriented approach named Smart-Its Friends. The approach is based on smart devices that can interact when are located in their communication range and gather comparable collected sensor data. Specifically, when they are colocated, handheld smart devices containing accelerometer sensors can be used to detect common shaking patterns among them. However, this is a sort of emulation of a handshake since Smart-Its devices are not purposely designed to recognize handshakes but are only able to detect interactions among smart devices near each other. A BSN-based system which could be also used to detect handshakes is presented by Kern et al. in [40]. In order to develop a wearable, context-aware application, the authors have designed a specific sensor platform that can collect and store motion data

coming from many acceletometer sensors positioned on the human body. Their system is composed by a hw platform, that is able to capture 3-axial acceleration data from up to 48 locations of the human body and a sw component implementing a recognition algorithm based on a Bayesian classifier. The acquisition system is based on the Smart-Its core board [39], which can provide wired and wireless communication to a host system (e.g. laptop) to store the data. The sensors are embedded in small boards that are connected to an add-on board that can handle several signals. By using the system, the authors try to detect basic user postures and movements (sitting, standing, etc.) and even a few gestures to infer the user’s social interactions like the handshake. In [41] the authors introduce a system for secure pairing of mobile devices using a common shaking movement as shared secret supporting subsequent mutual authentication. Two threshold-based classification algorithms for recognition of common object shaking are presented: ShaVe, based on a coherence measure between the acceleration samples, and ShaCe, based on rate of matching feature vectors. The evaluation performed using a wired apparatus shows that under optimal parameter selection, the classifiers can achieve a relatively low false negative rate of 12%. 5.2. Multi-sensor data fusion process for collaborative handshake detection The collaborative handshake detection process between CBSN i (composed of BSi and WSi ) and CBSN j (composed of BSj and WSj ) is defined by six succeeding phases (see Fig. 13): 1. Proximity Detection: BSi and BSj detect they are in proximity by means of the BSN Proximity Detection component (see Section 4.3).

Please cite this article in press as: G. Fortino et al., A framework for collaborative computing and multi-sensor data fusion in body sensor networks, Informat. Fusion (2014), http://dx.doi.org/10.1016/j.inffus.2014.03.005

14

G. Fortino et al. / Information Fusion xxx (2014) xxx–xxx

Fig. 13. The phases of the handshake detection process.

2. Node Activation: after proximity detection, both BSi and BSj activate sensing, feature extraction and in-node classification on their WS nodes (WSi and WSj , respectively). This would enable energy conservation of the WS node that is (re)activated only if necessary. In-node classification specifically relies on a lightweight decision tree-based classifier enabling detection of oneside handshaking (see below). 3. Potential Handshake: if a potential one-side handshake occurs, WSi notifies its BSi . This permits to save the energy of the sensor node and the bandwidth of the communication channel as the communication with BSi is actually established only upon the detection of a potential handshake. 4. Advertisement: BSi transmits a message to the partner BSj (and vice-versa) to signal the detection of a potential handshake. 5. Collaborative Handshake Detection: to recognize a handshake, the interaction between BSi and BSj is carried out by means of the cooperative handshake detection protocol [13]. Finally, a handshake detection occurs only if both BSi and BSj agree on it (see below for the description of the handshake detection process). In Fig. 14 the handshake detection process is modeled through a UML sequence diagram. The diagram shows the causal/temporal flow of messages exchanged between the components of the two

CBSNs, from proximity detection to handshake detection. In particular, messages are grouped according to the aforementioned five phases of the handshake detection process. As aforementioned, the process for the detection of handshakes (see Fig. 15) relies on two classification steps performed at the WS and BS sides and based on J48 classifiers. The classifier installed on the WS carries out local classification of the features computed on the accelerometer sensor readings and, upon detecting a potential handshake, transmits them to the BS. The classifier installed on the BS carries out classification after the merge of the features received from the partner CBSN. The two classifiers have different requirements: (i) the one on WS is simple as WS has very constrained resources and has to be fast to classify; (ii) the one on BS is more complex, as BS is more powerful than WS, and also has to quickly classify. Furthermore, the classifiers are synthesized on the basis of different data sets: the WS one derives from a small data set whereas the BS one is derived from a larger data set. Thus, at the node side, a classifier that relies on a decision tree is defined. The main motivation is that it is fast in the decision-making and straightforward to be developed on resource-constrained WSs. A decision tree model can be simply developed as a conditional statement sequence. It is worth noting that the classifier on the WS is trained offline using a dataset built on a large number of handshake trials. The features extracted from the collected sensor

Fig. 14. Sequence diagram of the handshake detection process.

Please cite this article in press as: G. Fortino et al., A framework for collaborative computing and multi-sensor data fusion in body sensor networks, Informat. Fusion (2014), http://dx.doi.org/10.1016/j.inffus.2014.03.005

G. Fortino et al. / Information Fusion xxx (2014) xxx–xxx

15

Fig. 15. Collaborative sensor data fusion process for handshake detection.

readings constitute the input data to the decision tree which classifies them as potential handshake or not. The classification is performed sequentially on each string of features extracted. The selection of significant features is a critical operation that has effects on the final accuracy of the classification algorithm. The selection of a minimal set of features can also reduce the communication bandwidth between WS and BS in the BSN and between the partner CBSNs, and consequently decreasing energy depletion. Moreover, features should be simple to calculate and capable of precisely representing handshakes. On the basis of an analysis of the accelerometer sensor data during handshakes, features were carefully chosen with the aforementioned characteristics. The selection of such features was carried out according both to previous experiences [10] in defining features for activity recognition and to specific feature selection experiments aiming at identifying an increase in accuracy. The final set of identified features exploited for the tree-based classifiers are reported in Table 2: Zero Crossing (ZC), Root Mean Square (RMS), Amplitude (A), Mean (M), Standard Deviation (SD), Total Energy (TE), Near Zero (computed as Below Threshold – BT). Apart from TE, the features are computed for each accelerometer channel. In particular:

 The 3-axial accelerometer is sampled at ST = 10 ms.  Features are calculated on a data window of 32 samples with a shift of 50%. They are so calculated each 160 ms and analyzed on a time frame window (hereafter named frame) equal to 320 ms. Such parameter values have been experimentally set up to trade off fast detection and good classification accuracy. To carry out joint classification, the BS classifier merges the features obtained from the associated WS and those received from the partner BS. An important requirement of the joint classification is the framebased synchronization of the features coming from the two associated CBSNs. The classification is therefore run on a pair of synchronized frames. Feature synchronization is easily carried out through the buffering of the features obtained from the node at the BS and the consequent transmission of the buffer of features to the BS of the other CBSN. In order to improve classification accuracy, additional frames can be transmitted and buffered. Upon the buffer reception, the two CBSN features are classified sequentially on a frame-by-frame basis. Each classification result is merged by means of a proper combination function (or combiner) to have a handshake detection outcome. Feature classification uses the same

Please cite this article in press as: G. Fortino et al., A framework for collaborative computing and multi-sensor data fusion in body sensor networks, Informat. Fusion (2014), http://dx.doi.org/10.1016/j.inffus.2014.03.005

16

G. Fortino et al. / Information Fusion xxx (2014) xxx–xxx Table 2 Set of features for handshake detection.

decision tree types run on the WS; indeed, the used classification tree is more complex as it is constructed on a data set based on a doubled attribute number (i.e. the features of both the CBSNs) and using a larger data set. The combiner specifically relies on an array of partial binary results filled with:

 Handshake = 1;  No Handshake = 0. Different combination functions can be applied on the vector of partial results to obtain the final result, also relying on the number of combined frames:

Please cite this article in press as: G. Fortino et al., A framework for collaborative computing and multi-sensor data fusion in body sensor networks, Informat. Fusion (2014), http://dx.doi.org/10.1016/j.inffus.2014.03.005

G. Fortino et al. / Information Fusion xxx (2014) xxx–xxx

17

 Single Frame: the result derives from just 1 frame.  Double Frame: the result consists of the AND of the two partial results (derived from two combined frames).  Triple Tolerant Frame: on the basis of three combined frames, two consecutive positive results are at least required.  Triple Frame: the result consists of the AND of the three partial results (derived from the three combined frames). 5.3. Accuracy evaluation The accuracy of the handshake detection is evaluated by experimentally measuring the percentage of correct/erroneous detections of handshakes under different operating conditions of the handshake detection component under-test. Moreover the computation of the false positives and false negatives, which should be minimized, is fundamental for the evaluation phase. In particular, a false positive is a detected handshake that is not so whereas a false negative corresponds to a non-detected real handshake. The dataset was constructed by collecting accelerometer sensor data from different types of handshakes, random non-handshake movements, movements of the hand in everyday life, and handshake-like movements specifically performed to emulate a handshake between two hands (hereafter named ‘‘fake’’ handshakes). In fake handshakes, hands are shaked close to each other but without any touch. In particular, real handshakes were categorized according to two parameters:  Shaking strength, which is categorized as: – strong: more than two shakes with large amplitude. – medium: one or two shakes with medium amplitude. – weak: one or two light shakes with narrow amplitude.  Shaking duration, which is categorized as: – fast: less than 1 s. – medium: between 1 s and 3 s. – slow: more than 3 s. Such rich experimentally constructed dataset is the basis for building up the local and joint classifiers. Before providing set-up and results of the accuracy evaluation of the handshake detection system, it is worth noting that the one-sided handshake classification is 100% accurate for real handshake: this means that, with the detection of one-sided handshake, we are only able to precisely understand if one hand is shaking in proximity of another hand. One-sided handshake is not enough to detect a two-sided handshake as it must be performed a collaborative classification to detect a handshake. To evaluate the performance of the handshake detection component in a real scenario, we executed several trials with a proper number of persons (25 subjects) having different biases (i.e. persons with previous contact/use of the system and persons with no knowledge about the system). To analyze the overall behavior of the handshake detection system, we set up a real environment where two individuals, equipped with a BSN executing C-HanDS, meet by shaking their hands. We carried out a trial involving one hundred handshakes in which the number of successes in their detection is counted. As the frame number to utilize in the collaborative classification can be easily configured in the handshake detection system, we re-executed such trial by changing the number of frames exchanged by the collaborative handshake protocol and selecting the four combiners previously defined (i.e. single frame, double frame, tolerant triple frame, and triple frame). Fig. 16 shows the percentages of true positives and false negatives for each set-up. The system possesses good accuracy particularly if the tolerant triple frame combiner is adopted. It is worthy to note that the tests were performed in a real environment involving

Fig. 16. Real accuracy on detecting 100 handshakes.

different people making handshakes of different durations and strengths. Moreover such results indicates that three frames are good enough to enable accurate cooperative decisions. Furthermore, the exploitation of cooperation dramatically minimizes the chance of enabling interactions between BSNs with no handshaking; this is an issue when the decision should be taken without using data coming from other BSNs. Moreover, to test the ability of the system to minimize false positives, the same test was carried out on 100 fake handshakes. Fig. 17 shows the obtained results. As can be seen, false positives are also minimized by using the Triple Frame (Tolerant) configuration. It is worth noting that a fake handshake is a handshake with the hands very close and with no touch, so that it can be considered an unrealistic case in the daily life. Finally it is worth to report that the accuracy of detecting real handshakes reaches 100% when the duration of the handshake is slow and the strength of the handshake is at least medium. This is an important result as it allows to effectively and properly test the whole e-Shake system (see Section 5.5). 5.4. Synchronized integration of handshake events and heart rate Handshake events and heart rate are integrated through the event synchronization framework that is able to synchronize continuous sensor streams and events coming from real and virtual sensors on the same time-line by using an approach called Axes-Based Synchronization [42] in the research area of multimedia systems. In particular, sensor data streams are time-dependent media objects that are generated according to specific temporal relations between consecutive units of the media stream. They are continuous in time and different from sporadic events that sensor nodes can generate upon detection of specific events (e.g. object recognition, threshold activation, etc.). By exploiting Axes-Based Synchronization, the

Fig. 17. Error on misdetection of 100 fake handshakes.

Please cite this article in press as: G. Fortino et al., A framework for collaborative computing and multi-sensor data fusion in body sensor networks, Informat. Fusion (2014), http://dx.doi.org/10.1016/j.inffus.2014.03.005

18

G. Fortino et al. / Information Fusion xxx (2014) xxx–xxx

framework is able to synchronize multiple continuous sensor streams and sporadic events. Specifically, the synchronization relies on a global timer which implies that all objects are placed onto a timing axis representing a realtime abstraction. This technique has the following advantages: removing objects does not affect the synchronization of the others and the integration of time-independent objects (sporadic sensor events) is easy to obtain. In our system, the global timer abstraction (H) is provided at the SPINE Coordinator (see Fig. 11). Thus, the heart rate stream can be formalized as continuous timed pairs hHRi ; ti i , where HRi is the i-th value of the heart rate and ti the measurement time of HRi , whereas handshake event can be represented by sporadic timed pairs hHD; t j i, where HD is true, which means Handshake Detected, and t j is the time in which handshake is detected. Both t i and tj refers to the global time H. 5.5. Emotion reaction analysis Here we discuss the most significant results obtained from the experimental evaluation of e-Shake. In particular, the aim of the evaluation is to analyze the effectiveness of e-Shake in detecting emotion reactions between handshaking people. Real experiments were defined and performed in a controlled experimental testbed which specifically refers to a university computer lab, where subjects (professors, students, and tutors), which wear e-Shake, can meet. To set up a real meeting environment, a given number of subjects with and without mutual acquaintance were seleted. Moreover, for the purpose of not introducing any bias, students were

fully non aware of the experiment objective, whereas professors and tutors were aware about the experiment objective and played a fundamental role to ease social interactions among students and enable student reactions thanks to the academic professor-student hierarchy. Specifically, the selected subjects were 15 students of MSc in Computer Engineering (age in the range 22–25 and 20% women) and 15 students of MSc in Statistics (age in the range 22–25 and 55% women). While students of the same course knew each other, students of different courses, which belong to different faculties, did not know each other. e-Shake was able to detect three different kinds of reciprocal emotion reactions to the organized meetings (see Fig. 18(a–c)):  (a) no emotion;  (b) emotion reaction from just one individual;  (c) emotion reaction from both individuals. In particular, A and B refer to general subjects that are different in the considered cases: HR(A) and HR(B) are the heart rate streams of the A and B subjects, respectively. It is worth noting that the event ‘‘Handshake Detected’’, reported with a black vertical arrow in the plot exactly at the time it occurred, is emitted by the Handshake Detection component based on the HS sensor (see Fig. 11). Of course, cases similar to the left graph of (a) are the most frequent ones. Indeed, the right graph of (a) also represents the same case but involves a tachycardic subject, characterized by HR(B). Case (b) is primarily correlated to meetings between a professor and a student. However, such case occurred in meetings

Fig. 18. Emotion reactions to handshake-enabled meeting between individuals A and B: (a) no reaction; (b) one-side reaction; and (c) two-side reaction.

Please cite this article in press as: G. Fortino et al., A framework for collaborative computing and multi-sensor data fusion in body sensor networks, Informat. Fusion (2014), http://dx.doi.org/10.1016/j.inffus.2014.03.005

G. Fortino et al. / Information Fusion xxx (2014) xxx–xxx Table 3 System accuracy in detecting emotion reactions. Situation

Occurrences

Nr. of correct classifications

No emotion Emotion

83 17

78 14

Table 4 Accuracy of the emotion reaction detection algorithm at different values of T and K parameters. In bold, the values related to the highest accuracy. T

K (%) 10 15 20

20 s

30 s

40 s

50 s

60 84 72

64 92 77

70 80 74

69 65 68

involving two students too. Occurrence of case (c) is rare. In particular, case (c) was stimulated by establishing a meeting between two individuals that know themselves very well but have not been meeting for long time. It is worthy to note that while the left graph of (c) highlights a reaction before handshaking, the right graph of (c) refers to a reaction during handshaking. In order to evaluate the system accuracy for the detection of emotion reactions, the experiment was organized in a quiet laboratory room with two separate entrances. Participant couples were asked to enter the room at the same time, each of them in one of the two doors, and to shake their hands as they meet. The average duration of the meeting was about 20 s. At the end of each experiment, the participants have been briefly (and separately, for privacy reasons) interviewed to annotate whether the meeting inducted some sort of emotion reaction. The handshake detection algorithm is able to recognize different type of handshakes; however, for the sake of analyzing the accuracy of our emotion reaction detection algorithm, we instructed the participants to shake their hands with a sufficient strength and duration so that the handshake detection system always recognizes the handshake (see Section 5.3). In particular, we collected handshake and cardiac data from 100 meetings. Table 3 summarizes the different meeting situations; the ‘‘emotion reaction’’ situation includes both the ‘‘one-side’’ and ‘‘two-side’’ emotion reaction (see Fig. 18). In e-Shake, we introduced a simple yet effective emotion reaction detection algorithm that is based on the assumption that, often, a sudden emotion state change causes a temporary heartrate increase. In particular, we analyze a window of the heart-rate signal in the interval ±T/2 from the handshake event. If the heartrate variation is above K% with respect to its baseline, the system triggers the occurrence of an emotion reaction. We performed an analysis with different values of T and K; the obtained results are summarized in Table 4, evaluating the corresponding accuracy of emotion reaction detection. We observe that the highest accuracy of 92% is reached by setting T = 30 s and K = 15%.

6. Conclusion BSNs are gaining enormous interest as they enable novel realworld systems aiming at helping people in many aspect of their life. Particularly, BSNs allow continuous, real-time, non-invasive, anywhere and anytime monitoring of assisted livings in many different areas: from mobile health care to e-Emergency, from e-Entertainment to e-Sociality. Nowadays, BSNs are usually applied for the monitoring of single assisted livings. As a consequence,

19

the BSN programming frameworks and middleware developed to date (e.g. SPINE, SPINE2, CodeBlue, Titan, RehabSpot) mainly provide programming support for star-topology-based BSN architectures consisting of a collection of wireless sensor nodes managed by a basestation (e.g. PC, PDA, tablet, smartphone). However, different types of BSN architectures can be defined in diversified application domains where single assisted livings’ monitoring is not sufficient to address more complex system requirements. To fill this lack, this paper has defined and proposed the innovative concept of CBSN (i.e. a novel type of BSN architecture that enables BSNs to collaborate with each other) that targets to support novel collaborative systems and application services in diverse application areas, e.g. e-Health, e-Entertainment, e-Emergency, e-Sport, e-Sociality, and e-Factory. The C-SPINE framework for the design and implementation of applications and systems based on CBSNs has been detailed. Specifically, C-SPINE in an extension of the well-known SPINE middleware with flexible and customizable service and communication layers enabling inter-BSN communication, service discovery, activation and execution, and multi-sensor data fusion involving multiple CBSNs. To the best of our knowledge, C-SPINE is currently the only BSN framework fully supporting CBSN applications covering all four logical BSN architectures: Single Body – Single Base station, Single Body – Multiple Base stations, Multiple Bodies – Single Base station, and Multiple Bodies – Multiple Base stations. Finally, to demonstrate the effectiveness of C-SPINE, the e-Shake system has been developed. In particular, e-Shake is a CBSN system aiming to detect emotion reactions between two individuals shaking their hands. e-Shake combines a cooperative handshake detection component, which relies on a wrist-worn sensor, with a heart rate computation component based on an ECG sensor node. The handshake detection process fully relies on the architecture for multi-sensor data fusion provided by C-SPINE to implement complex classification algorithms for multi-user activity recognition. Results obtained from an experimental controlled testbed show that e-Shake is able to detect emotion reactions of subjects during meetings driven by handshaking, with a good accuracy. In particular, three kinds of reactions can be captured during a handshakedriven meeting: no emotions, one-side emotion reaction, and two-side emotion reactions. By using C-SPINE, CBSN application prototyping can be actually boosted; this is confirmed by our experience in developing the handshake detection system by exploiting C-SPINE and, previously, by using only SPINE and low-level nesC programming to program collaboration mechanisms. The use of C-SPINE therefore reduces the development efforts, at design and implementation levels, and, as a consequence, the development time can be significantly reduced. In the specific case of the handshake detection system, the development time was reduced by almost four times using the same number of resources with similar skills. Finally, it is worth noting that naive/straightforward solutions to move from BSN to CBSN are not feasible as the basic collaborative BSN mechanisms (i.e. inter-BSN data communication, proximity detection, inter-BSN communication protocols, service discovery, activation and execution) needed for CBSNs are complex and time-consuming to implement. C-SPINE specifically provides such mechanisms so any application implementation hurdles can be easily overcome by design. On the basis of the developed C-SPINE framework and the availability of the novel e-Shake system, research efforts are currently devoted to: (i) defining intelligent techniques for the recognition of emotion states on the basis of collected heart rate patterns triggered by handshake detections and/or proximity of people. (ii) designing and implementing the collaborative applications, delineated in Section 3.1, through an holistic exploitation of C-SPINE

Please cite this article in press as: G. Fortino et al., A framework for collaborative computing and multi-sensor data fusion in body sensor networks, Informat. Fusion (2014), http://dx.doi.org/10.1016/j.inffus.2014.03.005

20

G. Fortino et al. / Information Fusion xxx (2014) xxx–xxx

and some reusable patterns for mobile collaborative applications [43]. (iii) defining a new mechanism for the time synchronization of the two local interacting CBSNs to analyze the impact of using tight time synchronization [44] instead of the currently used causal (or loosely-coupled time) synchronization during the collaboration among CBSNs.

[16]

[17]

Acknowledgments [18]

The authors wish to thank Antonio Augimeri, Antonio Condello and Giuseppe Scabellone for their precious help in the implementation of the e-Shake system. This research has been partially funded by the joint bilateral Italy/China Project ‘‘Smart Personal Mobility Systems for Human Disabilities in Future Smart Cities’’ (N. CN13MO7), by CONET, the Cooperating Objects Network of Excellence, funded by the European Commission under FP7 with contract number FP7-2007-2-224053, by TETRis TETRA Innovative Open Source Services, funded by the Italian Government (PON 01-00451), by the Ministry of Science and Technology of China (Grant Number: 2012BAJ05B07), and by International cooperation Projects of Hubei province (2011BFA012). Stefano Galzarano was supported by a Grant funded in the framework ‘‘POR Calabria FSE 2007/2013’’. References [1] P.J. Marron, S. Karnouskos, D. Minder, A. Ollero, Applications, in: P.J. Marron, S. Karnouskos, D. Minder, A. Ollero (Eds.), The Emerging Domain of Cooperating Objects, Springer, Berlin, Heidelberg, 2011, pp. 125–176. [2] J. Yick, B. Mukherjee, D. Ghosal, Wireless sensor network survey, Comput. Networks 52 (12) (2008) 2292–2330, http://dx.doi.org/10.1016/j.comnet. 2008.04.002. [3] A. Pantelopoulos, N.G. Bourbakis, A survey on wearable sensor-based systems for health monitoring and prognosis, Trans. Syst. Man Cyber. 40 (2010) 1–12, http://dx.doi.org/10.1109/TSMCC.2009.2032660. [4] M. Domingo, A context-aware service architecture for the integration of body sensor networks and social networks through the IP multimedia subsystem, IEEE Commun. Mag. 49 (1) (2011) 102–108, http://dx.doi.org/10.1109/ MCOM.2011.5681022. [5] F. Bellifemine, G. Fortino, R. Giannantonio, R. Gravina, A. Guerrieri, M. Sgroi, SPINE: a domain-specific framework for rapid prototyping of WBSN applications, Softw.: Pract. Exp. 41 (3) (2011) 237–265, http://dx.doi.org/ 10.1002/spe.998. [6] N. Raveendranathan, S. Galzarano, V. Loseu, R. Gravina, R. Giannantonio, M. Sgroi, R. Jafari, G. Fortino, From modeling to implementation of virtual sensors in body sensor networks, IEEE Sens. J. 12 (3) (2012) 583–593, http://dx.doi.org/ 10.1109/JSEN.2011.2121059. [7] K. Lorincz, D. Malan, T. Fulford-Jones, A. Nawoj, A. Clavel, V. Shnayder, G. Mainland, M. Welsh, S. Moulton, Sensor networks for emergency response: challenges and opportunities, Pervasive Comput., IEEE 3 (4) (2004) 16–23, http://dx.doi.org/10.1109/MPRV.2004.18. [8] C. Lombriser, D. Roggen, M. Stäger, G. Tröster, Titan: a tiny task network for dynamically reconfigurable heterogeneous sensor networks, in: 15. Fachtagung Kommunikation in Verteilten Systemen (KiVS), Informatik aktuell, Springer, 2007, pp. 127–138, http://dx.doi.org/10.1007/978-3-54069962-0_11. [9] M. Zhang, A. Sawchuk, A customizable framework of body area sensor network for rehabilitation, in: 2nd International Symposium on Applied Sciences in Biomedical and Communication Technologies, 2009. ISABEL 2009, 2009, pp. 1– 6, doi: 10.1109/ISABEL.2009.5373703. [10] R. Gravina, A. Guerrieri, G. Fortino, F. Bellifemine, R. Giannantonio, M. Sgroi, Development of body sensor network applications using SPINE, in: IEEE International Conference on Systems, Man and Cybernetics, 2008. SMC 2008, 2008, pp. 2810–2815, doi: 10.1109/ICSMC.2008.4811722. [11] SPINE website, 2013. . [12] A. Augimeri, G. Fortino, S. Galzarano, R. Gravina, Collaborative body sensor networks, in: 2011 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2011, pp. 3427–3432, doi: 10.1109/ICSMC.2011.6084199. [13] A. Augimeri, G. Fortino, M. Rege, V. Handziski, A. Wolisz, A cooperative approach for handshake detection based on body sensor networks, in: IEEE International Conference on Systems, Man and Cybernetics, 2010. SMC 2010, 2010. [14] R. Gravina, A. Alessandro, A. Salmeri, L. Buondonno, N. Raveendranathan, V. Loseu, R. Giannantonio, E. Seto, G. Fortino, Enabling multiple BSN applications using the SPINE framework, in: 2010 International Conference on Body Sensor Networks (BSN), 2010, pp. 228–233, doi: 10.1109/BSN.2010.34. [15] T. Terada, K. Tanaka, A framework for constructing entertainment contents using flash and wearable sensors, in: Proceedings of the 9th International

[19]

[20] [21] [22] [23] [24]

[25]

[26]

[27]

[28] [29]

[30]

[31] [32]

[33]

[34]

[35]

[36]

[37]

[38]

[39]

[40]

Conference on Entertainment Computing, ICEC’10, Springer-Verlag, Berlin, Heidelberg, 2010, pp. 334–341. R. Aylward, J.A. Paradiso, A compact, high-speed, wearable sensor network for biomotion capture and interactive media, in: Proceedings of the 6th International Conference on Information Processing in Sensor Networks, IPSN ’07, ACM, New York, NY, USA, 2007, pp. 380–389, http://dx.doi.org/ 10.1145/1236360.1236408. . C. Park, P. Chou, Y. Sun, A wearable wireless sensor platform for interactive dance performances, in: Fourth Annual IEEE International Conference on Pervasive Computing and Communications, 2006. PerCom 2006, 2006, pp. 6– 59, doi: 10.1109/PERCOM.2006.12. S. Coyle, D. Morris, K.-T. Lau, D. Diamond, N. Moyna, Textile-based wearable sensors for assisting sports performance, in: Proceedings of the 2009 Sixth International Workshop on Wearable and Implantable Body Sensor Networks, BSN ’09, IEEE Computer Society, Washington, DC, USA, 2009, pp. 307–311, http://dx.doi.org/10.1109/BSN.2009.57. J.-Y. Huang, C.-H. Tsai, A wearable computing environment for the security of a large-scale factory, in: Proceedings of the 12th International Conference on Human-computer Interaction: Interaction Platforms and Techniques, HCI’07, Springer-Verlag, Berlin, Heidelberg, 2007, pp. 1113–1122. TinyOS website, 2014. . Shimmer website, 2014. . Z-Stack website, 2014. . Sun SPOT website, 2014. . B. Khaleghi, A. Khamis, F.O. Karray, S.N. Razavi, Multisensor data fusion: A review of the state-of-the-art, Inf. Fusion 14 (1) (2013) 28–44, http:// dx.doi.org/10.1016/j.inffus.2011.08.001. T. Bourdenas, M. Sloman, Towards self-healing in wireless sensor networks, in: Proceedings of the 2009 Sixth International Workshop on Wearable and Implantable Body Sensor Networks, BSN ’09, IEEE Computer Society, Washington, DC, USA, 2009, pp. 15–20, http://dx.doi.org/10.1109/ BSN.2009.14. . L. Dong, J. Wu, X. Chen, Real-time physical activity monitoring by data fusion in body sensor networks, in: 2007 10th International Conference on Information Fusion, 2007, pp. 1–7, doi: 10.1109/ICIF.2007.4408176. W. Li, J. Bao, X. Fu, G. Fortino, S. Galzarano, Human postures recognition based on D–S Evidence theory and multi-sensor data fusion, in: Proc. of the 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, ccGRID 2012, IEEE Computer Society, Washington, DC, USA, 2012, pp. 912– 917, http://dx.doi.org/10.1109/CCGrid.2012.144. S. Thiemjarus, A Framework for Contextual Data Fusion in Body Sensor Networks, Ph.D. Thesis, Imperial College London, 2007. W. Dong, A. Pentland, Multi-sensor data fusion using the influence model, in: Proceedings of the International Workshop on Wearable and Implantable Body Sensor Networks, BSN’06, IEEE Computer Society, Washington, DC, USA, 2006, pp. 72–75, doi: 10.1109/BSN.2006.41. . G. Fortino, R. Giannantonio, R. Gravina, P. Kuryloski, R. Jafari, Enabling effective programming and flexible management of efficient body sensor network applications, IEEE Trans. Hum.-Mach. Syst. 43 (1) (2013) 115–133, http:// dx.doi.org/10.1109/TSMCC.2012.2215852. I.H. Witten, E. Frank, M.A. Hall, Data Mining: Practical Machine Learning Tools and Techniques, Morgan Kaufmann Publishers, 2011. F. Ingelrest, N. Mitton, D. Simplot-Ryl, A turnover based adaptive HELLO protocol for mobile Ad Hoc and sensor networks, Proceedings of the 2007 15th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS), IEEE Computer Society, Washington, DC, USA, 2007, pp. 9–14. doi: 10.1109/MASCOTS.2007.5. A. Ahmad Kassem, N. Mitton, Adapting dynamically neighbourhood table entry lifetime in wireless sensor networks, in: Proc. 10th International Conference on Wireless Communications and Signal Processing (WCSP), China, 2010. L. Hejjel, E. Roth, What is the adequate sampling interval of the ECG signal for heart rate variability analysis in the time domain?, Physiological Measurement 25 (6) (2004), doi: 10.1088/0967-3334/25/6/006. T.H.N. Vu, N. Park, Y.K. Lee, Y. Lee, J.Y. Lee, K.H. Ryu, Online discovery of heart rate variability patterns in mobile healthcare services, J. Syst. Softw. 83 (10) (2010) 1930–1940, http://dx.doi.org/10.1016/j.jss.2010.05.074. R. Covello, G. Fortino, R. Gravina, A. Aguilar, J.G. Breslin, Novel method and real-time system for detecting the cardiac defense response based on the ecg, in: Proc. of IEEE MEMEA, 2013. H. Chen, S. Chen, A moving average based filtering system with its application to real-time qrs detection, in: Computers in Cardiology, 2003, 2003, pp. 585– 588, doi: 10.1109/CIC.2003.1291223. M. Kanis, N. Winters, S. Agamanolis, A. Gavin, C. Cullinan, Toward wearable social networking with iband, in: Proc. of Computer-Human Interaction (CHI) – Extended Abstracts on Human Factors in Computing Systems, ACM, New York, NY, USA, 2005, pp. 1521–1524, http://dx.doi.org/10.1145/ 1056808.1056956. L.E. Holmquist, F. Mattern, B. Schiele, P. Alahuhta, M. Beigl, H.-W. Gellersen, Smart-its friends: a technique for users to easily establish connections between smart artefacts, in: Proceedings of the 3rd International Conference on Ubiquitous Computing (UbiComp), Springer-Verlag, London, UK, 2001, pp. 116–122. N. Kern, B. Schiele, A. Schmidt, Multi-sensor activity context detection for wearable computing, in: In Proc. EUSAI, LNCS, 2003, pp. 220–232.

Please cite this article in press as: G. Fortino et al., A framework for collaborative computing and multi-sensor data fusion in body sensor networks, Informat. Fusion (2014), http://dx.doi.org/10.1016/j.inffus.2014.03.005

G. Fortino et al. / Information Fusion xxx (2014) xxx–xxx [41] R. Mayrhofer, H. Gellersen, Shake well before use: intuitive and secure pairing of mobile devices, IEEE Trans. Mobile Comput. 8 (6) (2009) 792–806, http:// dx.doi.org/10.1109/TMC.2009.51. [42] G. Blakowski, R. Steinmetz, A media synchronization survey: reference model, specification, and case studies, IEEE J. Sel. Areas Commun. 14 (1) (1996) 5–35. [43] A. Neyem, S.F. Ochoa, J.A. Pino, R.D. Franco, A reusable structural design for mobile collaborative applications, J. Syst. Softw. 85 (3) (2012) 511–524, http:// dx.doi.org/10.1016/j.jss.2011.05.046. [44] J. Wahslen, I. Orhan, T. Lindh, Local time synchronization in bluetooth piconets for data fusion using mobile phones, in: 2011 International Conference on Body Sensor Networks (BSN), 2011, pp. 133–138, doi: 10.1109/BSN.2011.11.

Giancarlo Fortino received his Laurea (B.S. and M.S.) degree in Computer Engineering from University of Calabria, Italy in 1995, and Ph.D. degree in Computer Engineering from University of Calabria, Italy in 2000. He is currently Associate Professor of Computer Engineering (since 2006) at the Dept. of Informatics, Modeling, Electronics and Systems (DIMES) of the University of Calabria (Unical), Rende (CS), Italy. He has been a visiting researcher at the International Computer Science Institute, Berkeley (CA), USA, in 1997 and 1999, and visiting professor at Queensland Univ. of Technology, Brisbane, Australia, in 2009. He was nominated Guest Professor in Computer Engineering of Wuhan Univ. of Technology (WUT) on April, 18 2012. His research interests include distributed computing, Wireless Sensor Networks, software agents, cloud computing, multimedia networks. He authored over 200 publications in journals, conferences and books. He is the founding editor of the Springer Book Series on Internet of Things: Technology, Communications and Computing and serves in the editorial board of Journal of Networks and Computer Applications, Engineering Applications of Artificial Intelligence, Information Fusion, etc. He is co-founder and CEO of SenSysCal S.r.l., a spinoff of Unical, focused on innovative sensor-based systems for e-Health and domotics.

Stefano Galzarano received both the B.S. and M.S. degrees in computer engineering from the University of Calabria (UNICAL), Italy. He is currently pursuing a joint-Ph.D. degree from UNICAL and the Eindhoven University of Technology (TU/e), Netherlands. His main research interest is on Wireless Sensor Networks (WSNs) and specifically in highlevel programming methodologies and frameworks, agent-oriented software engineering, machine learning and autonomic techniques. He is involved in several projects including MAPS (multi-agent system for WSNs) and SPINE2 (programming framework for WSNs). He is author of more than 15 papers on journals and international conferences.

21

Raffaele Gravina (http://plasma.deis.unical.it/rg/ index.html) served as Postdoctoral research fellow at the University of Calabria (Italy) for two years. He received the Ph.D. degree in computer engineering from the same University in 2012. He is author of more than 30 papers on journals and international conferences. His research interests are focused on high-level programming methods for WSNs and on innovative m-Health systems based on Wireless Body Sensor Networks (WBSNs). He is Responsible for the ADPERSONAS project, founded by the 2007–2013 Italian NOP for Research and Competitiveness for the Convergence Regions. He is the main designer of the SPINE Framework and responsible for the open-source contributions. He spent two years as researcher at the Telecom Italia WSN Lab at Berkeley, California. He is involved in several research projects on WSNs, including MAPS and the REWSN Cluster of CONET FP7. He is cofounder of SenSysCal S.r.l, a spin-off of the University of Calabria (Italy), founded on April 2010 and whose offer includes innovative services and products derived from academic research results in the Wireless Sensor Networks domain.

Wenfeng Li is a full professor at the school of Logistics Engineering, Wuhan University of Technology. He set up the institute of Logistics and Robotics, the center of IOT and Logistics Technologies, and the international joint lab of Internet of Things. He graduated in mechanical engineering at Zhengzhou Light Industry Institute, and received his Master Diploma from Huazhong University of Science and Technology and his Ph.D. degree from Wuhan University of Technology. Dr. Wenfeng Li is a senior member of both IEEE and China Society of Logistics, TC member of IEEE CSCWD, a Committee member of Teaching Guiding Committee for logistics education of Ministry of Education of China, a Member of Intelligent Manufacture Committee of China. He has contributed to IEEE as a Session Co-Chair of IEEE SMC 2009 to 2013, a Program Committee Co-Chair of the International Working Group on CSCW in Design (2012) and a Program Committee Co-Chair of the international Conference of ICMA 2010 and ICIA 2010. His research interests include Internet of Things, Swarm Robots, Logistics and Supply Chain, Complex Systems.

Please cite this article in press as: G. Fortino et al., A framework for collaborative computing and multi-sensor data fusion in body sensor networks, Informat. Fusion (2014), http://dx.doi.org/10.1016/j.inffus.2014.03.005