- Email: [email protected]
A Real-time Fall Detection System for Maintenance Activities in Indoor Environments*
3rd IFAC Workshop on Advanced Maintenance Engineering, Service and Technology 3rd Workshop Advanced Maintenance October 2016.on Biarritz, France 3rd IFAC IFAC19-21, Workshop on Advanced Maintenance Engineering, Engineering, Service Service and and Technology Technology 3rd IFAC Workshop on Advanced Maintenance Engineering, Service and Technology October October 19-21, 19-21, 2016. 2016. Biarritz, Biarritz, France France Available online at www.sciencedirect.com October 19-21, 2016. Biarritz, France
ScienceDirect A A A D. D. D. D.
IFAC-PapersOnLine 49-28 (2016) 286–290
Real-time Fall Detection System for Real-time Fall Detection System for Real-time FallActivities DetectioninSystem for Maintenance Indoor Maintenance Activities in Indoor Maintenance Activities in Indoor Environments Environments Environments ∗ ∗ ∗ ∗∗
Triantafyllou ∗ S. Krinidis ∗ D. Ioannidis ∗ I.N. Metaxa ∗∗ ∗ S. Krinidis Triantafyllou D. Ioannidis Metaxa ∗∗ ∗∗ ∗ ∗ ∗ Triantafyllou S.Ziazios Krinidis D. Ioannidis I.N. ∗D. ∗ I.N. Triantafyllou ∗C. Krinidis D.Tzovaras Ioannidis I.N. Metaxa Metaxa ∗∗ ∗∗ ∗ ∗∗ D. ∗ C.S.Ziazios Ziazios Tzovaras C. ∗∗ D. Tzovaras ∗ C. Ziazios D. Tzovaras ∗ Information Technologies Institute, Center for Research and ∗ ∗ Information Technologies Institute, Center for Research and Technologies Institute, Research ∗ Information Technology Hellas, Thermi-Thessaloniki, Information Technologies Institute, Center Center for forGreece. Research and and Technology Hellas, Thermi-Thessaloniki, Greece. ∗∗ Technology Hellas, Thermi-Thessaloniki, Greece. Antlantis Engineering, Thessaloniki, Greece. Technology Hellas, Thermi-Thessaloniki, Greece. ∗∗ ∗∗ Antlantis Engineering, Thessaloniki, Greece. ∗∗ Antlantis Engineering, Thessaloniki, Greece. Antlantis Engineering, Thessaloniki, Greece. Abstract: A real-time, multi-camera incident detection system for indoor environments is Abstract: real-time, multi-camera incident for indoorwhile environments is Abstract: A real-time, multi-camera incident detection system for environments is presented inA this paper. The paper focuses on thedetection detectionsystem of fall incidents it highlights Abstract: Athis real-time, multi-camera incident detection system for indoor indoorwhile environments is presented in paper. The paper focuses on the detection of fall incidents it highlights presented in this paper. The paper focuses on the detection of fall incidents while it highlights the leverage that such a system can provide to the human resources department of a shop-floor presented in this paper. The paper focuses on the detection of fall incidents while it highlights the leverage that such system can provide to the human resources department of a shop-floor the leverage that a system can to resources department of especially referring to a the maintenance procedures. The proposed detection method extracts the leverage that such such athe system can provide provide to the the human human resources detection department of aa shop-floor shop-floor especially referring to maintenance procedures. The proposed method extracts especially referring to the maintenance procedures. The proposed detection method extracts features that characterize a falling person’s trajectory, like vertical velocity and area variance, especially referring to the amaintenance procedures. The proposed detectionand method extracts features falling person’s like vertical velocity area features that characterize a person’s trajectory, like velocity and variance, while thethat fall characterize is described by Hidden Markovtrajectory, Models (HMM). The system utilizes onlyvariance, privacy features that characterize a falling falling person’s trajectory, like vertical vertical velocityutilizes and area area variance, while the fall is described by Hidden Markov Models (HMM). The system only privacy while the fall is described by Hidden Markov Models system utilizes only privacy preserving sensors. Experimental results illustrate its (HMM). efficiency.The while the fall is described by Hidden Markov Models (HMM). The system utilizes only privacy preserving sensors. Experimental results illustrate its efficiency. preserving sensors. Experimental results illustrate its preserving sensors. Experimental results illustrate its efficiency. efficiency. © 2016, IFAC (International Federation of Automatic Control) Hosting by Elsevier Ltd. All rights reserved. Keywords: Incident detection, fall, shop-floor, Hidden Markov Models, human resources. Keywords: Incident detection, fall, shop-floor, Hidden Markov Models, human human resources. Keywords: Incident Incident detection, detection, fall, fall, shop-floor, shop-floor, Hidden Hidden Markov Keywords: Markov Models, Models, human resources. resources. 1. INTRODUCTION approaches utilizing the distance from the top or the cen1. INTRODUCTION INTRODUCTION approaches utilizing the distance from the top or 1. approaches utilizing the distance from the the centroid of a person to the as a basic forthe fallcende1. INTRODUCTION approaches utilizing thefloor distance from criterion the top top or or the centroid of aa person to the floor as aa basic criterion for fall detroid of person to the floor as basic criterion for fall tection (Diraco et al. (2010), Kepksi and Kwolek (2014)). Maintenance is not an easy task, especially when main- troid of a person to the floor as a basic criterion for fall dedetection (Diraco et al. (2010), Kepksi and Kwolek (2014)). Maintenance is not an easy task, especially when maintection (Diraco al. (2010), Kwolek (2014)). particular, inet Diraco et al.Kepksi (2010),and a certain threshold Maintenance is not an easy task, especially when maintenance technicians are prompted to approach remote or In tection (Diraco et al. (2010), Kepksi and Kwolek (2014)). Maintenance is not an easy task, especially when mainIn particular, inaDiraco Diraco ettime al. period (2010),ofaa immobility certain threshold threshold tenance technicians are prompted to approach approach or particular, al. (2010), certain to the floor andin specificet on the tenance technicians prompted to remote or In dangerous areas dueare to irritant chemical agents,remote climatic In particular, inaDiraco ettime al. period (2010),ofa immobility certain threshold tenance technicians are prompted to approach remote or to the floor and specific on the dangerous areas due to irritant chemical agents, climatic to the floor and a specific time period of immobility on floor are used as criteria for fall detection while in Kepksi dangerous areas due to irritant chemical agents, climatic conditions etc. Detection of risky or dangerous incidents to the floor and a specific time period of immobility on the the dangerous areas due to irritant chemical agents, climatic floor are used as criteria for fall detection while in Kepksi conditions etc.environment, Detection of of such risky as or adangerous dangerous incidents floor are used as criteria for fall detection while Kepksi Kwolek (2014) a k-NN classifier trained onin features conditions etc. Detection risky or incidents in an indoor shop floor where and floor are used as criteria for fall detection while in Kepksi conditions etc. Detection of risky or dangerous incidents and Kwolek (2014) aa k-NN classifier on features in an indoor indoor activities environment, such as as is shop floor where where and Kwolek (2014) classifier trained on as head-floor areatrained and shape’s major in an environment, such aa floor maintenance are ongoing, a practical prob- such and Kwolek (2014)distance, a k-NN k-NNperson classifier trained on features features in an indoor activities environment, such as is a shop shop floor where such as head-floor distance, person area and shape’s major maintenance are ongoing, a practical probsuch as head-floor distance, person area and shape’s major length to width is utilized. Moreover, there are methods maintenance activities are ongoing, is a practical problem highly affecting workers’ safety. A system monitoring such as head-floor distance, person area and shape’s major maintenance activities are ongoing, issystem a practical prob- length to width is utilized. Moreover, there are methods lem highly affecting workers’ safety. A monitoring length to width is utilized. Moreover, there are methods analysing the person’s movement in a world coordinate lem highly affecting workers’ safety. A system monitoring and recognizing such events could automatically trigger length to width is utilized. Moreover, there are methods lem highly affecting safety. A system monitoring analysing the person’s movement aa world and recognizing suchworkers’ events could automatically trigger analysing the person’s movement in coordinate Skubic (2014), in Zhang et al.coordinate (2012b), and recognizing such automatically trigger the appropriate alarm so thatcould measures dealing with the system(Stone analysing the and person’s movement in a world world coordinate and recognizing alarm such events events could automatically trigger system(Stone and Skubic (2014), Zhang et al. (2012b), the appropriate so that measures dealing with the system(Stone and Skubic (2014), Zhang et al. (2012b), Mastorakis and Makris (2014)). In this category, Stone and the appropriate alarm so that measures dealing with the incident can be taken immediately. A Human Resources and Skubic (2014), Zhang et al.Stone (2012b), the appropriate alarm so that measures dealingResources with the system(Stone Mastorakis and Makris (2014)). In this category, and incident can be taken immediately. A Human Mastorakis and Makris (2014)). In this category, Stone and Skubic (2014) propose an ensemble of decision trees for incident can be taken immediately. A Human Resources Management software tool, connected to the ComputerMastorakis and Makris (2014)). In this category, Stone incident can be taken tool, immediately. A Human Resources Skubic (2014) propose an ensemble of decision trees and for Management software connected to the ComputerSkubic (2014) propose an ensemble of decision trees for the fall’s confidence computation, whereas in Zhang et al. Management software tool, connected to the Computerized Maintenance Management System (CMMS) or the Skubic (2014) propose an ensemble of decision trees for Management software tool, connected to the Computerthe fall’s confidence computation, whereas in Zhang et al. ized Maintenance Management System (CMMS) or the the fall’s confidence computation, whereas in Zhang et (2012b) a Bayesian framework is utilized. In Mastorakis ized Maintenance Management System (CMMS) or the Enterprise Asset Management (EAM) can react fast and fall’saconfidence computation, whereas in Zhang et al. al. ized Maintenance Management(EAM) Systemcan (CMMS) or and the the (2012b) Bayesian framework is utilized. In Mastorakis Enterprise Asset Management react fast (2012b) aa Bayesian utilized. Mastorakis and Makris (2014) framework the velocityis which is In measured by Enterprise Asset Management can react fast and cope with the event depending (EAM) on its importance. (2012b) Bayesian framework is utilized. In Mastorakis Enterprise Asset Management (EAM) can react fast and and Makris (2014) the velocity which is measured by cope with with the the event event depending depending on on its its importance. importance. and Makris the which is by the expansion and contraction of the tracked person’s 3D cope Makris (2014) (2014) the velocity velocity which is measured measured by cope with thecomponent event depending its importance. the expansion and contraction of the tracked person’s 3D An essential of suchonsystem is a fall detector. and the expansion and contraction of the tracked person’s 3D bounding box constitutes the criterion for fall detection. the expansion and contraction of the tracked person’s 3D An essential component of such system is a fall detector. An of is detector. box constitutes the criterion fall detection. Fallsessential could becomponent caused either fromsystem just a simple bounding box constitutes the for fall detection. An essential of such such system is aa fall fallstumbling, detector. bounding Finally, other skeletal for joints in bounding box techniques constitutesutilize the criterion criterion for falltracking detection. Falls could becomponent causedfrom either from just a simple simple stumbling, Falls could be caused either from just a stumbling, Finally, other techniques utilize skeletal joints tracking in could be originated a health problem (faint, heart Finally, other techniques utilize skeletal joints tracking in Falls could be caused either from just a simple stumbling, order to achieve fall detection (Zhang et al. (2012a), Bian other techniques utilize skeletal joints tracking in could beororiginated originated from a health health problem such (faint, heart Finally, could be from a problem (faint, heart order to achieve fall detection (Zhang et al. (2012a), Bian attack) even be the result of an accident, as being order to achieve fall detection (Zhang et al. (2012a), Bian could beororiginated from a health problem such (faint, heart order et al. (2014)). A good survey on(Zhang vision et based fall detection to achieve fall detection al. (2012a), Bian attack) even be the result of an accident, as being attack) or even be the result of an accident, such as being et al. (2014)). A good survey on vision based fall detection hit by an operating device. Furthermore, they could occur et al. (2014)). A good vision attack) oroperating even be the resultFurthermore, of an accident, such as being found in et al. on (2015). et al.be (2014)). A Zhang good survey survey on vision based based fall fall detection detection hit by an device. could occur can hit an operating device. Furthermore, they occur can be found in Zhang et al. (2015). in aby restricted or dangerous area increasingthey the could importance can be found in Zhang et al. (2015). hit by an operating device. Furthermore, they could occur can be found in Zhang et al. (2015). in a restricted or dangerous area increasing the importance in restricted or dangerous dangerous area increasing increasing the the importance importance This paper extends previous work (Krinidis et al. (2014)) of atheir immediate detection. in restricted or area This paper extends previous work (Krinidis et al. (2014)) of atheir their immediate detection. This paper previous work et on a real-time, multi-space, tracking system of immediate detection. paper extends extends previousmulti-camera work (Krinidis (Krinidis et al. al. (2014)) (2014)) of their immediate detection. on a real-time, multi-space, multi-camera tracking system Fall detection has preoccupied researchers for many years. This on real-time, multi-space, multi-camera tracking system to a fall detection system with all the deriving qualities, on a fall real-time, multi-space, multi-camera tracking system Fall detection has preoccupied preoccupied researchers forgroups: many years. years. Fall detection has researchers for many to a detection system with all the deriving qualities, Existing approaches can be divided in two techto a fall detection system with all the deriving qualities, Fall detection has preoccupied researchers for many years. i.e. monitoring any area regardless its size by the use to a fall detection system with all the deriving qualities, Existing approaches can be divided in two groups: techExisting approaches can be divided in two groups: techi.e. monitoring any area regardless size by the use niques using non-vision sensors (Wu and groups: Xue (2008), i.e. monitoring any area regardless its size by the use Existing approaches can be divided in two tech- of multiple depth sensors that retain its workers’ individual i.e. monitoring any area regardless its size by the use niques using non-vision sensors (Wu and Xue (2008), niques non-vision sensors (Wu and Xue multiple depth sensors that retain workers’ individual Shany etusing al. (2012)), especially accelerometers, and(2008), exclu- of of multiple depth sensors that retain workers’ individual niques using non-vision sensors (Wu and Xue (2008), privacy. The introduced detection system is based on key of multiple depth sensors that retain workers’ individual Shany et al. (2012)), especially accelerometers, and excluShany et (2012)), especially accelerometers, and The system is basedvelocity on key sively vision based methods. Since wearable equipment can privacy. privacy. The introduced detection system is on Shany et al. al. based (2012)), especially accelerometers, and excluexclufeatures that introduced characterizedetection a fall, such as vertical privacy. The introduced detection system is based basedvelocity on key key sively vision methods. Since wearable equipment can sively vision based methods. Since wearable equipment can features that characterize a fall, such as vertical be annoying for workers, vision based methods, that are features that characterize a fall, such as vertical velocity sively vision based methods. Sincebased wearable equipment can and areathat variance, while falling process is modelled by features characterize a fall, such as vertical velocity be annoying for workers, vision methods, that are be annoying for workers, vision based methods, that are and area variance, while falling process is modelled by lessannoying intrusive,for areworkers, preferred. Furthermore, depththat sensors and area variance, while falling process is modelled by be vision based methods, are HMM. Furthermore, the system produces alarms that can and area variance, while falling process is modelled by less intrusive, are preferred. Furthermore, depth sensors less intrusive, are preferred. Furthermore, depth sensors HMM. Furthermore, the system produces alarms that can constitute a good solution that takes care of the ethical and HMM. Furthermore, the system produces alarms that can less intrusive, are preferred. Furthermore, depth sensors be used by the human resources department of a shopHMM. Furthermore, the system produces alarms that can constitute a good solution that takes care of the ethical and constitute good solution that care ethical and used by the human resources department of a shoplegal issuesaa of individual privacy. Previous work includes be used by resources department of constitute good solution privacy. that takes takes care of of the the ethical and be floor, triggering immediate response. To our knowledge, be used by the the human human resources department of a a shopshoplegal issues issues of of individual Previous work includes legal individual privacy. Previous work includes floor, triggering immediate response. To our knowledge, floor, triggering immediate response. To our knowledge, legal issues of individual privacy. Previous work includes floor, it is the first work presenting a systemTothat combines all triggering immediate response. our knowledge, This work has been partially supported by the European Commisit is the first work presenting a system that combines all it is the first work presenting aathe system that combines all these characteristics to support maintenance operation This work has been partially supported by the European Commis it is the first work presenting system that combines all sion through the been project HORIZON 2020-INNOVATION This work has partially supported by the EuropeanACTIONS Commis these characteristics to support the maintenance operation This work has been partially supported by the EuropeanACTIONS Commisthese characteristics to support the maintenance operation and in general. sion through the project HORIZON 2020-INNOVATION these characteristics to support the maintenance operation (IA)-636302-SATISFACTORY. sion through the project HORIZON 2020-INNOVATION ACTIONS and in general. sion through the project HORIZON 2020-INNOVATION ACTIONS and (IA)-636302-SATISFACTORY. (IA)-636302-SATISFACTORY. and in in general. general. (IA)-636302-SATISFACTORY. Copyright © 2016, 2016 IFAC 286Hosting by Elsevier Ltd. All rights reserved. 2405-8963 © IFAC (International Federation of Automatic Control) Copyright © 2016 IFAC 286 Copyright ©under 2016 responsibility IFAC 286Control. Peer review of International Federation of Automatic Copyright © 2016 IFAC 286 10.1016/j.ifacol.2016.11.049
2016 IFAC AMEST October 19-21, 2016. Biarritz, France
D. Triantafyllou et al. / IFAC-PapersOnLine 49-28 (2016) 286–290
287
To sum up, the main contributions of this paper are: (1) The introduction of a real-time, multi-space, multicamera fall detection system. (2) The fall process modelling by an HMM based on the falling person’s velocity and area variance from top view. The remainder of the paper is organized as follows: section 2 describes the methodology of the presented approach, section 3 analyses the importance and leverage that a system like the proposed one can offer to the human resources department of a shop-floor while section 4 includes the experimental results. The paper concludes with discussion on the proposed approach. 2. INCIDENT DETECTION METHOD The presented incident detection methodology comprises three steps: 1) detection and tracking of moving items, 2) extraction of event features that are indicative of the items’ state, 3)an HMM method that recognizes the occurring incidents based on the event features. 2.1 Detection and tracking of moving items In the first stage of the proposed method, a real-time, robust tracking system is used ( Krinidis et al. (2014) ). The utilized camera calibration algorithm allows the use of multiple cameras that refer to a common coordinate system located on the architectural map of the shopfloor’s building. This fact enables the monitoring of any area regardless its size. Furthermore, partial occlusions are handled by deploying a virtual top-view camera based on calibration data. Thus, the overall detection - tracking procedure remains unaffected since it is performed on the horizontal plane. In addition, dynamic changes of the environment can be encountered by a dual-band algorithm that incorporates to the background low objects, e.g. a chair, in a small period of time while retains for a longer period higher objects, such as humans. 2.2 Extraction of event features While the detection of an event like intrusion to a forbidden area is straightforward once tracking is achieved, incidents such as a worker’s fall need further processing. Therefore, event features, indicative of the tracked items state, are extracted.
where h (t) represents the tracked person’s height derivative (for brevity it will be referred as h), tc is the current time and tc − t0 = CT is a constant time window. There are cases where partial occlusion might abruptly cut a big portion of the worker’s blob upper part. In this case velocity forms a step signal and can lead to false alarms for fall detection. In order to deal with this phenomenon, once a step signal is detected, the velocities that include it in their calculation are set to zero. (2) Area variance σ 2 . As a person is falling, its area, measured from a top view, is gradually augmenting. This feature is captured by the variance of the area, in the same time window as velocity, in order to be independent from the initial area of a person before falling and relatively robust to noise. The area variance formula is the following: 2 tc tc 1 A(t ) A(t) − dt dt, (2) σ2 = tc − t 0 t c − t0 t0
2.3 Incident recognition A three state Markov model that takes into account the aforementioned event features is used in order to achieve fall detection. The first state (S1 ) refers to a non-falling state, e.g. a human walking or standing. The second state (S2 ) represents the actual fall which is characterized by highly decreasing vertical velocity (when the height decreases the velocity takes negative values) and augmenting area variance. The third state (S3 ) signifies the end of the fall and declares the detection of the incident. The transition probabilities are based on the event features and are defined in the following matrix: (1-F ) F 0 P = (1-F )(1-u) F (1-F )u , (3) 1 0 0 where F =
For the fall detection incident the aforementioned features are: (1) Vertical velocity v. A characteristic feature of a fall is the vertical velocity of the tracked person’s highest point. Nevertheless, depth sensors often provide noisy data that affect the height’s value. Thus, the mean velocity of a constant time window is calculated while a six order, low pass FIR filter with cut-off frequency 3Hz is applied on the corresponding heights (the order and cut off frequency of the filter were determined experimentally). The velocity’s formula is: tc 1 v= h (t)dt, (1) tc − t 0 t0
287
t0
where A(t) represents the area of the tracked human calculated from top view. (3) Height h. Apart from its importance to vertical velocity calculation, the value of the highest point of the person under detection facilitates the avoidance of false alarms. For example, the final height of a fallen person cannot be higher than 1 meter.
and u=
1 , 1 + evσ2 +T
(4)
0, HT -h≥ 0 , 1, HT -h<0
(5)
where T is a constant defined by training and HT is a constant threshold. The probability F constitutes a sigmoid function that favours with high values close to 1 cases with high (negative) vertical velocity and high area covariance, i.e. cases that correspond to the state of falling. Moreover function u declares that state S3 that signifies the detection of the fall cannot be reached if the fallen person is not below a loose threshold HT .
2016 IFAC AMEST 288 October 19-21, 2016. Biarritz, France
D. Triantafyllou et al. / IFAC-PapersOnLine 49-28 (2016) 286–290
allocating the minimum of personnel in the problematic area. It is of interest to more carefully examine the possibilities deriving from the detection of a fall of a person in a working environment in industrial manufacturing facilities. Several alternative cases can be discussed to support this point (Fig. 2). In case A there is a person who falls and then gets up. Here, there is no need to take immediate action. However, at the backlog of health and safety issues this could remain as a near-accident to be investigated. It could lead to the improvement of the facilities and the removal of potential causes of small accidents. In case B1 a person falls and does not get up in an area where no special conditions are imposed or dangerous material are used. The detection of such an incident triggers an alarm or event to the HR management tool for sending at the specific site a member of the team to check on the well-being of their co-worker. Thus, it would facilitate the provision of immediate assistance to the person hurt. In the case B2 a person falls and does not get up in a dangerous area (climatic conditions that do not allow for long human presence, use of chemicals, leakage of chemicals, in the vicinity of machinery with moving parts etc.). It is imperative then for the system to rise an alarm for human intervention as soon as possible, having as a first target the removal of the hurt person. The quick response can make a big difference in this case. This is the reason why the common practice is that the maintenance personnel involved in such tasks has been training in providing first aid.
Fig. 1. Three state Markov Model
Fig. 2. Human Resources coping mechanism’s diagram for fall incidents. 3. EVENT DETECTION AND HUMAN RESOURCE MANAGEMENT The incident detection methodology described refers mainly at this point at the detection of a person from the maintenance team entering in a restricted area and of falls of personnel. The ability to detect such types of incidents can be capitalized for the improvement of human resources (HR) management and allocation. It is possible to obtain information about personnel’s presence and movement at the shop floor by using the depth cameras or even with small wearable devices embedded in the suit of the maintenance technicians. This knowledge can be not only useful, but crucial in the case of a severe accident during the implementation of maintenance works. Tools such as those presented here offer a great potentiality for incident management and the ability to improve the management and safety of the maintenance operation. When the analysis shows a potential event, based on specific features, the incident can be fed as a trigger to an automated system to assess its importance. The system could detect the presence of maintenance -in our case- personnel at a restricted area. In the case of ongoing corrective or preventive maintenance activities at hazardous areas, this information can be very important in order to remotely inspect and supervise such an endeavour. It also enables 288
In general, each incident detected has the potential of being assessed based on a set of criteria such as the following: the area where the incident is indicated, the status of the area (restricted to all, work ongoing, under maintenance, accessible to a specific group of people), the condition of the area (ongoing production process, leakage of material detected, there is knowledge of an accident that has occurred there, explosive environment etc.). Consequently, the information deducted form the described method can be proven extremely valuable for human resource management of the maintenance team, especially coupled with health and safety issues. 4. EXPERIMENTS 4.1 Dataset To evaluate the proposed fall detection method a dataset with 70 events performed by 7 different persons was acquired. These events include 40 falls and 30 events with features similar to falling, such as a person bending to tie his shoes or to pick up an item from the floor (Fig. 3). These events are examined in order to test the robustness of the algorithm in cases that affect the event features that determine the transition to the HMM states, i.e. vertical velocity and area variance. Furthermore, the participants were allowed to fall in ways that felt natural to them, leading to a variety of falls, e.g. fall forwards, backwards, sideways, fall while walking, like stumbling, or while standing, as fainting. Finally, since the incident detection scenario takes place in a shop-floor, there were people walking or standing in the monitoring area during the experiments.
2016 IFAC AMEST October 19-21, 2016. Biarritz, France
D. Triantafyllou et al. / IFAC-PapersOnLine 49-28 (2016) 286–290
289
Fig. 4. Filtering of the height. The first image shows the raw height signal while in the second the filtered output is depicted.
Fig. 5. Step signal of height.
Fig. 3. Events included in the dataset. The first two rows correspond to fall events, the third and forth row to bending and picking up items while the last to bending in order to tie shoes. The experiment was conducted in two different working environments: CPERI and ITI institutes of CERTH in Greece. In both cases two depth sensors (Kinect cameras for Xbox360) monitoring subsequent areas were used. 4.2 Event features processing As it is already mentioned in section 2.2 the noisy data acquired from depth sensors can lead to false alarms. Apart from FIR filtering that smooths the height (Fig. 4), the algorithm detects step signals occurring (Fig. 5) either from occlusions with real items or sometimes, with holes in the depth images that can be caused by shiny or black items. Such an example can be seen in Fig. 5. The abrupt height change from 1200mm to less than 600mm indicates a step signal caused by occlusion, hence it should not be mistaken as a fall. Thus, the two aforementioned measures prevent big deviations between the calculated velocities and their actual values. In Fig. 6 graphs of the event features during a fall are depicted. As it can be seen in the height graph (Fig. 6.a) the tracked person is initially in a standing position, then falls, remains lying on the ground for a while and finally stands up. Velocity and area variance values change drastically because of the fall (Fig. 6.b and Fig. 6.c) but 289
with a very small time delay (less than a second) due to averaging (section 2.2). Moreover, Fig. 6.d depicts the probability F that is crucial for the HMM’s transitions. Its value raises fast due to fall, then decreases and remains low for the time the person is lying on the floor and finally slightly augments, remaining below 0.5, while the person rises. At this point, it should be mentioned that sensors’ frames were acquired with 15 fps (this value depends on the utilized computer computational power). Furthermore, time window CT is about 0.65 seconds (so that about 10 different values are averaged for the velocity and area calculation) , threshold T is 10−6 (based on a training set of 10 falls) while HT is 1 meter. 4.3 Experimental results Table 1. Fall Detection Results True Positives 40
True Negatives 28
False Positives 2
Table 2. Fall Detection Performance Precision 95.24%
Recall 100%
F1-score 97.56%
The proposed algorithm was tested in more than 50000 frames including falls, events similar to falls but also people walking or standing in the monitored area. All the 40 falls were detected correctly, while there were 2 false positives in the whole dataset. These false alarms corresponded to 2 out of the 30 similar to fall events where a participant bended and picked up the mattress used for falls. It has to be mentioned that this task was
2016 IFAC AMEST 290 October 19-21, 2016. Biarritz, France
D. Triantafyllou et al. / IFAC-PapersOnLine 49-28 (2016) 286–290
Incorporation of more incidents to the presented system, like a falling item or collision events, are planned as future work. REFERENCES
Fig. 6. Event features and transition probability F graphs during a fall: (a) height, (b) velocity, (c) area variance, (d) transition probability. replicated 5 times but only in 2 of them produced false alarm. Fall detection results on the acquired dataset are summarized at tables 1 and 2. The recall rate is 100% while precision is 95.24% leading to a rather high F1-score that equals 97.56%. The results refer only to falls and similar to fall events. Nevertheless, people just walking or standing at the monitoring area during the experiments did not produce any false positives, even though they were not taken into account at the results. Furthermore, the algorithm was tested in 15 falling cases of an on-line dataset (http://fenix.univ.rzeszow.pl/∼mkepski/ds/uf.html) and detected them successfully without any false alarms. 5. CONCLUSION This paper presents a real-time, multi-camera incident detection system for shop-floor applications. The proposed system can be a great asset for maintenance activities especially if they take place in remote or dangerous areas since it can notify of accidents or hazardous events. Thus, the appropriate coping mechanism can be immediately triggered and deal with the incident. Furthermore, the proposed system provides a sense of safety to workers without entrenching their privacy rights since only depth sensors are utilized. The paper focuses on the detection of fall incidents. Fall process is modelled by HMM based on features that characterize a fall event such as vertical velocity and area variance from top view of the tracked person. Experimental results are very promising since only cases of bending to pick up huge items, such as the mattress, caused false alarms. Nevertheless, such cases are probably rare in a shop-floor. Moreover, all the real falls were detected without misses. 290
Bian, Z., Hou, J., Chau, L., and Magnenat-Thalmann, N. (2014). Fall detection based on body part tracking using a depth camera. IEEE journal of biomedical and health informatics. Diraco, G., Leone, A., and Siciliano, P. (2010). An active vision system for fall detection and posture recognition in elderly health care. Design, Automation and Test in Europe Conference and Exhibition, 1536–1541. Kepksi, M. and Kwolek, B. (2014). Fall detection using ceiling-mounted 3d depth camera. International Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, 640–647. Krinidis, S., Stavropoulos, G., Ioannidis, D., and Tzovaras, D. (2014). A robust and real-time multispace occupancy extraction system exploiting privacy-preserving sensors. ISCCSP. Mastorakis, G. and Makris, D. (2014). Fall detection system using kinect as infrared sensor. Real-Time Image Processing, 635–646. Shany, T., Redmond, S., Narayanan, M., and Lovell, N. (2012). Sensors-based wearable systems for monitoring of human fals. IEEE Sensors journal, 658–670. Stone, E. and Skubic, M. (2014). Fall detection in homes of older adults using the microsoft kinect. IEEE journal of biomedical and health informatics, 1590–1594. Wu, G. and Xue, S. (2008). Portable preimpact fal detector with wearable sensors. IEEE Transactions on Neural Systems and Rehabiliation Enginneering, 178–183. Zhang, C., Tian, Y., and Capezuti, E. (2012a). Privacy preserving automatic fall detection for eldery using rgbd cameras. International Conference on Computers Helping People with Special Needs, 625–633. Zhang, Z., Conly, C., and Anthitsos, V. (2015). A survey on vision-based fall detection. International Conference on pervasive technologies related to assistive environments. Zhang, Z., Liu, W., Metsis, V., and Anthitsos, V. (2012b). A viewpoint-independent statistical method for fall detection. IEEE Conference on Pattern Recognition, 3626– 3630.
ScienceDirect A A A D. D. D. D.
IFAC-PapersOnLine 49-28 (2016) 286–290
Real-time Fall Detection System for Real-time Fall Detection System for Real-time FallActivities DetectioninSystem for Maintenance Indoor Maintenance Activities in Indoor Maintenance Activities in Indoor Environments Environments Environments ∗ ∗ ∗ ∗∗
Triantafyllou ∗ S. Krinidis ∗ D. Ioannidis ∗ I.N. Metaxa ∗∗ ∗ S. Krinidis Triantafyllou D. Ioannidis Metaxa ∗∗ ∗∗ ∗ ∗ ∗ Triantafyllou S.Ziazios Krinidis D. Ioannidis I.N. ∗D. ∗ I.N. Triantafyllou ∗C. Krinidis D.Tzovaras Ioannidis I.N. Metaxa Metaxa ∗∗ ∗∗ ∗ ∗∗ D. ∗ C.S.Ziazios Ziazios Tzovaras C. ∗∗ D. Tzovaras ∗ C. Ziazios D. Tzovaras ∗ Information Technologies Institute, Center for Research and ∗ ∗ Information Technologies Institute, Center for Research and Technologies Institute, Research ∗ Information Technology Hellas, Thermi-Thessaloniki, Information Technologies Institute, Center Center for forGreece. Research and and Technology Hellas, Thermi-Thessaloniki, Greece. ∗∗ Technology Hellas, Thermi-Thessaloniki, Greece. Antlantis Engineering, Thessaloniki, Greece. Technology Hellas, Thermi-Thessaloniki, Greece. ∗∗ ∗∗ Antlantis Engineering, Thessaloniki, Greece. ∗∗ Antlantis Engineering, Thessaloniki, Greece. Antlantis Engineering, Thessaloniki, Greece. Abstract: A real-time, multi-camera incident detection system for indoor environments is Abstract: real-time, multi-camera incident for indoorwhile environments is Abstract: A real-time, multi-camera incident detection system for environments is presented inA this paper. The paper focuses on thedetection detectionsystem of fall incidents it highlights Abstract: Athis real-time, multi-camera incident detection system for indoor indoorwhile environments is presented in paper. The paper focuses on the detection of fall incidents it highlights presented in this paper. The paper focuses on the detection of fall incidents while it highlights the leverage that such a system can provide to the human resources department of a shop-floor presented in this paper. The paper focuses on the detection of fall incidents while it highlights the leverage that such system can provide to the human resources department of a shop-floor the leverage that a system can to resources department of especially referring to a the maintenance procedures. The proposed detection method extracts the leverage that such such athe system can provide provide to the the human human resources detection department of aa shop-floor shop-floor especially referring to maintenance procedures. The proposed method extracts especially referring to the maintenance procedures. The proposed detection method extracts features that characterize a falling person’s trajectory, like vertical velocity and area variance, especially referring to the amaintenance procedures. The proposed detectionand method extracts features falling person’s like vertical velocity area features that characterize a person’s trajectory, like velocity and variance, while thethat fall characterize is described by Hidden Markovtrajectory, Models (HMM). The system utilizes onlyvariance, privacy features that characterize a falling falling person’s trajectory, like vertical vertical velocityutilizes and area area variance, while the fall is described by Hidden Markov Models (HMM). The system only privacy while the fall is described by Hidden Markov Models system utilizes only privacy preserving sensors. Experimental results illustrate its (HMM). efficiency.The while the fall is described by Hidden Markov Models (HMM). The system utilizes only privacy preserving sensors. Experimental results illustrate its efficiency. preserving sensors. Experimental results illustrate its preserving sensors. Experimental results illustrate its efficiency. efficiency. © 2016, IFAC (International Federation of Automatic Control) Hosting by Elsevier Ltd. All rights reserved. Keywords: Incident detection, fall, shop-floor, Hidden Markov Models, human resources. Keywords: Incident detection, fall, shop-floor, Hidden Markov Models, human human resources. Keywords: Incident Incident detection, detection, fall, fall, shop-floor, shop-floor, Hidden Hidden Markov Keywords: Markov Models, Models, human resources. resources. 1. INTRODUCTION approaches utilizing the distance from the top or the cen1. INTRODUCTION INTRODUCTION approaches utilizing the distance from the top or 1. approaches utilizing the distance from the the centroid of a person to the as a basic forthe fallcende1. INTRODUCTION approaches utilizing thefloor distance from criterion the top top or or the centroid of aa person to the floor as aa basic criterion for fall detroid of person to the floor as basic criterion for fall tection (Diraco et al. (2010), Kepksi and Kwolek (2014)). Maintenance is not an easy task, especially when main- troid of a person to the floor as a basic criterion for fall dedetection (Diraco et al. (2010), Kepksi and Kwolek (2014)). Maintenance is not an easy task, especially when maintection (Diraco al. (2010), Kwolek (2014)). particular, inet Diraco et al.Kepksi (2010),and a certain threshold Maintenance is not an easy task, especially when maintenance technicians are prompted to approach remote or In tection (Diraco et al. (2010), Kepksi and Kwolek (2014)). Maintenance is not an easy task, especially when mainIn particular, inaDiraco Diraco ettime al. period (2010),ofaa immobility certain threshold threshold tenance technicians are prompted to approach approach or particular, al. (2010), certain to the floor andin specificet on the tenance technicians prompted to remote or In dangerous areas dueare to irritant chemical agents,remote climatic In particular, inaDiraco ettime al. period (2010),ofa immobility certain threshold tenance technicians are prompted to approach remote or to the floor and specific on the dangerous areas due to irritant chemical agents, climatic to the floor and a specific time period of immobility on floor are used as criteria for fall detection while in Kepksi dangerous areas due to irritant chemical agents, climatic conditions etc. Detection of risky or dangerous incidents to the floor and a specific time period of immobility on the the dangerous areas due to irritant chemical agents, climatic floor are used as criteria for fall detection while in Kepksi conditions etc.environment, Detection of of such risky as or adangerous dangerous incidents floor are used as criteria for fall detection while Kepksi Kwolek (2014) a k-NN classifier trained onin features conditions etc. Detection risky or incidents in an indoor shop floor where and floor are used as criteria for fall detection while in Kepksi conditions etc. Detection of risky or dangerous incidents and Kwolek (2014) aa k-NN classifier on features in an indoor indoor activities environment, such as as is shop floor where where and Kwolek (2014) classifier trained on as head-floor areatrained and shape’s major in an environment, such aa floor maintenance are ongoing, a practical prob- such and Kwolek (2014)distance, a k-NN k-NNperson classifier trained on features features in an indoor activities environment, such as is a shop shop floor where such as head-floor distance, person area and shape’s major maintenance are ongoing, a practical probsuch as head-floor distance, person area and shape’s major length to width is utilized. Moreover, there are methods maintenance activities are ongoing, is a practical problem highly affecting workers’ safety. A system monitoring such as head-floor distance, person area and shape’s major maintenance activities are ongoing, issystem a practical prob- length to width is utilized. Moreover, there are methods lem highly affecting workers’ safety. A monitoring length to width is utilized. Moreover, there are methods analysing the person’s movement in a world coordinate lem highly affecting workers’ safety. A system monitoring and recognizing such events could automatically trigger length to width is utilized. Moreover, there are methods lem highly affecting safety. A system monitoring analysing the person’s movement aa world and recognizing suchworkers’ events could automatically trigger analysing the person’s movement in coordinate Skubic (2014), in Zhang et al.coordinate (2012b), and recognizing such automatically trigger the appropriate alarm so thatcould measures dealing with the system(Stone analysing the and person’s movement in a world world coordinate and recognizing alarm such events events could automatically trigger system(Stone and Skubic (2014), Zhang et al. (2012b), the appropriate so that measures dealing with the system(Stone and Skubic (2014), Zhang et al. (2012b), Mastorakis and Makris (2014)). In this category, Stone and the appropriate alarm so that measures dealing with the incident can be taken immediately. A Human Resources and Skubic (2014), Zhang et al.Stone (2012b), the appropriate alarm so that measures dealingResources with the system(Stone Mastorakis and Makris (2014)). In this category, and incident can be taken immediately. A Human Mastorakis and Makris (2014)). In this category, Stone and Skubic (2014) propose an ensemble of decision trees for incident can be taken immediately. A Human Resources Management software tool, connected to the ComputerMastorakis and Makris (2014)). In this category, Stone incident can be taken tool, immediately. A Human Resources Skubic (2014) propose an ensemble of decision trees and for Management software connected to the ComputerSkubic (2014) propose an ensemble of decision trees for the fall’s confidence computation, whereas in Zhang et al. Management software tool, connected to the Computerized Maintenance Management System (CMMS) or the Skubic (2014) propose an ensemble of decision trees for Management software tool, connected to the Computerthe fall’s confidence computation, whereas in Zhang et al. ized Maintenance Management System (CMMS) or the the fall’s confidence computation, whereas in Zhang et (2012b) a Bayesian framework is utilized. In Mastorakis ized Maintenance Management System (CMMS) or the Enterprise Asset Management (EAM) can react fast and fall’saconfidence computation, whereas in Zhang et al. al. ized Maintenance Management(EAM) Systemcan (CMMS) or and the the (2012b) Bayesian framework is utilized. In Mastorakis Enterprise Asset Management react fast (2012b) aa Bayesian utilized. Mastorakis and Makris (2014) framework the velocityis which is In measured by Enterprise Asset Management can react fast and cope with the event depending (EAM) on its importance. (2012b) Bayesian framework is utilized. In Mastorakis Enterprise Asset Management (EAM) can react fast and and Makris (2014) the velocity which is measured by cope with with the the event event depending depending on on its its importance. importance. and Makris the which is by the expansion and contraction of the tracked person’s 3D cope Makris (2014) (2014) the velocity velocity which is measured measured by cope with thecomponent event depending its importance. the expansion and contraction of the tracked person’s 3D An essential of suchonsystem is a fall detector. and the expansion and contraction of the tracked person’s 3D bounding box constitutes the criterion for fall detection. the expansion and contraction of the tracked person’s 3D An essential component of such system is a fall detector. An of is detector. box constitutes the criterion fall detection. Fallsessential could becomponent caused either fromsystem just a simple bounding box constitutes the for fall detection. An essential of such such system is aa fall fallstumbling, detector. bounding Finally, other skeletal for joints in bounding box techniques constitutesutilize the criterion criterion for falltracking detection. Falls could becomponent causedfrom either from just a simple simple stumbling, Falls could be caused either from just a stumbling, Finally, other techniques utilize skeletal joints tracking in could be originated a health problem (faint, heart Finally, other techniques utilize skeletal joints tracking in Falls could be caused either from just a simple stumbling, order to achieve fall detection (Zhang et al. (2012a), Bian other techniques utilize skeletal joints tracking in could beororiginated originated from a health health problem such (faint, heart Finally, could be from a problem (faint, heart order to achieve fall detection (Zhang et al. (2012a), Bian attack) even be the result of an accident, as being order to achieve fall detection (Zhang et al. (2012a), Bian could beororiginated from a health problem such (faint, heart order et al. (2014)). A good survey on(Zhang vision et based fall detection to achieve fall detection al. (2012a), Bian attack) even be the result of an accident, as being attack) or even be the result of an accident, such as being et al. (2014)). A good survey on vision based fall detection hit by an operating device. Furthermore, they could occur et al. (2014)). A good vision attack) oroperating even be the resultFurthermore, of an accident, such as being found in et al. on (2015). et al.be (2014)). A Zhang good survey survey on vision based based fall fall detection detection hit by an device. could occur can hit an operating device. Furthermore, they occur can be found in Zhang et al. (2015). in aby restricted or dangerous area increasingthey the could importance can be found in Zhang et al. (2015). hit by an operating device. Furthermore, they could occur can be found in Zhang et al. (2015). in a restricted or dangerous area increasing the importance in restricted or dangerous dangerous area increasing increasing the the importance importance This paper extends previous work (Krinidis et al. (2014)) of atheir immediate detection. in restricted or area This paper extends previous work (Krinidis et al. (2014)) of atheir their immediate detection. This paper previous work et on a real-time, multi-space, tracking system of immediate detection. paper extends extends previousmulti-camera work (Krinidis (Krinidis et al. al. (2014)) (2014)) of their immediate detection. on a real-time, multi-space, multi-camera tracking system Fall detection has preoccupied researchers for many years. This on real-time, multi-space, multi-camera tracking system to a fall detection system with all the deriving qualities, on a fall real-time, multi-space, multi-camera tracking system Fall detection has preoccupied preoccupied researchers forgroups: many years. years. Fall detection has researchers for many to a detection system with all the deriving qualities, Existing approaches can be divided in two techto a fall detection system with all the deriving qualities, Fall detection has preoccupied researchers for many years. i.e. monitoring any area regardless its size by the use to a fall detection system with all the deriving qualities, Existing approaches can be divided in two groups: techExisting approaches can be divided in two groups: techi.e. monitoring any area regardless size by the use niques using non-vision sensors (Wu and groups: Xue (2008), i.e. monitoring any area regardless its size by the use Existing approaches can be divided in two tech- of multiple depth sensors that retain its workers’ individual i.e. monitoring any area regardless its size by the use niques using non-vision sensors (Wu and Xue (2008), niques non-vision sensors (Wu and Xue multiple depth sensors that retain workers’ individual Shany etusing al. (2012)), especially accelerometers, and(2008), exclu- of of multiple depth sensors that retain workers’ individual niques using non-vision sensors (Wu and Xue (2008), privacy. The introduced detection system is based on key of multiple depth sensors that retain workers’ individual Shany et al. (2012)), especially accelerometers, and excluShany et (2012)), especially accelerometers, and The system is basedvelocity on key sively vision based methods. Since wearable equipment can privacy. privacy. The introduced detection system is on Shany et al. al. based (2012)), especially accelerometers, and excluexclufeatures that introduced characterizedetection a fall, such as vertical privacy. The introduced detection system is based basedvelocity on key key sively vision methods. Since wearable equipment can sively vision based methods. Since wearable equipment can features that characterize a fall, such as vertical be annoying for workers, vision based methods, that are features that characterize a fall, such as vertical velocity sively vision based methods. Sincebased wearable equipment can and areathat variance, while falling process is modelled by features characterize a fall, such as vertical velocity be annoying for workers, vision methods, that are be annoying for workers, vision based methods, that are and area variance, while falling process is modelled by lessannoying intrusive,for areworkers, preferred. Furthermore, depththat sensors and area variance, while falling process is modelled by be vision based methods, are HMM. Furthermore, the system produces alarms that can and area variance, while falling process is modelled by less intrusive, are preferred. Furthermore, depth sensors less intrusive, are preferred. Furthermore, depth sensors HMM. Furthermore, the system produces alarms that can constitute a good solution that takes care of the ethical and HMM. Furthermore, the system produces alarms that can less intrusive, are preferred. Furthermore, depth sensors be used by the human resources department of a shopHMM. Furthermore, the system produces alarms that can constitute a good solution that takes care of the ethical and constitute good solution that care ethical and used by the human resources department of a shoplegal issuesaa of individual privacy. Previous work includes be used by resources department of constitute good solution privacy. that takes takes care of of the the ethical and be floor, triggering immediate response. To our knowledge, be used by the the human human resources department of a a shopshoplegal issues issues of of individual Previous work includes legal individual privacy. Previous work includes floor, triggering immediate response. To our knowledge, floor, triggering immediate response. To our knowledge, legal issues of individual privacy. Previous work includes floor, it is the first work presenting a systemTothat combines all triggering immediate response. our knowledge, This work has been partially supported by the European Commisit is the first work presenting a system that combines all it is the first work presenting aathe system that combines all these characteristics to support maintenance operation This work has been partially supported by the European Commis it is the first work presenting system that combines all sion through the been project HORIZON 2020-INNOVATION This work has partially supported by the EuropeanACTIONS Commis these characteristics to support the maintenance operation This work has been partially supported by the EuropeanACTIONS Commisthese characteristics to support the maintenance operation and in general. sion through the project HORIZON 2020-INNOVATION these characteristics to support the maintenance operation (IA)-636302-SATISFACTORY. sion through the project HORIZON 2020-INNOVATION ACTIONS and in general. sion through the project HORIZON 2020-INNOVATION ACTIONS and (IA)-636302-SATISFACTORY. (IA)-636302-SATISFACTORY. and in in general. general. (IA)-636302-SATISFACTORY. Copyright © 2016, 2016 IFAC 286Hosting by Elsevier Ltd. All rights reserved. 2405-8963 © IFAC (International Federation of Automatic Control) Copyright © 2016 IFAC 286 Copyright ©under 2016 responsibility IFAC 286Control. Peer review of International Federation of Automatic Copyright © 2016 IFAC 286 10.1016/j.ifacol.2016.11.049
2016 IFAC AMEST October 19-21, 2016. Biarritz, France
D. Triantafyllou et al. / IFAC-PapersOnLine 49-28 (2016) 286–290
287
To sum up, the main contributions of this paper are: (1) The introduction of a real-time, multi-space, multicamera fall detection system. (2) The fall process modelling by an HMM based on the falling person’s velocity and area variance from top view. The remainder of the paper is organized as follows: section 2 describes the methodology of the presented approach, section 3 analyses the importance and leverage that a system like the proposed one can offer to the human resources department of a shop-floor while section 4 includes the experimental results. The paper concludes with discussion on the proposed approach. 2. INCIDENT DETECTION METHOD The presented incident detection methodology comprises three steps: 1) detection and tracking of moving items, 2) extraction of event features that are indicative of the items’ state, 3)an HMM method that recognizes the occurring incidents based on the event features. 2.1 Detection and tracking of moving items In the first stage of the proposed method, a real-time, robust tracking system is used ( Krinidis et al. (2014) ). The utilized camera calibration algorithm allows the use of multiple cameras that refer to a common coordinate system located on the architectural map of the shopfloor’s building. This fact enables the monitoring of any area regardless its size. Furthermore, partial occlusions are handled by deploying a virtual top-view camera based on calibration data. Thus, the overall detection - tracking procedure remains unaffected since it is performed on the horizontal plane. In addition, dynamic changes of the environment can be encountered by a dual-band algorithm that incorporates to the background low objects, e.g. a chair, in a small period of time while retains for a longer period higher objects, such as humans. 2.2 Extraction of event features While the detection of an event like intrusion to a forbidden area is straightforward once tracking is achieved, incidents such as a worker’s fall need further processing. Therefore, event features, indicative of the tracked items state, are extracted.
where h (t) represents the tracked person’s height derivative (for brevity it will be referred as h), tc is the current time and tc − t0 = CT is a constant time window. There are cases where partial occlusion might abruptly cut a big portion of the worker’s blob upper part. In this case velocity forms a step signal and can lead to false alarms for fall detection. In order to deal with this phenomenon, once a step signal is detected, the velocities that include it in their calculation are set to zero. (2) Area variance σ 2 . As a person is falling, its area, measured from a top view, is gradually augmenting. This feature is captured by the variance of the area, in the same time window as velocity, in order to be independent from the initial area of a person before falling and relatively robust to noise. The area variance formula is the following: 2 tc tc 1 A(t ) A(t) − dt dt, (2) σ2 = tc − t 0 t c − t0 t0
2.3 Incident recognition A three state Markov model that takes into account the aforementioned event features is used in order to achieve fall detection. The first state (S1 ) refers to a non-falling state, e.g. a human walking or standing. The second state (S2 ) represents the actual fall which is characterized by highly decreasing vertical velocity (when the height decreases the velocity takes negative values) and augmenting area variance. The third state (S3 ) signifies the end of the fall and declares the detection of the incident. The transition probabilities are based on the event features and are defined in the following matrix: (1-F ) F 0 P = (1-F )(1-u) F (1-F )u , (3) 1 0 0 where F =
For the fall detection incident the aforementioned features are: (1) Vertical velocity v. A characteristic feature of a fall is the vertical velocity of the tracked person’s highest point. Nevertheless, depth sensors often provide noisy data that affect the height’s value. Thus, the mean velocity of a constant time window is calculated while a six order, low pass FIR filter with cut-off frequency 3Hz is applied on the corresponding heights (the order and cut off frequency of the filter were determined experimentally). The velocity’s formula is: tc 1 v= h (t)dt, (1) tc − t 0 t0
287
t0
where A(t) represents the area of the tracked human calculated from top view. (3) Height h. Apart from its importance to vertical velocity calculation, the value of the highest point of the person under detection facilitates the avoidance of false alarms. For example, the final height of a fallen person cannot be higher than 1 meter.
and u=
1 , 1 + evσ2 +T
(4)
0, HT -h≥ 0 , 1, HT -h<0
(5)
where T is a constant defined by training and HT is a constant threshold. The probability F constitutes a sigmoid function that favours with high values close to 1 cases with high (negative) vertical velocity and high area covariance, i.e. cases that correspond to the state of falling. Moreover function u declares that state S3 that signifies the detection of the fall cannot be reached if the fallen person is not below a loose threshold HT .
2016 IFAC AMEST 288 October 19-21, 2016. Biarritz, France
D. Triantafyllou et al. / IFAC-PapersOnLine 49-28 (2016) 286–290
allocating the minimum of personnel in the problematic area. It is of interest to more carefully examine the possibilities deriving from the detection of a fall of a person in a working environment in industrial manufacturing facilities. Several alternative cases can be discussed to support this point (Fig. 2). In case A there is a person who falls and then gets up. Here, there is no need to take immediate action. However, at the backlog of health and safety issues this could remain as a near-accident to be investigated. It could lead to the improvement of the facilities and the removal of potential causes of small accidents. In case B1 a person falls and does not get up in an area where no special conditions are imposed or dangerous material are used. The detection of such an incident triggers an alarm or event to the HR management tool for sending at the specific site a member of the team to check on the well-being of their co-worker. Thus, it would facilitate the provision of immediate assistance to the person hurt. In the case B2 a person falls and does not get up in a dangerous area (climatic conditions that do not allow for long human presence, use of chemicals, leakage of chemicals, in the vicinity of machinery with moving parts etc.). It is imperative then for the system to rise an alarm for human intervention as soon as possible, having as a first target the removal of the hurt person. The quick response can make a big difference in this case. This is the reason why the common practice is that the maintenance personnel involved in such tasks has been training in providing first aid.
Fig. 1. Three state Markov Model
Fig. 2. Human Resources coping mechanism’s diagram for fall incidents. 3. EVENT DETECTION AND HUMAN RESOURCE MANAGEMENT The incident detection methodology described refers mainly at this point at the detection of a person from the maintenance team entering in a restricted area and of falls of personnel. The ability to detect such types of incidents can be capitalized for the improvement of human resources (HR) management and allocation. It is possible to obtain information about personnel’s presence and movement at the shop floor by using the depth cameras or even with small wearable devices embedded in the suit of the maintenance technicians. This knowledge can be not only useful, but crucial in the case of a severe accident during the implementation of maintenance works. Tools such as those presented here offer a great potentiality for incident management and the ability to improve the management and safety of the maintenance operation. When the analysis shows a potential event, based on specific features, the incident can be fed as a trigger to an automated system to assess its importance. The system could detect the presence of maintenance -in our case- personnel at a restricted area. In the case of ongoing corrective or preventive maintenance activities at hazardous areas, this information can be very important in order to remotely inspect and supervise such an endeavour. It also enables 288
In general, each incident detected has the potential of being assessed based on a set of criteria such as the following: the area where the incident is indicated, the status of the area (restricted to all, work ongoing, under maintenance, accessible to a specific group of people), the condition of the area (ongoing production process, leakage of material detected, there is knowledge of an accident that has occurred there, explosive environment etc.). Consequently, the information deducted form the described method can be proven extremely valuable for human resource management of the maintenance team, especially coupled with health and safety issues. 4. EXPERIMENTS 4.1 Dataset To evaluate the proposed fall detection method a dataset with 70 events performed by 7 different persons was acquired. These events include 40 falls and 30 events with features similar to falling, such as a person bending to tie his shoes or to pick up an item from the floor (Fig. 3). These events are examined in order to test the robustness of the algorithm in cases that affect the event features that determine the transition to the HMM states, i.e. vertical velocity and area variance. Furthermore, the participants were allowed to fall in ways that felt natural to them, leading to a variety of falls, e.g. fall forwards, backwards, sideways, fall while walking, like stumbling, or while standing, as fainting. Finally, since the incident detection scenario takes place in a shop-floor, there were people walking or standing in the monitoring area during the experiments.
2016 IFAC AMEST October 19-21, 2016. Biarritz, France
D. Triantafyllou et al. / IFAC-PapersOnLine 49-28 (2016) 286–290
289
Fig. 4. Filtering of the height. The first image shows the raw height signal while in the second the filtered output is depicted.
Fig. 5. Step signal of height.
Fig. 3. Events included in the dataset. The first two rows correspond to fall events, the third and forth row to bending and picking up items while the last to bending in order to tie shoes. The experiment was conducted in two different working environments: CPERI and ITI institutes of CERTH in Greece. In both cases two depth sensors (Kinect cameras for Xbox360) monitoring subsequent areas were used. 4.2 Event features processing As it is already mentioned in section 2.2 the noisy data acquired from depth sensors can lead to false alarms. Apart from FIR filtering that smooths the height (Fig. 4), the algorithm detects step signals occurring (Fig. 5) either from occlusions with real items or sometimes, with holes in the depth images that can be caused by shiny or black items. Such an example can be seen in Fig. 5. The abrupt height change from 1200mm to less than 600mm indicates a step signal caused by occlusion, hence it should not be mistaken as a fall. Thus, the two aforementioned measures prevent big deviations between the calculated velocities and their actual values. In Fig. 6 graphs of the event features during a fall are depicted. As it can be seen in the height graph (Fig. 6.a) the tracked person is initially in a standing position, then falls, remains lying on the ground for a while and finally stands up. Velocity and area variance values change drastically because of the fall (Fig. 6.b and Fig. 6.c) but 289
with a very small time delay (less than a second) due to averaging (section 2.2). Moreover, Fig. 6.d depicts the probability F that is crucial for the HMM’s transitions. Its value raises fast due to fall, then decreases and remains low for the time the person is lying on the floor and finally slightly augments, remaining below 0.5, while the person rises. At this point, it should be mentioned that sensors’ frames were acquired with 15 fps (this value depends on the utilized computer computational power). Furthermore, time window CT is about 0.65 seconds (so that about 10 different values are averaged for the velocity and area calculation) , threshold T is 10−6 (based on a training set of 10 falls) while HT is 1 meter. 4.3 Experimental results Table 1. Fall Detection Results True Positives 40
True Negatives 28
False Positives 2
Table 2. Fall Detection Performance Precision 95.24%
Recall 100%
F1-score 97.56%
The proposed algorithm was tested in more than 50000 frames including falls, events similar to falls but also people walking or standing in the monitored area. All the 40 falls were detected correctly, while there were 2 false positives in the whole dataset. These false alarms corresponded to 2 out of the 30 similar to fall events where a participant bended and picked up the mattress used for falls. It has to be mentioned that this task was
2016 IFAC AMEST 290 October 19-21, 2016. Biarritz, France
D. Triantafyllou et al. / IFAC-PapersOnLine 49-28 (2016) 286–290
Incorporation of more incidents to the presented system, like a falling item or collision events, are planned as future work. REFERENCES
Fig. 6. Event features and transition probability F graphs during a fall: (a) height, (b) velocity, (c) area variance, (d) transition probability. replicated 5 times but only in 2 of them produced false alarm. Fall detection results on the acquired dataset are summarized at tables 1 and 2. The recall rate is 100% while precision is 95.24% leading to a rather high F1-score that equals 97.56%. The results refer only to falls and similar to fall events. Nevertheless, people just walking or standing at the monitoring area during the experiments did not produce any false positives, even though they were not taken into account at the results. Furthermore, the algorithm was tested in 15 falling cases of an on-line dataset (http://fenix.univ.rzeszow.pl/∼mkepski/ds/uf.html) and detected them successfully without any false alarms. 5. CONCLUSION This paper presents a real-time, multi-camera incident detection system for shop-floor applications. The proposed system can be a great asset for maintenance activities especially if they take place in remote or dangerous areas since it can notify of accidents or hazardous events. Thus, the appropriate coping mechanism can be immediately triggered and deal with the incident. Furthermore, the proposed system provides a sense of safety to workers without entrenching their privacy rights since only depth sensors are utilized. The paper focuses on the detection of fall incidents. Fall process is modelled by HMM based on features that characterize a fall event such as vertical velocity and area variance from top view of the tracked person. Experimental results are very promising since only cases of bending to pick up huge items, such as the mattress, caused false alarms. Nevertheless, such cases are probably rare in a shop-floor. Moreover, all the real falls were detected without misses. 290
Bian, Z., Hou, J., Chau, L., and Magnenat-Thalmann, N. (2014). Fall detection based on body part tracking using a depth camera. IEEE journal of biomedical and health informatics. Diraco, G., Leone, A., and Siciliano, P. (2010). An active vision system for fall detection and posture recognition in elderly health care. Design, Automation and Test in Europe Conference and Exhibition, 1536–1541. Kepksi, M. and Kwolek, B. (2014). Fall detection using ceiling-mounted 3d depth camera. International Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, 640–647. Krinidis, S., Stavropoulos, G., Ioannidis, D., and Tzovaras, D. (2014). A robust and real-time multispace occupancy extraction system exploiting privacy-preserving sensors. ISCCSP. Mastorakis, G. and Makris, D. (2014). Fall detection system using kinect as infrared sensor. Real-Time Image Processing, 635–646. Shany, T., Redmond, S., Narayanan, M., and Lovell, N. (2012). Sensors-based wearable systems for monitoring of human fals. IEEE Sensors journal, 658–670. Stone, E. and Skubic, M. (2014). Fall detection in homes of older adults using the microsoft kinect. IEEE journal of biomedical and health informatics, 1590–1594. Wu, G. and Xue, S. (2008). Portable preimpact fal detector with wearable sensors. IEEE Transactions on Neural Systems and Rehabiliation Enginneering, 178–183. Zhang, C., Tian, Y., and Capezuti, E. (2012a). Privacy preserving automatic fall detection for eldery using rgbd cameras. International Conference on Computers Helping People with Special Needs, 625–633. Zhang, Z., Conly, C., and Anthitsos, V. (2015). A survey on vision-based fall detection. International Conference on pervasive technologies related to assistive environments. Zhang, Z., Liu, W., Metsis, V., and Anthitsos, V. (2012b). A viewpoint-independent statistical method for fall detection. IEEE Conference on Pattern Recognition, 3626– 3630.