- Email: [email protected]
Hybrid video surveillance systems using P300 based computational cognitive threat signature library
Available online at www.sciencedirect.com Available online at www.sciencedirect.com
ScienceDirect ScienceDirect
Procediaonline Computer 00 (2019) 000–000 Available at Science www.sciencedirect.com Procedia Computer Science 00 (2019) 000–000
ScienceDirect
www.elsevier.com/locate/procedia www.elsevier.com/locate/procedia
Procedia Computer Science 145 (2018) 512–519
Postproceedings of the 9th Annual International Conference on Biologically Inspired Cognitive Postproceedings of the 9thBICA Annual International Conference Inspired Cognitive Architectures, 2018 (Ninth Annual MeetingonofBiologically the BICA Society) Architectures, BICA 2018 (Ninth Annual Meeting of the BICA Society)
Hybrid video surveillance systems using P300 based computational Hybrid video surveillance using P300 based computational cognitivesystems threat signature library cognitive threat signature library Jeevanandam Jotheeswaran*, Anurag Singh, Sushama, Sanjeev Pippal Jeevanandam Jotheeswaran*, Anurag Singh, Sushama, Sanjeev Pippal School of Computing Science and Engineering, Galgotias School University, Greater Noida 201307, Uttar Pradesh, India of Computing Science and Engineering, Galgotias University, Greater Noida 201307, Uttar Pradesh, India a
a
Abstract Abstract Comprehensive Integrated Border Management System (CIBMS) is aimed to reduce the infiltration at borders of countries and to overcome risk. ItIntegrated is an integration manpower,System sensor, (CIBMS) networks,isintelligence, command and control research Comprehensive Border of Management aimed to reduce the infiltration at solutions. borders ofThis countries and to work is intended a reference library on threat using cognitivecommand technology which willsolutions. enhance the overcome risk. It to is create an integration of manpower, sensor,signature networks, intelligence, and control Thisintelligence research for CIBMS to reduce the human effortlibrary using cognitive science through Brain Computer Interface application. The system work is intended to create a reference on threat signature using cognitive technology which(BCI) will enhance the intelligence proposes to to usereduce an Electroencephalogram (EEG) cap toscience monitorthrough the operators brain signals when operator see any abnormal for CIBMS the human effort using cognitive Brain Computer Interface (BCI) application. The system activity area and then records the exact video framethe when the observer detectswhen a threat. The see combination of proposesacross to useborder an Electroencephalogram (EEG) cap to monitor operators brain signals operator any abnormal cognitive algorithm and EEG not only falseframe alarms, it also operators threats thatofwould activity across border area andfiltering then records the reduces exact video when the helps observer detectstoa detect threat.signs The of combination be overlooked, such and as flying birds, swaying branches nonalarms, threat object the action recognition. research aims cognitive algorithm EEG filtering not only reducesorfalse it alsoaccording helps operators to detect signs of This threats that would to a better optimization technique on branches P300 brain andobject build an operational libraryrecognition. of threat signature which can be be suggest overlooked, such as flying birds, swaying or signals non threat according the action This research aims used for future automated surveillance system with brain more signals accuracy. to suggest a better optimization technique on P300 and build an operational library of threat signature which can be used for future automated surveillance system with more accuracy. © 2019 The Authors. Published by Elsevier B.V. © 2018 The Authors. by Elsevier B.V. This is an open accessPublished article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) © 2019 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Peer-review under responsibility of the the CC scientific committee of the 9th Annual International Conference on Biologically Inspired This is an open access article under BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Peer-review under responsibility of the scientific committee of the 9th Annual International Conference on Biologically Inspired Cognitive Architectures. Peer-review under responsibility of the scientific committee of the 9th Annual International Conference on Biologically Inspired Cognitive Architectures. Cognitive Architectures. Keywords: Brain Computer Interface (BCI); Fast Independent Component Analysis (Fast ICA); Comprehensive Integrated Border Management System (CIBMS); Event Related Potentials (ERP); HumanAnalysis computer(Fast Interaction Keywords: Brain Computer Interface (BCI); Fast Independent Component ICA); (HCI). Comprehensive Integrated Border Management System (CIBMS); Event Related Potentials (ERP); Human computer Interaction (HCI).
* Corresponding author. E-mail address:author. [email protected] * Corresponding E-mail address: [email protected] 1877-0509© 2019 The Authors. Published by Elsevier B.V. This is an open access article under the CCby BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) 1877-0509© 2019 The Authors. Published Elsevier B.V. Peer-review under responsibility of the committee of the(https://creativecommons.org/licenses/by-nc-nd/4.0/) 9th Annual International Conference on Biologically Inspired Cognitive This is an open access article under thescientific CC BY-NC-ND license Architectures. Peer-review under responsibility of the scientific committee of the 9th Annual International Conference on Biologically Inspired Cognitive Architectures.
1877-0509 © 2018 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Peer-review under responsibility of the scientific committee of the 9th Annual International Conference on Biologically Inspired Cognitive Architectures. 10.1016/j.procs.2018.11.115
2
Jeevanandam Jotheeswaran et al. / Procedia Computer Science 145 (2018) 512–519 Author name / Procedia Computer Science 00 (2019) 000–000
513
1. Introduction Border patrolling is the high risk task among all cross border operations which leads to higher operational costs and loss of lives. Despite of using sensors and camera with GPS, GIS information with wireless access to the locations using Unmanned Aerial Vehicle (UAVs), but it is required security persons and soldiers for patrolling at the borders. Research organizations and security agencies started developing more advance security technologies which reduce the human lives and wastage of ammunitions by dropping the false alarm and increasing the accuracy in target detection. Many countries started working on a holistic solution at border patrolling operations called Comprehensive Integrated Border Management System (CIBMS). It works based on the principle of near line security and proposed to have a new radar system being installed in the border regions which will relay a 120 degree view of the location to the control room. Once the control room receives any information about an infiltration attempt, the specialized cameras at the border will automatically set itself according to available coordinates to capture images of militants trying to sneak in. The success of executing this system is purely lying on reduction of false alarms and subsequent miss hits by the automatic weapons. In surveillance environment the automatic detection of abnormal activities on cross border can be used to detect the potential threat such as a person with arms or throwing bomb or any dangerous behaviour. The human brain is one of the most complex systems in the universe and various technologies exist nowadays to record brain signals. Electroencephalogram (EEG) is one of the brain signals capturing and processing technique that allows gaining and understanding of the complex inner mechanisms of the brain. It records electrical signals generated by brain and any abnormal brain waves shown in EEG monitoring devices are to be associated with particular brain disorders. One of the brain signals P300 used for Event Related Potential (ERP) and processing it to decision making.ERP is a cognitive response generated by motor event, attention. In this work we are proposing a real time video frame capturing algorithm which captures the frame containing abnormal activity based on the highly peeked observers P300 brain signals when observer identifies the abnormal activity from the video frame and creating the library for future supervised automated systems. The system will not only reduce the false alarm for identification of threat but also analyze the accuracy between the observer and surveillance system, so that computer vision algorithm can be improved for more accuracy. 2. Related Work
The number of research works has been carried out independently in human detection and real time video surveillance. In [1] proposed a method to address human action recognition which includes internal technology which consists of human activity recognition and applications from low level to high level representation. In the review the author proposed many abnormal activity within crowd, multiple people interaction and single person abnormal activity can be recognized. Abnormal event detection can be captured under surveillance environment like crowded area, monitoring system and cross borders. 2.1. Abnormal Activity Detection in Video Abnormality detection in video is one of the eminent areas in the field of surveillance system and many researchers have worked on the area by using and optimizing different algorithms. Researchers in [2] have implemented a pedestrian detection system which combines both motion and appearance. Many methods implement a general pipeline based framework at first moving objects are detected, then they are classified and tracked over a certain number of frames. Finally, the resulting paths are used to distinguish ‘normal’ behaviour of objects from the ‘abnormal’. In the research [3] proposed an algorithm in which combination learning for detection with high structure redundancy in surveillance videos was used. Method obtained to carry out by them is sparse based combination to
514
Jeevanandam Jotheeswaran et al. / Procedia Computer Science 145 (2018) 512–519 Author name / Procedia Computer Science 00 (2019) 000–000
3
form spatio temporal cube. Researchers in [4] have implemented automatic motion tracking in sequence of video frames using static video capturing with twenty four hour surveillance video. It is almost impossible for a human observer to remain attentive round the clock and hence moving object detection is done by detecting the change in gray levels in consecutive frames. It is discussed under [5], a method to track and interpret the abnormal behaviour in a real time surveillance video. As the proposed method is based on the combination of visual feature extraction and image segmentation and it works without the need of a training phase. Feature Segmentation Crowd Behavior (FSCB) algorithm used for detection using instant entropy and temporal occupancy variations. In paper [6] explained the skelitonization method that can be computed based on distance transform and specified subsets of the transformed image are a distance skeleton by thinning approach using Matlab in static camera. Multi-view human action recognition is one of the trending research areas and [7] has proposed multi view human action recognition based on arbitrary number of cameras on publically available database. [8] experimented object detection and tracking using 7 synchronized cameras in which 2 are static which shows top down view and 5 were dynamic which always seen from static cameras. They have used temporal difference for moving camera between two consecutive frames and used static camera for background. Researchers in their research have proposed various techniques in identifying the object in real time video surveillance. They have tested the abnormal action detection in crowded area with morphological process using bounding box to find the suspect. Researchers used skelitonization on image rather than video frames. 2.2. Threat Detection using P300 EEG Signal Brain is the central and complex organ that controls human body which consist of various fluids and electric signals passes through neurons and this signal/waves are captured by EEG .It is used in medical area and researchers are introducing with technology along with Artificial Intelligence(AI) also known as cognitive technology. Many methods and proving mechanisms are available to capture, synthesize and analysis of EEG signals. In the research proposed in [9] about an algorithm for pattern matching using template based method to extract the morphological information from EEG Signals and applied linear discriminant function for pattern classification. Segmentation of video as discussed in the research is categorized as static camera segmentation and moving camera segmentation. Static camera performs on background subtraction, segmentation by tracking, Gaussian Mixture Model (GMM) whereas dynamic camera performs temporal difference and optical flow. Classification of videos discussed by following classification algorithm namely sliding Hierarchal Discriminant Classification Algorithm (sHDCA), Artificial Neural Network (ANN), Dynamic Time Wrapping (DTW). In the research proposed in[10] depicts target image detection on a Rapid Serial Visual presentation (RSVP) paradigm is a typical BCI application with the help of Stepwise Linear Discriminant Analysis (SWLDA) which detected P300 components. A tenfold cross-validation was conducted to determine the accuracy of all classification algorithms applied to the EEG data. The research proposed in [11] shows the performance of brain waves was evaluated based on the area under the Receiver Operating Characteristic (ROC) curve, Area under Curve (AUC) to capture the target location Minimum Distance to Mean (MDM).This dataset is used to evaluate offline performances of the methods in the canonical training test paradigm. Feature extraction can be done for segmented object using Space Time Volume (STV), Model based and local descriptors. The last stage is to classify the image or a video using classification algorithm such as Dynamic Time Wrapping (DTW) for measuring between two temporal sequences, Generative models for dynamic classification, Discriminative models for static classifier. Research for BCI is in evolving stage and researchers have only used it for analyzing brain disorder and medical area. Many researchers have proposed the EEG signals to identify the characters rather than analyzing the pattern of decisions. The research is suitable for incorporating brain signals to identify the potential threat from sequence of videos and creating a library for future use.
4
Author name / Procedia Computer Science 00 (2019) 000–000
Jeevanandam Jotheeswaran et al. / Procedia Computer Science 145 (2018) 512–519
.
515
3. Proposed System The Proposed method discussed in Fig.1. consists of live video processing and classification of P300 signals processed by EEG signal processing and recording it to data set. The system first takes input video frames border area camera which displays the real time video on the screen of surveillance centre for identification of threats.
Fig. 1. Block diagram for cognitive threat warning system
A parallel process to perform a frame difference between two consecutive frames using frame difference algorithm. W4 algorithm is used for sequencing of video frames extracted after the frame difference. The proposed system is using non invasive technique to capture brain waves because it cost low and no need to do any surgery that are done in invasive and partially invasive techniques. The observer wearing the EEG headset for capturing the Event Related Potential (ERP) such as abnormal actions, crawling under the fence etc. Many researchers used non invasive devices for recording of brain waves for various applications like capturing alphabet on computer screen, wheelchair for disabled people is a recent innovation in the field of neuroscience using brain waves capturing using non invasive EEG headset. 3.1. Frame Difference The formula discussed in (1)(2)(3) is about frame difference between two consecutive frames. The input frames f is the latest frame and fk-1 is the old frame.D is defined for the frame difference between two consecutive frames. The frames should be always greater then 0.The formula to calculate the frame difference between difference image D and Frame image F is as follow. k
k
k
k
Differential: D" =
f" − f"'( if f" − f"'( > 0 0 else
(1)
5
Author name / Procedia Computer Science 00 (2019) 000–000
Jeevanandam Jotheeswaran et al. / Procedia Computer Science 145 (2018) 512–519
516
Negative Differential:
D" =
f" − f"'( if f" − f"'( < 0 0 else
(2)
D" = |f" – f"'( |
(3)
Fully Differential:
D = Difference in image sequence F = Frame difference with some delay k
k
It is a parallel process for identifying the potential threat which was identified by the observer wearing EEG headset. The frame difference process gives the clear image of the abnormal activity generated and identified by the observer. 3.2. Object Tracking Tracking of object along with time storage is the process of detecting and following the moving object in sequence of video frames. Smart and digital cameras having the features of recording the video and as input sensors. Few algorithms are used for extracting the image quality under bad conditions such as (rain, storm, cyclone etc) to remove noise from the videos. In the research proposed [12] block matching algorithm for detecting moving object in video frames. Distance of the object from the camera and velocity of the object is calculated by using centroid and Euclidean distance formula for velocity of object between frames of object between frames discussed in (4)(5). Distance =
(x8 − x( )8 + (y8 − y( )8
Velocity = Distance Travelled/Frame Rate Table 1: Centroid formula x previous pixel position x present pixel position in width y previous pixel position y present pixel position in height. 1
2
1
2
(4) (5)
6
Author name / Procedia Computer Science 00 (2019) 000–000
Jeevanandam Jotheeswaran et al. / Procedia Computer Science 145 (2018) 512–519
517
Algorithm: Cognitive based Threat Detection Algorithm Input: Source real time video frames from cross border Output: EEG and video frame signal dataset after detection of threat Start { Read input video frames f; Loop for i= 1 to Ni do { Perform function frame difference (fk, fk-1) Perform function frame sequencing using W4 () Calculate velocity and distance of the object Print Time1; { Read signal by EEG Cap; Loop for i= 1 to Ni do Perform function feature extraction () Perform function Translation and Classification () Increase video frame by 1 iteration Print Time2; If(Time1= =Time2) { go to Cognitive library; save in Cognitive DB; return go to Device Command return to trigger Alarm; } Else {Read video frames f;} Loop end}} Loop End} End 4. EEG Analysis 4.1. EEG Processing A raw P300 signals which is generated from EEG cap are then processed under Matlab. Research proposed in [10] a Hierarchical Discriminant Principal Component Analysis (HDPCA) algorithm for all single-trial ERPs induced by dual-RSVP for a specific subject classification of component between target and non target classes. HDPCA is more improved then Hierarchical Discriminant Principal Component Analysis (HDCA) in terms of accuracy. The P300 signals are generated while the abnormal event detection in video frame will trigger the alarm for threat and then stored in the dataset to create the library for the future threat detection and trigger the threat alarm and proposed Template Matching (TM) in conjunction with LDA (TMLDA) only needed 15 trials to achieve a 100% recognition rate. The video frame containing abnormal activity is identified by an operator wearing EEG headset and the captured Video frame undergoes storage in library of data. The processed EEG signal undergoes for signal processing using classification and feature extraction algorithm and generates an alarm after detecting abnormal activity from the sets of video frames and stored in library for future detection. As discussed in Fig.2. the graph shows the P300 EEG data compared to normal EEG data.
Author name / Procedia Computer Science 00 (2019) 000–000
7
Jeevanandam Jotheeswaran et al. / Procedia Computer Science 145 (2018) 512–519
518
Fig. 2.P300 signal and abnormal activity matching
5. Conclusion According to the proposed methodology in processing the surveillance video and abnormal action detection is possible by using segmentation, filtering algorithms and BCI using EEG signal. P300 allows user to communicate with system via sequence of video frames and enhancing the quality of video in every bad conditions such as rain, storm etc. The system uses an EEG cap to monitor the operators brain signals and then records when the observer detects a threat. The combination of computational algorithm and EEG P300 signals not only reduces false alarms, it also helps observers to detect signs of threats that would be overlooked. Our system stores the video frame that contains potential threat captured by video camera when observers P300 brain waves were highly peeked after training for the specific area for a period of time into the dataset for using it further without human interaction and making the system man portable. The sHDCA algorithm automatically triggers the alarm using dataset P300 signal when the dynamic camera captures any abnormal activity under that area. Under supervised learning, the basic threat signature library is made available and in future it can be enabled with unsupervised mode. References [1] [2] [3] [4]
Shian-Ru ke1, Hoang Le Uyen Thuc2, Yong-Jin Lee 1, Jenq-Neng Hwang 1, Jang-Hee Yoo 3, & Kyoung-Ho Choi4. (2013). “A Review on Video-Based Human Activity Recognition”. EISSN 2073-431X.
P. Viola, M. J. Jones, & D. Snow. (2003) “Detecting pedestrians using patterns of motion and appearance”. Pro-ceedings Ninth IEEE International Conference on Computer Vision, (pp. pp. 734-741 vol. 2). Nice, France.
C. Lu, J. Shi, & J. Jia. (2013). “Abnormal Event Detection at 150 FPS in MATLAB”. IEEE International Conference on Computer Vision, (pp. pp. 2720-2727). Sydney, VIC.
M. A. AlGhamdi, M. A. Khan, & S. H. AlMotiri. (2015). “Automatic motion tracking of a human in a surveillance video”. IEEE First International Smart Cities Conference (ISC2), (pp. pp. 1-4). Guadalajara.
8
Jeevanandam Jotheeswaran et al. / Procedia Computer Science 145 (2018) 512–519 Author name / Procedia Computer Science 00 (2019) 000–000 [5] [6] [7] [8] [9]
519
Andrea Pennisia, & Domenico D. (2016). “On-line Real-time Crowd Behavior Detection in Video Sequences”. Computer Vision and Image Understanding, Elsevier Science Inc. , New York,NY,USA.
Ahmed Taha, & Hala H. Zayed. (2015). “Skeleton-Based Human Activity Recognition for Video Surveillance”. International Journal of Scientific & Engineering Research, vol. 6, issue 1, ISSN 2229-5518.
Iosifidis, A. Tefas, & I. Pitas. (2013). “Multi-view Human Action Recognition: A Survey”. 2013 Ninth International Conference on Intelligent Information Hiding and Multimedia Signal Processing, (pp. pp. 522-525). Beijing.
Kroeger T., Dragon R., & Van Gool L. (2014). “Multi-view Tracking of Multiple Targets with Dynamic Cameras”. In Jiang X., Hornegger J., & Koch R., Pattern Recognition. Springer. Chen, SW. & Lai, & YC. EURASIP J. (2014). “A signal-processing-based technique for P300 evoked potential detection with the applications into automated character recognition”. Adv. Signal Process.
[10] Zhimin Lin, Ying Zeng, & Hui Gao. (2017). “Multirapid Serial Visual Presentation Framework for EEG-Based Target Detection”. BioMed Research International, 12 pages.
[11] Congedo M., & Barachant A. (2014). “A Plug&Play P300 BCI Using Information Geometry” White paper - arXiv. - arXiv: 1409. 0107.
[12] B. Tharanidevi1, R. Vadivu, & . (2013). “Moving Object Tracking Distance and Velocity Determination based on Back-ground Subtraction Algorithm”. IOSR Journal of Electronics and Communication Engineering (IOSR-JECE) e-ISSN: 2278-2834.Volume 8, Issue 1, Volume 8, Issue 1.
ScienceDirect ScienceDirect
Procediaonline Computer 00 (2019) 000–000 Available at Science www.sciencedirect.com Procedia Computer Science 00 (2019) 000–000
ScienceDirect
www.elsevier.com/locate/procedia www.elsevier.com/locate/procedia
Procedia Computer Science 145 (2018) 512–519
Postproceedings of the 9th Annual International Conference on Biologically Inspired Cognitive Postproceedings of the 9thBICA Annual International Conference Inspired Cognitive Architectures, 2018 (Ninth Annual MeetingonofBiologically the BICA Society) Architectures, BICA 2018 (Ninth Annual Meeting of the BICA Society)
Hybrid video surveillance systems using P300 based computational Hybrid video surveillance using P300 based computational cognitivesystems threat signature library cognitive threat signature library Jeevanandam Jotheeswaran*, Anurag Singh, Sushama, Sanjeev Pippal Jeevanandam Jotheeswaran*, Anurag Singh, Sushama, Sanjeev Pippal School of Computing Science and Engineering, Galgotias School University, Greater Noida 201307, Uttar Pradesh, India of Computing Science and Engineering, Galgotias University, Greater Noida 201307, Uttar Pradesh, India a
a
Abstract Abstract Comprehensive Integrated Border Management System (CIBMS) is aimed to reduce the infiltration at borders of countries and to overcome risk. ItIntegrated is an integration manpower,System sensor, (CIBMS) networks,isintelligence, command and control research Comprehensive Border of Management aimed to reduce the infiltration at solutions. borders ofThis countries and to work is intended a reference library on threat using cognitivecommand technology which willsolutions. enhance the overcome risk. It to is create an integration of manpower, sensor,signature networks, intelligence, and control Thisintelligence research for CIBMS to reduce the human effortlibrary using cognitive science through Brain Computer Interface application. The system work is intended to create a reference on threat signature using cognitive technology which(BCI) will enhance the intelligence proposes to to usereduce an Electroencephalogram (EEG) cap toscience monitorthrough the operators brain signals when operator see any abnormal for CIBMS the human effort using cognitive Brain Computer Interface (BCI) application. The system activity area and then records the exact video framethe when the observer detectswhen a threat. The see combination of proposesacross to useborder an Electroencephalogram (EEG) cap to monitor operators brain signals operator any abnormal cognitive algorithm and EEG not only falseframe alarms, it also operators threats thatofwould activity across border area andfiltering then records the reduces exact video when the helps observer detectstoa detect threat.signs The of combination be overlooked, such and as flying birds, swaying branches nonalarms, threat object the action recognition. research aims cognitive algorithm EEG filtering not only reducesorfalse it alsoaccording helps operators to detect signs of This threats that would to a better optimization technique on branches P300 brain andobject build an operational libraryrecognition. of threat signature which can be be suggest overlooked, such as flying birds, swaying or signals non threat according the action This research aims used for future automated surveillance system with brain more signals accuracy. to suggest a better optimization technique on P300 and build an operational library of threat signature which can be used for future automated surveillance system with more accuracy. © 2019 The Authors. Published by Elsevier B.V. © 2018 The Authors. by Elsevier B.V. This is an open accessPublished article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) © 2019 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Peer-review under responsibility of the the CC scientific committee of the 9th Annual International Conference on Biologically Inspired This is an open access article under BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Peer-review under responsibility of the scientific committee of the 9th Annual International Conference on Biologically Inspired Cognitive Architectures. Peer-review under responsibility of the scientific committee of the 9th Annual International Conference on Biologically Inspired Cognitive Architectures. Cognitive Architectures. Keywords: Brain Computer Interface (BCI); Fast Independent Component Analysis (Fast ICA); Comprehensive Integrated Border Management System (CIBMS); Event Related Potentials (ERP); HumanAnalysis computer(Fast Interaction Keywords: Brain Computer Interface (BCI); Fast Independent Component ICA); (HCI). Comprehensive Integrated Border Management System (CIBMS); Event Related Potentials (ERP); Human computer Interaction (HCI).
* Corresponding author. E-mail address:author. [email protected] * Corresponding E-mail address: [email protected] 1877-0509© 2019 The Authors. Published by Elsevier B.V. This is an open access article under the CCby BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) 1877-0509© 2019 The Authors. Published Elsevier B.V. Peer-review under responsibility of the committee of the(https://creativecommons.org/licenses/by-nc-nd/4.0/) 9th Annual International Conference on Biologically Inspired Cognitive This is an open access article under thescientific CC BY-NC-ND license Architectures. Peer-review under responsibility of the scientific committee of the 9th Annual International Conference on Biologically Inspired Cognitive Architectures.
1877-0509 © 2018 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Peer-review under responsibility of the scientific committee of the 9th Annual International Conference on Biologically Inspired Cognitive Architectures. 10.1016/j.procs.2018.11.115
2
Jeevanandam Jotheeswaran et al. / Procedia Computer Science 145 (2018) 512–519 Author name / Procedia Computer Science 00 (2019) 000–000
513
1. Introduction Border patrolling is the high risk task among all cross border operations which leads to higher operational costs and loss of lives. Despite of using sensors and camera with GPS, GIS information with wireless access to the locations using Unmanned Aerial Vehicle (UAVs), but it is required security persons and soldiers for patrolling at the borders. Research organizations and security agencies started developing more advance security technologies which reduce the human lives and wastage of ammunitions by dropping the false alarm and increasing the accuracy in target detection. Many countries started working on a holistic solution at border patrolling operations called Comprehensive Integrated Border Management System (CIBMS). It works based on the principle of near line security and proposed to have a new radar system being installed in the border regions which will relay a 120 degree view of the location to the control room. Once the control room receives any information about an infiltration attempt, the specialized cameras at the border will automatically set itself according to available coordinates to capture images of militants trying to sneak in. The success of executing this system is purely lying on reduction of false alarms and subsequent miss hits by the automatic weapons. In surveillance environment the automatic detection of abnormal activities on cross border can be used to detect the potential threat such as a person with arms or throwing bomb or any dangerous behaviour. The human brain is one of the most complex systems in the universe and various technologies exist nowadays to record brain signals. Electroencephalogram (EEG) is one of the brain signals capturing and processing technique that allows gaining and understanding of the complex inner mechanisms of the brain. It records electrical signals generated by brain and any abnormal brain waves shown in EEG monitoring devices are to be associated with particular brain disorders. One of the brain signals P300 used for Event Related Potential (ERP) and processing it to decision making.ERP is a cognitive response generated by motor event, attention. In this work we are proposing a real time video frame capturing algorithm which captures the frame containing abnormal activity based on the highly peeked observers P300 brain signals when observer identifies the abnormal activity from the video frame and creating the library for future supervised automated systems. The system will not only reduce the false alarm for identification of threat but also analyze the accuracy between the observer and surveillance system, so that computer vision algorithm can be improved for more accuracy. 2. Related Work
The number of research works has been carried out independently in human detection and real time video surveillance. In [1] proposed a method to address human action recognition which includes internal technology which consists of human activity recognition and applications from low level to high level representation. In the review the author proposed many abnormal activity within crowd, multiple people interaction and single person abnormal activity can be recognized. Abnormal event detection can be captured under surveillance environment like crowded area, monitoring system and cross borders. 2.1. Abnormal Activity Detection in Video Abnormality detection in video is one of the eminent areas in the field of surveillance system and many researchers have worked on the area by using and optimizing different algorithms. Researchers in [2] have implemented a pedestrian detection system which combines both motion and appearance. Many methods implement a general pipeline based framework at first moving objects are detected, then they are classified and tracked over a certain number of frames. Finally, the resulting paths are used to distinguish ‘normal’ behaviour of objects from the ‘abnormal’. In the research [3] proposed an algorithm in which combination learning for detection with high structure redundancy in surveillance videos was used. Method obtained to carry out by them is sparse based combination to
514
Jeevanandam Jotheeswaran et al. / Procedia Computer Science 145 (2018) 512–519 Author name / Procedia Computer Science 00 (2019) 000–000
3
form spatio temporal cube. Researchers in [4] have implemented automatic motion tracking in sequence of video frames using static video capturing with twenty four hour surveillance video. It is almost impossible for a human observer to remain attentive round the clock and hence moving object detection is done by detecting the change in gray levels in consecutive frames. It is discussed under [5], a method to track and interpret the abnormal behaviour in a real time surveillance video. As the proposed method is based on the combination of visual feature extraction and image segmentation and it works without the need of a training phase. Feature Segmentation Crowd Behavior (FSCB) algorithm used for detection using instant entropy and temporal occupancy variations. In paper [6] explained the skelitonization method that can be computed based on distance transform and specified subsets of the transformed image are a distance skeleton by thinning approach using Matlab in static camera. Multi-view human action recognition is one of the trending research areas and [7] has proposed multi view human action recognition based on arbitrary number of cameras on publically available database. [8] experimented object detection and tracking using 7 synchronized cameras in which 2 are static which shows top down view and 5 were dynamic which always seen from static cameras. They have used temporal difference for moving camera between two consecutive frames and used static camera for background. Researchers in their research have proposed various techniques in identifying the object in real time video surveillance. They have tested the abnormal action detection in crowded area with morphological process using bounding box to find the suspect. Researchers used skelitonization on image rather than video frames. 2.2. Threat Detection using P300 EEG Signal Brain is the central and complex organ that controls human body which consist of various fluids and electric signals passes through neurons and this signal/waves are captured by EEG .It is used in medical area and researchers are introducing with technology along with Artificial Intelligence(AI) also known as cognitive technology. Many methods and proving mechanisms are available to capture, synthesize and analysis of EEG signals. In the research proposed in [9] about an algorithm for pattern matching using template based method to extract the morphological information from EEG Signals and applied linear discriminant function for pattern classification. Segmentation of video as discussed in the research is categorized as static camera segmentation and moving camera segmentation. Static camera performs on background subtraction, segmentation by tracking, Gaussian Mixture Model (GMM) whereas dynamic camera performs temporal difference and optical flow. Classification of videos discussed by following classification algorithm namely sliding Hierarchal Discriminant Classification Algorithm (sHDCA), Artificial Neural Network (ANN), Dynamic Time Wrapping (DTW). In the research proposed in[10] depicts target image detection on a Rapid Serial Visual presentation (RSVP) paradigm is a typical BCI application with the help of Stepwise Linear Discriminant Analysis (SWLDA) which detected P300 components. A tenfold cross-validation was conducted to determine the accuracy of all classification algorithms applied to the EEG data. The research proposed in [11] shows the performance of brain waves was evaluated based on the area under the Receiver Operating Characteristic (ROC) curve, Area under Curve (AUC) to capture the target location Minimum Distance to Mean (MDM).This dataset is used to evaluate offline performances of the methods in the canonical training test paradigm. Feature extraction can be done for segmented object using Space Time Volume (STV), Model based and local descriptors. The last stage is to classify the image or a video using classification algorithm such as Dynamic Time Wrapping (DTW) for measuring between two temporal sequences, Generative models for dynamic classification, Discriminative models for static classifier. Research for BCI is in evolving stage and researchers have only used it for analyzing brain disorder and medical area. Many researchers have proposed the EEG signals to identify the characters rather than analyzing the pattern of decisions. The research is suitable for incorporating brain signals to identify the potential threat from sequence of videos and creating a library for future use.
4
Author name / Procedia Computer Science 00 (2019) 000–000
Jeevanandam Jotheeswaran et al. / Procedia Computer Science 145 (2018) 512–519
.
515
3. Proposed System The Proposed method discussed in Fig.1. consists of live video processing and classification of P300 signals processed by EEG signal processing and recording it to data set. The system first takes input video frames border area camera which displays the real time video on the screen of surveillance centre for identification of threats.
Fig. 1. Block diagram for cognitive threat warning system
A parallel process to perform a frame difference between two consecutive frames using frame difference algorithm. W4 algorithm is used for sequencing of video frames extracted after the frame difference. The proposed system is using non invasive technique to capture brain waves because it cost low and no need to do any surgery that are done in invasive and partially invasive techniques. The observer wearing the EEG headset for capturing the Event Related Potential (ERP) such as abnormal actions, crawling under the fence etc. Many researchers used non invasive devices for recording of brain waves for various applications like capturing alphabet on computer screen, wheelchair for disabled people is a recent innovation in the field of neuroscience using brain waves capturing using non invasive EEG headset. 3.1. Frame Difference The formula discussed in (1)(2)(3) is about frame difference between two consecutive frames. The input frames f is the latest frame and fk-1 is the old frame.D is defined for the frame difference between two consecutive frames. The frames should be always greater then 0.The formula to calculate the frame difference between difference image D and Frame image F is as follow. k
k
k
k
Differential: D" =
f" − f"'( if f" − f"'( > 0 0 else
(1)
5
Author name / Procedia Computer Science 00 (2019) 000–000
Jeevanandam Jotheeswaran et al. / Procedia Computer Science 145 (2018) 512–519
516
Negative Differential:
D" =
f" − f"'( if f" − f"'( < 0 0 else
(2)
D" = |f" – f"'( |
(3)
Fully Differential:
D = Difference in image sequence F = Frame difference with some delay k
k
It is a parallel process for identifying the potential threat which was identified by the observer wearing EEG headset. The frame difference process gives the clear image of the abnormal activity generated and identified by the observer. 3.2. Object Tracking Tracking of object along with time storage is the process of detecting and following the moving object in sequence of video frames. Smart and digital cameras having the features of recording the video and as input sensors. Few algorithms are used for extracting the image quality under bad conditions such as (rain, storm, cyclone etc) to remove noise from the videos. In the research proposed [12] block matching algorithm for detecting moving object in video frames. Distance of the object from the camera and velocity of the object is calculated by using centroid and Euclidean distance formula for velocity of object between frames of object between frames discussed in (4)(5). Distance =
(x8 − x( )8 + (y8 − y( )8
Velocity = Distance Travelled/Frame Rate Table 1: Centroid formula x previous pixel position x present pixel position in width y previous pixel position y present pixel position in height. 1
2
1
2
(4) (5)
6
Author name / Procedia Computer Science 00 (2019) 000–000
Jeevanandam Jotheeswaran et al. / Procedia Computer Science 145 (2018) 512–519
517
Algorithm: Cognitive based Threat Detection Algorithm Input: Source real time video frames from cross border Output: EEG and video frame signal dataset after detection of threat Start { Read input video frames f; Loop for i= 1 to Ni do { Perform function frame difference (fk, fk-1) Perform function frame sequencing using W4 () Calculate velocity and distance of the object Print Time1; { Read signal by EEG Cap; Loop for i= 1 to Ni do Perform function feature extraction () Perform function Translation and Classification () Increase video frame by 1 iteration Print Time2; If(Time1= =Time2) { go to Cognitive library; save in Cognitive DB; return go to Device Command return to trigger Alarm; } Else {Read video frames f;} Loop end}} Loop End} End 4. EEG Analysis 4.1. EEG Processing A raw P300 signals which is generated from EEG cap are then processed under Matlab. Research proposed in [10] a Hierarchical Discriminant Principal Component Analysis (HDPCA) algorithm for all single-trial ERPs induced by dual-RSVP for a specific subject classification of component between target and non target classes. HDPCA is more improved then Hierarchical Discriminant Principal Component Analysis (HDCA) in terms of accuracy. The P300 signals are generated while the abnormal event detection in video frame will trigger the alarm for threat and then stored in the dataset to create the library for the future threat detection and trigger the threat alarm and proposed Template Matching (TM) in conjunction with LDA (TMLDA) only needed 15 trials to achieve a 100% recognition rate. The video frame containing abnormal activity is identified by an operator wearing EEG headset and the captured Video frame undergoes storage in library of data. The processed EEG signal undergoes for signal processing using classification and feature extraction algorithm and generates an alarm after detecting abnormal activity from the sets of video frames and stored in library for future detection. As discussed in Fig.2. the graph shows the P300 EEG data compared to normal EEG data.
Author name / Procedia Computer Science 00 (2019) 000–000
7
Jeevanandam Jotheeswaran et al. / Procedia Computer Science 145 (2018) 512–519
518
Fig. 2.P300 signal and abnormal activity matching
5. Conclusion According to the proposed methodology in processing the surveillance video and abnormal action detection is possible by using segmentation, filtering algorithms and BCI using EEG signal. P300 allows user to communicate with system via sequence of video frames and enhancing the quality of video in every bad conditions such as rain, storm etc. The system uses an EEG cap to monitor the operators brain signals and then records when the observer detects a threat. The combination of computational algorithm and EEG P300 signals not only reduces false alarms, it also helps observers to detect signs of threats that would be overlooked. Our system stores the video frame that contains potential threat captured by video camera when observers P300 brain waves were highly peeked after training for the specific area for a period of time into the dataset for using it further without human interaction and making the system man portable. The sHDCA algorithm automatically triggers the alarm using dataset P300 signal when the dynamic camera captures any abnormal activity under that area. Under supervised learning, the basic threat signature library is made available and in future it can be enabled with unsupervised mode. References [1] [2] [3] [4]
Shian-Ru ke1, Hoang Le Uyen Thuc2, Yong-Jin Lee 1, Jenq-Neng Hwang 1, Jang-Hee Yoo 3, & Kyoung-Ho Choi4. (2013). “A Review on Video-Based Human Activity Recognition”. EISSN 2073-431X.
P. Viola, M. J. Jones, & D. Snow. (2003) “Detecting pedestrians using patterns of motion and appearance”. Pro-ceedings Ninth IEEE International Conference on Computer Vision, (pp. pp. 734-741 vol. 2). Nice, France.
C. Lu, J. Shi, & J. Jia. (2013). “Abnormal Event Detection at 150 FPS in MATLAB”. IEEE International Conference on Computer Vision, (pp. pp. 2720-2727). Sydney, VIC.
M. A. AlGhamdi, M. A. Khan, & S. H. AlMotiri. (2015). “Automatic motion tracking of a human in a surveillance video”. IEEE First International Smart Cities Conference (ISC2), (pp. pp. 1-4). Guadalajara.
8
Jeevanandam Jotheeswaran et al. / Procedia Computer Science 145 (2018) 512–519 Author name / Procedia Computer Science 00 (2019) 000–000 [5] [6] [7] [8] [9]
519
Andrea Pennisia, & Domenico D. (2016). “On-line Real-time Crowd Behavior Detection in Video Sequences”. Computer Vision and Image Understanding, Elsevier Science Inc. , New York,NY,USA.
Ahmed Taha, & Hala H. Zayed. (2015). “Skeleton-Based Human Activity Recognition for Video Surveillance”. International Journal of Scientific & Engineering Research, vol. 6, issue 1, ISSN 2229-5518.
Iosifidis, A. Tefas, & I. Pitas. (2013). “Multi-view Human Action Recognition: A Survey”. 2013 Ninth International Conference on Intelligent Information Hiding and Multimedia Signal Processing, (pp. pp. 522-525). Beijing.
Kroeger T., Dragon R., & Van Gool L. (2014). “Multi-view Tracking of Multiple Targets with Dynamic Cameras”. In Jiang X., Hornegger J., & Koch R., Pattern Recognition. Springer. Chen, SW. & Lai, & YC. EURASIP J. (2014). “A signal-processing-based technique for P300 evoked potential detection with the applications into automated character recognition”. Adv. Signal Process.
[10] Zhimin Lin, Ying Zeng, & Hui Gao. (2017). “Multirapid Serial Visual Presentation Framework for EEG-Based Target Detection”. BioMed Research International, 12 pages.
[11] Congedo M., & Barachant A. (2014). “A Plug&Play P300 BCI Using Information Geometry” White paper - arXiv. - arXiv: 1409. 0107.
[12] B. Tharanidevi1, R. Vadivu, & . (2013). “Moving Object Tracking Distance and Velocity Determination based on Back-ground Subtraction Algorithm”. IOSR Journal of Electronics and Communication Engineering (IOSR-JECE) e-ISSN: 2278-2834.Volume 8, Issue 1, Volume 8, Issue 1.