- Email: [email protected]
Heterogeneous sensor arrays: Merging cameras and gas sensors into innovative fire detection systems
Sensors and Actuators B 231 (2016) 497–505
Contents lists available at ScienceDirect
Sensors and Actuators B: Chemical journal homepage: www.elsevier.com/locate/snb
Heterogeneous sensor arrays: Merging cameras and gas sensors into innovative fire detection systems Angelika Hackner a,∗ , Helmut Oberpriller a , Alexander Ohnesorge a , Volker Hechtenberg b , Gerhard Müller a,c a
Airbus Group Innovations, D-81663 Munich, Germany Innovation Team Bremen, D-17111 Meesiger, Germany c Munich University of Applied Sciences, Fachbereich 06, Lothstraße 34, D-80335 Munich, Germany b
a r t i c l e
i n f o
Article history: Received 22 October 2015 Received in revised form 9 February 2016 Accepted 18 February 2016 Available online 19 March 2016 Keywords: Multi-criteria fire detection Image processing Gas detection False alarm rate Alarm verification
a b s t r a c t An innovative way of multi-criteria fire detection is introduced. In this concept image information about the monitored area is combined with knowledge about the reactive gas concentrations inside this same area to produce a fire alarm and to enable alarm verification. Key component in this new mode of fire detection is a low-cost CCD camera which is operated in a difference image (DI) mode. In the unsupervised DI mode, the average grey scale values in consecutive difference images are evaluated and compared with the outputs of a metal oxide gas sensor array which measures the reactive gas concentrations in this same area. Fire alarms are generated upon observing concurrent changes in all sensor outputs. Once obtained, the camera can be switched to the supervised imaging mode and the validity of an alarm can be assessed by normal and/or difference imaging of the monitored area. In this paper the concept of DI-based fire detection is explained, demonstrated and possible enhancements and ramifications of this new technique are discussed. © 2016 Elsevier B.V. All rights reserved.
1. Introduction Fires represent severe danger to human lives, property and environment. In order to cope with such dangers, efficient methods of fire detection and firefighting are required. As far as fire detection is concerned, this field can be broadly subdivided into two sub-fields: (i) wide-area surveillance for the detection of fires in unconfined spaces, such as in forests, road tunnels or underground stations, and (ii) point-detection where fire hazards occur in closely confined spaces such as in office and in residential buildings or in factory plants. Whereas camera surveillance is the method of choice in the first class of environments, point detectors such as smoke, ionization, heat and gas detectors are being used in confined spaces. State of the art reviews of these existing fire detection technologies can be found in references [1–8]. As revealed there, video image processing focuses on the detection of spatial-temporal features such as color probability, contour irregularity in individual frames, and on temporal changes in between consecutive frames, i.e., on all pieces of information that can be conveyed over large distances via electromagnetic radiation. Point detectors, in contrast, concen-
∗ Corresponding author. Fax: +49 89 60724001. E-mail address: [email protected] (A. Hackner). http://dx.doi.org/10.1016/j.snb.2016.02.081 0925-4005/© 2016 Elsevier B.V. All rights reserved.
trate on those fire indicators which derive from localized chemical reactions that occur in open and smouldering fires and that are transported much more slowly over smaller distances via convection and diffusion processes. As both kinds of detectors are only sensitive to a limited sub-set of all possible fire features, sensorand application-specific false-alarm issues arise. A fire detection application with a very high damage potential, which is plagued by false alarm issues, is aeronautic fire detection [9–11]. Active fire protection inside commercial airplanes comprises fire detection by photo-electric automatic fire detectors as well as firefighting with extinguishing equipment [12,13]. Such equipment is installed in all those areas that are inaccessible during flight. These include electronic compartments such as avionics and in-flight entertainment equipment and, last not least cargo compartments and cargo containers. Among those the cargo compartments bear a particularly high danger potential because very few requirements actually restrict the flammability of any potential cargo. In the current state of the art, fire detection in such critical areas is performed with the help of smoke detectors [5–8], similar to those that are routinely used in office and in residential buildings. Such sensors detect open fires by scattering light on airborne particles. As smoke particles are only one sort of airborne particulate matter that might turn up, false alarms will almost certainly occur [9–11]. Well known causes of false alarms are dust particles and/or
498
A. Hackner et al. / Sensors and Actuators B 231 (2016) 497–505
condensing humidity. These latter forms of false alarms can easily occur in aircraft operation when an airplane is loaded on ground in a hot and humid environment and as the air inside the cargo compartment cools down as the airplane climbs to its cruising altitude. Statistics issued by the Federal Aviation Agency (FAA) [9], show that the probability of such false alarms is indeed very high. Investigations over a five year period showed that in hindsight only 1 out of 200 reported alarms could be unambiguously traced to a true smoke event. While false alarms of this type may be acceptable on ground, these cause severe problems when they occur in flight. As a fire, that may be harmless on ground, might become catastrophic in flight and as false alarms cannot be verified as such during flight, tight authority requirements exist on how to deal with such alarms [14]. These require that any fire alarm—whether true or not—needs to be answered by fire-fighting measures and by emergency landings. Statistics show that such events occur with a rate of about 200–300 per year with most of them ending in unscheduled flight interruptions [9–11]. The associated cost of such flight interruptions is estimated to range in between 30–50 thousand D /per event, depending on the size and occupancy of the airplanes [11]. Trying to decrease false alarm rates from the current rate of about 1 in 105 flight hours to the newly requested standard of 1 in 107 flight hours [12], several attempts have been made to develop multi-criteria fire detectors which measure several fire criteria at the same time, thereby enabling lower false alarm rates [15–17]. All these existing multi-sensor approaches employ heterogeneous arrays of sensors with scalar output signals, chiefly smoke, heat, and CO. Our current work contributes to this effort of attaining lower false-alarm rates by employing heterogeneous sensor arrays. In our approach conventional gas sensor arrays are combined with low-cost CCD cameras to detect fire gases and concurrent optical changes in the confined spaces of aircraft compartments. To the best of our knowledge, this is the first attempt to include cameras into heterogeneous sensor arrays to attain higher levels of false alarm rejection and unpresented possibilities of alarm verification. The fire detection system discussed below can be operated in an unsupervised monitoring mode in which the CCD camera is operated in a difference image (DI) mode in which grey scale changes in between consecutive images are monitored alongside with the potential appearance of fire gases which might correlate with the observed visual changes. In case concurrent changes in the gas evolution and consecutive image frames are detected, an alarm is issued and the system is switched into its verification mode. In this latter mode the CCD camera provides a straight-forward picture of the monitored scene which allows the pilot to take an informed decision as to the actual necessity of emergency measures. In the following, this innovative sensor concept is introduced (§2) and the processes underlying DI signal generation and DI monitoring (DIM) are explained (§3). §4 presents the results of feasibility tests in which two realistic fire detection scenarios are monitored using a heterogeneous array of sensors consisting of a DI camera and a metal oxide gas sensor array. In the concluding §5 possible enhancements and ramifications of the DIM approach are discussed.
smoke detectors [5,6,8]. As illustrated in the left-hand part of Fig. 1, reactive gas monitoring is performed with the help of a gas sensor array consisting of three MOX gas sensors [18]. These sensors contain different sensitive materials and/or are operated at different temperatures to exhibit cross sensitivity profiles oriented toward different components in the fire gas mix. Such an array produces six analog output signals, which form a signal vector. These signals consist of the sensitive layer resistances of the three MOX gas sensors (resistive/RES response) and of the electrical heater powers that need to be supplied to the substrate heater meanders to maintain constant sensor operation temperatures (catalytic/CAT response). Whereas the first three signals contain information concerning the oxidizing or reducing properties of the adsorbed fire gases, the latter three carry information with regard to the catalytic combustion powers generated by the adsorbed gases [18]. Low false alarm rate requires additional and independent fire indicators to be assessed inside the controlled area. In our approach we chose to monitor optical changes which coincide with open and smouldering fires. These are chiefly flames, smoke or aerosols of combustible liquids. As indicated on the right-hand side of Fig. 1, such visual changes are observed with the help of a low-cost CCD camera [19]. For our purpose a commercial CCD camera, which simply produces straight-forward images of the monitored scene, was modified to produce difference images (DI), which highlight visible changes in between two successive normal images (NI). As a second feature our camera allows to be operated in an unsupervised monitoring and in a supervised verification mode. In the unsupervised mode the DI data are compressed into a sequence of average grey scale values in each difference image. In this way, an additional analogue output signal is generated which can be combined with the six output signals of the gas sensor array to form an output signal vector. This vectorial output can be processed using standard pattern recognition techniques to form a multi-criteria fire detector with a sharply decreased false alarm probability. In the supervised verification mode, the full two-dimensional (2d) image information contained in the NI and/or DI images is used to enable alarm verification by a human observer. This observer (pilot) can then take informed decisions as to the necessity and extent of possible emergency measures. 3. Difference-image monitoring of controlled areas In order to introduce the concept of difference image monitoring (DIM), consider Fig. 2. For the sake of clarity Fig. 2a shows the DI camera module itself with its lens facing a candle in front of a uniform piece of cardboard. Fig. 2b and c, on the other hand, com-
2. Sensor concept The sensor concept evaluated in this paper is presented in Fig. 1. The monitored area in the center is assumed to be one of the aircraft’s critical compartments (avionic, in-flight entertainment, cargo containers and cargo compartments). As explained above, these areas need to be monitored in an unsupervised manner for the presence of certain fire indicators. While in the current state of the art this task is performed by smoke detectors, we chose in our case a metal oxide (MOX) gas sensor array which presents an interesting enhancement and/or an alternative to the currently employed
Fig. 1. Schematics of the DI fire monitoring and verification concept.
A. Hackner et al. / Sensors and Actuators B 231 (2016) 497–505
499
Fig. 2. (a) Lab prototype of DI camera looking at a simple test scene; (b, c): images and corresponding matrix representations M1 and M2. The two horizontal axes in the 3D surface plots below represent the two pixel indices i and j and the vertical one the grey scale values in each pixel.
pare two normal images (NI) of this same scene taken at different times. The first one shows the scene without and the second with the candle being lighted. The bottom panel of Fig. 2b and c shows the matrix representations of both images in the form of 3d surface plots. While three image sections stand out in the matrix representation of Fig. 2b, which originate from the text messages in this same picture, a forth feature in the matrix of Fig. 2c stands out, which is due to the burning flame. Fig. 3a and b display the difference images extracted from the normal images in Fig. 2b and c. The operation of difference image generation is visualized by the matrix M3 displayed in Fig. 3a. This latter matrix was obtained by subtracting the original image matrices M1 and M2 on a pixel-by-pixel basis and by subtracting the noisy background. Fig. 3b shows the result of digitalisation. In this process each pixel with a non-zero grey scale value in M3 was set to maximum brightness to produce matrix M4. In the corresponding picture (Fig. 3b) the fire feature “flame” clearly stands out. Some error pixels originate from differences in the text messages in both original pictures. The above example clearly demonstrates that difference images help to focus attention on those parts of a scene that have changed in between two successive images of that same scene. As explained in the introduction those scenes which are really relevant for the purpose of aeronautic fire detection, i.e., the interior of electronic boxes, cargo containers or cargo compartments, are not expected to change at all, at least as long as a fire is not initiated. In those non-fire situations, fire monitoring is supposed to proceed in an unsupervised manner, i.e., without human attention. During these unsupervised periods, the difference image information may be compressed in the way shown in Eq. (1). GSn =
1 imax jmax
imax jmax Mn i=0 j=0
max
(1)
In this latter equation GSn is the average grey scale in difference image n, represented by the matrix Mn . The pixel values i,j are divided by max , the maximum possible value of i,j which corresponds to “white”. The sums are over all row and column values that i and j may take. Repeatedly performing this procedure for any pair of subsequent images, an analogue signal is generated which can be compared with the outputs of other sensors that measure complementary fire features such as heat, smoke, CO or other reactive gases. The concept of difference images is further illustrated by the series of normal and difference images displayed in Fig. 4. In the
Fig. 3. Difference images (top) and corresponding matrix representations (bottom) extracted from the original pictures Fig. 2b and c after noise removal (a, M3) and after digitalization (b, M4).
first pair of images (a, b), the difference is simply flame off/flame on while in the second pair (b, c) the flame is only disturbed by a small air flow. The average grey scales in the bottom row illustrate that the data compression effectively distinguishes between large and small optical changes. 4. Difference-image-based multi-criteria fire detection In order to turn DIM into a useful fire detection technology, the DIM signals need to be combined with sensor signals which contain complementary information about independent fire indicators. This latter measure needs to be taken to distinguish a true fire event from any accidental optical change that might have happened for a multitude of reasons. As already mentioned above, useful alternative fire indicators are heat, smoke, reactive gases and CO. In order to arrive at such an innovative kind of multi-criteria fire detector we have built a heterogeneous multi-sensor array consisting of a MOX gas sensor array and a custom-built DI camera module as suggested in Fig. 1. Both elements of this array are shown in Fig. 5. Fig. 5a shows three commercial gas sensors connected through USB ports to a Labview application on a PC. This array consisted of three commercial sensors purchased from UST [18]. The individual sensors had cross sensitivity profiles which peaked at the main breakdown products of electrolytes emerging from damaged Lithium (Li) batteries [20] and all sensors were operated at the optimum operation temperatures Top recommended by the manufacturer [18]: • GGS6000T (MOX 1): Sensor for hydrogen with low cross sensitivity to methane, humidity and alcohol, Top = 350 ◦ C; • GGS2000T (MOX 2): Sensor for carbon monoxide, hydrogen and alcohol, low cross-sensitivity to methane, Top = 400 ◦ C; • GGS1000T (MOX 3): Universal sensor especially suitable for leak detection of combustible gases, Top = 400 ◦ C. The other part is a custom-built DI camera, also connected through a USB port to another Labview application (Fig. 5b). As discussed in more detail in the appendix, the DI camera module is able
500
A. Hackner et al. / Sensors and Actuators B 231 (2016) 497–505
Fig. 4. Monitoring a small flame in front of a complex background. Top row: normal images; bottom row: corresponding difference images. The numerical values inside both difference images represent the associated average grey scale values.
Fig. 5. Components of the heterogeneous fire sensor array: (a) MOX gas sensor array provided by UST [18]; (b) DI camera with data acquisition PC. Computer screen: difference image in response to hand waving before camera, Labview image: compressed DI data as a function of time.
to produce both normal and difference images of the monitored scene on a PC screen. Alternatively, the DI camera can work in an unsupervised mode producing a single time-dependent analogue signal whose magnitude is proportional to the average change in grey scales in consecutive difference images. Both data acquisition modules sample the sensing layer resistances (RES) and the heater powers (CAT) of the MOX gas sensors and of the DI camera with a rate of one per second, thus producing a 7 × n matrix of output values. In the following we describe how such MOX/DI combinations perform in realistic fire detection scenarios. Both of these scenarios are concerned with the overheating and destruction of (Li) batteries. As such batteries contain flammable electrolytes and a built-in source of energy, which can cause ignition, such batteries are classified hazardous materials both when transported as payload in aircrafts as well as when used as on-board energy sources. 4.1. Scenario 1: battery failure and inflammation following overcharging In this first scenario a Li battery was overcharged (DOW KOKAM SLPB554374H pouch cell; capacity: 1.25 Ah; end-of-charge voltage: 4.2 V). In this overcharging experiment the battery was charged with a constant voltage of 6 V, well beyond its end-of-charge voltage of 4.2 V. As a result of this overcharging, the cell temperature rose to about 70 ◦ C as the cell voltage reached 5.3 V. In the course of
this treatment the cell visibly swelled until it became leaky, evolved gases and suddenly caught fire. This latter event occurred after about 15 min of overcharging. Laboratory analysis of the evolved gases identified these as H2 , O2 , CO, CO2 and dimethyl carbonate [20]. These processes of cell swelling and destruction are shown in the form of direct images and difference images in Figs. 6 and 7. Fig. 8 shows the variation of the individual sensor responses at and around the moment of cell destruction. Very obviously, a strong, sudden and simultaneous change in the DI camera output and in the RES responses of all MOX gas sensor occurs at around tdes = 0s. Multiplication of all signals generates a simple trigger signal. Upon receiving this trigger signal, a fire alarm is issued and the direct and difference images taken around the trigger event (Fig. 7) can be displayed on the computer screen for alarm verification. A look at the images in Fig. 7 immediately shows that a true fire event has occurred and that emergency actions need to be taken. In comparison to the DI and RES responses, a CAT response of the MOX gas sensors cannot be observed. This missing CAT response can be explained by the fact that almost all combustible electrolyte fluid had already been burnt down in the cell destruction event, leaving only few combustible molecules for a direct interaction with the MOX gas sensor layers. The gaseous combustion products, which actually arrive there, obviously do no longer produce appreciable amounts of combustion power but still sizeable RES responses. Fig. 9 shows the variation of the DI and MOX gas sensor signals once again, considering, however, larger spans of time before cell
A. Hackner et al. / Sensors and Actuators B 231 (2016) 497–505
501
Fig. 6. Evolution of Li battery during the overcharging experiment and before destruction at tdes = 0s: (top row) direct images taken at 1000s, 200s and 1s before destruction, (bottom row) difference images derived from direct images a and b (DIab ) and from images b and c (DIbc ). GSab and GSac denote the corresponding average grey scale values in each difference image.
Fig. 7. Evolution of Li battery during the overcharging experiment at times immediately before and after cell destruction at tdes = 0s: (top row) direct images taken at −1s, 0s and 1s; (bottom row) difference images derived from the direct images in the top row. GSab and GSac denote average grey scale values in both difference images.
destruction. This latter data set shows that the DI/MOX sensor array is also able to provide a certain degree of pre-warning. Comparing the two different versions of DI sensor signals, it is evident that the blowing up of the cell volume becomes measurable at around 500s before the actual cell destruction at tdes = 0s. At around this time, the swelling of the cell volume is reflected by a monotonous increase in the DI signals when these are referenced with regard to the initial frame at start-up. At around tdes -130s the development of small cell leakages becomes evident from small, sudden changes in both DI signals. Reference to the direct image video sequences shows that these sudden DI changes derive from small, sudden recoil displacements of the cell as first small gas leaks develop. Reference to the gas sensor data shows that these sharply focussed, directed gas jets do not necessarily hit the MOX gas sensors and thus go undetected. The actual cell destruction, on the other hand, produces a widely dispersed gas cloud, which is easily detected. 4.2. Scenario 2: battery failure following short-circuiting In this second scenario we consider a case in which a gas alarm is initiated but in which no open fire can be detected. This latter
scenario is interesting because it demonstrates the value of using uncompressed difference images in the alarm verification process. In this latter scenario a Li battery was slowly heated toward increasingly higher temperatures to simulate the case of an unintentional short circuit situation (Panasonic NCR-18650, capacity: 2.9Ah; end-of-charge voltage: 4.2 V). Similar as in scenario 1 above, overpressure was generated inside the cell, causing it to become leaky and to eject a fine spray of electrolyte fluid. Unlike in scenario 1, this hot hydrocarbon spray did not catch fire, and therefore no easily visible open fire was initiated. This latter situation was monitored with the same virtual sensor array as above, yielding the sensor outputs summarised in Fig. 10 and the verification images displayed in Fig. 11. Taken together, the data in Fig. 10 reveal that the MOX gas sensors clearly detect the expulsion of the hot electrolyte both in the form of RES and CAT responses. Additionally, the electrolyte expulsion is also detected by the sequence of compressed DI camera signals. Again, as in scenario 1, a concurrent change in the DI and RES responses can be observed which allows these signals to be merged into a single trigger signal that issues a pre-alarm. Because of the absence of an open fire, however, this pre-alarm is much
502
A. Hackner et al. / Sensors and Actuators B 231 (2016) 497–505
Fig. 8. Sensor signal changes in the immediate vicinity of the destruction event (tdes = 0s). Top panel: Normalized grey scale values produced at the DI camera output (red: difference images taken in between consecutive frames); blue difference images at time t as referenced to the initial frame after start-up of the overcharging experiment. Middle panel: Normalized CAT response of MOX gas sensors in response to cell inflammation; (bottom panel): Normalized RES response of MOX gas sensors in response to cell inflammation. Battery failure is indicated by a sudden change in all DI and RES sensor outputs. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
Fig. 10. Normalized output signals of the three MOX gas sensors (black, blue, red) and of the compressed DI camera output (red, blue). Battery failure is again indicated by a sudden change in all sensor signals. In addition to the normal resistive (RES) responses, the MOX gas sensors also show sizeable but delayed CAT responses. When compared to scenario 1, the much stronger CAT response derives from the impact of much higher concentrations of unburnt gaseous electrolyte components on the MOX gas-sensing layers. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
hazard. In this latter case a potential fire hazard is indicated but immediate emergency measures are less indicated than in scenario 1. Another interesting difference with regard to scenario 1 is the occurrence of a measurable and delayed CAT response. Whereas the much larger CAT response can be explained by the interaction of unburnt electrolyte directly with the gas sensitive MOX layers, the 5s delay occurs because of the thermal inertia of the ceramic heaters and the heater control circuits [18]. 5. Conclusions and outlook
Fig. 9. Top panel: Normalized output signals of the compressed DI camera output (blue; bottom panel): MOX gas sensor outputs (red, blue, black) before, during and after cell destruction at tdes = 0s. Battery failure is clearly indicated by a sudden change in all sensor signals thus forming a trigger event. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
harder to identify as a true fire alarm when simply considering the normal images (a, b) following the trigger event. This problem is illustrated in the top row of Fig. 11. In contrast, the difference images derived from this same sequence of normal images clearly reveal the electrolyte expulsion and the presence of a potential fire
The above-described sensor concept has arisen out of the special and extremely tight false alarm rate requirements in the field of aeronautic fire detection. Key enabling factor for this development was the increasing availability of cheap CMOS cameras for the visible (VIS) and near infrared (NIR) ranges. To date, prices and power consumption of VIS cameras are approaching levels of more conventional fire detectors such as smoke detectors and/or MOX gas sensors, which call for combinations of these new sensors and the more established ones. More recently, mid infrared (MIR) camera chips have also appeared on the market, following a similar progression of price decay [21], which again makes them attractive for monitoring fire detection scenarios. A key step forward in the realization of image-based and image-enhanced multi-criteria fire detectors is the process of DI monitoring. DI monitoring and DI data compression reduces the complexity of the camera outputs to simple analogue signals, which allows cameras to be merged with more conventional fire detectors to form multi-criteria detectors. Possible alternative sensor combi-
A. Hackner et al. / Sensors and Actuators B 231 (2016) 497–505
503
Fig. 11. (top row) Li battery in heatable metal block (a) before and (b) after electrolyte ejection; (bottom row) difference images revealing the electrolyte ejection; (c) as obtained, (d) image c 5-fold amplified.
Fig. A1. Block circuit diagram of the DI camera module (center) interacting with MOX gas sensor array and system software (Labview).
nations are DI cameras with smoke, CO and infrared optical gas detectors. The combination with optical gas sensors is particularly interesting as such sensors are inherently self-testable, a property which is not shared by their MOX counterparts. In addition to selftestability, optical gas sensors also feature excellent selectivity and potentially very high sensitivities, in particular to CO [22–24]. All kinds of multi-criteria fire detectors produce multi-dimensional output vectors of orthogonal fire signals, which can be processed using the whole range of signal evaluation procedures which have arisen out of the field of olfactory research. Significantly reduced false alarm rates should be achievable in this way.
Low-cost, solid-state cameras potentially are in competition with conventional smoke detectors as both detect visible changes within the monitored environment. In comparison to smoke detectors, VIS and IR cameras have the added advantage that pre-alarms that had been generated in the unsupervised mode, can be verified by visual inspection using the full 2d image information at the occurrence of a potential fire event. In this way, important cost savings due to the avoidance of unnecessary emergency measures can be realized.
504
A. Hackner et al. / Sensors and Actuators B 231 (2016) 497–505
Acknowledgment The authors acknowledge financial support under the EU FP7 project “SNOOPY” (Sniffer for Concealed People Discovery); SEC2012.3.4-4 Innovative, cost-efficient and reliable technology to detect humans hidden in vehicles/closed compartments.
Appendix : Difference image camera architecture Following the above-described concepts we have realized a prototype DI camera. This camera builds on a commercially available low-cost CCD camera module [19], which in its conventional normal image (NI) mode, yields direct images of the monitored scenes. In order to turn such a camera into a DI one, an electronic circuit is required that can be interfaced in between the video output of a CCD camera and an external monitor. The block circuit diagram of this circuit is shown in Fig. A1. In order to obtain difference images, a fraction of the video signal is sent through a Sync Separator block, where the row and picture synchronisation signals are removed from the analogue pixel grey scale values. These remaining analogue signals are converted into digital signals and the digital picture information contained in pairs of subsequent images is dynamically stored in the micro controller unit (MCU) [25]. Once stored, the grey scales in both images are subtracted from each other on a pixel-by-pixel basis. In the end, the difference values are reconverted into analogue values and recombined with the Sync signals to arrive at a full-fletched piece of image information again. At this stage, every image simply highlights the optical change that has taken place in between every pair of subsequent normal images. In the mixer, finally, the normal (NI) and difference (DI) images may be mixed, i.e., linearly superimposed on each other, for easy viewing of the monitored scene. On the output screen, a human observer can then easily verify which item in the monitored scene is affected by a smouldering or open fire and to what extent. This mode of operation of the DI camera module obviously is concerned with the process of alarm verification. In the vast majority of time, where an alarm situation does not exist and in which the controlled area needs to be monitored for the eventual occurrence of such a situation, the camera needs to work in an unsupervised mode in which it simply contributes an analogue signal to the output signal vector of the multi-criteria fire detector. During these periods, the pixel grey scale values, dynamically stored inside the MCU block, are averaged on an image-by-image basis, yielding an average grey scale value for each subsequent image. In this way a time-dependent analogue signal is obtained that is output at the bottom of Fig. A1 and that can be combined with the MOX sensor outputs into a multi-dimensional signal vector, which can be processed with a whole range of methods well known in the field of multi-variate analysis.
References [1] Enis C¸etin, Kosmas Dimitropoulos, Benedict Gouverneur, Nikos Grammalidis, Osman Günay, Y. Hakan Habiboˇglu, B. Uˇgur Töreyin, S. Verstockt, Video fire detection—review, Dig. Signal Process. 23 (December (6)) (2013) 1827–18433. [2] A.A. Ahmad Alkhatib, A review on forest fire detection techniques, Int. J. Distrib. Sensor Netw. 2014 (2014) 12, http://dx.doi.org/10.1155/2014/ 597368, Article ID 597368. [3] Giuseppe Marbach, Markus Loepfe, Thomas Brupbacher, An image processing technique for fire detection in video images, Fire Saf. J. 41 (June (4)) (2006) 285–289. [4] Byoungmoo Lee, Dongil Han, Real-time fire detection using camera sequence image in tunnel environment, Int. Conf. Intell. Comput.−ICIC (2007) 1209–1220, http://dx.doi.org/10.1007/978-3-540-74171-8 123. [5] Rob Bogue, Sensors for fire detection, Sensor Rev. 33 (2) (2013) 99–103. [6] Zhigang Liu, Andrew Kim, Review of recent developments in fire detection technologies, J. Fire Prot. Eng. 13 (May (2)) (2003) 129–151. [7] https.//en.wikipedia.org/wiki/Smoke detector (accessed: 30.01.16).
[8] Peter Reimann, Andreas Schütze, Fire detection in coal mines based on semiconductor gas sensors, Sensor Rev. 32 (1) (2012) 47–58, http://dx.doi. org/10.1108/02602281211197143. [9] http://aviation-safety.net/statistics/period/stats.php?cat=A1. [10] W.L. Grosshandler, Nuisance Alarms in Aircraft Cargo Areas and Critical Telecommunications Systems: Third (3rd) NIST Fire Detector Workshop. Proceedings. National Institute of Standards and Technology. December 4–5 (1997) Gaithersburg, MD, USA. [11] David Blake Aircraft Cargo Compartment Smoke Detector Alarm Incidents on U.S.-Registered Aircraft, 1974–1999 Federal Aviation Administration, U.S. Department of Transportation, June (2000). [12] Klaus Schmötzer, Aircraft Automatic Fire Detection: Requirements, Qualification and Certification Aspects, AUBE 01, Proc. 12th Int. Conf. on Automatic Fire Detection, National Institute of Standards and Technology (NIST), March 25–28, 2001, Gaithersburg, Maryland, USA, http://fire.nist.gov/ bfrlpubs/fire01/PDF/f01060.pdf. [13] Fire protection: Cargo Compartments, http://www.boeing.com/commercial/ aeromagazine/articles/2011 q2/3/. [14] http://www.faa.gov/regulations policies/handbooks manuals/aircraft/amt airframe hand book/media/ama Ch17.pdf. [15] Susan L. Rose-Pehrsson, Ronald E. Shaffer, Sean J. Hart, Frederick W. Williams, Daniel T. Gottuk, Brooke D. Strehlen, Scott A. Hill, Multi-criteria fire detection systems using a probabilistic neural network, Sens. Actuators B 69 (2000) 325–335. [16] http://www.buildingtechnologies.siemens.com/bt/global/en/firesafety/firedetection/Pages/fire-detection.aspx (last accessed 01.02.16). [17] http://us.boschsecurity.com/us product/05 news and extras 2/01 productnews 2/06 productnews fire 1/fire detection (last accessed 01.02.16). [18] www.umweltsensortechnik.de; data sheets of the sensors used in this work can be downloaded from the company website. The temperature control circuits, which maintain constant sensor operation temperatures and which allow CAT response signals to be extracted, have been developed by UST upon request by AIRBUS INNOVATIONS. [19] Platinenkamera 8,47 mm (1/3) CCD S/W-Kamera mit 6 mm Linse—EAN: 4016138350708 Distributor Conrad Electronic SE; http://www.conrad.de. [20] Gases emerging from the stressed Li batteries were analysed with conventional laboratory equipment: (i) FTIR absorption spectrometer (Ansyco GASMET Dx-4000 (Temet), (ii) Flame ionisation detector (Agilent 6890 N with HP-MOLESIEVE, 30 m x 0,530 mm x 50 (m), (iii) Gas chromatograph (Agilent 6890 N mit HP-MOLESIEVE, 30 m x 0,530 mm x 50 (m). Main ejected gases were identified as H2, O2, CO2, CO and dimethyl-carbonate. [21] http://www.flir.de/home/. [22] O. Schulz, Th. Becker, G. Müller, IR sensor systems for hazard detection in aerospace applications using fire signature analysis, in: Proc. AMA Conference, Nurnberg, 2004. [23] Kari Schjølberg-Henriksen, Olaf Schulz, Alain Ferber, Sigurd Moe, Martin Lloyd, Gerhard Müller, Karl-Heinz Suphan, Dag T. Wang, Ralph W. Bernstein, Sensitive and selective photoacoustic gas sensor suitable for high-volume manufacturing, IEEE Sens. J. 8 (2008) 1539–1545. [24] Stefanie Heine, Optische Gassensorik zur Überwachung der Kabinenluft in Flugzeugen, Masterthesis, Munich Univ. Appl. Sci. (2014). [25] PIC18F26k22—8-bit MCU from Microchip Technology Inc. Arizona, USA.
Biographies A. Hackner (1966) received a first degree in Technical Physics at the Munich University of Applied Science in 1991. After several years of industrial employment she received a second MSc. degree in Micro and Nano-Technology from this same University in 2008. Currently she is working in the Sensor Integration group within the Department of Electronics, Communication & Intelligent Systems at Airbus Group Innovations in Munich. Her expertise is in the fields of silicon micromachining and microsensor technology. Her current focus is on solid state chemical sensors, in particular surface ionization gas detectors and optical MID IR absorption detectors. H. Oberpriller (1959) received his degree in physical chemistry at the Munich University of Applied Science in 1981. Since 1981 he is working mainly in the fields of analytical chemistry, failure analysis and cabin air quality. Currently he is part of the Power Generation team within the Department of Energy & Propulsion at Airbus Group Innovations in Munich. A. Ohnesorge (1974), Process engineer, received his diploma in process engineering at the Technical University of Dresden (TUD) in 2001. From 2002–2005 worked as a PhD student at EADS Dornier GmbH in Friedrichshafen with focal points at laser spectroscopy and infrared technology. After his PhD. in 2008 at the TUD he was employed as a researcher in the R&D department of the Solar World AG in Freiberg. Since 2009 he is a research engineer at Airbus Defence and Space/Airbus Innovations in Munich, working on H2 generation and fuel cells. He has experience in chemical and process engineering, laser technology; his current focus is on advanced batteries and other electrochemical storage devices. Kurt-Volker Hechtenberg graduated in Electrical Engineering from the University of Kaiserslautern in 1975. From 1975 to 2005 he was employed as a development engineer at MBB, DASA and EADS. Sensor systems deriving from his development activities included an acoustical bearing system for submarines based on digital
A. Hackner et al. / Sensors and Actuators B 231 (2016) 497–505 correlation, sensors for antitank missiles, sensors for laser-warning receivers, signal processing for a high-dynamic camera, an IR-camera, a stereo-camera, a camera with an artificial retina and a sensor aircraft based maintenance system. From 2005 he is self-employed and still affiliated with EADS, AIRBUS and Lufthansa. Recent activities have been focused on a flight approved security GPS-GSM based tracker module. Gerhard Müller received a PhD. degree in Physics from the University of Heidelberg in 1976. After academic positions at the Max-Planck Institute of Nuclear Physics in Heidelberg and at the University of Dundee in the period 1976–1980, he started
505
a career in industrial research and development working for several employers, including EADS Innovation Works where he served as “Senior Expert Sensor Physics” up to his retirement in 2013. In 2012 he was inducted into the EADS Hall of Fame in the category “Great Inventor”. After his retirement he continues to be a lecturer in the Munich University of Applied Sciences. He is author and co-author of more than 250 papers, including more than 30 invited papers at international conferences, in the fields of semiconductor physics, photovoltaics, thin film and sensor technology as well as sensor applications.
Contents lists available at ScienceDirect
Sensors and Actuators B: Chemical journal homepage: www.elsevier.com/locate/snb
Heterogeneous sensor arrays: Merging cameras and gas sensors into innovative fire detection systems Angelika Hackner a,∗ , Helmut Oberpriller a , Alexander Ohnesorge a , Volker Hechtenberg b , Gerhard Müller a,c a
Airbus Group Innovations, D-81663 Munich, Germany Innovation Team Bremen, D-17111 Meesiger, Germany c Munich University of Applied Sciences, Fachbereich 06, Lothstraße 34, D-80335 Munich, Germany b
a r t i c l e
i n f o
Article history: Received 22 October 2015 Received in revised form 9 February 2016 Accepted 18 February 2016 Available online 19 March 2016 Keywords: Multi-criteria fire detection Image processing Gas detection False alarm rate Alarm verification
a b s t r a c t An innovative way of multi-criteria fire detection is introduced. In this concept image information about the monitored area is combined with knowledge about the reactive gas concentrations inside this same area to produce a fire alarm and to enable alarm verification. Key component in this new mode of fire detection is a low-cost CCD camera which is operated in a difference image (DI) mode. In the unsupervised DI mode, the average grey scale values in consecutive difference images are evaluated and compared with the outputs of a metal oxide gas sensor array which measures the reactive gas concentrations in this same area. Fire alarms are generated upon observing concurrent changes in all sensor outputs. Once obtained, the camera can be switched to the supervised imaging mode and the validity of an alarm can be assessed by normal and/or difference imaging of the monitored area. In this paper the concept of DI-based fire detection is explained, demonstrated and possible enhancements and ramifications of this new technique are discussed. © 2016 Elsevier B.V. All rights reserved.
1. Introduction Fires represent severe danger to human lives, property and environment. In order to cope with such dangers, efficient methods of fire detection and firefighting are required. As far as fire detection is concerned, this field can be broadly subdivided into two sub-fields: (i) wide-area surveillance for the detection of fires in unconfined spaces, such as in forests, road tunnels or underground stations, and (ii) point-detection where fire hazards occur in closely confined spaces such as in office and in residential buildings or in factory plants. Whereas camera surveillance is the method of choice in the first class of environments, point detectors such as smoke, ionization, heat and gas detectors are being used in confined spaces. State of the art reviews of these existing fire detection technologies can be found in references [1–8]. As revealed there, video image processing focuses on the detection of spatial-temporal features such as color probability, contour irregularity in individual frames, and on temporal changes in between consecutive frames, i.e., on all pieces of information that can be conveyed over large distances via electromagnetic radiation. Point detectors, in contrast, concen-
∗ Corresponding author. Fax: +49 89 60724001. E-mail address: [email protected] (A. Hackner). http://dx.doi.org/10.1016/j.snb.2016.02.081 0925-4005/© 2016 Elsevier B.V. All rights reserved.
trate on those fire indicators which derive from localized chemical reactions that occur in open and smouldering fires and that are transported much more slowly over smaller distances via convection and diffusion processes. As both kinds of detectors are only sensitive to a limited sub-set of all possible fire features, sensorand application-specific false-alarm issues arise. A fire detection application with a very high damage potential, which is plagued by false alarm issues, is aeronautic fire detection [9–11]. Active fire protection inside commercial airplanes comprises fire detection by photo-electric automatic fire detectors as well as firefighting with extinguishing equipment [12,13]. Such equipment is installed in all those areas that are inaccessible during flight. These include electronic compartments such as avionics and in-flight entertainment equipment and, last not least cargo compartments and cargo containers. Among those the cargo compartments bear a particularly high danger potential because very few requirements actually restrict the flammability of any potential cargo. In the current state of the art, fire detection in such critical areas is performed with the help of smoke detectors [5–8], similar to those that are routinely used in office and in residential buildings. Such sensors detect open fires by scattering light on airborne particles. As smoke particles are only one sort of airborne particulate matter that might turn up, false alarms will almost certainly occur [9–11]. Well known causes of false alarms are dust particles and/or
498
A. Hackner et al. / Sensors and Actuators B 231 (2016) 497–505
condensing humidity. These latter forms of false alarms can easily occur in aircraft operation when an airplane is loaded on ground in a hot and humid environment and as the air inside the cargo compartment cools down as the airplane climbs to its cruising altitude. Statistics issued by the Federal Aviation Agency (FAA) [9], show that the probability of such false alarms is indeed very high. Investigations over a five year period showed that in hindsight only 1 out of 200 reported alarms could be unambiguously traced to a true smoke event. While false alarms of this type may be acceptable on ground, these cause severe problems when they occur in flight. As a fire, that may be harmless on ground, might become catastrophic in flight and as false alarms cannot be verified as such during flight, tight authority requirements exist on how to deal with such alarms [14]. These require that any fire alarm—whether true or not—needs to be answered by fire-fighting measures and by emergency landings. Statistics show that such events occur with a rate of about 200–300 per year with most of them ending in unscheduled flight interruptions [9–11]. The associated cost of such flight interruptions is estimated to range in between 30–50 thousand D /per event, depending on the size and occupancy of the airplanes [11]. Trying to decrease false alarm rates from the current rate of about 1 in 105 flight hours to the newly requested standard of 1 in 107 flight hours [12], several attempts have been made to develop multi-criteria fire detectors which measure several fire criteria at the same time, thereby enabling lower false alarm rates [15–17]. All these existing multi-sensor approaches employ heterogeneous arrays of sensors with scalar output signals, chiefly smoke, heat, and CO. Our current work contributes to this effort of attaining lower false-alarm rates by employing heterogeneous sensor arrays. In our approach conventional gas sensor arrays are combined with low-cost CCD cameras to detect fire gases and concurrent optical changes in the confined spaces of aircraft compartments. To the best of our knowledge, this is the first attempt to include cameras into heterogeneous sensor arrays to attain higher levels of false alarm rejection and unpresented possibilities of alarm verification. The fire detection system discussed below can be operated in an unsupervised monitoring mode in which the CCD camera is operated in a difference image (DI) mode in which grey scale changes in between consecutive images are monitored alongside with the potential appearance of fire gases which might correlate with the observed visual changes. In case concurrent changes in the gas evolution and consecutive image frames are detected, an alarm is issued and the system is switched into its verification mode. In this latter mode the CCD camera provides a straight-forward picture of the monitored scene which allows the pilot to take an informed decision as to the actual necessity of emergency measures. In the following, this innovative sensor concept is introduced (§2) and the processes underlying DI signal generation and DI monitoring (DIM) are explained (§3). §4 presents the results of feasibility tests in which two realistic fire detection scenarios are monitored using a heterogeneous array of sensors consisting of a DI camera and a metal oxide gas sensor array. In the concluding §5 possible enhancements and ramifications of the DIM approach are discussed.
smoke detectors [5,6,8]. As illustrated in the left-hand part of Fig. 1, reactive gas monitoring is performed with the help of a gas sensor array consisting of three MOX gas sensors [18]. These sensors contain different sensitive materials and/or are operated at different temperatures to exhibit cross sensitivity profiles oriented toward different components in the fire gas mix. Such an array produces six analog output signals, which form a signal vector. These signals consist of the sensitive layer resistances of the three MOX gas sensors (resistive/RES response) and of the electrical heater powers that need to be supplied to the substrate heater meanders to maintain constant sensor operation temperatures (catalytic/CAT response). Whereas the first three signals contain information concerning the oxidizing or reducing properties of the adsorbed fire gases, the latter three carry information with regard to the catalytic combustion powers generated by the adsorbed gases [18]. Low false alarm rate requires additional and independent fire indicators to be assessed inside the controlled area. In our approach we chose to monitor optical changes which coincide with open and smouldering fires. These are chiefly flames, smoke or aerosols of combustible liquids. As indicated on the right-hand side of Fig. 1, such visual changes are observed with the help of a low-cost CCD camera [19]. For our purpose a commercial CCD camera, which simply produces straight-forward images of the monitored scene, was modified to produce difference images (DI), which highlight visible changes in between two successive normal images (NI). As a second feature our camera allows to be operated in an unsupervised monitoring and in a supervised verification mode. In the unsupervised mode the DI data are compressed into a sequence of average grey scale values in each difference image. In this way, an additional analogue output signal is generated which can be combined with the six output signals of the gas sensor array to form an output signal vector. This vectorial output can be processed using standard pattern recognition techniques to form a multi-criteria fire detector with a sharply decreased false alarm probability. In the supervised verification mode, the full two-dimensional (2d) image information contained in the NI and/or DI images is used to enable alarm verification by a human observer. This observer (pilot) can then take informed decisions as to the necessity and extent of possible emergency measures. 3. Difference-image monitoring of controlled areas In order to introduce the concept of difference image monitoring (DIM), consider Fig. 2. For the sake of clarity Fig. 2a shows the DI camera module itself with its lens facing a candle in front of a uniform piece of cardboard. Fig. 2b and c, on the other hand, com-
2. Sensor concept The sensor concept evaluated in this paper is presented in Fig. 1. The monitored area in the center is assumed to be one of the aircraft’s critical compartments (avionic, in-flight entertainment, cargo containers and cargo compartments). As explained above, these areas need to be monitored in an unsupervised manner for the presence of certain fire indicators. While in the current state of the art this task is performed by smoke detectors, we chose in our case a metal oxide (MOX) gas sensor array which presents an interesting enhancement and/or an alternative to the currently employed
Fig. 1. Schematics of the DI fire monitoring and verification concept.
A. Hackner et al. / Sensors and Actuators B 231 (2016) 497–505
499
Fig. 2. (a) Lab prototype of DI camera looking at a simple test scene; (b, c): images and corresponding matrix representations M1 and M2. The two horizontal axes in the 3D surface plots below represent the two pixel indices i and j and the vertical one the grey scale values in each pixel.
pare two normal images (NI) of this same scene taken at different times. The first one shows the scene without and the second with the candle being lighted. The bottom panel of Fig. 2b and c shows the matrix representations of both images in the form of 3d surface plots. While three image sections stand out in the matrix representation of Fig. 2b, which originate from the text messages in this same picture, a forth feature in the matrix of Fig. 2c stands out, which is due to the burning flame. Fig. 3a and b display the difference images extracted from the normal images in Fig. 2b and c. The operation of difference image generation is visualized by the matrix M3 displayed in Fig. 3a. This latter matrix was obtained by subtracting the original image matrices M1 and M2 on a pixel-by-pixel basis and by subtracting the noisy background. Fig. 3b shows the result of digitalisation. In this process each pixel with a non-zero grey scale value in M3 was set to maximum brightness to produce matrix M4. In the corresponding picture (Fig. 3b) the fire feature “flame” clearly stands out. Some error pixels originate from differences in the text messages in both original pictures. The above example clearly demonstrates that difference images help to focus attention on those parts of a scene that have changed in between two successive images of that same scene. As explained in the introduction those scenes which are really relevant for the purpose of aeronautic fire detection, i.e., the interior of electronic boxes, cargo containers or cargo compartments, are not expected to change at all, at least as long as a fire is not initiated. In those non-fire situations, fire monitoring is supposed to proceed in an unsupervised manner, i.e., without human attention. During these unsupervised periods, the difference image information may be compressed in the way shown in Eq. (1). GSn =
1 imax jmax
imax jmax Mn i=0 j=0
max
(1)
In this latter equation GSn is the average grey scale in difference image n, represented by the matrix Mn . The pixel values i,j are divided by max , the maximum possible value of i,j which corresponds to “white”. The sums are over all row and column values that i and j may take. Repeatedly performing this procedure for any pair of subsequent images, an analogue signal is generated which can be compared with the outputs of other sensors that measure complementary fire features such as heat, smoke, CO or other reactive gases. The concept of difference images is further illustrated by the series of normal and difference images displayed in Fig. 4. In the
Fig. 3. Difference images (top) and corresponding matrix representations (bottom) extracted from the original pictures Fig. 2b and c after noise removal (a, M3) and after digitalization (b, M4).
first pair of images (a, b), the difference is simply flame off/flame on while in the second pair (b, c) the flame is only disturbed by a small air flow. The average grey scales in the bottom row illustrate that the data compression effectively distinguishes between large and small optical changes. 4. Difference-image-based multi-criteria fire detection In order to turn DIM into a useful fire detection technology, the DIM signals need to be combined with sensor signals which contain complementary information about independent fire indicators. This latter measure needs to be taken to distinguish a true fire event from any accidental optical change that might have happened for a multitude of reasons. As already mentioned above, useful alternative fire indicators are heat, smoke, reactive gases and CO. In order to arrive at such an innovative kind of multi-criteria fire detector we have built a heterogeneous multi-sensor array consisting of a MOX gas sensor array and a custom-built DI camera module as suggested in Fig. 1. Both elements of this array are shown in Fig. 5. Fig. 5a shows three commercial gas sensors connected through USB ports to a Labview application on a PC. This array consisted of three commercial sensors purchased from UST [18]. The individual sensors had cross sensitivity profiles which peaked at the main breakdown products of electrolytes emerging from damaged Lithium (Li) batteries [20] and all sensors were operated at the optimum operation temperatures Top recommended by the manufacturer [18]: • GGS6000T (MOX 1): Sensor for hydrogen with low cross sensitivity to methane, humidity and alcohol, Top = 350 ◦ C; • GGS2000T (MOX 2): Sensor for carbon monoxide, hydrogen and alcohol, low cross-sensitivity to methane, Top = 400 ◦ C; • GGS1000T (MOX 3): Universal sensor especially suitable for leak detection of combustible gases, Top = 400 ◦ C. The other part is a custom-built DI camera, also connected through a USB port to another Labview application (Fig. 5b). As discussed in more detail in the appendix, the DI camera module is able
500
A. Hackner et al. / Sensors and Actuators B 231 (2016) 497–505
Fig. 4. Monitoring a small flame in front of a complex background. Top row: normal images; bottom row: corresponding difference images. The numerical values inside both difference images represent the associated average grey scale values.
Fig. 5. Components of the heterogeneous fire sensor array: (a) MOX gas sensor array provided by UST [18]; (b) DI camera with data acquisition PC. Computer screen: difference image in response to hand waving before camera, Labview image: compressed DI data as a function of time.
to produce both normal and difference images of the monitored scene on a PC screen. Alternatively, the DI camera can work in an unsupervised mode producing a single time-dependent analogue signal whose magnitude is proportional to the average change in grey scales in consecutive difference images. Both data acquisition modules sample the sensing layer resistances (RES) and the heater powers (CAT) of the MOX gas sensors and of the DI camera with a rate of one per second, thus producing a 7 × n matrix of output values. In the following we describe how such MOX/DI combinations perform in realistic fire detection scenarios. Both of these scenarios are concerned with the overheating and destruction of (Li) batteries. As such batteries contain flammable electrolytes and a built-in source of energy, which can cause ignition, such batteries are classified hazardous materials both when transported as payload in aircrafts as well as when used as on-board energy sources. 4.1. Scenario 1: battery failure and inflammation following overcharging In this first scenario a Li battery was overcharged (DOW KOKAM SLPB554374H pouch cell; capacity: 1.25 Ah; end-of-charge voltage: 4.2 V). In this overcharging experiment the battery was charged with a constant voltage of 6 V, well beyond its end-of-charge voltage of 4.2 V. As a result of this overcharging, the cell temperature rose to about 70 ◦ C as the cell voltage reached 5.3 V. In the course of
this treatment the cell visibly swelled until it became leaky, evolved gases and suddenly caught fire. This latter event occurred after about 15 min of overcharging. Laboratory analysis of the evolved gases identified these as H2 , O2 , CO, CO2 and dimethyl carbonate [20]. These processes of cell swelling and destruction are shown in the form of direct images and difference images in Figs. 6 and 7. Fig. 8 shows the variation of the individual sensor responses at and around the moment of cell destruction. Very obviously, a strong, sudden and simultaneous change in the DI camera output and in the RES responses of all MOX gas sensor occurs at around tdes = 0s. Multiplication of all signals generates a simple trigger signal. Upon receiving this trigger signal, a fire alarm is issued and the direct and difference images taken around the trigger event (Fig. 7) can be displayed on the computer screen for alarm verification. A look at the images in Fig. 7 immediately shows that a true fire event has occurred and that emergency actions need to be taken. In comparison to the DI and RES responses, a CAT response of the MOX gas sensors cannot be observed. This missing CAT response can be explained by the fact that almost all combustible electrolyte fluid had already been burnt down in the cell destruction event, leaving only few combustible molecules for a direct interaction with the MOX gas sensor layers. The gaseous combustion products, which actually arrive there, obviously do no longer produce appreciable amounts of combustion power but still sizeable RES responses. Fig. 9 shows the variation of the DI and MOX gas sensor signals once again, considering, however, larger spans of time before cell
A. Hackner et al. / Sensors and Actuators B 231 (2016) 497–505
501
Fig. 6. Evolution of Li battery during the overcharging experiment and before destruction at tdes = 0s: (top row) direct images taken at 1000s, 200s and 1s before destruction, (bottom row) difference images derived from direct images a and b (DIab ) and from images b and c (DIbc ). GSab and GSac denote the corresponding average grey scale values in each difference image.
Fig. 7. Evolution of Li battery during the overcharging experiment at times immediately before and after cell destruction at tdes = 0s: (top row) direct images taken at −1s, 0s and 1s; (bottom row) difference images derived from the direct images in the top row. GSab and GSac denote average grey scale values in both difference images.
destruction. This latter data set shows that the DI/MOX sensor array is also able to provide a certain degree of pre-warning. Comparing the two different versions of DI sensor signals, it is evident that the blowing up of the cell volume becomes measurable at around 500s before the actual cell destruction at tdes = 0s. At around this time, the swelling of the cell volume is reflected by a monotonous increase in the DI signals when these are referenced with regard to the initial frame at start-up. At around tdes -130s the development of small cell leakages becomes evident from small, sudden changes in both DI signals. Reference to the direct image video sequences shows that these sudden DI changes derive from small, sudden recoil displacements of the cell as first small gas leaks develop. Reference to the gas sensor data shows that these sharply focussed, directed gas jets do not necessarily hit the MOX gas sensors and thus go undetected. The actual cell destruction, on the other hand, produces a widely dispersed gas cloud, which is easily detected. 4.2. Scenario 2: battery failure following short-circuiting In this second scenario we consider a case in which a gas alarm is initiated but in which no open fire can be detected. This latter
scenario is interesting because it demonstrates the value of using uncompressed difference images in the alarm verification process. In this latter scenario a Li battery was slowly heated toward increasingly higher temperatures to simulate the case of an unintentional short circuit situation (Panasonic NCR-18650, capacity: 2.9Ah; end-of-charge voltage: 4.2 V). Similar as in scenario 1 above, overpressure was generated inside the cell, causing it to become leaky and to eject a fine spray of electrolyte fluid. Unlike in scenario 1, this hot hydrocarbon spray did not catch fire, and therefore no easily visible open fire was initiated. This latter situation was monitored with the same virtual sensor array as above, yielding the sensor outputs summarised in Fig. 10 and the verification images displayed in Fig. 11. Taken together, the data in Fig. 10 reveal that the MOX gas sensors clearly detect the expulsion of the hot electrolyte both in the form of RES and CAT responses. Additionally, the electrolyte expulsion is also detected by the sequence of compressed DI camera signals. Again, as in scenario 1, a concurrent change in the DI and RES responses can be observed which allows these signals to be merged into a single trigger signal that issues a pre-alarm. Because of the absence of an open fire, however, this pre-alarm is much
502
A. Hackner et al. / Sensors and Actuators B 231 (2016) 497–505
Fig. 8. Sensor signal changes in the immediate vicinity of the destruction event (tdes = 0s). Top panel: Normalized grey scale values produced at the DI camera output (red: difference images taken in between consecutive frames); blue difference images at time t as referenced to the initial frame after start-up of the overcharging experiment. Middle panel: Normalized CAT response of MOX gas sensors in response to cell inflammation; (bottom panel): Normalized RES response of MOX gas sensors in response to cell inflammation. Battery failure is indicated by a sudden change in all DI and RES sensor outputs. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
Fig. 10. Normalized output signals of the three MOX gas sensors (black, blue, red) and of the compressed DI camera output (red, blue). Battery failure is again indicated by a sudden change in all sensor signals. In addition to the normal resistive (RES) responses, the MOX gas sensors also show sizeable but delayed CAT responses. When compared to scenario 1, the much stronger CAT response derives from the impact of much higher concentrations of unburnt gaseous electrolyte components on the MOX gas-sensing layers. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
hazard. In this latter case a potential fire hazard is indicated but immediate emergency measures are less indicated than in scenario 1. Another interesting difference with regard to scenario 1 is the occurrence of a measurable and delayed CAT response. Whereas the much larger CAT response can be explained by the interaction of unburnt electrolyte directly with the gas sensitive MOX layers, the 5s delay occurs because of the thermal inertia of the ceramic heaters and the heater control circuits [18]. 5. Conclusions and outlook
Fig. 9. Top panel: Normalized output signals of the compressed DI camera output (blue; bottom panel): MOX gas sensor outputs (red, blue, black) before, during and after cell destruction at tdes = 0s. Battery failure is clearly indicated by a sudden change in all sensor signals thus forming a trigger event. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
harder to identify as a true fire alarm when simply considering the normal images (a, b) following the trigger event. This problem is illustrated in the top row of Fig. 11. In contrast, the difference images derived from this same sequence of normal images clearly reveal the electrolyte expulsion and the presence of a potential fire
The above-described sensor concept has arisen out of the special and extremely tight false alarm rate requirements in the field of aeronautic fire detection. Key enabling factor for this development was the increasing availability of cheap CMOS cameras for the visible (VIS) and near infrared (NIR) ranges. To date, prices and power consumption of VIS cameras are approaching levels of more conventional fire detectors such as smoke detectors and/or MOX gas sensors, which call for combinations of these new sensors and the more established ones. More recently, mid infrared (MIR) camera chips have also appeared on the market, following a similar progression of price decay [21], which again makes them attractive for monitoring fire detection scenarios. A key step forward in the realization of image-based and image-enhanced multi-criteria fire detectors is the process of DI monitoring. DI monitoring and DI data compression reduces the complexity of the camera outputs to simple analogue signals, which allows cameras to be merged with more conventional fire detectors to form multi-criteria detectors. Possible alternative sensor combi-
A. Hackner et al. / Sensors and Actuators B 231 (2016) 497–505
503
Fig. 11. (top row) Li battery in heatable metal block (a) before and (b) after electrolyte ejection; (bottom row) difference images revealing the electrolyte ejection; (c) as obtained, (d) image c 5-fold amplified.
Fig. A1. Block circuit diagram of the DI camera module (center) interacting with MOX gas sensor array and system software (Labview).
nations are DI cameras with smoke, CO and infrared optical gas detectors. The combination with optical gas sensors is particularly interesting as such sensors are inherently self-testable, a property which is not shared by their MOX counterparts. In addition to selftestability, optical gas sensors also feature excellent selectivity and potentially very high sensitivities, in particular to CO [22–24]. All kinds of multi-criteria fire detectors produce multi-dimensional output vectors of orthogonal fire signals, which can be processed using the whole range of signal evaluation procedures which have arisen out of the field of olfactory research. Significantly reduced false alarm rates should be achievable in this way.
Low-cost, solid-state cameras potentially are in competition with conventional smoke detectors as both detect visible changes within the monitored environment. In comparison to smoke detectors, VIS and IR cameras have the added advantage that pre-alarms that had been generated in the unsupervised mode, can be verified by visual inspection using the full 2d image information at the occurrence of a potential fire event. In this way, important cost savings due to the avoidance of unnecessary emergency measures can be realized.
504
A. Hackner et al. / Sensors and Actuators B 231 (2016) 497–505
Acknowledgment The authors acknowledge financial support under the EU FP7 project “SNOOPY” (Sniffer for Concealed People Discovery); SEC2012.3.4-4 Innovative, cost-efficient and reliable technology to detect humans hidden in vehicles/closed compartments.
Appendix : Difference image camera architecture Following the above-described concepts we have realized a prototype DI camera. This camera builds on a commercially available low-cost CCD camera module [19], which in its conventional normal image (NI) mode, yields direct images of the monitored scenes. In order to turn such a camera into a DI one, an electronic circuit is required that can be interfaced in between the video output of a CCD camera and an external monitor. The block circuit diagram of this circuit is shown in Fig. A1. In order to obtain difference images, a fraction of the video signal is sent through a Sync Separator block, where the row and picture synchronisation signals are removed from the analogue pixel grey scale values. These remaining analogue signals are converted into digital signals and the digital picture information contained in pairs of subsequent images is dynamically stored in the micro controller unit (MCU) [25]. Once stored, the grey scales in both images are subtracted from each other on a pixel-by-pixel basis. In the end, the difference values are reconverted into analogue values and recombined with the Sync signals to arrive at a full-fletched piece of image information again. At this stage, every image simply highlights the optical change that has taken place in between every pair of subsequent normal images. In the mixer, finally, the normal (NI) and difference (DI) images may be mixed, i.e., linearly superimposed on each other, for easy viewing of the monitored scene. On the output screen, a human observer can then easily verify which item in the monitored scene is affected by a smouldering or open fire and to what extent. This mode of operation of the DI camera module obviously is concerned with the process of alarm verification. In the vast majority of time, where an alarm situation does not exist and in which the controlled area needs to be monitored for the eventual occurrence of such a situation, the camera needs to work in an unsupervised mode in which it simply contributes an analogue signal to the output signal vector of the multi-criteria fire detector. During these periods, the pixel grey scale values, dynamically stored inside the MCU block, are averaged on an image-by-image basis, yielding an average grey scale value for each subsequent image. In this way a time-dependent analogue signal is obtained that is output at the bottom of Fig. A1 and that can be combined with the MOX sensor outputs into a multi-dimensional signal vector, which can be processed with a whole range of methods well known in the field of multi-variate analysis.
References [1] Enis C¸etin, Kosmas Dimitropoulos, Benedict Gouverneur, Nikos Grammalidis, Osman Günay, Y. Hakan Habiboˇglu, B. Uˇgur Töreyin, S. Verstockt, Video fire detection—review, Dig. Signal Process. 23 (December (6)) (2013) 1827–18433. [2] A.A. Ahmad Alkhatib, A review on forest fire detection techniques, Int. J. Distrib. Sensor Netw. 2014 (2014) 12, http://dx.doi.org/10.1155/2014/ 597368, Article ID 597368. [3] Giuseppe Marbach, Markus Loepfe, Thomas Brupbacher, An image processing technique for fire detection in video images, Fire Saf. J. 41 (June (4)) (2006) 285–289. [4] Byoungmoo Lee, Dongil Han, Real-time fire detection using camera sequence image in tunnel environment, Int. Conf. Intell. Comput.−ICIC (2007) 1209–1220, http://dx.doi.org/10.1007/978-3-540-74171-8 123. [5] Rob Bogue, Sensors for fire detection, Sensor Rev. 33 (2) (2013) 99–103. [6] Zhigang Liu, Andrew Kim, Review of recent developments in fire detection technologies, J. Fire Prot. Eng. 13 (May (2)) (2003) 129–151. [7] https.//en.wikipedia.org/wiki/Smoke detector (accessed: 30.01.16).
[8] Peter Reimann, Andreas Schütze, Fire detection in coal mines based on semiconductor gas sensors, Sensor Rev. 32 (1) (2012) 47–58, http://dx.doi. org/10.1108/02602281211197143. [9] http://aviation-safety.net/statistics/period/stats.php?cat=A1. [10] W.L. Grosshandler, Nuisance Alarms in Aircraft Cargo Areas and Critical Telecommunications Systems: Third (3rd) NIST Fire Detector Workshop. Proceedings. National Institute of Standards and Technology. December 4–5 (1997) Gaithersburg, MD, USA. [11] David Blake Aircraft Cargo Compartment Smoke Detector Alarm Incidents on U.S.-Registered Aircraft, 1974–1999 Federal Aviation Administration, U.S. Department of Transportation, June (2000). [12] Klaus Schmötzer, Aircraft Automatic Fire Detection: Requirements, Qualification and Certification Aspects, AUBE 01, Proc. 12th Int. Conf. on Automatic Fire Detection, National Institute of Standards and Technology (NIST), March 25–28, 2001, Gaithersburg, Maryland, USA, http://fire.nist.gov/ bfrlpubs/fire01/PDF/f01060.pdf. [13] Fire protection: Cargo Compartments, http://www.boeing.com/commercial/ aeromagazine/articles/2011 q2/3/. [14] http://www.faa.gov/regulations policies/handbooks manuals/aircraft/amt airframe hand book/media/ama Ch17.pdf. [15] Susan L. Rose-Pehrsson, Ronald E. Shaffer, Sean J. Hart, Frederick W. Williams, Daniel T. Gottuk, Brooke D. Strehlen, Scott A. Hill, Multi-criteria fire detection systems using a probabilistic neural network, Sens. Actuators B 69 (2000) 325–335. [16] http://www.buildingtechnologies.siemens.com/bt/global/en/firesafety/firedetection/Pages/fire-detection.aspx (last accessed 01.02.16). [17] http://us.boschsecurity.com/us product/05 news and extras 2/01 productnews 2/06 productnews fire 1/fire detection (last accessed 01.02.16). [18] www.umweltsensortechnik.de; data sheets of the sensors used in this work can be downloaded from the company website. The temperature control circuits, which maintain constant sensor operation temperatures and which allow CAT response signals to be extracted, have been developed by UST upon request by AIRBUS INNOVATIONS. [19] Platinenkamera 8,47 mm (1/3) CCD S/W-Kamera mit 6 mm Linse—EAN: 4016138350708 Distributor Conrad Electronic SE; http://www.conrad.de. [20] Gases emerging from the stressed Li batteries were analysed with conventional laboratory equipment: (i) FTIR absorption spectrometer (Ansyco GASMET Dx-4000 (Temet), (ii) Flame ionisation detector (Agilent 6890 N with HP-MOLESIEVE, 30 m x 0,530 mm x 50 (m), (iii) Gas chromatograph (Agilent 6890 N mit HP-MOLESIEVE, 30 m x 0,530 mm x 50 (m). Main ejected gases were identified as H2, O2, CO2, CO and dimethyl-carbonate. [21] http://www.flir.de/home/. [22] O. Schulz, Th. Becker, G. Müller, IR sensor systems for hazard detection in aerospace applications using fire signature analysis, in: Proc. AMA Conference, Nurnberg, 2004. [23] Kari Schjølberg-Henriksen, Olaf Schulz, Alain Ferber, Sigurd Moe, Martin Lloyd, Gerhard Müller, Karl-Heinz Suphan, Dag T. Wang, Ralph W. Bernstein, Sensitive and selective photoacoustic gas sensor suitable for high-volume manufacturing, IEEE Sens. J. 8 (2008) 1539–1545. [24] Stefanie Heine, Optische Gassensorik zur Überwachung der Kabinenluft in Flugzeugen, Masterthesis, Munich Univ. Appl. Sci. (2014). [25] PIC18F26k22—8-bit MCU from Microchip Technology Inc. Arizona, USA.
Biographies A. Hackner (1966) received a first degree in Technical Physics at the Munich University of Applied Science in 1991. After several years of industrial employment she received a second MSc. degree in Micro and Nano-Technology from this same University in 2008. Currently she is working in the Sensor Integration group within the Department of Electronics, Communication & Intelligent Systems at Airbus Group Innovations in Munich. Her expertise is in the fields of silicon micromachining and microsensor technology. Her current focus is on solid state chemical sensors, in particular surface ionization gas detectors and optical MID IR absorption detectors. H. Oberpriller (1959) received his degree in physical chemistry at the Munich University of Applied Science in 1981. Since 1981 he is working mainly in the fields of analytical chemistry, failure analysis and cabin air quality. Currently he is part of the Power Generation team within the Department of Energy & Propulsion at Airbus Group Innovations in Munich. A. Ohnesorge (1974), Process engineer, received his diploma in process engineering at the Technical University of Dresden (TUD) in 2001. From 2002–2005 worked as a PhD student at EADS Dornier GmbH in Friedrichshafen with focal points at laser spectroscopy and infrared technology. After his PhD. in 2008 at the TUD he was employed as a researcher in the R&D department of the Solar World AG in Freiberg. Since 2009 he is a research engineer at Airbus Defence and Space/Airbus Innovations in Munich, working on H2 generation and fuel cells. He has experience in chemical and process engineering, laser technology; his current focus is on advanced batteries and other electrochemical storage devices. Kurt-Volker Hechtenberg graduated in Electrical Engineering from the University of Kaiserslautern in 1975. From 1975 to 2005 he was employed as a development engineer at MBB, DASA and EADS. Sensor systems deriving from his development activities included an acoustical bearing system for submarines based on digital
A. Hackner et al. / Sensors and Actuators B 231 (2016) 497–505 correlation, sensors for antitank missiles, sensors for laser-warning receivers, signal processing for a high-dynamic camera, an IR-camera, a stereo-camera, a camera with an artificial retina and a sensor aircraft based maintenance system. From 2005 he is self-employed and still affiliated with EADS, AIRBUS and Lufthansa. Recent activities have been focused on a flight approved security GPS-GSM based tracker module. Gerhard Müller received a PhD. degree in Physics from the University of Heidelberg in 1976. After academic positions at the Max-Planck Institute of Nuclear Physics in Heidelberg and at the University of Dundee in the period 1976–1980, he started
505
a career in industrial research and development working for several employers, including EADS Innovation Works where he served as “Senior Expert Sensor Physics” up to his retirement in 2013. In 2012 he was inducted into the EADS Hall of Fame in the category “Great Inventor”. After his retirement he continues to be a lecturer in the Munich University of Applied Sciences. He is author and co-author of more than 250 papers, including more than 30 invited papers at international conferences, in the fields of semiconductor physics, photovoltaics, thin film and sensor technology as well as sensor applications.