- Email: [email protected]
HYDRO-DAMS SECURITY ASSESSMENT BY VISIBLE AND INFRARED IMAGE FUSION
IFAC Workshop ICPS'07 2007, July 09-11 Cluj-Napoca, Romania
SAMPLE PAGES TO BE FOLLOWED EXACTLY IN PREPARING SCRIPTS
HYDRO-DAMS SECURITY ASSESSMENT BY VISIBLE AND INFRARED IMAGE FUSION
Mihaela Gordan1, Ovidiu Dancea2, Aurel Vlaicu1, Ioan Stoian2, Odysseas Tsatos3, Gabriel Oltean1
1
Technical Univesity of Cluj-Napoca, C. Daicoviciu 15, Cluj-Napoca 400020, Romania 2 S.C. IPA S.A. CIFATT Cluj, Zorilor 15, Cluj-Napoca 400335, Romania 3 ALBIODATA S.A. Information Systems - Consulting, Periferiaki Odos Trikallon, 41335 Larissa, Greece
Abstract: Hydro dams are very important economical and social structures that have a great impact on the population living in surrounding area. Surveillance of dam status consists of a complex process which involves data acquisition and analysis techniques, implying measurements from sensors and transducers and visual inspection. To enhance the visual inspection process made by experts, we propose a computer vision technique that will allow assessment and quantification of water infiltrations inside the dam wall, based on infrared and visible spectrum image analysis and data fusion techniques. Copyright © 2007 IFAC Keywords: infrared imaging, data fusion, dam monitoring, image processing
1. INTRODUCTION
sections, and relatively large horizontal sliding movements of concrete blocks. However, leakage may develop, and because of hydrostatic pressure water losses may be a concern. Also, pouring water is eroding the concrete material, the crack may further enlarge, leading to a chain reaction, and becoming a serious threat to dam safety. Also, some of the cracks formed in certain areas could remain undetected by visual inspection, the area may be covered by sediments and any leakage caused by cracks penetrating the grout curtain may eventually be plugged by the sediments. Cracks may only be detected if they intersect with the foundation gallery (Wieland, 2005).
Dam’s behaviour surveillance and monitoring are the main issues that ensure their security. Controlling and monitoring the dam’s state leads to determination and implementation of different measures in order to avoid its failure. Surveillance of dam behaviour is a complex process which involves data acquisition and analysis techniques, implying both measurements from sensors and transducers placed in the dam body and visual inspection. Using information and automation systems, the results are accurate, can be rapidly processed, and data can be passed to the alarm system when the limits are overdue. The safety of a dam can be improved, and its life period could be increased if there is a carefully implemented monitoring scheme. A key role in this process is played by visual inspection of the dam structure, reservoir, and other accessories. Observing the extent of the cracks, assessing water infiltration, basement movements or other dysfunctions, is vital in dam’s security and represents the object of a periodical visual supervision done by human experts. Cracks are an accepted fact in concrete dams. Their appearance is not always an alarm condition, as they are likely to occur, due to large size of the concrete
Computer vision techniques can be employed to enhance the visual observations made by human experts. The approach is to acquire images and analyse them using digital image processing algorithms. Also, a periodical recording of these images in a database can be very useful to monitor the overall condition of the dam walls in time. Less interest was oriented on incorporating image processing and analysis algorithms to automatically detect, diagnose and predict the behaviour of the dam and the possible faults affecting its structure. The main interest so far was in creating 3-D dam maps, to be further investigated by the human operators. Taking into account the wide variety of computer vision -234-
algorithms currently available, and the possibility of extending the research in this field, the automation of the visual inspection process of the dam, aiming to detect, diagnose and predict possible faults is very likely to be extended. In this paper we investigate the applicability of some of the state-of-the-art algorithms for image analysis in the particular case of infrared and optical images of dam walls. Bimodal analysis of optical and infrared images is a problem still needed to be tackled with. Few such applications are reported, mainly in the fields of surveillance, people counting and tracking, robust skin detection (face detection), forest fires detection, or land mines detection (Bebis et. al., 2006; O’Conaire et. al., 2006). However, for the diagnosis of dams such works are scarce, although infrared imaging is used extensively in assessing temperature loss, or poor isolations in buildings. Thermal images can provide information about the scene being scanned which is not available from a visual image. Little efforts have been made for the integration of complementary information extracted from the two modalities.
2.1 Image acquisition and plot delimitation; visible and infrared plot images registration The image acquisition and plot delimitation is performed in each of the two modalities: visible and infrared, independently. For the visible spectrum image acquisition, a standard color digital camera is used, whereas for the infrared image acquisition we used a thermal camera with temperature coding capabilities (providing a thermal map). The plot delimitation consists in the selection of the same plot in the images from both modalities. This selection can be performed either manually or automatically. The plot is identified as an (approximately) squared area delineated by two successive horizontal and vertical joints. For an automatic selection, a solution could be the one in (Gordan and Georgakis., 2006). In the following we will refer the image of the currently analysed plot in the visible spectrum as “the visible image” and the image of the same plot in the infrared domain will be referred as “the infrared image”. Although these two images are pixel-wise representations of the same physical scene, some difference between them can exist in size, aspect ratio, planar rotation and spatial rotation, considering that we have acquired them independently (however this might not be the case in a future acquisition setup, if we will use fixed cameras with a-priori setup for automatic plot image registration). A simple yet effective registration approach is given in (Yin and Malcolm, 2000). In the particular case of the images used in our experiments, no rotation compensation is needed, thus only scaling and translation compensation should be performed for their alignment. An example of such a registered (visible, infrared) image pair for a plot is given in Figure 2.
The work presented in this paper concerns detection and assessment of water infiltrations within the dam body. Analysing the images of the dam wall acquired in visible spectrum provides information about the areas with calcite deposits. This areas clearly experienced leakages in the past, as the calcite deposit forms after the water dried out. The analysis of the infrared images is based on the principle that during summertime the water (moisture) within the dam body is colder than the dry dam wall. Thus, infiltrations can be detected by infrared cameras, as they appear in different colour than dry areas on the infrared scale. During winter, the process is reversed. We aim to see how the infrared and visible information can be combined to enhance the water infiltration diagnosis from each modality alone. 2. INFILTRATION ASSESSMENT AND QUANTIFICATION The proposed approach is illustrated in Figure 1. Much of the processing runs in parallel, as: visible and infrared spectrum image acquisition, segmentation and image to infiltration severity degree mapping for the quantitative description of water infiltration. On the output of the corresponding stages, we simultaneously have the two water infiltration maps decided by the two modalities, to integrate these decisions by a simple fusion process.
Fig. 2. A pair of images for a hydro-dam wall plot acquired in the two modalities: visible spectrum modality (left) and infrared modality (right)
Fig. 1. Block diagram of the proposed method for infiltration assessment within the dam body
-235-
Ntrn,c of training calcite pixels, represented by their color in the quantized RGB space, denoted § R k , G k , B k · , where k represents the index ¨ q ,trn q ,trn q ,trn ¸ ¹ © of a training sample, k=0,1,..., Ntrn. Having this training set one can build a color normalised histogram in the quantized RGB space corresponding to calcite colors over the set of Ntrn training samples, on 16 ⋅ 16 ⋅ 16 = 4096 bins. Since we build the normalised histogram of the calcite color, each value in this histogram can be directly interpreted as the probability of that color from the quantized RGB color space to represent calcite. Therefore the histogram can be directly used in the image analysis/ segmentation phase to identify the calcite versus noncalcite pixels, once a probabilistic decision threshold to separate the two classes is chosen. According to this interpretation of the calcite color histogram, let us denote the normalised color histogram for the calcite color distribution obtained from the training set by the function p : {0,1,...,15}× {0 ,1,...,15}× {0,1,...,15} → [0;1], with:
2.2 Visible image analysis and segmentation for calcite detection and creation of the water infiltration map from the visible modality The purpose of this step is to identify in the visible spectrum image of the plot the possible regions where calcite is present. Calcite patches are good indicators of rather significant/severe and time persistent water infiltrations; they are most likely to occur as being transported by the water infiltrations from concrete, which happens only in the case of a repetitive water infiltration in a certain area of the hydro-dam. Therefore an accurate identification of the calcite deposits can be very important. The employed algorithm must take into account that, although the most discriminative attribute of the calcite versus the hydro-dam wall is its much whiter color, there is still a significant variability in the particular perceived calcite color, depending on factors as the shadow and regional humidity of the wall (more humidity – calcite appears more grey since is less dry), the thickness of the calcite layers on the wall, etc. Under these circumstances, having also several instances and appearances of calcite deposits at hand, we chose to derive and use a probabilistic model for the calcite areas identification and segmentation, from a large number of training samples, in the (previously roughly quantized) RGB color space. In both training and segmentation phase, we apply a uniform quantization of the RGB space with 16 quantization levels per color component. This is done to obtain a probabilistic model that can reliably cover the entire color space, since the chosen probabilistic model is a non-parametric one, unable to interpolate between the colours present in the training set and accurately classify any other color that can appear in the image to be analysed.
( )
( )
Count c q
p cq =
max
k∈{0 ,1,...,NTrn }
( ),
(1)
Count c kq
where: cq [3×1] represents any color in the quantized
[
RGB color space, in the form c q = Rq
Gq
Bq
]T ;
Count(cq) denotes the number of pixels in the training set having the color cq; Count(cqk) denotes the number of pixels in the training set having the color cqk, where k denotes the training sample. The probabilities given by Equation (1) represent the normalised LUT used as probabilistic calcite model. Then in the image analysis/ color pixel classification phase, we simply classify each pixel from the visible image of the hydro-dam plot, based on its color representation in the quantized RGB color space, as calcite or not. Let us denote the quantized color representation of the visible image of the plot by the matrices IRq,Vis[H×W], IGq,Vis[H×W], IBq,Vis[H×W], and by SVis[H×W] – the the segmented visible image of the plot into calcite and not calcite areas. Then this matrix is found as:
Several probabilistic color models used for colorbased image segmentation / pixel classification can be found in the literature. A good review of these techniques, for the particular case when we aim to identify in an image the pixels/regions with an approximately known color is given in (Vezhnevets et al., 2003). Among these methods, a very simple yet effective approach in our particular application of calcite detection in the visible spectrum plot images is the one based on the non-parametric modelling of the calcite color distribution in the quantized RGB space using a normalised look up table (LUT). Actually, the normalised LUT built for the desired color is simply a normalised color histogram built for all the colors present in the quantized RGB space from a (usually large) set of training data. The training data set is manually obtained from regions of several visible plot images corresponding to calcite deposits. That is, these regions are manually selected so that all the pixels in each region are only calcite deposit pixels. Then all the samples obtained in the training set represent only positive examples (this method does not require negative examples at all – as opposed to other probabilistic color models). Let us consider that after such a training process from several calcite patches, we obtain a number
(
)
1, if p c q (i , j ) > θ SVis ( i , j ) = ® , ¯0, otherwise
[
]
(2)
c q ( i , j ) = I Rq ,Vis (i , j ) I Gq ,Vis (i , j ) I Bq ,Vis (i , j ) . T
θ is the probabilistic threshold (the default value is 0.5, but to minimise the false rejection, we adopt a value of θ=0.4 in this paper).
After only the calcite area was definitely identified in the visible plot image, we quantify the “amount” or severity of calcite deposit. This is proportional to the severity of the water infiltration (as more severe and for much longer time the water infiltration is in this area, the brighter will the pixels in the calcite area be). We consider the severity degree of water infiltration mapped to an intensity range -236-
water infiltrations in our application (the severity of the infiltrations is stronger as the color is closer to violet and dark violet than to red), is illustrated in Figure 3. The red color is considered to be already not cold at all, whereas the rightmost part of the scale (very dark violet) is considered to be the coldest possible. To convert this color (thus 3D) scale into a scalar scale, similarly to the calcite case, we derive a scalar mapping of the “degree of coldness” in the range {0,1,...,255}, with the following meaning: 0 represents the minimum coldness (not cold at all) and we should map the red color to this value; 255 represents the maximum coldness and we map the darkest violet to this value. Since, as we go from red to dark violet, the intensity of the red color component goes from maximum (255) to minimum (0), we can use the negative of this color component of the scale image as the desired scalar mapping of the RGB representation of the “cold temperature” scale, as shown in Figure 3.
{0,1,...,255}, with 0 for the lack of any infiltration to 255 for maximum severity infiltration. Accordingly we can convert the segmented visible modality plot image (with calcite areas identified by the normalised LUT segmentation procedure) into a visible infiltration severity degree map as follows. Let us assume with very good approximation that the luminance component Y of the visible spectrum plot image is a good indicator of the "whiteness" of the calcite – therefore, of the severity of the infiltration also. We denote by IR,Vis [H×W], IG,Vis [H×W] and IB,Vis [H×W] – the original intensity matrices for the three color primaries in the visual spectrum image of the plot. Then the luminance component of the visible spectrum image of the plot can be as well represented in matrix form as IY [H×W], with elements in the range {0,1,...,255}. Let us consider the segmented plot image in the visible domain represented solely as the binary map SVis[H×W] given by Equation (2). Then the water infiltration severity degree map in the visible domain, denoted as the matrix MapVisible[H×W], is given by the following expression: S ( i , j ) ⋅ I Y ( i , j ) ⋅ 255 MapVisible ( i , j ) = Vis , YMax ,Calcite
Then the segmentation process of the infrared image of the plot into cold areas (corresponding to water infiltration) and not cold areas is done pixel wise, based on the pixel color, as follows. First the RGB space is uniformly quantized with only 4 bits per color component, as in the case of the visible images, to compensate for the imperfect acquisition which can make a color in the infrared image be not present exactly on the infrared scale. We denote by IR,IR [H×W], IG,IR [H×W] and IB,IR [H×W] – the original intensity matrices for the three color primaries in the thermal image of the plot, and by IRq,IR [H×W], IGq,IR [H×W] and IBq,IR [H×W] – their uniformly quantized versions. We also gather and denote by SCold the set of the quantized color intensities in the RGB representation of the IR scale corresponding to cold from Figure 3, SCold = {(Cold Rq , ColdGq , Cold Bq )}.
(3)
where YMax,Calcite is the maximum possible intensity for calcite areas. This parameter is derived from a set of training images corresponding to calcite patches on the hydro-dam wall. In our experiments, the soderived value YMax,Calcite was 190 of a scale 0 ... 255 for the luminance (0 – black, 255 – white). The resulting segmentation for the plot image in Figure 2, represented as a binary image (white – calcite, black – no calcite), and the corresponding water infiltration severity degree map, are given in Figure 4. 2.3 Infrared image analysis and segmentation for cold areas identification and creation of the water infiltration map from the visible modality
Then the segmented infrared image of the plot into cold vs. not cold areas, described by SIR [H×W], is given by:
The purpose of the processing done in this step is to provide complementary information regarding the water infiltrations in the plot, based on the principle that, at least in the spring/summer, when the ambient temperature is rather high, the areas of the plot with water infiltrations appear colder in the plot’s thermal map. The more significant the water infiltration is, the colder is the local part of the plot, thus the lower the temperature on the plot’s thermal map. However we can expect that in such areas, little evidence of calcite will be identified in the visible image, since if a particular area is wet, the calcite (which is white when is dried) will appear less white/more grey. Thus the two information sources can favourably complement each other.
(
)
* 1, if ∃ Cold *Rq , Cold Gq , Cold *Bq ∈ SCold ° °°so that Cold * , Cold * , Cold * ≡ Rq Gq Bq , (4) S IR ( i , j ) = ® °I ( i , j ), I Gq ,IR ( i , j ), I Bq ,IR ( i , j ) ° Rq ,IR °¯0 , otherwise
(
(
)
)
Similarly to the creation of the water infiltration severity degree map in the visible domain, we build the water infiltration severity degree map in the infrared domain. However in the infrared case, we consider as infiltration severity degree indicator – the negative of the red color component in each pixel position previously classified as cold, as discussed earlier. Denoting by MapInfrared[H×W] the severity degree map of the water infiltration in the infrared modality, represented in the range {0,1,...,255} as in the visible modality case, we obtain this matrix as:
Since in the case of the thermal maps we always have available exactly the color-temperature conversion scale, we can use this scale and a-priori knowledge on what “cold temperature corresponding color” means to obtain the accurate identification of the water infiltration areas. An example of the selected scale portion, as considered to represent
Map Infrared ( i , j ) = S IR ( i , j ) ⋅ (255 − I R ,IR ( i , j )). (5)
-237-
An example of the resulting water infiltration severity degree map after bimodal image fusion, for the plot presented in Fig. 2, is given in Fig. 6. Then this overall decision map can be used to compute quantitative descriptors of the water infiltration amount and local severity on the plot. Examples of such simple quantitative descriptors are the ones computed in our paper: a) the percentage of the water infiltration area from the total plot area; this is computed as:
Fig. 3. The cold temperature part of the infrared scale: original (left); its scalar mapping (right) An example of segmentation result in "cold" and "not cold" areas for the infrared plot image given in Figure 2, where the segmented image is presented as a binary image (black – "not cold", white – "cold"), and its associated water infiltration severity degree map, are given in Figure 5.
WaterInfiltArea =
2.4 The bimodal fusion of visible and infrared water infiltration severity information for global infiltration severity assessment
WaterInfiltPixels ⋅100[%], H ⋅W
(7)
where WaterInfiltPixels is the number of pixels (i,j) in the plot for which WaterInfiltMap(i,j)>0, i=0,1,…,H-1, j=0,1,….,W-1. b) the maximum local severity degree of water infiltration, assessed as the accumulated severity of the infiltration reported to the total area exhibiting infiltration:
In this final processing step, we use the two individual information sources already provided by the independent image processing and analysis stages: the visible and infrared image processing, to obtain the overall assessment and quantification of the water infiltration amount in the current plot.
H −1 W −1
¦ ¦ InfMap(i, j )
Several fusion schemes are available, varying from very simple (pixel-based) to complex ones, to perform the information integration from two or more modalities; the most used in particular for visible and infrared bimodal information fusion can be found in (Yin and Malcolm, 2000, O’Conaire et. al, 2006). Among these, one of the simplest schemes is by weighted averaging of the decisions given by each modality alone at pixel level, provided that the visible and infrared image registration was previously performed (as described in our algorithm). Let us denote the decision about the plausibility of presence of a certain event in the spatial position (i,j) in the visible spectrum modality by dVis(i,j) and the decision about the plausibility of presence of the same event in the spatial position (i,j) in the infrared modality by dIR(i,j). We also consider the weights (confidences) assigned to each modality denoted by wVis and wIR, chosen to satisfy the constraints: wVis ∈ (0;1); w IR ∈ (0;1); wVis + w IR = 1. The confidences wVis and wIR assigned to each modality are derived either based on expert’s knowledge or from a set of training data. The latter is however more difficult to obtain for the particular application addressed here. Therefore in this case, we use expert knowledge/intuition about the relative significance of each modality in assessing the severity of the water infiltration. The presence of calcite shows longer duration water infiltration in the plot, thus its weight should be higher than the infrared’s information source. We chose as confidence values in our application: wVis=0.65 and wIR=0.35. As information sources to be aggregated, we use the individual water infiltration severity degrees maps, MapVis [H×W] and MapIR [H×W]. The overall water infiltration severity degree map, represented as an intensity image in the range {0,1,…,255}, with 255 – maximum infiltration severity, is obtained as InfMap[H×W],
InfiltSev =
i =0 j =0
WaterInfiltPixels
⋅
100 [%]. 255
(8)
3. EXPERIMENTAL RESULTS In order to test our method we used images acquired from TarniĠa dam, near Cluj-Napoca. The dam was constructed in 1974 on Someúul Cald River. It has 97 meters in height, and 237 meters in length. The reservoir has a capacity of 74 millions cubic meters, and the lake area is 220 ha. We selected 5 pairs of plots acquired in both modalities (visible and infrared) and registered them with the strategy described in 2.1., to assess the functionality of our algorithm. We developed a Windows application to implement the proposed algorithms and verify its performance. A ground truth for visible image segmentation into calcite areas and non-calcite areas can be easily obtained, and the same – a ground truth for pixel classification into cold areas for the infrared images. Thus we can assess the functionality of these processing stages very accurately. However, this is not the case for the assessment of water infiltrations severity, which in general can only be subjectively estimated by human observers. Therefore we can only roughly compare the results provided by our algorithm, converted to subjective scales, to subjective (human) evaluation of the water infiltrations based on the visible and infrared plot image evaluation. These comparative results for the 5 pairs of plots are presented in Table 1. Examining the results in Table 1, one can see the good functionality of the proposed algorithm. Thus, although this novel application is now at its beginning, we consider it promising. The only difference from the human expert’s opinion is in the 4th line in Table 1, for the plot exhibiting water infiltration in a very small area, in respect to the local severity of the water
InfMap(i, j ) = wVis ⋅ MapVis (i, j ) + wIR ⋅ MapIR (i, j ). (6) -238-
infiltration: although the numerical results show a large local value, the human expert identifies it as not significant, and this could be explained by the overall assessment done by the human expert, with almost no attention to local details when the water infiltration region size is not significant.
Table 1 Quantitative results of our algorithm vs. subjective human expert’s opinion
The segmentation results, both for the visible and infrared plot images show in all cases good accuracy. An example of visible and infrared modality image segmentation and the water infiltration severity degree map in these modalities for the plot given in Figure 2 are presented in Figure 4 (visible) and Figure 5 (infrared). One can see based on the resulting infiltration severity degrees maps that the two information sources both confirm water infiltration in only a small portion of the plot (in the calcite area on the left side of the visible image) whereas in a large portion of the plot, they give complementary water infiltration information. The resulting water infiltration severity degree map after the bimodal image fusion is shown in Figure 6.
Plot pair
WaterInfiltArea
InfiltSev
#1 #2 #3 #4 #5
32.05% 23.63% 24.46% 2.4% 43.7%
81% 58% 64.7% 72% 78%
Infiltration amount (subjective) Medium/Large Medium Medium/Small Almost none Large
Infiltration severity (subjective) Severe Moderate Moderate Reduced Severe
4. CONCLUSIONS We proposed a novel application of bimodal fusion between visible and infrared imaging modalities (which is a promising strategy in many application areas), namely in the concrete hydro dams monitoring and diagnosis in respect to the water infiltration in the concrete walls. Although for now we employ one of the most simple fusion schemes, we can see how the use of the two modalities can lead to better results than the analysis of each imaging modality alone. Also, the computer implementation of the joint analysis of visible and infrared images has the advantage of providing numerical estimates of the extension of the water infiltrations and severity of the water infiltrations in the plots, reducing the risk of human observer subjectivity and image display quality. ACKNOWLEDGEMENTS The work described in this paper was performed in the frame of project no. 705/2006 in the frame of CEEX research and development programme, financed by Romanian Government.
Fig. 4. Plot image segmentation result in the visible domain: calcite vs. not calcite segmented image (left) and infiltration severity degree map (right)
REFERENCES Bebis, G.N., Gyaourova, A., Singh, S. and Pavlidis, I. (2006) Face recognition by fusing thermal infrared and visible imagery. In Image and Vision Computing No. 7, 2006, pp. 727-742 Gordan, M. and Georgakis, A. (2006) A novel fuzzy edge detection and classification scheme to aid hydro-dams surface examination, Proc. of Swedish Society for Automated Image Analysis (SSBA'06), pp. 121-124 O’Conaire C, O'Connor N, Cooke E and Smeaton A.F.. (2006) Comparison of Fusion Methods for Thermo-Visual Surveillance Tracking, In Proc. FUSION 2006 - 9th International Conference on Information Fusion, pp. 1-7 Vezhnevets, V., Sazonov, V. , and Andreeva, A. (2003) A survey on pixel-based skin color detection techniques, Proc. GRAPHICON03, Moscow, Russia, pp. 85-92 Wieland, M. (2005) Stress Management. In International water power & dam construction, Wilmington Media Ltd. Yin, Zhang and Malcolm, A. A (2000) Thermal and Visual Image Processing and Fusion - SIMTech Technical Report AT/00/016/MVS
Fig. 5. Plot image segmentation result in the infrared domain: cold vs. not cold segmented image (left) and infiltration severity degree map (right)
Fig. 6. Resulting water infiltration severity degree map after the bimodal image fusion -239-
SAMPLE PAGES TO BE FOLLOWED EXACTLY IN PREPARING SCRIPTS
HYDRO-DAMS SECURITY ASSESSMENT BY VISIBLE AND INFRARED IMAGE FUSION
Mihaela Gordan1, Ovidiu Dancea2, Aurel Vlaicu1, Ioan Stoian2, Odysseas Tsatos3, Gabriel Oltean1
1
Technical Univesity of Cluj-Napoca, C. Daicoviciu 15, Cluj-Napoca 400020, Romania 2 S.C. IPA S.A. CIFATT Cluj, Zorilor 15, Cluj-Napoca 400335, Romania 3 ALBIODATA S.A. Information Systems - Consulting, Periferiaki Odos Trikallon, 41335 Larissa, Greece
Abstract: Hydro dams are very important economical and social structures that have a great impact on the population living in surrounding area. Surveillance of dam status consists of a complex process which involves data acquisition and analysis techniques, implying measurements from sensors and transducers and visual inspection. To enhance the visual inspection process made by experts, we propose a computer vision technique that will allow assessment and quantification of water infiltrations inside the dam wall, based on infrared and visible spectrum image analysis and data fusion techniques. Copyright © 2007 IFAC Keywords: infrared imaging, data fusion, dam monitoring, image processing
1. INTRODUCTION
sections, and relatively large horizontal sliding movements of concrete blocks. However, leakage may develop, and because of hydrostatic pressure water losses may be a concern. Also, pouring water is eroding the concrete material, the crack may further enlarge, leading to a chain reaction, and becoming a serious threat to dam safety. Also, some of the cracks formed in certain areas could remain undetected by visual inspection, the area may be covered by sediments and any leakage caused by cracks penetrating the grout curtain may eventually be plugged by the sediments. Cracks may only be detected if they intersect with the foundation gallery (Wieland, 2005).
Dam’s behaviour surveillance and monitoring are the main issues that ensure their security. Controlling and monitoring the dam’s state leads to determination and implementation of different measures in order to avoid its failure. Surveillance of dam behaviour is a complex process which involves data acquisition and analysis techniques, implying both measurements from sensors and transducers placed in the dam body and visual inspection. Using information and automation systems, the results are accurate, can be rapidly processed, and data can be passed to the alarm system when the limits are overdue. The safety of a dam can be improved, and its life period could be increased if there is a carefully implemented monitoring scheme. A key role in this process is played by visual inspection of the dam structure, reservoir, and other accessories. Observing the extent of the cracks, assessing water infiltration, basement movements or other dysfunctions, is vital in dam’s security and represents the object of a periodical visual supervision done by human experts. Cracks are an accepted fact in concrete dams. Their appearance is not always an alarm condition, as they are likely to occur, due to large size of the concrete
Computer vision techniques can be employed to enhance the visual observations made by human experts. The approach is to acquire images and analyse them using digital image processing algorithms. Also, a periodical recording of these images in a database can be very useful to monitor the overall condition of the dam walls in time. Less interest was oriented on incorporating image processing and analysis algorithms to automatically detect, diagnose and predict the behaviour of the dam and the possible faults affecting its structure. The main interest so far was in creating 3-D dam maps, to be further investigated by the human operators. Taking into account the wide variety of computer vision -234-
algorithms currently available, and the possibility of extending the research in this field, the automation of the visual inspection process of the dam, aiming to detect, diagnose and predict possible faults is very likely to be extended. In this paper we investigate the applicability of some of the state-of-the-art algorithms for image analysis in the particular case of infrared and optical images of dam walls. Bimodal analysis of optical and infrared images is a problem still needed to be tackled with. Few such applications are reported, mainly in the fields of surveillance, people counting and tracking, robust skin detection (face detection), forest fires detection, or land mines detection (Bebis et. al., 2006; O’Conaire et. al., 2006). However, for the diagnosis of dams such works are scarce, although infrared imaging is used extensively in assessing temperature loss, or poor isolations in buildings. Thermal images can provide information about the scene being scanned which is not available from a visual image. Little efforts have been made for the integration of complementary information extracted from the two modalities.
2.1 Image acquisition and plot delimitation; visible and infrared plot images registration The image acquisition and plot delimitation is performed in each of the two modalities: visible and infrared, independently. For the visible spectrum image acquisition, a standard color digital camera is used, whereas for the infrared image acquisition we used a thermal camera with temperature coding capabilities (providing a thermal map). The plot delimitation consists in the selection of the same plot in the images from both modalities. This selection can be performed either manually or automatically. The plot is identified as an (approximately) squared area delineated by two successive horizontal and vertical joints. For an automatic selection, a solution could be the one in (Gordan and Georgakis., 2006). In the following we will refer the image of the currently analysed plot in the visible spectrum as “the visible image” and the image of the same plot in the infrared domain will be referred as “the infrared image”. Although these two images are pixel-wise representations of the same physical scene, some difference between them can exist in size, aspect ratio, planar rotation and spatial rotation, considering that we have acquired them independently (however this might not be the case in a future acquisition setup, if we will use fixed cameras with a-priori setup for automatic plot image registration). A simple yet effective registration approach is given in (Yin and Malcolm, 2000). In the particular case of the images used in our experiments, no rotation compensation is needed, thus only scaling and translation compensation should be performed for their alignment. An example of such a registered (visible, infrared) image pair for a plot is given in Figure 2.
The work presented in this paper concerns detection and assessment of water infiltrations within the dam body. Analysing the images of the dam wall acquired in visible spectrum provides information about the areas with calcite deposits. This areas clearly experienced leakages in the past, as the calcite deposit forms after the water dried out. The analysis of the infrared images is based on the principle that during summertime the water (moisture) within the dam body is colder than the dry dam wall. Thus, infiltrations can be detected by infrared cameras, as they appear in different colour than dry areas on the infrared scale. During winter, the process is reversed. We aim to see how the infrared and visible information can be combined to enhance the water infiltration diagnosis from each modality alone. 2. INFILTRATION ASSESSMENT AND QUANTIFICATION The proposed approach is illustrated in Figure 1. Much of the processing runs in parallel, as: visible and infrared spectrum image acquisition, segmentation and image to infiltration severity degree mapping for the quantitative description of water infiltration. On the output of the corresponding stages, we simultaneously have the two water infiltration maps decided by the two modalities, to integrate these decisions by a simple fusion process.
Fig. 2. A pair of images for a hydro-dam wall plot acquired in the two modalities: visible spectrum modality (left) and infrared modality (right)
Fig. 1. Block diagram of the proposed method for infiltration assessment within the dam body
-235-
Ntrn,c of training calcite pixels, represented by their color in the quantized RGB space, denoted § R k , G k , B k · , where k represents the index ¨ q ,trn q ,trn q ,trn ¸ ¹ © of a training sample, k=0,1,..., Ntrn. Having this training set one can build a color normalised histogram in the quantized RGB space corresponding to calcite colors over the set of Ntrn training samples, on 16 ⋅ 16 ⋅ 16 = 4096 bins. Since we build the normalised histogram of the calcite color, each value in this histogram can be directly interpreted as the probability of that color from the quantized RGB color space to represent calcite. Therefore the histogram can be directly used in the image analysis/ segmentation phase to identify the calcite versus noncalcite pixels, once a probabilistic decision threshold to separate the two classes is chosen. According to this interpretation of the calcite color histogram, let us denote the normalised color histogram for the calcite color distribution obtained from the training set by the function p : {0,1,...,15}× {0 ,1,...,15}× {0,1,...,15} → [0;1], with:
2.2 Visible image analysis and segmentation for calcite detection and creation of the water infiltration map from the visible modality The purpose of this step is to identify in the visible spectrum image of the plot the possible regions where calcite is present. Calcite patches are good indicators of rather significant/severe and time persistent water infiltrations; they are most likely to occur as being transported by the water infiltrations from concrete, which happens only in the case of a repetitive water infiltration in a certain area of the hydro-dam. Therefore an accurate identification of the calcite deposits can be very important. The employed algorithm must take into account that, although the most discriminative attribute of the calcite versus the hydro-dam wall is its much whiter color, there is still a significant variability in the particular perceived calcite color, depending on factors as the shadow and regional humidity of the wall (more humidity – calcite appears more grey since is less dry), the thickness of the calcite layers on the wall, etc. Under these circumstances, having also several instances and appearances of calcite deposits at hand, we chose to derive and use a probabilistic model for the calcite areas identification and segmentation, from a large number of training samples, in the (previously roughly quantized) RGB color space. In both training and segmentation phase, we apply a uniform quantization of the RGB space with 16 quantization levels per color component. This is done to obtain a probabilistic model that can reliably cover the entire color space, since the chosen probabilistic model is a non-parametric one, unable to interpolate between the colours present in the training set and accurately classify any other color that can appear in the image to be analysed.
( )
( )
Count c q
p cq =
max
k∈{0 ,1,...,NTrn }
( ),
(1)
Count c kq
where: cq [3×1] represents any color in the quantized
[
RGB color space, in the form c q = Rq
Gq
Bq
]T ;
Count(cq) denotes the number of pixels in the training set having the color cq; Count(cqk) denotes the number of pixels in the training set having the color cqk, where k denotes the training sample. The probabilities given by Equation (1) represent the normalised LUT used as probabilistic calcite model. Then in the image analysis/ color pixel classification phase, we simply classify each pixel from the visible image of the hydro-dam plot, based on its color representation in the quantized RGB color space, as calcite or not. Let us denote the quantized color representation of the visible image of the plot by the matrices IRq,Vis[H×W], IGq,Vis[H×W], IBq,Vis[H×W], and by SVis[H×W] – the the segmented visible image of the plot into calcite and not calcite areas. Then this matrix is found as:
Several probabilistic color models used for colorbased image segmentation / pixel classification can be found in the literature. A good review of these techniques, for the particular case when we aim to identify in an image the pixels/regions with an approximately known color is given in (Vezhnevets et al., 2003). Among these methods, a very simple yet effective approach in our particular application of calcite detection in the visible spectrum plot images is the one based on the non-parametric modelling of the calcite color distribution in the quantized RGB space using a normalised look up table (LUT). Actually, the normalised LUT built for the desired color is simply a normalised color histogram built for all the colors present in the quantized RGB space from a (usually large) set of training data. The training data set is manually obtained from regions of several visible plot images corresponding to calcite deposits. That is, these regions are manually selected so that all the pixels in each region are only calcite deposit pixels. Then all the samples obtained in the training set represent only positive examples (this method does not require negative examples at all – as opposed to other probabilistic color models). Let us consider that after such a training process from several calcite patches, we obtain a number
(
)
1, if p c q (i , j ) > θ SVis ( i , j ) = ® , ¯0, otherwise
[
]
(2)
c q ( i , j ) = I Rq ,Vis (i , j ) I Gq ,Vis (i , j ) I Bq ,Vis (i , j ) . T
θ is the probabilistic threshold (the default value is 0.5, but to minimise the false rejection, we adopt a value of θ=0.4 in this paper).
After only the calcite area was definitely identified in the visible plot image, we quantify the “amount” or severity of calcite deposit. This is proportional to the severity of the water infiltration (as more severe and for much longer time the water infiltration is in this area, the brighter will the pixels in the calcite area be). We consider the severity degree of water infiltration mapped to an intensity range -236-
water infiltrations in our application (the severity of the infiltrations is stronger as the color is closer to violet and dark violet than to red), is illustrated in Figure 3. The red color is considered to be already not cold at all, whereas the rightmost part of the scale (very dark violet) is considered to be the coldest possible. To convert this color (thus 3D) scale into a scalar scale, similarly to the calcite case, we derive a scalar mapping of the “degree of coldness” in the range {0,1,...,255}, with the following meaning: 0 represents the minimum coldness (not cold at all) and we should map the red color to this value; 255 represents the maximum coldness and we map the darkest violet to this value. Since, as we go from red to dark violet, the intensity of the red color component goes from maximum (255) to minimum (0), we can use the negative of this color component of the scale image as the desired scalar mapping of the RGB representation of the “cold temperature” scale, as shown in Figure 3.
{0,1,...,255}, with 0 for the lack of any infiltration to 255 for maximum severity infiltration. Accordingly we can convert the segmented visible modality plot image (with calcite areas identified by the normalised LUT segmentation procedure) into a visible infiltration severity degree map as follows. Let us assume with very good approximation that the luminance component Y of the visible spectrum plot image is a good indicator of the "whiteness" of the calcite – therefore, of the severity of the infiltration also. We denote by IR,Vis [H×W], IG,Vis [H×W] and IB,Vis [H×W] – the original intensity matrices for the three color primaries in the visual spectrum image of the plot. Then the luminance component of the visible spectrum image of the plot can be as well represented in matrix form as IY [H×W], with elements in the range {0,1,...,255}. Let us consider the segmented plot image in the visible domain represented solely as the binary map SVis[H×W] given by Equation (2). Then the water infiltration severity degree map in the visible domain, denoted as the matrix MapVisible[H×W], is given by the following expression: S ( i , j ) ⋅ I Y ( i , j ) ⋅ 255 MapVisible ( i , j ) = Vis , YMax ,Calcite
Then the segmentation process of the infrared image of the plot into cold areas (corresponding to water infiltration) and not cold areas is done pixel wise, based on the pixel color, as follows. First the RGB space is uniformly quantized with only 4 bits per color component, as in the case of the visible images, to compensate for the imperfect acquisition which can make a color in the infrared image be not present exactly on the infrared scale. We denote by IR,IR [H×W], IG,IR [H×W] and IB,IR [H×W] – the original intensity matrices for the three color primaries in the thermal image of the plot, and by IRq,IR [H×W], IGq,IR [H×W] and IBq,IR [H×W] – their uniformly quantized versions. We also gather and denote by SCold the set of the quantized color intensities in the RGB representation of the IR scale corresponding to cold from Figure 3, SCold = {(Cold Rq , ColdGq , Cold Bq )}.
(3)
where YMax,Calcite is the maximum possible intensity for calcite areas. This parameter is derived from a set of training images corresponding to calcite patches on the hydro-dam wall. In our experiments, the soderived value YMax,Calcite was 190 of a scale 0 ... 255 for the luminance (0 – black, 255 – white). The resulting segmentation for the plot image in Figure 2, represented as a binary image (white – calcite, black – no calcite), and the corresponding water infiltration severity degree map, are given in Figure 4. 2.3 Infrared image analysis and segmentation for cold areas identification and creation of the water infiltration map from the visible modality
Then the segmented infrared image of the plot into cold vs. not cold areas, described by SIR [H×W], is given by:
The purpose of the processing done in this step is to provide complementary information regarding the water infiltrations in the plot, based on the principle that, at least in the spring/summer, when the ambient temperature is rather high, the areas of the plot with water infiltrations appear colder in the plot’s thermal map. The more significant the water infiltration is, the colder is the local part of the plot, thus the lower the temperature on the plot’s thermal map. However we can expect that in such areas, little evidence of calcite will be identified in the visible image, since if a particular area is wet, the calcite (which is white when is dried) will appear less white/more grey. Thus the two information sources can favourably complement each other.
(
)
* 1, if ∃ Cold *Rq , Cold Gq , Cold *Bq ∈ SCold ° °°so that Cold * , Cold * , Cold * ≡ Rq Gq Bq , (4) S IR ( i , j ) = ® °I ( i , j ), I Gq ,IR ( i , j ), I Bq ,IR ( i , j ) ° Rq ,IR °¯0 , otherwise
(
(
)
)
Similarly to the creation of the water infiltration severity degree map in the visible domain, we build the water infiltration severity degree map in the infrared domain. However in the infrared case, we consider as infiltration severity degree indicator – the negative of the red color component in each pixel position previously classified as cold, as discussed earlier. Denoting by MapInfrared[H×W] the severity degree map of the water infiltration in the infrared modality, represented in the range {0,1,...,255} as in the visible modality case, we obtain this matrix as:
Since in the case of the thermal maps we always have available exactly the color-temperature conversion scale, we can use this scale and a-priori knowledge on what “cold temperature corresponding color” means to obtain the accurate identification of the water infiltration areas. An example of the selected scale portion, as considered to represent
Map Infrared ( i , j ) = S IR ( i , j ) ⋅ (255 − I R ,IR ( i , j )). (5)
-237-
An example of the resulting water infiltration severity degree map after bimodal image fusion, for the plot presented in Fig. 2, is given in Fig. 6. Then this overall decision map can be used to compute quantitative descriptors of the water infiltration amount and local severity on the plot. Examples of such simple quantitative descriptors are the ones computed in our paper: a) the percentage of the water infiltration area from the total plot area; this is computed as:
Fig. 3. The cold temperature part of the infrared scale: original (left); its scalar mapping (right) An example of segmentation result in "cold" and "not cold" areas for the infrared plot image given in Figure 2, where the segmented image is presented as a binary image (black – "not cold", white – "cold"), and its associated water infiltration severity degree map, are given in Figure 5.
WaterInfiltArea =
2.4 The bimodal fusion of visible and infrared water infiltration severity information for global infiltration severity assessment
WaterInfiltPixels ⋅100[%], H ⋅W
(7)
where WaterInfiltPixels is the number of pixels (i,j) in the plot for which WaterInfiltMap(i,j)>0, i=0,1,…,H-1, j=0,1,….,W-1. b) the maximum local severity degree of water infiltration, assessed as the accumulated severity of the infiltration reported to the total area exhibiting infiltration:
In this final processing step, we use the two individual information sources already provided by the independent image processing and analysis stages: the visible and infrared image processing, to obtain the overall assessment and quantification of the water infiltration amount in the current plot.
H −1 W −1
¦ ¦ InfMap(i, j )
Several fusion schemes are available, varying from very simple (pixel-based) to complex ones, to perform the information integration from two or more modalities; the most used in particular for visible and infrared bimodal information fusion can be found in (Yin and Malcolm, 2000, O’Conaire et. al, 2006). Among these, one of the simplest schemes is by weighted averaging of the decisions given by each modality alone at pixel level, provided that the visible and infrared image registration was previously performed (as described in our algorithm). Let us denote the decision about the plausibility of presence of a certain event in the spatial position (i,j) in the visible spectrum modality by dVis(i,j) and the decision about the plausibility of presence of the same event in the spatial position (i,j) in the infrared modality by dIR(i,j). We also consider the weights (confidences) assigned to each modality denoted by wVis and wIR, chosen to satisfy the constraints: wVis ∈ (0;1); w IR ∈ (0;1); wVis + w IR = 1. The confidences wVis and wIR assigned to each modality are derived either based on expert’s knowledge or from a set of training data. The latter is however more difficult to obtain for the particular application addressed here. Therefore in this case, we use expert knowledge/intuition about the relative significance of each modality in assessing the severity of the water infiltration. The presence of calcite shows longer duration water infiltration in the plot, thus its weight should be higher than the infrared’s information source. We chose as confidence values in our application: wVis=0.65 and wIR=0.35. As information sources to be aggregated, we use the individual water infiltration severity degrees maps, MapVis [H×W] and MapIR [H×W]. The overall water infiltration severity degree map, represented as an intensity image in the range {0,1,…,255}, with 255 – maximum infiltration severity, is obtained as InfMap[H×W],
InfiltSev =
i =0 j =0
WaterInfiltPixels
⋅
100 [%]. 255
(8)
3. EXPERIMENTAL RESULTS In order to test our method we used images acquired from TarniĠa dam, near Cluj-Napoca. The dam was constructed in 1974 on Someúul Cald River. It has 97 meters in height, and 237 meters in length. The reservoir has a capacity of 74 millions cubic meters, and the lake area is 220 ha. We selected 5 pairs of plots acquired in both modalities (visible and infrared) and registered them with the strategy described in 2.1., to assess the functionality of our algorithm. We developed a Windows application to implement the proposed algorithms and verify its performance. A ground truth for visible image segmentation into calcite areas and non-calcite areas can be easily obtained, and the same – a ground truth for pixel classification into cold areas for the infrared images. Thus we can assess the functionality of these processing stages very accurately. However, this is not the case for the assessment of water infiltrations severity, which in general can only be subjectively estimated by human observers. Therefore we can only roughly compare the results provided by our algorithm, converted to subjective scales, to subjective (human) evaluation of the water infiltrations based on the visible and infrared plot image evaluation. These comparative results for the 5 pairs of plots are presented in Table 1. Examining the results in Table 1, one can see the good functionality of the proposed algorithm. Thus, although this novel application is now at its beginning, we consider it promising. The only difference from the human expert’s opinion is in the 4th line in Table 1, for the plot exhibiting water infiltration in a very small area, in respect to the local severity of the water
InfMap(i, j ) = wVis ⋅ MapVis (i, j ) + wIR ⋅ MapIR (i, j ). (6) -238-
infiltration: although the numerical results show a large local value, the human expert identifies it as not significant, and this could be explained by the overall assessment done by the human expert, with almost no attention to local details when the water infiltration region size is not significant.
Table 1 Quantitative results of our algorithm vs. subjective human expert’s opinion
The segmentation results, both for the visible and infrared plot images show in all cases good accuracy. An example of visible and infrared modality image segmentation and the water infiltration severity degree map in these modalities for the plot given in Figure 2 are presented in Figure 4 (visible) and Figure 5 (infrared). One can see based on the resulting infiltration severity degrees maps that the two information sources both confirm water infiltration in only a small portion of the plot (in the calcite area on the left side of the visible image) whereas in a large portion of the plot, they give complementary water infiltration information. The resulting water infiltration severity degree map after the bimodal image fusion is shown in Figure 6.
Plot pair
WaterInfiltArea
InfiltSev
#1 #2 #3 #4 #5
32.05% 23.63% 24.46% 2.4% 43.7%
81% 58% 64.7% 72% 78%
Infiltration amount (subjective) Medium/Large Medium Medium/Small Almost none Large
Infiltration severity (subjective) Severe Moderate Moderate Reduced Severe
4. CONCLUSIONS We proposed a novel application of bimodal fusion between visible and infrared imaging modalities (which is a promising strategy in many application areas), namely in the concrete hydro dams monitoring and diagnosis in respect to the water infiltration in the concrete walls. Although for now we employ one of the most simple fusion schemes, we can see how the use of the two modalities can lead to better results than the analysis of each imaging modality alone. Also, the computer implementation of the joint analysis of visible and infrared images has the advantage of providing numerical estimates of the extension of the water infiltrations and severity of the water infiltrations in the plots, reducing the risk of human observer subjectivity and image display quality. ACKNOWLEDGEMENTS The work described in this paper was performed in the frame of project no. 705/2006 in the frame of CEEX research and development programme, financed by Romanian Government.
Fig. 4. Plot image segmentation result in the visible domain: calcite vs. not calcite segmented image (left) and infiltration severity degree map (right)
REFERENCES Bebis, G.N., Gyaourova, A., Singh, S. and Pavlidis, I. (2006) Face recognition by fusing thermal infrared and visible imagery. In Image and Vision Computing No. 7, 2006, pp. 727-742 Gordan, M. and Georgakis, A. (2006) A novel fuzzy edge detection and classification scheme to aid hydro-dams surface examination, Proc. of Swedish Society for Automated Image Analysis (SSBA'06), pp. 121-124 O’Conaire C, O'Connor N, Cooke E and Smeaton A.F.. (2006) Comparison of Fusion Methods for Thermo-Visual Surveillance Tracking, In Proc. FUSION 2006 - 9th International Conference on Information Fusion, pp. 1-7 Vezhnevets, V., Sazonov, V. , and Andreeva, A. (2003) A survey on pixel-based skin color detection techniques, Proc. GRAPHICON03, Moscow, Russia, pp. 85-92 Wieland, M. (2005) Stress Management. In International water power & dam construction, Wilmington Media Ltd. Yin, Zhang and Malcolm, A. A (2000) Thermal and Visual Image Processing and Fusion - SIMTech Technical Report AT/00/016/MVS
Fig. 5. Plot image segmentation result in the infrared domain: cold vs. not cold segmented image (left) and infiltration severity degree map (right)
Fig. 6. Resulting water infiltration severity degree map after the bimodal image fusion -239-