- Email: [email protected]
Multisensor and multiresolution image fusion using the linear mixing model
3 Multisensor and multiresolution image fusion using the linear mixing model Jan G.P.W. Clevers and Raul Zurita-Milla Wageningen University, Centre for Geo-Information, Wageningen, The Netherlands
In satellite sensor design we observe a trade-off between sensors with a high spatial resolution having only a few spectral bands and a low revisit frequency on the one hand, and sensors with a medium to low spatial resolution having many spectral bands and a high revisit time on the other hand. Many applications require a combination of a high spatial, spectral, and temporal resolution. In this chapter image fusion of a high spatial resolution (Landsat Thematic Mapper) and a high spectral resolution (Envisat MERIS) image based on the linear mixing model is presented. This approach is also known as spatial unmixing or unmixing-based data fusion. An optimisation of the number of classes used to classify the high spatial resolution image and the size of the neighbourhood, for which the unmixing equations are solved, is presented. It is illustrated for a test area in the Netherlands. Results show the feasibility of this approach yielding fused images with the spatial resolution of the high resolution image and with the spectral information from the low spatial resolution image. The quality of the fused images is evaluated using the spectral and spatial ERGAS index. Main advantage of the presented technique based on the linear mixing model is that the fused images do not include the spectral information of the high spatial resolution image in the final result in any way.
3.1 Introduction
Terrestrial vegetation plays an important role in biochemical cycles, like the global carbon cycle. Information on the type of vegetation is important for such studies. This is thematic information referring to the biome type or the land cover type. In addition, once we have the thematic class information, we need quantitative information on vegetation properties. 67
68
Image Fusion: Algorithms and Applications
This quantitative information refers to the so-called continuous fields [1]. In order to quantify the role of vegetation in biochemical cycles, variables describing the area of the green photosynthetic elements (mainly the leaves), the amount of photosynthetically active radiation (PAR) absorbed by the vegetation and the chlorophyll content of the leaves are important [2]. Key variables required for the estimation of, for instance, the primary production of vegetation are the leaf area index (LAI) and the fraction of PAR absorbed by vegetation (FAPAR) [3]. The LAI refers to the one-sided green leaf area per unit ground area. The FAPAR refers to the fraction of the incoming PAR absorbed. PAR (0.4–0.7 µm) is used as energy source in the photosynthesis process, whereby atmospheric CO2 and H2 O from the soil are converted into carbohydrates, enabling growth of the vegetation. The estimation of both LAI and FAPAR from spectral measurements also depends on other vegetation parameters, like leaf properties and canopy structure. Therefore, vegetation class-dependent relationships are often used. In general, to describe the relationship between spectral measurements and biophysical and chemical variables of vegetation both statistical and physical approaches have been used. As an example of statistical methods, numerous vegetation indices (VIs) have been developed for estimating variables like biomass and LAI for a range of vegetation types [4–9]. Physical-based methods often use radiative transfer models describing the interaction of radiation with the plant canopy based on physical principles. Subsequently, model inversion is used for estimating biophysical and chemical properties of the canopy [10–12]. Depending on the application, these techniques can be applied from local to global scales [13]. On the one hand, datasets derived from coarse resolution sensors provide information on land cover and vegetation properties globally. In the US, global land cover products have been derived using time series with 1 km data obtained from the National Oceanic and Atmospheric Administration’s (NOAA) Advanced Very High Resolution Radiometer (AVHRR) [14,15]. In addition, a 1 km land cover map has been compiled using a time series of data from the Moderate Resolution Imaging Spectroradiometer (MODIS) on the Terra platform [16]. An example of continuous fields is the MODIS LAI product, where biome specific algorithms are used for obtaining global maps of LAI [2]. Photosynthetic activity of vegetation canopies can be shown by the fraction of APAR, which can be estimated from satellite observation [17]. Using time-series of satellite data it is shown, for instance, that the length of the growing season is increasing at the northern hemisphere, which may be caused by global warming [2]. This coarse scale imagery is limiting the use for monitoring purposes due to the finer scale at which most land cover changes take place [18]. Still these sub-kilometre scale changes are critical for monitoring changes in, e.g., sinks and sources of the carbon cycle. On the other hand, many detailed studies at the regional and landscape scale at spatial resolutions between 10 and 30 m have been performed using the Thematic Mapper (TM) aboard the Landsat satellites [19], the multispectral imager (denoted by XS) aboard the SPOT satellites [20], or the Advanced Spaceborne Thermal Emission and Reflector Radiometer (ASTER) aboard the Terra platform [21]. The use of such data is usually not appropriate at the continental scale due to their limited spatial extent and low revisit time.
Multisensor and multiresolution image fusion using the linear mixing model
69
The gap between fine and coarse resolution sensors may be filled with imagery at spatial resolutions of 250 m using MODIS [22] or 300 m using the Medium Resolution Imaging Spectrometer (MERIS) [23]. MODIS and MERIS are spectrometers with a large number of spectral bands and high revisit time. However, for many applications one would like to have a spatial resolution better than 250 or 300 m as being provided by MODIS and MERIS, respectively. However, these high spatial resolution systems lack the large number of spectral bands and high revisit time. Image fusion techniques thrive on combining the advantages of the high spatial resolution systems with those of the high temporal resolution systems (which inherently have a low to medium spatial resolution). Moreover, the latter systems at medium spatial resolution often have many spectral bands in addition to a high temporal resolution. As an example, MERIS has 15 spectral bands in the region between 400 and 1000 nm. This opens the way for deriving biophysical properties, like the leaf chlorophyll content, which cannot be derived from sensors like Landsat-TM or SPOT-XS. Clevers et al. [24] showed that the so-called red-edge index can be derived from the MERIS standard band setting. This rededge index provides information on the leaf chlorophyll content, which cannot be derived from a combination of a near-infrared and visible broad spectral band. Concerning high spectral resolution data, this seems to be the major contribution of imaging spectrometry to applications in agriculture [25]. In this chapter we will focus on image fusion of TM and MERIS based on the linear mixing model.
3.2 Data fusion and remote sensing
The term data fusion groups all the methods that deal with the combination of data coming from different sources [26]. During recent years, data fusion has attracted a lot of attention from the remote sensing community because of the increasing need to integrate the vast amount of data being collected by Earth observation satellites. The main goal of such fusion techniques is to integrate various data sources by combining the best features of each of them. Fused images (i.e. images created by combining two or more types of data) generally offer increased interpretation capabilities and more reliable results [27]. Several data fusion methods have been described in literature. However, most of them are data type dependent [28]. For instance, most of the recent data fusion methods based on wavelet transformation [29] require that the ratio of the spatial resolutions of the images to be fused is a power of 2 [30] or they require that the images to be fused are in the same spectral domain [31]. Furthermore, most of the current data fusion methods do not properly preserve the spectral information of the input images because they are mainly concerned with the visual enhancement of the images. In this study, we selected an unmixing-based data fusion approach. The aim of this method is to combine two images that have been acquired at a different spatial resolution to produce an image with the spatial information of the high spatial resolution image and the spectral information of the low spatial resolution image, whereby no spectral information of the high resolution image contributes to the fused image [32–34]. As a result one may fuse images acquired at different dates as well, making use of the high temporal
70
Image Fusion: Algorithms and Applications
resolution of the low resolution image. Additionally, it ensures a physical interpretation of the fused image and facilitates the retrieval of landscape properties using, for instance, radiative transfer models.
3.3 The linear mixing model
The spatial resolution of medium resolution sensors like MODIS and MERIS is such, that one pixel is mostly covering several land cover types. Thus, the value of a pixel is composed of the signals coming from the individual components. It is a so-called mixed pixel. Only large, homogeneous areas will result in pure pixels in the medium spatial resolution image. The linear mixing model assumes that the spectrum of a mixed pixel is a linear combination of the pure spectra of the components present in that pixel weighted by their fractional coverage [35]. This linear mixing model is mathematically described in Equation (3.1): pi =
nc (rci · fc ) + ei ,
i = 1, . . . , nb
(3.1)
i=1
pi = reflectance (or radiance) of a mixed pixel in band i rci = reflectance (or radiance) of endmember c in band i fc = fraction of endmember c ei = residual error in band i nc = number of endmembers nb = number of spectral bands We can write this in matrix notation as follows: P(nb×1) = R(nb×nc) · F(nc×1) + E(nb×1)
(3.2)
Despite this apparent simplicity, the linear mixing model is widely used by the remote sensing community because it offers an effective framework to analyse mixed pixels. Thus, if we have a priori knowledge about the components that might be present in a given scene, then we can apply linear spectral unmixing of the data to retrieve the subpixel proportions of these components [36,37]. The success of the model outcome relies on the quality of the a priori knowledge on the scene composition. In other words, the results of any linear spectral unmixing method heavily depend on a proper identification of the main components present in the scene and their pure signal. This identification might be very difficult when the image has been acquired over very heterogeneous landscapes or when we work with coarse resolution data because in these cases most of the pixels are mixed (i.e. no pure signal can be found in the scene). In addition to this, the number of components that can be unmixed is limited by the number of spectral bands of the image. The underlying assumption of spectral unmixing using the linear mixing model is that knowledge of endmembers with known spectral profiles yields the fractions (abundances)
Multisensor and multiresolution image fusion using the linear mixing model
71
of the endmembers within a pixel. However, in the current fusion approach we would like to know the spectral profiles of the endmembers within a pixel. This requires knowledge of the abundances of the endmembers within a pixel. If we can derive the abundances, for instance, from an image with a high spatial resolution, then we may derive the spectral profiles by solving the same Equation (3.1) or (3.2). This application of the linear mixing model is known as spatial unmixing or unmixingbased data fusion. The aim of this kind of unmixing is to downscale the radiometric information of the low resolution image to the spatial resolution provided by the high spatial resolution image. Spatial unmixing does not require a priori knowledge of the main components present in the low spatial resolution scene because there is no need to identify their pure signals. In fact, these signals are the output of the spatial unmixing. Therefore, spatial unmixing can be applied to any pair of images, even if the low resolution image only has mixed pixels or a small number of spectral bands. The unmixing-based data fusion approach consists of 4 main steps [32]: (i) The high spatial resolution image is used to identify the different components of the scene. This is done by classifying the high spatial resolution image into nc unsupervised classes (the endmembers). (ii) The proportions of each of these nc classes that fall within each low spatial resolution pixel are then computed. (iii) Using these proportions and the spectral information provided by the low resolution sensor, the spectral behaviour of each nc class is unmixed. Here it is important to notice that in contrast to linear spectral unmixing, which is solved per pixel and for all bands at once, spatial unmixing is solved per pixel, per band, and for a given neighbourhood around the pixel that is being unmixed. This neighbourhood, represented by a square kernel of size k by k, is needed in order to get enough equations to solve the unmixing problem. (iv) Finally, a fused image is reconstructed by joining all the low resolution pixels that have been spatially unmixed. The third step of the unmixing-based data fusion approach can be written for a given band in a compact matrix notation as follows: P(k 2 ×1) = F(k 2 ×nc) · R(nc×1) + E(k 2 ×1)
(3.3)
where P is a (k 2 × 1) vector that contains the values of the band i for all the low spatial resolution pixels present in the neighbourhood k, F is a (k 2 × nc) matrix containing the fractions of each high spatial resolution component inside each low resolution pixel present in k, R is the (nc × 1) unknown vector of downscaled values of the band i for each of the high spatial resolution components inside k, E is a (k 2 × 1) vector of residual errors. This formulation of the unmixing-based data fusion indirectly implies that the number of classes used to classify the high resolution image (nc) and the size of the neighbourhood (k) need to be optimised. The first parameter, nc, needs to be optimised because it depends
72
Image Fusion: Algorithms and Applications
on the spectral variability present in the scene. Thus, heterogeneous scenes will require a larger nc value than homogeneous ones as they have more components. The neighbourhood size, k, also needs to be optimised because it has great influence on the spectral quality of the fused image. On the one hand, k should be kept as small as possible so that the fused image is spectrally dynamic but, on the other hand, k should be as large as possible because it determines the number of equations that are available to solve the spatial unmixing of each pixel. In other words, Equation (3.3) is a system of k 2 equations (one equation per low resolution pixel in the neighbourhood) with up to nc unknowns (depending on the number of classes present in such a neighbourhood). This means that k 2 must be equal to or greater than the number of classes inside the neighbourhood (otherwise the system of equations is undetermined and no solution can be found). However, if we use a large k value, the output image will have low spectral variability because each system of equations results in a unique solution. For instance, if the size of the neighbourhood matches the size of the scene, then we will have one unique set of equations. This implies that a unique spectral response will be assigned to each of the nc classes identified with the high spatial resolution image independently of their position in the scene. This means that the fused image will not be spectrally dynamic and that each of the components will be represented by a kind of spectral mean response. Minghelli-Roman et al. [33] applied this technique to a simulated MERIS image. For the high spatial resolution image (Landsat TM) an optimum number of 150 classes was obtained. Since the unmixing technique is applied to the whole medium resolution image, this results potentially in 150 different signatures. The resulting fused image will only be an approximation of what such a simulated sensor would measure, because all pixels belonging to one class have the same spectral profile. Therefore, a large number of classes are required. However, instead of using an unsupervised classification of a high spatial resolution image like TM, we may also use another source of high spatial resolution, as long as it may serve as a source for defining the objects we are interested in. This may be topographical information where all object (e.g., field) boundaries are identified, such as a land cover data base. As an example, every couple of years an update of the Dutch land cover data base (LGN) is produced. This data base may be used instead of the high resolution image. Such a land cover data base typically will have a limited number of classes, mostly considerably less than 100. The LGN has a legend of 39 classes. Applying the above methodology of Minghelli-Roman et al. [33] would mean that all pixels belonging to one specific land cover class would get the same spectral profile, eliminating all variation within a class. Therefore, we propose a regionalised approach as an alternative procedure. So, although the approach of Minghelli-Roman et al. [33] is computationally fast, we prefer to optimise the size of the neighbourhood k so that we can account for the natural variability of the components present in the scene. The spectral information of the high spatial resolution image or data set does not need to match that of the medium (or low) resolution image because the former spectral information is not directly used. The only requirement of the high resolution data is that it provides a mapping of all the classes of interest. Therefore, a temporal match is also not required as long as there are no significant changes in the landscape occurring.
Multisensor and multiresolution image fusion using the linear mixing model
73
3.4 Case study
3.4.1 Introduction This section describes the unmixing-based data fusion of the TM and the MERIS full resolution images that were available over the study area. MERIS is one of the payload components of the European Space Agency’s (ESA) environmental research satellite Envisat, launched in March 2002. MERIS is a 15 band imaging spectrometer. It is designed to acquire data at variable band width of 1.25 to 30 nm over the spectral range of 390– 1040 nm [23]. Data are acquired at 300 m full resolution (FR) mode or 1200 m reduced resolution (RR) mode over land. In contrast, TM has only four spectral bands in this range of the spectrum. Moreover, the TM bands are broader than those of MERIS. Figure 3.1 summarises the main steps of the data fusion process. First, the TM image was classified into 10, 20, 40, 60, and 80 classes using an unsupervised ISODATA classification rule. The aim of this classification was to identify, with different degrees of detail, the main components (spectral groups) of the scene. After this, a sliding window of k × k MERIS pixels was applied to each of the TM classified images to generate the corresponding class proportion matrices (F). In this study, 14 neighbourhood sizes were tested: from k = 5 to k = 53 in steps of 4. The sliding window used to generate the matrix of proportions F was also applied to all the MERIS bands to extract the k 2 elements that form the P vector. A constrained least-squares method was subsequently used to retrieve the per band MERIS downscaled radiances (R). The use of a constrained method is justified because the solution should fulfil the following two conditions: (i) the radiance values must be positive, and (ii) the radiance values cannot be larger than the MERIS radiance saturation value. The next step of the unmixing-based data fusion consisted of replacing each of the TM unsupervised classes present in the central pixel of the sliding window by their corresponding spectral signal in the MERIS band that is being unmixed. By repeating this operation for all the MERIS FR pixels, for all MERIS bands and for all the possible combinations of nc and k, a series of fused images was generated.
3.4.2 Study area and data The study area covers approximately 40 by 60 km of the central part of the Netherlands (52.19◦ N, 5.91◦ E). This area was selected considering both the heterogeneity of the landscape and the availability of cloud free high and medium spatial resolution satellite data acquired nearly simultaneously. A Landsat-5 TM image from 10 July 2003 and a MERIS full resolution level 1b image acquired 4 days later were available over this area. The TM image was geo-referenced to the Dutch national coordinate system (RD) using a cubic convolution resampling method and a pixel size of 25 m. The MERIS full resolution
74
Figure 3.1
Image Fusion: Algorithms and Applications
Flow chart of the linear mixing model used for the image fusion.
level 1b image was first transformed from digital numbers (DN) to top of atmosphere radiances (LTOA ) using the metadata provided with the file [38]. Then, the image was corrected for the so-called smile effect [39]. Finally, the image was re-projected into the RD coordinate system using the ground control points provided with the image file. After that, an image to image co-registration was performed using a nearest neighbour resampling method. Figure 3.2 shows the co-registered TM and MERIS FR images.
3.4.3 Quality assessment A quantitative analysis of the quality of the fused images has to be performed in order to find out the combination of nc and k that produces the best image from a spectral point of view. In general, this kind of assessments are not straightforward because the quality of the fused images depends on many factors like the difference in spatial or spectral resolution of the input images or the type of landscape under consideration [40]. We selected the ERGAS index as a main quality indicator because it is independent of
Multisensor and multiresolution image fusion using the linear mixing model
Figure 3.2
75
RGB composite of the TM band 4-3-2 (a) and MERIS FR band 13-7-5 (b) over the study area.
the units, the number of spectral bands, and the resolution of the input images [41]. The spectral and the spatial ERGAS indices [42] were used to assess the quality of the fused images at the MERIS and the TM spatial resolutions. Bearing in mind that any fused image should be as identical as possible to the original low resolution image once degraded to that resolution, we degraded the fused images to 300 m using a mean filter. After this, we compared the degraded fused images with the original MERIS FR image. The spectral ERGAS index was used to perform such a comparison: N h 1 ERGAS = 100 RMSE2i /Mi2 (3.4) l N i=1
where h is the resolution of the high spatial resolution image (TM), l is the resolution of the low spatial resolution image (MERIS FR), N is the number of spectral bands involved in the fusion, RMSEi is the root mean square error computed between the degraded fused image and the original MERIS image (for the band i), and Mi is the mean value of the band i of the reference image (MERIS FR). The spatial ERGAS index was used to evaluate the quality of the original fused image (i.e. 15 spectral bands and 25 m pixel size). The expression of the spatial ERGAS index is basically the same as the one used to compute the spectral ERGAS (Equation (3.4)) except that (i) the RMSEi is computed between the TM image and its spectrally corresponding band of the fused image, and (ii) the Mi is now the mean of band i of the TM image.
76
Image Fusion: Algorithms and Applications
Figure 3.3 The spectral (a) and spatial ERGAS (b) for all combinations of neighbourhood size and number of classes.
In order to select the optimal number of classes (nc) and size of the neighbourhood (k), the average of the spectral and spatial ERGAS is used [42].
3.4.4 Results and discussion First, the fused images were calculated for all combinations of 10, 20, 40, 60, and 80 classes with neighbourhood sizes from 5 up to 53 in steps of 4. Only the combinations of a neighbourhood of 5 with 40, 60, and 80 classes do not provide sufficient equations to solve the unmixing. The result is a series of fused images at a spatial resolution of 25 m with the MERIS spectral bands. Degrading the fused images to 300 m should yield the original MERIS image again. By comparing the degraded image with the original MERIS image one can calculate the spectral ERGAS. By comparing the fused image at 25 m with the TM image one can calculate the spatial ERGAS. Figure 3.3 illustrates the spectral and the spatial ERGAS index values for all fused images that were generated for the different combinations of nc and k. All fused images yielded spectral ERGAS values well below 3, which is the upper limit for a good data fusion [41]. This means that the unmixing-based data fusion succeeded
Multisensor and multiresolution image fusion using the linear mixing model
77
in preserving the spectral information of the MERIS image. A larger neighbourhood size and a smaller number of classes both give less differentiation of the spectral signatures in the fused image, thus yielding less correspondence with the MERIS image. This results in higher values of the spectral ERGAS with increasing neighbourhood size and decreasing number of classes. Most of the spatial ERGAS values were also below the upper limit for a good quality. However, the spatial ERGAS values of the images unmixed using a small neighbourhood exceed the empirical upper limit of 3. This might be due to two reasons: (i) The solution of the unmixing is not stable when few equations are used (remember that k determines the number of equations). Nevertheless, it is important to notice that the use of a large neighbourhood does not automatically guarantee a stable solution because if two components have a proportional coverage in that neighbourhood, then the matrix of proportions F will be rank deficient. In this case, the use of regularisation techniques will also be required to find a solution [43]. (ii) The upper limit might not be applicable here because this limit was defined for the spectral ERGAS which compares two images with the same spectral configuration. In this case we compare two images that have the same spatial resolution (25 m) but different band positions and bandwidths. Two other things can be noticed from Figure 3.3. First, the spectral and the spatial ERGAS indices are inversely correlated: the spectral ERGAS decreases when increasing the number of classes and increases with larger neighbourhood sizes, whereas the spatial ERGAS shows the opposite behaviour. This means that there is a trade-off between the spatial and the spectral reconstruction of the fused images and that we cannot find an optimum combination of nc and k that minimises both ERGAS values. Secondly, the spectral and the spatial ERGAS indices present saturation behaviour; this means that increasing nc or k beyond the values that were tested in this study will not improve the quality of the fused images. Figure 3.3 shows that there is a clear trade-off between the size of the neighbourhood and the number of classes. To obtain a spatial ERGAS of, e.g., 0.9, one can apply a small neighbourhood with, e.g., 10 classes, or a larger neighbourhood with a larger number of classes. Both combinations may be used for representing regional variability of the spectral profiles in the fused image. Which combination to use may depend on the high resolution data set that is available. For instance, if this data set only has a limited number of classes, a small neighbourhood should be used in order to get useful variability within the spectral profiles of the image pixels. A small neighbourhood is also feasible mathematically since a small number of classes require only a small number of equations for solving the unmixing equation. In this study we used the average of the spectral and spatial ERGAS for selecting the optimal number of classes and neighbourhood. Several combinations yielded a similar value. We selected the combination nc = 60 and k = 45 to illustrate the fusion result. Figure 3.4 shows the fused image for the test site using this parameter combination. When comparing this figure with Figure 3.2, we may conclude that the fused image exhibits the
78
Figure 3.4
Image Fusion: Algorithms and Applications
RGB colour composite of the fused image band 13-7-5 over the study area.
Figure 3.5 RGB colour composite of a 25 by 25 pixels subset of the fused (a) and the Landsat TM (b) image over the study area.
spatial detail of the Landsat-TM image, whereas it shows the spectral information of the MERIS image. In order to enable more detailed comparisons, Figure 3.5 shows a subset of the fused image (a) and the original TM image (b). Figure 3.6 shows the fused image after degradation to 300 m pixel size (a) and the original MERIS image (b). The fused image shows individual fields like the TM image because it now has a spatial resolution of 25 m (Figure 3.5). However, it is now possible to extract spectral signatures of individual land cover types. This is illustrated in Figure 3.7, showing top-of-
Multisensor and multiresolution image fusion using the linear mixing model
79
Figure 3.6 RGB colour composite of band 13-7-5 of the fused image degraded to 300 m (a) and the MERIS image (b) over the study area.
Figure 3.7
Example of spectral signatures of various land cover types derived from the fused image.
atmosphere radiance values. The general shape of these radiances corresponds to typical spectra of the respective classes. The first few bands show relatively high values due to atmospheric scattering in the blue. Band 11 shows very low values due to an oxygen
80
Image Fusion: Algorithms and Applications
Figure 3.8 Illustration of the red-edge index image based on the fused image over the study area (red-edge index values are linearly scaled between 710 and 730 nm). At the bottom the same subset area as in Figure 3.5 is depicted.
absorption band. The last band shows quite low values due to a water absorption feature. These spectral bands make it possible to calculate the red-edge index as presented by Clevers et al. [24] for MERIS data. The red-edge index based on the fused image is illustrated in Figure 3.8.
Multisensor and multiresolution image fusion using the linear mixing model
81
3.5 Conclusions
In this chapter we have studied the applicability of the linear mixing model to fuse a Landsat TM and a MERIS full resolution level 1b image. The method, known as unmixingbased data fusion, requires the optimisation of 2 parameters: the number of classes used to classify the TM image, nc, and the size of the MERIS neighbourhood, k, used to solve the unmixing equations. Several combinations of nc and k have been tested. The spectral and the spatial ERGAS indices were used to assess the quality of the fused images and to assist with the identification of the best fused image. The unmixing-based data fusion approach is particularly suitable for fusing MERIS FR time series with one or more TM images since the spectral information of the TM images is not included in the fused image in any way. This multitemporal data fusion exercise will be of great interest for land cover mapping and for monitoring vegetation dynamics (e.g., in terms of FAPAR, LAI, or chlorophyll content) at high spatial, spectral, and temporal resolutions. Nevertheless, it is important to notice that these fused images will only be an approximation of what the MERIS sensor would be measuring if it had a spatial resolution of 25 m. In addition, possible landscape changes between the dates of the Landsat TM acquisition and the MERIS images might reduce the quality of the fused images [34].
References
[1] M.E. Schaepman, ‘Spectrodirectional remote sensing: From pixels to processes’, International Journal of Applied Earth Observation and Geoinformation, Vol. 9, 2007, pp. 204–223. [2] R.B. Myneni, R.R. Nemani and S.W. Running, ‘Estimation of global leaf area index and absorbed par using radiative transfer models’, IEEE Transactions on Geoscience and Remote Sensing, Vol. 35, 1997, pp. 1380–1393. [3] A. Ruimy, B. Saugier and G. Dedieu, ‘Methodology for the estimation of terrestrial net primary production from remotely sensed data’, Journal of Geophysical Research, Vol. 99, 1994, pp. 5263–5283. [4] N.H. Broge and E. Leblanc, ‘Comparing prediction power and stability of broadband and hyperspectral vegetation indices for estimation of green leaf area index and canopy chlorophyll density’, Remote Sensing of Environment, Vol. 76, 2001, pp. 156–172. [5] C.S.T. Daughtry, C.L. Walthall, M.S. Kim, E. Brown De Colstoun and J.E. McMurtrey III, ‘Estimating corn leaf chlorophyll concentration from leaf and canopy reflectance’, Remote Sensing of Environment, Vol. 74, 2000, pp. 229–239. [6] D. Haboudane, J.R. Miller, E. Pattey, P.J. Zarco-Tejada and I.B. Strachan, ‘Hyperspectral vegetation indices and novel algorithms for predicting green LAI of crop canopies: Modeling and validation in the context of precision agriculture’, Remote Sensing of Environment, Vol. 90, 2004, pp. 337–352. [7] D. Haboudane, J.R. Miller, N. Tremblay, P.J. Zarco-Tejada and L. Dextraze, ‘Integrated narrow-band vegetation indices for prediction of crop chlorophyll content for application to precision agriculture’, Remote Sensing of Environment, Vol. 81, 2002, pp. 416–426.
82
Image Fusion: Algorithms and Applications
[8] P.S. Thenkabail, R.B. Smith and E. De Pauw, ‘Evaluation of narrowband and broadband vegetation indices for determining optimal hyperspectral wavebands for agricultural crop characterization’, Photogrammetric Engineering and Remote Sensing, Vol. 68, 2002, pp. 607–621. [9] M. Schlerf, C. Atzberger and J. Hill, ‘Remote sensing of forest biophysical variables using HyMap imaging spectrometer data’, Remote Sensing of Environment, Vol. 95, 2005, pp. 177–194. [10] C. Atzberger, ‘Object-based retrieval of biophysical canopy variables using artificial neural nets and radiative transfer models’, Remote Sensing of Environment, Vol. 93, 2004, pp. 53–67. [11] B. Combal, F. Baret, M. Weiss, A. Trubuil, D. Mace, A. Pragnere, R. Myneni, Y. Knyazikhin and L. Wang, ‘Retrieval of canopy biophysical variables from bidirectional reflectance – Using prior information to solve the ill-posed inverse problem’, Remote Sensing of Environment, Vol. 84, 2003, pp. 1–15. [12] S. Jacquemoud, C. Bacour, H. Poilve and J.P. Frangi, ‘Comparison of four radiative transfer models to simulate plant canopies reflectance: Direct and inverse mode’, Remote Sensing of Environment, Vol. 74, 2000, pp. 471–481. [13] W.A. Dorigo, R. Zurita-Milla, A.J.W. De Wit, J. Brazile, R. Singh and M.E. Schaepman, ‘A review on reflective remote sensing and data assimilation techniques for enhanced agroecosystem modeling’, International Journal of Applied Earth Observation and Geoinformation, Vol. 9, 2007, pp. 165–193. [14] T.R. Loveland, B.C. Reed, J.F. Brown, D.O. Ohlen, Z. Zhu, L. Yang and J.W. Merchant, ‘Development of a global land cover characteristics database and IGBP DISCover from 1 km AVHRR data’, International Journal of Remote Sensing, Vol. 21, 2000, pp. 1303–1330. [15] M.C. Hansen, R.S. Defries, J.R.G. Townshend and R. Sohlberg, ‘Global land cover classification at 1 km spatial resolution using a classification tree approach’, International Journal of Remote Sensing, Vol. 21, 2000, pp. 1331–1364. [16] M.A. Friedl, D.K. McIver, J.C.F. Hodges, X.Y. Zhang, D. Muchoney, A.H. Strahler, C.E. Woodcock, S. Gopal, A. Schneider, A. Cooper, A. Baccini, F. Gao and C. Schaaf, ‘Global land cover mapping from MODIS: Algorithms and early results’, Remote Sensing of Environment, Vol. 83, 2002, pp. 287–302. [17] G. Asrar, M. Fuchs, E.T. Kanemasu and J.L. Hatfield, ‘Estimating absorbed photosynthetic radiation and leaf-area index from spectral reflectance in wheat’, Agronomy Journal, Vol. 76, 1984, pp. 300–306. [18] C.A. Mücher, K.T. Steinnocher, F.P. Kressler and C. Heunks, ‘Land cover characterization and change detection for environmental monitoring of pan-Europe’, International Journal of Remote Sensing, Vol. 21, 2000, pp. 1159–1181. [19] W.B. Cohen and S.N. Goward, ‘Landsat’s role in ecological applications of remote sensing’, BioScience, Vol. 54, 2004, pp. 535–545. [20] E. Bartholome and A.S. Belward, ‘GLC2000: A new approach to global land cover mapping from Earth observation data’, International Journal of Remote Sensing, Vol. 26, 2005, pp. 1959–1977. [21] A.R.S. Marcal, J.S. Borges, J.A. Gomes and J.F.P. Da Costa, ‘Land cover update by supervised classification of segmented ASTER images’, International Journal of Remote Sensing, Vol. 26, 2005, pp. 1347–1362.
Multisensor and multiresolution image fusion using the linear mixing model
83
[22] X. Zhan, R.A. Sohlberg, J.R.G. Townshend, C. Dimiceli, M.L. Carroll, J.C. Eastman, M.C. Hansen and R.S. Defries, ‘Detection of land cover changes using MODIS 250 m data’, Remote Sensing of Environment, Vol. 83, 2002, pp. 336–350. [23] M. Rast, J.L. Bezy and S. Bruzzi, ‘The ESA medium resolution imaging spectrometer MERIS – A review of the instrument and its mission’, International Journal of Remote Sensing, Vol. 20, 1999, pp. 1681–1702. [24] J.G.P.W. Clevers, S.M. De Jong, G.F. Epema, F.D. Van Der Meer, W.H. Bakker, A.K. Skidmore and K.H. Scholte, ‘Derivation of the red edge index using the MERIS standard band setting’, International Journal of Remote Sensing, Vol. 23, 2002, pp. 3169–3184. [25] J.G.P.W. Clevers, ‘The use of imaging spectrometry for agricultural applications’, ISPRS Journal of Photogrammetry and Remote Sensing, Vol. 54, 1999, pp. 299– 304. [26] L. Wald, ‘Some terms of reference in data fusion’, IEEE Transactions on Geoscience and Remote Sensing, Vol. 37, 1999, pp. 1190–1193. [27] C. Pohl and J.L. Van Genderen, ‘Multisensor image fusion in remote sensing: Concepts, methods and applications’, International Journal of Remote Sensing, Vol. 19, 1998, pp. 823–854. [28] Y. Zhang, ‘Understanding image fusion’, Photogrammetric Engineering and Remote Sensing, Vol. 70, 2004, pp. 657–661. [29] F.W. Acerbi-Junior, J.G.P.W. Clevers and M.E. Schaepman, ‘The assessment of multi-sensor image fusion using wavelet transforms for mapping the Brazilian Savanna’, International Journal of Applied Earth Observation and Geoinformation, Vol. 8, 2006, pp. 278–288. [30] W. Shi, C. Zhu, Y. Tian and J. Nichol, ‘Wavelet-based image fusion and quality assessment’, International Journal of Applied Earth Observation and Geoinformation, Vol. 6, 2005, pp. 241–251. [31] X. Otazu, M. Gonzalez-Audicana, O. Fors and J. Nunez, ‘Introduction of sensor spectral response into image fusion methods. Application to wavelet-based methods’, IEEE Transactions on Geoscience and Remote Sensing, Vol. 43, 2005, pp. 2376–2385. [32] B. Zhukov, D. Oertel, F. Lanzl and G. Reinhackel, ‘Unmixing-based multisensor multiresolution image fusion’, IEEE Transactions on Geoscience and Remote Sensing, Vol. 37, 1999, pp. 1212–1226. [33] A. Minghelli-Roman, M. Mangolini, M. Petit and L. Polidori, ‘Spatial resolution improvement of MeRIS images by fusion with TM images’, IEEE Transactions on Geoscience and Remote Sensing, Vol. 39, 2001, pp. 1533–1536. [34] A. Minghelli-Roman, L. Polidori, S. Mathieu-Blanc, L. Loubersac and F. Cauneau, ‘Spatial resolution improvement by merging MERIS-ETM images for coastal water monitoring’, IEEE Geoscience and Remote Sensing Letters, Vol. 3, 2006, pp. 227– 231. [35] J.J. Settle and N.A. Drake, ‘Linear mixing and the estimation of ground cover proportions’, International Journal of Remote Sensing, Vol. 14, 1993, pp. 1159–1177. [36] S.L. Ustin, M.O. Smith and J.B. Adams, ‘Remote sensing of ecological processes: A strategy for developing and testing ecological models using spectral mixture analysis’, in J.R. Ehleringer and C.B. Field (eds.), Scaling Physicological Processes: Leaf to Globe, Academic Press, San Diego, CA, 1993, 388 pp.
84
Image Fusion: Algorithms and Applications
[37] J.B. Adams, D.E. Sabol, V. Kapos, R. Almeida, D.A. Roberts, M.O. Smith and A.R. Gillespie, ‘Classification of multispectral images based on fractions of endmembers – Application to land-cover change in the Brazilian Amazon’, Remote Sensing of Environment, Vol. 52, 1995, pp. 137–154. [38] J.G.P.W. Clevers, M.E. Schaepman, C.A. Mücher, A.J.W. De Wit, R. ZuritaMilla and H.M. Bartholomeus, ‘Using MERIS on Envisat for land cover mapping in the Netherlands’, International Journal of Remote Sensing, Vol. 28, 2007, pp. 637– 652. [39] R. Zurita-Milla, J.G.P.W. Clevers, M.E. Schaepman and M. Kneubuehler, ‘Effects of MERIS L1b radiometric calibration on regional land cover mapping and land products’, International Journal of Remote Sensing, Vol. 28, 2007, pp. 653–673. [40] C. Thomas and L. Wald, ‘Assessment of the quality of fused products’, in M. Oluic (ed.), 24th EARSeL Symposium, New Strategies for European Remote Sensing, Dubrovnik, Croatia, 25–27 May 2004. [41] L. Wald, Data Fusion Definitions and Architectures: Fusion of Images of Different Spatial Resolutions, Ecole des Mines Pres, 2002. [42] M. Lillo-Saavedra, C. Gonzalo, A. Arquero and E. Martinez, ‘Fusion of multispectral and panchromatic satellite sensor imagery based on tailored filtering in the Fourier domain’, International Journal of Remote Sensing, Vol. 26, 2005, pp. 1263– 1268. [43] R. Faivre and A. Fischer, ‘Predicting crop reflectances using satellite data observing mixed pixels’, Journal of Agricultural, Biological, and Environmental Statistics, Vol. 2, 1997, pp. 87–107.
In satellite sensor design we observe a trade-off between sensors with a high spatial resolution having only a few spectral bands and a low revisit frequency on the one hand, and sensors with a medium to low spatial resolution having many spectral bands and a high revisit time on the other hand. Many applications require a combination of a high spatial, spectral, and temporal resolution. In this chapter image fusion of a high spatial resolution (Landsat Thematic Mapper) and a high spectral resolution (Envisat MERIS) image based on the linear mixing model is presented. This approach is also known as spatial unmixing or unmixing-based data fusion. An optimisation of the number of classes used to classify the high spatial resolution image and the size of the neighbourhood, for which the unmixing equations are solved, is presented. It is illustrated for a test area in the Netherlands. Results show the feasibility of this approach yielding fused images with the spatial resolution of the high resolution image and with the spectral information from the low spatial resolution image. The quality of the fused images is evaluated using the spectral and spatial ERGAS index. Main advantage of the presented technique based on the linear mixing model is that the fused images do not include the spectral information of the high spatial resolution image in the final result in any way.
3.1 Introduction
Terrestrial vegetation plays an important role in biochemical cycles, like the global carbon cycle. Information on the type of vegetation is important for such studies. This is thematic information referring to the biome type or the land cover type. In addition, once we have the thematic class information, we need quantitative information on vegetation properties. 67
68
Image Fusion: Algorithms and Applications
This quantitative information refers to the so-called continuous fields [1]. In order to quantify the role of vegetation in biochemical cycles, variables describing the area of the green photosynthetic elements (mainly the leaves), the amount of photosynthetically active radiation (PAR) absorbed by the vegetation and the chlorophyll content of the leaves are important [2]. Key variables required for the estimation of, for instance, the primary production of vegetation are the leaf area index (LAI) and the fraction of PAR absorbed by vegetation (FAPAR) [3]. The LAI refers to the one-sided green leaf area per unit ground area. The FAPAR refers to the fraction of the incoming PAR absorbed. PAR (0.4–0.7 µm) is used as energy source in the photosynthesis process, whereby atmospheric CO2 and H2 O from the soil are converted into carbohydrates, enabling growth of the vegetation. The estimation of both LAI and FAPAR from spectral measurements also depends on other vegetation parameters, like leaf properties and canopy structure. Therefore, vegetation class-dependent relationships are often used. In general, to describe the relationship between spectral measurements and biophysical and chemical variables of vegetation both statistical and physical approaches have been used. As an example of statistical methods, numerous vegetation indices (VIs) have been developed for estimating variables like biomass and LAI for a range of vegetation types [4–9]. Physical-based methods often use radiative transfer models describing the interaction of radiation with the plant canopy based on physical principles. Subsequently, model inversion is used for estimating biophysical and chemical properties of the canopy [10–12]. Depending on the application, these techniques can be applied from local to global scales [13]. On the one hand, datasets derived from coarse resolution sensors provide information on land cover and vegetation properties globally. In the US, global land cover products have been derived using time series with 1 km data obtained from the National Oceanic and Atmospheric Administration’s (NOAA) Advanced Very High Resolution Radiometer (AVHRR) [14,15]. In addition, a 1 km land cover map has been compiled using a time series of data from the Moderate Resolution Imaging Spectroradiometer (MODIS) on the Terra platform [16]. An example of continuous fields is the MODIS LAI product, where biome specific algorithms are used for obtaining global maps of LAI [2]. Photosynthetic activity of vegetation canopies can be shown by the fraction of APAR, which can be estimated from satellite observation [17]. Using time-series of satellite data it is shown, for instance, that the length of the growing season is increasing at the northern hemisphere, which may be caused by global warming [2]. This coarse scale imagery is limiting the use for monitoring purposes due to the finer scale at which most land cover changes take place [18]. Still these sub-kilometre scale changes are critical for monitoring changes in, e.g., sinks and sources of the carbon cycle. On the other hand, many detailed studies at the regional and landscape scale at spatial resolutions between 10 and 30 m have been performed using the Thematic Mapper (TM) aboard the Landsat satellites [19], the multispectral imager (denoted by XS) aboard the SPOT satellites [20], or the Advanced Spaceborne Thermal Emission and Reflector Radiometer (ASTER) aboard the Terra platform [21]. The use of such data is usually not appropriate at the continental scale due to their limited spatial extent and low revisit time.
Multisensor and multiresolution image fusion using the linear mixing model
69
The gap between fine and coarse resolution sensors may be filled with imagery at spatial resolutions of 250 m using MODIS [22] or 300 m using the Medium Resolution Imaging Spectrometer (MERIS) [23]. MODIS and MERIS are spectrometers with a large number of spectral bands and high revisit time. However, for many applications one would like to have a spatial resolution better than 250 or 300 m as being provided by MODIS and MERIS, respectively. However, these high spatial resolution systems lack the large number of spectral bands and high revisit time. Image fusion techniques thrive on combining the advantages of the high spatial resolution systems with those of the high temporal resolution systems (which inherently have a low to medium spatial resolution). Moreover, the latter systems at medium spatial resolution often have many spectral bands in addition to a high temporal resolution. As an example, MERIS has 15 spectral bands in the region between 400 and 1000 nm. This opens the way for deriving biophysical properties, like the leaf chlorophyll content, which cannot be derived from sensors like Landsat-TM or SPOT-XS. Clevers et al. [24] showed that the so-called red-edge index can be derived from the MERIS standard band setting. This rededge index provides information on the leaf chlorophyll content, which cannot be derived from a combination of a near-infrared and visible broad spectral band. Concerning high spectral resolution data, this seems to be the major contribution of imaging spectrometry to applications in agriculture [25]. In this chapter we will focus on image fusion of TM and MERIS based on the linear mixing model.
3.2 Data fusion and remote sensing
The term data fusion groups all the methods that deal with the combination of data coming from different sources [26]. During recent years, data fusion has attracted a lot of attention from the remote sensing community because of the increasing need to integrate the vast amount of data being collected by Earth observation satellites. The main goal of such fusion techniques is to integrate various data sources by combining the best features of each of them. Fused images (i.e. images created by combining two or more types of data) generally offer increased interpretation capabilities and more reliable results [27]. Several data fusion methods have been described in literature. However, most of them are data type dependent [28]. For instance, most of the recent data fusion methods based on wavelet transformation [29] require that the ratio of the spatial resolutions of the images to be fused is a power of 2 [30] or they require that the images to be fused are in the same spectral domain [31]. Furthermore, most of the current data fusion methods do not properly preserve the spectral information of the input images because they are mainly concerned with the visual enhancement of the images. In this study, we selected an unmixing-based data fusion approach. The aim of this method is to combine two images that have been acquired at a different spatial resolution to produce an image with the spatial information of the high spatial resolution image and the spectral information of the low spatial resolution image, whereby no spectral information of the high resolution image contributes to the fused image [32–34]. As a result one may fuse images acquired at different dates as well, making use of the high temporal
70
Image Fusion: Algorithms and Applications
resolution of the low resolution image. Additionally, it ensures a physical interpretation of the fused image and facilitates the retrieval of landscape properties using, for instance, radiative transfer models.
3.3 The linear mixing model
The spatial resolution of medium resolution sensors like MODIS and MERIS is such, that one pixel is mostly covering several land cover types. Thus, the value of a pixel is composed of the signals coming from the individual components. It is a so-called mixed pixel. Only large, homogeneous areas will result in pure pixels in the medium spatial resolution image. The linear mixing model assumes that the spectrum of a mixed pixel is a linear combination of the pure spectra of the components present in that pixel weighted by their fractional coverage [35]. This linear mixing model is mathematically described in Equation (3.1): pi =
nc (rci · fc ) + ei ,
i = 1, . . . , nb
(3.1)
i=1
pi = reflectance (or radiance) of a mixed pixel in band i rci = reflectance (or radiance) of endmember c in band i fc = fraction of endmember c ei = residual error in band i nc = number of endmembers nb = number of spectral bands We can write this in matrix notation as follows: P(nb×1) = R(nb×nc) · F(nc×1) + E(nb×1)
(3.2)
Despite this apparent simplicity, the linear mixing model is widely used by the remote sensing community because it offers an effective framework to analyse mixed pixels. Thus, if we have a priori knowledge about the components that might be present in a given scene, then we can apply linear spectral unmixing of the data to retrieve the subpixel proportions of these components [36,37]. The success of the model outcome relies on the quality of the a priori knowledge on the scene composition. In other words, the results of any linear spectral unmixing method heavily depend on a proper identification of the main components present in the scene and their pure signal. This identification might be very difficult when the image has been acquired over very heterogeneous landscapes or when we work with coarse resolution data because in these cases most of the pixels are mixed (i.e. no pure signal can be found in the scene). In addition to this, the number of components that can be unmixed is limited by the number of spectral bands of the image. The underlying assumption of spectral unmixing using the linear mixing model is that knowledge of endmembers with known spectral profiles yields the fractions (abundances)
Multisensor and multiresolution image fusion using the linear mixing model
71
of the endmembers within a pixel. However, in the current fusion approach we would like to know the spectral profiles of the endmembers within a pixel. This requires knowledge of the abundances of the endmembers within a pixel. If we can derive the abundances, for instance, from an image with a high spatial resolution, then we may derive the spectral profiles by solving the same Equation (3.1) or (3.2). This application of the linear mixing model is known as spatial unmixing or unmixingbased data fusion. The aim of this kind of unmixing is to downscale the radiometric information of the low resolution image to the spatial resolution provided by the high spatial resolution image. Spatial unmixing does not require a priori knowledge of the main components present in the low spatial resolution scene because there is no need to identify their pure signals. In fact, these signals are the output of the spatial unmixing. Therefore, spatial unmixing can be applied to any pair of images, even if the low resolution image only has mixed pixels or a small number of spectral bands. The unmixing-based data fusion approach consists of 4 main steps [32]: (i) The high spatial resolution image is used to identify the different components of the scene. This is done by classifying the high spatial resolution image into nc unsupervised classes (the endmembers). (ii) The proportions of each of these nc classes that fall within each low spatial resolution pixel are then computed. (iii) Using these proportions and the spectral information provided by the low resolution sensor, the spectral behaviour of each nc class is unmixed. Here it is important to notice that in contrast to linear spectral unmixing, which is solved per pixel and for all bands at once, spatial unmixing is solved per pixel, per band, and for a given neighbourhood around the pixel that is being unmixed. This neighbourhood, represented by a square kernel of size k by k, is needed in order to get enough equations to solve the unmixing problem. (iv) Finally, a fused image is reconstructed by joining all the low resolution pixels that have been spatially unmixed. The third step of the unmixing-based data fusion approach can be written for a given band in a compact matrix notation as follows: P(k 2 ×1) = F(k 2 ×nc) · R(nc×1) + E(k 2 ×1)
(3.3)
where P is a (k 2 × 1) vector that contains the values of the band i for all the low spatial resolution pixels present in the neighbourhood k, F is a (k 2 × nc) matrix containing the fractions of each high spatial resolution component inside each low resolution pixel present in k, R is the (nc × 1) unknown vector of downscaled values of the band i for each of the high spatial resolution components inside k, E is a (k 2 × 1) vector of residual errors. This formulation of the unmixing-based data fusion indirectly implies that the number of classes used to classify the high resolution image (nc) and the size of the neighbourhood (k) need to be optimised. The first parameter, nc, needs to be optimised because it depends
72
Image Fusion: Algorithms and Applications
on the spectral variability present in the scene. Thus, heterogeneous scenes will require a larger nc value than homogeneous ones as they have more components. The neighbourhood size, k, also needs to be optimised because it has great influence on the spectral quality of the fused image. On the one hand, k should be kept as small as possible so that the fused image is spectrally dynamic but, on the other hand, k should be as large as possible because it determines the number of equations that are available to solve the spatial unmixing of each pixel. In other words, Equation (3.3) is a system of k 2 equations (one equation per low resolution pixel in the neighbourhood) with up to nc unknowns (depending on the number of classes present in such a neighbourhood). This means that k 2 must be equal to or greater than the number of classes inside the neighbourhood (otherwise the system of equations is undetermined and no solution can be found). However, if we use a large k value, the output image will have low spectral variability because each system of equations results in a unique solution. For instance, if the size of the neighbourhood matches the size of the scene, then we will have one unique set of equations. This implies that a unique spectral response will be assigned to each of the nc classes identified with the high spatial resolution image independently of their position in the scene. This means that the fused image will not be spectrally dynamic and that each of the components will be represented by a kind of spectral mean response. Minghelli-Roman et al. [33] applied this technique to a simulated MERIS image. For the high spatial resolution image (Landsat TM) an optimum number of 150 classes was obtained. Since the unmixing technique is applied to the whole medium resolution image, this results potentially in 150 different signatures. The resulting fused image will only be an approximation of what such a simulated sensor would measure, because all pixels belonging to one class have the same spectral profile. Therefore, a large number of classes are required. However, instead of using an unsupervised classification of a high spatial resolution image like TM, we may also use another source of high spatial resolution, as long as it may serve as a source for defining the objects we are interested in. This may be topographical information where all object (e.g., field) boundaries are identified, such as a land cover data base. As an example, every couple of years an update of the Dutch land cover data base (LGN) is produced. This data base may be used instead of the high resolution image. Such a land cover data base typically will have a limited number of classes, mostly considerably less than 100. The LGN has a legend of 39 classes. Applying the above methodology of Minghelli-Roman et al. [33] would mean that all pixels belonging to one specific land cover class would get the same spectral profile, eliminating all variation within a class. Therefore, we propose a regionalised approach as an alternative procedure. So, although the approach of Minghelli-Roman et al. [33] is computationally fast, we prefer to optimise the size of the neighbourhood k so that we can account for the natural variability of the components present in the scene. The spectral information of the high spatial resolution image or data set does not need to match that of the medium (or low) resolution image because the former spectral information is not directly used. The only requirement of the high resolution data is that it provides a mapping of all the classes of interest. Therefore, a temporal match is also not required as long as there are no significant changes in the landscape occurring.
Multisensor and multiresolution image fusion using the linear mixing model
73
3.4 Case study
3.4.1 Introduction This section describes the unmixing-based data fusion of the TM and the MERIS full resolution images that were available over the study area. MERIS is one of the payload components of the European Space Agency’s (ESA) environmental research satellite Envisat, launched in March 2002. MERIS is a 15 band imaging spectrometer. It is designed to acquire data at variable band width of 1.25 to 30 nm over the spectral range of 390– 1040 nm [23]. Data are acquired at 300 m full resolution (FR) mode or 1200 m reduced resolution (RR) mode over land. In contrast, TM has only four spectral bands in this range of the spectrum. Moreover, the TM bands are broader than those of MERIS. Figure 3.1 summarises the main steps of the data fusion process. First, the TM image was classified into 10, 20, 40, 60, and 80 classes using an unsupervised ISODATA classification rule. The aim of this classification was to identify, with different degrees of detail, the main components (spectral groups) of the scene. After this, a sliding window of k × k MERIS pixels was applied to each of the TM classified images to generate the corresponding class proportion matrices (F). In this study, 14 neighbourhood sizes were tested: from k = 5 to k = 53 in steps of 4. The sliding window used to generate the matrix of proportions F was also applied to all the MERIS bands to extract the k 2 elements that form the P vector. A constrained least-squares method was subsequently used to retrieve the per band MERIS downscaled radiances (R). The use of a constrained method is justified because the solution should fulfil the following two conditions: (i) the radiance values must be positive, and (ii) the radiance values cannot be larger than the MERIS radiance saturation value. The next step of the unmixing-based data fusion consisted of replacing each of the TM unsupervised classes present in the central pixel of the sliding window by their corresponding spectral signal in the MERIS band that is being unmixed. By repeating this operation for all the MERIS FR pixels, for all MERIS bands and for all the possible combinations of nc and k, a series of fused images was generated.
3.4.2 Study area and data The study area covers approximately 40 by 60 km of the central part of the Netherlands (52.19◦ N, 5.91◦ E). This area was selected considering both the heterogeneity of the landscape and the availability of cloud free high and medium spatial resolution satellite data acquired nearly simultaneously. A Landsat-5 TM image from 10 July 2003 and a MERIS full resolution level 1b image acquired 4 days later were available over this area. The TM image was geo-referenced to the Dutch national coordinate system (RD) using a cubic convolution resampling method and a pixel size of 25 m. The MERIS full resolution
74
Figure 3.1
Image Fusion: Algorithms and Applications
Flow chart of the linear mixing model used for the image fusion.
level 1b image was first transformed from digital numbers (DN) to top of atmosphere radiances (LTOA ) using the metadata provided with the file [38]. Then, the image was corrected for the so-called smile effect [39]. Finally, the image was re-projected into the RD coordinate system using the ground control points provided with the image file. After that, an image to image co-registration was performed using a nearest neighbour resampling method. Figure 3.2 shows the co-registered TM and MERIS FR images.
3.4.3 Quality assessment A quantitative analysis of the quality of the fused images has to be performed in order to find out the combination of nc and k that produces the best image from a spectral point of view. In general, this kind of assessments are not straightforward because the quality of the fused images depends on many factors like the difference in spatial or spectral resolution of the input images or the type of landscape under consideration [40]. We selected the ERGAS index as a main quality indicator because it is independent of
Multisensor and multiresolution image fusion using the linear mixing model
Figure 3.2
75
RGB composite of the TM band 4-3-2 (a) and MERIS FR band 13-7-5 (b) over the study area.
the units, the number of spectral bands, and the resolution of the input images [41]. The spectral and the spatial ERGAS indices [42] were used to assess the quality of the fused images at the MERIS and the TM spatial resolutions. Bearing in mind that any fused image should be as identical as possible to the original low resolution image once degraded to that resolution, we degraded the fused images to 300 m using a mean filter. After this, we compared the degraded fused images with the original MERIS FR image. The spectral ERGAS index was used to perform such a comparison: N h 1 ERGAS = 100 RMSE2i /Mi2 (3.4) l N i=1
where h is the resolution of the high spatial resolution image (TM), l is the resolution of the low spatial resolution image (MERIS FR), N is the number of spectral bands involved in the fusion, RMSEi is the root mean square error computed between the degraded fused image and the original MERIS image (for the band i), and Mi is the mean value of the band i of the reference image (MERIS FR). The spatial ERGAS index was used to evaluate the quality of the original fused image (i.e. 15 spectral bands and 25 m pixel size). The expression of the spatial ERGAS index is basically the same as the one used to compute the spectral ERGAS (Equation (3.4)) except that (i) the RMSEi is computed between the TM image and its spectrally corresponding band of the fused image, and (ii) the Mi is now the mean of band i of the TM image.
76
Image Fusion: Algorithms and Applications
Figure 3.3 The spectral (a) and spatial ERGAS (b) for all combinations of neighbourhood size and number of classes.
In order to select the optimal number of classes (nc) and size of the neighbourhood (k), the average of the spectral and spatial ERGAS is used [42].
3.4.4 Results and discussion First, the fused images were calculated for all combinations of 10, 20, 40, 60, and 80 classes with neighbourhood sizes from 5 up to 53 in steps of 4. Only the combinations of a neighbourhood of 5 with 40, 60, and 80 classes do not provide sufficient equations to solve the unmixing. The result is a series of fused images at a spatial resolution of 25 m with the MERIS spectral bands. Degrading the fused images to 300 m should yield the original MERIS image again. By comparing the degraded image with the original MERIS image one can calculate the spectral ERGAS. By comparing the fused image at 25 m with the TM image one can calculate the spatial ERGAS. Figure 3.3 illustrates the spectral and the spatial ERGAS index values for all fused images that were generated for the different combinations of nc and k. All fused images yielded spectral ERGAS values well below 3, which is the upper limit for a good data fusion [41]. This means that the unmixing-based data fusion succeeded
Multisensor and multiresolution image fusion using the linear mixing model
77
in preserving the spectral information of the MERIS image. A larger neighbourhood size and a smaller number of classes both give less differentiation of the spectral signatures in the fused image, thus yielding less correspondence with the MERIS image. This results in higher values of the spectral ERGAS with increasing neighbourhood size and decreasing number of classes. Most of the spatial ERGAS values were also below the upper limit for a good quality. However, the spatial ERGAS values of the images unmixed using a small neighbourhood exceed the empirical upper limit of 3. This might be due to two reasons: (i) The solution of the unmixing is not stable when few equations are used (remember that k determines the number of equations). Nevertheless, it is important to notice that the use of a large neighbourhood does not automatically guarantee a stable solution because if two components have a proportional coverage in that neighbourhood, then the matrix of proportions F will be rank deficient. In this case, the use of regularisation techniques will also be required to find a solution [43]. (ii) The upper limit might not be applicable here because this limit was defined for the spectral ERGAS which compares two images with the same spectral configuration. In this case we compare two images that have the same spatial resolution (25 m) but different band positions and bandwidths. Two other things can be noticed from Figure 3.3. First, the spectral and the spatial ERGAS indices are inversely correlated: the spectral ERGAS decreases when increasing the number of classes and increases with larger neighbourhood sizes, whereas the spatial ERGAS shows the opposite behaviour. This means that there is a trade-off between the spatial and the spectral reconstruction of the fused images and that we cannot find an optimum combination of nc and k that minimises both ERGAS values. Secondly, the spectral and the spatial ERGAS indices present saturation behaviour; this means that increasing nc or k beyond the values that were tested in this study will not improve the quality of the fused images. Figure 3.3 shows that there is a clear trade-off between the size of the neighbourhood and the number of classes. To obtain a spatial ERGAS of, e.g., 0.9, one can apply a small neighbourhood with, e.g., 10 classes, or a larger neighbourhood with a larger number of classes. Both combinations may be used for representing regional variability of the spectral profiles in the fused image. Which combination to use may depend on the high resolution data set that is available. For instance, if this data set only has a limited number of classes, a small neighbourhood should be used in order to get useful variability within the spectral profiles of the image pixels. A small neighbourhood is also feasible mathematically since a small number of classes require only a small number of equations for solving the unmixing equation. In this study we used the average of the spectral and spatial ERGAS for selecting the optimal number of classes and neighbourhood. Several combinations yielded a similar value. We selected the combination nc = 60 and k = 45 to illustrate the fusion result. Figure 3.4 shows the fused image for the test site using this parameter combination. When comparing this figure with Figure 3.2, we may conclude that the fused image exhibits the
78
Figure 3.4
Image Fusion: Algorithms and Applications
RGB colour composite of the fused image band 13-7-5 over the study area.
Figure 3.5 RGB colour composite of a 25 by 25 pixels subset of the fused (a) and the Landsat TM (b) image over the study area.
spatial detail of the Landsat-TM image, whereas it shows the spectral information of the MERIS image. In order to enable more detailed comparisons, Figure 3.5 shows a subset of the fused image (a) and the original TM image (b). Figure 3.6 shows the fused image after degradation to 300 m pixel size (a) and the original MERIS image (b). The fused image shows individual fields like the TM image because it now has a spatial resolution of 25 m (Figure 3.5). However, it is now possible to extract spectral signatures of individual land cover types. This is illustrated in Figure 3.7, showing top-of-
Multisensor and multiresolution image fusion using the linear mixing model
79
Figure 3.6 RGB colour composite of band 13-7-5 of the fused image degraded to 300 m (a) and the MERIS image (b) over the study area.
Figure 3.7
Example of spectral signatures of various land cover types derived from the fused image.
atmosphere radiance values. The general shape of these radiances corresponds to typical spectra of the respective classes. The first few bands show relatively high values due to atmospheric scattering in the blue. Band 11 shows very low values due to an oxygen
80
Image Fusion: Algorithms and Applications
Figure 3.8 Illustration of the red-edge index image based on the fused image over the study area (red-edge index values are linearly scaled between 710 and 730 nm). At the bottom the same subset area as in Figure 3.5 is depicted.
absorption band. The last band shows quite low values due to a water absorption feature. These spectral bands make it possible to calculate the red-edge index as presented by Clevers et al. [24] for MERIS data. The red-edge index based on the fused image is illustrated in Figure 3.8.
Multisensor and multiresolution image fusion using the linear mixing model
81
3.5 Conclusions
In this chapter we have studied the applicability of the linear mixing model to fuse a Landsat TM and a MERIS full resolution level 1b image. The method, known as unmixingbased data fusion, requires the optimisation of 2 parameters: the number of classes used to classify the TM image, nc, and the size of the MERIS neighbourhood, k, used to solve the unmixing equations. Several combinations of nc and k have been tested. The spectral and the spatial ERGAS indices were used to assess the quality of the fused images and to assist with the identification of the best fused image. The unmixing-based data fusion approach is particularly suitable for fusing MERIS FR time series with one or more TM images since the spectral information of the TM images is not included in the fused image in any way. This multitemporal data fusion exercise will be of great interest for land cover mapping and for monitoring vegetation dynamics (e.g., in terms of FAPAR, LAI, or chlorophyll content) at high spatial, spectral, and temporal resolutions. Nevertheless, it is important to notice that these fused images will only be an approximation of what the MERIS sensor would be measuring if it had a spatial resolution of 25 m. In addition, possible landscape changes between the dates of the Landsat TM acquisition and the MERIS images might reduce the quality of the fused images [34].
References
[1] M.E. Schaepman, ‘Spectrodirectional remote sensing: From pixels to processes’, International Journal of Applied Earth Observation and Geoinformation, Vol. 9, 2007, pp. 204–223. [2] R.B. Myneni, R.R. Nemani and S.W. Running, ‘Estimation of global leaf area index and absorbed par using radiative transfer models’, IEEE Transactions on Geoscience and Remote Sensing, Vol. 35, 1997, pp. 1380–1393. [3] A. Ruimy, B. Saugier and G. Dedieu, ‘Methodology for the estimation of terrestrial net primary production from remotely sensed data’, Journal of Geophysical Research, Vol. 99, 1994, pp. 5263–5283. [4] N.H. Broge and E. Leblanc, ‘Comparing prediction power and stability of broadband and hyperspectral vegetation indices for estimation of green leaf area index and canopy chlorophyll density’, Remote Sensing of Environment, Vol. 76, 2001, pp. 156–172. [5] C.S.T. Daughtry, C.L. Walthall, M.S. Kim, E. Brown De Colstoun and J.E. McMurtrey III, ‘Estimating corn leaf chlorophyll concentration from leaf and canopy reflectance’, Remote Sensing of Environment, Vol. 74, 2000, pp. 229–239. [6] D. Haboudane, J.R. Miller, E. Pattey, P.J. Zarco-Tejada and I.B. Strachan, ‘Hyperspectral vegetation indices and novel algorithms for predicting green LAI of crop canopies: Modeling and validation in the context of precision agriculture’, Remote Sensing of Environment, Vol. 90, 2004, pp. 337–352. [7] D. Haboudane, J.R. Miller, N. Tremblay, P.J. Zarco-Tejada and L. Dextraze, ‘Integrated narrow-band vegetation indices for prediction of crop chlorophyll content for application to precision agriculture’, Remote Sensing of Environment, Vol. 81, 2002, pp. 416–426.
82
Image Fusion: Algorithms and Applications
[8] P.S. Thenkabail, R.B. Smith and E. De Pauw, ‘Evaluation of narrowband and broadband vegetation indices for determining optimal hyperspectral wavebands for agricultural crop characterization’, Photogrammetric Engineering and Remote Sensing, Vol. 68, 2002, pp. 607–621. [9] M. Schlerf, C. Atzberger and J. Hill, ‘Remote sensing of forest biophysical variables using HyMap imaging spectrometer data’, Remote Sensing of Environment, Vol. 95, 2005, pp. 177–194. [10] C. Atzberger, ‘Object-based retrieval of biophysical canopy variables using artificial neural nets and radiative transfer models’, Remote Sensing of Environment, Vol. 93, 2004, pp. 53–67. [11] B. Combal, F. Baret, M. Weiss, A. Trubuil, D. Mace, A. Pragnere, R. Myneni, Y. Knyazikhin and L. Wang, ‘Retrieval of canopy biophysical variables from bidirectional reflectance – Using prior information to solve the ill-posed inverse problem’, Remote Sensing of Environment, Vol. 84, 2003, pp. 1–15. [12] S. Jacquemoud, C. Bacour, H. Poilve and J.P. Frangi, ‘Comparison of four radiative transfer models to simulate plant canopies reflectance: Direct and inverse mode’, Remote Sensing of Environment, Vol. 74, 2000, pp. 471–481. [13] W.A. Dorigo, R. Zurita-Milla, A.J.W. De Wit, J. Brazile, R. Singh and M.E. Schaepman, ‘A review on reflective remote sensing and data assimilation techniques for enhanced agroecosystem modeling’, International Journal of Applied Earth Observation and Geoinformation, Vol. 9, 2007, pp. 165–193. [14] T.R. Loveland, B.C. Reed, J.F. Brown, D.O. Ohlen, Z. Zhu, L. Yang and J.W. Merchant, ‘Development of a global land cover characteristics database and IGBP DISCover from 1 km AVHRR data’, International Journal of Remote Sensing, Vol. 21, 2000, pp. 1303–1330. [15] M.C. Hansen, R.S. Defries, J.R.G. Townshend and R. Sohlberg, ‘Global land cover classification at 1 km spatial resolution using a classification tree approach’, International Journal of Remote Sensing, Vol. 21, 2000, pp. 1331–1364. [16] M.A. Friedl, D.K. McIver, J.C.F. Hodges, X.Y. Zhang, D. Muchoney, A.H. Strahler, C.E. Woodcock, S. Gopal, A. Schneider, A. Cooper, A. Baccini, F. Gao and C. Schaaf, ‘Global land cover mapping from MODIS: Algorithms and early results’, Remote Sensing of Environment, Vol. 83, 2002, pp. 287–302. [17] G. Asrar, M. Fuchs, E.T. Kanemasu and J.L. Hatfield, ‘Estimating absorbed photosynthetic radiation and leaf-area index from spectral reflectance in wheat’, Agronomy Journal, Vol. 76, 1984, pp. 300–306. [18] C.A. Mücher, K.T. Steinnocher, F.P. Kressler and C. Heunks, ‘Land cover characterization and change detection for environmental monitoring of pan-Europe’, International Journal of Remote Sensing, Vol. 21, 2000, pp. 1159–1181. [19] W.B. Cohen and S.N. Goward, ‘Landsat’s role in ecological applications of remote sensing’, BioScience, Vol. 54, 2004, pp. 535–545. [20] E. Bartholome and A.S. Belward, ‘GLC2000: A new approach to global land cover mapping from Earth observation data’, International Journal of Remote Sensing, Vol. 26, 2005, pp. 1959–1977. [21] A.R.S. Marcal, J.S. Borges, J.A. Gomes and J.F.P. Da Costa, ‘Land cover update by supervised classification of segmented ASTER images’, International Journal of Remote Sensing, Vol. 26, 2005, pp. 1347–1362.
Multisensor and multiresolution image fusion using the linear mixing model
83
[22] X. Zhan, R.A. Sohlberg, J.R.G. Townshend, C. Dimiceli, M.L. Carroll, J.C. Eastman, M.C. Hansen and R.S. Defries, ‘Detection of land cover changes using MODIS 250 m data’, Remote Sensing of Environment, Vol. 83, 2002, pp. 336–350. [23] M. Rast, J.L. Bezy and S. Bruzzi, ‘The ESA medium resolution imaging spectrometer MERIS – A review of the instrument and its mission’, International Journal of Remote Sensing, Vol. 20, 1999, pp. 1681–1702. [24] J.G.P.W. Clevers, S.M. De Jong, G.F. Epema, F.D. Van Der Meer, W.H. Bakker, A.K. Skidmore and K.H. Scholte, ‘Derivation of the red edge index using the MERIS standard band setting’, International Journal of Remote Sensing, Vol. 23, 2002, pp. 3169–3184. [25] J.G.P.W. Clevers, ‘The use of imaging spectrometry for agricultural applications’, ISPRS Journal of Photogrammetry and Remote Sensing, Vol. 54, 1999, pp. 299– 304. [26] L. Wald, ‘Some terms of reference in data fusion’, IEEE Transactions on Geoscience and Remote Sensing, Vol. 37, 1999, pp. 1190–1193. [27] C. Pohl and J.L. Van Genderen, ‘Multisensor image fusion in remote sensing: Concepts, methods and applications’, International Journal of Remote Sensing, Vol. 19, 1998, pp. 823–854. [28] Y. Zhang, ‘Understanding image fusion’, Photogrammetric Engineering and Remote Sensing, Vol. 70, 2004, pp. 657–661. [29] F.W. Acerbi-Junior, J.G.P.W. Clevers and M.E. Schaepman, ‘The assessment of multi-sensor image fusion using wavelet transforms for mapping the Brazilian Savanna’, International Journal of Applied Earth Observation and Geoinformation, Vol. 8, 2006, pp. 278–288. [30] W. Shi, C. Zhu, Y. Tian and J. Nichol, ‘Wavelet-based image fusion and quality assessment’, International Journal of Applied Earth Observation and Geoinformation, Vol. 6, 2005, pp. 241–251. [31] X. Otazu, M. Gonzalez-Audicana, O. Fors and J. Nunez, ‘Introduction of sensor spectral response into image fusion methods. Application to wavelet-based methods’, IEEE Transactions on Geoscience and Remote Sensing, Vol. 43, 2005, pp. 2376–2385. [32] B. Zhukov, D. Oertel, F. Lanzl and G. Reinhackel, ‘Unmixing-based multisensor multiresolution image fusion’, IEEE Transactions on Geoscience and Remote Sensing, Vol. 37, 1999, pp. 1212–1226. [33] A. Minghelli-Roman, M. Mangolini, M. Petit and L. Polidori, ‘Spatial resolution improvement of MeRIS images by fusion with TM images’, IEEE Transactions on Geoscience and Remote Sensing, Vol. 39, 2001, pp. 1533–1536. [34] A. Minghelli-Roman, L. Polidori, S. Mathieu-Blanc, L. Loubersac and F. Cauneau, ‘Spatial resolution improvement by merging MERIS-ETM images for coastal water monitoring’, IEEE Geoscience and Remote Sensing Letters, Vol. 3, 2006, pp. 227– 231. [35] J.J. Settle and N.A. Drake, ‘Linear mixing and the estimation of ground cover proportions’, International Journal of Remote Sensing, Vol. 14, 1993, pp. 1159–1177. [36] S.L. Ustin, M.O. Smith and J.B. Adams, ‘Remote sensing of ecological processes: A strategy for developing and testing ecological models using spectral mixture analysis’, in J.R. Ehleringer and C.B. Field (eds.), Scaling Physicological Processes: Leaf to Globe, Academic Press, San Diego, CA, 1993, 388 pp.
84
Image Fusion: Algorithms and Applications
[37] J.B. Adams, D.E. Sabol, V. Kapos, R. Almeida, D.A. Roberts, M.O. Smith and A.R. Gillespie, ‘Classification of multispectral images based on fractions of endmembers – Application to land-cover change in the Brazilian Amazon’, Remote Sensing of Environment, Vol. 52, 1995, pp. 137–154. [38] J.G.P.W. Clevers, M.E. Schaepman, C.A. Mücher, A.J.W. De Wit, R. ZuritaMilla and H.M. Bartholomeus, ‘Using MERIS on Envisat for land cover mapping in the Netherlands’, International Journal of Remote Sensing, Vol. 28, 2007, pp. 637– 652. [39] R. Zurita-Milla, J.G.P.W. Clevers, M.E. Schaepman and M. Kneubuehler, ‘Effects of MERIS L1b radiometric calibration on regional land cover mapping and land products’, International Journal of Remote Sensing, Vol. 28, 2007, pp. 653–673. [40] C. Thomas and L. Wald, ‘Assessment of the quality of fused products’, in M. Oluic (ed.), 24th EARSeL Symposium, New Strategies for European Remote Sensing, Dubrovnik, Croatia, 25–27 May 2004. [41] L. Wald, Data Fusion Definitions and Architectures: Fusion of Images of Different Spatial Resolutions, Ecole des Mines Pres, 2002. [42] M. Lillo-Saavedra, C. Gonzalo, A. Arquero and E. Martinez, ‘Fusion of multispectral and panchromatic satellite sensor imagery based on tailored filtering in the Fourier domain’, International Journal of Remote Sensing, Vol. 26, 2005, pp. 1263– 1268. [43] R. Faivre and A. Fischer, ‘Predicting crop reflectances using satellite data observing mixed pixels’, Journal of Agricultural, Biological, and Environmental Statistics, Vol. 2, 1997, pp. 87–107.