Model-assisted estimation as a unifying framework for estimating the area of land cover and land-cover change from remote sensing

Remote Sensing of Environment 113 (2009) 2455–2462

Contents lists available at ScienceDirect

Remote Sensing of Environment j o u r n a l h o m e p a g e : w w w. e l s ev i e r. c o m / l o c a t e / r s e

Model-assisted estimation as a unifying framework for estimating the area of land cover and land-cover change from remote sensing Stephen V. Stehman SUNY College of Environmental Science & Forestry, Syracuse, NY 13210, USA

a r t i c l e

i n f o

Article history: Received 27 January 2009 Received in revised form 17 July 2009 Accepted 18 July 2009 Keywords: Accuracy assessment General regression estimator Sampling design Design-based inference

a b s t r a c t Two common approaches to estimate the area of land cover or land-cover change are to use a confusion matrix to adjust the area derived from pixel counting and to use a survey sampling regression estimator that takes advantage of auxiliary variables to improve the precision of the estimated area. These two seemingly divergent approaches to area estimation are encompassed within the general framework of model-assisted estimation. The theory and methods of model-assisted estimation can be applied to expand the options for area estimators and to extend these estimators to sampling designs beyond those currently in use. The objectives of area estimation and map accuracy assessment can be addressed by the same sample data. This suggests the need for research to identify or develop sampling designs that effectively and efficiently achieve this dual-purpose use of these data. © 2009 Elsevier Inc. All rights reserved.

1. Introduction Quantifying the area of each land-cover class or the area of landcover change and assessing the accuracy of a map of land cover or land-cover change are typically viewed as separate objectives addressed by different methodologies. Gallego (2004) provides a comprehensive review of area estimation methods, and Foody (2002) and Stehman and Foody (2009) review basic methods of accuracy assessment. The two objectives are brought together when a confusion matrix obtained from an accuracy assessment is used to adjust the area of each land-cover class or area of land-cover change obtained by pixel counting (Card, 1982; Czaplewski & Catts 1992). But the connection between area estimation and accuracy assessment extends well beyond this linkage via the confusion matrix. The objective of this article is to establish a stronger connection between area estimation and accuracy assessment and to present model-assisted estimation as a general extension of the confusion matrix approach to area estimation within the design-based inference framework. In design-based inference, the response observed on an element of the population is regarded as a fixed constant, not a random variable, and the randomization distribution resulting from the sampling design is the basis of inference. For example, variance and bias of an estimator of a population parameter are determined from the set of all possible samples (i.e. the sample space) and the probability associated with each sample in the sample space for the particular sampling design implemented. de Gruijter and ter Braak (1990), Gregoire (1998), Särndal et al. (1992), and Stehman (2000)

E-mail address: [email protected]. 0034-4257/$ – see front matter © 2009 Elsevier Inc. All rights reserved. doi:10.1016/j.rse.2009.07.006

provide extensive discussion of design-based inference and contrast it with model-based inference. To define terminology used in this article, an “estimator” is a statistic calculated from the sample data (i.e. a rule or formula) and an “estimate” is a value obtained by applying the estimator to a particular sample realization (Särndal et al., 1992, pp. 38–40). The phrase “reference condition” or “reference class” will be used to describe the best available determination of the true land cover or land-cover change at a specified location. In some applications, the reference condition may be determined by a ground visit, and in other cases it may be determined by interpreting aerial photos or videography, or by classifying satellite imagery. “Reference data” will be used to represent the general protocol and information collected used to obtain the reference condition, and “reference area” will refer to the area of a land-cover class or to the area of land-cover change when these areas are based on the reference condition. Numerous potential sources of error in reference data exist (Congalton & Green, 1999; Foody, 2002), as, for example, when an incorrect reference label is assigned by an interpreter or when the map and reference data locations are spatially mis-registered. The complications arising from reference data error will not be addressed. Rather, it will be assumed that the reference data represent the best available information regarding the true or ground condition, and the quality of the estimates of accuracy or area will be diminished to the degree that reference data error is present. The area estimation objective focuses on estimating the reference area of a land-cover class or the reference area of land-cover change. The area estimation objective does not target the area of a land-cover class as represented by the map, but instead targets the area of the class as it is determined from the reference condition. In reality, it is generally not practical to obtain the reference condition for the full

2456

S.V. Stehman / Remote Sensing of Environment 113 (2009) 2455–2462

region of interest (e.g. the reference condition is obtained for only a sample of ground plots, or in a case where satellite imagery is used to obtain the reference condition, only a sample of the imagery is actually classified). For the area estimation objective, other auxiliary variables potentially associated with the reference condition may be available for the entire region. These auxiliary variables may be useful to improve the precision of the area estimates. For example, if estimating the area of land-cover change is the objective and most change in the region is associated with human activity, auxiliary variables related to road density or population density may be useful. The area estimation problem has been approached from two different perspectives. In the first, area estimation is viewed as an offshoot or value-added analysis appended to an accuracy assessment. That is, the primary objective is to assess the accuracy of a land cover or land-cover change classification, but the sample obtained for accuracy assessment can also be used to estimate the area of each class mapped. The confusion matrix typically used to summarize an accuracy assessment is then employed to adjust the area of each class obtained from counting map pixels (Card, 1982; Czaplewski & Catts 1992). The area estimators derived via this approach will be called “confusion matrix area estimators”. An alternative perspective is to view area estimation as the primary objective. In such cases, survey sampling regression estimators (Cochran, 1977, Chapter 7) are commonly employed to estimate area. A land-cover map or other complete coverage information may be used to provide auxiliary variables for a regression estimator of area, but assessing the accuracy of the complete coverage information is often not considered an objective when this second perspective is taken. The following example is typical of the approach. Suppose the reference condition is forest cover loss determined from classified Landsat imagery and the auxiliary variable is area of forest clearing as determined from classified coarser resolution imagery (e.g. 250m × 250-m, 500-m × 500-m, or 1-km × 1-km data from the MODerate Resolution Imaging Spectroradiometer, MODIS). The target parameter is the total area of forest cover loss as determined from the higher resolution Landsat imagery (i.e. area of the reference condition). However, cost considerations prevent producing a complete coverage map of forest cover loss from the Landsat imagery, so the Landsatderived forest cover loss is obtained for a sample of areal units (e.g. 10km × 10-km blocks). A complete coverage forest clearing map derived from the MODIS imagery is available, and this information on forest clearing serves as the auxiliary variable (xu) that can be used to improve the precision of the estimator of the area of the reference condition (i.e. forest cover loss as determined from Landsat imagery). The confusion matrix area estimators and the regression estimators are typically viewed as distinct approaches to estimating area. In this article, the two approaches to area estimation are shown to share a common structure as model-assisted estimators. Recognizing this underlying commonality reveals a broad variety of estimators available within the class of model-assisted estimators that are applicable to the area estimation problem. Several of these previously unused estimators are highlighted for specific area estimation applications. The strong link between area estimation and land-cover mapping also reveals the opportunity to use sample data to address both area estimation and accuracy assessment objectives simultaneously. The characteristics of sampling designs that would enhance the value of the sample data for both objectives are described, and some of the key sampling design issues that must be resolved are also provided. 2. Area estimators 2.1. Estimating area directly from an accuracy assessment sample A sample obtained to assess the accuracy of a classification of land cover or land-cover change can be used to directly estimate the area of land cover or land-cover change. The accuracy assessment is based on

a reference sample of locations at which the reference condition of land cover or land-cover change is obtained. Because these data represent the best available determination of the true land cover or land-cover change (i.e. the reference condition), they also provide the best available information for estimating area. The estimated area is thus based on the sample of the reference condition. In the following example, the approach of estimating area directly from an accuracy assessment sample is illustrated for simple random and stratified random sampling. Although the example addresses accuracy and area of a land-cover classification, the extension to a land-cover change classification is straightforward. Suppose the accuracy assessment is a per-pixel evaluation of a classification in which each pixel is assigned to one of the k land-cover classes mapped (i.e. a crisp classification). A probability sample of pixels is selected and the reference class obtained for each pixel. The confusion matrix is constructed such that the rows represent the reference class and the columns represent the map class (this is the confusion matrix structure used by Card (1982) and Czaplewski and Catts (1992), but in other presentations of a confusion matrix, the rows and columns may be reversed). The population pixel count for each cell of the confusion matrix (i.e. the count based on a census of reference data) is denoted Nij to represent the number of pixels that are reference class i and map class j. The corresponding confusion matrix cell entries obtained from the sample counts are denoted nij. Let n denote the sample size and N the population size (i.e. N is the total number of pixels in the region of interest). The parameter of interest is the population proportion of area in reference class c, Pc + =

k X

Ncj = N = Nc + = N;

ð1Þ

j=1

which corresponds to the row proportion for class c of the population confusion matrix. For simple random sampling, an unbiased estimator of this ̂ = nc+ / n, where nc+ is the number of population proportion is P c+ sample pixels in reference class c. The total area of reference class c is ̂ by the total area of the region. estimated simply by multiplying P c+ Stratified random sampling, with the strata based on the map classes, is often implemented to increase the sample size in the rare map classes. The true number of pixels mapped as class c, N+ c, is known, and the sample size obtained from each map class (stratum) is fixed at a specified value, n+ c. Following standard stratified random sampling theory (Cochran 1977, Eq. (5.52)), the estimated proportion of area of reference class c is Pˆ c +; str = ð1 = NÞ

k X j=1

N+j

! ncj : n+j

ð2Þ

If the accuracy assessment sampling design is not simple random or stratified random, the general theory of Horvitz–Thompson estimation (Särndal et al., 1992) would be applicable to construct an area estimator for any other probability sampling design. Recognizing that a probability sample implemented to assess map accuracy is also a legitimate sample for estimating the reference area of each class is consistent with the standard methodology employed to estimate non-site-specific accuracy. Non-site-specific accuracy examines the accuracy of the proportion of area mapped of each class without regard to location (Congalton & Green 1999, p. 43). That is, the parameter representing non-site-specific accuracy of class c compares the proportion of area mapped as class c, N+ c / N, to the proportion of area of class c as determined from the reference classification, Nc+ / N. The latter proportion can be estimated directly from the accuracy assessment sample, thereby re-affirming that the sample used for accuracy assessment can also be used to provide an estimate of reference area (for each class) independent of the map

S.V. Stehman / Remote Sensing of Environment 113 (2009) 2455–2462

being subject to the accuracy assessment. In the next subsection, the area estimation problem is approached from a different perspective. In this alternate view, the area obtained from counting pixels for each map class must be adjusted based on information contained in the confusion matrix obtained from an accuracy assessment. 2.2. Confusion matrix area estimators The area of each land-cover class or area of change in land cover can be computed by counting pixels in a land-cover map or land-cover change map. However, because of classification error, the area derived from pixel counting is usually biased (see Czaplewski, 1992, Gallego, 2004, or Stehman, 2005 for specification of this bias). Because the pixel count is based on a complete census of the region, the bias of this pixel count area is viewed as a “measurement bias” rather than as an “estimator bias” (Särndal et al., 1992, p. 608; Stehman 2005). A confusion matrix provides the classification error information that allows for adjusting the area obtained from pixel counting to account for this measurement bias. Similar to the estimators presented in the previous subsection, these confusion matrix area estimators are obtained from the sample selected for accuracy assessment and the reference class determined at each sample location. The parameter of interest is still the total area or proportion of area of a land-cover class or land-cover change class as determined from the reference condition (Eq. (1)). Two approaches that use the confusion matrix to adjust the pixel count area are the “inverse” estimator and the “direct” estimator (the terminology follows Gallego, 2004, p. 3032). The formulas and rationale for these estimators are thoroughly described by Gallego (2004, Section 3.4), Czaplewski and Catts (1992), and Walsh and Burk (1993). The equations for the two estimators provided below follow the derivation and notation provided by Czaplewski and Catts (1992), but the labels assigned to these estimators follow Gallego's (2004) terminology. Suppose the population consists of N pixels and there are k classes of interest. Let P represent a (k × k) population confusion matrix with cell entry pij denoting the population proportion of pixels with reference class i and map class j. Let the (k × 1) column vector t = (t1, t2,…, tk)′ denote the population proportion of area in each class based on the reference classification, and the (k × 1) column vector r = (r1, r2,…, rk)′ denote the population proportion of area in each class based on the map classification. The unknown parameter of interest is t, whereas r is known. Then for a simple random sample of size n, the direct estimator of t is given by   −1 tˆdir = Pˆ Rˆ r;

ð3Þ

where P̂ is the estimated confusion matrix with cell entries estimated by p̂ij = nij / n, and R̂ = diag(P̂′1) where 1 denotes a (k × 1) column vector of 1's and diag denotes a diagonal (k × k) matrix with the diagonal entries obtained from the (k × 1) column vector P̂′1 (see Czaplewski and Catts 1992, Eq. (A10)). The diagonal entries of R̂ are the estimated marginal proportions of the map classification. Gallego (2004) labels (Eq. (3)) the “direct” estimator, whereas Czaplewski and Catts (1992) apply the term “inverse” estimator. An alternative estimator, called the “classical” estimator by Czaplewski and Catts (1992, Eq. (A11)) but called the “inverse” estimator by Gallego (2004), is given by   −1 −1 tˆinv = Pˆ VTˆ r

ð4Þ

where T ̂ = diag(P ̂1) is a diagonal (k × k) matrix. The diagonal entries of T ̂ are the estimated marginal proportions of the reference classification. Based on their evaluation of example populations covering different numbers of land-cover classes and different classification

2457

error structures, Czaplewski and Catts (1992) and Walsh and Burk (1993) demonstrated that the direct estimator (Eq. (3)) was generally superior to the inverse estimator (Eq. (4)). Yuan (1996) suggested that the problems associated with the inverse estimator, instability and non-existence of the inverse of (P ̂′T ̂− 1)in Eq. (4), were diminished if the classification met the condition of “minimal practically acceptable”, which translates to having producer's accuracy for each class exceeding 50%. Adequate performance of the inverse estimator was obtained in a simulation study (Yuan 1997) when the evaluation was limited to classifications that met this minimal practically acceptable criterion. Buckland and Elston (1994) asserted that the inverse estimator was not applicable for sampling designs in which the pixels were sampled with unequal inclusion probabilities (i.e. all pixels do not have an equal chance of being selected in the sample), and they presented empirical evidence showing a potentially large bias of the inverse estimator. Confusion matrix area estimators have been derived for simple random and stratified random sampling. The need to extend confusion matrix area estimators and variance estimators to additional sampling designs, particularly cluster sampling, has been noted. For example, Czaplewski and Catts (1992) suggested the need for further study if “heterogeneous reference sites” are used, Walsh and Burk (1993, p. 282) mentioned that for “certain sampling situations, such as cluster sampling, the data are often evaluated differently,” and Gallego (2004, p. 3033) suggested that in regard to variance estimation, “… adaptations are needed if each sampling unit is a segment (for example a square of 1 km2) containing a number of pixels that do not necessarily belong to the same class.” Buckland and Elston (1994) derived a version of the direct estimator for a sampling design in which stratification by distance to nearest access point is employed to guide allocation of a first-stage sample of 1-km × 1-km squares, and then one point location is selected at random from within each first-stage sample 1-km × 1-km square. 2.3. Model-assisted estimators A model-assisted estimator takes advantage of a model of the relationship between the target (response) variable and one or more auxiliary variables to improve the precision over the estimators presented in Section 2.1. Suppose the target region is partitioned into N segments. These segments could be equal area units such as 30-m × 30-m pixels or 10-km× 10-km blocks, or segments could be unequal area, irregularly shaped units such as watersheds, map delineated polygons, or administrative regions (e.g., counties or parishes). Let yu denote the area of the reference condition of class c for segment u. The target parameter, the total reference area of class c, can be expressed as PN T= u = 1 yu . One or more auxiliary variables may be available for all N segments. A simple version of the model-assisted strategy is based on a simple random sample of n segments from the N available. If a single auxiliary variable (xu) is measured on each of the N segments, then the simple regression estimator of the population total T is    Tˆreg = N y + b X − x

ð5Þ

where x ̅ and y ̅ are the sample means of xu and yu, X ̅ is the mean of xu for all N segments, and b is the ordinary least square estimator of the slope for the regression of yu on xu. Scheaffer et al. (2006, Section 6.6) present a numerical example illustrating the calculations required for the simple regression estimator (Eq. (5)). If xu and yu are correlated, then T ̂reg generally has a smaller variance than the estimator T ̂ = Ny ̅ that does not use the auxiliary information (Särndal et al., 1992). The simple regression estimator is a special case of a general model-assisted estimator called a general regression (GREG) estimator (Särndal et al., 1992, Eq. (6.4.1), p. 225). The GREG estimator encompasses a wide range of models beyond the linear model used in the

2458

S.V. Stehman / Remote Sensing of Environment 113 (2009) 2455–2462

simple regression estimator (Eq. (5)) and extends to probability sampling designs other than simple random sampling. The GREG estimator is approximately unbiased for large samples and its variance and variance estimators are documented (Särndal et al., 1992, Result 6.6.1, p. 235). While the improvement in the precision of a model-assisted estimator is dependent on how well the specified model corresponds to the actual relationship between yu and xu, the validity of the inference is not dependent on correct model specification (Särndal et al., 1992, Secs. 6.4.1, 6.7) but instead remains based on the randomization distribution associated with the sampling design. That is, model-assisted estimators are “approximately (asymptotically) design unbiased irrespective of whether the working model is correct or not, and are particularly efficient if a working model is correct” (Wu & Sitter 2001). The strength of model-assisted estimation is the variety of sampling designs and models that can be accommodated. The fact that the variance and variance estimators exist for the GREG estimator along with the approximate unbiased property of the GREG estimator provides strong underlying theoretical support for this approach to area estimation. 3. Synthesis of approaches to area estimators The connection between the confusion matrix area estimators and the model-assisted estimators is established by recognizing that the direct estimator (Eq. (3)) is a poststratified estimator, and then noting that poststratified estimators fall within the class of model-assisted estimators. As defined previously (Section 2.1), the confusion matrix is constructed with the rows representing the reference classes and the columns the map classes. The confusion matrix column Pk margin, N+ j = i = 1 Nij , the number of pixels mapped as class j, is known for each of the k classes. For simple random sampling, Card (1982, Eq. (22)) provides the direct estimator of Nc+, which when divided by N, yields the direct confusion matrix area estimator of the proportion of area of class c,

ð1 = NÞ

k X j=1

N +j

! ncj : n+j

where xc,u is 1 if element u of the population has map class c and xc,u is 0 if element u is not map class c. The population total of xc,u is N+ c, the total number of elements that have map class c. For some response yu defined on element u of the population, the group mean model is yu = β1 x1;u + β2 x2;u + N + βk xk;u + eu

ð7Þ

where β1, β2,…,βk are population regression coefficients and εu is the random (model) error. The GREG estimator derived from this model in combination with a simple random sampling design would lead to the poststratified estimator (Eq. (6)). The simple regression estimator incorporates different auxiliary information from the poststratified estimator and therefore also employs a different model. The model underlying the regression estimator (Eq. (5)) is a common simple regression model (Särndal et al., 1992, Section 7.8), yu = α + βxu + eu

ð8Þ

where α and β are the population y-intercept and slope, and εu is the random error. The poststratified estimator (Eq. (6)) and the simple regression estimator (Eq. (5)), both derived for simple random sampling, are just two of a wide variety of potential estimators that can be applied to the area estimation problem. Recognizing the model-assisted basis of these estimators creates the opportunity to explore the much richer class of estimators available from other models taking advantage of different auxiliary variables, and the general theory of model-assisted estimation extends the applicability of these estimators to other probability sampling designs. While the model-assisted framework includes the direct confusion matrix calibration estimator as a special case, it has not been proven that the inverse calibration estimator (Eq. (4)) based on the confusion matrix can similarly be identified as a model-assisted estimator. Uncovering a model underlying the inverse estimator would be a useful unifying result. 4. Model-assisted area estimators

ð6Þ

Eq. (6) is equivalent to Eq. (2), the stratified random sampling estimator of a proportion. Therefore, although he does not identify it as such, Card's (1982, Eq. (22)) direct estimator is a poststratified estimator of the proportion of area of class c. The difference between Card's (1982) direct confusion matrix area estimator and the stratified sampling estimator (Eq. (2)) is that the sample size in each stratum (n+ j) is fixed for the stratified random sampling design leading to Eq. (2), whereas these sample sizes vary from sample to sample for the simple random sampling design underlying the poststratified estimator (Eq. (6)). For the case of simple random sampling, instead of using the sample proportion of reference area in class c, nc+ / n, as the estimator of Pc+ (Section 2.1), auxiliary information provided by the land-cover map, specifically the known map marginal proportions N+j, is incorporated in the poststratified estimator (Eq. (6)). This establishes that the direct confusion matrix area estimator can be viewed as an estimator that takes advantage of auxiliary information in the form of the map land-cover marginal proportions to improve the precision of the estimator of area. Särndal et al. (1992, Secs. 7.5, 7.6) show that a poststratified estimator is a model-assisted estimator based on a group mean model, thus completing the link between Card's (1982) direct estimator (Eq. (6)) based on the confusion matrix and the model-assisted estimators that also include the simple regression estimator (Eq. (5)) as a special case. The group mean model may be specified as follows for the case in which the map classes are used as strata (groups). Suppose there are k dummy or indicator variables x1,u, x2,u,…, xk,u,

The flexibility exists to choose from a variety of models for model-assisted estimation. Distinguishing whether the reference data for the N segments partitioning the study area produces a categorical response (e.g. a land-cover class) or a continuous response (e.g. area of land cover or area of change) is a useful first consideration when deciding upon a model. For example, if the segments are pixels and each pixel is assigned to a single reference class, the response is categorical. If the segments are pixels and each pixel is assigned a vector of proportions representing the reference area of each class present (i.e. a mixed pixel), the response is continuous. A categorical response will be viewed as binary for purposes of estimating the reference area of one class at a time. That is, to estimate the area of class c, the binary response would take on the value y u = 1 if segment u is reference class c, and yu = 0 if segment u is not reference class c. A univariate approach will also be used for a continuous response, with the area of each class estimated separately. Some model-assisted estimators will apply to only one of the response types (categorical or continuous) whereas other estimators can be applied to both types. 4.1. Poststratification Poststratification is a general estimation technique typically applied with a simple random sampling design. The stratification is not used when selecting the sample, but is instead employed after the sample has been selected and stratified estimators are used in the analysis (Särndal et al., 1992, p. 265). In Section 3, it was established that the direct estimator from the family of confusion matrix area estimators is a poststratified estimator in which the map classes are

S.V. Stehman / Remote Sensing of Environment 113 (2009) 2455–2462

the poststrata. In the direct confusion matrix area estimator, the response is categorical because the reference class is observed for each sample segment (e.g. a pixel). In the broader view of poststratification, the auxiliary information is the proportion of area in each of the poststrata, and the poststrata can be constructed from any available information related to the reference condition, not just the mapped classes. Further, poststratification can be applied to either a categorical or continuous response. For the continuous case, if yu,j = area of reference class c in segment u in poststratum j, a poststratified estimator of the total reference area of class c is Tˆpost =

Xk j=1

N +j

X uaSj

yu; j = n + j ;

ð9Þ

where the second summation is over all sample segments in poststratum j, Sj. The poststratified estimators for simple random sampling are equivalent to the estimators that would be used if a stratified random sampling design had been implemented. A variance estimator for the poststratified estimator under simple random sampling is provided by Särndal et al. (1992, Sections 7.6, 7.10.2). Stehman et al. (2005) evaluated poststratified estimation for the continuous response of area of land-cover change on equal area square segments with poststrata constructed from population census data. In another example in which the response defined on each segment was continuous, Hansen et al. (2008) employed poststratified area estimation where equal area square segments were poststratified by intact forest landscape (Greenpeace International, 2006) to improve the estimation of forest area cleared in the tropics. McRoberts et al. (2002, 2005) extensively explored stratified estimation using maps based on remotely sensed variables. Applications in which the direct estimator has been employed are all examples of poststratified estimation for a categorical response (see Card, 1982 for a numerical example). 4.2. Raking Raking estimators can be applied when the auxiliary information includes more than one categorical variable, but only the marginal population totals of the auxiliary variables are used in the estimator. Lohr (1999, Section 8.5.2.2) provides a simple numerical example of raking estimation, and Deville et al. (1993) provide a detailed exposition of the theory, including variance and variance estimator formulas. Raking estimators are applicable for both types of response, categorical and continuous. A specific application in which a raking estimator of area would be considered is when an accuracy assessment is conducted to evaluate two land-cover maps (assuming the maps share a common legend). The accuracy assessment sample data provide the reference condition for estimating area, and the two land-cover maps provide the complete coverage auxiliary information to improve the precision of the area estimator. Because the proportion of area of each land-cover class mapped is known for both maps, it would be possible to use just one of the maps for poststratified estimation of area (i.e. employ the direct confusion matrix area estimator). But the raking estimator allows for using two or more maps to improve the precision of the estimator of area, and thus potentially improves upon the poststratified estimator. Table 1 illustrates the data required for the raking estimator when two land-cover maps are available, each map based on a four-class legend, natural, developed, agriculture, and water. The cell entries Mij represent the proportion of area mapped as class i by Map A and class j by Map B. Although it may be possible to poststratify by all 16 cells of the cross-classification, the risk is that some of these cross-classes comprise a small area and the sample size in one or more of these cross-classes will be 0. The raking estimator uses only the marginal

2459

Table 1 Data required for using a raking estimator of land-cover area when two complete coverage land-cover maps provide the auxiliary information. Land-cover Map B Natural Developed Agriculture Water Total Land-cover Map A Natural Developed Agriculture Water Total

M11 M21 M31 M41 M+1

M12 M22 M32 M42 M+2

M13 M23 M33 M43 M+3

M14 M24 M34 M44 M+4

M1+ M2+ M3+ M4+ 1.00

The scenario assumes a common land-cover legend of 4 classes for both maps. Mij denotes the proportion of area that is mapped as class i by Land-cover Map A and class j by Land-cover Map B.

proportions, Mi+ and M+ j, and weights each sample observation to produce estimates that are consistent with the row and column margins of the table. That is, if the raking estimator is applied to the sample observations to estimate the proportion of area in each class of each map, the raking estimates would reproduce the known map proportions Mi+ and M+ j. These same raking estimation weights are then employed to estimate any parameter of interest for the study region, in particular the reference area of each class. If both maps are reasonably accurate, the auxiliary information contained in the map marginal proportions Mi+ and M+ j will result in the raking estimator having a smaller variance than an area estimator in which no auxiliary information is used (Section 2.1) or one in which only one of the maps is used in a poststratified area estimator. The application of a raking estimator would be a natural choice for use with the global validation reference database envisioned by Strahler et al. (2006, Section 6.1) to validate existing land-cover products. This global validation database would be obtained from a probability sampling design and would include the reference class for each sample location. Not only would it be possible to estimate the reference area of each class from this accuracy assessment sample, but the various complete coverage maps that are the subject of validation may be advantageously used to improve the precision of the estimated areas via application of a raking estimator. Such an application merging accuracy assessment and area estimation objectives would be a highly efficient use of accuracy assessment reference data obtained to validate multiple maps.

4.3. Model-assisted estimators for a binary (categorical) response For a binary response, the class of model-assisted area estimators can be extended beyond the simple group mean model of the poststratified estimator to more complex models. A logistic regression model is an obvious option when the outcome (response) variable is binary. The logistic regression model accommodates both continuous and categorical auxiliary variables. For example, the binary outcome presence or absence of reference land-cover class c at each location could be modeled as a function of a continuous auxiliary variable such as elevation and a categorical auxiliary variable indicating the presence or absence of class c as determined from a lower quality classification of the target region. Even more complex logistic models could incorporate the spatial correlation of the presence of the reference class. Firth and Bennett (1998) describe an application of a logistic regression model as well as other more complex models within the model-assisted estimation framework and Opsomer et al. (2007) present an application in which a generalized additive model was employed to estimate the area of forest cover. These approaches are clearly more complex model-assisted estimation strategies than the poststratified and raking estimators, but the logistic and generalized additive models are viable extensions of the model-assisted approach.

2460

S.V. Stehman / Remote Sensing of Environment 113 (2009) 2455–2462

4.4. Simple special cases of GREG estimators Simple ratio and regression estimators have a long history of use in area estimation problems (Gallego, 2004, Carfagna and Gallego, 2005). Early applications of these estimators focused on estimating the area of crops (Heydorn, 1984, Chhikara et al., 1986, May et al., 1986), and more recently they have been used to estimate the area of land-cover change (e.g., Stehman et al., 2003, 2005, Hansen et al., 2008, Potapov et al., 2008). For these ratio and regression models, the auxiliary variables are typically continuous (e.g. area of crop type or area of land-cover change), and the sampling design is simple random or stratified random. The GREG estimator provides the generalization of these simpler ratio and regression estimators of area to other sampling designs and more complex models. 4.5. Other potential area estimators For completeness, several additional estimators potentially applicable to the area estimation problem are briefly mentioned. Survey sampling calibration estimators (Deville & Särndal, 1992; Theberge, 1999) represent another class of estimators incorporating auxiliary information to improve precision. The fundamental premise of calibration estimation is to use the auxiliary variables to construct a weight for each sampled segment such that the sample estimate of a population mean or total of an auxiliary variable matches the known population value. It is assumed that when these same weights are incorporated in a sample-based estimator of the mean or total of the target response (e.g. area), precision of the calibration estimator will be improved over an estimator not using the auxiliary information. The GREG estimator is a special case of a calibration estimator when a chi-square distance function is used in the calibration estimator (Deville & Särndal, 1992). Baffetta et al. (2009) developed a nearest neighbor estimator that could be applied to estimate area when either a categorical or continuous response is defined on the segments. Lastly, Gallego (2004) noted the importance of small-area estimation methods when the estimation objective targets a small region and the sample size is insufficient to yield a precise area estimate. Small-area estimators are typically viewed in a model-dependent (also sometimes called “model-based”) inference framework because the inference is dependent on the model specified. 5. Sampling design for simultaneously assessing map accuracy and estimating area The objectives of area estimation and map accuracy assessment may be addressed simultaneously with the same reference sample data applicable to both purposes. The features common to both objectives are the sample information regarding the reference condition, which provides the basis for both the area and accuracy estimates, and the potential availability of complete coverage auxiliary information that may be useful for improving the precision of the area estimates. The confusion matrix area estimators emerge from the common perspective that sample data collected for the primary objective of assessing accuracy can serve a secondary purpose for area estimation. The reverse perspective, although less widely held, is that sample data collected for the primary objective of area estimation also have utility for assessing map accuracy. Recognizing the dual-purpose use of the sample of reference data for both accuracy assessment and area estimation objectives greatly enhances the opportunity to extract valuable information from these data. The area estimation objective is prominently in evidence as sample-based approaches taking advantage of classified satellite imagery for land-cover monitoring have become more common. For example, in the United States Geological Survey's Land Cover Trends project (Loveland et al., 2002), a sampling approach is used to estimate changes in land cover in the United States between 1973 and

2000. Forest change is a particularly active focus of recent application of sample-based methods to estimate area (e.g., Achard et al., 2002; Leckie et al., 2002; Mayaux et al., 2005; Corona et al., 2007a,b; Hansen et al., 2008; Potapov et al., 2008; Duveiller et al., 2008; Dymond et al., 2008), and sampling of satellite imagery is an important component of the forest change monitoring conducted by the United Nations' Food and Agriculture Organization (FAO 1996). In each of these studies area estimation is the primary objective, but the sample data obtained for the purpose of estimating area could also provide the reference data for an accuracy assessment of land-cover or land-cover change maps. For example, Hansen et al. (2008) and Potapov et al. (2008) interpreted Landsat imagery to map forest clearing for a sample of 18.5-km × 18.5-km blocks, and these data were the basis of the area estimates of forest cover loss. Because Landsat imagery is often used as reference data to assess the accuracy of land-cover products derived from coarser resolution imagery, the sample of Landsat-based forest cover loss data could be used to assess the accuracy of various MODIS forest change products developed for the same regions. The land-cover change data collected by Loveland et al. (2002) are obtained by intensive manual interpretation of sample blocks using available satellite imagery and aerial photography for each date. These sample blocks may be a source of reference landcover data to assess the accuracy of the National Land-Cover Data (NLCD) of the United States (Vogelmann et al., 2001; Homer et al., 2007). Obviously several conditions must be met to ensure that the sample data obtained for an area estimation objective can be appropriately used as reference data for an accuracy assessment. The reference data must be such that it is possible to match each sample location with a map location, a common classification legend must exist, and the dates of the map and reference data must be compatible. Because sampling designs implemented for the objective of area estimation are typically not chosen to address accuracy assessment objectives, such designs would not necessarily serve well the objectives of accuracy assessment. A more effective approach would be to plan the sampling design to accommodate the simultaneous use of the sample data for both area estimation and accuracy assessment objectives. The question of which sampling designs would be jointly advantageous for both objectives has not been addressed as most effort has been invested in evaluating sampling designs targeting each of these objectives separately. Sampling designs for accuracy assessment objectives have been extensively studied (Congalton & Green 1999; Stehman & Czaplewski 1998; Stehman, 1999, 2001, 2009), and considerable exploration has been invested to determine effectiveness of sampling designs for estimating area (Corona 2007a,b; Czaplewski 2003; Gallego 2005; Levy & Milne 2004; Stehman et al., 2003, 2005; Tomppo et al., 2002). But little effort has been invested in evaluating sampling designs regarding their efficacy for merging the area estimation and accuracy assessment objectives. Czaplewski and Catts (1992) explored the sample sizes required to achieve precise estimates when the direct confusion matrix area estimator was employed, but comparing simple random to other sampling designs was not within the scope of their work. The fact that the confusion matrix area estimators have generally been studied only for simple random sampling demonstrates the scarcity of information pertaining to sampling design comparisons. Fattorini et al. (2004, Section 5) provide a rare example in which accuracy assessment and area estimation are linked as they point out that their two-phase sampling scheme, designed to estimate the area of k land-cover classes, could also be used to estimate a confusion matrix for accuracy assessment purposes. The choice of sampling unit (e.g. pixel versus cluster), whether to use strata and which strata to use, and which designs yield acceptable precision for both the accuracy estimates and the area estimates are key considerations to address when evaluating sampling designs effective for both accuracy assessment and area estimation objectives.

S.V. Stehman / Remote Sensing of Environment 113 (2009) 2455–2462

Choosing the sampling unit for such a dual-purpose sampling design will require resolving differences engrained in established practice. At present, the two general approaches to area estimation, confusion matrix and regression estimators, are generally associated with different sampling units and sampling designs. Because the confusion matrix area estimators arise from an accuracy assessment orientation, this approach is typically associated with a pixel-based sampling design. In contrast, the regression estimators are generally associated with sampling designs that employ a larger areal segment as the primary sampling unit, and thus these designs would be viewed as cluster sampling designs for purposes of accuracy assessment (i.e. clusters of pixels). Segments (primary sampling units) employed in area estimation include an 18.5-km × 18.5-km block (Hansen et al., 2008; Potapov et al., 2008), a 10-km × 10-km block (Loveland et al., 2002; Mayaux et al., 2005; Duveiller et al., 2008), a 2-km × 2-km block (Leckie et al., 2002), Landsat scenes or quarter scenes (Achard et al., 2002), and a 35.6 ha (88 acre) rectangular segment (Bauer et al., 1994). Cluster sampling is sometimes viewed with trepidation for accuracy assessment applications because of the concern that large clusters do not yield precise estimates on a per pixel cost basis (cf. Congalton & Green 1999, p. 23). However, the common use of clusters for area estimation objectives highlights the practical utility of clusters for collecting a large quantity of reference data relatively inexpensively. Because there will likely be high interest in class-specific accuracy and in estimating area by class, the sampling design must be constructed to enhance the precision of these class-specific estimates. Rare classes play a key role in both accuracy assessment and area estimation. For example, accuracy assessments typically must be able to address the question of how well rare land-cover classes or rare land-cover changes are mapped. Similarly, area estimates may be highly desirable for one or more rare land-cover types or land-cover changes (e.g. deforestation). Thus stratified sampling becomes prominent in the design considerations as the sampling design must be constructed to increase the sample size of rare classes or conditions (e.g. land-cover change). Stratification options that are effective for both area estimation and accuracy assessment objectives will need to be developed. If the sampling unit is a pixel, it is straightforward to use an existing map to stratify by the map classes. This stratification works well for estimating class-specific accuracy, and it would seem to be a logical approach to obtain precise estimates of area for rare classes. Fattorini et al.'s (2004) results are promising in this regard because stratification by map land-cover class, which would be favorable for the objective of estimating class-specific accuracy in an accuracy assessment, led to improved precision of area estimates relative to simple random sampling. If the sampling unit (segment) is larger than a pixel, choosing strata becomes more complex because these segments will often contain a mix of different classes. Rules for defining strata and assigning segments to strata will need to account for this mixedclass character of the segments. For example, strata could be defined based on the majority class present in each segment, the diversity of the class composition of the segment, or the presence of one or more rare classes within the segment. It is not clear how these different stratification options affect precision of class-specific accuracy or area estimates. To support rigorous design-based inference, accuracy assessments should be based on probability sampling designs (Stehman, 2001). If these reference sample data are also going to be used for area estimation, adhering to a probability sampling protocol is more strongly motivated by the need for the area estimates to be statistically rigorous. Questionable practices violating the protocol of probability sampling that are sometimes employed in accuracy assessment such as restricting the sample to conveniently accessible locations or to areas of homogeneous land cover would not be acceptable for the area estimation objective because of the risk of significant bias.

2461

6. Conclusions Accuracy assessment and area estimation share common underlying structures that can be used to advantage to choose a precise area estimator and to construct a sampling design that provides data for both objectives. Model-assisted estimation provides a framework unifying the area estimators commonly used in practice. The generalized regression (GREG) estimator can be employed with any probability sampling design and it allows great flexibility to specify different models of the relationship between the reference class or reference area and auxiliary variables related to these reference data. Because the GREG estimator is approximately unbiased and variance and variance estimator formulas exist for the GREG estimator, the theoretical support is in place for practical application of the GREG estimator to area estimation. The GREG estimator includes as special cases both the direct confusion matrix area estimator (Card, 1982; Czaplewski, & Catts, 1992) that has commonly been used when area estimation is combined with accuracy assessment and the regression estimators that have historically been used in area estimation studies that have no accuracy assessment component (Gallego, 2004). The benefit of the model-assisted estimation perspective is that it reveals a much broader variety of potential area estimators and extends their utility to any probability sampling design, not just simple random and stratified random sampling. In terms of sampling design, the reference data collected for assessing accuracy can also be used to estimate area. Recognizing this dual value of the sample data offers a golden opportunity to maximize the utility of these reference data, which are typically very expensive to collect. An important direction for research is to develop or identify cost-effective sampling designs that will provide reference data that have high value for both area estimation and accuracy assessment objectives. Acknowledgments Partial funding for this research was provided by an Intergovernmental Personnel Agreement between the U. S. Geological Survey Earth Resources Observation and Science (EROS) Center and SUNY ESF. The suggestions by three anonymous reviewers led to improvements in the manuscript and their efforts are much appreciated. References Achard, F., Eva, H., Stibig, H. J., Mayaux, P., Gallego, J., Richards, T., & Malingreau, J. P. (2002). Determination of deforestation rates of the world’s humid tropical forests. Science, 297, 999−1002. Baffetta, F., Fattorini, L., Franceschi, S., & Corona, P. (2009). Design-based approach to k-nearest neighbors technique for coupling field and remotely sensed data in forest surveys. Remote Sensing of Environment, 113, 463−475. Bauer, M. E., Burk, T. E., Ek, A. R., Coppin, P. R., Lime, S. D., Walsh, T. A., Walters, D. K., Befort, W., & Heinzen, D. F. (1994). Satellite inventory of Minnesota forest resources. Photogrammetric Engineering & Remote Sensing, 60, 287−298. Buckland, S. T., & Elston, D. A. (1994). Use of groundtruth data to correct land cover area estimates from remotely sensed data. International Journal of Remote Sensing, 15, 1273−1282. Card, D. H. (1982). Using known map category marginal frequencies to improve estimates of thematic map accuracy. Photogrammetric Engineering & Remote Sensing, 48, 431−439. Carfagna, E., & Gallego, F. J. (2005). Using remote sensing for agricultural statistics. International Statistical Review, 73, 389−404. Chhikara, R. S., Lundgren, J. C., & Houston, A. G. (1986). Crop acreage estimation using a Landsat-based estimator as an auxiliary variable. IEEE Transactions on Geoscience and Remote Sensing, 24, 157−168. Cochran, W. G. (1977). Sampling techniques, 3rd ed. New York, NY: John Wiley & Sons. Congalton, R. G., & Green, K. (1999). Assessing the accuracy of remotely sensed data: Principles and practices. Boca Raton, FL: CRC Press. Corona, P., Fattorini, L., Chirici, G., Valentini, R., & Marchetti, M. (2007). Estimating forest area at the year 1990 by two-phase sampling on historical remotely sensed imagery in Italy. Journal of Forest Research, 12, 8−13. Corona, P., Fattorini, L., & Pompei, E. (2007). Aerial assessment of landscape net change by means of two-phase network sampling: An application to central Italy. Environmetrics, 18, 205−215. Czaplewski, R. L. (1992). Misclassification bias in areal estimates. Photogrammetric Engineering & Remote Sensing, 58, 189−192.

2462

S.V. Stehman / Remote Sensing of Environment 113 (2009) 2455–2462

Czaplewski, R. L. (2003). Can a sample of Landsat sensor scenes reliably estimate the global extent of tropical deforestation? International Journal of Remote Sensing, 24, 1409−1412. Czaplewski, R. L., & Catts, G. P. (1992). Calibration of remotely sensed proportion or area estimates for misclassification error. Remote Sensing of Environment, 39, 29−43. de Gruijter, J. J., & Ter Braak, C. J. F. (1990). Model-free estimation from spatial samples: A reappraisal of classical sampling theory. Mathematical Geology, 22, 407−415. Deville, J. C., & Särndal, C. E. (1992). Calibration estimators in survey sampling. Journal of the American Statistical Association, 87, 376−382. Deville, J. C., Särndal, C. E., & Sautory, O. (1993). Generalized raking procedures in survey sampling. Journal of the American Statistical Association, 88, 1013−1020. Duveiller, G., Defourny, P., Descleé, B., & Mayaux, P. (2008). Deforestation in Central Africa: Estimates at regional, national and landscape levels by advanced processing of systematically-distributed Landsat extracts. Remote Sensing of Environment, 112, 1969−1981. Dymond, J. R., Shepherd, J. D., Arnold, G. C., & Trotter, C. M. (2008). Estimating area of forest change by random sampling of change strata mapped using satellite imagery. Forest Science, 54, 475−480. FAO (1996). Forest Resources Assessment 1990: Survey of tropical forest cover and study of change processes. FAO Forestry Paper 130 Rome: FAO. Fattorini, L., Marcheselli, M., & Pisani, C. (2004). Two-phase estimation of coverages with second-phase corrections. Environmetrics, 15, 357−368. Firth, D., & Bennett, K. E. (1998). Robust models in probability sampling. Journal of the Royal Statistical Society Series B, 60, 3−21. Foody, G. M. (2002). Status of land cover classification accuracy assessment. Remote Sensing of Environment, 80, 185−201. Gallego, F. J. (2004). Remote sensing and land cover area estimation. International Journal of Remote Sensing, 25, 3019−3047. Gallego, F. J. (2005). Stratified sampling of satellite images with a systematic grid of points. ISPRS Journal of Photogrammetry & Remote Sensing, 59, 369−376. Greenpeace International (2006). Roadmap to recovery: The world's last intact forest landscapes. Amsterdam: Greenpeace International. Gregoire, T. G. (1998). Design-based and model-based inference in survey sampling: Appreciating the difference. Canadian Journal of Forest Research, 28, 1429−1447. Hansen, M. C., Stehman, S. V., Potapov, P. V., Loveland, T. R., Townshend, J. R. G., Pittman, K. W., Stolle, F., Steininger, M. K., Carrol, M., & DiMiceli, C. (2008). Humid tropical forest clearing from 2000 to 2005 quantified using multi-temporal and multiresolution remotely sensed data. Proceedings of the National Academy of Sciences, 105, 9439−9444. Heydorn, R. P. (1984). Using satellite remotely sensed data to estimate crop proportions. Communications in Statistics — Theory and Methods, 13, 2881−2903. Homer, C., Dewitz, J., Fry, J., Coan, M., Hossain, N., Larson, C., Herold, N., McKerrow, A., Van Driel, J. N., & Wickham, J. D. (2007). Completion of the 2001 National Land Cover Database for the conterminous United States. Photogrammetric Engineering & Remote Sensing, 73, 337−341. Leckie, D. G., Gillis, M. D., & Wulder, M. A. (2002). Deforestation estimation for Canada under the Kyoto Protocol: A design study. Canadian Journal of Remote Sensing, 28, 672−678. Levy, P. E., & Milne, R. (2004). Estimation of deforestation rates in Great Britain. Forestry, 77, 9−16. Lohr, S. L. (1999). Sampling: Design and analysis. New York, NY: Duxbury Press. Loveland, T. R., Sohl, T. L., Stehman, S. V., Gallant, A. L., Sayler, K. L., & Napton, D. E. (2002). A strategy for estimating rates of recent United States land cover changes. Photogrammetric Engineering & Remote Sensing, 68, 1091−1099. May, G. A., Holko, M. L., & Jones, N., Jr (1986). Landsat large-area estimates for land cover. IEEE Transactions on Geoscience and Remote Sensing, GE-24, 175−184. Mayaux, P., Holmgren, P., Achard, F., Eva, H., Stibig, H. J., & Branthomme, A. (2005). Tropical forest cover change in the 1990s and options for future monitoring. Philosophical Transactions of the Royal Society B, 360, 373−384. McRoberts, R. E., Wendt, D. G., Nelson, M. D., & Hansen, M. H. (2002). Using a land cover classification based on satellite imagery to improve the precision of forest inventory area estimates. Remote Sensing of Environment, 81, 36−44.

McRoberts, R. E., Holden, G. R., Nelson, M. D., Liknes, G. C., & Gormanson, D. D. (2005). Using satellite imagery as ancillary data for increasing the precision of estimates for the Forest Inventory and Analysis program of the USDA Forest Service. Canadian Journal of Forest Research, 35, 2968−2980. Opsomer, J. D., Breidt, F. J., Moisen, G. G., & Kauermann, G. (2007). Model-assisted estimation of forest resources with generalized additive models. Journal of the American Statistical Association, 102, 400−409. Potapov, P., Hansen, M. C., Stehman, S. V., Loveland, T. R., & Pittman, K. (2008). Combining MODIS and Landsat imagery to estimate and map boreal forest cover loss. Remote Sensing of Environment, 112, 3708−3719. Särndal, C. E., Swensson, B., & Wretman, J. (1992). Model-assisted survey sampling. New York, NY: Springer-Verlag. Scheaffer, R. L., Mendenhall, W., & Ott, R. L. (2006). Elementary survey sampling, (6th edn), Belmont, CA: Duxbury Press. Stehman, S. V. (1999). Basic probability sampling designs for thematic map accuracy assessment. International Journal of Remote Sensing, 20, 2423−2441. Stehman, S. V. (2000). Practical implications of design-based sampling inference for thematic map accuracy assessment. Remote Sensing of Environment, 72, 35−45. Stehman, S. V. (2001). Statistical rigor and practical utility in thematic map accuracy assessment. Photogrammetric Engineering & Remote Sensing, 67, 727−734. Stehman, S. V. (2005). Comparing estimators of gross change derived from complete coverage mapping versus statistical sampling of remotely sensed data. Remote Sensing of Environment, 96, 466−474. Stehman, S. V. (2009). Sampling designs for accuracy assessment of land cover. International Journal of Remote Sensing, 30. Stehman, S. V., & Czaplewski, R. L. (1998). Design and analysis for thematic map accuracy assessment: Fundamental principles. Remote Sensing of Environment, 64, 331−344. Stehman, S. V., & Foody, G. M. (2009). Accuracy assessment. In T. A. Warner, M. D. Nellis, & G. M. Foody (Eds.), The SAGE Handbook of Remote Sensing (pp. 297−309). New York: Sage Publications. Stehman, S. V., Sohl, T. L., & Loveland, T. R. (2003). Statistical sampling to characterize land-cover change in the U. S. Geological Survey land-cover trends project. Remote Sensing of Environment, 86, 517−529. Stehman, S. V., Sohl, T. L., & Loveland, T. R. (2005). An evaluation of sampling strategies to improve precision of estimates of gross change in land use and land cover. International Journal of Remote Sensing, 26, 4941−4957. Strahler, A. H., Boschetti, L., Foody, G. M., Friedl, M. A., Hansen, M. C., Herold, M., Mayaux, P., Morisette, J. T., Stehman, S. V., & Woodcock, C. E. (2006). Global land cover validation: Recommendations for evaluation and accuracy assessment of global land cover maps, EUR 22156 EN - DG. Luxembourg: Office for Official Publications of the European Communities. Theberge, A. (1999). Extensions of calibration estimators in survey sampling. Journal of the American Statistical Association, 94, 635−644. Tomppo, E., Czaplewski, R., & Mäkisara, K. (2002). The role of remote sensing in global forest assessments. Forest Resource Assessment Programme, Working Paper, vol. 61. Rome: FAO. Vogelmann, J. E., Howard, S. M., Yang, L., Larson, C. R., Wylie, B. K., & Van Driel, N. (2001). Completion of the 1990s national land cover data set for the conterminous United States from Landsat Thematic mapper data and ancillary data sources. Photogrammetric Engineering & Remote Sensing, 67, 650−662. Walsh, T. A., & Burk, T. E. (1993). Calibration of satellite classifications of land area. Remote Sensing of Environment, 46, 281−290. Wu, C., & Sitter, R. R. (2001). A model-calibration approach to using complete auxiliary information from survey data. Journal of the American Statistical Association, 96, 185−193. Yuan, D. (1996). Natural constraints for inverse area estimate corrections. Photogrammetric Engineering & Remote Sensing, 62, 413−417. Yuan, D. (1997). A simulation comparison of three marginal area estimators for image classification. Photogrammetric Engineering & Remote Sensing, 63, 385−392.