PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING
October 2016
789
Modeling the Effects of Horizontal Positional
Error on Classification Accuracy Statistics
Henry B. Glick, Devin Routh, Charlie Bettigole, Lindsi Seegmiller, Catherine Kuhn, and Chadwick D. Oliver
Abstract
Using a concept proposed by Stehman and Czaplewski
(1997), we implemented spatially-explicit Monte Carlo simu-
lations to test the effects of manually introduced horizontal
positional error on standard inter-rater statistics derived
from twelve classified high-resolution images. Through
simulations we found that both overall and kappa accuracies
decrease markedly with increasing error distance, varying
greatly across distances relevant to practical application. The
use of ground reference sites falling solely in homogeneous
patches significantly improves inter-rater statistics and calls
into question the use of kernel-smoothed data in one-time
accuracy assessments. Our simulations offer insight into the
scale of both structural and cover type heterogeneity across
our landscapes, and support a new method for minimizing
the effects of positional error on map accuracy. We recom-
mend that analysts use caution when applying traditional
accuracy assessment strategies to categorical maps, par-
ticularly when working with high-resolution imagery.
Introduction
The assessment of thematic map accuracy, defined as the de-
gree to which classified or categorical mapped feature labels
correspond with true feature labels, has been central to the
field of remote sensing for approximately 40 years (Congalton
and Green, 2009). During this time, practitioners have ad-
opted increasingly rigorous assessment strategies. The use of
error matrices and inter-rater agreement statistics (e.g., overall
accuracy, Cohen’s (1960) kappa coefficient, errors of omission
and commission, user’s and producer’s accuracies) has been
common in recent decades (Congalton, 1994; Congalton and
Mead, 1983; Congalton
et al
., 1983; Liu et
al
., 2007; Story and
Congalton, 1983), although the remote sensing community
has not reached consensus on general standards, statistical
reporting, or target accuracies (Foody, 2002; Liu
et al
., 2007;
Stehman, 1997). That thematic accuracy assessment has not
been standardized in the way that positional accuracy assess-
ment has (i.e., ASPRS, 2015; FGDC, 1998) may be due to this
lack of consensus, its shorter history (40 years versus over 70
years; Congalton and Green, 2009), and/or the rapidity with
which remote sensing technology has changed (Campbell and
Wynne, 2011; Graham, 1999). As the resolutions (i.e., spatial,
spectral, radiometric, and temporal) of remotely sensed data
sets become continually finer, we face the ongoing challenge
of how best to evaluate our spatial models.
Error-matrix based accuracy assessment relies on the ability
to relate known information from ground reference sites to the
predicted information for those sites. However, for the compar-
ison to take place, the locations must be described in theoret-
ical space using coordinate systems (usually a function of
GPS
or survey-based geometry), datums, and geospatial projections
that rely on transformation functions and generalized models
of the Earth’s shape. If we were to trust that two datasets with
the same coordinate systems and no positional error were per-
fectly co-registered (a questionable assumption given satellite
orbiting speeds, changes in satellite altitude, and rounding
error, among others) then we could be confident that our
comparison of the datasets makes sense at all spatial scales.
Unfortunately, we can rarely, if ever, satisfy these conditions.
The most spatially limiting sampling unit size occurs when
treating the imagery as point-sampled data in which each
point has the spatial extent of a single pixel (Janssen and van
der Wel, 1994), and where a single pixel is used to represent a
ground control plot. To accurately compare ground and image
data using this format, one needs a portable Global Position-
ing System (
GPS
) receiver and a georeferenced image whose
combined root mean squared horizontal georeferencing error
is less than one-half the length of the shortest side of a pixel.
These hypothetical circumstances are seldom achievable.
Many analysts often ignore one form of georeferencing error,
limiting registration hurdles to either (a) obtaining a highly
accurate image, or (b) using a
GPS
receiver whose horizontal
error is less than one-half the length of the shortest side of a
pixel (Weber, 2006). Recreational-grade
GPS
technology has
improved markedly since its early years, and many afford-
able units can now achieve locational accuracies of ±5 m or
less. For moderate to coarse resolution datasets (e.g., Landsat,
EO-1, MODIS), such
GPS
units work well. However, as Weber
(2006) points out, they are generally insufficient to assess the
accuracy of high spatial resolution datasets (here defined as
imagery with a ground resolved distance of 5 m or less), and
failure to understand
GPS
unit receiver-specific accuracy and
error propagation may lead analysts astray.
High spatial resolution satellite imagery has been avail-
able on the commercial market for roughly 15 years. Though
it offers a unique and often alluring level of detail, as image
resolution increases the analyst must contend with poten-
tially less accurate co-registration between image and ground
reference sites, as well as a concomitant decrease in class
separability that may reduce classification accuracy (Carleer
and Wolff, 2005). When measured as a function of linear
Henry B. Glick, Devin Routh, Charlie Bettigole, and Chadwick
D. Oliver are with the Ucross High Plains Stewardship Initia-
tive, Yale School of Forestry and Environmental Studies, 195
Prospect Street, New Haven, CT, 06511, USA (henry.glick@
yale.edu;
).
Lindsi Seegmiller is an independent geospatial consultant,
3319 North Stone Creek Circle, Madison, WI, 53719; and
formerly with the Ucross High Plains Stewardship Initiative,
Yale School of Forestry and Environmental Studies, Yale
University.
Catherine Kuhn is with the School of Environmental and For-
est Sciences, University of Washington, Box 325100, Seattle,
WA, 98195; and formerly with the Ucross High Plains Stew-
ardship Initiative, Yale School of Forestry and Environmental
Studies, Yale University.
Photogrammetric Engineering & Remote Sensing
Vol. 82, No. 10, October 2016, pp. 789–802.
0099-1112/16/789–802
© 2016 American Society for Photogrammetry
and Remote Sensing
doi: 10.14358/PERS.82.10.789
