assess image segmentation quality. A common way of apply-
ing these methods it to define a reference dataset consisting of
reference polygons spread across the area of interest and com-
pare the geometric match between them and the correspond-
ing objects obtained by the segmentation under evaluation
(Möller
et al
., 2007; Whiteside
et al.
, 2014). The segmentation
quality over the reference dataset is assumed to be representa-
tive of the whole segmentation output quality.
The geometric focus of the typical supervised methods,
however, limits their capability to include the user’s perspec-
tive, especially with regard to the severity of thematic errors.
That is, the severity of segmentation errors depends on the
classes involved in the classification since individual users
may vary greatly in their sensitivity to thematic misclassifica-
tions. In spite of this, the traditional methods used in remote
sensing to assess image segmentation quality are geometric-
only (Clinton
et al.
, 2010; Whiteside
et al.
, 2014).
In this paper, it is argued that a more appropriate basis on
which to evaluate the quality of image segmentations is to con-
sider their geometric quality as well as thematic content, that
is, the class composition of the objects. This is possible to im-
plement within a supervised framework, which uses a dataset
as reference. The land cover classes associated with a reference
dataset are typically known as a supervised image classification
in which class labels are included in the reference dataset is
used, and this class information may also be used in the assess-
ment of image segmentation quality. As a result, segmentation
errors can be weighted as a function of the classes involved and
the perspective of the map user taken into account.
A new approach to assess image segmentation quality is
proposed which combines a traditional geometric-only meth-
od with a thematic method. The latter requires the reference
dataset to include land cover information associated with the
objects and the user to express explicitly and quantitatively
their sensitivity to misclassification errors. This information
allows the measurement of the geometric and the thematic
quality of the objects to be evaluated. To demonstrate the po-
tential of the proposed approach, the geometric-only method
of Möller
et al.
(2013) and the thematic similarity index
(Costa and Foody, 2013) are combined to provide a means to
evaluate objects based on their geometric and thematic prop-
erties from the perspective of specific users.
Methods
The proposed approach to assess the quality of image seg-
mentations is based on a traditional geometric-only method
and includes user specific information on thematic misclas-
sification severity. In essence, the geometric errors committed
in segmentation are weighted as a function of the thematic
classes mixed in the objects defined by the segmentation and
the relative importance of the misclassification to the user.
The method of Möller
et al.
(2013) was selected to assess
the geometric match between the segmentation under evalu-
ation and a reference dataset. Although this method is fully
described in Möller
et al.
(2013), some salient details are
repeated in this paper together with a discussion of a refine-
ment made to it.
The thematic similarity index (
TSI
) was designed to aid the
assessment of the thematic quality of objects. The
TSI
is based
on the thematic content of the objects delimited in a segmen-
tation, and the thematic quality of the objects is calculated ac-
cording to the user’s perspective. For example, if an object is
under-segmented and hence mixing two classes defined by the
user as thematically similar, the thematic quality of such ob-
ject is not substantially damaged for that user. Had the classes
mixed been of very different value to the user, however, the
thematic quality of the object would have been reduced. By
combining the assessment of geometric and thematic quality
an integrated assessment of image segmentation quality is ob-
tained. An overview of the methodology is shown in Figure 1.
Figure 1. Geometric-thematic approach for image segmentation
quality assessment.
Möller
et al.’s
(2013) Method
The method proposed by Möller
et al.
(2013) starts by
overlaying the segmentation under evaluation with a refer-
ence dataset to perform an operation of spatial intersection
(commonly found in geographical information systems), from
which new objects are created (Figure 2a). Geometric met-
rics are then calculated for the new objects to measure both
their overlap and position in relation to the reference and the
segmentation.
Using Möller
et al.
’s (2013) terminology, the intersection
of a reference polygon R with an object F creates object S (S
= R
Ç
F). Also objects R* are created, if any, which correspond
to the area of R that does not intersect with F (R* = R
Ç
¬F).
Similarly, F* is defined too (F* – R
Ç
¬ F) (Figure 2a).
The area of R, F, and S (A
R
, A
F
, and A
S
, respectively) are
used to calculate the metrics O
R
and O
F
as O
X
= A
S
/A
X
, with
X
∈
[R,F]. Metrics O
R
and O
F
calculate the overlap of S in
relation to R and F, respectively. If an object F is geometri-
cally similar to its reference, then the overlap of S with R and
F will be high as well as the value of O
R
and O
F
. Over-seg-
mented and under-segmented objects will correspond to low
values of O
R
and O
F
, respectively. The value of these metrics
ranges between 0 and 1.
A complementary way to assess the geometric match
between shapes is to consider their relative positions, which
can be measured based on the distance between centroids
(Figure 2b and 2c). If an object F is geometrically similar to its
reference, then the centroid of S and that of R and F will lay
close to each other. The centroids of R, F, and S (c
R
, c
F
, and c
S
,
respectively) are used to calculate the metrics P
R
and P
F
:
P
dist c c
dist c c
X
S X
S X* max
= −
(
)
(
)
1
,
,
,
with X
∈
[R,F]
(1)
These metrics use a normalization factor (the ratio’s de-
nominator) which is the distance between c
S
and the further-
most centroid of either R* (c
R*,max
) or F* (c
F*,max
). Thus, these
metrics also range between 0 and 1. Over-segmented and
under-segmented objects will correspond to low values of P
R
and P
F
, respectively.
452
June 2015
PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING
