indexes cannot be applied to one-to-many or many-to-one
matching relationships.
Arithmetic Discrepancy Indexes
The indexes based on arithmetic discrepancy evaluate the
segmentation result by measuring the arithmetic discrepancy
between the reference objects and the corresponding objects.
The arithmetic relationship between objects includes one-to-
many, one-to-one, and many-to-one (Figure 2b). The above
three arithmetic relationships have been well-defined and
explained in the literature (Liu
et al
., 2012). However, the
arithmetic relationship between all reference objects and its
corresponding object should be one-to-one in an ideal seg-
mentation result.
Fragmentation-Based Indexes
Strasters and Gerbrands (1991) proposed the Fragmentation
(F
RA
G)
index:
FRAG
p m v
q
=
+
−
1
1
.
(27)
where
m
and
v
are the number of the reference objects and
the corresponding objects.
p
and
q
are scale parameters which
should be set properly according to actual situation and ap-
plication. The range of
F
RA
G
is [0, 1], and each corresponding
object have one-to-one relationship with its reference object.
A value of 1 indicates an ideal segmentation result.
Number of Segments Ratio Indexes
Liu
et al
. (2012) defined the Number of Segments Ratio
(
NSR
)
index.
NSR
m v
m
=
−
(28)
where
m
and
v
are the number of the reference objects and
corresponding objects. When
NSR
’s value is 0, all reference
objects have a one-to-one relationship with corresponding
objects, indicating that the segmentation result is ideal. The
bigger the
NSR
value, the more many-to-one or one-to-many
quantitative relationships exist, which is caused by the
increasingly serious phenomena of over-segmentation and
under-segmentation.
Ratio Index of Good Objects and Invading Objects Number
Schöpfer and Lang (2006) described the Offspring Loyalty
(
OL
)
index and the Interference
(I)
index based on the object-fate
matching method. The better the segmentation quality, the
higher the proportion of good objects to the corresponding
object, and the lower the proportion of invading objects to
segmented objects.
OL
n
n n
Good
Good Expanding
=
+
(29)
I
n
n n
n
Invading
Good Expanding Invading
=
+
+
(30)
where
n
stands for the number of objects; the ideal value of
OL
and
I
are 1 and 0, respectively.
Pros and Cons of Arithmetic Discrepancy Indexes
Evaluating the quantitative discrepancy is as important as
evaluating the geographic discrepancy during the segmenta-
tion evaluation process. For an ideal segmentation result,
geometric discrepancies between the reference object and
the corresponding object is definitely small, whereas small
geometric discrepancies cannot guarantee ideal segmentation
results. In some extreme cases, such as when the size of
each corresponding objects is one pixel, the corresponding
objects and the reference objects will be completely over-
lapped. Thus, there will not be any under-segmentation or
over-segmentation phenomena according to the definition of
geographic discrepancy, which is obviously unreasonable.
Therefore, the indexes based on arithmetic discrepancies are
gaining academic attention, and combining these indexes
with the indexes based on area is the future of the segmenta-
tion evaluation field.
Mixed Indexes
Besides the single indexes mentioned above, comprehensive
use of various indexes for supervised evaluation of segmenta-
tion is also available by combining the above indexes into a
composite index (Table 1). Using a composite index combines
the advantages of both, and makes up for the limitations of
each index on its own, which makes the segmentation evalu-
ation process more objective and comprehensive. Also, if the
range of the index has no strict definition or is in a different
magnitude with the index it is being combined with, stan-
dardizing and normalizing processing needs to be completed
on the indexes to be combined.
Table 1. Typical mixed indexes of the supervised evaluation
method.
Index
Combination
ED1
(Clinton et al
. 2010)
Under-Segmentation index and Over-
Segmentation index
ED2
(Liu
et al
. 2012)
Under-Segmentation index and Arithmetic
index
ED3
(Yang
et al
. 2014)
Local Under-Segmentation index and
Over-Segmentation index
SEI
(Yang
et al
. 2015a)
Local Under-Segmentation index and
Over-Segmentation index
M
(Möller
et al
. 2007)
Region and Location based index
ADI
(Cheng
et al
. 2014)
Under-Segmentation index and Over
Segmentation index
Problems and Analysis of Supervised Evaluation
Compared to the subjective evaluation method, the indirect
evaluation and analytical evaluation methods previously
discussed, the supervised evaluation method overcomes the
human error to a certain extent, and provides a more accurate
evaluation result, both objectively and quantitatively. How-
ever, supervised evaluation methods require the dataset to
be built-up manually. The segmentation results derived from
the
GeOBIA
multi-scale segmentation algorithm also need to be
evaluated. Although some scholars have put forward a multi-
scale supervised evaluation method (Anders
et al
., 2011;
Dr
ă
gu
ţ
et al
., 2014; Myint et al., 2011; Trias-Sanz
et al
.,2008;
Zhang
et al
., 2015c), building the whole reference dataset for
high spatial resolution remote sensing imaging is tedious and
time consuming and involves some subjectivity (Corcoran
et
al
., 2010). Moreover, the whole multi-scale reference dataset
is hard to define by visual interpretation (Martin
et al
., 2001).
When the supervised evaluation method used in single-scale
segmentation evaluation for typical object recognition, only
the reference data of the interested object needs to be built
when the boundary is clear and will not be confused with
other objects. Overcoming the subjectivism in the refer-
ence data building process would save both time and effort.
Generally, the supervised evaluation method is more suitable
for target recognition task especially when a standard target
dataset has been available.
636
October 2018
PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING
