PE&RS January 2016 - page 27

in under-segmentation errors. The computation indicates that
our technique decreased the fragment rate from 7.4 percent
to 0.9 percent, and the mixing rate from 14.8 percent to 0.9
percent.
Performance Evaluation
Besides of the above simple comparison, the evaluation of the
performance in a more strict and comprehensive way is need-
ed. There is substantial literature discussing of the evaluation
of the building extraction results. In the literature of Aw-
rangjeb et al. (2010), the authors proposed a comprehensive
set of evaluation metrics, including two types of evaluation
systems about the accuracy of the building footprint extrac-
tion, the pixel-based (Song et al., 2005) and the object-based.
The object-based evaluation technique takes the whole roof
as a unit and can measure the result in the sense of matching
number and gives the evaluation of the extraction rate, while
the pixel-based evaluation technique can respond to both the
shape and direction accuracy of the extracted buildings.
We adopted a one-to-one correspondence based evaluation
system. The reference data were created by manual operation.
The metrics are listed in the left column of Table 1, Table 2,
and Table 3. They were computed from true positive (TP),
false positive (FP), true negative (TN), and false negative (FN).
For one-to-one correspondence, TP equals TN. Roof-level met-
rics are for object-based evaluation system, while pixel-level
metrics are for pixel-based evaluation system. Completeness
Cm and correctness Cr (Pfeifei et al., 2007), respectively, are
also referred to as user’s accuracy and producer’s accuracy.
Qualification Ql (Rutzinger et al., 2009) measures the accu-
racy considering both directions. Detection cross-lap Crd and
reference cross-lap Crr (Awrangjeb et al., 2010) respectively
evaluates the under- and over-segmentation situations. Crd is
the percentage of the extracted boundaries that overlap more
than one reference boundary. Crr is on the contrary. Area
omission error Aoe and area commission error Ace (Aw-
rangjeb et al., 2014; Song et al., 2005) represent the areas of
successfully and unsuccessfully detected buildings respec-
tively in terms of pixel. The root-mean-square error RMSExy
(Song et al., 2005) reflects the geometric accuracy in planime-
try. All the metrics are percentage or ratio thus have no units,
except for RMSExy, which has the unit of pixel.
All the three data sets were used for the evaluation and the
evaluation results were listed in Table 1, Table 2, and Table 3
(a)
(b)
(c)
Figure 8. Comparison with other segmentation methods: (a)
proposed method, (b)general marker-watershed method, and (c)
Pesarasi’s method
T able 1. Quality Evaluation for Dataset 1
Roof-level Pixel-level/ Geometric
Cm: 0.9204 Cmp: 0.7650 RMSExy:8.4395 pixels
Cr: 0.9811 Crp: 0.8600
Ql: 0.9043 Qlp: 0.6803
Crd rate 0.0472 Aoe: 0.2350
Crr rate: 0.0088 Ace: 0.1628
T able 2. Quality Evaluation for Dataset 2
Roof-level Pixel-level Geometric
Cm: 0.9147 Cmp: 0.8014 RMSExy:2.337 pixels
Cr: 0.9415 Crp: 0.8254
Ql: 0.8655 Qlp: 0.6376
Crd rate: 0.0183 Aoe: 0.2630
Crr rate: 0.0355 Ace: 0.2115
T able 3. Quality Evaluation for Dataset 3
Roof-level Pixel-level Geometric
Cm: 0.7971 Cmp: 0.8444 RMSExy:5.2509 pixels
Cr: 0.9483 Crp: 0.9329
Ql: 0.7639 Qlp: 0.7961
Crd rate: 0.0345 Aoe: 0.1556
Crr rate: 0.0290 Ace: 0.0719
PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING January 2016 27