of segmentation algorithm being evaluated. The evaluation in-
dexes of the analytical method can be broken down into two
categories: qualitative and quantitative. The main qualitative
indexes are as follows:
Prior Knowledge
With the continuous improvement and innovation of segmen-
tation algorithms, some algorithms with prior knowledge in
specific applications have achieved better results than others.
The use of prior knowledge has a greater effect on the perfor-
mance of the segmentation algorithm, by combining various
types and amounts of prior knowledge with the principle of
segmentation algorithms. Through analysis, we can learn their
advantages and disadvantages to a certain extent, so that the
stability, reliability, and efficiency of segmentation algorithms
can be improved (Zhang 1996b). Liedtke
et al
. (1987) pro-
posed an analytical evaluation method based on the type and
amount of prior knowledge to evaluate several segmentation
algorithms, which were theoretically based on the photomet-
ric and geometric knowledge of the image to be segmented.
However, this index is not suitable for all kinds of segmenta-
tion algorithms, because of most prior knowledge is heuristic.
At present, there is no way to describe prior knowledge quan-
titatively, which leads to difficulty in comparing the different
types of prior knowledge. In addition, to consider the type
and amount of prior knowledge, knowing how to combine
prior knowledge into segmentation algorithms is also impor-
tant in improving an algorithm’s performance. In comparison,
the difficulty of the latter is higher than the former.
Theoretical Background of Segmentation Algorithm
Nowadays, there are a growing number of image segmenta-
tion algorithms with different theoretical backgrounds. An
algorithm’s theoretical characteristics will affect its perfor-
mance and scope of application. Theoretical analysis of the
segmentation algorithm requires a background of professional
knowledge and an in-depth understanding of an algorithm’s
principles and operating mechanism. Ming
et al
. (2009) evalu-
ated the maximum entropy method, split-merge method,
improved Markov random field method, and the directional
phase method qualitatively by analyzing their theoretical
characteristics, which provided theoretical support for the
subsequent quantitative experiments.
Processing Strategy
The processing strategy of a segmentation algorithm will also
directly affect its performance. Segmentation of an algorithm’s
processing strategies can be categorized into serial, parallel,
iterative, and mixed processing, which is similar to computer
graphics processing techniques. The serial strategy achieved
high segmentation precision and stability by fully consider-
ing local and global information, while it exhibited a slower
information processing. The parallel strategy dramatically
improves the speed of the segmentation calculation, but un-
like the serial strategy, it only considers local information,
which would result in poor anti-noise capability. Therefore,
the precision (anti-noise capability) and the speed of the
segmentation calculation contradict each other in the process-
ing strategy. One solution to improve the overall performance
of the algorithm is using an iterative or a mixed processing
strategy to balance the relationship between the precision and
running speed. Therefore, we can qualitatively evaluate the
efficiency of segmentation algorithms through their process-
ing strategy.
As for quantitative analysis criteria, the main indexes in-
clude: detection probability ratio (Abdou and Pratt 1979), spa-
tial complexity, time complexity, and segmentation resolution.
Quantitative analysis methods enable the obtainment of
quantitative indicators for the algorithm evaluation without
segmentation experiments.
The significance of the analytical evaluation method is to
find the substantial defects in the segmentation algorithm,
and indicate the direction of algorithm improvement, to deter-
mine which is suitable for qualitative evaluation (Zhang
et al
.,
2009), and which can only be applied to some specific models
or algorithms with certain expectations (Zhang, 1996a). In
other words, this method is only applicable to evaluate the
segmentation algorithm itself and its implementation charac-
teristics, which are usually independent of the segmentation
result. Although a series of analysis evaluation indicators
are listed above, not all features for each algorithm can be
characterized by analysis (Zhang, 2001); different evaluation
indicators can be used to analyze different segmentation al-
gorithms. Besides, quantitative and qualitative index analysis
cannot be applied to universal indexes of all types of segmen-
tation algorithms. Therefore, the analytical evaluation method
is suitable for certain applications with particular demand
and cannot applied to the comparison of different algorithms’
performance (Pal and Pal,1993), especially to the algorithms
with a large difference in an ideal and theoretical background
(Cardoso and Corte-Real, 2005).
Empirical Evaluation Method
Empirical evaluation is to quantitatively evaluate the segmen-
tation quality with some certain measurements. According
to whether referenced data are involved in the segmentation
evaluation, empirical evaluation methods can be divided into
two aspects, supervised evaluation methods and supervised
evaluation methods. These two methods are the most widely
evaluation methods in remote sensing field, for it could pro-
vide quantitative evaluation result based on practical result of
segmentation, which is vary from method and scale param-
eters selection of segmentation for high spatial remote sensing
images. The next two Sections respectively review the state of
art of supervised evaluation method (also known as empirical
discrepancy method) and unsupervised evaluation method
(also known as empirical goodness method) in detail.
Supervised Evaluation Method
The supervised evaluation method (Yang
et al
., 1995), also
known as the empirical discrepancy method (Zhang, 1996a),
was considered the optimal evaluation method (Hoover
et al
.,
1996; McCane, 1997), as it does not require subjective input
during the evaluation process. It evaluates segmentation algo-
rithms quantitatively by using discrepancy criterion between
segmentation results and manually selected segmentation ref-
erence datasets (Gold Standard Image
(
de Graaf
et al
., 1994))
and the boundary and range of the reference datasets are pre-
cisely defined. The supervised evaluation method is gradually
replacing the subjective evaluation method and is becoming a
frequently-used segmentation evaluation method.
Process of Supervised Evaluation
There are three main steps in the supervised evaluation
method: establishment of segmentation reference datasets,
objects matching, and discrepancy calculation.
Establishment of Segmentation Reference Dataset
Segmentation of the reference dataset to create a standard ref-
erence which is the optimal and ideal segmentation result for
the study area and consists of several vector polygon objects.
The reference dataset of high spatial resolution remote sens-
ing images is usually obtained through visual interpretation
or digitalized data collected in the field. In this paper, it is
expressed as a reference object set
R
= {
r
i
;
i
= 1 …
n
}. Because
the reference map needs to be chosen by manual methods,
acquisition of the reference data is a subjective and a time-
consuming task. Creating a digitalized segmentation reference
map for one image and multiple different images, especially
PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING
October 2018
631
