comprehensive evaluation algorithms or frameworks exclu-
sively for high spatial resolution remote sensing images have
not yet been proposed.
In this paper, from the point of the application of geosci-
ence, we systematically summarize the presented segmenta-
tion evaluation methods and explore its application domain
in high spatial resolution remote sensing images. Possible
future direction for improvements and propose potential ap-
plications for high spatial resolution remote sensing images
are also discussed.
Hierarchy of Existing Segmentation Evaluation System
Although research on segmentation evaluation is less ad-
vanced compared with that on segmentation algorithms,
plenty of evaluation methods with varied feasibility and
applicability have already been proposed. These evaluation
methods can be divided into the following five methodologies
(Figure 1):
Figure 1. The hierarchy of segmentation evaluation methods
according to Zhang
et al
. (2008).
Depending on whether visual interpretation by a person is
needed, segmentation evaluation methods can be classified
into the subjective evaluation methods and objective evalu-
ation methods. For the objective evaluation methods, some
evaluate the segmentation process, while others evaluate the
result and accuracy of subsequent process operations, such
as classification and target extraction. Therefore, we can
divide the objective evaluation methods into direct evalu-
ation methods and indirect evaluation methods. The direct
evaluation method evaluates the segmentation algorithm, and
also evaluates the results of segmentation. Thus, the direct
evaluation method can be further divided into analytical
evaluation methods, which are qualitative, and empirical
evaluation methods, which are quantitative. In the end, based
on whether the reference data is needed or not, empirical
evaluation methods can be divided into supervised evalua-
tion method, also known as empirical discrepancy method,
and unsupervised evaluation method, which are empirical
goodness methods.
It should be noted that the evaluation methods mentioned
above are not mutually independent and exclusive, and each
type of evaluation method has its own characteristics and
limitations. A combination comprising one or more evalu-
ation methods can be used in segmentation evaluation and
applied for different images.
Subjective Evaluation Method
The subjective evaluation method is based on human visual
judgment. Using the average score of several evaluators, sub-
jective evaluation methods have the ability to make qualita-
tive evaluations of the segmentation methods, and can also
make qualitative comparison between multiple segmentation
methods. For example, Neubert (Neubert and Meinel, 2003;
Neubert
et al
., 2006; Neubert and Herold, 2008) assessed and
compared multiple segmentation programs for high spatial
resolution remote sensing images through visual assessment.
Yang
et al
. (2015b) evaluated the automatic selection method
of segmentation scale parameters which he proposed, using
visual interpretation.
While subjective evaluation methods are the most widely
used evaluation methods nowadays (Neubert and Meinel,
2003; Neubert and Herold, 2008; Pesaresi and Benedikts-
son, 2001), this method is qualitative and certainly subjec-
tive. Each evaluator has their own evaluation criteria, so the
evaluation scores can vary significantly between evaluators
(Gelasca
et al
., 2004; Paglieroni, 2004). Specifically, the final
evaluation results will be affected by the order of observa-
tion, knowledge background of the evaluators, the evaluators’
experience, and can even be affected by the evaluators’ age
and gender. In order to reduce the impact of the factors men-
tioned above, a sufficient number of evaluators, who through
professional training, can represent normal human evaluation
criteria, needs to be chosen to compare and evaluate a large
number of representative images’ segmentation results. The
purpose of image segmentation in the field of remote sens-
ing is to achieve automatic analysis of remote images (Baatz,
2000). However, the subjective evaluation method is tedious
and time-consuming, and cannot provide quantitative evalu-
ation indexes. Therefore, the subjective evaluation method
often fails to select an applicable segmentation algorithm or
set optimal parameters for the segmentation algorithm, and
cannot be applied in real-time for automatic information
analysis and extraction (Zhang
et al
., 2008).
Indirect Evaluation Method
The indirect evaluation method is also known as the system-
level evaluation method or application evaluation method.
This method treats the segmentation process as a part of a
whole processing system, by evaluating the segmentation
method through the dialectical relationship between parts
and the whole. Researchers and system designers were able
to indirectly evaluate segmentation algorithms through the
accuracy and results of the subsequent experiments. For
example, this would be completed by evaluating the segmen-
tation result, based on the classification accuracy and object
extraction quality, while examining the classification and
object extraction as the subsequent processes in segmentation
(Dronova
et al
., 2012; Hofmann
et al
., 2015; Johnson and Xie,
2013; Laliberte and Rango, 2009; Li
et al
. 2011; Shin
et al
.,
2001; Zhang
et al
., 2015b).
However, evaluating segmentation results using this
method is not direct (Kim
et al
., 2009; Zhang
et al
., 2005);
obtaining ideal results in the subsequent process operations
does not mean that segmentation results were superior, and
vice versa
. Each segmentation algorithm has many essential
properties which are independent of the applications. In ad-
dition, the system is often based on a particular type or loca-
tion of remote sensing images and have specific application
requirements. So, the evaluation results of indirect evaluation
methods for segmentation algorithms can only explain that
the segmentation algorithm being evaluated is more suitable
to be applied to the particular system.
Analytical Evaluation Method
The analytical evaluation method is one of the direct evalua-
tion methods for segmentation results. With this method, we
can evaluate the segmentation algorithm itself without any
segmentation experiments on the experimental images. Per-
formance and applicability of the segmentation algorithm can
be obtained by analyzing the basic idea and functional theory
630
October 2018
PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING
