PE&RS December 2015 - page 907

training. For testing, there were two test sample sets, the first
set being provided by the contest committee for the whole im-
age, while the second set was collected by the authors on the
edges near the missing strips, to evaluate the performance of
the inpainting step. In the second test set, the road class was
omitted, and the bare soil class was mostly absent in the edge
parts. Details of the training and test samples for each class of
the reference data are shown in Table 3.
General Classification Performance
The different classification approaches described in Table
4 and the top-two results in the contest were selected for
comparison. Each thematic map is visually shown in Plate
1. The quantitative evaluation consisted of the classification
accuracy for each class, the overall accuracy (OA), the average
accuracy (AA), and the kappa coefficient (κ) (Congalton, 1991),
as shown in Table 5. For the spectral-spatial feature fusion
related approaches, the related parameters were selected with
consideration of the spatial distribution and resolution, as
suggested in Dalla Mura et al. (2010). In addition, the κ values
for the winners of the classification contest are given in the
bottom row of Table 5. In this table, the best results for each
quality index are highlighted in bold, and the second-best
results are underlined.
It can be observed that most of the accuracy statistics in
Table 5 suggest that the proposed method can obtain the best
or at least a desirable performance, except for the bare soil
pixels, which are often mixed with grass and tree pixels in
the study area. In Plate 1d through 1i), the classification ac-
curacies of the thematic results improve one by one. In Plate
1c, it is suggested that although the road pixels still show
a satisfactory discrimination, the imaging environmental
changes (i.e., the temperature change, pressure variation, etc.,
shown in Table 1) reduce the discriminative ability for the
rest of the classes, especially the pixels in the second strip, in
which most of the pixels (except for the road pixels) are clas-
sified as concrete roof. In both Plate 1a and 1b, it can be seen
that many corridors existing between the roofs in the commu-
nity are misclassified as concrete roof pixels, as the material
of these corridors is also concrete. Since there is no corridor
class type in the reference data, this kind of misclassification
is ignored in the quantitative statistics. The proposed method
interpreted these pixels as road, which is more suitable in
the semantic-level exploitation. Compared with the other
thematic maps, Plate 1h and 1i show the best road detection
performance, while the latter figure also shows the best bare
soil land-cover recognition.
Table 6 shows a statistical comparison between the dif-
ferent approaches. The ΔOA values are the difference in the
quantity disagreement between practical Classifier 1 and Clas-
sifier 2, and they confirm the inferiority of the former classi-
fier when ΔOA <0. McNemar’s test (Foody 2004), which is a
non-parametric statistical significance test of the difference
Table 5. Classification Accuracy (%) for the Study Area
Class Name TI-HSI VIS VIS-SF VIS-VS InVIS-VS R-InVIS-VS Proposed
Road 96.30 88.29 89.86 91.49 94.52 98.17 98.17
Trees 0.90 12.15 90.16 90.11 88.56 85.94 91.07
Red roof 27.71 93.66 96.93 97.85 98.40 94.72 95.14
Gray roof 50.10 53.20 67.42 90.24 88.74 95.71 98.99
Concrete roof 71.53 93.77 92.15 91.49 91.82 90.38 92.29
Vegetation 26.51 99.40 91.46 93.46 93.58 94.74 99.14
Bare soil 45.54 88.89 65.55 88.12 88.38 87.69 89.70
OA (%) 70.10 81.29 87.80 91.90 93.42 95.57 96.81
AA (%) 45.51 75.63 84.79 91.82 92.00 92.48 94.93
κ 0.5470 0.7187 0.8160 0.8795 0.9009 0.9324 0.9514
Note: according to the result of the IADF 2014 Data Fusion Contest, the κ value of the runner-up was 0.9217, and 0.9438 for the winner.
T able 3. T he S even Ground-Reference C lasses in the Study Area, and the
Training and T est Sample Sets for Each C lass
No. Class
name
Provided by the contest committee Edge
sample
set
Training
samples Global test set
1 Road 112457 809098 -
2 Trees 27700 100749 21116
3 Red roof 46578 136697 55702
4 Gray roof 53520 142868 118723
5 Concrete
roof 97826 109539 37718
6 Vegetation 185329 103583 105473
7 Bare soil 44738 49212 -
Total 568148 1451746 338732
Table 4. Comparison of the Classification Approaches
Acronym Approach
TI-HSI Original spectral feature of the TI-HSI data
VIS Original spectral feature of the VIS data
VIS-SF Single optimal spatial feature of the VIS data
VIS-VS Vector stacking (VS) of the five spatial features of
the VIS data
InVIS-VS Inpainting of the strips of the VIS data, and then VS
of the five spatial features of the VIS data
R-InVIS-VS
Road extraction with the OOIFA approach, then
inpainting of the strips, and VS for the remaining
class classification
Proposed R-InVIS-VS, with post-classification
Table 6. Summary of the Classification Comparisons Undertaken in the Study
Area. A Resampling Method was Used to C onduct the M cNemar’s T est, to
Compare the Proportions of the C orrectly Allocated Pixels. All the tests
Shown were One-sided, and a 5 Percent Level of Significance was Selected
Classifier 1 Classifier 2
Comparison of the proportions
and disagreement
ΔOA(%) |z| Significant?
TI-HSI VIS −10.19 1.8204 No
VIS-SF VIS 6.51 1.6713 No
VIS-VS VIS 10.61 3.6742 Yes
Proposed VIS 15.52 4.5962 Yes
PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING December 2015 907