Integrated Image Matching and Segmentation for
3D Surface Reconstruction in Urban Areas
Lei Ye and Bo Wu
Abstract
High-resolution imagery, which features the advantages of
high-quality imaging, a short revisit time, and lower costs, is
an attractive option for 3D reconstruction applications. Pho-
togrammetric 3D reconstruction requires reliable and dense
image matching. In urban areas, however, image matching
is particularly difficult because of the complexity of urban
textures and the severe occlusion problems caused by build-
ings. This paper presents an integrated image matching and
segmentation approach (named
SATM
+) for 3D reconstruction
in urban areas.
SATM
+ is based on our existing self-adaptive
triangulation-constrained matching (
SATM
) framework and
incorporates three novel aspects to address image match-
ing challenges in urban areas: (1) image segmentation-based
occlusion filtering, (2) segment-adaptive similarity mea-
surement to reduce matching ambiguity, and (3) local and
regional dense matching propagation to generate reliable and
dense matches. We performed an experimental analysis of
two sets of high-resolution urban images, and the 3D point
clouds generated using the proposed
SATM
+ were compared
with airborne light detection and ranging (lidar) data and the
point clouds generated using the semi-global matching (SGM)
method. The results indicate that
SATM
+ can generate 3D
point clouds with a geometric accuracy comparable to that of
lidar data but a much higher point density.
SATM
+ performs
similarly to SGM in relatively flat areas, but is superior in
built-up areas. The proposed approach is a promising option
for image-based 3D surface reconstruction in urban areas.
Introduction
In recent decades, urban 3D modelling has emerged as an im-
portant issue in the fields of photogrammetry and computer
vision, with various applications in urban planning, urban
monitoring, and urban management. Although urban 3D data
are in high demand worldwide, these data are still generated
using rudimentary and relatively expensive methods (Gruen,
2008; Vanegas
et al
., 2010; Zhu
et al
., 2010). To date, two
main techniques have been used to generate 3D data: airborne
light detection and ranging (lidar) techniques and image-
based photogrammetric techniques.
Since the 1990s, lidar techniques have been increasingly
used to collect 3D urban data (Gamba and Houshmand, 2000).
The current systems allow the measurement of approximately
four surface points per square meter, with a vertical accuracy
of approximately 15 cm (Hyyppae
et al
., 2000). It should be
noted, however, that airborne lidar surveys are usually expen-
sive (Shan and Toth, 2008).
Image-based photogrammetric techniques have been widely
used for 3D reconstruction due to the advantages of high-qual-
ity imaging, a short revisit time, and lower costs. A success-
ful photogrammetric 3D data derivation requires automated,
reliable, and dense image matching. The techniques used for
image-based 3D reconstruction have advanced since the late
1980s (Ackermann and Krzystek, 1991), and suitable image
matching algorithms can be used to generate 3D point clouds
with favorable levels of accuracy, reliability, and detail in
relatively favorable texture conditions (Gruen, 2008; Bleyer
et
al
., 2011; Wu
et al
., 2011 and 2012). However, image matching
in urban areas is particularly difficult. Most traditional digital
photogrammetry systems require considerable human labor to
process images in urban areas (Helpke, 1995; Wu
et al
., 2011),
especially in metropolitan regions containing densely packed
areas of vast skyscrapers and tall buildings. These challenges
are mainly attributed to the intrinsic problem of image match-
ing, which is caused by the complexities of urban textures and
severe occlusion problems associated with buildings.
However, the textural complexity of urban areas can actually
increase the accuracy and convenience of feature-based image
matching, assuming an effective matching strategy is avail-
able. And the high density of buildings tends to offer regularly
shaped image segments. These advantages could potentially
facilitate image-based 3D surface reconstruction in urban areas.
The present study used high-resolution imagery of urban
areas for 3D reconstruction, with the aim of developing a
cheaper, better automated image-based 3D reconstruction
method. Following the literature review in the next section,
we present our novel integrated image matching and segmen-
tation approach to 3D surface reconstruction in urban areas.
Then, we describe the results of experiments performed using
two sets of high-resolution images, each representing a typical
urban type. Finally, the concluding remarks are presented and
discussed.
Related Works
Image matching is used to identify corresponding pixels
between images that can be used for 3D reconstruction using
photogrammetric space intersection (Wu
et al
., 2012). Im-
age matching is also a difficult, essential task in the fields of
photogrammetry and computer vision (Lhuillier and Quan,
2002; Zhang and Gruen, 2006; Wu
et al
., 2011). Currently, two
image matching strategies are normally used: sparse point
matching, which is often used for image registration, and
dense matching, which is often used for 3D reconstruction.
Sparse point matching normally includes interest point
detection and matching. The interest point detectors widely
used in photogrammetry include the Moravec detector
(Moravec, 1981), the Förstner detector (Förstner, 1986), and
the Harris detector (Harris and Stephens, 1988). The Moravec
detector measures the gray value differences between a win-
dow and windows shifted in several directions, and detects
interest points if the minimum of these differences is superior
Department of Land Surveying and Geo-Informatics, The
Hong Kong Polytechnic University, Hung Hom, Kowloon,
Hong Kong (
).
Photogrammetric Engineering & Remote Sensing
Vol. 84, No. 3, March 2018, pp. 135–148.
0099-1112/17/135–148
© 2018 American Society for Photogrammetry
and Remote Sensing
doi: 10.14358/PERS.84.3.135
PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING
March 2018
135
