October 2019 Layout Flipping Full - page 766

View and GeoEye satellites were taken to construct simulated
and real data experiments, respectively. Besides, several dif-
ferent types of terrains were selected to test the performance
of this method in different terrain conditions. Experiments
verified the feasibility and practicality of this kind of appli-
cation, and a subpixel level DSM with higher-fidelity can be
obtained compared to directly DSM interpolation.
The rest of this paper is organized as follows: Section “Re-
lated Work” gives a brief overview on CNN-based SR methods
and the applications of SR in remote sensing fields. Section
“Application to Subpixel Level DSM Generation” describes
the main application details and tells the evaluation metrics
of the proposed approach. Section “Experiments” confirms
the effectiveness and practicability of this kind of applica-
tion with simulated experiments and real data experiments.
Finally, the “Conclusion” section concludes this paper.
Related Work
In this section, we will introduce and analyze related methods
on image super resolution and give a brief overview on some
relevant applications of SR technology in remote sensing fields.
Image SR
Image SR aims to restore a high-resolution (HR) image from
one or more low-resolution (LR) images. Numerous methods
have been proposed to deal with the image SR problems in
recent years, which can be mainly classified into reconstruc-
tion-based SR and learning-based SR (Zeng et al. 2017).
The reconstruction-based SR methods assume that the LR
images are from the observations of their related LR images
after transformation, deformation, and noise disturbance. As
SR is an ill-posed task, reconstruction-based methods attempt
to optimize the problem by imposing some extra assumptions,
such as smoothness and limited bands (Sun et al. 2008; Tai et
al. 2010; Zhang et al. 2010). However, the added assumptions
will heavily limit the performance of this kind of method
since they are always unsuitable in most image situations.
Different from the reconstruction-based methods, the
learning-based SR methods learn the high frequency infor-
mation difference through constantly training high/low-
resolution image pairs, and then build a robust model. After
that, given a low-resolution image into the learned model, its
corresponding HR image can be restored. Specifically, accord-
ing to the different strategy of learning, the learning-based
method can be further divided into sparse-coding-based
methods(Yang et al. 2008;Yang et al. 2010), manifold-learn-
ing-based methods(Chang et al. 2004; Lu et al. 2013) and
CNN-based methods(Dong et al. 2016; Kim et al. 2016). Since
more high-frequency details can be learnt from the numerous
image samples, CNN-based methods visibly outperform other
methods. A pioneer work of CNN-based SR methods is called
super-resolution convolution neural network (SRCNN) (Dong
et al. 2016), which learns an end-to-end nonlinear mapping
between LR and HR image pairs via a three-hidden-layers CNN
architecture. However, this method cannot extract satisfactory
image feature for HR image reconstruction due to the shallow
architecture. Considering this, Kim et al. (2016) developed a
much deeper SR model VDSR, of which the number of convolu-
tion layers is increased to 20. As the difference of LR /HR image
pairs are mainly located in high frequency part, Kim et al.
(2015) proposed another network Deeply-Recursive Convo-
lutional Network, which utilized residual learning (He et al.
2016) as optimization strategy to accelerate the convergence
speed of their proposed network. Bicubic down-sampling is
commonly used in these CNN -based methods to generate LR
images from their HR counterparts, which results in the un-
satisfactory performance on real data. Therefore, Zhang et al.
(2018) proposed an effective model super-resolution network
for multiple degradations (SRMD ) to address this problem
to some extent, which established a degradation formula
between the HR/LR image pairs and took the LR images and the
degradation maps as input to reconstruct the corresponding
HR results.
SR in Remote Sensing Fields
As an effective technology for improving image spatial resolu-
tion, image super resolution is highly related to remote sens-
ing fields. And some classical SR methods in computer vision
fields were transferred into remote sensing fields to better
solve those ill-posed problems. Luo et al. (2017) refined VDSR
to deal with image compression problem that occurred during
video satellite image transmission process, Song et al (2018)
applied SRCNN to image fusion tasks as a transitional image
generation tool, and He et al. (2018) adopted cascaded deep
network and multiple receptive fields for infrared image super
resolution. Besides, considering the special characteristic of
remote sensing images, Huang et al (2017) developed remote
sensing deep residual-learning specifically for remote sensing
images super-resolution, while Lei et al. (2017) used local-
global combined network (LGCNet) to learn multilevel repre-
sentations of remote sensing images. Moreover, Jeon et al.
(2018) utilized a parallax prior in stereo images to reconstruct
HR images for subpixel registration accuracy, and Song et al.
(2016) extended the success of deep convolutional neural
network to depth image super resolution.
To the best of our knowledge, even though SR methods are
widely applicated into remote sensing fields and Dai et al.
(2016) have already verified the effectiveness and usefulness
of SR technology in several vision applications, there is not a
work to discuss whether it is feasible to bring the SR method
into traditional dense matching procedures for obtaining
subpixel level DSM .
Since the main purpose of this paper is to explore the
feasibility and practicality of this kind of application and to show
the effect of bringing different SR models into the framework, for
fair comparison and better explaining the experimental results,
we refined several SR models to show the performance difference
and analyze the practicality of the proposed method in this
paper, rather than designed a new SR model.
Application to Subpixel Level DSM Generation
As illustrated in Figure 1, the workflow of this application
can be mainly divided into two stages. Separately, in step I,
the image quality and spatial resolution is improved by CNN-
based SR module. The S R module is adopted to enhance the
object texture and edge detection, which helps extract more
reliable matching points for subsequent DSM generation pro-
cess. Then, with the obtained HR images from step I as input,
a subpixel level DSM with high fidelity can be obtained from a
standard DSM generation procedure in step II. In this section,
the details of this application will be explained in the follow-
ing aspects: refinement details of SR models, DSM generation
process, and quality assessment metrics.
Refinement of SR Models
In this subsection, we will introduce how to reconstruct a HR
image from its counterpart LR image. For the ease of repre-
sentation, the LR image is denoted as xi, its corresponding HR
image is denoted as yi, and yi′ is the reconstructed result of LR
image through CNN architecture. Specifically, xi is obtained
from yi according to down-sampling or a special degradation
formula. Therefore, as illustrated in Figure 2, a robust archi-
tecture can be obtained by repeatedly minimizing the differ-
ence between HR image yi and yi′ . To measure the difference
between real HR image yi and predicted HR image yi′, mean
squared error (MSE), is usually used as Equation 1,
MSE
width height
= × − 1 2
y yi i ′ (1)
766 October 2019 PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING