PE&RS May 2019 Public - page 389

Self-Supervised Convolutional Neural Networks
for Plant Reconstruction Using Stereo Imagery
Yuanxin Xia, Pablo d’Angelo, Jiaojiao Tian, Friedrich Fraundorfer, and Peter Reinartz
Abstract
Stereo matching can provide complete and dense three-
dimensional reconstruction to study plant growth. Recently,
high-quality stereo matching results were achieved combining
Semi-Global Matching (SGM) with deep learning. However,
due to a lack of suitable training data, this technique is not
readily applicable for plant reconstruction. We propose a
self-supervised Matching Cost with a Convolutional Neu-
ral Network (MC-CNN) scheme to calculate matching cost
and test it for plant reconstruction. The MC-CNN network is
retrained using the initial matching results obtained from
the standard MC-CNN weights. For the experiment, close-
range photogrammetric imagery of an in-house plant is used.
The results show that the performance of self-supervised
MC-CNN is superior to the Census algorithm and compa-
rable to MC-CNN trained by a Light Detection and Ranging
point cloud. Another experiment is performed using stereo
imagery of a field beech tree. The proposed self-training
strategy is tested and has proved capable of identifying the
drought condition of trees from the reconstructed leaves.
Introduction
Forest management is an interdisciplinary topic involved
in numerous fields such as environment, politics, econom-
ics, climate, and ecology (Strigul 2012). Remote sensing, as a
technique to take measurements from a distance, is appropri-
ate to assist forest management because it can observe the
target with no need to approach it and provide time series
data sets for constant monitoring. Spaceborne and airborne
remote sensing instruments offer broad observation of trees to
estimate the biomass, monitor the living condition, measure
the forest canopy cover, etc. (Ahmed et al. 2014; Freeman et
al. 2016; Wu et al. 2016). Some high-resolution stereo imaging
sensors are capable of deriving detailed digital surface models
to acquire geometric parameters of the forest, however, only
some large-scale properties such as forest canopy height can
actually be estimated (Tian et al. 2017).
In order to obtain detailed information about the forest,
single tree growth patterns should be observed. The size,
shape, color, and leaf distribution of individual trees are all
important factors and worth measuring in detail so that the
health situation of the tree and even the whole ecosystem can
be better understood (Levin 1999; Gatziolis et al. 2015). The
terrestrial Light Detection and Ranging (LiDAR ) technique can
provide accurate and dense point clouds of trees to support
the geometric survey for tree-level parameters estimation
(Kankare et al. 2013; Tao et al. 2015). Nevertheless, the data
acquisition can require considerable manpower and material
resources and can even be dangerous in extreme terrain.
In the past decade, dense matching using optical stereo
images has been widely used for three-dimensional (3D)
reconstruction. Among the different techniques, Semi-Global
Matching (SGM ) has outperformed most existing approaches
in accuracy and efficiency (especially in remote sensing),
and is used in many applications, for example building
reconstruction, digital surface model generation, robot
navigation, and driver assistance (Hirschmüller 2011; Kuschk
et al. 2014; Qin et al. 2015). However, the performance
varies when different matching cost calculation approaches
are adopted. Many local features (e.g. Census, Mutual
Information) have been used for the matching cost calculation
(Hirschmüller 2008; Hirschmüller and Scharstein 2009). But
tree leaf matching remains very difficult due to the lack of
unique features, many occlusions, and repetitive structure.
Convolutional Neural Networks (CNN) (LeCun et al. 1998)
are a popular topic in computer vision and have been used
to solve many vision problems. Recently, an algorithm
computing Matching Cost based on CNN (MC-CNN) was
proposed (Zbontar and LeCun 2016) in which a net is trained
with supervised learning based on pairs of small image
patches with known true disparity. Combined with SGM,
MC-CNN has proved to outperform most previous algorithms
thanks to a good extraction of the local image features and a
trained similarity measure to compare the extracted feature
descriptors. However, the ground truth collection is always
a bottleneck for deep neural network-based algorithms,
which require huge amount of labeled data to train the net
(Krizhevsky et al. 2012; Knöbelreiter et al. 2018). Ground
truth acquisition for tree reconstruction via LiDAR sensors is
complicated by the long scanning time required for capturing
a dense point cloud. Any tiny movement of the leaf or branch
during the laser scanning will cause the scanned point cloud
to be inconsistent with the images, which limits its use for
further training and evaluation. Hence, in this paper we
follow the work of (Knöbelreiter et al. 2018) and propose a
dense matching strategy combining SGM and a self-trained MC-
CNN for plant reconstruction.
This paper is organized as follows: The MC-CNN based
dense matching and the proposed training schemes are
described in the section “Methodology”. The section
“Experiments” describes an indoor and an outdoor
experiment, which demonstrate the feasibility of the proposed
self-training strategy. Conclusions are drawn and an outlook
for future research is provided in Conclusion.
Department of Photogrammetry and Image Analysis, Remote
Sensing Technology Institute, German Aerospace Center
(DLR), 82234 Wessling, Germany (Yuanxin.Xia, Pablo.Angelo,
Jiaojiao.Tian, Peter.Reinartz)@dlr.de.
Friedrich Fraundorfer is also with the Institute of Computer
Graphics and Vision, Graz University of Technology (TU
Graz), 8010 Graz, Austria (fraundorfer@icg.tugraz.at).
Photogrammetric Engineering & Remote Sensing
Vol. 85, No. 5, May 2019, pp. 389–399.
0099-1112/18/389–399
© 2019 American Society for Photogrammetry
and Remote Sensing
doi: 10.14358/PERS.85.5.389
PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING May 2019 389