unstructured accuracies were very low, with an overall ac-
curacy of 64% and a weighted F-score of 61%. The temporal
modeling approach improved corresponding
OA
and weighted
F-score by 12% and 9%, respectively, confirming the con-
tribution of temporal structure even if the accuracy of the
parcel-wise prediction was low. As for Site77, the structured
approach slightly improved the weighted F-scores by 3%,
4%, and 3% for radar, optical, and combined optical/radar at-
tributes, respectively. The contribution of temporal structure
is lower than that in Site04, as the initial parcel-wise accura-
cies were already high (weighted F-scores >0.88).
As seen in Table 5, temporal structure modeling signifi-
cantly improved per-class accuracies of permanent crops
(fruit trees +36%, vineyards +30%, olive groves +25%,
aromatic groves +19% for Site04 ), which reached F-scores
higher than 93%. The meadows class F-score was improved
by 4%. These results were expected since the permanent
crops have the highest transition probability as shown in the
section “Transition Matrix Assessment”. However, F-scores of
annual crops classes decreased when using temporal struc-
ture; only slightly so for corn (
-
11%), other cereals (
-
7%), and
sunflowers (
-
8%), but rapeseed (
-
21%), barley (
-
22%), other
oilseeds (
-
16%), protein (
-
49%), and forage crops (
-
46%)
are significantly more often misclassified (cf
.
Table 7). This
may be explained, in our opinion, by two facts: first, the crop
rotations are modeled by a temporal regularization between
the observation-based term (classification without temporal
structure) and the crop transition probabilities. The predic-
tion is a trade-off between both data and regularization terms.
If the data-term is high (good prediction with observations),
adding crop rotation information does not impact the results
significantly (as for corn, rapeseed, and sunflower classes).
Second, this may be due to the fact that the first order transi-
tions of annual crops are less stable and highly variable with
agricultural practices and operators in this area. Hence, tem-
poral modeling does not add useful information and may even
wrongly correct an initially correct parcel-wise prediction.
As for Site77, similarly to Site04, the best improvements
occur on permanent crops, such as meadows and fruit trees
(cf. Table 5). Moreover, the temporal structure improved the
prediction of some annual crops, such as other cereals, rape-
seeds, and proteins since they have a high first order transition
probability to other cereals. The prediction of forage crops is
also highly improved using crop rotations information.
On both sites, including rotation knowledge improved the
overall accuracy of crop classification. The proposed model is
a trade-off between observation-based classification and tem-
poral regularization using learned rotation knowledge. If the
precision of observation-based classification is already high
and the transition patterns inconclusive or poorly followed,
integrating rotation knowledge may decrease the accuracy.
However, ambiguous observation-based prediction can be im-
proved by modeling the temporal structure, especially if the
temporal aspect is very influential, as with permanent crops
or crops alternating with other cereals. The detrimental effect
of temporal modeling on some annual crops were can also be
explained by the limited availability
LPIS
(only one edition,
2016, was available at the time), which might not be sufficient
to model crop rotation schemes occurring over two or three
years. Indeed, in this paper, only first order crop rotation was
modeled. However, our approach could be straightforwardly
extended to rotations over multiple years, provided more data
is available. Further tests should be processed with a newer
LPIS
edition and over larger areas in order to assess the effect
of modeling the temporal structure.
Conclusion and Perspectives
This study focused on improving the automatic prediction
of crop types using Sentinel-1 and -2 time series and learned
rotation knowledge. This study demonstrated the efficiency of
multi-temporal and multi-modal Sentinel (optical and radar)
images for crop type classification using a fine nomenclature
(>10 classes) and without filtering small parcels. The joint use
of optical and radar features ensured more stable and accurate
results. However, results varied highly depending on sites
depending on cloud cover, crop types, and parcel size.
We modeled the temporal structure (i.e., crop knowledge)
with conditional random fields and automatically learning
the probability of crop rotations from previous
LPIS
editions.
This rotation knowledge markedly improved the prediction of
crop types. However, while a positive impact is demonstrated
on permanent crops using first order crop transitions; this
impact is fairly limited or even detrimental for some annual
crops. Higher transition orders should be investigated to con-
firm the interest of temporal structure for annual crops and
larger areas with more representative classes should be used.
Finally, thanks to the growing volume of available
LPIS
data
and the free availability of numerous Sentinel images, deep
learning approaches for both parcel-wise feature extraction
and temporal modeling should be investigated.
Acknowledgments
This work is supported by the French National Research
Agency under the grant ANR-18-CE23-0023. The authors want
to thank the
ASP
(French payment agency) and the French ag-
ricultural ministry for the rich discussions and their guidance
for the nomenclature choice and crop rotation schemes.
References
Aurdal, L., R. B. Huseby, L. Eikvil, R. Solberg, D. Vikhamar and A.
Solberg. 2005. Use of hidden Markov models and phenology
for multitemporal satellite image classification: Applications to
mountain vegetation classification. Pages 16–18 in
Proceedings
of International Workshop Analysis Multi-Temporal Remote
Sensing Images
.
Belgiu, M. and O. Csillik. 2018. Sentinel-2 cropland mapping using
pixel-based and object-based time-weighted dynamic time
warping analysis.
Remote Sensing of Environment
204:509–523.
Berzsenyi, Z., B. Györffy and D. Q. Lap. 2000. Effect of crop rotation
and fertilisation on maize and wheat yields and yield stability
in a long-term experiment.
European Journal of Agronomy
13
(2):225–244.
Boryan, C., Z. Yang, R. Mueller and M. Craig. 2011. Monitoring
us agriculture: The US department of agriculture, national
agricultural statistics service, cropland data layer program.
Geocarto International
26 (5):341–358.
Castellazzi, M. S., G. A. Wood, P. J. Burgess, J. Morris, K. F. Conrad
and J. N. Perry. 2008. A systematic representation of crop
rotations.
Agricultural Systems
97 (1):26–33.
Defourny, P., S. Bontemps, N. Bellemans, C. Cara, G. Dedieu, E.
Guzzonato, O. Hagolle, J. Inglada, L.-t. Nicola, T. Rabaute, M.
Savinaud, C. Udroiu, S. Valero, A. Bégué, J.-F. Dejoux, A. El
Harti, J. Ezzahar, N. Kussul, K. Labbassi, V. Lebourgeois, Z. Miao,
T. Newby, A. Nyamugama, N. Salh, A. Shelestov, V. Simonneaux,
P.S. Traore, S.S. Traore and B. Koetz. 2019. Near real-time
agriculture monitoring at national scale at parcel resolution:
Performance assessment of the Sen2-agri automated system in
various cropping systems around the world.
Remote Sensing of
Environment
221:551–568.
Deledalle, C.-A., L. Denis, F. Tupin, A. Reigber and M. Jäger. 2015.
NL-SAR: A unified nonlocal framework for resolution-preserving
(pol)(in) SAR denoising.
IEEE Transactions on Geoscience and
Remote Sensing
53 (4):2021–2038.
440
July 2020
PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING
