In July 2021, Prof. Tao Lin's group published a research paper on RemoteSensing of Environment (JCR,Q1), which is one of the top journals in the fieldof remote sensing, with 5-year impact factor of 11.057, entitled “Towardsinterpreting multi-temporal deep learning models in crop mapping”.

Multi-temporal deep learning approaches have exhibited excellentclassification performance in large-scale crop mapping. These approachesefficiently and automatically transform remote sensing time series into high-dimensionalfeature representations to identify crop types. The lack of interpretation,however, is regarded as a major drawback of these high-performance approaches.Interpreting deep learning approaches in multitemporal crop mapping is criticalfor verifying their reliability. This study aims to quantify the impact of multi-temporalinformation in input time series on classification performance and develop amulti-perspective interpretation pipeline for deep learning models. Thepipeline involves three interpretation approaches: evaluating input featureimportance, analyzing hidden features, and monitoring temporal changes inmodel's soft output. An experiment is conducted to classify corn and soybean inthe U.S corn belt in 2018. The study area consists of three sites each encompassingmillions of pixel-level samples at 30 m resolution. The Landsat Analysis ReadyData are used as the input remote sensing time series and Cropland Data Layeris used as the ground reference. Attention-based Long Short-Term Memory(AtLSTM) and Transformer models are built as multitemporal deep learningmodels, and compared to Random Forest (RF). Complete time series input in thecorrect order achieves a higher overall accuracy of 97.8% than usingsingle-window or out-of-order inputs, indicating multi-temporal informationfacilitates crop classification. An assessment of the input feature importance demonstratesthat the AtLSTM, Transformer, and RF models all consider the period from weeks11 to 20 (early- July to late-August) as a key growth period and the shortwaveinfrared band as the critical band for corn and soybean discrimination. Hiddenfeature analysis suggests that the AtLSTM model accumulates the usefulinformation over the growth period, while the Transformer model extracts thetemporal dependencies that contribute important information to high-levelfeature learning. The learned features contain more effective and refined informationthan the raw input features and thus are better suited for crop classification.The soft output analysis in the in-season classification scenario demonstratesthat increased length of input time series improves the model's confidence inthe classification results. The further comparison of input feature importancein different sites and years demonstrates the applicability of theinterpretation approach at larger spatiotemporal extents with heterogeneouslandscapes and interannual variability. This study provides a multi-perspectiveevaluation to identify key features in multi-spectral and multi-temporal remotesensing data, and yields a practical approach to integrate agronomy knowledgein deep learning-based crop mapping.