Crop Identification Using Deep Learning on LUCAS Crop Cover Photos
Abstract
:1. Introduction
Objectives
- To select and publish a subset of LUCAS cover photos representative of major and mature crops across the EU for training purposes.
- To deploy and benchmark a set of MobileNet computer vision models to recognize crops on close-up pictures and identify the best-performing model.
- To explore the use of probability- and entropy-based metrics to threshold and filter correct and incorrect classifications.
- To illustrate the applications and limitations of the model for inference in a practical and agricultural-policy-relevant context.
2. Materials and Methods
2.1. Data
2.1.1. LUCAS Cover Photos
2.1.2. Crop Calendars and Harmonization
2.1.3. Expert Knowledge Gap Filling and Mature Pre-Harvest Stages
2.1.4. Manual Photo Pre-Processing by Visual Assessment
2.2. Method
2.2.1. Sample Selection for Training and Inference Set(s)
2.2.2. Hyper-Parameter Search and Best Model Selection
2.2.3. Operational Use
2.2.4. Computational Infrastructure
2.2.5. Equivalent Reference Probability Filter
3. Results
3.1. Mature Major European Crops
3.2. Best Performing Model
3.3. Confusion Matrix
3.4. Equivalent Reference Probability Filter
3.5. Unfavorable Conditions
4. Discussion
4.1. Context
4.2. ERP Filtering
4.3. Limitations
4.4. Recommendations
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
Appendix A
Sources | Crops |
---|---|
AGRI4CAST 1 | Corn, winter wheat, durum wheat, rice |
USDA 2 | Sunflower, barley, rye, soybeans, spring wheat, rapeseed, oats |
EUROPABIO 3 | Potato, sugar beet |
Range of Pixels | W × H Included | % of Images |
---|---|---|
Less than 1 million | 640 × 480, 1024 × 768, 800 × 600 | 1.39 |
1–2 million | 1600 × 1200, 1280 × 960, 1632 × 1224, 1200 × 1600, 1200 × 900, 1605 × 1204, 1728 × 1152, 1288 × 966, 1600 × 1198, 1612 × 1212, 1700 × 1130, 1600 × 963, 1261 × 817, 1600 × 900, 1397 × 1048, 1593 × 1200, 1319 × 989, 1280 × 1024, 1552 × 1164 | 64.73 |
2–3 million | 1664 × 1248, 2048 × 1360, 1920 × 1080, 1824 × 1216, 1733 × 1300, 1792 × 1312, 1936 × 1288, 1656 × 1242, 1984 × 1488, 2048 × 1104, 2000 × 1333, 1824 × 1368, 1936 × 1296, 1936 × 1452, 1920 × 1440, 1662 × 1246, 2080 × 1368, 1360 × 2048, 2048 × 1376, 1800 × 1350, 1632 × 1232 | 6.68 |
3–4 million | 2048 × 1536, 2304 × 1728, 2272 × 1704, 2288 × 1712, 1536 × 2048, 2352 × 1568, 2592 × 1458, 2200 × 1650, 2042 × 1532, 2133 × 1600, 2240 × 1680, 2080 × 1544 | 22.32 |
4–5 million | 2560 × 1712, 2560 × 1920, 2400 × 1800, 2344 × 1758, 2464 × 1632, 1932 × 2580, 2576 × 1932 | 2.33 |
5–6 million | 2592 × 1944, 2816 × 2112, | 3.46 |
6–7 million | 2848 × 2136, 3072 × 2048, 3456 × 1946, 2896 × 2172, | 0.95 |
7–8 million | 3072 × 2304, 3264 × 2448, 3584 × 2016 | 3.08 |
Over 8 million | 3968 × 2976, 3488 × 2616, 4320 × 2432, 3664 × 2748, 4672 × 3504, 3840 × 2880 | 0.81 |
References
- Alzubaidi, L.; Zhang, J.; Humaidi, A.J.; Al-Dujaili, A.; Duan, Y.; Al-Shamma, O.; Santamaría, J.; Fadhel, M.A.; Al-Amidie, M.; Farhan, L. Review of deep learning: Concepts, CNN architectures, challenges, applications, future directions. J. Big Data 2021, 8, 1–74. [Google Scholar]
- Howard, A.G.; Zhu, M.; Chen, B.; Kalenichenko, D.; Wang, W.; Weyand, T.; Andreetto, M.; Adam, H. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv 2017, arXiv:1704.04861. [Google Scholar]
- Howard, A.; Sandler, M.; Chu, G.; Chen, L.C.; Chen, B.; Tan, M.; Wang, W.; Zhu, Y.; Pang, R.; Vasudevan, V.; et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea, 27 October–2 November 2019; pp. 1314–1324. [Google Scholar]
- d’Andrimont, R.; Yordanov, M.; Martinez-Sanchez, L.; van der Velde, M. Monitoring crop phenology with street-level imagery using computer vision. Comput. Electron. Agric. 2022, 196, 106866. [Google Scholar] [CrossRef]
- Matvienko, I.; Gasanov, M.; Petrovskaia, A.; Jana, R.B.; Pukalchik, M.; Oseledets, I. Bayesian aggregation improves traditional single image crop classification approaches. arXiv 2020, arXiv:2004.03468. [Google Scholar]
- Bogaert, P.; Waldner, F.; Defourny, P. An information-based criterion to measure pixel-level thematic uncertainty in land cover classifications. Stoch. Environ. Res. Risk Assess. 2017, 31, 2297–2312. [Google Scholar] [CrossRef] [Green Version]
- Columbus, L. 10 Ways AI Has the Potential to Improve Agriculture in 2021. Available online: https://www.forbes.com/sites/louiscolumbus/2021/02/17/10-ways-ai-has-the-potential-to-improve-agriculture-in-2021/ (accessed on 10 July 2023).
- Tian, H.; Wang, T.; Liu, Y.; Qiao, X.; Li, Y. Computer vision technology in agricultural automation—A review. Inf. Process. Agric. 2020, 7, 1–19. [Google Scholar] [CrossRef]
- Santoso, I.; Purnomo, M.; Sulianto, A.; Choirun, A. Machine learning application for sustainable agri-food supply chain performance: A review. In Proceedings of the IOP Conference Series: Earth and Environmental Science, Malang, Indonesia, 6–7 July 2021; IOP Publishing: Bristol, UK, 2021; Volume 924, p. 012059. [Google Scholar]
- Kollia, I.; Stevenson, J.; Kollias, S. AI-Enabled Efficient and Safe Food Supply Chain. Electronics 2021, 10, 1223. [Google Scholar] [CrossRef]
- Zhu, L.; Spachos, P.; Pensini, E.; Plataniotis, K.N. Deep learning and machine vision for food processing: A survey. Curr. Res. Food Sci. 2021, 4, 233–249. [Google Scholar] [CrossRef]
- Schiller, C.; Schmidtlein, S.; Boonman, C.; Moreno-Martínez, A.; Kattenborn, T. Deep learning and citizen science enable automated plant trait predictions from photographs. Sci. Rep. 2021, 11, 1–12. [Google Scholar]
- Affouard, A.; Goëau, H.; Bonnet, P.; Lombardo, J.C.; Joly, A. Pl@ ntnet app in the era of deep learning. In Proceedings of the ICLR: International Conference on Learning Representations, Toulon, France, 24–26 April 2017. [Google Scholar]
- Wu, Z.; Chen, Y.; Zhao, B.; Kang, X.; Ding, Y. Review of weed detection methods based on computer vision. Sensors 2021, 21, 3647. [Google Scholar] [CrossRef]
- Mohanty, S.P.; Hughes, D.P.; Salathé, M. Using deep learning for image-based plant disease detection. Front. Plant Sci. 2016, 7, 1419. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Kapach, K.; Barnea, E.; Mairon, R.; Edan, Y.; Ben-Shahar, O. Computer vision for fruit harvesting robots–state of the art and challenges ahead. Int. J. Comput. Vis. Robot. 2012, 3, 4–34. [Google Scholar] [CrossRef] [Green Version]
- Lu, Y.; Young, S. A survey of public datasets for computer vision tasks in precision agriculture. Comput. Electron. Agric. 2020, 178, 105760. [Google Scholar] [CrossRef]
- Zheng, Y.Y.; Kong, J.L.; Jin, X.B.; Wang, X.Y.; Su, T.L.; Zuo, M. CropDeep: The crop vision dataset for deep-learning-based classification and detection in precision agriculture. Sensors 2019, 19, 1058. [Google Scholar] [CrossRef] [Green Version]
- Sudars, K.; Jasko, J.; Namatevs, I.; Ozola, L.; Badaukis, N. Dataset of annotated food crops and weed images for robotic computer vision control. Data Brief 2020, 31, 105833. [Google Scholar] [CrossRef]
- Tseng, G.; Zvonkov, I.; Nakalembe, C.L.; Kerner, H. Cropharvest: A global dataset for crop-type classification. In Proceedings of the Thirty-Fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), Online, 7–10 December 2021. [Google Scholar]
- d’Andrimont, R.; Verhegghen, A.; Meroni, M.; Lemoine, G.; Strobl, P.; Eiselt, B.; Yordanov, M.; Martinez-Sanchez, L.; van der Velde, M. LUCAS Copernicus 2018: Earth-observation-relevant in situ data on land cover and use throughout the European Union. Earth Syst. Sci. Data 2021, 13, 1119–1133. [Google Scholar] [CrossRef]
- d’Andrimont, R.; Yordanov, M.; Martinez-Sanchez, L.; Haub, P.; Buck, O.; Haub, C.; Eiselt, B.; van der Velde, M. LUCAS Cover photos 2006–2018 over the EU: 874,646 spatially distributed geo-tagged close-up photos with land cover and plant species label. Earth Syst. Sci. Data Discuss. 2022, 2022, 1–14. [Google Scholar]
- Devos, W.; Sima, A.; Milenov, P. Conceptual Basis of Checks by Monitoring. 2021. European Union, JRC Technical Report JRC127678. [Online]. Available online: https://wikis.ec.europa.eu/download/attachments/86968800/JRC127678_final.pdf?version=1&modificationDate=1682601334749&api=v2 (accessed on 10 July 2023).
- Sima, A.; Loudjani, P.; Devos, W. Use of Geotagged Photographs in the Frame of Common Agriculture Policy Checks. Ispra 2020, JRC120223. European Union, JRC Report Investigating the Potential of Specific Technology and/or Domain. [Online]. Available online: https://wikis.ec.europa.eu/download/attachments/86968800/Geotagged_JRC_ReportV5b.pdf?version=1&modificationDate=1682604679430&api=v2 (accessed on 10 July 2023).
- van der Velde, M.; van Diepen, C.; Baruth, B. The European crop monitoring and yield forecasting system: Celebrating 25 years of JRC MARS Bulletins. Agric. Syst. 2019, 168, 56–57. [Google Scholar] [CrossRef]
- AGRI4CAST. AGRI4CAST Bulletin Archive 2018. 2018. Available online: https://agri4cast.jrc.ec.europa.eu/BulletinsArchive (accessed on 30 June 2021).
- Koch, E.; Dittmann, E.; Lipa, W.; Menzel, A.; Nekovar, J.; van Vliet, A. COST Action 725: Establishing a European phenological data platform for climatological applications. In Proceedings of the 17th International Congress of Biometeorology (ICB 20050), Offenbach am Main 2005, Garmisch-Partenkirchen, Germany, 5–9 September 2005; pp. 554–558. [Google Scholar]
- Soekhoe, D.; Van Der Putten, P.; Plaat, A. On the impact of data set size in transfer learning using deep neural networks. In International Symposium on Intelligent Data Analysis; Springer: Berlin/Heidelberg, Germany, 2016; pp. 50–60. [Google Scholar]
- Bergstra, J.; Bengio, Y. Random search for hyper-parameter optimization. J. Mach. Learn. Res. 2012, 13, 281–305. [Google Scholar]
- Opitz, J.; Burst, S. Macro f1 and macro f1. arXiv 2019, arXiv:1911.03347. [Google Scholar]
- Soille, P.; Burger, A.; De Marchi, D.; Kempeneers, P.; Rodriguez, D.; Syrris, V.; Vasilev, V. A versatile data-intensive computing platform for information retrieval from big geospatial data. Future Gener. Comput. Syst. 2018, 81, 30–40. [Google Scholar] [CrossRef]
- Bruha, I.; Famili, A. Postprocessing in machine learning and data mining. ACM SIGKDD Explor. Newsl. 2000, 2, 110–114. [Google Scholar] [CrossRef]
- Bansal, S.; Kumar, A. A Post-processing Fusion Framework for Deep Learning Models for Crop Disease Detection. In Proceedings of the IOP Conference Series: Materials Science and Engineering, Chennai, India, 16–17 September 2020; IOP Publishing: Bristol, UK, 2020; Volume 998, p. 012065. [Google Scholar]
- Gao, X.; Ram, S.; Philip, R.C.; Rodríguez, J.J.; Szep, J.; Shao, S.; Satam, P.; Pacheco, J.; Hariri, S. Selecting Post-Processing Schemes for Accurate Detection of Small Objects in Low-Resolution Wide-Area Aerial Imagery. Remote Sens. 2022, 14, 255. [Google Scholar] [CrossRef]
- Ma, Y.; He, H. Imbalanced Learning: Foundations, Algorithms, and Applications; John Wiley & Sons, Inc.: Hoboken, NJ, USA, 2013. [Google Scholar]
- Gao, J.; Liu, C.; Han, J.; Lu, Q.; Wang, H.; Zhang, J.; Bai, X.; Luo, J. Identification Method of Wheat Cultivars by Using a Convolutional Neural Network Combined with Images of Multiple Growth Periods of Wheat. Symmetry 2021, 13, 2012. [Google Scholar] [CrossRef]
- Waldner, F.; Hansen, M.C.; Potapov, P.V.; Löw, F.; Newby, T.; Ferreira, S.; Defourny, P. National-scale cropland mapping based on spectral-temporal features and outdated land cover information. PLoS ONE 2017, 12, e0181911. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Owusu, M.; Kuffer, M.; Belgiu, M.; Grippa, T.; Lennert, M.; Georganos, S.; Vanhuysse, S. Towards user-driven earth observation-based slum mapping. Comput. Environ. Urban Syst. 2021, 89, 101681. [Google Scholar] [CrossRef]
B11 | B12 | B13 | B14 | B15 | B16 | B21 | B22 | B31 | B32 | B33 | B55 | Total | Total MMEC | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
AT | 136 | 32 | 139 | 103 | 29 | 69 | 48 | 67 | 18 | 85 | 122 | 124 | 972 | 3595 |
BE | 59 | 2 | 71 | 4 | 3 | 39 | 93 | 49 | 0 | 23 | 0 | 62 | 405 | 2127 |
BG | 179 | 5 | 129 | 19 | 12 | 148 | 11 | 0 | 110 | 90 | 4 | 3 | 710 | 2855 |
CY | 15 | 1 | 29 | 0 | 2 | 0 | 6 | 0 | 0 | 0 | 0 | 0 | 53 | 207 |
CZ | 72 | 4 | 28 | 32 | 20 | 47 | 30 | 45 | 4 | 194 | 12 | 55 | 543 | 6691 |
DE | 156 | 40 | 157 | 176 | 133 | 114 | 177 | 152 | 17 | 220 | 6 | 212 | 1560 | 24,055 |
DK | 132 | 2 | 93 | 105 | 27 | 55 | 21 | 24 | 0 | 184 | 0 | 175 | 818 | 3226 |
EE | 20 | 0 | 16 | 7 | 8 | 1 | 5 | 0 | 0 | 26 | 0 | 14 | 97 | 612 |
EL | 62 | 81 | 88 | 0 | 25 | 62 | 6 | 7 | 22 | 9 | 0 | 4 | 366 | 1386 |
ES | 68 | 58 | 85 | 105 | 80 | 22 | 59 | 58 | 34 | 50 | 0 | 178 | 797 | 19,582 |
FR | 121 | 97 | 116 | 130 | 118 | 82 | 139 | 143 | 75 | 186 | 142 | 193 | 1542 | 40,989 |
HR | 38 | 0 | 22 | 4 | 7 | 53 | 6 | 2 | 16 | 9 | 34 | 16 | 207 | 434 |
HU | 74 | 34 | 103 | 66 | 30 | 53 | 17 | 8 | 49 | 135 | 49 | 8 | 626 | 7354 |
IT | 136 | 54 | 175 | 17 | 130 | 54 | 50 | 98 | 17 | 21 | 378 | 166 | 1296 | 13,387 |
LT | 82 | 6 | 65 | 72 | 50 | 1 | 28 | 4 | 0 | 147 | 0 | 31 | 486 | 2313 |
LU | 3 | 0 | 7 | 2 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 6 | 20 | 149 |
LV | 66 | 3 | 72 | 44 | 33 | 9 | 12 | 1 | 0 | 110 | 0 | 42 | 392 | 1763 |
NL | 150 | 0 | 53 | 23 | 2 | 134 | 200 | 115 | 0 | 3 | 0 | 54 | 734 | 1805 |
PL | 66 | 12 | 40 | 105 | 61 | 98 | 89 | 93 | 1 | 173 | 7 | 67 | 812 | 20,542 |
PT | 28 | 7 | 24 | 40 | 50 | 22 | 19 | 0 | 1 | 0 | 0 | 68 | 259 | 877 |
RO | 71 | 28 | 39 | 13 | 25 | 159 | 22 | 16 | 189 | 51 | 88 | 32 | 733 | 3649 |
SE | 89 | 0 | 67 | 47 | 85 | 11 | 34 | 55 | 0 | 95 | 0 | 167 | 650 | 2742 |
SI | 29 | 6 | 26 | 2 | 2 | 30 | 6 | 0 | 1 | 5 | 0 | 27 | 134 | 364 |
SK | 89 | 13 | 95 | 27 | 16 | 51 | 18 | 25 | 42 | 164 | 55 | 18 | 613 | 3091 |
UK | 155 | 0 | 134 | 8 | 84 | 78 | 126 | 105 | 0 | 182 | 0 | 179 | 1051 | 5665 |
Total | 2096 | 485 | 1873 | 1151 | 1032 | 1393 | 1222 | 1067 | 596 | 2163 | 897 | 1901 | 15,876 | - |
Total MMEC | 47,143 | 8062 | 31,500 | 7296 | 6582 | 32,175 | 4113 | 4414 | 6830 | 13,958 | 1603 | 5784 | - | 169,460 |
Ranking | 1 | 2 | 3 | Best |
---|---|---|---|---|
Model | 78 | 88 | 4 | 78 |
Level | Augm | Augm | Augm | Best Model |
LR | 0.0035 | 0.0073 | 0.0096 | 0.0035 |
BS | 1024 | 512 | 512 | 1024 |
Momentum | 0 | 0 | 0 | 0 |
Optimizer | GD | GD | GD | GD |
Number of Images | 1020 | 1020 | 1020 | 8642 |
Validation Accuracy | 0.7768 | 0.7789 | 0.7747 | 0.7768 |
Training Accuracy | 0.8945 | 0.8965 | 0.9238 | 0.8945 |
Test Accuracy | 0.7941 | 0.7775 | 0.7755 | 0.7854 |
M-F1 | 0.7941 | 0.7775 | 0.7755 | 0.7572 |
False | True | OA | |
---|---|---|---|
Blurry | 27 | 32 | 0.54 |
Close | 27 | 32 | 0.54 |
Early | 41 | 18 | 0.31 |
Landscape | 35 | 24 | 0.41 |
Object | 37 | 22 | 0.37 |
Post-harvest | 47 | 12 | 0.20 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Yordanov, M.; d’Andrimont, R.; Martinez-Sanchez, L.; Lemoine, G.; Fasbender, D.; van der Velde, M. Crop Identification Using Deep Learning on LUCAS Crop Cover Photos. Sensors 2023, 23, 6298. https://doi.org/10.3390/s23146298
Yordanov M, d’Andrimont R, Martinez-Sanchez L, Lemoine G, Fasbender D, van der Velde M. Crop Identification Using Deep Learning on LUCAS Crop Cover Photos. Sensors. 2023; 23(14):6298. https://doi.org/10.3390/s23146298
Chicago/Turabian StyleYordanov, Momchil, Raphaël d’Andrimont, Laura Martinez-Sanchez, Guido Lemoine, Dominique Fasbender, and Marijn van der Velde. 2023. "Crop Identification Using Deep Learning on LUCAS Crop Cover Photos" Sensors 23, no. 14: 6298. https://doi.org/10.3390/s23146298
APA StyleYordanov, M., d’Andrimont, R., Martinez-Sanchez, L., Lemoine, G., Fasbender, D., & van der Velde, M. (2023). Crop Identification Using Deep Learning on LUCAS Crop Cover Photos. Sensors, 23(14), 6298. https://doi.org/10.3390/s23146298