EasyIDP: A Python Package for Intermediate Data Processing in UAV-Based Plant Phenotyping
Abstract
:1. Introduction
2. Materials and Methods
2.1. Study Sites and Image Acquisition
2.2. Three-Dimensional Reconstruction by SfM-MVS
2.3. ROI Making
2.4. ROI-Cropping Module
2.5. ROI-Reversing Module
2.5.1. Camera Parameter Loading
2.5.2. Reverse Calculation
2.5.3. Camera Distortion Calibration
2.5.4. Performance Evaluation
2.6. Implementation
3. Results
3.1. ROI Cropping
3.2. ROI Reversing
3.2.1. Reversing Results for Pix4D and Metashape
3.2.2. Reverse Accuracy Evaluation
4. Discussion
4.1. Reconstruction Quality Control and Assessment in Agriculture
4.2. Whole DOM Cropping to Small Parts
4.3. Reverse Calculation
4.4. Future Works
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
Abbreviations
DOM | Digital orthomosaic |
DSM | Digital surface model |
GAN | Generative adversarial network |
GCP | Ground control point |
GIS | Geographic information system |
GUI | Graphical user interface |
HTP | High-throughput phenotyping |
LiDAR | Light detection and ranging |
MKL | (Intel) Math kernel library |
MVS | Multi-view stereo |
PCD | Point cloud |
RGB | Red, green, and blue |
ROI | Region of interest |
RTK | Real-time kinematic |
SfM | Structure from motion |
UAV | Unmanned aerial vehicle |
Appendix A
Appendix B
Appendix C
References
- Wu, S.; Wen, W.; Xiao, B.; Guo, X.; Du, J.; Wang, C.; Wang, Y. An Accurate Skeleton Extraction Approach From 3D Point Clouds of Maize Plants. Front. Plant Sci. 2019, 10, 248. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Ten Harkel, J.; Bartholomeus, H.; Kooistra, L. Biomass and Crop Height Estimation of Different Crops Using UAV-Based Lidar. Remote Sens. 2019, 12, 17. [Google Scholar] [CrossRef] [Green Version]
- Jin, S.; Su, Y.; Song, S.; Xu, K.; Hu, T.; Yang, Q.; Wu, F.; Xu, G.; Ma, Q.; Guan, H.; et al. Non-Destructive Estimation of Field Maize Biomass Using Terrestrial Lidar: An Evaluation from Plot Level to Individual Leaf Level. Plant Methods 2020, 16, 69. [Google Scholar] [CrossRef]
- Sun, S.; Li, C.; Chee, P.W.; Paterson, A.H.; Jiang, Y.; Xu, R.; Robertson, J.S.; Adhikari, J.; Shehzad, T. Three-Dimensional Photogrammetric Mapping of Cotton Bolls in Situ Based on Point Cloud Segmentation and Clustering. ISPRS J. Photogramm. Remote Sens. 2020, 160, 195–207. [Google Scholar] [CrossRef]
- Zhu, B.; Liu, F.; Xie, Z.; Guo, Y.; Li, B.; Ma, Y. Quantification of Light Interception within Image-Based 3D Reconstruction of Sole and Intercropped Canopies over the Entire Growth Season. Ann. Bot. 2020, 126, mcaa046. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Jay, S.; Rabatel, G.; Hadoux, X.; Moura, D.; Gorretta, N. In-Field Crop Row Phenotyping from 3D Modeling Performed Using Structure from Motion. Comput. Electron. Agric. 2015, 110, 70–77. [Google Scholar] [CrossRef] [Green Version]
- Zermas, D.; Morellas, V.; Mulla, D.; Papanikolopoulos, N. 3D Model Processing for High Throughput Phenotype Extraction—The Case of Corn. Comput. Electron. Agric. 2020, 172, 105047. [Google Scholar] [CrossRef]
- Duan, T.; Zheng, B.; Guo, W.; Ninomiya, S.; Guo, Y.; Chapman, S.C. Comparison of Ground Cover Estimates from Experiment Plots in Cotton, Sorghum and Sugarcane Based on Images and Ortho-Mosaics Captured by UAV. Funct. Plant Biol. 2017, 44, 169. [Google Scholar] [CrossRef]
- Hu, P.; Chapman, S.C.; Zheng, B. Coupling of Machine Learning Methods to Improve Estimation of Ground Coverage from Unmanned Aerial Vehicle (UAV) Imagery for High-Throughput Phenotyping of Crops. Funct. Plant Biol. 2021, 48, 766–779. [Google Scholar] [CrossRef]
- Oats, R.; Escobar-Wolf, R.; Oommen, T. Evaluation of Photogrammetry and Inclusion of Control Points: Significance for Infrastructure Monitoring. Data 2019, 4, 42. [Google Scholar] [CrossRef] [Green Version]
- Feldman, A.; Wang, H.; Fukano, Y.; Kato, Y.; Ninomiya, S.; Guo, W. EasyDCP: An Affordable, High-throughput Tool to Measure Plant Phenotypic Traits in 3D. Methods Ecol. Evol. 2021. [Google Scholar] [CrossRef]
- Young, D. Ucdavis/Metashape: Easy, Reproducible Metashape Workflows. Available online: https://github.com/ucdavis/metashape (accessed on 2 June 2021).
- Mortensen, A.K.; Laursen, M.S.; Jørgensen, R.N.; Gislum, R. Drone dataflow—A MATLAB toolbox for extracting plots from images captured by a UAV. In Precision Agriculture ’19; Wageningen Academic Publishers: Montpellier, France, 2019; pp. 959–965. [Google Scholar]
- Guo, W.; Fukano, Y.; Noshita, K.; Ninomiya, S. Field-based Individual Plant Phenotyping of Herbaceous Species by Unmanned Aerial Vehicle. Ecol. Evol. 2020, 10, 12318–12326. [Google Scholar] [CrossRef]
- Fukano, Y.; Guo, W.; Aoki, N.; Ootsuka, S.; Noshita, K.; Uchida, K.; Kato, Y.; Sasaki, K.; Kamikawa, S.; Kubota, H. GIS-Based Analysis for UAV-Supported Field Experiments Reveals Soybean Traits Associated With Rotational Benefit. Front. Plant Sci. 2021, 12, 637694. [Google Scholar] [CrossRef]
- Tresch, L.; Mu, Y.; Itoh, A.; Kaga, A.; Taguchi, K.; Hirafuji, M.; Ninomiya, S.; Guo, W. Easy MPE: Extraction of Quality Microplot Images for UAV-Based High-Throughput Field Phenotyping. Plant Phenomics 2019, 2019, 1–9. [Google Scholar] [CrossRef] [Green Version]
- Guo, W.; Zheng, B.; Potgieter, A.B.; Diot, J.; Watanabe, K.; Noshita, K.; Jordan, D.R.; Wang, X.; Watson, J.; Ninomiya, S.; et al. Aerial Imagery Analysis—Quantifying Appearance and Number of Sorghum Heads for Applications in Breeding and Agronomy. Front. Plant Sci. 2018, 9, 1544. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Ghosal, S.; Zheng, B.; Chapman, S.C.; Potgieter, A.B.; Jordan, D.R.; Wang, X.; Singh, A.K.; Singh, A.; Hirafuji, M.; Ninomiya, S.; et al. A Weakly Supervised Deep Learning Framework for Sorghum Head Detection and Counting. Plant Phenomics 2019, 2019, 1–14. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Zhang, Y.; Teng, P.; Shimizu, Y.; Hosoi, F.; Omasa, K. Estimating 3D Leaf and Stem Shape of Nursery Paprika Plants by a Novel Multi-Camera Photography System. Sensors 2016, 16, 874. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Andújar, D.; Calle, M.; Fernández-Quintanilla, C.; Ribeiro, Á.; Dorado, J. Three-Dimensional Modeling of Weed Plants Using Low-Cost Photogrammetry. Sensors 2018, 18, 1077. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Zhou, J.; Fu, X.; Zhou, S.; Zhou, J.; Ye, H.; Nguyen, H.T. Automated Segmentation of Soybean Plants from 3D Point Cloud Using Machine Learning. Comput. Electron. Agric. 2019, 162, 143–153. [Google Scholar] [CrossRef]
- Martinez-Guanter, J.; Ribeiro, Á.; Peteinatos, G.G.; Pérez-Ruiz, M.; Gerhards, R.; Bengochea-Guevara, J.M.; Machleb, J.; Andújar, D. Low-Cost Three-Dimensional Modeling of Crop Plants. Sensors 2019, 19, 2883. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Pix4D Support Menu Process > Processing Options > 1. Initial Processing > Calibration. Available online: https://support.pix4d.com/hc/en-us/articles/205327965-Menu-Process-Processing-Options-1-Initial-Processing-Calibration (accessed on 14 May 2021).
- Agisoft LCC. Agisoft Metashape User Manual—Professional Edition, Version 1.7. Available online: https://www.agisoft.com/metashape-pro_1_7_en (accessed on 14 May 2021).
- Guo, W.; Carroll, M.E.; Singh, A.; Swetnam, T.; Merchant, N.; Sarkar, S.; Singh, A.K.; Ganapathysubramanian, B. UAS Based Plant Phenotyping for Research and Breeding Applications. Plant Phenomics 2021, 2021, 9840192. [Google Scholar] [CrossRef]
- Zhou, Q.-Y.; Park, J.; Koltun, V. Open3D: A Modern Library for 3D Data Processing. arXiv 2018, arXiv:1801.09847. [Google Scholar]
- van der Walt, S.; Colbert, S.C.; Varoquaux, G. The NumPy Array: A Structure for Efficient Numerical Computation. Comput. Sci. Eng. 2011, 13, 22–30. [Google Scholar] [CrossRef] [Green Version]
- Hunter, J.D. Matplotlib: A 2D Graphics Environment. Comput. Sci. Eng. 2007, 9, 90–95. [Google Scholar] [CrossRef]
- Pix4D Support How Are the Internal and External Camera Parameters Defined? Available online: https://support.pix4d.com/hc/en-us/articles/202559089-How-are-the-Internal-and-External-Camera-Parameters-defined (accessed on 21 October 2020).
- Pix4D Support Yaw, Pitch, Roll and Omega, Phi, Kappa Angles. Available online: https://support.pix4d.com/hc/en-us/articles/202558969-Yaw-Pitch-Roll-and-Omega-Phi-Kappa-angles (accessed on 21 October 2020).
- Everingham, M.; Van Gool, L.; Williams, C.K.I.; Winn, J.; Zisserman, A. The Pascal Visual Object Classes (VOC) Challenge. Int. J. Comput. Vis. 2010, 88, 303–338. [Google Scholar] [CrossRef] [Green Version]
- Liu, F.; Hu, P.; Zheng, B.; Duan, T.; Zhu, B.; Guo, Y. A Field-Based High-Throughput Method for Acquiring Canopy Architecture Using Unmanned Aerial Vehicle Images. Agric. For. Meteorol. 2021, 296, 108231. [Google Scholar] [CrossRef]
- Xiao, S.; Chai, H.; Shao, K.; Shen, M.; Wang, Q.; Wang, R.; Sui, Y.; Ma, Y. Image-Based Dynamic Quantification of Aboveground Structure of Sugar Beet in Field. Remote Sens. 2020, 12, 269. [Google Scholar] [CrossRef] [Green Version]
- Bauer, A.; Bostrom, A.G.; Ball, J.; Applegate, C.; Cheng, T.; Laycock, S.; Rojas, S.M.; Kirwan, J.; Zhou, J. Combining Computer Vision and Deep Learning to Enable Ultra-Scale Aerial Phenotyping and Precision Agriculture: A Case Study of Lettuce Production. Hortic. Res. 2019, 6, 1–12. [Google Scholar] [CrossRef] [Green Version]
- Zhou, C.; Hu, J.; Xu, Z.; Yue, J.; Ye, H.; Yang, G. A Monitoring System for the Segmentation and Grading of Broccoli Head Based on Deep Learning and Neural Networks. Front. Plant Sci. 2020, 11, 402. [Google Scholar] [CrossRef] [PubMed]
- Zhou, C.; Ye, H.; Yu, G.; Hu, J.; Xu, Z. A Fast Extraction Method of Broccoli Phenotype Based on Machine Vision and Deep Learning. Smart Agric. 2020, 2, 121. [Google Scholar] [CrossRef]
- Desai, S.V.; Balasubramanian, V.N.; Fukatsu, T.; Ninomiya, S.; Guo, W. Automatic Estimation of Heading Date of Paddy Rice Using Deep Learning. Plant Methods 2019, 15, 76. [Google Scholar] [CrossRef] [Green Version]
- Lin, Y.-C.; Zhou, T.; Wang, T.; Crawford, M.; Habib, A. New Orthophoto Generation Strategies from UAV and Ground Remote Sensing Platforms for High-Throughput Phenotyping. Remote Sens. 2021, 13, 860. [Google Scholar] [CrossRef]
- Feng, A.; Zhou, J.; Vories, E.; Sudduth, K.A. Evaluation of Cotton Emergence Using UAV-Based Imagery and Deep Learning. Comput. Electron. Agric. 2020, 177, 105711. [Google Scholar] [CrossRef]
- Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Kai, L.; Li, F.-F. ImageNet: A Large-Scale Hierarchical Image Database. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 20–25 June 2009; IEEE: Piscataway, NJ, USA, 2009; pp. 248–255. [Google Scholar]
- Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; Zitnick, C.L. Microsoft COCO: Common Objects in Context. In Proceedings of the Computer Vision—ECCV 2014; Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T., Eds.; Springer International Publishing: Cham, Switzerland, 2014; pp. 740–755. [Google Scholar]
- David, E.; Madec, S.; Sadeghi-Tehran, P.; Aasen, H.; Zheng, B.; Liu, S.; Kirchgessner, N.; Ishikawa, G.; Nagasawa, K.; Badhon, M.A.; et al. Global Wheat Head Detection (GWHD) Dataset: A Large and Diverse Dataset of High-Resolution RGB-Labelled Images to Develop and Benchmark Wheat Head Detection Methods. Plant Phenomics 2020, 2020, 1–12. [Google Scholar] [CrossRef] [PubMed]
- David, E.; Serouart, M.; Smith, D.; Madec, S.; Velumani, K.; Liu, S.; Wang, X.; Espinosa, F.P.; Shafiee, S.; Tahir, I.S.A.; et al. Global Wheat Head Dataset 2021: More Diversity to Improve the Benchmarking of Wheat Head Localization Methods. arXiv 2021, arXiv:2105.07660. [Google Scholar]
- Mikolajczyk, A.; Grochowski, M. Data Augmentation for Improving Deep Learning in Image Classification Problem. In Proceedings of the 2018 International Interdisciplinary PhD Workshop (IIPhDW), Swinoujście, Poland, 9–12 May 2018; IEEE: Piscataway, NJ, USA, 2018; pp. 117–122. [Google Scholar]
- Han, J.; Shi, L.; Yang, Q.; Huang, K.; Zha, Y.; Yu, J. Real-Time Detection of Rice Phenology through Convolutional Neural Network Using Handheld Camera Images. Precis. Agric. 2020, 22, 154–178. [Google Scholar] [CrossRef]
- Perez, L.; Wang, J. The Effectiveness of Data Augmentation in Image Classification Using Deep Learning. arXiv 2017, arXiv:1712.04621. [Google Scholar]
- Beck, M.A.; Liu, C.-Y.; Bidinosti, C.P.; Henry, C.J.; Godee, C.M.; Ajmani, M. An Embedded System for the Automated Generation of Labeled Plant Images to Enable Machine Learning Applications in Agriculture. PLoS ONE 2020, 15, e0243923. [Google Scholar] [CrossRef]
- Perez, F.; Lebret, R.; Aberer, K. Weakly Supervised Active Learning with Cluster Annotation. arXiv 2019, arXiv:1812.11780. [Google Scholar]
- Chandra, A.L.; Desai, S.V.; Balasubramanian, V.N.; Ninomiya, S.; Guo, W. Active Learning with Point Supervision for Cost-Effective Panicle Detection in Cereal Crops. Plant Methods 2020, 16, 34. [Google Scholar] [CrossRef]
- Zhang, W.; Chen, K.; Wang, J.; Shi, Y.; Guo, W. Easy Domain Adaptation Method for Filling the Species Gap in Deep Learning-Based Fruit Detection. Hortic. Res. 2021, 8, 119. [Google Scholar] [CrossRef] [PubMed]
- Hui, F.; Zhu, J.; Hu, P.; Meng, L.; Zhu, B.; Guo, Y.; Li, B.; Ma, Y. Image-Based Dynamic Quantification and High-Accuracy 3D Evaluation of Canopy Structure of Plant Populations. Ann. Bot. 2018, 121, 1079–1088. [Google Scholar] [CrossRef] [PubMed]
- Muangprakhon, R.; Kaewplang, S. Estimation of Paddy Rice Plant Height Using UAV Remote Sensing. Eng. Access 2021, 7, 93–97. [Google Scholar] [CrossRef]
Dataset | Crop | Field Location | Flight Date (yy/mm/dd) | UAV Model | CameraModel | Flight Height (m) | Image Num. | Size of Images (px) |
---|---|---|---|---|---|---|---|---|
1 | Soybean | Tanashi 1 | 19/08/07 | DJI Inspire 1 | FC550 | 30 | 202 | 4608 × 3456 |
2 | Sugar beet | Memuro 2 | 18/06/26 | DJI Phantom 4 v1 | FC6310 | 30 | 120 | 5472 × 3648 |
3 | Wheat | Tanashi | 19/03/14 | DJI Inspire 1 | FC550 | 30 | 138 | 4608 × 3456 |
4 | Orchard | Tanashi | 20/08/06 | DJI Phantom 4 v2 | FC6310S | 50 | 119 | 5472 × 3648 |
5 | Lotus | Tanashi | 17/05/31 | DJI Inspire 1 | FC550 | 30 | 142 | 4608 × 3456 |
6 | Maize | Tanashi | 19/07/29 | DJI Inspire 1 | FC550 | 30 | 138 | 4608 × 3456 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Wang, H.; Duan, Y.; Shi, Y.; Kato, Y.; Ninomiya, S.; Guo, W. EasyIDP: A Python Package for Intermediate Data Processing in UAV-Based Plant Phenotyping. Remote Sens. 2021, 13, 2622. https://doi.org/10.3390/rs13132622
Wang H, Duan Y, Shi Y, Kato Y, Ninomiya S, Guo W. EasyIDP: A Python Package for Intermediate Data Processing in UAV-Based Plant Phenotyping. Remote Sensing. 2021; 13(13):2622. https://doi.org/10.3390/rs13132622
Chicago/Turabian StyleWang, Haozhou, Yulin Duan, Yun Shi, Yoichiro Kato, Seishi Ninomiya, and Wei Guo. 2021. "EasyIDP: A Python Package for Intermediate Data Processing in UAV-Based Plant Phenotyping" Remote Sensing 13, no. 13: 2622. https://doi.org/10.3390/rs13132622