# Leaf Counting with Multi-Scale Convolutional Neural Network Features and Fisher Vector Coding

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Related Work

## 3. Materials

## 4. Proposed Method

#### 4.1. Setting Label According to the Number of Maize Leaves

#### 4.2. Leaf Count Net

#### 4.3. Coding Multi-Scale Feature Maps by Using Fisher Vector(FV)

`×`W, the feature dimension of each feature point is D. Then we can use $X=\{{x}_{H\times W},t=1,2\dots H\times W\}$ to describe the image.

## 5. Results and Discussion

#### 5.1. Implentation Details

#### 5.2. Image Data

#### 5.3. Experimental Results and Comparison with Other Methods

#### 5.4. Misclassified Image Analysis

#### 5.5. The Relationship Between Maize Leaf Number and Water Content

## 6. Conclusions

## Author Contributions

## Funding

## Acknowledgments

## Conflicts of Interest

## References

- Gebbers, R.; Adamchuk, V.I. Precision agriculture and food security. Science
**2010**, 327, 828–831. [Google Scholar] [CrossRef] [PubMed] - Li, Q.; Dong, B.; Qiao, Y.; Liu, M.; Zhang, J. Root growth, available soil water, and water-use efficiency of winter wheat under different irrigation regimes applied at different growth stages in north China. Agric. Water Manag.
**2010**, 97, 1682. [Google Scholar] [CrossRef] - Knezevic, S.Z.; Evans, S.P.; Blankenship, E.E.; Lindquist, A.J.L. Critical period for weed control: The concept and data analysis. Weed Sci.
**2002**, 50, 773–786. [Google Scholar] [CrossRef] - Furbank, R.T.; Tester, M. Phenomics–technologies to relieve the phenotyping bottleneck. Trends Plant Sci.
**2011**, 16, 635–644. [Google Scholar] [CrossRef] [PubMed] - Telfer, A.; Bollman, K.M.; Poethig, R.S. Phase change and the regulation of trichome distribution in Arabidopsis thaliana. Development
**1997**, 124, 645–654. [Google Scholar] [PubMed] - Walter, A.; Schurr, U. The modular character of growth in Nicotiana tabacum plants under steady-state nutrition. J. Exp. Bot.
**1999**, 50, 1169–1177. [Google Scholar] [CrossRef] - Koornneef, M.; Hanhart, C.; van Loenen-Martinet, P.; Blankestijn de Vries, H. The effect of daylength on the transition to flowering in phytochrome-deficient, late-flowering and double mutants of Arabidopsis thaliana. Physiol. Plant.
**1995**, 95, 260–266. [Google Scholar] [CrossRef] - Giuffrida, M.V.; Minervini, M.; Tsaftaris, S.A. Learning to Count Leaves in Rosette Plants. Available online: http://tsaftaris.com/preprints/Giuffrida_CVPPP2015.pdf (accessed on 11 March 2019).
- Ye, M.; Cao, Z.; Yu, Z. An image-based approach for automatic detecting tasseling stage of maize using spatio-temporal saliency. In MIPPR 2013: Remote Sensing Image Processing, Geographic Information Systems, and Other Applications; International Society for Optics and Photonics: Bellingham, WA, USA, 2013. [Google Scholar]
- Zhuang, S.; Wang, P.; Jiang, B.; Li, M.; Gong, Z. Early detection of water stress in maize based on digital images. Comput. Electron. Agric.
**2017**, 140, 461–468. [Google Scholar] [CrossRef] - Cui, S.L.; Tian, F. Face recognition method based on sift feature and fisher. Comput. Eng.
**2009**, 35, 195–197. [Google Scholar] - Szegedy, C.; Vanhoucke, V.; Ioffe, S.; Shlens, J.; Wojna, Z. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 26 June–1 July 2016; pp. 2818–2826. [Google Scholar]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 26 June–1 July 2016; pp. 770–778. [Google Scholar]
- Jeon, W.S.; Rhee, S.Y. Plant leaf recognition using a convolution neural network. Int. J. Fuzzy Logic Intell. Syst.
**2017**, 17, 26–34. [Google Scholar] [CrossRef] - Choi, H. CNN output optimization for more balanced classification. Int. J. Fuzzy Logic Intell. Syst.
**2017**, 17, 98–106. [Google Scholar] [CrossRef] - Li, E.; Xia, J.; Du, P.; Lin, C.; Samat, A. Integrating multilayer features of convolutional neural networks for remote sensing scene classification. IEEE Trans. Geosci. Remote Sens.
**2017**, 55, 1–13. [Google Scholar] [CrossRef] - Yu, K.; Salzmann, M. Second-order convolutional neural networks. Clinic. Immunol. Immunopathol.
**2017**, 66, 230–238. [Google Scholar] - Huang, Z.; Van Gool, L. A Riemannian network for SPD matrix learning. arXiv
**2016**, arXiv:1608.04233. [Google Scholar] - Akodad, S.; Bombrun, L.; Yaacoub, C.; Berthoumieu, Y.; Germain, C. Image classification based on log-Euclidean Fisher Vectors for covariance matrix descriptors. In Proceedings of the 2018 Eighth International Conference on Image Processing Theory, Tools and Applications (IPTA), Xi’an, China, 7–10 November 2018; IEEE: Piscataway, NJ, USA, 2018; pp. 1–6. [Google Scholar]
- Ren, M.; Zemel, R.S. End-to-end instance segmentation with recurrent attention. arXiv
**2017**, arXiv:1605.09410. [Google Scholar] - Donahue, J.; Anne Hendricks, L.; Guadarrama, S.; Rohrbach, M.; Venugopalan, S.; Saenko, K.; Darrell, T. Long-term recurrent convolutional networks for visual recognition and description. arXiv
**2015**, arXiv:1411.4389v4. [Google Scholar] - Pape, J.M.; Klukas, C. Utilizing machine learning approaches to improve the prediction of leaf counts and individual leaf segmentation of rosette plant images. In Proceedings of the Computer Vision Problems in Plant Phenotyping Workshop, Swansea, UK, 7–10 September 2015. [Google Scholar]
- Pape, J.M.; Klukas, C. 3-D Histogram-Based Segmentation and Leaf Detection for Rosette Plants. Asia-pacific Conference on Conceptual Modelling; Australian Computer Society, Inc.: Sydney, Australia, 2015. [Google Scholar]
- Dobrescu, A.; Giuffrida, M.V.; Tsaftaris, S.A. Leveraging multiple datasets for deep leaf counting. In Proceedings of the 2017 IEEE International Conference on Computer Vision Workshop (ICCVW), Venice, Italy, 22–29 October 2017; IEEE: Piscataway, NJ, USA, 2017; pp. 2072–2079. [Google Scholar]
- Aich, S.; Stavness, I. Leaf counting with deep convolutional and deconvolutional networks. arXiv
**2017**, arXiv:1708.07570. [Google Scholar] - Aslam, M.; Maqbool, M.A.; Ceng, R. Drought Stress in Maize (Zea mays L.); Springer Briefs in Agriculture: Berlin, Germany, 2015; Chapter Effect of Drought on Maize; pp. 5–17. [Google Scholar]
- Lowe, D.G. Object recognition from local scale-invariant features. In Proceedings of the 1999 Seventh IEEE International Conference on Computer Vision, Kerkyra, Greece, 20–27 September 1999; IEEE: Piscataway, NJ, USA, 1999; Volume 2, pp. 1150–1157. [Google Scholar]
- Simonyan, K.; Parkhi, O.M.; Vedaldi, A.; Zisserman, A. Fisher Vector Faces in the Wild; University of Oxford: Oxford, UK, 2013; Volume 2, p. 4. [Google Scholar]
- Jaakkola, T.; Haussler, D. Exploiting Generative Models in Discriminative Classifiers. Available online: https://papers.nips.cc/paper/1520-exploiting-generative-models-in-discriminative-classifiers.pdf (accessed on 11 March 2019).
- Perronnin, F.; Dance, C. Fisher Kernels on Visual Vocabularies for Image Categorization. In Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, USA, 17–22 June 2007; IEEE: Piscataway, NJ, USA, 2008. [Google Scholar]
- Koh, J.E.; Ng, E.Y.; Bhandary, S.V.; Hagiwara, Y.; Laude, A.; Acharya, U.R. Automated retinal health diagnosis using pyramid histogram of visual words and Fisher vector techniques. Comput. Biol. Med.
**2018**, 92, 204–209. [Google Scholar] [CrossRef] [PubMed] - Krizhevsky, A.; Sutskever, I.; Hinton, G.E. Imagenet Classification with Deep Convolutional Neural Networks. Available online: https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf (accessed on 11 March 2019).
- Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv
**2014**, arXiv:1409.1556. [Google Scholar]

**Figure 1.**(

**a**)–(

**d**) shows maize image samples at different growth stage. From left to right, the growth days increased in turn.

**Figure 2.**In the left picture, the horizontal axis shows the number of leaves, and the vertical axis represents the number of samples. The content of the right picture is similar to the left one, we reset the sample label according to the distribution of leaf number.

**Figure 7.**In (

**a**

**,b**), the vertical axis represents the correct rate and the loss value, respectively. And the horizontal axis represents the number of iterations.

**Figure 8.**The result of MSE corresponding to different k values. The horizontal axis represents the different k values and the vertical axis represents the mean square error of true value and predict value.

**Figure 9.**Comparison between predicted and real values. (

**a**) Train data. (

**b**) Test data. The horizontal axis represents the absolute value of the difference between the predicted value and the true value, and the vertical axis represents the proportion of samples that satisfying the condition.

**Figure 10.**The performance of MSE. The vertical axis represents the MSE value. And the horizontal axis represents the validation times.

**Figure 11.**(

**a**)–(

**d**) are the samples of detection errors within 2 leaves. We preprocessed the original image. Because when obtaining the image, some leaves of other maize may as the noise sample appear in the image. These noise leaves are artificially filtered out. (

**e**,

**f**) are the samples of detection errors more than 2 leaves.

**Figure 12.**The number of leaves changing with time. Four different color lines represent four different samples.

Sample | Soil Moisture | Depth of Moisture Test (cm) |
---|---|---|

Sample1 | 65%–80% | 20 |

Sample2 | 50%–60% | 20 |

Sample3 | 40%–50% | 20 |

Sample4 | <40% | 20 |

Growth Stage | Feature Description |
---|---|

VE | Emergence |

V1 | One leaf with collar visible |

V2 | Two leaves with collar visible |

Vn | (n) leaves with collar visible |

VT | Last branch of tassel is completely visible |

Range of Leaf Number | Reset Label |
---|---|

[0,6] | 0 |

(6,7] | 1 |

(7,9] | 2 |

(9,13] | 3 |

Batchsize | Epochs | Learning Rate | L2 Weight Decay |
---|---|---|---|

32 | 200 | ${10}^{-3}$ | ${10}^{-4}$ |

Methods | AbsCountDiff | CountDiff | MSE |
---|---|---|---|

(1) Alex-net | Train:1.39 | 0.065 | 3.58 |

Test:1.43 | 0.038 | 3.84 | |

(2) VGG | Train:1.36 | 0.063 | 3.37 |

Test:1.43 | −0.019 | 3.78 | |

(3) Proposed (Leaf-count-net+FV) | Train:0.17 | −0.003 | 0.069 |

Test:0.35 | 0.0018 | 0.31 | |

(4) Sift+FV | Train:0.40 | 0.013 | 0.31 |

Test:0.91 | 0.017 | 1.70 |

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Jiang, B.; Wang, P.; Zhuang, S.; Li, M.; Li, Z.; Gong, Z.
Leaf Counting with Multi-Scale Convolutional Neural Network Features and Fisher Vector Coding. *Symmetry* **2019**, *11*, 516.
https://doi.org/10.3390/sym11040516

**AMA Style**

Jiang B, Wang P, Zhuang S, Li M, Li Z, Gong Z.
Leaf Counting with Multi-Scale Convolutional Neural Network Features and Fisher Vector Coding. *Symmetry*. 2019; 11(4):516.
https://doi.org/10.3390/sym11040516

**Chicago/Turabian Style**

Jiang, Boran, Ping Wang, Shuo Zhuang, Maosong Li, Zhenfa Li, and Zhihong Gong.
2019. "Leaf Counting with Multi-Scale Convolutional Neural Network Features and Fisher Vector Coding" *Symmetry* 11, no. 4: 516.
https://doi.org/10.3390/sym11040516