# Image Generation of Tomato Leaf Disease Identification Based on Adversarial-VAE

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Related Work

#### 2.1. Generative Adversarial Network (GAN)

#### 2.2. Variational Auto-Encoder (VAE)

#### 2.3. VAE-GAN

#### 2.4. Two-Stage VAE

## 3. Materials and Methods

#### 3.1. Dataset

#### 3.2. Adversarial-VAE Model for Generating Tomato Leaf Disease Images

#### 3.2.1. Adversarial-VAE Model

#### 3.2.2. Components of Stage 1

#### 3.2.3. Components of Stage 2

#### 3.3. Improved Adversarial-VAE Model

#### 3.3.1. Multi-Scale Convolution

#### 3.3.2. Dense Connection Strategy

#### 3.4. Loss Function

Algorithm 1: The training pipeline of the stage 1. |

Initial parameters of the models: ${\theta}_{e}$$,{\theta}_{g}$$,{\theta}_{d}$ |

while training do |

${x}^{real}\leftarrow $ batch of images sampled from the dataset. |

${{z}_{\mu}}^{real},{{z}_{\sigma}}^{real}\leftarrow {E}_{{\theta}_{e}}({x}^{real})$ |

${z}^{real}\leftarrow {{z}_{\mu}}^{real}+\epsilon {{z}_{\sigma}}^{real}$$\mathrm{with}\epsilon \sim N\left(0,Id\right)$ |

${\overline{x}}^{real}\leftarrow {G}_{{\theta}_{g}}({z}^{real})$ |

${z}^{fake}\leftarrow $$\mathrm{prior}P(z)$ |

${x}^{fake}\leftarrow {G}_{{\theta}_{g}}({z}^{fake})$ |

{Compute losses gradients and update parameters.} |

${\theta}_{e}\overline{\leftarrow}\Vert {\overline{x}}^{real}-{x}^{real}\Vert +KL(P({z}^{real}|{x}^{real})\Vert P(z))$ |

${\theta}_{g}\overline{\leftarrow}\Vert {\overline{x}}^{real}-{x}^{real}\Vert -{D}_{{\theta}_{d}}({\overline{x}}^{real})-{D}_{{\theta}_{d}}({x}^{fake})$ |

${\theta}_{d}\overline{\leftarrow}{D}_{{\theta}_{d}}({\overline{x}}^{real})+{D}_{{\theta}_{d}}({x}^{fake})-{D}_{{\theta}_{d}}({x}^{real})$ |

end while |

Algorithm 2: The training pipeline of the stage 2. |

Initial parameters of the models: ${\theta}_{e}$$,{\theta}_{d}$. |

while training do |

${z}^{real}\leftarrow $ Gaussian distribution. |

${{u}_{\mu}}^{real},{{u}_{\sigma}}^{real}\leftarrow {E}_{{\theta}_{e}}({z}^{real})$. |

${u}^{real}\leftarrow {{u}_{\mu}}^{real}+{\epsilon {u}_{\sigma}}^{real}$$\mathrm{with}\epsilon \sim N\left(0,Id\right)$. |

${\overline{z}}^{real}\leftarrow {D}_{{\theta}_{d}}({u}^{real})$. |

${u}^{fake}\leftarrow $$\mathrm{prior}P(u)$. |

${z}^{fake}\leftarrow {D}_{{\theta}_{d}}({u}^{fake})$. |

{Compute losses gradients and update parameters.} |

${\theta}_{e}\overline{\leftarrow}\Vert {\overline{z}}^{real}-{z}^{real}\Vert +KL(P({u}^{real}|{z}^{real})\Vert P(u))$. |

${\theta}_{d}\overline{\leftarrow}\Vert {\overline{z}}^{real}-{z}^{real}\Vert $. |

end while |

#### 3.5. Experimental Setup

#### 3.6. Performance Evaluation Metrics

## 4. Results and Discussion

#### 4.1. Generation Results and Analysis

#### 4.2. Identification Results and Analysis

## 5. Conclusions

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Conflicts of Interest

## References

- Stewart, E.L.; Wiesner-Hanks, T.; Kaczmar, N.; DeChant, C.; Wu, H.; Lipson, H.; Nelson, R.J.; Gore, M.A. Quantitative Phenotyping of Northern Leaf Blight in UAV Images Using Deep Learning. Remote Sens.
**2019**, 11, 2209. [Google Scholar] [CrossRef][Green Version] - Zhang, Y.; Song, C.; Zhang, D. Deep learning-based object detection improvement for tomato disease. IEEE Access
**2020**, 8, 56607–56614. [Google Scholar] [CrossRef] - Xie, X.; Ma, Y.; Liu, B.; He, J.; Li, S.; Wang, H. A Deep-Learning-Based Real-Time Detector for Grape Leaf Diseases Using Improved Convolutional Neural Networks. Front. Plant Sci.
**2020**, 11, 751. [Google Scholar] [CrossRef] [PubMed] - Saleem, M.H.; Potgieter, J.; Arif, K.M. Plant Disease Detection and Classification by Deep Learning. Plants
**2019**, 11, 468. [Google Scholar] [CrossRef] [PubMed][Green Version] - Afzaal, H.; Farooque, A.A.; Schumann, A.W.; Hussain, N.; McKenzie-Gopsill, A.; Esau, T.; Abbas, F.; Acharya, B. Detection of a Potato Disease (Early Blight) Using Artificial Intelligence. Remote Sens.
**2021**, 13, 411. [Google Scholar] [CrossRef] - Wu, Y.; Xu, L. Crop Organ Segmentation and Disease Identification Based on Weakly Supervised Deep Neural Network. Agronomy
**2019**, 9, 737. [Google Scholar] [CrossRef][Green Version] - Tetila, E.C.; Machado, B.B.; Menezes, G.K.; Oliveira, A.D.S.; Alvarez, M.; Amorim, W.P.; De Souza Belete, N.A.; Goncalves Da Silva, G.; Pistori, H. Automatic recognition of soybean leaf diseases using UAV images and deep convolutional neural networks. IEEE Geosci Remote Sens Lett.
**2019**, 17, 903–907. [Google Scholar] [CrossRef] - Yu, H.J.; Son, C.H. Leaf spot attention network for apple leaf disease identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Seattle, WA, USA, 14–19 June 2020; pp. 52–53. [Google Scholar]
- Liu, X.; Min, W.; Mei, S.; Wang, L.; Jiang, S. Plant Disease Recognition: A Large-Scale Benchmark Dataset and a Visual Region and Loss Reweighting Approach. IEEE Trans. Image Process.
**2021**, 30, 2003–2015. [Google Scholar] [CrossRef] [PubMed] - Jiang, P.; Chen, Y.; Liu, B.; He, D.; Liang, C. Real-Time Detection of Apple Leaf Diseases Using Deep Learning Approach Based on Improved Convolutional Neural Networks. IEEE Access
**2019**, 7, 59069–59080. [Google Scholar] [CrossRef] - Zhu, J.; Park, T.; Isola, P.; Efros, A. Unpaired Image-To-Image Translation Using Cycle-Consistent Adversarial Networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 22–29 October 2017; pp. 2223–2232. [Google Scholar]
- Ke, X.; Zou, J.; Niu, Y. End-to-End Automatic Image Annotation Based on Deep CNN and Multi-Label Data Augmentation. IEEE Trans. Multimed.
**2019**, 21, 2093–2106. [Google Scholar] [CrossRef] - Tran, N.T.; Tran, V.H.; Nguyen, N.B.; Nguyen, T.K.; Cheung, N.M. On data augmentation for GAN training. IEEE Trans. Image Process.
**2021**, 30, 1882–1897. [Google Scholar] [CrossRef] [PubMed] - Konidaris, F.; Tagaris, T.; Sdraka, M.; Stafylopatis, A. Generative adversarial networks as an advanced data augmentation technique for MRI data. In Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2019), Prague, Czech Republic, 25–27 February 2019; Volume 5, pp. 48–59. [Google Scholar]
- Liu, B.; Tan, C.; Li, S.; He, J.; Wang, H. A Data Augmentation Method Based on Generative Adversarial Networks for Grape Leaf Disease Identification. IEEE Access
**2020**, 8, 102188–102198. [Google Scholar] [CrossRef] - Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative adversarial networks. arXiv
**2014**, arXiv:1406.2661. [Google Scholar] - Kingma, D.P.; Welling, M. Auto-encoding variational Bayes. arXiv
**2013**, arXiv:1312.6114. [Google Scholar] - Radford, A.; Metz, L.; Chintala, S. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. arXiv
**2015**, arXiv:1511.06434. [Google Scholar] - Chen, X.; Duan, Y.; Houthooft, R.; Schulman, J.; Sutskever, I.; Abbeel, P. InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets. In Proceedings of the 30th Annual Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain, 5–10 December 2016; pp. 2180–2188. [Google Scholar]
- Arjovsky, M.; Chintala, S.; Bottou, L. Wasserstein GAN. arXiv
**2017**, arXiv:1701.07875. [Google Scholar] - Tolstikhin, I.; Bousquet, O.; Gelly, S.; Schölkopf, B. Wasserstein Auto-Encoders. arXiv
**2017**, arXiv:1711.01558. [Google Scholar] - Dai, B.; Wipf, D. Diagnosing and Enhancing VAE Models. arXiv
**2019**, arXiv:1903.05789. [Google Scholar] - Larsen, A.B.L.; Snderby, S.K.; Larochelle, H.; Winther, O. Autoencoding beyond pixels using a learned similarity metric. In Proceedings of the 33rd International Conference on Machine Learning (ICML 2016), New York City, NY, USA, 19–24 June 2016; Volume 4, pp. 2341–2349. [Google Scholar]
- Hughes, D.; Salathe, M. An open access repository of images on plant health to enable the development of mobile disease diagnostics. arXiv
**2015**, arXiv:1511.08060. [Google Scholar] - Szegedy, C.; Liu, W.; Jia, Y.; SErmanet, P.; Reed, S.; Anguelov, D.; Erhan, D.; Vanhoucke, V.; Rabinovich, A. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 1–9. [Google Scholar]
- Huang, G.; Liu, Z.; Van Der Maaten, L.; Weinberger, K.Q. Densely connected convolutional networks. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Los Alamitos, CA, USA, 21–26 July 2017; pp. 4700–4708. [Google Scholar]
- Heusel, M.; Ramsauer, H.; Unterthiner, T.; Nessler, B.; Hochreiter, S. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Proceedings of the 31st Annual Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, 4–9 December 2017; pp. 6629–6640. [Google Scholar]
- Salimans, T.; Goodfellow, I.; Zaremba, W.; Cheung, V.; Radford, A.; Chen, X. Improved techniques for training gans. In Proceedings of the 30th Annual Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain, 5–10 December 2016; pp. 2234–2242. [Google Scholar]
- Barratt, S.; Sharma, R. A Note on the Inception Score. arXiv
**2018**, arXiv:1801.01973. [Google Scholar]

**Figure 2.**Examples of tomato leaf diseases: healthy, Tomato bacterial spot (TBS), Tomato early blight (TEB), Tomato late blight (TLB), Tomato leaf mold (TLM), Tomato mosaic virus (TMV), Tomato septoria leaf spot (TSLS), Tomato target spot (TTS), Tomato two-spotted spider mite (TTSSM), and Tomato yellow leaf curl virus (TYLCV), respectively.

**Figure 11.**Examples of tomato diseased leaf generated by improved Adversarial-VAE networks: healthy, TBS, TEB, TLB, TLM, TMV, TSLS, TTS, TTSSM, and TYLCV, respectively.

**Figure 12.**Examples of tomato diseased leaf generated by VAE networks: healthy, TBS, TEB, TLB, TLM, TMV, TSLS, TTS, TTSSM, and TYLCV, respectively.

Layer | Input | Conv | Scale 0 | Downsample 0 | Scale 1 | Downsample 1 |
---|---|---|---|---|---|---|

Size | 128 × 128 × 3 | 128 × 128 × 16 | 128 × 128 × 16 | 64 × 64 × 32 | 64 × 64 × 32 | 32 × 32 × 64 |

Layer | …… | Downsample 3 | Scale 4 | Reducemean | Scale_fc | FC |

Size | …… | 8 × 8 × 256 | 8 × 8 × 256 | 256 | 256 | 256 |

Layer | Input | FC | Reshape | Upsample 0 | Scale 0 | Upsample 1 |
---|---|---|---|---|---|---|

Size | 256 | 4096 | 2 × 2 × 1024 | 4 × 4 × 512 | 4 × 4 × 512 | 8 × 8 × 256 |

Layer | …… | Upsample 4 | Scale 4 | Upsample 5 | Scale 5 | Conv |

Size | …… | 64 × 64 × 32 | 64 × 64 × 32 | 128 × 128 × 16 | 128 × 128 × 16 | 128 × 128 × 3 |

Layer | Input | Conv | Scale 0 | Downsample 0 | Scale 1 | Downsample 1 |
---|---|---|---|---|---|---|

Size | 128 × 128 × 3 | 128 × 128 × 16 | 128 × 128 × 16 | 64 × 64 × 32 | 64 × 64 × 32 | 32 × 32 × 64 |

Layer | …… | Downsample 3 | Scale 4 | Reducemean | Scale_fc | FC |

Size | …… | 8 × 8 × 256 | 8 × 8 × 256 | 256 | 256 | 1 |

Class | All Sample Numbers | 60% of Sample Numbers |
---|---|---|

healthy | 1592 | 954 |

TBS | 2127 | 1276 |

TEB | 1000 | 600 |

TLB | 1910 | 1145 |

TLM | 952 | 571 |

TMV | 373 | 223 |

TSLS | 1771 | 1062 |

TTS | 1404 | 842 |

TTSSM | 1676 | 1005 |

TYLCV | 5357 | 3214 |

ALL | 18,162 | 10,892 |

Reconstruction-FID | InfoGAN [19] | WAE [21] | VAE [17] | VAE-GAN [23] | 2VAE [22] | Adversarial-VAE |
---|---|---|---|---|---|---|

healthy | 172.61 | 129.47 | 155.64 | 130.08 | 155.64 | 130.08 |

TBS | 135.29 | 103.11 | 148.07 | 114.24 | 148.07 | 114.24 |

TEB | 126.96 | 106.69 | 138.87 | 100.59 | 138.87 | 100.59 |

TLB | 180.10 | 111.81 | 169.80 | 119.23 | 169.80 | 119.23 |

TLM | 160.93 | 133.79 | 161.37 | 147.08 | 161.37 | 147.08 |

TMV | 144.71 | 125.86 | 157.20 | 140.23 | 157.20 | 140.23 |

TSLS | 120.24 | 90.43 | 139.41 | 108.57 | 139.41 | 108.57 |

TTS | 107.88 | 81.74 | 137.89 | 99.67 | 137.89 | 99.67 |

TTSSM | 114.22 | 91.23 | 141.42 | 106.89 | 141.42 | 106.89 |

TYLCV | 140.11 | 83.23 | 133.05 | 79.76 | 133.05 | 79.76 |

AVERAGE | 140.31 | 105.74 | 148.27 | 114.63 | 148.27 | 114.63 |

Generation-FID | InfoGAN [19] | WAE [21] | VAE [17] | VAE-GAN [23] | 2VAE [22] | Adversarial-VAE |
---|---|---|---|---|---|---|

healthy | 221.86 | 202.06 | 186.37 | 167.46 | 179.83 | 162.57 |

TBS | 232.88 | 221.85 | 190.71 | 178.75 | 187.09 | 179.96 |

TEB | 183.09 | 169.42 | 158.43 | 132.42 | 153.65 | 133.65 |

TLB | 277.65 | 227.51 | 192.38 | 184.64 | 199.17 | 180.71 |

TLM | 235.07 | 219.42 | 200.15 | 200.90 | 196.47 | 197.45 |

TMV | 210.91 | 211.38 | 191.24 | 214.60 | 196.78 | 210.54 |

TSLS | 199.31 | 182.59 | 156.61 | 148.31 | 152.93 | 146.11 |

TTS | 199.87 | 208.23 | 191.90 | 163.99 | 185.01 | 161.07 |

TTSSM | 195.08 | 210.70 | 175.97 | 147.95 | 173.95 | 146.83 |

TYLCV | 182.74 | 172.82 | 151.22 | 99.60 | 146.89 | 98.76 |

AVERAGE | 213.85 | 202.60 | 179.50 | 163.86 | 177.18 | 161.77 |

Generation-FID | Adversarial-VAE Alone | Adversarial-VAE + Multi-Scale Convolution | Adversarial-VAE + Dense Connection Strategy | Improved Adversarial-VAE |
---|---|---|---|---|

healthy | 162.57 | 162.64 | 167.63 | 171.63 |

TBS | 179.96 | 170.29 | 176.3 | 167.53 |

TEB | 133.65 | 128.28 | 130.81 | 126.84 |

TLB | 180.71 | 175.15 | 170.42 | 166.92 |

TLM | 197.45 | 194.81 | 191.42 | 187.79 |

TMV | 210.54 | 202.39 | 198.28 | 189.09 |

TSLS | 146.11 | 151.91 | 147.11 | 151.8 |

TTS | 161.07 | 155.89 | 166.72 | 165.84 |

TTSSM | 146.83 | 144.54 | 143.74 | 142.32 |

TYLCV | 98.76 | 98.31 | 98.64 | 99.79 |

AVERAGE | 161.77 | 158.42 | 159.11 | 156.96 |

**Table 8.**Classification accuracy of the classification network trained with the expanded training set generated by different generative methods.

Classification Alone | InfoGAN + Classification | WAE + Classification | VAE + Classification | VAE-GAN + Classification | 2VAE + Classification | Improved Adversarial-VAE + Classification | |
---|---|---|---|---|---|---|---|

Accuracy | 82.87% | 82.42% | 82.16% | 84.65% | 86.86% | 85.43% | 88.43% |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Wu, Y.; Xu, L.
Image Generation of Tomato Leaf Disease Identification Based on Adversarial-VAE. *Agriculture* **2021**, *11*, 981.
https://doi.org/10.3390/agriculture11100981

**AMA Style**

Wu Y, Xu L.
Image Generation of Tomato Leaf Disease Identification Based on Adversarial-VAE. *Agriculture*. 2021; 11(10):981.
https://doi.org/10.3390/agriculture11100981

**Chicago/Turabian Style**

Wu, Yang, and Lihong Xu.
2021. "Image Generation of Tomato Leaf Disease Identification Based on Adversarial-VAE" *Agriculture* 11, no. 10: 981.
https://doi.org/10.3390/agriculture11100981