# End-to-End Multimodal 16-Day Hatching Eggs Classification

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

## 1. Introduction

- In order to solve the problem of different categories possibly having the same image or heartbeat characteristics, this paper designed a network structure that can simultaneously use the time series heartbeat signals and the egg embryo images.
- In order to solve time-series classification problems, this paper designed a six layer-deep temporal convolutional network (TCN) architecture that can model the heartbeat signal.
- We used a pre-training ResNet model to shorten the training time and create a more accurate image classification model.

## 2. Methods

#### 2.1. PicNet Design

#### 2.2. HeartNet Design

#### 2.3. Fusion and Decision Layers Design

## 3. Experiments and Results

#### 3.1. Dataset

#### 3.2. Unimodal Training

#### 3.2.1. PicNet Training

#### 3.2.2. HeartNet Training

#### 3.3. Multimodal Training

^{−4}, the decay rate is 0.9, and the momentum is 0.1. The batch size is 32. With eight NVIDIA GTX 1080 Ti GPUs, it took approximately 3 minutes for one epoch. The loss curve of the training process is shown in Figure 8.

#### 3.4. Results Evaluation

## 4. Conclusions

## Author Contributions

## Funding

## Conflicts of Interest

## References

- Shan, B. Fertility Detection of Middle-stage Hatching Egg in Vaccine Production Using Machine Vision. In Proceedings of the 2nd International Workshop on Education Technology and Computer Science, ETCS 2010, Wuhan, China, 6–7 March 2010; pp. 95–98. [Google Scholar] [CrossRef]
- Lawrence, K.C.; Smith, D.P.; Windham, W.R.; Heitschmidt, G.W.; Park, B. Egg embryo development detection with hyperspectral imaging. In Optics for Natural Resources, Agriculture, and Foods; SPIE: Boston, MA, USA, 2006. [Google Scholar]
- Liu, L.; Ngadi, M.O. Detecting Fertility and Early Embryo Development of Chicken Eggs Using Near-Infrared Hyperspectral Imaging. Food Bioprocess Technol.
**2013**, 6, 2503–2513. [Google Scholar] [CrossRef] - Xu, Q.; Cui, F. Non-destructive Detection on the Fertility of Injected SPF Eggs in Vaccine Manufacture. In Proceedings of the 26th Chinese Control and Decision Conference, CCDC 2014, Changsha, China, 31 May–2 June 2014; pp. 1574–1579. [Google Scholar]
- Krizhevsky, A.; Sutskever, I.; Hinton, G.E. ImageNet Classification with Deep Convolutional Neural Networks. Commun. ACM
**2017**, 60, 84–90. [Google Scholar] [CrossRef] - Szegedy, C.; Liu, W.; Jia, Y.; Sermanet, P.; Reed, S.; Anguelov, D.; Erhan, D.; Vanhoucke, V.; Rabinovich, A. Going Deeper with Convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 8–10 June 2015; pp. 1–9. [Google Scholar]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep Residual Learning for Image Recognition. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition(CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 770–778. [Google Scholar]
- Geng, L.; Yan, T.; Xiao, Z.; Xi, J.; Li, Y. Hatching eggs classification based on deep learning. Multimed. Tools Appl.
**2018**, 77, 22071–22082. [Google Scholar] [CrossRef] - Geng, L.; Hu, Y.; Xiao, Z.; Xi, J. Fertility Detection of Hatching Eggs Based on a Convolutional Neural Network. Appl. Sci.
**2019**, 9, 1408. [Google Scholar] [CrossRef] - Huang, L.; He, A.; Zhai, M.; Wang, Y.; Bai, R.; Nie, X. A Multi-Feature Fusion Based on Transfer Learning for Chicken Embryo Eggs Classification. Symmetry
**2019**, 11, 606. [Google Scholar] [CrossRef] - Graves, A.; Mohamed, A.R.; Hinton, G. Speech Recognition with Deep Recurrent Neural Networks. In Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP), Vancouver, BC, Canada, 26–31 May 2013; pp. 6645–6649. [Google Scholar]
- Brattoli, B.; Buchler, U.; Wahl, A.S.; Schwab, M.E.; Ommer, B. LSTM Self-Supervision for Detailed Behavior Analysis. In Proceedings of the 30th Ieee Conference on Computer Vision and Pattern Recognition (CVPR 2017), Honolulu, HI, USA, 21–26 July 2017; pp. 3747–3756. [Google Scholar]
- Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; Ma, S.; Huang, Z.H.; Karpathy, A.; Khosla, A.; Bernstein, M.; et al. ImageNet Large Scale Visual Recognition Challenge. Int. J. Comput. Vis.
**2015**, 115, 211–252. [Google Scholar] [CrossRef] [Green Version] - Lea, C.; Vidal, R.; Reiter, A.; Hager, G.D. Temporal Convolutional Networks: A Unified Approach to Action Segmentation. In European Conference on Computer Vision(ECCV); Springer: Berlin/Heidelberg, Germany, 2016; pp. 47–54. [Google Scholar] [Green Version]
- Wang, Z.; Yan, W.; Oates, T. Time Series Classification from Scratch with Deep Neural Networks: A Strong Baseline. In Proceedings of the 2017 International Joint Conference on Neural Networks(IJCNN), Anchorage, AK, USA, 14–19 May 2017; pp. 1578–1585. [Google Scholar]
- Hochreiter, S.; Schmidhuber, J. Long short-term memory. Neural comput.
**1997**, 9, 1735–1780. [Google Scholar] [CrossRef] [PubMed] - Graves, A. Supervised sequence labelling. In Supervised Sequence Labelling with Recurrent Neural Networks; Springer: Berlin/Heidelberg, Germany, 2012; pp. 5–13. [Google Scholar]
- Karim, F.; Majumdar, S.; Darabi, H.; Chen, S. LSTM Fully Convolutional Networks for Time Series Classification. IEEE Access
**2018**, 6, 1662–1669. [Google Scholar] [CrossRef] - Allen, J. Photoplethysmography and its application in clinical physiological measurement. Physiol. Meas.
**2007**, 28, R1–R39. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Cho, K.; Van Merriënboer, B.; Bahdanau, D.; Bengio, Y. On the properties of neural machine translation: Encoder-decoder approaches. arXiv
**2014**, arXiv:1409.1259. [Google Scholar] - Kingma, D.P.; Ba, J. Adam: A method for stochastic optimization. arXiv
**2014**, arXiv:1412.6980. [Google Scholar] - Ruder, S. An overview of gradient descent optimization algorithms. arXiv
**2016**, arXiv:1609.04747. [Google Scholar] - Duchi, J.; Hazan, E.; Singer, Y. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. J. Mach. Learn. Res.
**2011**, 12, 2121–2159. [Google Scholar]

**Figure 1.**Three categories of 16-day hatching eggs. The first row contains pictures of the hatching eggs. The second row contains photoplethysmography (PPG) signals. (

**a**) The fertile egg. (

**b**) The recovered egg has three sets of characteristics. (

**c**) The waste egg.

**Figure 2.**The proposed multimodal network architecture. It is divided into a picture processing network and a heartbeat signal processing network.

**Figure 5.**The signal acquisition equipment. The laser source uses a near-infrared source with a wavelength of 808 nm. The receiving terminal module uses the AFE4490 chip, which designed by Texas Instruments for signal denoising and A/D conversion.

**Figure 6.**Processed 16-days hatching egg signal. The first row shows pictures of hatching eggs. The blood vessels of the hatching egg are apparent. The second row shows the PPG signal, which reflects heartbeat information. (

**a**) The fertile egg. (

**b**) The recovered egg. (

**c**) The waste egg.

**Figure 7.**Loss and accuracy curves of different models on picture dataset. (

**a**) Loss curve of different models. (

**b**) Accuracy curve of different models.

**Figure 8.**Varying curves of loss. (

**a**) Loss curve of training dataset. (

**b**) Loss curve of validation dataset.

Network | Layer Name | Layer Type | Related Parameters |
---|---|---|---|

HeartNet | Conv1 | Conv1D | 5 kernelsize,1stride,128 |

Pool1 | Max Pooling | 5 kernelsize,1stride | |

Conv2 | Conv1D | 5 kernelsize,1stride,256 | |

Pool2 | Max Pooling | 5 kernelsize,1stride | |

Conv3 | Conv1D | 5 kernelsize,1stride,128 | |

Pool3 | Average Pooling | 4 kernelsize,1stride | |

PicNet | ResNet-50 [7] | \ | \ |

Fusion and decision | LSTM | LSTM | 350 hidden units |

Dropout | Dropout | dropout-ratio 0.5 | |

FC | Fully connected | \ |

Type | Train | Valid | Test | Total |
---|---|---|---|---|

fertile eggs | 1253 | 418 | 417 | 2088 |

waste eggs | 1296 | 432 | 432 | 2160 |

recovered eggs | 1728 | 576 | 576 | 2880 |

total | 4277 | 1426 | 1425 | 7128 |

Model | Accuracy |
---|---|

AlexNet | 82.56% |

VGG-13 | 85.34% |

VGG-16 | 85.78% |

ResNet-50 | 90.92% |

k | Accuracy |
---|---|

3 | 75.23% |

4 | 77.56% |

5 | 77.78% |

6 | 77.68% |

Layer Name | Layer Type | Related Parameters |
---|---|---|

LSTM1 | LSTM | 150 hidden units |

LSTM2 | LSTM | 75 hidden units |

Dropout | Dropout | dropout-ratio 0.5 |

FC | Fully connected |

Layer Name | Layer Type | Related Parameters |
---|---|---|

GRU1 | GRU | 150 hidden units |

GRU2 | GRU | 75 hidden units |

Dropout | Dropout | dropout-ratio 0.5 |

FC | Fully connected |

Model | Accuracy |
---|---|

LSTM | 60.23% |

GRU | 58.31% |

Ours | 77.78% |

Model | Dataset | Signal Type | Accuracy | Recall _{micro} | F1 _{micro} |
---|---|---|---|---|---|

PicNet | Egg picture | Picture | 90.92% | 89.86% | 89.99% |

HeartNet | Egg heart | Sequence | 77.78% | 77.82% | 77.80% |

Multimodal | Mixed | Mixed | 98.98% | 98.95% | 98.90% |

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Geng, L.; Peng, Z.; Xiao, Z.; Xi, J.
End-to-End Multimodal 16-Day Hatching Eggs Classification. *Symmetry* **2019**, *11*, 759.
https://doi.org/10.3390/sym11060759

**AMA Style**

Geng L, Peng Z, Xiao Z, Xi J.
End-to-End Multimodal 16-Day Hatching Eggs Classification. *Symmetry*. 2019; 11(6):759.
https://doi.org/10.3390/sym11060759

**Chicago/Turabian Style**

Geng, Lei, Zhen Peng, Zhitao Xiao, and Jiangtao Xi.
2019. "End-to-End Multimodal 16-Day Hatching Eggs Classification" *Symmetry* 11, no. 6: 759.
https://doi.org/10.3390/sym11060759