# Evolutionary Hierarchical Sparse Extreme Learning Autoencoder Network for Object Recognition

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Background and Preliminaries

#### 2.1. Differential Evolution

- Initialize the population randomly or according to some specific distribution.
- Pick out two individuals from the population randomly and compute the weighted difference.
- Carry out the mutation according to Equation (1).
- Conduct crossover between the mutated and the original individuals.
- Evaluate the fitness values of all the individuals in both the original population and the one after crossover.
- Select individuals according to the fitness values to form the population of the next generation.

#### 2.2. Hierarchical Extreme Learning Machine

## 3. Evolutionary Hierarchical Extreme Learning Network

#### 3.1. Initialize the Population and Define the Fitness Function

#### 3.2. Mutate and Crossover the Individuals

#### 3.3. Select Predominant Individuals

## 4. Experimental Results and Discussion

#### 4.1. Comparison with HELM and Analysis

#### 4.2. Comparison with Relevant State-of-the-Art Methods and Analysis

#### 4.3. Application on a Real Complex Dataset

## 5. Conclusions

## Author Contributions

## Funding

## Acknowledgments

## Conflicts of Interest

## Abbreviations

ELM | Extreme Learning Machine |

HELM | Hierarchical Extreme Learning Machine |

EHELN | Evolutionary Hierarchical Extreme Learning Network |

DL | Deep Learning |

SAE | Stacked Autoencoder |

SDA | Stacked Denoising Autoencoder |

DBN | Deep Belief Network |

DBM | Deep Boltzmann Machine |

ML-ELM | Multi-Layer Extreme Learning Machine |

SLFN | Single Hidden Layer Feedforward Network |

## References

- Bengio, Y. Learning deep architectures for AI. Found. Trends Mach. Learn.
**2009**, 2, 1–127. [Google Scholar] [CrossRef] - Bengio, Y.; Courville, A.; Vincent, P. Representation learning: A review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell.
**2013**, 35, 1798–1828. [Google Scholar] [CrossRef] [PubMed] - LeCun, Y.; Bengio, Y.; Hinton, G. Deep learning. Nature
**2015**, 521, 436–444. [Google Scholar] [CrossRef] [PubMed] - Hinton, G.E.; Salakhutdinov, R.R. Reducing the dimensionality of data with neural networks. Science
**2006**, 313, 504–507. [Google Scholar] [CrossRef] [PubMed] - Dahl, G.E.; Yu, D.; Deng, L.; Acero, A. Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition. IEEE Trans. Audio Speech Lang. Process.
**2012**, 20, 30–42. [Google Scholar] [CrossRef] - Graves, A.; Mohamed, A.R.; Hinton, G. Speech recognition with deep recurrent neural networks. In Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, BC, Canada, 26–31 May 2013; pp. 6645–6649. [Google Scholar]
- Jarrett, K.; Kavukcuoglu, K.; LeCun, Y. What is the best multi-stage architecture for object recognition? In Proceedings of the 2009 IEEE 12th International Conference on Computer Vision, Kyoto, Japan, 29 September–2 October 2009; pp. 2146–2153. [Google Scholar]
- Krizhevsky, A.; Sutskever, I.; Hinton, G.E. ImageNet Classification with Deep Convolutional Neural Networks. In Proceedings of the Neural Information and Processing Systems, Lake Tahoe, NV, USA, 3–6 December 2012. [Google Scholar]
- Mesnil, G.; Dauphin, Y.; Glorot, X.; Rifai, S.; Bengio, Y.; Goodfellow, I.; Lavoie, E.; Muller, X.; Desjardins, G.; Warde-Farley, D.; et al. Unsupervised and transfer learning challenge: A deep learning approach. J. Mach. Learn. Res.
**2011**, 7, 97–110. [Google Scholar] - Bengio, Y. Deep learning of representations for unsupervised and transfer learning. J. Mach. Learn. Res.
**2012**, 27, 17–37. [Google Scholar] - Huang, G.B.; Zhu, Q.Y.; Siew, C.K. Extreme learning machine: Theory and applications. Neurocomputing
**2006**, 70, 489–501. [Google Scholar] [CrossRef] [Green Version] - Huang, G.B. An insight into extreme learning machines: Random neurons, random features and kernels. Cognit. Comput.
**2014**, 6, 376–390. [Google Scholar] [CrossRef] - Huang, G.B.; Zhu, Q.Y.; Siew, C.K. Extreme learning machine: A new learning scheme of feedforward neural networks. In Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN2004), Budapest, Hungary, 25–29 July 2004; pp. 985–990. [Google Scholar]
- Huang, G.B.; Zhou, H.; Ding, X.; Zhang, R. Extreme learning machine for regression and multiclass classification. IEEE Trans. Syst. Man Cybern. B
**2012**, 42, 513–529. [Google Scholar] [CrossRef] [PubMed] - Huang, Z.; Yu, Y.; Gu, J.; Liu, H. An Efficient Method for Traffic Sign Recognition Based on Extreme Learning Machine. IEEE Trans. Cybern.
**2017**, 47, 920–933. [Google Scholar] [CrossRef] [PubMed] - Zeng, Y.; Xu, X.; Fang, Y.; Zhao, K. Traffic sign recognition using deep convolutional networks and extreme learning machine. In Proceedings of the International Conference on Intelligent Science and Big Data Engineering, Suzhou, China, 14–16 June 2015; pp. 272–280. [Google Scholar]
- Zeng, Y.; Xu, X.; Shen, D.; Fang, Y.; Xiao, Z. Traffic sign recognition using kernel extreme learning machines with deep perceptual features. IEEE Trans. Intell. Transp. Syst.
**2016**, 18, 1647–1653. [Google Scholar] [CrossRef] - Zhang, L.; He, Z.; Liu, Y. Deep object recognition across domains based on adaptive extreme learning machine. Neural Comput.
**2017**, 239, 194–203. [Google Scholar] [CrossRef] - Kasun, L.L.C.; Zhou, H.; Huang, G.B.; Vong, C.M. Representational learning with extreme learning machine for big data. IEEE Intell. Syst.
**2013**, 28, 31–34. [Google Scholar] - Tissera, M.D.; McDonnell, M.D. Deep extreme learning machines: Supervised autoencoding architecture for classification. Neural Comput.
**2016**, 174, 42–49. [Google Scholar] [CrossRef] - Beck, A.; Teboulle, M. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci.
**2009**, 2, 183–202. [Google Scholar] [CrossRef] - Tang, J.; Deng, C.; Huang, G.B. Extreme learning machine for multilayer perceptron. IEEE Trans. Neural Netw. Learn. Syst.
**2016**, 27, 809–821. [Google Scholar] [CrossRef] [PubMed] - LeCun, Y.; Bottou, L.; Bengio, Y.; Haffner, P. Gradient-Based Learning Applied to Document Recognition. Proc. IEEE
**1998**, 86, 2278–2324. [Google Scholar] [CrossRef] - LeCun, Y.; Huang, F.J.; Bottou, L. Learning Methods for Generic Object Recognition with Invariance to Pose and Lighting. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Washington, DC, USA, 27 June–2 July 2004; pp. II-97–CII-104. [Google Scholar]
- Storn, R.; Price, K. Differential evolution–A simple and efficient heuristic for global optimization over continuous spaces. J. Glob. Optim.
**1997**, 11, 341–359. [Google Scholar] [CrossRef] - Huang, G.B.; Chen, L.; Siew, C.K. Universal approximation using incremental constructive feedforward networks with random hidden nodes. IEEE Trans. Neural Netw.
**2006**, 17, 879–892. [Google Scholar] - Zhu, Q.Y.; Qin, A.K.; Suganthan, P.N.; Huang, G.B. Evolutionary extreme learning machine. Pattern Recognit.
**2005**, 38, 1759–1763. [Google Scholar] [CrossRef] - Huang, G.B.; Bai, Z.; Kasun, L.L.C.; Vong, C.M. Local receptive fields based extreme learning machine. IEEE Comput. Intell. Mag.
**2015**, 10, 18–29. [Google Scholar] [CrossRef] - Bartlett, P.L. The sample complexity of pattern classification with neural networks: The size of the weights is more important than the size of the network. IEEE Trans. Inf. Theory
**1998**, 44, 525–536. [Google Scholar] [CrossRef] [Green Version] - Vincent, P.; Larochelle, H.; Bengio, Y.; Manzagol, P.A. Extracting and Composing Robust Features with Denoising Autoencoders. In Proceedings of the 25th International Conference on Machine Learning, Helsinki, Finland, 5–9 July 2008. [Google Scholar]
- Hinton, G.E.; Osindero, S.; Teh, Y.W. A fast learning algorithm for deep belief nets. Neural Comput.
**2006**, 18, 1527–1554. [Google Scholar] [CrossRef] [PubMed] - Salakhutdinov, R.; Larochelle, H. Deep Boltzmann machines. In Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics, Clearwater Beach, FL, USA, 16–18 April 2009; pp. 448–455. [Google Scholar]
- Stallkamp, J.; Schlipsing, M.; Salmen, J.; Igel, C. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural Netw.
**2012**, 32, 323–332. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Zaklouta, F.; Stanciulescu, B.; Hamdoun, O. Traffic sign classification using K-d trees and random forests. In Proceedings of the 2011 International Joint Conference on Neural Networks, San Jose, CA, USA, 31 July–5 August 2011; pp. 2151–2155. [Google Scholar]
- Sun, Z.L.; Wang, H.; Lau, W.S.; Seet, G.; Wang, D. Application of BW-ELM model on traffic sign recognition. Neural Comput.
**2014**, 29, 153–159. [Google Scholar] [CrossRef]

**Figure 4.**Recognition performance comparison between the proposed method and HELM on five multiple-class recognition benchmarks.

**Figure 5.**Part image examples in (

**a**) the Mixed National Institute of Standards and Technology (MNIST) dataset and (

**b**) the NORBdataset.

**Figure 6.**Recognition performance comparison with relevant rival algorithms on (

**a**) the MNIST dataset and (

**b**) the NORB dataset.

**Figure 7.**Training time comparison with relevant rival algorithms on (

**a**) the MNIST dataset and (

**b**) the NORB dataset.

Benchmark Database Name | Number of Hidden Nodes in Each Layer |
---|---|

Letter | L1 = L2 = 200, L3 = 2000 |

Iris | L1 = L2 = 20, L3 = 200 |

Glass | L1 = L2 = 20, L3 = 200 |

Wine | L1 = L2 = 20, L3 = 500 |

Satimage | L1 = L2 = 100, L3 = 1000 |

**Table 2.**Recognition accuracy (%) vs. different hidden nodes of the proposed method and HELM on the MNIST dataset.

Hidden Node Number | 300 | 400 | 500 | 600 | 700 | 800 | 900 |
---|---|---|---|---|---|---|---|

HELM | 0.9895 | 0.9893 | 0.9890 | 0.9899 | 0.9913 | 0.9892 | 0.9891 |

the proposed method | 0.9917 | 0.9920 | 0.9919 | 0.9119 | 0.9923 | 0.9919 | 0.9919 |

**Table 3.**Recognition accuracy (%) vs. different hidden nodes of the proposed method and HELM on the NORB dataset.

Hidden Node Number | 1000 | 1500 | 2000 | 2500 | 3000 | 3500 | 4000 |
---|---|---|---|---|---|---|---|

HELM | 0.9021 | 0.8984 | 0.9026 | 0.9001 | 0.9128 | 0.9004 | 0.9007 |

the proposed method | 0.9123 | 0.9120 | 0.9119 | 0.9136 | 0.9246 | 0.9118 | 0.9117 |

**Table 4.**Discriminative rate comparison of the features encoded by the proposed method and HELM on the MNIST dataset.

Method | HELM | The Proposed Method |
---|---|---|

Discriminative rate | 0.9274 | 0.9372 |

**Table 5.**Discriminative rate comparison of the features encoded by the proposed method and HELM on the NORB dataset.

Method | HELM | The Proposed Method |
---|---|---|

Discriminative rate | 0.4711 | 0.4724 |

Method | HOG-LDA | HOG-Random Forests | BW-ELM | HELM |

Accuracy (%) | 95.68 | 96.14 | 97.19 | 97.85 |

Method | Human Performance | HOGv-ELM | Proposed Method | |

Accuracy (%) | 98.84 | 99.09 | 98.91 |

© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Zeng, Y.; Qian, L.; Ren, J.
Evolutionary Hierarchical Sparse Extreme Learning Autoencoder Network for Object Recognition. *Symmetry* **2018**, *10*, 474.
https://doi.org/10.3390/sym10100474

**AMA Style**

Zeng Y, Qian L, Ren J.
Evolutionary Hierarchical Sparse Extreme Learning Autoencoder Network for Object Recognition. *Symmetry*. 2018; 10(10):474.
https://doi.org/10.3390/sym10100474

**Chicago/Turabian Style**

Zeng, Yujun, Lilin Qian, and Junkai Ren.
2018. "Evolutionary Hierarchical Sparse Extreme Learning Autoencoder Network for Object Recognition" *Symmetry* 10, no. 10: 474.
https://doi.org/10.3390/sym10100474