Classification of Similar Sports Images Using Convolutional Neural Network with HyperParameter Optimization
Abstract
:Featured Application
Abstract
1. Introduction
 We prepared a new image dataset of four similar sports—American football, rugby, soccer, and field hockey.
 We developed a method for the classification of sports disciplines from images using the transfer learning of CNN with HPO.
 We compared our proposed method with a conventional CNN as well as with the CNN using the transfer learning methodology with handpicked hyperparameters for fine tuning.
 We provide detailed results of the performed experiments alongside with the statistical analysis and the analysis of interpretable representations of the trained image classification models.
2. Related Work
3. Background and Methods
3.1. Transfer Learning
3.2. HyperParameter Optimization in Machine Learning
3.3. Differential Evolution
3.4. Interpretable Representation
4. Image Classification with CNN Using Optimized Transfer Learning
Algorithm 1 CNNTLDE algorithm. 
Input: training dataset D, pretrained CNN model, number of hyperparameters to be optimized n 
Output: trained CNN model 

 the dropout probability value of the spatial twodimensional dropout layer $SDO\_2D$;
 the number of neurons in the last fully connected dense layer $FC\_2+Sigmoid$;
 the dropout probability value of the second dropout layer;
 the optimizer type; and
 the learning rate.
HyperParameter Optimization of Transfer Learning with Differential Evolution
5. Experiments
 CNN, where the conventional approach for training the adapted VGG19 CNN model was used;
 CNNTL, where the transfer learning methodology was used with handpicked hyperparameters for fine tuning; and
 CNNTLDE, where our proposed method which utilizes the DE for the optimization of hyperparameters for fine tuning of the transfer learning was exploited.
5.1. Dataset
5.2. CNNTLDE Settings
5.3. Experimental Settings
6. Results
6.1. Classification Performance
6.1.1. Statistical Comparison
6.1.2. Using Alternative Optimization Techniques
 CNNTLGA, where GA was used instead of DE for the optimization of hyperparameters for fine tuning of the transfer learning; and
 CNNTLPSO, where PSO was used instead of DE for the same purpose.
6.1.3. Using Different CNN Architectures
6.2. The Analysis of Confusion Matrices
6.3. The Analysis of the Obtained HyperParameters
6.4. Explaining the Classification Model
7. Conclusions
Author Contributions
Funding
Conflicts of Interest
References
 Hatem, Y.; Rady, S. Exploring feature dimensionality reduction methods for enhancing automatic sport image annotation. Multimed. Tools Appl. 2018, 77, 9171–9188. [Google Scholar] [CrossRef]
 Pilar, P.M.; Rafael, M.C.; Félix, Z.O.; Gabriel, G.V. Impact of sports mass media on the behavior and health of society. A systematic review. Int. J. Environ. Res. Public Health 2019, 16, 486. [Google Scholar] [CrossRef] [PubMed] [Green Version]
 Minhas, R.A.; Javed, A.; Irtaza, A.; Mahmood, M.T.; Joo, Y.B. Shot classification of field sports videos using AlexNet Convolutional Neural Network. Appl. Sci. 2019, 9, 483. [Google Scholar] [CrossRef] [Green Version]
 Fister, I.; Fister, I., Jr.; Fister, D. Computational Intelligence in Sports; Springer: Berlin/Heidelberg, Germany, 2019. [Google Scholar]
 Wang, S.Y.; Liao, W.S.; Hsieh, L.C.; Chen, Y.Y.; Hsu, W.H. Learning by expansion: Exploiting social media for image classification with few training examples. Neurocomputing 2012, 95, 117–125. [Google Scholar] [CrossRef] [Green Version]
 Lu, C.; Zhai, F. Weaklysupervised largescale image modeling for sport scenes and its applications. J. Vis. Commun. Image Represent. 2020, 71, 102718. [Google Scholar] [CrossRef]
 Ciregan, D.; Meier, U.; Schmidhuber, J. Multicolumn deep neural networks for image classification. In Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, 16–21 June 2012; pp. 3642–3649. [Google Scholar]
 Karpathy, A.; Toderici, G.; Shetty, S.; Leung, T.; Sukthankar, R.; FeiFei, L. Largescale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 24–27 June 2014; pp. 1725–1732. [Google Scholar]
 Moreira, D.; Vasconcelos, C.; Paes, A.; Velho, L. Deep Image Classification of a Wild Data Set for Olympic Sports. 2016. Available online: https://www.visgraf.impa.br/Data/RefBib/PS_PDF/workpedia2016/Workpedia_2016_paper_18.pdf (accessed on 20 August 2020).
 Hussain, M.; Bird, J.J.; Faria, D.R. A study on cnn transfer learning for image classification. In UK Workshop on Computational Intelligence; Springer: Berlin/Heidelberg, Germany, 2018; pp. 191–202. [Google Scholar]
 Yang, L.; Shami, A. On hyperparameter optimization of machine learning algorithms: Theory and practice. Neurocomputing 2020, 415, 295–316. [Google Scholar] [CrossRef]
 Vrbančič, G.; Fister, I.J.; Podgorelec, V. Parameter Setting for Deep Neural Networks Using Swarm Intelligence on Phishing Websites Classification. Int. J. Artif. Intell. Tools 2019, 28, 1960008. [Google Scholar] [CrossRef]
 Vrbančič, G.; Zorman, M.; Podgorelec, V. Transfer Learning Tuning Utilizing Grey Wolf Optimizer for Identification of Brain Hemorrhage from Head CT Images. In Proceedings of the 2019 6th Student Computer Science Research Conference, Koper, Slovenia, 10 October 2019; pp. 61–66. [Google Scholar]
 Vrbančič, G.; Pečnik, Š.; Podgorelec, V. Identification of COVID19 Xray Images using CNN with Optimized Tuning of Transfer Learning. In Proceedings of the 2020 International Conference on INnovations in Intelligent SysTems and Applications (INISTA), Novi Sad, Serbia, 24–26 August 2020; pp. 1–8. [Google Scholar]
 Jung, Y.; Hwang, E.; Kim, W. Sports image classification through Bayesian classifier. In Lecture Notes in Computer Science, Proceedings of the Conference on Technology Transfer, San Sebastian, Spain, 12–14 November 2003; Springer: Berlin/Heidelberg, Germany, 2003; pp. 546–555. [Google Scholar]
 Capodiferro, L.; Costantini, L.; Mangiatordi, F.; Pallotti, E. SVM for historical sport video classification. In Proceedings of the 2012 5th International Symposium on Communications, Control and Signal Processing, Rome, Italy, 2–4 May 2012; pp. 1–4. [Google Scholar]
 Hatem, Y.; Rady, S.; Ismail, R.; Bahnasy, K. Automatic content description and annotation of sport images using classification techniques. In Proceedings of the 10th International Conference on Informatics and Systems, Cairo, Egypt, 9–11 May 2016; pp. 88–94. [Google Scholar]
 Breen, C.; Khan, L.; Ponnusamy, A. Image classification using neural networks and ontologies. In Proceedings of the 13th International Workshop on Database and Expert Systems Applications, AixenProvence, France, 6 September 2002; pp. 98–102. [Google Scholar]
 Kanellopoulos, I.; Wilkinson, G.G. Strategies and best practice for neural network image classification. Int. J. Remote Sens. 1997, 18, 711–725. [Google Scholar] [CrossRef]
 Foody, G. Supervised image classification by MLP and RBF neural networks with and without an exhaustively defined set of classes. Int. J. Remote Sens. 2004, 25, 3091–3104. [Google Scholar] [CrossRef]
 Giacinto, G.; Roli, F. Design of effective neural network ensembles for image classification purposes. Image Vis. Comput. 2001, 19, 699–707. [Google Scholar] [CrossRef]
 Fukushima, K.; Miyake, S. Neocognitron: A selforganizing neural network model for a mechanism of visual pattern recognition. In Competition and Cooperation in nEural Nets; Amari, S., Arbib, M.A., Eds.; Springer: Berlin/Heidelberg, Germany, 1982; pp. 267–285. [Google Scholar]
 Rawat, W.; Wang, Z. Deep convolutional neural networks for image classification: A comprehensive review. Neural Comput. 2017, 29, 2352–2449. [Google Scholar] [CrossRef] [PubMed]
 Vrbančič, G.; Podgorelec, V. Automatic Classification of Motor Impairment Neural Disorders from EEG Signals Using Deep Convolutional Neural Networks. Elektron. Elektrotech. 2018, 24, 3–7. [Google Scholar] [CrossRef]
 Vrbančič, G.; Fister, I.J.; Podgorelec, V. Automatic Detection of Heartbeats in Heart Sound Signals Using Deep Convolutional Neural Networks. Elektron. Elektrotech. 2019, 25, 71–76. [Google Scholar] [CrossRef] [Green Version]
 McQuin, C.; Goodman, A.; Chernyshev, V.; Kamentsky, L.; Cimini, B.A.; Karhohs, K.W.; Doan, M.; Ding, L.; Rafelski, S.M.; Thirstrup, D.; et al. CellProfiler 3.0: Nextgeneration image processing for biology. PLoS Biol. 2018, 16, e2005970. [Google Scholar] [CrossRef] [PubMed] [Green Version]
 Javaid, A.; Niyaz, Q.; Sun, W.; Alam, M. A deep learning approach for network intrusion detection system. In Proceedings of the 9th EAI International Conference on BioInspired Information and Communications Technologies (Formerly BIONETICS), New York, NY, USA, 3–5 December 2015; pp. 21–26. [Google Scholar]
 Kamilaris, A.; PrenafetaBoldú, F.X. Deep learning in agriculture: A survey. Comput. Electron. Agric. 2018, 147, 70–90. [Google Scholar] [CrossRef] [Green Version]
 Ching, J.Y.; Wong, A.K.C.; Chan, K.C.C. Classdependent discretization for inductive learning from continuous and mixedmode data. IEEE Trans. Pattern Anal. Mach. Intell. 1995, 17, 641–651. [Google Scholar] [CrossRef]
 Dumais, S.; Platt, J.; Heckerman, D.; Sahami, M. Inductive Learning Algorithms and Representations for Text Categorization. In Proceedings of the 7th International Conference on Information and Knowledge Management, Bethesda, MD, USA, 3–7 November 1998; pp. 148–152. [Google Scholar]
 Zhu, X.; Wu, X. Class noise handling for effective costsensitive learning by costguided iterative classification filtering. IEEE Trans. Knowl. Data Eng. 2006, 18, 1435–1440. [Google Scholar]
 Yang, Q.; Ling, C.; Chai, X.; Pan, R. Testcost sensitive classification on data with missing values. IEEE Trans. Knowl. Data Eng. 2006, 18, 626–638. [Google Scholar] [CrossRef] [Green Version]
 Diaz, G.I.; FokoueNkoutche, A.; Nannicini, G.; Samulowitz, H. An effective algorithm for hyperparameter optimization of neural networks. IBM J. Res. Dev. 2017, 61, 9:1–9:11. [Google Scholar] [CrossRef] [Green Version]
 Gambella, C.; Ghaddar, B.; NaoumSawaya, J. Optimization models for machine learning: A survey. arXiv 2019, arXiv:1901.05331. [Google Scholar]
 Bergstra, J.S.; Bardenet, R.; Bengio, Y.; Kégl, B. Algorithms for hyperparameter optimization. In Proceedings of the Advances in Neural Information Processing Systems, Granada, Spain, 12–17 December 2011; pp. 2546–2554. [Google Scholar]
 Bergstra, J.; Bengio, Y. Random search for hyperparameter optimization. J. Mach. Learn. Res. 2012, 13, 281–305. [Google Scholar]
 Bacanin, N.; Bezdan, T.; Tuba, E.; Strumberger, I.; Tuba, M. Optimizing Convolutional Neural Network Hyperparameters by Enhanced Swarm Intelligence Metaheuristics. Algorithms 2020, 13, 67. [Google Scholar] [CrossRef] [Green Version]
 Bibaeva, V. Using metaheuristics for hyperparameter optimization of convolutional neural networks. In Proceedings of the 2018 IEEE 28th International Workshop on Machine Learning for Signal Processing (MLSP), Aalborg, Denmark, 17–20 September 2018; pp. 1–6. [Google Scholar]
 Storn, R.; Price, K. Differential evolution–a simple and efficient heuristic for global optimization over continuous spaces. J. Glob. Optim. 1997, 11, 341–359. [Google Scholar] [CrossRef]
 Nakisa, B.; Rastgoo, M.N.; Rakotonirainy, A.; Maire, F.; Chandran, V. Long short term memory hyperparameter optimization for a neural network based emotion recognition framework. IEEE Access 2018, 6, 49325–49338. [Google Scholar] [CrossRef]
 Xue, N.; Triguero, I.; Figueredo, G.P.; LandaSilva, D. Evolving Deep CNNLSTMs for Inventory Time Series Prediction. In Proceedings of the 2019 IEEE Congress on Evolutionary Computation (CEC), Wellington, New Zealand, 10–13 June 2019; pp. 1517–1524. [Google Scholar]
 Brezočnik, L.; Fister, I.; Vrbančič, G. Applying Differential Evolution with Threshold Mechanism for Feature Selection on a Phishing Websites Classification. In New Trends in Databases and Information Systems; Welzer, T., Eder, J., Podgorelec, V., Wrembel, R., Ivanović, M., Gamper, J., Morzy, M., Tzouramanis, T., Darmont, J., Kamišalić Latifić, A., Eds.; Springer International Publishing: Cham, Switzerland, 2019; pp. 11–18. [Google Scholar]
 Piotrowski, A.P. Review of differential evolution population size. Swarm Evol. Comput. 2017, 32, 1–24. [Google Scholar] [CrossRef]
 Mininno, E.; Neri, F. A memetic differential evolution approach in noisy optimization. Memetic Comput. 2010, 2, 111–135. [Google Scholar] [CrossRef]
 Das, S.; Suganthan, P.N. Differential Evolution: A Survey of the StateoftheArt. IEEE Trans. Evol. Comput. 2011, 15, 4–31. [Google Scholar] [CrossRef]
 Ribeiro, M.T.; Singh, S.; Guestrin, C. “Why should i trust you?” Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, 13–17 August 2016; pp. 1135–1144. [Google Scholar]
 Zhang, Y.; Song, K.; Sun, Y.; Tan, S.; Udell, M. “Why Should You Trust My Explanation?” Understanding Uncertainty in LIME Explanations. arXiv 2014, arXiv:1904.12991. [Google Scholar]
 Schallner, L.; Rabold, J.; Scholz, O.; Schmid, U. Effect of Superpixel Aggregation on Explanations in LIME–A Case Study with Biological Data. In Communications in Computer and Information Science, Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Würzburg, Germany, 16–20 September 2019; Springer: Cham, Switzerland, 2019; pp. 147–158. [Google Scholar]
 Lundberg, S.M.; Lee, S.I. A Unified Approach to Interpreting Model Predictions. In Advances in Neural Information Processing Systems 30; Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R., Eds.; Curran Associates, Inc.: Red Hook, NY, USA, 2017; pp. 4765–4774. [Google Scholar]
 Lundberg, S. SHAP Documentation. 2018. Available online: https://shap.readthedocs.io/en/latest/index.html (accessed on 28 September 2020).
 Molnar, C. Interpretable Machine Learning: A Guide for Making Black Box Models Explainable, 1st ed.; Lulu: Morrisville, NC, USA, 2019. [Google Scholar]
 Simonyan, K.; Zisserman, A. Very deep convolutional networks for largescale image recognition. arXiv 2014, arXiv:1409.1556. [Google Scholar]
 Deng, J.; Dong, W.; Socher, R.; Li, L.J.; Li, K.; FeiFei, L. Imagenet: A largescale hierarchical image database. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 20–25 June 2009; pp. 248–255. [Google Scholar]
 GómezValverde, J.J.; Antón, A.; Fatti, G.; Liefers, B.; Herranz, A.; Santos, A.; Sánchez, C.I.; LedesmaCarbayo, M.J. Automatic glaucoma classification using color fundus images based on convolutional neural networks and transfer learning. Biomed. Opt. Express 2019, 10, 892–913. [Google Scholar] [CrossRef] [Green Version]
 Mayer, D.G.; Kinghorn, B.; Archer, A.A. Differential evolution–an easy and efficient evolutionary algorithm for model optimisation. Agric. Syst. 2005, 83, 315–328. [Google Scholar] [CrossRef]
 Fawcett, T. Using rule sets to maximize ROC performance. In Proceedings of the 2001 IEEE International Conference on Data Mining, San Jose, CA, USA, 29 November–2 December 2001; pp. 131–138. [Google Scholar]
 Fleiss, J.L.; Levin, B.; Paik, M.C. Statistical Methods for Rates and Proportions, 3rd ed.; WileyInterscience: Hoboken, NJ, USA, 2003. [Google Scholar]
 Demsar, J. Statistical Comparisons of Classifiers over Multiple Data Sets. J. Mach. Learn. Res. 2006, 1–30. [Google Scholar] [CrossRef]
 Szegedy, C.; Vanhoucke, V.; Ioffe, S.; Shlens, J.; Wojna, Z. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 26 June–1 July 2017; pp. 2818–2826. [Google Scholar]
 Huang, G.; Liu, Z.; Van Der Maaten, L.; Weinberger, K.Q. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Hoholulu, HI, USA, 21–26 July 2017; pp. 4700–4708. [Google Scholar]
 Szegedy, C.; Liu, W.; Jia, Y.; Sermanet, P.; Reed, S.; Anguelov, D.; Erhan, D.; Vanhoucke, V.; Rabinovich, A. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 1–9. [Google Scholar]
Sports Discipline  Number of Images 

American football  342 (25.17%) 
field hockey  363 (26.71%) 
rugby  321 (23.62%) 
soccer  333 (24.50%) 
Parameter  CNN  CNNTL  CNNTLDE 

max number of epochs  100  100  5100 
batch size  32  32  32 
learning rate  1 $\times {10}^{5}$  $1\times {10}^{5}$   
optimizer type  RMSprop  RMSprop   
SpatialDropout2D probability  0.5  0.5   
number of neurons in last FC  1024  1024   
dropout probability  0.5  0.5   
Parameter  Value 

Dimension of the problem  5 
Population size $\mathit{Np}$  10 
Scale factor F  0.5 
Crossover rate $\mathit{CR}$  0.9 
Number of function evaluations  50 
Metric  CNN  CNNTL  CNNTLDE 

accuracy  48.63 ± 6.07  75.72 ± 6.07  81.31 ± 3.51 
AUNP  65.73 ± 4.05  83.81 ± 4.11  87.56 ± 2.30 
precision  51.00 ± 5.67  77.81 ± 3.49  83.13 ± 3.33 
recall  48.79 ± 6.13  75.51 ± 6.37  81.30 ± 3.41 
F1  49.75 ± 5.49  76.59 ± 4.90  82.20 ± 3.31 
kappa  31.51 ± 8.11  67.57 ± 8.19  75.08 ± 4.64 
time  319.47 ± 80.16  102.62 ± 35.04  3430.43 ± 315.78 
epochs  18.5 ± 5.43  24.5 ± 8.86  790.3 ± 85.49 
Friedman Test  Wilcoxon Signed Rank Test  

Metric  All Three  CNNTLDE vs. CNN  CNNTLDE vs. CNNTL 
accuracy  <0.001  0.005  0.033 
AUNP  <0.001  0.005  0.022 
precision  <0.001  0.005  0.022 
recall  <0.001  0.005  0.022 
F1  <0.001  0.005  0.022 
kappa  <0.001  0.005  0.022 
time  <0.001  0.005  0.005 
epochs  <0.001  0.005  0.005 
Metric  CNNTLGA  CNNTLPSO  CNNTLDE 

accuracy  79.03 ± 6.18  78.07 ± 5.89  81.31 ± 3.51 
AUNP  85.98 ± 4.17  85.41 ± 3.89  87.56 ± 2.30 
precision  82.06 ± 3.49  81.30 ± 5.89  83.13 ± 3.33 
recall  78.74 ± 6.49  78.11 ± 5.78  81.30 ± 3.41 
F1  80.32 ± 4.98  79.62 ± 4.19  82.20 ± 3.31 
kappa  71.97 ± 8.31  70.77 ± 7.81  75.08 ± 4.64 
time  3311.46 ± 271.46  3667.14 ± 272.13  3430.43 ± 315.78 
epochs  744.7 ± 66.09  859.5 ± 79.45  790.3 ± 85.49 
Metric  InceptionV3  DenseNet121  VGG19 

accuracy  76.82 ± 5.77  82.49 ± 5.26  81.31 ± 3.51 
AUNP  84.69 ± 3.77  88.33 ± 3.53  87.56 ± 2.30 
precision  80.38 ± 3.59  82.49 ± 5.26  83.13 ± 3.33 
recall  77.30 ± 5.60  82.29 ± 5.38  81.30 ± 3.41 
F1  78.81 ± 4.59  82.39 ± 5.32  82.20 ± 3.31 
kappa  69.20 ± 7.61  76.62 ± 7.04  75.08 ± 4.64 
time  4579.74 ± 341.49  5408.79 ± 77.73  3430.43 ± 315.78 
epochs  720.6 ± 104.21  678.6 ± 51.90  790.3 ± 85.49 
Fold  Learning Rate  Optimizer Type  Spatial Dropout2D Prob.  Num. of Neurons  Dropout Prob. 

0  0.0001  adam  0.1  2048  0.6 
1  0.00005  adam  0.1  512  0.4 
2  0.00001  rmsprop  0.1  64  0.1 
3  0.00005  adam  0.3  2048  0.9 
4  0.0001  adam  0.1  1024  0.3 
5  0.005  sgd  0.3  2048  0.5 
6  0.00001  adam  0.4  2048  0.7 
7  0.00005  adam  0.4  2048  0.2 
8  0.0001  adam  0.3  64  0.4 
9  0.00005  adam  0.2  512  0.7 
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. 
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Podgorelec, V.; Pečnik, Š.; Vrbančič, G. Classification of Similar Sports Images Using Convolutional Neural Network with HyperParameter Optimization. Appl. Sci. 2020, 10, 8494. https://doi.org/10.3390/app10238494
Podgorelec V, Pečnik Š, Vrbančič G. Classification of Similar Sports Images Using Convolutional Neural Network with HyperParameter Optimization. Applied Sciences. 2020; 10(23):8494. https://doi.org/10.3390/app10238494
Chicago/Turabian StylePodgorelec, Vili, Špela Pečnik, and Grega Vrbančič. 2020. "Classification of Similar Sports Images Using Convolutional Neural Network with HyperParameter Optimization" Applied Sciences 10, no. 23: 8494. https://doi.org/10.3390/app10238494