Next Article in Journal
Method of Forming Various Configurations of Telecommunication System Using Moving Target Defense
Next Article in Special Issue
Event-Based PID Control of a Flexible Manufacturing Process
Previous Article in Journal
Determination of the Expected Value of Losses Caused by the Cargo Transportation Insurance Risks by Water Transport
Previous Article in Special Issue
Wind Variation near the Black Sea Coastal Areas Reflected by the ERA5 Dataset
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Sensitivity Analysis of Artificial Neural Networks Identifying JWH Synthetic Cannabinoids Built with Alternative Training Strategies and Methods

by
Catalina Mercedes Burlacu
,
Adrian Constantin Burlacu
and
Mirela Praisler
*
Department of Chemistry, Physics and Environment, Faculty of Science and Environment, “Dunarea de Jos” University of Galati, 47 Domneasca Street, 800008 Galati, Romania
*
Author to whom correspondence should be addressed.
Inventions 2022, 7(3), 82; https://doi.org/10.3390/inventions7030082
Submission received: 13 July 2022 / Revised: 29 August 2022 / Accepted: 29 August 2022 / Published: 13 September 2022

Abstract

:
This paper presents the alternative training strategies we tested for an Artificial Neural Network (ANN) designed to detect JWH synthetic cannabinoids. In order to increase the model performance in terms of output sensitivity, we used the Neural Designer data science and machine learning platform combined with the programming language Python. We performed a comparative analysis of several optimization algorithms, error parameters and regularization methods. Finally, we performed a new goodness-of-fit analysis between the testing samples in the data set and the corresponding ANN outputs in order to investigate their sensitivity. The effectiveness of the new methods combined with the optimization algorithms is discussed.

1. Introduction

Artificial neural networks (ANN) contain a set of parameters that can be adjusted to perform different tasks. These structures have universal approximation properties, which means that they can approximate any function in any size and, generally, up to a desired degree of accuracy [1,2,3,4].
In this article, we present a series of deep learning training and optimization strategies that have been applied to improve the performance of an ANN identifying JWH-synthetic-cannabinoid-class membership. In order to increase the system sensitivity, we trained and optimized an initial model on four new architectures. For this purpose, we used the data science and machine learning platform Neural Designer. The best version was implemented in the Python 3.10 programming language for further development and improvement.
The classification efficiencies (output results) obtained for several combinations of algorithms, error parameters and regularization methods were compared. The good fit between the test samples and the corresponding ANN outputs was also analyzed. The effectiveness of the methods was analyzed and is presented in detail.

2. Materials and Methods

The initial input database of 150 synthetic chemicals included JWH synthetic cannabinoids, others synthetic cannabinoids and other substances of abuse. These designer drugs were divided into three classes referred to as “Class 1—JWH”, “Class 2—non-JWH Cannabinoids” and “Class 3—Others”. The group of positives contained 50 JWH synthetic cannabinoids, while the group of negatives included 100 compounds, i.e., 50 non-JWH cannabinoids and 50 other substances of forensic interest [5].
We used the quantitative structure–activity relationship (QSAR) method to estimate and predict the pharmacokinetics, drug-likeness and medicinal chemistry friendliness of each input compound by calculating a number of 300 molecular descriptors in terms of their physical and chemical properties, as well as 50 indices characterizing their chemical absorption, distribution, metabolism, excretion and toxicity activity (ADMET). The descriptors were selected from three blocks, i.e., topological, 3D-MoRSE (molecule representation of structure based on electron diffraction) and toxicity [6].
Only the first 150 most relevant descriptors were selected and used for the final computational and modelling stage. Hence, the input database was a matrix consisting of 150 samples × 150 variables. The shape, feature and target types of the data set, including the list of the computed and tested input molecular descriptors was presented in a previously published article [7].
The data set was divided into three subsets of samples: training, selection and testing. Hence, we used 90 training samples (60%), 30 selection samples (20%) and 30 testing samples (20%). To discover redundancies between the input variables, we used a correlation matrix, which represents a numerical value between −1 and 1 that expresses the strength of the relationship between two variables [8]. The types of layers the most frequently used in our classification model were the perceptron layer, the probabilistic layer and the scaling and bounding layers. The objective of the selection was to find the best-performing network architecture in terms of system sensitivity.
To avoid underfitting and overfitting, the neuron selection algorithm responsible for finding the optimal number of neurons in the networks was the Growing neurons algorithm [9]. The Neural Designer data science and machine learning platform was used to generate the mathematical expression represented by ANNs in order to export and incorporate them into the programming language Python 3.10 in the so-called production mode.
Our general training strategy consisted of two different concepts, i.e., the loss index and the optimization algorithms. The error was the essential term in the loss expression. The most important errors that we estimated were the sum squared error, the mean squared error, the root mean squared error, the normalized squared error and the Minkowski error. We used the L1 and L2 regularization methods, which involve the sum of the absolute values of all parameters and the square sum of all the parameters in the ANN. The loss index was measured on the data set and could be represented as a hyper-surface with the parameters as coordinates (see Figure 1) [10].
In order to train the ANN, we generated a sequence of parameter vectors so that the loss index was reduced at each iteration of the algorithm.

3. Results

Five different optimization algorithms were used and compared, each with a variety of different calculation and storage requirements: gradient descent [11], conjugate gradient, quasi-Newton method, Levenberg-Marquardt algorithm [12] and adaptative linear momentum [13].
In order to scale the inputs, we calculated the following parameters: the minimum, the maximum, the mean and the standard deviation (see Table 1). The ANN architecture is presented in Figure 2 for version 1. The architectures of the following versions (2, 3 and 4) were also with one hidden layer perceptrons and had the same input and output layers as version 1. On the other hand, their hidden layers contained three (version 2), one (version 3) and six (version 4) nodes respectively.
We used the adaptive moment estimation (version 1), the Levenberg–Marquardt (version 2), the gradient descent (version 3) and the conjugate gradient optimization algorithms, as well as the growing neuron selection (all versions) method with the L1 (versions 2, 3 and 4) and L2 (version 1) regularization methods.

4. Discussion

The confusion matrices, calculated for each architecture and 30 testing samples, are presented in Table 2, Table 3, Table 4 and Table 5 and the error results are highlighted in Table 6. The activation functions used were the hyperbolic tangent (version 1), the rectified linear (versions 2, 3 and 4) and Softmax (all versions).
In order to test and compare the performances of the analyzed ANNs, we used the weighted average derived from the confusion matrix, i.e., the accuracy, the recall and the F1 score (see Table 7) [14].

5. Conclusions

In terms of the system performance, the results obtained for the four ANNs designed to recognize the class identity of JWH synthetic cannabinoids lead to the following conclusions:
  • Accuracy [(true positives + true negatives)/total instances]:
In comparison with the accuracy (93.3%) obtained for the initial ANN model presented in a previous article, the ANN—amended version 1—generated a higher score (96.7%), while the other three ANNs generated a lower score (86.7% for the amended version 3 and 90.0% for the amended versions 2 and 4).
2.
Sensitivity, or true positive rate (true positives/positive instances):
All the ANN architectures (the initial ANN and its four amended versions), had an exceptional sensitivity in detecting JWH synthetic cannabinoids (class I); all the tested JWHs were recognized as such without exception, the rate of true positives (TP) being 100% in all cases.
3.
Specificity, or true negative rate (specificity = true negative/negative instances):
Compared to the specificity (90.0%) of the initial ANN model, one of the four new architectures recorded a higher score (95.0% for the amended version 1); the other three were characterized by a lower specificity (81.8% for amended version 3, 86.3% for amended version 2 and 86.3% for amended version 4).
4.
Error rate [accuracy = (false positives + false negatives)/total instances]:
While the initial ANN model was characterized by an error rate of 6.6%, its amended version 1 recorded a better (lower) error rate (3.3%), while the other three ANNs recorded a higher error rate (10.0% for amended version 2, 3 and 4).
Regarding the goodness-of-fit analysis, the best results were recorded for the ANN—amended version 1 (99.97%), followed by the amended version 2 (99.94%), amended version 3 (98.41%) and amended version 4 (99.04%). Hence, we may conclude that the best performing ANN architecture was the one that included the following elements:
  • Activation function: hyperbolic tangent, Softmax;
  • Loss index: normalized squared error;
  • Regularization: L2;
  • Neuron selection: growing neurons;
  • Inputs selection: growing inputs;
  • Optimization algorithm: Adaptive Moment Estimation.
The results indicate that very good classification rates were obtained although the data set was complex. We intend to continue this study by applying various metaheuristic algorithms on these data sets and compare the results.

Author Contributions

Conceptualization, C.M.B. and A.C.B.; methodology, M.P.; software, A.C.B.; validation, M.P.; formal analysis, C.M.B.; investigation, C.M.B.; resources, A.C.B.; writing—original draft preparation, A.C.B.; writing—review and editing, C.M.B.; supervision, M.P.; project administration, M.P. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Data Availability Statement

Not applicable.

Acknowledgments

An appreciation of the “Wiley Online Library” and the data science platform and machine learning “Neural Designer”, important and useful tools, used in the construction of system architectures presented in this paper.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Benitez, J.M.; Castro, J.L.; Requena, I. Are artificial neural networks black boxes? IEEE Trans. Neural. Netw. 1997, 8, 1156–1164. [Google Scholar] [CrossRef] [PubMed]
  2. Doulamis, A.D.; Doulamis, N.D.; Kollias, S.D. On-line retrainable neural networks: Improving the performance of neural networks in image analysis problems. IEEE Trans. Neural. Netw. 2000, 11, 137–155. [Google Scholar] [CrossRef] [PubMed]
  3. Yeung, D.S.; Cloete, I.; Shi, D.; Ng, W.W.Y. Sensitivity Analysis for Neural Networks; Springer: Berlin/Heidelberg Germany, 2010. [Google Scholar]
  4. Montano, J.J.; Palmer, A. Numeric sensitivity analysis applied to feedforward neural networks. Neural. Comput. Appl. 2003, 12, 119–125. [Google Scholar] [CrossRef]
  5. Burlacu, C.M.; Burlacu, A.C.; Praisler, M. Physico-chemical analysis, systematic benchmarking, and toxicological aspects of the JWH aminoalkylindole class-derived synthetic JWH cannabinoids. Ann. Univ. Dunarea Jos Galati Fascicle II Math. Phys. Theor. Mech. 2021, 44, 34–45. [Google Scholar] [CrossRef]
  6. Todeschini, R.; Consonni, V. Handbook of Molecular Descriptors; Wiley: Weinheim, Germany, 2000; pp. 1–667. [Google Scholar]
  7. Burlacu, C.M.; Gosav, S.; Burlacu, B.A.; Praisler, M. Convolutional Neural Network detecting synthetic cannabinoids. In Proceedings of the 2021 International Conference on e-Health and Bioengineering (EHB), Iasi, Romania, 18–19 November 2021; pp. 1–4. [Google Scholar]
  8. Egger, J.; Pepe, A.; Gsaxner, C.; Jin, Y.; Li, J.; Kern, R. Deep learning-a first meta-survey of selected reviews across scientific disciplines, their commonalities, challenges and research impact. PeerJ Comput. Sci. 2021, 7, e773. [Google Scholar] [CrossRef] [PubMed]
  9. Gupta, R.; Srivastava, D.; Sahu, M.; Tiwari, S.; Ambasta, R.K.; Kumar, P. Artificial intelligence to deep learning: Machine intelligence approach for drug discovery. Mol. Divers. 2021, 25, 1315–1360. [Google Scholar] [CrossRef] [PubMed]
  10. Jing, Y.; Bian, Y.; Hu, Z.; Wang, L.; Xie, X.Q.S. Deep learning for drug design: An artificial intelligence paradigm for drug discovery in the big data era. AAPS J. 2018, 20, 1–10. [Google Scholar]
  11. Smith, S.L.; Dherin, B.; Barrett, D.G.T.; De, S. On the origin of implicit regularization in stochastic gradient descent. arXiv 2021, arXiv:2101.12176. [Google Scholar]
  12. Wang, M.; Xu, X.; Yan, Z.; Wang, H. An online optimization method for extracting parameters of multi-parameter PV module model based on adaptive Levenberg-Marquardt algorithm. Energy Convers. Manag. 2021, 245, 114611. [Google Scholar] [CrossRef]
  13. Singarimbun, R.N.; Nababan, E.B.; Sitompul, O.S. Adaptive moment estimation to minimize square error in backpropagation algorithm. In Proceedings of the 2019 International Conference of Computer Science and Information Technology (ICoSNIKOM), Medan, Indonesia, 28–29 November 2019; pp. 1–7. [Google Scholar]
  14. Powers, D.M.W. Evaluation: From precision, recall and F-measure to ROC, informedness, markedness and correlation. arXiv 2011, arXiv:2010.16061. [Google Scholar]
Figure 1. Loss index viewed as a hyper-surface with parameters as coordinates.
Figure 1. Loss index viewed as a hyper-surface with parameters as coordinates.
Inventions 07 00082 g001
Figure 2. Network architecture, version 1: scaling layer (yellow), perceptron layer (blue), probabilistic layer (red).
Figure 2. Network architecture, version 1: scaling layer (yellow), perceptron layer (blue), probabilistic layer (red).
Inventions 07 00082 g002
Table 1. Values used for scaling the inputs for all ANN versions.
Table 1. Values used for scaling the inputs for all ANN versions.
InputMinimumMaximumMeanDeviationScaler
Topological4.3000007.9000005.8433300.828066Mean St. Dev.
3D-MoRSE2.0000004.4000003.0540000.433594Mean St. Dev.
ADMET1.0000006.9000003.7586701.764420Mean St. Dev.
Table 2. Confusion matrix for the analyzed ANN—version 1: 29 (96.7%) tested compounds were correctly classified and 1 (3.3%) was misclassified.
Table 2. Confusion matrix for the analyzed ANN—version 1: 29 (96.7%) tested compounds were correctly classified and 1 (3.3%) was misclassified.
Predicted
JWH
Predicted
Non-JWH Cannabinoids
Predicted Others
Real JWH Cannabinoids1000
Real non-JWH Cannabinoids070
Real Others0112
Table 3. Confusion matrix for the analyzed ANN—version 2: 27 (90.0%) tested compounds were correctly classified and 3 (10.0%) were misclassified.
Table 3. Confusion matrix for the analyzed ANN—version 2: 27 (90.0%) tested compounds were correctly classified and 3 (10.0%) were misclassified.
Predicted
JWH
Predicted
Non-JWH Cannabinoids
Predicted Others
Real JWH Cannabinoids800
Real non-JWH Cannabinoids091
Real Others0210
Table 4. Confusion matrix for the analyzed ANN—version 3: 26 (86.7%) tested compounds were correctly classified and 4 (13.3%) were misclassified.
Table 4. Confusion matrix for the analyzed ANN—version 3: 26 (86.7%) tested compounds were correctly classified and 4 (13.3%) were misclassified.
Predicted
JWH
Predicted
Non-JWH Cannabinoids
Predicted Others
Real JWH Cannabinoids800
Real Non-JWH Cannabinoids091
Real Others039
Table 5. Confusion matrix for the analyzed ANN—version 4: 27 (90.0%) tested compounds were correctly classified and 3 (10.0%) were misclassified.
Table 5. Confusion matrix for the analyzed ANN—version 4: 27 (90.0%) tested compounds were correctly classified and 3 (10.0%) were misclassified.
Predicted
JWH
Predicted
Non-JWH Cannabinoids
Predicted Others
Real JWH Cannabinoids800
Real non-JWH Cannabinoids0100
Real Others039
Table 6. Data errors for the analyzed ANNs.
Table 6. Data errors for the analyzed ANNs.
Root Mean Squared Error
Number of Nodes
in the Hidden Layer
ANNTraining
Set
Selection
Set
Testing
Set
1Version 30.1875580.2761550.284885
3Version 20.1806180.2767620.284408
4Version 10.1784300.2410470.213058
6Version 40.1733800.2682220.280003
Table 7. Classification metrics for the target variables of each analyzed ANNs.
Table 7. Classification metrics for the target variables of each analyzed ANNs.
Weighted Average
Number of Nodes
in the Hidden Layer
ANNAccuracyRecallF1 Score
1Version 30.8766670.8666670.866667
3Version 20.9030300.9000000.900207
4Version 10.9708330.9666670.967111
6Version 40.9230770.9000000.899379
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Burlacu, C.M.; Burlacu, A.C.; Praisler, M. Sensitivity Analysis of Artificial Neural Networks Identifying JWH Synthetic Cannabinoids Built with Alternative Training Strategies and Methods. Inventions 2022, 7, 82. https://doi.org/10.3390/inventions7030082

AMA Style

Burlacu CM, Burlacu AC, Praisler M. Sensitivity Analysis of Artificial Neural Networks Identifying JWH Synthetic Cannabinoids Built with Alternative Training Strategies and Methods. Inventions. 2022; 7(3):82. https://doi.org/10.3390/inventions7030082

Chicago/Turabian Style

Burlacu, Catalina Mercedes, Adrian Constantin Burlacu, and Mirela Praisler. 2022. "Sensitivity Analysis of Artificial Neural Networks Identifying JWH Synthetic Cannabinoids Built with Alternative Training Strategies and Methods" Inventions 7, no. 3: 82. https://doi.org/10.3390/inventions7030082

Article Metrics

Back to TopTop