MisRoBÆRTa: Transformers versus Misinformation
Abstract
:1. Introduction
- (1)
- Increasing the size of the dataset from small to large; and
- (2)
- Moving the focus of fake news detection from binary to multi-class classification.
- (Q1)
- Which transformer obtains the overall best accuracy for the task of multi-class classification of misinformation?
- (Q2)
- Which transformer obtains the overall best runtime performance without the model having a significant decrease in accuracy?
- (Q3)
- Can there be a balance between accuracy and runtime required for fine-tuning and training?
- (1)
- We propose MisRoBÆRTa, a new transformer-based deep learning architecture for misinformation detection;
- (2)
- We conducted an in-depth analysis of the current state-of-the-art deep learning, word embeddings, and transformer-based models for the task of fake news and misinformation detection;
- (3)
- We propose a new balanced dataset for misinformation containing 100,000 articles extracted from the FakeNewsCorpus, where, for each news article, we (1) manually verified its content to make sure that it was correctly labeled; (2)verified the URLs to point to the correct article by matching the titles and authors;
- (4)
- We performed an in-depth analysis of the proposed dataset;
- (5)
- We extended the SimpleTransformers package with a multi-class implementation for BART;
- (6)
- We performed a detailed benchmark analysis using multiple transformers and transfer learning on the task of misinformation and compared the results with the state-of-the-art model FakeBERT [8].
2. Related Work
3. Methodology
3.1. Transformers
3.1.1. BERT
3.1.2. DistilBERT
3.1.3. RoBERTa
- (1)
- It removes the next-sentence pre-training objective; and
- (2)
- It increases both the mini-batches and learning rates.
3.1.4. DistilRoBERTa
3.1.5. ALBERT
3.1.6. DeBERTa
3.1.7. XLNet
3.1.8. ELECTRA
3.1.9. XLM
3.1.10. XLM-RoBERTa
3.1.11. BART
3.2. Classification Network
4. MisRoBÆRTa: Misinformation RoBERTa-BART Ensemble Detection Model
- (1)
- The sentence embeddings constructed with RoBERTa base of size 768; and
- (2)
- The sentence embeddings extracted with BART, a large of size 1024.
4.1. MisRoBÆRTa Components
4.1.1. Long Short-Term Memory Networks
- (1)
- is the input at time step t;
- (2)
- is the output, or next hidden state;
- (3)
- is the previous hidden state;
- (4)
- is the cell input activation vector;
- (5)
- is the current memory state;
- (6)
- is the previous memory state;
- (7)
- , , , and are the weights for each gate’s current input;
- (8)
- , , , and are the weights for each gate’s previous hidden state;
- (9)
- , , , and are the bias vectors;
- (10)
- is the sigmoid activation function;
- (11)
- is the hyperbolic tangent activation function;
- (12)
- ⊙ operator is the Hadamard product, i.e., the element-wise multiplication function.
4.1.2. Convolutional Neural Networks
4.1.3. Max Pooling
4.1.4. Dense Layer
4.2. MisRoBÆRTa Description
4.3. MisRoBÆRTa Architectural Choices
5. Experimental Results
5.1. Dataset
5.2. Fine-Tuning
5.3. Classification Results
- (1)
- FakeBERT is developed for binary classification, while we used it for multi-class classification;
- (2)
- To change FakeBERT from a binary classifier to a multi-class classifier, we modified the number of units of its final DENSE layer from 2 to the number of classes, i.e., 10;
- (3)
- We only trained FakeBERT for 10 epochs, to be consistent in our performance experiments (note: this is also the number of epochs used in the original paper).
- (1)
- BART large and RoBERTa base;
- (2)
- BiLSTM layers that combined two hidden states to preserve information from both past and future;
- (3)
- CNNs to generate multi-word expressions, which better represent the feature space.
5.4. Runtime Evaluation
5.5. Ablation Testing
6. Discussion
- (1)
- Multi-modal (i.e., visual, audio, network propagation, and immunization, etc.) techniques are rarely used; this is due to a lack of complete, high quality, and well labeled corpora and the tedious work that is required to correctly annotate such datasets;
- (2)
- Source verification is rarely taken into consideration for misinformation detection [56], this is due to the instability of links on the internet;
- (3)
- Author credibility should be an important factor that is not really discussed or weighted in the feature selection for the models;
- (4)
- The perpetual need to retrain or fine-tune models to capture the subtle shifts that appear over time in news articles that spread misinformation.
7. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
Abbreviations
LSTM | long short-term memory |
BiLSTM | bidirectional long short-term memory |
CNN | convolutional neural network |
BERT | bidirectional encoder representations from transformers |
DistilBERT | distilled BERT |
RoBERTa | robustly optimized BERT pre-training approach |
DistilRoBERTa | distilled RoBERTa |
ALBERT | a lite BERT |
DeBERTa | decoding-enhanced BERT with disentangled attention |
ELECTRA | efficiently learning an encoder that classifies token replacements Accurately |
XLM | cross-lingual language model |
BART | bidirectional and autoregressive transformer |
References
- Ruths, D. The misinformation machine. Science 2019, 363, 348. [Google Scholar] [CrossRef] [PubMed]
- Shu, K.; Wang, S.; Le, T.; Lee, D.; Liu, H. Deep Headline Generation for Clickbait Detection. In Proceedings of the IEEE International Conference on Data Mining, Singapore, 17–20 November 2018. [Google Scholar] [CrossRef]
- Ilie, V.I.; Truica, C.O.; Apostol, E.S.; Paschke, A. Context-Aware Misinformation Detection: A Benchmark of Deep Learning Architectures Using Word Embeddings. IEEE Access 2021, 9, 162122–162146. [Google Scholar] [CrossRef]
- Zannettou, S.; Sirivianos, M.; Blackburn, J.; Kourtellis, N. The web of false information: Rumors, fake news, hoaxes, clickbait, and various other shenanigans. J. Data Inf. Qual. 2019, 11, 1–37. [Google Scholar] [CrossRef] [Green Version]
- Bovet, A.; Makse, H.A. Influence of fake news in Twitter during the 2016 US presidential election. Nat. Commun. 2019, 10, 7. [Google Scholar] [CrossRef]
- Marco-Franco, J.E.; Pita-Barros, P.; Vivas-Orts, D.; González-de-Julián, S.; Vivas-Consuelo, D. COVID-19, Fake News, and Vaccines: Should Regulation Be Implemented? Int. J. Environ. Res. Public Health 2021, 18, 744. [Google Scholar] [CrossRef]
- Commision, E. Fighting Disinformation. 2021. Available online: https://ec.europa.eu/info/live-work-travel-eu/health/coronavirus-response/fighting-disinformation_en (accessed on 6 September 2021).
- Kaliyar, R.K.; Goswami, A.; Narang, P. FakeBERT: Fake news detection in social media with a BERT-based deep learning approach. Multimed. Tools Appl. 2021, 80, 11765–11788. [Google Scholar] [CrossRef]
- Conroy, N.K.; Rubin, V.L.; Chen, Y. Automatic deception detection: Methods for finding fake news. Proc. Assoc. Inf. Sci. Technol. 2015, 52, 1–4. [Google Scholar] [CrossRef] [Green Version]
- Gravanis, G.; Vakali, A.; Diamantaras, K.; Karadais, P. Behind the cues: A benchmarking study for fake news detection. Expert Syst. Appl. 2019, 128, 201–213. [Google Scholar] [CrossRef]
- Kaliyar, R.K.; Goswami, A.; Narang, P.; Sinha, S. FNDNet—A deep convolutional neural network for fake news detection. Cogn. Syst. Res. 2020, 61, 32–44. [Google Scholar] [CrossRef]
- Pennington, J.; Socher, R.; Manning, C. GloVe: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar, 25–29 October 2014; pp. 1532–1543. [Google Scholar] [CrossRef]
- Li, Q.; Hu, Q.; Lu, Y.; Yang, Y.; Cheng, J. Multi-level word features based on CNN for fake news detection in cultural communication. Pers. Ubiquitous Comput. 2020, 24, 259–272. [Google Scholar] [CrossRef]
- Mikolov, T.; Chen, K.; Corrado, G.; Dean, J. Efficient estimation of word representations in vector space. In Proceedings of the International Conference on Learning Representations, Scottsdale, AZ, USA, 2–4 May 2013; pp. 1–12. [Google Scholar]
- Mishra, R.; Setty, V. Sadhan: Hierarchical attention networks to learn latent aspect embeddings for fake news detection. In Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval, Santa Clara, CA, USA, 2–5 October 2019; pp. 197–204. [Google Scholar] [CrossRef]
- Trueman, T.E.; Kumar, A.; Narayanasamy, P.; Vidya, J. Attention-based C-BiLSTM for fake news detection. Appl. Soft Comput. 2021, 110, 107600. [Google Scholar] [CrossRef]
- Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; Stoyanov, V. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv 2019, arXiv:1907.11692. [Google Scholar]
- Liu, C.; Wu, X.; Yu, M.; Li, G.; Jiang, J.; Huang, W.; Lu, X. A Two-Stage Model Based on BERT for Short Fake News Detection. In Proceedings of the International Conference on Knowledge Science, Engineering and Management, Athens, Greece, 28–30 August 2019; pp. 172–183. [Google Scholar] [CrossRef]
- Kurasinski, L.; Mihailescu, R.C. Towards Machine Learning Explainability in Text Classification for Fake News Detection. In Proceedings of the IEEE International Conference on Machine Learning and Applications, Miami, FL, USA, 14–17 December 2020; pp. 775–781. [Google Scholar] [CrossRef]
- Jwa, H.; Oh, D.; Park, K.; Kang, J.; Lim, H. exBAKE: Automatic Fake News Detection Model Based on Bidirectional Encoder Representations from Transformers (BERT). Appl. Sci. 2019, 9, 4062. [Google Scholar] [CrossRef] [Green Version]
- Mersinias, M.; Afantenos, S.; Chalkiadakis, G. CLFD: A Novel Vectorization Technique and Its Application in Fake News Detection. In Proceedings of the Language Resources and Evaluation Conference, Marseille, France, 11–16 May 2020; pp. 3475–3483. [Google Scholar]
- Shaar, S.; Babulkov, N.; Da San Martino, G.; Nakov, P. That is a Known Lie: Detecting Previously Fact-Checked Claims. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, 5–10 July 2020; pp. 3607–3618. [Google Scholar] [CrossRef]
- Kula, S.; Choraś, M.; Kozik, R. Application of the BERT-Based Architecture in Fake News Detection. In Proceedings of the Conference on Complex Intelligent, and Software Intensive Systems, Asan, Korea, 1–3 July 2020; pp. 239–249. [Google Scholar] [CrossRef]
- Shu, K.; Zheng, G.; Li, Y.; Mukherjee, S.; Awadallah, A.H.; Ruston, S.; Liu, H. Leveraging multi-source weak social supervision for early detection of fake news. arXiv 2020, arXiv:2004.01732. [Google Scholar]
- Gautam, A.; Venktesh, V.; Masud, S. Fake News Detection System using XLNet model with Topic Distributions: CONSTRAINT@AAAI2021 Shared Task. In Proceedings of the CONSTRAINT Shared Task in AAAI-2021, Virtual Event, 8 February 2021; pp. 1–12. [Google Scholar] [CrossRef]
- Lan, Z.; Chen, M.; Goodman, S.; Gimpel, K.; Sharma, P.; Soricut, R. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In Proceedings of the International Conference on Learning Representations, Addis Ababa, Ethiopia, 26–30 April 2020; pp. 1–17. [Google Scholar]
- Tian, L.; Zhang, X.; Peng, M. FakeFinder: Twitter Fake News Detection on Mobile. In Proceedings of the Companion Proceedings of the Web Conference 2020, Taipei, Taiwan, 20–24 April 2020; pp. 79–80. [Google Scholar] [CrossRef]
- Khan, J.Y.; Khondaker, M.T.I.; Afroz, S.; Uddin, G.; Iqbal, A. A benchmark study of machine learning models for online fake news detection. Mach. Learn. Appl. 2021, 4, 100032. [Google Scholar] [CrossRef]
- Peters, M.E.; Neumann, M.; Iyyer, M.; Gardner, M.; Clark, C.; Lee, K.; Zettlemoyer, L. Deep Contextualized Word Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers); Association for Computational Linguistics: New Orleans, LA, USA, 2018; pp. 2227–2237. [Google Scholar] [CrossRef] [Green Version]
- Devlin, J.; Chang, M.W.; Lee, K.; Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, Minneapolis, MN, USA, 2–7 June 2019; pp. 4171–4186. [Google Scholar] [CrossRef]
- Sanh, V.; Debut, L.; Chaumond, J.; Wolf, T. DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. In Proceedings of the Workshop on Energy Efficient Machine Learning and Cognitive Computing, Vancouver, BC, Canada, 13 December 2019; pp. 1–5. [Google Scholar]
- Sajjad, H.; Dalvi, F.; Durrani, N.; Nakov, P. Poor Man’s BERT: Smaller and Faster Transformer Models. arXiv 2020, arXiv:2004.03844. [Google Scholar]
- He, P.; Liu, X.; Gao, J.; Chen, W. DeBERTa: Decoding-enhanced BERT with Disentangled Attention. arXiv 2021, arXiv:2006.03654. [Google Scholar]
- Yang, Z.; Dai, Z.; Yang, Y.; Carbonell, J.; Salakhutdinov, R.; Le, Q.V. XLNet: Generalized Autoregressive Pretraining for Language Understanding. In Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada, 8–14 December 2019; pp. 5753–5763. [Google Scholar]
- Dai, Z.; Yang, Z.; Yang, Y.; Carbonell, J.; Le, Q.; Salakhutdinov, R. Transformer-XL: Attentive Language Models beyond a Fixed-Length Context. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, Florence, Italy, 28 July–2 August 2019; pp. 2978–2988. [Google Scholar] [CrossRef] [Green Version]
- Clark, K.; Luong, M.T.; Le, Q.V.; Manning, C.D. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. In Proceedings of the International Conference on Learning Representations, Addis Ababa, Ethiopia, 26–30 April 2020; pp. 1–17. [Google Scholar]
- Conneau, A.; Lample, G. Cross-lingual Language Model Pretraining. In Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada, 8–14 December 2019; pp. 1–11. [Google Scholar]
- Conneau, A.; Khandelwal, K.; Goyal, N.; Chaudhary, V.; Wenzek, G.; Guzmán, F.; Grave, E.; Ott, M.; Zettlemoyer, L.; Stoyanov, V. Unsupervised Cross-lingual Representation Learning at Scale. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, Online, 5–10 July 2020; pp. 8440–8451. [Google Scholar] [CrossRef]
- Lewis, M.; Liu, Y.; Goyal, N.; Ghazvininejad, M.; Mohamed, A.; Levy, O.; Stoyanov, V.; Zettlemoyer, L. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, Online, 5–10 July 2020; pp. 7871–7880. [Google Scholar] [CrossRef]
- Seide, F.; Li, G.; Chen, X.; Yu, D. Feature engineering in Context-Dependent Deep Neural Networks for conversational speech transcription. In Proceedings of the IEEE Workshop on Automatic Speech Recognition & Understanding, Waikoloa, HI, USA, 11–15 December 2011. [Google Scholar] [CrossRef]
- Hochreiter, S.; Schmidhuber, J. Long Short-Term Memory. Neural Comput. 1997, 9, 1735–1780. [Google Scholar] [CrossRef]
- Guo, B.; Zhang, C.; Liu, J.; Ma, X. Improving text classification with weighted word embeddings via a multi-channel TextCNN model. Neurocomputing 2019, 363, 366–374. [Google Scholar] [CrossRef]
- Szpakowski, M. Fake News Corpus. 2021. Available online: https://github.com/several27/FakeNewsCorpus (accessed on 6 August 2021).
- Geva, M.; Goldberg, Y.; Berant, J. Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, 3–7 November 2019; pp. 1161–1166. [Google Scholar] [CrossRef]
- Kuwatly, H.A.; Wich, M.; Groh, G. Identifying and Measuring Annotator Bias Based on Annotators’ Demographic Characteristics. In Proceedings of the Fourth Workshop on Online Abuse and Harms. Association for Computational Linguistics, Online, 20 November 2020; pp. 184–190. [Google Scholar] [CrossRef]
- Jie, Z.; Xie, P.; Lu, W.; Ding, R.; Li, L. Better Modeling of Incomplete Annotations for Named Entity Recognition. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA, 2–7 June 2019; pp. 729–734. [Google Scholar] [CrossRef] [Green Version]
- Bird, S.; Loper, E.; Klein, E. Natural Language Processing with Python; O’Reilly Media Inc.: Sebastopol, CA, USA, 2009. [Google Scholar]
- Grootendorst, M. PolyFuzz. 2021. Available online: https://maartengr.github.io/PolyFuzz/ (accessed on 14 October 2021).
- Arora, S.; Ge, R.; Moitra, A. Learning Topic Models—Going beyond SVD. In Proceedings of the Annual Symposium on Foundations of Computer Science, New Brunswick, NJ, USA, 20–23 October 2012; pp. 1–10. [Google Scholar]
- Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Dubourg, V.; et al. Scikit-learn: Machine Learning in Python. J. Mach. Learn. Res. 2011, 12, 2825–2830. [Google Scholar]
- Rajapakse, T. SimpleTransformers. 2021. Available online: https://simpletransformers.ai (accessed on 12 October 2021).
- Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M.; et al. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, Punta Cana, Dominican Republic, 8–12 November 2020; pp. 38–45. [Google Scholar]
- Abadi, M.; Agarwal, A.; Barham, P.; Brevdo, E.; Chen, Z.; Citro, C.; Corrado, G.S.; Davis, A.; Dean, J.; Devin, M.; et al. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. 2015. Available online: https://www.tensorflow.org/ (accessed on 8 October 2021).
- Chollet, F.; Jin, H.; Nozawa, K. Keras. 2015. Available online: https://keras.io (accessed on 8 October 2021).
- Zampieri, M.; Nakov, P.; Rosenthal, S.; Atanasova, P.; Karadzhov, G.; Mubarak, H.; Derczynski, L.; Pitenis, Z.; Çağrı, Ç. SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020). In Proceedings of the Workshop on Semantic Evaluation, Online, 12–13 December 2020; pp. 1425–1447. [Google Scholar]
- Parikh, S.B.; Atrey, P.K. Media-rich fake news detection: A survey. In Proceedings of the 2018 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), Miami, FL, USA, 10–12 April 2018; pp. 436–441. [Google Scholar] [CrossRef]
- Mitroi, M.; Truică, C.O.; Apostol, E.S.; Florea, A.M. Sentiment Analysis using Topic-Document Embeddings. In Proceedings of the IEEE 16th International Conference on Intelligent Computer Communication and Processing, Cluj-Napoca, Romania, 3–5 September 2020; pp. 75–82. [Google Scholar] [CrossRef]
- Truică, C.O.; Apostol, E.S.; Șerban, M.L.; Paschke, A. Topic-Based Document-Level Sentiment Analysis Using Contextual Cues. Mathematics 2021, 9, 2722. [Google Scholar] [CrossRef]
Class | Description |
---|---|
Fake News | Fabricated or distorted information |
Satire | Humorous or ironic information |
Extreme Bias | Propaganda articles |
Conspiracy Theory | Promote conspiracy theories |
Junk Science | Scientifically dubious claims |
Hate News | Promote discrimination |
Clickbait | Credible content, but misleading headline |
Proceed With Caution | May be reliable or not |
Political | Promote political orientations |
Credible | Reliable information |
Class | Number of Tokens per Document | Number of Tokens per Class | ||||
---|---|---|---|---|---|---|
Mean | Min | Max | StdDev | Unique | All | |
Fake News | 660.86 | 7 | 18,179 | 899.56 | 318,808 | 6,808,099 |
Satire | 245.19 | 11 | 4942 | 232.01 | 153,509 | 2,502,701 |
Extreme Bias | 539.71 | 5 | 17,402 | 1192.27 | 308,827 | 5,549,908 |
Conspiracy Theory | 800.17 | 7 | 17,448 | 947.65 | 315,146 | 8,182,202 |
Junk Science | 511.05 | 7 | 12,716 | 732.69 | 253,004 | 5,208,868 |
Hate News | 930.92 | 5 | 17,871 | 2064.67 | 348,295 | 9,516,495 |
Clickbait | 361.20 | 7 | 5544 | 361.62 | 193,844 | 3,702,766 |
Unreliable Sources | 505.63 | 7 | 17,232 | 982.76 | 236,495 | 5,149,370 |
Political Bias | 547.17 | 9 | 15,221 | 795.81 | 258,269 | 5,581,168 |
Credible | 737.98 | 6 | 15,049 | 818.02 | 257,475 | 7,524,605 |
Entire dataset statistics | 583.99 | 5 | 18,179 | 1037.13 | 1,337,708 | 59,726,182 |
Class | Top-10 Unigrams | Bert Similarity | FastText Similarity |
---|---|---|---|
Entire Dataset | People Time Government American World System Year America State Public | ||
Fake News | people time government world year story market American day God | 0.89 | 0.84 |
Satire | order close continue user policy agree send deny click advertising | 0.63 | 0.44 |
Extreme Bias | talk people American government time Russia America state war country | 0.87 | 0.84 |
Conspiracy | people government time American America world report year country state | 0.94 | 0.91 |
Junk Science | health people food free time world found body cancer study | 0.76 | 0.67 |
Hate News | people black white snip percent American whites time country race | 0.79 | 0.71 |
Clickbait | people Donald time state Clinton government America told country American | 0.87 | 0.84 |
Unreliable Sources | system Tor operating computer submission stick people public time order | 0.79 | 0.71 |
Political Bias | people government time American percent year state president tax political | 0.89 | 0.83 |
Credible | people God Christian government American time world war told Iraq | 0.85 | 0.81 |
Class | Top-1 Topic | Bert Similarity | FastText Similarity |
---|---|---|---|
Entire Dataset | Banner Preference Navigation Consent Technical Advertising Profile Element User Click | ||
Fake News | not people Trump year day government time state world no | 0.63 | 0.41 |
Satire | navigation banner advertising consent technical preference element click profile access | 0.97 | 0.95 |
Extreme Bias | talk hide link page category template user supply file previous | 0.71 | 0.51 |
Conspiracy | not report government people president year world state no time | 0.65 | 0.41 |
Junk Science | food free health reference offer documentary herb nutrition list program | 0.68 | 0.42 |
Hate News | not black white snip people year percent school no student | 0.64 | 0.41 |
Clickbait | Trump not president people twitter video state woman year no | 0.64 | 0.41 |
Unreliable Sources | Tor system tail browser submission computer stick communication GNU bundle | 0.70 | 0.45 |
Political Bias | Trump not president year state people house government no white | 0.64 | 0.41 |
Credible | not church people Trump God president no war Bush state | 0.63 | 0.39 |
Model | Pretrained Model Name | Encoder Layers | Hidden State | Attention Heads | Parameters |
---|---|---|---|---|---|
BERT base | bert-base-cased | 12 | 768 | 12 | 110 M |
BERT large | bert-large-cased | 24 | 1024 | 16 | 335 M |
DistilBERT | distilbert-base-cased | 6 | 768 | 12 | 65 M |
RoBERTa base | roberta-base | 12 | 768 | 12 | 125 M |
RoBERTa large | roberta-large | 24 | 1024 | 16 | 355 M |
DistilRoBERTa | distilroberta-base | 6 | 768 | 12 | 82 M |
XLNet base | xlnet-base-cased | 12 | 768 | 12 | 110 M |
XLNet large | xlnet-large-cased | 24 | 1024 | 16 | 340 M |
ALBERT base v1 | albert-base-v1 | 12 | 768 | 12 | 11 M |
ALBERT base v2 | albert-base-v2 | 12 | 768 | 12 | 11 M |
ALBERT xxlarge v1 | albert-xxlarge-v1 | 12 | 4096 | 64 | 223 M |
ALBERT xxlarge v2 | albert-xxlarge-v2 | 12 | 4096 | 64 | 223 M |
DeBERTa base | microsoft/deberta-base | 12 | 768 | 12 | 140 M |
DeBERTa large | microsoft/deberta-large | 24 | 1024 | 16 | 400 M |
ELECTRA base | google/electra-base-discriminator | 12 | 768 | 12 | 110 M |
ELECTRA large | google/electra-large-discriminator | 24 | 1024 | 16 | 335 M |
XLM | xlm-mlm-100-1280 | 16 | 1280 | 16 | 550 M |
XLM-RoBERTa base | xlm-roberta-base | 12 | 768 | 8 | 270 M |
XLM-RoBERTa large | lm-roberta-large | 24 | 1024 | 16 | 550 M |
BART base | facebook/bart-base | 12 | 768 | 16 | 139 M |
BART large | facebook/bart-large | 12 | 1024 | 16 | 406 M |
Model | Accuracy | Micro Precision | Macro Precision | Micro Recall | Macro Recall | Execution Time (Hours) |
---|---|---|---|---|---|---|
MisRoBÆRTa | 92.50 ± 0.26 | 92.50 ± 0.26 | 92.69 ± 0.21 | 92.50 ± 0.26 | 92.50 ± 0.26 | 1.83 ± 0.01 |
BERT base | 90.03 ± 0.19 | 90.03 ± 0.19 | 90.05 ± 0.21 | 90.03 ± 0.19 | 90.03 ± 0.19 | 5.12 ± 0.22 |
BERT large | 89.12 ± 0.14 | 89.12 ± 0.14 | 89.11 ± 0.15 | 89.12 ± 0.14 | 89.12 ± 0.14 | 8.97 ± 0.05 |
DistilBERT | 89.36 ± 0.15 | 89.36 ± 0.15 | 89.40 ± 0.16 | 89.36 ± 0.15 | 89.36 ± 0.15 | 2.30 ± 0.01 |
RoBERTa base | 91.36 ± 0.15 | 91.36 ± 0.15 | 91.39 ± 0.16 | 91.36 ± 0.15 | 91.36 ± 0.15 | 3.95 ± 0.15 |
RoBERTa large | 88.60 ± 0.23 | 88.61 ± 0.22 | 88.85 ± 0.23 | 88.59 ± 0.24 | 88.62 ± 0.21 | 6.55 ± 0.32 |
DistilRoBERTa | 91.32 ± 0.10 | 91.32 ± 0.10 | 91.34 ± 0.08 | 91.32 ± 0.10 | 91.32 ± 0.10 | 1.93 ± 0.01 |
ALBERT base v1 | 86.92 ± 0.17 | 86.92 ± 0.17 | 86.95 ± 0.18 | 86.92 ± 0.17 | 86.92 ± 0.17 | 2.16 ± 0.19 |
ALBERT base v2 | 85.05 ± 0.20 | 85.05 ± 0.20 | 85.08 ± 0.17 | 85.05 ± 0.20 | 85.05 ± 0.20 | 2.76 ± 0.17 |
ALBERT xxlarge v1 | 89.20 ± 0.04 | 89.20 ± 0.04 | 89.36 ± 0.02 | 89.20 ± 0.04 | 89.20 ± 0.04 | 9.98 ± 0.09 |
ALBERT xxlarge v2 | 86.51 ± 2.23 | 86.51 ± 2.23 | 86.66 ± 2.21 | 86.51 ± 2.23 | 86.51 ± 2.23 | 11.14 ± 0.04 |
DeBERTa base | 90.56 ± 0.08 | 90.56 ± 0.08 | 90.56 ± 0.10 | 90.56 ± 0.08 | 90.56 ± 0.08 | 6.90 ± 0.35 |
DeBERTa large | 89.93 ± 0.27 | 89.93 ± 0.27 | 89.96 ± 0.29 | 89.93 ± 0.27 | 89.93 ± 0.27 | 11.06 ± 1.10 |
XLNet base | 89.94 ± 0.09 | 89.94 ± 0.09 | 89.94 ± 0.09 | 89.94 ± 0.09 | 89.94 ± 0.09 | 5.50 ± 0.06 |
XLNet large | 88.05 ± 0.39 | 88.04 ± 0.38 | 88.38 ± 0.39 | 88.03 ± 0.37 | 88.07 ± 0.40 | 10.44 ± 1.54 |
ELECTRA base | 87.10 ± 0.24 | 87.10 ± 0.24 | 87.12 ± 0.22 | 87.10 ± 0.24 | 87.10 ± 0.24 | 4.49 ± 0.34 |
ELECTRA large | 86.92 ± 0.09 | 86.92 ± 0.09 | 86.92 ± 0.08 | 86.92 ± 0.09 | 86.92 ± 0.09 | 8.74 ± 0.43 |
XLM | 85.82 ± 0.19 | 85.82 ± 0.19 | 85.90 ± 0.23 | 85.82 ± 0.19 | 85.82 ± 0.19 | 5.16 ± 0.01 |
XLM-RoBERTa base | 89.78 ± 0.17 | 89.78 ± 0.17 | 89.77 ± 0.17 | 89.78 ± 0.17 | 89.78 ± 0.17 | 6.97 ± 0.38 |
XLM-RoBERTa large | 87.50 ± 0.61 | 87.50 ± 0.61 | 87.58 ± 0.58 | 87.50 ± 0.61 | 87.50 ± 0.61 | 9.46 ± 2.16 |
BART base | 91.35 ± 0.04 | 91.35 ± 0.04 | 91.35 ± 0.04 | 91.35 ± 0.04 | 91.35 ± 0.04 | 4.17 ± 0.16 |
BART large | 91.94 ± 0.15 | 91.94 ± 0.15 | 91.97 ± 0.16 | 91.94 ± 0.15 | 91.94 ± 0.15 | 6.79 ± 0.29 |
FakeBERT [8] | 70.18 ± 0.01 | 70.18 ± 0.01 | 70.21 ± 0.03 | 70.18 ± 0.01 | 70.18 ± 0.01 | 2.59 ± 0.01 |
32 Units/Layer | MisRoBÆRTa with LSTM Cells | MisRoBÆRTa with BiLSTM Cells | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
BART Branch | RoBERTa Branch | Ensemble Branch | Accuracy | Precision Micro | Precision Macro | Recall Micro | Recall Macro | Execution Time (Hours) | Accuracy | Precision Micro | Precision Macro | Recall Micro | Recall Macro | Execution Time (Hours) |
1 × [Bi]LSTM | 1 × [Bi]LSTM | 1 × [Bi]LSTM | 97.10 ± 0.43 | 97.10 ± 0.43 | 97.16 ± 0.42 | 97.10 ± 0.43 | 97.07 ± 0.43 | 0.01 ± 0.00 | 97.35 ± 0.37 | 97.35 ± 0.37 | 97.40 ± 0.36 | 97.35 ± 0.37 | 97.33 ± 0.37 | 0.01 ± 0.00 |
1 × [Bi]LSTM | 1 × [Bi]LSTM | 2 × [Bi]LSTM | 97.14 ± 0.35 | 97.14 ± 0.35 | 97.17 ± 0.35 | 97.14 ± 0.35 | 97.12 ± 0.35 | 0.02 ± 0.00 | 97.30 ± 0.42 | 97.30 ± 0.42 | 97.33 ± 0.38 | 97.30 ± 0.42 | 97.29 ± 0.45 | 0.03 ± 0.00 |
1 × [Bi]LSTM | 1 × [Bi]LSTM | 3 × [Bi]LSTM | 97.10 ± 0.57 | 97.10 ± 0.57 | 97.13 ± 0.56 | 97.10 ± 0.57 | 97.08 ± 0.57 | 0.03 ± 0.00 | 97.32 ± 0.37 | 97.32 ± 0.37 | 97.34 ± 0.35 | 97.32 ± 0.37 | 97.34 ± 0.36 | 0.05 ± 0.00 |
1 × [Bi]LSTM | 2 × [Bi]LSTM | 1 × [Bi]LSTM | 97.06 ± 0.28 | 97.06 ± 0.28 | 97.09 ± 0.28 | 97.06 ± 0.28 | 97.05 ± 0.27 | 0.04 ± 0.00 | 97.02 ± 0.48 | 97.02 ± 0.48 | 97.10 ± 0.44 | 97.02 ± 0.48 | 96.99 ± 0.49 | 0.06 ± 0.00 |
1 × [Bi]LSTM | 2 × [Bi]LSTM | 2 × [Bi]LSTM | 97.15 ± 0.35 | 97.15 ± 0.35 | 97.16 ± 0.35 | 97.15 ± 0.35 | 97.15 ± 0.35 | 0.05 ± 0.00 | 97.43 ± 0.28 | 97.43 ± 0.28 | 97.44 ± 0.26 | 97.43 ± 0.28 | 97.43 ± 0.30 | 0.08 ± 0.00 |
1 × [Bi]LSTM | 2 × [Bi]LSTM | 3 × [Bi]LSTM | 97.16 ± 0.37 | 97.16 ± 0.37 | 97.18 ± 0.37 | 97.16 ± 0.37 | 97.17 ± 0.35 | 0.07 ± 0.00 | 97.36 ± 0.32 | 97.36 ± 0.32 | 97.38 ± 0.30 | 97.36 ± 0.32 | 97.38 ± 0.33 | 0.11 ± 0.00 |
1 × [Bi]LSTM | 3 × [Bi]LSTM | 1 × [Bi]LSTM | 97.09 ± 0.46 | 97.09 ± 0.46 | 97.10 ± 0.45 | 97.09 ± 0.46 | 97.11 ± 0.44 | 0.08 ± 0.00 | 97.33 ± 0.32 | 97.33 ± 0.32 | 97.35 ± 0.30 | 97.33 ± 0.32 | 97.32 ± 0.34 | 0.13 ± 0.00 |
1 × [Bi]LSTM | 3 × [Bi]LSTM | 2 × [Bi]LSTM | 97.07 ± 0.37 | 97.07 ± 0.37 | 97.08 ± 0.38 | 97.07 ± 0.37 | 97.08 ± 0.36 | 0.09 ± 0.00 | 97.14 ± 0.41 | 97.14 ± 0.41 | 97.16 ± 0.39 | 97.14 ± 0.41 | 97.17 ± 0.39 | 0.15 ± 0.00 |
1 × [Bi]LSTM | 3 × [Bi]LSTM | 3 × [Bi]LSTM | 97.30 ± 0.37 | 97.30 ± 0.37 | 97.35 ± 0.38 | 97.30 ± 0.37 | 97.27 ± 0.36 | 0.11 ± 0.00 | 97.24 ± 0.53 | 97.24 ± 0.53 | 97.31 ± 0.48 | 97.24 ± 0.53 | 97.22 ± 0.55 | 0.18 ± 0.01 |
2 × [Bi]LSTM | 1 × [Bi]LSTM | 1 × [Bi]LSTM | 97.56 ± 0.29 | 97.56 ± 0.29 | 97.57 ± 0.27 | 97.56 ± 0.29 | 97.55 ± 0.30 | 0.12 ± 0.00 | 97.57 ± 0.29 | 97.57 ± 0.29 | 97.58 ± 0.28 | 97.57 ± 0.29 | 97.57 ± 0.31 | 0.19 ± 0.01 |
2 × [Bi]LSTM | 1 × [Bi]LSTM | 2 × [Bi]LSTM | 97.34 ± 0.36 | 97.34 ± 0.36 | 97.38 ± 0.35 | 97.34 ± 0.36 | 97.32 ± 0.36 | 0.13 ± 0.00 | 97.44 ± 0.38 | 97.44 ± 0.38 | 97.44 ± 0.38 | 97.44 ± 0.38 | 97.44 ± 0.37 | 0.21 ± 0.01 |
2 × [Bi]LSTM | 1 × [Bi]LSTM | 3 × [Bi]LSTM | 97.20 ± 0.38 | 97.20 ± 0.38 | 97.24 ± 0.36 | 97.20 ± 0.38 | 97.18 ± 0.39 | 0.15 ± 0.00 | 97.31 ± 0.44 | 97.31 ± 0.44 | 97.34 ± 0.42 | 97.31 ± 0.44 | 97.30 ± 0.46 | 0.23 ± 0.01 |
2 × [Bi]LSTM | 2 × [Bi]LSTM | 1 × [Bi]LSTM | 97.31 ± 0.47 | 97.31 ± 0.47 | 97.38 ± 0.41 | 97.31 ± 0.47 | 97.28 ± 0.50 | 0.16 ± 0.00 | 97.41 ± 0.37 | 97.41 ± 0.37 | 97.42 ± 0.38 | 97.41 ± 0.37 | 97.41 ± 0.37 | 0.25 ± 0.01 |
2 × [Bi]LSTM | 2 × [Bi]LSTM | 2 × [Bi]LSTM | 97.44 ± 0.26 | 97.44 ± 0.26 | 97.45 ± 0.26 | 97.44 ± 0.26 | 97.42 ± 0.27 | 0.18 ± 0.00 | 97.26 ± 0.37 | 97.26 ± 0.37 | 97.30 ± 0.36 | 97.26 ± 0.37 | 97.25 ± 0.36 | 0.28 ± 0.01 |
2 × [Bi]LSTM | 2 × [Bi]LSTM | 3 × [Bi]LSTM | 97.30 ± 0.33 | 97.30 ± 0.33 | 97.34 ± 0.25 | 97.30 ± 0.33 | 97.29 ± 0.35 | 0.19 ± 0.00 | 97.40 ± 0.44 | 97.40 ± 0.44 | 97.44 ± 0.42 | 97.40 ± 0.44 | 97.37 ± 0.44 | 0.30 ± 0.01 |
2 × [Bi]LSTM | 3 × [Bi]LSTM | 1 × [Bi]LSTM | 97.37 ± 0.41 | 97.37 ± 0.41 | 97.40 ± 0.41 | 97.37 ± 0.41 | 97.36 ± 0.41 | 0.21 ± 0.00 | 97.40 ± 0.37 | 97.40 ± 0.37 | 97.42 ± 0.36 | 97.40 ± 0.37 | 97.39 ± 0.37 | 0.33 ± 0.01 |
2 × [Bi]LSTM | 3 × [Bi]LSTM | 2 × [Bi]LSTM | 97.39 ± 0.36 | 97.39 ± 0.36 | 97.39 ± 0.37 | 97.39 ± 0.36 | 97.39 ± 0.36 | 0.22 ± 0.00 | 97.46 ± 0.22 | 97.46 ± 0.22 | 97.47 ± 0.20 | 97.46 ± 0.22 | 97.47 ± 0.20 | 0.35 ± 0.01 |
2 × [Bi]LSTM | 3 × [Bi]LSTM | 3 × [Bi]LSTM | 97.09 ± 0.35 | 97.09 ± 0.35 | 97.14 ± 0.34 | 97.09 ± 0.35 | 97.10 ± 0.33 | 0.24 ± 0.00 | 97.16 ± 0.37 | 97.16 ± 0.37 | 97.17 ± 0.38 | 97.16 ± 0.37 | 97.16 ± 0.36 | 0.38 ± 0.01 |
3 × [Bi]LSTM | 1 × [Bi]LSTM | 1 × [Bi]LSTM | 97.43 ± 0.39 | 97.43 ± 0.39 | 97.46 ± 0.37 | 97.43 ± 0.39 | 97.41 ± 0.41 | 0.26 ± 0.00 | 97.45 ± 0.39 | 97.45 ± 0.39 | 97.48 ± 0.39 | 97.45 ± 0.39 | 97.44 ± 0.39 | 0.40 ± 0.01 |
3 × [Bi]LSTM | 1 × [Bi]LSTM | 2 × [Bi]LSTM | 97.42 ± 0.33 | 97.42 ± 0.33 | 97.44 ± 0.34 | 97.42 ± 0.33 | 97.40 ± 0.32 | 0.27 ± 0.01 | 97.36 ± 0.35 | 97.36 ± 0.35 | 97.40 ± 0.35 | 97.36 ± 0.35 | 97.35 ± 0.35 | 0.43 ± 0.01 |
3 × [Bi]LSTM | 1 × [Bi]LSTM | 3 × [Bi]LSTM | 97.19 ± 0.29 | 97.19 ± 0.29 | 97.30 ± 0.23 | 97.19 ± 0.29 | 97.15 ± 0.32 | 0.29 ± 0.00 | 97.41 ± 0.27 | 97.41 ± 0.27 | 97.42 ± 0.29 | 97.41 ± 0.27 | 97.40 ± 0.27 | 0.46 ± 0.01 |
3 × [Bi]LSTM | 2 × [Bi]LSTM | 1 × [Bi]LSTM | 97.25 ± 0.44 | 97.25 ± 0.44 | 97.28 ± 0.41 | 97.25 ± 0.44 | 97.25 ± 0.42 | 0.31 ± 0.01 | 97.33 ± 0.47 | 97.33 ± 0.47 | 97.38 ± 0.44 | 97.33 ± 0.47 | 97.33 ± 0.43 | 0.48 ± 0.01 |
3 × [Bi]LSTM | 2 × [Bi]LSTM | 2 × [Bi]LSTM | 97.31 ± 0.32 | 97.31 ± 0.32 | 97.36 ± 0.28 | 97.31 ± 0.32 | 97.29 ± 0.34 | 0.32 ± 0.01 | 97.30 ± 0.39 | 97.30 ± 0.39 | 97.34 ± 0.32 | 97.30 ± 0.39 | 97.29 ± 0.42 | 0.51 ± 0.01 |
3 × [Bi]LSTM | 2 × [Bi]LSTM | 3 × [Bi]LSTM | 97.23 ± 0.41 | 97.23 ± 0.41 | 97.26 ± 0.40 | 97.23 ± 0.41 | 97.23 ± 0.38 | 0.34 ± 0.01 | 97.48 ± 0.42 | 97.48 ± 0.42 | 97.48 ± 0.42 | 97.48 ± 0.42 | 97.50 ± 0.41 | 0.54 ± 0.01 |
3 × [Bi]LSTM | 3 × [Bi]LSTM | 1 × [Bi]LSTM | 97.41 ± 0.34 | 97.41 ± 0.34 | 97.43 ± 0.35 | 97.41 ± 0.34 | 97.39 ± 0.32 | 0.36 ± 0.01 | 97.31 ± 0.30 | 97.31 ± 0.30 | 97.35 ± 0.31 | 97.31 ± 0.30 | 97.30 ± 0.30 | 0.57 ± 0.01 |
3 × [Bi]LSTM | 3 × [Bi]LSTM | 2 × [Bi]LSTM | 97.24 ± 0.29 | 97.24 ± 0.29 | 97.29 ± 0.31 | 97.24 ± 0.29 | 97.21 ± 0.28 | 0.38 ± 0.01 | 97.42 ± 0.45 | 97.42 ± 0.45 | 97.44 ± 0.43 | 97.42 ± 0.45 | 97.42 ± 0.46 | 0.60 ± 0.02 |
3 × [Bi]LSTM | 3 × [Bi]LSTM | 3 × [Bi]LSTM | 97.15 ± 0.31 | 97.15 ± 0.31 | 97.17 ± 0.30 | 97.15 ± 0.31 | 97.13 ± 0.32 | 0.40 ± 0.01 | 97.30 ± 0.44 | 97.30 ± 0.44 | 97.32 ± 0.43 | 97.30 ± 0.44 | 97.31 ± 0.43 | 0.63 ± 0.02 |
64 Units/Layer | MisRoBÆRTa with LSTM Cells | MisRoBÆRTa with BiLSTM Cells | ||||||||||||
BART Branch | RoBERTa Branch | Ensemble Branch | Accuracy | Precision Micro | Precision Macro | Recall Micro | Recall Macro | Execution Time (Hours) | Accuracy | Precision Micro | Precision Macro | Recall Micro | Recall Macro | Execution Time (Hours) |
1 × [Bi]LSTM | 1 × [Bi]LSTM | 1 × [Bi]LSTM | 97.24 ± 0.78 | 97.24 ± 0.78 | 97.29 ± 0.72 | 97.24 ± 0.78 | 97.25 ± 0.77 | 0.01 ± 0.00 | 97.42 ± 0.47 | 97.42 ± 0.47 | 97.49 ± 0.40 | 97.42 ± 0.47 | 97.39 ± 0.48 | 0.01 ± 0.00 |
1 × [Bi]LSTM | 1 × [Bi]LSTM | 2 × [Bi]LSTM | 97.36 ± 0.42 | 97.36 ± 0.42 | 97.41 ± 0.39 | 97.36 ± 0.42 | 97.34 ± 0.43 | 0.02 ± 0.00 | 97.29 ± 0.48 | 97.29 ± 0.48 | 97.35 ± 0.40 | 97.29 ± 0.48 | 97.29 ± 0.50 | 0.03 ± 0.00 |
1 × [Bi]LSTM | 1 × [Bi]LSTM | 3 × [Bi]LSTM | 97.42 ± 0.47 | 97.42 ± 0.47 | 97.44 ± 0.41 | 97.42 ± 0.47 | 97.41 ± 0.48 | 0.03 ± 0.00 | 97.20 ± 0.50 | 97.20 ± 0.50 | 97.25 ± 0.46 | 97.20 ± 0.50 | 97.20 ± 0.48 | 0.04 ± 0.00 |
1 × [Bi]LSTM | 2 × [Bi]LSTM | 1 × [Bi]LSTM | 97.48 ± 0.37 | 97.48 ± 0.37 | 97.48 ± 0.37 | 97.48 ± 0.37 | 97.48 ± 0.38 | 0.04 ± 0.00 | 97.26 ± 0.28 | 97.26 ± 0.28 | 97.36 ± 0.17 | 97.26 ± 0.28 | 97.24 ± 0.33 | 0.06 ± 0.00 |
1 × [Bi]LSTM | 2 × [Bi]LSTM | 2 × [Bi]LSTM | 97.33 ± 0.39 | 97.33 ± 0.39 | 97.34 ± 0.39 | 97.33 ± 0.39 | 97.33 ± 0.38 | 0.05 ± 0.00 | 97.13 ± 0.39 | 97.13 ± 0.39 | 97.25 ± 0.30 | 97.13 ± 0.39 | 97.10 ± 0.43 | 0.08 ± 0.00 |
1 × [Bi]LSTM | 2 × [Bi]LSTM | 3 × [Bi]LSTM | 97.36 ± 0.42 | 97.36 ± 0.42 | 97.38 ± 0.40 | 97.36 ± 0.42 | 97.35 ± 0.40 | 0.06 ± 0.00 | 97.35 ± 0.39 | 97.35 ± 0.39 | 97.40 ± 0.37 | 97.35 ± 0.39 | 97.34 ± 0.39 | 0.10 ± 0.00 |
1 × [Bi]LSTM | 3 × [Bi]LSTM | 1 × [Bi]LSTM | 97.32 ± 0.46 | 97.32 ± 0.46 | 97.35 ± 0.45 | 97.32 ± 0.46 | 97.31 ± 0.46 | 0.07 ± 0.00 | 97.29 ± 0.77 | 97.29 ± 0.77 | 97.37 ± 0.67 | 97.29 ± 0.77 | 97.29 ± 0.73 | 0.12 ± 0.00 |
1 × [Bi]LSTM | 3 × [Bi]LSTM | 2 × [Bi]LSTM | 97.53 ± 0.34 | 97.53 ± 0.34 | 97.56 ± 0.33 | 97.53 ± 0.34 | 97.50 ± 0.35 | 0.09 ± 0.00 | 97.20 ± 0.79 | 97.20 ± 0.79 | 97.30 ± 0.66 | 97.20 ± 0.79 | 97.22 ± 0.73 | 0.14 ± 0.00 |
1 × [Bi]LSTM | 3 × [Bi]LSTM | 3 × [Bi]LSTM | 97.32 ± 0.40 | 97.32 ± 0.40 | 97.32 ± 0.39 | 97.32 ± 0.40 | 97.33 ± 0.38 | 0.10 ± 0.00 | 97.33 ± 0.50 | 97.33 ± 0.50 | 97.38 ± 0.45 | 97.33 ± 0.50 | 97.35 ± 0.47 | 0.16 ± 0.00 |
2 × [Bi]LSTM | 1 × [Bi]LSTM | 1 × [Bi]LSTM | 97.84 ± 0.50 | 97.84 ± 0.50 | 97.85 ± 0.50 | 97.84 ± 0.50 | 97.84 ± 0.48 | 0.11 ± 0.00 | 97.85 ± 0.23 | 97.85 ± 0.23 | 97.88 ± 0.24 | 97.85 ± 0.23 | 97.83 ± 0.22 | 0.18 ± 0.00 |
2 × [Bi]LSTM | 1 × [Bi]LSTM | 2 × [Bi]LSTM | 97.56 ± 0.38 | 97.56 ± 0.38 | 97.58 ± 0.40 | 97.56 ± 0.38 | 97.55 ± 0.37 | 0.12 ± 0.00 | 97.40 ± 0.38 | 97.40 ± 0.38 | 97.51 ± 0.34 | 97.40 ± 0.38 | 97.36 ± 0.38 | 0.20 ± 0.00 |
2 × [Bi]LSTM | 1 × [Bi]LSTM | 3 × [Bi]LSTM | 97.56 ± 0.48 | 97.56 ± 0.48 | 97.60 ± 0.42 | 97.56 ± 0.48 | 97.56 ± 0.50 | 0.14 ± 0.00 | 97.39 ± 0.36 | 97.39 ± 0.36 | 97.45 ± 0.37 | 97.39 ± 0.36 | 97.37 ± 0.33 | 0.22 ± 0.00 |
2 × [Bi]LSTM | 2 × [Bi]LSTM | 1 × [Bi]LSTM | 97.53 ± 0.48 | 97.53 ± 0.48 | 97.57 ± 0.48 | 97.53 ± 0.48 | 97.50 ± 0.47 | 0.15 ± 0.00 | 97.33 ± 0.46 | 97.33 ± 0.46 | 97.41 ± 0.35 | 97.33 ± 0.46 | 97.32 ± 0.50 | 0.24 ± 0.01 |
2 × [Bi]LSTM | 2 × [Bi]LSTM | 2 × [Bi]LSTM | 97.56 ± 0.45 | 97.56 ± 0.45 | 97.61 ± 0.42 | 97.56 ± 0.45 | 97.55 ± 0.43 | 0.16 ± 0.00 | 97.38 ± 0.65 | 97.38 ± 0.65 | 97.42 ± 0.60 | 97.38 ± 0.65 | 97.38 ± 0.63 | 0.26 ± 0.01 |
2 × [Bi]LSTM | 2 × [Bi]LSTM | 3 × [Bi]LSTM | 97.53 ± 0.38 | 97.53 ± 0.38 | 97.56 ± 0.36 | 97.53 ± 0.38 | 97.52 ± 0.39 | 0.18 ± 0.00 | 97.35 ± 0.41 | 97.35 ± 0.41 | 97.43 ± 0.34 | 97.35 ± 0.41 | 97.34 ± 0.41 | 0.28 ± 0.01 |
2 × [Bi]LSTM | 3 × [Bi]LSTM | 1 × [Bi]LSTM | 97.52 ± 0.35 | 97.52 ± 0.35 | 97.55 ± 0.34 | 97.52 ± 0.35 | 97.52 ± 0.35 | 0.19 ± 0.00 | 97.47 ± 0.21 | 97.47 ± 0.21 | 97.52 ± 0.20 | 97.47 ± 0.21 | 97.44 ± 0.22 | 0.30 ± 0.01 |
2 × [Bi]LSTM | 3 × [Bi]LSTM | 2 × [Bi]LSTM | 97.52 ± 0.37 | 97.52 ± 0.37 | 97.57 ± 0.34 | 97.52 ± 0.37 | 97.50 ± 0.39 | 0.21 ± 0.00 | 97.46 ± 0.34 | 97.46 ± 0.34 | 97.52 ± 0.29 | 97.46 ± 0.34 | 97.47 ± 0.31 | 0.33 ± 0.01 |
2 × [Bi]LSTM | 3 × [Bi]LSTM | 3 × [Bi]LSTM | 97.63 ± 0.31 | 97.63 ± 0.31 | 97.65 ± 0.30 | 97.63 ± 0.31 | 97.60 ± 0.31 | 0.22 ± 0.01 | 97.46 ± 0.25 | 97.46 ± 0.25 | 97.49 ± 0.18 | 97.46 ± 0.25 | 97.46 ± 0.28 | 0.36 ± 0.01 |
3 × [Bi]LSTM | 1 × [Bi]LSTM | 1 × [Bi]LSTM | 97.72 ± 0.25 | 97.72 ± 0.25 | 97.73 ± 0.26 | 97.72 ± 0.25 | 97.70 ± 0.25 | 0.24 ± 0.01 | 97.60 ± 0.37 | 97.60 ± 0.37 | 97.65 ± 0.40 | 97.60 ± 0.37 | 97.56 ± 0.37 | 0.38 ± 0.01 |
3 × [Bi]LSTM | 1 × [Bi]LSTM | 2 × [Bi]LSTM | 97.67 ± 0.43 | 97.67 ± 0.43 | 97.72 ± 0.43 | 97.67 ± 0.43 | 97.65 ± 0.41 | 0.25 ± 0.01 | 97.66 ± 0.34 | 97.66 ± 0.34 | 97.66 ± 0.33 | 97.66 ± 0.34 | 97.66 ± 0.33 | 0.40 ± 0.01 |
3 × [Bi]LSTM | 1 × [Bi]LSTM | 3 × [Bi]LSTM | 97.58 ± 0.43 | 97.58 ± 0.43 | 97.64 ± 0.35 | 97.58 ± 0.43 | 97.55 ± 0.47 | 0.27 ± 0.01 | 97.57 ± 0.31 | 97.57 ± 0.31 | 97.60 ± 0.32 | 97.57 ± 0.31 | 97.55 ± 0.31 | 0.42 ± 0.01 |
3 × [Bi]LSTM | 2 × [Bi]LSTM | 1 × [Bi]LSTM | 97.59 ± 0.30 | 97.59 ± 0.30 | 97.62 ± 0.29 | 97.59 ± 0.30 | 97.58 ± 0.31 | 0.28 ± 0.01 | 97.52 ± 0.29 | 97.52 ± 0.29 | 97.54 ± 0.28 | 97.52 ± 0.29 | 97.50 ± 0.30 | 0.44 ± 0.01 |
3 × [Bi]LSTM | 2 × [Bi]LSTM | 2 × [Bi]LSTM | 97.57 ± 0.37 | 97.57 ± 0.37 | 97.60 ± 0.38 | 97.57 ± 0.37 | 97.56 ± 0.36 | 0.30 ± 0.01 | 97.38 ± 0.47 | 97.38 ± 0.47 | 97.41 ± 0.43 | 97.38 ± 0.47 | 97.41 ± 0.46 | 0.47 ± 0.01 |
3 × [Bi]LSTM | 2 × [Bi]LSTM | 3 × [Bi]LSTM | 97.63 ± 0.38 | 97.63 ± 0.38 | 97.65 ± 0.35 | 97.63 ± 0.38 | 97.63 ± 0.39 | 0.31 ± 0.01 | 97.56 ± 0.34 | 97.56 ± 0.34 | 97.57 ± 0.33 | 97.56 ± 0.34 | 97.57 ± 0.34 | 0.50 ± 0.01 |
3 × [Bi]LSTM | 3 × [Bi]LSTM | 1 × [Bi]LSTM | 97.36 ± 0.48 | 97.36 ± 0.48 | 97.41 ± 0.47 | 97.36 ± 0.48 | 97.35 ± 0.46 | 0.33 ± 0.01 | 97.30 ± 0.48 | 97.30 ± 0.48 | 97.41 ± 0.42 | 97.30 ± 0.48 | 97.24 ± 0.50 | 0.52 ± 0.01 |
3 × [Bi]LSTM | 3 × [Bi]LSTM | 2 × [Bi]LSTM | 97.62 ± 0.48 | 97.62 ± 0.48 | 97.65 ± 0.45 | 97.62 ± 0.48 | 97.61 ± 0.49 | 0.35 ± 0.01 | 97.52 ± 0.20 | 97.52 ± 0.20 | 97.54 ± 0.22 | 97.52 ± 0.20 | 97.50 ± 0.20 | 0.55 ± 0.01 |
3 × [Bi]LSTM | 3 × [Bi]LSTM | 3 × [Bi]LSTM | 97.46 ± 0.35 | 97.46 ± 0.35 | 97.47 ± 0.35 | 97.46 ± 0.35 | 97.46 ± 0.36 | 0.37 ± 0.01 | 97.50 ± 0.39 | 97.50 ± 0.39 | 97.51 ± 0.38 | 97.50 ± 0.39 | 97.51 ± 0.37 | 0.58 ± 0.01 |
128 Units/Layer | MisRoBÆRTa with LSTM Cells | MisRoBÆRTa with BiLSTM Cells | ||||||||||||
BART Branch | RoBERTa Branch | Ensemble Branch | Accuracy | Precision Micro | Precision Macro | Recall Micro | Recall Macro | Execution Time (Hours) | Accuracy | Precision Micro | Precision Macro | Recall Micro | Recall Macro | Execution Time (Hours) |
1 × [Bi]LSTM | 1 × [Bi]LSTM | 1 × [Bi]LSTM | 97.17 ± 0.54 | 97.17 ± 0.54 | 97.22 ± 0.50 | 97.17 ± 0.54 | 97.17 ± 0.54 | 0.01 ± 0.00 | 97.23 ± 0.52 | 97.23 ± 0.52 | 97.26 ± 0.49 | 97.23 ± 0.52 | 97.25 ± 0.49 | 0.01 ± 0.00 |
1 × [Bi]LSTM | 1 × [Bi]LSTM | 2 × [Bi]LSTM | 93.98 ± 0.97 | 93.98 ± 0.97 | 97.06 ± 0.84 | 93.98 ± 0.97 | 97.01 ± 0.92 | 0.02 ± 0.00 | 97.32 ± 0.27 | 97.32 ± 0.27 | 97.36 ± 0.24 | 97.32 ± 0.27 | 97.31 ± 0.29 | 0.03 ± 0.00 |
1 × [Bi]LSTM | 1 × [Bi]LSTM | 3 × [Bi]LSTM | 97.04 ± 0.55 | 97.04 ± 0.55 | 97.10 ± 0.47 | 97.04 ± 0.55 | 97.04 ± 0.59 | 0.03 ± 0.00 | 97.19 ± 0.32 | 97.19 ± 0.32 | 97.21 ± 0.30 | 97.19 ± 0.32 | 97.20 ± 0.32 | 0.04 ± 0.00 |
1 × [Bi]LSTM | 2 × [Bi]LSTM | 1 × [Bi]LSTM | 97.18 ± 0.61 | 97.18 ± 0.61 | 97.25 ± 0.60 | 97.18 ± 0.61 | 97.18 ± 0.58 | 0.03 ± 0.00 | 97.33 ± 0.42 | 97.33 ± 0.42 | 97.38 ± 0.44 | 97.33 ± 0.42 | 97.33 ± 0.42 | 0.06 ± 0.00 |
1 × [Bi]LSTM | 2 × [Bi]LSTM | 2 × [Bi]LSTM | 97.13 ± 0.81 | 97.13 ± 0.81 | 97.19 ± 0.76 | 97.13 ± 0.81 | 97.14 ± 0.79 | 0.05 ± 0.00 | 97.37 ± 0.26 | 97.37 ± 0.26 | 97.37 ± 0.25 | 97.37 ± 0.26 | 97.38 ± 0.27 | 0.08 ± 0.00 |
1 × [Bi]LSTM | 2 × [Bi]LSTM | 3 × [Bi]LSTM | 97.34 ± 0.48 | 97.34 ± 0.48 | 97.37 ± 0.47 | 97.34 ± 0.48 | 97.34 ± 0.47 | 0.06 ± 0.00 | 97.03 ± 0.86 | 97.03 ± 0.86 | 97.11 ± 0.72 | 97.03 ± 0.86 | 97.06 ± 0.80 | 0.10 ± 0.00 |
1 × [Bi]LSTM | 3 × [Bi]LSTM | 1 × [Bi]LSTM | 97.38 ± 0.65 | 97.38 ± 0.65 | 97.40 ± 0.64 | 97.38 ± 0.65 | 97.37 ± 0.64 | 0.07 ± 0.00 | 97.04 ± 0.53 | 97.04 ± 0.53 | 97.14 ± 0.44 | 97.04 ± 0.53 | 97.03 ± 0.54 | 0.12 ± 0.01 |
1 × [Bi]LSTM | 3 × [Bi]LSTM | 2 × [Bi]LSTM | 97.23 ± 0.41 | 97.23 ± 0.41 | 97.25 ± 0.38 | 97.23 ± 0.41 | 97.22 ± 0.43 | 0.08 ± 0.00 | 97.15 ± 0.42 | 97.15 ± 0.42 | 97.17 ± 0.40 | 97.15 ± 0.42 | 97.17 ± 0.42 | 0.14 ± 0.01 |
1 × [Bi]LSTM | 3 × [Bi]LSTM | 3 × [Bi]LSTM | 97.43 ± 0.40 | 97.43 ± 0.40 | 97.45 ± 0.41 | 97.43 ± 0.40 | 97.42 ± 0.38 | 0.09 ± 0.00 | 97.24 ± 0.28 | 97.24 ± 0.28 | 97.24 ± 0.28 | 97.24 ± 0.28 | 97.28 ± 0.28 | 0.16 ± 0.01 |
2 × [Bi]LSTM | 1 × [Bi]LSTM | 1 × [Bi]LSTM | 97.55 ± 0.38 | 97.55 ± 0.38 | 97.63 ± 0.37 | 97.55 ± 0.38 | 97.51 ± 0.38 | 0.10 ± 0.00 | 97.58 ± 0.73 | 97.58 ± 0.73 | 97.55 ± 0.65 | 97.58 ± 0.73 | 97.58 ± 0.70 | 0.18 ± 0.01 |
2 × [Bi]LSTM | 1 × [Bi]LSTM | 2 × [Bi]LSTM | 97.43 ± 0.35 | 97.53 ± 0.35 | 97.48 ± 0.36 | 97.43 ± 0.35 | 97.42 ± 0.34 | 0.11 ± 0.00 | 97.18 ± 0.46 | 97.18 ± 0.46 | 97.24 ± 0.40 | 97.18 ± 0.46 | 97.16 ± 0.48 | 0.19 ± 0.01 |
2 × [Bi]LSTM | 1 × [Bi]LSTM | 3 × [Bi]LSTM | 97.35 ± 0.47 | 97.35 ± 0.47 | 97.40 ± 0.46 | 97.35 ± 0.47 | 97.34 ± 0.45 | 0.13 ± 0.00 | 97.36 ± 0.27 | 97.36 ± 0.27 | 97.37 ± 0.28 | 97.36 ± 0.27 | 97.35 ± 0.26 | 0.22 ± 0.01 |
2 × [Bi]LSTM | 2 × [Bi]LSTM | 1 × [Bi]LSTM | 97.00 ± 0.97 | 97.00 ± 0.97 | 97.16 ± 0.79 | 97.00 ± 0.97 | 93.97 ± 0.97 | 0.14 ± 0.00 | 97.04 ± 0.54 | 97.04 ± 0.54 | 97.13 ± 0.44 | 97.04 ± 0.54 | 97.05 ± 0.51 | 0.23 ± 0.01 |
2 × [Bi]LSTM | 2 × [Bi]LSTM | 2 × [Bi]LSTM | 97.36 ± 0.57 | 97.36 ± 0.57 | 97.39 ± 0.54 | 97.36 ± 0.57 | 97.36 ± 0.57 | 0.15 ± 0.00 | 97.25 ± 0.39 | 97.25 ± 0.39 | 97.29 ± 0.35 | 97.25 ± 0.39 | 97.26 ± 0.39 | 0.25 ± 0.01 |
2 × [Bi]LSTM | 2 × [Bi]LSTM | 3 × [Bi]LSTM | 97.23 ± 0.47 | 97.23 ± 0.47 | 97.29 ± 0.43 | 97.23 ± 0.47 | 97.23 ± 0.50 | 0.16 ± 0.00 | 97.23 ± 0.21 | 97.23 ± 0.21 | 97.26 ± 0.21 | 97.23 ± 0.21 | 97.24 ± 0.18 | 0.28 ± 0.01 |
2 × [Bi]LSTM | 3 × [Bi]LSTM | 1 × [Bi]LSTM | 97.31 ± 0.60 | 97.31 ± 0.60 | 97.37 ± 0.58 | 97.31 ± 0.60 | 97.30 ± 0.59 | 0.18 ± 0.01 | 97.24 ± 0.44 | 97.24 ± 0.44 | 97.32 ± 0.37 | 97.24 ± 0.44 | 97.21 ± 0.47 | 0.30 ± 0.01 |
2 × [Bi]LSTM | 3 × [Bi]LSTM | 2 × [Bi]LSTM | 97.47 ± 0.51 | 97.47 ± 0.51 | 97.53 ± 0.49 | 97.47 ± 0.51 | 97.45 ± 0.50 | 0.19 ± 0.01 | 97.26 ± 0.17 | 97.26 ± 0.17 | 97.31 ± 0.17 | 97.26 ± 0.17 | 97.24 ± 0.16 | 0.32 ± 0.01 |
2 × [Bi]LSTM | 3 × [Bi]LSTM | 3 × [Bi]LSTM | 97.26 ± 0.43 | 97.26 ± 0.43 | 97.28 ± 0.41 | 97.26 ± 0.43 | 97.29 ± 0.43 | 0.21 ± 0.01 | 97.23 ± 0.31 | 97.23 ± 0.31 | 97.28 ± 0.25 | 97.23 ± 0.31 | 97.23 ± 0.35 | 0.35 ± 0.01 |
3 × [Bi]LSTM | 1 × [Bi]LSTM | 1 × [Bi]LSTM | 97.40 ± 0.58 | 97.40 ± 0.58 | 97.45 ± 0.55 | 97.40 ± 0.58 | 97.39 ± 0.57 | 0.22 ± 0.01 | 97.40 ± 0.35 | 97.40 ± 0.35 | 97.40 ± 0.34 | 97.40 ± 0.35 | 97.43 ± 0.34 | 0.37 ± 0.01 |
3 × [Bi]LSTM | 1 × [Bi]LSTM | 2 × [Bi]LSTM | 97.37 ± 0.47 | 97.37 ± 0.47 | 97.42 ± 0.48 | 97.37 ± 0.47 | 97.38 ± 0.45 | 0.23 ± 0.01 | 97.33 ± 0.34 | 97.33 ± 0.34 | 97.38 ± 0.32 | 97.33 ± 0.34 | 97.32 ± 0.32 | 0.39 ± 0.01 |
3 × [Bi]LSTM | 1 × [Bi]LSTM | 3 × [Bi]LSTM | 97.41 ± 0.61 | 97.41 ± 0.61 | 97.44 ± 0.61 | 97.41 ± 0.61 | 97.41 ± 0.60 | 0.25 ± 0.01 | 97.35 ± 0.30 | 97.35 ± 0.30 | 97.37 ± 0.29 | 97.35 ± 0.30 | 97.35 ± 0.31 | 0.41 ± 0.01 |
3 × [Bi]LSTM | 2 × [Bi]LSTM | 1 × [Bi]LSTM | 97.35 ± 0.41 | 97.35 ± 0.41 | 97.40 ± 0.40 | 97.35 ± 0.41 | 97.33 ± 0.40 | 0.26 ± 0.01 | 97.06 ± 0.79 | 97.06 ± 0.79 | 97.19 ± 0.57 | 97.06 ± 0.79 | 97.03 ± 0.85 | 0.43 ± 0.01 |
3 × [Bi]LSTM | 2 × [Bi]LSTM | 2 × [Bi]LSTM | 97.25 ± 0.58 | 97.25 ± 0.58 | 97.29 ± 0.57 | 97.25 ± 0.58 | 97.27 ± 0.58 | 0.27 ± 0.01 | 97.35 ± 0.41 | 97.35 ± 0.41 | 97.37 ± 0.38 | 97.35 ± 0.41 | 97.33 ± 0.43 | 0.46 ± 0.01 |
3 × [Bi]LSTM | 2 × [Bi]LSTM | 3 × [Bi]LSTM | 97.24 ± 0.60 | 97.24 ± 0.60 | 97.29 ± 0.59 | 97.24 ± 0.60 | 97.26 ± 0.56 | 0.29 ± 0.01 | 97.13 ± 0.32 | 97.13 ± 0.32 | 97.15 ± 0.30 | 97.13 ± 0.32 | 97.15 ± 0.31 | 0.48 ± 0.01 |
3 × [Bi]LSTM | 3 × [Bi]LSTM | 1 × [Bi]LSTM | 97.47 ± 0.59 | 97.47 ± 0.59 | 97.52 ± 0.57 | 97.47 ± 0.59 | 97.47 ± 0.58 | 0.30 ± 0.01 | 97.34 ± 0.29 | 97.34 ± 0.29 | 97.35 ± 0.29 | 97.34 ± 0.29 | 97.36 ± 0.28 | 0.51 ± 0.02 |
3 × [Bi]LSTM | 3 × [Bi]LSTM | 2 × [Bi]LSTM | 97.45 ± 0.53 | 97.45 ± 0.53 | 97.49 ± 0.52 | 97.45 ± 0.53 | 97.46 ± 0.51 | 0.32 ± 0.01 | 97.23 ± 0.44 | 97.23 ± 0.44 | 97.24 ± 0.43 | 97.23 ± 0.44 | 97.24 ± 0.41 | 0.53 ± 0.02 |
3 × [Bi]LSTM | 3 × [Bi]LSTM | 3 × [Bi]LSTM | 97.48 ± 0.47 | 97.48 ± 0.47 | 97.49 ± 0.47 | 97.48 ± 0.47 | 97.48 ± 0.48 | 0.34 ± 0.01 | 97.23 ± 0.50 | 97.23 ± 0.50 | 97.27 ± 0.45 | 97.23 ± 0.50 | 97.23 ± 0.48 | 0.57 ± 0.02 |
32 Units/Layer | MisRoBÆRTa with LSTM Cells | MisRoBÆRTa with BiLSTM Cells | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
BART Branch | RoBERTa Branch | Ensemble Branch | Accuracy | Precision Micro | Precision Macro | Recall Micro | Recall Macro | Execution Time (Hours) | Accuracy | Precision Micro | Precision Macro | Recall Micro | Recall Macro | Execution Time (Hours) |
1 × [Bi]LSTM | 1 × [Bi]LSTM | 1 × [Bi]LSTM | 91.38 ± 0.24 | 91.38 ± 0.24 | 91.55 ± 0.26 | 91.38 ± 0.24 | 91.38 ± 0.24 | 0.03 ± 0.00 | 91.91 ± 0.30 | 91.91 ± 0.30 | 92.16 ± 0.20 | 91.91 ± 0.30 | 91.91 ± 0.30 | 0.03 ± 0.00 |
1 × [Bi]LSTM | 1 × [Bi]LSTM | 2 × [Bi]LSTM | 91.35 ± 0.36 | 91.35 ± 0.36 | 91.57 ± 0.32 | 91.35 ± 0.36 | 91.35 ± 0.36 | 0.06 ± 0.00 | 91.90 ± 0.21 | 91.90 ± 0.21 | 92.08 ± 0.20 | 91.90 ± 0.21 | 91.90 ± 0.21 | 0.06 ± 0.00 |
1 × [Bi]LSTM | 1 × [Bi]LSTM | 3 × [Bi]LSTM | 90.87 ± 0.34 | 90.87 ± 0.34 | 91.16 ± 0.35 | 90.87 ± 0.34 | 90.87 ± 0.34 | 0.11 ± 0.00 | 91.61 ± 0.21 | 91.61 ± 0.21 | 91.79 ± 0.17 | 91.61 ± 0.21 | 91.61 ± 0.21 | 0.10 ± 0.01 |
1 × [Bi]LSTM | 2 × [Bi]LSTM | 1 × [Bi]LSTM | 90.90 ± 0.35 | 90.90 ± 0.35 | 91.13 ± 0.30 | 90.90 ± 0.35 | 90.90 ± 0.35 | 0.14 ± 0.01 | 91.62 ± 0.29 | 91.62 ± 0.29 | 91.84 ± 0.28 | 91.62 ± 0.29 | 91.62 ± 0.29 | 0.13 ± 0.01 |
1 × [Bi]LSTM | 2 × [Bi]LSTM | 2 × [Bi]LSTM | 90.87 ± 0.25 | 90.87 ± 0.25 | 91.13 ± 0.25 | 90.87 ± 0.25 | 90.87 ± 0.25 | 0.18 ± 0.01 | 91.60 ± 0.16 | 91.60 ± 0.16 | 91.75 ± 0.17 | 91.60 ± 0.16 | 91.60 ± 0.16 | 0.17 ± 0.01 |
1 × [Bi]LSTM | 2 × [Bi]LSTM | 3 × [Bi]LSTM | 90.60 ± 0.35 | 90.60 ± 0.35 | 90.79 ± 0.30 | 90.60 ± 0.35 | 90.60 ± 0.35 | 0.23 ± 0.01 | 91.45 ± 0.21 | 91.45 ± 0.21 | 91.60 ± 0.24 | 91.45 ± 0.21 | 91.45 ± 0.21 | 0.21 ± 0.01 |
1 × [Bi]LSTM | 3 × [Bi]LSTM | 1 × [Bi]LSTM | 90.64 ± 0.43 | 90.64 ± 0.43 | 90.77 ± 0.43 | 90.64 ± 0.43 | 90.64 ± 0.43 | 0.26 ± 0.01 | 91.51 ± 0.22 | 91.51 ± 0.22 | 91.70 ± 0.24 | 91.51 ± 0.22 | 91.51 ± 0.22 | 0.25 ± 0.01 |
1 × [Bi]LSTM | 3 × [Bi]LSTM | 2 × [Bi]LSTM | 90.83 ± 0.21 | 90.83 ± 0.21 | 91.09 ± 0.18 | 90.83 ± 0.21 | 90.83 ± 0.21 | 0.31 ± 0.01 | 91.60 ± 0.21 | 91.60 ± 0.21 | 91.78 ± 0.24 | 91.60 ± 0.21 | 91.59 ± 0.21 | 0.29 ± 0.01 |
1 × [Bi]LSTM | 3 × [Bi]LSTM | 3 × [Bi]LSTM | 90.56 ± 0.34 | 90.56 ± 0.34 | 90.79 ± 0.32 | 90.56 ± 0.34 | 90.56 ± 0.34 | 0.36 ± 0.02 | 91.32 ± 0.18 | 91.32 ± 0.18 | 91.53 ± 0.16 | 91.32 ± 0.18 | 91.32 ± 0.18 | 0.35 ± 0.02 |
2 × [Bi]LSTM | 1 × [Bi]LSTM | 1 × [Bi]LSTM | 91.98± 0.19 | 91.98 ± 0.19 | 92.17± 0.15 | 91.98 ± 0.19 | 91.98 ± 0.19 | 0.39 ± 0.02 | 92.08± 0.18 | 92.08 ± 0.18 | 92.25 ± 0.19 | 92.08 ± 0.18 | 92.08 ± 0.18 | 0.38 ± 0.02 |
2 × [Bi]LSTM | 1 × [Bi]LSTM | 2 × [Bi]LSTM | 91.60 ± 0.19 | 91.60 ± 0.19 | 91.78 ± 0.25 | 91.60 ± 0.19 | 91.60 ± 0.19 | 0.43 ± 0.02 | 91.80 ± 0.17 | 91.80 ± 0.17 | 91.94 ± 0.18 | 91.80 ± 0.17 | 91.80 ± 0.17 | 0.42 ± 0.02 |
2 × [Bi]LSTM | 1 × [Bi]LSTM | 3 × [Bi]LSTM | 91.19 ± 0.28 | 91.19 ± 0.28 | 91.37 ± 0.27 | 91.19 ± 0.28 | 91.19 ± 0.28 | 0.48 ± 0.02 | 91.53 ± 0.30 | 91.53 ± 0.30 | 91.67 ± 0.30 | 91.53 ± 0.30 | 91.53 ± 0.30 | 0.47 ± 0.02 |
2 × [Bi]LSTM | 2 × [Bi]LSTM | 1 × [Bi]LSTM | 91.73 ± 0.12 | 91.73 ± 0.12 | 91.90 ± 0.09 | 91.73 ± 0.12 | 91.73 ± 0.12 | 0.51 ± 0.02 | 91.88 ± 0.17 | 91.88 ± 0.17 | 92.07 ± 0.20 | 91.88 ± 0.17 | 91.88 ± 0.17 | 0.51 ± 0.02 |
2 × [Bi]LSTM | 2 × [Bi]LSTM | 2 × [Bi]LSTM | 91.35 ± 0.24 | 91.35 ± 0.24 | 91.51 ± 0.22 | 91.35 ± 0.24 | 91.35 ± 0.24 | 0.56 ± 0.02 | 91.71 ± 0.20 | 91.71 ± 0.20 | 91.86 ± 0.19 | 91.71 ± 0.20 | 91.71 ± 0.20 | 0.56 ± 0.02 |
2 × [Bi]LSTM | 2 × [Bi]LSTM | 3 × [Bi]LSTM | 90.92 ± 0.37 | 90.92 ± 0.37 | 91.15 ± 0.34 | 90.92 ± 0.37 | 90.92 ± 0.37 | 0.61 ± 0.02 | 91.20 ± 0.25 | 91.20 ± 0.25 | 91.32 ± 0.24 | 91.20 ± 0.25 | 91.20 ± 0.25 | 0.62 ± 0.03 |
2 × [Bi]LSTM | 3 × [Bi]LSTM | 1 × [Bi]LSTM | 91.55 ± 0.18 | 91.55 ± 0.18 | 91.68 ± 0.21 | 91.55 ± 0.18 | 91.55 ± 0.18 | 0.64 ± 0.02 | 91.56 ± 0.21 | 91.56 ± 0.21 | 91.75 ± 0.19 | 91.56 ± 0.21 | 91.56 ± 0.21 | 0.67 ± 0.03 |
2 × [Bi]LSTM | 3 × [Bi]LSTM | 2 × [Bi]LSTM | 91.10 ± 0.46 | 91.10 ± 0.46 | 91.23 ± 0.44 | 91.10 ± 0.46 | 91.10 ± 0.46 | 0.69 ± 0.03 | 91.49 ± 0.21 | 91.49 ± 0.21 | 91.58 ± 0.24 | 91.49 ± 0.21 | 91.49 ± 0.21 | 0.72 ± 0.03 |
2 × [Bi]LSTM | 3 × [Bi]LSTM | 3 × [Bi]LSTM | 90.69 ± 0.37 | 90.69 ± 0.37 | 90.92 ± 0.34 | 90.69 ± 0.37 | 90.69 ± 0.37 | 0.75 ± 0.03 | 91.19 ± 0.25 | 91.19 ± 0.25 | 91.35 ± 0.32 | 91.19 ± 0.25 | 91.19 ± 0.25 | 0.79 ± 0.03 |
3 × [Bi]LSTM | 1 × [Bi]LSTM | 1 × [Bi]LSTM | 91.74 ± 0.23 | 91.74 ± 0.23 | 91.92 ± 0.22 | 91.74 ± 0.23 | 91.74 ± 0.23 | 0.80 ± 0.03 | 91.92 ± 0.22 | 91.92 ± 0.22 | 92.06 ± 0.23 | 91.92 ± 0.22 | 91.92 ± 0.22 | 0.84 ± 0.04 |
3 × [Bi]LSTM | 1 × [Bi]LSTM | 2 × [Bi]LSTM | 91.39 ± 0.20 | 91.39 ± 0.20 | 91.58 ± 0.17 | 91.39 ± 0.20 | 91.39 ± 0.20 | 0.85 ± 0.03 | 91.72 ± 0.32 | 91.72 ± 0.32 | 91.91 ± 0.31 | 91.72 ± 0.32 | 91.72 ± 0.32 | 0.92 ± 0.04 |
3 × [Bi]LSTM | 1 × [Bi]LSTM | 3 × [Bi]LSTM | 90.86 ± 0.20 | 90.86 ± 0.20 | 91.06 ± 0.16 | 90.86 ± 0.20 | 90.86 ± 0.20 | 0.90 ± 0.04 | 91.48 ± 0.26 | 91.48 ± 0.26 | 91.64 ± 0.23 | 91.48 ± 0.26 | 91.48 ± 0.26 | 0.97 ± 0.04 |
3 × [Bi]LSTM | 2 × [Bi]LSTM | 1 × [Bi]LSTM | 91.62 ± 0.19 | 91.62 ± 0.19 | 91.82 ± 0.17 | 91.62 ± 0.19 | 91.62 ± 0.19 | 0.95 ± 0.04 | 91.84 ± 0.25 | 91.84 ± 0.25 | 92.00 ± 0.28 | 91.84 ± 0.25 | 91.84 ± 0.25 | 1.02 ± 0.04 |
3 × [Bi]LSTM | 2 × [Bi]LSTM | 2 × [Bi]LSTM | 91.16 ± 0.26 | 91.16 ± 0.26 | 91.31 ± 0.26 | 91.16 ± 0.26 | 91.16 ± 0.26 | 1.01 ± 0.04 | 91.51 ± 0.19 | 91.51 ± 0.19 | 91.66 ± 0.17 | 91.51 ± 0.19 | 91.51 ± 0.19 | 1.09 ± 0.05 |
3 × [Bi]LSTM | 2 × [Bi]LSTM | 3 × [Bi]LSTM | 90.36 ± 0.00 | 90.36 ± 0.00 | 90.79 ± 0.00 | 90.36 ± 0.00 | 90.36 ± 0.00 | 1.03 ± 0.04 | 91.25 ± 0.27 | 91.25 ± 0.27 | 91.49 ± 0.27 | 91.25 ± 0.27 | 91.25 ± 0.27 | 1.17 ± 0.06 |
3 × [Bi]LSTM | 3 × [Bi]LSTM | 1 × [Bi]LSTM | 91.23 ± 0.28 | 91.23 ± 0.28 | 91.37 ± 0.26 | 91.23 ± 0.28 | 91.23 ± 0.28 | 1.09 ± 0.04 | 91.50 ± 0.23 | 91.50 ± 0.23 | 91.63 ± 0.21 | 91.50 ± 0.23 | 91.50 ± 0.23 | 1.23 ± 0.06 |
3 × [Bi]LSTM | 3 × [Bi]LSTM | 2 × [Bi]LSTM | 90.77 ± 0.28 | 90.77 ± 0.28 | 90.92 ± 0.30 | 90.77 ± 0.28 | 90.77 ± 0.28 | 1.15 ± 0.04 | 91.32 ± 0.19 | 91.32 ± 0.19 | 91.47 ± 0.18 | 91.32 ± 0.19 | 91.32 ± 0.19 | 1.30 ± 0.06 |
3 × [Bi]LSTM | 3 × [Bi]LSTM | 3 × [Bi]LSTM | 88.50 ± 0.36 | 88.50 ± 0.36 | 88.87 ± 0.24 | 88.50 ± 0.36 | 88.50 ± 0.36 | 1.18 ± 0.05 | 91.22 ± 0.25 | 91.22 ± 0.25 | 91.41 ± 0.20 | 91.22 ± 0.25 | 91.22 ± 0.25 | 1.36 ± 0.08 |
64 Units/Layer | MisRoBÆRTa with LSTM Cells | MisRoBÆRTa with BiLSTM Cells | ||||||||||||
BART Branch | RoBERTa Branch | Ensemble Branch | Accuracy | Precision Micro | Precision Macro | Recall Micro | Recall Macro | Execution Time (Hours) | Accuracy | Precision Micro | Precision Macro | Recall Micro | Recall Macro | Execution Time (Hours) |
1 × [Bi]LSTM | 1 × [Bi]LSTM | 1 × [Bi]LSTM | 92.05 ± 0.18 | 92.05 ± 0.18 | 92.23 ± 0.19 | 92.05 ± 0.18 | 92.05 ± 0.18 | 0.02 ± 0.00 | 92.26 ± 0.23 | 92.26 ± 0.23 | 92.40 ± 0.21 | 92.26 ± 0.23 | 92.26 ± 0.23 | 0.02 ± 0.00 |
1 × [Bi]LSTM | 1 × [Bi]LSTM | 2 × [Bi]LSTM | 91.77 ± 0.29 | 91.77 ± 0.29 | 91.96 ± 0.25 | 91.77 ± 0.29 | 91.77 ± 0.29 | 0.04 ± 0.00 | 92.12 ± 0.19 | 92.12 ± 0.19 | 92.22 ± 0.19 | 92.12 ± 0.19 | 92.12 ± 0.19 | 0.05 ± 0.00 |
1 × [Bi]LSTM | 1 × [Bi]LSTM | 3 × [Bi]LSTM | 91.72 ± 0.21 | 91.72 ± 0.21 | 91.90 ± 0.21 | 91.72 ± 0.21 | 91.72 ± 0.21 | 0.07 ± 0.00 | 91.90 ± 0.17 | 91.90 ± 0.17 | 92.07 ± 0.18 | 91.90 ± 0.17 | 91.90 ± 0.17 | 0.08 ± 0.00 |
1 × [Bi]LSTM | 2 × [Bi]LSTM | 1 × [Bi]LSTM | 91.72 ± 0.28 | 91.72 ± 0.28 | 91.92 ± 0.29 | 91.72 ± 0.28 | 91.72 ± 0.28 | 0.09 ± 0.00 | 92.02 ± 0.20 | 92.02 ± 0.20 | 92.16 ± 0.18 | 92.02 ± 0.20 | 92.02 ± 0.20 | 0.11 ± 0.01 |
1 × [Bi]LSTM | 2 × [Bi]LSTM | 2 × [Bi]LSTM | 91.63 ± 0.13 | 91.63 ± 0.13 | 91.83 ± 0.13 | 91.63 ± 0.13 | 91.63 ± 0.13 | 0.11 ± 0.00 | 91.92 ± 0.28 | 91.92 ± 0.28 | 92.14 ± 0.21 | 91.92 ± 0.28 | 91.92 ± 0.28 | 0.14 ± 0.01 |
1 × [Bi]LSTM | 2 × [Bi]LSTM | 3 × [Bi]LSTM | 91.48 ± 0.35 | 91.48 ± 0.35 | 91.73 ± 0.28 | 91.48 ± 0.35 | 91.48 ± 0.35 | 0.15 ± 0.01 | 91.97 ± 0.15 | 91.97 ± 0.15 | 92.06 ± 0.15 | 91.97 ± 0.15 | 91.97 ± 0.15 | 0.18 ± 0.01 |
1 × [Bi]LSTM | 3 × [Bi]LSTM | 1 × [Bi]LSTM | 91.57 ± 0.23 | 91.57 ± 0.23 | 91.78 ± 0.20 | 91.57 ± 0.23 | 91.57 ± 0.23 | 0.17 ± 0.01 | 92.01 ± 0.16 | 92.01 ± 0.16 | 92.16 ± 0.19 | 92.01 ± 0.16 | 92.01 ± 0.16 | 0.21 ± 0.01 |
1 × [Bi]LSTM | 3 × [Bi]LSTM | 2 × [Bi]LSTM | 91.66 ± 0.16 | 91.66 ± 0.16 | 91.79 ± 0.18 | 91.66 ± 0.16 | 91.66 ± 0.16 | 0.19 ± 0.01 | 91.90 ± 0.21 | 91.90 ± 0.21 | 92.08 ± 0.22 | 91.90 ± 0.21 | 91.90 ± 0.21 | 0.25 ± 0.01 |
1 × [Bi]LSTM | 3 × [Bi]LSTM | 3 × [Bi]LSTM | 91.37 ± 0.26 | 91.37 ± 0.26 | 91.55 ± 0.27 | 91.37 ± 0.26 | 91.37 ± 0.26 | 0.23 ± 0.01 | 91.64 ± 0.22 | 91.64 ± 0.22 | 91.77 ± 0.24 | 91.64 ± 0.22 | 91.64 ± 0.22 | 0.29 ± 0.01 |
2 × [Bi]LSTM | 1 × [Bi]LSTM | 1 × [Bi]LSTM | 92.30± 0.30 | 92.30 ± 0.30 | 92.47 ± 0.24 | 92.30 ± 0.30 | 92.30 ± 0.30 | 0.25 ± 0.01 | 92.41± 0.18 | 92.41 ± 0.18 | 92.57 ± 0.13 | 92.41 ± 0.18 | 92.41 ± 0.18 | 0.32 ± 0.01 |
2 × [Bi]LSTM | 1 × [Bi]LSTM | 2 × [Bi]LSTM | 92.00 ± 0.35 | 92.00 ± 0.35 | 92.14 ± 0.32 | 92.00 ± 0.35 | 92.00 ± 0.35 | 0.28 ± 0.01 | 92.22 ± 0.17 | 92.22 ± 0.17 | 92.40 ± 0.16 | 92.22 ± 0.17 | 92.22 ± 0.17 | 0.35 ± 0.01 |
2 × [Bi]LSTM | 1 × [Bi]LSTM | 3 × [Bi]LSTM | 91.94 ± 0.29 | 91.94 ± 0.29 | 92.06 ± 0.31 | 91.94 ± 0.29 | 91.94 ± 0.29 | 0.31 ± 0.01 | 92.12 ± 0.27 | 92.12 ± 0.27 | 92.24 ± 0.25 | 92.12 ± 0.27 | 92.12 ± 0.27 | 0.40 ± 0.01 |
2 × [Bi]LSTM | 2 × [Bi]LSTM | 1 × [Bi]LSTM | 92.06 ± 0.21 | 92.06 ± 0.21 | 92.18 ± 0.23 | 92.06 ± 0.21 | 92.06 ± 0.21 | 0.33 ± 0.01 | 92.32 ± 0.23 | 92.32 ± 0.23 | 92.49 ± 0.24 | 92.32 ± 0.23 | 92.32 ± 0.23 | 0.43 ± 0.02 |
2 × [Bi]LSTM | 2 × [Bi]LSTM | 2 × [Bi]LSTM | 91.83 ± 0.25 | 91.83 ± 0.25 | 91.99 ± 0.25 | 91.83 ± 0.25 | 91.83 ± 0.25 | 0.36 ± 0.01 | 92.16 ± 0.25 | 92.16 ± 0.25 | 92.31 ± 0.15 | 92.16 ± 0.25 | 92.16 ± 0.25 | 0.47 ± 0.02 |
2 × [Bi]LSTM | 2 × [Bi]LSTM | 3 × [Bi]LSTM | 91.61 ± 0.19 | 91.61 ± 0.19 | 91.77 ± 0.21 | 91.61 ± 0.19 | 91.61 ± 0.19 | 0.40 ± 0.01 | 91.74 ± 0.25 | 91.74 ± 0.25 | 91.91 ± 0.26 | 91.74 ± 0.25 | 91.74 ± 0.25 | 0.52 ± 0.02 |
2 × [Bi]LSTM | 3 × [Bi]LSTM | 1 × [Bi]LSTM | 91.97 ± 0.27 | 91.97 ± 0.27 | 92.10 ± 0.24 | 91.97 ± 0.27 | 91.97 ± 0.27 | 0.43 ± 0.01 | 92.01 ± 0.25 | 92.01 ± 0.25 | 92.19 ± 0.16 | 92.01 ± 0.25 | 92.01 ± 0.25 | 0.56 ± 0.02 |
2 × [Bi]LSTM | 3 × [Bi]LSTM | 2 × [Bi]LSTM | 91.81 ± 0.31 | 91.81 ± 0.31 | 91.97 ± 0.31 | 91.81 ± 0.31 | 91.81 ± 0.31 | 0.46 ± 0.02 | 91.81 ± 0.17 | 91.81 ± 0.17 | 91.96 ± 0.23 | 91.81 ± 0.17 | 91.81 ± 0.17 | 0.60 ± 0.02 |
2 × [Bi]LSTM | 3 × [Bi]LSTM | 3 × [Bi]LSTM | 91.47 ± 0.26 | 91.47 ± 0.26 | 91.64 ± 0.24 | 91.47 ± 0.26 | 91.47 ± 0.26 | 0.50 ± 0.02 | 91.51 ± 0.43 | 91.51 ± 0.43 | 91.70 ± 0.41 | 91.51 ± 0.43 | 91.51 ± 0.43 | 0.66 ± 0.02 |
3 × [Bi]LSTM | 1 × [Bi]LSTM | 1 × [Bi]LSTM | 92.29 ± 0.26 | 92.29 ± 0.26 | 92.43 ± 0.21 | 92.29 ± 0.26 | 92.29 ± 0.26 | 0.52 ± 0.02 | 92.19 ± 0.31 | 92.19 ± 0.31 | 92.31 ± 0.34 | 92.19 ± 0.31 | 92.19 ± 0.31 | 0.70 ± 0.02 |
3 × [Bi]LSTM | 1 × [Bi]LSTM | 2 × [Bi]LSTM | 91.90 ± 0.21 | 91.90 ± 0.21 | 92.09 ± 0.16 | 91.90 ± 0.21 | 91.90 ± 0.21 | 0.56 ± 0.02 | 91.97 ± 0.21 | 91.97 ± 0.21 | 92.09 ± 0.20 | 91.97 ± 0.21 | 91.97 ± 0.21 | 0.74 ± 0.03 |
3 × [Bi]LSTM | 1 × [Bi]LSTM | 3 × [Bi]LSTM | 91.85 ± 0.28 | 91.85 ± 0.28 | 91.98 ± 0.26 | 91.85 ± 0.28 | 91.85 ± 0.28 | 0.60 ± 0.02 | 91.91 ± 0.15 | 91.91 ± 0.15 | 92.06 ± 0.17 | 91.91 ± 0.15 | 91.91 ± 0.15 | 0.80 ± 0.03 |
3 × [Bi]LSTM | 2 × [Bi]LSTM | 1 × [Bi]LSTM | 92.08 ± 0.19 | 92.08 ± 0.19 | 92.20 ± 0.19 | 92.08 ± 0.19 | 92.08 ± 0.19 | 0.63 ± 0.02 | 92.16 ± 0.23 | 92.16 ± 0.23 | 92.31 ± 0.19 | 92.16 ± 0.23 | 92.16 ± 0.23 | 0.84 ± 0.03 |
3 × [Bi]LSTM | 2 × [Bi]LSTM | 2 × [Bi]LSTM | 91.73 ± 0.27 | 91.73 ± 0.27 | 91.93 ± 0.27 | 91.73 ± 0.27 | 91.73 ± 0.27 | 0.67 ± 0.02 | 91.91 ± 0.13 | 91.91 ± 0.13 | 92.03 ± 0.16 | 91.90 ± 0.13 | 91.91 ± 0.13 | 0.92 ± 0.03 |
3 × [Bi]LSTM | 2 × [Bi]LSTM | 3 × [Bi]LSTM | 91.43 ± 0.16 | 91.43 ± 0.16 | 91.63 ± 0.16 | 91.43 ± 0.16 | 91.43 ± 0.16 | 0.70 ± 0.02 | 91.59 ± 0.07 | 91.59 ± 0.07 | 91.71 ± 0.11 | 91.59 ± 0.07 | 91.59 ± 0.07 | 0.93 ± 0.04 |
3 × [Bi]LSTM | 3 × [Bi]LSTM | 1 × [Bi]LSTM | 91.92 ± 0.22 | 91.92 ± 0.22 | 92.03 ± 0.22 | 91.92 ± 0.22 | 91.92 ± 0.22 | 0.73 ± 0.02 | 91.97 ± 0.23 | 91.97 ± 0.23 | 92.13 ± 0.24 | 91.97 ± 0.23 | 91.97 ± 0.23 | 0.98 ± 0.04 |
3 × [Bi]LSTM | 3 × [Bi]LSTM | 2 × [Bi]LSTM | 91.61 ± 0.28 | 91.61 ± 0.28 | 91.77 ± 0.27 | 91.61 ± 0.28 | 91.61 ± 0.28 | 0.77 ± 0.03 | 91.80 ± 0.16 | 91.80 ± 0.16 | 91.93 ± 0.18 | 91.81 ± 0.16 | 91.80 ± 0.16 | 1.04 ± 0.04 |
3 × [Bi]LSTM | 3 × [Bi]LSTM | 3 × [Bi]LSTM | 89.81 ± 0.21 | 89.81 ± 0.21 | 89.91 ± 0.27 | 89.81 ± 0.21 | 89.81 ± 0.21 | 0.79 ± 0.03 | 85.53 ± 0.11 | 85.53 ± 0.11 | 85.64 ± 0.17 | 85.53 ± 0.11 | 85.53 ± 0.11 | 1.08 ± 0.04 |
128 Units/Layer | MisRoBÆRTa with LSTM Cells | MisRoBÆRTa with BiLSTM Cells | ||||||||||||
BART Branch | RoBERTa Branch | Ensemble Branch | Accuracy | Precision Micro | Precision Macro | Recall Micro | Recall Macro | Execution Time (Hours) | Accuracy | Precision Micro | Precision Macro | Recall Micro | Recall Macro | Execution Time (Hours) |
1 × [Bi]LSTM | 1 × [Bi]LSTM | 1 × [Bi]LSTM | 92.16 ± 0.27 | 92.16 ± 0.27 | 92.35 ± 0.22 | 92.16 ± 0.27 | 92.16 ± 0.27 | 0.02 ± 0.00 | 92.20 ± 0.30 | 92.20 ± 0.30 | 92.35 ± 0.29 | 92.20 ± 0.30 | 92.20 ± 0.30 | 0.03 ± 0.00 |
1 × [Bi]LSTM | 1 × [Bi]LSTM | 2 × [Bi]LSTM | 92.11 ± 0.37 | 92.11 ± 0.37 | 92.26 ± 0.35 | 92.11 ± 0.37 | 92.11 ± 0.37 | 0.03 ± 0.00 | 92.23 ± 0.14 | 92.23 ± 0.14 | 92.37 ± 0.13 | 92.23 ± 0.14 | 92.23 ± 0.14 | 0.05 ± 0.00 |
1 × [Bi]LSTM | 1 × [Bi]LSTM | 3 × [Bi]LSTM | 91.95 ± 0.29 | 91.95 ± 0.29 | 92.14 ± 0.23 | 91.95 ± 0.29 | 91.95 ± 0.29 | 0.05 ± 0.00 | 92.04 ± 0.25 | 92.04 ± 0.25 | 92.17 ± 0.24 | 92.04 ± 0.25 | 92.04 ± 0.25 | 0.09 ± 0.00 |
1 × [Bi]LSTM | 2 × [Bi]LSTM | 1 × [Bi]LSTM | 92.02 ± 0.29 | 92.02 ± 0.29 | 92.17 ± 0.24 | 92.02 ± 0.29 | 92.02 ± 0.29 | 0.07 ± 0.00 | 92.17 ± 0.26 | 92.17 ± 0.26 | 92.36 ± 0.23 | 92.17 ± 0.26 | 92.17 ± 0.26 | 0.11 ± 0.00 |
1 × [Bi]LSTM | 2 × [Bi]LSTM | 2 × [Bi]LSTM | 91.87 ± 0.24 | 91.87 ± 0.24 | 91.98 ± 0.26 | 91.87 ± 0.24 | 91.87 ± 0.24 | 0.09 ± 0.00 | 91.94 ± 0.23 | 91.94 ± 0.23 | 92.03 ± 0.25 | 91.94 ± 0.23 | 91.94 ± 0.23 | 0.15 ± 0.00 |
1 × [Bi]LSTM | 2 × [Bi]LSTM | 3 × [Bi]LSTM | 91.56 ± 0.32 | 91.56 ± 0.32 | 91.72 ± 0.32 | 91.56 ± 0.32 | 91.56 ± 0.32 | 0.11 ± 0.00 | 91.62 ± 0.27 | 91.62 ± 0.27 | 91.76 ± 0.27 | 91.62 ± 0.27 | 91.62 ± 0.27 | 0.18 ± 0.00 |
1 × [Bi]LSTM | 3 × [Bi]LSTM | 1 × [Bi]LSTM | 91.86 ± 0.30 | 91.86 ± 0.30 | 92.01 ± 0.31 | 91.86 ± 0.30 | 91.86 ± 0.30 | 0.13 ± 0.00 | 91.97 ± 0.18 | 91.97 ± 0.18 | 92.09 ± 0.18 | 91.97 ± 0.18 | 91.97 ± 0.18 | 0.21 ± 0.00 |
1 × [Bi]LSTM | 3 × [Bi]LSTM | 2 × [Bi]LSTM | 91.58 ± 0.52 | 91.58 ± 0.52 | 91.74 ± 0.51 | 91.58 ± 0.52 | 91.58 ± 0.52 | 0.15 ± 0.00 | 91.53 ± 0.28 | 91.53 ± 0.28 | 91.65 ± 0.33 | 91.53 ± 0.28 | 91.53 ± 0.28 | 0.25 ± 0.00 |
1 × [Bi]LSTM | 3 × [Bi]LSTM | 3 × [Bi]LSTM | 91.09 ± 0.43 | 91.09 ± 0.43 | 91.21 ± 0.42 | 91.09 ± 0.43 | 91.09 ± 0.43 | 0.17 ± 0.00 | 91.07 ± 0.32 | 91.07 ± 0.32 | 91.17 ± 0.34 | 91.07 ± 0.32 | 91.07 ± 0.32 | 0.29 ± 0.00 |
2 × [Bi]LSTM | 1 × [Bi]LSTM | 1 × [Bi]LSTM | 92.50± 0.38 | 92.50 ± 0.38 | 92.67 ± 0.35 | 92.50 ± 0.38 | 92.50 ± 0.38 | 0.19 ± 0.00 | 92.50± 0.26 | 92.50 ± 0.26 | 92.69 ± 0.21 | 92.50 ± 0.26 | 92.50 ± 0.26 | 0.32 ± 0.00 |
2 × [Bi]LSTM | 1 × [Bi]LSTM | 2 × [Bi]LSTM | 92.30 ± 0.28 | 92.30 ± 0.28 | 92.46 ± 0.25 | 92.30 ± 0.28 | 92.30 ± 0.28 | 0.21 ± 0.01 | 92.35 ± 0.26 | 92.35 ± 0.26 | 92.52 ± 0.22 | 92.35 ± 0.26 | 92.35 ± 0.26 | 0.35 ± 0.01 |
2 × [Bi]LSTM | 1 × [Bi]LSTM | 3 × [Bi]LSTM | 92.04 ± 0.37 | 92.04 ± 0.37 | 92.20 ± 0.38 | 92.04 ± 0.37 | 92.04 ± 0.37 | 0.24 ± 0.00 | 92.11 ± 0.25 | 92.11 ± 0.25 | 92.20 ± 0.21 | 92.11 ± 0.25 | 92.11 ± 0.25 | 0.39 ± 0.00 |
2 × [Bi]LSTM | 2 × [Bi]LSTM | 1 × [Bi]LSTM | 92.36 ± 0.31 | 92.36 ± 0.31 | 92.48 ± 0.25 | 92.36 ± 0.31 | 92.36 ± 0.31 | 0.26 ± 0.00 | 92.40 ± 0.20 | 92.40 ± 0.20 | 92.52 ± 0.21 | 92.40 ± 0.20 | 92.40 ± 0.20 | 0.42 ± 0.01 |
2 × [Bi]LSTM | 2 × [Bi]LSTM | 2 × [Bi]LSTM | 92.05 ± 0.23 | 92.05 ± 0.23 | 92.17 ± 0.21 | 92.05 ± 0.23 | 92.05 ± 0.23 | 0.28 ± 0.00 | 92.19 ± 0.25 | 92.19 ± 0.25 | 92.33 ± 0.22 | 92.19 ± 0.25 | 92.19 ± 0.25 | 0.46 ± 0.01 |
2 × [Bi]LSTM | 2 × [Bi]LSTM | 3 × [Bi]LSTM | 91.70 ± 0.47 | 91.70 ± 0.47 | 91.87 ± 0.45 | 91.70 ± 0.47 | 91.70 ± 0.47 | 0.31 ± 0.01 | 91.69 ± 0.35 | 91.69 ± 0.35 | 91.88 ± 0.40 | 91.69 ± 0.35 | 91.69 ± 0.35 | 0.51 ± 0.01 |
2 × [Bi]LSTM | 3 × [Bi]LSTM | 1 × [Bi]LSTM | 92.13 ± 0.28 | 92.13 ± 0.28 | 92.36 ± 0.27 | 92.13 ± 0.28 | 92.13 ± 0.28 | 0.33 ± 0.01 | 92.16 ± 0.28 | 92.16 ± 0.28 | 92.33 ± 0.27 | 92.16 ± 0.28 | 92.16 ± 0.28 | 0.54 ± 0.01 |
2 × [Bi]LSTM | 3 × [Bi]LSTM | 2 × [Bi]LSTM | 91.94 ± 0.32 | 91.94 ± 0.32 | 92.06 ± 0.33 | 91.94 ± 0.32 | 91.94 ± 0.32 | 0.35 ± 0.01 | 91.69 ± 0.32 | 91.69 ± 0.32 | 91.79 ± 0.37 | 91.69 ± 0.32 | 91.69 ± 0.32 | 0.58 ± 0.01 |
2 × [Bi]LSTM | 3 × [Bi]LSTM | 3 × [Bi]LSTM | 91.18 ± 0.25 | 91.18 ± 0.25 | 91.30 ± 0.27 | 91.18 ± 0.25 | 91.18 ± 0.25 | 0.38 ± 0.01 | 91.18 ± 0.23 | 91.18 ± 0.23 | 91.29 ± 0.24 | 91.18 ± 0.23 | 91.18 ± 0.23 | 0.63 ± 0.01 |
3 × [Bi]LSTM | 1 × [Bi]LSTM | 1 × [Bi]LSTM | 92.38 ± 0.25 | 92.38 ± 0.25 | 92.50 ± 0.22 | 92.38 ± 0.25 | 92.38 ± 0.25 | 0.40 ± 0.01 | 92.34 ± 0.21 | 92.34 ± 0.21 | 92.45 ± 0.23 | 92.34 ± 0.21 | 92.34 ± 0.21 | 0.67 ± 0.01 |
3 × [Bi]LSTM | 1 × [Bi]LSTM | 2 × [Bi]LSTM | 92.13 ± 0.30 | 92.13 ± 0.30 | 92.23 ± 0.30 | 92.13 ± 0.30 | 92.13 ± 0.30 | 0.43 ± 0.01 | 92.16 ± 0.27 | 92.16 ± 0.27 | 92.26 ± 0.26 | 92.16 ± 0.27 | 92.16 ± 0.27 | 0.71 ± 0.01 |
3 × [Bi]LSTM | 1 × [Bi]LSTM | 3 × [Bi]LSTM | 91.87 ± 0.26 | 91.87 ± 0.26 | 92.13 ± 0.24 | 91.87 ± 0.26 | 91.87 ± 0.26 | 0.45 ± 0.01 | 92.06 ± 0.23 | 92.06 ± 0.23 | 92.16 ± 0.26 | 92.06 ± 0.23 | 92.06 ± 0.23 | 0.76 ± 0.01 |
3 × [Bi]LSTM | 2 × [Bi]LSTM | 1 × [Bi]LSTM | 92.28 ± 0.28 | 92.28 ± 0.28 | 92.42 ± 0.26 | 92.28 ± 0.28 | 92.28 ± 0.28 | 0.48 ± 0.01 | 92.22 ± 0.17 | 92.22 ± 0.17 | 92.37 ± 0.11 | 92.22 ± 0.17 | 92.22 ± 0.17 | 0.80 ± 0.01 |
3 × [Bi]LSTM | 2 × [Bi]LSTM | 2 × [Bi]LSTM | 92.04 ± 0.25 | 92.04 ± 0.25 | 92.20 ± 0.31 | 92.04 ± 0.25 | 92.04 ± 0.25 | 0.50 ± 0.01 | 92.04 ± 0.25 | 92.04 ± 0.25 | 92.17 ± 0.26 | 92.04 ± 0.25 | 92.04 ± 0.25 | 0.84 ± 0.01 |
3 × [Bi]LSTM | 2 × [Bi]LSTM | 3 × [Bi]LSTM | 91.14 ± 0.21 | 91.14 ± 0.21 | 91.19 ± 0.24 | 91.14 ± 0.21 | 91.14 ± 0.21 | 0.52 ± 0.01 | 91.30 ± 0.11 | 91.30 ± 0.11 | 91.43 ± 0.12 | 91.30 ± 0.11 | 91.30 ± 0.11 | 0.88 ± 0.01 |
3 × [Bi]LSTM | 3 × [Bi]LSTM | 1 × [Bi]LSTM | 92.19 ± 0.23 | 92.19 ± 0.23 | 92.31 ± 0.25 | 92.19 ± 0.23 | 92.19 ± 0.23 | 0.55 ± 0.01 | 92.19 ± 0.18 | 92.19 ± 0.18 | 92.27 ± 0.21 | 92.19 ± 0.18 | 92.19 ± 0.18 | 0.92 ± 0.02 |
3 × [Bi]LSTM | 3 × [Bi]LSTM | 2 × [Bi]LSTM | 92.10 ± 0.13 | 92.10 ± 0.13 | 92.19 ± 0.12 | 92.10 ± 0.13 | 92.10 ± 0.13 | 0.58 ± 0.02 | 91.57 ± 0.32 | 91.57 ± 0.32 | 91.70 ± 0.35 | 91.57 ± 0.32 | 91.57 ± 0.32 | 0.97 ± 0.02 |
3 × [Bi]LSTM | 3 × [Bi]LSTM | 3 × [Bi]LSTM | 91.01 ± 0.09 | 91.01 ± 0.09 | 91.09 ± 0.08 | 91.01 ± 0.09 | 91.01 ± 0.09 | 0.60 ± 0.02 | 91.03 ± 0.17 | 91.03 ± 0.17 | 91.14 ± 0.20 | 91.03 ± 0.17 | 91.03 ± 0.17 | 1.02 ± 0.02 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Truică, C.-O.; Apostol, E.-S. MisRoBÆRTa: Transformers versus Misinformation. Mathematics 2022, 10, 569. https://doi.org/10.3390/math10040569
Truică C-O, Apostol E-S. MisRoBÆRTa: Transformers versus Misinformation. Mathematics. 2022; 10(4):569. https://doi.org/10.3390/math10040569
Chicago/Turabian StyleTruică, Ciprian-Octavian, and Elena-Simona Apostol. 2022. "MisRoBÆRTa: Transformers versus Misinformation" Mathematics 10, no. 4: 569. https://doi.org/10.3390/math10040569
APA StyleTruică, C.-O., & Apostol, E.-S. (2022). MisRoBÆRTa: Transformers versus Misinformation. Mathematics, 10(4), 569. https://doi.org/10.3390/math10040569