Improving Access to Justice with Legal Chatbots
Abstract
:1. Introduction
2. State of the Art
2.1. Evaluation Metrics
2.2. Representation Learning and Text Classification
2.2.1. Bag-of-Words and Variants
A: “All foreign students can apply for a visa online.”,
B: “How to get on a student visa?”,
C: “How can I work while I am a student?”.
I: “I received my work visa.”,
J: “I need a student visa”,
K: “I received a letter.”,
2.2.2. Word Embeddings
2.3. Language Models and Recurrent Networks
2.4. Chatbots
2.4.1. Question-Answering Systems
- -
- Student visa, work authorization.
- -
- I’m on a student visa. Can I work?
- -
- : “If you are an American citizen, you need a job offer to apply to a work visa. To get a student visa, you only need to prove your student status. There is no working holiday visa option.
- -
- : “If you are French, you will need […]”
- -
- : “What are the prerequisites for a work permit for U.S. citizens?”
- -
- : “What are the prerequisites for Americans to get a work permit?”
2.4.2. General Chatbots
2.4.3. Task-Specific Chatbots
started: TRUE; citizenship: UNKNOWN, visa_type: STUDENT; requirements_given: FALSE
3. Methodology
3.1. Experiment Tracking and Reproducibility
3.2. Data
3.2.1. Immigration Canada FAQ
3.2.2. Internal Legal QAS
4. Experiments and Results
4.1. Immigration Chatbot
4.1.1. Baseline Using StarSpace
4.1.2. Approach Based on Information Retrieval
4.1.3. Results
4.2. Legal Information Chatbot in a Corporate Environment
4.2.1. Baseline Using StarSpace
4.2.2. Enhanced Baseline
4.2.3. Approach Based on Information Retrieval
4.2.4. Results
5. Conclusions
Author Contributions
Funding
Acknowledgments
Conflicts of Interest
Abbreviations
BERT | Bidirectionnal Encoder Representations from Transformers |
BoW | Bag-of-Words |
CNN | Convolutional Neural Network |
FAQ | Frequently Asked Questions |
IR | Information Retrieval |
ML | Machine Learning |
NBC | National Bank of Canada |
NLP | Natural Language Processing |
NRLs | Non-Represented Litigants |
QAS | Question Answering System |
RNN | Recurrent Neural Network |
TF-IDF | Term Frequency-Inverse Document Frequency |
UQAM | Université du Québec à Montréal |
References
- Laniel, R.A.; Bahary-Dionne, A.; Bernheim, E. Agir seul en justice: Du droit au choix—État de la jurisprudence sur les droits des justiciables non représentés. Les Cah. Droit 2018, 59, 495–532. [Google Scholar] [CrossRef] [Green Version]
- Schneider, R.K. Illiberal Construction of Pro Se Pleadings. Univ. Pa. Law Rev. 2010, 159, 585. [Google Scholar]
- Manning, C.D.; Manning, C.D.; Schütze, H. Foundations of Statistical Natural Language Processing; MIT Press: Cambridge, MA, USA, 1999. [Google Scholar]
- Manning, C.D.; Raghavan, P.; Schütze, H. Introduction to Information Retrieval; Cambridge University Press: New York, NY, USA, 2008. [Google Scholar]
- Jones, K.S. A Statistical Interpretation of Term Specificity and its Application in Retrieval. J. Doc. 1972, 28, 11–21. [Google Scholar] [CrossRef]
- Shannon, C.E.; Weaver, W. A Mathematical Theory of Communication; University of Illinois Press: Champaign, IL, USA, 1998. [Google Scholar]
- Robertson, S. Understanding inverse document frequency: On theoretical arguments for IDF. J. Doc. 2004, 60, 503–520. [Google Scholar] [CrossRef] [Green Version]
- Harris, Z.S. Distributional Structure. Word 1954, 10, 146–162. [Google Scholar] [CrossRef]
- Landauer, T.K.; Dumais, S.T. A Solution to Plato’s Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge. Psychol. Rev. 1997, 104, 211. [Google Scholar] [CrossRef]
- Mikolov, T.; Chen, K.; Corrado, G.; Dean, J. Efficient Estimation of Word Representations in Vector Space. arXiv 2013, arXiv:1301.3781. [Google Scholar]
- Mikolov, T.; Sutskever, I.; Chen, K.; Corrado, G.S.; Dean, J. Distributed Representations of Words and Phrases and Their Compositionality. In Proceedings of the Advances in Neural Information Processing Systems, Lake Tahoe, NV, USA, 5–10 December 2013; pp. 3111–3119. [Google Scholar]
- Arora, S.; Liang, Y.; Ma, T. A Simple but Tough-to-Beat Baseline for Sentence Embeddings. In Proceedings of the ICLR 2017 Conference Submission, Toulon, France, 24–26 April 2017. [Google Scholar]
- Wu, L.Y.; Fisch, A.; Chopra, S.; Adams, K.; Bordes, A.; Weston, J. Starspace: Embed all the things! In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, LA, USA, 2–7 February 2018. [Google Scholar]
- Kenter, T.; Borisov, A.; De Rijke, M. Siamese CBOW: Optimizing word embeddings for sentence representations. arXiv 2016, arXiv:1606.04640. [Google Scholar]
- Xu, W.; Rudnicky, A. Can Artificial Neural Networks Learn Language Models? In Proceedings of the Sixth International Conference on Spoken Language Processing, Beijing, China, 16–20 October 2000. [Google Scholar]
- Bengio, Y.; Ducharme, R.; Vincent, P.; Jauvin, C. A Neural Probabilistic Language Model. J. Mach. Learn. Res. 2003, 3, 1137–1155. [Google Scholar]
- Cho, K.; Van Merriënboer, B.; Gulcehre, C.; Bahdanau, D.; Bougares, F.; Schwenk, H.; Bengio, Y. Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation. arXiv 2014, arXiv:1406.1078. [Google Scholar]
- Bahdanau, D.; Cho, K.; Bengio, Y. Neural Machine Translation by Jointly Learning to Align and Translate. arXiv 2014, arXiv:1409.0473. [Google Scholar]
- Devlin, J.; Chang, M.W.; Lee, K.; Toutanova, K. BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. arXiv 2018, arXiv:1810.04805. [Google Scholar]
- Jurafsky, D.; Martin, J.H. Question Answering. In Speech and Language Processing; Pearson: London, UK, 2014; Volume 3. [Google Scholar]
- Auer, S.; Bizer, C.; Kobilarov, G.; Lehmann, J.; Cyganiak, R.; Ives, Z. DBpedia: A Nucleus for a Web of Open Data. In Proceedings of the 6th International The Semantic Web and 2nd Asian Conference on Asian Semantic Web Conference, ISWC’07/ASWC’07, Busan, Korea, 11–15 November 2007; Springer: Berlin/Heidelberg, Germany, 2007; pp. 722–735. [Google Scholar]
- Deriu, J.; Rodrigo, A.; Otegi, A.; Echegoyen, G.; Rosset, S.; Agirre, E.; Cieliebak, M. Survey on Evaluation Methods for Dialogue Systems. arXiv 2019, arXiv:1905.04071. [Google Scholar] [CrossRef]
- Weizenbaum, J. ELIZA—A Computer Program for the Study of Natural Language Communication between Man and Machine. Commun. ACM 1966, 9, 36–45. [Google Scholar] [CrossRef]
- Wallace, R.; Tomabechi, H.; Aimless, D. Chatterbots Go Native: Considerations for an Eco-System Fostering the Development of Artificial Life Forms in a Human World. 2003. Available online: http://www.pandorabots.com/pandora/pics/chatterbotsgonative.doc (accessed on 1 December 2019).
- Satu, M.S.; Parvez, M.H. Review of Integrated Applications With AIML Based Chatbot. In Proceedings of the 2015 International Conference on Computer and Information Engineering (ICCIE), Rajshahi, Bangladesh, 26–27 November 2015; pp. 87–90. [Google Scholar]
- Vinyals, O.; Le, Q. A Neural Conversational Model. arXiv 2015, arXiv:1506.05869. [Google Scholar]
- Sutskever, I.; Vinyals, O.; Le, Q.V. Sequence to Sequence Learning with Neural Networks. In Proceedings of the Advances in Neural Information Processing Systems 27 (NIPS 2014), Montreal, QC, Canada, 8–13 December 2014; pp. 3104–3112. [Google Scholar]
- Qiu, M.; Li, F.L.; Wang, S.; Gao, X.; Chen, Y.; Zhao, W.; Chen, H.; Huang, J.; Chu, W. AliMe Chat: A Sequence to Sequence and Rerank Based Chatbot Engine. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, BC, Canada, 30 July–4 August 2017; Volume 2, pp. 498–503. [Google Scholar]
- Turing, A.M. Computing Machinery and Intelligence. Mind 1950, 59, 433. [Google Scholar] [CrossRef]
- Radziwill, N.M.; Benton, M.C. Evaluating Quality of Chatbots and Intelligent Conversational Agents. arXiv 2017, arXiv:1704.04579. [Google Scholar]
- Ramos, R. Screw the Turing Test-Chatbots Don’t Need to Act Human. VentureBeat. 2017. Available online: https://venturebeat.com/2017/02/03/screw-the-turing-test-chatbots-dont-need-to-act-human/ (accessed on 13 March 2017).
- Papineni, K.; Roukos, S.; Ward, T.; Zhu, W.J. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, Philadelphia, PA, USA, 7–12 July 2002; pp. 311–318. [Google Scholar]
- Lin, C.Y. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out; Association for Computational Linguistics: Barcelona, Spain, 2004; pp. 74–81. [Google Scholar]
- Lowe, R.; Noseworthy, M.; Serban, I.V.; Angelard-Gontier, N.; Bengio, Y.; Pineau, J. Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses. arXiv 2017, arXiv:1708.07149. [Google Scholar]
- Bobrow, D.G.; Kaplan, R.M.; Kay, M.; Norman, D.A.; Thompson, H.; Winograd, T. GUS, a Frame-Driven Dialog System. Artif. Intell. 1977, 8, 155–173. [Google Scholar] [CrossRef]
- Meurs, M.J.; Duvert, F.; Lefevre, F.; De Mori, R. Markov Logic Networks for Spoken Language Interpretation. Inf. Syst. J. 2008, 1978, 535–544. [Google Scholar]
- Meurs, M.J.; Lefevre, F.; De Mori, R. Learning Bayesian Networks for Semantic Frame Composition in a Spoken Dialog System. In Proceedings of the Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Boulder, CO, USA, 31 May–5 June 2009; pp. 61–64. [Google Scholar]
- Jurafsky, D.; Martin, J.H. Dialog Systems and Chatbots. In Speech and Language Processing; Pearson: London, UK, 2017; Volume 3, pp. 6–10. [Google Scholar]
- Bocklisch, T.; Faulkner, J.; Pawlowski, N.; Nichol, A. Rasa: Open source language understanding and dialogue management. arXiv 2017, arXiv:1712.05181. [Google Scholar]
- Baker, M. 1500 Scientists Lift the Lid on Reproducibility. Nat. News 2016, 533, 452. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Gundersen, O.E.; Kjensmo, S. State of the art: Reproducibility in artificial intelligence. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, LA, USA, 2–7 February 2018. [Google Scholar]
- Zaharia, M.; Chen, A.; Davidson, A.; Ghodsi, A.; Hong, S.A.; Konwinski, A.; Murching, S.; Nykodym, T.; Ogilvie, P.; Parkhe, M.; et al. Accelerating the Machine Learning Lifecycle with MLflow. IEEE Data Eng. Bull. 2018, 41, 39–45. [Google Scholar]
- Charton, E.; Meurs, M.J.; Jean-Louis, L.; Gagnon, M. Mutual Disambiguation for Entity Linking. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, Baltimore, MD, USA, 23–25 June 2014; Volume 2, pp. 476–481. [Google Scholar]
- Queudot, M.; Meurs, M.J. Artificial Intelligence and Predictive Justice: Limitations and Perspectives. In Proceedings of the International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems, Montreal, QC, Canada, 25–28 June 2018; pp. 889–897. [Google Scholar]
Predicted | ||||
---|---|---|---|---|
A | B | C | ||
true | A | 3 | 1 | 1 |
B | 4 | 5 | 1 | |
C | 0 | 1 | 6 |
a | All | Am | Apply | Can | for | Foreign | Get | How | I | on | Online | Student | to | Visa | While | Work | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
A | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 1 | 0 | 1 | 0 | 0 |
B | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 0 | 1 | 0 | 1 | 1 | 1 | 0 | 0 |
C | 1 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 0 | 0 | 1 | 1 |
Nationality | Type | Requirements |
---|---|---|
American | work | You need a job offer. |
American | study | You only need a document mentioning the equivalent degree. |
American | PVT | This visa is not offered to American citizens. |
French | work | You need a job offer and a report from a competitivity analysis. |
French | study | You need to show an acceptation letter and have $10,000. |
French | PVT | You need to register to the random drawing. |
microP | microR | microF | macroP | macroR | macroF | wP | wR | wF | |
---|---|---|---|---|---|---|---|---|---|
softmax | 0.92 | 0.52 | 0.67 | 0.51 | 0.52 | 0.52 | 0.51 | 0.52 | 0.52 |
marge | 1 | 0.60 | 0.75 | 0.60 | 0.60 | 0.60 | 0.60 | 0.60 | 0.60 |
IR | 1 | 0.60 | 0.75 | 0.60 | 0.60 | 0.60 | 0.60 | 0.60 | 0.60 |
microP | microR | microF | macroP | macroR | macroF | wP | wR | wF | |
---|---|---|---|---|---|---|---|---|---|
StarSpace | 0.61 | 0.61 | 0.61 | 0.52 | 0.48 | 0.48 | 0.73 | 0.61 | 0.63 |
BERT | 0.70 | 0.67 | 0.66 | 0.75 | 0.75 | 0.75 | 0.85 | 0.75 | 0.76 |
IR | 0.78 | 0.64 | 0.70 | 0.64 | 0.62 | 0.60 | 0.72 | 0.64 | 0.65 |
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Queudot, M.; Charton, É.; Meurs, M.-J. Improving Access to Justice with Legal Chatbots. Stats 2020, 3, 356-375. https://doi.org/10.3390/stats3030023
Queudot M, Charton É, Meurs M-J. Improving Access to Justice with Legal Chatbots. Stats. 2020; 3(3):356-375. https://doi.org/10.3390/stats3030023
Chicago/Turabian StyleQueudot, Marc, Éric Charton, and Marie-Jean Meurs. 2020. "Improving Access to Justice with Legal Chatbots" Stats 3, no. 3: 356-375. https://doi.org/10.3390/stats3030023
APA StyleQueudot, M., Charton, É., & Meurs, M.-J. (2020). Improving Access to Justice with Legal Chatbots. Stats, 3(3), 356-375. https://doi.org/10.3390/stats3030023