Topic Editors

Utrecht University School of Governance, Utrecht University, 3511 ZC Utrecht, The Netherlands
Faculty of Communication and Public Relations, National University of Political Studies and Public Administration, 012244 București, Romania

Getting Insight into How Different Chatbots Answer the Same Questions and How Reliable They Are

Abstract submission deadline
1 July 2024
Manuscript submission deadline
1 September 2024
Viewed by
493

Topic Information

Dear Colleagues,

From November 2023 onward, ChatGPT has been used by a growing number of people looking for answers to a variety of questions in fields such as education, science, law, finance, marketing, public administration, health, and leisure. Since it was made publicly available, a steady flow of scientific studies has been published. Many of these studies use a SWOT analysis to describe its strengths, weaknesses, opportunities, and threats (e.g., Farrokhnia, 2023). Others have analyzed ChatGPT’s conversational characteristics or the extent to which ChatGPT is capable of self-reflection (Loos et al., 2023). Ethical questions have also attracted attention (e.g., Balmer, 2023; Loos & Radicke (submitted)). But ChatGPT is not the only chatbot generating human-like text. Other examples of such chatbots include Google Bard and Perplexity AI. It struck us that there are almost no studies (with studies such Liesenfeld et al.’s in 2023 being a notable exception) systematically comparing how different chatbots answer the same questions and how reliable these answers are. For this reason, we invite researchers to submit cutting-edge empirical studies that showcase how at least two chatbots answer the same question(s) and then present an analysis of the reliability of the answer(s). It is of the utmost importance that the researchers define and operationalize the notion of a ‘reliable answer’. This calls for a thorough knowledge of the field from which the questions are raised. We welcome empirical contributions pertaining to topics including, but not limited to, the reliability of the answers given by at least two chatbots related to the same question(s) in order to analyze the following:

  • The extent to which the language used in the prompts plays a role in the way the chatbots’ answers are generated;
  • If the countries in which the chatbots are used influence their answers;
  • The degree to which chatbots can generate answers in a specific language style (for example, for a twelve-year-old person);
  • If they are able to generate their answer in a specific textual form (e.g., as an essay, a recipe, a scientific paper, etc.);
  • The extent to which the chatbots can execute different tasks, such as writing a text, translating a text, formulating a research design, collecting data, coding, conducting data analysis, summarizing a text, correcting a text, or giving feedback on a text;
  • The sources the chatbots refer to and how reliable they are;
  • The interactional characteristics between users’ prompts and chatbots’ answers.
  • What answers chatbots generate when confronted with an ethical dilemma or politically or societally sensitive issue;
  • When chatbots hallucinate and the characteristics of these hallucinations;
  • If chatbots can answer questions related to current topics.

References

Balmer, A. A Sociological Conversation with ChatGPT about AI Ethics, Affect and Reflexivity. Sociology 2023, 00380385231169676.

Farrokhnia, M.; Banihashem, S.K.; Noroozi, O.; Wals, A. A SWOT analysis of ChatGPT: Implications for educational practice and research. Innovations in Education and Teaching International, 2023, 1-15.

Liesenfeld, A.; Lopez, A.; Dingemanse, M. Opening up ChatGPT: Tracking openness, transparency, and accountability in instruction-tuned text generators. In Proceedings of the 5th International Conference on Conversational User Interfaces, July 2023; pp. 1-6.

Loos, E.; Gröpler, J.; Goudeau, M.L.S. Using ChatGPT in Education: Human Reflection on ChatGPT’s Self-Reflection. Societies 2023, 13, 196.

Loos, E.; Radicke, J. (submitted). Using ChatGPT-3 as a writing tool: An educational assistant or a moral hazard? Current ChatGPT-3 media representations compared to Plato’s critical stance on writing in Phaedrus.

Dr. Eugène Loos
Dr. Loredana Ivan
Topic Editors

Keywords

  • AI
  • LLMs
  • chatbots
  • ChatGPT
  • Google Bard
  • perplexity AI
  • reliability

Participating Journals

Journal Name Impact Factor CiteScore Launched Year First Decision (median) APC
Algorithms
algorithms
2.3 3.7 2008 15 Days CHF 1600 Submit
Behavioral Sciences
behavsci
2.6 3.0 2011 21.5 Days CHF 2200 Submit
Societies
societies
2.1 2.3 2011 32.6 Days CHF 1400 Submit
Technologies
technologies
3.6 5.5 2013 19.7 Days CHF 1600 Submit

Preprints.org is a multidiscipline platform providing preprint service that is dedicated to sharing your research from the start and empowering your research journey.

MDPI Topics is cooperating with Preprints.org and has built a direct connection between MDPI journals and Preprints.org. Authors are encouraged to enjoy the benefits by posting a preprint at Preprints.org prior to publication:

  1. Immediately share your ideas ahead of publication and establish your research priority;
  2. Protect your idea from being stolen with this time-stamped preprint article;
  3. Enhance the exposure and impact of your research;
  4. Receive feedback from your peers in advance;
  5. Have it indexed in Web of Science (Preprint Citation Index), Google Scholar, Crossref, SHARE, PrePubMed, Scilit and Europe PMC.

Published Papers

This Topic is now open for submission.
Back to TopTop