Getting Insight into How Different Chatbots Answer the Same Questions and How Reliable They Are

Topic Information

Dear Colleagues,

From November 2023 onward, ChatGPT has been used by a growing number of people looking for answers to a variety of questions in fields such as education, science, law, finance, marketing, public administration, health, and leisure. Since it was made publicly available, a steady flow of scientific studies has been published. Many of these studies use a SWOT analysis to describe its strengths, weaknesses, opportunities, and threats (e.g., Farrokhnia, 2023). Others have analyzed ChatGPT’s conversational characteristics or the extent to which ChatGPT is capable of self-reflection (Loos et al., 2023). Ethical questions have also attracted attention (e.g., Balmer, 2023; Loos & Radicke (submitted)). But ChatGPT is not the only chatbot generating human-like text. Other examples of such chatbots include Google Bard and Perplexity AI. It struck us that there are almost no studies (with studies such Liesenfeld et al.’s in 2023 being a notable exception) systematically comparing how different chatbots answer the same questions and how reliable these answers are. For this reason, we invite researchers to submit cutting-edge empirical studies that showcase how at least two chatbots answer the same question(s) and then present an analysis of the reliability of the answer(s). It is of the utmost importance that the researchers define and operationalize the notion of a ‘reliable answer’. This calls for a thorough knowledge of the field from which the questions are raised. We welcome empirical contributions pertaining to topics including, but not limited to, the reliability of the answers given by at least two chatbots related to the same question(s) in order to analyze the following:

• The extent to which the language used in the prompts plays a role in the way the chatbots’ answers are generated;
• If the countries in which the chatbots are used influence their answers;
• The degree to which chatbots can generate answers in a specific language style (for example, for a twelve-year-old person);
• If they are able to generate their answer in a specific textual form (e.g., as an essay, a recipe, a scientific paper, etc.);
• The extent to which the chatbots can execute different tasks, such as writing a text, translating a text, formulating a research design, collecting data, coding, conducting data analysis, summarizing a text, correcting a text, or giving feedback on a text;
• The sources the chatbots refer to and how reliable they are;
• The interactional characteristics between users’ prompts and chatbots’ answers.
• What answers chatbots generate when confronted with an ethical dilemma or politically or societally sensitive issue;
• When chatbots hallucinate and the characteristics of these hallucinations;
• If chatbots can answer questions related to current topics.

References

Balmer, A. A Sociological Conversation with ChatGPT about AI Ethics, Affect and Reflexivity. Sociology 2023, 00380385231169676.

Farrokhnia, M.; Banihashem, S.K.; Noroozi, O.; Wals, A. A SWOT analysis of ChatGPT: Implications for educational practice and research. Innovations in Education and Teaching International, 2023, 1-15.

Liesenfeld, A.; Lopez, A.; Dingemanse, M. Opening up ChatGPT: Tracking openness, transparency, and accountability in instruction-tuned text generators. In Proceedings of the 5th International Conference on Conversational User Interfaces, July 2023; pp. 1-6.

Loos, E.; Gröpler, J.; Goudeau, M.L.S. Using ChatGPT in Education: Human Reflection on ChatGPT’s Self-Reflection. Societies 2023, 13, 196.

Loos, E.; Radicke, J. (submitted). Using ChatGPT-3 as a writing tool: An educational assistant or a moral hazard? Current ChatGPT-3 media representations compared to Plato’s critical stance on writing in Phaedrus.

Dr. Eugène Loos
Dr. Loredana Ivan
Topic Editors

Keywords

Participating Journals

Journal Name	Impact Factor	CiteScore	Launched Year	First Decision (median)	APC
Algorithms algorithms	1.8	4.1	2008	18.9 Days	CHF 1600
Behavioral Sciences behavsci	2.5	2.6	2011	28.7 Days	CHF 2200
Societies societies	1.7	3.1	2011	32.1 Days	CHF 1400
Technologies technologies	4.2	6.7	2013	21.1 Days	CHF 1600

Preprints.org is a multidisciplinary platform offering a preprint service designed to facilitate the early sharing of your research. It supports and empowers your research journey from the very beginning.

MDPI Topics is collaborating with Preprints.org and has established a direct connection between MDPI journals and the platform. Authors are encouraged to take advantage of this opportunity by posting their preprints at Preprints.org prior to publication:

1. Share your research immediately: disseminate your ideas prior to publication and establish priority for your work.
2. Safeguard your intellectual contribution: Protect your ideas with a time-stamped preprint that serves as proof of your research timeline.
3. Boost visibility and impact: Increase the reach and influence of your research by making it accessible to a global audience.
4. Gain early feedback: Receive valuable input and insights from peers before submitting to a journal.
5. Ensure broad indexing: Web of Science (Preprint Citation Index), Google Scholar, Crossref, SHARE, PrePubMed, Scilit and Europe PMC.

Published Papers