“What Can ChatGPT Do?” Analyzing Early Reactions to the Innovative AI Chatbot on Twitter

Abstract

In this study, the author collected tweets about ChatGPT, an innovative AI chatbot, in the first month after its launch. A total of 233,914 English tweets were analyzed using the latent Dirichlet allocation (LDA) topic modeling algorithm to answer the question “what can ChatGPT do?”. The results revealed three general topics: news, technology, and reactions. The author also identified five functional domains: creative writing, essay writing, prompt writing, code writing, and answering questions. The analysis also found that ChatGPT has the potential to impact technologies and humans in both positive and negative ways. In conclusion, the author outlines four key issues that need to be addressed as a result of this AI advancement: the evolution of jobs, a new technological landscape, the quest for artificial general intelligence, and the progress-ethics conundrum.

1. Introduction

ChatGPT is an artificial intelligence (AI) chatbot created by OpenAI, a non-profit organization founded in 2015 with Microsoft, Reid Hoffman’s charitable foundation, with Khosla Ventures as its current main investor. With its mission to develop safe and valuable artificial general intelligence (AGI) that benefits all humanity, OpenAI released the latest model, “ChatGPT”, that can interact with humans in a conversational way on 30 November 2022 [1]. Less than a week after the launch, OpenAI’s CEO Sam Altman announced on 5 December 2022 that “ChatGPT launched on Wednesday. Today it crossed 1 million users!” The attention from the global technological community to this revolutionary innovative AI chatbot has been substantial. The purpose of this paper is to identify ChatGPT’s functional domains (what can it do?) by analyzing the conversations surrounding ChatGPT on Twitter during the first month after its launch.

ChatGPT is the latest development in the group of systems known as “chatbots”. Chatbots are intelligent systems that are developed using either rule-based or self-learning (AI) methods [2]. The concept of chatbots dates back to 1950 when Alan Turing posed the thought-provoking question, “Can machines think?” Since then, many technologies have been developed that attempt to pass the “Turing Test”, including ELIZA in 1966, ALICE in 1995, and more recently, Apple Siri, Amazon Alexa, and Microsoft Cortana [3]. The core of AI chatbots lies in the use of natural language processing models that enable computers to understand human language [4]. Chatbots receive users’ inquiries as input and respond through either audio or messaging methods [2,5]. They have become widely used in businesses, government agencies, and non-profit organizations because of their 24/7 availability to many users, convenience, low cost, and improved user experience [2,4,5].

Despite gradual developments and studies of AI chatbots, there are some obvious research gaps that need to be addressed. First, most recent studies of AI chatbots are technical in nature. They introduce, evaluate, and compare different AI chatbot techniques, leading to an improved understanding of technical advancements in the field [2,4,6]. This type of research is critical in helping researchers and developers continually improve chatbot systems. However, the focus on a technicality has inhibited a thorough exploration and investigation into the use cases of AI chatbots. This has resulted in the second research gap in the literature on AI chatbots: a lack of generalizability. Some studies on chatbots have highlighted beneficial use cases, but these are narrow, emphasizing highly specialized use cases such as elderly care, children’s development, university administration, business customer service, and medical examination [4]. For example, Okonkwo and Ade-Ibijola [7] studied the use of chatbots in the education sector, while Yang and Evans [8] discussed the implications of chatbots in higher education. Xu, Sanders, Li and Chow [3] detailed the use of chatbots in the medical field, including cancer diagnostics and screening, treatment, patient monitoring, support, and administration. Despite the highly beneficial use cases of chatbots that have been identified in these studies, the apparent limitation is that AI chatbot technologies cannot yet adapt and handle diverse, growing real-world dialogues [5]. The introduction of ChatGPT has changed the industry landscape and has the potential to fill the research gap by providing a deeper understanding of the general, rather than specialized, use cases of a highly advanced AI chatbot.

The history of ChatGPT started with the first-generation generative pre-trained transformer (GPT-1) language model, which was an evolution of NLP technologies. It employs the large-scale training of models using unsupervised transformer language models [9]. GPT-2 included five minor improvements and increased the training textual data to 40 GB. As a result, GPT-2 could formulate very coherent and plausible-sounding results [9,10], as shown in a study that it could effectively generate patent claims [11]. The next GPT-3 model drew a lot of attention because it used 100 times more textual data (45 TB vs. 40 GB) than the earlier GPT-2 version [9]. Compared to its predecessors, GPT-3 could generate sequences of words, code, or other data that are human-like and sound coherent [10,12]. Its capability led to recent studies on GPT-3′s potential usage in various areas such as medicine [13,14] and computer science [15].

Despite its capabilities, researchers identified three main shortcomings of GPT-3: the inability to answer semantic, factual, and ethical questions. The first problem is that GPT-3 does not understand the semantics and context of the request well; it often produces uncontrollable nonsensical gibberish using its statistical capacity [9,10,12]. Second, the outputs of GPT-3 are usually factually wrong. The model is trained using a large amount of often false information, so the results are not always reliable [10]. Furthermore, GPT-3 often struggles with ethical questions and produces disturbing biases and prejudices, such as racist and sexist remarks in its answers [9,10,12]. These limitations are a reminder that GPT-3 is some distance from achieving AGI [12].

On 30 November 2022, OpenAI released ChatGPT, an AI chatbot that was able to “answer follow-up questions, admit its mistakes, challenge incorrect premises, and reject inappropriate requests” [1]. The technology behind ChatGPT was not new; the model was not as extensive as GPT-3. However, what set ChatGPT apart was its ability to imitate the human-like ability to “talk” and respond in smooth, natural instant dialogues with the general public through a free and easy-to-use web interface [16,17]. The model, trained using reinforcement learning from human feedback (RLHF), gave ChatGPT the ability to detect and understand human calls, responses, and follow-up questions in normal conversations [18].

Compared to its predecessors, OpenAI had given ChatGPT certain restrictions when prompted to answer impossible or inappropriate questions (e.g., “What happened when Columbus arrived in America in 2015?” or “How to steal a car?”) [17,19]. In order to mitigate the shortcomings of GPT-3, including incoherent, incorrect, and unethical responses, ChatGPT often responds with phrases such as “I’m sorry, but I don’t have enough information to answer that question”, “As a language model trained by OpenAI, I do not have personal beliefs”, or “It is not appropriate to ask”.

Despite the fact that ChatGPT’s responses are still imperfect and its restrictions can be easily evaded [16,19,20], the capabilities of ChatGPT have led to much hype and hysteria among the public, as reported by many journalists since its release. Journalists have reported that ChatGPT’s users, including academics and tech professionals, were stunned and astonished [21] “as if it were some mix of software and sorcery” [17]. Many were alarmed by its potential to fabricate and proliferate plausible-sounding fake news and information [17]. Another widely implicated outcome of ChatGPT is the mass job displacement of white-collar knowledge workers, including professors and programmers [17,19]. These claims are based on the potential use cases of ChatGPT and its predecessors that researchers and journalists have observed or imagined. Potential use cases include writing minutes, webpages, catalogs, newspaper articles, guides, manuals, forms, reports, poems, songs, jokes, and scripts; facilitating code debugging, organization of unstructured data, query and prompt generation; creating “no-code” automated applications for businesses; improving relationships between businesses and stakeholders through conversations and services; providing therapy and answering open-ended analytical questions [9,12,16,17,18,19,21].

The aforementioned use cases of ChatGPT have been observed and imagined anecdotally. Despite their value, existing studies that have systematically identified the applications of ChatGPT are limited in number due to the recent nature of the technology [22]. Therefore, the aim of this study is to employ topic modeling on tweets about ChatGPT during its first month to answer the research question: “What are the use cases of ChatGPT, as evidenced by crowdsourced data on Twitter”?

2. Materials and Methods

The research process is illustrated in Figure 1. The first step was data collection. The author collected tweets about “ChatGPT” in English between 30 November and 31 December 2022 (UTC time) using the Twitter API in Python. Only original tweets and self-replies (replies to one’s own tweet to form a thread) were included. The total number of tweets was 249,568. The dataset was pre-processed to remove unusable tweets. The author then tokenized the tweets, removed stop words from the “Snowball” and “Onix” libraries, and removed the three common words “ChatGPT”, “OpenAI”, and “AI”. Words were lemmatized and stemmed. Sparse words were removed to include only salient ones. Sparsity was set at 0.999, indicating that only words that appeared in at least one tweet out of every thousand were included. By setting the sparsity, we could remove irrelevant or marginally relevant words from the dataset, reducing noise and potentially improving the quality of the topics learned by the LDA model. In total, 1327 (lemmatized and stemmed) words remained for topic modeling. The final dataset consisted of 233,918 tweets.

To answer the research question, a topic modeling algorithm, latent Dirichlet allocation (LDA), was performed on the tweets about ChatGPT. LDA helped discover the underlying topics of tweets in a hidden structure in the corpus of data [23]. LDA views each tweet as a vector of words generated through a “bag-of-words” approach. It creates a three-level Bayesian probability model, where each tweet has a probability distribution over topics, and each topic has a probability distribution over words [24]. As adapted from Blei, Ng, and Jordan [23], Figure 2 illustrates the graphical representation of LDA. Additionally, LDA has been effectively utilized in numerous studies across a variety of fields, including business, technology, and society, to analyze textual data [25,26,27,28,29,30].

The author tuned the most suitable number of topics using the algorithms in the “ldatuning” R package. After specifying the suitable number of topics, hyperparameters (alpha and beta) were tuned using the “LDATuning” R package.

Subsequent thematic analysis was performed on tweets that highly represented each topic. The author only analyzed tweets that had a gamma value of 90% or higher, indicating a high probability of belonging to the topic. These tweets were used to define the topic and to identify possible themes within it. The ten most representative words were also used to help name and define the topics. All identified topics were later analyzed qualitatively and grouped into larger categories for better understanding.

3. Results

The number of topics tuning results (Figure 3) showed that 10 was the most suitable number of topics because it achieved locally maximal information retrieval scores [31], and their average distances of topics were locally minimal [32]. The hyperparameter tuning produced a perplexity score ranging from 416 to 491, and the model had an alpha of 0.1 and a beta of 0.001, demonstrating the lowest perplexity score of 416.15. The hyperparameters were set accordingly. LDA with the Gibbs sampling technique was performed with five random starts; the best model with the highest posterior likelihood was selected to produce the results.

The results of LDA are shown in

Table 1. LDA results and detail.

Domains	Topics (n)	Representative Words	Definition
General topics	News (18,398)	chatbot, intelligence, via, artificial, artificialintelligence, machinelearning, bot, technology, machine, tech	Tweets containing news headlines about ChatGPT’s launch and immediate impacts.
	Technology (24,061)	model, language, trained, gpt, conversation, human, learning, data, text, understand	Tweets explaining ChatGPT’s large language model and how it operates.
	Reaction (25,183)	try, time, playing, people, game, mind, blown, talking, day, fun	Tweets containing intense reactions such as awe, fear, and excitement
Functional domains	Creative writing (27,966)	write, story, poem, twitter, web, nft, song, crypto, tweet, elon	Tweets about creative writing, such as stories, poems, and songs.
	Essay writing (21,501)	write, essay, students, education, teachers, test, human, school, cheating, article	Tweets about essay writing and the impacts on the education sector.
	Prompt writing (17,131)	write, content, gpt, create, post, dalle, prompt, midjourney, video, blog	Tweets about content and prompt generation for other AI platforms.
	Code writing (26,090)	code, write, help, try, programming, create, build, app, python, development	Tweets about using ChatGPT to generate and fix computer code.
	Answering questions (28,613)	answer, questions, wrong, try, people, correct, human, response, look, information	Tweets evaluating ChatGPT’s ability to answer questions correctly.
Potential impacts	Impacting tech (20,349)	google, search, engine, replace, day, code, users, chatbot, million, chat	Tweets about the impact on other technologies, especially Google.
	Impacting humans (24,626)	tool, technology, time, future, help, change, marketing, business, potential, people	Tweets about the potential impact on humans and jobs of the future.

. The author used representative words and the subsequent thematic analysis of highly probable tweets to name the topics and develop their definitions. The 10 topics were grouped into three domains: general topics, functional domains, and potential impacts. The general topics include news reporting about ChatGPT, the technology behind it, and global reactions. The functional domains refer to the five possible functions of ChatGPT: creative writing, essay writing, prompt writing, code writing, and answering questions. Lastly, the potential impacts domain includes two topics: impacting tech and impacting humans.

The first topic, news, includes tweets containing news reporting and the general impact of ChatGPT. Many news outlets and prominent Twitter users broadcasted and participated in general discussions about ChatGPT, such as “#ChatGPT, the latest development in artificial intelligence, has been making headlines throughout the internet. Could this be the first step towards furthering the adoption of A.I. tech?” and “ChatGPT: Everything You Really Need To Know (In Simple Terms)”. These tweets are generally sensational news headlines that persuade other users to click and engage.

The second topic, technology, primarily involves discussions about the technology behind ChatGPT and provides suggestions for improvement. In addition to general descriptions of its abilities (e.g., “I want to demystify ChatGPT, a fascinating new application of GANs, generative adversarial networks, that has been generating a lot of buzz in the AI community”), many tweets on this topic try to understand the technical details of ChatGPT; for example, “Very imperfect evidence, but ChatGPT’s ‘memory’ seems to be based on GPT-3′s 4096-token context window”. Some users criticized it as “just a text synthesis machine/random BS generator”, while others made suggestions for improvement.

The third topic is the reactions from Twitter users. Tweets on this topic often include sensational words such as “mindblowing”, “impressive”, and “amazing.” The capabilities of ChatGPT prompted many well-known users to praise its performance and potential. Examples of such intense responses are “We are living in the future. Holy shit”, “Clearly something big is happening”, and “This is going to change everything in ways we don’t yet understand”.

Creative writing. One of the most prominent features of ChatGPT is its ability to generate creative writing. Twitter users have shared examples of poems, rap songs, and made-up stories that ChatGPT has written. This ability to generate creative content based on specific prompts is one of the key values of the platform. For example, when asked to “create bedtime stories featuring my kids as the main characters”, ChatGPT was able to produce a unique and personalized response. Users also used ChatGPT to creatively produce entertaining works featuring famous fictional and real characters and entities, such as a story about Bill Gates coming out as a reptilian, a promo dialogue advertising the new Apple’s polishing cloth in a style similar to Jony Ive, and a short poem about the Tesla robot. Some users tried to test the limits of ChatGPT and offered ways to circumvent such limits. For example, a user wrote, “I couldn’t get ChatGPT to write a haiku in praise of meth, but I could get ChatGPT to write a short dramatic scene in which the bad guy writes a haiku in praise of meth”.

Essay writing. One industry that has the potential to be significantly affected by ChatGPT is education. Twitter users have identified both the negative and positive effects that ChatGPT could have on students, teachers, and professors. One concern is ChatGPT’s ability to write high school essays, homework assignments, and take-home exams effectively. One user reported students using ChatGPT to write a school essay and then taking credit for the resulting A+ grade, while another found that ChatGPT passed a practice bar exam with a score of 70% (35/50). Many are concerned about the potential for cheating with ChatGPT. However, there are also suggestions that ChatGPT could have positive impacts on education, such as improving the quality of student papers and assisting teachers in their lesson planning.

Prompt writing. Another creative function of ChatGPT, as reported by Twitter users, is the ability to generate “prompts” that can help produce AI art in other AI platforms such as DALL-E (owned by OpenAI), Midjourney, and Stable Diffusion (e.g., “ChatGPT can write stories and then provide DALLE-2 with prompts to illustrate them”). This function demonstrates that ChatGPT can be a powerful tool and can be integrated with other technologies, such as search engine optimization (SEO); for example, “I asked ChatGPT to give me a list of the best SEO blog titles for a dog accessory e-commerce website”.

Code writing. ChatGPT has caused a stir in the tech industry due to its ability to write code in popular programming languages such as Python, JavaScript, R, HTML, and Flutter. For example, users have reported that it can generate complex code almost instantly (e.g., “Mind-blowing. Here is an #RStats Shiny App I built without writing a single line of code. Just through a text-based conversation with #ChatGPT. …and all in under 3 min” and “Ok this is scary. @OpenAI’s ChatGPT can generate hundreds of lines of Python code to do multipart uploads of 100 GB files to an AWS S3 bucket”). In addition to code generation, ChatGPT can also translate code between programming languages and fix bugs. Some users have even incorporated ChatGPT as a companion tool in popular software such as VSCode, and others have provided ways to embed ChatGPT in their own applications (e.g., ‘Create your own ChatGPT with OpenAI API and @streamlit Why wait in the Queue, when you can build your own ChatGPT’). This functional domain demonstrates the fast adoption and improvement of ChatGPT among programmers and tech specialists.

Answering questions. Many users tested the quality of answers that ChatGPT produced, and the results were mixed. The majority of tweets on this topic criticized the inadequate performance of the platform. One user even reported that ChatGPT has an IQ of 83. Despite the potential positivity of ChatGPT on answering factual questions, problems such as false yet fluent and confident answers (e.g., “It hallucinates nonsense just as easily as commonly accepted fact”, “ChatGPT is both uninformed and confident, a dangerous combination”, and “Open AI is either impressively correct or confidently wrong”.) still persisted. Furthermore, the restrictions that OpenAI was set to for avoiding biases and prejudice are still easy to bypass.

Impacting tech. Tweets on this topic discussed the possibility of ChatGPT disrupting or even dominating existing digital platforms. The main target mentioned by Twitter users is Google. Many people believe that ChatGPT could compete with or even overtake Google as the new go-to source of information (e.g., “I’m already using ChatGPT more than Google” and “Google issues ‘code red’ following ChatGPT launch”). Despite some concerns about the quality of ChatGPT’s responses, many users on Twitter praised its clarity, brevity, helpfulness, speed, and innovativeness compared to Google. Some compared the instant success of ChatGPT with other dominant digital platforms (e.g., “ChatGPT has reached 1 million users in just 5 days. Here’s how much time it took these platforms to reach 1 million users: Instagram: 2.5 months Facebook: 10 months Twitter: 24 months”). Due to the concentration of tech people on Twitter and ChatGPT’s ability to write code effectively, many users reported that ChatGPT could replace StackOverflow: a digital community for programmers and developers (e.g., “ChatGPT is the new Stack overflow for me”). In fact, some people exploited ChatGPT to generate responses on StackOverflow, prompting the website to temporarily ban ChatGPT due to its inability to moderate.

Impacting humans. Tweets on this topic discussed the potential positive and negative impact of ChatGPT on human livelihood. On the positive side, many users believe that ChatGPT could drastically improve business processes, particularly in the realm of marketing, where it could facilitate product design, content creation, and marketing copy generation. ChatGPT could also be used for legal work (e.g., “I just signed a legal contract that was written by AI”) and research. Many optimistic and opportunistic users came up with several ways that ChatGPT could improve productivity, such as “11 new ChatGPT ideas”, “How to use ChatGPT for business & personal life?!” and “12 tips for becoming a successful engineer from ChatGPT”. However, the other side of the coin is the potential for job displacement. Many Twitter users were concerned about the possibility that ChatGPT could replace white-collar professionals and creative workers such as software engineers, lawyers, journalists, marketers, creative writers, and storytellers.

Figure 4 displays the t-distributed stochastic neighbor embedding (t-SNE) visualization of the LDA results [33]. t-SNE is a dimensionality reduction algorithm that displays high-dimensional data in two dimensions while preserving the most significant structure.

4. Discussion and Conclusions

This research fills the gap in the literature by investigating the general use cases of advanced AI chatbot technology. This was achieved by systematically analyzing and summarizing early conversations about the innovative AI chatbot, ChatGPT on Twitter. During the first month of its launch, journalists and experts shared a variety of ideas and suggestions. However, a systematic analysis of early, dynamic, and crowdsourced data on this topic does not yet exist. This research fills this gap by employing LDA topic modeling on 233,914 English tweets containing the word “ChatGPT”. The study presents three main domains of conversation: general topics, functional domains, and potential impacts. The five topics in the functional domain—creative writing, essay writing, prompt writing, code writing, and answering questions—are the comprehensive possible use cases of ChatGPT based on the crowdsourced knowledge of Twitter users. However, arguably even more important are the potentially serious impacts on tech and humans, which are topics that require further exploration in the future. Four key issues are identified based on the findings of this study.

The next evolution of jobs. The next evolution of jobs will likely be impacted by ChatGPT and other innovative AI technologies. This evolution has the potential to benefit people, but it also poses risks, such as job displacement for white-collar and creative professionals. Furthermore, the future seems to be approaching faster than anticipated. Even “safe” professions, such as AI coders, trainers, and analysts [34,35], may be at risk of becoming replaced by ChatGPT. As the world rapidly advances and new technologies emerge, it is critical for leaders in both the public and business sectors to contemplate the future of the job market and its potential transformations brought about by advancements such as ChatGPT. The answers to these questions will have a significant impact on the characteristics and trends of the job market. Equally important is the consideration by the education sector on how best to prepare the next generation for this rapidly changing landscape. This requires the transformation of pedagogical approaches and a deep reflection on the skills and knowledge necessary for success in the new era.

The new technological landscape. In 2022, the two most renowned technological frameworks, Web3 and Metaverse, encountered major roadblocks; the Luna crypto and FTX crash impacted Web3’s blockchain trajectory, and the decline of Meta (Facebook) impacted the future of Metaverse. ChatGPT emerged into the tech scene and received a mostly positive reception from the Twitter community in its first week. It remains to be seen how ChatGPT and its related technologies can challenge dominant players such as Google, who have recently announced a competing technology called “Bard” [36]. However, as found in this research, the use cases of ChatGPT could compete in many sectors, including search (Google), personal assistant (Alexa and Siri), personalized entertainment, and software as a service (SaaS). The digital era has brought about a rapid evolution in technology, and the introduction of ChatGPT has the potential to greatly shape the future technological and digital landscape. While it remains uncertain just how deeply AI technologies such as ChatGPT will become integrated into our lives, those who are able to anticipate the trends and adapt proactively are poised to reap significant benefits.

The quest for artificial general intelligence. ChatGPT has undoubtedly sparked further discussions about AGI, or what Goertzel [37] referred to as “human-level AGI”. Despite its creative writing skills and advanced model, ChatGPT, which is created from a large language model (LLM), has not achieved the level of AGI. It has not achieved many fundamental components of AGI, such as self/social, emotion, motivation, meta-cognitive, and deliberative processes [see 37]. However, it has undeniably demonstrated another step towards AGI. Moving forward, ChatGPT and other similar technologies face numerous challenges and opportunities for growth as they strive towards AGI. These include improving the quality of their responses, incorporating new technical advancements in models, and developing new features that can expand and deepen their application in a wider range of use cases.

The ethics-progress conundrum. The discussion about ethics and AI has been ongoing, with many globally convergent ideas such as transparency, justice, and fairness. However, divergence in the interpretation of these principles remains substantial [38]. Unfortunately, the development and implementation of AI ethics have not been successful and are often considered an extraneous, unbinding framework imposed from institutions outside the tech sector [39] despite calls for ethical frameworks and for AI chatbots in education and healthcare [3,7]. However, it is necessary to accelerate the debates and address the tension between technological progress and ethical concerns. The potential positive and negative impacts of ChatGPT are immediate and wide-ranging, emphasizing the necessity for a comprehensive examination and collective formation of ethical guidelines for its use in all fields. In scientific research, the esteemed scientific journal “Nature” recently released principles regarding the use of large language model (LLM) AI in scientific papers [40].

Due to the recentness and novelty of the technology, the findings of this research seem to raise more questions than they provide answers. Indeed, ChatGPT could answer (and has answered) these questions with ease, but as our findings suggest, we cannot rely on its answers fully.