Review Reports - Bibliometric Overview of ChatGPT: New Perspectives in Social Sciences

Round 1

Reviewer 1 Report (Previous Reviewer 1)

Comments and Suggestions for Authors

Dear Authors,

I recognize that the authors have made several improvements to the manuscript in the resubmitted version. However, I still think there is room to develop the paper further and polish it more. See comments in the box below above language and flow.

Specific comments:

Title: What do you mean by "new perspectives"?

I would consider incorporating some more recent bibliometric analyses of ChatGPT. These studies have all been published in 2024: I recognize that you cite some bibliometric studies on line 89.

Zheltukhina, M. R., et al. "A bibliometric analysis of publications on ChatGPT in education: Research patterns and topics." Online Journal of Communication and Media Technologies 14.1 (2024): e202405.

Farhat, Faiza, et al. "The scholarly footprint of ChatGPT: a bibliometric analysis of the early outbreak phase." Frontiers in Artificial Intelligence 6 (2023).

Liu, Jun, et al. "A bibliometric analysis of generative AI in education: current status and development." Asia Pacific Journal of Education (2024): 1-20.

Good luck!

Comments on the Quality of English Language

I still think the language and flow could be improved. However, I leave to the editors to decide the extent to which this is necessary.

Author Response

Firstly, we would like to express our sincere thanks for your suggestions. Below, we have included comments on the changes that have been made in the article, as well as possible responses to any doubts arising in connection with the substantive and technical construction of the text.

Fragment of the article

Reply and comment

Title

The title "New Perspectives in Social Sciences" highlights the dual nature of ChatGPT as a tool that opens new possibilities and presents challenges in this field. Our article responds to the growing interest in the topic of using chatbots, based on natural language analysis, in social sciences. At the same time, our considerations aim to provide guidance for further research in this area, highlighting aspects that should be taken into account.

Introduction

Following your recommendations, we used the following publications in our article:

1. Zheltukhina, M. R.; Sergeeva, O. V.; Masalimova, A. R.; Budkevich, R. L.; Kosarenko, N. N.; Nesterov, G. V. A bibliometric analysis of publications on ChatGPT in education: Research patterns and topics. Online Journal of Communication and Media Technologies 2024, 11 (1), e202405. https://doi.org/10.30935/ojcmt/14103.

2. Farhat, F.; Silva, S. E.; Hassani, H.; Madsen, D. Ø.; Sohail, S. S.; Himeur, Y.; Alam, M. A.; Zafar, A. The scholarly footprint of ChatGPT: a bibliometric analysis of the early outbreak phase. Frontiers in Artificial Intelligence 2024, 6. https://doi.org/10.3389/frai.2023.1270749.

3. Liu, J.; Wang, C.; Liu, Z.; Gao, M.; Xu, Y.; Chen, J.; Cheng, Y. A bibliometric analysis of generative AI in education: current status and development. Asia Pacific Journal of Education 2024. https://doi.org/10.1080/02188791.2024.2305170.

Reviewer 2 Report (Previous Reviewer 2)

Comments and Suggestions for Authors

The paper presents a bibliometric overview of ChatGPT usage in the domain of social sciences with an emphasisis on ethical, technological, and sociological aspects in mind. I think that this resubmitted version of the paper is much more comprehensive, focused, and matured. Although still lacking some more traditional bibliometric indicators, it covers the landscape of ChatGPT usage in social sciences pretty well, also supported by the discussion section.

Lines 47-53: Added text gives deeper insight into the role of AI in scientific research. However, it should be supported with some references and examples.

Lines 79-87. Organization of the paper should go to the end of the Introduction section.

Fig. 3 caption is not in its place below the Fig. 3.

Table 2. Are those WoS impact factors? From which year? There is nothing about it mentioned in the text.

Author Response

Fragment of the article

Reply and comment

Introduction

In lines 47-53, following your recommendations, more publications were added that provide the theoretical basis to support the conclusions drawn.

The paragraph located in lines 79-87 was moved to the end of the introduction, which improved the overall structure of the text.

Results

Technical issues related to Figure 3 were corrected.

The text now includes information about the Impact Factor indicators downloaded from WoS for the year 2023.

Reviewer 3 Report (New Reviewer)

Comments and Suggestions for Authors

Abstract:

I consider missing:

- Highlighting one or two key insights or surprising trends could make the abstract more compelling and underscore the study's value.

- A brief mention of how these insights contribute to the field or could influence future research, policy, or practice would add depth.

Keywords:

Increase to at least 6 keywords.

Introduction:

Nothing to comment

2.1.1. Collection of bibliographic data on.

- What was the keyword for the research on Scopus? What was the keyword criteria selection? Briefly mention them. Table 1 is already giving more details.

2.1.3. Transformation and statistical analysis of the data

- Meaning of VOSviewer?

2.2. Literature review protocol

- Explain why the source was limited to journals.

2.3. Classification of publications

- As a recommendation, the groups can fit into a table for a better reading. But in enumerations is also good.

3.1. Content analysis

- Why more than 4 citations is green? I mean, what are the criteria behind it? Is there any trend in social sciences that a paper over 4 citations within the first year considers relevant? If there are no criteria, Figures 1 and 2 are redundant.

- The other category and their references should be removed. It is not relevant.

3.2 Analysis of citations

- During the review of this paper, maybe Figure 2 and Table 2 should update the values.

Additionally, the caption of the table should say: The 20-most frequently cited.......

Page 10

- The Figure 3 is between the text. Edition mistake.

- The figure 3 Looks a bit messy. Should be good to use another kind of Figure. It is a lot of data and relationship among them. That's why looks like that. Another graphic could fix this problem.

3.3. Analysis of the source of articles

- Should be Table 3 not 2. and Table 3 should be 4.

- In this table specify the year of the IF report.

3.4. Analysis of keywords

- Reduce the keyterms to the most relevant. Otherwise the graph look messy.

Discussion / Conclussion:

Where exactly are you answering the research questions? This most be the core of the discussion/conclusions

Author Response

Fragment of the article	Reply and comment
Abstract	Following your suggestions, the abstract has been enriched with additional insights and emerging trends identified in the study. It also includes a brief mention of the recommendations.
Keywords	The number of keywords in the article has been expanded from 3 to 6.
Materials and Methods	In the paragraph under Table 1, the process of selecting scientific articles was commented on, justifying the choice of the keyword "ChatGPT" and the exclusive selection of scientific articles. The research focused solely on scientific articles due to their recency and decisive advantage in the structure of scientific publication resources on this topic. Scientific articles allow for precise identification of citations and co-citations, due to their indexing in databases, which was very important given the nature of the conducted research. Subchapter titles were corrected, following your recommendations, to be more appropriate.
Results	Articles that were cited four times were the most numerous compared to the citation count of other scientific works, excluding those not cited at all. Therefore, it was decided to present a chart showing which areas of scientific research attract the most interest, measured by the number of citations. This is a key observation, as the field with the most publications is not necessarily the field that is most frequently cited. The 'Other' category was removed, allowing the reader to focus on the most important elements of the study. The name of Table 2 was changed to include the number of the most frequently cited scientific articles. Technical errors related to the improper placement of figures and their presentation were corrected, improving the overall structure of the text and the visual quality of the article.
Discussion	The paragraph in the first subsection was expanded, focusing on answering the research questions posed.

Round 2

Reviewer 1 Report (Previous Reviewer 1)

Comments and Suggestions for Authors

I would like to the authors for taking the reviewer comments seriously.

There is a small typo in the title (the word "title" is probably unnecessary).

Comments on the Quality of English Language

Some edits and language improvements would strengthen the paper.

Author Response

Thank you for your suggestions. The manuscript has been edited.

Reviewer 3 Report (New Reviewer)

Comments and Suggestions for Authors

I really appreciate that you followed my recommendations.

I think the paper is suitable for publication now.

Author Response

Thank you.

This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The subject of the manuscript (ChatGPT) is notably relevant and will likely engage the journal's audience. However, significant revisions are required for it to meet the publication standards. The following are some constructive suggestions for improving the manuscript in a future revision:

Title: While succinct, exploring alternative titles may benefit the manuscript.

Abstract: Though brief, the abstract should be revised for improved clarity and reader engagement.

Introduction: Strengthen the introduction by articulating the study's contribution and value more clearly. The final paragraph should outline the article's structure. Clarify the term "field of social science" and its usage in this context.

Literature Review: Include references to other bibliometric studies focusing on different facets of ChatGPT. For example, one was published just yesterday: The beginning of ChatGPT – a systematic and bibliometric review of the literature | Emerald Insight. In addition, there are several other bibliometric papers on ChatGPT that you could cite. Make an argument for how your bibliometric study differs from these other bibliometric reviews.

Materials and Methods:

Discuss in more detail the advantages and disadvantages of utilizing Scopus data.

Compare and contrast the use of VOSviewer bibliometric software with other alternatives, such as Bibliometrix, highlighting their strengths and limitations.

Information in lines 144-151 could be more effectively presented in a table.

Subsections should be numerically labeled (in my opinion)

The manuscript seems to omit the data extraction date from Scopus (or I somehow missed it), which is crucial given the rapid growth of ChatGPT-related publications in 2023.

Results (Section 3):

This section appears lengthy and could benefit from better structuring. Consider dividing it into two parts, with a separate, more detailed section for content analysis.

Elaborate on the methodology used for the content analysis of abstracts. Also, consider presenting these results in a table.

Expand on the analysis of the most cited publications, providing more details about these articles, their content, and the journals in which they are published. This could be included in Table 1.

Discussion (Section 4): The manuscript currently lacks a Section 4 (the authors jump from 3.4 to 5 Conclusion), which could be a discussion section. In this new section, discuss the findings of the bibliometric analysis in greater detail.

Conclusion (Section 5): You should also work on strengthening this section.

Minor Issues:

There is a missing word in the caption of Figure 3 ("... ChatGPT in social").

Inconsistencies in font sizes are noted, particularly in section 3.4.

These recommendations are intended to enhance the manuscript's overall quality and readability.

Comments on the Quality of English Language

The manuscript needs polishing and proofreading.

Reviewer 2 Report

Comments and Suggestions for Authors

The paper presents a study on usage of ChatGPT in social sciences through bibliometrics analysis. Based on Scopus index database, the authors provide an overview of papers published in recent period (year 2023) with an emphasisis on the use of ChatGPT in social sciences and the way it (negatively) affects the modern science.

Abstract of the paper provides an overview. However, I would like to see a more focused research hypotesis and results in the abstract. It is stated that "ChatGPT is considered by certain experts to pose a threat to contemporary science". No conclusion on such bold state is mentioned, both in the abstract or in the conclusion.

Introduction line 50: "Numerous studies have suggested that the implementation of artificial intelligence can revolutionise the process of scientific discovery, significantly affecting the structural organisation of scientific disciplines". It is correct, but still I would like to see some elaboration on how utilization of ChatGPT can can revolutionise the process of scientific discovery. Please add some more motivation.

Some contributions of the paper are given in lines 66-80. However, some of them are more specific and results of the study, while others, such as "to promote the ethical use of AI" are long way to go. Please, state bullet-by-bullet the obvious contributions and novelty of the paper, separately from the possible long term effects.

Section 3.1. Before content analysis of thematic categories, I would like to see a table with statistical descriptions of the papers to have some overview. Number of papers per identified thematic category, average authors per paper - normal and fractional analysis, total citations per group, and similar.

Section 3.1. Give some concise examples of "Other" category beside just references. In general, for each category you can add a sentence on the topic of each paper mentioned.

Table 2. It would be beneficial to add WoS impact factors of the journals, if they have.

Fig. 3 is nice. However, I would also advise to present a co-authorship network and analyze it. It would be nice to check how well-connected are authors in the field for a such a short period. Are there some prominent communities or the are is still in its infancy? If you are familiar with social network analysis methods, they can be also used to verify some of the hypotesis and reveal prominent authors, both i co-authorship and co-citation networks.

For most of the topics and conclusions, you present up to three references from the obtained dataset. However, how did you chose them? Are they the most cited ones, the earliest or you used some other criteria? For example: "This 332 journal has published articles by authors such as Hayward [84], Henley [85] and Bugaj et al. [86].". Please explain.

In Table 3 and the following explanation, you presented the countries of origin. Notable is the presence of the developed countries. You mentioned that China and Indida has a growing role. However, in other scientometrics studies, those countries are usually the leading ones, especially China, together with USA. Please give us some context and compare findings in this field to other scientific fields. Maybe there are some ChatGPT studies for technology field? You can compare also to other, non-ChatGPT studies as well.

Generally, the paper is nicely written with a lot of data and conclusions. However, a proper discussion and comparison with other studies and fields is needed before the conclusion. We need to put this study in the wider context, scope of the bibliometrics and scientometrics research.

Technical remarks
----------------------

The references in the paper are well-prepared with a lot of recent ones. It might be even a too many of them, but considering this being a bibliometrics/scientometrics study, it is understandable.

English is correct, easy to follow.

Section 3.1. numbered list is not justified by both margins, but only left justified.

Table 2. Caption is incomplete.