Tailoring Scientific Knowledge: How Generative AI Personalizes Academic Reading Experiences

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

This research intends to show how generative AI can dynamically personalize scholarly content by tailoring summaries and key takeaways to individual user profiles.

The paper is well-written and clear.

However, I cannot consider it as a research paper because of the following reasons:

1. The literature review does not show an in-depth study of similar works. If there is none, it should properly be justified. In the absence of similar studies, the closest works to this one should be mentioned.
2. The method should come from the results of point 1, which, in turn, should clearly show the gaps in the current research. As an example, based on what the person selection was decided?
3. The number of samples and participants is not enough to generalize the outcomes. That could be the reason for a lack of thorough discussion of the results.
4. There are some repetitions in many sections that could be avoided.

Therefore, in its current form, I consider this paper more of a letter or a review, not a research that could be generalized. However, I find the idea attractive and even novel. For that, I appreciate the work.

 

Author Response

The response to the review can be found in the attached file.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

The study effectively offers meaningful insights into AI's potential role in scholarly publishing.

Strengths

Introduction:
- The introduction effectively outlines the problem of information overload in academic research and presents a compelling case for the need for personalized content.
- The background references relevant literature to establish the existing gap in personalized academic content, creating a strong probable cause for the paper's objectives.

Methodology:
- The detailed persona creation process adds realism and ensures that AI-generated content aligns with distinct reader profiles.
- The use of structured AI prompts to curate persona-specific content provides clear implementation flow.
- The use of two distinct professional personas effectively highlights the flexibility of generative AI models, as well as helps in testing its scale of flexibility.

Results:
- The comparison between persona-specific outputs clearly illustrates how AI-driven content modification can improve user-experience and understanding and comprehension.
- The Likert scale analysis is particularly valuable in providing a quantifiable measure of content relevance and engagement to the different personas.

Discussion:
- The exploration of the ethical implications and risks associated with AI-driven content modification, as well as lapses and biases that AI's sometimes have, demonstrates consideration of potential challenges.
- The emphasis on developing dynamic persona profiles to enhance content personalization shows a forward-thinking approach.

Areas for Improvement

Introduction:
- While the introduction presents a strong argument for AI-driven personalization, the section could benefit from additional examples showcasing the limitations of traditional content filtering methods to better highlight the study's novelty. A larger dataset of papers could have been used. 

Methodology:
- The description of the selected academic articles is somewhat brief. Expanding this section with clearer justifications for their inclusion could add context to the dataset's relevance.
- While the AI prompt design is explained well, more technical details on how GPT-4o was adapted for persona-specific customization would improve replicability for future researchers.

Results:
- A dive into the time it took the model to respond and how many trials were done for the final result could have enhanced the paper's novelty. 
- The paper does not clearly explain potential limitations in the AI's performance, such as misinterpretations of content relevance or biases in highlighting certain themes. It does not take into account AI lapses and errors. 

Discussion and Conclusion:
- While the AI performs accurately for extreme cases, for fields which are in between the two personas, the results and discussion are a bit loose. 
- The implications for improving interdisciplinary research access are valuable but could be expanded with specific examples of how these findings could shape academic publishing practices.

The paper is well-structured and easy to follow, with clear language and concise presentation of ideas. While the ethical discussion is strong, the paper could further address the potential misuse of AI-driven content modification, and point out the various biases and lapses in judgement/understanding AI has with large input or a variety of input.

Author Response

The response to the review can be found in the attached file.

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

The author explored whether AI could adjust the presentation of scientific articles based on the distinct profiles of a researcher from academia and a specialist from the industry. To do this, they used prompt engineering to generate personalized abstracts and content highlights, and then evaluated the effectiveness of these AI-generated content modifications in enhancing readability, comprehension, and relevance for different audiences. In a larger sense, the paper identifies a relevant gap in the application of generative AI, specifically for real-time, reader-specific customization of scientific content, which presents a degree of novelty. It moves beyond existing AI applications that primarily focus on summarization and search optimization by aiming to dynamically adjust content presentation based on user expertise and preferences. However, there are a few areas of concern:

- The paper's discussion of limitations is weak due to its reliance on generic challenges like content oversimplification and bias, rather than addressing specific shortcomings of the study's methodology. This lack of specificity hinders the reader's understanding of the results' scope and generalizability. The choice of only two personas and the use of a single LLM (ChatGPT-4) also could be seen as limiting factors in the methodology's robustness and generalizability. Consequently, the paper misses an opportunity to provide a more rigorous and insightful evaluation of its research.

- The tables and listings are difficult to read due to small font size.

- The text-heavy description of the methodology makes it difficult for the reader to quickly grasp the sequence of steps involved. It's hard to visualize the process of persona creation, prompt engineering, and evaluation. A flowchart or diagram could clearly illustrate the entire research process, from article selection and persona definition to prompt engineering and output evaluation. This would provide a high-level overview and improve comprehension.

- The paper doesn't mention how readability, comprehension, and relevance were measured. Were they using quantitative metrics (e.g., readability scores, comprehension quizzes)? Or was it qualitative (e.g., user surveys, expert reviews)? Without clear metrics or methods, any assessment of "readability, comprehension, and relevance" becomes highly subjective. It's difficult to see how the authors could have arrived at any objective conclusions.

To put it altogether, this paper successfully presents the idea that generative AI could be used to personalize scientific content. It highlights the potential benefits and proposes a general approach. However, it falls short of providing strong evidence that this personalization actually leads to measurable improvements in readability, comprehension, or relevance. The weak evaluation doesn't support the claims of effectiveness. So, while the conceptual aspect of the paper has merit, it needs that "additional stuff" – the rigor, the evidence – to become a more convincing and valuable contribution to the field.

Comments on the Quality of English Language

The author explored whether AI could adjust the presentation of scientific articles based on the distinct profiles of a researcher from academia and a specialist from the industry. To do this, they used prompt engineering to generate personalized abstracts and content highlights, and then evaluated the effectiveness of these AI-generated content modifications in enhancing readability, comprehension, and relevance for different audiences. In a larger sense, the paper identifies a relevant gap in the application of generative AI, specifically for real-time, reader-specific customization of scientific content, which presents a degree of novelty. It moves beyond existing AI applications that primarily focus on summarization and search optimization by aiming to dynamically adjust content presentation based on user expertise and preferences. However, there are a few areas of concern:

• The paper's discussion of limitations is weak due to its reliance on generic challenges like content oversimplification and bias, rather than addressing specific shortcomings of the study's methodology. This lack of specificity hinders the reader's understanding of the results' scope and generalizability. The choice of only two personas and the use of a single LLM (ChatGPT-4) also could be seen as limiting factors in the methodology's robustness and generalizability. Consequently, the paper misses an opportunity to provide a more rigorous and insightful evaluation of its research.
• The tables and listings are difficult to read due to small font size.
• The text-heavy description of the methodology makes it difficult for the reader to quickly grasp the sequence of steps involved. It's hard to visualize the process of persona creation, prompt engineering, and evaluation. A flowchart or diagram could clearly illustrate the entire research process, from article selection and persona definition to prompt engineering and output evaluation. This would provide a high-level overview and improve comprehension.
• The paper doesn't mention how readability, comprehension, and relevance were measured. Were they using quantitative metrics (e.g., readability scores, comprehension quizzes)? Or was it qualitative (e.g., user surveys, expert reviews)? Without clear metrics or methods, any assessment of "readability, comprehension, and relevance" becomes highly subjective. It's difficult to see how the authors could have arrived at any objective conclusions. To put it altogether, this paper successfully presents the idea that generative AI could be used to personalize scientific content. It highlights the potential benefits and proposes a general approach. However, it falls short of providing strong evidence that this personalization actually leads to measurable improvements in readability, comprehension, or relevance. The weak evaluation and limited scope severely restrict the generalizability of the findings. So, while the conceptual aspect of the paper has merit, it needs greater methodological rigor, more diverse testing conditions, and objective evaluation metrics to become a more convincing and valuable contribution to the field.

Author Response

The response to the review can be found in the attached file.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

Thank you for the update.

My concerns remain as they were.

Reviewer 3 Report

Comments and Suggestions for Authors

Thank you for improving the manuscript.  Your study introduces an interesting and timely direction on personalized reading using LLMs, but I encourage you to conduct a more rigorous investigation in future work.