Analysis of Reliability and Efficiency of Information Extraction Using AI-Based Chatbot: The More-for-Less Paradox

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Dear Authors

The paper titled “Analysis of the Reliability and Efficiency of Information Extraction  Using AI-Based Chatbot: The More for Less Paradox” addresses important issues however it needs more improvements as follows.

1. Revise the abstract to clarify the problem statement.
2. What is the novelty of this investigation?
3. Provide more details on the dataset used for validation.
4. Explain the choice of the 10-point Likert scale for measuring satisfaction.
5. Clarify the "more-for-less" paradox with simpler examples.
6. How generalizable are the findings to other AI chatbot models?
7. See and add Enhancing disaster response and public safety with advanced social media analytics and natural language processing; Cyberbullying detection using machine learning and deep learning.
8. Discuss potential biases in human user satisfaction ratings.
9. Provide more details on the iterative algorithm's computational complexity.
10. How does the discount factor impact the results?
11. Address limitations of the proposed index-based policy.
12. Compare the results with non-index-based search policies.
13. Discuss practical implications for AI chatbot developers.
14. How does the paradox manifest in real-world chatbot deployments?
15. Provide more insights into the failure cases of the algorithm.
16. Suggest future research directions to extend this work.

Author Response

Please find our responses attached. 

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

This paper presented an interesting study of AI chatbot search efficiency. The authors revealed a counterintuitive "more-for-less" paradox: under certain conditions, less reliable chatbots can yield higher user satisfaction.

The authors model the chatbot interaction as a sequential discrete search problem and developed an optimal index-based search policy.

The methodology is sound and supported by detailed simulations using numerical examples.

Real world observations or validations of the "more-for-less" paradox in practical AI chatbot systems would enhance this study.

Author Response

Please find our responses attached. 

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

The paper presented a study of the reliability and efficiency of information extraction using AI chatbots. Generally, the paper is well-written, has rich content, and covers many aspects. Here are the strengths of the paper:

1. Sections 1 and 2 provide enough information on the discrete search problems related to information extraction problems using AI chatbots, as well as a brief summary of this research and the paper's contributions.
2. Section 3 mathematically formulated the AI-based search problem with an efficiency-maximizing objective and compared it with discrete search problems. Then, the authors summarized related works using index-based algorithms for discrete search problems and introduced the paradoxes in Operations Research.
3. Sections 5 and 6 discussed the proposed methodologies in this research, showing how to find the optimal policy for AI-based search problems. The authors explained the propositions, the pseudocode, and the algorithm steps very well.
4. In section 7, the authors presented the collected results of numerical examples to support the hypothesis mentioned in previous sections.
5. In section 8, the authors summarized the research problems, discussed the research outcome, and suggested further research ideas.

However, there are some major and minor problems which should be addressed:

1. This research problem was formulated similarly to the Markov Decision Process, the foundation for Reinforcement Learning. The formation of finding an optimal policy for AI-based search problems is very similar to reinforcement learning. Recently, many Large Language Model (LLM) based chatbots, like OpenAI ChatGPT, have been trained with Reinforcement Learning from Human Feedback (RLHF). Such methodologies help improve search efficiency, reliability, and user experiences, as well as deal with ethical issues or censor “sensitive” topics. These bring a huge question about the research methodology and the contributions of this research to the domain. What contributions does this research make compared to RLHF in training LLMs or AI-based chatbots? How significant are they?
2. In section 7, the authors presented the numerical examples. It is not clear where the input data in Table 2 comes from. Were they randomly generated? The experimental setup and configurations should be provided in more detail for verification purposes, such as the details of two chatbots, the related search queries and responses of the mentioned 12 consecutive rounds, the experiment environment, and related tools. More information should be provided about how many experiments were conducted, how many runs were needed for each experiment, and/or hyperparameter settings.
3. Literature Review: The authors should review related works using reinforcement learning in developing AI-based chatbots or related LLMs.

Comments for author File: Comments.pdf

Comments on the Quality of English Language

1. Many places in this paper should be improved in written English or formats. Here are some common problems:

- Use acronyms without full spelling the first time they were introduced in the manuscripts, such as AI (in the abstract), PRM (line 111), OR (line 259),

- Missing articles, such as a/the, or punctuations between sentence phrases

- Verb tenses.

- Singular or plural words.

Please check the attached PDF file with the detailed highlighted comments to improve the writing of this article.

Author Response

Please find our responses attached. 

Author Response File: Author Response.pdf

Reviewer 4 Report

Comments and Suggestions for Authors

The study is devoted to solving an operations research problem, which consists in finding the optimal strategy for searching for a target among N objects. The problem proposed for consideration in the manuscript is an extension of the (well-known in operations research) discrete search problem. Unlike the traditional formulation, the proposed setup involves not one, but two players: the first performs the search, while the second evaluates the first player's performance at each search iteration.

In the paper, the first player is referred to as an AI-powered chatbot, and the second as the user, who attempts to find their target through a dialogue with the chatbot, rating its response in each round on a ten-point scale.

I found the study quite interesting from the perspective of operations research theory. However, the chatbot's behavior in the search process described in the paper seems to differ from how real-world chatbots operate.

The introduction, abstract, and keywords reflect the content of the work to some extent, but they are somewhat misleading. The issue is that they highlight artificial intelligence and AI-chat bots as the main research focus of the article. However, an expert in AI and search engine technologies will find only an operations research problem in the manuscript. Conversely, an expert in operations research might overlook this work, as it cannot be found via keyword, title, or abstract searches in relevant databases.

In this regard, I believe the authors should more clearly define their target audience and revise the abstract and keywords accordingly.

I would also note that although the literature review is fairly comprehensive, it focuses exclusively on works related to operations research and does not cover related issues in intelligent search.

The authors proposed a search concept using a chatbot and presented an algorithm to solve the corresponding problem in an abstract setting. In the empirical part of the study, the authors test the algorithm on synthetic data (an abstract task). I believe it is fine for now.

It would be appreciated to receive a comment on the computational complexity of the problem considered in the study.

Overall, I found the study interesting. The paper contains novel and significant results.

I would recommend that the authors slightly modify the manuscript based on these comments.

Author Response

You can find the answers attached. 

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

The manuscript shows some ethical considerations need to be addressed.

Author Response

Please find our response attached. 

Author Response File: Author Response.docx

Reviewer 3 Report

Comments and Suggestions for Authors

Compared to the previous version, this revision shows many improvements:
a. The abstract has been improved to clarify the problem settings and research methodologies. However, it is quite long and should be shortened. 
b. The paper's contributions have been revised and clarified in the abstract, introduction, and conclusion.
c. The “more-or-less” paradox examples have been revised in section 7, and future research directions have been included in the conclusion section.
d. Include the time complexity analysis of the proposed iterative algorithm.
e. Improvements in written language and/or figures have been made.
However, two minor problems still can be improved: 
f. I am still not fully convinced of the numerical examples in section 7, as the data collected from sensors in your lab may not reflect the possible data derived from similar experiments on real AI-powered chatbots. Either the authors justify that they are in the same data distribution or acknowledge the limitations in the discussion/conclusion. 
g. The authors should acknowledge different limitations of this research in the conclusion: a systematic comparison of the behavior of real-world AI-powered chatbots, potential biases in human user satisfaction ratings, and other issues that are out of the scope of this study.

Author Response

Please find our response attached.

Author Response File: Author Response.docx

Round 3

Reviewer 1 Report

Comments and Suggestions for Authors

 The authors have failed to address critical queries from my review.

Author Response

You can find the answers attached. 

Author Response File: Author Response.docx