Analysis of Reliability and Efficiency of Information Extraction Using AI-Based Chatbot: The More-for-Less Paradox

This paper addresses the problem of information extraction using an AI-powered chatbot. The problem concerns searching and extracting relevant information from large databases in response to a human user’s query. Expanding the traditional discrete search problem well known in operations research, this problem introduces two players; the first player—an AI chatbot such as ChatGPT—sequentially scans available datasets to find an appropriate answer to a given query, while the second—a human user—conducts a dialogue with the chatbot and evaluates its answers in each round of the dialogue. The goal of an AI-powered chatbot is to provide maximally useful and accurate information. During a natural language conversation between a human user and an AI, the human user can modify and refine queries until s/he is satisfied with the chatbot’s output. We analyze two key characteristics of human–AI interaction: search reliability and efficiency. Search reliability is defined as the ability of a robot to understand user queries and provide correct answers; it is measured by the frequency (probability) of correct answers. Search efficiency of a chatbot indicates how accurate and relevant the information returned by the chatbot is; it is measured by the satisfaction level a human user receives for a correct answer. An AI chatbot must perform a sequence of scans over the given databases and continue searching until the human user declares, in some round, that the target has been found. Assuming that the chatbot is not completely reliable, each database may have to be scanned infinitely often; in this case, the objective of the problem is to determine a search policy for finding the optimal sequence of chatbot scans that maximizes the expected user satisfaction over an infinite time horizon. Along with these results, we found a counterintuitive relationship between AI chatbot reliability and search performance: under sufficiently general conditions, a less reliable AI chatbot may have higher expected search efficiency; this phenomenon aligns with other well-known “more-for-less” paradoxes. Finally, we discussed the underlying mechanism of this paradox.