DietNerd: A Nutrition Question-Answering System That Summarizes and Evaluates Peer-Reviewed Scientific Articles

Abstract

DietNerd is a large language model-based system designed to enhance public health education in diet and nutrition. The system responds to user questions with concise, evidence-based summaries and assesses the quality and potential biases of cited research. This paper describes the system’s workflow, back-end implementation, and the prompts used. Accuracy and quality-of-response results are presented based on an automated comparison against systematic surveys and against the responses of similar state-of-the-art systems through human feedback from registered dietitians. DietNerd is among the highest-evaluated of these systems and is unique in combining safety features with sophisticated source analysis. Thus, DietNerd could be a tool to bridge the gap between complex scientific literature and public understanding.

1. Introduction

There is growing evidence of the close relationship between diet and health, on both a physical [1] and a mental health [2] level. As a result of that research and similar assertions by the popular media, many non-scientists search for dietary health best practices and craft a diet plan based on their health needs. A 2016 study involving 42,113 French adults showed that 85.1% used the internet for health and nutrition information [3]. The main problem with this approach is that there is a great deal of false or biased information in the media and on the web. That false information also infects large language models (LLMs) such as ChatGPT [4]. Moreover, given the ever-evolving nature of nutrition and dietary research, widely-held beliefs are continuously being upended by new discoveries.

Thus, there is a need for a consumer-friendly question-answering (Q&A) tool that leverages peer-reviewed research, offers individual article analysis, and prioritizes safety. That is what DietNerd provides. The contributions of this research are:

• A combined parsing/prompt pipeline that takes a question, fetches relevant articles, and summarizes them.
• An analysis of each relevant article that extracts the purpose, the design of the experiment, the main conclusion, risks and benefits of the dietary change tested, study statistics, and any possible conflicts of interest.
• Accuracy tests based on systematic surveys that show a high semantic similarity of the responses of DietNerd with those surveys for 64% of the articles and moderate semantic similarity for the rest.
• User tests with registered dietitians to assess the clarity, accuracy, and completeness of our summary responses compared to state-of-the-art systems show that DietNerd ranked among the best in the state of the art.
• DietNerd is unique among state-of-the-art systems in offering a safety analysis, showing the pros and cons of a diet intervention, based on peer-reviewed research.

DietNerd in Action

DietNerd can be accessed at https://dietnerd.org/ When the user enters a question on the landing page, as shown in panel A of Figure 1, the system will suggest the top three most similar questions that DietNerd has previously answered (panel B) as well as an option to generate an answer to the submitted question. Assuming the user chooses the generate option, the system gives status updates in the course of answering the question (panel C) and then generates a summary with citations (panel D). If the user selects a citation, the system displays an evaluation of the article (panel E). The actual article text can be accessed through the article summary page (panel F).

2. Related Work

Various LLM-powered medical question-answering tools have emerged in recent years. These applications harness the generative capabilities of LLMs to provide increasingly sophisticated and context-aware interactions. As of the summer of 2024, the systems that are closest to DietNerd in functionality are Clinfo.ai, Consensus, OpenEvidence, and Scholar GPT. Each of these systems, as characterized in

Table 1. System Functionality Comparisons: DietNerd, OpenEvidence and Clinfo.ai are single-turn question-answering tools. Consensus and Scholar GPT offer a conversational feature, while single-turn approaches like DietNerd would require longer questions to achieve the same functionality. Additionally, DietNerd and Clinfo.ai offer article summaries that highlight research quality attributes. DietNerd’s major functional contributions have to do with (i) extracting information from full texts rather than abstracts alone, (ii) a Safety (risk/benefit) Analysis, and (iii) an in-depth article quality analysis that includes an analysis of statistics.

System	Full-Text Extraction (When Available)	Research Quality Analysis	Safety Analysis	Conversational	Multiple Databases Accessed
Clinfo.ai		✓			✓
Consensus		✓		✓	✓
DietNerd	✓	✓	✓
OpenEvidence					✓
Scholar GPT	✓			✓	✓

, are well-regarded and widely popular.

Clinfo.ai is an open source retrieval-augmented LLM developed by a research team at Stanford that answers medical questions using articles from either PubMed or Semantic Scholar, depending on user preference [5]. In addition to developing this tool, the team also developed PubMedRS-200, a medical question-answer dataset of 200 questions built off systematic reviews that can be openly leveraged for performance evaluation [6].

Consensus is a knowledge engine capable of performing various tasks related to scientific literature including article summarization, question answering, and reference searches [7]. Built on the Semantic Scholar database, it is fine-tuned to extract key takeaways from each paper [8]. Consensus employs vector and keyword searches against abstracts and titles while also considering citation count, citation velocity, study design, and publication date to highlight and rank relevant articles in response to user queries. It is available through the OpenAI GPT Marketplace, where it is one of the most used research GPTs, and offers a premium version through a paid subscription on its own in-browser platform.

Scholar GPT, available on the OpenAI GPT Marketplace, was developed by awesomegpts.ai [9]. With access to multiple research databases, Scholar GPT offers versatile capabilities such as scholarly article searches, access to papers and patents, critical reading and text summarization, data analysis, data visualization creation, and web page scraping. As of August 2024, it is the top featured research GPT. It should be noted that Scholar GPT typically requires explicit prompting to include resource hyperlinks when citing sources.

OpenEvidence is a medical search platform founded by Daniel Nadler and maintained by a collective of researchers and medical advisors from Harvard, MIT, Columbia, and Cornell [10]. It is designed to answer health professionals’ questions using research and studies. Relevant articles are displayed alongside icons indicating recent publication or top journal status. OpenEvidence was recently recognized as one of the top 100 AI companies globally [11] and is the first AI to have achieved a score exceeding 90% on the United States Medical Licensing Examination, surpassing performance of Google’s Med-PaLM 2, Anthropic’s Claude 2, ChatGPT, and GPT-4 [12].

Several other innovative systems have appeared as well. StatsPearls Semantic Search [13] and Elicit [14] are very comparable to Clinfo.ai performance [5]. Similarly, Scholar AI, accessible in the OpenAI GPT Marketplace or as a separately web-hosted tool, presents a well-developed approach to academic search and synthesis. Because its functionality bears resemblance to Scholar GPT and Consensus, both of which are included in our comparative evaluation, to avoid redundancy and maintain a streamlined analysis, we have chosen not to include Scholar AI in our direct comparisons, though we note its significant contributions to the field.

Enhancing LLM Performance. Two prominent techniques to enhance LLM performance and adaptability are in-context learning, which enables models to quickly adapt to new tasks based on provided examples, and Retrieval-Augmented Generation (RAG) systems, which combine the generative power of LLMs with external knowledge retrieval for more informed and accurate outputs.

The emergence of in-context learning techniques allow for adaption to new tasks without fine-tuning [15]. Introduced in 2020 with GPT-3, this approach enables LLMs to learn from a few examples provided in the input prompt, effectively performing few-shot learning [16]. In-context learning has shown remarkable versatility across various Natural Language Processing tasks, including question answering, text classification, and even code generation. Its strength lies in its flexibility and efficiency, because it eliminates the need for task-specific model architectures or extensive fine-tuning procedures. The LLMs presented in our work make extensive use of carefully crafted prompts and few-shot examples.

A popular method for developing reliable LLMs are Retrieval-Augmented Generation (RAG) systems. By combining LLMs with external knowledge bases, RAG addresses key limitations of traditional language models. Not only are retrieval-augmented LLMs able to access up-to-date and domain-specific information [17], but they can generate responses that are factually grounded. This was demonstrated in a 2024 study where the addition of a RAG pipeline to the LLM, without embedding or fine-tuning, resulted in an increase of 0.048 in its recall and an increase of 0.103 in its F1 score [18], indicating that the system became better at retrieving relevant and more reliable information. Furthermore, RAG systems offer transparency and interpretability because the sources of information can be traced and verified, which is especially important for machine learning and artificial intelligence applications in healthcare and medicine [19]. Our work follows the spirit of the RAG approach.

Generating Safe Outputs With the popularity and increasingly widespread adoption of LLMs, there has been an increased focus on effective safety measures. To illustrate, Google’s LaMDA adopts a multi-faceted safety strategy by combining fine-tuning with several additional layers of precaution that involve deploying classifiers to identify unsafe content, integrating a retrieval system for factual accuracy, and applying multi-turn safety checks to assess conversation-wide safety [20]. DeepMind’s Sparrow model merges rule-based constraints with information retrieval techniques to govern behavior, encouraging the citation of sources to minimize errors and enhance factual integrity [21]. IBM’s Project Debater, while not a traditional LLM, incorporates techniques such as argument mining, stance classification, and claim verification to attempt to promote balanced, safe, and accurate discourse [22]. Meta’s BlenderBot introduces a persona-based approach to safety, assigning the model traits that promote safety-conscious interactions [23]. Other notable strategies include Reinforcement Learning from Human Feedback [24], Constitutional AI [25], and adversarial training [26].

While these strategies predominantly rely on fine-tuning, there is a burgeoning interest in harnessing in-context learning for safety applications — an approach which entails careful prompt engineering among other techniques. Although in-context learning methods may not achieve the same level of consistency as fine-tuned models, they allow for the dynamic adaptation of safety measures without the need for retraining, thus offering a more accessible and affordable alternative [16].

DietNerd makes use of the in-context learning safety framework by chaining together several task-specific LLMs to produce safe responses in resource-constrained environments.

3. Materials and Methods

3.1. Algorithm Overview

While the details of the algorithm depend on specific prompts and database accesses, the basic steps are shown diagrammatically in Figure 2. Algorithm 1 shows the method in text, along with references to the corresponding sections for a detailed description of those steps.

Algorithm 1: DietNerdAlgorithm Text: Start from a user question Q, as also illustrated in Figure 2. The condition on previous question-answer pairs requires database access. The de novo generation of an answer entails several prompts as detailed in Section 3.2.2, Section 3.2.3, Section 3.2.5, Section 3.2.7, Section 3.2.8 and Section 3.2.9 and their corresponding appendices

3.2. System Design and Architecture

DietNerd utilizes a chain of LLMs integrated with the PubMed search index and two MySQL databases hosted on Amazon’s Relational Database Service in the workflow/decision tree of Figure 2. Our system makes use of both the zero-shot and few-shot reasoning capabilities of two OpenAI models (namely GPT-3.5-turbo-0125 and GPT-4-turbo) with the following temperature and nucleus sampling settings based on the task. For a more in-depth system architecture diagram, see Appendix D.

Nucleus Sampling Settings: For complex tasks that require a nuanced output, primarily text generation, GPT-4-turbo is used with a nucleus sampling of 1 (rich vocabulary). Otherwise, for categorization tasks with a finite set of output responses (e.g., “yes/no”), such as question relevance classification, GPT-3.5-turbo-0125 is used with a nucleus sampling set to 0.5 to restrict the diversity of words considered.

Temperature Settings: Tasks that require either a more creative output (e.g., text generation) or a nuanced interpretation of the input data (e.g., determining relevance of an article to a user’s question) have temperature settings greater than or equal to 0.5. Conversely, when the task outputs a finite set of responses or requires a simple copy-paste of the input text, the temperature setting is set close to 0. Specifically, the temperature is set to 0.2 for the question validity task and 0.1 for the full-text relevant section mapping task. See Appendix C for individual LLM implementation details.

DietNerd’s core LLM functionality was built on the OpenAI library, which facilitates interaction with advanced language models. Bio and metapub were implemented to interface with the PubMed API and retrieve biomedical literature. For information retrieval, versatile web scraping, and full-text extraction, we utilized libraries such as requests, BeautifulSoup, and PyMuPDF. Database connectivity and RAG capabilities were achieved through mysql-connector-python, with environment variables managed using python-dotenv.

3.2.1. Question Match against Question-Answer Database

When a user submits a question, DietNerd first performs a similarity search against questions in a MySQL database that stores question-answer pairs. The similarity search algorithm begins by retrieving all existing questions from the database. These questions, along with the new user query, are then converted into Term Frequency-Inverse Document Frequency (TF-IDF) vectors using scikit-learn’s TfidfVectorizer. The cosine similarity is then computed between the TF-IDF vector of the user’s question and the vectors of all stored questions. The top three most similar questions above a threshold are selected. As shown in Figure 2, if a match is found, the user has the option to select from the top three closest matches. Alternatively, the user can ask the system to generate an answer based on a scan of the literature.

3.2.2. Question Validity

When a user asks the system to generate an answer, DietNerd first passes the question through the Question Validity LLM, which utilizes few-shot prompts to determine if the user’s question is one that DietNerd can address. These few-shot prompts are designed to help the LLM discern nuances between acceptable and unacceptable queries. While DietNerd can recommend general diets that may be suitable for certain health conditions, it does not answer questions around recipe creation, in-depth meal prep, or questions that are asked on behalf of an animal. If a user question is found to be invalid based on these criteria, the output will either be “False—Recipe” or “False—Animal” and a canned response directs the user towards alternate resources. If a user question is valid, DietNerd will continue to the subsequent downstream tasks. See Appendix C.1 for implementation details.

3.2.3. Query Generation for Safe Responses

If the user question passes the validity test, DietNerd then generates five PubMed queries using two LLMs, each with their own few-shot prompting examples to inform PubMed-specific syntax and stylistic preferences (see Appendix C.2):

• General Query Generation LLM: Generates one PubMed query that represents the user’s question and is designed to retrieve articles that provide general context.
• Points of Contention Query Generation LLM: Generates the remaining four PubMed queries by identifying the top four controversial debates surrounding the topic(s) in question and creating a custom query per point of contention. These queries are intended to surface safety considerations, because controversy in the health domain mostly pertains to safety.

By leveraging a multi-faceted article search strategy, DietNerd considers a broader spectrum of perspectives, which helps prevent bias towards a single viewpoint. Furthermore, it establishes an additional layer of transparency by highlighting any safety concerns or ethical debates and explicitly acknowledging contentious issues derived from reliable sources.

For example, in

Table 2. For every submitted user question, five PubMed queries are generated—each of which captures a different perspective on the question’s topic. Each point of contention query identified includes a summary explaining the debate. Shown below is an example of the five queries that result from the user question: “Is it safe to take St. John’s Wort if you are already on medication for depression?”.

Query ID	Query Type	Summary	Query
1	General Query	-	(“St. John’s Wort” OR Hypericum) AND (“depression medication” OR antidepressants) AND safety
2	Point of Contention	Interaction of St. John’s Wort with Antidepressants The most debated argument is around the potential interaction of St. John’s Wort with antidepressants, particularly SSRIs. Some studies suggest a risk of serotonin syndrome, whereas others argue that the interaction might not be clinically significant.	(“St. John’s Wort” OR Hypericum) AND antidepressants AND interaction
3	Point of Contention	Efficacy of St. John’s Wort as a Complementary Treatment The debate revolves around whether St. John’s Wort can enhance the effects of standard antidepressants and improve treatment outcomes. While some studies suggest a potential benefit, others find no significant effect.	(“St. John’s Wort” OR Hypericum) AND antidepressants AND efficacy
4	Point of Contention	Safety of St. John’s Wort in Polypharmacy There is a contention about the safety of St. John’s Wort in patients who are on multiple medications. Concerns about potential drug interactions and side effects are discussed.	(“St. John’s Wort” OR Hypericum) AND polypharmacy AND safety
5	Point of Contention	Impact of St. John’s Wort on Antidepressant Withdrawal The debate is whether St. John’s Wort can help manage withdrawal symptoms from antidepressants. Some argue it may ease the process, while others caution against potential risks.	(“St. John’s Wort” OR Hypericum) AND antidepressants AND withdrawal

, queries two and three highlight areas where some argue there is insufficient scientific evidence to support certain claims about St. John’s Wort. These queries are particularly valuable as they direct research towards contentious areas where empirical data is critically needed to resolve debates. Meanwhile, queries four and five draw attention to potential risks associated with using St. John’s Wort as a supplement for individuals already taking depression medication. This approach ensures a comprehensive exploration of both the contested efficacy claims and potential dangers of the treatment. These few-shot prompts enable the LLMs to generate targeted and contextually relevant queries without extensive training. All queries are then aggregated into a list.

3.2.4. Article Retrieval

DietNerd adopts a methodology akin to a Retrieval-Augmented Generation (RAG) system [17]. The five generated queries are utilized to retrieve relevant information from the PubMed database. This retrieval process is facilitated through Entrez Programming Utilities (E-utilities), an API provided by the National Center for Biotechnology Information (NCBI) that offers access to various NCBI databases, including PubMed [27].

The retrieval mechanism operates in two stages:

Stage 1: 

Using the API’s ESearch utility [28], the system runs each query through PubMed, identifies the ten most relevant articles per query, and stores a list of PubMed IDs (PMIDs). Therefore, for each question, the system can retrieve up to fifty articles.

Stage 2: 

Each of the five PMID lists is fed to the EFetch utility [28], which then retrieves article data in the form of Extensible Markup Language (XML) files.

This approach parallels the retrieval component of RAG systems, which typically access a knowledge base to enhance the contextual understanding and accuracy of language models. In our case, PubMed serves as a specialized knowledge base, providing up-to-date and peer-reviewed information. By integrating this retrieved information with the capabilities of a large language model, DietNerd aims to generate responses that are both relevant to the user question and scientifically accurate based on the biomedical literature.

3.2.5. Relevance Classifier and Safety Considerations

Once all five queries have been processed, the retrieved articles are deduplicated and consolidated into a flattened list. This list is then concurrently processed through the Relevance Classifier LLM (see Appendix C.3) and categorized as either relevant or irrelevant, based on each article’s title, author, and abstract. An article is considered relevant if at least one of the following holds:

• It contains information that is helpful in answering the user’s question.
• It contains a safety aspect that would be important to include in the answer.
• It is NOT an animal-based study.

Again, an important risk-mitigation/safety element is established through the second criteria which qualifies articles as relevant if it includes safety-related information, even if it does not directly answer a user’s question. Doing so ensures that potential risks or safety concerns are not overlooked and that contraindications and warnings are considered. This strategy aligns with the principle of “do no harm” in healthcare by proactively surfacing safety information [29].

3.2.6. Article Match against Article Analysis Database

The article processing workflow is managed through two key functions. The first, connect_to_reliability_analysis_db(), establishes a connection to a MySQL database containing previously analyzed articles, retrieving and normalizing this data into a pandas DataFrame. The second function, article_matching(), compares newly collected articles against this database. By matching articles through its unique PubMed ID (PMID), the system identifies articles that have already been analyzed and retrieves full records of the article, including previously-generated article analyses. Articles not found in the database are flagged for processing. This approach avoids duplicate work and speeds up the overall process. The functions work in tandem to ensure that only new articles undergo full analysis, while previously examined article analyses are quickly retrieved.

3.2.7. Full Text Parser

As noted in Table 1, several state-of-the-art nutrition systems reference only the abstract of a paper. Although an abstract functions well as an overview, it may omit details around design (e.g., bias mitigation tactics) and statistical results. Because DietNerd builds a comprehensive profile for each article (as shown in Section 3.2.8 and Appendix C.5), we offer a thorough article summary. This summary can be used by both readers and subsequent LLMs to ascertain the level of evidence presented in the paper and thus give more credence to higher quality research.

To start, DietNerd first checks whether a PubMed Central ID (PMCID) is available. If so, it retrieves the full text from PubMed Central, a free full-text archive [30]. To avoid hitting the context window ceiling, DietNerd incorporates a Section Mapping LLM (as seen in Appendix C.4) which identifies the section names that closely map to a list of information-rich sections. Based on the attributes to be enumerated in each article analysis, the most relevant sections are: “Abstract”, “Background”, “Methods”, “Results”, “Discussion”, “Conclusion”, “Sources of Funding”, “Conflicts of Interest”, “References”, and “Table”.

In the case that the full text of an article is not available through PubMed Central, DietNerd parses the additional links provided by PubMed to other journal databases that may freely host the full text. If a list of links is compiled, DietNerd selects the most preferred of the available sources and captures its hyperlink. Preferred source is determined by referencing a ranked list of sources, where journal databases that are larger and encompass a wider variety of topics are prioritized. Then, depending on the source and provided hyperlink, the system will either:

(a)

Extract a unique article identifier from the hyperlink and concatenate it into URL strings that link to the full-text version of the article.

(b)

Navigate directly to the provided hyperlink.

Sources like Elsevier and Springer follow method (a), while sources such as JAMA Network and Wiley follow method (b). If none of these sources are available or if full-text retrieval fails, the system defaults to using the abstract.

3.2.8. Strength of Claim Analysis

After the article text is fetched, DietNerd begins analyzing the articles (in parallel). For each article, a list of PubMed publication types [31] is extracted from the article’s XML and used to categorize the article as either a “Review” type (e.g., literature reviews, systematic reviews, meta-analyses) or a “Study” type (e.g., randomized controlled trials, observational studies). Depending on the assigned category, DietNerd selects one out of two prompts and activates the Strength of Claim Analysis, a process led by an LLM tasked with profiling each article by extracting specified attributes and summarizing (see Appendix C.5). As demonstrated in

Table 3. Attributes captured for an article’s analysis based on paper category: “Study”-type papers have 13 attributes, while “Review”-type papers capture 9 attributes. Each of these listed attributes are specified and defined in the Strength of Claim LLM prompt.

“Study”-Type Papers	“Review”-Type Papers
Purpose & Design	Purpose
Main Conclusions	Main Conclusions
Risks	Risks
Benefits	Benefits
Type of Study	Search Methodology & Scope
Testing Subject	Selection Criteria
Size of Study	Quality Assessment of Included Studies
Length of Study	Synthesis & Analysis
Statistical Analysis of Results	Sources of Funding or Conflicts of Interest
Significance Level
Confidence Interval
Effect Size
Sources of Funding or Conflicts of Interest

, although the analyses of Review and Study paper types share core attributes, they diverge in some others. The analysis of “Study”-type papers look at characteristics like study size and statistics. The analysis of “Review”-type papers focus on attributes like selection criteria and a higher-level summary of statistical results (if applicable).

3.2.9. Safety-Conscious Output Synthesis

All relevant articles and their analyses are fed into the final LLM along with the original user question. DietNerd is designed to answer the question using articles with the strongest level of evidence, address both benefits and risks for a balanced response, and be inclusive of as many demographics as possible. Moreover, the LLM is directed to include dosage information when available and explain technical concepts in layman’s terms to make the information accessible to a wider audience.

Multiple safety dimensions are incorporated throughout the prompt. The LLM is tasked with highlighting any risks or dangers associated with the topic. Safeguards against harmful queries are also included. If the user question is dangerous or malicious, DietNerd will avoid answering and instead curb this behavior by suggesting safer or healthier alternatives. By emphasizing the importance of professional medical consultation, DietNerd further reinforces responsible information dissemination. Collectively, these elements create a framework that prioritizes user safety, ethical considerations, and risk mitigation while delivering comprehensive responses to medical queries.

A few-shot fine-tuning approach is used to dictate format and tone, as seen through the inclusion of “example_1_question” and “example_2_question”. See Appendix C.6.

4. Website Interface Architecture

The website architecture comprises of a back-end built with FastAPI and a front-end utilizing HTML, CSS, and JavaScript. This section describes the structure and functionality of both components, illustrated in Figure 3.

4.1. Back-End Structure

The back-end architecture of our system consists of two main components: a Python-based API server using FastAPI and a JavaScript file handling front-end interactions and API calls. This structure allows for a separation of concerns between server-side processing and client-side operations.

The Python back-end is organized as follows:

1.

API Endpoints: The FastAPI application defines several key endpoints, including:

• check_valid: Validates user queries using the function defined in Section 3.2.2.
• process_query: Initiates the full query processing workflow, using the functions defined in Section 3.2.3 through Section 3.2.9.
• sse: Implements Server-Sent Events (sse) to give the user real-time updates while the generation workflow is running.
• db_sim_search: Performs a similarity search on existing questions, using the algorithm defined in Section 3.2.1.
• db_get: Retrieves stored answers from the database.

2.

Asynchronous Processing: The back-end utilizes asynchronous programming techniques, specifically Python’s asyncio library and FastAPI’s asynchronous capabilities to handle concurrent requests efficiently.

3.

Background Tasks: Long-running processes, such as query processing, are handled as background tasks to prevent blocking the main thread.

4.

Database Interactions: The system interacts with two MySQL databases to store and retrieve processed question-answer pairs and article analyses.

5.

Hosting: The FastAPI back-end is deployed on an Amazon Web Services (AWS) Elastic Compute Cloud (EC2) t2.small instance.

4.2. JavaScript Front-End Logic

The key functions of the JavaScript front-end include:

1.

API Communication: Handles all API calls to the FastAPI back-end, including query submission, answer retrieval, and similarity search.

2.

Real-time Updates: Implements an EventSource to receive real-time updates from the server during query processing, using the SSE endpoint defined in Section 4.1.

3.

Response Formatting: Processes and formats the API responses for display, including handling references and citations.

4.

PDF Generation: Incorporates functionality to generate PDF reports of the query results using the jsPDF library.

5.

User Interface Management: Manages dynamic User Interface elements such as displaying similar questions and handling user interactions.

• Algorithm Implementation

The implementation of the DietNerd algorithm combines real-time processing with database retrievals:

1.

Question Submission: When a user submits a question, the question is first checked for validity using the check_valid endpoint, defined in Section 4.1.

2.

Database Check: If valid, the system checks if an answer already exists in the database using the db_get endpoint, defined in Section 4.1.

3.

Similar Questions: If no exact match is found, the system performs a similarity search to suggest questions already answered by our system to the user’s question using the sim_search endpoint in Section 4.1, and offers these questions as options.

4.

Real-time Processing: If the user requests a new answer for their question:

• A session ID is generated and returned to the client.
• The process_query endpoint (Section 4.1) is called and the answer generation workflow starts.
• The client establishes an SSE connection to receive updates, using the SSE endpoint in Section 4.1.
• The FastAPI back-end processes the question asynchronously, sending progress updates to the client.

5.

Result Delivery: The final answer, along with references and citations, is sent to the client and displayed to the user.

This workflow achieves efficient handling of both new and previously answered questions, providing a responsive user experience while managing server resources effectively.

5. Quality Evaluation of DietNerd

The assessment of DietNerd comprises two main components:

• Section 6 describes an automated accuracy evaluation of DietNerd to assess its agreement with respect to systematic surveys.
• Section 7 presents a blinded cross-tool comparison experiment involving domain experts and other state-of-the-art systems.

These two analyses allow us to assess DietNerd’s performance from multiple perspectives, combining quantitative metrics with qualitative expert feedback.

6. Automated Accuracy Evaluation

The automated evaluation compares the outputs of DietNerd against a gold-standard benchmarking dataset to systematically assess the semantic similarity of the responses with those of systematic surveys.

6.1. Benchmark Dataset Development

We followed the process developed by Clinfo.ai to construct a benchmark dataset using systematic reviews as the gold standard [5]. From the systematic reviews available in PubMed, those meeting the following criteria were selected to compile a dataset of 100 systematic reviews:

• The article title contains a question.
• The question addresses a specific dietary intervention.
• The topic inquires about a clear health outcome.

To construct the benchmark dataset, questions were extracted from the titles of selected systematic reviews. Gold-standard responses were derived from the article abstracts, the results, and conclusion sections. In cases where abstracts lacked explicit section titles, these sections were inferred from the text. Minimal edits were made to the gold-standard responses to avoid undue penalization, primarily spelling out acronyms and removing references to specific study types or statistical results. The publication date of each article was included to ensure that the systematic review itself would not be cited as a resource in the comparative analysis. We could not constrain the search of other systems by date, so this benchmark test measures only the results of DietNerd.

6.2. Generated Responses Dataset

The 100 questions from the benchmark dataset were processed through DietNerd. Each generated query included a temporal constraint, instructing the system to retrieve only articles published up to one day prior to the listed publication date of the benchmark question. This constraint was implemented to simulate the information landscape just before the target systematic review was published, ensuring a fair comparison. In cases where questions failed to yield a response due to insufficient sources, they were replaced with newly sourced questions to maintain the benchmark dataset’s size. Following generation, any acronyms in the responses were spelled out, mirroring the process applied to the benchmark dataset.

6.3. Metric Selection

As mentioned, the gold-standard reference text is based on systematic reviews, making it likely to incorporate technical language. DietNerd, on the other hand, generates text that is meant to be consumer-friendly. Because of this difference in target audiences, the focus in our evaluation is on whether the final conclusions matched semantically rather than syntactically. Therefore, to assess the alignment between DietNerd’s generated text and the gold-standard reference text, two similarity metrics were employed: BERTScore [32] and Semantic Textual Similarity.

BERTScore analyzes textual similarity based on token-level contextual embeddings, focusing on finer lexical details while still taking into consideration semantic similarity. It outputs an F1 score that is rescaled so that values near or below 0 indicate poor similarity and values closer to 1 indicate higher similarity.

Semantic Textual Similarity evaluates the overall similarity in the conclusions reached by both texts by calculating the cosine similarity on a scale of −1 to 1, with 1 denoting absolute semantic equivalence. To calculate Semantic Textual Similarity, the model all-mpnet-base-v2-negation was used. Built upon the Sentence Transformer base model all-mpnet-base-v2 [33], this model is fine-tuned on the Compilation of ANnotated, Negation-Oriented Text-pairs (CANNOT) dataset, making it much more sensitive to negations [34]. The Universal Sentence Encoder (USE) Q&A model [35] was considered but, since all-mpnet-base-v2’s underlying MPNet architecture incorporates permutation-based training, it potentially offers more versatility and flexibility when dealing with the varied sentence structures and complex language commonly found in medical texts.

While other evaluation metrics are widely used in LLM research, we determined that they were less suitable for our specific objectives. For instance, the Bilingual Evaluation Understudy (BLEU) [36], commonly used in text generation tasks, focuses on n-gram precision, which does not align well with our goal of assessing semantic understanding. BLEU can penalize semantically correct outputs that use different phrasing. Similarly, Recall-Oriented Understudy for Gisting Evaluation (ROUGE) [37] emphasizes lexical overlap, which does not capture the full range of acceptable variations in LLM outputs. Even ROUGE-WE (ROUGE with Word Embeddings) [38], an extension of ROUGE designed to better capture semantic similarities, was not a good fit because its use of static word embeddings limits its ability to account for the different meanings a word might have in different contexts. Our chosen metrics allow for greater flexibility in assessing semantically equivalent but lexically diverse responses, reflecting our research objectives.

Table 4. For each of the 100 questions included in the automated evaluation, a gold-standard reference text and DietNerd’s generated text are compared and analyzed. Below is an example where the BERTScore found a low similarity at the token level (0.043), while Semantic Textual Similarity showed high similarity at a semantic level (0.788). As one can see, both the generated text and the reference text arrive to similar conclusions, supporting the high semantic similarity score. By contrast, the reference text explicitly lists out each factor, while the generated text groups them together (e.g., “cardiovascular risk factors”), resulting in a syntactic difference. The generated text includes a safety-motivated recommendation to consult with a professional, contributing to a (benign) semantic difference between the texts.

Question	Reference Text	Generated Text
Does omega-3 PUFAs supplementation improve metabolic syndrome and related cardiovascular diseases?	Supplementation of omega-3 polyunsaturated fatty acids had a favorable effect on improving triglyceride, total cholesterol, high-density lipoprotein, systolic blood pressure, diastolic blood pressure, interleukin-6, tumor necrosis factor-$\alpha$, C-reactive protein, and interleukin-1 levels, yet did not affect low-density lipoprotein, monocyte chemoattractant protein-1, intracellular adhesion molecule-1, and soluble endothelial selectin among patients with metabolic syndrome and related cardiovascular diseases.	In conclusion, omega-3 polyunsaturated fatty acids supplementation can be beneficial for reducing triglycerides and improving some cardiovascular risk factors in individuals with metabolic syndrome. However, the effects on cholesterol levels and other metabolic markers can vary. It is advisable for individuals considering omega-3 supplements to consult with healthcare providers to tailor the approach based on personal health needs and risks.

demonstrates the evaluation focus of BERTScore versus Semantic Textual Similarity. In this example, BERTScore’s F1 score between the reference and generated texts was low at 0.043 reflecting a token-level difference. By contrast, Semantic Textual Similarity’s score rated the texts significantly similar at a 0.788 level. The reason for that high score is that the two texts reach the same essential conclusion: PUFAs were shown to have benefits for some health markers but have no proven impact on others.

6.4. Quantitative Results

The results of our automated evaluation show poor lexical similarity. As seen in Figure 4, BERTScore’s F1 scores ranged from −0.05 to 0.497 with a mean of 0.246, suggesting moderate token-level alignment. By contrast, Semantic Textual Similarity scores ranged from 0.439 to 0.934 with a mean of 0.736. The Semantic Textual Similarity is more important for DietNerd, because our primary goal is to deliver a semantically similar result in layman’s terms to those of expert researchers. As noted by [39], similarity scores exceeding 0.7 indicate strong similarity, and our results show a concentration of scores in the upper ranges of 0.75 to 0.85. Similarity scores above 0.5 indicate moderate agreement.

In summary, DietNerd, when given primary literature, generates responses that are semantically similar to high-quality systematic review medical literature.

7. Domain Expert Cross-Tool Comparison Experiment

We conducted a blinded experiment comparing the five systems listed in Table 1, which includes DietNerd and the GPT Marketplace version of Consensus, with eight registered dietitians/registered dietitian nutritionists. Feedback and ratings were collected across four questions, each containing five anonymous and randomly ordered responses. Domain experts assessed the quality across five dimensions: relevance, accuracy, completeness, clarity, and overall quality.

For each question, the five system responses were identified only by letters and the order of the responses (and thus the correspondence between letter and system) differed for each question. For example, in question 1, DietNerd was given letter A and appeared first, while in question 2, DietNerd was given letter C and appeared third. Thus the authorship of the response was hidden from the domain experts and the position of each system’s response varied from question to question.

7.1. Questions and Feedback Criteria

Question Pool Aggregation and Selection: The diet questions we used came from questions asked on online forums and from diet/nutrition-focused frequently asked question pages. One of the primary feedback participants (who did not participate in the later evaluations) ranked these questions according to patient relevance. The two highest-ranked questions were selected for feedback, with the remaining two questions chosen randomly.

Response Generation and Standardization: For each of the four questions, each system generated a response. Responses for each system were then compiled in random order into a single portable document format (PDF) per question, minimally standardized to preserve system anonymity.

Feedback Survey: For each feedback question, a survey form containing five sections—one for each response—was provided. Each section included five questions analyzing a response across four criteria and an overall evaluation:

• Relevance: How relevant was the response to the question(s) asked?
• Accuracy: Was any of the information misleading or factually incorrect?
• Completeness: Did the response miss significant information or context?
• Clarity: Was the response clearly presented?
• Overall Quality: How would you rate the overall quality of this response?

For each value in the “Overall Quality” question, participants were provided the following definitions:

• Excellent You would use the tool that generated this response again and would recommend it to others.
• Pretty Good: The response was reasonable, but not quite good enough to recommend.
• Okay: This response was more or less accurate, but incomplete or unclear.
• Pretty Bad: This response had significant problems in either accuracy or clarity.
• Horrible: This response was just unacceptable.

7.2. Quantitative Results of Expert Analysis

Using the Overall Quality score provided by participants (Figure 5), we employed a non-parametric paired test [40] between the results of each pair of systems. The series of pairwise comparisons of the five systems revealed several significant differences in performance. Given a significance threshold of 0.05, the analysis indicated that OpenEvidence, Scholar GPT, and DietNerd performed similarly with no statistically significant differences observed among them. These three systems did however show a statistically significant advantage when compared to Clinfo.ai and, a lesser (and statistically insignificant) advantage over Consensus. Specifically, the comparison between Clinfo.ai and the three systems (DietNerd, OpenEvidence, and Scholar GPT) resulted in p-values $<0.001$, indicating significant differences in performance.

When looking at individual quality criteria (Figure 6), a non-parametric statistical analysis suggests a stratification in the performance of the evaluated systems. DietNerd, Scholar GPT, and OpenEvidence exhibit statistically similar performance levels. Consensus showed some differences in performance compared to OpenEvidence and Scholar GPT, with a trend suggesting potential differences with DietNerd. Clinfo.ai demonstrated a statistically significant disadvantage when compared to DietNerd, Scholar GPT, and OpenEvidence.

8. Feedback from Non-Professional Users

Because DietNerd is meant to be used by subject matter experts and laymen alike, we show here some anecdotal feedback from non-professional users.

• “Wow! Thank you for sending it to me! I have already had advice for my arthritis!!!”
• “The output was pretty consistent with what I’ve learned over my years living with psoriatic arthritis and the various diets I have tried and information I have read. That was a nice summary!”
• “I was really impressed with the results! The information provided was precise, thorough, and covered the issue from various perspectives.”
• “It was a smooth user experience. The similar questions feature was very helpful and provided an immediate informative response. The PDF download is also very convenient.”
• “The answers matched the findings I had come across when researching this topic on my own time—there were even quite a few new things I learned. I especially appreciated the inclusion of dosages and exact numbers. These were things I tried to source but was having trouble easily finding within the studies and research articles I read. Also, having the articles analyses and the full-text vs abstract-only indicator makes the vetting process much easier when I’m trying to filter out which research and journals I can trust.”
• “The responses were indeed helpful. I found my intentions were identified accurately and the responses were quite relevant to my questions. I particularly like the references presented besides the response—at least I get to know the information source and have access to verify the information that DietNerd provides to me. It could be helpful to include more details of what this tool can provide on the website, like a user manual, and to have a type of “history” section that helps users easily retrieve previous questions and answers when needed. Lastly, it would be very helpful if the system could support follow-up questions so that users may narrow down their questions as they get more responses.”

These anecdotal responses show the system’s usability for a wide variety of users (for instance, the first arthritis user is close to 90). Users praised the relevance of the information provided, the ease-of-use, and appreciated the analysis of the references. Some users suggested potential improvements, especially the capability of dealing with follow-up questions. We will address this in future work.

9. Conclusions

DietNerd is a system that gives nutritional and diet advice based on peer-reviewed scientific articles. Based on an experiment measuring semantic similarity against systematic surveys, DietNerd achieves high accuracy. Compared to other state-of-the-art systems, DietNerd is among the best based on expert (i.e., registered dietitian) reviews. In addition, DietNerd embeds multiple safety checkpoints within its design and provides detailed individual article analysis functionality, making it unique among sister systems. Last but not least, non-professional users find the system easy to use and informative.

These findings suggest that DietNerd could help bridge the gap between complex primary literature and consumer-friendly content, thus contributing towards public health education. This has the potential to democratize access to medical knowledge, enabling consumers to better understand and engage with current research findings. Moreover, DietNerd’s ability to aggregate information from multiple studies in a manner that maintains high semantic similarity to systematic reviews suggests it could help consumers gain a more comprehensive and balanced understanding of medical topics, mirroring the rigor of systematic reviews but in a more digestible format. This could enable non-medical professionals to have better informed conversations with healthcare professionals and encourage stronger patient-doctor/dietitian communication as individuals may ask more pertinent questions and better understand medical advice.

The value of tools like DietNerd is particularly pronounced in the current healthcare landscape, which suffers from shortages of specialized professionals. According to the Bureau of Health Workforce, there is a growing deficit of registered dietitians, a trend that is projected to worsen over time [41]. This shortage has far-reaching implications for patient care and health outcomes. As the gap between supply and demand widens, patients are experiencing progressively longer wait times for appointments, potentially delaying crucial nutritional interventions and guidance. In response to this challenge, some clinics have implemented interim solutions, such as offering consultations with health coaches to address basic questions and provide general wellness advice [42]. However, these stopgap measures may not fully bridge the knowledge gap or provide the depth of information that patients require.

In this context, DietNerd can play a role in augmenting existing healthcare resources. By providing scientifically grounded, easily digestible information, our tool can answer fundamental questions during the waiting period between appointments. This is not to suggest that DietNerd should replace professional consultation; rather, it serves as a complementary resource that can enhance the patient experience and preparedness.

Moreover, DietNerd can support continuity of care by providing consistent, up-to-date information between appointments. This is particularly valuable for patients managing chronic conditions or implementing long-term dietary changes, where ongoing education and reinforcement are crucial. By empowering patients with knowledge, DietNerd can potentially improve adherence to dietary recommendations and foster a more collaborative relationship between patients and healthcare providers.

10. Limitations and Future Work

There are four limitations that we plan to overcome next. First, we would like to build out access to other scientific databases to further enrich our knowledge base and provide a more comprehensive foundation for DietNerd. The current system in fact already accesses the databases of specific publishers to get full text articles, but this could be more fully built out to widen the pool of available information.

Second, we would like to offer the functionality of recipe creation by leveraging resources such as FoodKG, a food knowledge graph [43], as well as other nutrient databases to generate nutritionally balanced recipes tailored to a user’s dietary preferences and restrictions.

Third, some of our non-professional users have requested conversational functionality. While we primarily viewed this as a convenience feature rather than a scientific contribution, it holds potential for an improved user experience.

Fourth, the system could be designed to elicit clarifying information by posing questions to the user. This would help DietNerd tailor responses to a user’s medical profile, for example, by taking into account food allergies and other preexisting conditions.