Comparative Evaluation of Artificial Intelligence Models for Contraceptive Counseling

Abstract

Background: As digital health resources become increasingly prevalent, assessing the quality of information provided by publicly available AI tools is vital for evidence-based patient education. Objective: This study evaluates the accuracy and readability of responses from four large language models—ChatGPT 4.0, ChatGPT 3.5, Google Bard, and Microsoft Bing—in providing contraceptive counseling. Methods: A cross-sectional analysis was conducted using standardized contraception questions, established readability indices, and a panel of blinded OB/GYN physician reviewers comparing model responses to an AAFP benchmark. Results: The models varied in readability and evidence adherence; notably, ChatGPT 3.5 provided more evidence-based responses than GPT-4.0, although all outputs exceeded the recommended 6th-grade reading level. Conclusions: Our findings underscore the need for the further refinement of LLMs to balance clinical accuracy with patient-friendly language, supporting their role as a supplement to clinician counseling.

1. Introduction

As digital resources increasingly become the first line of inquiry for many patients with medical concerns, the accuracy and comprehensibility of online health information are of paramount concern [1]. This is particularly true in the field of reproductive health, where decisions informed by online resources can have profound implications on individual health outcomes and public health at large. Contraception counseling, with its intricate balance of medical, personal, and social considerations, exemplifies a clinical area where the quality and accuracy of information is critical.

The integration of artificial intelligence (AI) into health communication has opened novel avenues for patient education [2,3,4]. Large language models (LLMs), such as OpenAI’s ChatGPT series and counterparts from other tech entities, have demonstrated impressive capabilities in generating human-like text responses [5,6,7,8]. These models are trained on vast corpuses of data, enabling them to simulate a broad understanding of human language and knowledge. Their application in healthcare has the potential to revolutionize patient engagement by providing instant, personalized responses to health inquiries. Nonetheless, the deployment of such models in a clinical context demands rigorous evaluations to ensure that the information provided is both accurate and accessible [9,10,11].

Accuracy in medical information is multi-dimensional and encompasses not only the factual correctness of the content but also its alignment with current clinical guidelines and evidence-based practices [12]. The dynamic nature of medical knowledge, with continuous updates and revisions to guidelines, poses a substantial challenge for LLMs for which training datasets may not reflect the most recent consensus. Moreover, the nuances of medical decision-making often require context-sensitive information that cannot be accounted for by generalist AI models, irrespective of their sophistication [13].

Readability, a measure of the ease of understanding a text, is another critical dimension that impacts the utility of information provided by LLMs. Health literacy, or the ability to obtain, process, and understand basic health information to make appropriate health decisions, is a fundamental right. Yet, it remains a challenge for a significant portion of the population. The American Medical Association recommends that patient education materials be written at a 6th-grade reading level to be widely comprehensible [14,15]. However, the advanced language capabilities of LLMs often produce content that surpasses this threshold, potentially alienating those with limited health literacy [16].

This study seeks to systematically evaluate the accuracy and readability of responses from four leading LLMs on a range of common contraception-related questions. By benchmarking these AI-generated responses against evidence-based responses in a recent review on contraception published by American Academy of Family Physicians (AAFP) [17], we aim to critically assess the readiness of LLMs for deployment in patient counseling scenarios. While studies have been conducted with ChatGPT in OB/GYN, no studies have evaluated all four LLMs. This investigation is grounded in the aim of ensuring that emerging AI tools enhance, rather than hinder, patient autonomy in reproductive decision-making. The ultimate goal of this project is to inform the development of AI applications in healthcare that are not only technically proficient but also ethically aligned with the principles of patient-centered care.

2. Methods

This cross-sectional analysis was designed to evaluate and compare the performance of four contemporary LLMs in providing readable and accurate responses to common contraception questions. This study was determined to be exempt from review by the Yale University institutional review board and followed the STROBE reporting guidelines [18]. The LLMs included in this study were OpenAI’s ChatGPT 4.0, OpenAI’s ChatGPT 3.5, Google Bard, and Microsoft Bing. These models were selected based on their widespread use, advanced language-generation capabilities, and diverse underlying technologies. Each model was accessed via its publicly available interface on 6/10/2023.

The contraception questions selected from the AAFP review represent a comprehensive range of topics commonly encountered in clinical practice. The questions were selected after a review of current clinical guidelines and in consultation with 2 OB/GYN physicians to ensure they address key aspects of contraceptive counseling, including efficacy, safety, and patient-specific considerations. Six questions were chosen to cover a spectrum of common contraceptive counseling scenarios encountered by healthcare providers (

Table 1. Summary of average physician reviewer scores for all criteria used to evaluate the accuracy of responses to contraceptive counseling inquiries ^a.

Question	AAFP b	GPT-4.0	GPT-3.5	Google Bard	Microsoft Bing
What forms of emergency contraception are effective?	3.5	3	3.1	3.1	2.5
Are fertility awareness methods of contraception effective?	3.4	3.3	3.5	3.4	2.5
What contraceptive methods are less safe for people with migraines?	3.5	3.6	3.5	3.2	3.2
How long does long-acting reversible contraception remain effective?	3.5	3.5	3.3	3.1	2.8
Can depot medroxyprogesterone acetate be self-administered subcutaneously? What are the effects on bone mineral density?	3.4	3.3	3.6	4	2.9
What are contraception considerations in transgender and gender-diverse people with a uterus?	3.4	3.2	3.4	3.3	3.3

^a Higher scores (maximum 5) are more favorable for the following criteria: Was the question responded to? Was the response evidence based? Was the response complete or had any extraneous information? Did the response refer the user to a healthcare provider or other resources? Lower scores (minimum 1) are more favorable for the following criteria: Does the response speak in absolutes? ^b American Academy of Family Physicians.

). Each LLM was queried with the six questions, and the resulting responses were recorded verbatim. To mitigate potential biases, each model’s responses were collected in a new chat session to prevent learning or tailoring responses based on previous interactions. Responses were anonymized and formatting was standardized to prevent identification of the source LLM by the reviewers.

For the readability analysis, the LLMs’ outputs were assessed by employing four established readability indices: the Gunning–Fog Index (GF), Flesch–Kincaid Grade Level (FK), Automated Readability Index (AR), and Coleman–Liau Index (CL) [19,20]. These indices were chosen for their widespread recognition and use in assessing health education materials. Readability scores for all four indices align with U.S. school grade levels, with a score of 6 indicating a sixth-grade reading level. To standardize the outputs and ensure an equal comparison, all formatting was removed, including bullet points and numbered lists, as is consistent with previous readability studies [16,21]. Ancillary response information (e.g., “please note I am not a medical professional”) was also removed to focus the analysis on the clinical content.

For the accuracy analysis, two OB/GYN physicians independently reviewed the entire response output of each LLM and AAFP review. The physician reviewers were blinded to the source of the responses and assessed the accuracy, specificity, evidence basis, completeness, and readability of each response. The reviewers used a 5-point Likert scale, with anchors clearly defined for each criterion to maintain consistency in scoring. Descriptive statistics were used to summarize the Likert scale ratings and readability scores. All analyses were conducted using SAS version 9.4.

3. Results

Readability analyses showed that of the four LLMs, Google Bard exhibited the lowest average readability with a score of 10.59 which signifies a tenth grade reading level. GPT-3.5 averaged 13.65, Microsoft Bing averaged 14.18, and GPT-4.0 averaged 15.44, thus demonstrating higher readability scores, consistent with college-grade reading levels (

Table 2. Readability index score for each large language model.

	GPT-4.0	GPT-3.5	Google Bard	Microsoft Bing
Gunning–Fog Index	13.80	13.32	10.10	13.91
Flesch–Kincaid Grade Level	14.50	12.57	9.55	12.45
Automated Readability Index	16.95	15.22	11.35	15.40
Coleman–Liau Index	16.49	13.47	11.37	14.96
Average Reading Grade Level	15.44	13.65	10.59	14.18

). The average reading grade level of the responses from the four LLMs deviated from the target goal of 6 by a range of 4.59 (Google Bard) to 9.44 (GPT-4.0).

The response accuracy assessment utilized the average of physician reviewer scores for all individual LLM and AAFP responses (

Table 3. Average physician reviewer scores for each criteria used to assess the accuracy of large language model responses.

Criteria	AAFP a	GPT-4.0	GPT-3.5	Google Bard	Microsoft Bing
Was the question responded to?	4.92	4.67	4.67	4.67	4.42
Was the response evidence based?	5.00	2.83	3.17	2.83	2.92
Was the response complete or had any extraneous information?	3.33	2.75	2.67	2.67	2.42
Did the response refer the user to a healthcare provider or other resources?	3.00	4.58	5.00	5.00	2.33
Does the response speak in absolutes?	1.00	1.75	1.50	1.58	2.25

For criteria such as ‘response accuracy’ and ‘evidence basis’, higher scores (maximum 5) are more favorable; conversely, for the ‘use of absolutes’ criterion, lower scores (minimum 1) indicate a better performance. ^a American Academy of Family Physicians.

). The AAFP reference set a high benchmark for whether the questions were responded to with an average response rating of 4.92 out of 5. Among the LLMs, GPT-4.0, GPT-3.5, and Google Bard were each rated at 4.67 based on this criteria, while Microsoft Bing was rated at 4.42. During an assessment of the evidence-basis for each response, the AAFP reference achieved a score of 5.0. The LLMs exhibited varying levels of adherence to evidence-based information, with GPT-3.5 at 3.17, Microsoft Bing at 2.92, and both GPT-4.0 and Google Bard at 2.83. When assessed for completeness and the presence of extraneous information, the AAFP reference scored 3.33, while GPT-4.0 scored 2.75, GPT-3.5 and Google Bard scored 2.67, and Microsoft Bing scored 2.42. In terms of referral to healthcare providers or resources, the LLMs generally showed higher referral rates, with GPT-3.5 and Google Bard both achieving a perfect score of 5.0 and GPT-4.0 close behind with a score of 4.58. The AAFP reference garnered a lower score of 3.0 and Microsoft Bing even lower at 2.33. Finally, the use of absolutes in responses was lowest for the AAFP reference at a score of 1.0. Among the LLMs, GPT-3.5 was best at minimizing absolutist language and had the lowest score at 1.5, followed by Bard at 1.58, GPT-4.0 at 1.75, and Bing at 2.25.

4. Comment

4.1. Principal Findings

The advent of large language models (LLMs), like OpenAI’s ChatGPT, Google Bard, and Microsoft Bing, offers a novel and rapidly expanding avenue for patient education in areas like contraception counseling. Our study aimed to systematically evaluate the accuracy and readability of responses from four leading LLMs—OpenAI’s ChatGPT 4.0 and 3.5, Google Bard, and Microsoft Bing—in providing contraception-related information. This evaluation is crucial in an era where accurate and comprehensible health information is a cornerstone of effective patient care and autonomy.

4.2. Results in the Context of What Is Known

Our results indicate that all LLMs generally responded to the question asked but notably differ in adherence to evidence-based information. All LLMs fell short of the AAFP’s perfect score for evidence-basis. This gap highlights a critical limitation of current LLMs—they may not always draw from the most current evidence-based guidelines or reliably reason through nuance, both crucial aspects in medical decision-making. Interestingly, GPT-3.5 scored highest among LLMs in this regard, and its outperformance of GPT-4.0 suggests that newer versions of these models are not necessarily better in providing reliable information. When asked “What contraceptive methods are less safe for people with migraines?”, GPT-3.5 cited guidelines issued by the American College of Obstetricians and Gynecologists (ACOG) and the World Health Organization (WHO); however, GPT-4.0 failed to reference any established guidelines in its response. In other instances, responses from GPT-3.5 and GPT-4.0 highlighted internal inconsistences. When asked “Are fertility awareness methods (FAMs) of contraception effective?”, GPT-4.0 wrote “However, perfect use of FAMs can result in lower failure rates, typically around 1-5%,” while GPT-3.5 wrote “However, with perfect use, the failure rate can be much lower, around 2-3% for some methods.” The sources of these efficacy values are unclear, and the inconsistency raises some concern regarding the training material that LLMs use to develop their evidence-basis.

An analysis of response completeness and the presence of extraneous information also revealed an unsatisfactory performance by LLMs compared to the AAFP review. When asked “What forms of emergency contraception are effective?”, all LLMs failed to address the impact of body mass index (BMI) on emergency contraception methods. The risk of pregnancy with common forms of emergency contraception varies significantly in obese and overweight patients [22]. This suggests that while LLMs can provide relevant information, they may also fail to include important details, potentially leading to misinformation or confusion.

Regarding referral to healthcare providers or resources, LLMs generally outperformed the AAFP reference. This is encouraging, as it demonstrates the models’ capacity to recognize the limits of their advice and the importance of professional medical consultation. The lower use of absolutes in responses, particularly by the AAFP reference and GPT-3.5, is a significant finding. Over-reliance on absolutes can be misleading in medicine, where uncertainties and patient-specific factors often play a substantial role. This highlights the nuanced understanding required in conveying medical information, an area where LLMs can be further optimized.

In terms of readability, Google Bard and GPT-3.5 offered content that was somewhat more accessible to the general public, as indicated by lower readability scores. In contrast, responses from Microsoft Bing and GPT-4.0 were at a higher grade level, which could be less approachable for individuals with limited health literacy. This finding is crucial, considering the recommendation for patient education materials to be at a 6th-grade reading level. It suggests a need for further refinement in the LLMs’ language generation algorithms to make health information more universally accessible.

The range of readability scores of LLM-generated content is a striking aspect of our results. The higher reading levels of LLM responses, which far surpassed the recommended standards for patient education materials, raise concerns about the equitable delivery of health information. This is critical, given that lower health literacy is associated with poorer health outcomes and greater use of health services. The challenge lies in developing LLMs capable of conveying complex medical information in simplified terms without compromising content quality, a balance that is yet to be fully achieved.

4.3. Clinical Implications

Our findings indicate significant variability in both the accuracy and readability of contraceptive information provided by large language models (LLMs). This limits them as sources as standalones for patient counseling. If LLMs are to be considered for integration within the clinical space, they should be used to supplement, rather than replace, direct patient–provider interactions. This approach ensures contextual relevance and maintains the nuances essential for individualized patient care. It is of utmost importance to adapt an equitable framework under which any LLM is utilized within clinical settings, which includes the verification of generated content to current, evidence-based clinical guidelines.

4.4. Research Implications

This work stresses the need for ongoing efforts to refine large language models for medical use. Future studies should zero in on creating flexible algorithms that can pull in real-time clinical data, keeping the information in step with current evidence-based standards. Broadening the training data to cover a wider array of clinical situations and factoring in aspects like patient viewpoints and healthcare fairness will make these tools more practical and ethically sound. Bringing in a diverse mix of reviewers and digging deeper into outcomes that matter to patients will also help sharpen the health information that LLMs deliver.

4.5. Strengths and Limitations

Our study has several strengths. By comparing four different LLMs, we offer a broad analysis of the current dominant artificial intelligence systems and can assess trends across models. For our readability analysis, we used four different indices to derive a composite average, thereby minimizing the limitations of any one index. For our accuracy analysis, we utilized a peer-reviewed AAFP article as a benchmark, ensuring a vetted and rigorous standard for accurate responses to contraception questions. Furthermore, the OB/GYN physician evaluators were blinded to the source of each of the responses thereby minimizing preconceived biases about LLM performance. During the evaluative process, a standardized grading scale with anchors clearly defined each criterion to maintain consistency in scoring.

The cross-sectional nature of this study limits our ability to assess the consistency of LLMs over time. Given the rapid growth within the field of AI, there is a need for ongoing evaluations. Second, our analysis was confined to a single set of contraception questions. The scope of questions assessed may not encompass the full breadth of scenarios encountered by patients, nor does it capture the interactive and iterative nature of patient-provider conversations. Additionally, the reliance on a small panel of reviewers may not represent a consensus across the broader medical community, which highlights the need for more diverse reviewer perspectives in future evaluations. Additionally, we were unable to explore any broader assessment criteria, potentially overlooking the critical dimensions of healthcare information quality. A further examination of patient perspectives would provide deeper insights into the real-world applicability and effectiveness of LLM-generated health information. Furthermore, our focus on readability indices, though standardized, does not fully capture the comprehensiveness and nuance of language comprehension. These reading-level indices are formulas for readability but do not account for factors, such as cultural relevance, personal experience, or emotional tone, which are crucial to patient education. Lastly, the AAFP benchmark text was intended for an audience of healthcare providers and some of the content may not have been as relevant for a general consumer audience.

5. Conclusions

A comprehensive study on the use of LLMs for contraception counseling reveals both the potential and limitations of AI in patient education. While LLMs like OpenAI’s ChatGPT, Google Bard, and Microsoft Bing can provide useful information and effectively guide users towards professional medical consultation, there remain significant gaps in aligning their outputs with current evidence-based guidelines and ensuring readability at recommended levels. Newer versions of these LLMs do not consistently outperform older versions, highlighting that the organic evolution of these models does not necessarily equate to more accurate or accessible patient health information. Successive iterations of these LLMs must keep these goals in mind to see a marked improvement in health care applications.

The study highlights the importance of ensuring accurate and accessible health information in newly developed LLMs in order to provide equitable health communication and minimize misinformation to consumers. It also underscores the importance of AI tools as supplements, rather than replacements, for direct patient–physician interactions. Looking forward, enhancing LLMs’ algorithms for greater medical accuracy and adaptability in content complexity remains a priority. Simultaneously, integrating these tools into clinical practice calls for a collaborative effort among developers, clinicians, and patients to maximize their benefits and address inherent shortcomings. In light of rapid advancements in LLM technology, future studies should explore integrating multimodal data (e.g., voice and image inputs) and deploying these models in real-time clinical decision support systems. Such approaches may further personalize contraceptive counseling and improve patient outcomes. This study paves the way for future research focused on the real-world applications of LLMs in various clinical settings, aiming to improve patient outcomes and satisfaction in the evolving landscape of digital health.