Building Safe AI Chatbots for Rural Mothers Seeking Breastfeeding Support: Understanding Hallucinations and How to Mitigate Them

Abstract

AI-enabled chatbots are increasingly positioned as a remedy for breastfeeding support gaps in rural maternal health, offering private, immediate assistance amid persistent shortages of lactation specialists and limited access to care. However, their clinical promise remains constrained by the probabilistic nature of large language models, which can generate hallucinations that undermine maternal–infant safety. This article argues that safely integrating AI into breastfeeding support requires treating hallucination not as a singular technical flaw but as a systems-level risk shaped by design, governance, and use context. We identified key risks of AI systems that could result in hallucination such as, false citations, transcription errors, prompt injection and jailbreaking, and incorrect generalization or personalization, and analyze how each error introduces distinct safety vulnerabilities. Drawing from systems thinking, we outline mitigation strategies including retrieval-augmented generation grounded in authoritative breastfeeding sources, layered guardrails, adversarial testing, uncertainty-aware messaging, and domain-specific fine-tuning. By linking AI system design choices to downstream health consequences in resource-constrained settings, this paper reframes AI-assisted breastfeeding support as a governance challenge central to equitable, safe maternal health innovation.

1. Introduction

The rapid expansion of generative artificial intelligence (AI) is transforming how population groups and communities seek and search for health information (Yun and Bickmore 2025). Increasingly, rural mothers are using AI-enabled digital tools and chatbots as a source of on-demand breastfeeding guidance during the postpartum period, particularly when in-person lactation support is limited (Agudelo-Pérez et al. 2024; Arora et al. 2021; Igwama et al. 2024). This shift from reliance on traditional, clinician-based breastfeeding support in the pre-AI era to the increasing use of AI-enabled tools in the current AI era reflects long-standing structural inequities in maternal health care, stemming from severe shortages of obstetricians, pediatricians, and certified lactation specialists in rural regions. At the same time, traditional sources of breastfeeding guidance, such as grandmothers, aunts, and older siblings, may no longer be as accessible or trusted due to generational shifts, family dispersion, or conflicting advice that may diverge from current clinical breastfeeding recommendations, such as early breast milk complementation before six months or inadequate guidance for managing breast complications (Angelo et al. 2020; Luna et al. 2022). Evidence also suggests rural–urban disparities in postpartum care access, with rural postpartum mothers being less likely to see an obstetrician–gynecologist and more likely to experience barriers to care compared to urban postpartum mothers (Handley et al. 2025). When professional lactation support is distant, AI assistants provide essential, around-the-clock guidance for families.

Despite the potential reach of AI, this system presents meaningful risks such as hallucinated clinical guidance, fabricated citations, and unsafe overgeneralization of breastfeeding recommendations (Macrae 2019). The challenge of ensuring reliability in large language models (LLMs) stems from their fundamental design. Rather than drawing solely from verified medical knowledge or structured clinical reasoning, LLMs generate text through autoregressive prediction, selecting each token based on statistical patterns in prior tokens (Ahn 2025). This probabilistic guessing can lead to what is referred to as hallucination statements that are inaccurate, fabricated, or presented with unwarranted certainty. Because misinformation on infant feeding can directly and indirectly affect neonatal health, hallucinations may cause delayed mastitis care, unsafe supplementation, poor infant nutrition, or early breastfeeding cessation. Ensuring safe deployment, therefore, requires attention to both end-user literacy and institutional governance. This article identifies key risks of AI-generated content that could result in hallucination in AI breastfeeding support chatbots used in rural health settings, with strategies to mitigate their risks (

Table 1. AI Failure Types, Associated Harm, and Mitigation Strategies in Breastfeeding Support Chatbots.

Failure Type	Associated Harms	Mitigation Methods
False Citations and Evidence	Fabricated or unsupported attribution (including guideline/policy statements) can create artificial legitimacy and mislead rural mothers into unsafe feeding practices	RAG grounded in pre-approved resources High-quality data structuring (“data as a product”: cleaned, validated, machine-readable) Encourage users to verify references before applying recommendations
Errors in Speech Transcription	Transcription may introduce text never spoken, altering the interpretation of feeding practices and influencing clinical decisions in telemedicine	Stress test with diverse speech samples Secondary model to flag uncertain segments Human review for high-stakes interactions Users review visit summaries and confirm key details verbally
Prompt Injection and Jailbreaking	Can bypass safety restrictions or produce off-topic content, undermining trust and creating safety/security risks	Retrieval guardrails (vetted breastfeeding-specific repositories) Generation guardrails (reject/redirect unsafe or off-topic queries) Adversarial testing Continuous monitoring after deployment
Incorrect Generalization and False Personalization	Overgeneralized advice can miss important etiologies and misalign recommendations with user goals, affecting breastfeeding duration and maternal confidence	Domain-specific fine-tuning on breastfeeding content Uncertainty-aware messaging (avoid categorical language) “Confession reports” Offer differential considerations and prompt professional evaluation when appropriate

).

2. False Citations and Evidence

Chatbots may confidently attribute recommendations, including guideline- or policy-based statements, to authoritative organizations such as the American Academy of Pediatrics without any corresponding scientific documentation. This artificial legitimacy can mislead rural mothers into adopting feeding practices that are unsafe. Preventing fabricated evidence requires institutional governance, such as retrieval-augmented generation (RAG) systems that ensure that chatbots draw from pre-approved resources (Li et al. 2025).

However, RAG’s effectiveness depends on high-quality data structuring. Treating data as a product that is cleaned, validated, and optimized for machine readability provides a practical path for rural populations and community-based health organizations with limited access to data and resources. Parents and health professionals should remain critical consumers by seeking verifiable references before applying recommendations.

3. Errors in Speech Transcription

Hallucination also occurs in automated transcription used in telemedicine consultations (Koenecke et al. 2024). Many women in rural communities rely on telehealth services for reproductive health (Sundstrom et al. 2020). Popular tools, such as “Whisper,” may introduce text that was never spoken when audio quality is compromised, or accents differ from the training data (Frieske and Shi 2024; Koenecke et al. 2024). Even minor errors can alter the interpretation of feeding practices, potentially influencing clinical decisions.

Because maternal–infant care is a high-risk domain, developers must stress-test transcription models with diverse speech samples, integrate secondary models to flag uncertain segments, and ensure human review of high-stakes interactions. Mothers can assist accuracy by reviewing visit summaries and requesting verbal confirmation of key details.

4. Prompt Injection and Jailbreaking

Even a chatbot intentionally designed to support breastfeeding can be manipulated to circumvent safety restrictions or provide off-topic responses. This can occur through prompt injections, in which users introduce hidden instructions into their questions, or jailbreaking, in which they intentionally coax the model to ignore its safety boundaries. These vulnerabilities can undermine trust in clinical communication tools and expose families and institutions to breastfeeding safety and security risks.

Mitigating these threats requires the deliberate use of guardrails, described as protective mechanisms integrated into the AI system. In RAG, guardrails can operate at two levels. First, retrieval guardrails ensure that only documents from vetted, breastfeeding-specific repositories are provided to the model. By restricting the knowledge source, retrieval guardrails reduce the likelihood that inappropriate or unsafe breastfeeding instructions will surface. Second, generation guardrails then constrain the chatbot’s output to remain aligned with breastfeeding support norms and clinical safety standards. These may include rejecting or redirecting off-topic queries, preventing speculative medical advice, or enforcing responses that encourage professional care when symptoms are high-risk.

Adversarial testing during development and continuous monitoring after deployment are essential to verifying that jailbreaking attempts do not compromise system behavior. Together, retrieval and generation guardrails provide a multilayered approach to secure breastfeeding chatbots and maintain their reliability, specifically in rural maternal health contexts.

5. Incorrect Generalization and False Personalization

Generative models tend to rely on dominant patterns in their training data, often producing advice that appears confident yet fails to account for individual variation (Haines et al. 2025). In breastfeeding contexts, this may lead a chatbot to attribute all nipple pain to poor latch technique or recurrent plugged ducts while overlooking less common but clinically important etiologies such as vasospasm, dermatoses, infection, allodynia, or maternal oversupply (hypergalactia) as factors (Berens et al. 2016). Similarly, the model may infer user characteristics or goals not explicitly stated. For example, a mother may ask an AI chatbot for help with low milk supply or infant fussiness, and the chatbot may prematurely recommend formula supplementation despite the mother not expressing any desire to supplement. These forms of overgeneralization and false personalization can subtly redirect caregivers toward choices misaligned with their clinical needs or infant feeding intentions, potentially negatively affecting breastfeeding duration and maternal confidence.

Mitigation strategies involve explicitly embedding nuance and safety into the model’s design. Fine-tuning on domain-specific breastfeeding content, annotated to reinforce decision thresholds and diverse clinical presentations, can reduce the likelihood of advice that treats all cases as typical. Additionally, designing the model to acknowledge uncertainty by generating “confession reports,” avoiding categorical language, and proactively offering differential considerations can prompt users to seek timely professional evaluation when appropriate (OpenAI 2025; Zhou et al. 2025). Together, these measures strengthen the alignment of AI responses with individual needs, an essential requirement in rural maternal health environments where follow-up care is less accessible.

6. Discussion, Implications and Future Directions

Breastfeeding in rural settings is often shaped by geographic isolation, limited availability of lactation support, and barriers to timely postpartum follow-up care (Handley et al. 2025). These structural constraints will increase reliance on informal advice or on-demand digital tools when urgent breastfeeding concerns arise (Angelo et al. 2020; Luna et al. 2022). This Opinion highlights that AI-enabled breastfeeding support chatbots may help address gaps in postpartum support in rural settings, but their usefulness depends on whether their outputs are safe, accurate, and aligned with clinical breastfeeding guidance (Arora et al. 2021; Macrae 2019). The failure types discussed in this manuscript include false citations, transcription-related errors, prompt injection and jailbreaking vulnerabilities, and incorrect generalization or false personalization; they show how AI-generated content can introduce risks that may affect maternal decision-making and infant health (Mukherjee and Venugopal 2018). These risks reinforce the need for careful design, governance, and validation before such tools are relied upon in sensitive maternal–child health contexts.

Implications of AI Hallucination in Breastfeeding Support for Rural Mothers

As digital literacy becomes an increasingly strong predictor of health behavior, it is essential to consider how emerging technologies intersect with existing structural inequities (Corvo et al. 2024). In rural settings, breastfeeding mothers often face compounded challenges, including geographic isolation, shortages of lactation professionals, limited broadband infrastructure, and variable access to reliable internet services (Demirci et al. 2019; Grubesic and Durbin 2020). These constraints shape not only whether AI-enabled tools can be used, but also how their outputs are interpreted, trusted, and acted upon.

Limited digital literacy in some rural communities may increase vulnerability to AI-generated misinformation, particularly when chatbots present guidance in a fluent, confident, and authoritative language (Mathur 2025). Mothers with fewer opportunities for in-person postpartum support may rely more heavily on AI tools during urgent breastfeeding challenges, amplifying the downstream impact of hallucinated guidance, false citations, or overgeneralized recommendations. In this context, AI literacy, defined as the ability to critically evaluate AI-generated information, understand uncertainty, and recognize when professional care is needed, becomes especially important.

Addressing AI risks in rural breastfeeding support, therefore, requires more than technical safeguards alone. Efforts to deploy AI chatbots in rural maternal health contexts should incorporate user-centered education, clear communication about uncertainty, and design choices that account for variable connectivity and literacy levels. Integrating AI literacy into broader maternal health and digital health initiatives may help ensure that AI-enabled breastfeeding support tools complement, rather than exacerbate, existing rural health disparities.

Future work should prioritize rigorous evaluation of chatbot outputs using expert review and structured quality controls. This includes having lactation consultants and maternal–child health experts assess responses using strong rubrics and checklists, and building a dedicated test set (dataset) of realistic breastfeeding scenarios to measure whether the system gives clinically appropriate answers rather than only fluent or “nice-sounding” ones (Johnson and Straub 2024). Safety testing should also include red teaming, where experts intentionally try to break the system (including jailbreaking attempts) to identify vulnerabilities before real users are exposed (Jiang et al. 2024). Finally, LLMs as judges may be useful for scalable screening of responses against predefined criteria, with expert assessment remaining essential for high-stakes breastfeeding and postpartum scenarios (Szymanski et al. 2025).

In addition to technical safeguards, future work should examine how user perceptions and anthropomorphic tendencies shape reliance on AI breastfeeding support chatbots. Because chatbots often communicate in fluent, confident, and human-like language, users may attribute expertise, intent, or trustworthiness to the system even when outputs are uncertain or incorrect (Heersmink et al. 2024). These tendencies may amplify risks such as hallucinated guidance, false personalization, or inappropriate reassurance, particularly in high-stress postpartum contexts where mothers may seek rapid answers. Incorporating user-centered evaluation, including how caregivers interpret tone, certainty, and perceived “personality” of AI systems, can strengthen mitigation strategies and support safer, more realistic deployment in maternal and child health settings (Maeda and Quan-Haase 2024).

7. Conclusions

AI-driven breastfeeding assistance holds significant promise for addressing long-standing geographic inequities in maternal postpartum support (Handley et al. 2025). Yet the benefits of accessibility must be matched by responsible system design and deployment. Reducing hallucination risks is essential for safeguarding infant health and supporting informed decision-making among rural mothers who already experience structural barriers to care. With rigorous governance, grounding in authoritative evidence, and increased user literacy, AI chatbots can complement clinical care and contribute to a future in which all mothers, regardless of where they live, have timely, reliable access to breastfeeding guidance.