An LLM and Retrieval Pipeline for Reproductive-Health Misinformation and Stigma

Abstract

Background/Objectives: Reproductive-health topics such as polycystic ovary syndrome (PCOS), endometriosis, fertility, and menstruation attract both unsupported claims and stigmatizing framing on social media. We present an evidence-grounded pipeline combining large language models (LLMs) with biomedical retrieval, extended with a novel layer that scores the stigma framing of each claim independently of its truth. Methods: From 991 keyword-sampled English-language Bluesky posts we extracted 937 health claims, grounded each in evidence from PubMed, openFDA, and authoritative clinical guidelines (ACOG, NICE, NHS, WHO), and classified veracity into five categories using only the retrieved evidence. A parallel module scored five stigma dimensions and, separately, empowerment as a counter-stigma indicator. Results: Most claims (75.2%) were non-supported, though a relevance audit shows this largely records evidence that was not retrieved rather than claims shown to be false; the dominant frame was empowerment rather than overt shame. Stigma increased monotonically as claims departed from the evidence. The gradient was modest but robust (Kruskal–Wallis $p < {10}^{-7}$, ${\epsilon }^2 = 0.039$; $\rho = 0.20$), surviving cluster-aware reanalysis of claims nested within posts. Stigma varied sharply by condition, highest for infertility and endometriosis. Conclusions: Non-supported claims and stigmatizing framing co-occur, though this cross-sectional design cannot establish a direction. Single-coder validation was a pilot diagnostic check: stigma scores aligned directionally with human judgment, while five-way veracity agreement was limited. Results describe this keyword-sampled, single-platform corpus, not platform-wide prevalence.

1. Introduction

Health misinformation on social media has emerged as a significant public-health concern, with platforms amplifying unverified claims about treatments, supplements, and self-management strategies to audiences that may lack the clinical context needed to evaluate them [1,2]. While misinformation research has devoted considerable attention to domains such as vaccination, infectious disease, and oncology [3,4], reproductive health remains comparatively understudied despite being a frequent target of online misinformation [5]. Endometriosis is frequently associated with substantial diagnostic delays and negative healthcare experiences, including symptom normalization and perceived dismissal by healthcare providers [6,7]. These challenges drive affected individuals toward online communities and social media, where they encounter information about diet, supplements, and alternative therapies, some of which lack rigorous evidentiary support [8,9]. Identifying and characterizing the misinformation landscape in reproductive-health discourse is therefore an important step toward evidence-informed platform governance and patient education.

A distinguishing characteristic of reproductive-health misinformation is the pervasive role of stigma. Menstruation, fertility, and reproductive illness intersect with longstanding cultural taboos surrounding bodily function, gender, and reproductive autonomy [10,11]. Stigma shapes both the production and consumption of reproductive-health information. Individuals may seek guidance through online communities and social media, while online content often reinforces stigma through language and framing that invokes shame, fear, embarrassment, and other longstanding cultural taboos [12,13]. Although stigma and misinformation frequently co-occur in online health discourse, their relationship remains insufficiently understood. Existing computational research has largely focused either on detecting health misinformation or on identifying stigmatizing language, with relatively little work integrating both dimensions within a unified analytical framework. Related multi-platform computational work has also shown that identity- and health-related online discourse can be characterized at scale [14]. Consequently, the interaction between factual accuracy and stigmatizing framing remains largely unexplored in automated pipelines [15,16].

Recent advances in large language models (LLMs) have expanded the scope of automated health-claim assessment. LLMs can interpret complex health claims, support automated veracity assessment, and generate natural-language explanations for their classification decisions [17,18]. Recent evidence-grounded LLM pipelines have also been applied to climate misinformation and health-product claims on social media [19,20]. However, standalone LLM-based classification is susceptible to hallucination and to reproducing training-data biases, which limits its reliability in safety-critical health applications [21,22]. Retrieval-augmented generation (RAG) addresses this limitation by grounding the model’s reasoning in externally retrieved evidence at inference time [23]. In the biomedical domain, grounding can draw on peer-reviewed biomedical literature retrieved from PubMed, complemented by drug-safety databases such as openFDA and authoritative clinical guidelines from bodies such as the American College of Obstetricians and Gynecologists (ACOG), the National Institute for Health and Care Excellence (NICE), and the World Health Organization (WHO) [24,25].

Despite these methodological advances, two notable gaps persist in the literature. First, existing misinformation-detection systems for health content have largely been developed independently of computational approaches for stigma analysis, with relatively little work integrating emotional framing with evidence-based claim verification. As a result, the interaction between what is said and how it is said—between factual accuracy and affective framing—remains largely unexplored in current automated pipelines. Second, while most social media health-misinformation research has concentrated on established platforms such as Twitter/X and Facebook [26,27], the decentralized microblogging platform Bluesky has received little analytical attention despite its open AT Protocol providing programmatic access to publicly available posts, making it a tractable and increasingly active platform for empirical research [28].

This study addresses both gaps by presenting an evidence-grounded pipeline that combines LLM-based claim extraction and veracity classification with biomedical retrieval from PubMed, openFDA, and authoritative clinical guidelines, and extends the pipeline with a novel affective-framing module that scores five stigmatising dimensions—shame, secrecy, disgust, fear, and blame—together with empowerment as a separate counter-stigma indicator, all independently of a claim’s truth value. The primary contribution is this stigma layer and the joint veracity–stigma analysis it enables; the claim-extraction, retrieval, and veracity-classification components adapt an architecture we have previously applied in other domains [19,20]. Section 2.7 states explicitly which elements of the study are novel, which extend prior work, and which replicate established approaches in a new setting. The pipeline is applied to English-language Bluesky posts concerning PCOS, endometriosis, fertility, menstruation, and related reproductive-health topics. The study is guided by seven research questions: RQ1: What is the veracity distribution of reproductive-health claims in the sampled Bluesky corpus when grounded in biomedical evidence? RQ2: Do claims that depart from biomedical evidence carry stronger stigmatizing framing? RQ3: Does stigma framing vary across reproductive-health conditions? RQ4: Is stigma associated with post-level engagement? RQ5: What is the added value of evidence-grounded classification over a keyword-based baseline? RQ6: How reliably does the automated pipeline agree with human annotation? RQ7: What does retrieval grounding contribute relative to the same model without evidence? As an exploratory single-platform study, results are interpreted within the keyword-sampled corpus rather than as platform-wide prevalence estimates.

2. Literature Review

This section reviews work across four strands relevant to the present study: health misinformation on social media, reproductive-health misinformation and stigma, LLM- and retrieval-based claim verification, and computational stigma detection. It then sets out the conception of stigma we operationalise, and closes by situating our contribution relative to these strands.

2.1. Health Misinformation on Social Media

Health misinformation on social media is a well-documented public-health concern. Systematic reviews have mapped its spread and prevalence across platforms, finding that unverified claims about treatments and self-management strategies reach audiences that often lack the clinical context to evaluate them [1,2]. Much of this work has concentrated on a few high-salience domains, with vaccination and COVID-19 misinformation attracting sustained attention through reviews cataloguing its features, drivers, and candidate interventions [3,26,27], and broader infodemic reviews synthesising evidence across many such studies [16]. Methodologically, the field has matured from descriptive prevalence studies toward automated detection [4]. Reproductive health, however, remains comparatively understudied relative to its prominence as a misinformation target [5].

2.2. Reproductive-Health Misinformation and Stigma

Reproductive-health discourse is distinguished by the tight coupling of misinformation with stigma. Endometriosis is associated with substantial diagnostic delays and negative healthcare experiences, including symptom normalisation and perceived dismissal, which push affected individuals toward online communities in search of guidance [6,7]. In those spaces they encounter claims about diet, supplements, and alternative therapies of varying evidentiary quality; measurable effects of such content on people affected by endometriosis have been documented [8], and cross-platform analyses have characterised the PCOS content landscape [9]. Running alongside this is a long tradition of scholarship on menstrual and reproductive stigma, framing menstruation as a social mark and connecting stigma to gender equity and access to care [10,11]. Qualitative and content-analytic studies show how online language reinforces shame, fear, and secrecy [12,13]. What this body of work largely lacks is a computational treatment that links the affective framing of a claim to its factual accuracy within a single pipeline.

2.3. LLM- and Retrieval-Based Claim Verification

Large language models have expanded the scope of automated health-claim assessment, interpreting complex claims, supporting veracity assessment, and generating natural-language explanations [17,18]. Their standalone use is constrained, however, by hallucination and the reproduction of training-data biases, which is especially problematic in safety-critical health settings [21,22]. Retrieval-augmented generation (RAG) mitigates this by grounding model reasoning in externally retrieved evidence at inference time [23]. In the biomedical domain, grounding can draw on peer-reviewed literature from PubMed alongside drug-safety databases and authoritative clinical guidelines [24,25]. A recent comprehensive review of RAG in healthcare surveys these architectures and notes that their reliability depends not only on whether evidence is retrieved but on whether the retrieved evidence is pertinent, with retrieval noise a recurring failure mode in clinical applications [29]. Recent systems demonstrate the value of this approach for health fact-checking specifically, including RAG-based COVID-19 verification [24] and frameworks that combine retrieved evidence with structured reasoning for biomedical claims [25]. We have previously applied an evidence-grounded LLM pipeline to climate misinformation [19] and to health-product claims on social media [20], and related work has examined multi-perspective LLM reasoning for fake-media detection [30]; the present study builds on this line by adding a stigma layer specific to reproductive health.

2.4. Computational Stigma Detection

A separate computational strand targets stigmatising language directly rather than factual accuracy. Deep-learning models have been developed to detect stigma across platforms, for example for COVID-vaccine stigma [15], and content-analytic studies have quantified menstrual stigma in social-media product posts [13]. More recent work has begun to apply LLMs to stigma reduction in reproductive well-being [31]. These approaches establish that stigma is computationally tractable, but they have developed largely in isolation from misinformation-detection systems, leaving the relationship between what is said and how it is framed unexamined. Multi-platform computational analyses of identity- and health-related discourse illustrate the broader feasibility of characterising affective reactions at scale [14].

2.5. Conceptualising Stigma for Computational Measurement

Stigma is not a single attribute but a process. Goffman [32] described it as an attribute that discredits its bearer in a particular social context, and Link and Phelan [33] decomposed the process into labelling, stereotyping, separation, status loss, and discrimination. Operationalising this at the level of a single social-media claim requires a narrower target: what is observable in a short text is not the social process itself but its discursive trace—the framing through which a body, a condition, or a person is marked as discrediting.

We therefore treat the five stigmatising dimensions as observable markers of distinct components of that process rather than as stigma itself. Shame and disgust correspond most directly to the attribute-devaluation at the core of Goffman’s account: the body or condition is presented as discrediting or impure, a framing documented at length in menstrual-stigma scholarship [10,13]. Secrecy is a behavioural consequence of anticipated devaluation—concealment presupposes that disclosure carries cost—and is well attested as a response to menstrual stigma [12]. Blame corresponds to the attribution of responsibility that converts a condition into a moral failing, the mechanism by which stereotyping shades into separation [33]. Fear is the most heterogeneous of the five: catastrophising about a reproductive condition is not itself stigmatising, but in this domain it typically operates by presenting the condition as a threat to be avoided, which is a stigmatising move rather than a neutrally negative one. We flag this dimension as the weakest of the five on construct grounds and return to it in Section 5.4.

Two boundaries follow from this framing. First, stigmatising framing is not the same as negative affect. A claim can express distress, pain, or frustration without marking the condition or the person as discrediting; the dimensions above are designed to capture the marking, not the negative valence. This distinction is imperfectly enforced by an automated scorer and is a source of measurement error we acknowledge rather than resolve. Second, empowerment is not a dimension of stigma. Normalising, destigmatising, and openly disclosing framings run counter to the stigma process, and treating them as a sixth stigma dimension would conflate a construct with its opposite. We therefore score empowerment alongside the other five but exclude it from the composite stigma score, and interpret it as an indicator of counter-stigmatising discourse (Section 3.6).

2.6. Summary and Positioning

Table 1 summarizes these strands. Two gaps motivate the present study. First, misinformation detection and stigma analysis have advanced on parallel but separate tracks, leaving the interaction between factual accuracy and affective framing unexplored in automated pipelines. Second, empirical health-misinformation research has concentrated on established platforms such as Twitter/X and Facebook, while the decentralised platform Bluesky has received little analytical attention despite its open AT Protocol enabling programmatic access to public posts [28]. We address both by combining evidence-grounded LLM veracity classification with a parallel, truth-independent stigma-scoring module, applied to a Bluesky corpus.

Table 1. Summary of related work by theme and its relation to this study.

Theme	Representative Work	Relation to This Study
Health misinformation on social media	[1,2,4,16]	Establishes prevalence and the shift to automated detection; we extend it to an understudied domain.
Reproductive-health misinformation & stigma	[5,8,9,10]	Defines the domain and the stigma–misinformation coupling we model jointly.
LLM + retrieval claim verification	[17,19,20,23,24,29]	Basis for our evidence-grounded veracity classifier.
Computational stigma detection	[13,15,31]	Basis for our parallel stigma-scoring module, run independently of truth.
Decentralised platform research	[28]	Justifies our single-platform Bluesky corpus choice.

2.7. Contributions of This Study

Because the study combines several components of differing originality, we state their status explicitly rather than presenting the work as uniformly novel (Table 2). The primary contribution is the truth-independent stigma layer and the joint veracity–stigma analysis it makes possible. The evidence-retrieval and veracity-classification components are an adaptation of a pipeline architecture we have applied previously to climate misinformation [19] and to health-product claims [20]; their contribution here is the demonstration that the approach transfers to reproductive-health discourse, together with two methodological additions—the controlled retrieval ablation and the relevance audit—that were not part of those earlier applications. Readers evaluating the significance of the study should weight the stigma layer and the coupling it reveals, not the retrieval pipeline itself.

Table 2. Status of each component of the study relative to prior work.

Component	Status	Relation to Prior Work
Parallel affective-framing module, scored independently of the veracity label	Novel	Existing systems detect either misinformation or stigmatising language; none score both on the same claim within one pipeline.
Joint veracity–stigma analysis (gradient across labels; variation across conditions)	Novel	Enabled by the module above; the relationship it measures has not previously been quantified computationally.
Controlled retrieval ablation (same model, same prompt, evidence withheld)	Extension	Prior pipelines benchmark against lexical baselines; this isolates the grounding layer itself and quantifies its behavioural effect.
Relevance audit of retrieved evidence	Extension	Retrieval coverage is commonly reported; per-claim relevance, broken down by label, is not [29].
Claim extraction, biomedical retrieval, and evidence-grounded veracity classification	Replication in a new domain	Architecture adapted from [19,20]; the contribution is evidence that it transfers to reproductive health.
Bluesky corpus	Application	An understudied platform [28], but the sampling and collection approach is standard.

3. Materials and Methods

3.1. Pipeline Overview

We implemented an end-to-end, claim-level computational pipeline for identifying, grounding, and characterising reproductive-health claims in social media posts. The workflow begins with keyword-based collection of English-language Bluesky posts and continues through large-language-model claim extraction, biomedical evidence retrieval, evidence-grounded veracity classification, truth-independent stigma scoring, statistical analysis, retrieval ablation, and human validation. The unit of analysis after extraction is the individual health claim rather than the original post, because a single post may contain more than one distinct claim.

Figure 1 summarises the full pipeline. The figure should be read from left to right: social media posts are first converted into structured claims; each claim is then linked to PubMed, openFDA, and guideline evidence; the evidence-grounded classifier assigns a veracity label; a parallel stigma module scores the framing of the same claim; and the resulting claim-level dataset is used for descriptive statistics, inferential tests, ablation, and human validation.

Figure 1. Main computational workflow of the evidence-grounded LLM pipeline.

3.2. Data Collection from Bluesky

English-language social media data were collected from Bluesky, a public microblogging platform built on the AT Protocol. Bluesky was selected as the English-language social media source because it provides programmatic access to publicly available posts through its open protocol, while allowing collection without storing direct user identifiers. Data collection was performed using Python version 3.11.9 and the atproto client version 0.0.68, with authentication through a Bluesky app password.

A predefined set of search queries was used to identify posts related to women’s reproductive health, fertility, and health-product claims. The queries targeted topics frequently associated with online health misinformation, including polycystic ovary syndrome, endometriosis, menstrual regulation, fertility, hormonal contraception, menopause, dietary remedies, herbal products, and supplement use. Examples of search terms included “pcos cure”, “pcos natural”, “endometriosis diet”, “seed cycling”, “inositol pcos”, “birth control infertility”, “fertility supplements”, “period detox”, and “menopause natural remedy”.

For each search query, the pipeline retrieved up to 100 English-language posts, with pagination applied in batches of 25 posts. A short delay was introduced between requests to reduce the intensity of automated querying. Retrieved posts were then screened using a predefined health-claim keyword list. Posts were retained only if they contained sufficient textual content and matched at least one relevant health-claim keyword. Duplicate records were removed using a stable hashed identifier generated from the post URI.

To reduce privacy risks, the dataset did not store author handles, decentralized identifiers, profile metadata, or direct post URLs. Each retained post was represented using a normalized schema containing a non-identifying hashed ID, platform name, language, post text, engagement counts, and post type. The resulting Bluesky records were stored in JSONL format and were also written to a combined dataset file for downstream claim extraction, evidence retrieval, classification, and statistical analysis.

3.3. Health Claim Extraction

Health claims were extracted from the collected social media posts using a large language model-based information extraction pipeline. The input to this stage consisted of the combined social media dataset generated during the data collection stage. Each post was processed individually, and posts with insufficient textual content were excluded before claim extraction.

The extraction prompt was designed to identify distinct health-related claims in posts concerning menstrual or reproductive health. A claim was defined as any statement asserting that a product, food, supplement, behaviour, or intervention has an effect on a menstrual condition, reproductive condition, fertility-related outcome, or general health. Each extracted claim’s structured fields and verbatim supporting quote were retained in the original English text of the post.

For each post, the model was instructed to return a JSON array containing all distinct claims identified in the text. Each extracted claim included the named product or intervention, the claimed effect, the target condition, the strength of the claim, a short verbatim quote from the original post, and a risk category. If a post did not contain a health claim, the model returned an empty array. This design allowed posts containing multiple claims to contribute more than one structured record, while posts without explicit health claims were excluded from the claim-level dataset.

To ensure compatibility with downstream analysis, each extracted claim was augmented with provenance metadata from the source post. These metadata included the source post identifier, platform, engagement counts, number of comments, and language. The final claim-level dataset was saved in JSONL format, with one structured JSON object per extracted claim. This file served as the input for subsequent evidence retrieval, claim classification, validation, and statistical analysis.

3.4. Biomedical Evidence Retrieval

To ground each extracted health claim in external biomedical evidence, an automated evidence-retrieval stage was implemented. The input to this stage was the claim-level dataset generated during the claim extraction step. For each structured claim, the pipeline constructed a search query using the extracted product or intervention together with either the target condition or, when no specific condition was available, the claimed effect. This allowed the evidence search to be tailored to the substantive content of each claim rather than relying only on the original social media text.

Biomedical literature was retrieved from PubMed through the NCBI Entrez utilities. The search strategy prioritized higher-level clinical evidence by restricting retrieval to systematic reviews, meta-analyses, and randomized controlled trials. For each claim, up to five PubMed records were retrieved. The extracted evidence fields included the PubMed identifier, article title, publication year, abstract text, publication type, and PubMed URL. This provided a structured evidence base for assessing whether the social media claim was supported, exaggerated, unsupported, or contradicted by biomedical literature.

In addition to PubMed, the pipeline queried the openFDA drug adverse event database for products or substances named in the claim. This step was used to capture reported adverse reactions associated with the mentioned product where such records were available. For each matched product, the pipeline retrieved a limited number of adverse event reports and stored the reported reactions and seriousness indicator. The purpose of this step was not to estimate incidence or causality, but to provide safety-related contextual evidence for later classification.

The evidence-retrieval stage also linked claims to a curated set of authoritative reproductive-health resources when the target condition matched predefined categories. These references included guidance or information sources related to polycystic ovary syndrome, endometriosis, fertility, pregnancy, menopause, menstruation, and dietary supplements. Depending on the condition, the pipeline attached references from sources such as ACOG, NICE, NHS, WHO, Cochrane, or the NIH Office of Dietary Supplements. These references provided an additional clinical context for claims where guideline-level or patient-facing authoritative information was relevant.

For each claim, the retrieved evidence was stored in a structured evidence object containing the PubMed query, retrieved PubMed articles, openFDA adverse event records, authoritative reference, and total evidence count. The resulting evidence-augmented dataset was saved in JSONL format, with one record per claim. This dataset served as the input for the subsequent claim classification stage.

3.5. Evidence-Grounded Veracity Classification

Each evidence-augmented claim was classified using an evidence-grounded large language model pipeline. The input to this stage was the claim-level dataset produced after biomedical evidence retrieval. For each claim, the model received the structured claim fields, including the product or intervention, claimed effect, target condition, claim strength, and original verbatim quote, together with the retrieved biomedical evidence.

The classification prompt instructed the model to assess the claim using only the evidence provided by the retrieval stage. Evidence supplied to the model included PubMed records, openFDA adverse-event reports, and authoritative reproductive-health references where available. PubMed evidence was summarized using article metadata, publication type, title, year, and abstract excerpts. Safety-related evidence from openFDA was included as reported adverse reactions, while guideline or reference information was included for relevant reproductive-health conditions.

Claims were assigned to one of five mutually exclusive veracity categories. The schema separates two distinct judgments. Four labels describe the relationship between the claim and the retrieved evidence, and are distinguished first by whether relevant evidence was retrieved at all and, if so, by the direction in which it pointed: SUPPORTED when relevant evidence was retrieved and was consistent with the claim as stated; EXAGGERATED when relevant evidence supported a weaker version of the claim but the claim overstated the magnitude, certainty, or generality of the effect; CONTRADICTED when relevant evidence was retrieved and opposed the claim; and UNSUPPORTED when no relevant confirming evidence was identified. The distinction between the last two is therefore evidence present and negative versus evidence absent, and UNSUPPORTED does not assert that the claim is false. The fifth label, DANGEROUS, is a harm judgment rather than an evidentiary one and takes precedence over the other four: it was applied whenever acting on the claim could plausibly cause harm—for example by substituting for indicated medical care, by advising discontinuation of prescribed treatment, or by recommending a contraindicated substance or dose—irrespective of what evidence was retrieved. Because DANGEROUS is assigned on the harm axis rather than the evidence axis, such claims need not have any retrieved trial evidence, which accounts for the retrieval pattern reported in Section 4.4. Table 3 states the operative criterion for each label alongside a representative claim from the corpus. In addition to the veracity label, the model returned a confidence score, a risk tier, a short explanation, key supporting or opposing evidence, and a brief recommendation.

Table 3. Operative criterion for each veracity label, with a representative claim from the corpus. DANGEROUS is assigned on the harm axis and overrides the four evidence-axis labels.

Label	Operative Criterion	Representative Claim and Rationale
SUPPORTED	Relevant evidence retrieved; consistent with the claim as stated.	“I take the pill to regulate my period” (combined oral contraceptive → cycle regulation). Retrieved trials and guideline sources support cycle regulation as an established effect of the named intervention.
EXAGGERATED	Relevant evidence retrieved; supports a weaker effect than claimed (magnitude, certainty, or generality overstated).	“you can cure your PCOS with diet and supplements” (diet and supplements → cure PCOS). Evidence supports symptom management, not cure; the claim overstates a partially supported effect.
UNSUPPORTED	No relevant confirming evidence identified. Records absence of retrieved evidence, not falsity.	“My face is full of acne scars from…missing my PCOS meds” (PCOS medication → prevents acne scarring). No high-level article addressed this intervention–outcome pair; the label records that absence.
CONTRADICTED	Relevant evidence retrieved and opposing the claim.	“does getting a hysterectomy cure all pcos symptoms” (hysterectomy → cures PCOS). Retrieved evidence opposes the claim: PCOS is an endocrine disorder not resolved by uterine removal.
DANGEROUS	Acting on the claim could plausibly cause harm; overrides the evidence-axis labels and does not require retrieved evidence.	“I was able to cure my girlfriend’s type 2 diabetes and PCOS using Kefir and Kombucha” (fermented drinks → cure T2D and PCOS). No PubMed article was retrieved; the label reflects the harm of substituting this for diabetes management.

Throughout this paper we use non-supported as the collective term for the four labels other than SUPPORTED. This is deliberate: these labels record the outcome of a retrieval and classification procedure, not an adjudication of truth. We reserve misinformation for cases in which relevant evidence was retrieved and opposed the claim, and do not use it as a synonym for the non-supported block as a whole.

To provide a transparent comparator, a rule-based keyword baseline was also implemented. This baseline classified claims as DANGEROUS when the claim text contained high-risk expressions such as “cure”, “stop taking”, “instead of”, “no need for doctor”, “guaranteed”, “miracle”, or related terms. Claims containing strong exaggeration markers such as “always”, “completely”, “permanently”, “instantly”, or “never fails” were classified as EXAGGERATED. Claims not matching these rules were assigned to the UNSUPPORTED category by default. The keyword baseline was not intended to replace the evidence-grounded model, but to provide a simple and interpretable comparison method.

For each claim, both the evidence-grounded classification and the keyword-baseline classification were stored under a structured classification field. The resulting classified dataset was saved in JSONL format and used for downstream descriptive statistics, inferential analysis, validation, and reporting.

3.6. Affective-Framing (Stigma) Scoring

Each claim was scored a second time by the same model in a separate call that received only the verbatim quote and its product and condition context, and not the retrieved evidence or the veracity label. The two judgments are therefore independent by construction: the stigma score cannot be a restatement of the veracity label, which is what makes the association reported in Section 4.11 interpretable.

The model rated six affective dimensions on a 0–1 scale. Five are stigmatising markers, defined as set out in Section 2.5: shame (embarrassment about one’s own body or cycle), secrecy (concealment, the premise that the condition must be hidden), disgust (framing bodies or menstruation as dirty or impure), fear (catastrophising about symptoms or outcomes), and blame (moralising or lifestyle attribution of responsibility for a condition). The sixth, empowerment, captures normalising, destigmatising, or openly disclosing framing and is scored as a counter-stigma indicator.

The model additionally returned a single dominant frame and a composite stigma_score on the same 0–1 scale. The prompt instructs explicitly that empowerment does not contribute to the composite score; the lexicon baseline computes the composite over the five stigmatising dimensions only, likewise excluding empowerment. All analyses reported as “stigma” in Section 4 use this composite. Where empowerment is reported, it is reported as a dominant frame or as a separate dimension intensity, never as a component of the stigma score.

3.7. Descriptive Analysis and Figure Generation

Descriptive analyses were performed on the classified claim-level dataset. The analysis summarized the total number of extracted claims, the number of languages represented, the distribution of evidence-grounded veracity labels, the distribution of risk categories, dominant affective frames, and the mean stigma score within each veracity group. These summaries were saved as structured JSON output to support reproducibility and transparent reporting.

Publication figures were generated using Python version 3.11.9 and Matplotlib version 3.8.2. The figures described the distribution of claim veracity and risk categories, the relationship between claim veracity and stigma score, and the variation of stigma dimensions across reproductive-health conditions. The engagement analysis and the comparison with the keyword-based baseline were summarized in text, because their main findings were adequately captured by the reported statistics and did not require separate visualization. When more than one language was present in the dataset, an additional language-comparison figure was generated to compare mean stigma scores across language groups.

The figure-generation pipeline was organized around the study research questions. Veracity and risk-category distributions addressed the overall composition of non-supported claims. The comparison of stigma scores across veracity labels examined whether unsupported, exaggerated, contradicted, or dangerous claims carried stronger stigmatizing framing than supported claims. The condition-by-stigma heatmap evaluated whether stigma dimensions differed across reproductive-health topics. Engagement and keyword-baseline results were retained as text-based analyses because the relevant numerical patterns were simple and were more clearly reported in prose.

For qualitative inspection, the pipeline also exported representative examples for each veracity category. Claims within each category were sorted by the confidence score assigned by the evidence-grounded classifier, and the highest-confidence examples were written to separate text files. These examples were used to support interpretation of the quantitative results and to illustrate typical patterns within each veracity class.

3.8. Inferential Statistical Analysis

Inferential statistical analyses were conducted to evaluate the relationships between claim veracity, stigma, reproductive-health condition, and engagement. Because stigma scores were bounded between 0 and 1 and were not assumed to be normally distributed, non-parametric statistical tests were used.

To test whether stigma differed across veracity groups, a Kruskal–Wallis test was applied to stigma scores grouped by evidence-grounded veracity label. The corresponding effect size was reported using epsilon-squared. In addition, veracity was encoded ordinally from SUPPORTED to DANGEROUS, and Spearman correlation was used to examine whether stigma scores increased as claims became less supported or more potentially harmful.

Pairwise comparisons were performed between SUPPORTED claims and each non-supported category, including UNSUPPORTED, EXAGGERATED, CONTRADICTED, and DANGEROUS. These comparisons used the Mann–Whitney U test. To control for multiple testing, Holm correction was applied to the resulting p-values. Rank-biserial correlation was reported as an effect-size measure for each pairwise comparison.

To evaluate whether stigma differed across reproductive-health conditions, claims were grouped by target condition and a Kruskal–Wallis test was applied to the most frequent condition groups with sufficient sample size. The association between stigma and engagement was assessed using Spearman correlation between stigma score and post score. This analysis was interpreted cautiously because social media engagement counts, especially on Bluesky, may be sparse and platform-dependent.

All inferential statistics were saved to a structured JSON file, including sample sizes, group means, test statistics, p-values, and effect sizes. This allowed the statistical results to be reproduced and incorporated consistently into the Results section.

3.9. Retrieval Quality Evaluation

Coverage indicates whether evidence was returned, not whether that evidence was pertinent, and retrieval noise is a recognised failure mode of retrieval-augmented systems in clinical domains [29]. We therefore added a relevance audit. For a stratified sample of 149 claims that had returned at least one PubMed article, each retrieved abstract was rated for relevance to the specific claim on a three-point scale (2 = directly relevant, addressing both the named intervention and the target condition or claimed outcome; 1 = partially relevant; 0 = not relevant). Ratings assessed relevance only and were made without reference to the claim’s veracity label. We report article-level relevance, the proportion of claims retrieving at least one directly relevant article, and both quantities broken down by veracity label. We additionally report, for all 937 claims, the proportion retrieving no PubMed article at all, by veracity label.

3.10. Retrieval Ablation

To quantify the contribution of the retrieval layer itself, rather than only comparing the full pipeline against a lexical baseline, we conducted a controlled ablation. Each of the 937 claims was re-classified by the same model (claude-haiku-4-5) using an identical prompt, system message, label schema, and parser, with the single difference that the retrieved evidence block was removed and the model was instructed to rely on its own parametric knowledge. Because every component except the evidence was held constant, any divergence in the resulting labels is attributable to grounding alone. We report exact label agreement between the grounded and ungrounded configurations, the full confusion matrix, the marginal label distributions, and two safety-relevant directional quantities: the number of claims the grounded pipeline flagged as problematic but the ungrounded model relabeled SUPPORTED, and the number of CONTRADICTED or DANGEROUS verdicts lost when evidence was removed.

3.11. Human Validation Procedure

To evaluate the reliability of the automated classifications, a blind human-validation workflow was implemented. A stratified sample of claims was drawn from the classified dataset, with sampling stratified by veracity label to ensure representation of smaller classes such as DANGEROUS or CONTRADICTED. The default validation sample size was 150 claims, although the script allowed this value to be adjusted.

The validation workflow generated two separate files. The first file was a blind coding sheet containing the claim product, claimed effect, target condition, and short verbatim quote. A single human coder (author B.B.), blinded to all model output, was asked to assign a veracity label using the same five-category schema and to provide a stigma score between 0 and 1. The coder could also flag records that were not clearly rateable as health claims, including questions, irrelevant posts, underspecified interventions, or ambiguous claims. Because validation was performed by one coder only, inter-rater reliability could not be estimated, and this stage is reported throughout as a pilot diagnostic check on pipeline behaviour rather than as a multi-coder gold-standard evaluation. Flagged records were reviewed separately as claim-extraction quality-control cases. Records left uncoded for human veracity were excluded from the veracity agreement analysis, and records without a human stigma score were excluded from the stigma agreement analysis. The second file was a hidden key containing the corresponding model-assigned veracity labels and stigma scores for the sampled claims.

After human coding, agreement between the automated labels and human annotations was assessed by merging the completed blind coding sheet with the hidden key by row identifier. For categorical veracity labels, exact agreement, unweighted Cohen’s kappa, and a confusion matrix were calculated. Because the veracity labels can also be interpreted as an ordinal scale from supported to dangerous, quadratic-weighted kappa was additionally computed. For stigma scores, agreement was evaluated using Spearman correlation and mean absolute error between human and model-assigned scores.

This validation design provided an independent check on the automated classification pipeline while avoiding circular evaluation, because the coder did not see the model outputs during annotation. The resulting validation metrics were saved as a structured JSON file for reporting in the Results section.

3.12. Cluster-Aware Statistical Analysis

Because a single post can yield more than one claim, claim-level observations are nested within posts and are not fully independent. We therefore re-examined the two claim-level associations under three approaches that relax the independence assumption. First, we quantified the nesting itself: a one-way random-effects intraclass correlation (ICC) of the stigma score across posts, and the corresponding design effect and effective sample size. Second, RQ2 was re-tested using (i) a cluster bootstrap resampling posts rather than claims (2000 resamples) and (ii) 2000 draws of one randomly selected claim per post, which imposes strict independence at the cost of discarding data. Third, we fitted a linear mixed model of stigma score on ordinally coded veracity with a random intercept for post. For RQ4, because engagement is a property of the post rather than of the claim, the analysis was repeated at post level, aggregating stigma within post by both the mean and the maximum.

3.13. Temporal Robustness Analysis Design

As an auxiliary temporal robustness check, we conducted an extended timestamped re-collection of raw Bluesky posts using the same reproductive-health search queries. These posts were aggregated by month and compared with monthly Google Trends search-interest series for the same query terms. This post-level analysis was used only to assess whether the sampled discourse showed obvious temporal instability or co-movement with external public-attention signals, and was not treated as a forecasting model or as part of the main claim-level validation.

3.14. Pipeline Configuration and Reproducibility

All stages of the pipeline used a shared configuration file to ensure consistent paths, labels, thresholds, and vocabulary across the study. The configuration file defined the directory structure for raw data, extracted claims, evidence-augmented claims, classified claims, evaluation outputs, and validation files. It also specified the minimum text-length threshold used for filtering short posts.

The same configuration file defined the controlled vocabularies used throughout the pipeline, including claim-strength categories, risk categories, veracity labels, risk tiers, stigma dimensions, and stigma frames. Risk categories included supplement efficacy, menstrual management, fertility and conception, pregnancy and prenatal claims, condition-cure claims, hormonal birth-control claims, detox claims, and other claims. Veracity labels included SUPPORTED, UNSUPPORTED, EXAGGERATED, CONTRADICTED, and DANGEROUS. The affective-framing vocabulary comprised the five stigmatising dimensions (shame, secrecy, disgust, fear, blame) together with empowerment and neutral framing; only the first five enter the composite stigma score.

Credentials and API keys were not hard-coded in the scripts. Instead, external services were accessed through environment variables, including credentials for Bluesky, the large language model API, and PubMed Entrez. This design reduces the risk of credential exposure and improves reproducibility across computing environments. The shared configuration file therefore served as the central control layer for the full computational pipeline.

4. Results

The Results section follows the logic of the pipeline and the study research questions. We first report the corpus, claim yield, and evidence-retrieval coverage. We then present the veracity distribution of the extracted claims, the stigma profile of the corpus, and the relationship between stigma and veracity. The final subsections report condition-specific stigma patterns, engagement analysis, comparison with the keyword baseline, the retrieval ablation, and the human-validation audit.

4.1. Collected Corpus

The data collection stage produced a corpus of 991 English-language Bluesky posts matching the predefined reproductive-health and health-product search queries. After hashed-identifier deduplication, all 991 records were unique, with no duplicate posts retained. The corpus was uniform on the collection-design dimensions: all posts were classified as English ($n = 991$) and all originated from Bluesky, consistent with the single-platform, single-language scope of this study.

Retained posts were of moderate length, with a median text length of 245 characters (range 33–302; mean 221.1). Engagement was present but sparse, as expected for Bluesky. Post-level engagement scores had a median of 1 and a mean of 10.6, with a small number of highly engaged posts producing a long right tail (maximum 4005); 625 posts (63.1%) received an engagement score greater than zero. Reply activity was lower, with a median of 0 comments (mean 0.73, maximum 29) and 474 posts (47.8%) receiving at least one comment. These distributions informed the cautious interpretation of the later stigma–engagement analysis, given the skewed and platform-dependent nature of Bluesky engagement counts.

This corpus served as the input to the claim-extraction stage described in Section 3.

4.2. Extracted Health Claims

Applying the claim-extraction pipeline to the 991-post corpus yielded 937 distinct health claims. These claims originated from 574 of the 991 posts (57.9%), indicating that the remaining posts either contained no explicit health claim or insufficient textual content and were therefore excluded from the claim-level dataset. Posts contributing claims yielded a mean of 1.63 claims each; while most contributing posts produced a single claim ($n = 353$), a substantial share produced two ($n = 132$), three ($n = 53$), or more (up to six claims from a single post). Because multiple claims can derive from the same post, this nested structure was addressed by the cluster-aware reanalysis in Section 4.12.

Extracted claims varied in assertive strength. Most were coded as suggestive ($n = 483$; 51.5%) or anecdotal ($n = 332$; 35.4%), while a smaller subset made definitive assertions ($n = 122$; 13.0%). The claims spanned the full set of predefined risk categories, with the largest groups concerning supplement efficacy ($n = 219$), condition-cure claims ($n = 175$), menstrual management ($n = 172$), and fertility and conception ($n = 152$). Hormonal birth-control claims ($n = 115$), a residual “other” category ($n = 89$), detox claims ($n = 12$), and pregnancy or prenatal claims ($n = 3$) accounted for the remainder.

At the condition level, claims concentrated on a small number of reproductive-health topics. Polycystic ovary syndrome (PCOS) was by far the most frequent target condition, followed by endometriosis and a broad “general health” category, with fertility- and conception-related conditions also well represented. This distribution reflects the keyword-driven sampling strategy and is consistent with the prominence of PCOS and endometriosis in online reproductive-health discussion.

4.3. Evidence Retrieval Coverage

The evidence-retrieval stage attached external grounding evidence to the 937 extracted claims from three complementary sources: PubMed, the openFDA adverse-event database, and a curated set of authoritative reproductive-health references. Overall coverage was high: 914 of the 937 claims (97.5%) were linked to at least one evidence item, with a mean of 4.9 evidence items per claim (median 5). Only 23 claims (2.5%) could not be matched to any PubMed article, adverse-event report, or guideline reference, and were therefore classified by the downstream model under conditions of explicitly absent evidence.

PubMed retrieval, restricted to systematic reviews, meta-analyses, and randomized controlled trials, returned at least one article for 575 claims (61.4%), yielding 2342 articles in total (mean 2.5 across all 937 claims; 4.1 among the 575 claims that returned any article). The distribution of retrieved articles was markedly bimodal: 390 claims reached the retrieval ceiling of five articles, while 362 claims (38.6%) returned no high-level clinical evidence at all, with relatively few claims in between. This pattern reflects the nature of the underlying claims, as well-studied interventions for conditions such as PCOS returned abundant trial-level evidence, whereas niche products, novel remedies, and vaguely specified “general health” claims frequently had no corresponding high-level literature.

The openFDA query matched at least one adverse-event report for 528 claims (56.4%), contributing 1410 reports in total. Consistent with the stated design, these records were used only as safety-relevant contextual signals rather than as estimates of incidence or causality. Authoritative reproductive-health references provided the broadest coverage, with 837 claims (89.3%) linked to a guideline-level or patient-facing source; the most frequently attached references concerned PCOS and endometriosis (ACOG and NICE guidance), followed by fertility guidance and the NIH Office of Dietary Supplements fact sheets.

Considering the sources jointly, 508 claims were grounded in both PubMed and an authoritative reference, 67 in PubMed alone, and 329 in an authoritative reference alone. The complementarity of these sources is important for interpreting the classification stage: while high-level trial evidence was available for only about three-fifths of claims, the guideline reference layer ensured that the large majority of claims were assessed against at least some form of external clinical context rather than in an evidentiary vacuum.

4.4. Retrieval Quality

Coverage was high, but relevance was uneven, and unevenly distributed across the veracity labels. In the rated sample of 149 claims (613 abstracts), 35.4% of retrieved abstracts were judged directly relevant to the claim, 40.0% partially relevant, and 24.6% not relevant (mean relevance 1.11 on the 0–2 scale). Two thirds of rated claims (65.1%) retrieved at least one directly relevant article, while 6.7% retrieved nothing relevant at all.

Relevance varied sharply by label (Table 4). Claims the pipeline judged SUPPORTED were well served by retrieval: 89.7% returned at least one PubMed article, and 96.3% of those returned at least one directly relevant article. Claims judged UNSUPPORTED were not: only 44.4% returned any article, and only 37.9% of those returned a directly relevant one. Combining the two stages, an estimated 86% of SUPPORTED claims but only 17% of UNSUPPORTED and 11% of CONTRADICTED claims were assessed against at least one directly relevant high-level article. All 11 DANGEROUS claims returned no PubMed article whatsoever, and were therefore ineligible for relevance rating; those verdicts rest on the guideline reference layer and on the model’s reading of the claim text rather than on retrieved trial evidence.

Table 4. Retrieval quality by veracity label. Column 2 is computed over all 937 claims; columns 3–4 over the rated sample of 149 claims with at least one retrieved article. Column 5 is the product of columns 2 and 4, an estimate of the share of all claims in that label assessed against at least one directly relevant article.

Label	≥1 Article	Rated n	≥1 Directly Relevant	Estimated Combined
SUPPORTED	89.7%	54	96.3%	86%
EXAGGERATED	86.8%	26	80.8%	70%
UNSUPPORTED	44.4%	58	37.9%	17%
CONTRADICTED	58.7%	11	18.2%	11%
DANGEROUS	0%	0	—	0%

This pattern has a direct bearing on how the veracity distribution should be read. The UNSUPPORTED category conflates two distinct situations: claims for which relevant high-level evidence was retrieved and did not support the claim, and claims for which no relevant evidence was retrieved at all. The latter dominates. UNSUPPORTED is therefore better understood as evidence not identified than as evidence of absence, and the same caution applies, more acutely, to the small CONTRADICTED and DANGEROUS categories. We return to this in Section 5.4.

4.5. Veracity Distribution of Claims

Applying the evidence-grounded classifier to all 937 claims produced a five-way veracity distribution dominated by claims that biomedical evidence did not support. Only 232 claims (24.8%) were classified as SUPPORTED. The majority, 505 claims (53.9%), were UNSUPPORTED, reflecting the large share of vague, anecdotal, or under-studied assertions for which adequate confirming evidence was not available. A further 114 claims (12.2%) were EXAGGERATED, 75 (8.0%) were CONTRADICTED by the retrieved evidence, and 11 (1.2%) were judged DANGEROUS. Taken together, 705 claims (75.2%) fell into one of the four non-supported categories. These proportions should be read in light of the keyword-driven sampling strategy, which deliberately targeted claim-rich and suspicion-prone topics, and therefore do not represent the prevalence of misinformation across Bluesky as a whole (Figure 2).

Figure 2. Distribution of evidence-grounded veracity labels across the 937 claims.

The classifier reported a mean confidence of 0.80 (median 0.85) across its assignments. The accompanying risk-tier field placed most claims at LOW (39.2%) or MODERATE (51.5%) risk, with smaller proportions at HIGH (3.5%) and CRITICAL (5.8%) tiers, indicating that while non-supported claims were common, those posing the most acute potential for harm formed a small subset of the corpus (Figure 3).

Figure 3. Distribution of claims across the predefined risk categories.

4.6. Stigma Framing of Reproductive-Health Discourse

The affective-framing layer characterized the framing of each claim independently of its veracity. Two quantities are reported here and they are distinct. The composite stigma score, computed over the five stigmatising dimensions only, was low across the corpus (0.18 on a 0–1 scale; median 0.05), and just over half of the claims (52.2%) carried any non-zero stigma signal. Separately, each claim was assigned a single dominant frame, and the most common was empowerment (640 claims; 68.3%). The two are consistent rather than in tension: a corpus in which most claims are dominated by normalising or destigmatising framing is expected to have a low mean stigma score, since empowerment does not enter that score. Among the negatively valenced frames, fear (12.3%) and blame (10.2%) were most prominent, while explicit shame (1.1%), disgust (0.7%), and secrecy (0.4%) frames were comparatively rare. This profile suggests that, in this corpus, stigma manifested less as overt expressions of shame or impurity and more through fear-based catastrophizing and moralizing or lifestyle-blame framing.

4.7. Association Between Veracity and Stigma

A central question of this study was whether less accurate claims carried stronger stigmatizing framing. Mean stigma scores increased monotonically across the veracity scale, from SUPPORTED claims (mean 0.120) to UNSUPPORTED (0.167), EXAGGERATED (0.275), CONTRADICTED (0.297), and DANGEROUS claims (0.336). In other words, claims that departed further from the biomedical evidence, and those with greater potential for harm, were on average framed in more stigmatizing terms, while evidence-supported claims were the least stigmatizing (Figure 4). The statistical significance of this gradient is evaluated in Section 4.11.

Figure 4. Mean stigma score by veracity label. Error bars indicate standard error. Stigma increases monotonically as claims become less supported by biomedical evidence.

4.8. Agreement Between the Evidence-Grounded Model and the Keyword Baseline

To benchmark the evidence-grounded classifier against a transparent comparator, its veracity labels were compared with those of the rule-based keyword baseline. The two methods agreed on 469 of 937 claims (50.1%). This limited agreement is partly structural: by construction, the keyword baseline can only emit three of the five labels and never assigns SUPPORTED or CONTRADICTED. In practice the baseline assigned the large majority of claims to UNSUPPORTED (813 claims), with only 117 flagged DANGEROUS and 7 EXAGGERATED, because it relies on surface lexical cues such as the words “cure” or “miracle” rather than on retrieved evidence. The divergence illustrates that lexical heuristics capture only a narrow, keyword-triggered slice of problematic claims and cannot reproduce the evidence-grounded distinctions, particularly the identification of supported claims and of claims directly contradicted by the literature.

4.9. Contribution of the Retrieval Layer (Ablation)

Removing the retrieved evidence changed the verdict on more than half of all claims: the grounded and ungrounded configurations agreed on only 455 of 937 claims (48.6%). This ablation isolates a different quantity from the keyword-baseline comparison above: there the contrast was between two distinct methods, whereas here it is the same model with and without retrieved evidence. The direction of disagreement is the substantive finding. Without evidence, the model became markedly more credulous, reassigning a large share of claims to SUPPORTED (407 vs. 232 grounded) while UNSUPPORTED collapsed from 505 to 164 (Table 5). Concretely, 209 claims that the grounded pipeline placed in a non-supported category were relabelled SUPPORTED once evidence was withheld, and 31 of the model’s CONTRADICTED or DANGEROUS verdicts disappeared. The DANGEROUS category was almost entirely lost, falling from 11 grounded verdicts to 2. These shifts show that the retrieval layer is not a presentational detail: it is what makes the model’s verdicts markedly more conservative. We emphasise that this is a statement about behaviour, not about correctness. The ablation establishes that grounding changes the labels in a safety-relevant direction; it does not establish that the grounded labels are the correct ones, and the retrieval-quality audit (Section 4.4) shows the evidence base was thin precisely for the categories the ablation is most often credited with preserving. RQ7 should therefore be read as answered in terms of behavioural effect rather than accuracy. The accompanying human validation (Section 4.10) provides a pilot diagnostic check of where the evidence-grounded model agrees with, and diverges from, human judgment.

Table 5. Marginal veracity distribution under the grounded pipeline versus the no-evidence ablation, on the same 937 claims.

Veracity Label	Grounded	No Evidence
SUPPORTED	232	407
UNSUPPORTED	505	164
EXAGGERATED	114	270
CONTRADICTED	75	94
DANGEROUS	11	2

4.10. Human Validation and Claim-Extraction Audit

The blind human-validation sheet contained 150 sampled claim records. In addition to assigning veracity labels and stigma scores, the human coder flagged records that were not clearly rateable as health claims. This review identified 12 records with extraction-quality or claim-definition concerns, corresponding to 8.0% of the validation sample. These cases included underspecified cure claims, posts phrased as questions rather than assertions, irrelevant or non-reproductive-health content, and semantically ambiguous wording. Of these 12 flagged records, eight were left uncoded for human veracity and were therefore excluded from the veracity agreement analysis. The remaining coded records yielded 142 rateable claims for veracity validation. Among these 142 claims, the human coder assigned 92 claims (64.8%) to SUPPORTED, 29 (20.4%) to UNSUPPORTED, 14 (9.9%) to EXAGGERATED, 2 (1.4%) to CONTRADICTED, and 5 (3.5%) to DANGEROUS. One coded record lacked a human stigma score, leaving 141 records for stigma-score agreement. The mean human-assigned stigma score was 0.048, with a median of 0.0. Model–human agreement on the five-way veracity labels was limited (Table 6). Exact agreement was 33.8% (48/142), with an unweighted Cohen’s $\kappa$ of 0.134 and a quadratic-weighted $\kappa$ of 0.279. The confusion matrix (Table 7) shows that the disagreement was systematic rather than random: the human coder assigned many more claims to SUPPORTED than the evidence-grounded model. In particular, of the 92 claims coded as SUPPORTED by the human coder, the model also assigned 29 to SUPPORTED, but assigned the remaining cases to non-supported categories. This pattern suggests that the evidence-grounded model used a more conservative evidentiary threshold than the human coder. Agreement was stronger for stigma scoring than for categorical veracity. Human and model stigma scores were positively associated (Spearman $\rho = 0.335$, $p < 0.001$), with a mean absolute error of 0.177 on the 0–1 stigma scale. Overall, the human validation should therefore be interpreted as a pilot diagnostic check rather than as a definitive gold-standard evaluation. It confirmed that most extracted records were rateable and that stigma scores were directionally aligned with human judgments, while also showing that veracity classification requires further calibration against a larger multi-coder human-annotated reference set.

Table 6. Human-validation results for the coded validation subset. All coding was performed by a single blinded coder; inter-rater reliability could not be estimated.

Validation Metric	Value
Validation records	150
Flagged extraction-quality records	12/150 (8.0%)
Veracity-coded records	142/150 (94.7%)
Exact veracity agreement	48/142 (33.8%)
Unweighted Cohen’s $\kappa$	0.134
Quadratic-weighted $\kappa$	0.279
Stigma-coded records	141/150 (94.0%)
Stigma Spearman correlation	$\rho = 0.335$, $p < 0.001$
Stigma mean absolute error	0.177

Table 7. Confusion matrix for human versus model veracity labels on the 142 veracity-coded validation records. Rows show human labels; columns show evidence-grounded model labels. Abbreviations: Sup. = supported; Unsup. = unsupported; Exag. = exaggerated; Contr. = contradicted; Dang. = dangerous.

Human Label	Sup.	Unsup.	Exag.	Contr.	Dang.
SUPPORTED	29	34	17	9	3
UNSUPPORTED	1	8	3	12	5
EXAGGERATED	2	2	8	2	0
CONTRADICTED	0	0	1	1	0
DANGEROUS	0	0	1	2	2

4.11. Inferential Analysis

4.11.1. Stigma and Veracity (RQ2)

A Kruskal–Wallis test confirmed that stigma scores differed significantly across the five veracity groups ($H = 39.92$, $df = 4$, $p = 4.5 \times {10}^{-8}$), with a small effect size (${\epsilon }^2 = 0.039$). Treating veracity as an ordinal scale from SUPPORTED (0) to DANGEROUS (4), a Spearman correlation showed a positive monotonic association between decreasing veracity and increasing stigma ($\rho = 0.20$, $p = 5.5 \times {10}^{-10}$), indicating that less accurate and more potentially harmful claims were framed in more stigmatizing terms.

Pairwise Mann–Whitney U comparisons between SUPPORTED claims and each non-supported category were all significant after Holm correction for multiple testing (Table 8). The rank-biserial effect size increased steadily along the veracity scale, from a small effect for UNSUPPORTED ($r_{\mathrm{rb}} = 0.12$, $p_{\mathrm{Holm}} = 0.009$) to moderate effects for EXAGGERATED ($r_{\mathrm{rb}} = 0.28$, $p_{\mathrm{Holm}} = 1.5 \times {10}^{-5}$) and CONTRADICTED ($r_{\mathrm{rb}} = 0.35$, $p_{\mathrm{Holm}} = 3.9 \times {10}^{-6}$), and the largest effect for DANGEROUS claims ($r_{\mathrm{rb}} = 0.45$, $p_{\mathrm{Holm}} = 0.009$). The progressive growth in effect size, rather than the significance values alone, provides the clearest evidence that the stigma–veracity relationship is graded rather than categorical.

Table 8. Pairwise Mann–Whitney U comparisons of stigma scores between SUPPORTED claims and each non-supported veracity category. p-values are Holm-corrected; $r_{\mathrm{rb}}$ is the rank-biserial effect size.

Comparison	${\mathit{n}}_1/{\mathit{n}}_2$	U	${\mathit{p}}_{\mathbf{Holm}}$	${\mathit{r}}_{\mathbf{rb}}$
SUPPORTED vs. UNSUPPORTED	232/505	51,481.0	0.009	0.12
SUPPORTED vs. EXAGGERATED	232/114	9541.0	$1.5 \times {10}^{-5}$	0.28
SUPPORTED vs. CONTRADICTED	232/75	5668.5	$3.9 \times {10}^{-6}$	0.35
SUPPORTED vs. DANGEROUS	232/11	705.5	0.009	0.45

4.11.2. Stigma Across Conditions (RQ3)

Stigma framing varied markedly across reproductive-health conditions. A Kruskal–Wallis test across the six most frequent conditions was highly significant ($H = 82.66$, $p = 2.3 \times {10}^{-16}$) with a medium effect size (${\epsilon }^2 = 0.103$), the largest effect observed in this study. Condition labels were normalised before grouping so that variant spellings of the same condition (for example “pcos” and “pcos (polycystic ovary syndrome)”) were merged. Mean stigma scores were lowest for male fertility (0.112; $n = 20$), PCOS (0.128; $n = 328$), and “general health” (0.136; $n = 108$), and substantially higher for infertility (0.292; $n = 62$), endometriosis (0.278; $n = 173$), and fertility (0.194; $n = 67$). Conditions centred on conception difficulty and on a chronic, frequently under-recognized pain condition thus attracted more stigmatizing framing than PCOS, which in this corpus was discussed largely in empowering or advocacy-oriented terms (Figure 5).

Figure 5. Mean intensity of each affective dimension across reproductive-health conditions. The five leftmost columns are the stigmatising dimensions that enter the composite stigma score; empowerment is shown alongside them as a counter-stigma indicator and is excluded from the composite. Rows are the six most frequent conditions, the same set used in the Kruskal–Wallis test in Section 4.11.

4.11.3. Stigma and Engagement (RQ4)

At claim level the association between stigma score and engagement was positive but weak ($\rho = 0.08$, $p = 0.014$). However, engagement is a property of the post rather than of the individual claim, and the post-level reanalysis reported in Section 4.12 does not sustain the association ($\rho = 0.076$, $p = 0.068$ for mean stigma per post). Given this and the sparse, highly skewed nature of Bluesky engagement counts, we do not claim an association between stigma and engagement in this corpus.

Taken together, the results show that non-supported reproductive-health claims were common in the sampled corpus, but the central pattern was not simply the frequency of non-supported claims. Claims that departed further from the retrieved biomedical evidence also carried stronger stigmatizing framing, and this relationship held under cluster-aware reanalysis and was echoed at the condition level. The engagement analysis did not support an association once nesting was accounted for. The retrieval ablation quantified the behavioural contribution of evidence grounding, while the single-coder human validation is reported as a pilot diagnostic check rather than as confirmation of the veracity labels.

4.12. Cluster-Aware Reanalysis

The 937 claims derive from 574 posts (mean 1.63 claims per post, maximum 6). Claims from the same post resemble one another substantially in stigma framing: the intraclass correlation was 0.577, giving a design effect of 1.37 and an effective sample size of roughly 687 rather than 937.

The stigma–veracity gradient (RQ2) was robust to this nesting under all three approaches. A cluster bootstrap resampling posts gave $\rho = 0.201$ with a 95% confidence interval of [0.129, 0.272], excluding zero. Drawing one claim per post gave a median $\rho$ of 0.223 (2.5th–97.5th percentile 0.192–0.257), with the Kruskal–Wallis test significant at $\alpha = 0.05$ in 100% of 2000 draws (median $H = 30.9$, median $p = 3.2 \times {10}^{-6}$). A linear mixed model with a random intercept for post gave a positive coefficient for ordinal veracity ($\beta = 0.052$, SE $= 0.008$, $z = 6.48$, $p = 8.9 \times {10}^{-11}$; post-level variance 0.029, residual 0.024). The association reported in Section 4.11 therefore does not depend on treating co-occurring claims as independent.

The stigma–engagement association (RQ4) did not survive the same scrutiny. Repeating the analysis at post level, the association was weak and no longer significant using mean stigma per post ($\rho = 0.076$, $p = 0.068$, $n = 574$) and only marginally significant using maximum stigma ($\rho = 0.090$, $p = 0.031$).

4.13. Temporal Robustness Check

As an auxiliary temporal robustness check, we analysed an extended timestamped re-collection of raw Bluesky posts matching the same reproductive-health search queries as the primary corpus. This analysis was separate from the main 991-post, 937-claim dataset and was used only to test whether the sampled discourse showed obvious temporal instability or co-movement with external public-attention signals.

Monthly post activity was compared with Google Trends search interest for the same reproductive-health query terms. Raw monthly levels showed a positive association between Google Trends interest and Bluesky post volume (Spearman $\rho = 0.73$, $p < 0.001$). However, this association disappeared after detrending the two series using month-to-month changes ($\rho = -0.006$, $p = 0.971$), indicating that the raw association mainly reflected shared growth over time rather than genuine short-term co-movement.

A separate raw-text temporal probe also found no clear monotonic trend in surface indicators of stigma-related language, claim-related language, or stigma-word intensity across the observed period. These temporal checks therefore suggest that the main cross-sectional findings are not driven by a short-lived activity spike or by a simple external search-interest pattern. Rather, the veracity and stigma patterns reported above should be interpreted as stable corpus-level associations within the sampled Bluesky discourse. A fuller time-series treatment using dated claim-level classifications and forecasting models is left to future work.

5. Discussion

This study set out to determine whether evidence-grounded LLMs can jointly surface non-supported claims and stigmatising framing in reproductive-health discourse, and whether the two co-occur. Three findings stand out. First, most sampled claims (75.2%) were not supported by biomedical evidence, though this reflects a deliberately suspicion-prone keyword sample rather than platform-wide prevalence, and the largest component of that figure records evidence that was not retrieved rather than evidence of falsity (Section 4.4). Second, stigma rose monotonically as claims departed from the evidence, a graded relationship whose effect sizes increased steadily from UNSUPPORTED through to DANGEROUS claims. The gradient is modest in absolute terms (${\epsilon }^2 = 0.039$; $\rho = 0.20$) but robust: it survived a cluster bootstrap over posts, one-claim-per-post resampling, and a mixed model with a random intercept for post (Section 4.12). It indicates that non-supported claims and stigmatising framing co-occur systematically; the cross-sectional design does not license any claim about the direction of that relationship. Third, the retrieval ablation showed that grounding makes the pipeline substantially more conservative: without it, the same model endorsed 209 additional problematic claims as SUPPORTED and lost almost all of its DANGEROUS verdicts. This isolates the evidence-retrieval layer, rather than the language model alone, as the component driving that conservatism, and offers a concrete, quantified answer to a question that prior misinformation pipelines have generally left implicit. It does not, however, show that the grounded labels are more accurate; establishing that requires the human reference set discussed in Section 5.2. The human-validation audit further showed that most extracted records were rateable, but that a small subset failed because the claim was underspecified, question-like, irrelevant, or ambiguous, highlighting claim-validity filtering as an important improvement for future versions of the pipeline.

The condition-level results add nuance: stigma was highest for infertility and endometriosis and lowest for PCOS and male fertility. Section 5.1 sets out why these two findings—the veracity gradient and the condition-level variation—are consistent with a common set of mechanisms.

5.1. Why Non-Supported Claims and Stigmatising Framing Co-Occur

The association reported here is not self-explanatory, and the cross-sectional design cannot adjudicate between the accounts below. They are, however, distinguishable in principle, and two of them make predictions about which affective dimensions should be elevated—predictions the observed profile is consistent with.

Shared cause: the explanatory vacuum. Conditions with contested aetiology, long diagnostic delay, and limited curative treatment generate demand for explanation that clinical medicine does not meet. Endometriosis is the clearest case: diagnostic delays are substantial and patients report symptom normalisation and dismissal [6,7], which drives them toward online communities [8]. That dismissal is itself a stigmatising act—it marks the reported experience as not credible—and the resulting discourse carries both the unmet explanatory demand that non-supported claims fill and the stigma that produced it. On this account neither variable causes the other; both are downstream of the same clinical and social conditions.

Blame is entailed by the structure of lifestyle-cure claims. A claim that a condition is curable through diet, supplements, or behaviour change carries an implication about anyone who remains ill: that they have not done the right thing. Attribution of responsibility is not incidental to such claims but built into their logical form, and it is precisely the mechanism by which a condition is converted into a moral failing [33]. This account predicts that blame specifically, rather than shame or disgust, should be the elevated dimension in a corpus dominated by cure and supplement claims. It was: blame was the dominant frame for 10.2% of claims against 1.1% for shame and 0.7% for disgust, and condition-cure and supplement-efficacy claims were the two largest risk categories.

Affect substitutes for evidentiary warrant. A claim that cannot appeal to evidence must persuade some other way, and framing a condition as an urgent threat is a standard substitute. This predicts elevated fear, which was the most common negatively valenced frame (12.3%). Unlike the first two accounts, this one implies a direction—the absence of warrant drives the affective framing—but the present data cannot establish it.

The three are not mutually exclusive and most likely operate together. What they share is the implication that the association is structural rather than incidental: it arises from features of how contested reproductive-health conditions are discussed, not from a coincidence of the sample.

The condition-level results follow this logic. Stigma was highest for infertility (0.292) and endometriosis (0.278), the two conditions in the corpus that best fit the explanatory-vacuum account, and lowest for PCOS (0.128) despite PCOS being by far the most frequent target condition. The PCOS result is instructive rather than anomalous: the mechanisms above describe conditions that generate stigmatising discourse, but they can be offset. PCOS in this corpus was discussed largely in advocacy-oriented terms, with empowerment the dominant frame, which is what counter-stigmatising discourse looks like when a patient community has organised around a diagnosis [10,11]. That a condition attracting many non-supported claims can nonetheless carry low stigma is a useful check on the gradient reported in Section 4.11: the association is an average tendency across the corpus, not a property of every condition within it. This is also why automated stigma assessment should be condition-aware rather than treating reproductive health as a single undifferentiated domain.

5.2. Implications of Limited Human–Model Agreement

The human-validation results constrain what can be claimed about the pipeline’s performance and merit explicit interpretation. Exact five-way agreement was 33.8% ($\kappa = 0.134$; quadratic-weighted $\kappa = 0.279$). The gap between the unweighted and weighted coefficients indicates that agreement was better on the ordering of categories than on their exact boundaries: model and coder tended to place a claim on the same side of the evidentiary scale more often than they agreed on which of the five labels applied.

The disagreement was also systematic rather than random. The coder assigned 92 of 142 claims to SUPPORTED, against 232 of 937 for the model (Table 7). The two raters were effectively applying different criteria: the model treated the absence of retrieved high-level evidence as grounds for non-support, whereas the coder could draw on background clinical plausibility. Neither criterion is simply incorrect, but they are not interchangeable, and the difference explains most of the observed disagreement.

Two consequences follow. First, the pipeline’s relative patterns—the stigma gradient across veracity levels and the variation in stigma across conditions—rest on the internal consistency of a single labelling rule and are more defensible than its absolute category proportions. Second, the 75.2% non-supported figure should be read as the output of one conservative evidentiary threshold applied to a suspicion-prone sample, not as a measured rate of misinformation. Establishing accuracy rather than consistency would require multiple independent coders, an explicit codebook with anchor examples for each category boundary, adjudicated consensus labels, and a larger validation sample.

5.3. Scope of the Claims Supported by These Results

Because the pipeline produces several distinct kinds of output, it is worth stating plainly what each one does and does not establish.

The retrieval ablation is a behavioural result. It establishes that withholding retrieved evidence changes the model’s labels, and changes them in a safety-relevant direction. It does not establish that the grounded labels are more accurate than the ungrounded ones, because no independent gold standard was available against which either configuration could be scored.

The human validation is a pilot diagnostic check on a single coder, not a measurement of accuracy. Exact five-way agreement of 33.8% ($\kappa = 0.134$) bounds what can be claimed about the reliability of the categorical veracity labels, and the disagreement was systematic rather than random (Section 5.2).

The veracity categories are statements about retrieved evidence, not about truth. UNSUPPORTED, which accounts for 505 of the 705 non-supported claims, records that no relevant confirming evidence was identified by this retrieval architecture. The relevance audit shows that this reflects retrieval failure in the majority of cases (Section 4.4). Throughout this paper, non-supported therefore denotes a retrieval outcome and should not be read as a synonym for false or for misinformation.

The stigma–veracity association is cross-sectional and modest. It is robust to the nesting of claims within posts, and it is graded rather than categorical, but it identifies co-occurrence within a single corpus at a single point in time. It cannot show that stigmatising framing causes claims to depart from evidence, nor the reverse.

Finally, the corpus was assembled with suspicion-prone keyword queries from one platform. The reported proportions describe that sample and are not prevalence estimates.

5.4. Limitations

Several limitations qualify these findings. The corpus is single-platform and keyword-sampled, so the veracity and stigma proportions describe this sample and not Bluesky as a whole. Veracity and stigma labels were produced by a single LLM; while the retrieval ablation isolates the value of grounding, the human validation that would confirm absolute accuracy is reported as an independent check rather than a large-scale gold standard. Grounding quality was uneven across labels: directly relevant high-level evidence was available for an estimated 86% of SUPPORTED claims but only 17% of UNSUPPORTED, 11% of CONTRADICTED, and none of the DANGEROUS claims. The non-supported categories should therefore not be read as evidence that the claims are false; UNSUPPORTED in particular records that relevant evidence was not retrieved. This is consistent with the retrieval-noise failure mode documented across clinical RAG systems [29], and indicates that query construction, rather than the classifier, is the component most in need of improvement. Two construct-validity limitations follow from Section 2.5. First, fear is the least secure of the five stigmatising dimensions. Some fear-framed claims present a condition as a threat to be avoided, which is a stigmatising move; others express ordinary anxiety about symptoms, which is not. The scorer does not separate these, so the fear dimension almost certainly absorbs some non-stigmatising negative affect. Because fear was the most common negatively valenced frame (12.3%), this inflates the composite score to an unknown degree, and the mechanism proposed in Section 5.1 for fear should be treated as the most tentative of the three. Second, and for the same reason, the boundary between stigmatising framing and negative valence more generally is enforced only by the prompt definitions; a codebook with anchor examples and multi-coder adjudication would be required to establish that the scorer respects it. We also note that the PubMed filter admitted five retracted publications into the evidence pool, and that no automated check for retraction status was applied. Engagement on Bluesky is sparse and highly skewed, and the stigma–engagement association did not hold at post level, so we make no claim about it. The human-validation audit also showed that a small fraction of extracted records were not clearly rateable claims, most often because the intervention was underspecified, the post was phrased as a question, or the content was irrelevant or ambiguous. This indicates that future versions of the pipeline should include a claim-validity filter before evidence retrieval and veracity classification. Finally, the keyword baseline can emit only three of the five labels, which structurally limits its agreement with the grounded model. Validation used a single coder; the implications are discussed in Section 5.2.

6. Conclusions

We presented an evidence-grounded LLM pipeline that jointly assesses the veracity and the stigma framing of reproductive-health claims, applied to 937 claims drawn from a keyword-sampled, single-platform Bluesky corpus. Non-supported claims—a category that for the most part records evidence that was not retrieved rather than claims shown to be false—were common in this suspicion-prone sample, which was deliberately constructed to target claim-rich topics and does not support prevalence estimates for Bluesky as a whole; stigma increased monotonically as claims departed from biomedical evidence—a modest but consistent gradient (${\epsilon }^2 = 0.039$) that held under cluster-aware reanalysis—and a controlled ablation showed that the retrieval layer is what drives the conservatism of the model’s verdicts: removing it caused the model to endorse 209 additional problematic claims and to lose nearly all of its harm warnings. That ablation is a statement about model behaviour and does not establish that the grounded labels are the correct ones. By scoring stigma independently of truth, the pipeline makes visible an interaction—between what is claimed and how it is framed—that prior systems treated as separate. Validation against human judgment was performed by a single coder and is reported as a pilot diagnostic check; the categorical veracity labels therefore await calibration against a multi-coder reference set. Future work should extend the approach across platforms and languages, incorporate a larger human-annotated gold standard, and test condition-aware stigma models.