Stance and Engagement: How Community Notes Influence HPV Vaccine Conversations on X in Japan ^†

Abstract

This study examines the impact of Community Notes on users’ stance and posting behavior regarding the human papillomavirus (HPV) vaccine on X. Unlike previous research focusing on affected posts and authors, this study analyzes users who viewed Community Notes and their posting behavior before and after exposure. We analyzed posts related to the HPV vaccine using X’s official Community Notes dataset (January 2021–July 2024). Posts were classified as “Support,” “Oppose,” or “Neutral” using a large language model (GPT-4o, OpenAI), and changes in stance and posting frequency were evaluated. Findings show that 73% of users maintained their stance after viewing Community Notes. However, posting frequency increased sharply immediately after the note was added, especially among opposing users. This suggests that, since most Community Notes support vaccination, opposing users may have actively responded by posting critical responses. This study contributes by examining viewer behavior, not just post authors. However, limitations include GPT-4o’s classification accuracy and the restricted scope of topics and users covered in this study. Future research should improve the evaluation of Community Notes by verifying users who viewed Community Notes and enhancing stance classification through better prompts and model comparisons. Additionally, expanding the analysis beyond the HPV vaccine will help assess the broader applicability of the findings.

1. Introduction

In recent years, the spread of misinformation on the internet has become a critical social issue. Misinformation circulates across various domains, including epidemics, political, and environmental issues, and its impact can rapidly extend across a wide audience, particularly on social media [1]. For example, misinformation regarding the safety of the human papillomavirus (HPV) vaccine led to a sharp decline in vaccination rates in Japan after 2013, causing a critical public health issue [2]. Such misinformation can also reinforce political polarization and influence public policy decisions. Therefore, effective strategies are needed to counter misinformation and promote accurate information spread. A good way to fight misinformation is to help people and society change their actions based on reliable information. It is not enough to just give correct information. People need to believe it and use it to make good choices. However, cognitive biases and social media algorithms influence the process of information reception, making the correction of misinformation challenging [3].

In this context, Community Notes on X (formerly known as Twitter, https://x.com/), have gained attention as a new user-driven initiative to counter misinformation. Community Notes (formerly known as Birdwatch, https://communitynotes.x.com/guide/en/about/introduction (accessed on 18 August 2025)) are supplementary information added by X users, aiming to provide context and factual background to posts that may be misleading. In contrast to traditional one-way fact-checking conducted by expert organizations, this system employs collective intelligence, allowing for a broader range of perspectives and potentially enhancing the accuracy of information. Recent studies have focused on the impact of providing supplementary information on misinformation in social media. Wojcik et al. conducted an experiment using the Community Notes feature and found that users who viewed posts with Community Notes were less likely to “like” or “repost” them compared to those who did not [4]. This finding suggests that providing supplementary information may help reduce the spread of misinformation. Kim et al. conducted a study to compare two groups of users. One group received supplementary information through Community Notes. The other group received it via replies to their posts. The researchers then analyzed how each method influenced users’ behavior. The results showed a clear difference between the two groups. Users who received supplementary information through Community Notes were more likely to discuss topics different from their original political stance. They were also more likely to introduce new subjects in their following posts. In contrast, users who received information through replies showed less change in their posting behavior [5].

Previous studies have analyzed posts with Community Notes and the users who created those posts. However, research focusing on users who view Community Notes remains limited, and its effects have not been sufficiently examined. However, identifying those who have definitely seen the note is technically and practically challenging. This study defines “users who replied to posts with Community Notes” as “users who viewed the note” and conducts an analysis.

Selecting an appropriate research target is crucial for analyzing the impact of Community Notes. Whether specific information is considered misinformation depends on its content and context. Therefore, focusing on highly controversial topics where Community Notes are frequently applied is an effective approach. For example, discussions on politics and healthcare involve active debates among users with opposing perspectives. These topics also receive a large number of Community Notes. As a result, they are well-suited for analyzing how Community Notes influence changes in user behavior. Given the constraints on data collection and experiment scale, this study selects the HPV vaccine as the research subject on X in Japan. Despite its scientifically established efficacy and safety, the HPV vaccine is a notable case where misinformation led to a decline in vaccination rates [2].

This study focuses on users who viewed Community Notes related to the HPV vaccine. By analyzing posts created before and after these notes were shown, this study aims to examine changes in users’ stance on the HPV vaccine and their posting behavior.

2. Related Work

2.1. The Impact and Spread of Misinformation

Misinformation can significantly impact society by deepening social divisions, disrupting public policy, and increasing risks to public health. Vosoughi et al. analyzed the effects of misinformation on policy decisions and public health. They pointed out that misinformation spread on social media can contribute to misunderstandings regarding the adoption of vaccines and environmental policies [1]. Their study highlighted the 2016 U.S. presidential election as an example, suggesting that the spread of false information may have influenced voting behavior. The widespread use of social media has enabled misinformation to spread instantly, affecting a large number of people. In particular, false information, especially in the political domain, spreads more rapidly and reaches a broader audience than other types of information. Furthermore, the spread of misinformation may be driven by factors such as novelty and emotional influence. False information is often reported to be more novel than accurate information. It is also more likely to evoke strong emotions, such as fear and surprise. As a result, users tend to share it.

Roozenbeek et al. analyzed survey data from multiple countries and found that a tendency to believe misinformation about COVID-19 is connected to lower vaccine acceptance and a decreased commitment to public health guidelines [6]. In particular, individuals who are frequently exposed to misinformation are more likely to refuse vaccination. They are also less likely to engage in basic preventive behaviors such as handwashing and mask-wearing. The spread of such misinformation can undermine the effectiveness of epidemic containment efforts.

2.2. Approaches to Combating Misinformation

Several approaches have been investigated to limit the spread of misinformation on social media. Clayton et al. examined the effectiveness of warning labels on misinformation [7]. They found that labels clearly marking information as “false” were more effective in reducing people’s trust in misinformation. In contrast, labels indicating that the information was “under discussion” had a weaker impact.

Pennycook et al. proposed a method to enhance the visibility of reliable information and suppress the spread of misinformation by utilizing crowdsourced evaluations of news sources [8]. The research demonstrated that accuracy ratings of news sources by the general public were strongly correlated with those of professional fact-checkers. These studies suggest that the proper use of fact-checking tags and crowdsourced evaluations of news sources are promising methods for countering misinformation.

2.3. HPV Vaccine Misinformation

HPV is considered a major cause of various cancers. To prevent cervical cancer and other HPV-related diseases, the HPV vaccine was developed. Research shows it works. The World Health Organization (WHO) reports that the HPV vaccine has a high preventive effect. In particular, when given to young individuals before sexual debut, it has the potential to significantly reduce future cervical cancer incidence rates [9].

However, in Japan, a major newspaper published an article in 2013 reporting harmful reactions after HPV vaccination. Following this, negative media coverage continued, leading to a sharp decline in vaccination rates [2]. As a result of these misleading reports, the Japanese government halted its active promotion of the vaccine in June 2013. Scientific reviews later confirmed the safety and effectiveness of the vaccine, and the Japanese government resumed its recommendation in 2021. However, misinformation about the vaccine continues to spread, and public debate over vaccination remains unresolved (https://www.jsog.or.jp/medical/582/ (accessed on 17 February 2025)). On social media, HPV vaccine misinformation spreads easily, and some posts distort the risks of vaccination [10]. In this situation, Community Notes may be an effective way to counter HPV vaccine misinformation.

2.4. Impact of Community Notes on Misinformation Suppression

Community Notes is a system introduced by X in January 2021. It allows users to provide additional explanations for posted information. These notes can highlight the accuracy of the information or point out its potential for misinterpretation. Each Community Note has a status that indicates its current state. When first posted, it is labeled as “Needs More Ratings.” Once it receives sufficient evaluations, it is categorized as either “Helpful” or “Not Helpful.” The status of a note is not determined by a simple majority vote. Instead, evaluations from users with diverse opinions are required. This mechanism helps prevent biased assessments by specific groups and aims to provide more objective information (https://communitynotes.x.com/guide/en/contributing/diversity-of-perspectives (accessed on 17 February 2025)).

Chuai et al. analyzed the effects of Community Notes on X and found that the frequency of misinformation spread decreased by an average of 61.4%. Their study was based on an analysis of 31,758 fact-checked posts and confirmed that the number of reposts significantly dropped after a note was displayed [11]. However, they pointed out an issue with timing. Community Notes typically appeared 75.5 h after the initial post. This lag reduces the effectiveness of intervention at the peak of content spread.

Kim et al. compared two groups of users. One group received additional information through Community Notes, while the other received it via replies to their posts. They analyzed how different methods of providing supplementary information affect the original poster [5]. According to their study, replies were often seen as aggressive because they were direct responses from individuals. As a result, users who received replies tended to reduce the diversity of their posts. On the other hand, Community Notes involve a voting process by users with different perspectives, ensuring a more objective and neutral evaluation. This makes it easier for users to accept differing viewpoints. As a result, users who received additional information through Community Notes were more likely to discuss topics outside their original political stance or engage with new subjects compared to those who received replies.

Previous studies have analyzed posts with Community Notes and their authors. There is little research on users who read these notes, and their effects are not well understood. This study focuses on users who viewed Community Notes related to the HPV vaccine. It analyzes how their stance on the vaccine and their posting frequency change before and after viewing the notes. The following sections provide details on data collection and analysis methods.

3. Methods

In this study, we established data collection and analysis methods to examine behavioral changes in users who viewed Community Notes. We first describe the data collection procedure in Section 3.1. Then, in Section 3.2, we explain how the collected data were analyzed.

3.1. Data Collection

To ensure transparency, X publicly shares data on Community Notes on its official website (https://communitynotes.x.com/guide/en/under-the-hood/download-data (accessed on 17 February 2025)). In this study, we used data from “Notes” and “Note Status History” (“Notes” refers to data related to Community Notes, including the content of the notes and the post ID. “Note Status History” represents the status information of Community Notes) covering the period from 28 January 2021 to 23 July 2024.

3.1.1. Collection of Community Notes on the HPV Vaccine

Community Notes written in Japanese were collected and filtered to identify entries related to HPV vaccination. The filtering process extracted notes containing the keywords “HPV” and “HPV vaccine.” Additionally, we select only notes that have remained visible on X since their initial evaluation. To do this, we choose notes where the firstNonNMRStatus and currentStatus columns in the Note Status History data wer both marked as “CURRENTLY RATED HELPFUL.”

3.1.2. Collection of Users Who Replied After a Community Note

In this study, we defined “users who replied after a Community Note was added” as “users who viewed the Community Note.” To detect these users, we first used the tweetId column in the Notes data to find posts with Community Notes. Then, we referred to the timestampMillisOfFirstNonNMRStatus column in the Note Status History data to determine when the Community Note was first added. Based on this timestamp, we collected users who replied after the note was added. If the same user replied multiple times, we used only the most recent reply.

3.1.3. Collection of Users’ Posting History

To analyze changes in posts related to the HPV vaccine, we collected posts that included the keywords “HPV” or “HPV vaccine” during two periods. The first period covered 62 days before the user replied to a post with a Community Note, and the second period includes 62 days after the reply. Additionally, we included “quote reposts” and “replies” to other users’ posts that contained the same keywords. In this study, we defined “posts made during the 62 days before replying” as “posts made before viewing the Community Note.” Similarly, we defined “posts made during the 62 days after replying” as “posts made after viewing the Community Note.” It is important to note that “replies made after a Community Note was added” were counted as “posts made during the 62 days after replying.” Additionally, we excluded users who did not post at least once in both the before and after periods because our focus was on changes before and after viewing a Community Note. The distribution of the collected posts is shown in Figure 1. The analysis included 100 users, with a total of 1235 posts. The distribution shows that while many users made only a few posts, some users contributed a large number of posts.

3.2. Data Analysis

We analyzed posts made before and after viewing Community Notes to score users’ stances on the HPV vaccine. First, we determined whether posts made before viewing the note expressed “Support,” “Oppose,” or “Neutral” regarding the effectiveness and social promotion of the HPV vaccine. However, manually assessing and categorizing a large volume of data would take significant time and effort. To solve this issue, we utilized GPT-4o (https://openai.com/index/hello-gpt-4o/ (accessed on 17 February 2025)), a large language model (LLM) developed by OpenAI.

In recent years, LLMs have made significant progress in natural language processing, particularly in text classification and information extraction accuracy. Traditionally, classification was performed through manual annotation or machine learning models. However, the ability of recent LLMs to understand context has been improved. They have also been shown to achieve accuracy comparable to or exceedingt hat traditional methods, even in tasks requiring advanced expertise [12]. Since user posts included two types—“regular posts” and “quote reposts or replies”—we designed prompts suitable for each.

For regular posts, we used the prompt shown in

Table 1. Prompt for regular posts in English.

You are given the following post.

Determine whether the content of the post expresses “Support,” “Oppose,” or “Neutral” regarding the effectiveness of the HPV vaccine and its social recommendation.

Additionally, consider the following three points:

1. The terms “anti-vax” and “anti-vaccine” refer to individuals who oppose vaccines.

2. If the post contains critical expressions toward pro-vaccine individuals, it is likely expressing opposition to the HPV vaccine.

3. If the post contains critical expressions toward anti-vaccine individuals, it is likely to express support for the HPV vaccine.

The output should be one of “Support,” “Oppose,” or “Neutral,” along with a justification for the decision.

Example Output:

Conclusion: Support

Justification: The phrases “Vaccination is important” and “Essential for maintaining health” are used, and the overall tone suggests an intent to promote the HPV vaccine.

—– Post —–

“Post Content”

. Here, we provide the English translation of the original Japanese prompts. The original Japanese prompts are shown in

Table A1. Prompt for regular posts.

以下にある投稿を与える。

投稿の内容がHPVワクチンの有効性やHPVワクチンの社会的な推奨に対して「賛成」か「反対」あるいは「どちらでもない」か判断せよ。

また、以下の3点に注意せよ:

1. 「反ワク」「反ワクチン」という言葉はワクチン反対派を意味する言葉であること。

2. ワクチン賛成派に対して批判的な表現が使われている場合、HPVワクチンに対して否定的な意見である可能性があること。

3. ワクチン反対派に対して批判的な表現が使われている場合、HPVワクチンに対して賛成的な意見である可能性があること。

出力は「賛成」「反対」「どちらでもない」のいずれかとし、判断の根拠を加えよ。

出力例:

結論:賛成

根拠:「予防接種は大事」「健康維持には必須」というフレーズが使用されており、トーンもHPVワクチン推奨の意図が感じられるため。

—–投稿—–

“投稿の内容”

of the Appendix A. For quote reposts and replies, we added additional instructions to the end of the Table 1 prompt to explain for their context. The specific prompts are provided in

Table A2. Prompt for quote reposts or replies.

以下にある投稿とそれに対する返信を与える。

ある投稿とそれに対する返信の文脈を理解した上で、返信の内容がHPVワクチンの有効性やHPVワクチンの社会的な推奨に対して「賛成」か「反対」あるいは「どちらでもない」か判断せよ。

また、以下の3点に注意せよ:

1. 「反ワク」「反ワクチン」という言葉はワクチン反対派を意味する言葉であること。

2. ワクチン賛成派に対して批判的な表現が使われている場合、HPVワクチンに対して否定的な意見である可能性があること。

3. ワクチン反対派に対して批判的な表現が使われている場合、HPVワクチンに対して賛成的な意見である可能性があること。

出力は「賛成」「反対」「どちらでもない」のいずれかとし、判断の根拠を加えよ。

出力例:

結論:賛成

根拠:「予防接種は大事」「健康維持には必須」というフレーズが使用されており、トーンもHPVワクチン推奨の意図が感じられるため。

—–投稿—–

“投稿の内容”

—–返信—–

“投稿に対する返信の内容”

of the Appendix A. To ensure more reliable results, each post was classified 10 times, and the final classification was determined by a majority vote.

Once GPT-4o classified the opinions in the posts, we used the data to calculate each user’s stance on the HPV vaccine before viewing the note. The calculation followed Equation (1), where $P_{\mathrm{before}}$ represents the number of posts supporting the vaccine before viewing the note, and $N_{\mathrm{before}}$ represents the number of posts opposing it.

$$S_{\mathrm{before}}={\displaystyle \frac{P_{\mathrm{before}}-N_{\mathrm{before}}}{P_{\mathrm{before}}+N_{\mathrm{before}}}}$$

(1)

$S_{\mathrm{before}}$ represents the score stance of a user on the HPV vaccine before viewing the note. It is designed to range from $-1$ to 1, where values closer to $-1$ indicate Oppose, and values closer to 1 indicate Support. For example, if all posts before viewing the Community Note are supportive, then $S_{\mathrm{before}}=1$. If there is one supportive post and four opposing posts, then $S_{\mathrm{before}}={\displaystyle \frac{1-4}{1+4}}=-0.6$.

By applying the same process to posts made after viewing the Community Note, we scored the user’s stance on the HPV vaccine after viewing the note. By comparing the scored stance before and after, we quantitatively evaluated the impact of Community Notes on users’ stance on the HPV vaccine. If the total number of posts was low and the scored stance changed drastically (e.g., $S_{\mathrm{before}}=1$ and $S_{\mathrm{after}}=-1$), misclassification by GPT-4o was a possibility. To solve this, we manually classified users who had three or fewer posts in total and whose absolute stance change was 1 or greater($|S_{\mathrm{after}}-S_{\mathrm{before}}| \geqq 1$). For manual classification, four science-major university students, including the author, reviewed the same posts. The final classification for each post was determined based on the most frequent classification result among them.

4. Results

The results of the user stance analysis are shown in Figure 2. Figure 2 illustrates how much users’ stances on the HPV vaccine changed before and after viewing the Community Note. The figure shows that many users (73%) remain in the same position. Some users (27%) have arrows, indicating a change in stance, but overall, they are in the minority. This suggests that while Community Notes influence some users, they do not significantly change the stance of the majority, who tend to maintain their original position.

The findings suggest that the majority of users are not influenced by Community Notes and tend to maintain their existing stance. To further analyze this trend, we calculated each user’s overall stance on the HPV vaccine across the entire period, regardless of whether they viewed the Community Note before or after. Then, We examined the posting frequency for each stance. The overall stance was calculated using the following formula:

$${\displaystyle \frac{P_{\mathrm{before}}+P_{\mathrm{after}}-(N_{\mathrm{before}}+N_{\mathrm{after}})}{P_{\mathrm{before}}+P_{\mathrm{after}}+N_{\mathrm{before}}+N_{\mathrm{after}}}}$$

This metric ranges from $-1$ to 1, similar to Equation (1) in Section 3.2. Values closer to $-1$ indicate an Oppose stance, while values closer to 1 indicate a Support stance. The results of the posting frequency analysis for each stance are shown in Figure 3.

The figure shows that posting frequency is highest around day 0. This means that users post more frequently immediately after a Community Note is added to a post. This suggests that Community Notes attract user attention and stimulate posting activity. Additionally, posts with an “Oppose” stance (around $-1$) appear more frequently after a Community Note is added. This increase is more clear compared to posts with a “Support” stance (around +1).

5. Discussion

In this study, we analyzed users who viewed Community Notes related to the HPV vaccine. We examined whether their stance or posting frequency changed from before to after the note was displayed. The results suggest that most users tend to maintain their existing stance even after viewing the Community Note, without making significant changes. Posting frequency increased immediately after the Community Note was added. This effect was particularly noticeable among opposing users. They showed a higher rise in posts.

These results may be influenced by the long-term polarization surrounding the HPV vaccine in Japan. According to data analysis by Lim et al., pro-vaccine and anti-vaccine groups have remained in clear opposition for 11 years [10]. In a highly polarized environment, new supplementary information is unlikely to cause significant shifts in existing opinions. Additionally, this phenomenon may be influenced by confirmation bias. Confirmation bias is the tendency to accept information that supports one’s beliefs and expectations. At the same time, they tend to ignore or dismiss information that contradicts their views [13]. As a result, rather than transforming users’ perspectives and behaviors, the supplementary information in Community Notes may serve to confirm their current opinions.

Changes in posting frequency showed a sharp increase in user activity immediately after a Community Note was added. This increase was especially noticeable among opposing users (Figure 3). This suggests that the stance of the Community Note may influence user behavior. Following the same classification method as in Section 3.2, we categorized the content of Community Notes as “Support,” “Oppose,” or “Neutral” regarding the HPV vaccine. The results show that 87% of Community Notes related to the HPV vaccine express support for it. In other words, since most Community Notes take a supportive stance, opposing users may have actively posted in response to challenge them. These findings indicate that Community Notes serve not only as a means of correcting misinformation but also as a trigger for new discussions.

This study has several limitations. First, this study defines “users who replied to posts with Community Notes” as “users who viewed the note.” However, not all users who replied necessarily read the content of the note. Therefore, it is possible that the analysis does not fully capture the exact impact of the note. Additionally, this study focuses on users who replied to posts. Since this analysis is limited to users who actively engage with posts, there may be a potential bias. For a more accurate analysis, it is necessary to include users who do not engage in such active participation. Second, it is important to carefully assess the reliability of stance classification using GPT-4o. While large language models are highly capable of understanding context, they may have difficulty accurately interpreting subtle nuances or sarcasm. Future research needs to improve classification accuracy by comparing results with human-annotated data and developing more advanced prompt techniques. Third, this study focuses specifically on the HPV vaccine, and it remains unclear whether similar results apply to other misinformation topics. Further analysis on different topics is needed to assess the impact of Community Notes. This will help determine whether their effect is limited to specific subject areas or applies more broadly.

6. Conclusions

This study focused on users who viewed posts with Community Notes and analyzed whether their stance on the HPV vaccine and posting frequency changed before and after viewing the note. The results suggest that most users maintain their existing stance and do not change much, even after viewing the Community Note. Additionally, the analysis of posting frequency revealed that user activity immediately after a Community Note was added. This effect was particularly among users with an opposing stance on the HPV vaccine, as their posting frequency increased after the note was attached. This outcome may be influenced by the fact that the majority of Community Notes express Support for the HPV vaccine. In other words, opposing users may have perceived the Community Notes as information conflicting with their own stance and responded by increasing their posts in opposition.

The significance of this study is in analyzing the impact of Community Notes not only on the posters but also on the behavioral changes of viewers. Previous research has primarily focused on posts with Community Notes and the behavior of their authors. However, this study examines how users who view these posts are influenced. This provides valuable insights for evaluating the effectiveness of Community Notes.

However, this study has several limitations. First, users who replied to posts with Community Notes were not necessarily those who actually viewed the notes. Some users may reply without reading the note, while others may have seen the note but did not reply. As a result, the analysis may not fully capture the actual impact of Community Notes. Furthermore, this study specifically examines users who responded to posts. Since the analysis is restricted to users who actively engage with posts, it may introduce some bias. Second, the validity of stance classification using GPT-4o requires careful evaluation. While large language models effectively understanding context, they have limitations in interpreting nuances or sarcasm. Depending on prompt design and model performance, misclassification may occur. Finally, this study focuses specifically on Community Notes related to the HPV vaccine, which is just one topic. It remains uncertain whether similar findings apply to other misinformation topics.

Therefore, future research should conduct a more comprehensive evaluation of the impact of Community Notes. First, a direct method should be introduced to confirm whether users have viewed the notes. This will allow for a more accurate assessment of the influence of Community Notes on user behavior. Second, to improve the accuracy of stance classification using GPT-4o, comparing results with human-annotated data and better prompt construction are needed. Third, it is crucial to examine the influence of Community Notes on user behavior. This analysis should extend to misinformation topics beyond the HPV vaccine. This will help determine whether the insights from this study can be applied to other fields.