Leakage-Guarded Next-Window Superchat Prediction from VTuber Live Chat Dynamics

Abstract

Predicting near-future monetization in virtual livestreaming remains methodologically challenging because paid-support events are sparse, temporally dependent, and vulnerable to leakage under inappropriate evaluation designs. This study develops a leakage-guarded, window-based machine-learning framework for predicting next-window Superchat occurrence from VTuber live-chat dynamics. Public VTuber live-chat and Superchat logs were reconstructed into non-overlapping five-minute windows, and features were organized into audience activity, member composition, message intensity, donation-state information, and short-horizon dynamic groups. To reduce optimistic bias, the primary evaluation used video-level grouped splitting and compared a strict setting that excluded direct current-window donation-state variables with an extended donation-state-aware setting. HistGradientBoosting achieved the strongest performance. In the strict setting, it reached PR-AUC = 0.899, ROC-AUC = 0.920, F1 = 0.822, and Brier score = 0.171, while the extended setting produced only modest additional gains. Additional zero-chat sensitivity, repeated grouped split, channel-level robustness, graph-proxy baseline, feature-ablation, and calibration analyses supported the stability and interpretability of the framework. The results suggest that next-window Superchat occurrence can be predicted from participation breadth, chat activity, message intensity, and temporally shifted behavioral dynamics under leakage-aware evaluation.

1. Introduction

Live streaming has evolved from a passive content delivery format into an interaction-intensive socio-technical environment in which entertainment, social exchange, and monetization unfold simultaneously in real-time. Prior studies have shown that continued watching, trust formation, social presence, streamer–viewer interaction, and purchase-related engagement are central to the value-creation logic of contemporary livestreaming platforms [1,2,3,4,5,6]. In this context, monetization is not merely a by-product of accumulated attention; rather, it is closely embedded in how audiences participate, communicate, and publicly express support during a live session.

A central monetization mechanism in YouTube livestreaming is Superchat. Unlike ordinary chat messages, Superchat is a platform-mediated paid message that allows viewers to increase the visibility and salience of their messages during a livestream. A Superchat event therefore combines communication, attention purchase, public support, and visible financial contribution. This makes Superchat behavior a useful event-level indicator of monetized audience engagement, particularly in creator-centered communities where streamer response, public recognition, and collective visibility are important components of audience participation [7,8,9,10,11].

Donation, gifting, and digital patronage behavior have become important research topics in livestreaming studies. Prior studies have shown that viewer engagement is associated with gift-giving in live video streaming [3], that interactive chat mechanisms can stimulate paid gifting [4], and that attachment, flow, identity, reciprocity, and social influence help explain donation or sponsorship intentions [7,8,9,10,11]. Research on Twitch and related game-streaming ecosystems further shows that social motivations, parasocial processes, and digital patronage norms shape why viewers remain engaged and financially supportive [12,13,14,15,16,17,18,19]. These findings suggest that donation-like behavior is not random noise, but a structured response to observable interactional and relational signals.

The emergence of Virtual YouTubers (VTubers) adds a distinctive layer to this phenomenon. Unlike traditional livestreamers, VTubers perform through avatar-mediated identities, which changes how authenticity, intimacy, parasociality, and community participation are configured [20,21,22,23]. Viewers interact with virtual personas in a highly mediated space where stylized identity, fictional character performance, anonymity, and community-specific norms may intensify affective attachment and public support behavior [20,21,22,23]. Recent ecosystem-scale analyses have documented channel dynamics, fan engagement, subscriber behavior, and Superchat inequality in VTuber environments [24,25], while adjacent research on virtual streamers and virtual influencers has examined how social presence, anthropomorphism, authenticity, coolness, and perceived value influence support-related or purchase-related intentions [20,21,22,26,27]. However, much of this literature remains descriptive, explanatory, or attitudinal rather than predictive and event-level. It explains why VTuber audiences may support virtual performers and how monetization is distributed, but it provides limited evidence on whether immediate live-chat dynamics can predict near-future Superchat occurrence.

This gap motivates the present study. If Superchat is understood as a monetized form of attention-seeking interaction, then the relevant empirical question is not only which channels accumulate more revenue over time, but whether short-term audience dynamics within a livestream contain predictive signals for upcoming monetization events. Prior predictive work has shown that real-time donation events in YouTube livestreaming can be modeled using dynamic interaction structures [28]. However, existing predictive donation research has not fully addressed the leakage-sensitive next-window prediction problem in VTuber livestreaming, where prediction must be based only on information observable before the target window. The present study therefore formulates VTuber Superchat prediction as a leakage-guarded next-window binary classification task using interpretable window-level live-chat features.

From a computational perspective, predicting near-future monetization events is practically valuable but methodologically fragile. Empirical implementations can be distorted by temporal leakage, inappropriate split design, severe class imbalance, arbitrary temporal aggregation, and poorly calibrated probability estimates [28,29,30,31,32,33,34,35,36,37,38,39]. For sparse events such as Superchats, conventional accuracy is uninformative and ROC-only reporting can be insufficient; precision–recall-oriented evaluation, prevalence-aware interpretation, calibration-sensitive metrics, and transparent error analysis are therefore required [29,30,31,32,36,38,39]. In the present study, these issues are not treated as general background concerns but as explicit design constraints for task formulation, feature construction, data partitioning, baseline comparison, robustness analysis, and probability calibration.

To address these challenges, this study develops a leakage-guarded, window-based machine-learning framework for predicting next-window Superchat occurrence from VTuber live-chat dynamics. Each livestream is divided into five-minute windows, and the model predicts whether at least one Superchat event will occur in the immediately following window. To address feature-level leakage, the framework distinguishes a strict setting, which excludes direct current-window Superchat state variables, from an extended setting, which includes current-window donation-state information as an upper-bound condition. In the strict setting, rolling Superchat-related summaries are computed only from past shifted windows, thereby preventing focal-window or future target information from entering the feature representation.

The evaluation protocol is designed to address partition-level leakage and creator-level heterogeneity. Instead of relying on random row-level splitting, the main evaluation uses video-level grouped splitting so that all windows from a given livestream are assigned to only one of the training, validation, or test subsets. This prevents the model from being trained and evaluated on different windows from the same livestream. In addition, channel identifiers recovered from the raw VTuber 1B data are used to conduct channel-level grouped evaluation and repeated channel-level robustness checks. These experiments examine whether the model retains predictive value for unseen channels rather than merely exploiting channel-specific donation propensity.

To strengthen the empirical comparison, the proposed strict model is evaluated against several interpretable baselines. These include previous-window Superchat occurrence, past three-window Superchat occurrence rate, cumulative past within-video Superchat rate, activity-only models, basic behavioral models, and train-derived channel-rate baselines. In addition, graph-proxy features derived from raw chat interactions, including author diversity, repeat-author ratio, top-author share, and author entropy, are used to provide an interaction-structure baseline under the same leakage-guarded window-level formulation. This graph-proxy comparison is not intended as a full reimplementation of prior dynamic-graph donation prediction, but it provides an additional quantitative reference for assessing whether raw interaction structure alone explains next-window Superchat occurrence.

Relative to existing approaches, this study makes five contributions. First, it introduces a VTuber-specific next-window Superchat prediction task that moves prior VTuber monetization research from retrospective ecosystem-level description toward short-horizon event prediction. Second, it proposes a leakage-guarded feature formulation that separates strict past-observable behavioral signals from direct current-window donation-state variables. Third, it provides a grouped evaluation design that combines video-level and channel-level splitting, thereby addressing both within-video partition contamination and creator-level generalization. Fourth, it benchmarks the proposed model against temporal heuristics, cumulative donation-rate baselines, activity-only and behavioral baselines, channel-rate baselines, and graph-proxy interaction baselines. Fifth, it evaluates robustness under zero-chat inclusion, repeated split protocols, balanced held-out compositions, chronological video-level evaluation, feature-space visualization, and probability calibration. These contributions position the novelty of the study not in the use of a new classifier, but in the VTuber-specific task formulation, leakage-guarded design, and comprehensive evaluation framework.

The remainder of this paper is organized as follows. Section 2 reviews related work on livestreaming engagement, VTuber monetization, predictive donation modeling, and leakage-aware evaluation. Section 3 describes the dataset, task formulation, feature construction, and evaluation protocol. Section 4 reports the main results, baseline comparisons, robustness checks, channel-level evaluation, graph-proxy comparison, and calibration analysis. Section 5 discusses implications and limitations, and Section 6 concludes the paper.

2. Related Work

2.1. Livestreaming Engagement, Gifting, and Patronage

Livestreaming platforms have transformed online media consumption into a real-time, interaction-oriented environment in which content viewing, social exchange, and monetization occur simultaneously. Unlike asynchronous video platforms, livestreaming services allow viewers to communicate with streamers and other viewers during the broadcast, creating a sense of immediacy, co-presence, and collective participation. Prior studies have shown that continued watching and engagement in livestreaming environments are shaped by multiple factors, including entertainment value, social interaction, information seeking, trust, perceived responsiveness, and social presence [1,2,6,12,13,40,41,42]. These findings suggest that viewer behavior in livestreams should not be understood merely as passive content consumption, but as a socially embedded and temporally dynamic process. A key theoretical concept in this context is parasocial interaction. In livestreaming, viewers often perceive streamers as socially accessible and responsive, even though the relationship remains asymmetric. This perception can strengthen attachment, identification, and repeated participation [14,15]. Compared with traditional media, livestreaming intensifies parasocial dynamics because viewers can send messages, receive immediate feedback, participate in collective chat activity, and observe other viewers’ responses in real-time. As a result, audience participation becomes both an individual behavior and a group-level signal of collective attention. These interactional conditions are directly relevant to monetization because financial support often occurs in visible public settings rather than through private or delayed transactions. Donation, gifting, paid memberships, and digital patronage extend engagement dynamics into observable monetization behavior. Earlier studies found that viewer engagement is positively associated with gift-giving in live video streaming [3], that danmaku-mediated interaction can stimulate paid gifting [4], and that emotional attachment, flow experience, social influence, reciprocity, and identity-based motivations shape viewers’ willingness to provide financial support [7,8,9,10,11]. In game-streaming and creator-centered communities, digital patronage is also associated with social recognition, community belonging, and the desire to support creators whose content or persona viewers value [12,13,14,15,16,17,18,19]. Taken together, prior work establishes that livestreaming monetization is not independent of audience interaction. Rather, paid support is often embedded in the same real-time social processes that drive engagement, visibility, and collective participation. However, much of this literature has focused on explaining engagement, intention, motivation, or aggregate gifting behavior rather than predicting short-horizon monetization events from time-localized behavioral signals. This distinction is important for the present study. If paid support is partly reflected in observable audience dynamics, then window-level chat activity, member participation, and message intensity may provide useful signals for predicting whether a Superchat event will occur in the near-future. This motivates a transition from engagement explanation to leakage-aware, short-horizon monetization prediction.

2.2. Virtual Streamers, VTubers, and Monetization

VTubers represent a distinctive form of livestreaming creator because their public identity is mediated through animated avatars, fictionalized personas, and performance-based character presentation. Unlike conventional livestreamers, VTubers interact with viewers through a hybrid identity that combines streamer agency, character design, voice performance, narrative framing, and community-specific norms. Prior studies have shown that avatar-mediated performance can shape perceptions of authenticity, intimacy, attractiveness, social presence, and parasocial attachment [20,21,22,23,26,27]. These characteristics make VTuber livestreaming an important context for examining how technologically mediated identity affects audience engagement and financial support. Superchat behavior is particularly important in VTuber communities because paid messages are not only financial transactions, but also public interactional acts. Viewers may use Superchat to attract the streamer’s attention, display support to the community, participate in collective rituals, or strengthen their perceived relationship with the virtual persona. In this sense, Superchat can function simultaneously as donation, communication, visibility purchase, and symbolic participation. This differs from purely private subscription or payment behavior because the act is temporally embedded in the live-chat stream and publicly visible to other viewers. Therefore, Superchat occurrence can be interpreted as an event-level manifestation of monetized engagement within the unfolding dynamics of a livestream. Recent studies on the VTuber ecosystem have documented channel growth, fan participation, subscriber behavior, engagement patterns, and revenue inequality [24,25]. These studies are important because they show that VTuber monetization is highly uneven and shaped by creator popularity, audience scale, and community behavior. Related works on virtual streamers and virtual influencers have also examined how perceived authenticity, anthropomorphism, coolness, social presence, and viewer identification affect purchase intention, support intention, or engagement-related outcomes [20,21,22,26,27]. Together, these findings suggest that VTuber monetization is shaped by both platform-level visibility and relational audience dynamics. However, existing VTuber monetization research remains limited in two respects. First, much of the literature focuses on retrospective, channel-level, or ecosystem-level patterns, such as subscriber growth, aggregate Superchat revenue, or fan engagement distributions. These approaches are useful for explaining long-term monetization inequality, but they do not directly address whether short-term live-chat behavior can predict an upcoming Superchat event. Second, studies based on surveys or attitudinal models can explain why viewers may support VTubers, but they cannot directly test how observable behavioral signals within a livestream relate to near-future monetized events. As a result, there remains a methodological gap between explanatory research on VTuber support behavior and predictive research on time-localized Superchat occurrence.

The present study addresses this gap by treating Superchat prediction as a short-horizon event prediction problem within VTuber livestreams. Rather than predicting aggregate channel revenue or long-term donation propensity, the task is to estimate whether at least one Superchat event will occur in the next five-minute window based on chat-derived signals available up to the current window. This formulation allows VTuber monetization to be analyzed not only as a retrospective ecosystem outcome, but also as a dynamic process reflected in real-time audience activity, member participation, message intensity, and short-horizon behavioral changes.

2.3. Predictive Modeling, Leakage Prevention, and Evaluation Under Imbalance

The studies reviewed above establish that livestream monetization is strongly connected to audience engagement, social interaction, parasocial attachment, and community participation. However, explaining why viewers engage or donate is different from predicting when a monetization event will occur. Predictive modeling requires a more operational formulation: the outcome must be defined at a specific temporal resolution, the input signals must be observable before the target event, and the evaluation protocol must reflect how the model will be used on unseen livestream data. This distinction is especially important for Superchat prediction because paid-message events are temporally sparse, socially visible, and often embedded in short bursts of audience activity. Existing predictive studies on livestreaming and online engagement can be broadly grouped into three research streams. The first stream focuses on viewer engagement prediction, such as continued watching, participation, purchase intention, or interaction intensity. These studies show that social presence, trust, perceived responsiveness, entertainment value, and interaction quality can predict engagement-related outcomes [1,2,3,4,5,6,19,40,41,42,43]. Although such studies provide useful evidence that livestream behavior is predictable from social and behavioral signals, their target variables are often attitudinal, aggregate, or session-level outcomes rather than event-level monetization occurrences. As a result, they do not directly answer whether a short-term behavioral state within a livestream can predict an upcoming Superchat event. The second stream focuses on donation, gifting, and digital patronage. Prior studies have examined how viewer engagement, danmaku interaction, emotional attachment, flow, reciprocity, social influence, identity, and community belonging shape viewers’ willingness to donate or provide financial support [3,4,7,8,9,10,11,12,13,14,15,16,17,18]. These studies are directly relevant to Superchat behavior because they show that monetary support is associated with observable interactional and relational mechanisms. However, many of these studies are explanatory rather than predictive. They often analyze survey responses, long-term support intentions, aggregate gifting behavior, or platform-level monetization patterns. Therefore, they offer limited guidance on how to construct a leakage-safe, time-localized prediction task in which current audience dynamics are used to estimate near-future monetization occurrence. The third stream focuses on computational prediction of donation-like events from live interaction traces. The closest prior predictive work to the present study modeled real-time donation events in YouTube livestreaming using dynamic interaction structures [28]. This work is important because it demonstrates that donation events can be predicted from evolving interaction patterns rather than being treated as random or purely exogenous outcomes. Nevertheless, its methodological emphasis differs from the objective of the present study. Dynamic-graph donation prediction focuses on modeling evolving interaction structures in general livestreaming contexts, whereas the present study focuses on VTuber Superchat occurrence as a next-window binary event prediction problem using interpretable window-level features. The goal is not to replace graph-based modeling, but to examine whether short-horizon monetization events in VTuber livestreams can be predicted under a feature-timing discipline and grouped evaluation protocol that reduce leakage and partition contamination. This distinction leads to several methodological requirements. First, the prediction unit must be explicitly defined. In the present study, each livestream is divided into non-overlapping five-minute windows, and the model predicts whether at least one Superchat event occurs in the immediately following window. Second, the feature space must be interpretable and temporally aligned with the prediction moment. Accordingly, the proposed framework uses audience activity, participation breadth, member composition, message intensity, donation-state information, and short-horizon dynamics as structured feature groups. Third, predictive performance must be compared not only against generic machine-learning classifiers, but also against simple behavioral and temporal heuristics. Previous-window Superchat occurrence, past-window Superchat rate, cumulative within-video Superchat rate, activity-only models, and channel-rate baselines are important because they test whether the proposed model captures broader audience dynamics rather than merely reproducing donation continuation or creator-level propensity.

The present study is positioned at the intersection of these research streams. It builds on livestream engagement research by treating real-time audience activity as a predictive signal, extends donation and patronage research by modeling Superchat as an event-level monetization outcome, and relates to computational donation prediction by providing a leakage-guarded window-level alternative to graph-centered modeling. The contribution is therefore not the introduction of a novel classifier, but the formulation of a VTuber-specific short-horizon prediction task, the use of temporally disciplined feature construction, and the evaluation of the model against stronger temporal, behavioral, channel-level, and graph-proxy baselines.

2.4. Positioning Relative to Prior Predictive Donation Studies

Short-horizon prediction in livestream environments requires particular attention to leakage-aware evaluation. In temporal event prediction, leakage can occur when information that would not be available at the prediction moment is included in the feature representation or when temporally related observations are split across training and test subsets in a way that inflates apparent generalization performance [28,29]. This issue is especially relevant for window-based livestream data because adjacent windows from the same broadcast may share highly similar audience composition, conversational context, streamer behavior, and donation momentum. If windows from the same livestream are randomly assigned to both training and test sets, the model may benefit from within-video similarity rather than learning patterns that generalize to held-out streams. Leakage in this setting can be understood at two levels. The first is feature-level leakage, which occurs when current or future donation information is used in a way that overlaps with the target definition. For next-window Superchat prediction, features must be observable no later than the current window, whereas the target is defined over the immediately following window. Therefore, rolling summaries, lagged variables, and donation-state features must be constructed with explicit temporal alignment. The second is partition-level leakage, which occurs when the evaluation split allows the same livestream, creator, or closely related temporal sequence to influence both the training and test performance. Grouped splitting by video is therefore more appropriate than random row-level splitting for evaluating whether the model generalizes beyond the specific livestreams used during training. Creator-level heterogeneity introduces an additional evaluation concern. Livestream monetization can vary substantially across channels because of differences in audience size, fan culture, membership composition, donation norms, and streamer popularity. A model evaluated only under video-level grouping may still encounter channels during testing that were represented in training through other videos. This does not necessarily invalidate video-level evaluation, but it limits the strength of claims about unseen-channel generalization. For this reason, channel-level grouped evaluation provides a stricter robustness check when channel identifiers are available. It helps distinguish whether a model captures short-term audience dynamics or merely exploits channel-specific donation propensity. Class imbalance is another central challenge in Superchat prediction. Paid-message events are relatively rare compared with ordinary chat activity, and the positive class may be highly concentrated in particular videos or channels. In such cases, accuracy is not informative because a model can achieve high accuracy by predicting the majority class. ROC-AUC can also appear optimistic under severe imbalance because it summarizes ranking performance across both positive and negative classes without directly reflecting the precision cost of rare-event detection [30,31]. Precision–recall-oriented metrics, such as PR-AUC, precision, recall, and F1 score, are therefore important for evaluating whether the model identifies positive Superchat windows without producing excessive false positives [30,31,33,34,35,36]. Because PR-AUC is sensitive to the positive-class prevalence of the test set, prevalence-aware interpretation and robustness checks across multiple split compositions are also necessary. Calibration is equally important when predictive scores are intended to support monitoring or decision-support applications. A model may rank high-risk windows effectively while still producing probability estimates that do not correspond to observed event frequencies. Poorly calibrated probabilities can lead to misleading alert thresholds, unreliable dashboard outputs, and inappropriate operational interpretation. Calibration methods such as sigmoid scaling and isotonic regression are commonly used to adjust predicted probabilities after model training, while Brier score, expected calibration error, and reliability diagrams are used to assess probability quality [32,37,38,39]. Importantly, calibration should be fitted on validation data and evaluated on an independent test set to avoid overestimating reliability. The discussion above shows that the present study is situated at the intersection of livestream engagement research, VTuber monetization studies, predictive donation modeling, and leakage-aware machine-learning evaluation.

Table 1. Methodological positioning of the proposed framework relative to related research streams.

Research Stream	Main Focus in Prior Work	Remaining Limitation for the Present Task	Position of the Present Study
Livestream engagement and digital gifting	Viewer engagement, continued watching, social presence, trust, interaction, gifting, and patronage behavior	Often explains engagement or support behavior at the viewer, session, or intention level, rather than predicting time-localized monetization events	Uses live-chat dynamics as observable behavioral signals for next-window Superchat occurrence prediction
VTuber monetization and virtual performer studies	Avatar-mediated identity, parasocial attachment, fan engagement, channel growth, Superchat revenue, and monetization inequality	Mainly retrospective, ecosystem-level, channel-level, or attitudinal; limited focus on short-horizon event prediction within live sessions	Formulates VTuber Superchat as a short-horizon event-level prediction task
Predictive donation and interaction-structure modeling	Real-time donation prediction using evolving interaction structures, including graph-based representations	Not centered on a VTuber-specific, leakage-guarded, window-level prediction setting with grouped evaluation	Provides a leakage-guarded window-level formulation and includes graph-proxy interaction features as an additional baseline
Leakage-aware temporal evaluation	Avoiding feature leakage, temporal leakage, and partition contamination in predictive modeling	Often treated separately from livestream monetization studies, despite the temporal dependence of livestream windows	Uses past-observable features, shifted rolling summaries, video-level grouped splitting, and channel-level grouped robustness checks
Imbalanced and calibrated event prediction	Evaluation of rare-event prediction, precision–recall analysis, probability calibration, and reliability assessment	Accuracy or ROC-only reporting can be insufficient for sparse Superchat events; probability estimates may be poorly calibrated	Reports PR-AUC, precision, recall, F1, prevalence-aware robustness checks, Brier score, expected calibration error, and reliability diagrams

summarizes the methodological positioning of the proposed framework relative to these related research streams. In contrast to a narrow comparison with a single predictive donation model, the table clarifies how the present study extends prior work by combining a VTuber-specific next-window prediction task, leakage-guarded feature construction, grouped evaluation, imbalanced-event metrics, calibration analysis, and graph-proxy baseline comparison.

As shown in Table 1, the contribution of the present study is not limited to applying a machine-learning classifier to Superchat data. Rather, the proposed framework is methodologically positioned as a leakage-guarded, event-level prediction design for VTuber livestream monetization. It extends prior livestream engagement and donation research by converting monetized audience support into a short-horizon predictive task, while also addressing feature timing, data partitioning, class imbalance, channel-level heterogeneity, and probability calibration within a unified evaluation framework.

3. Materials and Methods

3.1. Dataset and Analytical Scope

This study uses publicly released VTuber live-chat and Superchat logs distributed through the VTuber ecosystem and related public repositories [24,25]. The source data consist of timestamped event records for ordinary chat messages and paid Superchat events. We retained videos for which chat records and Superchat records could be consistently parsed, temporally aligned, and reconstructed into window-level observations. The final analytical subset covered livestreams observed between 18 March 2021 and 31 July 2022. After preprocessing, the retained dataset consisted of 300 livestream videos. By recovering video-to-channel mappings from the raw VTuber data files, all 300 videos were mapped to 89 unique VTuber channels. The unit of analysis is a non-overlapping five-minute time window. For each video, chat and Superchat events were aggregated into consecutive five-minute windows according to their timestamps. Each focal window contained observable audience activity up to that time window, whereas the prediction target was defined over the immediately following window. This construction enables short-horizon event prediction while preserving the temporal order of livestream interactions. The five-minute window length was used as the main temporal resolution because it provides a practical balance between responsiveness and sufficient behavioral aggregation. Additional window-size sensitivity analyses are reported in the robustness results. The initial reconstruction produced 648,619 five-minute candidate windows. However, many livestream periods contained no ordinary chat messages. Because the purpose of the main analysis was to predict near-future Superchat occurrence from observable audience behavior, the primary dataset focused on behaviorally active windows in which at least one ordinary chat message was present in the focal window. Zero-chat focal windows were excluded from the main analysis because chat-derived behavioral features, such as audience activity, member composition, and message intensity, are either undefined or uninformative when no chat activity is observed. This filtering produced 190,065 active-chat windows. Among the 190,065 active-chat windows, 21,714 were positive cases in which at least one Superchat occurred in the immediately following five-minute window, and 168,351 were negative cases. The resulting positive ratio was 11.42%, indicating that next-window Superchat prediction remains an imbalanced binary event-prediction problem even after restricting the analysis to active-chat periods. In comparison, the full all-window reconstruction before zero-chat filtering had a substantially lower positive ratio of 3.88%. Therefore, the active-chat setting should be interpreted as the main monitoring-oriented setting in which audience behavior is observable, while the all-window setting is used as a sensitivity analysis to examine the effect of zero-chat filtering.

Table 2. Dataset construction and analytical scope.

Dataset Characteristic	Value	Interpretation
Source data	Public VTuber live-chat and Superchat logs	Timestamped ordinary chat and paid Superchat events
Observation period	18 March 2021–31 July 2022	Period covered by the retained processable records
Retained videos	300	Livestream videos with processable chat and Superchat records
Recovered channels	89	Unique VTuber channels mapped from raw video-to-channel metadata
Window length	5 min	Main temporal aggregation unit
Initial reconstructed windows	648,619	All five-minute candidate windows before zero-chat filtering
Positive ratio before zero-chat filtering	3.88%	Next-window Superchat occurrence rate in the all-window reconstruction
Excluded zero-chat focal windows	458,554	Windows without ordinary chat activity in the focal window
Final active-chat windows	190,065	Main analytical dataset used for model training and evaluation
Positive next-window Superchat cases	21,714	Active-chat windows followed by at least one Superchat in the next window
Negative next-window cases	168,351	Active-chat windows not followed by a Superchat in the next window
Positive ratio after zero-chat filtering	11.42%	Next-window Superchat occurrence rate in the active-chat setting
Main analytical setting	Active-chat windows	Monitoring-oriented setting with observable audience behavior
Sensitivity setting	All reconstructed windows	Used to evaluate the effect of zero-chat inclusion

summarizes the dataset construction and filtering process. The table reports both the full reconstructed window set and the active-chat analytical subset so that the effect of zero-chat filtering is transparent. The retained 300 videos and 89 channels provide a multi-channel VTuber livestream sample, but the dataset should not be interpreted as a complete or language-stratified representation of the entire VTuber ecosystem. The source logs are platform event records rather than a manually balanced sample by language, creator agency, region, or content genre. Accordingly, the empirical claims of this study are limited to the processed VTuber livestream subset and are evaluated through video-level and channel-level grouped protocols rather than through population-level representativeness assumptions.

The zero-chat filtering step is therefore not intended to make the task artificially easier, but to align the main prediction setting with the study objective: estimating near-future Superchat occurrence from observable live-chat dynamics. To verify that the conclusions were not solely an artifact of this filtering decision, we additionally evaluated the model on the all-window reconstruction including zero-chat periods and report the corresponding sensitivity results in Section 4.

3.2. Task Formulation

The objective of this study was to predict whether a Superchat event would occur in the immediately following time window of a VTuber livestream. The prediction task was formulated at the level of non-overlapping five-minute windows. Let v denote a livestream video, and let V denote the set of retained videos. Each video v is divided into a sequence of five-minute windows as follows:

$$W_v= \left\{w_{v,1},w_{v,2},\cdots ,w_{v,Tv}\right\}$$

(1)

Equation (1) defines the temporal window sequence for each livestream video. Here, W_v,t denotes the t-th five-minute window of video v and T_v denotes the total number of reconstructed windows for that video. Each window contains timestamped chat and Superchat events aggregated within the corresponding five-minute interval. For each focal window w_v,t, the prediction target is defined over the immediately following window w_v,t+1. Let SC_v,t+1 denote the number of Superchat events observed in the next window. The binary target variable is defined as follows:

$$y_{v,t+1}= \left\{\begin{array}{c}1, \mathrm{if} {SC}_{v,t+1} \ge 1\\ 0, \mathrm{if} {SC}_{v,t+1} = 0\end{array}\right.$$

(2)

Equation (2) defines the supervised target variable. Specifically, y_v,t+1 = 1 indicates that at least one Superchat event occurs in the next five-minute window, whereas y_v,t+1 = 0 indicates that no Superchat event occurs in that window. Because the target is defined over the following window, the final window of each video is not used as a supervised focal window unless a valid subsequent target window is available. For each focal window w_v,t, the feature vector is constructed only from information observable up to the focal window. The general feature representation is defined as follows:

$$x_{v,t}= \left[f\_act(\mathrm{v},\mathrm{t}),f\_comp(\mathrm{v},\mathrm{t}),f\_int(\mathrm{v},\mathrm{t}),f\_state(\mathrm{v},\mathrm{t}),f\_dyn(\mathrm{v},\mathrm{t})\right]$$

(3)

Equation (3) defines the window-level feature vector used for next-window prediction. The term f_act(v, t) denotes audience activity features, f_comp(v, t) denotes member-composition features, f_int(v, t) denotes message-intensity features, f_state(v, t) denotes donation-state features, and f_dyn(v, t) denotes short-horizon dynamic features. The bracketed expression in Equation (3) indicates feature concatenation. The detailed feature groups and their timing constraints are described in Section 3.3. The prediction problem is formulated as a binary classification task. Given the feature vector x_v,t, the model estimates the conditional probability that at least one Superchat event will occur in the next window:

$${\hat{p}}_{v,t+1}=\mathrm{P}(y_{v,t}+1=1|x_{v,t})$$

(4)

Equation (4) defines the predicted probability of next-window Superchat occurrence. Specifically, ${\hat{p}}_{\mathrm{v},\mathrm{t}+1}$ denotes the model-estimated probability that at least one Superchat event will occur in window w_v,t+1, conditional on the feature vector w_v,t observed at the focal window w_v,t. The model therefore does not estimate aggregate revenue or long-term donation propensity. Instead, it estimates the probability of a short-horizon monetization event based on currently observable live-chat dynamics. To convert the predicted probability ${\hat{p}}_{\mathrm{v},\mathrm{t}+1}$ into a binary prediction, a decision threshold τ is applied. The threshold τ is selected only on the validation set and then fixed for test-set evaluation:

$${\hat{y}}_{v,t+1}=\left\{\begin{array}{c}1, \mathrm{if} {\hat{p}}_{v,t+1} \ge \tau \\ 0, \mathrm{if} {\hat{p}}_{v,t+1} < \tau \end{array}\right.$$

(5)

Equation (5) defines the threshold-based binary prediction rule. This procedure prevents the test set from being used for model selection or decision-rule tuning. The thresholded prediction $\widehat{y}$_v,t+1 is used to compute classification metrics such as precision, recall, and F1 score, whereas the predicted probability ${\hat{p}}_{\mathrm{v},\mathrm{t}+1}$ is used for ranking and calibration metrics such as PR-AUC, ROC-AUC, Brier score, and expected calibration error. The supervised dataset is represented as follows:

$$D=\left\{\left(x_{v,t},y_{v,t+1}\right) |\mathrm{v}\in \mathrm{V}, \mathrm{t}=1,\cdots ,T_v-1\right\}$$

(6)

Equation (6) defines the final supervised learning dataset. Each training example consists of a focal-window feature vector x_v,t and its immediately following next-window target y_v,t+1. The main analytical setting uses active-chat focal windows, as described in Section 3.1, so that x_v,t contains observable audience behavior. The all-window setting, including zero-chat focal windows, is used only for sensitivity analysis. This formulation enforces a clear temporal ordering between features and targets. The model observes chat-derived behavioral signals from the focal window and past windows, whereas the label is defined only by Superchat occurrence in the next window. Therefore, information from w_v,t+1 is not used to construct x_v,t. This temporal separation is essential for preventing feature-level leakage in short-horizon livestream prediction. The next-window formulation also distinguishes the present task from aggregate Superchat revenue prediction or long-term channel-level donation propensity modeling. Rather than estimating how much revenue a video or channel will eventually receive, the task focuses on whether a monetized interaction event will occur in the next short time interval. This design is suitable for evaluating real-time monitoring scenarios, where the model must make predictions from currently observable live-chat dynamics without access to future Superchat outcomes.

3.3. Window Construction and Feature Engineering

For each focal window w_v,t, we constructed a set of window-level features from ordinary chat messages, member indicators, message-length statistics, Superchat-state variables, and short-horizon temporal dynamics. Following the prediction problem and feature-space formulation introduced in Section 3.2, all predictors were constrained to information observable no later than the focal window w_v,t, whereas the prediction target was defined over the next window w_v,t+1. This separation between the feature window and the target window is central to the leakage-guarded design of the study. As defined in Equation (3), the feature vector x_v,t consists of five conceptual feature groups: audience activity, member composition, message intensity, current donation state, and short-horizon dynamics. Audience activity features summarize the immediate volume and breadth of viewer participation in the focal window. Representative variables include the total number of ordinary chat messages and the number of unique chatters. These features capture whether the livestream is experiencing active audience mobilization at the prediction moment. Member-composition features describe the participation of accounts marked as members. In particular, share_members is defined as the proportion of ordinary chat messages in the focal window that were authored by member accounts. When at least one ordinary chat message is present in the focal window, share_members is computed as the number of member-authored ordinary chat messages divided by the total number of ordinary chat messages in that window. This variable uses only focal-window chat metadata and does not require any future Superchat information. Because the main analysis uses active-chat windows, the denominator is positive in the primary analytical dataset. Message intensity features summarize the expressive volume of ordinary chat communication. Representative variables include the total and mean body length of ordinary chat messages in the focal window. These variables approximate the extent to which viewers are producing longer or more concentrated textual responses. Although message length does not directly measure sentiment or semantic content, it provides a lightweight behavioral proxy for conversational intensity that can be computed consistently across large scale live-chat logs. Current donation-state features summarize Superchat activity that has already occurred within the focal window. Representative variables include the number of Superchat events and the total Superchat amount in the focal window. These variables can be informative because recent donation activity may indicate ongoing monetization momentum. However, they represent a stronger information condition because they use current-window donation-state information. Therefore, direct current donation-state variables are excluded from the strict setting and included only in the extended setting. The extended setting is interpreted as a donation-state-aware upper-bound comparison rather than the primary leakage-guarded setting. Short-horizon dynamic features summarize recent temporal trajectories around the focal prediction point. These features include rolling means or rolling sums of chat activity, unique chatters, member composition, message intensity, and past Superchat-related signals. To prevent feature-level leakage, all rolling features were computed after applying a one-window shift within each video. Thus, a three-window rolling feature for focal window t uses information from windows t − 1, t − 2, and t − 3, rather than information from the next-window target t + 1. This timing rule ensures that rolling summaries represent past dynamics available at the prediction moment.

Table 3. Main feature groups used in the window-level prediction framework.

Feature Group	Representative Variables	Behavioral Interpretation	Role in Evaluation
Audience activity	n_chats; n_unique_chatters	Captures immediate participation intensity and breadth of audience mobilization	Included in both strict and extended settings
Member composition	share_members	Measures the proportion of ordinary chat messages authored by member accounts in the focal window	Included in both strict and extended settings
Message intensity	sum_bodyLength; mean_bodyLength	Represents expressive effort and conversational concentration in ordinary chat messages	Included in both the strict and extended settings
Current donation state	n_superchats; sc_total	Summarizes already realized Superchat momentum within the focal window	Included only in the extended setting
Short-horizon dynamics	Rolling means or sums of chats, unique chatters, member composition, message length, and past Superchat-related signals	Encodes recent temporal trajectories beyond the current single-window snapshot	Included in both settings when computed from past-shifted windows; direct current donation-state variables are excluded from the strict setting

summarizes the main feature groups used in the window-level prediction framework. The table also clarifies whether each feature group is included in the strict and extended settings.

To further verify the temporal validity of the rolling variables, we audited representative rolling features used in the modeling pipeline.

Table 4. Rolling feature timing audit for representative features.

Rolling Feature	Expected Timing Definition	Past-Only Verified
n_chats_roll3_mean	n_chats.shift(1).rolling(3).mean() within each video	True
n_chats_roll3_sum	n_chats.shift(1).rolling(3).sum() within each video	True
n_unique_chatters_roll3_mean	n_unique_chatters.shift(1).rolling(3).mean() within each video	True
n_unique_chatters_roll3_sum	n_unique_chatters.shift(1).rolling(3).sum() within each video	True
share_members_roll3_mean	share_members.shift(1).rolling(3).mean() within each video	True
share_members_roll3_sum	share_members.shift(1).rolling(3).sum() within each video	True
mean_bodyLength_roll3_mean	mean_bodyLength.shift(1).rolling(3).mean() within each video	True
mean_bodyLength_roll3_sum	mean_bodyLength.shift(1).rolling(3).sum() within each video	True
sum_bodyLength_roll3_mean	sum_bodyLength.shift(1).rolling(3).mean() within each video	True
sum_bodyLength_roll3_sum	sum_bodyLength.shift(1).rolling(3).sum() within each video	True
n_superchats_roll3_mean	n_superchats.shift(1).rolling(3).mean() within each video	True
sc_total_roll3_sum	sc_total.shift(1).rolling(3).sum() within each video	True

reports examples of the expected timing definitions for these variables. All audited rolling variables followed the same rule: each base variable was shifted by one window within each video before the rolling aggregation was computed. Therefore, the rolling feature value at focal window t summarizes only previous windows and does not use information from the next-window target.

The timing audit in Table 4 directly addresses the risk that rolling variables may inadvertently include current or future target information. Because rolling features are computed after applying shift(1) within each video, the feature value at window t depends only on windows preceding t. This is particularly important for Superchat-related rolling variables in the strict setting. Although direct current-window Superchat-state variables were excluded from the strict setting, past Superchat-related rolling summaries may still be used when they are computed from shifted historical windows. In other words, the strict setting does not prohibit all historical donation signals; rather, it prohibits direct use of current-window donation-state variables and any future information overlapping with the next-window target.

The distinction between strict and extended feature settings was used throughout the experiments. The strict setting was treated as the primary leakage-guarded setting because it relies on audience activity, member composition, message intensity, and past-shifted short-horizon dynamics without direct current-window Superchat-state variables. The extended setting adds current donation-state variables to examine how much predictive performance changes when already realized focal-window Superchat information is available. This design allows the analysis to separate behavior-only predictive signals from donation-state-aware prediction and provides a clearer interpretation of the empirical results.

3.4. Leakage-Guarded Evaluation Design

A central methodological concern in event prediction from temporally adjacent livestream windows is information leakage. Leakage occurs when training and evaluation observations are not sufficiently separated or when the model can exploit information that would not be available under realistic prediction conditions [29]. In the present task, this risk is particularly important because adjacent windows from the same livestream may share similar audience states, streamer context, topic flow, and donation momentum. If windows from the same video are randomly divided across training and test sets, the model may appear to generalize while actually benefiting from near-duplicate temporal context. To reduce this risk, the primary evaluation design uses video-level grouping. All windows from a given livestream are assigned to only one subset among training, validation, and testing. This design prevents the model from being trained on one portion of a livestream and evaluated on another portion of the same livestream. The resulting evaluation is therefore stricter than row-level random splitting and better aligned with the intended use case of predicting Superchat occurrence in unseen livestream sessions. In other words, the main evaluation is designed to assess video-level generalization rather than within-video memorization. In addition to partition-level leakage control, we define two feature settings, WITHOUT_CURRENT_SC and WITH_CURRENT_SC, to separate the strict leakage-guarded condition from the extended donation-state-aware condition. Here, SC denotes Superchat. The ‘WITHOUT_CURRENT_SC’ setting is the primary strict condition. It excludes direct current-window Superchat-state variables and tests whether next-window Superchat occurrence can be predicted from ordinary chat-derived engagement dynamics, member composition, message intensity, and past-shifted short-horizon dynamics. In contrast, the ‘WITH_CURRENT_SC’ setting represents an extended condition in which current-window Superchat-state variables are included. This setting may be realistic in dashboard environments where current donation summaries are already visible at prediction time, but it also represents a stronger information condition because current-window donation momentum is directly available.

Table 5. Summary of the strict and extended evaluation settings.

Evaluation Setting	Feature Composition	Methodological Rationale
WITH_CURRENT_SC (Extended)	Includes current-window Superchat-state variables, such as n_superchats and sc_total, in addition to the strict feature set	Represents a donation-state-aware upper-bound condition in which current-window donation summaries are already available at prediction time
WITHOUT_CURRENT_SC (Strict)	Excludes direct current-window Superchat-state variables, while retaining audience activity, member composition, message intensity, and past-shifted short-horizon dynamic features	Tests whether near-future Superchat events remain predictable from chat-derived engagement dynamics alone, without relying on direct current-window donation momentum

summarizes the difference between the strict and extended settings. The strict setting is used as the primary leakage-guarded condition, whereas the extended setting is retained as a donation-state-aware upper-bound comparison.

The leakage-control design operates at both the feature and partition levels. At the feature level, the strict setting removes direct current-window donation-state variables, and rolling features are computed using shifted historical windows, as described in Section 3.3. Thus, no feature in the strict setting uses information from the next-window target. At the partition level, model evaluation is based on grouped splits so that temporally related observations from the same livestream are not simultaneously placed in training and test subsets. The main evaluation uses video-level grouping, and additional robustness checks use channel-level grouping after recovering video-to-channel mappings from the raw VTuber data.

Figure 1 illustrates the overall leakage-guarded prediction and evaluation pipeline. The pipeline begins with raw VTuber chat and Superchat logs, constructs five-minute active-chat windows, generates past-observable features, predicts next-window Superchat occurrence, and evaluates the model on held-out video and channel test sets. The lower part of the figure highlights the three main safeguards used in the study: feature-level leakage control through shifted rolling summaries, partition-level leakage control through grouped splitting, and robustness and calibration analysis through prevalence-aware reporting.

This design directly addresses the two main leakage risks in window-level livestream prediction. First, the model cannot use future Superchat outcomes because the target is defined over window w_v,t+1, while features are constructed from information observable no later than window w_v,t. Second, the grouped evaluation protocol reduces partition contamination by preventing windows from the same livestream from appearing in both training and test sets. The channel-level extension further examines whether the model retains predictive value for unseen channels rather than relying only on creator-specific donation propensity. The strict and extended settings are therefore not two competing deployment claims, but two complementary diagnostic conditions. The strict setting evaluates the central research question of whether chat-derived behavioral dynamics can predict near-future Superchat occurrence under conservative timing constraints. The extended setting estimates the incremental value of direct current-window donation-state information. This distinction supports a more transparent interpretation of the model results and avoids overstating the practical meaning of performance obtained with stronger information assumptions.

3.5. Models, Metrics, and Interpretation Strategy

The primary evaluation protocol was designed to assess held-out livestream generalization while reducing partition-level leakage. Instead of using a random row-level split, the dataset was partitioned at the video level. All active-chat windows from the same livestream video were assigned to only one subset among training, validation, and testing. This grouped design prevents windows from the same livestream from appearing simultaneously in the training and test data, thereby reducing the risk that the model benefits from within-video temporal similarity or near-duplicate context. The training set was used to fit the predictive models, the validation set was used for threshold selection and post hoc calibration, and the test set was used only for final held-out evaluation. This separation is important because the decision threshold and calibration models can otherwise be indirectly tuned to the test set. The main evaluation therefore follows a three-stage protocol: model fitting on the training videos, threshold and calibration selection on the validation videos, and final reporting on the held-out test videos.

Table 6. Video-level train/validation/test split summary.

Split	Videos	Windows	Positive Cases	Negative Cases	Positive Ratio
Training	180	170,430	14,675	155,755	8.61%
Validation	45	7833	2626	5207	33.52%
Test	75	11,802	4413	7389	37.39%

summarizes the resulting video-level train/validation/test split. The split contains 180 videos for training, 45 videos for validation, and 75 videos for testing. Because Superchat behavior is unevenly distributed across livestreams, the positive ratio differs across subsets. This imbalance reflects the heterogeneous distribution of Superchat occurrence across videos; therefore, the subset-specific positive ratios are explicitly reported and considered in the robustness analyses.

As shown in Table 6, the grouped split eliminates within-video overlapped across the training, validation, and test subsets. However, it also produced different positive ratios across subsets. This occurs because Superchat events are concentrated in particular livestreams rather than uniformly distributed across videos. Therefore, the main split is interpreted as a held-out video-level evaluation rather than a prevalence-balanced benchmark. To avoid relying on a single split composition, we conducted additional robustness analyses, including repeated video-level grouped splits, balanced video-level robustness checks, chronological video-level evaluation, random row-level split diagnostics, and channel-level grouped evaluation. The results of these analyses are presented in Section 4. The main learner was a histogram-based gradient boosting classifier, which belongs to the family of gradient-boosted decision tree models [44,45,46]. This model was selected because it can handle nonlinear relationships among window-level behavioral features while remaining efficient for large tabular datasets. The main configuration used a learning rate of 0.05, a maximum of 300 boosting iterations, and a maximum of 31 leaf nodes. To address class imbalance, balanced sample weights were used for the histogram-based gradient boosting model. For linear or tree-based baseline models, where applicable, class-weight balancing was applied. The decision threshold was selected on the validation set by maximizing F1 score and was then fixed for test-set evaluation. This procedure ensures that threshold-dependent metrics such as precision, recall, and F1 score are not optimized directly on the held-out test set. Probability-based metrics, including PR-AUC and ROC-AUC, were computed from predicted probabilities. Calibration analyses were conducted using raw probabilities, sigmoid calibration, and isotonic calibration. Calibration models were fitted using validation predictions and then evaluated on the independent test set using Brier score, expected calibration error, and reliability diagrams.

Table 7. Model and evaluation configuration.

Component	Configuration
Split protocol	Video-level grouped train/validation/test split; repeated video-level grouped robustness checks; channel-level grouped robustness checks
Primary learner	Histogram-based gradient boosting classifier; learning rate = 0.05; maximum iterations = 300; maximum leaf nodes = 31
Class imbalance handling	Balanced sample weights for the histogram-based gradient boosting model; class–weight balancing for linear or tree-based baselines where applicable
Threshold selection	Validation-set F1 maximization; fixed threshold applied to the held-out test set
Probability evaluation	PR-AUC and ROC-AUC computed from predicted probabilities
Calibration	Raw probability, sigmoid calibration, and isotonic calibration; Brier score and expected calibration error reported
Robustness checks	Repeated video-level grouped splits, balanced video-level split, chronological video-level split, zero-chat sensitivity analysis, random row-level split diagnostic, and channel-level grouped evaluation
Interpretation analyses	Feature-group ablation, permutation importance, graph-proxy comparison, and feature-space visualization

summarizes the main model and evaluation configuration used in the study. The table is intended to make the experimental procedure reproducible and clarify how model fitting, imbalance handling, threshold selection, calibration, and interpretation were conducted.

In addition to the main video-level split, several robustness analyses were conducted to address split composition, class imbalance, and creator-level generalization. Repeated video-level grouped splits were used to examine whether model performance remained stable across different held-out video sets. A balanced video-level split was used to reduce the effect of extreme prevalence differences across subsets. A chronological video-level split was used to evaluate whether the model retained predictive value when videos were separated according to video start time. A random row-level split was included as a diagnostic comparison to illustrate potential partition contamination under non-grouped splitting. Because videos from the same channel may share creator-specific audience behavior and donation norms, channel-level grouped evaluation was also conducted after recovering video-to-channel mappings from the raw VTuber data. In this setting, all videos from the same channel were assigned to only one subset, preventing train–test channel overlap. This provides a stricter robustness check than video-level grouping and helps assess whether the model captures short-horizon audience dynamics rather than merely exploiting channel-specific donation propensity. Overall, the evaluation protocol combines grouped data partitioning, validation-only threshold selection, prevalence-aware metrics, repeated robustness checks, and calibration analysis. This design provides a more conservative and transparent assessment of next-window Superchat prediction than would be obtained from a single random row-level split or accuracy-centered evaluation.

3.6. Computational Environment and Software

No physical instruments, reagents, or biological/chemical agents were used in this study; therefore, company names and addresses for instruments and agents are not applicable. All experiments were implemented and executed in Python 3.14.2 using Jupyter Notebook on Windows 11, build 10.0.22631. Data preprocessing, window-level aggregation, and numerical operations were conducted using NumPy 2.4.2, pandas 3.0.2, SciPy 1.17.0, PyArrow 23.0.1, and joblib 1.5.3. Machine-learning models, grouped data splitting, evaluation metrics, post-hoc calibration, and feature-space diagnostics were implemented using scikit-learn 1.8.0. Figures were generated using Matplotlib 3.10.8.

4. Results

4.1. Overall Predictive Performance

The first analysis evaluated overall predictive performance on the primary held-out video-level test split. Candidate learners were trained under both the strict WITHOUT_CURRENT_SC setting and the extended WITH_CURRENT_SC setting. For threshold-dependent metrics, including F1, precision, and recall, the operating threshold was selected on the validation set by maximizing F1 score and was then applied unchanged to the held-out test set. Probability-based metrics, including PR-AUC, ROC-AUC, and Brier score, were computed from the predicted probabilities before thresholding.

Across all candidate models, HistGradientBoosting (HGB) provided the strongest and most stable performance under both evaluation settings.

Table 8. Best-model performance under the strict and extended evaluation settings.

Setting	PR-AUC	ROC-AUC	F1	Precision	Recall	Brier	Threshold
WITHOUT_CURRENT_SC (HGB)	0.899	0.920	0.822	0.824	0.809	0.171	0.856
WITH_CURRENT_SC (HGB)	0.909	0.926	0.825	0.847	0.804	0.168	0.873

summarizes the best-model performance on the held-out test set. For threshold-dependent metrics, including F1, precision, and recall, the operating threshold was selected on the validation set by maximizing F1 score and was then applied unchanged to the held-out test set. The practical importance of this comparison lies not only in the absolute performance values, but also in the fact that the strict WITHOUT_CURRENT_SC setting remains highly competitive even after current-window donation-state variables are removed. The extended WITH_CURRENT_SC setting shows only a modest improvement, indicating that current-window Superchat information provides additional predictive signal but is not the sole basis of the model’s performance. This result supports the central claim that chat-derived engagement dynamics contain meaningful information for next-window Superchat prediction.

The practical importance of Table 8 is not only the absolute score level, but also the comparison between the two information conditions. The strict model remains highly competitive even after direct current-window Superchat-state variables are removed. This supports the central claim that chat-derived engagement dynamics, member composition, message intensity, and past-shifted short-horizon dynamics contain meaningful information for next-window Superchat prediction. The extended setting should therefore be interpreted as a donation-state-aware upper-bound comparison rather than the primary leakage-guarded result.

Figure 2 compares PR-AUC across the candidate learners. The dummy baseline reflects the difficulty of the imbalanced task and remained close to the positive-class prevalence of the held-out test set. Logistic regression and random forest models provided substantially stronger discrimination than the dummy baseline, but HGB achieved the highest PR-AUC in both the strict and extended settings. The extended setting improved PR-AUC for the non-dummy learners, yet the small gap between the strict and extended HGB models suggests that direct current-window donation-state variables provide additional signal but do not fully explain the model’s predictive performance.

Because next-window Superchat occurrence is an imbalanced event-prediction problem, the precision–recall curve in Figure 3 provides a more diagnostic view than accuracy alone. The strict-setting HGB model maintained high precision across a broad range of recall values, indicating that the model can identify many positive monetization windows without producing an excessive number of false alarms. This pattern is important for monitoring scenarios in which alert quality is as important as overall ranking performance.

Figure 4 presents a threshold-based evaluation of the best strict-setting model. At the validation-selected threshold, the model correctly identified 3572 positive windows and 6628 negative windows while producing 761 false positives and 841 false negatives. Based on these values, the false-positive rate was 10.30%, and the false-negative rate was 19.06%. This operating point provides a practical balance between event coverage and false-alarm control. In a deployment-oriented monitoring context, the threshold could be adjusted depending on whether the application prioritizes lower false alarms or higher coverage of potential monetization events. Finally, the Brier scores in Table 8 provide an initial view of probability quality, but raw discrimination performance does not guarantee well-calibrated probabilities. Therefore, the probability reliability of the strict HGB model is examined separately through calibration analysis in the subsequent Results section. Likewise, because Table 8 and Figure 2, Figure 3 and Figure 4 are based on the primary video-level split, additional robustness analyses are reported in later subsections to examine zero-chat sensitivity, repeated video-level splits, channel-level generalization, and graph-proxy comparisons.

4.2. Robustness Analysis Across Repeated Video-Level Splits

Section 4.1 established the overall predictive performance of the proposed model under the primary held-out video-level split. This section further examines whether the observed performance can be explained by simpler continuation heuristics, reduced behavioral feature sets, or a particular split composition. This analysis is important because next-window Superchat occurrence may exhibit short-term persistence, and because Superchat-positive windows are unevenly distributed across livestream videos.

Table 9. Temporal, behavioral, and proposed-model baseline comparison.

Setting	Model	PR-AUC	ROC-AUC	F1	Precision	Recall	Brier
Baseline	Global train prevalence	0.374	0.500	0.544	0.374	1.000	0.317
Temporal heuristic	Previous-window Superchat heuristic	0.701	0.832	0.790	0.786	0.793	0.158
Temporal heuristic	Past 3-window Superchat-rate heuristic	0.798	0.882	0.794	0.815	0.774	0.123
Temporal heuristic	Cumulative past within-video Superchat rate	0.699	0.803	0.665	0.595	0.755	0.196
HGB baseline	Activity-only HGB	0.827	0.862	0.758	0.788	0.731	0.208
HGB baseline	Basic behavioral HGB	0.837	0.867	0.766	0.853	0.695	0.206
Proposed	Strict full HGB	0.899	0.920	0.822	0.856	0.790	0.171

compares the proposed strict HGB model with temporal, behavioral, and prevalence-based baselines under the same video-level grouped split. The global train-prevalence baseline provides a lower reference point that reflects the positive-class prevalence of the held-out test set. The previous-window Superchat occurrence heuristic and the past three-window Superchat occurrence-rate heuristic test whether prediction can be explained mainly by immediate or recent donation continuation. The activity-only and basic behavioral HGB baselines test how much predictive value is provided by ordinary chat activity, member composition, and message-intensity features before the full strict feature set is used.

The results in Table 9 show that simple temporal heuristics are informative but insufficient. The previous-window and past three-window baselines achieved substantially higher PR-AUC than the prevalence baseline, indicating that short-term donation history contains predictive signal. However, the strict full HGB model achieved the highest PR-AUC and F1 score among the compared models. This suggests that the proposed model captures broader chat-derived engagement dynamics beyond immediate donation continuation alone. The comparison between the activity-only, basic behavioral, and strict full HGB models further clarifies the contribution of feature design. The activity-only model already performed well, indicating that audience participation intensity and breadth are important predictors. Adding member composition and message-intensity features provided additional behavioral signal, while the strict full feature set yielded the strongest overall performance. These results support the view that next-window Superchat prediction benefits from combining multiple live-chat dynamics rather than relying on a single activity or donation-history signal. To complement the primary held-out test evaluation, repeated video-level grouped split experiments were conducted using multiple random seeds. This additional analysis follows the methodological concern that temporally structured behavioral logs require group-aware evaluation to reduce leakage and improve the reliability of reported predictive performance [29]. In each repetition, the grouping unit was the video, ensuring that windows from the same livestream were assigned to only one of the training, validation, or test subsets. The repeated-split analysis therefore provides an additional assessment of performance stability under different held-out video compositions.

Table 10. Repeated video-level grouped split robustness for HGB (mean ± SD over 10 splits).

Setting	PR-AUC	ROC-AUC	F1	Precision	Recall
WITHOUT_CURRENT_SC (strict)	0.805 ± 0.065	0.915 ± 0.019	0.750 ± 0.052	0.815 ± 0.036	0.698 ± 0.080
WITH_CURRENT_SC (extended)	0.821 ± 0.061	0.922 ± 0.018	0.762 ± 0.050	0.837 ± 0.031	0.700 ± 0.071

reports the mean and standard deviation of the main performance metrics across repeated video-level grouped splits. In the strict WITHOUT_CURRENT_SC setting, HGB achieved PR-AUC = 0.805 ± 0.065, ROC-AUC = 0.915 ± 0.019, F1 = 0.750 ± 0.052, precision = 0.815 ± 0.036, and recall = 0.698 ± 0.080. In the extended WITH_CURRENT_SC setting, HGB achieved PR-AUC = 0.821 ± 0.061, ROC-AUC = 0.922 ± 0.018, F1 = 0.762 ± 0.050, precision = 0.837 ± 0.031, and recall = 0.700 ± 0.071. The extended setting provides a modest improvement, while the strict setting retains substantial predictive performance even without direct current-window Superchat-state variables. These results indicate that the model’s performance is not confined to a single held-out video composition and that chat-derived behavioral dynamics remain informative under repeated grouped evaluation. The standard deviations in Table 10 indicate that performance varies across split repetitions, which is expected given the heterogeneous distribution of Superchat events across videos. The repeated-split results should therefore not be interpreted as eliminating all dataset heterogeneity. Rather, they show that the main performance pattern is preserved across alternative video-level partitions. Together with the baseline comparison in Table 9, these findings indicate that the model captures predictive information beyond simple prevalence, immediate donation continuation, and basic audience activity signals.

Overall, the baseline and repeated-split analyses strengthen the interpretation of the main result. The strict HGB model performs well against heuristic and reduced-feature baselines, while repeated grouped evaluation provides evidence that the performance pattern remains meaningful under alternative video-level partitions. Additional robustness dimensions, including zero-chat sensitivity, split-diagnostic analysis, channel-level generalization, graph-proxy comparison, and probability calibration, are examined in the following sections.

4.3. Robustness to Zero-Chat Inclusion and Split Protocols

Section 4.2 examined whether the proposed model outperforms temporal and behavioral baselines and whether the repeated video-level grouped split results remain stable across alternative held-out video compositions. This section further evaluates two additional robustness issues: the effect of including zero-chat focal windows and the sensitivity of performance interpretation to alternative split protocols. These analyses are important because Superchat occurrence is sparse, unevenly distributed across videos, and sensitive to how the prediction population and held-out subsets are constructed.

The zero-chat sensitivity analysis evaluates whether the main active-chat formulation is responsible for the observed performance pattern. In the main setting, focal windows without ordinary chat activity were excluded because the model is intended to predict next-window Superchat occurrence from observable live-chat dynamics. However, to test whether this filtering decision materially changes the conclusion, we also reconstructed and evaluated the task using all five-minute windows, including zero-chat focal windows.

As shown in

Table 11. Zero-chat sensitivity analysis.

Dataset	Windows	Positive Ratio	PR-AUC	ROC-AUC	F1	Brier
Active-chat windows only	190,065	11.42%	0.899	0.920	0.822	0.171
All windows including zero-chat windows	648,619	3.88%	0.876	0.923	0.802	0.237

, including zero-chat windows substantially changed the event prevalence, reducing the positive ratio from 11.42% to 3.88%. This lower prevalence made the task more sparse and led to a reduction in PR-AUC from 0.899 to 0.876 and F1 from 0.822 to 0.802. The Brier score also increased from 0.171 to 0.237, indicating poorer probability accuracy under the all-window formulation. Nevertheless, the model remained predictive even when zero-chat periods were included. This result supports the interpretation that the active-chat setting is a meaningful monitoring-oriented formulation, while also showing that the performance pattern is not entirely dependent on excluding zero-chat windows. We next examined whether the interpretation of model performance changes under alternative split protocols. Because PR-AUC is sensitive to the positive-class prevalence of the test set, split-level comparisons should not be interpreted using PR-AUC alone. Therefore, the diagnostic table reports test prevalence, AP lift, and normalized AP in addition to the standard discrimination and threshold-based metrics. AP lift is the ratio of PR-AUC to test prevalence, and normalized AP rescales PR-AUC relative to the prevalence baseline and the maximum achievable AP. These prevalence-aware indicators help separate ranking quality from changes in test-set class balance.

Table 12. Split-protocol diagnostic with prevalence-aware AP interpretation.

Protocol	Role	Test Prevalence	PR-AUC	AP Lift	Normalized AP	ROC-AUC	F1	Video Overlap
Video-level grouped split	Main leakage-guarded evaluation	37.39%	0.899	2.404	0.838	0.920	0.822	0
Random row-level split	Leakage-prone diagnostic	11.43%	0.800	7.006	0.775	0.923	0.748	300
Greedy balanced video-level split	Balanced video-level robustness	10.17%	0.748	7.358	0.719	0.897	0.695	0
Chronological video-level split	Temporal generalization robustness	22.07%	0.885	4.009	0.852	0.938	0.811	0

shows that the video-level grouped split preserved the intended leakage-guarded evaluation design, with zero train–test video overlap. In contrast, the random row-level split produced complete train–test video overlap, with all 300 videos appearing in both the training and test subsets. This confirms that row-level splitting contaminates the evaluation by allowing windows from the same livestream to cross subset boundaries. Although the random row-level split achieved a PR-AUC = 0.800, this value should be interpreted as a diagnostic result rather than as evidence of generalization because the model can be exposed to within-video context during training. The balanced video-level split and chronological video-level split provide additional robustness views. The greedy balanced video-level split lowered test prevalence to 10.17% and yielded a PR-AUC = 0.748 while maintaining zero video overlap. The chronological video-level split yielded PR-AUC = 0.885 and ROC-AUC = 0.938 under a video start-time-based separation, also with zero video overlap. These results indicate that performance varies with the composition and prevalence of the held-out subset, as expected in a heterogeneous Superchat dataset. At the same time, the model retains meaningful predictive value across grouped and time-ordered split diagnostics.

Overall, the zero-chat and split-protocol analyses clarify the scope of the main evaluation. The active-chat formulation is appropriate for monitoring settings in which audience behavior is observable, but the all-window sensitivity result shows that the model remains informative when zero-chat periods are included. The split-protocol diagnostics further show that grouped evaluation is necessary for avoiding within-video partition contamination. These findings support the use of video-level and channel-level grouped evaluation protocols in the subsequent robustness analyses.

4.4. Window-Size Sensitivity and Channel-Level Robustness

This section examines two complementary robustness dimensions. First, window-size sensitivity analysis evaluates whether the main five-minute temporal aggregation is dependent on a single temporal resolution. Second, channel-level grouped evaluation assesses whether the model retains predictive value when held-out data are separated by creator channel rather than only by livestream video.

4.4.1. Window-Size Sensitivity Analysis

To examine the robustness of the temporal aggregation choice, we conducted a window-size sensitivity analysis using 1, 2, 5, 10, and 15 min non-overlapping windows. For each window size, the window-level dataset was reconstructed from the source chat and Superchat event logs, the same zero-chat filtering rule was applied, and the strict and extended settings were evaluated under the same video-level grouped split protocol. This analysis was designed to assess whether the main five-minute specification provides a reasonable balance between short-horizon responsiveness, event sparsity, and predictive stability.

As shown in

Table 13. Window-size sensitivity analysis across 1, 2, 5, 10, and 15 min windows.

Size (Min)	Setting	PR-AUC	ROC-AUC	F1	Brier	ECE
1	Strict	0.886	0.929	0.799	0.159	0.205
1	Extended	0.899	0.936	0.807	0.154	0.199
2	Strict	0.897	0.928	0.818	0.160	0.210
2	Extended	0.909	0.935	0.825	0.153	0.204
5	Strict	0.896	0.919	0.821	0.172	0.219
5	Extended	0.910	0.926	0.826	0.166	0.214
10	Strict	0.892	0.909	0.812	0.185	0.221
10	Extended	0.909	0.920	0.821	0.175	0.215
15	Strict	0.882	0.896	0.809	0.190	0.214
15	Extended	0.901	0.910	0.818	0.179	0.210

, predictive performance remained stable across short-horizon window sizes. In the strict WITHOUT_CURRENT_SC setting, the 2 min window achieved the highest PR-AUC of 0.897, while the 5 min window achieved a nearly identical PR-AUC of 0.896 and the highest F1 score of 0.821. In the extended WITH_CURRENT_SC setting, the 5 min window achieved the strongest overall performance, with PR-AUC = 0.910 and F1 = 0.826. The 10 and 15 min windows also remained predictive, but their discrimination metrics gradually decreased relative to the shorter 2 and 5 min settings.

Figure 5 visualizes the sensitivity results reported in Table 13. The figure shows that the 2 and 5 min windows formed the strongest performance region across the evaluated temporal resolutions. The 1 min window provided high temporal responsiveness but produced a larger number of sparse windows, whereas the longer 10 and 15 min windows provided more interactional aggregation at the cost of reduced short-horizon specificity. The 5 min window therefore offers a practical operational compromise: it preserves short-horizon responsiveness while maintaining stable predictive performance and a manageable level of temporal aggregation.

Overall, the sensitivity analysis supports the use of the 5 min window as the main specification. The results indicate that the reported findings are not tied to a single temporal aggregation choice and that the proposed framework remains effective across nearby short-horizon window settings. This strengthens the methodological basis for using five-minute windows in the main analysis and aligns the prediction task with realistic dashboard-based monitoring and retrospective stream analysis scenarios.

4.4.2. Channel-Level Robustness

The main evaluation protocol uses video-level grouped splitting to prevent windows from the same livestream from appearing in both the training and test data. However, videos from the same VTuber channel may share creator-specific audience behavior, fan culture, membership composition, and donation norms. Therefore, video-level grouping alone does not fully establish unseen-channel generalization. To address this concern, we recovered video-to-channel mappings from the raw VTuber 1B parquet files. All 300 analyzed videos were mapped to 89 unique channels, enabling channel-level grouped evaluation.

Table 14. Single video-level and channel-level grouped split comparison.

Protocol	Model	Test Prevalence	PR-AUC	AP Lift	ROC-AUC	F1	Precision	Recall
Video-level grouped split	Strict full HGB	37.39%	0.899	2.404	0.920	0.822	0.856	0.790
Video-level grouped split	Train-derived channel-rate baseline	37.39%	0.395	1.055	0.532	0.544	0.374	1.000
Channel-level grouped split	Strict full HGB	7.88%	0.624	7.925	0.856	0.591	0.765	0.482
Channel-level grouped split	Train-derived channel-rate baseline with global fallback	7.88%	0.079	1.000	0.500	0.146	0.079	1.000

compares the strict full HGB model with a train-derived channel-rate baseline under video-level and channel-level grouped settings. The channel-rate baseline uses only channel-level Superchat propensity estimated from the training data and therefore provides a direct test of whether performance can be explained by creator-specific donation propensity alone.

Under the single channel-level grouped split, the strict HGB model achieved PR-AUC = 0.624 and ROC-AUC = 0.856 despite a lower test prevalence of 7.88%. In contrast, the train-derived channel-rate baseline collapsed to PR-AUC equal to the test prevalence and ROC-AUC = 0.500 under the same channel-level split. This contrast indicates that the proposed model does not merely reproduce channel-level donation propensity. Although the absolute PR-AUC was lower under the stricter channel-level split than under the video-level split, the AP lift remained high because the test-set prevalence was substantially lower. Therefore, channel-level results should be interpreted with prevalence-aware metrics rather than by PR-AUC alone. To assess whether this pattern was stable across different held-out channel compositions, we further conducted repeated channel-level grouped splitting across 10 random seeds. In every repetition, videos from the same channel were assigned to only one subset, and train–test channel overlap remained zero.

As shown in

Table 15. Repeated channel-level grouped split robustness.

Model	Repeats	Test Prevalence	PR-AUC	AP Lift	ROC-AUC	F1	Precision	Recall
Strict full HGB	10	0.138 ± 0.058	0.795 ± 0.087	6.421 ± 1.792	0.911 ± 0.033	0.742 ± 0.069	0.801 ± 0.031	0.699 ± 0.110
Train-derived channel-rate baseline with global fallback	10	0.138 ± 0.058	0.138 ± 0.058	1.000 ± 0.000	0.500 ± 0.000	0.239 ± 0.087	0.138 ± 0.058	1.000 ± 0.000

, the strict full HGB model achieved PR-AUC = 0.795 ± 0.087, AP lift = 6.421 ± 1.792, ROC-AUC = 0.911 ± 0.033, and F1 = 0.742 ± 0.069 across repeated channel-level grouped splits. In contrast, the train-derived channel-rate baseline achieved PR-AUC = 0.138 ± 0.058 and ROC-AUC = 0.500 ± 0.000, matching the prevalence-only expectation. These results provide stronger evidence that the model captures short-horizon audience dynamics rather than relying solely on channel-specific donation propensity.

The channel-level evaluation should be interpreted as a robustness check rather than as a claim of universal creator-level generalization. The analyzed data were still drawn from the same platform and historical period, and unseen channels may differ in audience culture, content genre, and monetization norms. Nevertheless, the zero-overlap channel grouping and the large gap between the strict HGB model and the channel-rate baseline strengthen the conclusion that the proposed framework retains predictive value beyond video-level memorization and channel-level propensity alone.

4.5. Feature-Group Ablation, Behavioral Interpretability, and Graph-Proxy Baseline

This section examines which behavioral components contribute to next-window Superchat prediction and whether the proposed leakage-guarded window-level feature set captures information beyond simple interaction-structure proxies. The analysis combines feature-group ablation, permutation importance, partial dependence plots, and a graph-proxy interaction baseline.

4.5.1. Feature-Group Ablation and Behavioral Interpretability

To examine how each behavioral component contributes to prediction, we conducted feature-group ablation using the feature structure defined in Section 3.3. Because the task is highly imbalanced, PR-AUC was used as the primary comparison metric, while ROC-AUC and F1 score were retained as complementary indicators of ranking and threshold-based performance. The ablation analysis evaluates whether prediction depends mainly on a single feature group or whether multiple behavioral dimensions jointly support the model.

Table 16. Feature-group ablation results for next-window Superchat prediction.

Setting	PR-AUC	ROC-AUC	F1	Interpretation
Audience activity only	0.827	0.862	0.758	Broad participation provides strong baseline signal
Activity + member composition	0.831	0.863	0.758	Member composition adds limited additional signal
Activity + composition + message intensity	0.837	0.867	0.766	Message intensity provides incremental improvement
Strict full model	0.896	0.918	0.817	Chat-derived engagement dynamics support strong prediction
Extended full model	0.909	0.926	0.825	Current-window donation-state variables add modest gain
Strict full without short-horizon dynamics	0.837	0.867	0.766	Dynamic features are important for short-horizon prediction

shows that audience activity alone provided a strong baseline signal, achieving a PR-AUC = 0.827. Adding member composition produced only a small improvement, while adding message-intensity variables increased the PR-AUC to 0.837. The strict full model achieved substantially stronger performance with PR-AUC = 0.896, indicating that the full set of leakage-guarded behavioral and past-dynamic features provides additional predictive value. When short-horizon dynamic features were removed from the strict full model, PR-AUC decreased to 0.837, which indicates that recent temporal trajectories are important for near-future Superchat prediction. The extended full model achieved a PR-AUC = 0.909, suggesting that current-window donation-state variables provide additional information, while the strict full model remained highly competitive without direct current-window donation-state features.

Figure 6 further clarifies which individual features drive prediction within the strict setting. The permutation-importance results show that n_unique_chatters was the most influential predictor, followed by past-shifted Superchat-related rolling information, n_chats, and message-length variables. This ranking is consistent with the ablation results in Table 16, where features associated with audience activity and short-horizon dynamics contributed most strongly to predictive performance, whereas member-composition features provided a smaller marginal contribution. The importance of n_unique_chatters is especially meaningful because it reflects participation breadth rather than only message volume. Windows in which many distinct viewers participate appear more predictive of near-future monetization than windows dominated by a small number of highly active users.

Superchat-related rolling variables should be interpreted carefully. In the strict setting, direct current-window donation-state variables are excluded, but rolling summaries computed from shifted historical windows can still summarize recent temporal context from preceding windows. Their importance therefore indicates that short-horizon dynamics matter, not that the model relies on direct focal-window donation labels. This distinction is central to the leakage-guarded interpretation of the strict setting.

To complement permutation importance, Figure 7 presents partial dependence plots for top behavioral predictors. The PDP results provide a model-level view of how predicted Superchat risk changes as each feature varies while averaging over the empirical distribution of other features. The plots show that predicted risk generally increases with participation breadth, especially for n_unique_chatters. Chat volume and aggregate message length also show gradually increasing relationships with predicted risk, suggesting that both audience scale and expressive intensity contribute to monetization likelihood. Rolling dynamic features capture recent momentum in audience activity and past Superchat-related behavior, reinforcing the importance of temporal context.

The PDP results should be interpreted as marginal model-response patterns rather than causal effects. Nevertheless, they provide useful interpretability evidence by showing that the strongest model responses are behaviorally plausible and consistent with the theoretical feature groups. Together, Table 16, Figure 6 and Figure 7 indicate that the model’s predictions are supported primarily by audience activity, participation breadth, message intensity, and short-horizon dynamics rather than by arbitrary or uninterpretable tabular signals.

4.5.2. Graph-Proxy Interaction Baseline

To provide an additional interaction-structure baseline, we constructed graph-proxy features from raw chat interactions. These features capture window-level interaction-structure patterns, including author diversity, repeat-author ratio, top-author share, author entropy, and related interaction-density characteristics. The purpose of this comparison is not to claim equivalence with a full dynamic-graph donation model, but to test whether lightweight interaction-structure proxies alone can account for next-window Superchat prediction under the same grouped evaluation protocol.

Table 17. Graph-proxy interaction baseline comparison.

Model	Features	PR-AUC	ROC-AUC	F1	Precision	Recall	Brier
Strict full HGB	27	0.899	0.920	0.822	0.856	0.790	0.171
Graph-proxy only HGB	6	0.832	0.864	0.759	0.819	0.707	0.216
Strict full + graph-proxy HGB	33	0.898	0.920	0.818	0.875	0.769	0.175

shows that graph-proxy interaction features alone provided meaningful predictive signal, achieving PR-AUC = 0.832 and ROC-AUC = 0.864. This indicates that viewer-interaction structure contains information relevant to next-window Superchat occurrence. However, the graph-proxy-only model remained below the strict full HGB model, which achieved a PR-AUC = 0.899 and ROC-AUC = 0.920. Moreover, adding graph-proxy features to the strict full feature set did not materially improve the PR-AUC. This suggests that the original leakage-guarded window-level feature set already captures much of the relevant short-horizon interaction structure through audience activity, participation breadth, message intensity, and past-dynamic variables.

Overall, the interpretability and graph-proxy analyses support the empirical interpretation of the proposed framework. The results suggest that predictive performance is driven by a combination of participation breadth, chat activity, message intensity, and temporally shifted behavioral dynamics. This interpretation is consistent with the leakage-guarded task formulation and supports the use of window-level live-chat dynamics for short-horizon Superchat prediction.

4.6. Calibration and Feature-Space Diagnostics

This section evaluates two diagnostic aspects of the proposed framework, namely probability calibration and feature-space structure. The calibration analysis examines whether the probability estimates produced by the strict HGB model are aligned with observed next-window Superchat frequencies, rather than functioning only as relative ranking scores. The feature-space diagnostics provide auxiliary evidence on whether the engineered strict feature representation exhibits clearer class-structure patterns than a basic activity-only representation. Calibration is important because the proposed framework may be used in monitoring-oriented settings where model outputs are interpreted as short-horizon risk indicators. In imbalanced event prediction, discrimination metrics such as PR-AUC and ROC-AUC are useful for assessing whether positive windows are ranked above negative windows. However, these metrics do not directly evaluate the agreement between the predicted probabilities and empirical event frequencies. Therefore, we evaluated raw HGB probabilities and post hoc calibrated probabilities using Brier score, expected calibration error (ECE), and reliability diagrams [32,37,38,47]. The sigmoid and isotonic calibration models were fitted on validation predictions and evaluated once on the held-out test set [37,38,47].

As shown in

Table 18. Corrected probability-calibration metrics for the strict HGB model.

Calibration	PR-AUC	ROC-AUC	Brier	ECE (10 Bins)
Raw HGB	0.899	0.920	0.171	0.218
Sigmoid calibrated HGB	0.899	0.920	0.110	0.078
Isotonic calibrated HGB	0.891	0.919	0.098	0.015

, post hoc calibration substantially improved the probability reliability while preserving ranking performance. The raw HGB model achieved a PR-AUC = 0.899 and ROC-AUC = 0.920, but its Brier score and ECE were 0.171 and 0.218, respectively. Sigmoid calibration preserved PR-AUC and ROC-AUC while reducing the Brier score to 0.110 and ECE to 0.078. Isotonic calibration produced the strongest calibration performance, reducing the Brier score to 0.098 and ECE to 0.015, although PR-AUC slightly decreased to 0.891. These results indicate that calibration is beneficial when model scores are interpreted as probability-like risk estimates rather than only as ranking scores.

Figure 8 visualizes the calibration behavior of the raw and post hoc calibrated HGB probabilities. The diagonal reference line indicates perfect calibration, where predicted probabilities match the observed Superchat occurrence rates. The raw HGB curve was systematically below the diagonal across much of the probability range, indicating the underestimation of observed positive fractions in several bins. In contrast, the post hoc calibrated curves moved closer to the diagonal, with isotonic calibration showing the closest alignment in this evaluation. The confidence intervals provide bin-level uncertainty and help avoid overinterpreting sparsely populated probability regions.

In addition to calibration, we examined feature-space diagnostics to assess whether engineered strict features provide a richer representation than basic activity-only features. These visualizations are intended as auxiliary diagnostic evidence and do not replace the quantitative evaluation, but they help illustrate whether the engineered feature space contains class-structure patterns that are consistent with the observed predictive performance.

Figure 9 shows the t-SNE projection of the engineered strict feature space [48]. The visualization suggests that positive and negative next-window outcomes are not completely separable, which is expected in a noisy livestream prediction task. Nevertheless, the projection reveals regions in which positive windows become more concentrated, indicating that the engineered feature representation contains meaningful class-structure information beyond a purely random or prevalence-only pattern. This diagnostic pattern is consistent with the ablation and baseline results reported in the preceding sections.

Overall, the calibration and feature-space diagnostics complement the predictive-performance results. Calibration analysis shows that the model’s probability outputs require post hoc adjustment before being interpreted as probability-like risk estimates. Feature-space visualization provides additional diagnostic evidence that the engineered strict features encode behavioral structure relevant to next-window Superchat occurrence. Together, these analyses support the use of the proposed framework for ranking-oriented prediction and calibrated risk interpretation under a leakage-guarded evaluation design.

5. Discussion

The present study advances research on livestreaming monetization by formulating VTuber Superchat occurrence as a leakage-guarded, window-based next-event prediction problem. Prior studies have primarily examined gifting, donation, sponsorship, and digital patronage through survey-based constructs, retrospective behavioral analysis, or ecosystem-level monetization patterns [3,4,7,8,9,10,11,16,17,18,19,24,25]. In contrast, this study demonstrates that near-future Superchat occurrence can be predicted from window-level audience dynamics observed in naturally occurring live-chat logs. This shift from retrospective explanation to short-horizon event prediction is important because livestreaming monetization unfolds in temporally concentrated interactional contexts, where audience activity, message intensity, community composition, and recent engagement dynamics may change rapidly within a single broadcast. A central finding is that the strict WITHOUT_CURRENT_SC setting retained strong predictive performance even after direct current-window donation-state variables were excluded. If the model had depended mainly on donation continuation, removing current-window Superchat variables would have caused a substantial collapse in performance. Instead, the strict model remained competitive, whereas the extended WITH_CURRENT_SC setting produced only a modest additional gain. This result suggests that the proposed framework captures broader interactional precursors of monetization rather than merely extending an already visible donation trend. Methodologically, the strict–extended comparison strengthens the leakage-guarded design of the study by separating chat-derived engagement dynamics from direct current-window donation-state information. The robustness analyses further reinforce this interpretation. The repeated video-level grouped split results show that the overall performance pattern was preserved across alternative held-out video partitions. This is important because Superchat activity is unevenly distributed across livestream videos, and a single video-level split may reflect the specific difficulty or prevalence structure of the selected held-out videos. The zero-chat sensitivity analysis further shows that the active-chat formulation is appropriate for a monitoring-oriented prediction task while also confirming that the model remains informative when zero-chat windows are included. In addition, the comparison among grouped video-level, balanced video-level, chronological video-level, and random row-level split protocols clarifies the importance of leakage-aware evaluation. The random row-level split is useful as a diagnostic condition, but its complete train–test video overlap illustrates why non-grouped partitioning can produce contaminated evaluations in temporally adjacent livestream data. Together, these analyses support the claim that the reported results are not limited to a single favorable split or to a narrowly defined active-window subset. The channel-level robustness analysis provides an additional test of whether the model merely exploits creator- or channel-specific donation propensity. By recovering channel identifiers from the raw VTuber 1B parquet files, the study evaluated a stricter grouped setting in which held-out channels did not appear in training. The strict HGB model retained meaningful predictive performance under channel-level grouping and substantially outperformed a train-derived channel-rate baseline. This finding indicates that the model is not simply memorizing channel-level donation tendency. At the same time, channel-level generalization remains more demanding than video-level generalization because creator communities differ in audience size, chat culture, donation norms, and membership composition. Therefore, the channel-level results should be interpreted as robustness evidence rather than as a final claim of universal cross-creator generalization. The window-size sensitivity analysis addresses the temporal design of the framework. The main five-minute window specification was selected to balance operational responsiveness and event sparsity, but the sensitivity analysis shows that the framework is not dependent on a single arbitrary temporal resolution. Performance remained strong across one-, two-, five-, ten-, and fifteen-minute windows, with the two- and five-minute settings forming the strongest performance region. The one-minute setting provides finer temporal responsiveness but produces a larger number of sparse windows, whereas longer windows aggregate more interactional context at the cost of reduced short-horizon immediacy. The five-minute window therefore offers a practical compromise between prediction stability and temporal usefulness for dashboard-based monitoring and retrospective stream analysis. The feature-group ablation, permutation-importance, partial-dependence, and graph-proxy analyses clarify how the model uses behavioral information. Audience activity alone provided a strong baseline signal, showing that broad participation is a major precursor of near-future Superchat occurrence. However, the strict full model outperformed activity-only and basic behavioral baselines, indicating that multiple behavioral dimensions jointly support prediction. In particular, the contribution of temporally shifted dynamic features suggests that recent trajectories of audience behavior are important for distinguishing stable, declining, and intensifying interaction states. The prominence of participation breadth, especially the number of unique chatters, further suggests that monetization-prone windows may reflect moments when a wider segment of the audience becomes collectively engaged, rather than moments dominated only by a small number of highly active users. This interpretation is consistent with prior work emphasizing social interaction, parasocial responsiveness, community participation, and reciprocal communication as important factors in livestream gifting and digital patronage [3,4,12,13,14,15,16,17]. The graph-proxy interaction baseline provides a useful additional perspective. Lightweight interaction-structure proxies, including author diversity, repeat-author ratio, top-author share, author entropy, and related interaction-density characteristics, achieved meaningful performance on their own. However, they did not fully match the strict full feature set, and adding graph-proxy features to the strict full model did not materially improve the predictive performance. This result suggests that interaction-structure signals are relevant to Superchat prediction, but that the proposed window-level behavioral representation already captures much of the short-horizon information needed for the task. Importantly, the graph-proxy experiment should not be interpreted as a full dynamic-graph donation model. Rather, it serves as a lightweight baseline showing that the proposed feature set provides predictive value beyond simple interaction-structure summaries. The calibration analysis adds another layer to the interpretation of the model outputs. In imbalanced event prediction, strong PR-AUC and ROC-AUC indicate that the model ranks higher-risk windows above lower-risk windows, but probability-based monitoring requires additional evidence regarding the agreement between the predicted probabilities and observed event frequencies [30,31,32,33,49]. The post hoc calibration results show that sigmoid and isotonic calibration improved the Brier score and expected calibration error while maintaining high discrimination performance. This distinction is practically meaningful. The raw HGB model is useful for ranking-oriented screening, whereas calibrated outputs are more appropriate when scores are presented as probability-like short-horizon risk indicators in a monitoring interface. Therefore, the proposed framework should be understood as supporting both discrimination-oriented event detection and calibration-aware risk interpretation. From an applied perspective, the proposed framework contributes to cognitive-computing-oriented livestream analytics by translating high-frequency behavioral logs into short-horizon monetization risk estimates. Such estimates may support retrospective stream analysis, presentation-oriented monitoring, and model explanation.

However, any replay-oriented visualization or dashboard prototype should be interpreted as an exploratory interface for making window-level prediction dynamics more legible, rather than as a validated operational decision-support system. User-centered validation remains necessary before making claims about practical decision impact for streamers, moderators, analysts, or platform operators. Several limitations should be noted. First, although the study includes video-level and channel-level grouped evaluations, the dataset remains limited to the available and successfully processed subset of VTuber 1B-derived monthly files. Broader coverage across time periods, stream types, creator sizes, and platform conditions would improve representativeness. Second, the channel-level analysis provides robustness evidence, but broader creator-level generalization should be examined with larger numbers of channels and more balanced creator-level sampling. Third, the graph-proxy experiment was designed as a lightweight interaction-structure baseline, not as a full dynamic network model of viewer–streamer or viewer–viewer interactions. Future work could construct temporal interaction graphs, dynamic community features, or heterogeneous graph representations to examine whether richer relational modeling improves prediction. Fourth, although post hoc calibration improved the probability reliability, calibrated risk presentation should be further evaluated in real monitoring interfaces. Fifth, the current analysis focuses on behavioral and timing features derived from chat logs. Future research could incorporate multimodal stream context, semantic content, streamer actions, visual events, or audio cues to better contextualize donation-prone livestream moments beyond behavioral timing signals alone.

Overall, the findings suggest that next-window Superchat prediction is feasible when the task is formulated with leakage-aware validation, interpretable feature groups, imbalance-sensitive metrics, robustness checks, channel-level diagnostics, and calibration analysis. The study therefore contributes not only a predictive model, but also a methodological template for short-horizon monetization prediction in virtual livestreaming environments.

6. Conclusions

This study developed a leakage-guarded, window-based machine-learning framework for predicting next-window Superchat occurrence from VTuber live-chat dynamics. By formulating each livestream as a sequence of non-overlapping temporal windows, the study moved beyond retrospective descriptions of monetization behavior and addressed Superchat occurrence as a short-horizon event-prediction problem. The proposed framework organized audience activity, member composition, message intensity, donation-state information, and short-horizon behavioral dynamics into an interpretable feature-space design for modeling monetization-related audience behavior. The empirical results show that next-window Superchat occurrence can be predicted with strong performance under leakage-aware grouped evaluation. The strict WITHOUT_CURRENT_SC setting retained substantial predictive performance even after direct current-window donation-state variables were excluded, while the extended WITH_CURRENT_SC setting provided only a modest additional gain. This finding indicates that predictive performance is not driven solely by direct donation momentum, but also by broader chat-derived engagement signals such as participation breadth, message intensity, and recent audience dynamics. The repeated video-level grouped split analysis, zero-chat sensitivity analysis, split-protocol diagnostics, and channel-level robustness analysis further support the stability and methodological reliability of the proposed framework. The additional analyses strengthen the empirical interpretation of the model. Window-size sensitivity analysis showed that the five-minute specification provides a practical balance between temporal responsiveness, event sparsity, and predictive stability. Feature-group ablation, permutation importance, partial dependence analysis, and graph-proxy comparison showed that audience activity, participation breadth, message intensity, interaction-structure signals, and temporally shifted behavioral dynamics jointly contribute to next-window prediction. The graph-proxy baseline further suggests that lightweight interaction-structure features are informative, but do not fully account for the predictive value of the proposed leakage-guarded window-level representation. The calibration analysis demonstrated that post hoc sigmoid and isotonic calibration improved probability-level reliability, supporting the use of calibrated outputs when model scores are interpreted as short-horizon risk indicators. This distinction is important because discrimination-oriented metrics such as PR-AUC and ROC-AUC indicate whether higher-risk windows are ranked above lower-risk windows, whereas calibrated outputs are more appropriate for probability-like monitoring and risk interpretation. Therefore, the proposed framework not only contributes a predictive model, but also a calibration-aware and interpretation-oriented methodological template for short-horizon monetization prediction in virtual livestreaming environments. Several directions remain for future research. First, although the study includes video-level and channel-level grouped evaluations, broader creator-level validation should be conducted using larger and more balanced samples of channels, stream types, languages, and time periods. Second, future work could examine richer temporal and relational models, including sequence models, dynamic interaction graphs, and heterogeneous graph representations. Third, future work could incorporate multimodal stream context, semantic chat content, streamer actions, visual events, and audio cues to better contextualize donation-prone livestream moments beyond behavioral timing signals alone. Fourth, calibrated risk presentation and replay-oriented visualization should be evaluated through user-centered studies with streamers, moderators, analysts, or platform operators.

Collectively, the findings suggest that next-window Superchat occurrence can be predicted from VTuber live-chat dynamics when the task is formulated with leakage-aware validation, interpretable feature groups, imbalance-sensitive metrics, robustness checks, channel-level diagnostics, and calibration analysis. Therefore, this study not only provides a predictive model, but also a methodological framework for short-horizon monetization prediction in virtual livestreaming environments.