Source-Specific Lexical Salience in Korean Online Information About Artificial Intelligence in Sports: Cluster-Aware Inference and Expert Validation

Abstract

The co-occurrence of sports and artificial intelligence in online search results does not necessarily indicate meaningful relevance of AI applications in sports. This study examined source-specific lexical salience in Korean online information and assessed how source dependence and semantic screening uncertainty affect interpretation. From 10,878 news, blog, and online community records retrieved from Naver sources through TEXTOM, 9567 remained after deduplication, and 3161 met the operational relevance criteria. We applied document-prevalence-based keyness analysis, cluster-aware generalized estimating equations, Benjamini–Yekutieli correction, and sensitivity analyses. Two external sports-domain experts independently evaluated 360 documents and 333 lexical contexts. Organizational terms were generally more prevalent in news, whereas use and prediction were more prevalent in both blogs and online communities; several review- and officiating-related contrasts were more specific to blogs. Of 185 tested lexical groups, 51 showed a BY-adjusted overall source effect in the primary three-source GEE analysis. Expert agreement was 85.28% (Cohen’s κ = 0.696). With 54 unresolved cases retained as uncertain, conditional weighted precision and recall ranged from 53.41–70.79% and 54.22–76.48%, respectively. Major contextual contrasts were largely preserved in expert-defined relevant subsets, although the near-equal prevalence of technology was not. These findings identify candidate source-specific lexical contrasts within the retrieved corpus and show that statistical stability and semantic validity should be evaluated separately in search-based text analysis.

1. Introduction

Extracting technology-related knowledge from online text involves more than collecting documents and counting word frequencies. In automated text analysis, decisions about preprocessing and the unit of analysis can shape both the results and their interpretation, while the reliability and semantic validity of extracted patterns must also be established if those patterns are to represent the concepts researchers intend to study [1,2,3]. It is therefore necessary to consider whether retrieved documents are substantively relevant to the research topic, whether detected expressions actually represent the intended concepts, and whether repeated documents from the same publishing source are being treated as independent observations. Previous research on automated content analysis and text preprocessing has shown that transparent and reproducible computational procedures do not, by themselves, guarantee valid substantive interpretation [4,5].

Artificial intelligence in sports provides a particularly suitable setting in which to examine these issues. AI is now used across a wide range of sports-related tasks, including injury-risk assessment, performance prediction, player-action analysis, tactical decision support, and health management, each involving different data structures and analytical objectives [6,7,8]. As a result, apparently similar terms such as prediction, analysis, and support may refer to quite different technical functions and use contexts. Understanding sports AI therefore requires attention not only to the performance of particular technologies, but also to the activities, problems, and application settings through which those technologies are represented in online information. Rather than reassessing the technical performance of individual AI applications, this study examines how sports AI is contextually represented in searchable online texts.

Text-based research on sports AI is not entirely new. Bibliometric studies have examined the conceptual structure and research trends of artificial intelligence, machine learning, and deep learning in sports, while Korean research has used news topic modeling and social network analysis to explore industrial and technological trends in sports AI [9,10]. These studies provide an important foundation for understanding the major topics and conceptual structure of the field. What remains less clear, however, is how news, blogs, and online community posts retrieved through comparable search terms differ in the contexts through which sports AI is described, and whether those differences remain stable when document selection, deduplication, source composition, and inferential assumptions are changed. Distinguishing genuine source-related differences from the effects of semantically irrelevant retrieved documents is particularly important when comparing organizational and industry-oriented descriptions in news with use- and application-oriented descriptions in blogs and online communities.

This problem also relates to a broader concern in computational linguistics and social media research: the validity of interpretation. Heaton et al. [11] compared the outputs of topic modeling, sentiment analysis, and emotion detection with human judgments and contextual evidence, showing that computational outputs may not always align with the meanings researchers attribute to them. Stella [12] evaluated conceptual relationships extracted from text against an independent semantic benchmark, while Gurcan [13] incorporated domain-expert review into the evaluation of topics derived from data-science Q&A posts. Taken together, these studies suggest that applying a computational method and assigning substantive meaning to its output are distinct analytical tasks. Contextual review and human assessment are therefore important components of validation.

The methodological gap addressed in this study arises from the tendency to conflate three different forms of uncertainty. First, document prevalence can reduce the influence of repeated mentions within a single document, but it does not remove dependence among documents produced by the same publishing source. Second, cluster-aware statistical inference can account for such dependence, but it cannot establish whether the analyzed documents meaningfully connect sports and AI. Third, high agreement between expert raters does not necessarily imply that a document-screening rule has high precision or recall. In other words, lexical distribution, statistical dependence, and semantic relevance are related but analytically distinct concerns [14,15,16].

To address these issues, the study integrates three levels of evaluation within a single analytical framework. First, source-specific lexical salience is examined using document-prevalence-based keyness, thereby reducing the influence of repeated mentions within individual documents. Second, cluster-aware inference and multiple-testing correction are used to assess whether observed lexical contrasts remain statistically stable when dependence among documents from the same publishing source is taken into account. Third, two sports-domain experts independently evaluate documents from both the selected and excluded sets, while the actual contexts of several potentially ambiguous lexical items are examined separately. These procedures are designed to address different sources of uncertainty and to clarify the extent to which the final interpretations are empirically supported.

The purpose of this study is therefore to identify source-specific lexical contrasts in Korean online texts related to sports AI and to evaluate how consistently their direction, effect size, statistical support, and substantive interpretation are maintained under different validation conditions. Accordingly, the following research questions are addressed:

RQ1. 

Which lexical items show source-specific prevalence differences across news, blogs, and online communities in rule-selected sports AI search texts?

RQ2. 

To what extent do the direction and statistical support of source-specific lexical contrasts remain stable when blogs and online communities are pooled and when source clustering, deduplication, screening stringency, and the scope of analyzed text are varied?

RQ3. 

When independent expert judgments and unresolved ratings are taken into account, within what range does the document-screening rule perform, and to what extent are the main lexical contrasts and their substantive interpretations preserved in expert-defined relevant subsets?

This study therefore compares source-specific lexical salience in sports AI information while jointly assessing the statistical stability and semantic validity of the observed contrasts. News, blogs, and online communities are analyzed within a common lexical space, and the robustness of source differences is examined through document prevalence, cluster-aware inference, and sensitivity analysis. Independent expert evaluation of both selected and excluded documents, together with contextual validation of key lexical items, is then used to incorporate uncertainty in search-based screening and interpretation. In this way, the study moves beyond simple frequency rankings to provide a more explicit basis for determining how far lexical differences observed in searchable online texts can be considered reliable and substantively interpretable.

2. Background and Related Work

2.1. Task Diversity in Sports AI and the Unit of Interpretation

Artificial intelligence in sports is difficult to define as a single technology or analytical task. In practice, it encompasses a wide range of applications, including injury-risk estimation, match outcome prediction, player-action evaluation, performance analysis, tactical decision support, and health management, each involving different objectives and data structures. Claudino et al. [6] systematically reviewed the use of AI for injury-risk assessment and performance prediction in team sports, while Hammes et al. [7] and Mateus et al. [8] showed that AI applications in elite sport now extend across training, performance, and health management. Sports AI is therefore better understood not as a homogeneous application domain, but as a broader technological field in which multiple data sources, analytical objectives, and decision environments intersect.

This heterogeneity is also evident in individual empirical studies. Rossi et al. [17] investigated injury forecasting using GPS-based training data from soccer players, whereas Van Eetvelde et al. [18] highlighted substantial variation in data design, methodological quality, and performance evaluation across machine-learning studies of sports injury prediction. Bunker and Susnjak [19], in their review of match outcome prediction in team sports, likewise showed that prediction problems and evaluation criteria depend on the sport and the structure of the available data. Other studies illustrate even more specific applications. TacticAI, for example, links prediction with tactical recommendations in the highly constrained context of football corner kicks [20], while Decroos et al. [21] proposed an approach for estimating the value of individual player actions within the broader context of a match rather than focusing only on goals. Together, these studies demonstrate that apparently similar terms such as AI or prediction may refer to substantially different analytical targets and decision functions.

Interpreting sports AI also requires attention to the data, environmental, and organizational conditions within which algorithms are developed and used. Araújo et al. [22] emphasized the relationship between athlete behavior and environmental conditions in AI-based performance analysis, while Rein and Memmert [23] discussed the importance of data management and organizational infrastructure in the use of big data and tactical analysis in elite football. Public event datasets, such as the soccer match data introduced by Pappalardo et al. [24], provide infrastructure for comparison and reuse, whereas research on multi-agent reinforcement learning in StarCraft II illustrates a different setting in which esports serves as an experimental domain for AI research [25]. These examples further show that the meaning of sports AI depends not only on the algorithm itself, but also on the data environment and the context in which the technology is applied.

Beyond these sports-specific applications, recent AI research also demonstrates continued methodological advances toward more adaptive, transferable, and data-efficient intelligent modeling. Meta-learning-based graph convolutional wavelet networks have been developed to integrate multi-scale feature representation with adaptive learning, while adaptive fused domain-cycling variational generative adversarial networks address data scarcity through synthetic-data generation and multi-domain fusion. Dynamic collaborative adversarial domain adaptation networks further illustrate advances in learning across heterogeneous domains without labeled target data [26,27,28]. Although these methods were developed primarily in engineering contexts rather than sports, they illustrate the broader evolution of AI toward adaptive, cross-domain, and data-efficient modeling.

The purpose of reviewing these studies is not to compare the superiority of individual AI models. Rather, they provide a basis for recognizing that a single search category such as sports AI can contain heterogeneous tasks, datasets, actors, and application settings. Consequently, expressions such as prediction, analysis, support, and use in online texts should not be treated as if they referred to a single predefined function. Their meaning instead depends on the sports task and action context in which they occur. From this perspective, the unit of interpretation in this study is not the AI technology itself, but the way sports AI is connected to particular activities and application contexts in searchable online information.

2.2. Source-Mediated Information Environments and Lexical Salience

Technology-related information observed online does not simply reflect the inherent characteristics of a technology; it is also shaped by the environments in which that information is produced, distributed, and made visible. The concept of social media logic proposed by van Dijck and Poell [29] emphasizes that the production, circulation, and visibility of online information are conditioned by technological infrastructures as well as organizational and economic arrangements. Applied to the present study, news, blogs, and online communities are therefore better understood not as categories representing fixed identities or attitudes of their authors, but as distinct information environments shaped by posting practices, document genres, communicative purposes, and patterns of search visibility.

These source environments may emphasize different aspects of the same sports AI technology. News reports, for example, may foreground organizational activities such as business initiatives, technology adoption, partnership agreements, and policy support, whereas blogs and online communities may more often feature concrete application contexts such as use, reviews, match prediction, product-related applications, or officiating. Such differences, however, should not be interpreted directly as differences between institutional and user attitudes. The present data do not independently identify authors’ occupations, organizational affiliations, consumer status, commercial intent, or actual experience with the technologies being discussed.

Accordingly, this study does not interpret source differences as causal effects whereby a platform or source type produces particular forms of language. Instead, it examines whether particular lexical items are distributed more widely across documents from one source than across those from another within the same search domain. The emphasis is therefore on describing source-conditioned patterns of representation rather than estimating the causal effect of a platform on language use.

Entman’s [30] discussion of salience provides a useful supplementary perspective for interpreting these patterns. A central element of framing theory is that selected aspects of reality may become more prominent when they are given greater emphasis. The present study, however, does not directly code the combination of problem definition, causal attribution, moral evaluation, and treatment recommendation described by Entman. Accordingly, lexical salience is used here in a narrower statistical sense: it refers to the relative tendency for a particular expression to occur across a larger proportion of documents in one source than in a comparison source. It should not be treated as equivalent to a fully specified media frame.

This distinction is important for avoiding overinterpretation. For example, even if terms such as industry, agreement, and support are more salient in news, this alone does not establish that news sources intentionally frame sports AI in industrial terms. Likewise, a greater prevalence of use, reviews, or prediction in blogs and online communities does not demonstrate that these texts represent genuine consumer experience or user attitudes. In this study, labels such as organizational-activity vocabulary and use- and application-related vocabulary are therefore descriptive summaries of observed lexical patterns, rather than psychological constructs or direct measures of authors’ motives, perceptions, or collective attitudes.

2.3. Interpretability and Validation in Online Text Analysis

In computational research on large-scale online text, analytical performance and the validity of substantive interpretation are not necessarily the same problem. Some studies focus primarily on improving classification accuracy or predictive performance, whereas others place greater emphasis on whether computational outputs retain the meanings that researchers attribute to them in context. Zhu et al. [31], for example, proposed a sentiment-analysis model that integrates LDA, RealNVP, BiLSTM, and multi-head self-attention, and evaluated its performance against alternative models using measures including accuracy and F1 score. The central concern in such work is how to construct a more accurate predictive model. By contrast, Heaton et al. [11] applied widely used computational linguistic methods, including topic modeling, sentiment analysis, and emotion detection, to social media data and then examined the resulting outputs through human review and contextual inspection. Their analysis showed that automated classifications may diverge from human interpretation because of linguistic features such as negation, sarcasm, and contextual variation, highlighting the need to examine computational outputs alongside the contexts in which they occur.

The present study is closer to the latter perspective. Its aim is not to propose a new machine-learning classifier for sports AI documents or to outperform existing models in predictive accuracy. Instead, it asks whether documents retained by a rule-based screening procedure meaningfully connect sports and AI, and whether detected lexical expressions are used in the sense intended by the analytical dictionary. Accordingly, document relevance and contextual validity of lexical detection are treated as separate validation problems. Stella’s [12] work on textual forma mentis networks provides another useful methodological reference because it linked computationally extracted conceptual structures to an independent benchmark. The study evaluated whether concepts identified from text were semantically related to the underlying topics by comparing them with external behavioral and semantic data. The present study does not adopt Stella’s cognitive-network framework: it does not construct syntactic, synonym, sentiment, or emotion networks, nor does it infer users’ mental lexica or psychological attitudes from online texts. The relevant principle is instead that computationally extracted patterns should be evaluated separately for their substantive semantic adequacy. Gurcan [13] is also relevant because domain-expert validation was incorporated into the analysis of topics derived from data-science Q&A posts. Two domain experts reviewed sampled posts assigned to the identified topics, linking computational analysis to substantive assessment. The data-generating setting, however, differs from that of the present study. Gurcan examined posts from a single specialized Q&A platform, whereas the current corpus consists of search-retrieved documents drawn from heterogeneous information environments, including news, blogs, and online communities. Expert validation in the present study is therefore used not to refine topic labels, but to quantify the possibility of semantic misclassification introduced by search retrieval and rule-based screening.

2.4. Document-Prevalence-Based Keyness and Independent Evaluation

Keyness analysis identifies expressions that are distinctive relative to a comparison corpus, but what counts as distinctive depends on both the reference corpus and the statistic used. Frequency profiling and log-likelihood provide a basis for detecting distributional differences, while log ratio is useful for interpreting the relative magnitude of those differences [32,33]. Statistical significance and effect size, however, answer different questions, making the choice of metric dependent on the analytical purpose [34,35].

Total token frequency can be strongly influenced by repeated use of the same expression within a small number of documents. Research on text dispersion and sampling units therefore cautions against treating concentrated repetition in a few texts as a general property of an entire corpus [14,15]. More recent work on keyness similarly argues that frequency, association, and dispersion capture related but distinct aspects of lexical distinctiveness [36,37,38]. In the present study, the primary measure is document prevalence, defined as the proportion of documents in which a lexical item occurs at least once. This choice reduces the influence of within-document repetition, but it is not presented as a substitute for, or implementation of, all existing dispersion measures.

Reproducibility of dictionary rules must also be distinguished from semantic validity. A researcher may be able to reproduce exactly which strings were detected without establishing that those strings express the intended concept in context [4,11]. The present study therefore evaluates two levels separately. At the document level, the question is whether a retrieved text meaningfully concerns sports AI. At the lexical level, the question is whether a detected expression can substantively support interpretations such as actual use, sports analysis, or the role of human judgment. These are related validation tasks, but one cannot substitute for the other.

Disagreement between independent raters may arise not only from coding error but also from limited information or differences in interpretation. Pavlick and Kwiatkowski [39], in their study of natural-language inference, showed that some human disagreements persist even after additional judgments and contextual information are provided. This does not imply that all disagreements in the present study are inherently ambiguous. It does, however, provide a methodological reason not to force every unresolved case into a single definitive label. Accordingly, inter-rater agreement is distinguished from consensus labeling, and cases without an agreed reference judgment are retained as uncertainties and incorporated into conditional performance ranges rather than arbitrarily assigned to one class [16].

3. Methods

The analysis proceeded in five stages: corpus construction and screening, document-level lexical detection, estimation of source-specific effects, sensitivity analyses addressing clustering and analytical choices, and independent expert validation. The document was the unit of observation in the primary analysis, whereas the document–term pair was the unit of observation in contextual validation. The primary inferential analysis treated source as a three-level factor comprising news, blogs, and online communities. The pooled news–online comparison was retained as a secondary descriptive and sensitivity analysis. Hereafter, online refers to the combined set of blogs and online communities, excluding news.

3.1. Study Design and Data Scope

This study used an exploratory comparative-corpus design based on Korean online search texts retrieved through TEXTOM, a commercial web-based data collection platform. Data collection was conducted on 3 September 2026 through four separate collection runs defined by two search expressions and two Naver source settings. The search expressions recorded in TEXTOM were “스포츠 + AI” [sports + AI] and “스포츠 + 인공지능” [sports + artificial intelligence]. Each expression was collected separately from Naver News and from Naver Blogs/Cafes, resulting in four source-query combinations. The downloaded records were subsequently analyzed by source as news, blogs, and online communities. All four collection runs covered the period from 1 January 2024 to 31 August 2026 and used annual collection units within the specified coverage period. The collection scope included title, body-text, and URL fields. No morphological analyzer, part-of-speech restriction, user dictionary, or spam-filtering option was applied during the TEXTOM collection stage. TEXTOM’s built-in duplicate-removal option based on title and contents was enabled for all four collection runs. Additional post-download deduplication and standardization procedures applied by the researchers are described separately in Section 3.2.

The data were obtained through the TEXTOM web interface and downloaded from the platform rather than collected through a researcher-operated Naver API. Accordingly, the researchers specified the query expressions, source channels, coverage period, temporal collection unit, and downloadable fields through the TEXTOM interface. Platform-side procedures, including the underlying Naver search endpoint, ranking algorithm, pagination mechanism, and retrieval limits, were controlled by TEXTOM and were not directly exposed to the researchers. The “+” symbol in the two search expressions is reported exactly as recorded in the original TEXTOM collection settings; its underlying implementation by the platform was not independently verified. The specified coverage periods should therefore be understood as collection settings rather than independently verified publication dates for every retrieved document. For subsequent analysis, the downloaded title and body-text fields of each record were concatenated. Because the retrieved body-text fields could represent search-visible excerpts rather than complete original webpages, they were not treated as full-text reproductions of the original news articles or posts. In the full deduplicated corpus, median body-text length was 162 characters for news, 193 for blogs, and 215 for online community posts, and 20 online community records had empty body fields. These characteristics further indicate that the available text should be interpreted as the content retrieved through the collection platform rather than as a complete archive of the original webpages.

The supplied frequency tables were not used directly for lexical counts because they contained interface-related strings, including expressions equivalent to “open in a new window.” Instead, all lexical occurrences were recounted from the downloaded source-text fields. Preprocessing included Unicode NFKC normalization, HTML entity decoding, whitespace normalization, and removal of interface-related text. Accordingly, the estimand of the present study is lexical prevalence within the TEXTOM-retrieved Naver corpus under the documented collection conditions, rather than prevalence in a probability sample or exhaustive representation of Korean online information. The documented TEXTOM retrieval settings for the four collection runs are summarized in Table A1.

3.2. Deduplication and Relevance Screening

URLs were normalized with respect to host capitalization, the www prefix, tracking parameters, and trailing delimiters. For records sharing the same normalized URL, the row with the longest concatenated title-and-body text was retained; ties were resolved by preserving the original record order. Exact-text duplicates were then identified after removing whitespace and standardizing case. Search-term provenance and original row identifiers were retained internally. This procedure resulted in 9567 deduplicated documents for subsequent analysis. The relevance rule used two sets of markers. Sports-related markers included terms referring to sports, physical activity, specific sports, performance, and coaching, whereas AI-related markers included terms referring to AI, artificial intelligence, machine learning, deep learning, and generative AI. Expressions likely to generate false matches—such as sports-media brand names, automobile model names, and AI-related recommendation tags—were masked before applying explicit exclusion rules for clearly non-sport contexts.

For the primary analysis, a document was retained when both a sports marker and an AI marker were present and either appeared together in the title or occurred within a minimum distance of 20 characters in the body text. The 20-character threshold refers to distance in the normalized character string, not to a number of words or a semantic-distance measure. This criterion was treated as an operational screening rule rather than as a definitive indicator of semantic relevance. The screening logic, marker families, masking rules, and exclusion logic described in this section were fixed before inferential testing.

Alternative screening conditions included a strict title-based rule requiring both marker types in the title, a broader rule requiring both marker types without the distance constraint, and analysis of the complete deduplicated corpus. Near-duplicate documents were additionally assessed using character 3–5-g TF–IDF vectors (min_df = 2; maximum 50,000 features). Within each source type, documents with cosine similarity ≥0.90 were connected, and the longest document in each connected component was retained. This procedure does not imply that every pair of documents within a connected component exceeded the similarity threshold. Cross-source near-duplicate pairs were not removed.

3.3. Lexical Dictionary and Document Prevalence

A regular-expression dictionary containing 189 conceptual lexical groups was constructed by the researchers on the basis of prior sports AI literature and inspection of the study data. The dictionary is not presented as the output of morphological analysis or as a previously validated measurement scale. For each lexical group, the working dictionary documented the included surface forms, regular expression, conceptual category, literature linkage, whether the item was data-informed, and its validation status. No contemporaneous record exists showing that each individual rule was preregistered or originally derived from a specific literature source. Because contemporaneous provenance was not available for every individual item, literature links documented retrospectively were used only to provide conceptual context and were not treated as evidence of prior specification. All dictionary rules were considered part of an exploratory lexical resource.

Matching was case-insensitive. Where matches overlapped, priority was given first to the earliest starting position, then to the longer expression when positions were identical, and finally to dictionary order when both remained tied. Lexical presence was coded as 1 when a lexical group occurred at least once in a document and 0 otherwise. Of the 189 lexical groups, 185 occurred in at least 10 documents in the primary-analysis corpus and were included in inferential testing. Four lexical groups—those corresponding to fatigue, heart rate, bias, and privacy—did not meet this threshold. They were retained in the dictionary and document–term matrix but excluded from inferential testing. The same set of 185 lexical groups was used for effect estimation and sensitivity comparisons.

The binary document–term matrix retained both detected occurrences and zero values. The 185 tested groups therefore remained identifiable within the full 189-column working matrix rather than being restricted to statistically significant terms. Because the dictionary was developed after inspection of the study data, it was not treated as a confirmatory predictive model or as a fixed lexical measurement instrument validated on an independent holdout sample. The 189-group dictionary was treated as an exploratory, data-informed lexical resource. Because its categories and regular-expression rules were developed and refined after inspection of the same corpus subsequently used for inference, the inferential results are conditional on the final dictionary specification and should not be interpreted as confirmatory tests of a prespecified lexical measurement instrument. The complete dictionary, including surface forms, regular expressions, conceptual categories, literature links, data-informed status, document frequency, test inclusion, and contextual-validation status, is available from the corresponding author upon reasonable request.

3.4. Effect Sizes and Cluster-Aware Inference

For each lexical group, document prevalence was calculated separately for news, blogs, and online communities as the proportion of documents containing the lexical group at least once. Source-specific percentage-point differences were used to describe absolute contrasts among the three source types. For continuity with the initial analysis, the pooled news–online comparison was retained as a secondary descriptive analysis, with LR_D defined as the base-2 logarithm of the continuity-corrected news-to-online prevalence ratio.

$${\mathit{LR}}_D(t)={\mathit{log}}_2[{\displaystyle \frac{\frac{a_t+0.5}{N_N+1}}{\frac{b_t+0.5}{N_U+1}}}]$$

$${\Delta }_t=100\times ({\displaystyle \frac{a_t}{N_N}}-{\displaystyle \frac{b_t}{N_U}})$$

The 0.5 continuity correction was applied to each cell of the corresponding 2 × 2 table only for calculation of LR_D. No continuity correction or pseudo-observation was used for raw prevalence, percentage-point differences, Fisher’s exact test, GEE estimation, or cluster bootstrap resampling. For the secondary pooled news–online comparison, two-sided Fisher’s exact tests were retained as conventional comparisons under document-level independence. Benjamini–Yekutieli (BY) adjustment was used to control false discovery under dependence among lexical tests [40]. BY adjustment addresses multiplicity across lexical tests but does not account for clustering among documents. For the primary analysis, a logistic generalized estimating equation (GEE) was fitted separately for each lexical group, with document-level lexical presence \(Y_{it}\) as the dependent variable and source as a three-level categorical predictor (news, blogs, and online communities), with news as the reference category [41]. An independence working-correlation structure and cluster-robust sandwich variance estimator were used. A two-degree-of-freedom Wald test jointly evaluated the blog and online-community coefficients, providing the primary omnibus test of source heterogeneity.

logit{P(Y_it = 1)} = β_0t + β_1tSource_i

Cluster proxies were defined as the URL domain for news, the blogId or account path for blogs, and the clubid or community path for online communities. When no such identifier could be extracted, the record identifier was used. The two-degree-of-freedom omnibus Wald test used a chi-square approximation, whereas single planned contrasts used the normal approximation; no small-sample variance correction was applied. The 185 omnibus Wald test p-values were adjusted using the BY procedure, with q < 0.05 used as the primary reporting criterion. Planned contrasts compared news with blogs, news with online communities, and blogs with online communities. To control multiplicity across the source-specific contrasts, the 555 contrast p-values (185 lexical groups × three contrasts) were jointly BY-adjusted. Pairwise contrasts were interpreted primarily for lexical groups meeting the omnibus GEE–BY criterion. Odds ratios from the logistic models were distinguished from prevalence ratios. For the secondary pooled news–online comparison, a corresponding binary-source logistic GEE was also fitted separately for each lexical group using the same cluster definitions and cluster-robust sandwich variance estimator. The resulting 185 p-values were BY-adjusted as a separate secondary test family.

For the three-source model, coefficients were estimated for blogs and online communities relative to news, and the omnibus Wald statistic jointly tested both source coefficients. Cases in which boundary prevalence prevented estimation of finite logistic coefficients were retained as non-estimable rather than assigned finite effect estimates. For multiplicity adjustment, non-estimable tests were assigned an adjustment input of 1 solely to preserve the complete test family; this did not imply that a finite Wald p-value of 1 had been estimated. When observed prevalence in both groups fell strictly between 0 and 1, the source coefficient was obtained as the difference between the two group logits, and sandwich variance was calculated from cluster-level sums of residual scores. When prevalence in either group was exactly 0 or 1, no finite logit coefficient existed. Such cases were explicitly marked as non-estimable. A value of 1 was supplied only as the multiplicity-adjustment input needed to preserve the full test family; this does not mean that a finite Wald p-value of 1 had been estimated. The corresponding coefficient and odds ratio remained unestimated.

For the secondary pooled news–online comparison, uncertainty in absolute prevalence differences was assessed using 2000 cluster-bootstrap replications (seed = 20,260,912). Clusters were sampled with replacement and equal probability separately within the news and online groups, with all documents from a selected cluster retained. Prevalence differences were recalculated in each replication, and the 2.5th and 97.5th percentiles were reported. These intervals are lexical-item-specific bootstrap intervals rather than simultaneous multiple-comparison intervals. Because the search corpus is not a probability sample of the internet, they should not be interpreted as design-based confidence intervals for an internet-wide population. Additional cluster diagnostics were calculated for the principal reported lexical groups, including the number of event-containing clusters. For the sparse terms review, referee, and officiating decision, news–blog and news–online-community prevalence differences were additionally evaluated using 2000 cluster-bootstrap replications as robustness checks (Table A2 and Table A3).

3.5. Independent Expert Evaluation and Weighted Performance Ranges

Two external sports-domain experts, neither of whom was an author of this study, independently completed the evaluation forms. Both held doctoral degrees in physical education and had research interests related to artificial intelligence applications in sports. One evaluator was a university professor, and the other was a doctoral-level researcher. The validation sample was drawn from six strata defined by crossing the three source types with screening status (selected vs. excluded). Sixty documents were sampled without replacement from each stratum (seed = 20,260,913), producing a total of 360 documents. Including both selected and excluded strata made it possible to evaluate not only false inclusion but also potentially relevant documents missed by the screening rule. The two external evaluators completed their assessments independently, and their original ratings were retained without post hoc reconciliation or forced consensus. Disagreements and uncertain judgments were preserved as unresolved cases for the subsequent range analysis. The evaluators were not shown source type, screening status, the other evaluator’s ratings, or the study’s statistical results. They received the same written coding instructions before assessment; contextual validation additionally provided the target lexical item and its surrounding textual context.

For stratum h, with population size N_h and sample size n_h = 60, each sampled document received a weight of N_h/n_h. Based on the available title, body text, and contextual information, each expert independently classified whether sports and AI were meaningfully connected as relevant (1), irrelevant (0), or uncertain/insufficient information (U). These judgments applied only to the text available in the study files; they did not constitute re-verification of the full content of the linked webpages. Independence of the ratings was based on the evaluation procedure confirmed by the researchers. No joint consensus table was created after disagreements were observed, and the original ratings of both experts were retained. Inter-rater agreement was therefore distinguished from a consensus-derived reference classification [16].

Nominal Cohen’s κ, including all three categories, was calculated [42]. A corresponding κ based on the weighted stratified contingency table was also reported. Percentile intervals for κ were obtained from 4000 bootstrap replications conducted within strata, resampling 60 documents with replacement from each stratum (seed = 20,260,919). These intervals are technical resampling intervals for the observed coding process and are not exact finite-population intervals. Rather than constructing a single consensus gold standard, three reference schemes were examined. Under the A and B schemes, only the respective evaluator’s uncertain ratings were treated as unresolved. Under the agreement-based scheme, only matching binary judgments (0/0 or 1/1) were treated as fixed; all disagreements and cases in which both evaluators selected U were retained as unresolved.

For selected documents, the lower bound of weighted true positives consisted of documents with fixed relevant judgments, whereas the upper bound additionally included the weighted unresolved cases. Weighted false negatives in the excluded group were defined analogously. Bounds for precision, recall, false-positive rate, false-negative rate, the proportion of irrelevant documents among selected records, and the proportion of relevant documents among excluded records were then obtained using the monotonicity of the corresponding performance functions. Proportions calculated directly from the balanced validation sample were not reported as full-corpus performance estimates because sampling probabilities differed across strata [43].

To distinguish uncertainty due to unresolved judgments from sampling uncertainty, an additional conservative simultaneous 95% outer range was calculated. Within each stratum, finite-population totals for two indicators—fixed relevant and fixed relevant or unresolved—were bounded by inversion of the hypergeometric distribution. A Bonferroni correction of α/12 was applied across the 12 stratum-level quantities. The resulting stratum-specific lower and upper bounds were aggregated and propagated through the corresponding performance functions. These outer ranges remain conditional on the correctness of agreed ratings and on stability of the coding process; they do not account for the possibility that both experts made the same incorrect judgment.

Source-specific screening performance was additionally evaluated for news, blogs, and online communities using the same agreement-based treatment of unresolved judgments. To assess whether screening uncertainty could alter the principal lexical contrasts, we conducted a validation-weighted screening-error sensitivity analysis using all 9567 deduplicated documents. Within each source, documents retained by the operational screening rule were assigned relevance weights spanning the source-specific agreement-based precision bounds, whereas excluded documents were assigned weights spanning the corresponding bounds for the proportion of relevant documents among excluded records. For each lexical group, weighted document prevalence was recalculated across all combinations of these bounds, and source-specific prevalence differences were examined. These ranges are sensitivity scenarios rather than confidence intervals or error-corrected population estimates; they assume that, within each source-by-screening stratum, the probability of semantic relevance can be represented by the corresponding validation-derived weight. Because this stratum-level weighting assigns a common relevance probability to term-present and term-absent documents within each source-by-screening stratum, it cannot represent lexical-item-specific screening error.

To address the possibility that screening error differed according to lexical presence within a source-by-screening stratum, an additional term-conditioned sensitivity analysis was conducted using the 360 validation documents. For each principal lexical group, expert relevance judgments were cross-classified by source, screening status, and lexical presence, while the original stratification weights were retained. Concordant relevant and irrelevant judgments were treated as fixed, whereas disagreements and jointly uncertain ratings were propagated as unresolved to obtain identification bounds for news–blog, news–online-community, and blog–online-community prevalence differences. Sampling uncertainty was additionally incorporated through finite-population hypergeometric inversion with Bonferroni adjustment, producing conservative 95% outer ranges reported in Table A4.

3.6. Contextual Validation and Sensitivity of Interpretation

Six lexical groups—human, accuracy, use, prediction, referee, and decision/officiating—were purposively selected for contextual validation because of their high potential for contextual ambiguity or their importance to substantive interpretation. For each term-by-source combination, up to 20 detected documents were sampled, yielding 333 document–term pairs. Because one document could contribute to more than one lexical group, these 333 pairs represented 287 unique documents. The contextual-validation exercise was not intended to constitute a representative validation of the full 189-group dictionary, and no semantic-validity claim is made for lexical groups that were not evaluated.

Experts independently assessed whether each detected expression was contextually consistent with the intended concept and separately coded its more specific semantic category. Weighted contextual-validity rates were calculated using weights defined by the number of detected documents in each term-by-source stratum divided by the corresponding sample size. Ranges reflecting evaluator-specific uncertain ratings were reported. Cohen’s κ was interpreted together with raw agreement and the marginal distributions. Items identified through cross-checking between document relevance and contextual lexical validity were retained as review flags; the original expert judgments were not overwritten. An additional lexical sensitivity analysis was conducted using all 360 validation documents. Three expert-defined relevant subsets were constructed: documents rated relevant by evaluator A, documents rated relevant by evaluator B, and documents rated relevant by both evaluators. Within each subset, weighted lexical prevalence was calculated separately for news and online documents by dividing the weighted number of lexical occurrences by the weighted number of relevant documents.

This analysis included relevant documents that had originally been excluded by the screening rule. It was restricted to 14 key lexical groups and treated as a descriptive sensitivity analysis. The principal subset sizes and key effect estimates are reported in Section 4.5. The analysis was not presented as a single expert-corrected version of the full 3161-document corpus, and no additional significance testing was imposed on these relatively small and sometimes sparse subsets.

3.7. Reproducibility and Sensitivity Analyses

Sensitivity analyses examined alternative search-term histories, screening stringency, near-duplicate removal, betting-related expressions, repeated series introductions, exclusion of titles, restriction to the first 150 characters of body text, and alternative duplicate-representative selection. Because these analyses were not preregistered as confirmatory tests, all were interpreted as exploratory robustness checks. To assess whether source differences in available text length influenced the principal lexical contrasts, body-text length and missing-body distributions were summarized separately within the 3161-document rule-selected corpus. A common-exposure analysis excluded documents with missing or fewer than 150 body characters and reran lexical detection using the complete title plus exactly the first 150 body characters, retaining 1341 news, 1235 blog, and 446 online-community documents. Title-only and body-only complete-case analyses were also conducted using the same lexical dictionary. A secondary 1:1 body-length matching analysis with a ±10-character caliper was examined as a diagnostic; because overlap between source-specific length distributions was limited, the common-exposure analysis was treated as the principal text-length robustness check.

All analyses were conducted in Python 3.12 using NumPy 2.3.5, SciPy 1.17.0, pandas 2.2.3, and scikit-learn 1.8.0. During manuscript revision, the analysis workflow was rerun, reproducing all primary estimates, bootstrap intervals, performance ranges, and expert-defined subset results. Across the pairwise comparisons, 10 cases with boundary prevalence yielded no finite logistic coefficient; these were retained as non-estimable and were not counted as successfully estimated models.

4. Results

4.1. Corpus and Validation Sample

Of the 10,878 original records, 9567 remained after URL-based and normalized-text deduplication. Applying the operational relevance rule—requiring sports- and AI-related markers to co-occur in the title or within 20 characters in the body text—yielded 3161 documents for the primary analysis. These consisted of 1404 news records, 1256 blog posts, and 501 online community posts. Within this rule-selected corpus, median body-text lengths were 163 characters for news (IQR = 159–167), 194 for blogs (IQR = 189–198), and 216 for online communities (IQR = 206–223). No selected news or blog records had missing body text, whereas seven online-community records (1.4%) had empty body fields. Blogs accounted for 71.49% of the pooled online group of 1757 documents. Importantly, the 3161 documents represent records selected by the screening rule rather than documents whose relevance had been confirmed by experts.

The primary-analysis corpus contained 348 news-domain clusters, 1071 blog account-path clusters, and 331 online-community clusters. The largest cluster accounted for 9.83% of news documents and 2.16% of documents in the pooled online group. Because multiple documents could therefore originate from the same publishing unit, analyses assuming document-level independence were compared with cluster-aware analyses. For independent validation, 60 documents were sampled from each of six strata defined by source and screening status, yielding 360 documents in total. Two experts independently evaluated these documents, as well as 333 detected lexical contexts. The number of documents retained at each processing and screening stage is summarized in Table 1.

Table 1. Number of documents by processing and screening stage.

Stage	News	Blogs	Online Communities	Total
Original records	5000	3646	2232	10,878
After URL deduplication	4025	3454	2143	9622
After longest-record retention and exact-text deduplication	3978	3448	2141	9567
Primary-analysis selected	1404	1256	501	3161
Primary-analysis excluded	2574	2192	1640	6406
Strict title-based screening	510	742	238	1490
Broad dual-mention screening	3460	2789	1457	7706
Primary analysis after near-duplicate removal	1225	1253	476	2954

Note: Strict screening, broad screening, and near-duplicate removal were evaluated separately. Selected documents were not necessarily semantically confirmed as relevant.

4.2. Agreement on Document Relevance and Screening Performance (RQ3)

The two experts assigned the same relevance category to 307 of the 360 documents. Raw agreement was 85.28%, with Cohen’s κ = 0.696. A 4000-replication within-stratum bootstrap yielded a 95% interval of 0.620–0.766 for κ. After accounting for the stratified sampling weights, agreement was 86.64% and κ was 0.697. This weighted κ is a nominal agreement statistic incorporating unequal sampling probabilities across strata; it is not an ordinal weighted κ based on distances between categories.

Both experts classified 111 documents as relevant and 195 as irrelevant. The remaining 54 cases—53 disagreements and one case in which both experts selected the uncertain category—were retained as unresolved rather than adjudicated. A post hoc descriptive review of the evaluators’ written rationales was conducted to characterize the sources of disagreement without changing the original ratings. The dominant sources of uncertainty were whether AI and sports were functionally linked rather than merely co-occurring or being mentioned in parallel (24/54, 44.4%), boundary cases concerning what should count as a sports-AI application—including mind sports, e-sports, gaming or betting, sports media, products, and educational contexts (22/54, 40.7%)—and insufficient or ambiguous textual evidence, such as missing body text, title- or link-only records, and unclear program or product labels (8/54, 14.8%). These patterns indicate that disagreement was concentrated primarily at the semantic boundary of sports-AI relevance rather than being attributable to a single source type. The stratified validation sample and independent expert judgments are summarized in Table 2. Among the 180 sampled documents selected by the screening rule, 54 were classified as irrelevant by both experts. Among the 180 sampled excluded documents, 18 were classified as relevant by both experts. Because an equal number of documents was sampled from each stratum, these unweighted proportions should not be interpreted as error rates for the full corpus.

Table 2. Sampling strata and independent expert judgments for document relevance.

Source	Screening Status	Stratum Size	Sample	Both Relevant	Both Irrelevant	Unresolved
News	Selected	1404	60	38	13	9
News	Excluded	2574	60	12	44	4
Blogs	Selected	1256	60	27	22	11
Blogs	Excluded	2192	60	1	48	11
Online communities	Selected	501	60	28	19	13
Online communities	Excluded	1640	60	5	49	6

Note: Unresolved cases comprise disagreements between evaluators or cases in which both evaluators selected the uncertain category.

After applying stratum weights and fixing agreement judgments while allowing unresolved cases to take any admissible classification, weighted precision ranged from 53.41% to 70.79% and weighted recall from 54.22% to 76.48%. The corresponding false-positive rate ranged from 13.90% to 22.82%, and the false-negative rate from 23.52% to 45.78%. The proportion of irrelevant documents among selected records ranged from 29.21% to 46.59%, whereas the proportion of relevant documents among excluded records ranged from 10.74% to 22.25%. The resulting weighted performance ranges are presented in Table 3.

Table 3. Weighted performance ranges based on the original expert ratings.

Reference Scheme	Precision (%)	Recall (%)	False-Positive Rate (%)	False-Negative Rate (%)	Irrelevant Among Selected (%)	Relevant Among Excluded (%)
Evaluator A	58.54–58.80	66.23–66.33	19.25–19.35	33.67–33.77	41.20–41.46	14.73–14.73
Evaluator B	64.87–65.92	62.95–65.89	16.82–17.60	34.11–37.05	34.08–35.13	16.84–18.83
Agreement-based	53.41–70.79	54.22–76.48	13.90–22.82	23.52–45.78	29.21–46.59	10.74–22.25

Note: Evaluator A had one uncertain rating and Evaluator B eight; 54 cases remained unresolved. The ranges are conditional bounds, not 95% confidence intervals. See Methods for metric definitions.

Screening performance also varied across source types. Under the agreement-based scheme, conditional precision ranged from 63.33–78.33% for news, 45.00–63.33% for blogs, and 46.67–68.33% for online communities. The corresponding recall ranges were 56.44–68.12%, 56.32–95.61%, and 43.74–71.47%, respectively. These source-specific ranges indicate that screening uncertainty was not uniform across information environments (Table A5).

These ranges quantify the effect of unresolved expert judgments under the assumption that agreed classifications are correct. They do not represent sampling confidence intervals and do not establish the absolute accuracy of the experts themselves. When sampling uncertainty was additionally incorporated, the conservative simultaneous 95% outer ranges widened to 35.37–85.07% for precision and 30.19–92.03% for recall. Treating two additional cross-check cases as unresolved increased the upper bounds to 71.79% for precision and 76.74% for recall. This additional analysis was a sensitivity check and did not alter the original expert ratings.

Overall, the independent evaluation showed substantial agreement between experts but did not support characterizing the screening rule as a high-precision filter. The analysis retained the 54 unresolved cases instead of forcing a consensus classification.

4.3. Three-Source Lexical Differences (RQ1)

In the primary three-source GEE analysis, 51 of 185 lexical groups met the BY-adjusted omnibus criterion (q < 0.05), while five groups had boundary prevalence and did not yield finite coefficients. Among omnibus-significant groups, 41 news–blog, 18 news–online-community, and three blog–online-community contrasts met the jointly BY-adjusted criterion. Because the dictionary was developed after inspection of the study corpus, these findings are interpreted as exploratory candidate contrasts conditional on the final dictionary specification. Industry, agreement, and expansion were more prevalent in news than in both online sources, whereas use and prediction were more prevalent in both blogs and online communities. Other patterns were source-dependent: support differed mainly between news and blogs, and review was particularly concentrated in blogs, including a significant blog–online-community contrast (q = 0.024). Referee and officiating decision also showed stronger news–blog than news–online-community differences. Overall, the results indicate that blogs and online communities should not be treated as a homogeneous inferential category (Table A6).

4.4. Pooled Online Comparison and Sensitivity to Analytical Choices (RQ2)

Although the three-source GEE served as the primary inferential model, the pooled news–online comparison was retained as a secondary descriptive and sensitivity analysis. In this comparison, industry, agreement, support, and expansion were more prevalent in news, whereas use, review, prediction, referee, and officiating decision were more prevalent in the pooled online group. Of the 185 tested lexical groups, 53 met the cluster-aware GEE–BY criterion. For the secondary pooled news–online comparison, selected lexical effects are summarized in Table 4, while Figure 1 shows the corresponding absolute prevalence differences and 95% cluster-bootstrap intervals.

Table 4. Document prevalence and effect sizes for selected lexical groups.

Lexical Group	News n/N (%)	Online n/N (%)	LR_D	Difference (pp)	Fisher q_BY	GEE q_BY
More salient in news
Industry	350/1404 (24.93)	224/1757 (12.75)	0.97	12.18	<0.001	<0.001
Agreement	99/1404 (7.05)	27/1757 (1.54)	2.18	5.51	<0.001	<0.001
Support	195/1404 (13.89)	102/1757 (5.81)	1.25	8.08	<0.001	<0.001
Expansion	121/1404 (8.62)	34/1757 (1.94)	2.14	6.68	<0.001	<0.001
More salient online
Use	10/1404 (0.71)	122/1757 (6.94)	−3.22	−6.23	<0.001	<0.001
Review	1/1404 (0.07)	99/1757 (5.63)	−5.73	−5.56	<0.001	<0.001
Prediction	34/1404 (2.42)	147/1757 (8.37)	−1.77	−5.94	<0.001	<0.001
Referee	20/1404 (1.42)	82/1757 (4.67)	−1.69	−3.24	<0.001	0.001
Officiating decision	21/1404 (1.50)	63/1757 (3.59)	−1.24	−2.09	0.004	0.022
Theoretically relevant terms without a clear source contrast
Technology	500/1404 (35.61)	607/1757 (34.55)	0.04	1.07	1	1
Performance	36/1404 (2.56)	43/1757 (2.45)	0.07	0.12	1	1
Fairness	22/1404 (1.57)	44/1757 (2.50)	−0.66	−0.94	0.794	0.956
Trust	8/1404 (0.57)	15/1757 (0.85)	−0.54	−0.28	1	1
Ethics	15/1404 (1.07)	13/1757 (0.74)	0.52	0.33	1	1

Note: Positive values of LR_D and the percentage-point difference indicate higher prevalence in news. Fisher and GEE q-values were each BY-adjusted across 185 lexical tests. Logistic GEE odds ratios are distinct from prevalence ratios. Nonsignificance does not establish equivalence.

Figure 1. Absolute differences in key lexical items and cluster-bootstrap intervals. Note: Points show news–online prevalence differences with 95% cluster-bootstrap intervals (2000 replications). Marker types indicate GEE–BY significance (q < 0.05). Intervals are not simultaneous or population-sampling confidence intervals. The vertical blue line at zero indicates no difference in document prevalence between news and the comparison source.

The pooled online estimates reflect the relative composition of blogs and online communities. For example, the prevalence of officiating decisions was 1.50% in news, 4.30% in blogs, and 1.80% in online communities. The news–blog comparison yielded GEE–BY q = 0.007, whereas the news–online-community comparison yielded q = 1.000. Review occurred in 7.09% of blog posts but 2.00% of online community posts, while support occurred in 4.46% and 9.18%, respectively.

Differences in statistical significance across separate pairwise comparisons were not interpreted as direct evidence that the corresponding effect sizes differed significantly from one another. The observed prevalence values nevertheless show that a single interpretation cannot always be applied to the pooled online category. Cluster diagnostics indicated that review was particularly sparse in news, occurring in only one document and one cluster, compared with 89 documents across 85 blog clusters and 10 documents across nine online-community clusters. Referee occurred across 17, 55, and 19 event-containing clusters in news, blogs, and online communities, respectively, and officiating decision across 15, 53, and eight. Sparse-term cluster-bootstrap checks preserved the main news–blog directions, whereas the news–online-community interval for officiating decision crossed zero (Table A5 and Table A6).

When blogs and online communities were technically standardized to contribute equally to the pooled online group, the news–online differences were 12.71 percentage points for industry, −6.24 for use, and −4.47 for review. The latter was smaller in absolute magnitude than the original −5.56-point difference. Standardizing the composition of the search-coverage intervals likewise preserved the directions of the main contrasts: 12.10 points for industry, 5.90 for agreement, and −6.11 for use. These standardizations were used to examine compositional sensitivity and were not interpreted as estimates of temporal change or causal platform effects.

Across 11 alternative analyses, Spearman correlations between the reference pooled news–online LR_D ranking and the alternative LR_D rankings for the 185 lexical groups ranged from 0.842 to 0.997. After removal of same-source near-duplicates, all 75 terms in the Fisher–BY reference set retained their direction and 65 continued to meet the same criterion. Under the strict title-based screening rule, 74 retained their direction and 30 retained both direction and the corrected criterion. Because available body-text length differed across sources, a common-exposure analysis was additionally conducted using documents with at least 150 non-missing body characters and restricting each document to its title plus the first 150 body characters. The principal news contrasts for industry, support, expansion, use, review, prediction, and referee retained their original directions, and the news contrasts for agreement also remained stable. The news–online-community contrast for officiating decision attenuated to approximately zero, whereas the news–blog contrast for technology changed direction from −2.21 to +1.31 percentage points. Title-only and body-only complete-case analyses generally preserved the major source patterns, although several effects were attenuated in the title-only analysis (Table A7). A secondary ±10-character body-length matching diagnostic yielded 246 news–blog, 61 news–community, and 329 blog–community pairs; because common support was limited, particularly for the news–community comparison, these matched results were treated as ancillary rather than as the primary robustness analysis.

Reductions in the number of terms meeting the criterion reflect both smaller sample sizes and reduced opportunities for lexical exposure. For consistency, the sensitivity summary uses the 75 Fisher–BY terms as the reference set and should not be interpreted as a reanalysis of the 53 GEE–BY terms. These sensitivity statistics are treated as robustness diagnostics rather than confirmatory tests.

4.5. Expert-Defined Relevant Subsets and Contextual Sensitivity (RQ3)

To link validation of the screening rule with sensitivity of the lexical results, an additional descriptive analysis was conducted using only documents rated as relevant by experts across both the selected and excluded strata. Evaluator A identified 130 relevant documents, Evaluator B identified 140, and both evaluators agreed on 111 relevant documents. Weighted prevalence estimates were calculated within each expert-defined subset.

These estimates represent different expert-defined relevance domains and should not be interpreted as a corrected version of the entire 3161-document primary corpus. Across all three expert-defined domains, industry, agreement, support, and expansion retained a news-oriented direction, whereas use, review, prediction, referee, and officiating decision retained an online-oriented direction. For industry, the news–online difference was 11.51 percentage points in Evaluator A’s subset, 6.22 in Evaluator B’s, and 8.43 in the jointly relevant subset. For use, the corresponding differences were −6.02, −11.17, and −8.18 points.

These consistent directions provide additional support for the main contextual contrasts. They do not, however, constitute confirmatory replication across all 185 lexical groups because the expert-defined subsets are small and some source-specific event counts were zero. Figure 2 compares the main lexical differences in the rule-selected corpus and the expert-defined relevant subsets. Technology was an important exception. Its news–online difference was only 1.07 percentage points in the rule-selected corpus but increased to 26.11 points in Evaluator A’s relevant subset, 15.27 in Evaluator B’s subset, and 28.42 in the jointly relevant subset. Thus, although technical vocabulary occurred in both sources, the near-equal prevalence observed in the rule-selected corpus was not stable under expert-defined relevance. The central finding is therefore not an “equivalent technical vocabulary layer,” but the coexistence of shared technical expressions and source-specific contextual emphasis.

Figure 2. Key lexical differences in the rule-selected and expert-defined relevant subsets. Note: Points show news–online prevalence differences for the primary corpus and Evaluators A and B relevant subsets (A: 130; B: 140). Distances are not confidence intervals; the jointly relevant subset is discussed in the text. The vertical blue line at zero indicates no difference in document prevalence between news and the comparison source.

The validation-weighted screening-error sensitivity analysis provided an additional check on whether the principal lexical contrasts depended on plausible source-specific screening error. Across the full range of agreement-based relevance weights, all nine main contextual contrasts retained their original direction in the pooled news–online comparison: industry, agreement, support, and expansion remained news-oriented, whereas use, review, prediction, referee, and officiating decision remained online-oriented. In the disaggregated comparisons, all nine contrasts retained their direction for news versus blogs, while eight of nine did so for news versus online communities. The exception was officiating decision, for which the sensitivity range for the news–online-community difference crossed zero (−0.36 to 0.23 percentage points). Technology was also unstable in the pooled analysis (−2.46 to 3.06 points), consistent with the expert-defined subset results above (Table A8).

The term-conditioned analysis showed that the news–blog and news–community directions for use, prediction, support, and expansion were preserved after jointly considering lexical presence and expert relevance. However, industry and agreement were less stable in some source contrasts, and technology showed a marked change after expert-based restriction. Review and referee were too sparse in the news validation sample for reliable source-specific term-conditioned estimation. When finite-population sampling uncertainty was additionally propagated, the conservative 95% outer envelopes crossed zero for all principal contrasts, indicating that the 360-document validation sample was insufficient for precise term-specific error correction. These results therefore support use of the validation analysis as a robustness check rather than as a corrected population estimate (Table A4).

Contextual validation covered 333 detected document–term pairs representing 287 unique documents. The experts disagreed on conceptual validity in 72 pairs and on the finer-grained semantic category in 112 pairs; the union of these two sets contained 113 pairs requiring review. Weighted contextual-validity rates for use differed substantially between Evaluator A (74.92%) and Evaluator B (46.70%), whereas the corresponding values for prediction were 83.12% and 88.62%.

Raw agreement for officiating decision was 95.92%. Nevertheless, κ was 0 for human, accuracy, and officiating decision because Evaluator A classified every sampled context for these terms as valid, producing a degenerate marginal distribution. These κ values should therefore be interpreted alongside raw agreement, consistent with the known high-agreement/low-κ phenomenon [44]. The contextual-validation results for the six selected lexical groups are summarized in Table 5.

Table 5. Expert evaluation of detected contexts for selected ambiguous lexical groups.

Lexical Group	Sampled Pairs	Agreement (%)	κ	Evaluator A Weighted Validity (%)	Evaluator B Weighted Validity (%)
Human	60	75	0	100.00–100.00	72.32–73.15
Accuracy	54	72.22	0	100.00–100.00	71.30–71.30
Use	50	68	0.375	74.92–74.92	46.70–46.70
Prediction	60	86.67	0.482	83.12–83.12	88.62–88.62
Referee	60	73.33	0.184	71.96–71.96	90.20–90.20
Officiating decision	49	95.92	0	100.00–100.00	95.54–95.54

Note: Identical bounds indicate no uncertain ratings. κ = 0 for human, accuracy, and officiating decision because Evaluator A rated all sampled contexts as valid. The 333 pairs represented 287 unique documents; validity rates apply only to detected instances, not dictionary recall.

This validation applies only to detected contexts for the six lexical groups examined. It does not establish semantic validity for all 189 lexical groups, nor does it estimate dictionary recall because undetected contexts were not sampled. In particular, occurrences of human were not automatically interpreted as concerns about human replacement, prediction was not automatically equated with performance analytics, and use was not treated as direct evidence of actual user experience. The evaluation therefore distinguishes between robustness in the direction of a lexical contrast and validity of the substantive meaning assigned to the detected term. The original results likewise explicitly limit the contextual evaluation to these six lexical groups. Accordingly, statistical significance for lexical groups outside these six terms should not be interpreted as evidence of contextual semantic validity.

5. Discussion

5.1. Source-Specific Contextual Emphasis and Interpretation

The findings reveal a combination of shared technical vocabulary and source-specific contextual emphasis in online information about sports AI. Industry, agreement, and expansion were more prevalent in news than in both online sources, whereas support was primarily a news–blog contrast. Use and prediction were more prevalent in both blogs and online communities, while review, referee, and officiating-related contrasts were stronger for blogs. The directions of these major contextual contrasts were largely preserved in the expert-defined relevant subsets. The near-equal prevalence of technology, however, was not. The main finding should therefore not be interpreted as evidence that technical vocabulary is distributed equally across sources. Rather, shared technical expressions coexist with source-specific patterns of contextual emphasis.

This pattern is consistent with the view that technology-related information is shaped by the environments in which it is produced and circulated [29]. Compared with Ning et al. [45], who examined information needs and topics in sports-related Q&A data, the present study focuses less on identifying topics themselves and more on the conditions under which lexical differences across information sources can be interpreted. Because the data do not independently identify authors’ roles, commercial intent, or actual experience with sports AI, the observed source differences should not be interpreted as direct contrasts between institutional and consumer attitudes or as causal effects of particular platforms. Lexical salience in this study refers to relative document-level prevalence rather than to a direct measure of framing, technology acceptance, or user attitudes.

The heterogeneity of blogs and online communities also matters when interpreting source contrasts. The prevalence of terms such as officiating decision and review differed across the two online sources, and standardizing their composition reduced the absolute news–online difference for review. Estimates for the pooled online group therefore partly reflect the relative contribution of its component sources. Different effect measures also convey different information. The large relative effect for agreement, the larger absolute difference for industry, and the very sparse occurrence of review in news describe distinct aspects of source-specific lexical patterns. The large relative contrast for review should nevertheless be interpreted cautiously because its news prevalence was based on only one event-containing document and cluster. For this reason, source contrasts are better interpreted using raw counts, document prevalence, relative effects, and percentage-point differences together rather than relying on a single ranking statistic [35,38].

5.2. Methodological Implications for Applied Text Analysis

A central methodological contribution of this study is the explicit separation of lexical distribution, document dependence, multiplicity, and semantic relevance. Document prevalence limits the influence of repeated occurrences within individual documents. Cluster-aware inference addresses dependence among documents generated by the same publishing unit. Multiple-testing correction addresses the problem of simultaneously testing many lexical groups, whereas independent expert evaluation assesses document relevance and the contextual validity of detected expressions. These procedures are complementary, but none can substitute for the others [4,14,15].

Statistical dependence altered the inferential results. In the primary three-source analysis, 51 lexical groups met the BY-adjusted omnibus criterion. In the secondary pooled news–online analysis, 75 groups met the Fisher–BY criterion and 53 remained significant under cluster-aware GEE–BY, illustrating the effect of accounting for within-source dependence. These findings should be regarded as robust candidate contrasts within the analyzed corpus rather than as externally confirmed lexical effects. Statistical support conditional on the finalized dictionary does not by itself establish measurement validity. The contextual-validation results therefore provide targeted evidence only for the six ambiguity-prone lexical groups evaluated; significance for the remaining groups indicates source-specific distributional contrast, not confirmed semantic validity.

In this respect, the present study is aligned with work that connects automated text analysis to human evaluation. Wunderlich and Memmert [46] compared lexicon-based sentiment analysis of sports-related Twitter data with human coding, while Heaton et al. [11] demonstrated the importance of examining computational outputs in their actual linguistic context. The present study extends this logic by separating two validation targets: document-level relevance and term-level contextual validity. It also evaluates both selected and excluded documents so that the validation captures not only potentially irrelevant inclusions but also relevant documents missed by the screening rule. Stratification weights were applied because the balanced validation sample did not reflect the source and screening proportions of the full corpus [43].

The expert evaluation further illustrates why inter-rater agreement and screening performance should be treated as different quantities. Under the agreement-based reference scheme, the conditional weighted precision of the screening rule ranged from 53.41% to 70.79% once unresolved cases were retained as uncertain. This range does not represent a sampling confidence interval, and it widened further when sampling uncertainty was incorporated. Nor does it account for cases in which both experts may have made the same incorrect judgment. The contribution of the validation exercise is therefore not that uncertainty was eliminated, but that its consequences were quantified and used to constrain the interpretation of the lexical results. The qualitative review further showed that unresolved judgments were concentrated at the boundary between direct sports-AI application and incidental, peripheral, or insufficiently documented associations, supporting the decision to preserve these cases as uncertainty rather than force consensus. This distinction is consistent with broader concerns regarding the validity of automated content analysis and persistent disagreement in human language judgments [4,39].

5.3. Practical Implications for Sports AI Information

The results also have implications for the presentation of sports AI information. Sports organizations and technology providers may benefit from explaining not only that an AI system has been adopted or that a partnership has been established, but also what task the system performs, what data it requires, how it is used, and where its judgments may be limited. In online content, distinctions between sports analytics and betting predictions, and between promotional material and accounts of actual use, should also be made explicit. When blogs and online communities are combined for communication analysis, differences in source composition and content type should be considered.

These implications should nevertheless be interpreted cautiously. They are derived from asymmetries in observed textual patterns rather than from experimental evidence that a particular form of communication increases understanding, trust, or intention to use AI. Similarly, the absence of clear source differences for explicit terms such as ethics, fairness, and trust should not be interpreted as evidence that ethical concerns are absent. Ethical issues in sports AI include fairness, explainability, privacy, and accountability [47], but such concerns may also appear indirectly through discussions of officiating errors, data use, or the role of human judgment. Information providers may therefore need to communicate not only abstract ethical principles but also concrete details concerning data provenance, human oversight, error handling, and accountability. Future experimental work could examine whether such forms of explanation influence users’ understanding and evaluation of sports AI systems.

5.4. Limitations and Future Research

Several limitations define the scope of the findings. First, the corpus consists of Korean-language texts retrieved through the Naver search environment and does not constitute a probability sample of the wider internet. The possibility that body fields contained excerpts rather than complete documents, together with incomplete documentation of some search and collection settings, further limits external validity. Although the common-exposure, title-only, and body-only analyses generally preserved the principal source contrasts, these checks cannot fully eliminate uncertainty arising from platform-specific excerpt construction and truncation. In addition, the screening rules and lexical dictionary were developed after inspection of the study data. The BY procedure controls multiplicity among the finalized lexical tests but does not account for the data-dependent process through which the screening rules and lexical dictionary were developed. The reported q-values are therefore conditional on the final exploratory specification and should not be interpreted as confirmatory evidence from a prespecified lexical instrument. Independent confirmation will require applying fixed screening rules and an unchanged dictionary to a separately collected corpus.

Second, contextual validation was restricted to six purposively selected lexical groups and to detected instances. It therefore does not establish semantic validity for the remaining dictionary entries or estimate dictionary-level recall. Broader validation across additional lexical groups and undetected contexts is required before the lexical resource can be treated as an externally validated measurement instrument. Likewise, directional consistency within the relatively small expert-defined relevant subsets should not be generalized to inferential replication across all lexical groups. The cluster proxies used for publishing units may also fail to capture all reposting or syndication relationships, and inference for sparse lexical items remains sensitive to small event counts. Moreover, the principal sensitivity analyses used the 75 Fisher–BY terms as their reference set; they should not be interpreted as complete re-evaluations of all 53 GEE–BY terms under every alternative specification.

Third, because the analysis was based on web-retrieved texts, complete end-to-end replication would require access to the original source documents and retrieval conditions. Future work could extend validation to undetected contexts and source-specific misclassification and compare rule-based screening with supervised or large-language-model-based classifiers. Such comparisons would be most informative if conducted using the same holdout sample, expert reference procedure, and treatment of unresolved judgments. Future research could also situate these discourse-oriented validation approaches alongside the rapidly expanding technical literature on sports AI, where deep-learning and computer-vision methods, including convolutional and recurrent neural-network architectures, are increasingly used for performance analysis [48]. These extensions remain directions for future research and were not evaluated in the present study.

6. Conclusions

This exploratory study identified candidate shared and source-specific lexical patterns in the TEXTOM-retrieved Korean sports-AI corpus. Organizational-activity vocabulary was generally more prevalent in news, whereas use and prediction were more prevalent in both blogs and online communities; several review- and officiating-related contrasts were more specific to blogs. The directions of the main contextual contrasts were largely preserved in expert-defined relevant subsets, although the distribution of some broadly shared technical terms changed after expert-based restriction. The presence of shared vocabulary should therefore be distinguished from evidence of equivalent prevalence across sources.

The broader contribution of the study lies in integrating document prevalence, cluster-aware inference, multiple-testing correction, and independent expert evaluation to define how far observed lexical contrasts can reasonably be interpreted. Unresolved expert judgments were retained as uncertainty rather than forced into a single reference label, allowing their effect on screening performance to be reflected in conditional ranges. This framework provides an applied example of how statistical support and semantic validity can be evaluated jointly in search-based text analysis. The findings characterize the Korean search texts examined here and should not be interpreted as estimates of public attitudes, the diffusion of sports AI, or the performance of AI technologies. The observed contrasts should therefore be interpreted as corpus-specific candidate patterns requiring confirmation with a fixed lexical resource and independently collected data.