Skip to Content
Applied SciencesApplied Sciences
  • Article
  • Open Access

27 July 2026

16 Pages

PU-Based Quality Classifier for LLM Training Texts

and
Institute of Information Technology, Warsaw University of Life Sciences, 02-787 Warsaw, Poland
*
Author to whom correspondence should be addressed.
This article belongs to the Section Computing and Artificial Intelligence

Abstract

In this study, we examine positive-unlabeled (PU) learning for classifying Polish texts intended for large language model training. We evaluate six PU methods, Naive, Clust, Strict-LassClust, Non-Strict-LassClust, LassoJoint, and Spy, under both classic and MVC non-SCAR labeling schemes. Across the benchmark, the best F1 reaches 0.906 on an open subtitles corpus under classic labeling and 0.900 under MVC labeling; the largest absolute gain over the Naive baseline is 0.811 F1 points on ulotki medyczne and 0.758 on plwiki in the classic setting. The experimental workflow supports multiple labeling rates, c calc, and repeated random seeds, and evaluates models with AUC, PR-AUC, and F1 as the main metrics. Our study focuses on six SpeakLeash datasets spanning different domains: plwiki, open subtitles corpus, job offers corpus, ISAP corpus, ulotki medyczne, and wolne lektury corpus. For each dataset, we construct a binary PU setup from the quality field, where HIGH is treated as positive and LOW is negative, while other values are removed. The resulting preprocessed tables retain only the selected linguistic and structural features, rescale numeric variables to the [0, 1] range, and remove the raw text column.

1. Introduction

Positive-unlabeled learning is a natural fit for text quality assessment when only a subset of high-quality examples is explicitly identified and the remaining documents are partially observed. This scenario arises naturally whenever collecting negative labels is costly or semantically ambiguous—a situation that directly characterizes large-scale text corpora such as those studied here [1].
The theoretical foundations of PU learning have been well-established. Bekker and Davis [1] provide a comprehensive survey covering label-frequency estimation, class-prior estimation, and algorithm design under partial supervision. Liu et al. [2] formalized this structure for text classification and introduced the Spy technique, in which a fraction of labeled positives is temporarily moved into the unlabeled set to calibrate a threshold for reliable negative extraction. Nigam et al. [3] showed that even a small labeled set combined with unlabeled documents via Expectation-Maximization substantially reduces classification error—an early demonstration that the unlabeled majority carries exploitable signal. The PU framework has since been applied across diverse domains: Fusilier et al. [4] demonstrated that a PU classifier trained solely on confirmed fake hotel reviews can achieve performance comparable to fully supervised approaches. Furmańczyk et al. [5,6] subsequently introduced propensity-based and cluster-assisted logistic regression methods, which provide the methodological foundation for the present study. Recent work has extended PU learning to settings with extremely few labeled positive examples. Dai et al. [7] derived theoretical conditions under which reliable learning remains possible and proposed the ScalePU framework for such highly label-scarce scenarios. A complementary line of research concerns class-prior estimation under distribution shift. Mielniczuk et al. [8] proposed a kernel-embedding estimator for PU data that explicitly accounts for prior shift and established non-asymptotic theoretical guarantees for the estimator. More recently, Liu et al. [9] proposed PUICL, an in-context learning framework based on pretrained transformers that performs PU classification without task-specific fine-tuning. These recent developments demonstrate the rapid expansion of PU learning toward both stronger theoretical guarantees and modern foundation-model architectures. However, they have not yet been systematically evaluated for stylometric quality assessment of Polish LLM training corpora. Mansouri et al. [10] provide the first theoretical analysis of the label complexity of active PU learning, studying how an adaptive querying strategy can reduce the number of labels needed to recover a reliable classifier.
A critical practical consideration is that the labeling mechanism in quality-annotation pipelines is unlikely to satisfy the Selected Completely At Random (SCAR) assumption. Formally, SCAR requires e ( x ) : = P ( S = 1 ∣ Y = 1 , x ) = c for some constant c independent of x —that is, every positive instance is equally likely to be labeled, regardless of its features. In practice, however, annotators preferentially label visually prominent documents—those with higher lexical density or greater length—making the labeling probability feature-dependent and thus violating the SCAR assumption [1]. This non-SCAR regime motivates the use of propensity-based labeling schemes that explicitly model e ( x ) [5]. From this perspective, document-quality annotation for LLM training texts can be viewed as a PU learning problem under a non-SCAR labeling mechanism: only a subset of truly high-quality documents receives an explicit positive label, while the remaining pool is left unlabeled and therefore contains a mixture of high- and low-quality texts. The probability that a high-quality document is annotated as such then depends on its observable attributes x (for example, length, lexical density, or domain), which matches the instance-dependent propensity setting e ( x ) = P ( S = 1 ∣ Y = 1 , x ) emphasized in recent PU learning work [1].
The practical stakes of quality filtering in Polish NLP are illustrated by the SpeakLeash-based Bielik model family [11,12,13]: successive versions (7B v0.1, 11B v2, 11B v3) were trained on best-quality SpeakLeash corpora and demonstrated that curated Polish data is essential for competitive downstream performance. Penedo et al. [14] reached a structurally identical conclusion for English web text: their FineWeb-Edu classifier—trained on a small LLM-annotated positive seed and a large unlabeled CommonCrawl pool—is, in effect, a PU learner, even though it is not explicitly framed as such. More recently, Emami Gohari et al. [15] demonstrated at scale that combining heuristic and classifier-based quality signals into a unified pipeline (GneissWeb) substantially improves LLM pretraining outcomes, reinforcing the importance of principled data-quality filtering for both English and multilingual corpora. Negoiță et al. [16] show that diversity-aware quality filtering strongly benefits Romanian LLM pretraining, highlighting the relevance of classifier-based filters for under-resourced languages beyond Polish. Li et al. [17] caution, however, that quality filters anchored to a fixed high-quality reference set introduce selection bias and reduce corpus diversity, because the filtering criterion is implicitly shaped by the characteristics of the seed collection rather than the downstream task. Nait Saada et al. [18] further argue that classifier-based quality scores can conflate changes in corpus composition with genuine quality improvements, challenging the view that classifier-based quality filtering (CQF) always captures an intrinsic notion of data quality. Langlais et al. [19] address the data availability problem from a different angle, releasing Common Corpus—the largest open and fully licensed collection for multilingual LLM pretraining—which covers Polish among dozens of European languages and establishes a reproducible baseline for open-science LLM research.
This observation reinforces the need for principled PU methods that explicitly model the labeling propensity e ( x ) and decouple it from the classification objective—exactly the distinction drawn by the non-SCAR framework evaluated in the present work. In this paper, we study whether dedicated PU methods can classify Polish texts in a reproducible way across heterogeneous domains and data sizes. The full experimental pipeline is implemented in Python 3.10 (selected classification scripts were rewritten by the authors from their earlier R 4.3.0 implementations (https://github.com/kapacc/mdai25-clusters-scripts (accessed on 22 July 2026)) into Python with assistance from GitHub Copilot, mainly using a GPT-Codex model). The current codebase is available at https://github.com/kapacc/pullm (accessed on 22 July 2026)) and separates labeling, preprocessing, and model training into distinct stages.

Related Work and Contributions

Modern data filtering for LLM training increasingly uses classifier-guided pipelines such as FineWeb-Edu [14], the GneissWeb framework [15], and scaling-aware quality filters such as ScalingFilter [17]. Those approaches motivate our study, but they still do not answer the PU-specific question of how to model positive-only labeling under non-SCAR conditions for Polish corpora. Recent work on low-resource languages demonstrates that similar classifier-based quality filtering pipelines can substantially improve pretraining data for Romanian [16]. At the same time, Nait Saada et al. [18] provide a critical analysis of CQF-style filters for LLM pretraining, showing that classifier scores may not always correspond to intrinsic data quality; we acknowledge this as a limitation of classifier-based filters and treat our PU classifiers primarily as tools for prioritizing data rather than as complete quality metrics. Langlais et al. [19] complement this picture by providing a large-scale multilingual resource with explicit provenance metadata, enabling future work to benchmark quality classifiers under reproducible and ethically vetted conditions. On the PU learning side, Wang et al. [20] recently proposed a standardized evaluation framework for PU algorithms that highlights the importance of realistic experimental conditions and reproducible benchmarking—a concern shared by the present work. Dai et al. [7] complement this perspective with a theoretical analysis of PU learning under extreme label scarcity and a variance-regularized ScalePU estimator. Mielniczuk et al. [8] address the related problem of class-prior estimation when the target distribution shifts, which is directly relevant to our cross-domain evaluation setting. The PUICL model by Liu et al. [9] demonstrates that in-context learning can be applied to PU classification without dataset-specific training—a direction that could further reduce the annotation overhead in future extensions of this work. Mansouri et al. [10] provide formal guarantees on the label complexity achievable by an adaptive querying strategy in the PU setting, motivating future work on active annotation for SpeakLeash corpora.
The concrete contributions of this paper are as follows:
  • First comprehensive benchmark of PU-learning methods for Polish LLM training corpora.
  • Evaluation under two different non-SCAR labeling mechanisms.
  • Cross-domain study across six heterogeneous SpeakLeash datasets.
  • Practical recommendations for PU-learning deployment in LLM data filtering.
  • Open-source Python implementation enabling reproducible research. The paper presents a Python reimplementation of the training and evaluation pipeline from earlier work, together with preprocessing tailored to the SpeakLeash corpora used in our experiments.
Table 1 and Table 2 summarizes the substantive differences between the previous our non-SCAR study and the present manuscript.
Table 1. Substantive novelty of the present manuscript relative to our earlier non-SCAR study.
Table 2. Comparison of selected related work and the present study.
Throughout this work, we adopt the single-scenario formulation of PU learning [1]. Specifically, we assume that the latent triples ( X i , Y i , S i ) , i = 1 , … , n , are i.i.d. realizations of the random vector ( X , Y , S ) . However, only the pairs ( X i , S i ) are observed, where S i = 1 implies Y i = 1 , while observations with S i = 0 are unlabeled and originate from a mixture of positive and negative instances. This stands in contrast to the case-control scenario, where separate i.i.d. samples from p ( x ∣ Y = 1 ) and the marginal p ( x ) are available. The single-scenario setting is the natural one for our use case: the SpeakLeash corpora arrive as a single collection in which only a fraction of high-quality documents carries an explicit HIGH label, while the remaining documents are left unlabeled regardless of their true quality.

2. Objectives and Methods

The primary objective of this work is to investigate whether positive–unlabeled (PU) learning methods can be effectively applied to text quality classification of Polish LLM training corpora. To achieve this objective, we evaluate six representative PU learning methods (Table 3) using the SpeakLeash corpora. We implemented a comprehensive benchmarking pipeline that controls for both the labeling mechanism and the estimation of the class-prior proxy c, enabling direct comparison of PU methods under non-SCAR settings.
Table 3. Characteristics of the PU learning methods considered in the benchmark.

2.1. PU Learning Methods

We evaluate six PU methods with increasing algorithmic complexity:
  • Naive: Standard logistic regression trained directly on the observed label S as a binary target, treating labeled positives ( S = 1 ) as the positive class and all unlabeled instances ( S = 0 ) as negatives, without any correction for the fact that the unlabeled pool contains a mixture of true positives and true negatives. As a result, the model implicitly assumes e ( x ) = 1 , i.e., that every positive instance is labeled—a strong and generally violated assumption that causes the predicted scores to be systematically downward-biased for true positives hidden in the unlabeled set.
  • Clust [6] applies an iterative “pecking-order” procedure to estimate logistic regression coefficients; the number of iterations and the pecked fraction are treated as parameters, set to 5 iterations and full labeled-positive set per iteration in this experiment. In each iteration, a random subset of labeled positive instances ( S = 1 ) is temporarily moved into the unlabeled pool (in our work param of pecked_part = 1); the combined set is standardized (Z-score) and partitioned into two clusters via K-means ( K = 2 ). The cluster with the higher proportion of labeled positives is treated as positive, and the remaining cluster as negative. A logistic regression is then fitted on the resulting pseudo-labeled data, and final coefficients are obtained by averaging estimates across all iterations.
  • Strict-LassClust [6] combines Clust (details above) with L1 regularization (Lasso). Coefficients are averaged across clusters using strict averaging (only features selected in all clusters are retained).
  • Non-Strict-LassClust [6] is similar to Strict-LassClust, but uses non-strict averaging (averaging across all clusters regardless of selection consistency).
  • LassoJoint [21] combines L1-regularized feature selection with a joint PU logistic model fitted on the selected support. In the first step, a Lasso logistic regression with cross-validated penalty ( λ selected via 10-fold CV) is used to identify informative features; only those with | β ^ j | > δ (where δ = 0.5 λ ) are retained. In the second step, a joint logistic model R ( b , c ) is fitted on the selected features by minimizing the negative log-likelihood
    − ∑ i S i log ( c σ ( x i ⊤ b ) ) + ( 1 − S i ) log ( 1 − c σ ( x i ⊤ b ) ) ,
    which simultaneously estimates the coefficient vector b and the class-prior proxy c = E [ S ∣ Y = 1 ] .
  • Spy [2] uses the Spy technique to extract reliable negatives from the unlabeled set, then trains a classifier on the augmented labeled pool. A fraction of labeled positive examples is temporarily moved into the unlabeled set, and predictions using these Spy samples calibrate a classification threshold for reliable negative extraction.
Deep generative PU architectures, including GAN-based models [22] and variational-autoencoder approaches, are most often analyzed in a case–control sampling scheme, where one observes separate samples from the positive class and from the mixture distribution over the general population. This assumption is less realistic for our SpeakLeash use case, which follows the single-scenario design with a single partially labeled sample. In principle, the single-scenario regime with selection bias can be addressed by variational methods such as the VAE-PU+OCC model of Wawrzeńczyk and Mielniczuk [23], but deploying such architectures on our largest corpora would be prohibitively expensive computationally, especially in terms of GPU memory and wall-clock training time.
The methods studied here were chosen to cover the most relevant trade-offs for this setting. Naive provides the strongest possible lower bound. Clust and the two LassClust variants test whether clustering and sparse logistic fitting can recover structure from weakly labeled features. LassoJoint is the most direct propensity-aware method in the set. Spy represents a classical PU baseline that is still popular in text classification. Together, the six methods span simple, cluster-assisted, sparse, and propensity-aware families without introducing a separate neural architecture family that would require a very different computational regime.

2.2. Benchmark Workflow

Table 4 and Figure 1 give a compact summary of the end-to-end benchmark workflow from raw SpeakLeash data to final evaluation.
Table 4. End-to-end benchmark workflow used in the manuscript.
Figure 1. Benchmark pipeline for a single dataset and labelling strategy.

2.3. Experimental Design and Evaluation

Each method was evaluated across multiple random seeds (10 repetitions) and three target labeling rates ( c c a l c ∈ { 0.2 ,   0.5 ,   0.8 } ). We used the union-train protocol. For each seed and each dataset, we first drew up to 10,000 examples from the preprocessed table (if a dataset was smaller, all rows were kept), preserving the class composition induced by Y. Next, each sampled dataset was split into train and test parts using a stratified 70 / 30 split. The training parts from all datasets were then concatenated into one shared PU training set, on which non-SCAR labeling was generated and all PU methods were fitted once. Since the experiment was repeated ten times, each with a different random 70/30 train–test split, the results are based on a multi-split evaluation rather than a single train–test split. Evaluation was then performed separately for each dataset: predictions were sliced back to the corresponding test subset, and AUC, PR-AUC, and F1 were computed per dataset. The experimental pipeline was implemented using dedicated Python scripts. The current codes used in this study are publicly available in the paper repository (https://github.com/kapacc/pullm (accessed on 22 July 2026)).
The sample size of 10,000 examples per corpus was chosen to balance computational tractability with statistical representativeness. While modern LLM data filters operate on corpora orders of magnitude larger, the present benchmark is designed to evaluate the methodological behaviour of PU classifiers under controlled conditions—with fixed feature spaces, shared preprocessing, and repeated seeds—rather than to simulate production-scale filtering. The 10,000-example cap ensures that results remain comparable across corpora of very different sizes (from ∼6000 to ∼1.4 M rows) while keeping wall-clock training time manageable across six methods, three labeling rates, and ten seeds. We discuss this choice as a limitation in Section 7.

2.4. Background: The Bielik Quality Classification Use Case

The original motivation for PU learning in Polish NLP stems from the Bielik model family. Quality assessment of documents is therefore a crucial step in training LLMs, especially in the Bielik line, where curated Polish data directly affects model performance. The Bielik 7B v0.1 model was adapted from Mistral 7B v0.1 due to limited access to high-quality Polish data. During pre-training preparation, 9000 documents were manually annotated with quality labels (HIGH, MEDIUM, LOW), and 266 stylometric features were computed per document (verb/noun frequencies, sentence counts, punctuation patterns). An XGBoost classifier was then used to classify the full corpus, retaining only documents predicted to be high quality for final LLM training. This historical approach motivates the PU framework: only a seed set of confirmed high-quality documents is available, while the remaining pool remains partially observed, making dedicated PU methods essential for scalable and principled quality assessment. The subsequent Bielik v3 series [13] extended this approach by replacing the universal Mistral-based tokenizer with a Polish-optimized vocabulary, further illustrating how data-quality decisions interact with architectural choices in the development of language-specific LLMs.
Both labeling strategies (Table 5) respect the class structure ( Y ∈ { 0 , 1 } ) but differ fundamentally in how they model the selection propensity e ( x ) = P ( S = 1 ∣ Y = 1 , x ) and calibrate the achieved class-prior proxy c = E [ S ∣ Y = 1 ] to match a target value c c a l c .
Table 5. Comparison of labeling strategies used in the benchmark.
The Classic (logistic) strategy assumes a linear propensity model e ( x ) = σ ( α + θ ⊤ x ) , where the propensity score uses all available variables and α is calibrated via grid search over [ − 10 , 10 ] to minimize | c c a l c − c | . This approach is stochastic: each run generates labels independently via Bernoulli sampling, leading to non-zero variance across random seeds. It follows established PU simulation practice [5]. In the Classic strategy, we use a homogeneous coefficient vector θ = ( 1 , … , 1 ) ⊤ ∈ R p (i.e., θ j = 1 for all j), so that the linear predictor reduces to α + ∑ j = 1 p x j . The intercept α is selected on a uniform grid of 401 points spanning [ − 10 , 10 ] to minimize E [ e ( X ) ∣ Y = 1 ] − c calc , where the expectation is approximated by the sample mean over positive instances in the training set. Once the optimal α ^ is found, selection labels are drawn as S i ∼ Bernoulli ( e ( x i ) ) independently for each Y i = 1 , while S i = 0 is set deterministically for all Y i = 0 .
The MVC (Most Variable Columns) strategy is a deterministic, rank-based mechanism based on feature variance ranking. In this work, samples are scored by summing only the two features with the highest variance and then ranked. A linear threshold is computed by interpolation at quantile levels { 0.1 ,   0.3 ,   0.5 ,   0.7 ,   0.9 } to match the target c c a l c . This method was introduced in [6].

3. Datasets

SpeakLeash is an open source ecosystem centered on Polish-language AI resources. According to the official package documentation, it provides a lightweight Python library for downloading and inspecting Polish datasets together with dataset manifests and metadata access [24].
Our experiments use six SpeakLeash datasets selected to cover distinct domains and scales [24]. The selected corpora span encyclopedia-like content, dialogue, job offers, legal documents, medical texts, and literary prose. This mix allows us to test PU methods on both shorter and longer documents, as well as on sources with different vocabulary richness and style.
  • plwiki is a Polish Wikipedia mirror containing encyclopedia entries across diverse topics; the dataset combines both longer structured articles and shorter stub entries.
  • open_subtitles_corpus consists of subtitles extracted from film and television dialogues, yielding shorter utterances with informal and conversational language patterns.
  • job_offers_pl_corpus aggregates job listings from online employment portals, typically featuring semi-structured content with repetitive formatting and domain-specific terminology.
  • ISAP_corpus comprises legal and administrative documents from Polish public institutions, characterized by long, formally structured sentences with specialized vocabulary.
  • ulotki_medyczne contains medical information leaflets and health-related texts featuring technical terminology and precise descriptive language.
  • wolne_lektury_corpus is a collection of Polish literary prose from the public domain, including novels and short stories with diverse narrative styles and rich vocabulary.
For each dataset, we use the quality metadata provided with the corpus to construct the PU target. Documents labeled HIGH are mapped to the positive class ( Y = 1 ), documents labeled LOW are mapped to the negative class ( Y = 0 ), and all remaining quality values are discarded. We then keep only the selected feature columns used by the benchmark pipeline, remove the raw text column, and rescale all numeric variables to the [ 0 , 1 ] range using min–max scaling. A detailed description of all 22 linguistic and stylometric features is provided in Table S1 of the Supplementary Material. Importantly, in line with the SpeakLeash curation perspective, these quality labels primarily reflect usefulness for LLM training (e.g., clarity, consistency, and preprocessing suitability), rather than artistic merit or stylistic richness.
The exclusion of MEDIUM is intentional rather than accidental. Those documents are ambiguous by construction and do not provide a clean negative class, so keeping them inside a binary HIGH/LOW benchmark would entangle class-label noise with PU noise. We therefore use the binary split as a controlled experimental proxy for the practical filtering task, but we interpret the results as methods for prioritizing high-quality data rather than as a complete quality-ranking solution. This is especially important for corpora such as ISAP_corpus, where the positive class is extremely rare and any method can become sensitive to small changes in the label mechanism.
Table 6 summarizes the datasets used in the current scope and combines corpus metadata with PU-related summary statistics. Mean Abs Corr denotes the average absolute Pearson correlation between each feature.
Table 6. SpeakLeash datasets used in the latest benchmark run, with corpus metadata and PU summary statistics.
Figure 2 presents a condensed comparison of the shared feature distributions for all six preprocessed SpeakLeash corpora used in this benchmark (plwiki, open_subtitles_corpus, job_offers_pl_corpus, ISAP_corpus, ulotki_medyczne, wolne_lektury_corpus). The left panel shows per-feature medians, while the right panel shows interquartile ranges (IQRs). Two patterns are visible: first, central tendencies differ most strongly for ratio-style features such as lexical density, noun ratio, and verb ratio; second, several count-like features have very low IQR after min–max scaling, indicating narrow within-dataset spread despite clear between-dataset differences.
Figure 2. Condensed comparison of shared feature distributions for all six preprocessed SpeakLeash corpora used in this benchmark. (top): feature medians. (bottom): IQR ( Q 0.75 − Q 0.25 ).
In conclusion, Figure 2 confirms that the visualized corpora are not interchangeable in feature space, even after uniform preprocessing and min–max scaling. The largest cross-dataset contrasts are concentrated in ratio-based variables, which suggests that stylistic and lexical composition differs more strongly than raw count intensity. At the same time, low IQR values for many count-like variables indicate internally stable feature profiles within each corpus, supporting dataset-specific benchmarking rather than pooling all sources into one training distribution.
The same interpretation applies to the binary PU setup. The dataset statistics show that ISAP_corpus contains only 0.84% positive instances, so even small fluctuations in the labeling mechanism can visibly affect the apparent difficulty of the task. By contrast, corpora such as open_subtitles_corpus and plwiki contain much richer positive support, which makes them better suited for comparing method behaviour across labeling strategies.

4. Stability of Non-SCAR Labeling

To pre-assess stability of non-SCAR labeling, we repeated label generation five times per dataset using different random seeds ( 11 ,   22 ,   33 ,   44 ,   55 ) and a fixed target value of c c a l c = 0.2 . The seed sequence { 11 ,   22 ,   33 ,   44 ,   55 } was chosen as a simple, evenly-spaced arithmetic progression to ensure reproducibility and avoid any implicit bias that could arise from selecting seeds based on prior experimental outcomes. We report two quantities per method: (i) achieved class-prior proxy c = E [ S ∣ Y = 1 ] after labeling, and (ii) absolute deviation from the target, | c c a l c − c | . We additionally report wall-clock labeling time. The experiment covers all six preprocessed SpeakLeash datasets used in the benchmark.
Table 7 summarizes the results as the mean ± standard deviation over five runs. We use shortened names in the table: mvc and classic. The mvc variant is a rank-based non-SCAR mechanism calibrated to match a target labeling rate and was used previously in [6]. The classic variant uses logistic propensity calibration to match the target class-prior proxy and follows prior PU simulation practice [5]. As expected, classic is stochastic and shows non-zero variance across seeds, while mvc is deterministic for fixed data and therefore has close to zero variance in the achieved c and deviation. In methodological terms, classic reflects a widely used non-SCAR simulation pattern in the PU-learning literature, whereas mvc is the authors’ original rank-based approach introduced in our earlier work [6].
Table 7. Stability of non-SCAR labeling on six preprocessed datasets ( c c a l c = 0.2 , five seeds).
The two strategies also differ in their practical cost and calibration behaviour. The average runtime of the classic generator is substantially higher on large corpora, most clearly on plwiki, where the mean labeling time exceeds 30 s because the propensity grid search must evaluate a much larger positive pool. MVC is cheaper and fully deterministic, but it can undershoot the target c c a l c when the two highest-variance features are not sufficient to express the desired selection propensity.
A compact aggregated comparison of wall-clock runtime by method and labeling strategy is given in Supplementary Table S3.

5. Benchmark Results

In this section, all reported metrics are computed per dataset from a model trained once per seed on the union of training subsets. We evaluated all six PU methods across six SpeakLeash datasets using both non-SCAR labeling strategies: mvc and classic. Each configuration was run for three target labeling rates, c c a l c ∈ { 0.2 ,   0.5 ,   0.8 } , and ten random seeds. We report AUC, PR-AUC, and F1 as the metrics in the Tables S5–S16 of Supplementary Material (https://github.com/kapacc/pullm/blob/master/paper/supplement_mdpi.pdf (accessed on 22 July 2026)), while the main text summarizes mean F1 aggregated over the three c c a l c values. We focus on the F1 score because our goal is to ensure clear class separation rather than to optimize decision thresholds, which is typically impractical in LLM training due to its substantial computational cost. As AUC reflects performance across a range of thresholds, we use it only as a supportive, illustrative metric in this analysis.
Table 8 and Table 9 provide side-by-side summaries for the two strategies. The split makes strategy-dependent behavior explicit: for example, LassoJoint achieves notably high F1 under classic on plwiki (0.491) and ulotki_medyczne (0.665). The spy method demonstrates strong results on plwiki under both strategies and on ulotki_medyczne under classic. For plwiki under classic labeling, LassoJoint clearly dominates PR-AUC (0.841/0.718/0.851), remaining well above Spy (0.407 across all c c a l c ) and cluster-based variants, which supports the view that propensity-aware modeling can materially improve positive-class retrieval quality on this corpus (Table S14 in the Supplementary Material).
Table 8. Benchmark results summary (mvc): average F1 over c c a l c ∈ { 0.2 ,   0.5 ,   0.8 } .
Table 9. Benchmark results summary (classic): average F1 over c c a l c ∈ { 0.2 ,   0.5 ,   0.8 } .
For a compact summary of the best observed F1 scores, the Naive baseline, and the absolute gain per dataset and strategy, see Supplementary Table S2.
To compare the MVC and Classic labeling strategies, we performed paired Wilcoxon signed-rank tests separately for each PU method at a fixed value of c. The paired observations were defined by the same ( dataset ,   seed ) combination, yielding 6 × 10 = 60 paired observations for each method (six corpora and ten random train–test splits). To account for multiple comparisons across the six methods, the resulting p-values were adjusted using Holm’s procedure. The detailed results of these statistical comparisons for F1-score, AUC, and AUC-PR are reported in Tables S18–S20 of the Supplementary Material. The comparison revealed that only a subset of methods exhibited statistically significant differences in terms of F1-score after Holm correction, whereas almost all comparisons were significant for AUC (16 out of 18) and all comparisons remained significant for AUC-PR. These findings indicate that the effect of the non-SCAR labeling strategy is much more pronounced for ranking-based performance measures (AUC and AUC-PR) than for the threshold-dependent F1-score.

6. Discussion

The results show that the most reliable conclusions are method- and corpus-specific rather than global. LassoJoint is the strongest general-purpose choice under the classic mechanism, especially for job_offers_pl_corpus, plwiki, and ulotki_medyczne, while Spy is more stable when the data are highly imbalanced or when the feature signal is sparse. Under MVC, the ranking changes more often: Spy remains competitive on ISAP_corpus, ulotki_medyczne, and wolne_lektury_corpus, while cluster-based methods can be preferable for job_offers_pl_corpus.
The failure modes are also informative. On ISAP_corpus, many methods reach F1 values close to zero because the class prior is extremely small (0.84%) and the 22 available stylometric features do not cleanly separate legal high-quality text from the rest of the corpus. This is not a random artifact of one split; the same pattern appears across seeds and both labeling strategies. The near-zero F1 on ISAP_corpus is therefore not a sign that PU methods are intrinsically broken, but rather a consequence of extreme class imbalance combined with a feature space that captures surface-level stylometric properties rather than domain-specific legal quality signals. In other words, the legal domain is a hard case for low-dimensional PU filtering, and the benchmark correctly exposes that limitation. This picture is consistent with recent theoretical analyses of PU learning under extreme label scarcity, such as the ScalePU framework of Dai et al. [7], which highlights how generalization bounds in PU risk estimation are dominated by the positive-term complexity when very few labeled positives are available. The kernel-embedding approach of Mielniczuk et al. [8] suggests that prior-shift-aware estimation could further improve calibration in such imbalanced regimes, and we leave this extension for future work.
From a practical perspective, the choice of method should depend on the data regime. If the corpus is moderately balanced and the goal is a strong high-quality filter, LassoJoint is usually the first candidate. If the corpus is very noisy, highly imbalanced, or the selection mechanism is likely weakly informative, Spy is a safer baseline. If the selection mechanism itself is being stress-tested or simulated, the difference between classic and MVC should be treated as part of the benchmark design rather than as a single fixed truth.

7. Limitations

This study has several limitations. First, the benchmark only uses the 22 stylometric features currently available in SpeakLeash, so it does not test whether richer representations would change the ranking of the methods. Second, we intentionally discarded MEDIUM documents to keep the task binary and reproducible, which means that the setup was an approximation of real filtering rather than a full ternary quality assessment. Third, the current benchmark logs capture wall-clock runtime for model fitting and evaluation together; isolated inference timing is not yet measured separately. Finally, deep generative PU models remain outside the present comparison because they would require a substantially different computational budget and training protocol.

8. Conclusions

This work demonstrates that PU learning can be used as a practical and reproducible framework for quality classification of Polish texts intended for LLM training. We implemented an end-to-end Python pipeline that separates preprocessing, non-SCAR labeling, and benchmark evaluation, and we validated six PU methods across six heterogeneous SpeakLeash datasets.
The main empirical conclusion is that there is no single globally best PU method across all domains and labeling strategies. Performance is strongly strategy- and dataset-dependent. Under classic labeling, lassojoint achieves the strongest overall results on most datasets, while under mvc, the best choice shifts more often toward spy or cluster-based methods, depending on the corpus. The paired Wilcoxon signed-rank tests presented in the Supplementary Material use (dataset, seed) pairs as matched observations, with Holm’s correction applied across the six hypotheses, corresponding to the evaluated methods for each performance metric (https://github.com/kapacc/pullm/blob/master/paper/supplement_mdpi.pdf (accessed on 22 July 2026)).
Supplementary Table S2 gives the per-dataset best-F1 summary underlying these headline statements, together with the Naive baseline and absolute gain.
The obtained results may be related to the findings of Ociepa et al. [11], for whom one component of the study involved separating text-quality classes. They reported an F1 score of 0.84 (on test subset about 1000 samples) using an XGBoost model; however, their experiments were conducted on fully labeled samples (about 9000 samples). Moreover, they relied on 266 stylometric features, whereas the present work uses only the 22 features directly available in the SpeakLeash repository, which substantially simplifies the computational pipeline. Additional advantages of our approach include scalability without manual annotation of negative examples: PU learning, by design, operates without costly labeling of the LOW class, which is a key operational benefit when filtering large corpora. Furthermore, methods such as LassoJoint allow us to model propensity, acknowledging that the probability of a document being labeled depends on its characteristics—a realistic assumption for SpeakLeash annotation systems.
Notably, a substantial portion of the SpeakLeash corpora carries intermediate quality labels (MEDIUM), which are currently discarded in binary setups. As discussed by the SpeakLeash team at conferences, this MEDIUM subset represents untapped potential for LLM training after refined text separation. PU learning directly addresses this scenario: documents labeled MEDIUM may contain genuine positives that remain unlabeled, making the problem structurally equivalent to the single-scenario PU setting where the true positive class is partially observed.
In practical terms, the benchmark suggests the following guidance: Use LassoJoint when the corpus has enough signal for propensity-aware feature selection and when strong retrieval of high-quality texts matters most. Use Spy when the corpus is extremely imbalanced or the feature space is too weak for sparse propensity estimation. Treat cluster-based methods as reasonable middle-ground baselines, especially when the selection mechanism itself may be coarse. Future work should extend the feature set, isolate inference timing, and add deeper PU baselines before drawing broader conclusions about deployment at web scale.
A compact decision-oriented summary of this guidance is provided in Supplementary Table S4.

Supplementary Materials

The following supporting information can be downloaded at https://www.mdpi.com/article/10.3390/app16157481/s1. Tables S1–S4: Methods and data overview; Tables S5–S16: Detailed benchmark results per dataset and labeling strategy; Table S17: Feature descriptions; Tables S18–S20: Statistical comparison.

Author Contributions

Conceptualization, K.F. and K.P.; methodology, K.F. and K.P.; software, K.P.; validation, K.F. and K.P.; formal analysis, K.F. and K.P.; investigation, K.P.; resources, K.F.; data curation, K.P.; writing—original draft preparation, K.P.; writing—review and editing, K.F.; supervision, K.F.; project administration, K.F. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Data Availability Statement

The Python code and supplementary results are available at https://github.com/kapacc/pullm (accessed on 22 July 2026). The SpeakLeash datasets are publicly available via https://pypi.org/project/speakleash/ (accessed on 22 July 2026).

Acknowledgments

The authors thank the SpeakLeash community for providing open-access Polish corpora used in this study.

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Bekker, J.; Davis, J. Learning from Positive and Unlabeled Data: A Survey. Mach. Learn. 2020, 109, 719–760. [Google Scholar] [CrossRef] [Scilit]
  2. Liu, B.; Lee, W.S.; Yu, P.S.; Li, X. Partially Supervised Classification of Text Documents. In Proceedings of the 19th International Conference on Machine Learning (ICML 2002); Morgan Kaufmann: San Francisco, CA, USA, 2002; pp. 387–394. [Google Scholar]
  3. Nigam, K.; McCallum, A.K.; Thrun, S.; Mitchell, T. Text Classification from Labeled and Unlabeled Documents Using EM. Mach. Learn. 2000, 39, 103–134. [Google Scholar] [CrossRef] [Scilit]
  4. Fusilier, D.H.; Montes-y-Gómez, M.; Rosso, P.; Cabrera, R.G. Detecting Positive and Negative Deceptive Opinions Using PU-Learning. Inf. Process. Manag. 2015, 51, 433–443. [Google Scholar] [CrossRef] [Scilit]
  5. Furmańczyk, K.; Mielniczuk, J.; Rejchel, W.; Teisseyre, P. Double Logistic Regression Approach to Biased Positive-Unlabeled Data. In ECAI 2023: 26th European Conference on Artificial Intelligence; Frontiers in Artificial Intelligence and Applications; IOS Press: Amsterdam, The Netherlands, 2023; Volume 372, pp. 764–771. [Google Scholar] [CrossRef] [Scilit]
  6. Furmańczyk, K.; Paczutkowski, K. A Proposal for PU Classification under Non-SCAR Using Clustering and Logistic Model. In Proceedings of the 22nd International Conference on Modeling Decisions for Artificial Intelligence (MDAI 2025); Torra, V., Narukawa, Y., Domingo-Ferrer, J., Eds.; Springer: Valencia, Spain, 2025. [Google Scholar]
  7. Dai, Y.; Li, X.; Wang, W.; Li, C.; Niu, G.; Sugiyama, M. Positive-Unlabeled Learning with Extreme Scarcity of Labeled Positives. In Proceedings of the 43rd International Conference on Machine Learning (ICML 2026), Seoul, Republic of Korea, 6–11 July 2026; Available online: https://icml.cc/virtual/2026/poster/61529 (accessed on 3 July 2026).
  8. Mielniczuk, J.; Rejchel, W.; Teisseyre, P. Prior Shift Estimation for Positive Unlabeled Data through the Lens of Kernel Embedding. arXiv 2026, arXiv:2502.21194v3. [Google Scholar] [CrossRef] [Scilit]
  9. Liu, S.; Chang, Y.; Cheng, M.; Tian, Q.; Li, P. In-Context Positive-Unlabeled Learning. arXiv 2026, arXiv:2605.05591. [Google Scholar] [CrossRef] [Scilit]
  10. Mansouri, F.; Zilles, S.; Ben-David, S. Active Learning from Positive and Unlabeled Examples. arXiv 2026, arXiv:2602.02081. [Google Scholar] [CrossRef] [Scilit]
  11. Ociepa, K.; Flis, Ł.; Wróbel, K.; Gwoźdźiej, A.; Kinas, R. Bielik 7B v0.1: A Polish Language Model—Development, Insights, and Evaluation. Comput. Sci. 2025, 26, 131–161. [Google Scholar] [CrossRef] [Scilit]
  12. Ociepa, K.; Flis, Ł.; Kinas, R.; Wróbel, K.; Gwoźdźiej, A. Bielik 11B v3: Multilingual Large Language Model for European Languages. arXiv 2025, arXiv:2601.11579. [Google Scholar] [CrossRef] [Scilit]
  13. Ociepa, K.; Flis, Ł.; Kinas, R.; Wróbel, K.; Gwoźdźiej, A. Advancing Polish Language Modeling through Tokenizer Optimization in the Bielik v3 7B and 11B Series. arXiv 2026, arXiv:2604.10799. [Google Scholar] [CrossRef] [Scilit]
  14. Penedo, G.; Kydlíček, H.; Ben Allal, L.; Lozhkov, A.; Mitchell, M.; Raffel, C.; Von Werra, L.; Wolf, T. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. In Proceedings of the 38th Conference on Neural Information Processing Systems (NeurIPS 2024), Track on Datasets and Benchmarks; Curran Associates: Red Hook, NY, USA, 2024. [Google Scholar]
  15. Emami Gohari, A.; Patel, D.; Bhattacharjee, B.; Belgodere, B.; Braz, F.A.; Chaudhary, V.; Dalmia, S.; Fadnis, K.; Ganti, R.; Ghosh, P.; et al. GneissWeb: Preparing High Quality Data for LLMs at Scale. In Proceedings of the 14th International Conference on Learning Representations (ICLR 2026); OpenReview: Vienna, Austria, 2026; Available online: https://openreview.net/forum?id=NRWUAo075J (accessed on 22 July 2026).
  16. Negoiță, V.-A.; Masala, M.; Rebedea, T. Improving Romanian LLM Pretraining Data Using Diversity and Quality Filtering. In Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026); Association for Computational Linguistics: Rabat, Morocco, 2026; pp. 140–148. [Google Scholar] [CrossRef] [Scilit]
  17. Li, R.; Wei, Y.; Zhang, M.; Yu, N.; Hu, H.; Peng, H. ScalingFilter: Assessing Data Quality Through the Lens of Scaling Laws. arXiv 2024, arXiv:2408.08310. [Google Scholar] [CrossRef] [Scilit]
  18. Nait Saada, T.; Béthune, L.; Klein, M.; Grangier, D.; Cuturi, M.; Ablin, P. The Data-Quality Illusion: Rethinking Classifier-Based Quality Filtering for LLM Pretraining. In Proceedings of the 14th International Conference on Learning Representations (ICLR 2026); OpenReview: Vienna, Austria, 2026; Available online: https://openreview.net/forum?id=vSBACt34gS (accessed on 22 July 2026).
  19. Langlais, P.-C.; Chizhov, P.; Arnett, C.; Hinostroza, C.; Nee, M.; Jones, E.; Girard, I.; Mach, D.; Stasenko, A.; Yamshchikov, I. Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training. In Proceedings of the 14th International Conference on Learning Representations (ICLR 2026); OpenReview: Vienna, Austria, 2026; Available online: https://openreview.net/forum?id=XeAvoKDOn0 (accessed on 22 July 2026).
  20. Wang, J.; Garg, S.; Schneider, A. Accessible, Realistic, and Fair Evaluation of Positive-Unlabeled Learning Algorithms. In Proceedings of the 14th International Conference on Learning Representations (ICLR 2026); OpenReview: Vienna, Austria, 2026; Available online: https://openreview.net/forum?id=5R11h5o44C (accessed on 22 July 2026).
  21. Furmańczyk, K.; Paczutkowski, K.; Dudziński, M.; Dziewa-Dawidczyk, D. Classification Methods Based on Fitting Logistic Regression to Positive and Unlabeled Data. In Computational Science—ICCS 2022; Lecture Notes in Computer Science; Springer: Cham, Switzerland, 2022; Volume 13350, pp. 33–46. [Google Scholar] [CrossRef] [Scilit]
  22. Hou, M.; Chaib-draa, B.; Li, C.; Zhao, Q. Generative Adversarial Positive-Unlabeled Learning. In Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI 2018); IJCAI Organization: Stockholm, Sweden, 2018; pp. 2255–2261. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  23. Wawrzeńczyk, A.; Mielniczuk, J. One-Class Classification Approach to Variational Learning from Biased Positive-Unlabeled Data. In ECAI 2023: 26th European Conference on Artificial Intelligence; Frontiers in Artificial Intelligence and Applications; IOS Press: Amsterdam, The Netherlands, 2023; Volume 372, Available online: https://home.ipipan.waw.pl/j.mielniczuk/VAEPUCC.pdf (accessed on 22 July 2026).
  24. SpeakLeash Team. SpeakLeash: Agnostic Dataset Library for the Polish Language, Version 0.3.51; PyPI: Warsaw, Poland, 2024. Available online: https://pypi.org/project/speakleash/ (accessed on 3 July 2026).
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Article Metrics

Citations

Article Access Statistics

Multiple requests from the same IP address are counted as one view.