LLM Post-Training to Enhance Knowledge Extraction from Specialist Domains: Teaching LLMs User Forum Creole

Abstract

Frontier large language models achieve broad linguistic competence but degrade on specialist domains underrepresented in pre-training corpora. Domain-adaptive post-training (DAPT) on curated professional text partially closes this gap, yet the dominant approach flattens structured discourse into isolated document units, discarding the collaborative reasoning embedded in multi-party exchanges. This paper investigates whether preserving the full recursive structure of user forum threads during post-training is a more effective first step toward knowledge extraction than flattened question-answer pairs. Four open-source decoder-only models (TinyLlama 1.1B, Phi-2 2.7B, LLaMA-2-7B 6.8B, LLaMA-2-13B 13B) are post-trained using parameter-efficient LoRA adaptation on 4970 threads from AgTalk, an agricultural producer forum, under three conditions: flattened Q → A pairs, full recursive threads preserving reply order, and shuffled recursive threads with randomly permuted intermediate replies. Five hypotheses are tested through paired Wilcoxon signed-rank comparisons across 29 thread positions. DAPT significantly reduces perplexity relative to pretrained baselines across all architectures (H0 supported). Recursive training reduces perplexity relative to flattened training, an advantage clearest for the two LLaMA-2 models under matched-context evaluation (Wilcoxon win rates near 72%) and present but obscured by outlier variance at the 1.1B and 2.7B scales (H₁ supported). However, ordered recursive training provides only a marginal advantage over shuffled (H₂ inconclusive), attention analysis reveals identical U-shaped endpoint-weighted profiles regardless of training condition (H₃: architectural not learned), and perplexity shows no systematic decrease with accumulating thread depth (H₄ not supported). These results are attributed to Rotary Position Embedding constraints in decoder-only architectures that systematically underweight middle-thread content. Encoder–decoder architectures with bidirectional cross-attention are identified as a promising next step for exploiting the full collaborative structure of forum discourse.

1. Introduction

Historically, advances in computer architecture (IBM 360 → Intel 8086 → NVIDIA H100) have focused on relieving humans of quantitative clerical, and record-keeping burdens, transforming structured workloads from paper files and ledgers into programmable digital systems [1]. Catalyzed by the internet’s information revolution, digital systems delivered major efficiency gains through scalable data processing, lower transaction costs, and faster coordination across networked workflows [2,3]. Global billion-dollar tech industries such as SaaS have been created, serving business, technical, and scientific domains [4]. While reaching new levels of autonomy, these digital systems remained primarily open-loop, requiring linguistic, creative, and interpretive skills of the human operator to provide input code, model development, execution, evaluation, etc. [5]. The traditional human dominance over its technology was maintained and highly rewarded for entrepreneurship, i.e., the creation of the Silicon Valley tech giants, and their venture-capital backing of the AI revolution [6].

The development of frontier large language models (LLM) has revolutionized the role of linguistics in the digital automation era [7,8]. Generative AI has transformed the human–digital relationship into a new paradigm envisioned by early AI pioneers [9]. NLP powered computers are increasingly capable of performing human level linguistic tasks, closing the digital loop, and threatening to place humans in a subservient role to silicon technology [10]. Current downward trends in SaaS and related white-collar employment, once considered a safe haven where human-capital was able to outpace digital advances, highlights the human–AI agent displacement reality in the workplace and beyond [11].

As humans begin to yield critical functions to NLP systems, trust and reliability in their relationship as a technical peer elevates [12,13]. Accuracy of CPU processing has rarely been challenged when confined to numerical tasks. With roles-reversed, NLP systems are expected to provide qualitative solutions that require high level of linguistic competency. The lure of these systems is the autonomous “closed loop” freeing up human resources, particularly in repetitive task areas such as customer service, financial reporting, etc. In more technically demanding settings, LLM-based agents are increasingly expected to function as domain-aware decision-support systems, providing diagnostic and analytical assistance to physicians, scientists, engineers, and other specialists working in high-complexity environments. In these emerging trends towards specialized knowledge extraction, AI isn’t just a search engine: it’s the decision-maker and executor sitting on both sides of the computer.

The near legion of recent LLM research has identified that frontier LLMs “only know what they know”, i.e., specialized domain corpora are typically not contained in the frontier LLM training [14]. Frontier LLM performance when tasked to provide expert and technically challenging questions, continue to fall short compared to expected benchmarks (see Section 2.1 below). Underlying these benchmark shortfalls is a more fundamental linguistic problem: a model that cannot fluently predict the next token cannot reliably perform downstream tasks such as knowledge extraction or reasoning based on that text either [15]. Perplexity, the standard measure of next-token prediction accuracy, is therefore the first-order check of whether a model has absorbed a domain, and the metric that has to improve before any downstream gains can be expected [16]. Even for the human operator, learning technical languages and terminology often takes years of formal training and on the job experience. Many technical communities in essence create their own “Creole”, a blend of native language (e.g., English) infused with exotic phrasing—often slang. The exotic nature of such Creole “surprises” the LLM decoding attention mechanisms, creating weak downstream task performance and, likely, hallucinations [17].

Transformer-based AI was originally designed for encoder–decoder translation between natural languages, proving the architecture could learn unfamiliar linguistic patterns [18]. Retraining an LLM in the vernacular is impractical since the standard language is still dominant. In its place, the research community has developed a handful of post-training techniques, including adapters, prefix tuning, and Low-Rank Adaptation (LoRA), requiring only minimally invasive retuning of model parameters, which can infuse the vernacular as a specialized form of corpora [19,20,21]. The BERT suite of models has demonstrated this on legal, biomedical, and scientific corpora: each a specialized vernacular distinct from the general standard language (see Section 2 below). Hu et al. [19] developed a post-training procedure (LoRa) that while requiring the retraining of less than one percent of LLM parameters maintains strong downstream task performance. Hu et al.’s [19] contribution is foundational: it provides a strong work around to the post-training solution which would otherwise be unavailable except to the frontier.

Post-training LLM reinforces the need for linguistic clarity: researchers typically report the sequential improvement of both perplexity followed by downstream tasks [22]. Sankar et al. [22] used per-token perplexity as their sole metric to study how transformer dialog models respond to perturbations of conversation history, framing it as a measure of ‘how the learned probability distribution behaves’ under structural manipulation. This paper applies the same metric for a similar purpose: measuring how a post-trained model’s perplexity responds to different forum thread representations, e.g., Q-A pairs vs. recursive vs. shuffled. Extracting knowledge from parochial community user forums has an even greater need for fluency since the agentic LLM will need to hear (decode) the native language through user prompt, as well as speak back to the user in the same native language. Mining technical information will involve slang and highly abbreviated phrasing, including abbreviations and conventions not included in stand language. Figure 1 illustrates the challenges when post-training user forums discussed in the next Section DAPT Challenges in Training User Forums.

DAPT Challenges in Training User Forums

Figure 1 lays out five design properties where DAPT on user forum corpora differs from DAPT on peer-reviewed corpora. Training unit. DAPT trained on user forum threads requires reconstruction into alternative forms to consider possible intra-thread dependencies that drive model performance. No such restructuring is typically needed for DAPT trained on peer-reviewed documents such as PubMed or court opinions. Context window. Post-training user forums—when full recursive threads are trained—can extend up to 4096 tokens, pushing the model into long-context territory, overwhelming the decoder attention mechanisms. BERT’s native context window of 512 tokens is much shorter, training only abstracts and short briefs, seldom encountering context length constraints. Training objective. Peer-reviewed DAPT trained encoders for downstream classification tasks. Evaluation was straightforward: F1 scores tally how often the model picked the right answer from a fixed set of categories. This paper trains decoders to generate forum-appropriate text token by token. Perplexity is based on how probabilistically “surprised” the decoder’s next-token prediction is compared to the actual token. As context grows longer, that surprise becomes harder to minimize. Register and vocabulary. Editorial DAPT corpora are written by trained professionals who have access to editors: vocabulary, spelling and grammar are standardized and symbology properly formatted. The tokenizer was designed for this kind of text and handles it cleanly. User forums lack that type of editing and formatting, and vernacular extends into slang. The tokenizer fragments these corpora unnaturally, challenging the decoder’s perplexity performance. Corpus scale. Editorial-based DAPT corpora are large, e.g., PubMed has tens of millions of abstracts, providing adequate data for model training and strong downstream F1 task performance. User forum corpora can be more limited in size, and its poorer quality content increases tokenization costs.

The previous issues are distilled into three additional training challenges unique to DAPT on user forum threads. First is whether threads should remain in “long form” rather than simpler Q-A pairs: a trade-off between the added signal from the full thread and the added computing cost and potential issues with context length overshooting the decoder architecture. A second is whether order matters: do users respond to one another sequentially? A third is attention: even if thread order does matter, can decoder attention architectures capture the recursive signal? Liu et al. point out that the RoPE mechanism used in decoders produces a “lost in the middle” effect, making it difficult to pick up recursiveness (see Section 2.2 below).

This paper post-trains four open-source decoder-only LLMs on AgTalk, a long-running agricultural producer forum where producers post technical problems and peers reply with diagnostic and operational advice. The aspirational goal is to start extracting that practitioner knowledge from the forum’s threads: this paper begins this knowledge extraction process by establishing an LLM fluent with the forum’s Creole. Figuratively, could the LLM post an understandable reply on the forum that sounds like it came from another producer? Operationally, this paper identifies the optimal thread structure that generates the top performing LLM as measured by perplexity, the standard evaluation metric used in decoder-based frontier LLMs.

The main experimental treatment is how the threads are structured during training: as flattened Q-A pairs, as full recursive threads with replies in their original order, or as recursive threads with the replies shuffled:

(i)

Flattened: B = {(Q, A_i)} that reproduces standard Q-A paired training

(ii)

Recursive: R = {(Q, A₁, …, A_i−1) → A_i} preserves the cumulative conditioning context

(iii)

Shuffled: R_s = {(Q, rand[(A₁, …, A_i−1)] → A_i} applies a random permutation to (A₁, …, A_i−1) while keeping Q and A_i at original positions, isolating chronological ordering effect from contextual presence.

The three treatments are designed to isolate three distinct effects of thread structure during training. Together they generate five testable hypotheses:

• H₀ (NULL): DAPT post training lowers perplexity compared to pre-trained base LLM. If supported, post-training and the additional thread-structure hypotheses are warranted.
• H₁ (Context): Post-training on recursive thread contexts produces lower perplexity on held-out forum replies than post-training on flattened Q-A pairs. That is, giving the model access to the prior replies during training produces a better fit to new thread content than training on question–reply pairs alone.
• H₂ (Order): Within recursive training, the chronological order of the intermediate replies carries training signal beyond simply having those replies in the context. That is, it matters not just whether the model sees the prior replies, but whether it sees them in the order they were actually written.
• H₃ (Attention): When a recursively trained model predicts a reply, its attention reflects something the model learned about forum discourse rather than the endpoint bias that decoder-only transformers are known to exhibit on long contexts regardless of training. That is, if the model attends strongly to the question and the most recent reply, that pattern should come from training on recursive threads, not from RoPE alone.
• H₄ (Accumulation): As more replies accumulate in a thread, each new reply becomes easier to predict because the growing context provides more information. That is, reply A_n+1 should have lower perplexity than reply A_n within the same thread.

All three treatments are applied to four open-source decoder-only base models spanning an order of magnitude in parameter count, namely TinyLlama 1.1B [23], Phi-2 2.7B [24], LLaMA-2-7B [25], and LLaMA-2-13B [25], using parameter-efficient LoRA adaptation [19] on a corpus of 4970 quality-filtered AgTalk threads.

Section 2 situates this work within the literatures on domain-adaptive post-training, forum discourse representation, and long-context decoder behavior. Section 3 describes the AgTalk corpus construction and preprocessing pipeline. Section 4 specifies the experimental design, training conditions, and evaluation metrics. Section 5 reports results for H1 through H4. Section 6 discusses implications and limitations. Section 7 concludes.

2. Related Work

2.1. The Specialist-Knowledge Gap in Frontier LLMs

General-purpose large language models (e.g., GPT-4, Claude, and Gemini) are Herculean when tasked with the kind of broad questions a well-educated generalist would handle, but continue to fall short of domain-expert performance on specialized technical questions [7,8]. GPQ-A, a benchmark of 448 graduate-level questions authored by PhD domain experts, shows the magnitude of this gap directly: domain PhDs reach 65 percent accuracy on questions within their own specialty, while GPT-4 with chain-of-thought prompting reaches only 39 percent [26]. MMLU-Pro, the latest professional-level benchmark, evaluates over 12,000 reasoning-intensive questions across 14 academic and technical domains [27]. It reveals that leading models like GPT-4o, Gemini-1.5-Pro, and Claude-3-Opus achieve an average accuracy of approximately 70% on graduate-level and professional-practice questions—a disappointing performance compared to human expectations.

2.1.1. Specialist-Knowledge Gap and Corpus Coverage

Kandpal et al. explain part of this expert knowledge gap simply: LLMs “only know what they know” [28]. While the scaling effect from ChatGPT-2 through ChatGPT-4 was met with exuberance, LLMs cannot perform magic on content their pretraining did not cover. Kandpal et al. show this empirically: factual Q-A accuracy on BLOOM-176B rises from 25 percent to above 55 percent as the count of pretraining documents containing the relevant entities grows from 10 to 10,000 [28]. Specialist domains whose terminology, equipment, and practices appear rarely in general web text therefore leave the model’s parameters weakly fit to that content. While few-shot prompting has been shown to improve LLM performance on downstream tasks where the relevant patterns were learned during pretraining [7], attempting to add required corpora through multi-shot prompting cannot be expected to substitute for the underlying parametric knowledge.

2.1.2. Domain Adaptive Post-Training (DAPT)

Domain-Adaptive Post-Training (DAPT) continues the model’s pretraining objective on a domain-specific corpus, adapting it to the target domain after general pretraining is complete (Figure 2). The literature splits along two lines: encoder-only adaptations like the BERT family, where full fine-tuning is computationally feasible, and larger LLMs, where parameter-efficient methods such as LoRA [19] make the same approach tractable (see Section 2.1.3). Gururangan et al. [29] provide the clearest demonstration of DAPTs performance gains. Their influential “don’t stop pretraining” finding showed that even strongly pretrained models can still benefit from additional in-domain training. Using RoBERTa across four domains (biomedical papers, computer science papers, news, and reviews) and eight downstream classification tasks, they reported sizable reductions in held-out masked-language-model loss (1.32 to 0.99 in biomedicine, 1.63 to 1.34 in computer science, 2.10 to 1.93 in reviews), along with accuracy gains on the downstream tasks. The core finding is that continued pretraining delivers measurable gains when the pretraining text and the downstream evaluation text come from the same domain; pretraining on biomedical papers improves performance on biomedical tasks but not on unrelated ones.

Five encoder-based DAPT studies have taken general-purpose BERT models and continued their pretraining on domain-specific corpora: BioBERT on PubMed [30] , LEGAL-BERT on legal text [31], SciBERT on scientific papers [32], AgriBERT on USDA food-nutrition text [33], and BERTOverflow on Stack Overflow posts [34]. These models function as retrieval engines for technical corpus chunks drawn from published documents in sensitive professional domains (biomedical, legal, scientific) where frontier-lab LLMs have not been able to provide reliable retrieval. All five report improved performance on downstream tasks within their respective domains, establishing that DAPT reliably improves a model’s fit to specialized text. Whether the same approach transfers to a generative decoder-only setting is the question this paper takes up.

2.1.3. Parameter-Efficient Fine-Tuning (PEFT)

Domain-Adaptive Post-Training of large decoder-only LLMs is computationally infeasible for most researchers if all model parameters are updated. Hu et al. [19] introduced LoRA, a parameter-efficient fine-tuning (PEFT) method that freezes the base model weights and inserts small low-rank adapter matrices alongside the attention projections, updating only a tiny fraction of the original parameter count. Tested on GPT-3 175B and GPT-2 Medium across natural language understanding and generation benchmarks (WikiSQL, MultiNLI, SAMSum, E2E NLG), LoRA matched or slightly exceeded full fine-tuning performance-for example, 73.8% vs. 73.0% on WikiSQL and 91.7% vs. 89.5% on MultiNLI for GPT-3. Computing requirements during training are reduced roughly threefold, and after training, the adapter matrices can be merged into the base LLM model generating output at rates no slower than running the original model. This paper applies LoRA at rank r = 16 to four open-source decoder-only LLMs (TinyLlama 1.1B, Phi-2 2.7B, LLaMA-2-7B, LLaMA-2-13B), making the DAPT comparisons in Section 4 and Section 5 tractable on a single high-performance computing cluster.

2.2. Perplexity

Perplexity is the standard evaluation metric for language models, the linguistic equivalent to the numerical SSE, measuring how accurately a model predicts the next token in a sequence. Foundational LLM papers report perplexity as a primary indicator of model quality. On LAMBADA, a benchmark of long-range word prediction where lower perplexity reflects better contextual understanding, GPT-2 reached a zero-shot perplexity of 8.63 [35], a substantial improvement over earlier RNN-based models that scored above 100. GPT-3 Brown et al. [7] cut it further to 3.00, demonstrating that scale alone produces meaningful perplexity reductions on the same benchmark. Within the DAPT literature, perplexity reductions are typically smaller in absolute terms because the starting model is already well-fit to general language. Chalkidis et al. [31] report drops of 1.1 to 5.6 perplexity points across three legal subdomains for LEGAL-BERT compared to base BERT, with downstream F1 gains commensurate with the perplexity reductions.

2.3. Forum Structure, Conversational Context, and Flattened Q-A Representations

Early forum and community Q-A work by Cong et al. [37] developed methods to extract (Q, A) pairs from general forum threads using an unsupervised graph-based ranker for candidate answers, culling non-related threads. Wang et al. [38] kept the forum thread structure intact instead of flattening it into (Q, A) pairs. Their research investigated who in a thread was replying to whom, building a reply tree from the conversation. That reply tree incorporating intra-thread conversation turned out to be useful: their search engine that took the recursive structure into account returned better-ranked search results compared to a baseline that ignored conversation structure on three technical domain forums (genealogy, tech support, and programming forums). Their work is about searching forums, not post-training a model on them, but the underlying finding lines up with what this paper does: the way replies connect to each other in a thread carries information that gets thrown away when you flatten the thread.

Shah and Pomerantz [39] tested 21 features for predicting which answer to a Yahoo Answers question would be selected as best by the asker. The single strongest predictor was reciprocal rank: essentially where the answer sat in the list of replies. A logistic regression model using reciprocal rank alone achieved 80.34% accuracy in predicting which reply the asker chose as best. The relevance to this paper is direct: Shah and Pomerantz found that at the inference stage, reply position within a thread carries most of the predictive signal about which reply the asker selected as best. This paper asks a parallel question, whether reply position within a user forum thread matters: does position carry significant training signal when a generative model is post-trained on the thread content?

Anderson et al. [40] also identified that the entire thread matters, and that the order of replies carries meaning. On Stack Overflow, the most experienced users typically respond first, followed by less experienced users whose contributions tend to be discounted by the community. This suggests that when post-training on a user forum, maintaining the recursive thread structure is worthwhile. The same pattern can be expected in technical forums like AgTalk: initial replies are likely to come from users most confident of an answer, followed by less experienced users who may build on the earlier replies but whose contributions are typically of lower quality. Preserving the original reply order during training preserves this implicit quality signal that is otherwise diminished when threads are flattened into independent Q-A pairs.

Sankar et al. [22] tested whether neural dialog models actually use the structure of community-style conversations-discussions where replies are not just responses to the original question but engage with successive replies to reject, correct, or augment what came before. They corrupted the conversation history at test time in nine different ways (shuffling utterances, reversing them, dropping them, shuffling words within utterances, reversing word order, dropping nouns or verbs) across four dialog datasets, and measured how perplexity changed in response. Their findings were that models barely noticed: perplexity stayed about the same even under extreme perturbations. Transformer models were particularly insensitive, more so than recurrent models with attention. Sankar et al.’s study applies directly to this paper: does preserving the original recursive thread order matter, suggesting that replies give attention not only to the original Q but also to previous replies-or, if shuffled, would they act more or less as simple Q-A pairs?

A related question is whether the order of training examples matters during model training, and this has been studied under the name curriculum learning. This is the standard practice of deliberately ordering training examples from easy to hard, on the assumption that the model can better understand a planned sequence the way a student learns from prior lessons. Campos [36] tested several such orderings for ELMo pretraining (LSTM model) on WikiText-2 and WikiText-103 and found no significant evidence that curriculum orderings improved model pretraining. The result is relevant to this paper because the underlying assumption is structurally similar. Curriculum learning assumes the model benefits from a planned ordering across examples analogous to the recursive training condition (H1) and assumes the model benefits from access to the cumulative thread context (Q, A₁, …, A_i−1) when predicting the next reply. The shuffle comparison (H2) assumes the model benefits specifically from the chronological order of those prior replies, where reply A₄ may build on A₃ and A₅ may build on both.

Among the seven prior studies reviewed, three (Anderson et al. [40], Shah et al. [39], Wang et al. [38]) found that reply order or thread structure carried meaningful signal. The remaining four (Cong et al. [37], Yang et al. [41], Sankar et al. [22], Campos et al. [36]) either flattened thread structure or found that models were insensitive to ordering. None of this prior work tested whether reply order within a single forum thread carries training signal for a decoder-only language model during post-training. That is the gap this paper fills, with the recursive (H1) and shuffle (H2) comparisons designed to test whether the model can extract within-thread sequential signal that earlier work has shown is hard for models to use even when it is present.

2.4. Attention Interpretation

Recent long-context work by Liu et al. [42] documents a U-shaped performance curve in decoder-based language models: when relevant information is placed in the middle of a long context, accuracy on retrieval tasks drops by more than 20 percentage points compared to placement at the beginning or end. Zhang et al. [43] subsequently attributed this U-shape, in models that use Rotary Position Embedding (RoPE), to RoPE’s long-term decay property—the encoding gives more weight to nearby tokens and less to distant ones. As discussed in the Introduction, this is an important consideration for forum threads, where the original question lies at the beginning of the context and the most recent reply lies near the end. The recursive threads in this paper reach up to 4000 tokens, placing them squarely in the long-context regime. Factual content within a thread is likely to be distributed across all replies; if endpoint-favoring attention dominates, the model may fail to absorb the content carried by intermediate replies.

Clark et al. [44] analyzed BERT’s attention patterns and found specific heads encoding syntactic relations: head 8–10 connects direct objects to their verbs, and a separate head connects determiners to their nouns at 94.3% accuracy. These findings establish that attention can be examined directly to identify what kinds of linguistic structure a model has absorbed. Vig and Belinkov [45] performed a parallel analysis on GPT-2 and found that different layers specialize in different syntactic patterns, with middle layers most predictive of dependency relations and deepest layers capturing the longest-range relationships. This paper applies the same kind of attention analysis to recursively trained decoder-only LLMs (H3), examining whether the trained models attend to within-thread structure or default to RoPE-driven endpoint bias.

3. Data Constructions

3.1. Data Collection and Preprocessing

Forum data was collected through systematic web scraping of publicly accessible threads using a Python 3.11 script with the requests library and BeautifulSoup for HTML parsing. AgTalk (talk.newagtalk.com date accessed 26 June 2026) is a free, openly available agricultural producer forum that does not require user registration to view threads, has no paywall, and serves all content through standard public HTTP requests. The corpus was restricted to the Machinery Talk subforum of AgTalk, with threads scraped from 1 April 2023 to 1 November 2025. The scraper accessed each thread’s display URL the same way an ordinary web browser would, with no login credentials, authentication tokens, API keys, or back-end access used. No security measures were circumvented and no rate limits were bypassed. A short delay between requests was used to avoid placing undue load on the server.

The scraping process preserved full thread structure including original questions, all responses, author identifiers, timestamps, and thread metadata (subforum category, view counts). From the complete archive, we extracted 4970 threads meeting the following quality criteria: (1) minimum three responses to enable comparison across training conditions; (2) total thread length within 2048 tokens to fit model context windows; (3) English language content; (4) no deleted or moderated posts that would break conversational coherence. These filters ensure that all threads represent complete, naturally occurring discussions suitable for studying recursive structure. These filters proved highly selective, yet preserved the vast majority of collected threads: 4970 threads (99.4%) passed all quality criteria. AgTalk discussions are, therefore, predominantly complete, coherent, and suitable for studying recursive structure. This high retention rate indicates that the forum maintains strong conversational quality with minimal thread abandonment or moderation.

The final corpus comprises 4970 threads with the following thread characteristics:

• Mean responses per thread: 8.3 (SD = 5.7);
• Range: 3 to 40 responses;
• Mean tokens per thread: 1247 (SD = 534);
• Mean tokens per response: 153 (SD = 98);
• Vocabulary size: 47,322 unique tokens;
• Agricultural-specific terms: 3847 tokens (identified as terms appearing ≥10 times in AgTalk but <1% frequency in general web corpora).

3.2. Computational Requirements

The 4970 LLaMA-2 model threads yield substantial training data when the recursive thread structure is preserved. With a mean of 8.3 responses per thread, the Recursive-Full condition generates approximately 41,251 training examples (4970 threads × 8.3 responses), the same as in the baseline condition given the pairwise comparisons. This recursive expansion of training examples from fixed threads represents an additional benefit of structure-preserving preprocessing but also required substantial computational resources. Training and subsequent attention analysis took nearly 72 h of HPCC time across all three LLM models (TinyLlama, Phi-2, LLaMA-2-7B, and LLaMA-2-13B). The sample size of 4970 threads represents a pragmatic balance between statistical power and computational feasibility, which was approached even with access to the Oklahoma State HPCC. Scaling to larger corpus sizes would require proportionally greater computational investment.

4. Experimental Design

The experimental design isolates the effect of thread representation during post-training by varying the quantity and ordering of the context preceding a target reply. Threads were partitioned into training, validation, and test sets at the thread level in an 80/10/10 split. Only thread content is supplied to the model; author names identified in the scraping process are not used.

4.1. Training and Testing Conditions

Three training conditions are compared, all predicting the same held-out reply A_n from a thread of n replies. The conditions differ in what conversational context is provided during training.

Flattened (B): isolated question–reply pairs. Each reply is predicted from the original question alone:

$$P_B(A_i\mid Q),i=1,2,\dots ,n$$

This reproduces the standard preprocessing step where forum threads are reduced to (Q, A_i) pairs and prior replies A₁, …, A_i−1 are discarded.

Recursive (R): full chronological thread context. Each reply is predicted from the original question together with all preceding replies in their original chronological order:

$$P_R(A_n\mid Q,A_1,A_2,\dots ,A_{n-1})$$

This preserves the cumulative discourse structure of the thread.

Shuffled (R_s): full context with permuted reply order. The intermediate replies A₁, …, A_n−1 are randomly permuted while Q and A_n remain at their original endpoint positions:

$$P_{R_s}(A_n\mid Q,\pi (A_1,A_2,\dots ,A_{n-1}\left)\right)$$

where π is a uniform random shuffling of the n − 1 intermediate replies. The model receives the same prior content as in the recursive condition, but loses the original chronological ordering. Comparison between R and R_s isolates the effect of reply order from the effect of contextual presence.

Testing conditions. Each trained model is evaluated by predicting held-out reply A_n on test threads with structure as follows. The Flattened condition is tested two ways. The first test presents the model with the same isolated (Q → A_n) context used during training. The second test presents the model with the full recursive context (Q, A₁, …, A_n−1) when predicting the target A_n. This can be considered a “context-equivalence” scenario to determine if the recursive structure is benefiting from context length rather than context ordering.

The Recursive and Shuffled conditions are both tested under the full recursive context (Q, A₁, …, A_n−1) when predicting the target A_n, maintaining the original chronological order. Holding the test context identical across these two conditions isolates the effect of training-time order: any perplexity difference between Recursive-trained and Shuffled-trained models on the same chronological test context can be attributed to training condition rather than test condition.

4.2. Model Selection and Training Procedure

Training experiments are conducted using four open-source base models spanning an order of magnitude in parameter count (

Table 1. Model architectures and training parameters.

Model	Total Params	Trainable (LoRA)	% Trainable	GPUs	VRAM	Training Time
TinyLlama	1.1 B	4.5 M	0.41%	1 × RTX 6000	24 GB	~5 h
Phi-2	2.7 B	7.9 M	0.28%	1 × RTX 6000	24 GB	~11 h
LLaMA-2-7B	6.8 B	16.8 M	0.25%	2 × RTX 6000	48 GB	~48 h
LLaMA-2-13B	13 B	26.2 M	0.20%	2 × RTX 6000	48 GB	~72 h

): TinyLlama, developed by Singapore University of Technology and Design [23]; Phi-2, developed by Microsoft [24]; and LLaMA-2-7B & LLAMA-2-13B (Touvron et al., 2023), developed by Meta [25]. All three models were selected for their strong performance on language tasks, open availability enabling reproducible research, and compatibility with parameter-efficient fine-tuning methods. TinyLlama contains 1.1 billion parameters, Phi-2 reaches 2.7 billion, and LLaMA-2 models comprise 6.8 billion and 13 billion (Table 1). This range of LLM size is used to identify any scaling effects associated with perplexity training.

Full fine-tuning of billion-parameter language models requires updating all model weights during training, a computationally prohibitive approach except at the frontier labs. To fully fine-tuning Phi-2 (2.7B parameters) would have required storing model weights, gradients, and optimizer states simultaneously, exceeding 80GB of GPU memory. Even with access to the Oklahoma State University Pete High Performance Computing Center (HPCC), equipped with RTX 6000 GPUs (24GB VRAM), full fine-tuning of models at this scale was infeasible. To reduce the computational burden, post-training used Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning method that holds the pre-trained model weights fixed [19]. LoRA injects a small set of trainable parameters to the attention projection matrices in each transformer layer, which adjust how the model attends to domain-specific tokens during inference. The frozen weights carry everything the base model already learned from web-scale pretraining; the added parameters carry the domain adaptation learned from the AgTalk corpus. The literature supports the fact that the computational easing maintains acceptable training accuracy. Hu et al. [19] demonstrated that LoRA performs on par with, or better than, full fine-tuning on RoBERTa, DeBERTa, GPT-2, and GPT-3, despite training fewer than 1% of total parameters. Using LoRA reduced trainable parameters to less than 0.5% across all models that still required substantial computing resources (Table 1). The HPCC was pushed to its practical limits: training LLaMA-2-13B required allocating all available GPUs on a single node and approximately 72 h of continuous computation per training condition.

Training hyperparameters were held constant across all conditions: learning rate of 2e-4 with cosine annealing, LoRA rank of 16 with alpha of 32, batch size of 4 with gradient accumulation over 4 steps, and training for a single epoch with early stopping on validation loss. We applied LoRA adapters to the query, key, value, and output projection matrices of the attention mechanism (q_proj, k_proj, v_proj, o_proj). Attention analysis was conducted on the trained models during inference by extracting attention weights from the final layer. All experiments are conducted on the Oklahoma State University Pete HPCC using RTX 6000 GPUs (24GB VRAM).

All model training and attention extraction were implemented using the HuggingFace Transformers library [46] with PEFT (Parameter-Efficient Fine-Tuning) [47] for LoRA integration and PyTorch 2.5.1 (CUDA 12.1 build) [48] as the underlying deep learning framework. For LLaMA-7B, we employed 4-bit quantization using BitsAndBytes [49] with NF4 quantization type and double quantization to reduce memory footprint, enabling training on RTX 6000 GPUs. The LoRA configuration specified rank r = 16, alpha = 32, dropout = 0.05, targeting four projection matrices: query, key, value, and output (q_proj, k_proj, v_proj, o_proj). Training used the paged_adamw_8bit optimizer with gradient checkpointing to further reduce memory requirements.

Attention weights were obtained during inference by accessing the model’s attention tensors from the final transformer layer with output_attentions = True. For each target answer position, we computed attention mass allocated to each input segment (Q, A₁, A₂, …, A_n−1) by summing attention weights over the token positions corresponding to each segment, then normalizing to percentages. This user reply segmentation exploited our explicit [QUESTION] and [RESPONSE N] formatting, enabling clean identification of segment boundaries. Attention analysis aggregated weights across attention heads using mean pooling. Statistical significance testing for attention patterns employed paired Wilcoxon signed-rank tests implemented via the SciPy Wilcoxon function, comparing adjacent segments (e.g., Q vs. A1, A1 vs. A2) to assess Q-anchoring decay and recency bias patterns.

4.3. Evaluation Metrics and Hypothesis Testing

We evaluated models and hypotheses using two categories of metrics: perplexity, a standard language modeling metric, and an attention-based measure (

Table 2. Evaluation metrics, hypotheses, and hypotheses test outcomes.

Metric	Definition	Hypothesis	Outcome
Paired perplexity-Wilcoxon signed-rank	Win rate: recursive-trained vs. baseline-trained, both evaluated on full recursive test context	H0: Does post-training on AgTalk threads produce lower held-out perplexity than the general-purpose base model: is domain adaptation necessary to absorb practitioner discourse?	Supported. Post-trained perplexity is substantially lower than base-model perplexity across all four model scales.
“  “	“ “	H1: Does training on full recursive thread context produce lower perplexity than training on isolated Q → A pairs?	Supported. Win rates of 72.6% (LLaMA-7B) and 72.2% (LLaMA-13B).
“  “	“ “	H2: Does sequential order of intermediate replies carry additional learning signal beyond mere context presence?	Inconclusive. Positive effect with scale but not statistically significant.
Segment-level attention weight vector and Attention Ratio ARn	Proportion of attention allocated to Q, A1 … An−1 when predicting An; compared between recursive-and shuffled-trained models	H3: Does the U-shaped attention pattern reflect genuine semantic Q-anchoring or RoPE positional artifact?	Inconclusive. Attention shape indistinguishable between recursive and shuffled conditions, indicating RoPE rather than learned discourse exploitation.
Consecutive position Wilcoxon signed-rank	Paired perplexity comparison between adjacent positions An and An+1 within same threads	H4: Does perplexity decrease as cumulative thread context accumulates within fixed-length threads?	Not supported. 0/18 consecutive pairs reach significance across all four models; win rates hover near 50%.

).

Perplexity. Perplexity measures how well a language model predicts text during the training process. Operationally, perplexity quantifies the probability of correctly predicting each token given the preceding tokens in the context window. A model that consistently assigns high probability to the correct next token will produce low perplexity, while a model that is frequently surprised by the actual next token will produce high perplexity.

Attention Ratio (AR). Recent work on transformer interpretability has demonstrated that analyzing attention weight distributions can reveal what linguistic structures and contextual elements models prioritize during predictive text generation [40,41]. In transformer self-attention, each token attends to all preceding tokens in the context window by computing a weighted sum over value vectors, where the weights are determined by the scaled dot-product of query and key vectors followed by softmax normalization:

$$AR(Q,K,V)=softmax\left({\displaystyle \frac{{QK}^T}{\sqrt{d_k}}}\right)V$$

(1)

where Q = query matrix, K = key matrix, V = value matrix, d_k = dimension of key vectors.

The resulting attention weights sum to 1.0 across all source tokens for each target token, representing the proportion of attention allocated to each position. To characterize how models distribute attention across forum thread segments, we extract attention weights from the final transformer layer, averaging across all attention heads. For each target position A_n, we compute the full segment-level attention weight vector:

$$w_n=(w_{n,Q}, w_{n,A_1}, w_{n,A_2}, \dots , w_{n,A_{n-1}})$$

(2)

where each element $w_{n,s}$ represents the proportion of total attention allocated to source segment $s$ when predicting $A_n$. The $w_n$ equation is computed by summing token-level attention weights within each segment and normalizing so that ${\sum }_s{w}_{n,s}=1.0$.

This produces a complete attention profile across all thread segments for each target position. To summarize the overall balance between Q-anchoring and prior reply attention across the full thread context, we define the Attention Ratio (AR) as:

$${AR}_n={\displaystyle \frac{w_{n,A_1}+w_{n,A_2}+\cdots +w_{n,A_{\mathrm{n}-1}}}{w_{n,Q}}}$$

(3)

where $w_{n,s}$ are the segment-level attention weights defined in $w_n$ above. AR_n directly quantifies the relative allocation of attention between the accumulated prior replies and the original question when predicting A_n.

Table 2 specifies the formal test for each hypothesis. For H₀, the pretrained base model is evaluated on the same full recursive test context used for the post-trained comparisons. The analyses share a common evaluation protocol: for H₁, H₂, and H₃, both models in each comparison are evaluated on the identical full recursive test context at inference time, isolating the effect of training condition alone from any context-window effects, including shuffling. For H₄, consecutive-position comparisons are restricted to threads that contain both positions being compared. At each answer position with sufficient sample size (n ≥ 10), paired Wilcoxon signed-rank tests are applied and statistical significance is assessed at α = 0.05. Results are stratified by answer position (A2 through A30) and by model architecture (TinyLlama 1.1B, Phi-2 2.7B, LLaMA-2-7B, LLaMA-2-13B) to assess whether advantages vary with conversational depth and model scale. For H₃ specifically, two decay patterns are tested in addition to the main attention-profile comparison: Q-anchoring decay (attention decreasing from Q outward) and recency bias decay (attention decreasing backward from the most recent reply). Both are tested with Wilcoxon signed-rank tests at α = 0.05.

4.4. Downstream Task Evaluation: Generating the Next Forum Reply

Lower perplexity does not necessarily translate into better performance on a downstream task. A model can predict the next token more accurately without producing a more useful reply, so the perplexity reductions established in H0–H4 do not by themselves show that recursive post-training improves the task practitioners actually care about. To test this directly, a downstream task was evaluated: generating the next reply in a user forum thread-the task the forum itself performs.

In this downstream task, each model was given the question and preceding replies of a held-out AgTalk thread and generated the next reply. Every reply was then scored on a one-to-ten scale by a single strong frontier judge (Claude Sonnet), applied uniformly across all four models. Each reply was scored in isolation, with no indication of its source-neither whether it was human- or machine-generated, nor which training condition produced it; therefore, the scoring is blind to condition and observed differences cannot arise from labeling cues. Because every candidate reply is produced by a non-Claude model, self-preference judgement toward its own family cannot bias the comparisons. The human reply that actually followed each thread was scored by the same judge under identical conditions and served as a calibration anchor: a high score for the human reply indicates the judge rewarded genuine answer quality rather than favoring machine text.

Each model was evaluated on the downstream task in three forms—its untrained baseline, its flattened-trained (B) variant, and its recursive-trained (R) variant—so the contribution of post-training, and of recursive structure specifically, to downstream performance could be read directly. Two rubrics were applied independently to every reply. The first rated forum Creole: how closely the reply reproduced the terse slang-bearing vernacular of AgTalk discourse. The second rated content: the substance and usefulness of the reply independent of its style. Separating the two isolates whether post-training improves the model’s command of the forum’s language, the content of its answers, or both.

5. Results

Figure 3 presents training and validation loss curves for all sixteen model configurations: TinyLlama (1.1B), Phi-2 (2.7B), LLaMA-2-7B (6.8B), and LLaMA-2-13B (13B), each under baseline, recursive, and shuffled training conditions. All models were trained for one epoch on 4970 threads with a maximum context length of 4096 tokens, using LoRA fine-tuning with the paged_adamw_8bit optimizer and cosine learning rate schedule (Section 4.3). Validation loss was evaluated every 50 training steps via the HuggingFace Transformers evaluation_strategy parameter, and model selection employed the Trainer class argument load_best_model_at_end = True, which retains the checkpoint with minimum validation loss for each configuration (

Table 3. Training convergence: minimum validation loss and steps to convergence for all model × condition configurations.

LLM Model	Condition	Min Validation Loss	Steps to Min Val Loss
TinyLlama (1.1B)	Baseline	2.2416	33
TinyLlama (1.1B)	Recursive	2.3400	88
TinyLlama (1.1B)	Shuffled	2.3447	108
Phi-2 (2.7B)	Baseline	2.4728	104
Phi-2 (2.7B)	Recursive	2.6337	118
Phi-2 (2.7B)	Shuffled	2.6370	183
LLaMA-7B (6.8B)	Baseline	1.9721	46
LLaMA-7B (6.8B)	Recursive	2.0512	88
LLaMA-7B (6.8B)	Shuffled	2.0644	96
LLaMA-13B (13B)	Baseline	1.9172	39
LLaMA-13B (13B)	Recursive	1.9889	73
LLaMA-13B (13B)	Shuffled	2.0050	93

). Training loss decreased smoothly across all configurations, with all models reaching their validation floor within the first 200 steps (Table 3).

Two patterns in the convergence behavior are noteworthy. First, recursive training consistently reached its minimum validation loss in fewer training steps than shuffled training across all four architectures (88 vs. 108 for TinyLlama, 118 vs. 183 for Phi-2, 88 vs. 96 for LLaMA-7B, 73 vs. 93 for LLaMA-13B), suggesting that ordered thread structure provides a more tractable optimization landscape than randomly permuted replies. Second, baseline conditions achieve lower absolute validation loss than recursive or shuffled conditions (e.g., 1.9172 vs. 1.9889 vs. 2.0050 for LLaMA-13B), reflecting the shorter input sequences in the Q → A_n format rather than superior generalization.

5.1. H0s: DAPT Training Improves Next-Token Prediction

The null hypothesis H0 is supported across all four model architectures—strongly for TinyLlama and the two LLaMA-2 models, and nominally for Phi-2 (

Table 4. H₀ paired perplexity (PPL) comparisons: pretrained base vs. LoRA post-trained models evaluated across 29 thread positions (A₂–A₃₀) under two train-test regimes (BB-RR).

Model	Pretrained Mean PPL	Post-Trained PPL	Reduction	Win Rate	Sig Pos
BB regime: Pretrained vs. B-trained, both tested under B (Q → An)
TinyLlama (1.1B)	9166.60	364.24	96.0%	98.1%	27/29
Phi-2 (2.7B)	28,645.23	14,540.20	49.2%	95.4%	25/29
LLaMA-2-7B (6.8B)	634.93	257.58	59.4%	84.3%	25/29
LLaMA-2-13B (13B)	219.46	156.29	28.8%	87.0%	29/29
RR regime: Pretrained vs. R-trained, both tested under R (Q + A1 + … + An−1)
TinyLlama (1.1B)	444.19	126.81	71.5%	80.5%	28/29
Phi-2 (2.7B)	3445.73	3018.17	12.4%	82.5%	10/29
LLaMA-2-7B (6.8B)	84.41	53.81	36.3%	83.6%	23/29
LLaMA-2-13B (13B)	71.22	38.85	45.5%	83.6%	24/29

Note. Win Rate = percentage of paired predictions where the post-trained model achieves lower perplexity than the pretrained base model. Sig Pos = number of answer positions (out of 29) where a paired Wilcoxon signed-rank test, under Bonferroni correction (α = 0.05/29 ≈ 0.0017), yields p < 0.05 favoring the post-trained model. B = baseline structural format Q → A_n (question-answer pairs); R = recursive structural format Q + A₁ + … + A_n−1 (full chronological threads). Two-letter codes denote (training, test) condition: e.g., BB = baseline-trained, baseline-tested. Phi-2 absolute perplexity is not directly comparable across models owing to tokenizer differences; within-model comparisons (base vs. B vs. R) remain valid.

). The “off-the-shelf” frontier LLMs all leave a measurable perplexity gap when compared to their LoRA post-trained counterparts, with gap reductions ranging from 28.8 percent (LLaMA-2-13B) to 96.0 percent (TinyLlama) and post-trained win rates from 80.5 to 98.1 percent of threads. Paired Wilcoxon signed-rank tests reach significance (p < 0.05) at 26 to 29 of the 29 evaluated thread positions for every model–regime combination except Phi-2 under recursive training, where Phi-2 still reaches significance at 15 of 29 positions.

Phi-2 starts out as the most poorly suited for post-training and remains so. Its absolute perplexity is the highest of the four models, but this largely reflects its tokenizer, which segments forum text onto a different scale and makes cross-model comparison of absolute perplexity erroneous. The valid comparison is within Phi-2 itself: under the more robust RR regime, post-training lowers its perplexity from 3446 to 3018, a reduction of only about 12 percent and the smallest among the four models. The substantive finding is therefore not that Phi-2 trails the other models in absolute terms, but that R-training moves it so little; the model and its tokenizer appear poorly matched to AgTalk forum text in a way that R-training cannot resolve. This is a methodological finding in its own right: post-training is not a remedy for an LLM pretrained on a corpus lexically too distant from the specialized target domain to be adequately tokenized. Phi-2′s pretraining data appears to have left too large a gap for the LoRA adapter to close within its rank-16 parameter budget.

TinyLlama gains the most from post-training. It shows the largest reductions in both evaluation regimes, 96.0 percent under BB and 71.5 percent under RR (Table 4). TinyLlama’s performance reflects that—being the smallest pretrained model—it was initially the most surprised by AgTalk forum text, and therefore had the most to gain from domain adaptation. Post-trained TinyLlama performs at roughly the same perplexity range as the pretrained LLaMA-2 models, while consuming substantially less compute at inference time. Under the BB regime, post-trained TinyLlama (364) falls squarely within the range of the pretrained LLaMA-2 baselines (219 to 635), and under the RR regime TinyLlama (441) lies above them by a factor of two (84 and 71).

The LLaMA-2 models enter post-training already well-suited to AgTalk, leaving less room for DAPT to contribute. Even without DAPT, the LLaMA-2 architectures can predict AgTalk replies better than the R-trained smaller LLMs when evaluated under R-test full thread context (Table 4). Pretrained LLaMA-2-7B reaches a perplexity of 84 and pretrained LLaMA-2-13B reaches 71: both below the post-trained TinyLlama at 127, and far below the post-trained Phi-2 at 3018. Post-training delivers a statistically robust gain over the pretrained baseline within each LLaMA-2 architecture, with the post-trained model winning 83.6 percent of paired comparisons under the RR regime for both LLaMA-2-7B and LLaMA-2-13B, and reaching significance at 26 of 29 thread positions in each case (Table 4).

The pretrained-versus-post-trained scatter plots (Figure 4) confirm the H₀ numerical results visually. Across all four models and both test regimes, nearly all points cluster below the 45-degree diagonal: DAPT improves next-token prediction with vertical distance measuring post-training perplexity gain. The H₀ pattern-perplexity gap closed by DAPT training is most clearly visible under the RR regime (Figure 4) which has more significant gains than the BB (Figure 4).

TinyLlama shows the most dramatic post-training effect under RR. A substantial cluster of blue lies clearly below 45-degree diagonal, indicating that even ordinary low-perplexity replies in the 0–10² range improve after R-training. Larger perplexity gains are obtained in the yellow (10²–10³) and red (>10³) regions where the pretrained TinyLlama would have likely produced incoherent or hallucinated next-token predictions. The visible cluster of red outliers in the 10³ to 10⁴ pretrained range drops sharply after post-training, typically one and often two orders of magnitude in depth. This vertical drop on a log scale is genuine thread rescue, not just refinement. The recursive context helps the pretrained TinyLlama, but the small model still lacks enough domain fluency to use that context fully. R-training appears to extract the linguistic features specific to the AgTalk Creole—the producer vocabulary, conversational style, and forum-discourse patterns—that were absent from the pretrained model’s training corpus. The same pattern is present in the BB plot but less dramatic in absolute terms likely due to BB’s shortened context window.

The two LLaMA-2 models show the cleanest and tightest scatter clouds of the four LLMs (Figure 4). Under RR, both LLaMA-2-7B and LLaMA-2-13B produce compact blue clusters within the 0 to 10² perplexity range, sitting modestly but consistently below the 45-degree line, with sparse yellow (10²–10³) and red (>10³) outlier tails (Figure 5). The visual pattern is best described as high-quality refinement: post-training sharpens already plausible recursive predictions from the pre-trained model rather than rescuing severe pre-trained failures. The 13B model produces the cloud that sits lowest in the panel, anchored in the smallest perplexity range of the four models. Even without DAPT, the larger pretrained model handles AgTalk replies reasonably well when given full thread context, which is why the cloud occupies the bottom-left corner where both pretrained and post-trained perplexities are smallest. The BB plots for both LLaMA-2 models show the same refinement-dominated pattern with a slightly larger rescue component, since the BB pretrained LLaMA-2 models have a heavier outlier tail than under RR.

The H0 results identify three groups of DAPT response. Phi-2 sits at the under-served end: its pretraining corpus is too distant from AgTalk for the rank-16 LoRA adapter to bridge, and post-training delivers only nominal improvement. TinyLlama sits in the DAPT “sweet spot”: its pretraining corpus is close enough that LoRA adaptation can extract the missing AgTalk Creole features, producing the largest perplexity reductions in the study and rescuing many threads that the pretrained model would have predicted poorly. The two LLaMA-2 models sit at the well-served end: their pretrained baselines are already strong enough that DAPT contributes meaningful but modest refinement rather than rescue, and the practitioner question for these models is whether the marginal gain justifies the training and inference cost. The next sections (H₁, H₂, H₃) examine the recursive thread structure benefit within these three groups, but the screening framework established by H₀ already constrains which architectures are candidate deployment targets and which are not.

5.2. H₁: Recursive Context Improves Training

H₁ is supported if LoRA post-training on full recursive thread contexts R = (Q + A₁ + … + A_n−1 → A_n) significantly outperforms baseline B = Q → A_n pairs when tested on identically structured context.

Table 5. H1 paired perplexity (PPL) comparisons across four models and alternative training and testing conditions.

Model	Mean PPL (Condition A)	Mean PPL (Condition B)	Reduction	Win Rate	Sig Pos
Combined Effect                     BB            vs.           RR
TinyLlama (1.1B)	364.24	126.81	65.2%	72.9%	12/29
Phi-2 (2.7B)	14,540.20	3018.17	79.2%	85.8%	21/29
LLaMA-2-7B (6.8B)	257.58	53.81	79.1%	89.4%	27/29
LLaMA-2-13B (13B)	156.29	38.85	75.1%	89.6%	27/29
Training Effect: Test on B           BB           vs.           RB “     “          “     “          “     “
TinyLlama (1.1B)	364.24	612.30	−68.1%	58.8%	4/29
Phi-2 (2.7B)	14,540.20	14,462.45	0.5%	55.1%	0/29
LLaMA-2-7B (6.8B)	257.58	371.92	−44.4%	67.1%	12/29
LLaMA-2-13B (13B)	156.29	99.09	36.6%	69.1%	17/29
Training Effect: Test on R           BR           vs.           RR “     “          “     “          “     “
TinyLlama (1.1B)	231.86	126.81	45.3%	55.2%	10/29
Phi-2 (2.7B)	1542.19	3018.17	−95.7%	61.4%	2/29
LLaMA-2-7B (6.8B)	77.93	53.81	31.0%	72.6%	21/29
LLaMA-2-13B (13B)	56.95	38.85	31.8%	72.2%	19/29
Testing Effect: Train on B            BB            vs.           BR “     “          “     “          “     “
TinyLlama (1.1B)	364.24	231.86	36.3%	72.7%	12/29
Phi-2 (2.7B)	14,540.20	1542.19	89.4%	85.9%	21/29
LLaMA-2-7B (6.8B)	257.58	77.93	69.7%	85.7%	28/29
LLaMA-2-13B (13B)	156.29	56.95	63.6%	85.8%	27/29
Testing Effect: Train on R           RB           vs.           RR “     “          “     “          “     “
TinyLlama (1.1B)	612.30	126.81	79.3%	68.6%	7/29
Phi-2 (2.7B)	14,462.45	3018.17	79.1%	86.0%	23/29
LLaMA-2-7B (6.8B)	371.92	53.81	85.5%	86.4%	27/29
LLaMA-2-13B (13B)	99.09	38.85	60.8%	86.2%	27/29

Note. Win Rate = percentage of paired predictions where the post-trained model achieves lower perplexity than the pretrained base model. Sig Pos = number of answer positions (out of 29) where a paired Wilcoxon signed-rank test, under Bonferroni correction (α = 0.05/29 ≈ 0.0017), yields p < 0.05 favoring the post-trained model. B = baseline Q → A_n; R = recursive Q + A₁ + … + A_n−1. Two-letter codes denote (training, test) condition: e.g., BR = baseline-trained, recursive-tested. Phi-2 absolute perplexity is not directly comparable across models owing to tokenizer differences; within-model comparisons (base vs. B vs. R) remain valid.

begins with a less rigorous, though common situation, where the training structure is maintained as the context window, resulting in a BB vs. RR comparison.

The BB vs. RR comparison strongly supports H₁ across all four LLM architectures (Table 4). The R-trained model tested under R produces lower perplexity than the B-trained model tested under B, with mean perplexity reductions of 65 to 79 percent across the four models, paired Wilcoxon win rates of 72.9 to 89.6 percent, and statistical significance reached at 13 to 29 of the 29 evaluated thread positions. The two LLaMA-2 models reach significance at every evaluated position. The pattern strengthens monotonically with model scale, with TinyLlama producing the smallest effect and the LLaMA-2 models the largest.

The BB vs. RR scatter plots (Figure 4) compare the B-trained perplexity on the x-axis and R-trained perplexity on the y-axis. Points below the 45 degree line are test outcomes where the RR structure produces lower perplexity than BB. Across all four models, the point cloud sits overwhelmingly below the 45-degree line, with the RR advantage strengthening with model scale. TinyLlama shows the weakest structure: it improves under RR, but its blue mass is more dispersed, suggesting the smallest model has less stable ability to exploit longer recursive context. Phi-2 shows much larger absolute perplexity values, but the recursive advantage is visually strong, with most points below the 45 degree line. LLaMA-2-7B and LLaMA-2-13B show the cleanest result. Their clouds are compact, strongly below the 45-degree line, and the recursive win rate is nearly 90%. The 13B model has the lowest absolute perplexity overall, but overall recursive thread context improves prediction across the bulk, middle, and outlier regions for all models.

The three colored regions of the plot, defined by the maximum perplexity within each paired observation, separate where the recursive gains are concentrated. The blue region (max perplexity ≤ 100) contains the bulk of ordinary forum replies. The dense “hanging mass” of blue points below the 45-degree line shows that recursive structure improves prediction not only for rare or pathological cases but for routine replies as well. Because the axes are log-scaled, vertical distance below the line represents multiplicative perplexity reduction; many blue points sit far enough below the diagonal to represent reductions approaching an order of magnitude in all models. The red region (max perplexity > 1000) contains the hardest outlier cases: highly specific, technical, or unusually phrased replies that overwhelm BB’s limited context window and often lead to hallucinations. Most red points fall below the 45-degree line, indicating that R-training combined with R-testing lowers next-token loss on these difficult replies and pulls these cases back toward grounded predictions.

If computational resources are not a constraint, the choice is obvious: deploy the RR configuration. Otherwise, a careful assessment of the BB vs. RR trade-offs is warranted. RR carries both substantially higher GPU compute time during training and higher tokenization costs at every deployment query; however, the BB vs. RR comparison itself does not reveal which of these tasks is responsible for the perplexity reduction. Assessing the tradeoff begins with separating the empirical contributions of R-training and R-testing, recognizing that training is a one-time fixed cost while deployment costs are recurrent.

The BB vs. RR comparison cannot perform this separation on its own. The two conditions differ in both training and inference. The BB model is B-trained and B-tested, while the RR model is R-trained and R-tested. Hence, any perplexity gap between them confounds the contribution of R-training with the contribution of R-testing. The next sections of Table 4 decompose this combined effect into a training-only contribution (BB vs. RB and BR vs. RR) and a testing-only contribution (BB vs. BR and RB vs. RR).

Training Effect: BB vs. RB and BR vs. RR

When tested under identical context windows, support for H1 varies by model size and testing context window (Table 5). LLaMA-2-13B is the only model where R-training produces a clear positive effect when evaluated using the B-test context window, with a 36.6% reduction in perplexity and a Wilcoxon paired win rate of 69.1% (20/29 significant). At 13 billion parameters, the pre-trained model appears to have a more general language structure: its baseline perplexity starts at 219.46, already much closer than TinyLlama’s 9166.60, Phi-2′s 28645.23, or LLaMA-2-7B’s 634.93. Hence, its post-training task is considerably less demanding compared to the others. The scaling advantage of LLaMA-2-13B is striking even under the B-test condition. The B-trained LLaMA-2-13B, with its smaller training context, produces a lower perplexity (156.29) than the other three models even when those three are R-trained with a substantially longer context. Specifically, when R-trained and tested under the same B-test condition, the smaller three models have substantially higher perplexity: TinyLlama (612.30), R-trained Phi-2 (14,462.45), and R-trained LLaMA-2-7B (371.92). The smaller architectures cannot recover through R-training what the 13B model achieves with simpler B-training, suggesting that the recursive thread training at small scale is a less powerful contributor to perplexity performance than LLM scale. This pattern is expected: the R-trained model has internalized attention patterns that look back at intermediate replies and condition next-token predictions on accumulated thread context. When that context is absent at test time, those learned patterns no longer have inputs to attend to, and perplexity performance of the smaller LLMs degrade.

When tested using the longer R context window (BR vs. RR), the recursive advantage in post-training remains scale dependent, though more uniform across models (Table 5). Post-training with the R structure generates the highest advantages in the two larger LLaMA-2 models (Table 5). Both of the R-trained LLaMA-2 models have 72 percent Wilcoxon paired comparison win rates over their B-trained counterparts, clearly distinct from chance, and reduce perplexity by 31%. This provides clear support for their R-training provided it remains in the context window during inference. Results generate weaker support for R-training TinyLlama and Phi-2 models. The matched-context mean perplexity reductions and Wilcoxon paired win rates are substantially lower than those of the larger LLaMA-2 models. Although TinyLlama had the largest perplexity reduction of all models, 45%, its 55.2 percent Wilcoxon paired comparison win rate is essentially at chance, and significant in less than half (12/29) of the thread positions. Overall, Phi-2 outperformed its B-trained counterpart, with a 61% Wilcoxon paired win rate, significant in 8/29 positions, though failed to reduce mean perplexity due to outliers as now explained.

The scatter plots in Figure 3 reveal a scaling tendency already apparent at the smaller scales. TinyLlama and Phi-2 perform well within the typical user forum Creole text domain, where thread predictions fall in the perplexity range between 10² and 10³. In this region, the recursive-trained models reduce perplexity relative to their baseline-trained counterparts commensurate with the larger LLaMA-2 models. Moreover, the smaller models R-training appears to rescue many B-trained threads from hallucination-prone predictions. These regions in the TinyLlama and Phi-2 scatter plots are distinguished by the cloud of outcomes beneath the 45 degree line, where R-trained perplexity values are typically an order of magnitude lower than B-trained ones.

The smaller models, however, appear to lack the trainable parameters needed to handle the full breadth of Creole forum text. As a result, their scatter plots show substantially wider variability than the larger LLaMA-2 models, with high-perplexity outliers reaching three to five orders of magnitude above the mean. Because perplexity is an exponentiated next-token loss, even a small number of these outliers disproportionately pulls/pushes the arithmetic mean upward/downward and weakens interpretation of mean perplexity including paired comparisons. Within the bulk of typical threads (perplexity below approximately 100), R-training nonetheless delivers consistent perplexity reductions for the smaller models.

The same scaling tendency appears across the entire model family. As model size increases, scatter plots tighten in both B- and R-training conditions: perplexity variability decreases, the high-perplexity outlier tail compresses, and the rescued scatter clouds of the smaller models fade. The LLaMA-2 panels in Figure 3 show this clearly, with both training conditions clustering tightly near the y = x diagonal and the R-trained clusters sitting visibly below it across the full perplexity range. In summary, the R-training advantage is structurally present at all scales tested in this study; what changes with scale is the outlier noise that the smaller models cannot eliminate, which unfavorably skews their perplexity performance.

To investigate whether R-training benefits vary with conversational depth, paired perplexity comparisons are stratified by answer position, A_i, within threads (

Table 6. H₁ recursive win rates by thread position for the training effect comparisons: BB vs. RB and BR vs. RR.

	TinyLlama (1.1B)		Phi-2 (2.7B)		LLaMA-7B (6.8B)		LLaMA-13B (13B)
Position	BB vs. RB	BR vs. RR	BB vs. RB	BR vs. RR	BB vs. RB	BR vs. RR	BB vs. RB	BR vs. RR
A2–A6	60.3 (2/5)	54.9 (1/5)	54.5 (0/5)	60.9 (0/5)	67.6 (5/5)	71.8 (5/5)	69.9 (5/5)	74.4 (5/5)
A7–A11	59.8 (2/5)	51.6 (0/5)	56.0 (0/5)	62.5 (0/5)	67.3 (3/5)	72.5 (4/5)	68.0 (4/5)	70.3 (5/5)
A12–A16	55.3 (0/5)	51.5 (0/5)	57.5 (0/5)	58.9 (0/5)	66.4 (3/5)	73.0 (5/5)	67.0 (5/5)	70.0 (5/5)
A17–A21	54.4 (0/5)	56.5 (0/5)	51.4 (0/5)	59.0 (0/5)	62.9 (0/5)	72.6 (3/5)	71.7 (2/5)	67.2 (3/5)
A22–A26	54.8 (0/5)	78.5 (5/5)	58.6 (0/5)	66.1 (1/5)	67.7 (0/5)	78.0 (2/5)	68.3 (0/5)	72.0 (1/5)
A27–A30	52.4 (0/4)	85.7 (4/4)	46.4 (0/4)	77.4 (1/4)	72.6 (1/4)	79.8 (2/4)	70.2 (1/4)	72.6 (0/4)
Overall	58.8 (4/29)	55.2 (10/29)	55.1 (0/29)	61.4 (2/29)	67.1 (12/29)	72.6 (21/29)	69.1 (17/29)	72.2 (19/29)

Note: Values show win rate % (significant positions/total positions per range). Wilcoxon signed-rank pair-wise comparison win-rates, α = 0.05, under Bonferroni correction (α = 0.05/29 ≈ 0.0017).

). With the deeper probe into thread depth and cumulative context windows, a second scaling pattern emerges under R-testing (BR vs. RR) (Table 6). For the smaller models, the matched-context recursive advantage is concentrated at deep thread positions beyond A21. TinyLlama performs essentially at chance through positions A2 through A21 with 51 to 57 percent Wilcoxon paired win rates and most position ranges showing zero or one significant position out of five. For longer threads, performance climbs sharply reaching significant win rates of 78.5 percent (A22–A26) and 85.7 percent A27–A30. Phi-2 follows the same pattern but at a slightly lower magnitude; it is indistinguishable from chance below position A21, then 66 to 77 percent win rates at A22 and beyond, with most positions reaching significance. The two larger LLaMA-2 models, by contrast, show no such depth dependence: their matched-context win rates remain in the 70 percent range across every depth from A2 through A30, with all or nearly all positions reaching significance at every range.

The depth-conditional pattern in the smaller models suggests an accumulation effect. Recursive training appears to encode useful signal that compounds as more prior replies become available at inference. At shallow thread depths, the accumulated context, though longer than Q-A pairs, remains too small for R-training to leverage with no significant advantage over the B-trained counterpart. At the deeper thread depths the accumulated context window grows large enough for the R-trained model to achieve significant gains. Whether the accumulation is merely providing additional context quantity, more tokens to condition on, or additional context structure from the threads sequential discourse cannot be determined from the perplexity-based comparison. Pairwise comparisons cannot distinguish between the two since quantity and structure increase together as thread depth grows.

The next two hypotheses (H₂ and H₃) further examine the structural ordering of the recursive signal directly. H₂ (Section 5.2) tests whether thread reply ordering generates improved training signal by comparing the R-trained models against models trained on the same threads with intermediate replies shuffled. A direct test of whether the chronological ordering of accumulated replies carries information beyond mere context length. H3 (Section 5.3) examines the attention patterns of recursive-trained and shuffled-trained models to determine whether: (1) the model is actually attending to intermediate replies in a manner consistent with the hypothesized attention that users place on prior replies, not merely the original Q, or (2) attention concentrates on positional endpoints, Q-A pairs.

The two testing-effect sections of Table 4 (BB vs. BR and RB vs. RR) compare testing outcomes between the shortened B-test (Q → A_n pair) and the full thread R-test (Q + A₁ + … + A_n−1), holding the training condition constant (B-training in BB vs. BR, R-training in RB vs. RR). In contrast to the R-training’s scale effect, which limited the benefits of R-training to essentially the larger LLaMA-2 models, the R-testing effect is large and uniform across all four models and both the B- and R-training conditions.

For the B-trained models (BB vs. BR), the full thread R-test reduces mean perplexity by 36 to 89 percent across the four LLMs, with paired win rates of 72.7 to 85.9 percent and statistical significance reached at 13 to 28 of the 29 evaluated thread positions (Table 4). Phi-2 shows the largest perplexity reduction in the table (89.4 percent), gaining the most from being given thread context it never saw during training. Its Wilcoxon paired win rate (85.9 percent) and significant-position count (21 of 29) sit in the same range as LLaMA-2-7B (85.7 percent, 28 of 29) and LLaMA-2-13B (85.8 percent, 28 of 29). Under R-testing the 2.7 billion parameter Phi-2 performs nearly like a smaller sibling of the LLaMA-2 family, in contrast to its weaker showing under R-training comparisons earlier. TinyLlama trails the rest, but even at 1.1 billion parameters it crosses a modest threshold of statistical support: 72.7 percent win rate with 13 of 29 positions significant.

The scatter plots in Figure 4 show a consistent shape across all four LLMs: a dense diagonal band that hangs below the 45-degree equality line, indicating that the full thread R-test lowers perplexity compared to the shortened B-test. Each panel’s “hanging mass” of R-test points sits at roughly the same x-axis location across all four models, near a perplexity of 10⁰ (≈1) on the B-test axis. The vertical extent of these masses is roughly 10¹, one order of magnitude, which on a log-scaled plot equals the perplexity reduction the R-test produces for those replies. In the previously discussed R-trained scatter plots, these “hanging mass” rescues appeared at high absolute perplexity where the model may otherwise have hallucinated. Here under R-testing the rescued replies sit at much lower perplexity, slightly below the mean for all four models. Hence, these are not likely hallucinations, rather, they are routine forum replies that the B-test predicted reasonably well. The R-test improves the prediction further by up to an order of magnitude. Prior replies contained in the R-test improve next-token prediction even on threads where the model was already producing reasonably good predictions from Q alone.

The R-test findings have a direct practical implication: R-training is not strictly required to capture most of the recursive structure benefit. A B-trained model given the full thread R-test during testing (BR) operates in the same perplexity range as the R-trained, R-tested counterpart (RR). For LLaMA-2-7B, BR perplexity is 77.93 against RR perplexity of 53.81; for LLaMA-2-13B, BR is 56.95 against RR’s 38.85. The R-trained model still performs better, but the B-trained model with full-context R-test captures most of the perplexity gain. Practitioners constrained without the infrastructure to support R-training can deploy a B-trained model under R-test conditions and recover substantial perplexity reduction at lower training cost, though at higher per-query tokenization cost during inference.

5.3. H₂: Does Sequential Order of Intermediate Replies Matter?

H₂ investigates whether the sequential order of intermediate replies carries additional training signal by testing it against shuffled-trained models, whose chaotic discourse should produce higher perplexity if H2 is supported.

H₂ is not supported at the two smaller model scales. Shuffling the training reply order failed to increase perplexity at test time for either TinyLlama or Phi-2, and for TinyLlama the effect actively reverses: the shuffled-trained model produces lower perplexity than the recursive-trained model (

Table 7. H₂ paired perplexity comparisons: recursive-trained (ordered) vs. shuffled-trained (randomly permuted intermediate replies) models, both evaluated on identical ordered recursive test contexts (Q + A₁ + … + A_n−1 → A_n).

Model	N	Recursive vs. Shuffled Total Wins	Recursive vs. Shuffled Win Rate	Sig Pos	Significant Positions (p < 0.05)
TinyLlama (1.1B)	4866	1644	33.8% *	4/29	A4, A7, A15, A16
Phi-2 (2.7B)	4866	2381	48.9%	0/29	none
LLaMA-2-7B (6.8B)	4866	2708	55.7%	0/29	none
LLaMA-2-13B (13B)	4866	2767	56.9%	2/29	A13, A28

Note: Win Rate = percentage of paired predictions where the recursive-trained model achieves lower perplexity. Sig Pos = positions with Wilcoxon signed-rank p < 0.05 favoring recursive. * Shuffled model wins 66.2% of comparisons; recursive win rate reported for consistency.

). TinyLlama’s results provide the least support: counterintuitively, the shuffled-trained model produces lower perplexity in the majority of cases—66.2 percent (Table 7). A plausible explanation is RoPE positional encoding, applied within the multi-head self-attention layers of each decoder block, which concentrates attention on the sequence endpoints, Q at the beginning and A_n-1 at the end, while attenuating attention to middle positions. Under ordered training, RoPE’s concentration on the sequence endpoints means TinyLlama over-attends to A₁ adjacent to Q at the beginning and A_n−1 adjacent to A_n at the recency end, learning their specific chronological roles as predictors of A_n. Under shuffled training, these same endpoint positions are occupied by random intermediate replies across training examples. The shuffled pattern appears to force the model to learn a more diffuse pattern that pulls signal from a broader range of replies rather than being constrained to A₁ and A_n−1. At test time, the shuffled-trained threads diffuse pattern generalizes better than the over-specialized recursive-trained ones.

Scale appears to soften the RoPE-driven attention constraints that limit H2 support but does not overcome them. Phi-2, at 2.7 billion parameters, produces a non-conclusive R-trained win rate of 48.9 percent with significance reached at only 1 of 29 positions (A28) (Table 7). The bigger LLaMA-2 models push the Wilcoxon paired win rate slightly into positive territory at 55.7 percent and 56.9 percent, respectively, but significance is reached at only 2 of 29 positions for LLaMA-2-7B and 5 of 29 for LLaMA-2-13B (Table 7). The monotonic upward progression in paired win rates suggests that larger model parameters and greater multi-head attention capacity allow larger models to extract some content-level signal from chronologically ordered middle positions despite RoPE concentrating attention on the endpoints. Even at the 13B scale, however, the order effect remains weak with most thread positions failing to reach significance. Whether the RoPE-driven attention explains this seemingly counterintuitive finding is examined in H₃ (Section 5.3).

5.4. H₃: The U-Shaped Attention Pattern Is Architectural, Not Learned

Figure 6 presents attention allocated to the two anchor segments, the original question Q (red) and the most recent reply A_n−1 (blue), across all thread positions and models. The central finding is immediately visible: within each model, the recursive and shuffled curves are nearly indistinguishable. The U-shaped dual-anchor pattern, providing high attention to Q and A_n−1 while attenuating attention to middle replies, is hence present in both training conditions. Paired Wilcoxon signed-rank tests at the Q and A_n−1 segments find statistically significant differences exist in the magnitude of each anchor. Shuffled models tend to allocate slightly more attention to Q while recursive models allocate slightly more to A_n−1 for Phi-2 and LLaMA-7B. The U-shape of the attention distribution, however, is indistinguishable between conditions.

The nearly identical endpoint-weighted attention profiles resolve H₃ in favor of the dual-anchor pattern being structurally imposed by RoPE rather than learned from discourse. This architectural bias is inherent to the decoder-only transformer. RoPE systematically concentrates attention on tokens near the beginning and end of the causal context window, the “lost in the middle” phenomenon documented by Liu et al. (2024) [42], which constrains H₂’s recursive discourse from obtaining adequate full thread attention mechanisms. This architectural constraint provides a reasonable explanation for the convergent findings of Section 5.1 and Section 5.2. R-training improves perplexity because it exposes the model to the presence of intermediate replies between the two RoPE-favored endpoints (H₁ supported), but sequential ordering of those replies provides minimal additional benefit because the attention mechanism cannot differentially access middle-thread content regardless of its arrangement (H₂ inconclusive). The 7B-to-13B plateau in H₁ win rates is likewise attributable to this ceiling. Additional model parameters and larger attention architecture cannot overcome the RoPe structural limitation in how attention is distributed across the input sequence.

5.5. H₄: Consecutive Position Analysis

H₄ posits that if accumulated thread context carries exploitable informational value, perplexity should decrease as context grows. Intuitively, the model should predict A_n+1 better than A_n within the same thread, since the additional reply provides one more chunk of collaborative information to condition on. H₄ is investigated using paired Wilcoxon signed-rank tests between 18 consecutive position pairs (A₂ → A₃ through A₁₉ → A₂₀), restricting each comparison to threads containing both positions to ensure valid pairing (

Table 8. H₄ consecutive position analysis across all four models.

Model	Consecutive Pairs	Sig Pairs (p < 0.05)	Mean Win Rate Range
TinyLlama (1.1B)	18	0/18	43.8–54.2%
Phi-2 (2.7B)	18	0/18	44.7–55.9%
LLaMA-7B (6.8B)	18	0/18	44.4–54.2%
LLaMA-13B (13B)	18	0/18	46.1–54.2%

Note: Paired Wilcoxon signed-rank tests between positions A_n and A_n+1 within the same threads (A2 → A3 through A19 → A20), testing whether perplexity decreases as cumulative thread context accumulates. Recursive-trained models evaluated on full recursive test context. Mean Win Rate Range shows the minimum and maximum A_n+1 < A_n win rate across all 18 consecutive pairs.

). The result is uniform: 0/18 consecutive pairs reach statistical significance for any of the four models and win rates hover near 50% at every pair. H₄ is therefore not supported: adding one more reply to the context does not measurably improve next-token prediction at the next thread position. This null result is a direct consequence of the architectural constraint identified in H₃. Because RoPE-driven attention concentrates on Q and A_n−1 while underweighting intermediate replies, the net information available to the model remains approximately constant regardless of how much collaborative context has accumulated in the middle of the thread.

The pattern across H₁, H₂, H₃, and H₄ is therefore consistent. H₁ could be supported for either of two reasons: the increased training that the recursive-trained model receives, or accumulated context within the recursive threads that reduces perplexity. Because H₄ is not supported and H₂ finds no advantage from reply ordering, the results attribute the H₁ gain to the added training rather than to the recursive structure of the threads. H₃ explains why that structure contributes little: attention concentrates on the question and the most recent reply and underutilizes the intervening replies regardless of training condition. For practitioners, this implies that the benefit of recursive training comes from its added compute effort, with no additional information extracted from the recursive nature of the threads. The Discussion that follows considers architectural directions for remedying this limitation, including encoder–decoder alternatives whose bidirectional attention could extract the sequential signal that decoder-only models leave unused in the middle of the thread.

5.6. Downstream Task Evaluation Results: Generating the Next Forum Reply

The downstream task succeeded: the post-trained models generated better forum replies than the base models (

Table 9. Downstream task next-reply generation results: Creole and Content scores by model and training condition.

Model	Base	B-Training	R-Training	Human
Creole (forum language): mean score 1–10
TinyLlama 1.1B	1.12 d	3.71 c	4.08 b	7.77 a
Phi-2 2.7B	1.52 c	4.88 b	4.91 b	7.36 a
LLaMA-2-7B	1.43 d	5.17 c	5.43 b	7.46 a
LLaMA-2-13B	1.22 d	5.41 b	5.11 c	7.63 a
Content (answer quality): mean score 1–10
TinyLlama 1.1B	1.42 b	1.16 d	1.35 c	4.49 a
Phi-2 2.7B	1.18 d	1.40 c	1.59 b	4.53 a
LLaMA-2-7B	1.54 d	1.75 c	2.03 b	4.32 a
LLaMA-2-13B	1.74 d	1.93 c	2.44 b	4.32 a

Superscripts are a compact letter display of within-row pairwise significance (Wilcoxon signed-rank, p < 0.05). Conditions sharing a letter are not significantly different: a > b > c > d. Human = the real forum reply, scored blind as a calibration basis.

). The clearest effect was on Creole, where both trained conditions scored far above the base at every model size—the strongest result in the study—meeting one of the paper’s central goals: a model fluent in the forum’s local language. On content the gain was real but smaller, with the trained models producing better answers than the base at the larger scales, though every model stayed below the human reply.

On Creole, the ranking was largely the same across the four models. The human reply scored highest, as expected (7.36 to 7.77). R-training came next, above the flattened-Q-A-pair B-training in three of the four models (4.08 vs. 3.71, 4.91 vs. 4.88, and 5.43 vs. 5.17 at 1.1B, 2.7B, and 7B) and below it only at 13B (5.11 vs. 5.41). Both trained conditions sat far above the base (1.12 to 1.52) at every size. The recursive advantage measured in perplexity (H1) was, thus, carried into the downstream result.

On content, the same ordering appeared at 2.7B, 7B, and 13B: human highest (4.32 to 4.53), then R-training, then B-training, then base. R-training scored above B-training at every model size (1.35, 1.59, 2.03, and 2.44 versus 1.16, 1.40, 1.75, and 1.93), again consistent with H1. The trained-over-base gain was clearest at 7B and 13B, where the base (1.54 and 1.74) fell below both trained conditions. At 1.1B the conditions scored low and close, and the base ranked above the trained models (1.42).

Together these results place the downstream finding within the knowledge-extraction framing: post-training first gives the model the forum’s language, and at the larger scales begins to improve the substance of its answers—a second step toward knowledge extraction. The gap to the human reply marks how far that extraction still has to go.

6. Discussion

Recent work has reported counterintuitive evidence that may help explain the weak support for H₂ through H₄ in the present study. The authors of LengthBenchmark [50] tested decoder-only transformers (LLaMA-3.2 and Qwen2.5 families at scales from 1B to 32B parameters) under two perplexity evaluation protocols: a sliding-window approach that delivers many small, recursively accumulating context chunks to the model, and a non-sliding approach that delivers fewer larger chunks without continuous recursive accumulation. The recursive sliding protocol consistently produced higher perplexity than the non-sliding protocol across all model scales, indicating that decoder-only architectures benefit less from continuous recursive context accumulation than the intuition behind that approach would suggest. The parallel to the present results is direct: the H2 null result shows that recursive ordering of training replies does not outperform shuffled ordering at the smaller scales, and the H4 null result shows that adding one more reply at the next thread position does not measurably improve next-token prediction. Both papers reach the same underlying observation from different angles. Decoder-only transformers with RoPE positional encoding extract less value from recursive accumulation than the structural intuition predicts, whether the recursion is applied at the evaluation-protocol level [50] or at the thread-reply level as in the present study.

Other literature provides a more architectural rather than contextual explanation for the same pattern. Prior studies have found that decoder-based neural dialog models extract limited signal from conversational structure [22,51]. Findings confirm from Liu et al. [42] that this is not a property of forum discourse, but of decoder-only architectures whose RoPE positional encoding produces attention weights concentrated on sequence endpoints. Encoder–decoder architectures offer a possible solution to this bottleneck. The addition of an encoder applies bidirectional self-attention across the full input thread, so each reply attends to every other reply regardless of position. The decoder gains expanded predictive power through cross-attention to the encoded representation, selecting whichever segments of the thread carry the most relevant signal for each next-token prediction step. Cross-attention is context-driven rather than RoPE position-driven, which is the property the H₃ results suggest decoder-only architectures cannot adequately produce. T5 [52] and its long-context variant LongT5 [53,54] are examples of this architecture, applying bidirectional self-attention in the encoder and context-driven cross-attention in the decoder.

The H₀–H₄ findings taken together suggest a deployment recommendation under typical compute-constrained environments. R-training plus R-testing is the best configuration when training infrastructure is available, but B-training plus R-testing captures most of the recursive benefit at lower training cost, at the price of higher per-query tokenization. The choice depends on deployment volume and the practitioner’s relative access to training versus inference compute. The H₀ corpus-distance screening also matters here-practitioners should assess whether their base model’s pretraining corpus is close enough to the target domain that LoRA adaptation can move it meaningfully, before committing to DAPT at all.

7. Conclusions

This paper establishes the first step toward knowledge extraction from specialized domains. Maintaining full user forum context during training and testing appears justified even with the greater compute requirements compared to flattened Q-A pairs. A clear scaling effect also emerged across the four models tested, supporting culling the smaller models, particularly Phi-2, from subsequent research stages. Such follow-up research should explore transformer and other training architectures to potentially gain further performance by utilizing the entire threads to overcome the decoder-only positional encoding (RoPE) that largely ignored intermediate thread positions. The next stages of this research arc move beyond training-versus-training comparisons toward delivering deployable knowledge extraction tools: systems that producers, extension professionals, and agricultural advisors can use directly to retrieve grounded answers from the collaborative reasoning preserved in forum threads.

The next step is supervised post-training on masked, domain-screened question-answer pairs derived from the same forum corpus used for recursive DAPT. Where the present paper measured next-token prediction across full thread sequences, this next stage would isolate the specific producer-question to validated-answer pairs that practitioners actually consult forums to obtain. Training on these screened pairs after the recursive DAPT stage would test whether perplexity improvement translates into accuracy improvement on knowledge-grounded responses, and would produce a model that begins to extract domain expertise rather than merely model domain text.

Continued research will be needed to optimally deploy the post-trained models so they can interact with producers and extension professionals in ways that deliver tangible benefit. Downstream tasks that DAPT LLMs are expected to perform include: expert chatbots grounded forum-derived knowledge in technical domain areas; AI assistants for extension professionals that act on extracted forum knowledge and synthesize actionable recommendations; and diagnostic agents that walk producers through structured problem-solving informed by the collaborative reasoning extracted from forum discourse. Each of these tools is a different deployment shape for the same underlying knowledge extraction capability, and each requires the perplexity-to-accuracy validation step described above before it can be deployed responsibly. Realizing these deployments is the goal toward which the recursive DAPT methodology established in this paper is the first step.

The ultimate goal would be shaping deployment towards autonomy using agents that could monitor forum activity, identify emerging issues across producer populations, and surface diagnostic patterns to extension professionals before individual producers know to ask about them. This direction connects the knowledge extraction methodology developed here to the broader agricultural information infrastructure in which producer-discourse forums sit as a substantial but largely under-utilized signal source.