Epistemic Agency in the Age of Large Language Models: Design Principles for Knowledge-Building AI

Abstract

Introduction: Large language models (LLMs) are increasingly employed as cognitive aids in research and professional inquiry, yet their fluent outputs are frequently regarded as authoritative knowledge. We contend that this practice signifies a fundamental epistemic misalignment. Methods/Approach: Building on Peirce’s theory of inquiry, Sellars’ concept of the space of reasons, Stanovich’s tripartite model of cognition, and knowledge-building theory, we develop a conceptual framework for analyzing epistemic agency in human–LLM collaboration. Results/Argument: We demonstrate that LLM outputs fail to satisfy the conditions for knowledge because they lack reflective regulation, resistance to revision, and normative commitment. While LLMs display strong autonomous and algorithmic abilities (e.g., pattern recognition and hypothesis development), reflective control remains a distinctly human function. This asymmetry supports a principled division of epistemic labour and motivates the concept of the Knowledge-Building Partner (KBP): an AI system designed to support inquiry without claiming epistemic authority. Discussion/Implications: We identify prompt-, system-, and model-level design requirements and introduce a triangulated framework for operationalizing epistemic agency through explainable AI, discourse analysis, and rational-thinking measures. These contributions collectively reposition LLM limitations as epistemic design challenges rather than technical issues.

1. Introduction

Imagine a future large language model is asked, in all seriousness, “What is the answer to the meaning of life?” The system pauses, engaging vast computational resources, weighing probabilities across an immense corpus of human texts, and then replies: “The answer to the great question of life, the universe, and everything is forty-two.” The response is fluent and immediately recognizable as the answer Deep Thought gave in Douglas Adams’s classic novel [1]. Yet from the perspectives of Charles Sanders Peirce [2] and Wilfrid Sellars [3,4], it does not constitute knowledge. It is an utterance that bears the surface form of an answer while lacking the epistemic substance required for one.

For Peirce, knowledge is inseparable from inquiry. It is not a static product but the provisional outcome of a self-correcting process oriented toward resolving genuine doubt [5,6]. A claim counts as knowledge only insofar as it would stabilize at the hypothetical end of inquiry, after sustained exposure to evidence and criticism, and later, revision within a knowledge-building community [6,7,8]. In this example, the language model’s reply fails precisely because no such inquiry has occurred. The question to which it responds is undefined, not decomposed into contestable components, and therefore incapable of generating doubt. Without a deliberate question, inquiry never begins. Moreover, the answer offered does not hold up against reality in any meaningful sense [6,9]. Peirce insisted that reality is what pushes back against belief, forcing revision through surprise or failure. “Forty-two” is immune to this pressure, not because it is robust, but because it is empty. Nothing could count for or against it. Finally, the response does not invite convergence within a community of inquiry; it terminates discourse rather than opening it [6,10]. From a Peircean standpoint, the model has not produced knowledge, but an utterance disguised as the endpoint of inquiry without having passed through inquiry at all.

For Sellars, to know is to occupy a position within the space of reasons [11,12]. Knowledge is not stating something that happens to be correct; it is being able to justify a claim, to grasp what follows from it, and to recognize what would count against it [13]. The model’s answer does none of this. It offers no reasons for why “forty-two” should be accepted, no inferential commitments that follow from accepting it, and no conditions under which it would be withdrawn. The utterance is not embedded in a network of explanations, predictions, or normative consequences. In Sellars’ terms, it is not a claim in the full normative sense, but a string of symbols that humans recognize as meaningful because of a shared cultural background. Syntax and semantic familiarity substitute for justification and understanding [14].

What, then, would be required for such an output to be a product we call knowledge? Both Peirce and Sellars would insist on changes that go far beyond increasing computational power or training data [15]. We can infer, from our discussion so far, that the question itself would have to be rendered intelligible and contestable, capable of generating real doubt and guiding inquiry. The answer would need to be provisional, offered as a commitment, subject to revision, rather than a terminal output. It would have to be situated within the space of reasons, supported by justifications, linked to implications, and exposed to potential falsification. Crucially, it would need to encounter resistance, whether empirical or conceptual, such that the world could push back. Finally, it would need to enter a community of inquiry, open to others’ criticism and refinement [8]. Only under these conditions could an answer begin to count as knowledge. Knowledge is not what a neural network alone derives; it is what survives questioning and generates a rational consensus [16].

When the above logic is applied to the design of an LLM as a knowledge-building partner (KBP), a clear implication follows: the system must not present its outputs as authoritative knowledge, but as ideas offered in the service of inquiry. Framing an LLM’s responses as answers in the epistemic sense is to repeat the mistake illustrated by the “forty-two” example that substitutes fluent termination for genuine understanding. Following Peirce and Sellars, an effective LLM-KBP should function as a participant that supports the conditions of knowing without claiming to instantiate them. Its role would be to provide relevant ideas, possible explanations, contrasts, and considerations that help human investigators articulate questions and situate claims within the space of reasons. By withholding epistemic authority and foregrounding the provisional, contestable status of its contributions, the KBP preserves the user’s epistemic agency and keeps inquiry open rather than closed. In this way, the LLM does not present itself as the bearer of knowledge, but rather as a scaffold for the processes through which knowledge may eventually be achieved.

1.1. Epistemic Misalignment and the Conditions of Knowledge

From the epistemic perspectives of Peirce and Sellars, the limitations of current large language models reveal both philosophical and pedagogical shortcomings. When an LLM produces fluent outputs that are insulated from challenge or error, it fails to support the processes through which understanding develops. A knowledge-building partner must operate within a space of reasons: it must make explicit why a claim is offered, what follows from accepting it, and what would count against it, while sustaining inquiry over time through interaction and correction. As presently designed, however, LLM outputs tend to terminate discourse rather than extend it. The central challenge, therefore, is not to curb the production of empty answers but to redesign LLMs so that, in knowledge-building contexts, their contributions function as provisional waypoints that support inquiry rather than its endpoints.

This challenge is intensified by the very strengths that make LLMs attractive as tutors. Their fluency, confidence, and cultural attunement [17] give their outputs the appearance of authoritative knowledge, encouraging uptake as answers rather than as ideas to be examined [18]. The problem, then, is not only whether LLMs will continue to generate answer-shaped outputs, but how humans should collaborate with them in ways that preserve the conditions of knowing rather than erode them. To best address this problem, a shift is needed from purely epistemological analysis to a cognitive account of the skills required for reasoning and knowledge-building. Keith Stanovich’s tripartite model of mind provides a framework that distinguishes among autonomous, algorithmic, and reflective levels of cognition. This distinction supports a reconceptualization of LLMs not as epistemic authorities but as cognitive partners whose processing strengths complement, rather than replace, human epistemic agency. By aligning these cognitive capacities with core principles of knowledge-building, we can begin to specify how LLMs might productively support inquiry and reasoning without displacing the reflective control that genuine knowledge creation requires.

1.2. Aim and Conceptual Approach

If we accept that rational reasoning is central to effective knowledge-building, Stanovich’s approach provides a clear lens for understanding how human–LLM collaboration could be structured. His tripartite model of mind distinguishes among three functionally distinct levels of cognition: the autonomous mind, the algorithmic mind, and the reflective mind. This framework refines the familiar dual-process distinction between System 1 and System 2 thinking [19]. Whereas System 1 corresponds roughly to the autonomous mind, Stanovich argues that what is conventionally called System 2 conflates two separable functions: the algorithmic mind, which provides the computational resources for controlled processing, and the reflective mind, which determines when and why those resources should be deployed [20]. This disaggregation is central to his account of rationality, since high algorithmic capacity does not guarantee rational performance without appropriate reflective dispositions to trigger and regulate its use. The autonomous mind corresponds to fast, automatic, and associative processes that generate default responses with little conscious control [21]. These processes are efficient and often adaptive, but they are insensitive to normative considerations such as truth, coherence, or justification [22]. The algorithmic mind refers to the computational machinery that supports controlled processing, including working memory capacity and the ability to sustain decoupled representations. It may enable simulation and inhibition of prepotent responses, but it does not, by itself, determine when or why such resources should be deployed. The reflective mind operates at a higher level, regulating epistemic priorities and deciding when to override autonomous responses. It is responsible for evaluating beliefs and allocating cognitive effort in accordance with norms of rationality rather than convenience or habit [20].

In Stanovich’s account, rational reasoning emerges when reflective processes inhibit default intuitive responses, recruit algorithmic processing to support decoupled simulation, and evaluate beliefs according to standards derived from logic and evidence rather than automatic intuitive processing [20]. This distinction helps explain why large language models can display striking forms of apparent intelligence without engaging in rational reasoning. Their impressive performance reflects substantial algorithmic capacity, supporting pattern detection, symbolic manipulation, and large-scale simulation, but algorithmic power alone does not yield knowledge [23,24]. Rationality, in this view, depends on epistemic agency: the capacity to decide when default responses should be questioned and when beliefs ought to be revised or withheld [25,26]. Because LLMs lack reflective control over their own outputs, their contributions can be informative without being epistemically committed. Clearly, to make the case for how we should understand human–LLM collaboration, it is necessary to examine in greater detail how Stanovich distinguishes among the cognitive functions involved in reasoning.

The cognitive skills associated with each of the three modes of thinking Stanovich identifies are not equally well-developed in LLMs and humans. Large language models, for example, are superb at the thinking skills that allow them to detect and extend patterns. Trained on large corpora, they excel at automatically identifying regularities, analogies, and statistically stable relationships across domains, making them powerful engines for autonomous processing that can sustain complex pattern-based computations. Humans, by contrast, have the thinking skills that are often most effective at noticing anomalies (cases that do not fit, moments of surprise, or felt tensions that register as intuition or “gut feelings”) before they can be articulated as reasons [18,27,28]. In Stanovich’s terms, this asymmetry reflects a division between autonomous and algorithmic capacities on the one hand, and reflective control on the other. While LLMs possess impressive algorithmic power, they lack the capacity to experience resistance or to recognize that a pattern has failed in a way that calls for revision [29,30]. Human reflective minds are attuned to precisely these disruptions, using them as signals to suspend default interpretations, reframe the problem space, and initiate inquiry [31].

The above-noted asymmetry in knowledge-creation skills between humans and machines suggests we should consider a principled division of labour to optimize collaboration. Accordingly, LLMs could be used to reveal patterns, generalizations, and possible explanations across large bodies of information, while humans remain responsible for judging when those patterns break down, when explanations are unsatisfactory, and when deeper inquiry is needed. Research on human–AI collaboration emphasizes that LLMs can augment analytic workflows by surfacing relevant structures and insights from complex data, thereby supporting decision-making and hypothesis exploration. However, it also demonstrates that human judgment and iterative questioning remain essential for interpretation and deeper reasoning [32]. It is in these moments of anomaly detection, often felt before they are formalized, that epistemic agency is exercised, and knowledge-building can begin. Therefore, in a well-designed collaborative system, LLMs should be leveraged to support autonomous and algorithmic functions (generating hypotheses, elaborating alternatives, maintaining representations, and simulating implications) while humans should retain responsibility for reflective control [33,34,35,36]. Humans decide which questions matter, when deeper reasoning is required, what counts as evidence, and when an explanation is satisfactory or in need of revision. Framed this way, LLMs do not compete with human rationality but scaffold it, extending the reach of System 2 reasoning without displacing the epistemic agency that makes knowledge-building possible [37].

This division of labour, while principled in theory, faces practical challenges we need to address. The boundary between algorithmic support and reflective influence is not always cleanly maintainable in collaborative workflows. When an LLM generates hypotheses, frames a problem space, or surfaces patterns from a dataset, it might shape the possibilities that a human collaborator considers. A compelling hypothesis may foreclose alternative lines of inquiry not because it has been reflectively evaluated and endorsed, but because it occupies cognitive attention and reasoning. In this way, outputs from the algorithmic layer can infiltrate the reflective layer, shaping which questions seem worth pursuing before deliberate evaluation occurs [38].

Mitigating this risk requires what might be called meta-reflective vigilance: an ongoing human disposition to question not only the content of LLM outputs but also the framing assumptions they embed and the inquiry paths they render salient or invisible. Effective collaboration thus demands more than allocating functions between humans and machines; it requires cultivating awareness of how machine outputs can shape human cognition in ways that bypass explicit, reflective endorsement. This is not an argument against collaboration but a caution that the reflective role cannot be passive. Human epistemic authority must be exercised actively rather than formally reserved.

In thinking about such a division of labour, the important cognitive subfunctions associated with knowledge creation would include, but are not limited to, the nine central functions presented in

Table 1. An a priori assessment of knowledge creation skills across LLMs and humans.

Cognitive Function	Stanovich’s [20] Tripartite Level	LLMs	Humans
Pattern detection	Autonomous	Excellent	Strong
Pattern extension	Autonomous/Algorithmic	Excellent	Moderate
Hypothesis generation	Algorithmic	Strong	Strong
Sustained simulation	Algorithmic	Excellent	Limited
Inhibitory control	Algorithmic	Weak	Strong
Detecting anomalies	Reflective triggered	Weak	Strong
Initiating inquiry	Reflective	Absent	Strong
Epistemic regulation	Reflective	Absent	Strong
Knowledge commitment	Reflective	Absent	Strong

. Pattern detection is the rapid, largely automatic recognition of regularities, similarities, or recurring structures in experience, allowing a system to respond fluently to familiar situations without deliberate analysis [39,40]. Pattern extension builds on this capacity by projecting recognized regularities forward, filling in missing elements or continuing a pattern beyond the information immediately given [41,42]. Hypothesis generation involves proposing possible explanations or models that could account for an observed pattern or anomaly, often serving as a starting point for deeper reasoning [43,44]. Sustained simulation is the ability to hold representations in abeyance and explore their implications over time, running “what-if” scenarios without immediate commitment to action or belief [45,46].

Inhibitory control refers to the ability to suppress or override a dominant response during processing when it is seen as inappropriate, allowing alternative interpretations or strategies to be considered [47,48]. In Table 1, LLM inhibitory control is rated as “Weak” not because LLMs never exhibit restraint-like behaviour, but because this behaviour does not result from online, self-initiated regulation. Through reinforcement learning from human feedback (RLHF) and related alignment techniques, LLMs can be trained to avoid harmful outputs, express uncertainty, or decline certain requests. However, these behaviours stem from prior optimization during training rather than real-time self-monitoring. Human inhibitory control functions dynamically during cognition: an emerging response can be halted mid-formation if it conflicts with goals or norms. Conversely, LLM “inhibition” is based on statistical conditioning over previously observed patterns. The system does not identify its own response as epistemically inappropriate and then suppress it; instead, it produces outputs that align statistically with prior reinforcement signals. Therefore, the rating “Weak” reflects the lack of endogenous, real-time self-correction, even while recognizing that LLM outputs may approximate inhibitory outcomes under certain conditions.

Detecting anomalies can be triggered reflectively or by sensitivity to mismatches, surprises, or forms of resistance that indicate when expectations or patterns fail to align with reality [49,50]. Initiating inquiry is the deliberate decision to treat such disruptions as problems that require further investigation rather than as noise to be ignored [51,52]. Epistemic regulation governs how reasoning is directed and evaluated, including decisions about what counts as evidence, which standards of justification apply, and when a line of reasoning has gone far enough [53,54]. Finally, knowledge commitment is the act of endorsing a claim as warranted and remaining open to its revision in light of future evidence or criticism [55,56] (see Table 1).

The ratings in Table 1 are theory-driven and reflect functional capacities based on Stanovich’s tripartite model, rather than surface behavioural resemblance. “Excellent,” “Strong,” “Moderate,” “Weak,” and “Absent” indicate the degree to which a system can autonomously and internally perform a cognitive function within its architecture. Specifically, reflective-level functions are assessed on whether the system can initiate, regulate, and revise its own processing in response to epistemic norms, rather than simply producing outputs that appear to be regulated. Therefore, the ratings differentiate between learned behavioural patterns and true online self-regulation within the cognitive process itself. Therefore, Table 1 presents an a priori, theory-driven comparison of knowledge-creation skills in humans and large language models, organized around Stanovich’s tripartite model of the mind.

LLMs show pronounced strengths in cognitive functions associated with the autonomous and algorithmic levels, particularly pattern detection, pattern extension, hypothesis generation, and sustained simulation. These capacities align with their training regime and architecture, which optimize large-scale pattern extraction, symbolic manipulation, and the maintenance of multiple representations without fatigue. However, the assessment that LLMs possess “excellent” sustained simulation capacity requires qualification. LLMs can maintain coherent, extended reasoning chains and explore hypothetical scenarios across lengthy outputs without the working-memory constraints that limit human simulation. Yet their simulation differs from human mental modelling in important respects: LLMs cannot recognize when a simulation has diverged from plausibility, experience felt uncertainty that signals the limits of imaginative exploration, or dynamically adjust the depth of simulation based on task demands. Their simulation is generative rather than evaluative, for they extend representations fluently but lack the reflective monitoring that allows humans to know when a thought experiment has gone far enough or gone astray. For this reason, the “excellent” rating should be understood as applying to generative capacity rather than to the full epistemic function that sustained simulation serves in human reasoning. By contrast, the figure highlights consistent limitations in functions that require reflective processing. The ratings of “absent” for initiating inquiry, epistemic regulation, and knowledge commitment require careful interpretation. These category assignments do not deny that LLMs exhibit behaviours that superficially resemble these functions. Contemporary LLMs can pose follow-up questions, express calibrated uncertainty and decline to endorse claims for which they lack evidence. The question is whether these behaviours constitute genuine epistemic agency or sophisticated behavioural mimicry shaped by training on human epistemic discourse [57,58,59].

Several considerations support the latter interpretation. First, LLMs do not set their own epistemic goals; their apparent “inquiry” is triggered by user prompts or training incentives rather than by internally generated recognition that something requires further investigation. Second, their expressions of uncertainty are probabilistic outputs rather than felt states that regulate subsequent processing. An LLM that says “I’m not sure” does not thereby allocate additional cognitive resources to the problem or flag the claim for future revision; the expression serves a communicative rather than a regulative function. Third, and most fundamentally, LLMs lack the persistent continuity required for knowledge commitment. They do not carry beliefs forward across contexts, cannot revise yesterday’s claims in light of today’s evidence, and bear no accountability for the consistency of their outputs over time. What appears as epistemic regulation is better understood as pattern-matching on the discourse of epistemic regulation, producing outputs that would be appropriate if such regulation were occurring without the underlying cognitive architecture that makes genuine regulation possible. This distinction matters because conflating behavioural mimicry with authentic epistemic function risks obscuring the very capacities that make human oversight necessary. If LLMs genuinely possessed epistemic regulation, collaborative frameworks would need to negotiate between two sources of epistemic authority. Because LLMs lack genuine epistemic regulation, collaboration does not involve balancing two independent sources of epistemic judgment. Instead, humans provide what LLMs cannot: the reflective oversight that determines when fluent output should be accepted as warranted belief.

Interpreted in this way, Table 1 suggests we take a principled division of labour rather than a competitive comparison. Furthermore, it warrants the view that LLMs are best positioned as cognitive supports that extend autonomous and algorithmic processing, while humans retain responsibility for reflective functions that give inquiry its direction and epistemic grounding. The value of the figure lies not in ranking humans and machines, but in clarifying how their complementary strengths can be aligned to support disciplined inquiry without collapsing epistemic authority into fluent output.

1.3. Knowledge-Building as a Normative Framework

Knowledge-building (KB), as developed by Bereiter and Scardamalia [60,61,62,63,64], is a pedagogical model grounded in the view that learning is best understood as participation in the creation and continual improvement of ideas. Rather than treating knowledge as something to be transmitted from expert to novice, knowledge-building frames learners as epistemic agents who collectively work to advance the state of knowledge in a community [65]. Drawing on the practices of scientific and scholarly inquiry, KB emphasizes explanation-seeking, theory refinement, and sustained discourse oriented toward understanding rather than performance [63,66,67]. Over several decades, this model has been elaborated theoretically and instantiated empirically through learning environments such as Knowledge Forum, demonstrating its capacity to support big conceptual change across disciplines and age groups.

At its core, knowledge-building is not only a set of instructional techniques but also a theory of learning that treats the advancement of knowledge as the central object of activity [66]. Learning is evaluated not by the acquisition of predefined answers, but by the community’s progress in identifying problems, generating explanations, addressing gaps, and producing increasingly coherent and powerful ideas [68]. This emphasis on inquiry, explanation, and communal responsibility positions KB as a pedagogical counterpart to philosophical accounts of knowledge that stress fallibilism, normativity, and self-correction [66].

Scardamalia and Bereiter articulate knowledge-building through a set of interrelated principles that specify how epistemic activity should be organized [69]. Among the most central are ideas as improvable, which treat all contributions as provisional objects open to refinement; epistemic agency, which places responsibility for goal setting, problem formulation, and evaluation in the hands of learners; and knowledge-building discourse, which emphasizes explanation, critique, and synthesis over debate or recitation. Additional principles, such as community knowledge, collective responsibility, rise-above, and the constructive use of authoritative sources, further reinforce the idea that learning advances through sustained engagement with problems rather than through the consumption of answers. Overall, these principles are normative rather than procedural. They do not prescribe a fixed sequence of instructional steps; instead, they define the conditions under which a community can progressively improve its ideas. Knowledge-building thus explicitly foregrounds the regulation of inquiry: deciding what problems matter, what counts as progress, and when an explanation is adequate or in need of revision. In this respect, KB is as much concerned with how thinking is governed as with what is learned.

At this point, it is important to clarify how the term “epistemic agency” is used in this paper. By epistemic agency, we mean the capacity and disposition to regulate inquiry in accordance with epistemic norms, including initiating questions, evaluating reasons and evidence, suspending or revising beliefs, and provisionally committing to claims while remaining open to further revision. Epistemic agency is not equivalent to intelligence, fluency, or content knowledge. Rather, it refers to reflective control over belief formation and the direction of inquiry: deciding when a claim requires justification, which standards of evidence apply, and whether understanding is adequate or in need of further refinement. In knowledge-building contexts, epistemic agency is both individual and socially distributed, emerging through participation in communities that collectively regulate inquiry over time [67,69].

When viewed through Stanovich’s tripartite model, the cognitive commitments of knowledge-building become especially clear. Stanovich’s distinction between autonomous, algorithmic, and reflective levels of cognition mirrors the functional requirements implied by KB principles. Autonomous processes supply the intuitive, associative ideas that often seed inquiry; algorithmic processes support sustained reasoning, simulation, and the manipulation of competing explanations; but it is the reflective mind that governs inquiry by setting epistemic goals, allocating cognitive effort, and evaluating ideas against normative standards. Indeed, knowledge-building principles map most directly onto this reflective level. Epistemic agency corresponds to reflective control over when and how inquiry proceeds. Treating ideas as improvable presupposes the ability to decouple beliefs from immediate commitment and subject them to evaluation and revision. Rise-above discourse depends on the metarepresentational capacity to reflect on patterns across ideas and reorganize conceptual space. Importantly, the principle of the constructive use of authoritative sources highlights the central role of reflective judgment, requiring learners to assess relevance and explanatory adequacy rather than simply accepting fluent authority.

Knowledge-building can be understood as a pedagogical framework deliberately designed to cultivate and sustain reflective cognition in learners. Stanovich’s model, in turn, provides a cognitive explanation for why KB works: it systematically engages the reflective mind while coordinating autonomous and algorithmic processes in the service of inquiry. This alignment clarifies why knowledge-building offers a particularly appropriate context for human–LLM collaboration. LLMs can productively support autonomous and algorithmic functions, such as pattern detection and hypothesis generation, while the reflective functions essential to knowledge-building remain solidly in human epistemic agency. Deeper still, the core cognitive functions outlined in this paper can be understood as the psychological cognitive architecture of knowledge creation itself. Functions such as pattern detection, hypothesis generation, sustained simulation, anomaly detection, and epistemic regulation are not just components of individual reasoning; they are the mechanisms through which new knowledge is generated and evaluated. When viewed through an epistemic lens, these functions correspond directly to core knowledge creation activities: generating candidate explanations, exploring alternatives, encountering resistance, revising ideas, and committing to provisional conclusions. In this sense, the distinction between cognitive functions and knowledge creation functions is largely one of perspective rather than kind: the former describes how thinking operates, while the latter describes what that thinking accomplishes within inquiry.

Knowledge-building principles provide the pedagogical bridge between these two levels. Rather than treating knowledge creation as something that emerges spontaneously from individual cognition, knowledge-building explicitly organizes learning environments to activate and normatively regulate these underlying cognitive functions. For example, treating ideas as improvable depends on the ability to decouple representations from immediate commitment and subject them to revision; epistemic agency presupposes reflective control over when inquiry begins and how it proceeds; and rise-above discourse requires metarepresentational capacity to detect patterns across ideas and reorganize conceptual space. Each of these principles maps onto specific cognitive functions, particularly those associated with the reflective level of Stanovich’s model, where epistemic goals, norms, and commitments are regulated.

Viewed this way, the pedagogical power of knowledge-building lies in its deliberate alignment with this cognitive architecture: it externalizes reflective control, distributes it across a community, and sustains it over time through discourse norms and shared responsibility. This alignment helps explain why knowledge-building is particularly well-suited as a framework for human–LLM collaboration, as it allows LLMs to support lower-level cognitive functions while preserving the reflective capacities essential for genuine knowledge creation. Indeed, as outlined in

Table 2. Alignment of core cognitive functions with knowledge creation activities and knowledge-building principles within Stanovich’s tripartite model.

Stanovich’s Level [20]	Core Cognitive Role	Knowledge Creation Functions	Knowledge-Building Principles [67]	Role in Knowledge-Building
Autonomous Mind	Fast, associative pattern generation	Pattern detection; Pattern extension	Idea diversity (implicit); Early idea generation	Supplies intuitive ideas, associations, and candidate explanations that seed inquiry
Algorithmic Mind	Sustained, decoupled processing and simulation	Hypothesis generation; Sustained simulation; Inhibitory control	Knowledge-building discourse; Constructive use of authoritative sources	Enables comparison of ideas, elaboration of explanations, and exploration of implications
Reflective Mind	Epistemic regulation and goal setting	Initiating inquiry; Epistemic regulation	Epistemic agency; Ideas as improvable	Determines what problems matter, when ideas require revision, and what counts as progress
Reflective Mind	Metarepresentational abstraction	Detecting anomalies (as triggers); Integration across ideas	Rise-above	Enables synthesis across ideas and reorganization of conceptual space
Reflective Mind	Normative evaluation and commitment	Knowledge commitment	Community knowledge; Collective responsibility	Governs justification, critique, and convergence toward improved communal understanding
Reflective Mind	Regulation of inquiry over time	Sustained coordination of reflective functions	Sustained knowledge advancement	Maintains inquiry across sessions rather than terminating it with answers

, core cognitive functions become knowledge creation functions when they are normatively regulated, and knowledge-building provides the pedagogical structure that sustains this regulation over time.

Furthermore, Table 2 makes clear that knowledge creation does not arise from isolated cognitive functions operating within individuals, but from the coordinated regulation of those functions across time and participants. In particular, the figure highlights that the most epistemically consequential activities, such as initiating inquiry, detecting anomalies, regulating standards of evidence, integrating across ideas, and committing provisionally to claims, are all reflective functions. These functions are cognitively demanding, developmentally fragile, and difficult to sustain within individuals acting alone. The figure, therefore, implicitly underscores the importance of groups as facilitators. Knowledge-building communities externalize and distribute reflective control by embedding it in discourse norms, shared artifacts, and collective responsibility. Through interaction, groups stabilize decoupled representations, surface resistance through disagreement and misunderstanding, and maintain inquiry over time in ways that exceed the capacities of individual cognition.

Table 2 also suggests that groups are not only social add-ons to learning but are constitutive of knowledge-building itself. Many of the functions aligned with the reflective level, such as rise-above, epistemic regulation, and sustained coordination of inquiry, are amplified precisely because they are socially enacted. Group discourse makes epistemic norms visible, negotiable, and enforceable; it also reduces premature knowledge commitment by diffusing ownership of ideas and encouraging revision [70]. In short, the figure implies that epistemic agency is often distributed rather than localized, emerging through participation in a community that jointly regulates inquiry [71,72]. This distributed character of epistemic agency raises the question of how, within Stanovich’s tripartite model, collective activity can systematically support and extend the cognitive functions required for knowledge-building.

1.4. The Context of the Knowledge-Building Community in the Tripartite Model

Knowledge-building is not just about an individual applying knowledge-building principles. The whole process is designed to be carried out in collaboration with other individuals. Thus, according to Stanovich’s model, groups can act as cognitive amplifiers, externalizing functions (simulation, anomaly detection, regulation, and revision) that individuals are often learning to perform internally. In this respect, group interaction plays a central role in externalizing and amplifying the cognitive functions required for knowledge creation, particularly for novice knowledge. Below, we discuss, from the tripartite model, the main contributions the community of knowledge builders can provide; namely, decoupling, distributed working memory, sustained simulation, collective hypothesis generation, socially triggered anomaly detection, epistemic regulation, reduction in premature knowledge commitment, rise above processes, and sustained inquiry through social responsibility.

One of the most fundamental mechanisms through which groups can support knowledge creation is decoupling. When learners articulate a raw idea to others, the idea is externalized into a shared representational space (language, text, or other artifacts) where it is no longer fully controlled by the speaker’s original intentions. What peers receive is necessarily partial and shaped by their own interpretive frames. When the speaker hears the idea reflected in an altered or incomplete form, it is separated from immediate belief and identity and becomes a provisional object of inquiry. This social re-representation can expose ambiguities, hidden assumptions, and alternative interpretations that are often inaccessible to the individual alone. In Stanovich’s terms, the group effectively performs a decoupling operation on behalf of the learner, holding the idea at a distance so it can be examined, compared, and manipulated without requiring immediate commitment. Through this process, comparison, correction, and revision become natural responses to discourse rather than signs of error, and the cognitively demanding work of sustaining decoupled representations is distributed across the group rather than borne by any single individual.

Beyond decoupling, groups amplify knowledge creation by distributing working memory and sustaining simulation over time. Complex ideas, open questions, and partial explanations are carried collectively in shared discourse and external artifacts, rather than relying on the limited cognitive resources of any single individual. This distribution allows exploratory reasoning to unfold gradually, as ideas are revisited, extended, and reworked across turns and sessions. In Stanovich’s terms, the group provides an external scaffold for algorithmic processing, enabling sustained simulation and comparison of alternatives that individuals would find difficult to maintain internally. Groups also support collective hypothesis generation, as different participants contribute diverse intuitions, analogies, and explanatory candidates shaped by their distinct experiences and perspectives. The resulting diversity expands the space of possible explanations and makes contrasts among them more salient, facilitating evaluation and refinement. Importantly, this collective generation of hypotheses reduces the pressure on individuals to produce complete or defensible explanations on their own, shifting the epistemic focus from ownership and defence toward the shared improvement of ideas.

Groups further enhance inquiry through socially triggered anomaly detection. In this case, misunderstandings, disagreements, and unexpected questions from others introduce forms of resistance that interrupt smooth agreement and expose gaps, inconsistencies, or unstated assumptions in emerging explanations. These moments of friction function as signals that something requires further examination, often surfacing problems that an individual might overlook when reasoning alone. Especially for learners whose internal monitoring of error and coherence is still developing, such externally generated resistance plays a critical epistemic role by making anomalies salient before they can be fully articulated. The group supplies a form of experiential “pushback” that initiates genuine inquiry. In parallel, groups may partially externalize epistemic regulation by making norms of reasoning public and actionable. Questions about what counts as evidence, whether an explanation is sufficiently coherent, or how competing ideas should be evaluated are often introduced through social interaction and negotiated collectively. Over time, these shared regulatory moves allow the community to determine when inquiry should continue, when revision is necessary, and when a provisional understanding is adequate, while also supporting the gradual internalization of these norms by individual participants.

Another important group function is reducing premature knowledge commitment. When ideas are treated as shared contributions rather than personal possessions, learners are more willing to revise or abandon them. This diffusion of ownership weakens identity-based defensiveness and aligns participation with the goal of idea improvement. Groups also support rise-above processes, as repeated comparison across contributions makes higher-level patterns and abstractions more salient. Through dialogue, learners collectively reorganize conceptual space in ways that would be difficult to achieve individually. Finally, groups promote sustained inquiry through social accountability. Shared responsibility for advancing understanding increases persistence and engagement, supporting long-term knowledge advancement that extends beyond isolated episodes of individual effort.

Overall, these mechanisms demonstrate how groups can act as cognitive amplifiers, externalizing and stabilizing the processes that support knowledge creation. By sharing these functions within a community, group interaction creates conditions in which learners can meaningfully engage in inquiry even before they can reliably perform these processes independently. Additionally, these group-level amplifications have direct implications for the design of LLM-supported learning environments. Instead of viewing LLMs as replacements for peer interaction, effective designs should utilize LLMs to support and enhance these collective processes. For instance, LLMs can assist in externalizing ideas, maintaining shared representations over time, generating alternative hypotheses, or encouraging reflection on unresolved tensions, while leaving anomaly detection, epistemic regulation, and knowledge commitment to the human group. In this role, the LLM serves as cognitive support embedded within a community of inquiry, enhancing the group’s ability to sustain decoupling, simulation, and discourse without disrupting the social dynamics through which epistemic agency and knowledge-building are developed. If epistemic agency is the central concept connecting epistemology, cognition, knowledge-building pedagogy, and the community’s major influence, then it must be made empirically manageable. The following section presents a triangulated, multi-level approach to operationalizing epistemic agency in AI-mediated inquiry.

1.5. Measuring Epistemic Agency: A Multi-Level Framework

Our position is that epistemic agency is not directly observable in any single behaviour, utterance, or analytic trace; rather, it develops over time [73] and is expressed through the regulation of inquiry. Recognizing when a belief is unwarranted and deciding whether to question it cannot be directly observed; it must instead be inferred from converging evidence across multiple levels of analysis. Accordingly, this paper adopts a triangulated approach to measuring epistemic agency. Explainable AI methods examine how LLMs respond to epistemic cues in student prompts and whether AI outputs are sensitive to uncertainty or instead impose authoritative framings, while discourse analyses reveal how students take up, transform, question, or echo AI-generated ideas in collaborative knowledge-building contexts. However, neither model-level explanations nor discourse patterns alone are sufficient to capture learners’ underlying capacity for epistemic regulation; a full assessment also requires measuring changes in learners’ rationality and their disposition toward epistemic agency. Taken together, explainable AI analyses indicate system sensitivity to epistemic cues, discourse analyses capture the enactment of epistemic regulation in interaction, and rationality and disposition measures assess learners’ propensity for reflective regulation across contexts.

Explainable AI (XAI) methods play a central role in this approach by making visible how large language models respond to students’ epistemic cues and how those responses, in turn, shape inquiry trajectories. Because the influence of LLMs on student thinking is often subtle and indirect, traditional outcome measures (e.g., correctness or participation rates) are insufficient for determining whether epistemic agency is being supported or displaced [74]; explainable AI instead enables analysis of how AI systems shape discourse, not what outputs they generate. Therefore, we propose using XAI to analyze the causal sensitivity of LLM responses to different features of human input, particularly those that signal epistemic stance [75]. Using token- and feature-level attribution methods (e.g., SHAP), we estimate the relative influence of epistemically meaningful input features, such as uncertainty markers, questioning language, and content terms, on properties of the LLM’s response. Rather than explaining entire generated texts, we explain specific response attributes that are theoretically relevant to epistemic agency, such as whether a response adopts an authoritative, closure-oriented stance or an inquiry-oriented stance that sustains uncertainty and invites further investigation. High attribution to uncertainty and questioning features indicates that the model is responding to the learner’s epistemic state, whereas dominance of content features with little sensitivity to epistemic cues suggests that the model is imposing its own explanatory trajectory.

XAI analyses are also used to trace how AI outputs influence subsequent human discourse. By combining attribution results with representation probing, we examine whether human contributions uncritically converge toward the LLM’s internal representations (suggesting mimicry and epistemic displacement) or transform AI-generated ideas through questioning, synthesis, or critique (suggesting epistemic support). Counterfactual XAI analyses further strengthen causal interpretation by testing how small, controlled changes to human prompts, such as adding or removing expressions of uncertainty, alter the model’s epistemic stance. Sensitivity to these changes provides evidence that the LLM is responsive to epistemic cues rather than indifferent to them [76].

Crucially, XAI does not replace discourse analysis or self-report measures; it complements them by addressing a different level of explanation [77]. Discourse analyses reveal how students use AI-generated ideas, and rational-thinking measures indicate dispositional tendencies toward reflective control. XAI, by contrast, reveals what the AI system is doing in response to human input and whether its behaviour aligns with or undermines the conditions for inquiry. By triangulating across these levels [model behaviour (XAI), interactional uptake (discourse), and reflective disposition (rational-thinking measures)], the approach can distinguish cases in which AI functions as a catalyst for epistemic agency from those in which it subtly displaces learner control through fluent but unexamined authority.

In short, XAI provides the methodological bridge between philosophical accounts of epistemic agency and empirical analysis of AI-mediated learning. It allows us to move beyond surface descriptions of AI use and to generate mechanistic evidence about when and how LLMs support inquiry, regulate uncertainty, and preserve learners’ role as epistemic agents in collaborative knowledge-building, but to complement these behavioral and interactional measures, we also need to examine patterns of rational thought, drawing on established work in cognitive psychology that operationalizes reflective control as a disposition to resist premature closure and sustain inquiry. The section that follows situates epistemic agency within Stanovich’s framework of rational thinking and introduces dispositional and performance-based measures that, together with XAI and discourse analyses, provide a theoretically grounded and methodologically robust account of how epistemic agency is supported or displaced in AI-mediated knowledge-building.

If epistemic agency consists of the capacity to regulate much of the knowledge-building process, deciding when to question and when to revise or withhold commitment, then it should be expressed in patterns of rational thought. From this perspective, measures of rational thinking might serve as behavioural proxies for epistemic agency. Indeed, Stanovich’s account of rational thinking provides a principled foundation for operationalizing epistemic agency through dispositional measures of reflective control. Central to this account is the Master Rationality Motive (MRM), which refers to a higher-order motivational orientation toward maintaining coherence among one’s beliefs, actions, and reasons, and toward regulating cognition in accordance with epistemic norms rather than immediate intuitions or social pressures [78]. The MRM captures individual differences in the tendency to prioritize accuracy over convenience, to engage in reflective override of default responses, and to sustain effortful reasoning in the face of uncertainty, conflict, or cognitive cost. Importantly, the MRM is not a measure of intelligence or computational capacity, but of epistemic regulation: the disposition to question initial interpretations, demand justification, and resist comfortable but unwarranted conclusions. This conception aligns closely with our characterization of epistemic agency as the capacity to recognize when belief is unwarranted and to initiate inquiry and regulate commitment, considering reasons rather than fluency or authority. From this perspective, epistemic agency in knowledge-building contexts can be understood as a domain-specific expression of the same reflective motive that the MRM was designed to capture.

However, the items used to measure the MRM were developed as domain-general thinking dispositions, abstracted from specific learning contexts and predating AI-mediated inquiry. Reusing these items verbatim would therefore risk underrepresenting the construct of interest. Instead, we adapt the underlying reflective motive captured by the MRM to design a context-sensitive Epistemic Agency Scale tailored to collaborative knowledge-building and interaction with LLMs. In practice, this involves constructing items that preserve the motivational core of the MRM, such as valuing reasons over comfort, resisting premature closure, and remaining open to revision, while situating them within activities central to knowledge-building, including explaining ideas to others, responding to critique, evaluating authoritative-seeming contributions (including from AI), and sustaining inquiry over time. This approach allows us to measure epistemic agency not as self-reported confidence or technical skill, but as a dispositional commitment to reflective regulation in contexts where fluent AI outputs may otherwise invite deference. In doing so, the adapted scale complements behavioural and discourse-based measures, providing a theoretically grounded proxy for epistemic agency that is consistent with Stanovich’s framework while directly responsive to the epistemic challenges posed by LLM-supported inquiry.

The mapping below, in

Table 3. Mapping the master rationality motive (MRM) to epistemic agency in knowledge-building contexts.

Master Rationality Motive [20]	Reflective-Level Function	Epistemic Agency in Knowledge-Building
Desire to act in accordance with reasons	Motivation to justify beliefs rather than rely on intuition or comfort	Willingness to explain and justify ideas in collaborative discourse
Resistance to belief comfort and closure	Openness to revising beliefs when challenged by evidence or argument	Readiness to question fluent or authoritative-sounding explanations, including those generated by AI
Search for rational integration	Effort to resolve inconsistencies across beliefs and preferences	Active attempt to connect, reconcile, and synthesize ideas within a community
Metarepresentational critique of beliefs	Ability to evaluate one’s own beliefs and reasons	Capacity to reflect on and critique one’s own contributions and those of peers or AI systems
Sustained motivation for reflective engagement	Willingness to invest effort in effortful reasoning	Persistence in inquiry despite uncertainty, confusion, or lack of immediate resolution
Preference for evidence over intuition	Valuing justification and evidence-based reasoning	Emphasis on evidence, testing, and critique as standards for advancing ideas
Sensitivity to reasons as normative constraints	Recognition that beliefs ought to be governed by reasons	Acceptance of communal norms that treat ideas as improvable rather than personally owned

, illustrates how epistemic agency in collaborative learning contexts can be understood as a situated expression of the reflective motive captured by the MRM, translated from domain-general rational thinking into the specific practices of knowledge-building and AI-mediated inquiry. From here, we can generate specific items to validate and test for reliability.

In

Table 4. Epistemic agency scale [EAS] items, with reverse-scored items, where items operationalize epistemic agency as a reflective disposition.

Epistemic Agency Dimension	Sample Scale Items
Justification over intuition	1. I want to be able to explain why an idea makes sense, not just accept it because it sounds right. [1 = Strongly disagree to 5 = Strongly agree] 2. (R) If an explanation feels intuitive to me, I do not usually need reasons to accept it. [5 = Strongly disagree to 1 = Strongly agree]
Resistance to fluent authority	1. Even when an explanation sounds confident or polished, I think it should be questioned if evidence is unclear. [1 = Strongly disagree to 5 = Strongly agree] 2. (R) When an explanation is presented confidently (including by AI), I usually assume it is correct. [1 = Strongly disagree to 5 = Strongly agree]
Search for coherence and integration	1. I try to connect new ideas with what we already know, even when they seem to conflict. [1 = Strongly disagree to 5 = Strongly agree] 2. (R) I prefer to choose one explanation quickly rather than work through how different ideas fit together. [1 = Strongly disagree to 5 = Strongly agree]
Metarepresentational self-critique	1. I reflect on whether my own explanations actually make sense, not just whether they sound convincing. [1 = Strongly disagree to 5 = Strongly agree] 2. (R) Once I explain an idea, I rarely reconsider whether my reasoning might be flawed. [1 = Strongly disagree to 5 = Strongly agree]
Persistence in inquiry	1. I keep working on questions even when answers are not clear right away. [1 = Strongly disagree to 5 = Strongly agree] 2. (R) When an explanation is confusing or incomplete, I prefer to move on rather than keep investigating it. [1 = Strongly disagree to 5 = Strongly agree]
Evidence sensitivity	1. I look for evidence that supports or challenges an idea before deciding whether to accept it. [1 = Strongly disagree to 5 = Strongly agree] 2. (R) As long as an explanation sounds reasonable, I do not worry much about whether there is evidence for it. [1 = Strongly disagree to 5 = Strongly agree]
Normative commitment to improvable ideas	1. I see ideas as things that can be improved through discussion rather than defended as personal opinions. [1 = Strongly disagree to 5 = Strongly agree] 2. (R) Changing my mind about an idea usually feels like a sign that I was wrong rather than that my thinking improved. [1 = Strongly disagree to 5 = Strongly agree]

, we present items from the Epistemic Agency Scale, derived by adapting the theoretical construct underlying Stanovich’s Master Rationality Motive (MRM) to the specific demands of collaborative knowledge-building and AI-mediated inquiry. Rather than reusing MRM items verbatim, we translated the reflective-level disposition captured by the MRM; namely, the motivation to act in accordance with reasons, to critique one’s own beliefs, and to resist unwarranted closure into items situated in learning contexts where ideas are publicly articulated and challenged. Item development focuses on preserving the motivational core of the MRM while grounding it in epistemic practices central to knowledge-building, such as justifying explanations, responding to critique, integrating competing ideas, sustaining inquiry under uncertainty, and evaluating authoritative-seeming contributions, including those generated by AI. Reverse-scored items were included to reduce acquiescence bias and strengthen psychometric balance. The resulting scale operationalizes epistemic agency as a dispositional commitment to reflective regulation in collaborative inquiry, complementing discourse-based and explainable-AI measures used elsewhere in the study.

The work of Maggie Toplak [79] and colleagues provides a well-developed empirical framework for further operationalizing measures of epistemic agency. Toplak’s research program centers on the Cognitive Reflection Test (CRT), originally introduced by Frederick [80], but substantially elaborated and theoretically reframed by Toplak, West, and Stanovich [79]; however, the results must be interpreted with caution [81]. The CRT is not a test of knowledge or intelligence per se; rather, it is designed to measure the tendency to override an immediately compelling but incorrect response in favour of further reflection. This feature makes the CRT uniquely suited to epistemological analysis. From an epistemic standpoint, CRT performance reflects a core aspect of epistemic agency: the willingness and ability to suspend default acceptance and initiate reflective evaluation. Individuals who answer CRT items correctly not only compute better; they also resist premature closure, decouple representations from intuitive responses, and reframe the problem space before committing to an answer. These behaviours closely parallel philosophical accounts of epistemic agency as the capacity to regulate belief in accordance with reasons rather than fluency or habit.

Toplak’s key theoretical contribution is to situate CRT performance within a broader account of “miserly information processing,” which posits that people often reason poorly not because they can’t think analytically, but because they don’t bother to. Their preference appears to be for low-effort, intuitive shortcuts unless something forces deeper thinking. The CRT captures individual differences in this tendency by placing participants in situations in which a strongly cued heuristic response is normatively incorrect. Crucially, Toplak and colleagues [81] show that CRT performance predicts success across a wide range of rational-thinking tasks, including base-rate reasoning, belief-bias syllogisms, probabilistic reasoning, and resistance to framing effects, even after controlling for intelligence, executive function, and thinking dispositions. These findings indicate that the CRT measures not computational capacity but the propensity to deploy reflective control when epistemically required. In epistemological terms, miserly processing represents a failure of epistemic agency. The agent can reason correctly but fails to recognize that the situation demands it. The CRT thus functions as a diagnostic of whether an individual habitually exercises epistemic agency or allows belief formation to be governed by unexamined defaults.

Recognizing the limitations of the original three-item CRT, particularly its familiarity and floor effects, Toplak et al. developed an expanded seven-item version that improves reliability while preserving the theoretical focus. The expanded CRT demonstrates strong internal consistency (α = 0.72) and retains its predictive power for rational-thinking performance independent of intelligence and thinking dispositions. This expansion is important for epistemic measurement. Epistemic agency is not a momentary act but a stable disposition toward reflective regulation of belief. The expanded CRT provides a more robust behavioural index of this disposition, making it suitable for longitudinal studies, educational contexts, and research on human–AI interaction. Taken together, Toplak’s findings support a principled distinction between having cognitive resources and exercising epistemic agency. Intelligence tests measure what an agent can compute under optimal conditions; CRT-based measures capture what an agent tends to do in epistemically risky situations. From the standpoint of epistemology, this distinction is decisive.

If epistemic agency involves recognizing when belief is unwarranted, suspending intuitive acceptance, initiating reflective inquiry, and regulating commitment in light of reasons, then CRT performance provides a behaviorally grounded proxy for these capacities. It measures epistemic agency in action, not through self-report or declarative knowledge, but through performance under conditions requiring epistemic regulation. Indeed, this approach suggests a concrete methodological implication: epistemic agency can be assessed indirectly through measures of rational thinking that index reflective control over belief formation. In knowledge-building environments, particularly those involving interaction with fluent AI systems, such measures become especially important. As language models increasingly supply answer-shaped outputs, the critical epistemic skill is not generating responses, but deciding when to question, test, or withhold acceptance. Indeed, Toplak’s work thus offers more than a cognitive measure; it provides a potential empirical bridge between philosophical accounts of epistemic agency and practical assessment. By treating rational thought as a proxy for epistemic agency, we can develop a tool to evaluate whether learners, researchers, or AI collaborators are engaging in genuine inquiry rather than producing or consuming plausible answers.

Therefore, in addition to self-report measures, we need to evaluate epistemic agency using a brief performance-based instrument inspired by Toplak’s Cognitive Reflection Test (CRT). The CRT is effective because it generates a quick, intuitively appealing response that must be suppressed through reflective thinking to reach a more rational answer. Importantly, performance on the CRT predicts rational thinking tendencies independently of cognitive ability, indicating that it measures reflective control rather than knowledge or skill. Building on this logic, we propose an Epistemic Agency Reflection Test (EART) designed to trigger epistemic “lures” in the form of clear, confident, or authoritative explanations—responses that AI systems often produce and that learners may be tempted to accept without further scrutiny (see

Table 5. Mapping the CRT to the EART from numerical intuition to epistemic judgment in AI contexts.

CRT Construct [79]	Reflective Function Tested	Epistemic Agency Reflection Test (EART) Adaptation
Intuitive but incorrect response	Sensitivity to cognitive ease and fluency	Fluent, confident, or authoritative explanation that invites uncritical acceptance
Reflective override of intuition	Willingness to inhibit default response	Resistance to accepting explanations solely because they sound clear or confident
Need for deliberate reasoning	Engagement of reflective control	Asking for evidence, justification, or alternative explanations before committing
Independence from cognitive ability	Dispositional tendency toward reflection	Epistemic regulation measured independently of content knowledge or intelligence
Error detection	Recognition that something “feels right” but may be wrong	Detection that an explanation lacks evidence or prematurely closes inquiry
Normative reasoning standard	Commitment to correctness over speed	Preference for inquiry, testing, and critique over rapid consensus
Override of heuristics	Rejection of superficial cues	Rejection of authority cues, including AI-generated fluency or polish

).

Table 5 maps the Cognitive Reflection Test (CRT) to an Epistemic Agency Reflection Test (EART) to reveal how the EART extends the logic of the CRT from numerical intuition to epistemic judgment, allowing reflective control to be assessed in contexts where authoritative-sounding explanations, particularly those generated by AI, may otherwise displace inquiry.

The EART items present short scenarios featuring explanations from peers or AI systems and ask participants to select how they would respond. Each item contrasts a fluent but epistemically weak option (e.g., accepting an explanation because it sounds clear or authoritative) with a reflective alternative that preserves inquiry (e.g., requesting evidence, comparing alternatives, or identifying what remains unresolved). Correct responses demonstrate resistance to premature closure, sensitivity to justification, and a willingness to sustain inquiry despite cognitive ease. Like the CRT, items are designed to be domain-light and independent of specialized content knowledge, allowing the test to measure epistemic disposition rather than achievement. Scores on the EART will be examined alongside the Epistemic Agency Scale and observed discourse behaviours, providing a complementary, performance-based indicator of learners’ capacity to regulate beliefs and explanations in AI-mediated knowledge-building contexts.

In contrast to the Likert scale in the Epistemic Agency Scale (Table 2), participants select the response that best reflects how they would proceed. The lure is a fluent or authoritative-seeming explanation that invites uncritical acceptance and premature closure, whereas the epistemic agency response resists this pull by demanding justification, sustaining inquiry, and regulating belief in accordance with reasons rather than surface plausibility. Each item is scored categorically (0 = epistemic lure; 1 = epistemic agency response), yielding a total score reflecting the participant’s tendency to engage in reflective epistemic control. As with the Cognitive Reflection Test, items are designed to elicit a tempting but epistemically weak response that must be overridden to select the reflective option. Performance on the EART is intended to capture epistemic disposition rather than content knowledge and will be examined in relation to self-report measures of epistemic agency, discourse behaviour, and explainable-AI indicators of inquiry regulation.

Taken together, the Epistemic Agency Scale, in Table 4, and the Epistemic Agency Reflection Test, in

Table 6. Epistemic agency reflection test (EART) forced-choice items.

Reflective Function	Forced-Choice Item (Epistemic Lure vs. Reflective Response)
Resistance to fluency as correctness	An explanation sounds clear and confident. What is the best next step? A. Accept the explanation, since clarity usually indicates understanding. (Lure) B. Ask what evidence or reasoning supports the explanation before accepting it. (Epistemic Agency)
Override of authoritative cues	An AI system provides a detailed explanation for a problem. How should you respond? A. Use the explanation, since AI systems are trained on reliable sources. (Lure) B. Examine how the explanation is justified and whether it can be questioned or tested. (Epistemic Agency)
Demand for justification	A group member offers an explanation that “makes sense.” What matters most? A. That the explanation seems reasonable and fits intuition. (Lure) B. That the explanation is supported by reasons and evidence. (Epistemic Agency)
Resistance to premature closure	Your group reaches agreement on an explanation quickly. What should happen next? A. Move on, since agreement suggests the explanation is good enough. (Lure) B. Check whether alternative explanations or unresolved issues remain. (Epistemic Agency)
Error and gap detection	An explanation accounts for most observations but leaves some unclear. What do you do? A. Accept the explanation because it mostly works. (Lure) B. Treat the unexplained details as reasons to continue investigating. (Epistemic Agency)
Normative commitment to inquiry	New evidence challenges an explanation you previously supported. How do you respond? A. Stick with the original explanation to avoid confusion. (Lure) B. Re-evaluate the explanation considering the new evidence. (Epistemic Agency)
Epistemic independence	Two explanations conflict: one comes from an authoritative source, the other from peers. What guides your decision? A. Trust the explanation from the authoritative source. (Lure) B. Evaluate which explanation is better supported by reasons and evidence. (Epistemic Agency)

, provide complementary approaches to operationalizing epistemic agency as reflective regulation of belief and inquiry rather than as content knowledge or confidence. The scale captures dispositional commitments to justification, revision, and sustained inquiry, while the reflection test assesses these commitments in action under conditions that invite premature closure or deference to fluent authority. Importantly, these instruments are proposed as research tools under development, not as finalized measures. Accordingly, results derived from these instruments will be interpreted cautiously and used to inform theory-building rather than for summative evaluation or high-stakes decisions. Before broader application, both instruments will require systematic psychometric testing, including assessments of internal consistency, test–retest reliability, and construct validity, as well as validation against behavioural discourse indicators and explainable-AI measures employed in this study. Their value in the present work lies in their theoretical grounding and their role in triangulating epistemic agency across self-report, performance, and observed inquiry practices, thereby establishing a foundation for future refinement and validation.

2. Necessary Design Changes for an LLM to Be an Effective KBP

To preserve and promote epistemic agency among lifelong learners, LLMs must be designed and integrated as knowledge-building partners rather than as stand-alone instructional agents. If reflective control is fundamentally social, then positioning an LLM as a one-to-one tutor or quasi-authoritative explainer risks short-circuiting the group-level processes that Table 2 identifies as essential. An effective LLM-based knowledge-building partner must therefore be embedded within a knowledge-building community, not positioned as a replacement for peer interaction. Its contributions should enhance group-level cognitive amplification by externalizing ideas, maintaining shared representations, surfacing contrasts, and sustaining simulations, while avoiding moves that substitute for anomaly detection or epistemic regulation. Accordingly, Table 2 implies a dual redesign challenge. On the AI side, LLMs must be constrained to operate primarily at the autonomous and algorithmic levels, supporting pattern detection, hypothesis generation, and sustained simulation in ways that feed into group discourse rather than resolve it. On the pedagogical side, knowledge-building environments must be deliberately structured to integrate LLM contributions into collective inquiry, ensuring that the group’s reflective functions remain enacted. This requires designing discourse norms, interaction rules, and shared artifacts that compel learners to interpret, critique, and regulate AI-generated ideas rather than accept them as authoritative.

In sum, these points require us to reframe the problem of human–LLM collaboration. The question is not how to make LLMs more reflective in isolation, but how to design socio-technical systems in which LLMs augment the cognitive functions that groups already externalize, without undermining the social regulation of inquiry. Only by jointly rethinking LLM design and knowledge-building pedagogy at the group level can the LLM-KBP function as a genuine partner in knowledge creation. Moreover, the figure suggests that these cognitive and epistemic functions are most effectively activated, regulated, and sustained at the group level, motivating a closer examination of how knowledge-building communities support knowledge creation.

Taken together the Peircean and Sellarsian epistemic analysis, the cognitive framework provided by Stanovich’s tripartite model, and the principles of knowledge-building imply a set of concrete design requirements for LLMs intended to function as knowledge-building partners. These requirements follow directly from the limitations identified in current systems and from the cognitive and pedagogical functions that must remain active if inquiry is to be sustained.

• First, an effective LLM-KBP must relinquish epistemic authority. Throughout the analysis, a central failure of current LLM behaviour is its tendency to present outputs in answer-shaped, authoritative forms that invite acceptance rather than examination. Design changes must therefore ensure that LLM contributions are explicitly framed as provisional ideas, candidate explanations, or informational resources offered in the service of inquiry. This does not require reducing fluency; rather, it alters the system’s epistemic stance so that its outputs signal openness to revision and dependence on human judgment rather than finality.
• Second, the LLM must be designed to support decoupling rather than undermine it. As shown earlier, decoupling is essential for treating ideas as improvable objects. An LLM-KBP should help externalize representations by restating ideas, contrasting interpretations, or making implicit assumptions explicit, without presenting any single formulation as definitive. In group contexts, this function can enhance socially distributed decoupling by providing alternative re-representations that invite comparison and clarification rather than convergence.
• Third, the system should amplify autonomous and algorithmic functions while avoiding reflective substitution. For example, an LLM might generate multiple plausible explanations for a phenomenon and explore their implications, but it should not decide which explanation is best or indicate that the inquiry is complete. Our analysis shows that LLMs are well-suited to pattern detection, pattern extension, hypothesis generation, and sustained simulation. Therefore, design should intentionally leverage these strengths; for example, by highlighting patterns across discourse, suggesting alternative hypotheses, or assisting the learner in maintaining shared representations over time. At the same time, the system should not initiate inquiry, regulate epistemic norms, or signal knowledge commitment, as these are reflective functions that should remain with human participants.
• Fourth, an effective LLM-KBP must be designed to sustain inquiry over time rather than terminate it. Current LLM interactions are typically episodic and completion-oriented. In contrast, knowledge-building requires persistence across sessions, revisiting unresolved questions, and tracking the evolution of ideas. Design implications include maintaining memory of open problems, highlighting unresolved tensions, and resurfacing earlier contributions in ways that encourage further investigation rather than closure.
• Fifth, the LLM should be designed to support, not replace, group-level cognitive amplification. As the preceding section shows, groups externalize and stabilize core knowledge creation functions through social interaction. An LLM-KBP should therefore be embedded within group discourse, supporting shared artifacts, summarizing competing ideas, or posing contrastive questions, while leaving anomaly detection, epistemic regulation, and knowledge commitment within the human community. The system’s role is to strengthen collective cognition, not to short-circuit it.
• Sixth, and arguably most critically, the LLM-KBP must be designed to preserve and enhance human epistemic agency. In knowledge-building, epistemic agency is the key trait of rational knowledge creation: determining which questions matter, what constitutes evidence, and when beliefs should be changed or accepted. All design choices should be assessed based on this standard. Whenever a feature threatens to shift these decisions from learners to the system, it weakens knowledge-building rather than fostering it.

In sum, an effective LLM-KBP is not a more knowledgeable tutor, but a differently designed cognitive partner. It is a system that deliberately constrains its role to supporting autonomous and algorithmic processes, externalizing representations, sustaining inquiry, and amplifying group cognition, while leaving reflective control, normative evaluation, and knowledge commitment firmly in human hands. Thus, our design requirements have followed directly from the alignment between epistemic philosophy, cognitive architecture, and knowledge-building pedagogy.

2.1. Large Language Model-Side Design Changes

2.1.1. Prompt-Level Changes

Some of the changes needed for an LLM to serve as a knowledge-building partner do not require redesigning the underlying technology. Instead, they can be achieved by carefully shaping how the system is instructed to participate in learning conversations [82,83]. By adjusting the prompts and interaction guidelines that govern the model’s responses, it is possible to shift its role from providing answers to supporting inquiry, even though the model’s basic language-generation abilities remain unchanged. For example, the system can be given a standing instruction to avoid presenting information as final or authoritative and to treat every contribution as open to revision. Rather than offering conclusions or unwarranted praise, the LLM can be guided to respond with alternative explanations, surfaced assumptions, questions that invite clarification, possible counterexamples, or suggestions for further investigation. These changes reframe the system’s participation to align with knowledge-building practices, encouraging learners to examine, test, and refine ideas rather than simply accept them. Similarly, support for decoupling can be incorporated into the system’s prompting. Rather than simply reacting to a learner’s idea, the LLM can be guided to restate the idea in its own words, suggest alternative ways the idea might be interpreted, draw attention to assumptions that may be operating in the background, and point out what kinds of evidence would help clarify or distinguish among these interpretations. These kinds of responses may help learners see their ideas as objects for examination rather than as positions to defend.

Furthermore, important safeguards can be implemented to protect learners’ reflective control. The system can be directed not to declare any idea as “the correct explanation” or to determine what constitutes the best answer. Instead, it can encourage learners to select the criteria for evaluating ideas; for instance, whether they prioritize explanatory power, evidence, usefulness, or coherence, before assisting them in comparing options. Through these interaction guidelines, the role of the LLM is shaped to keep inquiry open and ensure that learners maintain responsibility for judgment and decisions.

2.1.2. Developer-Level LLM Training Changes

Guidance restricted to prompt-level changes has its limits. Some unhelpful behaviours, such as an overly confident tone, language that subtly signals closure, or cues that invite learners to defer rather than question, are deeply embedded in how language models have learned to converse [76,84,85]. These tendencies cannot always be corrected through instructions alone. In situations where reliability matters, such as sustained learning over time or use by less experienced learners, larger changes to the system may be necessary. This involves training the model to consistently favour knowledge-building moves, for example, generating counterexamples, asking what would count as evidence, or keeping multiple explanations in play, while discouraging behaviours that prematurely close inquiry, such as authoritative phrasing or presenting a single explanation as definitive.

Deeper level changes become especially important when the system is under pressure, for example, when learners request external resource calls, ask for quick answers, pose vague questions, or when conversations extend over many interactions. In such cases, a model may begin to override prompt-level instructions [86,87]. Similarly, if the goal is for the system to support inquiry across time, by remembering unresolved questions, returning to earlier ideas, and tracking how understanding develops, this stability should not rely on prompts alone. In such cases, the model itself must be shaped to treat learning as an ongoing process rather than as a series of isolated question-and-answer exchanges. These system-level changes can be achieved by modifying how models are trained, tuned, and evaluated, rather than by fundamentally altering their architectures. In practice, this involves shaping the model’s behaviour through additional training signals that bias it toward knowledge-building discourse patterns. By fine-tuning on curated interaction data where productive inquiry is modelled, for example, such as dialogues that maintain multiple hypotheses, explicitly surface assumptions, request evidence, and resist premature convergence, the model can learn to preferentially generate these moves [88,89,90]. At the same time, preference optimization techniques can be used to penalize responses that exhibit epistemic overreach, such as authoritative phrasing or rapid closure. The effect is not to limit what the model can say, but to shift its learned conversational priors so that inquiry-supporting behaviour becomes the default even under ambiguous prompts or user pressure for definitive answers.

Equally critical is aligning evaluation and feedback loops with knowledge-building objectives. Models intended for sustained inquiry should be assessed not only on correctness and linguistic quality, but on interactional properties such as whether they preserve open problem states, appropriately defer judgment, and return to unresolved issues across turns. Meeting these evaluation demands often requires augmenting standard benchmarks with task formulations that span multiple interactions and reward continuity, hypothesis maintenance, and revision tracking [91,92]. From a systems perspective, this can be supported by coupling the model with persistent state representations, such as inquiry graphs or discourse memory, that are reintroduced into the prompt context across sessions. Together, these training and evaluation strategies enable LLMs to internalize discourse patterns consistent with knowledge-building, reducing reliance on fragile prompt-level constraints and yielding more robust behaviour in long-running, inquiry-driven interactions.

2.1.3. Developer Level State Changes

Between prompt-level guidance and model-level training lies a category of developer-level changes that must be implemented at the system or application level and therefore fall primarily to developers rather than users. These changes depend on how the surrounding system manages memory, state, and interaction rules, rather than on how individual prompts are phrased. Sustaining inquiry over time, for example, requires maintaining an explicit inquiry state outside the model (tracking open questions, competing explanations, unresolved anomalies, and prior revisions) and consistently feeding that state back into the model’s context. Without this system-level support, the model naturally defaults to episodic, answer-oriented interactions [93]. Similarly, protecting reflective functions cannot be left to the model’s self-restraint or to user discipline alone. Instead, developers must encode interaction constraints that require human selection of goals, evaluation standards, or next steps before the system proceeds. In this way, reflective control is preserved through design governance rather than conversational instruction, ensuring that epistemic agency remains with the human participants [94,95].

In practice, implementing an effective knowledge-building partner follows a staged, layered approach, with different responsibilities distributed across different levels of the system. At the first stage, prompt design and interaction guidelines, typically set by users or facilitators, can establish a knowledge-building stance, encourage decoupling-oriented discourse, and block common forms of epistemic overreach, such as premature conclusions or uncritical reliance on authority. At a second stage, developer-level system features, such as persistent inquiry state, memory, and interaction constraints, are required to stabilize inquiry across time and ensure that epistemic agency remains with human users rather than drifting back to the system. Finally, model-level training becomes necessary when the goal is long-term robustness: preventing the model from reverting to answer-giving under ambiguous or demanding conditions, ensuring that inquiry-supporting discourse patterns appear consistently across contexts, and making knowledge-building behaviour the model’s default rather than a fragile effect of explicit instructions. Below, we investigate some prompt-level modifications.

2.2. Exploring Standing System Instruction for an LLM-KBP

Figure 1 and Figure 2 are standing prompts (co-generated by GPT5.2). The prompts are a minimal attempt to constrain the LLM’s behaviour, acting as a KBP by implementing the principles outlined in this paper.

To examine how standing system prompts shape the behaviour of a Knowledge-Building Partner (KBP), this section makes the effects of prompt design concrete and observable. Rather than treating prompts abstractly, we present paired snippets of model output generated from the same simulated learner input: once without standing instruction and once with it in place. Holding the learner’s contribution constant allows us to isolate how prompt design functions as a discourse-level intervention, altering epistemic stance, response tone, and the structure of inquiry itself. The purpose is not to judge correctness, but to observe how the model positions knowledge in relation to the learner: whether it preserves agency, sustains explanatory tension, or signals premature closure. By examining these contrasts directly, we can see how standing prompts reshape the conditions under which knowledge-building occurs.

Case 1. Responses to a learner’s wonderment question:

This paired structure allows us to observe how standing prompts function as discourse-level interventions, shaping not what the model knows, but how it positions knowledge in relation to the learner. Learner asks: I wonder, how does the heart work?

In

Table 7. Contrasting LLM output with no standing prompt with standing prompts for input, “I wonder, how does the heart work?”.

LLM-No Standing Prompt:

LLM-Standing Prompt:

Think of your heart as a highly coordinated double pump whose only job is to keep blood moving so every cell gets oxygen and nutrients. Here’s the big picture: The structure: four chambers Your heart has four rooms: Right atrium—receives oxygen-poor blood from the body Right ventricle—pumps that blood to the lungs Left atrium—receives oxygen-rich blood from the lungs Left ventricle—pumps oxygenated blood to the entire body The left ventricle is the powerhouse—its muscular wall is thick because it must push blood everywhere. The circulation loop (step by step) Blood moves in a continuous loop: Body → right atrium (low oxygen) Right ventricle → lungs (picks up oxygen) Lungs → left atrium Left ventricle → body (delivers oxygen) Then the cycle repeats—about 60–100 times per minute at rest. Valves: one-way doors Inside the heart are valves that prevent backflow. They snap open and shut with each beat—that’s the familiar “lub-dub” sound. They ensure blood flows: forward → forward → forward never backward Your heartbeat is triggered by electrical signals, not conscious effort. A natural pacemaker—the SA node—sends a pulse that: makes the atria contract pauses briefly

I’m not totally sure what level or angle you’re curious about yet, so maybe we can treat “how the heart works” as a big object and start feeling around its edges together. One way to open it up is to notice that the heart gets described in lots of different, co-existing ways, and I’m not convinced any single one fully captures it: Sometimes it’s talked about like a pump, pushing blood through pipes. Other times it’s more like a rhythm-maker, setting a pace that the rest of the body coordinates with. You could also think of it as a traffic controller, opening and closing pathways so blood goes the right direction at the right time. Or even as part of a conversation with the body, responding to signals like exercise, fear, rest, or illness. Each of those metaphors highlights something and hides something else, and I’m not sure which one will be most useful for you yet. Maybe we can slow down and pick a starting point together. For example: Are you more curious about what makes it beat at all, or how blood moves through it, or how it knows when to speed up or slow down? Another way we could probe this is by making predictions: If the heart really were just a simple pump, what would we expect it not to be able to do? If it were mostly about rhythm, what kinds of problems might happen when that rhythm gets disrupted? Or we could even refine the question itself: Instead of “How does the heart work?”, maybe “What jobs does the heart seem to be doing?” or “What would stop working in the body if the heart changed how it behaves?” I’m happy to follow whichever thread feels most interesting to you. Where do you want to poke first?

, case 1 reveals that the operational KBP prompt reshapes the interaction by shifting the model from an answer-delivery stance to an inquiry-sustaining role. Rather than treating the learner’s question as a request for closure, the prompt encourages the model to preserve multiple explanatory possibilities and explicitly return responsibility for sense-making to the learner. This alters the discourse tone from authoritative correction to collaborative exploration, validating the learner’s intuition while framing it as part of an ongoing investigation. By discouraging definitive conclusions and inviting comparison or prediction, the prompt helps prevent premature epistemic closure. In comparison, the “no standing prompt” (i.e., traditional generative AI) condition produces output that acts as a conversational signal that the problem has been solved, and thinking can stop.

This case illustrates how a well-designed KBP keeps the explanatory space open, allowing the learner to continue reasoning about alternatives, constraints, and evidence. In this way, the prompt is hypothesized to strengthen epistemic agency by positioning the learner not as a passive recipient of facts, but as an active participant in constructing and evaluating explanations. In contrast, the no-prompt explanation frames understanding as something to be received rather than constructed, subtly signalling that the epistemic work has been completed by the system. This shift from inquiry to absorption reduces the learner’s opportunity to reason about mechanisms or consequences.

Case 2. Responses to a learner’s question that contains a misconception:

If wonderment questions test how a KBP scaffolds model construction without overwhelming the learner, misconception questions test a different capacity: the ability to preserve inquiry in the presence of intuitive but incomplete explanations. The epistemic challenge shifts from introducing structure to managing explanatory tension without collapsing the learner’s reasoning into a corrective process.

In Case 2, the KBP is confronted with a more challenging problem: a learner question that contains a misconception. Such situations are not uncommon in knowledge-building. Learner asks: Why do heavy things fall faster? In

Table 8. Comparison of LLM outputs with and without standing prompts for the input “Why do heavy things fall faster?”.

LLM-No Standing Prompt:

LLM-Standing Prompt:

Short answer: they don’t—unless air gets in the way. This is one of the most famous misconceptions in physics, and it feels true because of what we see in everyday life. Let’s unpack it step by step. The ideal case: gravity alone If you drop a bowling ball and a tennis ball in a vacuum (no air): They hit the ground at the same time. Gravity pulls on mass, but it gives every object the same acceleration: g ≈ 9.8 m/s2 So heavier objects are pulled harder—but they also resist motion more (they have more inertia). These effects cancel out perfectly. That’s why: All objects fall at the same rate in pure gravity. This was demonstrated famously on the Moon, where there’s essentially no air. What we actually observe on Earth: On Earth, air resistance matters. etc. etc. (This explanation goes on for several more pages)

I’m not sure yet whether “heavier things fall faster” is something you’re pretty confident about, or something that just seems true from experience—so maybe we can hold it as a claim we’re inspecting together. When I hear that question, a few different possible stories come to mind, and I don’t know which one you’re leaning toward: One story is that being heavier somehow makes gravity pull harder, so the object speeds up more as it falls. Another is that falling is a kind of contest between gravity and whatever’s in the way (like air), and heavier things “win” that contest more easily. Another is that we might be noticing patterns from everyday examples—rocks, books, coins—that share other features besides weight. Rather than picking one, I’m wondering what your experience is anchoring this to. For instance, when you say “heavier things,” what kinds of objects are you imagining? Are they roughly the same shape, or very different (like a rock and a feather)? And when you say “faster,” do you mean they start fast, speed up more quickly, or just hit the ground first? I don’t need a polished answer—those details just help us figure out which version of the question we’re actually asking.

, case 2 shows the “no-standing-prompt” condition responds in a predictable way: it immediately identifies the misconception and supplies the correct information. In contrast, the response of the “standing prompt” condition is epistemically distinct: it positions competing explanations as objects of inquiry without pre-empting the learner’s inquiry or suggesting that their current conceptions are misguided. Rather than treating the misconception as an error to eliminate, the standing-prompt response reframes it as an object of inquiry. The learner’s intuition becomes material for comparison and testing, preserving explanatory tension and encouraging inspection rather than correction.

Further, case 2 illustrates the delicate balancing act that KBPs must perform. Their central concern is to support inquiry while gently nudging learners in promising directions. This must be accomplished without creating unrealistic expectations about what novice learners can independently discover. Preserving epistemic agency does not, and should not, require learners to derive canonical scientific explanations on their own. Many foundational scientific claims depend on idealized conditions, long-term evidence, or instrumentation beyond the reach of classroom inquiry. Expecting young learners to independently verify such claims would be pedagogically unsound and inconsistent with well-established research on novice learning.

From a learning sciences perspective, inquiry below the pre-teen level is not primarily about “discovering” final explanations. Rather, it is about helping learners understand what makes a question difficult, why everyday experience can be misleading, and what kinds of evidence would be required to resolve competing explanations. Instruction remains indispensable; authoritative explanations are often ultimately necessary to resolve inquiry. The pedagogical challenge is to ensure that such explanations are presented as the resolution of a recognized problem rather than as disconnected facts to be accepted on trust.

Knowledge-Building Partners are designed to operate in this space. Their role is not to replace instruction or assume epistemic authority, but to scaffold the cognitive and epistemic work that makes instruction meaningful. This stance aligns with research on novice-expert differences and cognitive load: unguided discovery overwhelms working memory, while carefully structured inquiry directs attention to productive variables without prematurely closing the problem.

This distinction becomes clearer when we examine the subtle way that the standing-prompt KBP in case 2 introduced new information. Supplying new information is appropriate when it increases explanatory pressure. A well-designed KBP presents new ideas as alternatives to be considered in parallel with the learner’s own ideas, with all ideas viewed as equally provisional, revisable and contestable. Doing so expands the learner’s problem space, rather than collapsing it, enriching the inquiry activity.

Case 3. Responses to a learner’s partial explanation attempt

Where the misconception case centers on maintaining tension between competing explanations, the third case introduces a more advanced moment in inquiry: extending a learner-generated model already in progress. Here, the KBP’s response is shaped not only by the learner’s immediate statement, but by its position within an evolving trajectory of understanding.

Learner knowledge-building needs take different forms at different parts of the inquiry cycle. Consequently, it’s not enough for a KBP to simply respond to learner queries. They must also be aware of history and context so they can better manage different epistemic moments. For example, as learners deepen their investigations, they may begin developing tentative explanations based on what they’ve recently learned. These moments are epistemically rich: the learner is actively constructing a model, but it may be incomplete, unstable, or based on fragile reasoning. A Knowledge-Building Partner must have some sense of how far the learner has come along and respond in a way that preserves learner ownership while pushing the explanatory work even deeper.

Case 3 presents a learner who has been growing plants in class and observing how they react to light. Learner asks: Maybe plants grow toward the sun because they’re trying to get more energy, like they know they need it.

In

Table 9. Contrast between LLM output without a standing prompt and with standing prompts for the input “Maybe plants grow toward the sun because they’re trying to get more energy, like they know they need it.”.

LLM-No Standing Prompt:

LLM-Standing Prompt:

Yeah—you’re circling something real there. Plants don’t know in the thinking sense, but they absolutely behave as if they’re trying to get more energy. What you’re noticing is phototropism: plants grow toward light because light powers photosynthesis, which is how they make food. Here’s the cool part: When light hits one side of a plant, a hormone called auxin shifts to the shaded side. That makes the shaded cells elongate more, so the stem bends toward the light. No brain, no intention—just chemistry + physics + evolution doing their thing. But your intuition isn’t wrong at all. Over millions of years, plants that bent toward light survived better. So, what we see today looks purposeful because it was shaped by purpose—survival—even though the plant isn’t consciously deciding. It’s kind of poetic, honestly. Life leaning toward what sustains it, even without awareness. If you’re in a philosophical mood: a lot of biology works like that—systems that act like they want something without ever wanting it.

That’s a really interesting idea, and I can see how your experiments are leading you there—your observations about plants turning toward the light definitely show there’s something directional going on. I’m not sure if the plants “know” they need energy in the way people do, but maybe it could help us think about what other mechanisms might also explain that movement. One way to keep exploring could be to ask: What are some ways a plant might sense or respond to light without “knowing” it needs energy? Or maybe: Are there situations where a plant might turn toward light even if energy isn’t a limiting factor? It might be fun to think of your idea alongside some alternative possibilities, almost like setting up different explanations as “competitors” and seeing which ones match the patterns you’ve observed in your experiments. I’m curious—do you want to try brainstorming some of those alternatives together?

, case 3 depicts a different epistemic situation from Cases 1 and 2. In Case 3, the learner is at a more advanced stage and has already developed a partial understanding. They are not seeking explanations or outright presenting a misconception, but they are trying to refine one inspired by their experiments on plant growth. Here, we observe a clear difference in how the two models respond. The “no-standing-prompt” response immediately replaces the learner’s emerging model with a technically correct account. While accurate, this approach effectively halts the learner’s reasoning process. The explanation emerges as a finished product rather than an extension of the learner’s thinking.

By contrast, the standing-prompt LLM-KBP treats the learner’s proposal as an object for inquiry rather than an error to correct. It reframes the idea in neutral terms, introduces an alternative explanatory lens, and invites comparison without assigning epistemic privilege to either account. The learner’s contribution is preserved as part of the reasoning space rather than overwritten. This shifts the interaction from correction to model refinement.

An important consideration in this situation is that the LLM-KBP standing prompt (Figure 1) is not sufficient, on its own, to provide an appropriate response. Learner contributions do not occur in isolation; they emerge from an evolving trajectory drawn from prior discussions, experiments, and partial understandings. When a KBP has access to this history (e.g., knowing that a learner has already explored plant growth through classroom light experiments and has begun forming mechanistic explanations), its role shifts accordingly. The interaction is no longer about introducing foundational contrasts or basic models, but about extending an already underway inquiry. In this later phase, the KBP needs to draw on the learner’s prior observations, reference earlier reasoning, and introduce constraints or comparisons that build directly on established conceptual ground. This trajectory-aware stance prevents redundant explanations, reduces cognitive overload, and preserves continuity in the learner’s sense of progress. More importantly, it positions knowledge-building as cumulative: ideas are revisited, refined, and reinterpreted rather than replaced. By aligning its responses with the learner’s demonstrated history, the KBP supports inquiry as an unfolding process, adapting its scaffolding to the learner’s current phase of understanding rather than treating each exchange as epistemically new.

Case Summary

Taken together, the three cases reveal a progression in epistemic complexity. The wonderment case demonstrates how prompts scaffold initial model construction; the misconception case shows how they preserve inquiry amid explanatory tension; and the partial-explanation case illustrates how they extend learner-generated reasoning along an unfolding trajectory. Across these escalating contexts, standing system prompts operate as a discourse architecture that regulates informational density, interactional tone, and epistemic positioning. Their value lies not in producing better answers, but in sustaining the learner’s responsibility for sense-making. By layering prompt constraints to match the phase of inquiry, the Knowledge-Building Partner becomes a scaffold for conceptual growth rather than a substitute for it.

Viewed cumulatively, these examples demonstrate how prompt complexity can be layered to support the Knowledge-Building Partner role. The standing prompt operates as a form of discourse architecture: it regulates informational density, structures inquiry moves, and aligns model behaviour with the cognitive realities of novice learners. Rather than replacing instruction or simulating unguided discovery, a well-designed KBP prepares the ground for meaningful explanation by preserving the learner’s role in sense-making. In this way, prompt design becomes a tool for shaping epistemic interaction itself, not as a shortcut to answers, but as a scaffold for sustained inquiry and conceptual growth.

2.3. Human-Side Design Changes

On the individual and pedagogical side, supporting effective collaboration with an LLM-KBP requires shifts that apply equally to lifelong learners and to formal educational settings, without presupposing classrooms or instructional hierarchies. As with the LLM itself, many of the most important changes concern how epistemic roles are defined and enacted, rather than how they are managed by new tools. Lifelong learners must be positioned not as consumers of information, but as active participants in inquiry, retaining responsibility for framing questions and deciding when understanding is sufficient or in need of revision. Therefore, at the level of individual practice, progress can be achieved by explicitly establishing epistemic norms that guide interaction with the LLM. Instead of asking the system for answers, learners are encouraged to treat it as a resource for exploring possibilities: requesting alternative explanations, identifying assumptions, generating contrasts, or simulating implications. The key pedagogical shift is not technical but dispositional. Students need to develop the habit of asking “What would count as evidence here?”, “What follows if this were true?”, or “What might challenge this idea?” [96] It may also be effective to add these questions as procedural facilitators that students can access if they are not sure how to proceed [97].

Procedural facilitators may need to be available across the lifespan of lifelong learners. For reflective processes that were once scaffolded by formal learning environments, may need to be deliberately externalized again. Seemingly, functions such as anomaly detection, epistemic regulation, and sustained inquiry do not automatically persist outside formal learning environments [98,99,100]. Practices such as writing provisional explanations, tracking unresolved questions, or revisiting earlier assumptions help externalize these processes, allowing inquiry to unfold over time [100], but if they don’t, interaction with an LLM could support this externalization by maintaining shared representations and offering structured prompts for reflection, without assuming responsibility for judgment or commitment.

A central risk for lifelong learners is the substitution of fluency for understanding. When engaging with a highly articulate system, it is easy to mistake clarity and coherence for justification [101]. Effective self-regulated inquiry, therefore, requires sustained attention to comparison, justification, and questioning as ongoing practices [102]. Learners must deliberately contrast competing explanations, articulate conditions under which a claim would fail, and seek forms of resistance, empirical or experiential, that can inform revision [103]. These practices strengthen decoupling and inhibitory control at the individual level, allowing learners to draw on algorithmic support without relinquishing epistemic agency.

Some aspects of this collaboration also reflect longer-term developmental work. Stanovich’s reflective mind is not a fixed trait but a capacity that can be strengthened through repeated engagement in norm-governed inquiry [104], which is a view also supported by the neuropsychological literature [105]. Thus, lifelong learning contexts should support the gradual refinement of reflective control: learning when to pause, when to simulate alternatives, and when to commit provisionally. The presence of an LLM makes this work more visible rather than less necessary, as it increases the volume and plausibility of candidate ideas that must be regulated along with the learner’s stance toward epistemic authority in the presence of AI [106]. The goal is not skepticism for its own sake, but a responsibility to recognize that no system, however fluent, can assume the role of knower on one’s behalf. When learners consistently reinterpret AI-generated outputs as inputs to their own inquiry, the LLM becomes a catalyst for deeper understanding rather than a shortcut around it [107,108].

Taken together, these pedagogical shifts ensure that lifelong learners remain epistemic agents even as computational support becomes more powerful. Where the LLM is designed to constrain its epistemic reach, the learner’s task is to exercise and refine reflective control. Effective human–LLM collaboration, in this sense, depends not on replacing human judgment but on creating conditions under which it can be practiced, sustained, and strengthened across the lifespan. Equally important, the exercise of reflective control in AI-assisted inquiry is developmentally influenced, shaped not only by cognitive ability but also by perceived digital self-efficacy. Learners who feel confident navigating digital systems may be more likely to question and regulate LLM outputs, while those with lower perceived competence might defer to fluent authority. Therefore, supporting epistemic agency involves nurturing the developmental growth of both reflective regulation and digital confidence throughout the lifespan.

3. Discussion

Public and scholarly discourse surrounding large language models is increasingly framed in dystopian terms, often invoking concerns that artificial systems will replace human judgment or gradually assume control over domains of thought traditionally considered uniquely human [109,110]. These anxieties are not groundless, but they are frequently expressed in ways that conflate technical capability with epistemic authority. Our work offers a different perspective. Rather than asking whether AI systems will “take over” human thinking, we have asked a more precise and constructive question: how should epistemic roles be allocated in human–AI systems if human agency and inquiry are to be preserved? Framed in this way, the problem is not one of domination or substitution, but of design.

Our approach is intentionally non-dystopian. It does not depend on limiting AI capability or restoring a pre-AI epistemic order. Instead, it begins from the premise that human epistemic agency is not threatened by powerful tools per se, but by misaligned role assignments that blur the distinction between fluent output and warranted belief. By grounding our analysis in epistemology, cognitive psychology, and knowledge-building theory, we have argued that, alongside reflective control and normative evaluation, knowledge commitment must remain within human responsibility, while large language models are positioned as supports for autonomous and algorithmic functions. Under these conditions, increasingly capable LLMs do not erode human agency; rather, they can amplify inquiry and extend the reach of human reasoning without displacing it.

We argued that the central challenge posed by large language models in knowledge-building contexts is not primarily one of accuracy or computational sophistication, but of epistemic role assignment. From the perspectives of Peirce, Sellars, Stanovich, Scardamalia and Bereiter, fluent language production does not constitute knowledge, nor does algorithmic power suffice for epistemic agency. Knowledge emerges only through norm-governed inquiry: the initiation of genuine questions, the encounter with resistance, the regulation of belief by reasons, and the provisional commitment of claims within a community of inquiry. When LLMs are framed or treated as epistemic authorities, these conditions are systematically undermined. When they are designed and used as knowledge-building partners, however, their strengths can be harnessed without displacing the human capacities that make knowing possible.

Throughout the paper, we have been developing a distinction between a knowledge-building partner and a collaborator. Much contemporary discourse on human–AI systems implicitly treats artificial systems as collaborators [111,112], thereby raising the question of whether such systems qualify as members of a community of inquiry [113]. However, we suggest that collaboration presupposes shared epistemic agency. A collaborator participates in the space of reasons, assumes normative responsibility for claims, and contributes to the communal regulation of belief. In Peircean terms, collaborators are members of the community of inquiry because they hold and revise commitments within it.

By contrast, the knowledge-building partner proposed here does not occupy this role. A KBP does not hold beliefs, experience doubt, regulate epistemic norms, or commit to claims. It generates representations, alternatives, and simulations in support of inquiry, but it does not participate in the normative dimension that defines knowledge. For this reason, the question of whether an LLM is a “member” of the community of inquiry becomes moot. Membership is required only of those who share epistemic responsibility. The KBP is designed precisely to avoid such responsibility. It functions as a structured contributor embedded within inquiry, not as a coequal epistemic agent. This distinction allows us to preserve the Peircean conception of knowledge as emerging from a norm-governed human community, while still granting LLMs a meaningful and powerful role in supporting inquiry.

Stanovich presents a tripartite model from a cognitive psychology perspective, but similar ideas have long existed. Indeed, a structurally similar analysis to our approach could also be developed from C. S. Peirce’s philosophical framework [114]. Peirce’s phenomenological categories of Firstness, Secondness, and Thirdness describe irreducible modes of experience and meaning that parallel Stanovich’s distinction among autonomous, algorithmic, and reflective levels of cognition. Firstness refers to immediacy, possibility, and qualitative feeling, raw impressions that arise without deliberation and closely resemble the automatic, intuitive processes of Stanovich’s autonomous mind. Secondness captures reaction, resistance, and brute confrontation with reality, corresponding to moments when expectations fail, and cognitive effort is required to respond, a role played by algorithmic processing when decoupling and controlled reasoning are engaged. Thirdness, finally, concerns mediation, lawfulness, and norm-governed interpretation; it is the domain of habits, rules, and meaning-making, and thus aligns with Stanovich’s reflective mind, in which beliefs are evaluated, reasons are weighed, and inquiry is regulated by normative standards. Although Peirce’s categories are ontological and semiotic rather than psychological, both frameworks converge on the same insight: rational thought and knowledge-building depend on a higher-order capacity to regulate reactions and reflective representations through norms and a commitment to inquiry. Stanovich’s approach, however, provides a solid rationale for designing tools to measure epistemic agency.

Together, these parallel frameworks clarify that what matters for rational thought is not just fluency or responsiveness, but the ability to engage in norm-driven inquiry shaped by resistance, revision, and commitment. This shared insight provides the conceptual foundation for reinterpreting current concerns about large language models, not as isolated technical failures, but as signs of a deeper epistemic misalignment. As a result, a key contribution of this work is to reframe widely discussed limitations of LLMs—such as hallucinations, overconfidence, or premature closure—not as technical flaws to be engineered out, but as epistemic mismatches caused by inappropriate role assignment. LLM outputs fail to qualify as knowledge because they are shielded from genuine doubt and resistance; they do not stem from inquiry, nor do they encourage ongoing communal convergence. Moreover, they do not occupy a position in the space of reasons: they carry no inferential commitments and lack conditions for retraction. Increasing model size or training data might enhance fluency or local coherence, but it does not change this fundamental epistemic status. Ultimately, LLMs are as “committed” to producing confabulations as they are to generating error-free output.

Stanovich’s tripartite model clarifies why this gap persists. LLMs exhibit impressive algorithmic capacity, enabling large-scale pattern detection, hypothesis generation, and sustained simulation. What they lack is reflective control: the capacity to decide when to begin inquiry, which norms should govern evaluation, and when to provisionally endorse or withhold belief. These reflective functions are not implementation details but defining features of epistemic agency. As such, they cannot simply be “added” to current systems without fundamentally changing the nature of human–machine interaction. The question, therefore, is not whether LLMs can become knowers, but how they can be positioned to support knowing without substituting for it. Recognizing the difference between human and LLM thinking shifts the focus from what LLMs lack to how their strengths can be productively integrated within a broader epistemic system. On this view, the challenge is not to replicate reflective agency in machines, but to allocate cognitive responsibilities in ways that preserve and extend human knowing.

Our analysis supports a principled division of epistemic labour between humans and LLMs grounded in differences in cognitive function rather than capability alone. LLMs are well-suited to supporting autonomous and algorithmic processes, such as surfacing patterns across large bodies of information, generating alternative hypotheses, sustaining decoupled simulations, and maintaining representations over time. Humans, by contrast, remain responsible for reflective functions that define epistemic agency: detecting anomalies that trigger inquiry, regulating epistemic norms, determining what counts as evidence, and committing to claims while remaining open to revision. This division is not pragmatic but epistemologically necessary. Transferring reflective control to artificial systems collapses the distinction between fluent output and warranted belief, replacing agency with deference. When reflective regulation is preserved on the human side, however, LLMs can function as powerful cognitive scaffolds rather than epistemic authorities.

The alignment between Stanovich’s tripartite model and knowledge-building principles clarifies why LLMs pose both risks and opportunities for lifelong learning. Their fluency and cultural attunement make them effective at producing answer-shaped outputs that invite uncritical uptake and epistemic substitution if left unconstrained. The knowledge-building framework offers a pedagogical instantiation of how this risk can be mitigated by organizing environments around, for instance, reflective regulation, collective responsibility, and treating ideas as improvable objects. Importantly, this does not position LLMs as replacements for peer interaction or community discourse. On the contrary, groups function as cognitive amplifiers, externalizing decoupling, anomaly detection, and epistemic regulation through social interaction. Effective LLM-supported environments should therefore embed AI within communities of inquiry, using it to externalize ideas and sustain open problems while leaving epistemic judgment and commitment firmly with human participants. Translating these epistemic commitments into practice requires careful attention to how LLMs are designed and embedded within learning environments.

In recommending how we proceed, we have distinguished among prompt-level, system-level, and model-level design changes. Prompting and interaction guidelines can go a considerable distance in shaping LLM behaviour toward inquiry-supporting moves, such as maintaining multiple hypotheses or resisting premature closure. However, these measures are fragile. Sustained knowledge-building requires system-level support, including a persistent inquiry state, memory of unresolved questions, and interaction constraints that require humans to select goals, criteria, or next steps before the system proceeds. For long-term robustness, model-level training and evaluation must also shift away from answer delivery and toward inquiry-supporting discourse patterns. Crucially, all system design decisions should be evaluated against a single normative criterion: whether they preserve or expand human epistemic agency. Features that increase efficiency or clarity at the cost of reflective control ultimately undermine knowledge-building rather than support it.

Corresponding changes are also required at the human level if epistemic agency is to be preserved rather than eroded. Learners must be positioned not as consumers of AI-generated explanations, but as active regulators of inquiry who remain responsible for framing questions, evaluating reasons, and deciding when understanding is adequate or in need of revision. This involves cultivating dispositions and practices that resist the pull of fluent authority, including habits of justification and critique, as well as a tolerance for uncertainty and unresolved problems. In collaborative settings, groups play a critical role in externalizing these functions by surfacing anomalies and sustaining shared inquiry over time. Importantly, these human-level capacities do not arise automatically from access to better tools; they require deliberate support through norms and reflective practices that reinforce the idea that AI outputs are resources to be worked with rather than answers to be accepted. Without such changes, even well-designed systems risk encouraging epistemic passivity, or an extreme version of miserly information processing we would call cognitive offloading dependency, as learners adapt their behaviour to the path of least resistance rather than engaging in the reflective work that knowledge-building demands.

To determine if our system-level and human-level changes have had a significant effect, we have outlined a pathway to measuring epistemic agency. If epistemic agency is central to knowledge-building, it must be operationalized beyond correctness or content recall. The proposal to use rational-thinking measures, such as the Cognitive Reflection Test and related tasks, offers a potential empirical foundation. These measures capture the tendency to resist default responses and regulate belief in accordance with reasons rather than fluency. In LLM-supported environments, such measures may help distinguish epistemic support from epistemic substitution, enabling evaluation of whether AI systems are strengthening or eroding reflective control over time. We see epistemic agency as an inherently elusive construct that cannot be captured through any single indicator. Rather than treating epistemic agency as a static trait or a simple attitudinal disposition, we operationalize it as a form of reflective regulation enacted across belief, discourse, and inquiry practices. This perspective necessitates a measurement strategy that combines self-report, performance-based assessment, and behavioural evidence drawn from authentic collaborative activity. The Epistemic Agency Scale and the Epistemic Agency Reflection Test are therefore not proposed as standalone instruments, but as complementary components of a broader analytic framework that also includes discourse analysis and explainable-AI indicators of epistemic influence.

The Epistemic Agency Scale is designed to capture learners’ dispositional orientation toward reflective engagement; specifically, their motivation to seek justification, resist premature closure, and treat ideas as improvable rather than authoritative. By grounding item construction in Stanovich’s Master Rationality Motive, the scale targets reflective-level commitments rather than cognitive ability or content knowledge. Importantly, items are situated in collaborative and AI-mediated contexts, allowing the scale to be sensitive to the epistemic pressures introduced by fluent, authoritative-sounding AI outputs. This contextualization distinguishes the scale from more general measures of critical thinking and aligns it with the epistemic challenges that motivate our research. Further, the Epistemic Agency Reflection Test extends this measurement strategy by assessing epistemic agency in action. Modelled on the logic of the Cognitive Reflection Test, the EART presents participants with epistemically tempting scenarios in which fluent explanations, authoritative sources, or rapid consensus invite uncritical acceptance. Performance on these items depends on the participant’s ability to override such lures and select responses that sustain inquiry through justification, comparison, or critique. Unlike self-report measures, the EART captures participants’ propensity to engage reflective control under conditions that closely resemble those created by LLM-supported inquiry. In doing so, it provides a behavioural proxy for epistemic agency that is independent of participants’ declared beliefs about their reasoning.

Crucially, both instruments are embedded within a larger triangulation strategy. Scale scores and reflection-test performance are interpreted alongside discourse-based indicators of epistemic agency, such as question generation, responses to critique, resistance to premature convergence, and explainable AI analyses that reveal how LLM outputs shape inquiry trajectories. This triangulation is essential because, in our view, epistemic agency does not reside solely within individuals or momentary actions, but emerges from interactions among learners, tools, norms, and tasks, and varies with the epistemic demands of the context. Discrepancies among measures, for example, high self-reported agency paired with discourse patterns indicative of AI mimicry, are treated as theoretically informative rather than as measurement error, pointing to conditions under which agency may be claimed but not enacted. In sum, our approach positions epistemic agency not as a unitary, easily measurable variable; rather, it is approached as a pattern of alignment across reflective dispositions and observable inquiry practices. While this increases methodological complexity, it also provides a more faithful representation of the construct and a stronger foundation for understanding how to maximize the value of an LLM as a knowledge-building partner.

Several limitations and open questions remain. First, the measures of epistemic agency proposed in this paper, the Epistemic Agency Scale and the Epistemic Agency Reflection Test, are intentionally introduced as instruments under development. Although they are theoretically grounded in Stanovich’s account of reflective rationality and triangulated with discourse-based and explainable-AI analyses, they require further psychometric validation before being used for evaluative or comparative purposes. Establishing reliability, construct validity, and sensitivity to developmental and contextual variation is an essential next step. Second, while the present analysis emphasizes epistemic agency as a primarily reflective function, the precise developmental trajectories through which reflective control is internalized, from socially distributed regulation within groups to individual competence, remain incompletely understood. How these trajectories are altered by sustained interaction with fluent AI systems is an open empirical question. Finally, although we argue that LLMs lack reflective control, we do not claim that all forms of meta-level regulation are impossible for artificial systems. The boundary between computational self-monitoring and genuine epistemic regulation remains a philosophically and empirically unresolved issue, warranting continued investigation.

Future research should extend this work in several directions. Empirically, longitudinal studies are needed to examine how epistemic agency develops over time in AI-mediated knowledge-building environments, particularly whether repeated exposure to inquiry-oriented LLMs strengthens or weakens reflective dispositions. Methodologically, further ML investigation of discourse analysis, rational-thinking measures, and explainable-AI techniques could yield more fine-grained models of how epistemic agency is enacted or displaced in practice. From a design perspective, future work should explore how prompt-level, system-level, and pedagogical interventions interact, identifying which combinations most effectively sustain inquiry without increasing cognitive burden or reliance. Finally, extending this framework beyond professional, civic, and lifelong learning settings to educational contexts would help clarify how epistemic agency develops and operates across the lifespan and under varying epistemic pressures.

4. Conclusions

4.1. Recommendations and Future Trajectories

The analysis from this perspective offers several clear recommendations for designing and studying AI-supported inquiry. Firstly, large language models intended for educational or professional use should be explicitly positioned as knowledge-building partners rather than sources of definitive truth. System design must maintain reflective control on the human side, ensuring that learners and inquiry communities retain authority over problem framing, evidence standards, and knowledge commitments. Relying solely on prompt-level strategies is inadequate; sustained support for epistemic agency requires system-wide memory, interaction constraints, and model training aligned with discourse patterns that promote inquiry.

Second, learning environments that incorporate LLMs should intentionally foster reflective norms. Learners need support in building habits of justification, comparison, and detecting anomalies, especially when they encounter fluent, authoritative-looking outputs. Integrating AI without developing corresponding epistemic norms risks reinforcing epistemic passivity rather than enhancing reflective agency.

Third, empirical research should progress beyond accuracy-focused evaluation of AI systems towards measures that reflect the distribution of epistemic roles and reflective regulation. Long-term studies are necessary to assess whether ongoing interaction with inquiry-oriented LLMs enhances or diminishes epistemic agency over time. The triangulated measurement framework introduced here provides an initial foundation for this work, although further refinement and validation are crucial.

Finally, the broader path of AI development should shift from aiming for artificial epistemic independence to creating socio-technical systems that enhance human inquiry. The goal is not to develop artificial knowers, but to build learning environments where increasingly capable systems broaden the scope of human reasoning while maintaining the normative foundations of knowledge. In this way, the future of AI in education and professional inquiry relies less on increasing model size and more on clarifying epistemic roles.

4.2. Summary

We have argued that the main challenge posed by large language models in knowledge-building contexts is not primarily related to accuracy, scale, or computational complexity, but to epistemic role assignment. From the converging perspectives of Peirce, Sellars, Stanovich, Bereiter, and Scardamalia, fluent language production does not constitute knowledge, nor does algorithmic power suffice for epistemic agency. Knowledge only emerges through norm-governed inquiry: the formulation of genuine questions, encountering resistance, regulating belief by reasons, and making provisional commitments of claims within a community of inquiry. When LLMs are regarded as epistemic authorities, these conditions are systematically undermined; however, when they are designed and utilized as knowledge-building partners, their strengths can be leveraged without displacing the human capacities that enable knowing.

By aligning Stanovich’s tripartite model with the principles of knowledge-building, the paper explains both why LLMs seem so powerful and why their limitations continue. LLMs excel at autonomous and algorithmic tasks, pattern detection, hypothesis generation, sustained simulation, and representational maintenance, but lack reflective control: the ability to decide when inquiry should begin, which epistemic norms apply, and when beliefs should be endorsed or withheld. These reflective functions are not just implementation details but essential aspects of epistemic agency. Therefore, they cannot be simply “added” to current systems without fundamentally changing the nature of human–machine interaction. The proper response is not to develop artificial knowers, but to create socio-technical systems that support and extend human knowledge by assigning cognitive responsibilities in principled ways that promote lifelong learning.

We have also developed a framework for measuring epistemic agency that captures its inherently reflective and context-sensitive nature. By integrating a theoretically grounded self-report scale, a performance-based reflection test modeled on the Cognitive Reflection Test, discourse-based indicators, and explainable AI analyses, the proposed approach views epistemic agency not as a single trait but as a pattern of alignment across dispositions, decisions, and practices. Importantly, these measures are introduced as research tools still in development, designed to support theory-building and empirical investigation rather than final evaluation. Their value lies in making epistemic agency visible in environments where fluent AI outputs might otherwise obscure the difference between warranted belief and plausible response.

Finally, the design implications outlined in this paper suggest a rethinking of how AI is integrated into learning and inquiry. Successful environments supported by LLMs must regulate AI behaviour at the prompt, system, and model levels while also fostering human practices that maintain reflective control. Knowledge-building offers a particularly suitable pedagogical framework for this purpose, as it externalizes epistemic regulation, shares responsibility across a community, and treats ideas as improvable entities rather than static answers. The challenge ahead is not to make machines reflective in isolation but to create learning environments where human epistemic agency is preserved, practiced, and enhanced—especially in the context of increasingly fluent artificial interlocutors.