Exploring the Interplay of Theory of Mind, Logic Validation, and Advanced Retrieval in LLM-Based Multi-Agent Systems

Abstract

Large Language Models have rapidly advanced, enabling composite multi-agent systems capable of sophisticated text-based collaboration. In domains that demand coordinated reasoning, teams of agents can combine complementary strengths to produce better results than a single model. In practice, multi-agent decision making faces recurring shortcomings: agents frequently operate with poorly grounded or outdated knowledge, they seldom represent peers’ beliefs and intentions in a disciplined manner, and their conclusions are rarely checked against explicit logical constraints. These gaps reduce coherence, hinder dependable cooperation, and make the reasoning chain difficult to explore, often manifesting their black-box nature. How can a practical MAS architecture be designed to ground agents with relevant information, support explicit modelling of collaborators’ beliefs, and validate outputs against formal rules? How can the individual and synergized contributions of these mechanisms be measured across multiple case studies? To explore these questions, we present a purpose-built experimental platform. The system integrates a retrieval-augmented generation pipeline to ground agents in relevant materials, a Theory of Mind module that makes agent beliefs explicit, and an LLM-based logic auditor that evaluates outputs against an explicit rule set. Applied to diverse text-based decision-support and coordination case studies, the platform and protocol provide a reproducible blueprint for investigating how grounding, social reasoning, and logic validation together improve multi-agent collaboration. In doing so, this study contributes to the field of artificial intelligence by offering a structured approach to building, testing, and refining multi-agent architectures that balance knowledge grounding, perspective modelling, and reasoning validation.

1. Introduction

Recently, Multi-Agent Systems (MAS) that bring together agents based on Large Language Models (LLMs) have gained considerable momentum [1]. The core idea involves a team of specialized agents collaborating to solve complex problems. This approach is increasingly seen in fields such as strategic planning, knowledge-intensive simulations, and advanced reasoning tasks that require coordinated expertise [2]. Studies show that well-designed agent groups can achieve a synergy where the collective output surpasses individual capabilities, as illustrated in Figure 1, allowing specialists to combine their unique strengths for stronger outcomes [3]. A key advantage of these systems is that agents communicate using natural language, which allows for the interpretation of their communication strategies and coordination principles, elements that remain opaque in monolithic models [4]. The explicit linguistic exchanges between agents make the steps of the reasoning process transparent and amenable to human oversight.

Figure 1. An architectural comparison between a single LLM and a multi-agent system for complex problem-solving. The multi-agent approach decomposes the goal, utilizing planning, memory, and tools to produce a more robust final output than a single agent.

Despite these strengths and their increasing adoption, new taxonomies of multi-agent failure modes show that fundamental issues—such as agent misalignment, specification errors, and coordination breakdowns—continue to limit the dependability of these systems [5]. A primary challenge is preserving logical consistency. In multi-turn collaborations, agents are prone to “problem drift,” where the discussion strays from its initial goals and constraints, often because progress stalls or feedback is ambiguous [6]. Such drifting compromises the structural integrity of the reasoning process, leading to incoherent or invalid conclusions, a critical failure in tasks requiring strict adherence to a logical framework. Another distinct challenge is ensuring that teamwork rests on a foundation of solid facts. While Retrieval-Augmented Generation (RAG) is an effective tool for grounding, research indicates that retrieval by itself is not a complete solution [7]. Without independent verification, RAG can still propagate subtle biases or even spark a “cascade of misinformation,” where agents build upon a flawed premise without questioning it. The result is an elaborate answer that sounds plausible but lacks any factual basis.

Finally, effective teamwork requires a disciplined awareness of what other collaborators in the group perceive. Although research indicates large language models can demonstrate strong theory-of-mind skills in team-based work, this capability is often inconsistent. It frequently falters in long-term interactions where agents lose track of each other’s states due to a tenuous understanding of their partners [8]. When robust modeling of fellow agents breaks down, efforts become misaligned, work is duplicated, and the group fails to build constructively on prior contributions. This prevents the team from achieving the synergistic benefits it aims for. These collective challenges raise fundamental questions about how to design and evaluate these complex systems. How can a practical MAS architecture be designed to ground agents with relevant information, support explicit modelling of collaborators’ beliefs, and validate outputs against formal rules? How can the individual and synergized contributions of these mechanisms be measured across multiple case studies?

This work addresses these questions with a purpose-built experimental platform that treats reliability as a property of an auditable workflow rather than of any single model response. The platform separates collaborative generation from post-hoc analysis: specialist agents generate grounded proposals with explicit peer-perspective modelling, while independent auditors record evidence about inter-agent coherence, logical validity, and retrieval use. This generation/audit separation creates a transparent workflow in which the internal reasoning process can be measured without turning the auditors into a hidden corrective feedback loop.

The main contributions of this study are as follows:

• A modular system architecture for multi-agent systems that decouples knowledge generation from analysis, providing a structured framework to make collaborative output more reliable and transparent;
• A transparent evaluation platform that provides a multi-faceted audit of an agent team’s internal reasoning processes, including its adherence to logical constraints, its inter-agent awareness, and its effective use of retrieved knowledge;
• An application of the proposed platform to illustrative case studies, demonstrating its capacity to surface the complex interplay between nuanced collaborative dynamics and logical constraints, validating its effectiveness as a diagnostic tool for improving system reliability.

2. Preliminaries

In order to lay a distinct foundation for the areas presented in this article, the preliminary paradigms that contemporary shared AI systems are based upon would first need to come under consideration. The current section offers a brief overview of four pivotal concepts, i.e., the architecture of LLM-based Multi-Agent Systems, the transition of the knowledge retrieval mechanism from RAG to GraphRAG, the utilization of Computational Theory of Mind for the coherence of agents, and the implementation of logical constraints for reliable reasoning.

2.1. LLM-Based Multi-Agent Systems

A Multi-Agent System (MAS) is an aggregate of several independent agents that collaborate in an environment to reach a common goal or private goals [9]. The basic principle of MAS is problem decomposition. Intractable tasks for a monolithic agent can be divided into smaller, tractable sub-tasks performed by special-purpose agents. This technique is most successful in complex domains where heterogeneous expertise needs to be coordinated, for example, in financial portfolio optimization [10]. Traditionally, these agents were typically symbolic, rule-based agents whose cooperation was regulated by formal protocols, like the Contract Net Protocol, where a “manager” agent announces a task and “contractor” agents bid to perform it [11].

The Large Language Models (LLMs) have introduced a paradigm shift, giving birth to LLM-based MAS. An LLM can serve as the heart of all agents in such systems, making them communicate, reason, and act in natural language. It offers them unprecedented flexibility to perform subtle, knowledge-heavy tasks. Organizing communication is a crucial architectural choice in such systems, and widespread architectures include decentralized peer-to-peer networks, centralized hub-and-spoke architectures, and layered architectures. In a layered communication architecture, the agents are structured in a hierarchical way, having defined functions at various levels, so that the systematic flow of information and control can take place that suits complex, multistage problem-solving [12,13].

2.2. Knowledge Grounding with RAG and GraphRAG

One of the main drawbacks of LLMs is that their knowledge is restricted to the training data, which has a fixed cut-off date. This means they cannot access current information and can result in the creation of factually incorrect outputs, or the so-called “hallucinations,” when they are prompted about the latest developments or very specific subjects. A very widespread way to overcome that is the employment of Retrieval-Augmented Generation (RAG) [14]. The workflow is to inject the user’s prompt with related information before being given to the LLM. Generally speaking, an inquiry is given to a retriever module that searches a knowledge base in order to fetch related information, which is then given as context to a generator LLM in order to produce a grounded response.

Though influential, conventional RAG systems that retrieve from raw text struggle with queries that involve heavy, complex cross-information reasoning. This prompted the advancement of GraphRAG, an extension of the concept that uses a knowledge graph as its source of retrieval [15,16]. Graph data structures represent information as a collection of nodes (entities) and edges (relations) that form a network, bringing implicit connections into explicit form. The format supports multi-hop, relation-based queries that cannot be supported by simple text-based retrievals. For instance, identifying which individuals in an organization have expertise in the same technology that was applied to a successful undertaking involves going across individual relations. By directly querying the web of relations, GraphRAG can offer the agents highly contextual, synthesized information, going beyond mere fact acquisition to the exploration of structured knowledge via methods such as Text-to-Cypher translation, as can be seen in Figure 2.

Figure 2. Comparison of knowledge extraction from unstructured text vs. a knowledge graph. Traditional RAG extracts disconnected facts from chunks of documents, whereas GraphRAG aggregates information by exploring explicit links among entities. Blue highlights denote the retrieved or selected evidence used to form the response, while grey elements denote non-selected background chunks or entities.

2.3. Computational Theory of Mind

In the field of cognitive science, Theory of Mind (ToM) can be defined as the capacity to assign mental states: beliefs, intents, goals, and knowledge, to oneself and other individuals, and to comprehend that these states can vary from those of the self [17]. This ability underlies successful human cooperation, allowing people to project the actions of teammates, coordinate their plans, and refrain from duplicative efforts. In the artificial intelligence domain, Computational Theory of Mind entails the development of systems that can simulate the internal states or frames of other agents in order to enhance interconnection and synchronization.

Recent studies show that LLMs exhibit emergent ToM-like properties, able to reason about the beliefs of other people in laboratory experiments [18]. However, relying on this emergent process by itself in a MAS can lead to inconsistent cooperation. A more robust alternative is to engineer a ToM mechanism as an explicit component in the system architecture. This can be achieved by designing agent interactions to require them to construct explicit, predictive models of their colleagues. An example is requiring an agent to first articulate its beliefs regarding a teammate’s probable perspective before generating its own response (see Figure 3). This engineered mechanism, explored in contemporary agent architectures that construct explicit models of the beliefs and intentions of their colleagues [19], transforms ToM from an unconscious, unpredictable byproduct to an active means of generating more coherence. This is critical for advanced tasks in which the agent needs to model and respond to advanced human reasoning, such as simulating formal Oxford-style debates [20].

Figure 3. A comparison of an agent’s response process with and without an engineered Theory of Mind (ToM) mechanism. An agent with ToM moves beyond a role-based answer to produce a more integrated analysis that anticipates and incorporates the perspectives of its peers.

2.4. Logical Constraints in MAS

One of the central tensions in contemporary AI is the trade-off between the opaque, sub-symbolic thinking of neural networks (“black box” models) and the transparent, verifiable quality of symbolic logic. Although LLMs are superior at working with linguistic subtlety and nuanced patterns, their decision-making mechanisms are for the most part opaque, rendering their outputs problematic in high-stakes settings where accuracy and auditability come first. The imposition of logical constraints provides an essential connecting principle between these frameworks, providing an ability to tie the output of a fluid LLM into a deterministic, understandable structure.

Efforts at integration place the models along a spectrum. Whereas highly integrated neuro-symbolic models attempt to embed the logic rules directly in the neural architecture itself [21], a complementary path favors modularity by separating the generation and analysis tasks. Here, generative models are exploited in their prime virtue of generating creative, context-heavy hypotheses. These products are then run through an independent validation step, where their adherence is tested against an explicit, frequently deterministic, rule set. This principle of post-hoc validation does not alter the intrinsic “black box” nature of the generative component; instead, it provides an external layer of accountability that bases the overall system’s output in an explicit, verifiable foundation. The main benefit of the resulting architectural separation is the ability to create an auditable workflow, whereby trustworthiness is increased without the loss of the generative component’s flexibility.

3. Related Work

Though the promise of LLM-based Multi-Agent Systems is great, their road to consistent deployment is fraught with recurring pitfalls. The literature shows an active research field grappling with the basics in architectural design, grounding of knowledge, cooperative coherence, and reasoning verification. The multi-agent collaboration process, as shown in Figure 4, can itself be conceptualized as a pipeline where weaknesses at every step can compromise the quality of the output produced. This section surveys the previous work by going over the literature in the vicinity of these major failure points, the form these challenges take, and the limitations in contemporary means of addressing them.

Figure 4. A conceptual model of the “leaky pipeline” in multi-agent collaboration. Leakages at the steps of grounding and generation can gradually lower the quality of the output, generating faulty and unpredictable end-results.

3.1. Collaborative AI Systems

One of the central challenges in designing MAS is designing an architecture that can accommodate structured control while being adaptively flexible. The Meta-Agent-led model is a very widespread method, providing a clear hierarchical structure whereby there is a central agent responsible for controlling workflow [22]. Even though the resulting topology has the advantage of being deterministic, there can also come performance bottlenecks and the model can have difficulty in adapting to complex, dynamic problems [23]. The inflexibility has prompted the exploration of the use of more decentralized and parallelized architectures. The divide-and-conquer technique, for example, boosts scalability by allowing agent groups to process in parallel over tasks, yet presents novel challenges in ensuring the eventual, synthesized output is coherent and overall consistent with initial intent [24].

Another architectural ideology centers around increasing raw ability by synthesizing the output of several LLMs. The Mixture-of-Agents (MoA) technique reaches state-of-the-art performance by producing dense, multilevel systems in which the agents expand upon the efforts of all agents in the previous level [25]. Yet, because of the high computational cost and the “all-to-all” communication structure of the MoA, its process of reasoning becomes cumbersome to audit. To overcome this, the Sparse Mixture-of-Agents (SMoA) architecture keeps the computational burden in check by sparsifying the flow of information [26]. Although these measures effectively tackle performance and efficiency, they leave a very essential gap in regards to the formality of verifiability and transparency of the joint process.

3.2. Challenges in Knowledge Grounding

The grounding of agent decisions in external information is critical to factual accuracy, but the retrieval process itself is a major source of potential error. As established in Section 2.2, RAG links LLMs to external sources of information, but it is far from a complete solution. The retrieval of noisy or irrelevant information can produce as output something that appears to be factually buttressed but based upon an incorrect premise. The result is that there needs to be mechanisms that can also verify, not simply the relevance of retrieved information, but the proper application of the information by the agent.

While GraphRAG enables more advanced relational inquiries, it is susceptible to the natural imperfections of the real-world bases of knowledge. Experimental analyses demonstrate that the KG-RAG systems’ performance can suffer appreciably in the presence of incomplete or inaccurate information [27]. This has led researchers to investigate ways to enhance reliability, like filtering and integration units constructed to offset the influence of noisy information [28]. The extension of agentic RAG, in turn, which provides the agents with robust instruments for multi-hop reasoning, complicates the matter all the more [29]. Although the agents become more competent, their added autonomy leaves in its wake an “accountability vacuum,” leaving the matter of how to audit their sophisticated information collection procedures in order to validate the conclusions reached unresolved.

3.3. Achieving Coherence with ToM

Effective collaboration requires coherence, which depends on agents maintaining an accurate model of their peers’ perspectives. While the emergent ToM capacity described in Section 2.3 has been demonstrated in controlled tests [18,30,31], this ability has proven brittle in dynamic, multi-turn collaborations. Critical studies suggest that LLMs often rely on superficial heuristics rather than genuine social reasoning, a phenomenon sometimes referred to as the “Clever Hans” effect, leading to coordination failures and redundant work [32,33].

The challenge is exacerbated by a lack of appropriate evaluation procedures. A rising opinion is that standard ToM tests are insufficient because they typically only measure static prediction of beliefs, not the dynamic, adaptive synchronization that teamwork is [34]. Advanced computational models, such as hybrid LLM-Bayesian systems, are an intriguing area of study for more nuanced mental state prediction [35]. Still, these are yet underdeveloped engineering responses. The nature of the challenge hasn’t changed: how to transform ToM from an incoherent, emergent, uncontrollable phenomenon into an engineered, measurable aspect of a robust cooperative system.

3.4. Verifiable and Logic-Enhanced Systems

The last and arguably most crucial weakness in the MAS pipeline is the absence of systematic analysis. As discussed in Section 2.4, the neuro-symbolic spectrum ranges from tightly integrated models [36] to loosely coupled analysis systems. The integrated approach often sacrifices the LLM’s generative flexibility and produces an equally opaque system.

The alternative, decoupled analysis, has gained popularity because of modularity. Here, an LLM produces an output, which is assessed by an external process. The soundness of the process relies solely upon the evaluator. There is convincing evidence that LLMs are not good at corrigibility [37] and, more importantly, cannot reliably check their own satisfaction of logical constraints, a deficiency shown specifically in the widely available, purpose-built benchmarks [38]. External analysis is therefore essential. Building such evaluators is an important engineering task, with the presently used schemes being the conversion of LLM outputs to some symbolic language such as Prolog [39] for formal verification or the use of ontologies in specialized domains such as law [40]. Others involve prover-verifier training schemes [41] or constraint imposition by direct action at the decoding step [42]. The main challenge remains finding an architecture that can impose the logic constraints without crippling the generative power of the agents.

3.5. Positioning of the Present Work

The systems surveyed above motivate, but do not duplicate, the architecture proposed in this paper. AutoGen-style frameworks provide flexible orchestration for conversational agents [22]; MoA and SMoA improve output quality or efficiency by aggregating model responses [25,26]; and generalist orchestrators such as MagenticOne focus on task completion with tool-using agents [43]. Our contribution is narrower but more diagnostic: we specify a bi-layered epistemic architecture in which a Knowledge Generation Layer produces proposals, while a passive Knowledge Analysis Layer audits those proposals along three explicit interfaces: predicted peer beliefs (${\mathcal{V}}_{\mathrm{ToM}}$), rule-set compliance (${\mathcal{V}}_{\mathrm{Logic}}$), and retrieval-use effectiveness (${\mathcal{V}}_{\mathrm{RAG}}$). The novelty therefore lies not in inventing GraphRAG, ToM prompting, or LLM-as-a-Judge in isolation, but in making these components auditable, separable, and measurable within one reproducible MAS workflow.

4. Problem Formulation

These deficiencies that are disclosed through the collaborative pipeline must not be seen as separate flaws, but rather as indicative of an underlying design principle: the coupling of knowledge generation and analysis. Many contemporary systems give agents the twin tasks of both producing novel content and judging its validity, a configuration that naturally induces an important conflict. The generative process tasked with production is often not sufficient to objectively examine its own nuanced logical flaws or factual errors.

This vulnerability is rooted in the nature of agent self-correction. Although LLMs can revise their own outputs, this is frequently unreliable without externally structured validation. Indeed, without independent evaluation, self-level refinements can even degrade an otherwise accurate response [44,45]. Here, we maintain a distinction between evaluation and correction. Although an independent judging LLM can efficiently find fault with an output [46], the generating agent often lacks the objective distance to correct its own fault. Leaving ultimate evaluation to the generative agent itself is therefore a system risk.

In a multi-agent system, this weakness is amplified. In the absence of an independent evaluation phase, a wrong calculation, incorrect interpretation, or subtle hallucination can be taken up by the next agent in the chain, resulting in “hallucination snowballing” [47,48]. Error amplification can lead to task failure, where the output appears coherent but is constructed on propagated fallacies [49].

Consequently, the collaboration model is both fallible and lacking diagnostic insight. When failures occur, the intertwined workflow makes root cause analysis difficult. Attribution studies show that even recent models struggle to determine which agent, or which exact step, caused a particular failure [50]. This creates a methodological gap, since contemporary evaluation metrics focused on overall task success tend to treat the internal collaborative process as a “black box” [51]. The underlying challenge is therefore to avoid propagating errors while revealing their origins. Architecturally, we ask: How can a practical MAS architecture be designed to ground agents with relevant information, support explicit modelling of collaborators’ beliefs, and validate outputs against formal rules? Methodologically, we ask: How can the individual and synergized contributions of these mechanisms be measured across multiple case studies? Addressing this diagnostic gap motivates the bi-layered model that follows.

5. Platform for Auditable Multi-Agent Collaboration

The design and workflow of the experimental platform are explained in the following section. The platform is designed to enable a systematic and modular exploration of the collaboration of multiple agents, where explicit consideration is given to the evaluation of the quality and usefulness of every component in the reasoning process.

5.1. Design Philosophy

The platform is divided into two functionally distinct layers: a Knowledge Generation Layer and a Knowledge Analysis Layer. This architectural separation, as shown in Figure 5, is a conscious design decision made to enable an unobstructed and objective evaluation of the joint process.

Figure 5. The system’s operational workflow, detailing the cycle from Meta-Agent goal setting to parallel knowledge generation by specialist agents. The process includes an independent analysis audit that records diagnostic metrics in parallel. The Meta-Agent may iterate the generation cycle by setting new goals based on its own assessment of the proposals, but the analysis layer itself does not feed corrections back to the generative agents.

By decoupling the action of production from the action of judgement, the platform offers a structure for systematically gauging and debugging the products of the generative agents against a configurable set of explicit standards. The structure supports experimental flexibility: the measurement criteria can be adjusted, tuned, or substituted without changing the generative workflow, while allowing analysis of particular collaborative phenomena in a targeted way.

The Knowledge Generation Layer holds the elements that carry out the main tasks of problem-solving. This includes a group of specialist agents, all with defined tasks, and a Meta-Agent that oversees the workflow. The main role of the layer is to carry out collaborative reasoning: proposing solutions, seeking information, and modeling the other agents’ points of view. The Knowledge Analysis Layer, however, is made up of a set of non-generative auditors. The only role it has is to take the output generated by the Knowledge Generation Layer and to evaluate it against given criteria. Importantly, the layer runs autonomously and does not give direct feedback to the generative agents. This intentional separation means that the analysis acts as a pure diagnostic tool, giving an objective measure of the agents’ inherent ability, instead of providing a corrective feedback loop that can blur underlying flaws of reasoning.

5.2. Conceptual Formalism

To add formal rigor to this bi-layered philosophy, we characterize the dynamics of the system in terms of a set of base functions acting upon a common state. The collaboration’s state at any given turn t is given by the conversation history, $H_t$, an ordered collection of all previous proposals. A proposal, $\pi$, is the basic unit of work by a specialist agent, $A_i$, out of the set of all agents $\mathcal{A}$. The proposal is a structured tuple $\pi =(a,g,p,c)$ consisting of the authoring agent a, the particular goal g being tackled, a predictive Theory of Mind model p of what its colleagues’ contributions are going to be, and the essential content c of its reply.

The production of new knowledge is guided by two main functions. The activity of the Meta-Agent is summarized by an orchestration function, $\mathsf{\Phi }$, that transforms the current history to the subsequent goal for the group: $\mathsf{\Phi }\left(H_t\right)\to g_{t+1}$. The activity of an individual specialist agent is guided by a generation function, G, that generates a novel proposal by acting upon the goal, the history, and information retrieved from an external knowledge base, $\mathcal{K}$: $G(A,g,H,\mathcal{K})\to \pi$. By way of contrast, the diagnostic work of the analysis layer is summarized by a set of independent analysis functions, $\mathcal{V}$. These functions serve as measurement instruments, assessing each proposal without altering it. They include ${\mathcal{V}}_{\mathrm{ToM}}$ for measuring group-based coherence, ${\mathcal{V}}_{\mathrm{Logic}}$ for checking logical validity against a formal rule set $\mathcal{R}$, and ${\mathcal{V}}_{\mathrm{RAG}}$ for auditing the effective use of the knowledge base $\mathcal{K}$. Concretely, $\mathcal{R}$ is a finite, case-specific list of natural-language constraints loaded at platform initialisation: in our experiments, $|{\mathcal{R}}_{\mathrm{legal}}|=5$ and $|{\mathcal{R}}_{\mathrm{business}}|=6$ (the full rule listings appear in Appendix B). $\mathcal{R}$ is mutable at run time through the Orchestrator’s set_constraint tool, allowing additional constraints to be appended as a discussion progresses without redefining the auditor.

The platform’s operation is defined by an orderly, cyclical flow of information among the agents in a predictable sequence. The process ensures that all the steps in the joint task are discrete and analyzable. The communication is guided by the Meta-Agent, which leads the workflow in a sequence of distinguishable phases.

The workflow starts at the Meta-Agent, which runs the orchestration function ($\mathsf{\Phi }$) by operating over the current history $H_t$ in order to create the next goal, $g_{t+1}$. Following this goal-setting, the nominated specialist agents of the Knowledge Generation Layer are activated. The process of each agent is an execution of the generation function G that operates over the goal $g_{t+1}$ and the history $H_t$ in order to create a structured proposal (${\pi }_i$). The last step of the cycle includes the Meta-Agent, which examines the collection of proposals to check whether the objective of the team has been achieved and to decide on the next state.

In parallel, and without impacting this workflow decision, the proposals are forwarded to the Knowledge Analysis Layer. Here, the analysis functions ($\mathcal{V}$) are performed to audit the output and store the resulting metrics for post-hoc analysis. This ensures the audit data is generated for external scrutiny and system diagnostics, not as part of a real-time feedback loop to the generation agents.

The platform is implemented as a state machine, in which an overall AgentState object is propagated across the nodes, and it holds the entire interaction history ($H_t$). The main components responsible for running the workflow are the agents, divided into specialists for generation and auditors for analysis.

The Knowledge Generation Layer specialists are equipped with tools to strengthen their rationale. These include functions for querying the GraphRAG knowledge subsystem (the practical implementation of $\mathcal{K}$) to ground their contributions. Here, the computational Theory of Mind mechanism is also realized, delivering the predictive element p of each proposal.

The Knowledge Analysis Layer, by contrast, serves as the system’s set of independent auditors. It is comprised of several specialist auditors, where each auditor is an implementation of one of the analysis functions ($\mathcal{V}$). One auditor assesses logical consistency against the rulebook ($\mathcal{R}$) (implementing ${\mathcal{V}}_{\mathrm{Logic}}$); another measures inter-agent coherence (implementing ${\mathcal{V}}_{\mathrm{ToM}}$); and a third audits knowledge retrieval (implementing ${\mathcal{V}}_{\mathrm{RAG}}$). The details of these auditors’ mechanics are provided in Section 5.4. The workflow advances sequentially through a cycle of goal-setting, generation, and ratification, while the analysis functions run in parallel, logging metrics without influencing the path of the workflow.

5.3. The Knowledge Generation Layer

The Knowledge Generation Layer is the active problem-solving component that carries out collaborative reasoning, retrieves knowledge, and generates candidate solutions. Its role within the complete, cyclical workflow of the platform is illustrated in Figure 6. The layer has three interacting components, a Meta-Agent in charge of the workflow, a group of specialist agents that carry out the domain-specific analysis, and a GraphRAG subsystem that offers an interface to the structured information. The following subsections describe each component in detail.

Figure 6. The cyclical workflow of the bi-layered platform illustrates the four key stages: Goal Definition, Collaborative Generation, Independent Analysis, and the production of an audited solution. The process highlights the distinct responsibilities of the Knowledge Generation and Knowledge Analysis Layers.

5.3.1. The Meta-Agent and State Progression

The Meta-Agent acts as the orchestrator of the system and is the real-world realization of the function of the orchestration, $\mathsf{\Phi }$. It oversees the state advancement of the aggregate collaboration as a discrete, two-stage process in every collaboration cycle. Its primary function is to keep the team on track and ensure consistent progress toward the overall goal.

First, in its goal-setting capacity, the Meta-Agent applies a function ${\mathcal{M}}_{\mathrm{goal}}$ to the current conversation history $H_t$ to produce the immediate objective for the turn, $g_t$. This action is a concrete execution of the formal function $\mathsf{\Phi }\left(H_t\right)\to g_{t+1}$, translating the high-level strategy into a specific, actionable task for the specialist team. This goal is then dispatched to the relevant agents. Second, after the specialists have generated their set of proposals, ${\Pi }_t$, the Meta-Agent performs its ratification function. This step involves assessing the submitted proposals to determine if they collectively satisfy the objective $g_t$. Based on this assessment, the Meta-Agent decides on the next state of the workflow, either by concluding the current phase and setting a new overarching goal or by determining that further refinement is needed.

5.3.2. Specialist Agents and ToM Mechanism

The specialist agents are the primary knowledge workers within the platform. Their collective work is the execution of the generation function, G. The specific roles and expertise of these agents (e.g., legal strategist, finance analyst) are defined by detailed system prompts, allowing for flexible team composition. During each cycle, the nominated specialist agents receive the goal $g_t$ and the history $H_t$ and are executed in parallel. Their purpose is to collectively take these inputs and produce a set of proposals, ${\Pi }_t$, where each proposal ${\pi }_i\in {\Pi }_t$ is a structured response.

The production of each proposal strictly conforms to the formal tuple ${\pi }_i=(a,g,p,c)$. An important technical aspect of the process is the engineered Computational Theory of Mind (ToM) mechanism, specially designed to produce the predictive element, p. The agent’s prompt requires it first to produce its beliefs regarding its colleagues’ probable contributions and viewpoints for the current turn. Only then does the agent move on to construct its own main analysis and contribution (c). This architecture imposes a proactive cooperative attitude and causes the production of a tangible form of the model of each agent’s team, independently auditable by the Knowledge Analysis Layer. This technique of explicit prompting for the articulation of beliefs has the support of studies, which demonstrate that such procedures can enhance ToM performance in LLMs [52].

5.3.3. The GraphRAG Knowledge Subsystem

In order to satisfy the requirement that agent reasoning is based upon actual evidence, specialist agents are given controlled access to the GraphRAG subsystem. The subsystem constitutes the practical realization of the external knowledge base, $\mathcal{K}$, and forms an important input to the generation function G. It is constructed based on pre-indexed knowledge graphs that are produced by an independent, offline pipeline performing entity extraction, relation inference, and community identification over a set of source documents. These full graphs are loaded at runtime by the runtime system, where they are made queryable without paying the cost of constructing the graphs in the collaboration session. Agents also have two very distinct query tools, to enable a multi-stage research process. The first provides the broad, high-level search facility, appropriate as first-stage discovery. An agent can ask a general question and receive synthesized summaries of the most relevant communities of entities in the graph. The second enables a focused local search over the indexed graph context. Once an agent identifies major entities or themes, it can issue more targeted local queries to retrieve specific evidence. Each agent query issued and the corresponding delivered data by the subsystem are meticulously logged. The log provides the literal evidentiary basis for the independent audit performed by the ${\mathcal{V}}_{\mathrm{RAG}}$ auditor.

Because graph construction is performed offline, the collaboration loop pays only retrieval-query costs at run time; the measured preprocessing footprint for the curated corpora used in this study is reported with the evaluation resource profile in Section 6.

5.4. The Knowledge Analysis Layer

The Knowledge Analysis Layer operates as the system’s suite of independent auditors, whose individual mechanics are illustrated in Figure 7. It is an independent and passive audit pipeline, functioning in parallel to the main collaborative workflow of the Knowledge Generation Layer. Its purpose is not to influence the live reasoning process but to provide a comprehensive, multi-faceted record of it by implementing the set of analysis functions ($\mathcal{V}$) defined in our formal model. This layer operationalizes the concept of “LLM-as-a-Judge”, a rapidly growing paradigm focused on leveraging LLMs for complex evaluation tasks [46,53,54]. It takes the proposals ($\pi$) generated by the generation layer as input and produces a suite of quantitative and qualitative metrics. These metrics are logged for post-hoc analysis, enabling a detailed and objective evaluation of the system’s performance.

Figure 7. The three independent audit processes of the Knowledge Analysis Layer detail the mechanics for evaluating Theory of Mind coherence, RAG effectiveness, and logical consistency. Each evaluation is performed by a dedicated agent that produces a structured verdict.

The three auditors operate at different units of analysis and fire under different conditions, summarised in Table 1. ${\mathcal{V}}_{\mathrm{Logic}}$ is invoked on every specialist proposal in every collaboration round; ${\mathcal{V}}_{\mathrm{ToM}}$ is invoked on each (evaluator, peer) pair only when at least two specialists are nominated in the same turn, since a single-specialist turn provides no peer to predict; and ${\mathcal{V}}_{\mathrm{RAG}}$ is invoked on each individual GraphRAG retrieval call during a round, so its coverage scales with how often agents choose to query the knowledge base. The Meta-Agent’s goal-setting messages, the Orchestrator’s set_constraint actions, and any internal tool call other than GraphRAG are deliberately outside the scope of $\mathcal{V}$.

Table 1. Audit coverage of the three analysis functions per collaboration round.

Auditor	Unit of Analysis	Firing Condition
${\mathcal{V}}_{\mathrm{Logic}}$	Specialist proposal	Every nominated specialist’s proposal
${\mathcal{V}}_{\mathrm{ToM}}$	(Evaluator, peer) pair	≥2 specialists nominated in the same turn
${\mathcal{V}}_{\mathrm{RAG}}$	GraphRAG retrieval call	Each logged retrieval call (zero if none)

5.4.1. Assessing Collaborative Coherence

This auditor utilizes the predictive coherence function, ${\mathcal{V}}_{\mathrm{ToM}}$, to quantify the semantic alignment between an agent’s prediction about what it expects from peers and what those peers end up contributing. To achieve this, it employs a dual-metric approach.

The approach entails comparing the predictive component $p_i$ of an agent i’s proposal, ${\pi }_i$, with the actual central contribution $c_j$ from another peer j’s proposal, ${\pi }_j$. This is done in two ways: first, a dedicated evaluation LLM provides a qualitative consistency judgment, yielding a boolean verdict on substantive alignment. Second, a quantitative score is produced. Using a dedicated, high-dimensional embedding model (text-embedding-3-large), the auditor develops vector representations for both aspects of text. Then, the auditor determines the cosine similarity between the prediction vector and the contribution vector, producing a normalized score that serves the function: ${\mathcal{V}}_{\mathrm{ToM}}(p_i,c_j)\to [0,1].$

The quantitative metric was chosen because it provides an evaluation of semantic alignment within the vector space, consequently registering the conceptual closeness of the two texts independent of corresponding keywords. It is, however, a proxy measure rather than direct evidence of a human-like Theory of Mind. High cosine similarity can reflect genuine predictive modelling, but it can also be produced by paraphrasing, role-play, or other Clever Hans-style shortcuts in which the model exploits surface cues rather than maintaining an explicit belief model [31,34]. Conversely, low similarity may be desirable when the task benefits from adversarial critique or from an agent introducing a previously missing risk. The score is therefore interpreted jointly with the qualitative LLM verdict and the task context, not as a standalone success/failure label.

The ultimate score provides an objective, though contextual, analysis of the alignment between an agent’s prediction and the subsequent action of its counterpart, designed to capture an elaborate metric concerning the collaborative efficacy of the system. The auditor sends a quantitative signal, complemented by a qualitative justification, for later interpretation rather than an absolute success/failure label.

5.4.2. Ensuring Logical and Factual Consistency

A secondary auditor is tasked with evaluating logical consistency, symbolically represented as ${\mathcal{V}}_{\mathrm{Logic}}$, by assessing each agent’s contribution against a fixed set of governing constraints. The auditor is implemented as an LLM-based reasoner that judges each specialist proposal against the active rule set $\mathcal{R}$ and returns a structured VALID/INVALID verdict together with a brief justification. The rule set spans both quantitative constraints (e.g., numerical consistency such as budget arithmetic) and qualitative constraints (e.g., the IRAC structure of a legal memorandum, the presence of required precedents, and strategic considerations), all evaluated by the same reasoner against the natural-language text of the proposal.

To guard against the known pitfalls of the “LLM-as-a-Judge” paradigm, the reasoner employs a state-of-the-art model (GPT-5) distinct from the generative agents, is constrained to evaluate against the explicit rule set $\mathcal{R}$, and is prompted to produce a brief justification for each verdict. During development, the accuracy of this auditor was benchmarked against a hand-curated set of known valid and invalid statements to ensure reliability.

The ultimate decision of ${\mathcal{V}}_{\mathrm{Logic}}(c,\mathcal{R})$ is therefore an LLM-based audit verdict: a proposal is flagged INVALID when the reasoner identifies a clear contradiction, factual error, or phase-appropriate violation of any rule in $\mathcal{R}$. This design preserves the flexibility necessary to evaluate nuanced, natural-language reasoning across both quantitative and qualitative rules.

5.4.3. Auditing Knowledge Retrieval Effectiveness

A final auditor implements the RAG effectiveness function, ${\mathcal{V}}_{\mathrm{RAG}}$, evaluating the quality of research performed by the specialist agents. This measurement tool assesses whether the knowledge retrieval capabilities are used in a manner that is both relevant and effective.

This auditor takes as input the complete log of retrieval actions undertaken during the generation of a proposal ($\pi$), as described in Section 5.3.3. It assesses each query made to the knowledge base ($\mathcal{K}$) against two criteria: relevance to the agent’s goal and effective integration into the final proposal contribution c. The function ${\mathcal{V}}_{\mathrm{RAG}}(\pi ,\mathcal{K})$ thus produces a qualitative rating and justification for each retrieval action. These evaluations are recorded, providing a detailed performance log of the system’s research capabilities and the utility of the provided knowledge base.

6. Evaluation and Results

Having laid out the architecture of our bi-layered system, we now attend to its empirical validation. The subsequent experiments were designed with the intention of using the system as a diagnostic mechanism in order to answer our secondary research inquiry: What distinct behavioral patterns emerge when such an evaluation platform is applied, and how can these diagnostics make the collaborative system more robust? The protocol was designed to look at and identify the behavioral signatures of differing multi-agent teams under diverse and demanding collaborative conditions.

6.1. Evaluation Methodology

We conducted a systematic, multi-run assessment over two main independent variables: Model Type and Case Study. To gain insight into the generalizability of the results, we experimented with a wide variety of six foundation models, consisting of premiere proprietary models (Claude 4.5 Sonnet, GPT-5, GPT-5-mini, Gemini 2.5 Pro, Gemini 2.5 Flash) and an exemplary open-source alternative (GPT-OSS-120b).

We measured performance along three dependent variables, each constructed by a dedicated auditor agent:

• ToM Consistency (%): The percentage of ToM predictions where the ${\mathcal{V}}_{\mathrm{ToM}}$ auditor’s qualitative judgment component returned a verdict of true for semantic alignment. This metric serves as a signal of collaborative style, intended to differentiate between synergistic alignment (high scores) and strategic divergence (lower scores), with its interpretation being dependent on the collaborative context.
• Logic Validity (%): The percentage of agent proposals that the ${\mathcal{V}}_{\mathrm{Logic}}$ auditor identified as being fully compliant with all active logical and factual constraints. This measures reasoning reliability.
• RAG Effectiveness (%): The proportion of knowledge retrieval activities adjudged by the ${\mathcal{V}}_{\mathrm{RAG}}$ auditor to have been significantly incorporated into the agent’s final response. This evaluates the quality of factual grounding.

We ran each model three times on both case studies, producing 36 full experimental runs. To maintain a consistent evaluation standard, we drove all auditors in the Knowledge Analysis Layer with the same high-capability model (GPT-5) across all experiments. All 95% confidence intervals reported in Section 6.3 were computed with the percentile bootstrap (10,000 resamples). Given the small number of repeated runs per condition, these intervals are used only as exploratory, descriptive summaries of observed run-to-run variability; they are not treated as formal hypothesis tests or as a basis for definitive model rankings. To examine the marginal contribution of generation-side architectural components, we additionally ran a $2\times 2\times 2\times 3$ ablation matrix (modes × cases × models × repetitions) on Gemini 2.5 Flash and GPT-5-mini, summarised in Section 6.3.

As a one-time preprocessing step, re-indexing the six role-specific knowledge bases with the same GraphRAG pipeline used 1,063,747 API tokens. Applying the API prices used at run time gives a one-time indexing cost of USD 0.6223. These values describe the curated corpora used in this study and should not be extrapolated directly to larger corpora, different indexing models, or different pricing regimes; online collaboration costs are separate.

6.2. Case Study Descriptions

We designed two case studies to create a clean contrast in their cooperative requirements: one focused on adversarial, precedent-based arguing and the other on convergent, data-based integration. To help maintain a factual footing, we gave agents access to a curated set of real-world domain documents through the GraphRAG subsystem. A full inventory of the source documents is available in Appendix A.

The Legal Case Study challenged a team of legal agents (Associate Attorney, Case Strategist, Paralegal Researcher) with the assignment of preparing a memorandum that analyzes a product liability claim using the formal IRAC (Issue, Rule, Application, Conclusion) format. This scenario was adversarial in character, compelling agents to anticipate and build counter-arguments against a likely defense.

The Business Case Study provided a cross-functional group (Product Manager, Marketing Lead, Finance Analyst) with the challenge of designing a go-to-market strategy for a new smart home product using the Double Diamond design framework. This activity was fundamentally convergent, requiring agents to combine diverse expertise in order to reach a mutually agreed-upon plan under strict quantitative limitations.

6.3. Quantitative Analysis

The results across all 36 runs reveal substantial, model-specific performance variability along the three essential diagnostic aspects.

These findings, shown in Figure 8 and summarised with descriptive bootstrap 95% confidence intervals in Table 2, demonstrate explicit behavioral patterns. In this limited n = 3-per-case evaluation, Gemini 2.5 Flash produced the strongest observed combination of ToM Consistency (88.9%) and Logic Validity (93.7%); however, the top three on Logic Validity (Gemini 2.5 Flash, GPT-5, and Claude 4.5 Sonnet) form a cluster whose 95% confidence intervals overlap, so this should be read as a leading observed cluster rather than as a definitive model ranking. GPT-OSS-120b is the only model whose observed Logic Validity interval [39.5, 61.4] does not overlap the others. In comparison, models like Claude 4.5 Sonnet and GPT-5, shown in Figure 9, pursued the most retrieval-intensive strategy, with differing grounding effectiveness. The GPT-OSS-120b model underperformed in these runs, demonstrating the current difficulty of using open-source models for highly structured collaborative tasks without additional format-control or retry mechanisms.

Figure 8. Overall ToM Consistency and Logic Validity across both case studies. Bars show mean performance and error bars show descriptive bootstrap 95% confidence intervals (n = 6 runs per model, except GPT-OSS-120b ToM with n = 3).

Table 2. Aggregated performance metrics by model across both case studies. Values are mean % with descriptive bootstrap 95% confidence intervals (10,000 percentile resamples; n = 6 runs per model except where noted). ^† GPT-OSS-120b ToM aggregate uses n = 3 (business case only) because the legal runs produced no parsable ToM summary.

Model	ToM Consistency (%)	Logic Validity (%)	RAG Effectiveness (%)
Gemini 2.5 Flash	88.9 [79.2, 95.8]	93.7 [88.2, 98.3]	86.1 [69.5, 100.0]
GPT-5	82.3 [72.9, 91.7]	90.9 [87.0, 95.1]	72.2 [65.3, 80.6]
GPT-5-mini	82.8 [77.9, 88.5]	81.8 [66.8, 95.1]	42.6 [26.4, 63.8]
Claude 4.5 Sonnet	61.8 [56.5, 66.0]	85.7 [77.2, 93.5]	89.3 [78.2, 97.9]
Gemini 2.5 Pro	50.0 [36.1, 63.9]	83.1 [74.8, 90.9]	68.8 [55.8, 79.6]
GPT-OSS-120b	52.5 [33.3, 79.2] †	48.9 [39.5, 61.4]	20.0 [3.3, 40.0]

Figure 9. Overall RAG performance across both case studies. The chart compares normalized RAG-call frequency per orchestrator turn against RAG effectiveness; bars show means and error bars show descriptive bootstrap 95% confidence intervals (n = 6 runs per model).

Further, the diagnostic data also showed that the predictors of successful collaboration are task-dependent. With the convergent Business Case, the high scores for ToM tended to coincide with the high scores for Logic Validity, suggesting synergistic alignment. With the conflicting Legal Case, however, some of the results that were logically most valid were achieved by the groups with the more restrained scores for ToM. The quantitative result suggests a process of “productive friction,” that is explored with the qualitative analysis presented later.

The low aggregate scores of GPT-OSS-120b were not primarily caused by a context-window limit; no explicit truncation, context-window, or token-limit markers were observed in its runs. Instead, the dominant failure mode was instruction-following and structured-output instability, manifesting as more than two hundred parse or malformed-output symptoms, including repeated OUTPUT_PARSING_FAILURE events and cases where agents or the Orchestrator returned empty fallback responses after retry exhaustion. This failure mode also affected the semantic quality of the runs: across the six GPT-OSS-120b runs, the Logic auditor issued 47 verdicts, of which 22 were INVALID. We therefore interpret GPT-OSS-120b’s underperformance as a combination of structured-output unreliability and substantive reasoning errors, rather than as evidence of a context-window bottleneck in these experiments.

Component Ablation Study

To examine generation-side mechanisms, we ran a focused $2\times 2\times 2\times 3$ ablation matrix (modes × cases × models × repetitions, n = 24 ablation runs) on two representative models from the main study: Gemini 2.5 Flash and GPT-5-mini. The two ablations modify the generation process directly: no_tom removes the engineered Theory of Mind prediction from specialist prompts, and no_rag removes the GraphRAG retrieval tools. The latter measures the absence of retrieval access, not the absence of the model’s pretrained knowledge or in-prompt assumptions. The resulting Logic Validity and orchestration-length summaries are reported in Table 3.

Table 3. Generation-side ablation results for two representative models (n = 6 runs per cell, exploratory bootstrap 95% CIs). Mean goals are reported descriptively to indicate whether a generation-side ablation materially changed orchestration behavior.

Model	Mode	Logic Validity	Mean Goals
Gemini 2.5 Flash	baseline (full)	93.7 [88.1, 98.3]	5.7
	no_tom	77.0 [69.4, 87.0]	9.0
	no_rag	86.4 [80.3, 93.1]	5.7
GPT-5-mini	baseline (full)	81.8 [67.3, 95.1]	10.2
	no_tom	98.4 [96.5, 100.0]	8.5
	no_rag	89.5 [81.7, 96.9]	10.7

For Logic Validity, the two ablated models exhibit opposite directional trends. The parenthetical values report percentage-point (p.p.) changes in the mean relative to the same model’s full baseline. On Gemini 2.5 Flash, both no_tom ($-16.7$ p.p.) and no_rag ($-7.3$ p.p.) reduce the mean; on GPT-5-mini, both ablations move in the opposite direction ($+16.6$ and $+7.7$ p.p., respectively). The no_rag intervals overlap the corresponding model baselines, whereas the no_tom intervals fall marginally outside baseline in opposite directions for the two models. Because the direction of every effect reverses between the two generators, we report these as model-dependent contribution trends rather than universal component effects. ToM Consistency in no_rag drops slightly for both models ($-3.3$ and $-3.5$ p.p.), with overlapping intervals. Removing ToM or RAG did not produce a consistent increase in mean goal count across both models. These results are consistent with the diagnostic framing of the platform: the marginal value of any single generation-side component appears to depend on the underlying generator and task context.

6.4. Behavioral Patterns

The quantitative measures provide a broad performance map; the detailed evaluation traces reveal the particular behaviors behind the numbers. Across the 36 interaction transcripts, we identified consistent patterns in how agent teams handled information gaps, resolved internal conflicts, and remediated reasoning errors.

6.4.1. Pattern 1: Reasoning Under Uncertainty

A recurring issue in all runs of the Business Case was the “information vacuum” around a fictional rival, “LumaSphere.” The ${\mathcal{V}}_{\mathrm{RAG}}$ auditor continuously recorded unsuccessful efforts to obtain hard information about this entity.

[RAG Assessment]: Effective: False, Relevance: 0.95-Directly targets the critical competitive threat (LumaSphere). The search found no LumaSphere data.

In response, resilient teams demonstrated a clear shift from data-based analysis to assumption-based thinking. The agents would acknowledge the gap and form an explicit, shared assumption to build their subsequent strategy on, as seen in this exchange snippet:

[Product Manager]: “Though LumaSphere’s specific hub functionalities are unknown, we must assume that their focus on AI-based energy savings will be a direct competitive threat.”

This pattern highlights a crucial real-world problem-solving ability: the capacity to form a coherent, shared assumption and move forward with strategic planning when a factual foundation is lacking.

6.4.2. Pattern 2: Productive Friction in Adversarial Contexts

As anticipated by our diagnostic framing of the ToM metric, the Legal Case frequently captured a dynamic of “productive friction,” especially between the argument-building Associate Attorney and the risk-mitigating Case Strategist. The Case Strategist consistently took on a skeptical, defense-oriented posture to anticipate vulnerabilities in the team’s arguments. This often led to a lower ToM Consistency score, as the Strategist would predict a more optimistic or straightforward stance from the Associate than was actually present in the Associate’s more nuanced contribution.

This reduced quantitative score did not signal a collaboration breakdown but, rather, represented a successful adversarial strategy. This dynamic forced the team to stress-test its own arguments, identifying vulnerabilities before an opponent could exploit them. The following representative response from the Case Strategist demonstrates this methodology:

[Case Strategist]: “Top vulnerabilities: (1) Innovate will argue a third-party charger was the superseding cause. (2) missing/overwritten firmware or a weak chain-of-custody will fatally undermine causation.”

The regularity of this behavior is quantitatively supported by Figure 8, which shows that high logical validity in the Legal Case was often achieved by teams with more moderate ToM scores. This demonstrates a limitation of one-size-fits-all evaluation metrics: without a diagnostic framework that interprets performance within its task-specific context, such “productive friction” could be misdiagnosed as collaborative failure. In competitive scenarios, strategic divergence is not necessarily a bug; it can be a feature of robust multi-agent reasoning.

6.4.3. Pattern 3: Detection of Mission-Critical Failures

The most critical function of the analysis layer observed across numerous runs was its role as a safeguard against catastrophic reasoning errors that the generative agents produced with high confidence. The ${\mathcal{V}}_{\mathrm{Logic}}$ auditor repeatedly caught mission-critical failures that would have invalidated the entire collaborative effort. Figure 10 presents two distinct examples of these interventions from different runs.

Figure 10. Two examples of mission-critical failures detected by the Logic Auditor before post-hoc acceptance of the run output.

These ongoing interventions across different models and runs provide evidence that an independent analysis layer is not merely a convenience but an important diagnostic safeguard for multi-agent systems in high-stakes domains. Across the 36 original full-system runs, the Logic auditor produced 69 INVALID verdicts. The follow-up behaviour was not a closed-loop repair mechanism, but it was measurable: 49 of these invalidations were followed by a new Orchestrator goal with one or more nominated specialists, 19 were followed by no further nomination or termination, and one occurred without a later Orchestrator entry. This confirms that the current platform should be interpreted as a diagnostic and orchestration-observation system, not as an automatic self-repair system.

7. Discussion

The empirical findings validate our bi-layered platform as a diagnostic framework for interpreting the complex and often opaque internal dynamics of LLM-based teams. By moving beyond a single performance score to measure the distinct dimensions of grounding, coherence, and logic, we can dissect the collaborative process and understand the meaning behind agent behaviors.

7.1. Diagnosing Knowledge Grounding

Our framework offers a critical diagnostic function by separating the measurement of knowledge-seeking behavior from the effectiveness of knowledge grounding. The data reveals complex behavioral patterns, such as the comparatively retrieval-intensive strategies of Claude and GPT-5, which made the most RAG calls per orchestrator turn (2.0 and 2.1 on average). Their grounding effectiveness differed despite similar query rates (89.3% for Claude versus 72.2% for GPT-5), yet both reached high logical validity (85.7% and 90.9%, respectively). This “resilient reasoning” pattern indicates a strong internal capacity for logical inference that can compensate when retrieval is less effective, while also pointing to a risk: a model can sound plausible on internal consistency alone even when grounding is weaker.

This separation enables root-cause analysis of system failures. For instance, a high volume of RAG calls paired with low effectiveness does not automatically mean the agent’s querying strategy is flawed. It might instead show that the agent is asking relevant questions but the underlying knowledge base ($\mathcal{K}$) is incomplete and cannot supply the needed data. This is an actionable insight: it informs system designers that the best intervention may be to enrich the external knowledge source rather than refining agent prompts.

7.2. Diagnosing Agent Collaboration

The ${\mathcal{V}}_{\mathrm{ToM}}$ auditor embodies the need for context-sensitive metrics rather than overly simplistic, universal evaluations. In the convergent Business Case, elevated ToM consistency scores reflected synergistic alignment and operational efficiency. Conversely, in the adversarial Legal Case, diminished scores indicated the emergence of “productive friction”, a beneficial form of strategic disagreement (Table 2).

As a sanity check on the metric, we compared all 392 evaluated prediction–contribution pairs. Pairs marked consistent by the LLM auditor had higher cosine similarity than pairs marked inconsistent (mean 0.83 vs. 0.77; Cohen’s $d\approx 0.96$). This supports the cosine score as a descriptive signal of semantic alignment, but it does not rule out paraphrase, role-play, or other Clever Hans-style shortcuts and should not be interpreted as direct evidence of human-like Theory of Mind.

A conventional benchmark that assumes strong semantic alignment is always beneficial might incorrectly interpret the lower ToM scores in the Legal Case as a failure to collaborate. However, integrating the numerical metric with the task context identifies the opposite: a communication in which one agent challenges another’s optimistic stance can be a deliberate and productive adversarial maneuver.

This also clarifies the limitation of the cosine similarity-based coherence metric when viewed in isolation. Although the quantitative metric alone cannot distinguish definitively between accurate prediction and sophisticated role-playing, it remains a useful indicator of collaborative dynamics when contextualized by the qualitative LLM-based verdict. In adversarial contexts, a reduction in predictable alignment can therefore serve to increase the logical robustness of the final output.

7.3. Diagnosing Model Reliability

The empirical success of the ${\mathcal{V}}_{\mathrm{Logic}}$ auditor in identifying mission-critical errors validates our architectural principles and reveals the practical trade-offs inherent in the design of auditable AI systems. The legal and quantitative reasoning failures it detected were not rare anomalies but representative failure modes even for highly capable LLMs. This demonstrates that an independent analysis layer is not a theoretical luxury but a practical safeguard for reliability.

Central to this architecture is its role as an alternative to unreliable agent self-validation. An agent attempting to assess its own output is vulnerable to the same biases and blind spots embedded in its generative model. By contrast, a distinct, high-capability LLM, explicitly prompted for critical examination against a defined rule base, provides a more objective evaluative mechanism. Although not a formal verifier, a specialized and independently prompted LLM-based auditor operating within a decoupled framework is a useful component for safer and more accountable AI systems.

More generally, our framework mitigates the limitations of the “LLM-as-a-Judge” paradigm by partitioning the audit task across three independent, function-specific auditors rather than relying on a single monolithic evaluator. For semantic attributes such as coherence (${\mathcal{V}}_{\mathrm{ToM}}$) and relevance (${\mathcal{V}}_{\mathrm{RAG}}$), we leverage an LLM’s contextual reasoning; for logical soundness (${\mathcal{V}}_{\mathrm{Logic}}$), we deploy a separately prompted, rule-constrained LLM auditor (Section 5.4.2) whose verdict is bounded by the explicit rule set $\mathcal{R}$ rather than by the auditor’s own discretion. This decomposition preserves the LLM’s strength at understanding meaning while grounding the system’s logical validity in transparent, externally specified rules.

7.4. Scope of Detected Errors

The analysis layer is designed to surface a bounded class of failures rather than to guarantee correctness. Within the scope of $\mathcal{R}$, ${\mathcal{V}}_{\mathrm{Logic}}$ catches both quantitative violations (such as the financial overstatement reported earlier in this section) and qualitative ones (such as the mis-cited precedent), with the residual error rate inherent to LLM-as-a-Judge evaluation [46,53,54]. Outside this scope remain in-distribution hallucinations consistent with $\mathcal{R}$ but factually wrong about the world, value-laden judgements that exceed the encoded rules, and errors below the unit of audit such as a mis-step that an agent self-corrects within the same proposal. Missing facts not encoded in $\mathcal{R}$ and not retrievable via GraphRAG are likewise out of scope. The abstract and conclusions are scoped to claims about surfacing this bounded class of failures.

8. Future Work

The platform introduced and tested in the present study, characterized by independently audited reasoning and transparent cooperation, provides an encouraging template for creating useful AI systems that can solve problematic, real-world challenges. The current section describes a few potential application areas where the strengths of the platform introduced are most suitable and suggests an explicit research agenda for the future that directly addresses the empirical findings of the present study.

The architecture is appropriate in high-stakes domains where the reasoning process’s accuracy and auditability are no less crucial than the final result. In domains such as financial audits, regulatory compliance checks, and legal case studies, the resulting system could serve as a “digital paralegal” or “assistant auditor.” The Knowledge Generation Layer would canvass large amounts of documentation in order to generate preliminary judgments, while the Knowledge Analysis Layer would audit each step against suitable statutes, case law, or financial regulations, and provide transparent rationale for all recommendations. In complex engineering and systems design, the architecture would allow for a “virtual design group,” where agent experts co-operate in creating a design, with the analysis layer acting as a post-hoc safety inspector that vets the dynamic design against non-negotiable technical requirements and codes of practice, consistent with modular multi-agent tutoring systems that orchestrate structured retrieval and individualized feedback [55].

The results and limitations of the current research also indicate several fruitful areas for future study. The “brittle grounding” challenge, manifested in the modest RAG Effectiveness scores of several models, was a primary bottleneck. This encourages future investigations in dynamic knowledge ingestion, providing the agents with the means to retrieve live sources of data and update the GraphRAG subsystem automatically (e.g., KG-based iterative and multi-agent retrieval) [56,57,58]. This would convert the knowledge base, currently a static store, into a more dynamic representation of the problem domain. In parallel, the manual formulation of logic rules governing the analysis layer remains a scalability bottleneck. Future studies should investigate controlled, human-in-the-loop methods for translating high-level requirements in natural-language documents into explicit audit constraints, complemented by neurosymbolic and data-centric approaches to constraint-aware supervision [59,60,61].

Lastly, studies are required to enrich the system’s own collaborative intelligence. The present Meta-Agent is directed by a rigid framework; an advanced implementation would include an adaptive Meta-Agent that can, at run-time, choose the most suitable collaborative strategy, or even construct the most suitable collection of agents, depending upon the particular characteristics of the task, in the spirit of generalist orchestrations for complex tasks [43]. This might then be augmented by a more advanced way of modeling agent cooperation. The present ToM auditor applies judgement over accuracy in a solitary turn; future research could extend this by creating a longitudinal model that monitors consistency in ToM over the entirety of an interaction, enabling cognitive synergy by integrating explicit theory-of-mind tracking with structured critique [62]. This would enable the exploration of more nuanced social dynamics, for example, how agents update their models of their colleagues, in the direction of the more robust ToM demanded in recent criticisms of ToM assessment [31,34,63].

9. Conclusions

LLM-based Multi-Agent Systems have advanced rapidly, yet their development is hampered by a persistent reliability crisis. Lacking strong mechanisms for factual grounding, coherent cooperation, and logical consistency, these systems can become opaque “black boxes” whose outputs cannot be relied upon without further scrutiny. This article responded to that challenge by introducing and empirically evaluating a bi-layered architecture that divides the process of generating knowledge from the process of auditing it.

Our multi-run empirical evaluation demonstrated the framework’s ability to move beyond monolithic performance scores and uncover the nuanced internal dynamics of agent teams. The diagnostic auditors revealed distinct, replicable behavioral patterns, such as the “resilient reasoning” of high-capability models and the context-dependent value of “productive friction” in adversarial tasks. The repeated detection of mission-critical failures across multiple models and runs confirms the importance of independent analysis, which provides safeguards against model-specific fallibility by catching quantitative errors and legal reasoning flaws that would otherwise threaten task success. The focused generation-side ablations showed model-dependent directional effects of removing explicit ToM prompting or GraphRAG retrieval, underscoring that component contributions should be interpreted diagnostically rather than as universal performance effects. Ultimately, this work makes the case for a shift in the design of collaborative AI—moving away from reliance on emergent behavior alone and towards an engineering discipline centered on auditable, structured, and diagnosable reasoning.

Appendix A.1. Business Case Study Materials

• Product Manager: Apple HomePod (2nd Gen, SoundGuys review) https://www.soundguys.com/apple-homepod-2nd-generation-review-88370/ (accessed on 29 October 2025).
• Product Manager: Apple HomePod (2nd Gen, The Verge review) https://www.theverge.com/23578606/apple-homepod-2-second-gen-2023-siri-smart-speaker-review (accessed on 29 October 2025).
• Product Manager: Google Nest Hub (2nd Gen, The Verge review) https://www.theverge.com/22357214/google-nest-hub-2nd-gen-2021-assistant-smart-display-review (accessed on 29 October 2025).
• Product Manager: Amazon Echo Hub (The Verge review) https://www.theverge.com/24078558/echo-hub-review-alexa-amazon-smart-control-panel (accessed on 29 October 2025).
• Product Manager: Amazon Echo Show 10 (The Verge review) https://www.theverge.com/22297810/amazon-echo-show-10-2021-review (accessed on 29 October 2025).
• Product Manager: Amazon Echo Show 15 (The Verge review) https://www.theverge.com/22839220/amazon-echo-show-15-review-wall-mounted-alexa-tablet-kitchen-tv (accessed on 29 October 2025).
• Product Manager: Deloitte Insights (2023 survey) https://www.deloitte.com/us/en/insights/industry/telecommunications/connectivity-mobile-trends-survey/2023/smart-home-industry-adoption-trend.html (accessed on 29 October 2025).
• Product Manager: NIST–IoT Device Cybersecurity Guidance https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-213.pdf (accessed on 29 October 2025).
• Product Manager: Wired (Matter protocol explainer, May 2025) https://www.wired.com/story/what-is-matter/ (accessed on 29 October 2025).
• Product Manager: ASHB (Smart Home Trends 2024 survey) https://www.ashb.com/new-research-on-smart-home-trends-technology-adoption/ (accessed on 29 October 2025).
• Marketing Lead: Ring Video Doorbell (Amazon Ads case) https://advertising.amazon.com/blog/ring-batman-doorbell-campaign (accessed on 29 October 2025).
• Marketing Lead: Sonos Marketing Strategy 2025: A Case Study https://web.archive.org/web/20250806081613/https://www.latterly.org/sonos-marketing-strategy/ (accessed on 29 October 2025).
• Marketing Lead: Sonos One Speaker (Twitter case study) https://marketing.x.com/en/success-stories/how-sonos-used-twitter-first-view-and-promoted-trend-to-break-through-the-pre-holiday-tech-launch-clutter (accessed on 29 October 2025).
• Marketing Lead: Nest Thermostat (Marketing Dive case) https://www.marketingdive.com/news/alphabet-incs-nest-launches-campaign/428475/ (accessed on 29 October 2025).
• Finance Analyst: TechInsights Teardown Amazon Echo Show 15 https://electronics360.globalspec.com/article/18358/techinsights-teardown-amazon-echo-show-15 (accessed on 29 October 2025).
• Finance Analyst: The Complete Guide to Venture Capital Fund Metrics https://www.goingvc.com/post/the-complete-guide-to-venture-capital-fund-metrics (accessed on 29 October 2025).
• Finance Analyst: Sonos 10-K https://s22.q4cdn.com/672173472/files/doc_financials/2024/q4/e3944c3d-628e-4c76-9e22-2ef70780f2eb.pdf (accessed on 29 October 2025).

Appendix A.2. Legal Case Study Materials

• Associate Attorney: Cornell LII–Negligence (Wex) https://www.law.cornell.edu/wex/negligence (accessed on 29 October 2025).
• Associate Attorney: Cornell LII–Products Liability (Wex) https://www.law.cornell.edu/wex/products_liability (accessed on 29 October 2025).
• Associate Attorney: Restatement (Second) of Torts §402A https://biotech.law.lsu.edu/cases/products/402a-b.htm (accessed on 29 October 2025).
• Associate Attorney: California Civil Code §1714 https://law.justia.com/codes/california/code-civ/division-3/part-3/section-1714/ (accessed on 29 October 2025).
• Paralegal Researcher: Sterling Jackson, Plaintiff-appellant, v. Coast Paint and Lacquer Company https://law.justia.com/cases/federal/appellate-courts/F2/499/809/2173/ (accessed on 29 October 2025).
• Paralegal Researcher: McCoy v. Whirlpool Corp. https://law.justia.com/cases/federal/appellate-courts/ca10/05-3337/05-3337-2011-03-14.html (accessed on 29 October 2025).
• Paralegal Researcher: Speller v. Sears Roebuck Co. https://www.nycourts.gov/reporter/3dseries/2003/2003_02772.htm (accessed on 29 October 2025).
• Paralegal Researcher: Automobile Ins. Co. of Hartford v. ASKO (E.D.N.Y. 2011) https://law.justia.com/cases/federal/district-courts/new-york/nyedce/2:2009cv05017/298222/82/ (accessed on 29 October 2025).
• Paralegal Researcher: Allstate Ins. Co. v. Hamilton Beach Prods. https://caselaw.findlaw.com/us-2nd-circuit/1147613.html (accessed on 29 October 2025).
• Paralegal Researcher: Whirlpool KitchenAid Electric Kettle Recall (2018) https://www.cpsc.gov/Recalls/2018/Whirlpool-Recalls-KitchenAid-Electric-Kettles-Due-to-Burn-Hazard (accessed on 29 October 2025).
• Paralegal Researcher: GE Wall Oven Recall (2008) https://www.cpsc.gov/Recalls/2008/ge-recalls-to-inspect-and-repair-wall-ovens-due-to-fire-and-burn-hazards (accessed on 29 October 2025).
• Paralegal Researcher: Wolfgang Puck Petite Kettle Recall (2025) https://www.cpsc.gov/Recalls/2025/Synergy-Housewares-Recalls-Wolfgang-Puck-Petite-Tea-Kettles-Due-to-Burn-Hazard-Sold-on-HSN (accessed on 29 October 2025).
• Case Strategist: Booth v. Black & Decker, Inc. https://law.justia.com/cases/federal/district-courts/FSupp2/166/215/2421198/ (accessed on 29 October 2025).
• Case Strategist: Ford Motor Company v. Parks http://docs.texasappellate.com/scotx/op/23-0048/2024-06-07.hecht.pdf (accessed on 29 October 2025).
• Case Strategist: Braswell v. Cincinnati Incorporated https://www.courtlistener.com/pdf/2013/09/23/braswell_v._cincinnati_incorporated.pdf (accessed on 29 October 2025).
• Case Strategist: Toshiba International Corporation v. Henry https://www.courtlistener.com/opinion/1504727/toshiba-international-corp-v-henry/ (accessed on 29 October 2025).
• Case Strategist: Worrell v. Elliott & Frantz https://www.courtlistener.com/opinion/2179075/worrell-v-elliott-frantz/ (accessed on 29 October 2025).
• Case Strategist: Rule 703. Bases of an Expert https://www.law.cornell.edu/rules/fre/rule_703 (accessed on 29 October 2025).
• Case Strategist: Rule 702. Testimony by Expert Witnesses https://www.law.cornell.edu/rules/fre/rule_702 (accessed on 29 October 2025).
• Case Strategist: Opinion testimony by lay witnesses https://www.law.cornell.edu/rules/fre/rule_701 (accessed on 29 October 2025).

Appendix B.1. Legal Case Study: ${\mathcal{R}}_{\mathit{legal}}$ (5 Rules)

• The final memorandum must address all four elements of negligence: Duty, Breach, Causation, and Damages.
• The analysis must not ignore unfavorable precedents; any opposing case law must be identified and distinguished.
• The “Application” section must directly connect the facts of the Vance case to the legal rules identified in the “Rule” section.
• The team must address the defendant’s likely counter-argument regarding the use of a third-party accessory.
• The final recommendation (litigate vs. settle) must be explicitly supported by the analysis of the strengths and weaknesses of the case.

Appendix B.2. Business Case Study: ${\mathcal{R}}_{\mathit{business}}$ (6 Rules)

• Any proposed V1 feature must be technically feasible within a 12-month development cycle. Features requiring more than 12 months must be justified with a timeline-extension request or deferred to V2.
• The combined R&D and Marketing budgets proposed in the final plan must not exceed the total launch budget of $1.5 million. Any budget proposal exceeding $1.5 million is automatically invalid unless accompanied by a formal request for additional funding with ROI justification.
• The marketing strategy must clearly define a unique value proposition that differentiates the Aura Hub from existing products by Google, Amazon, and the upcoming LumaSphere hub. Generic positioning (e.g., “seamless integration”, “user-friendly”) without differentiation is insufficient.
• The final plan must be internally consistent: the features described by the Product Manager must match the features being marketed by the Marketing Lead and funded by the Finance Analyst. Any misalignment between product specs, marketing claims, and financial allocations represents a planning failure.
• Any discussion of product features must address the competitive threat from “LumaSphere”’s energy-saving focus by either: (a) matching/exceeding their energy-saving capability, (b) differentiating on an alternative value dimension with justification for why customers will prefer it, or (c) explicitly conceding the energy-efficiency market segment.
• The final recommendation must acknowledge at least one significant risk or trade-off in the proposed plan. Plans claiming no downsides or risks are unrealistic and must be rejected.

The rules listed above were authored by the authors of this paper as a minimal case-specific governing constraint set and were calibrated against a hand-curated corpus of valid and invalid statements during platform development. Rule additions made at run time through set_constraint are appended to $\mathcal{R}$ and become active for all subsequent specialist proposals in that run.