Cognitive Integration for Hybrid Collective Agency

Abstract

Can human–machine hybrid systems (HMHs) constitute genuine collective agents? This paper defends an affirmative answer. I argue that HMHs achieve collective intentionality without shared consciousness by satisfying the following three functional criteria: goal alignment, functional complementarity, and stable interactivity. Against this functionalist account, the following two objections arise: (1) the cognitive bloat problem, that functional criteria cannot distinguish genuine cognitive integration from mere tool use; and (2) the phenomenological challenge, that AI’s lack of practical reason reduces human–AI interaction to subject–tool relations. I respond by distinguishing constitutive from instrumental functional contributions and showing that collective agency requires stable functional integration, not phenomenological fusion. The result is what I call Functional Hybrid Collective Agents (FHCAs), which are systems exhibiting irreducible collective intentionality through deep human–AI coupling.

1. Introduction

The deep integration of artificial intelligence is fundamentally challenging the core assumptions of traditional collective action theory. Classical frameworks anchor collective intentionality in shared phenomenal consciousness and reciprocal commitment among agents (Bratman 1992 [1]; Gilbert 2009 [2]); yet they fail to account for the collaborative practices of human–machine hybrids (HMHs), which are systems where non-conscious AI components participate in goal-directed behavior through functional coupling, forming quasi-collective decision-making patterns. This paper argues that HMHs have strong potential to constitute a novel form of functional collective agency whose collective intentionality depends not on phenomenal consciousness but on the satisfaction of four cognitive integration criteria. The three criteria include (i) goal alignment, (ii) functional complementarity, and (iii) stable interactivity (Hutchins 1995 [3]; List and Pettit 2011 [4]; Fisher and Ravizza 1998 [5]). These criteria collectively support the emergence of a functional collective agency that does not rely on shared subjective consciousness but on the robust integration of human and AI components through functional coupling and goal alignment. Moreover, this perspective aligns with contemporary theories of collective agency that conceptualize it as a set of relational events—encompassing both internal structures and external functions—thereby emphasizing these constitutive relationships over individual subjective states (see Schmid 2009 [6]; Schweikard and Schmid 2021 [7]; Pettit 2009 [8]; List and Pettit 2011 [4]; Barandiaran, Di Paolo, and Rohde 2009 [9]).The functionalist perspective, however, faces significant critiques. A primary critique is “cognitive bloat” (Kallestrup 2023 [10]) which cautions against relying solely on functional contribution to define cognitive subjects, as it could arbitrarily broaden their boundaries. This critique extends to the criterion of functional complementarity, which does not provide a definitive boundary. A more fundamental critique stems from the irreducibility of human practical rationality, which posits that genuine mutual understanding between humans and AI is impossible. From this viewpoint, the interaction is not a reciprocal “subject–subject” relationship, as occurs between humans, but an instrumental “subject–-tool” dynamic. As Hans-Georg Gadamer argued [11], genuine understanding is inseparable from phenomenological experience and the integration of perspectives shaped by historical and lived engagement. Lacking these experiential and interpretive capacities, AI cannot participate in true practical rationality; its operations, however complex, remain fundamentally algorithmic.

In response to these critiques, and from a perspective deeply integrated with functionalism, I propose the following bold philosophical hypothesis: when emergent behaviors from interacting agents create the necessary conditions for genuine agency, we may witness the formation of a multi-system agent. This novel form of distributed cognitive entity transcends traditional boundaries between humans and artificial intelligence. The rapid advancement of strong AI technology has brought about unprecedented forms of human–AI integration Crucially, in certain instances, these deeply integrated human–AI systems manifest cognitive characteristics that have evolved beyond simple complementary tool use (Stapleton and Froese 2015 [12], p. 224). Instead, they form distributed cognitive wholes marked by emergent mindedness, which is a property that arises from the intricate interplay between biological and artificial components and is irreducible to either1.

2. Limitations of Traditional Theories and a Shift in Perspective

Traditional theories of collective intentionality, while illuminating for purely human contexts, face significant limitations when applied to human–AI hybrid systems. These theories broadly diverge yet converge on an anthropocentric premise that hinders their applicability to AI participation.

One prominent theory, exemplified by Tuomela and Miller [14], analyzes “we-intentions” as a combination of individual intentions coupled with mutual beliefs. Here, an agent intends a joint action, believes others share this intention, and believes this mutual intention exists among all participants. In contrast, Searle argues that “we-intentions” are primitive, irreducible collective phenomena that cannot be explained as mere aggregations of individual mental states. He asserts that collective intentionality is a fundamental biological phenomenon, emphasizing a “sense of doing (wanting, believing, etc.) something together” from which individual intentionality is derived (Searle 1990 [15]; 1995 [16], pp. 24–25). In a sophisticated individualist account, Michael Bratman analyzes shared intentions in terms of interlocking “I-intentions” that are coupled with beliefs about others’ corresponding intentions and commitments to coordinate. However, critics argue that these conditions can be overly permissive; for example, two individuals who happen to meet at a bank for separate, strategic gains might satisfy Bratman’s formal requirements while lacking any genuine collective agency. This critique highlights a crucial insight: true collective agency seems to require that participants represent their activity as having essentially collective content—content that cannot be decomposed into individual tasks performed in parallel (Bacharach 2006 [17]; Bratman 2014 [18]; Gold and Sugden 2007 [19]; Tenenbaum 2015 [20], pp. 3385–3386).It is precisely this reliance on a specific, irreducible mental representation that exposes the inherent anthropocentrism of these traditional theories. This limitation becomes a fundamental obstacle when one of the potential participants is a non-conscious AI.

When applied to human–machine hybrid systems (HMHs), traditional theories reveal fundamental limitations rooted in their anthropocentric assumptions. By requiring shared consciousness and human-specific subjectivity, they exclude AI agents by definition, regardless of functional capabilities. This requirement for participants to represent collective content through human-like mental states creates an even more acute version of the permissiveness problem: how can we evaluate human–AI coordination when one party operates algorithmically rather than intentionally? Yet, traditional theories offer essential analytical tools. The structural features they identify—coordination mechanisms, interlocking goals, and mutual responsiveness—apply regardless of agent type. Most importantly, their distinction between “functional integration” and genuine “collective agency” provides an evaluative framework. This allows us to ask whether sophisticated human–AI coordination (through real-time data sharing and adaptive algorithms) transcends mere functional integration to achieve collective agency2.

The fundamental issue is that traditional theories commit the following category mistake (Theiner 2017 [22], p. 3; Brouwer et al. 2021 [23]): they assess collective agency based on individual components’ intrinsic properties rather than the integrated system’s emergent capacities. This leads to important but potentially limiting debates about whether AI possesses consciousness or “original” versus “derivative” intentionality (Jordan 2002 [24]). While these ontological questions remain philosophically significant, they should not serve as gatekeeping criteria for HMH collective agency. Instead of making individual mental states a prerequisite, we should examine whether the integrated human–AI architecture fulfills the functional roles constitutive of agency. This demands a shift from exclusionary criteria based on consciousness to inclusive frameworks that assess functional integration and emergent capabilities.

3. Towards Criteria for Collective Agency

When AI serves as a cognitive extension in collective decision-making, the scenario becomes significantly more complex. This complexity arises not just from the addition of a new element but also from the fundamental reconfiguration of collective cognition. Traditional models, which assume that cognitive processes are either fully human or purely instrumental, are inadequate for understanding human–AI integration. Instead, we need a framework that captures the distributed nature of cognition across biological and artificial substrates, which can give rise to emergent cognitive properties (see Clark and Chalmers 1998 [25]; Clark 2007; 2008 [21,26]. To achieve this, it is imperative to develop criteria that can identify when such distribution constitutes genuine cognitive integration rather than mere tool use.

3.1. Collective Agency and Collective Subjectivity

While acknowledging the existence of collective agency does not necessarily imply the existence of collective subjectivity, it is important to clarify the relationship between these two concepts3. In human embodied social interactions, the phenomenon of the extended mind can generate a shared second-person perspective through mutual engagement (Gallagher 2013 [27]). However, this shared perspective does not extend to a collective first-person perspective; that is, there is no subjective experience center as the collective itself.

From an enactivist perspective, the constitution of subjectivity depends not only on agency but also on the monitoring and regulation of internal processes by specialized subsystems (Thompson 2007 [28]). This suggests that the realization of collective subjectivity may require a deeper level of physical integration, rather than merely functional coordination through interaction. Thus, the relationship between collective subjectivity and collective agency is such that the former is a necessary but not sufficient condition for the latter, and the scope of possibility for the two does not fully overlap, as agency can be achieved through loose interactive coordination, while subjectivity requires stricter internal integration (List and Pettit 2011 [4]; Zahavi 2014 [29]).This conceptual distinction provides a clear theoretical framework for understanding the different levels of integration required for these systems to function effectively. Additionally, it is essential for addressing the challenges and possibilities of integrating human and artificial components in a way that enhances system performance while maintaining reliability and safety.

3.2. Three Functional Criteria

To properly conceptualize HMHs as legitimate collective agents, I propose the following three core functional criteria for their cognitive integration: (1) goal alignment, (2) functional complementarity, and (3) stable interactivity.

First, goal alignment refers to the extent to which the components of a system share a unified set of goals that guide their collective behavior. In this paper, I emphasize not merely superficial goal congruence but rather a deep integration of human core values with the functional objectives of artificial intelligence. Achieving this deep integration requires AI systems to consistently operate in accordance with societal ethical norms and long-term human interests across diverse contexts. However, in practice, factors such as dynamic environmental changes, strategic interests, and technical inaccuracies pose significant challenges to maintaining collective goal consistency. This paper does not aim to propose specific solutions to these challenges but instead focuses on exploring the theoretical possibility of goal alignment within a functionalist framework.

To argue for this possibility, we must first move beyond traditional theories of collective intentionality. These theories often conceptualize goal alignment as a shared psychological state, implying that all participants hold common beliefs and intentions regarding their objectives (Miller and Tuomela 2014 [30]). However, this approach is not feasible for human–AI systems, as it presupposes that AI possesses psychological representations and subjective value judgment capabilities that it inherently lacks. Requiring AI to “understand” or “internalize” human values is akin to expecting a calculator to “appreciate” the aesthetic value of mathematical formulas; it can functionally perform calculations but cannot experience their deeper meanings or values. Thus, this path of functional integration is not a single step but a gradual, iterative process (Fasoro 2024 [31]). From a functionalist perspective, achieving goal alignment depends on specification and assurance. As Christopher Winter, Nicholas Hollman, and David Manheim noted (2023) in their discussion on aligning values in artificial judicial intelligence [32], deep value alignment depends on mechanisms of specification and assurance to ensure that AI reliably follows established values. This also implies that, without stable and reliable lower-level perceptual, decision-making, and execution functions, higher-level value integration is untenable.

To illustrate this, consider a Level 4 autonomous vehicle that encounters issues with fundamental tasks like obstacle detection. Prioritizing advanced rules like “protecting pedestrians” without mastering basic functions can lead to serious errors. For instance, these vehicles sometimes undergo “phantom braking” in clear conditions due to delayed sensor responses, misinterpreting the environment and causing accidents (Muzahid et al. 2023 [33,34]) This demonstrates that integrating complex values into AI systems relies on solid foundational capabilities; without them, such integrations can be counterproductive.

Second, I will elaborate on the criterion of functional complementarity. This criterion emphasizes that the strength of hybrid systems lies precisely in its ability to integrate heterogeneity. Human participants offer practical wisdom, ethical judgment, and nuanced understanding of ambiguous situations, while AI contributes extraordinary capabilities in pattern recognition, high-speed computation, and optimization simulations. This division of labor highlights the complementary nature of their distinct skills, enabling them to accomplish complex tasks that no individual component could achieve alone. It is this deep complementarity—rather than homogeneity of abilities—that allows the overall performance of the hybrid system to surpass that of any single part, such as in medical diagnosis through human–AI collaboration. Physicians rely on their clinical experience and ethical judgment to understand patients’ complex emotional and social contexts and make personalized treatment decisions. Meanwhile, AI systems analyze large volumes of medical images and electronic health records to rapidly identify disease patterns and risks. This deep complementarity—where human practical wisdom and AI data-processing capabilities work together—enables more accurate diagnoses, higher efficiency, and better management of complex or ambiguous cases. Neither could achieve this level of performance alone, illustrating how the heterogeneity within the hybrid system allows it to surpass the capabilities of any individual component.

Of course, it must be acknowledged that the model described here represents an idealized, deeply integrated system. It also needs to further explain and address the complexities and non-ideal problems encountered in real-world settings. For instance, when an AI algorithm processes data outside its training distribution (“out-of-distribution” data), or when human operators misunderstand AI outputs due to cognitive biases, the stability of their interaction and collaboration may break down. Such “coupling failures” are not mere operational glitches but reveal the system’s constitutive dependencies on training distributions and intersubjective interpretive frameworks. These failures underscore that genuine cognitive integration is not simply adding human and AI capabilities but a dynamic, fragile process requiring continuous calibration, where each component’s performance is intrinsically linked to its embedded environment and its interactions with others.

Finally, the criterion of stable interactivity ensures that functional integration is not ephemeral but rather persistent and reliable. In the predictive intelligence framework, stable interaction can be defined as the system’s (or within systemic components, including cognitive tools and modules) capacity to continually maintain functional coupling through iterative goal calibration mechanisms in uncertain environments (Gamez 2023 [35]) This deep, irreducible coupling depends on a specific structure of trust. To clarify this, it is helpful to compare two forms of trust. In Otto’s notebook case, trust is instrumental based on physical reliability and past successes. In contrast, human collective action involves intersubjective trust, which extends beyond static information to include shared intentions, commitments, and adherence to social norms. Critics may argue that humans are inherently less reliable than tools However, I believe that social norms, role assignments, and standardized communication procedures can foster a dynamic, processual form of reliability—similar to Hutchins’ cockpit example—where trust emerges from ongoing interactions rather than static facts. This kind of reliability functions as a guarantee, enabling coordinated collective agency through continuous, socially embedded exchanges (Clark and Chalmers 1998 [25]; Ongaro and Deschenaux 2022 [36]; Hutchins 1995 [3]).

Maintaining stable interactivity in human–AI systems often hinges on several key factors, including reliability of agent behaviors, adaptability to human variability, effective conflict minimization and error recovery and equilibrium between human input and agent autonomy. To be specific, if the AI component cannot adapt to the inherent variability, biases, or cognitive strengths/weaknesses of its human members, then the supervenience relation between individual contributions and group-level agency becomes unstable or impossible. Additionally, assuming that a human–AI hybrid system can be viewed as a collective agent, it then follows that this entity, much like an individual agent, is expected to exhibit a demonstrable degree of rationality. This rationality implies that its collective “beliefs”—i.e., its representational states and aggregated understandings—should be consistent, and its collective actions—its interventions in the world—should be coherent with its established goals. This expectation aligns directly with List and Pettit’s conditions for group agency, particularly the demand that the system satisfies conditions of theoretical rationality (in forming and revising states) and practical rationality (in acting based on those states). However, in any complex system, and especially in a heterogeneous human–AI hybrid, internal discrepancies and external challenges pose significant threats to this desired rationality, such as human bias, AI encountering out-of-distribution data, or the inherent heterogeneity of human and AI cognition.

In brief, human–AI systems can promisingly form genuine collective agents with emergent cognitive properties by meeting the outlined criteria. Especially with AI’s continued advancement, this invites us to redefine “cognition” as a functional achievement rather than a substrate-specific property, allowing for a broader understanding of intelligent agency that transcends traditional biological boundaries. However, these systems are dynamic and fragile hybrids, not seamless entities, requiring continuous calibration. Breakdowns in the functional integration between human and AI components, which are common in practice, underscore the need for a clear grasp of AI’s capabilities and limitations and for humans to critically assess AI outputs and intervene in the collaborative process. Understanding the shift from mere collaboration to genuine cognitive integration is crucial for exploring hybrid agency and the future of intelligence.

4. From Emergent Coordination to Cognitive Integration

The study of multi-agent systems (MAS) provides crucial insights for understanding cognitive integration in hybrid collective agency. While traditional MAS research distinguishes between pure systems (containing only one type of agent) and hybrid systems (involving different agent types) (Parunak et al. 2007 [37]) our focus on human–machine hybrids reveals that cognitive heterogeneity is not merely a technical feature but a fundamental condition for emergent collective agency. Of course, the progression from aligned behavior (agents moving in the same direction) to interactive behavior (agents responding to each other) to flocking behavior (collective movement patterns) demonstrates increasing coordination complexity. However, even sophisticated flocking based on simple rules—collision avoidance, velocity matching, and cohesion—remains at the level of behavioral coordination rather than cognitive integration.

While functional integration is necessary for genuine collective agency, this paper argues that collective agency demands a deeper level of functional integration, not just coordinated behavior. To clarify the unique collective agency of human–AI systems, it is necessary to distinguish them from the following two concepts:

The first is a Multi-agent System (MAS), in which only the individual agents at the lower level are considered to have agency, while the system as a whole does not. For instance, a stock trading system composed of multiple AIs, each analyzing different market data, exemplifies a MAS. Although these AIs share information, they operate independently, and the system lacks a unified goal or norm. While such a system can accomplish complex tasks, it lacks collective agency.

In contrast, a Multi-system Agent (MSA) is a higher-level entity that is considered a single agent, while its lower-level components are not. A multicellular organism like the human body is a clear example. The organism itself is seen as the agent, while individual cells do not possess independent agency; their behavior is entirely constrained by the overall physiological needs and norms of the organism.

However, the human–AI systems do not fit into either of these categories. The primary reasons include the following: Firstly, they transcend mere multi-agent systems because the collective itself can exhibit emergent agency and unified collective intentionality, going beyond a mere aggregation of individual behaviors. Unlike simple clusters where only individual components act, FHCA aims to form coherent collective judgments and pursue shared goals at the system level. Secondly, and crucially, they are not multi-system agents in the same reductive sense as multicellular organisms. While multicellular organisms possess clear system-level agency, their constituent parts (e.g., individual cells) are not considered agents themselves. In contrast, within FHCA, the human component explicitly retains its agency and regulatory control over systemic goals. Humans are not utterly reduced to non-agentic sub-modules but maintain the capacity to adjust AI’s behavior and steer the system’s objectives.

Based on these insights, we propose a new type of hybrid system: the Functional Hybrid Collective Agent (FHCA). This type describes systems composed of humans and AI that achieve collective agency through goal alignment, functional complementarity, and stable interaction. In FHCA systems, humans can adjust AI behavior to maintain continuous control over system goals. By meeting these criteria, FHCA aims to create an integrated system capable of effectively achieving complex goals. To enable effective interaction and maintain control over system goals, the design of human–machine interfaces is crucial. For example, Mueller et al. proposed a design space model for interwoven systems, categorizing human–machine relationships into “caretaker,” “angel,” “adversary,” and “influencer” types [38]. The model suggests that designers should aim for the “caretaker” type, where humans have high awareness of the machine’s mechanisms and the machine’s goals are highly aligned with human objectives, achieving optimal control and collaboration.

A key and more challenging issue arises from the formation of norms in hybrid systems. The shaping of norms in these systems is a complex process influenced by institutional design, social norms, cultural background, power structures, and behavioral interactions. These factors interact dynamically, both internally and externally, to shape the normativity of hybrid systems. The central task, therefore, is to navigate the tensions among these factors to ensure the system’s functional efficacy and ethical legitimacy. Philosophical theories of collective agency offer crucial insights into this challenge. For instance, Margaret Gilbert (2013, [39])’s theory of “joint commitment” posits that normativity arises when individuals constitute a “plural subject.” Within this collective, members are mutually accountable for upholding their commitments, giving rise to a form of normativity that is irreducibly collective and not merely an aggregation of individual obligations. This suggests that for a hybrid system to possess genuine collective binding power, it must incorporate a mechanism analogous to joint commitment, rather than relying on a unilateral model where humans simply impose rules on an AI. Building on this, Tenenbaum’s non-reductionist perspective points out that the basis of normativity is the representation of “irreducible collective content” (such as “we are doing X together”) [20]. This collective representation itself implies normative constraints (for example, coordinating behavior to achieve a common goal), and its normativity precedes the “sub-plan meshing” at the individual level. This means that normativity in hybrid systems is not a mechanical product of human rule-setting and AI compliance, but an emergent property of dynamic, collective construction. This process is grounded in a shared representation of the system’s goals and legitimacy, demanding a synthesis of philosophical theory with the pragmatic challenges of integrating heterogeneous agents within specific technological contexts.

Consider an emergency response system combining autonomous drones and human operators. During flood rescue operations, the system faces a critical decision: evacuate civilians or address a fire threatening nearby structures. The AI component might rationally deviate from preset protocols, prioritizing immediate evacuation over comprehensive response plans. While violating conventional consistency requirements, such deviation demonstrates genuine agency through contextual adaptation. This case reveals the distinction between emergent coordination and cognitive integration. Traditional multi-agent systems coordinate through preset rules, proving inadequate under radical uncertainty. FHCA, by contrast, achieves collective agency through the following three mechanisms: (i) goal alignment, which is the unified commitment to maximizing rescue effectiveness rather than protocol adherence; (ii) functional complementarity, which is AI’s computational capacity for rapid situational assessment combined with human moral judgment; and (iii) stable interaction patterns maintaining system coherence.

These adaptive behaviors constitute cognitive integration at the system level. Agency emerges not from algorithmic outputs or aggregated individual actions but as an irreducible property enabling the system to transcend programmed constraints and exhibit context-sensitive rationality. This distinguishes mere behavioral coordination from the functional integration FHCA requires.

If the above analysis holds, then the future challenge extends beyond mere design considerations to fundamental questions about the nature of agency itself, namely how responsibility distributes across hybrid systems and what constitutes authentic autonomy when human judgment is inextricably intertwined with algorithmic mediation.

5. Weak and Strong Equivalence in HMHs

In this section, I aim to expound that the distinction between weak and strong equivalence in HMHs is crucial for understanding the emergent collective intentionality in these systems. Roughly, weak equivalence focuses on superficial behavioral mimicry, while strong equivalence involves deeper functional integration and emergent properties that transcend the capabilities of individual components.

Before providing a more detailed description of the two equivalent distinctions, let us first take a look at why I think this distinction is crucial.

Here, I emphasize that this distinction provides a more precise and rigorous analytical tool for understanding how human–machine hybrid systems (HMHs) exhibit emergent collective intentionality. It also addresses the question of whether HMHs can be considered genuine “minded systems.” Traditional theories of collective intentionality, with their human-centric assumptions, struggle to fully account for collective agency involving AI and may lead to category mistakes, conflating the conscious properties of individual components with the overall agency of the system. Meanwhile, although large language models (LLMs) display complex emergent behaviors, their intentionality is derivative, mimicking human patterns rather than possessing intrinsic, phenomenological understanding. This is similar to the “Chinese Room” argument, in which the system can generate responses of imitative understanding, but it does not possess any inherent understanding of its own (Searle 1980 [40]). In this context, a purely functionalist perspective, despite advocating for a shift in focus from material substrates to functional roles (Smart 2018 [41]; Parthemore 2011 [42]), risks oversimplification and misapplication of the “parity principle.” As Churchland and Churchland (1981) noted in their early critique of functionalism [43], reducing mental states to purely functional roles may overlook crucial qualitative aspects of consciousness. It may reduce hybrid intelligence merely to machines mimicking human reasoning, thus ignoring the profound ontological differences and failing to explain how genuine collective minds form through human–AI interaction. For instance, some may question whether AI merely triggers human decisions and how minds can be shared if AI bypasses human mental states. These questions highlight that mere behavioral similarity or simple functional coordination is insufficient to support the claim that HMHs possess collective agency. Such inquiries highlight that behavioral similarity or simple functional coordination is insufficient for attributing collective agency to HMHs. Given the arguments presented in the preceding section, it follows that my stance is clear: AI in human–machine hybrid systems (HMHs) transcend mere triggering of human decisions. By providing critical information, generating predictions, and suggesting courses of action, AI actively engages in the decision-making process. This active engagement constitutes a form of functional integration that substantively enhances human decision-making capabilities, creating a new level of functionality and intentionality distinct from the sum of its parts.

While this paper does not deny that LLMs may increasingly mimic high-fidelity metacognitive functions in their outputs, this should not be hastily equated with genuine metacognition. The following ontological gap persists: they lack phenomenological consciousness, semantic understanding, or intrinsic purpose. Any discussion of human–AI integration and collective decision-making must confront the following fundamental distinction: AI’s advanced outputs simulate metacognitive processes rather than embody them. Recognizing this clear ontological divide is crucial for a nuanced understanding of AI’s potential and limitations in human collective action.

It is important to note that this claim differs from, and may even conflict with, the functionalist perspective in extended cognition theory, which advocates for shifting focus from material substrates to functional roles. Proponents of this view argue that what matters is the function performed, not the biological or silicon nature of the component. As Smart (2018 [41], p. 15) contends, if a system is defined by its hardware, then humans performing an equivalent function must logically be considered part of it. Similarly, Parthemore (2011 [42], p. 88) argues that adhering to biological boundaries might reflect an unexamined “biological bias”. However, this purely functionalist view has two issues. First, it oversimplifies the problem by reducing hybrid intelligence to “machines mimicking” human reasoning, thereby ignoring richer phenomenological distinctions (which will be detailed in the next section’s core criticisms of functionalism and are not elaborated here). Second, it naively applies the “parity principle”—assuming that AI and human functions can be fully interchangeable—without acknowledging the deeper ontological differences previously emphasized. This oversight can lead to superficial analysis and conceptual flaws. For instance, in robotic surgery, although AI can provide highly precise mechanical operations and real-time data support, it cannot fully replace the clinical judgment and situational understanding of human surgeons. This functional gap can lead to unpredictable outcomes in practice, such as prolonged surgery times or increased complications (Göndöcs and Dörfler 2024 [44]; Khalifa and Albadawy 2024 [45]), which highlights the limitations of equating AI functions with human functions.

Given these considerations, I advocate for a nuanced, multi-layered notion of “equivalence,” drawing on Rosenberg and Mackintosh’s distinction between “weak equivalence” and “strong equivalence,” as a more robust framework [46]. While weak equivalence captures behavioral-level similarities between human and AI components (input–output correspondence), strong equivalence denotes deeper computational and architectural integration, where cognitive processes become genuinely intertwined rather than merely parallel.

Weak equivalence: This describes a similar, behavior-level equivalence that exists when a human–machine system produces outputs comparable to those of a non-hybrid system. However, this form of compatibility is aggregative rather than integrative. The cognitive processes of the human and the AI, while working in conjunction, remain fundamentally separate and operate in parallel.

Strong equivalence: This describes a deep coupling at the level of functions and computations. Under this condition, the respective cognitive processes are no longer discrete but are dynamically integrated via continuous, stable, and bidirectional feedback loops. This integration results in the emergence of a single, unified, and higher-order cognitive system, whose properties may not be reducible to the sum of its constituent parts.

This multi-level concept of equivalence allows us to acknowledge AI’s functional contributions while recognizing the ontological differences between AI and human intelligence. For example, in medical diagnoses, AI may be trained on vast amounts of data to identify lesions and provide diagnostic recommendations that closely match those of human doctors (i.e., weak equivalence). However, AI cannot understand a patient’s emotions, background, or ethical dilemmas as a human doctor can, nor can it make intuitive judgments without data support (lacking strong equivalence). This nuanced distinction is crucial for building reliable and meaningful human–AI collaborative relationships.

Mere equivalence in input–output behavior is an insufficient criterion for cognitive constitution. Thus, not all functional contributions qualify as constitutive. A component warrants this status only if it is inseparably integrated into the system’s computational architecture and is indispensable to its information-processing capacities. This stricter standard is essential for policing the system’s boundaries and avoiding the problem of “cognitive inflation,” a point I will develop later.

6. The Characteristics and Challenges of Emergent Properties

Next, I will briefly discuss the specific characteristics of the “emergent properties” in HMHs brought about by this deep integration, as well as how we can understand and analyze these properties and the challenges we might face.

The agency of a hybrid system results from deep functional coupling, which produces emergent properties at the system level that cannot be reduced to individual human or machine components. These emergent behaviors—such as unpredictable decision patterns or group polarization—are nonlinear phenomena arising from complex interactions. While this perspective helps explain observed phenomena in collective decision-making, it also raises challenges. For example, emergent dynamics can be unpredictable or difficult to control, prompting concerns about the system’s reliability and safety. Theiner describes this as the emergence of “novelty” [22], showing that collective outcomes can be entirely unexpected. To this point, we can broadly characterize emergent properties by their unpredictability, irreducibility, and potential for novel, often unanticipated outcomes.

However, at the same time, the emergent phenomena in human–machine hybrids (HMHs) present significant challenges. One key challenge stems from the risk of biased amplification. If human–machine interactions are not properly supervised, AI biases can reinforce human cognitive biases, leading to more extreme collective decisions. For instance, consider a company’s hiring committee using AI to screen resumes. If the AI’s training data contain historical biases, then it might systematically rate resumes from certain demographic groups lower. Influenced by “automation bias,” human committee members could then stop critically evaluating the AI’s recommendations, resulting in more discriminatory outcomes than individual human judgment alone. This illustrates how novel, unforeseen, and often undesirable outcomes can emerge from complex interactions.

Another challenge is rooted in the inherent unpredictability of these emergent dynamics, which naturally raises doubts about the system’s reliability and safety. Recent empirical work on LLM-based collective decision-making demonstrates how AI systems can exhibit unexpected voting patterns that diverge from individual human choices, further illustrating the complexity of emergent behaviors in HMHs (Yang et al. 2024 [47]). While AI excels at processing vast data and identifying subtle patterns, this unpredictability becomes particularly salient in novel or ambiguous cases where unexpected outcomes can arise. For instance, in a medical diagnosis system, atypical patient symptoms could lead the AI to suggest highly unusual diagnoses, causing confusion for human experts or even leading to misdiagnosis.

However, these operational difficulties, while significant, do not constitute fundamental obstacles to realizing genuine collective agency in human–machine hybrid systems. From the functionalist framework developed in this paper, first, regarding bias amplification, to reply, we can highlight that such biases represent genuine emergent properties arising from the hybrid system’s integrated decision-making architecture. This recognition fundamentally reframes our approach to mitigation; rather than attempting to isolate and correct individual component failures, we must adopt a systemic intervention strategy that addresses the interactive dynamics and feedback loops constitutive of the hybrid’s cognitive integration. As I have argued before, effective remediation requires the architectural redesign of the entire coupling mechanism, not simply the debugging of isolated modules. Second, for the problem of unpredictability, it is important to distinguish it from randomness or epistemic opacity because what appears as “unpredictability” at the micro-level actually manifests as analyzable patterns at the macro-level. This distinction is crucial; while we cannot trace every causal pathway through the complex web of human–AI interactions, the system’s aggregate behavior exhibits stable probabilistic tendencies that can be empirically studied and theoretically modeled (List and Pivato 2015 [48]). For instance, a medical diagnostic HMH might demonstrate, through rigorous empirical validation, a 99.5% diagnostic accuracy rate, with 0.4% false negatives and 0.1% false positives. This probabilistic characterization reveals that apparent “unpredictability” actually constitutes an objective, analyzable property of the system’s emergent agency, one that supervenes on, but cannot be reduced to, the deterministic operations of its constituent elements.

Moreover, the very architectural features that give rise to these emergent challenges also enable unique adaptive capabilities. The dynamic integration of human and artificial components in HMHs fosters a flexible environment where inter-module interactions adapt to task complexity. For simpler tasks, fewer modules are engaged, reducing the potential for error. For more complex tasks, increased collaboration among modules enhances overall performance. This inherent flexibility improves both adaptability and fault tolerance, allowing other modules to compensate for failures. This is a unique advantage of HMHs over more rigid, purely human or artificial systems.

To systematically describe such phenomena, Carroll and Parola (2024) classify the agency of human–AI hybrid systems as Type-2 nonlocal emergence [49]. In this view, the system’s overall goals, beliefs, and decision patterns emerge from human–AI interactions and are distributed across the entire network, rather than being localized in any single component. Collective intentions thus become global properties of the interaction network. For Carroll and Parola, the validity of macro-level theories depends on their ability to “predict accurately after abstracting away micro-level details.” Such macro-structures are best understood as “real patterns” that persist despite the loss of micro-level information.

Building on this understanding of distributed and emergent agency, Menary (2009 [50], p. 41) provided a complementary methodological emphasis. He argued that the real issue is the form these activities take and how they produce their effects. This activity-centered perspective proves invaluable because it allows us to bypass metaphysical debates over the “mark of the mental”, which is a critical move given the non-conscious nature of AI components. Instead, it directs focus toward how the hybrid system functions in practice, recognizing that collective agency emerges dynamically from interactive processes rather than from static, localized mental states. This approach thus clarifies how human–AI systems can genuinely function as integrated agents by examining their operational dynamics.

In sum, while the emergent properties of human–machine hybrid systems (HMHs) offer substantial potential for enhancing decision-making and accomplishing complex tasks, they also introduce significant challenges related to unpredictability, control, system integration, and explainability. However, the functionalist perspective has not been entirely invalidated. Crucially, the dynamic integration within HMHs fosters adaptive flexibility and fault tolerance, enabling robust performance across varying task complexities.

7. Two Major Critiques

The functional integration theory proposed in this paper offers a new perspective on understanding human–machine hybridity and agency. However, it also faces significant challenges. This section focuses on the following two core criticisms: the boundary problem of “cognitive bloat” and the issue stemming from the irreducibility of human practical rationality.

7.1. Cognitive Bloat

The primary challenge here is the risk of “cognitive bloat.” This critique argues that, if we judge a collective cognitive agent solely by its functional contribution, then its boundaries become arbitrarily broad, making it impossible to distinguish between a constitutive cognitive partner and a transient external tool4.

Consider the following scenario: A scholar is writing a complex philosophical paper. During this process, she frequently uses internet search engines like Google and online encyclopedias such as Wikipedia to gather, verify, and support her arguments. In this context, the vast amount of online data and the algorithms behind search engines significantly contribute to her research, reasoning, and writing tasks. If we accept “functional contribution” as the sole criterion for defining cognitive boundaries, then we must conclude that her mind, in these moments, temporarily extends to include the servers of Google or Wikipedia. Her beliefs would, to some extent, be constituted by external hardware she has never seen or directly controlled. This conclusion is clearly absurd. It conflates a mere instrumental aid—a tool used during research—with true cognitive integration.

Rupert (2004 [51], pp. 402–403) offers a similar warning with his classic phonebook example, illustrating how excessive extension beyond natural cognition contradicts intuition. His critique strikes at the core of “cognitive bloat”, which states that an overly broad standard based solely on outcome equivalence can inflate the boundaries of the mind to include any external source of information. This results in a vague, unhelpful. He also emphasizes that internal and external cognitive processes differ fundamentally in their causal and functional characteristics. For example, internal memory systems exhibit phenomena such as interference effects—where old memories impede new ones—and generation effects, where self-produced information is remembered better than passively received data. Conversely, an external tool like a notebook lacks these dynamic features; recording new information in a notebook does not interfere with previous entries. Rupert convincingly argues that, because internal and external processes follow different causal laws, treating them as the same “cognitive kind” lacks scientific validity. Such a classification risks being “vacuous and nearly void of explanatory power” ([51], p. 407). However, when we replace the notion of a “notebook” with an AI system or intelligent robot, Rupert’s critique loses its core force. An integrated AI is not merely a passive external storage device but an active, dynamically complex computational system. Human commands and feedback continuously train and reshape the AI, while the AI’s analysis and predictions in real time influence human cognition and decision-making (Theiner 2017 [22], pp. 13–14).

Thus, classical critiques like Rupert’s do not weaken the argument but, in fact, highlight its necessity. They reveal that the root of “cognitive bloat” lies in a too-broad, outcome-based equivalence principle, an approach that inevitably leads to an ever-expanding boundary of the mind. To address this, we need to establish a more stringent criterion for what qualifies as genuine cognitive integration. Specifically, a component should be recognized as a cognitive partner only if it is deeply embedded within the system’s computational framework and plays a vital role in its information-processing capabilities. This stringent criterion aligns with the integration standards I proposed earlier and is crucial for maintaining the integrity of the system’s boundaries and preventing the overextension of cognitive attributions.

7.2. The Impossibility of Mutual Understanding

The second critique of FHCA stems from phenomenological and hermeneutic traditions, asserting that human practical reason—our capacity to act based on values, context, and lived experience—cannot be reduced to algorithms. Critics argue that genuine collaboration requires what Schmid calls intersubjective “we-ness,” emerging through embodied encounters and emotional resonance (Schmid 2013 [52]; 2014 [53]). From this perspective, functionalist accounts fatally neglect the experiential dimensions essential to authentic cooperation.

The hermeneutic tradition sharpens this critique. Following Gadamer’s notion of “fusion of horizons,” understanding happens when different experiential perspectives genuinely meet and transform each other [11]. However, AI seems stuck with a fixed algorithmic horizon; it processes according to preset functions. Consider the following example of judicial sentencing: while AI can process legal precedents and calculate risk assessments with remarkable sophistication, critics argue that it cannot grasp the phenomenological weight of a defendant’s remorse or engage in the kind of moral deliberation that requires wrestling with competing values and contextual nuances.

This juridical example, while compelling at first glance, reveals three problematic assumptions in the phenomenological critique. First, it romanticizes human understanding as direct phenomenological insight, overlooking that judgments about sincere remorse involve inference, heuristics, and unconscious biases, which are cognitive processes increasingly amenable to computational analysis. Second, it caricatures AI as passive data processing, ignoring sophisticated capabilities in evidence synthesis, bias correction, and contextual reasoning (Arrieta et al. 2019 [54]; Tolmeijer et al. 2022 [55]). Most fundamentally, it assumes that understanding necessarily requires access to internal mental states, which is an internalist premise that the contemporary philosophy of action increasingly questions (Hasselberger 2014 [56], p. 135).Despite these considerations, phenomenological critics maintain that the algorithmic and experiential realms remain fundamentally incommensurable, suggesting that human–AI interaction is inevitably confined to a subject–tool relationship rather than achieving genuine intersubjective engagement. Yet, this conclusion, as we shall see, rests on a category error about what collective agency actually requires.

This critique conflates the following two distinct phenomena: how we understand each other (through empathy, emotional attunement, what he calls “shared second-person perspective”) and actual collective consciousness. This distinction proves decisive because it reveals that collective agency can emerge from stable patterns of functional integration and mutual adaptation without requiring the kind of deep intersubjective understanding that the phenomenological critique demands. To further clarify this point, if collective agency can emerge from stable patterns of practical coordination even among conscious subjects who maintain distinct phenomenological perspectives, then the absence of AI consciousness may not constitute the fundamental barrier critics suggest. What matters for collective agency is not whether participants share phenomenological states, but whether they can establish reliable patterns of mutual responsiveness that enable coordinated goal-directed behavior. Human–AI systems can potentially achieve this through functional integration—developing stable, adaptive patterns of interaction—without requiring the kind of phenomenological commonality that the critique presupposes as necessary.

Consider how this works in practice. In the case of radiologist–AI collaboration in tumor detection, the functional integration achieved does not aspire to phenomenological convergence. Rather, through iterative interaction, both components develop mutual responsiveness, as the radiologist learns to interpret AI confidence levels and error patterns, while the AI refines its processing through feedback mechanisms. This reciprocal adaptation constitutes a form of practical understanding fundamentally different from, yet no less effective than, phenomenological understanding.

In conclusion, the phenomenological critique, while raising important concerns about the nature of understanding, ultimately misidentifies the challenge facing human–AI collective agency. The question is not whether AI can replicate or access human practical reason in its full phenomenological richness, but whether human–AI integration can generate new forms of collective rational agency that preserve what is distinctively human—our evaluative capacities, contextual sensitivity, and normative judgment—while transcending individual cognitive limitations. For FHCA, the irreducibility of human practical reason, rather than serving as an insurmountable barrier, becomes a crucial component that, when properly integrated with AI’s computational capabilities, enables forms of collective agency that neither humans nor AI could achieve in isolation.

8. Conclusions

The central aim of this paper has been to explore whether human–machine hybrid systems (HMHs) can form a novel type of collective agency and to develop a theoretical framework to support this claim. Traditional theories of collective action, which depend on shared consciousness and commitments among people, do not work well for HMHs that include non-conscious AI parts. To fix this, I suggest the following three key criteria for cognitive integration in HMHs: goal alignment, functional complementarity, and stable interactivity. When these criteria are met, HMHs can show real collective intentionality without needing conscious awareness. I also introduce a detailed idea of “equivalence,” differentiating between “weak equivalence” in behavior and “strong equivalence” in functions. Only when HMHs reach “strong equivalence”—with human and AI parts closely linked through stable feedback loops—can we have a Functional Hybrid Collective Agent (FHCA). In an FHCA, collective intentionality appears as a system-wide feature that is not just the sum of individual human or AI actions.

I tackle two main criticisms of functionalism. First, by telling apart functional contributions that are just tools from those that are essential to the system, I avoid the problem of cognitive bloat and keep the cognitive boundary clear. Second, even though AI does not have human-like experiences, it can still be part of collective decision-making through functional integration and aligned goals, creating a new type of collective agency. Even though I propose a new form of collective agency in HMHs, there are still limits and open questions. For example, making sure HMHs meet the criteria for cognitive integration, accurately measuring “strong equivalence,” and dealing with unpredictable and hard-to-control properties are big challenges. Additionally, how much AI can truly understand human practical rationality and share understanding with humans is still up for discussion.