What Can Game Theory Tell Us about an AI ‘Theory of Mind’?

Abstract

Game theory includes a rich source of methods for analysing strategic interactions where there are a small number of agents, each having only a few choices. In more complex settings though, where there are many choices over indefinite time horizons involving large social groups, these methods are unlikely to fully capture the causes of agent behaviour. If agents are able to simplify the task of understanding what others might do by modelling the constraints of others, particularly unobservable cognitive constraints, then the possible behavioural outcomes can be similarly restricted, thereby reducing the complexity of a social interaction. Having a cognitive representation of the unobserved causal states of others is an aspect of a ‘Theory of Mind’ and it plays a central role in the psychology of social interactions. In this article I examine a selection of results on the theory of mind and connect these with the ‘game theory of mind’ to draw conclusions regarding the complexity of one-on-one and large-scale social coordination. To make this explicit, I will illustrate the relationship between the two psychological terms ‘introspection’ and ‘theory of mind’ and the economic analysis of game theory, while retaining as much as possible of the richness of the psychological concepts. It will be shown that game theory plays an important role in modelling interpersonal relationships for both biological and artificial agents, but it is not yet the whole story, and some psychological refinements to game theory are discussed.

1. Introduction

The coordination between groups of artificial agents and humans will become one of the most challenging tasks for improving AI’s ability to support human endeavours [1,2]. In order to do this, we first need to understand how intelligent agents make choices in the context of other intelligent agents, a complex task for which humans have specific and highly developed cognitive processes that are detectable even in early childhood [3]. For example, in-group social dynamics are caused by people collectively orientating themselves towards ‘like minded’ others in social group formation (otherwise known as intra-group homophily) [4]. This reduces the cognitive load of having to try and understand everyone else in the group by assuming that everyone, to some reasonable approximation, is like everyone else. It also plays an important role in threat assessment: knowing the causal constraints of other agents, or collections of agents, helps to narrow the range of possible threats they pose, such antagonistic inter-group heterogeneity with intra-group homogeneity is often the source of the echo-chamber effect in social media, as shown by Colleoni et al. [5]. Notably, this study also showed that the individual network structures of different homophylic groups can be very different, suggesting that there is something in the specifics of the homophilous connections that influences social network topology (cf. Section 3). With both of these elements there is a need to better understand the mechanisms of both cooperation for achieving joint goals and competition for analysing strategic threats.

In a certain sense, two-agent games contain the basic foundations for experimental and theoretical studies into social interactions as well as forming the simplest social network containing two nodes (players or agents), with a bi-directional link between the nodes indicating a relationship exists from one node to the other. Despite its simplicity, this arrangement has provided a wealth of insights into the nature of collective behaviour of not just humans but also, for example, the behaviour of other species and how genes interact with one another. However, recent new developments have motivated novel analyses in game theory, three of which are discussed in this opinion piece: theory of mind (ToM), social network theory, and how game theory can connect these to develop new ideas in artificial intelligence (AI). In order to situate game theory in a psychological framework, a brief discussion of the meaning and significance of individual cognition and interpersonal cognition is introduced next before moving to the main topic of this article: the psychology of a ‘Game Theory of Mind’.

1.1. Individual Cognition

On the left in Figure 1 is Barrett’s ([6], Figure 1) analysis of recent research in cross-cultural cognitive sciences, illustrating the relationships between fields of study in the cognitive sciences. Individual cognition has also been studied recently by Peterson et al. [7] in the context of gambling and uncertainty. Some of the models they analysed are on the right of Figure 1 (simplified from Figure 1c in [7]). Peterson et al showed that well-known computational theories, such as expected value theory, prospect theory, and many others, are empirically sound using very large databases of people’s decisions. These results tell us not only what is necessary to accurately represent individual cognitive processes, but also how sophisticated a biological agent’s cognitive model of others needs to be in order to, at least partially, understand their individual cognitive processes.

1.2. Social Constraints

At the highest level of Barrett’s analysis is social cognition, which includes the elements of moral, cultural, and religious norms. In another recent result using large data analysis, Awad et al. [8] developed a questionnaire for what they called the Moral Machine in which they gathered 40 million decisions from participants from 233 countries. What they found is significant cross-cultural ethical variation with three large clusters of countries and that these differences correlate with institutions and cultural traits.

From a certain perspective, it may not be surprising that people with different backgrounds are different from one another, but this would miss two insights: (1) it is possible to quantify these differences; (2) that these differences have a measurable impact on choices in ambiguous scenarios. We can think of these large-scale patterns of behaviour as providing constraints on the decisions people make at the individual level when there is no strict logic, that is to say that knowing where someone comes from reduces the ambiguity another agent would have regarding their possible actions.

1.3. Theory of Mind and Introspection

A person is said to have a ToM if they are able to impute mental states, such as desires, intentions, and beliefs, to another person—see, for example, Frith and Frith [9] for a short summary. The emergence of a ToM is a key milestone in the interpersonal development of children [10], and the failure for it to develop is associated with autism, which, as a diagnosis, is in part characterised by a difficulty in understanding social contexts [11].

Introspection, on the other hand, is the facility to understand one’s own thoughts, for example to have a representation of our own thought processes. Recent work in this area has combined introspection with brain-mapping [12], and for an early historical review see Boring [13]. Introspection is also closely related to ToM, they are both central to early childhood development and there is an ongoing debate regarding how introspection and ToM interact with each other during early development; see, for example, Gonzales et al. [14] for a recent discussion. Figure 2 illustrates the relationship between introspection and ToM, and it will be shown that when they interact within a single agent the results have surprising complexities.

What follows in this article is an indication of the complexity of the relationship between certain psychological phenomena and their possible implementation in a working, socially adept AI where game theory underpins the theoretical aspects. This goes some way to addressing the concerns Shevlin and Halina [17] have in which ‘rich psychological terms’ should not misused. In what follows are two main sections. The first expands on introspection and ToM in a way that makes some of their key features operational in an AI, while illustrating the relationship with game theory. The second then discusses how this is related to building and maintaining complex social networks. In the final section I discuss areas for future work.

2. The ‘Game Theory of Mind’: Neuroscience and Economics in Strategic Interactions

The game theory of mind (GToM) model was introduced in a 2008 paper by Yoshida et al. [18] to describe how people represent the causal cognitive states of others in order to optimise interpersonal interactions. The premise is based on cooperating or defecting in a hunting game: an agent can hunt collectively with other agents, i.e., be socially cooperative, and earn a large reward by killing a large animal, or they can hunt as an individual, i.e., defect from the group, and earn a smaller reward. The goal is to develop a formal description, based on empirical evidence from human fMRI data, of “…a model of ‘theory of mind’ using ‘recursive sophistication’ in which my model of your goals includes a model of your model of my goals, and so on ad infinitum”. The people in the experiment played against a computer with two different strategies: a lower order competitive strategy and a higher order cooperative strategy, and the premise is that game theory can elucidate how agents build internal representations of strategic levels of social interactions from observations of their opponent’s behaviour. Experiments such as this have been extended beyond behavioural models to directly studying the behavioural and neural recordings in monkeys [19,20,21,22,23,24] and humans [25,26,27,28,29,30]. These and many other studies have significantly contributed to our understanding of how we attribute value to our decisions, where this is processed in the brain, and how our allocation of value to the decisions of others interacts with the value we allocate to our own decisions.

In order to accommodate these facets, research such as that of Griessinger and Coricelli [31] and Yoshida et al. [32] have posited models that allow for varying degrees (depth) of strategic reasoning. For example in Griessinger and Coricelli [31] they modelled reinforcement learning (level 0 strategic reasoning), fictitious play learning (level 1 strategic reasoning), and influence learning (level 2+ strategic reasoning). Much of this work has used the notion of strategic depth, the extent to which one agent reasons about the other agent reasoning about them, etc., as illustrated in Figure 3 (bottom, A.), which does indeed have elements of ToM. There is very little discussion of the relationship between ToM, introspection, and strategy though, arguably because it is not clear where introspection sits in an analysis of games despite its distinct psychological role independent of ToM. See, for example, Goeree and Holt [33], where introspection corresponds to ToM as Yoshida et al. [18] define it. Figure 3 shows where this distinction becomes important, not only does an agent $A_1$ need to be aware of $A_2$’s reasoning about $A_1$ (top row, A.), but $A_1$’s representation of the strategic interaction needs to include a representation of their own reasoning about the situation (top row, B.). With a model of their own cognitive abilities, $A_1$ can understand, for example, its limited understanding of $A_2$. The second row of Figure 3 shows the next step in deepening this strategic thinking.

To make the case more explicit, these concepts can be re-framed in terms related to game theory. In the work of Yoshida et al. [18,32], they modelled levels of cognitive reasoning but they also connect it, in a suitable limit, to the quantal response equilibrium (QRE) of McKelvey and Palfrey [34]. It is important to note that the QRE is explicitly an equilibrium model, and it contains within its limits the conventional notion of a Nash equilibrium, but we can also use it to illustrate how ToM relates to a specific GToM for two-player games, as follows. Given two players a and b each with a binary strategy set $S_i=\{s_i^1,s_i^2\}$, $i\in \{a,b\}$, the QRE can be represented as a self-consistent mapping ($-i$ denotes i’s opponent). We supplement the usual treatment of this model by assuming that i only possesses an approximate representation of $-i$’s decision model and subsequent behaviour, and we denote this internal representation with a prime on those terms that are internal to i rather than ‘true’ representations of the other agent:

$$\begin{array}{ccc}\hfill {\overline{s}}_i=p(s_i^1|{\overline{s}}_{-i}^{\prime })& \propto & exp({\beta }_{i}E(u_i|{\overline{s}}_{-i}^{\prime }))\hfill \end{array}$$

(1)

$$\begin{array}{ccc}\hfill {\overline{s}}_i& =& {\psi }_i({\overline{s}}_{-i}^{\prime }|c_i)\hfill \end{array}$$

(2)

$$\begin{array}{ccc}\hfill {\overline{s}}_i& =& {\psi }_i({\psi }_{-i}^{\prime }({\overline{s}}_i|c_{-i})|c_i)\hfill \end{array}$$

(3)

In the case where ${\overline{s}}_{-i}^{\prime }={\overline{s}}_{-i}$ and ${\psi }_{-i}^{\prime }({\overline{s}}_i|c_{-i})={\psi }_{-i}({\overline{s}}_i|c_{-i})$ Equation (3) defines the original QRE model [34]. In Equation (1) ${\overline{s}}_i\in [0,1]$ is the probability of player i playing strategy 1, $p(s_i^j|{\overline{s}}_{-i}^{\prime })$ is the probability of player i choosing strategy j subject to agent i’s approximation of $-i$’s choosing strategy 1, $E(u_i^j|{\overline{s}}_{-i}^{\prime })$ is the expected utility to i for playing j subject to $-i$’s probability of play, and ${\beta }_i$ is the rationality, or ‘persona’ [35], of agent i. There are also payoff matrix parameters (encoded in the expected utility function) and these, along with the ${\beta }_i$, are collected in the co-vector of i’s parameters denoted $c_i$. In Equation (3) we simply express the notion that i’s probability is a function of $-i$’s probability. Also note that this logit form of noisy decision-making is the game-theoretical analogue of the expected utility function Peterson et al. [7] used for individual cognition, discussed in the Introduction above.

While there is nothing novel about this representation, as the QRE has been used extensively as a model of bounded rationality [36,37,38,39,40], there are nuances that are worth making explicit. First note that Equation (2) has only a behavioural interpretation of the other player. Viewing ${\overline{s}}_{-i}$ as the behaviour of an inanimate object, the agent learns about $S_{-i}^{\prime }$ but it is limited in that ${\overline{s}}_{-i}^{\prime }$ does not have any sophisticated internal causal structure. In this interpretation, the agent only learns through observation the probabilities over possible outcomes and then, based on the rewards they receive, constructs a probability for their own choices: ${\overline{s}}_i$. However, once an agent learns that ${\overline{s}}_{-i}$ might be similar to themselves and has some form of internal structure of its own then Equation (3) illustrates how an agent can use this representation to modulate their own decision-making based on the other agent’s constraints $c_{-i}$.

Now suppose that i has an internal representation of the strategic interaction that i is involved in, i.e., Equations (1)–(3), but the internal representation is only an approximate representation, so we denote each term with a prime: ${\overline{s}}_i^{\prime }$, ${\psi }_i^{\prime }(·)$, and ${\psi }_{-i}^{\prime }(·)$, ⟵ denotes the direction of causation i has learned in constructing this representation, and ⟷ denotes a correspondence between two cognitive representations. The internal representation i has of the strategic interaction can be written in the following ways:

$${\Gamma }^i:\left\{\begin{array}{cc}{\overline{s}}_i^{\prime }\hfill & ⟵{\psi }_i^{\prime }({\psi }_{-i}^{\prime }({\overline{s}}_i^{\prime }|c_{-i})|c_i)\hfill \\ {{\psi }_i^{\prime }}^{-1}({\overline{s}}_i^{\prime }|c_i)\hfill & ⟷{\psi }_{-i}^{\prime }({\overline{s}}_i^{\prime }|c_{-i})\hfill \end{array}\right.$$

(4)

and ${\Gamma }^i$ is $A_i$’s encoding, in either an artificial or a biological neural network. In the first line of Equation (4) is the standard representation of the QRE. The second line is the correspondence between introspection on the left, given by the inverse of $A_i$’s representation of themselves: ${{\psi }_i}^{-1}(·)$, and $A_i$’s ToM representation of $A_{-i}$: ${\psi }_{-i}(·)$. Schematically, line one of Equation (4) is illustrated in Figure 4A. (introspection, the left-hand representation in $A_1$’s thought bubble where $A_1$ is thinking of $A_2$’s strategy) and line two is illustrated in Figure 4B. (the correspondence between ToM and introspection). Line two is an algebraic manipulation that makes explicit that $A_i$ has two distinct models, one of itself and the other of $A_i$. In practice ${{\psi }_{i^{\prime }}}^{-1}(·)$ may not be the inverse of the QRE as it can be any encoding that $A_i$ has of its own individual or interpersonal cognition (c.f. Figure 1), but this representation is in an equilibrium correspondence with $A_i$’s encoding of $A_{-i}$ in models such as the QRE.

Aspects of this formal approach have been developed into experiments in which the results support a strategic role for our ToM. In a 2004 study, Rilling et al. [41] used the Prisoner’s Dilemma (PD) and the Ultimatum Game (UG) in an fMRI study of the parts of the brain that were active in two different types of strategic interaction and if these regions corresponded to the known neural network that is activated during other ToM tasks. In both games, regions of the brain associated with a ToM were activated when humans were told they were playing both another human and a computer, but in the case of humans the activation was much stronger, suggesting that we differentiate our thinking of our opponents depending on what we know of their possible internal states, where the authors concluded: ‘The fact that computer partners are able to activate this network, albeit to a lesser extent than human partners, suggests that either this neural system can also be activated by reasoning about the unobservable states of nonhuman systems, or that participants imbue their computer partners with human attributes’. There were further differences between the neural activity of the test subjects that separated the UG from the PD, and this no doubt also points to subtle cognitive differences in how humans respond to strategic interactions based on the different hidden states of either their computer or human opponent. However, the final point I would like to make is that the QRE model, specifically its ‘persona’ version [35] in which the covector of parameters encodes the persona of the players, goes a significant way to explaining the differences between experimental results and the Nash equilibrium of the UG [42]. The interpretation of this result is that, in order to avoid subsequent punishment for an unfair split in this game, the agent who splits the value needs to take into account how likely the other agent is to punish them for an unfair split, and this pre-emptive accounting of future punishments is a type of understanding of the internal constraints the receiver has that the splitter needs to account for. These results have subsequently been borne out in subsequent studies as well [43,44,45].

3. The Importance of a Theory of Mind in Human-to-Human Interactions

In the field of anthropology there is a well-studied relationship between the surface area of the human brain and the size of our largest stable social grouping, approximately 150 individuals. This number, called Dunbar’s number [46,47], is also known to have a number of highly stratified layers within this grouping that cluster around mean values of ≃1.5, 5, 15, 35, and 150 people (note that there are multiple views on the exact numbers) [47,48,49]. These numerical values are related to the variety of cognitive adaptations people use to maintain long-term stable relationships, such as pair bonding, intimate friendships, casual acquaintances, and more distant associations. In particular, these numbers are a consequence of the topology of the social networks humans participate in, and in this case these are of the Erdös–Rényi variety of networks [49]. This topology is driven by people forming discrete links between one another that subsequently constrains the network topology. Notably, humans have the largest social network of any other comparable species [46], and at the same time humans have the largest ratio of neocortex size to total brain volume [50], and it is from this relationship that the Social Brain Hypothesis was developed to explain why the human brain has been so successful in manipulating the natural environment: it is due to human’s ability to develop and maintain complex social interactions that in turn leads to sophisticated solutions to tasks in the natural environment.

The inner layers of our social networks (called the support clique and the sympathy group) have been shown to be mediated by our ability to form social connections based on our capacity to use ToM and memory in order to form connections between individuals. In an MRI study, Powell et al. [51] showed that the volume of the orbital pre-frontal cortex positively correlates with the size of a subjects’ inner social network layers. This built on the earlier work of Stller and Dunbar [52] who showed that memory helps explain the sympathy group (the social layer of approximately 12–15 individuals) and perspective taking, i.e., ToM, helps explain the support clique of approximately 4–7 group members. As a final example, in the study by Lewis et al. [53] it was shown that an individual’s social network size is predicated upon the volume of neural material dedicated to specific types of mentalising.

The cognitive complexity of maintaining these networks is illustrated in Figure 5, each of the links $L_{1,i}$ that $A_1$ forms in the real world needs to have a cognitive representation $T{L}_{1,i}$ [16] and these links are not usually independent of one another. As mentioned above we also have specific strategies for managing these complexities and, to the extent that we will need socially adept AI, the next generation of AI agents will also need to have better internal representations of their human partners, how these partners see them, and how this influences collective behaviours.

4. Conclusions

Introspection and Theory of Mind are psychologically distinct, complex notions that are tightly inter-related, both developmentally and in later life. In this article I have used these two concepts to illustrate how a more precise use of psychological ideas can lead to a more nuanced understanding of strategic interactions, to look more closely ‘under the hood’ of cognition [54]. For example, a key point of this article is the distinction between how an agent makes a decision: ${\psi }_i(·)$ versus how this is cognitively encoded by an agent: ${\psi }_i^{\prime }(·)$. The first is a ‘true’ representation of the agent’s decision-making process, the second is an approximation of themselves an agent has learned. In contrast, many ‘game theory of mind’ approaches take the strategic levels beginning with a true representation of $A_i$ thinking of $A_{-i}$ who is thinking of $A_i$’s strategy i.e., ${\psi }_i({\psi }_{-i}^{\prime }(s_i))$. This takes the first level, ${\psi }_i(·)$, to be how $A_i$ makes a decision, and this then contains a representation of agent $A_{-i}$’s decision-making process: ${\psi }_{-i}^{\prime }(·)$. In this article, an agent can reason about their own cognition via introspection or about another agent’s cognition via ToM. The former is ${\psi }_i({\psi }_i^{\prime }(·))$ and the latter is ${\psi }_i({\psi }_{-i}^{\prime }(·))$; this is the distinction made in Figure 2. The consequence is an awareness of the relationship between ‘self’ and ‘other’, the representations being manipulated by the agent, and that these representations are consistent with respect to one another in an equilibrium models such as the QRE. This allows an agent to have greater flexibility when thinking of themselves in relation to others.

A further consequence is that if the equilibrium assumption is relaxed then encodings do not need to be related to one another (c.f. Figure 4A), which can be represented as: $T{A}_i[{\psi }_i^{\prime }(·),{\psi }_{-i}^{\prime }(·)]$ where $T{A}_i$ is the cognitive space of $A_i$, as described in the caption to Figure 2. These considerations bring to the fore three different notions of equilibrium: agents that are in equilibrium as in the conventional QRE (co-vectors are dropped to reduce notational clutter)—${\psi }_i({\psi }_{-i}(s_i))=s_i$; an agent who is using ToM as they think of another agent—${\psi }_i({\psi }_{-i}^{\prime }(s_i))=s_i$; and an agent that has has adapted both a ToM and introspection to interpret the interaction—$T{A}_i[{\psi }_i^{\prime }({\psi }_{-i}^{\prime }(s_i^{\prime }))=s_i^{\prime }]$, and so providing a clearer understanding of the different psychological states of the agents that each implies. Each ${\psi }_i(·)$ can also represent many different types of individual cognition (c.f. Figure 1), for example the seven described in Ert et al. [55] or the dozens listed in Peterson et al. [7]. The importance for the current article is that the models themselves are the constraints, they provide testable hypotheses for the limits of human cognition, how we represent ourselves and others, and how we can take advantage of these limits to understand and predict the behaviour of one another.

To make this specific in a realistically complex setting, consider the game of Go that the AI AlphaGo was so successful at playing [56]. The number of moves available at each stage of the game ranges from a relatively small number towards the end of the game to theoretically hundreds at the beginning of the game that branch out into an incomprehensibly large move space, so how do people play games like this? Clearly not by examining all, or even a representative sample of all, the moves available to them, and likewise their (human) opponent cannot examine a representative sample of moves. Of course, an enormous amount of experience and task-specific learning is involved, but ultimately people use a relatively small number of initial positions to investigate where to move next and then they explore a relatively narrow range of future plays from that small number of starting points, and this works because both players are subject to the same (cognitive) limitations. This is a common to all game experts and consequently there is not a great deal of variation in the number of starting positions considered between chess masters and intermediate players [57]. Not surprisingly, where they differ is in the quality of the moves they consider or the style of their play. While this is a complicated topic, the recent articles by Ericsson [58] and Gobet and Charness [59] review the key developments in the history of the psychology of expertise and how performance relates to perception, a topic I explored previously using an artificial neural network to model perceptual expertise in Go [60,61].

It is only through an interdisciplinary understanding of how the many different fields have approached the problem of coordinated social behaviour, from economics, to neuroscience, to psychology, and in AI, that we can begin to model and implement these intrinsically human-like aspects with the fidelity necessary to take advantage of the benefits AI can afford us.