Deep Hybrid Models: Infer and Plan in a Dynamic World

Abstract

To determine an optimal plan for complex tasks, one often deals with dynamic and hierarchical relationships between several entities. Traditionally, such problems are tackled with optimal control, which relies on the optimization of cost functions; instead, a recent biologically motivated proposal casts planning and control as an inference process. Active inference assumes that action and perception are two complementary aspects of life whereby the role of the former is to fulfill the predictions inferred by the latter. Here, we present an active inference approach that exploits discrete and continuous processing, based on three features: the representation of potential body configurations in relation to the objects of interest; the use of hierarchical relationships that enable the agent to easily interpret and flexibly expand its body schema for tool use; the definition of potential trajectories related to the agent’s intentions, used to infer and plan with dynamic elements at different temporal scales. We evaluate this deep hybrid model on a habitual task: reaching a moving object after having picked a moving tool. We show that the model can tackle the presented task under different conditions. This study extends past work on planning as inference and advances an alternative direction to optimal control.

1. Introduction

Imagine a baseball player striking a ball with a bat. State-of-the-art approaches to simulate such goal-directed movements in a human-like manner often rely on optimal control [1,2]. This theory rests upon the formulation of goals in terms of value functions and their optimization via cost functions. While this approach has advanced both robotics and our understanding of human motor control, its biological plausibility remains disputed [3]. Moreover, value functions for motor control might restrict the range of motions an agent can learn [4]. In contrast, complex movements like handwriting or walking can emerge naturally from generative models encoding goals as prior beliefs about the environment  [5]. The theory of active inference builds upon this premise, proposing that goal-directed behavior results from a biased internal representation of the world. This bias generates a cascade of prediction errors forcing the agent to sample those observations that make its beliefs true [6,7,8,9]. In this view, the tradeoff between exploration and exploitation arises naturally, driving the agent to minimize the uncertainty of its internal model before maximizing potential rewards [10]. Active inference shares foundational principles with predictive coding [11,12] and the Bayesian brain hypothesis, which postulates that the brain makes sense of the world by constructing a hierarchical generative model that continuously makes perceptual hypotheses and refines them [13].

This perspective offers a promising avenue for advancing current robotics and machine learning, particularly in research that frames control and planning as an inference process [14,15,16,17,18]. A distinctive feature of active inference is its ability to model the environment as a hierarchy of causes and dynamic states evolving at different timescales [19], which is fundamental for biological phenomena such as linguistic communication [20], and for advanced movements such as that of a baseball player. In the latter, several characteristics of the nervous system can be well captured by an active inference agent, providing a robust alternative to optimal control. First, the human brain is assumed to maintain a hierarchical representation of the body that generates the motor commands required to achieve the desired goal [21]. This representation must be flexible, in that the baseball player should relate the configuration of the bat to his body, acting as an extension of his hand [22]. Tool-use experiments in non-human primates revealed that parietal and motor regions rapidly adapt to incorporate tools into the body schema, enabling seamless interaction with objects [23]. Further, the player’s brain should maintain a dynamic representation of the moving ball in order to predict its trajectory before and after the hit. The posterior parietal cortex is known to encode multiple objects in parallel during action sequences, forming visuomotor representations that also account for object affordances [24]. In addition, distinct neural populations in the dorsal premotor cortex are known to encode multiple reaching options in decision-making tasks, with one population being activated and the others suppressed as the decision unfolds [25]. In short, the human brain can maintain multiple potential body configurations and potential trajectories appropriate for specific tasks.

Despite its potential, the development of deep hierarchical models within active inference remains limited. Adaptation to complex data still relies primarily on neural networks as generative models [26,27,28,29,30,31,32,33,34,35]. One study demonstrated that a deep hierarchical agent, equipped with independent dynamics functions for each degree of freedom (DoF), could continuously adjust its internal trajectories to align with prior expectations, enabling advanced control of complex kinematic chains [36]. This ability to learn and act across intermediate timescales offers significant advantages for solving control tasks. Moreover, simulating real-world scenarios can benefit from so-called hybrid or mixed models in active inference, which integrate discrete decision making with continuous motion. However, state-of-the-art applications of such models have simulated static contexts only [8,37,38,39,40].

In this work, we address these challenges from a unified perspective. Our key contributions are summarized as follows:

• We present an active inference agent affording robust planning in dynamic environments. The basic unit of this agent maintains potential trajectories, enabling a high-level discrete model to infer the state of the world and plan composite movements. The units are hierarchically combined to represent potential body configurations, incorporating object affordances (e.g., grasping a cup by the handle or with the whole hand) and their hierarchical relationships (e.g., how a tool can extend the agent’s kinematic chain).
• We introduce a modular architecture designed for tasks that involve deep hierarchical modeling, such as tool use. Its multi-input and multi-output connectivity resembles traditional neural networks and represents an initial step toward designing deep structures in active inference capable of generalizing across and learning novel tasks.
• We evaluate the agent’s performance in a common task: reaching a moving ball after reaching and picking a moving tool. The results highlight the interplay between the agent’s potential trajectories and the dynamic accumulation of sensory evidence. We demonstrate the agent’s ability to infer and plan under different conditions, such as random object positions and velocities.

2. Methods

2.1. Predictive Coding

According to predictive coding (PC), the human brain makes sense of the world by constructing an internal generative model of how hidden states of the environment generate the perceived sensations [11,12,41]. This internal model is continuously refined by minimizing the discrepancy (called prediction error) between sensations and the respective predictions. More formally, given some prior knowledge $p\left(\mathit{x}\right)$ over hidden states $\mathit{x}$ and partial evidence $p\left(\mathit{y}\right)$ over sensations $\mathit{y}$, the nervous system can find the posterior distribution of the hidden states given the sensations via Bayes rule:

$$p\left(\mathit{x}\right|\mathit{y})={\displaystyle \frac{p(\mathit{x},\mathit{y})}{p\left(\mathit{y}\right)}}$$

(1)

However, direct computation of the posterior $p\left(\mathit{x}\right|\mathit{y})$ is unfeasible since the evidence requires marginalizing over every possible outcome, i.e., $p\left(\mathit{y}\right)=\int p(\mathit{x},\mathit{y})d\mathit{x}$. Predictive coding supposes that, instead of performing exact Bayesian inference, organisms are engaged in a variational approach [42], i.e., they approximate the posterior distribution with a simpler recognition distribution $q\left(\mathit{x}\right)\approx p\left(\mathit{x}\right|\mathit{y})$, and minimize the difference between the two distributions. This difference is expressed in terms of a Kullback–Leibler (KL) divergence:

$$D_{KL}\left[q\left(\mathit{x}\right)\right|\left|p\left(\mathit{x}\right|\mathit{y})\right]={\int }_{\mathit{x}}q\left(\mathit{x}\right)ln{\displaystyle \frac{q\left(\mathit{x}\right)}{p\left(\mathit{x}\right|\mathit{y})}}d\mathit{x}$$

(2)

Given that the denominator $p\left(\mathit{x}\right|\mathit{y})$ still depends on the marginal $p\left(\mathit{y}\right)$, we express the KL divergence in terms of the log evidence and the Variational Free Energy (VFE), and minimize the latter quantity instead. The VFE is the negative of what in the machine learning community is known as the evidence lower bound or ELBO [43]:

$$\mathcal{F}=\underset{q\left(\mathit{x}\right)}{\mathbb{E}}\left[ln{\displaystyle \frac{q\left(\mathit{x}\right)}{p(\mathit{x},\mathit{y})}}\right]=\underset{q\left(\mathit{x}\right)}{\mathbb{E}}\left[ln{\displaystyle \frac{q\left(\mathit{x}\right)}{p\left(\mathit{x}\right|\mathit{y})}}\right]-lnp\left(\mathit{y}\right)$$

(3)

Since the KL divergence is always nonnegative, the VFE provides an upper bound on surprise, i.e., $\mathcal{F}\ge -lnp\left(\mathit{y}\right)$. Therefore, minimizing $\mathcal{F}$ is equivalent to minimizing the KL divergence with respect to $q\left(\mathit{x}\right)$. The more the VFE approaches 0, the closer the approximate distribution is to the real posterior, and the higher the model evidence (or, equivalently, the lower the surprise about sensory outcomes will be). The discrepancy between the two distributions also depends on the specific assumptions made over the recognition distribution; a common one is the Laplace approximation [42], which assumes Gaussian probability distributions, e.g., $q\left(\mathit{x}\right)=\mathcal{N}(\mathit{\mu },\mathbf{\Sigma })$, where $\mathit{\mu }$ represents the most plausible hypothesis—also called belief about the hidden states $\mathit{x}$—and $\mathbf{\Sigma }$ is its covariance matrix. Now, we factorize the generative model $p(\mathit{x},\mathit{y})$ into likelihood and prior terms, and we further parameterize it with some parameters $\mathit{\theta }$:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill p(\mathit{x},\mathit{y})& =p\left(\mathit{y}\right|\mathit{x},\mathit{\theta }\left)p\right(\mathit{x})\hfill \\ \hfill p\left(\mathit{y}\right|\mathit{x},\mathit{\theta })& =\mathcal{N}(\mathit{g}(\mathit{x},\mathit{\theta }),{\mathbf{\Sigma }}_y)\hfill \\ \hfill p\left(\mathit{x}\right)& =\mathcal{N}(\mathit{\eta },{\mathbf{\Sigma }}_{\eta })\hfill \end{array}\end{array}$$

(4)

where $\mathit{g}(\mathit{x},\mathit{\theta })$ is a likelihood function and $\mathit{\eta }$ is a prior. In this way, after applying the logarithm over the Gaussian terms, the VFE breaks down to the following simple formula:

$$\mathcal{F}=-{\displaystyle \frac{1}{2}}\left[{\mathbf{\Pi }}_y{\mathit{\epsilon }}_y^2+{\mathbf{\Pi }}_{\eta }{\mathit{\epsilon }}_{\eta }^2+ln2\pi {\mathbf{\Sigma }}_y+ln2\pi {\mathbf{\Sigma }}_{\eta }\right]$$

(5)

where we expressed the difference in the exponents of the Gaussian distributions in terms of prediction errors ${\mathit{\epsilon }}_y=\mathit{y}-\mathit{g}(\mathit{\mu },\mathit{\theta })$ and ${\mathit{\epsilon }}_{\eta }=\mathit{\mu }-\mathit{\eta }$, and we wrote the covariances in terms of their inverse, i.e., precisions ${\mathbf{\Pi }}_y$ and ${\mathbf{\Pi }}_{\eta }$. In order to minimize the KL divergence between the real and approximate posteriors, we can minimize the VFE with respect to the beliefs $\mathit{\mu }$—a process associated with perception—and the parameters $\mathit{\theta }$—generally referred to as learning. In practice, the update rules follow gradient descent:

$$\begin{array}{ccc}\hfill \begin{array}{cc}\hfill \mathit{\mu }& =\underset{\mathit{\mu }}{arg\; min}\mathcal{F}\hfill \\ \hfill \dot{\mathit{\mu }}& =-{\partial }_{\mu }\mathcal{F}={\partial }_{\mu }{\mathit{g}}^T{\mathbf{\Pi }}_y{\mathit{\epsilon }}_y-{\mathbf{\Pi }}_{\eta }{\mathit{\epsilon }}_{\eta }\hfill \end{array}& \hfill & \hfill \begin{array}{cc}\hfill \mathit{\theta }& =\underset{\mathit{\theta }}{arg\; min}\mathcal{F}\hfill \\ \hfill \dot{\mathit{\theta }}& =-{\partial }_{\theta }\mathcal{F}=-{\partial }_{\theta }{\mathit{g}}^T{\mathbf{\Pi }}_y{\mathit{\epsilon }}_y\hfill \end{array}\end{array}$$

(6)

In order to separate the timescales of fast perception and slow-varying learning, the two phases are treated as the steps of an EM algorithm, i.e., optimizing the beliefs while keeping the parameters fixed, and then optimizing the parameters while keeping the beliefs fixed.

Predictive coding, originally rooted in data compression [44], has inspired biologically plausible alternatives to traditional deep learning, such as Predictive Coding Networks (PCNs) [45,46]. In fact, the predictive coding algorithm can be scaled up to learn highly complex structures, composed of causal relationships of arbitrarily high depth—in a similar way to deep neural networks. In particular, we can factorize a generative model into a product of different distributions, wherein a specific level only depends on the level above:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill p({\mathit{x}}^{\left(0\right)},\dots ,{\mathit{x}}^{\left(l\right)})& =p\left({\mathit{x}}^{\left(0\right)}\right)\prod _{l=1}^{L}p\left({\mathit{x}}^{\left(l\right)}\right|{\mathit{x}}^{(l-1)})\hfill \\ \hfill p\left({\mathit{x}}^{\left(l\right)}\right|{\mathit{x}}^{(l-1)})& =\mathcal{N}(\mathit{g}({\mathit{x}}^{(l-1)},{\mathit{\theta }}^{(l-1)}),{\mathbf{\Sigma }}^{\left(l\right)})\hfill \end{array}\end{array}$$

(7)

where 0 indicates the highest level, and L is the number of levels in the hierarchy. In this way, ${\mathit{x}}^{\left(l\right)}$ acts as an observation for level $l-1$, and $\mathit{g}({\mathit{x}}^{\left(l\right)},{\mathit{\theta }}^{\left(l\right)})$ acts as a prior for level $l+1$. As a result, this factorization allows us to express the update of beliefs and parameters of a specific level only based on the level above and the level below—with a simple VFE minimization as in the previous case. Differently from the backpropagation algorithm of neural networks, the message passing of prediction errors implements a biologically plausible Hebbian rule between presynaptic and postsynaptic activities. In fact, the predictions $\mathit{u}$ computed by the likelihood functions are typically a weighted combination of neurons passed to a nonlinear activation function $\mathit{\varphi }$:

$${\mathit{u}}_i^{\left(l\right)}=\sum _{j=1}^J{\mathit{W}}_{i,j}^{(l-1)}\mathit{\varphi }\left({\mathit{x}}_j^{(l-1)}\right)$$

(8)

where $W_{i,j}^{(l-1)}$ are the weights from neuron j at level $l-1$ to neuron i at level l. Crucially, what is backpropagated in PCNs are not signals detecting increasingly complex features, but messages representing how much the model is surprised about sensory observations (e.g., a prediction equal to an observation means that the network structure is a good approximation of the real process, and no errors have to be conveyed).

2.2. Hierarchical Active Inference

The theory of active inference builds on the same assumptions of predictive coding, but with two critical differences. The first one—which is actually shared with some implementations of predictive coding such as temporal PC [47]—assumes that living organisms constantly deal with highly dynamic environments, and the internal models that they build must reflect the changes occurring in the real generative process [48]. This relation is usually expressed in terms of generalized coordinates of motion (encoding, e.g., position, velocity, acceleration, and so on) [49]; consequently, the environment is modeled with the following nonlinear system:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \tilde{\mathit{y}}& =\tilde{\mathit{g}}\left(\tilde{\mathit{x}}\right)+{\mathit{w}}_y\hfill \\ \hfill \mathcal{D}\tilde{\mathit{x}}& =\tilde{\mathit{f}}(\tilde{\mathit{x}},\tilde{\mathit{v}})+{\mathit{w}}_x\hfill \end{array}\end{array}$$

(9)

where $\tilde{\mathit{x}}$ are the generalized hidden states, $\tilde{\mathit{v}}$ are the generalized hidden causes, $\tilde{\mathit{y}}$ are the generalized sensory signals, $\mathcal{D}$ is a differential operator that shifts all the temporal orders by one, i.e., $\mathcal{D}\tilde{\mathit{x}}=[{\mathit{x}}^{\prime },{\mathit{x}}^{″},{\mathit{x}}^{‴},\dots ]$, and the letter w indicates (Gaussian) noise terms. The likelihood function $\tilde{\mathit{g}}$ defines how hidden states generate sensory observations (as in predictive coding), while the dynamics function $\tilde{\mathit{f}}$ specifies the evolution of the hidden states (see [8,50] for more details). The associated joint probability is factorized into independent distributions:

$$p(\tilde{\mathit{y}},\tilde{\mathit{x}},\tilde{\mathit{v}})=p\left(\tilde{\mathit{y}}\right|\tilde{\mathit{x}})p\left(\tilde{\mathit{x}}\right|\tilde{\mathit{v}})p\left(\tilde{\mathit{v}}\right)$$

(10)

where each distribution is Gaussian:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill p\left(\tilde{\mathit{y}}\right|\tilde{\mathit{x}})& =\mathcal{N}(\tilde{\mathit{g}}\left(\tilde{\mathit{x}}\right),{\tilde{\mathbf{\Sigma }}}_y)\hfill \\ \hfill p\left(\mathcal{D}\tilde{\mathit{x}}\right|\tilde{\mathit{v}})& =\mathcal{N}(\tilde{\mathit{f}}(\tilde{\mathit{x}},\tilde{\mathit{v}}),{\tilde{\mathbf{\Sigma }}}_x)\hfill \\ \hfill p\left(\tilde{\mathit{v}}\right|\mathit{\eta })& =\mathcal{N}(\mathit{\eta },{\tilde{\mathbf{\Sigma }}}_v)\hfill \end{array}\end{array}$$

(11)

As in predictive coding, these distributions are inferred through approximate posteriors $q\left(\tilde{\mathit{x}}\right)$ and $q\left(\tilde{\mathit{v}}\right)$, minimizing the related VFE $\mathcal{F}$. As a result, the updates of the beliefs $\tilde{\mathit{\mu }}$ and $\tilde{\mathit{\nu }}$, respectively, over the hidden states and hidden causes become:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \dot{\tilde{\mathit{\mu }}}-\mathcal{D}\tilde{\mathit{\mu }}& =-{\partial }_{\mu }\mathcal{F}=\partial {\tilde{\mathit{g}}}^T{\tilde{\mathbf{\Pi }}}_y{\tilde{\mathit{\epsilon }}}_y+{\partial }_{\mu }{\tilde{\mathit{f}}}^T{\tilde{\mathbf{\Pi }}}_x{\tilde{\mathit{\epsilon }}}_x-{\mathcal{D}}^T{\tilde{\mathbf{\Pi }}}_x{\tilde{\mathit{\epsilon }}}_x\hfill \\ \hfill \dot{\tilde{\mathit{\nu }}}-\mathcal{D}\tilde{\mathit{\nu }}& =-{\partial }_{\nu }\mathcal{F}={\partial }_{\nu }{\tilde{\mathit{f}}}^T{\tilde{\mathbf{\Pi }}}_x{\tilde{\mathit{\epsilon }}}_x-{\tilde{\mathbf{\Pi }}}_v{\tilde{\mathit{\epsilon }}}_v\hfill \end{array}\end{array}$$

(12)

where ${\tilde{\mathbf{\Pi }}}_y$, ${\tilde{\mathbf{\Pi }}}_x$, and ${\tilde{\mathbf{\Pi }}}_v$ are the precisions, and ${\tilde{\mathit{\epsilon }}}_y$, ${\tilde{\mathit{\epsilon }}}_x$, and ${\tilde{\mathit{\epsilon }}}_v$ are, respectively, the prediction errors of sensory signals, dynamics, and priors:

$$\begin{array}{cc}\hfill {\tilde{\mathit{\epsilon }}}_y& =\tilde{\mathit{y}}-\tilde{\mathit{g}}\left(\tilde{\mathit{\mu }}\right)\hfill \\ \hfill {\tilde{\mathit{\epsilon }}}_x& =\mathcal{D}\tilde{\mathit{\mu }}-\tilde{\mathit{f}}(\tilde{\mathit{\mu }},\tilde{\mathit{\nu }})\hfill \\ \hfill {\tilde{\mathit{\epsilon }}}_v& =\tilde{\mathit{\nu }}-\mathit{\eta }\hfill \end{array}$$

(13)

For a full account of free energy minimization in active inference, see [8]. Unlike the update rules of predictive coding, additional terms arise from the inferred dynamics, through which the model can capture the evolution of the environment. Also, note that since we are minimizing over (dynamic) paths and not (static) states, an additional term $\mathcal{D}\tilde{\mathit{\mu }}$ is present; this implies trajectory tracking, during which the VFE is minimized only when the belief of the generalized hidden states $\mathcal{D}\tilde{\mathit{\mu }}$ matches its instantaneous trajectory $\dot{\tilde{\mathit{\mu }}}$.

The second assumption made by active inference is that our brains not only perceive (and learn) the external generative process, but also interact with it to reach desired states (e.g., in order to survive, not only must one understand which cause leads to an increase or decrease in temperature, but also take actions to live in a narrow range around 37 degrees Celsius). The free energy principle, which is at the core of active inference, states that all living organisms act to minimize the free energy (or, equivalently, surprise). In fact, in addition to the perceptual inference of predictive coding, the VFE can be minimized by sampling those sensory observations that conform to some prior beliefs, e.g., $\mathit{a}={arg\; min}_{\mathit{a}}\mathcal{F}$, where $\mathit{a}$ are the motor commands. This process is typically called self-evidencing, implying that if I believe that I find myself in a narrow range of temperatures, minimizing surprise via action will lead to finding places with such temperatures—hence making my beliefs true. One role of the hidden causes is to define the agent’s priors that ensure survival. These priors generate proprioceptive predictions which are suppressed by motor neurons via classical reflex arcs [51]:

$$\dot{\mathit{a}}=-{\partial }_a{\mathcal{F}}_p=-{\partial }_a{\tilde{\mathit{y}}}_p{\tilde{\mathbf{\Pi }}}_p{\tilde{\mathit{\epsilon }}}_p$$

(14)

where ${\partial }_a{\tilde{\mathit{y}}}_p$ is an inverse model from (proprioceptive) observations to actions, and ${\tilde{\mathit{\epsilon }}}_p={\tilde{\mathit{y}}}_p-{\tilde{\mathit{g}}}_p\left(\tilde{\mathit{\mu }}\right)$ are the generalized proprioceptive prediction errors.

As in predictive coding, we can scale up this generative model to capture the causal relationships of several related entities (e.g., the joints of a kinematic structure), up to a certain depth [19,36,37,52]. Therefore, the prior becomes the prediction from the layer above, while the observation becomes the likelihood of the layer below:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\dot{\tilde{\mathit{\mu }}}}^{\left(l\right)}& =\mathcal{D}{\tilde{\mathit{\mu }}}^{\left(l\right)}+\partial {\tilde{\mathit{g}}}^{\left(l\right)T}{\tilde{\mathbf{\Pi }}}_v^{\left(l\right)}{\tilde{\mathit{\epsilon }}}_v^{\left(l\right)}+{\partial }_{\mu }{\tilde{\mathit{f}}}^{\left(l\right)T}{\tilde{\mathbf{\Pi }}}_x^{\left(l\right)}{\tilde{\mathit{\epsilon }}}_x^{\left(l\right)}-{\mathcal{D}}^T{\tilde{\mathbf{\Pi }}}_x^{\left(l\right)}{\tilde{\mathit{\epsilon }}}_x^{\left(l\right)}\hfill \\ \hfill {\dot{\tilde{\mathit{\nu }}}}^{\left(l\right)}& =\mathcal{D}{\tilde{\mathit{\nu }}}^{\left(l\right)}+{\partial }_{\nu }{\tilde{\mathit{f}}}^{\left(l\right)T}{\tilde{\mathbf{\Pi }}}_x^{\left(l\right)}{\tilde{\mathit{\epsilon }}}_x^{\left(l\right)}-{\tilde{\mathbf{\Pi }}}_v^{(l-1)}{\tilde{\mathit{\epsilon }}}_v^{(l-1)}\hfill \end{array}\end{array}$$

(15)

where:

$$\begin{array}{cc}\hfill {\tilde{\mathit{\epsilon }}}_x^{\left(l\right)}& =\mathcal{D}{\tilde{\mathit{\mu }}}_x^{\left(l\right)}-{\tilde{\mathit{f}}}^{\left(l\right)}({\tilde{\mathit{\mu }}}^{\left(l\right)},{\tilde{\mathit{\nu }}}^{\left(l\right)})\hfill \\ \hfill {\tilde{\mathit{\epsilon }}}_v^{\left(l\right)}& ={\tilde{\mathit{\mu }}}_v^{(l+1)}-{\tilde{\mathit{g}}}^{\left(l\right)}\left({\tilde{\mathit{\mu }}}^{\left(l\right)}\right)\hfill \end{array}$$

(16)

and the subscript indicates the index of the level, as before. Here, the role of the hidden causes is to link hierarchical levels. This allows the agent to construct a hierarchy of representations varying at different temporal scales, critical for realizing richly structured behaviors such as linguistic communication [20] or singing [53]. In this case, the motor commands minimize the generalized prediction errors of the lowest level of the hierarchy.

This continuous formulation of active inference is highly effective for dealing with realistic environments. However, minimizing the VFE (which is the free energy of the past and present) does not afford planning and decision making in the immediate future. To do this, we construct a discrete generative model in which we discretize the possible future states and encode their expectations with categorical distributions. We then minimize the Expected Free Energy (EFE) which, as the name suggests, is the free energy that the agents expect to perceive in the future [8,54]. This discrete generative model is similar to the continuous model defined above, with the difference that we condition the hidden states over policies $\mathit{\pi }$ (which in active inference are sequences of discrete actions):

$$p(\mathit{s},\mathit{o},\mathit{\pi })=p\left(\mathit{o}\right|\mathit{s},\mathit{\pi }\left)p\right(\mathit{s}\left|\mathit{\pi }\right)p\left(\mathit{\pi }\right)$$

(17)

Here, $\mathit{s}$ and $\mathit{o}$ are discrete states and outcomes, which represent past, present, and future states as in Hidden Markov Models. Then, the EFE is specifically constructed by considering future states as random variables that need to be inferred:

$${\mathcal{G}}_{\pi }=\underset{q(\mathit{s},\mathit{o}|\mathit{\pi })}{\mathbb{E}}\left[ln{\displaystyle \frac{q\left(\mathit{s}\right|\mathit{\pi })}{p(\mathit{s},\mathit{o}|\mathit{\pi })}}\right]\approx \underset{q(\mathit{s},\mathit{o}|\mathit{\pi })}{\mathbb{E}}\left[ln{\displaystyle \frac{q\left(\mathit{s}\right)}{q\left(\mathit{s}\right|\mathit{o},\mathit{\pi })}}\right]-\underset{q\left(\mathit{o}\right|\mathit{\pi })}{\mathbb{E}}[lnp\left(\mathit{o}\right|\mathit{C})]$$

(18)

where $q\left(\mathit{o}\right|\mathit{\pi })$ is a recognition distribution that the agent constructs to infer the real posterior of the generative process. Critically, the probability distribution $p\left(\mathit{o}\right|\mathit{C})$ encodes preferred outcomes, acting similarly to a prior over the hidden causes in the continuous counterpart. The last two terms are, respectively, called epistemic (uncertainty reducing) and pragmatic (goal seeking). In practice, this quantity is used by first factorizing the agent’s generative model as in POMDPs:

$$p({\mathit{s}}_{1:T},{\mathit{o}}_{1:T},\mathit{\pi })=p\left({\mathit{s}}_1\right)·p\left(\mathit{\pi }\right)·\prod _{\tau =1}^{T}p\left({\mathit{o}}_{\tau }\right|{\mathit{s}}_{\tau })·\prod _{\tau =2}^{T}p\left({\mathit{s}}_{\tau }\right|{\mathit{s}}_{\tau -1},\mathit{\pi })$$

(19)

Each of these elements can be represented with categorical distributions:

$$\begin{array}{ccc}\hfill \begin{array}{cc}\hfill p\left({\mathit{s}}_1\right)& =Cat\left(\mathit{D}\right)\hfill \\ \hfill p\left(\mathit{\pi }\right)& =Cat\left(\mathit{E}\right)\hfill \end{array}& \hfill & \hfill \begin{array}{cc}\hfill p\left({\mathit{o}}_{\tau }\right|{\mathit{s}}_{\tau })& =Cat\left(\mathit{A}\right)\hfill \\ \hfill p\left({\mathit{s}}_{\tau }\right|{\mathit{s}}_{\tau -1},\mathit{\pi })& =Cat\left({\mathit{B}}_{\pi ,\tau }\right)\hfill \end{array}\end{array}$$

(20)

where $\mathit{D}$ encodes beliefs about the initial state, $\mathit{E}$ encodes the prior over policies, $\mathit{A}$ is the likelihood matrix and ${\mathit{B}}_{\pi ,\tau }$ is the transition matrix. The minimization of EFE in discrete models follows the variational method used by predictive coding and continuous-time active inference; specifically, the use of categorical distributions breaks down the inference of hidden states and policies to a simple local message passing:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathit{s}}_{\pi ,\tau }& =\sigma (ln{\mathit{B}}_{\pi ,\tau -1}{\mathit{s}}_{\pi ,\tau -1}+ln{\mathit{B}}_{\pi ,\tau }^T{\mathit{s}}_{\pi ,\tau +1}+ln{\mathit{A}}^T{\mathit{o}}_{\tau })\hfill \\ \hfill \mathit{\pi }& =\sigma (ln\mathit{E}-\mathcal{G})\hfill \\ \hfill {\mathcal{G}}_{\pi }& \approx \sum _{\tau }\mathit{A}{\mathit{s}}_{\pi ,\tau }(ln\mathit{A}{\mathit{s}}_{\pi ,\tau }-lnp\left({\mathit{o}}_{\tau }\right|\mathit{C}))-{\mathit{s}}_{\pi ,\tau }diag({\mathit{A}}^{T}ln\mathit{A})\hfill \end{array}\end{array}$$

(21)

For a complete treatment of how the EFE and the approximate posteriors are computed—along with other insightful implications of the free energy principle in discrete models—see [8], while [55] provides a more practical tutorial with basic applications. Equation (21) shows that the update rule for the discrete hidden states at time $\tau$ and conditioned over a policy $\pi$ is a combination of messages coming from the previous and next discrete time steps, and a message coming from the discrete outcome. This combination is passed to a softmax function in order to obtain a proper probability. In addition, the optimal policy $\mathit{\pi }$ is found by a combination of a policy prior and the EFE, where the latter is a composition of epistemic and pragmatic behaviors. Policy inference can be refined by computing the EFE of several steps ahead in the future—a process called sophisticated inference. Then, at each discrete step $\tau$, the agent selects the most likely action u under all policies, i.e., $u_t={arg\; max}_u\mathit{\pi }·[U_{\pi ,t}=u]$.

As before, we can construct a hierarchical structure able to express more and more invariant representations of hidden states [56]. In discrete state-space, links between hierarchical levels are usually made between hidden states via the matrix $\mathit{D}$—although in some formulations, the hidden states of a level condition the policies of the subordinate levels. The update rules for this hierarchical alternative become:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathit{s}}_{\pi ,\tau }^{\left(l\right)}& =\sigma (ln{\mathit{B}}_{\pi ,\tau -1}^{\left(l\right)}{\mathit{s}}_{\pi ,\tau -1}^{\left(l\right)}+ln{\mathit{B}}_{\pi ,\tau }^{\left(l\right)T}{\mathit{s}}_{\pi ,\tau +1}^{\left(l\right)}+ln{\mathit{A}}^{\left(l\right)T}{\mathit{o}}_{\tau }^{\left(l\right)}+ln{\mathit{D}}^{(l+1)T}{\mathit{s}}_1^{(l+1)})\hfill \\ \hfill {\mathit{\pi }}^{\left(l\right)}& =\sigma (ln{\mathit{E}}^{\left(l\right)}-{\mathcal{G}}^{\left(l\right)})\hfill \\ \hfill {\mathcal{G}}_{\pi }^{\left(l\right)}& \approx \sum _{\tau }{\mathit{A}}^{\left(l\right)}{\mathit{s}}_{\pi ,\tau }^{\left(l\right)}(ln{\mathit{A}}^{\left(l\right)}{\mathit{s}}_{\pi ,\tau }^{\left(l\right)}-lnp\left({\mathit{o}}_{\tau }^{\left(l\right)}\right|\mathit{C}))-{\mathit{s}}_{\pi ,\tau }^{\left(l\right)}diag({\mathit{A}}^{\left(l\right)T}ln{\mathit{A}}^{\left(l\right)})\hfill \end{array}\end{array}$$

(22)

This implies that each level takes abstract actions to minimize its EFE, only depending on the levels immediately below and above.

2.3. Bayesian Model Comparison

Bayesian model comparison is a technique used to compare a posterior over some data with a few simple hypotheses known a priori [57]. Consider a generative model $p(\mathit{y},\mathit{\theta })$ with parameters $\mathit{\theta }$ and data $\mathit{y}$:

$$p(\mathit{\theta },\mathit{y})=p\left(\mathit{y}\right|\mathit{\theta }\left)p\right(\mathit{\theta })$$

(23)

We introduce additional distributions $p(\mathit{y},\mathit{\theta }|m)$ which are reduced versions of the first model if the likelihood of some data is the same under both models, i.e., $p\left(\mathit{y}\right|\mathit{\theta },m)=p(\mathit{y}\left|\mathit{\theta }\right)$, and the only difference rests upon the specification of the priors $p\left(\mathit{\theta }\right|m)$. We can express the posterior of the reduced models in terms of the posterior of the full model and the ratios of the priors and the evidence:

$$p\left(\mathit{\theta }\right|\mathit{y},m)=p\left(\mathit{\theta }\right|\mathit{y}){\displaystyle \frac{p\left(\mathit{\theta }\right|m\left)p\right(\mathit{y})}{p\left(\mathit{\theta }\right)p\left(\mathit{y}\right|m)}}$$

(24)

The procedure for computing the reduced posteriors is the following. First, we integrate over the parameters to obtain the evidence ratio of the two models:

$$p\left(\mathit{y}\right|m)=p\left(\mathit{y}\right)\int p\left(\mathit{\theta }\right|\mathit{y}){\displaystyle \frac{p\left(\mathit{\theta }\right|m)}{p\left(\mathit{\theta }\right)}}d\mathit{\theta }$$

(25)

Then, we define an approximate posterior $q\left(\mathit{\theta }\right)$ and we compute the reduced free energies of each model m:

$$\mathcal{F}\left[p\left(\mathit{\theta }\right|m)\right]\approx \mathcal{F}\left[p\left(\mathit{\theta }\right)\right]+ln\underset{q}{\mathbb{E}}\left[{\displaystyle \frac{p\left(\mathit{\theta }\right|m)}{p\left(\mathit{\theta }\right)}}\right]$$

(26)

This VFE acts as a hint to how well the reduced representation explains the full model. Similarly, the approximate posterior of the reduced model can be written in terms of the posterior of the full model:

$$lnq\left(\mathit{\theta }\right|m)=lnq\left(\mathit{\theta }\right)+ln{\displaystyle \frac{p\left(\mathit{\theta }\right|m)}{p\left(\mathit{\theta }\right)}}-ln\underset{q}{\mathbb{E}}\left[{\displaystyle \frac{p\left(\mathit{\theta }\right|m)}{p\left(\mathit{\theta }\right)}}\right]$$

(27)

The Laplace approximation [58] leads to a simple form of the approximate posterior and the reduced free energy. Assuming the following Gaussian distributions:

$$\begin{array}{ccc}\hfill \begin{array}{cc}\hfill p\left(\mathit{\theta }\right)& =\mathcal{N}(\mathit{\eta },\mathbf{\Sigma })\hfill \\ \hfill p\left(\mathit{\theta }\right|m)& =\mathcal{N}({\mathit{\eta }}_m,{\mathbf{\Sigma }}_m)\hfill \end{array}& \hfill & \hfill \begin{array}{cc}\hfill q\left(\mathit{\theta }\right)& =\mathcal{N}(\mathit{\mu },\mathit{C})\hfill \\ \hfill q\left(\mathit{\theta }\right|m)& =\mathcal{N}({\mathit{\mu }}_m,{\mathit{C}}_m)\hfill \end{array}\end{array}$$

(28)

the reduced free energy turns to:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \mathcal{F}\left[p\right(\mathit{\theta }\left|m\right)]& \approx \mathcal{F}\left[p\left(\mathit{\theta }\right)\right]+{\displaystyle \frac{1}{2}}ln|{\mathbf{\Pi }}_m\mathit{P}{\mathit{C}}_m\mathbf{\Sigma }|\hfill \\ \hfill & -{\displaystyle \frac{1}{2}}({\mathit{\mu }}^T\mathit{P}\mathit{\mu }-{\mathit{\mu }}_m^T{\mathit{P}}_m{\mathit{\mu }}_m-{\mathit{\eta }}^T\mathbf{\Pi }\mathit{\eta }+{\mathit{\eta }}_m^T{\mathbf{\Pi }}_m{\mathit{\eta }}_m)\hfill \end{array}\end{array}$$

(29)

expressed in terms of precision of priors $\mathbf{\Pi }$ and posteriors $\mathit{P}$. The reduced posterior mean and precision are computed via Equation (27):

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathit{\mu }}_m& ={\mathit{C}}_m(\mathit{P}\mathit{\mu }-\mathbf{\Pi }\mathit{\eta }+{\mathbf{\Pi }}_m{\mathit{\eta }}_m)\hfill \\ \hfill {\mathit{P}}_m& =\mathit{P}-\mathbf{\Pi }+{\mathbf{\Pi }}_m\hfill \end{array}\end{array}$$

(30)

In this way, two different hypotheses i and j can be easily compared to infer which is the most likely to have generated the observed data; this comparison has the form of a log-Bayes factor $\mathcal{F}\left[p\right(\mathit{\theta }\left|i\right)]-\mathcal{F}[p\left(\mathit{\theta }\right|j]$. For a more detailed treatment of Bayesian model comparison (in particular under the Laplace approximation), see [57,59].

3. Results

3.1. Deep Hybrid Models

Analyzing the emergence of distributed intelligence, Friston et al. [60] emphasized three kinds of depth within the framework of active inference: factorial, hierarchical, and temporal. Factorial depth assumes independent factors in the agent’s generative model (e.g., objects and qualities of an environment, or more abstract states), which can be combined to generate outcomes and transitions. Hierarchical depth introduces causal relationships between levels, inducing a separation of temporal scales whereby higher levels happen to construct more invariant representations, while lower levels better capture the rapid changes of sensory stimuli. Temporal depth entails, in discrete terms, a vision into the imminent future that can be used for decision making, or, in continuous terms, increasingly precise estimates of dynamic trajectories.

In the following, we present the main features of a deep hybrid model in terms of factorial, temporal, and hierarchical depths in the context of flexible behavior, iterative transformations of reference frames, and dynamic planning. By deep hybrid model, we mean an active inference model composed of hybrid units connected hierarchically. Here, hybrid means that discrete and continuous representations are encoded within each unit, wherein the communication between the two domains is achieved by Bayesian model reduction [57,59]. As a technical note, all the internal operations can be computed through automatic differentiation, i.e., by maintaining the gradient graph when performing each forward pass and propagating back the prediction errors. For a detailed treatment of predictive coding, hierarchical active inference in discrete and continuous state-spaces, and Bayesian model comparison, see Section 2.1, Section 2.2 and Section 2.3, respectively.

3.1.1. Factorial Depth and Flexible Behavior

Consider the case where one agent needs to infer which object is being followed by another agent. This can be achieved through a hybrid unit $\mathcal{U}$, whose factor graph is depicted in Figure 1a. The variables are continuous hidden states $\tilde{\mathit{x}}=[\mathit{x},{\mathit{x}}^{\prime }]$, observations $\tilde{\mathit{y}}=[\mathit{y},{\mathit{y}}^{\prime }]$, and (discrete) hidden causes $\mathit{v}$. Hidden states and observations comprise two temporal orders (e.g., $\mathit{x}$ encodes the position and ${\mathit{x}}^{\prime }$ the velocity), and the first temporal order will be indicated as the 0th order; see the Methods section for more information. The factors are dynamics functions $\mathit{f}$ and likelihood functions $\tilde{\mathit{g}}=[\mathit{g},{\mathit{g}}^{\prime }]$. Note also a prior over the 0th-order hidden states ${\mathit{\eta }}_x$, and a prior over the hidden causes ${\mathit{H}}_v$ encoding the agent’s goals. The generative model is the following:

$$p(\tilde{\mathit{x}},\mathit{v},\tilde{\mathit{y}})=p\left(\tilde{\mathit{y}}\right|\tilde{\mathit{x}})p\left({\mathit{x}}^{\prime }\right|\mathit{x},\mathit{v})p\left(\mathit{x}\right)p\left(\mathit{v}\right)$$

(31)

We assume that hidden causes and hidden states have different dimensions, resulting in two factorizations. The hidden states, with dimension N, are sampled from independent Gaussian distributions and generate predictions in parallel pathways:

$$\begin{array}{ccccc}\hfill p\left(\mathit{x}\right)& =\prod _n^N\mathcal{N}({\mathit{\eta }}_{x,n},{\mathbf{\Sigma }}_{\eta ,x,n})\hfill & \hspace{1em}\hspace{1em}& \hfill p\left(\tilde{\mathit{y}}\right|\tilde{\mathit{x}})& =\prod _n^N\mathcal{N}({\tilde{\mathit{g}}}_n\left({\tilde{\mathit{x}}}_n\right),{\tilde{\mathbf{\Sigma }}}_{y,n})\hfill \end{array}$$

(32)

where ${\mathbf{\Sigma }}_{\eta ,x,n}$ and ${\tilde{\mathbf{\Sigma }}}_{y,n}$ are their covariance matrices. In turn, the hidden causes, with dimension M, are sampled from a categorical distribution:

$$p\left(\mathit{v}\right)=Cat\left({\mathit{H}}_v\right)$$

(33)

This differs from state-of-the-art hybrid architectures which assume separate continuous and discrete models with continuous hidden causes (see Section 2.2) [48]. A discrete hidden cause $v_m$ concurs, with hidden states $\mathit{x}$, in generating a specific prediction for the 1st temporal order ${\mathit{x}}^{\prime }$:

$$p\left({\mathit{x}}^{\prime }\right|\mathit{x},m)=\mathcal{N}({\mathit{f}}_m\left(\mathit{x}\right),{\mathbf{\Sigma }}_{x,m})$$

(34)

This probability distribution entails a potential trajectory, or hypothetical evolution of the hidden states, which the agent maintains to infer the state of affairs of the world and act. More formally, we consider $p\left({\mathit{x}}^{\prime }\right|\mathit{x},m)$ as being the mth reduced version of a full model:

$$p\left({\mathit{x}}^{\prime }\right|\mathit{x},\mathit{v})=\mathcal{N}({\mathit{\eta }}_x^{\prime },{\mathbf{\Sigma }}_x)$$

(35)

This allows us—using the variational approach for approximating the true posterior distributions—to convert discrete signals into continuous signals and vice versa through Bayesian model average and Bayesian model comparison, respectively (see Section 2.3 and [57,59]). In particular, top-down messages combine the potential trajectories ${\mathit{f}}_m\left(\mathit{x}\right)$ with the related probabilities encoded in the discrete hidden causes $\mathit{v}$:

$${\mathit{\eta }}_x^{\prime }=\sum _m^M{v}_m{\mathit{f}}_m\left(\mathit{x}\right)$$

(36)

This computes a dynamic path that is an average of the agent’s hypotheses, based on its prior ${\mathit{H}}_v$. Conversely, bottom-up messages compare the agent’s prior surprise $-ln{\mathit{H}}_v$ with the log evidence $l_m$ of every reduced model, i.e., $\mathit{v}=\sigma (ln{\mathit{H}}_v+\mathit{l})$, where $\mathit{l}=[l_1,\dots ,l_M]$ and $\sigma$ is a softmax function. The log evidence is accumulated over a continuous time T:

$$l_m={\int }_0^T{\displaystyle \frac{1}{2}}({\mathit{\mu }}_m^{\prime T}{\mathit{P}}_{x,m}{\mathit{\mu }}_m^{\prime }-{\mathit{f}}_m{\left(\mathit{x}\right)}^T{\mathbf{\Pi }}_{x,m}{\mathit{f}}_m\left(\mathit{x}\right)-{\mathit{\mu }}^{\prime T}{\mathit{P}}_x{\mathit{\mu }}^{\prime }+{\mathit{\eta }}_x^{\prime T}{\mathbf{\Pi }}_x{\mathit{\eta }}_x^{\prime })dt$$

(37)

where ${\mathit{\mu }}_m$, ${\mathit{P}}_{x,m}$, and ${\mathbf{\Pi }}_{x,m}$ are the mean, posterior precision, and prior precision of the mth reduced model. In this way, the agent can infer which dynamic hypothesis is most likely to have generated the perceived trajectory; see Figure 1b and [61] for more details about this approach.

A hybrid unit has useful features deriving from the factorial depths of hidden states and causes. Consider the case where the hidden states encode the agent’s configuration and other environmental objects, while the hidden causes represent the agent’s intentions. A hybrid unit could dynamically assign the causes of its actions at a particular moment; this is critical, e.g., in a pick and place operation, during which an object is first the cause of the hand movements—resulting in a picking action—but then it is the consequence of another cause (i.e., a goal position)—resulting in a placing action. This approach differs from other solutions [6,62] that directly encode a target location in the hidden causes. Further, embedding environmental entities—and not just the self—into the hidden states permits inferring their dynamic trajectories, which is fundamental for interactions, e.g., in catching objects on the fly [63] or in tracking a hidden target with the eyes [64]. Considering the example in Figure 1b, the hidden states $\mathit{x}$ and ${\mathit{x}}^{\prime }$ may encode the angle and angular velocity of an agent’s arm, and two hidden causes $v_{circle}$ and $v_{square}$ may be associated with dynamics functions ${\mathit{f}}_{circle}$ and ${\mathit{f}}_{square}$ encoding (potential) reaching movements toward the two objects. The environment may be inferred by two likelihood functions, one—${\mathit{g}}_p$—predicting proprioceptive observations (i.e., arm and angular velocity), and another one—${\mathit{g}}_v$—predicting visual observations (i.e., the hand position and velocity). Since we are interested in inferring which object is being followed by the agent, we set a uniform discrete prior ${\mathit{H}}_v$. Then, Equation (37) compares the actual agent’s trajectory to the two reaching movements toward the two objects, and assigns a higher probability to the one better resembling the real trajectory.

3.1.2. Hierarchical Depth and Iterative Transformations

Hierarchical depth is critical in many tasks that require learning of modular and flexible functions. Considering motor control, forward kinematics is repeated throughout every element of the kinematic chain, computing the end effector position from the body-centered reference frame. Iterative transformations are also fundamental in computer vision, where camera models perform roto-translations and perspective projections in sequence. How can we express such hierarchical computations in terms of inference? We design a structure called Intrinsic-Extrinsic (or IE) module, performing iterative transformations between reference frames [36,65]. A unit ${\mathcal{U}}_e^{\left(i\right)}$—where the superscript indicates the ith level—encodes a signal ${\mathit{x}}_e^{\left(i\right)}$ in an extrinsic reference frame (e.g., Cartesian coordinates), while another unit ${\mathcal{U}}_i^{\left(i\right)}$ represents an intrinsic signal ${\mathit{x}}_i^{\left(i\right)}$ (e.g., polar coordinates). At each level i, a likelihood function ${\mathit{g}}_e^{\left(i\right)}$ applies a transformation to the extrinsic signal provided by the higher level based on the intrinsic information, and returns a new extrinsic state:

$${\mathit{x}}_e^{\left(i\right)}={\mathit{g}}_e^{\left(i\right)}({\mathit{x}}_i^{\left(i\right)},{\mathit{x}}_e^{(i-1)})+{\mathit{w}}_e={\mathit{T}}^{\left(i\right)}\left({\mathit{x}}_i^{\left(i\right)}\right)·{\mathit{x}}_e^{(i-1)}+{\mathit{w}}_e$$

(38)

where ${\mathit{w}}_e$ is a noise term and ${\mathit{T}}^{\left(i\right)}$ is a linear transformation matrix. This new state acts as a prior for the subordinate levels in a multiple-output system; indicating with the superscript $(i,j)$ the ith hierarchical level and the jth unit within the same level, we link the IE modules in the following way:

$${\mathit{y}}_e^{(i,j)}\equiv {\mathit{x}}_e^{(i+1,j)}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{\mathit{x}}_e^{(i-1,j)}\equiv {\mathit{\eta }}_e^{(i,j)}$$

(39)

as displayed in Figure 2; hence, the observation of level i becomes the prior over the hidden states of level $i+1$. Ill-posed problems that generally have multiple solutions—such as inverse kinematics or depth estimation—can be solved by inverting the agent’s generative model and backpropagating the sensory prediction errors, with two additional features compared to traditional methods: (i) the possibility of steering the optimization by imposing appropriate priors, e.g., for avoiding singularities during inverse kinematics; (ii) the possibility of acting over the environment to minimize uncertainty, e.g., with motion parallax during depth estimation.

Encoding signals in intrinsic and extrinsic reference frames also induces a decomposition over proprioceptive and exteroceptive predictions, as well as intrinsic and extrinsic dynamics functions, leading to simpler (and yet richer) attractor states. Figure 2b shows two examples of goal-directed behavior with complex kinematic structures—a 28-DoF kinematic tree and a 23-DoF human body. In these cases, an IE module of Figure 2a is employed for each DoF of the agent. Each IE module encodes the position and velocity of a specific limb, both in extrinsic (e.g., Cartesian)—${\mathit{x}}_e^{\left(i\right)}$ and ${\mathit{x}}_e^{\left(i\right)\prime }$—and intrinsic (e.g., polar)—${\mathit{x}}_i^{\left(i\right)}$ and ${\mathit{x}}_i^{\left(i\right)\prime }$—reference frames. These modules are connected hierarchically, i.e., the trunk position generates predictions for the positions of both arms and legs through the likelihood function ${\mathit{g}}_e^{\left(i\right)}$ computing forward kinematics. The goal-directed behavior of the kinematic tree is realized by defining four reaching dynamics functions (toward the red objects) at the last levels of the hierarchy representing the end effectors. Instead, the behavior of the human body is achieved by defining repulsive dynamics functions for the extrinsic reference frames of each IE module. The decomposition of independent dynamics is also useful for tasks requiring multiple constraints in both domains, e.g., when walking with a glass in hand [36], and it has also been applied to controlling robots in 3D environments [66], or for estimating the depth of an object by moving the eyes [65]. Further, the factorial depth previously described permits representing hierarchically not only the self, but also the objects in relation to the self, along with the kinematic chains of other agents [67]. In this way, an agent could maintain a potential body configuration whenever it observes a relevant entity. This representation also accounts for the affordances of objects to be manipulated, and can be realized efficiently as soon as necessary.

3.1.3. Temporal Depth and Dynamic Planning

Consider the following discrete generative model:

$$p({\mathit{s}}_{1:\tau },{\mathit{o}}_{1:\tau },\mathit{\pi })=p\left({\mathit{s}}_1\right)p\left(\mathit{\pi }\right)\prod _{\tau }p\left({\mathit{o}}_{\tau }\right|{\mathit{s}}_{\tau })p\left({\mathit{s}}_{\tau }\right|{\mathit{s}}_{\tau -1},\mathit{\pi })$$

(40)

where:

$$\begin{array}{ccc}\hfill \begin{array}{cc}\hfill p\left({\mathit{s}}_1\right)& =Cat\left(\mathit{D}\right)\hfill \\ \hfill p\left(\mathit{\pi }\right)& =\sigma (-\mathcal{G})\hfill \end{array}& \hfill & \hfill \begin{array}{cc}\hfill p\left({\mathit{o}}_{\tau }\right|{\mathit{s}}_{\tau })& =Cat\left(\mathit{A}\right)\hfill \\ \hfill p\left({\mathit{s}}_{\tau +1}\right|{\mathit{s}}_{\tau },\mathit{\pi })& =Cat\left({\mathit{B}}_{\pi ,\tau }\right)\hfill \end{array}\end{array}$$

(41)

Here, $\mathit{A}$, $\mathit{B}$, $\mathit{D}$ are the likelihood matrix, transition matrix, and prior, $\mathit{\pi }$ is the policy, ${\mathit{s}}_{\tau }$ are the discrete hidden states at time $\tau$, ${\mathit{o}}_{\tau }$ are discrete observations, and $\mathcal{G}$ is the expected free energy (see Section 2.2 for more details).

We can let the likelihood $p\left({\mathit{o}}_{\tau }\right|{\mathit{s}}_{\tau })$ directly bias the (discrete) hidden causes of a hybrid unit:

$${\mathit{H}}_{\tau }\equiv \mathit{A}{\mathit{s}}_{\tau }\hspace{1em}\hspace{1em}{\mathit{o}}_{\tau }\equiv {\mathit{v}}_{\tau }$$

(42)

Hence, the prior ${\mathit{H}}_{\tau }$ over the discrete hidden causes becomes the prediction by the discrete model, while the discrete observation becomes the discrete hidden causes of the hybrid unit. This is an alternative method to the state-of-the-art, which considers an additional level between discrete observations and static priors over continuous hidden causes [37]. Here, the discrete model can impose priors over trajectories even in the same period $\tau$, thus affording dynamic planning [61,67]. If a discrete model is linked to different hybrid units in parallel—as shown in Figure 3—the discrete hidden states are inferred by combining multiple pieces of evidence:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathit{s}}_{\pi ,\tau }& =\sigma (ln{\mathit{B}}_{\pi ,\tau -1}{\mathit{s}}_{\pi ,\tau -1}+{\mathit{B}}_{\pi ,\tau +1}^T{\mathit{s}}_{\pi ,\tau +1}+\sum _{n}ln{\mathit{A}}^{{\left(i\right)}^T}{\mathit{v}}_{\tau }^{\left(i\right)})\hfill \end{array}\end{array}$$

(43)

where ${\mathit{s}}_{\pi ,\tau }$ are the discrete hidden states conditioned over policy $\pi$ at time $\tau$, while the superscript i indicates the ith hybrid unit. These parallel pathways synchronize the behavior of all low-level units based on the same high-level plan, allowing, e.g., simultaneous coordination of every limb of the human body.

In Figure 3a, we notice the two kinds of temporal depths, peculiar to hybrid active inference. The first one comes from the discrete component and unfolds over future states $({\mathit{s}}_1,{\mathit{s}}_2,{\mathit{s}}_3,\dots )$ over a time horizon defined by the policy length; it allows the agent to make plans by computing the expected free energy of those states [68]. The second temporal depth derives from the continuous level and unfolds over the temporal derivatives of the hidden states, i.e., $(\mathit{x},{\mathit{x}}^{\prime },{\mathit{x}}^{″},\dots )$; this refines the estimated trajectories with an increasing sampling rate.

In addition to this, the overall deep hybrid model presents two hierarchical depths with different roles. First is a hybrid scale that separates the slow-varying representation of the discretized task with the fast update of continuous signals. Here, the temporal predictions from the continuous dynamics are used to accurately infer the discrete variables, so that the agent can make complex high-level plans even when the surrounding environment is changing frequently and is able to revise those plans when new evidence has been accumulated. Second is a continuous scale linking the hybrid units and inherent to the hierarchical representation of the agent’s kinematic structure, which can be appreciated from Figure 2. This induces a separation of temporal scales between high and low levels of the hierarchy (e.g., the trunk vs. the hands), as the predictions errors generated from the dynamics of the hand have less and less impact as they flow back to the shoulder and trunk dynamics.

Considering the pick-and-place operation shown in Figure 3b, we can encode the three key moments of the task (start position, ball picked, and ball placed) in terms of discrete hidden states $\mathit{s}$. At each discrete step $\tau$, these states make (intrinsic and extrinsic) predictions for the hybrid units representing the agent’s kinematic chain (as in Figure 2). For instance, the second discrete hidden state generates a potential body configuration with the hand closed and at the ball position. We can use an identity mapping for the likelihood matrices, so that the intrinsic and extrinsic hidden causes of the hybrid units all have the same decomposition into three steps, i.e., ${\mathit{v}}_i^{\left(i\right)}=[v_{i,start}^{\left(i\right)},v_{i,picked}^{\left(i\right)},v_{i,placed}^{\left(i\right)}]$ and ${\mathit{v}}_e^{\left(i\right)}=[v_{e,start}^{\left(i\right)},v_{e,picked}^{\left(i\right)},v_{e,placed}^{\left(i\right)}]$. Notably, since these hidden causes are related to potential trajectories, the agent can pick and place the ball even in dynamic contexts, e.g., if the ball is moving.

3.2. A Deep Hybrid Model for Tool Use

In this section, we show how a deep hybrid model can be used efficiently in a task that requires planning in a dynamic environment and coordination of all elements of the agent’s body. The implementation details are found hereafter in Section 3.2.1, while Appendix A illustrates the algorithms for the inference of the discrete model and hybrid units. Then, in Section 3.3 we analyze model performance and describe the effects of dynamic planning in terms of accumulated sensory evidence and transitions over discrete hidden states.

Reaching an object with a tool is a complex task that requires all the features delineated in the previous section. First, the task has to be decomposed into subgoals—reaching the tool and reaching the object—which requires high-level discrete planning. Second, the agent has to maintain distinct beliefs about its arm, the tool, and the object, all of which must be inferred from sensory observations if their locations are unknown or constantly changing. Third, if the tool has to be grasped at the origin while the object has to be reached with the tool’s extremity, the agent’s generative model should encode a hierarchical representation of the self and every entity, and specify goals (in the form of attractors) at different levels of the hierarchy.

As shown in the graphical representation of the virtual environment of Figure 4a, the agent controls an arm of 4 DoF. The agent receives proprioceptive information about its joint angles, and exteroceptive (e.g., visual) observations encoding the positions of its limbs, the tool, and the ball. For simplicity, we assume that the tool sticks to the agent’s end effector as soon as it is touched. The generative model provides an effective decomposition into three parallel pathways, displayed in Figure 4c: one maintaining an estimate of the agent’s actual configuration (indicated by the subscript 0), and two others representing potential configurations in relation to the tool and the ball (indicated by the subscripts t and b, respectively). In other words, the objects of interest are not just encoded by their qualities or location, but already define a body configuration appropriate to achieve a specific interaction. In our case, the potential configuration related to the tool represents not only the estimated tool’s location, but also the estimated end effector’s location needed to reach the tool. In addition, each body configuration is composed of as many IE modules as the agent’s DoF, following the hierarchical relationships of the forward kinematics and allowing the expression of both joint angles and limb positions. Message passing of extrinsic prediction errors, i.e., differences between the estimated limb positions and their predictions given by the estimated joint angles, allows the inference of the whole body configuration (either actual or potential) via exteroceptive observations. In addition, every component of a level exchanges lateral messages with the other components, in the form of dynamics prediction errors. These errors are caused by the potential dynamics functions described in Section 3.1.1, which define the interactions between entities for goal-directed behavior.

Two crucial aspects arise when modeling entities in a deep hierarchical fashion and related to the self. First, different configurations are inferred (hence, different movements) depending on the desired interaction with the object considered—for example, reaching a ball with either the elbow or the end effector. Second, entities could have their own hierarchical structures; in our application, the tool consists of two Cartesian positions and an orientation, and the agent should somehow represent this additional link. For these reasons, we consider a virtual level for the tool configuration, attached to the last IE module (i.e., end effector), as exemplified in Figure 4c; the visual observations of the tool are then linked to the last two levels. From these observations, the correct tool angle can be inferred as if it were a new joint angle of the arm. Additionally, since we want the agent to touch the ball with the tool’s extremity, we model the third (ball) configuration with a similar structure, in which a visual observation of the ball is attached to the virtual level. The overall architecture can be better understood from Figure 4b, showing the agent’s continuous beliefs of all three entities. As soon as the agent perceives the tool, it infers a possible kinematic configuration as if it had visual access only to its last two joints (which are actually the tool’s origin and extremity). Likewise, perceiving the ball causes the agent to find an extended kinematic configuration as if the tool were part of the arm.

With this task formalization, specifying the correct dynamics for goal-directed behavior is simple. First, we define two sets of dynamics functions implementing every subgoal, one for reaching the tool and another for reaching the ball with the tool’s extremity. As explained in Section 3.2.1, the second subgoal requires specifying an attractor at the virtual level, which makes the agent think that the tool’s extremity will be pulled toward the ball. The biased state generates an extrinsic prediction error that is backpropagated to the previous level encoding the tool’s origin and the end effector. Notably, defining discrete hidden states and discrete hidden causes related to the agent’s intentions allows the agent to accumulate evidence over trajectories from different modalities (e.g., intrinsic or extrinsic) and hierarchical locations (e.g., elbow or end effector), ultimately solving the task via inference. The four main processes of the task, i.e., perception, dynamic inference, dynamic planning, and action, are summarized in Figure 5.

3.2.1. Implementation Details

The agent’s sensory modalities are (i) a proprioceptive observation ${\mathit{y}}_p$ for the arm’s joint angles; (ii) a visual observation ${\mathit{y}}_v$ encoding the Cartesian positions of every link of the arm, both extremities of the tool, and the ball; (iii) a discrete tactile observation ${\mathit{o}}_t$ signaling whether or not the target is grasped.

We decompose intrinsic and extrinsic hidden states of every IE module into three components, the first one corresponding to the actual arm configuration, and the other two related to potential configurations for the tool and the ball. Hence:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathit{x}}_i^{\left(i\right)}& =\left[\begin{array}{ccc}{\mathit{x}}_{i,0}^{\left(i\right)}& {\mathit{x}}_{i,t}^{\left(i\right)}& {\mathit{x}}_{i,b}^{\left(i\right)}\end{array}\right]\hfill \\ \hfill {\mathit{x}}_e^{\left(i\right)}& =\left[\begin{array}{ccc}{\mathit{x}}_{e,0}^{\left(i\right)}& {\mathit{x}}_{e,t}^{\left(i\right)}& {\mathit{x}}_{e,b}^{\left(i\right)}\end{array}\right]\hfill \end{array}\end{array}$$

(44)

The end effector’s level and the virtual level (see Figure 4c) are indicated with the superscripts $\left(4\right)$ and $\left(5\right)$, respectively. Regarding the IE module of the virtual level, the intrinsic and extrinsic hidden states only have two components related to the potential states of the tool and the ball, i.e., ${\mathit{x}}_i^{\left(5\right)}=[{\mathit{x}}_{i,t}^{\left(5\right)},{\mathit{x}}_{i,b}^{\left(5\right)}]$ and ${\mathit{x}}_e^{\left(5\right)}=[{\mathit{x}}_{e,t}^{\left(5\right)},{\mathit{x}}_{e,b}^{\left(5\right)}]$.

For each entity, the intrinsic hidden states encode pairs of joint angles and limb lengths, e.g., ${\mathit{x}}_{i,0}^{\left(i\right)}=[{\theta }_0^{\left(i\right)},l_0^{\left(i\right)}]$ while the extrinsic reference frame is expressed in terms of the position of a limb’s extremity and its absolute orientation, e.g., ${\mathit{x}}_{e,0}^{\left(i\right)}=[p_{0,x}^{\left(i\right)},p_{0,y}^{\left(i\right)},{\varphi }_0^{\left(i\right)}]$. The likelihood function ${\mathit{g}}_e$ of Equation (38) computes extrinsic predictions independently for each entity:

$${\mathit{g}}_e({\mathit{x}}_i^{\left(i\right)},{\mathit{x}}_e^{(i-1)})=\left[\begin{array}{ccc}\mathit{T}({\mathit{x}}_{i,0}^{\left(i\right)},{\mathit{x}}_{e,0}^{(i-1)})& \mathit{T}({\mathit{x}}_{i,t}^{\left(i\right)},{\mathit{x}}_{e,t}^{(i-1)})& \mathit{T}({\mathit{x}}_{i,b}^{\left(i\right)},{\mathit{x}}_{e,b}^{(i-1)})\end{array}\right]$$

(45)

Here, the mapping $\mathit{T}({\mathit{x}}_i,{\mathit{x}}_e)$ reduces to a simple roto-translation:

$$\mathit{T}({\mathit{x}}_i,{\mathit{x}}_e)=\left[\begin{array}{c}{p}_x+l{c}_{\theta ,\varphi }\\ p_y+l{s}_{\theta ,\varphi }\\ \varphi +\theta \end{array}\right]$$

(46)

where ${\mathit{x}}_i=[\theta ,l]$, ${\mathit{x}}_e=[p_x,p_y,\varphi ]$, and we used a compact notation to indicate the sine and cosine of the sum of two angles, i.e., $c_{\theta ,\varphi }=cos\left(\theta \right)cos\left(\varphi \right)-sin\left(\theta \right)sin\left(\varphi \right)$. Each level then computes proprioceptive and visual predictions through likelihood functions ${\mathit{g}}_p$ and ${\mathit{g}}_v$, which in this case are simple mappings that extract the joint angles of the actual arm configuration and the Cartesian positions of the limbs and objects from the intrinsic and extrinsic hidden states, respectively:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathit{g}}_p\left({\mathit{x}}_i^{\left(i\right)}\right)& ={\theta }_0^{\left(i\right)}\hfill \\ \hfill {\mathit{g}}_v\left({\mathit{x}}_e^{\left(i\right)}\right)& =\left[\begin{array}{ccc}{p}_{0,x}^{\left(i\right)}& p_{t,x}^{\left(i\right)}& p_{b,x}^{\left(i\right)}\\ p_{0,y}^{\left(i\right)}& p_{t,y}^{\left(i\right)}& p_{b,y}^{\left(i\right)}\end{array}\right]\hfill \end{array}\end{array}$$

(47)

Reaching the tool’s origin with the end effector is achieved by a function (related to an agent’s intention) that sets the first component of the corresponding extrinsic hidden states equal to the second one:

$${\mathit{i}}_{e,t}^{\left(4\right)}\left({\mathit{x}}_e^{\left(4\right)}\right)=\left[\begin{array}{ccc}{\mathit{x}}_{e,t}^{\left(4\right)}& {\mathit{x}}_{e,t}^{\left(4\right)}& {\mathit{x}}_{e,b}^{\left(4\right)}\end{array}\right]$$

(48)

Then, we define a potential dynamics function by subtracting the current hidden states from this intentional state:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathit{f}}_{e,t}^{\left(4\right)}\left({\mathit{x}}_e^{\left(4\right)}\right)& ={\mathit{i}}_{e,t}^{\left(4\right)}\left({\mathit{x}}_e^{\left(4\right)}\right)-{\mathit{x}}_e^{\left(4\right)}=\left[\begin{array}{ccc}{\mathit{x}}_{e,t}^{\left(4\right)}-{\mathit{x}}_{e,0}^{\left(4\right)}& \mathit{0}& \mathit{0}\end{array}\right]\hfill \end{array}\end{array}$$

(49)

Note the decomposition into separate attractors. A non-zero velocity for the first component expresses the agent’s desire to move the end effector, while a zero velocity for the other two components means that the agent does not intend to manipulate the objects during the first step of the task. Since a potential kinematic configuration for the tool is already at the agent’s disposal, in order to speed up the movement, similar functions can be defined at every hierarchical level, both in intrinsic and extrinsic reference frames. The second step of the task involves reaching the ball with the tool’s extremity. Hence, we define two intentional states for the end effector’s and virtual levels, setting every component equal to the (potential) component related to the ball:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathit{i}}_{e,b}^{\left(4\right)}\left({\mathit{x}}_e^{\left(4\right)}\right)& =\left[\begin{array}{ccc}{\mathit{x}}_{e,b}^{\left(4\right)}& {\mathit{x}}_{e,b}^{\left(4\right)}& {\mathit{x}}_{e,b}^{\left(4\right)}\end{array}\right]\hfill \\ \hfill {\mathit{i}}_{e,b}^{\left(5\right)}\left({\mathit{x}}_e^{\left(5\right)}\right)& =\left[\begin{array}{cc}{\mathit{x}}_{e,b}^{\left(5\right)}& {\mathit{x}}_{e,b}^{\left(5\right)}\end{array}\right]\hfill \end{array}\end{array}$$

(50)

The attractors encoded in the second set of potential dynamics functions express the agent’s desire to modify the tool’s location:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathit{f}}_{e,b}^{\left(4\right)}\left({\mathit{x}}_e^{\left(4\right)}\right)& ={\mathit{i}}_{e,b}^{\left(4\right)}\left({\mathit{x}}_e^{\left(4\right)}\right)-{\mathit{x}}_e^{\left(4\right)}=\left[\begin{array}{ccc}{\mathit{x}}_{e,b}^{\left(4\right)}-{\mathit{x}}_{e,0}^{\left(4\right)}& {\mathit{x}}_{e,b}^{\left(4\right)}-{\mathit{x}}_{e,t}^{\left(4\right)}& \mathit{0}\end{array}\right]\hfill \\ \hfill {\mathit{f}}_{e,b}^{\left(5\right)}\left({\mathit{x}}_e^{\left(5\right)}\right)& ={\mathit{i}}_{e,b}^{\left(5\right)}\left({\mathit{x}}_e^{\left(5\right)}\right)-{\mathit{x}}_e^{\left(5\right)}=\left[\begin{array}{cc}{\mathit{x}}_{e,b}^{\left(5\right)}-{\mathit{x}}_{e,t}^{\left(5\right)}& \mathit{0}\end{array}\right]\hfill \end{array}\end{array}$$

(51)

Maintaining this set of dynamics eventually drives the hand into a suitable position that makes the tool’s extremity touch the ball. A representation of the relations between such dynamics is displayed in Figure 6.

Now, we define the hidden causes of the last two levels (for simplicity, we only describe the extrinsic hidden states):

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathit{v}}_e^{\left(4\right)}& =\left[\begin{array}{ccc}{v}_s^{\left(4\right)}& v_t^{\left(4\right)}& v_b^{\left(4\right)}\end{array}\right]\hfill \\ \hfill {\mathit{v}}_e^{\left(5\right)}& =\left[\begin{array}{cc}{v}_s^{\left(5\right)}& v_b^{\left(5\right)}\end{array}\right]\hfill \end{array}\end{array}$$

(52)

where the subscripts s, t, and b indicate the agent’s intentions to maintain the current state of the world (“stay”), reach the tool, and reach the ball, respectively. The first hidden cause, related to the following intentional states:

$${\mathit{i}}_{e,s}^{\left(4\right)}\left({\mathit{x}}_e^{\left(4\right)}\right)={\mathit{x}}_e^{\left(4\right)}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{\mathit{i}}_{e,s}^{\left(5\right)}\left({\mathit{x}}_e^{\left(5\right)}\right)={\mathit{x}}_e^{\left(5\right)}$$

(53)

is needed to ensure that ${\mathit{v}}_e^{\left(4\right)}$ and ${\mathit{v}}_e^{\left(5\right)}$ encode proper probabilities when the discrete model is in the initial state. The average trajectory is then found by weighting the potential dynamics functions with the corresponding hidden causes. Hence, having indicated with ${\mathit{\mu }}_e^{\left(4\right)}$ the belief of the extrinsic hidden states of the end effector, the generated trajectory is:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathit{\eta }}_{x,e}^{\prime \left(4\right)}& =v_s^{\left(4\right)}{\mathit{f}}_{e,s}^{\left(4\right)}\left({\mathit{\mu }}_e^{\left(4\right)}\right)+v_t^{\left(4\right)}{\mathit{f}}_{e,t}^{\left(4\right)}\left({\mathit{\mu }}_e^{\left(4\right)}\right)+v_b^{\left(4\right)}{\mathit{f}}_{e,b}^{\left(4\right)}\left({\mathit{\mu }}_e^{\left(4\right)}\right)\hfill \end{array}\end{array}$$

(54)

The belief is then updated according to the following update rules:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\dot{\mathit{\mu }}}_e^{\left(4\right)}& ={\mathit{\mu }}_e^{\prime \left(4\right)}-{\mathit{\pi }}_e^{\left(4\right)}{\mathit{\epsilon }}_e^{\left(4\right)}+\partial {\mathit{g}}_e^T{\mathit{\pi }}_e^{\left(5\right)}{\mathit{\epsilon }}_e^{\left(5\right)}+\partial {\mathit{g}}_v^T{\mathit{\pi }}_v^{\left(4\right)}{\mathit{\epsilon }}_v^{\left(4\right)}+\partial {\mathit{\eta }}_{x,e}^{\prime \left(4\right)T}{\mathit{\pi }}_{x,e}^{\left(4\right)}{\mathit{\epsilon }}_{x,e}^{\left(4\right)}\hfill \\ \hfill {\dot{\mathit{\mu }}}_e^{\prime \left(4\right)}& =-{\mathit{\pi }}_{x,e}^{\left(4\right)}{\mathit{\epsilon }}_{x,e}^{\left(4\right)}\hfill \end{array}\end{array}$$

(55)

In short, the 0th-order is subject to (i) a quantity proportional to the estimated trajectory; (ii) an extrinsic prediction error coming from the elbow, i.e., ${\mathit{\epsilon }}_e^{\left(4\right)}={\mathit{\mu }}_e^{\left(4\right)}-{\mathit{g}}_e({\mathit{\mu }}_i^{\left(4\right)},{\mathit{\mu }}_e^{\left(3\right)})$; (iii) a backward extrinsic prediction error coming from the virtual level, i.e., ${\mathit{\epsilon }}_e^{\left(5\right)}={\mathit{\mu }}_e^{\left(5\right)}-{\mathit{g}}_e({\mathit{\mu }}_i^{\left(5\right)},{\mathit{\mu }}_e^{\left(4\right)})$; (iv) a visual prediction error, i.e., ${\mathit{\epsilon }}_v^{\left(4\right)}={\mathit{y}}_v^{\left(4\right)}-{\mathit{\mu }}_e^{\left(4\right)}$; (v) a backward dynamics error encoding the generated trajectory, i.e., ${\mathit{\epsilon }}_{x,e}^{\left(4\right)}={\mathit{\mu }}_e^{\prime \left(4\right)}-{\mathit{\eta }}_{x,e}^{\prime \left(4\right)}$. For a more detailed treatment of inference and dynamics of kinematic configurations in hierarchical settings, see [36].

The actions $\mathit{a}$ are instead computed by minimizing proprioceptive prediction errors:

$$\dot{\mathit{a}}=-{\partial }_a{\mathit{g}}_p^T{\mathit{\pi }}_p{\mathit{\epsilon }}_p$$

(56)

where ${\partial }_a{\mathit{g}}_p$ performs an inverse dynamics from proprioceptive predictions to actions.

As concerns the discrete model, its hidden states $\mathit{s}$ express (i) whether the agent is at the tool position, at the ball position, or neither of the two; (ii) whether the agent has grasped the tool or not. These two factors combine in six process states in total. The first factor generates predictions for the extrinsic hidden causes of the hybrid units through likelihood matrices, i.e., ${\mathit{A}}_e^{\left(4\right)}\mathit{s}$ and ${\mathit{A}}_e^{\left(5\right)}\mathit{s}$. This allows the agent to synchronize the behavior of both the tool and end effector; additional likelihood matrices can be defined to impose priors for the intrinsic hidden states and at different levels of the hierarchy. The second factor returns a discrete tactile prediction, i.e., ${\mathit{A}}_t\mathit{s}$.

Finally, we define a discrete action for each step of the task, and a transition matrix $\mathit{B}$ such that the ball can be reached only when the tool has been grasped. Discrete actions are replanned every 10 continuous time steps, and transitions between discrete states occur dynamically depending on continuous evidence. In the example of Figure 7, the transition between reaching the tool and reaching the ball happens after 350 time steps. The extrinsic hidden causes ${\mathit{v}}_e^{\left(4\right)}$ and ${\mathit{v}}_e^{\left(5\right)}$ are found by Bayesian model comparison, as explained in Section 3.1.1:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathit{v}}_e^{\left(4\right)}& =\sigma (ln{\mathit{A}}_e^{\left(4\right)}\mathit{s}+{\mathit{l}}_e^{\left(4\right)})\hfill \\ \hfill {\mathit{v}}_e^{\left(5\right)}& =\sigma (ln{\mathit{A}}_e^{\left(5\right)}\mathit{s}+{\mathit{l}}_e^{\left(5\right)})\hfill \end{array}\end{array}$$

(57)

As noted above, ${\mathit{A}}_e^{\left(4\right)}\mathit{s}$ and ${\mathit{A}}_e^{\left(5\right)}\mathit{s}$ represent predictions of extrinsic hypotheses made by the discrete model for the end effector and virtual levels, e.g., a higher value of $v_b^{\left(4\right)}$ and $v_b^{\left(5\right)}$ means that the discrete model wants to reach the ball. Conversely, ${\mathit{l}}_e^{\left(4\right)}$ and ${\mathit{l}}_e^{\left(5\right)}$ are the bottom-up messages that accumulate continuous log evidence over some time T, that is, they provide information whether the end effector is reaching the tool or the ball based on the context. Comparison between such high-level expectations and low-level evidence permits inferring the discrete hidden states based on potential trajectories. In fact, the hidden causes act as additional observations for the discrete model, which infers the states at time $\tau$ by combining them with the tactile observation and the hidden states at time $\tau -1$:

$${\mathit{s}}_{\pi ,\tau }=\sigma (ln{\mathit{B}}_{\pi ,\tau -1}{\mathit{s}}_{\tau -1}+ln{\mathit{A}}_e^{\left(4\right)T}{\mathit{v}}_{e,\tau }^{\left(4\right)}+ln{\mathit{A}}_e^{\left(5\right)T}{\mathit{v}}_{e,\tau }^{\left(5\right)}+ln{\mathit{A}}_t^T{\mathit{o}}_t)$$

(58)

If we assume for simplicity that the agent’s preferences are encoded in a tensor $\mathit{C}$ in terms of expected states, the expected free energy breaks down to:

$${\mathcal{G}}_{\pi }\approx \sum _{\tau }{\mathit{s}}_{\pi ,\tau }[ln{\mathit{s}}_{\pi ,\tau }-lnp\left({\mathit{s}}_{\tau }\right|\mathit{C})]$$

(59)

Computing the softmax of the expected free energy returns the posterior probability over the policies $\mathit{\pi }$, which are used to infer the new discrete hidden states at time $\tau +1$.

3.3. Analysis of Model Performances

Figure 7 illustrates task progress during a sample trial. Although both objects are moving, the discrete model successfully infers and imposes continuous trajectories allowing the agent to operate correctly and achieve its goal. At the beginning of a trial, the beliefs of the hidden states are initialized with the actual starting configuration of the arm. The agent infers two potential kinematic configurations for the tool and the ball. While the two observations of the tool constrain the corresponding inference, the ball belief is only subject to its actual position, thus letting the agent initially overestimate the length of the virtual level. During the first phase, only the tool reaching intention is active; as a consequence, the tool belief constantly biases the arm belief, which in turn pulls the real arm. After 350 steps, both these beliefs are in the same configuration, while the ball belief has inferred the corresponding position. At this point, the tool is grasped, causing the discrete model to predict a different combination of hidden causes. Now, both the tool and arm beliefs are pulled toward the ball belief. After about 800 steps, the agent infers the same configuration for all three beliefs, successfully reaching the ball with the tool’s extremity, and tracking it until the trial ends. Note that even during the first reaching movement, the agent continuously updates its configuration in relation to the ball; as a result, the second reaching movement is faster.

The transitions can be better appreciated from Figure 8a, showing the bottom-up messages (i.e., accumulated evidence) ${\mathit{l}}_e^{\left(4\right)}$ and ${\mathit{l}}_e^{\left(5\right)}$ for the last two levels of the hierarchy (i.e., end effector’s and virtual levels), and the discrete hidden states ${\mathit{s}}_{\tau }$. As evident, the virtual level does not contribute to the inference of the first reaching trajectory, since this only involves the end effector. Note how the agent is able to dynamically accumulate the evidence over its discrete hypotheses; during the first phase, the evidence $l_{e,t}^{\left(4\right)}$ (related to the tool belief at the end effector’s level) increases as soon as the end effector approaches the tool’s origin, while $l_{e,b}^{\left(4\right)}$ and $l_{e,b}^{\left(5\right)}$ (related to the ball belief at the end effector’s and virtual levels, respectively) decrease as the ball moves away. During the second phase, the latter two rapidly increase as the end effector approaches the ball; finally, every probability of both levels slowly stabilizes as the extrinsic beliefs converge to the same value and the errors are minimized. The slow decrease in the initial state and the fast transition between the two steps are well summarized in the bottom graph. The trajectories of the hidden states show that the agent can plan new trajectories with a high frequency (in this case, 10 continuous time steps), allowing it to react rapidly to environmental stimuli.

The relationship between hidden causes and continuous dynamics is summarized in Figure 8b, showing the extrinsic potential and estimated dynamics for the end effector’s and virtual levels, as well as the dynamics of the extrinsic hidden causes. For simplicity, we considered the same discrete hidden causes for every hierarchical level, and the top plot recapitulates the state of the whole kinematic configuration. Here, we note a similar behavior to the discrete hidden states (i.e., slow decrease in the stay cause and increase in the first reaching movement, and rapid increase in the second reaching movement in the middle of the trial). Note that the agent maintains potential dynamics related to the three intentions for the duration of the whole trial; these dynamics are combined to produce a trajectory that the motor units accomplish. In fact, two spikes are evident in the dynamics of the end effector’s level, and one in the dynamics of the virtual level, regarding the ball reaching action.

In order to assess the model performances in dynamic planning tasks, we run three different experiments, each composed of 150 trials. The first experiment assessed the capacity of picking a moving tool and reaching a static target; hence, for each trial, we varied the tool velocity. The second experiment assessed the capacity of picking a static tool and tracking a moving target; here, we varied the ball velocity. The third experiment evaluated the performances of the agent in picking a moving tool and tracking a moving target; hence, we varied both tool and ball velocities. In all experiments, we randomly sampled tool and ball positions, and their directions where relevant. Also, velocity varied from 0 to 8 pixels per time step. The width and height of the virtual environment was 1300 × 1300 pixels, twice the total arm length plus the tool length; thus, the ball and tool were out of reach for a significant period of each trial. The duration of each trial was set to 3000 steps.

The results of the simulations are visualized in Figure 9, showing the task accuracy, the time needed to complete the task, and the average final error (see the caption for more details). With static or slow-varying environments, the agent completes the task in all conditions. The first condition (moving tool) achieved good performances even with high tool velocity, although with low velocities there is a slight decrease in accuracy; since the tool often moves out of reach, the agent cannot accomplish the tool-picking action. This specific behavior is not present in the second condition (moving ball), probably due to the increased operational space which allows the agent to move along the whole environment. However, the performance decreases for high ball velocities also due to the additional difficulty of tracking moving objects. The third combined condition (moving tool and ball) achieved slightly lower performances than the second condition, with a time needed to complete the task similar to the first condition. However, the average tracking error shown in the bottom panel remains restricted even with high ball and tool velocities.

Finally, the dynamic behavior of the extrinsic beliefs can be analyzed from Figure 10, showing the trajectories, for a sample trial of the third condition, of all the forces that make up the update of Equation (55), for the last two levels and every environmental entity. The transition between the two phases of the task is evident here; the 1st-order derivative of the arm belief ${\mathit{\mu }}_{e,a}^{\prime \left(4\right)}$ (blue line in the top left panel) is non-zero during the whole task, and presents two spikes at the beginning of each phase, signaling an increased prediction error due to the new intention. The arm movement during the first phase is the consequence of the non-zero 1st-order derivative of the tool belief ${\mathit{\mu }}_{e,t}^{\prime \left(4\right)}$ (blue line in the middle left panel). The dynamics of the corresponding extrinsic prediction error ${\mathit{\epsilon }}_{e,t}^{\left(5\right)}$ (green line in the middle left panel) combines both this derivative and the visual prediction error of the next level ${\mathit{\epsilon }}_{v,t}^{\left(5\right)}$ (red line in the middle right panel). Note that this extrinsic prediction error does not exist for the arm belief, and that the backward dynamics error ${\mathit{\epsilon }}_{e,x,a}^{\left(4\right)}$ has a smaller impact on the overall update with respect to the 1st-order derivative. The second phase begins with a spike in the 1st-order derivative of the tool belief at the virtual level ${\mathit{\mu }}_{e,t}^{\prime \left(5\right)}$ (blue line in the middle right panel), which is propagated back to the previous level as an extrinsic prediction error. Finally, note that the ball belief is only subject to its visual observation and the extrinsic prediction error coming from the previous levels.

4. Discussion

We proposed a computational method, based on hybrid active inference, that affords dynamic planning for hierarchical settings. Our goal was twofold. First, to show the effectiveness of casting control problems as inference and, in particular, of expressing entities in relation to a hierarchical configuration of the self. While there could be several ways to combine the units of the proposed architecture, we showed a specific design as a proof-of-concept to solve a typical task: reaching a moving object with a tool. The agent had to rely on three kinds of depth, i.e., it had to dynamically infer its intentions for decision making, and form different hierarchical generative models depending on the structure and affordances of the entities. The proposed model unifies several characteristics studied in the active inference literature: the modeling of objects, recently performed in the context of active object reconstruction [69,70,71]; the analyses of affordances in relation to the agent’s beliefs [72]; the modeling of itinerant movements, with a behavior similar to the Lotka–Volterra dynamics implemented in [73]; planning and control in extrinsic coordinates [6,74]; inference of discrete states based on continuous signals in dynamic environments, achieved through a different kind of post hoc Bayesian model selection [75] or other various approaches such as bio-inspired SLAM [33], dynamic Bayesian networks [76], recurrent switching linear dynamical systems [77]; the use of tools for solving complex tasks [78].

Our second goal was to show that a (deep) hierarchical formulation of active inference could lend itself to learning and generalization of novel tasks. Although we used a fixed generative model, we revealed that advanced behavior is possible by using likelihood and dynamics functions that could be easily implemented with neural connections, and by decomposing the model into small units linked together. In [79], a hierarchical kinematic model was used to learn the limbs of an agent’s kinematic chain, both during perception and action. The same mechanism could be used to infer the length of tools needed for object manipulation, extending the kinematic chain in a flexible way. Therefore, an encouraging research direction would be to design a deep hybrid model in the wake of PCNs, and let the agent learn appropriate structure and internal attractors for a specific goal via free energy minimization. PCNs have demonstrated robust performance in tasks like classification and regression [46,80], while approximating the backpropagation algorithm [81,82,83,84]. However, few studies have leveraged the modular and hierarchical nature of predictive coding to model complex dynamics [47,85,86,87,88] or enable interactions with the environment [18,89,90,91,92], mostly achieved through RL. Well-known issues of deep RL are data efficiency, explainability, and generalization [45]. Instead, the human brain is capable of learning new tasks with a small amount of samples, transferring the knowledge previously acquired in similar situations. Another common criticism is that deep RL lacks explainability, which is of greater concern as AI systems rapidly grow. A viable alternative is to learn a model of the environment [93], e.g., with Bayesian non-parametrics [18]; however, these approaches are still computationally demanding. Albarracin et al. described how active inference may find an answer to the black box problem [94], and we further showed how different elements of an active inference agent have practical and interpretable meanings. In this view, optimization of parameters in hybrid models could be an effective alternative to deep RL algorithms, or other approaches in active inference relying on the use of neural networks as generative models.

Besides the fixed generative model, another limitation of the proposed study is that we only used two temporal orders, while a more complete and effective model would make use of a greater set of generalized coordinates [49]. Nonetheless, every aspect we introduced can be extended by considering increasing temporal orders. For instance, discrete variables could depend on the position, velocity, and acceleration of an object, thus inferring a more accurate representation of dynamic trajectories. Also, flexible behavior could be specified in the 2nd temporal order, resulting in a more realistic force-controlled system.

An interesting direction of research regards the generation of states and paths, about which useful indications might come from planning and control with POMDP models [18]. Some implementations of discrete active inference models used additional connections between policies and between discrete hidden states [60,95]; hence, it might be beneficial to design similar connections in continuous and hybrid contexts as well. In this study, a single high-level discrete model imposed the behavior of every other hybrid unit; an alternative would be to design independent connections between hidden causes such that a high-level decision would be propagated down to lower levels with local message passing. This approach may also provide insights into how, by repetition of the same task, discrete policies adapt to construct composite movements (e.g., a reaching and grasping action) from simpler continuous paths.