Stackelberg Population Dynamics: A Predictive-Sensitivity Approach

Abstract

Hierarchical decision-making processes traditionally modeled as bilevel optimization problems are widespread in modern engineering and social systems. In this work, we deal with a leader with a population of followers in a hierarchical order of play. In general, this problem can be modeled as a leader–follower Stackelberg equilibrium problem using a mathematical program with equilibrium constraints. We propose two interconnected dynamical systems to dynamically solve a bilevel optimization problem between a leader and follower population in a single time scale by a predictive-sensitivity conditioning interconnection. For the leader’s optimization problem, we developed a gradient descent algorithm based on the total derivative, and for the followers’ optimization problem, we used the population dynamics framework to model a population of interacting strategic agents. We extended the concept of the Stackelberg population equilibrium to the differential Stackelberg population equilibrium for population dynamics. Theoretical guarantees for the stability of the proposed Stackelberg population learning dynamics are presented. Finally, a distributed energy resource coordination problem is solved via pricing dynamics based on the proposed approach. Some simulation experiments are presented to illustrate the effectiveness of the framework.

1. Introduction

Hierarchical decision-making processes, where a leader is making decisions affected by the response of the follower(s), are widespread in modern engineering and social systems. Hierarchical decision-making structures or leader–follower problems appear in diverse domains of applications such as terrorism analysis [1], communications and network security [2], traffic management [3], smart grids [4,5,6], machine learning [7], and market systems [8]. A particular promising application domain is the intersection between machine learning and game theory to model hierarchical interactions between learning agents. Classic simultaneous play games have been extended to problems in machine learning such as robust supervised learning [9] or generative adversarial network [10] and, most recently, in multiagent system (MAS) reinforcement learning [11]. For hierarchical machine-learning approaches based on game theory, just a few works have been proposed. A couple of relevant contributions can be found in [12], where a gradient descent with time scale separation was proposed, and in [7], where Stackelberg learning dynamics was developed; both works focused on a leader with a single follower.

Traditionally, hierarchical decision-making processes have been modeled as bilevel optimization problems, which can be classified into two main theoretical domains [13]. On the one hand are models based on game theory [14], which have used bilevel programming methods to develop the concept of Stackelberg equilibria. On the other hand is mathematical programming, which proposed a first attempt to solve bilevel optimization problems as an upper-level optimization problem with a lower-level optimization problem as a constraint [15]. Considering its nested nature, bilevel optimization problems have challenged the optimization and mathematical community ever since; they have been proven to be strongly NP-hard [16], and even the evaluation of the optimality of a solution is NP-hard [17,18]. Recent critical literature reviews recount the historical developments and current perspectives of the field [13,19,20].

In this work, we deal with a leader with a population of followers in a hierarchical order of play. In general, this problem can be modeled as a leader–follower Stackelberg equilibrium problem using a mathematical program with equilibrium constraints (MPEC) [21]. Some applications of this modeling have been recently proposed [22,23,24,25]. A preliminary MPEC model for a multiple leaders with multiple followers game was presented in [22]. A discrete-time iterative algorithm was proposed to compute a Stackelberg–Nash saddle-point in a leader with multiple followers problem. In [24], a Stackelberg differential game based on pricing for the bandwidth usage of the Internet was solved. Finally, a semi-decentralized algorithm was formulated to compute a local solution to the leader with multiple followers Stackelberg aggregate game in [25]. On the other hand, for power system applications, there is an emerging market-based control approach called transactive energy systems (TESs) [26,27,28]. This method is based on a privacy-preserving, market-based framework for the management of devices exchanging energy [6,29]. Hierarchical decision-making modeling has been studied as a cornerstone in this framework, where a coordinator agent is responsible for solving the market-clearing price of the distribution network. In several scenarios, the control signal of the leader agent or coordinator is not a price, but a pricing function depending on the preferences of the follower agents forming a reverse Stackelberg game [30,31]. Considering the multiple applications based on hierarchical decision-making structures, we propose a Stackelberg game learning framework to solve bilevel optimization problems as a dynamical system in a single time scale. In this paper, we focus on a single leader with a follower population of agents due to the motivation of the coordinator agent problem in TESs. For the upper-level optimization problem, we propose gradient descent dynamics in continuous-time [32,33], and for the lower-level optimization problem, we propose population games dynamics based on the replicator dynamics [34,35], in order to obtain two nested dynamical systems. This allows us to bring theoretical tools from the dynamical system theory for the design of algorithms to solve optimization problems. In contrast with traditional time scale separation methods for interconnected dynamical systems such as singular perturbation analysis [36], we used predictive-sensitivity conditioning [37,38] to integrate the two interconnected dynamical systems in a single time scale to guarantee the stability and convergence of the solution to a differential Stackelberg population equilibrium.

The main contribution of this paper is threefold. First, we propose two interconnected dynamical systems to dynamically solve a bilevel optimization problem between a leader and follower population in a single time scale by a predictive-sensitivity conditioning interconnection [37]. For the leader optimization problem, we developed a gradient descent algorithm based on the total derivative using the implicit function theorem [39]. For the follower MAS optimization problem, we used the population dynamics framework to model MAS interacting strategic agents, as shown in [34,35,40,41,42] and the reference therein. We extended the concept of the Stackelberg population equilibrium [43] to the differential Stackelberg population equilibrium for population dynamics. Second, theoretical guarantees for the stability of the proposed Stackelberg population learning dynamics are presented. Finally, a distributed energy resource (DER) coordination problem was solved via pricing dynamics based on the proposed approach. Some simulation experiments are presented to illustrate the effectiveness of the framework.

The rest of the paper is organized as follows. In Section 2, the formulation of the bilevel optimization problem and the main concepts of Stackelberg equilibria are introduced. Section 3 presents the population game’s essential concepts, and the Stackelberg population learning dynamics is developed based on the predictive-sensitivity conditioning interconnection. Stability results are established to guarantee the convergence of the proposed dynamics. In Section 4, an application of the proposed framework is developed for DER coordination. Finally, the main conclusions are drawn in Section 5.

2. Stackelberg Games and Bilevel Optimization Problems

In this section, we introduce a general formulation of a noncooperative game between an agent called a leader and a finite set of agents called followers. In this case, we have a Stackelberg game with multiple followers, where the leader plays first and then the followers make their decision. This Stackelberg game can be modeled by a bilevel optimization problem. Traditionally, the leader’s optimization problem is referred to as the upper-level problem, and the followers’ optimization problem is referred as the lower-level problem. Each level has its own set of variables, constraints, and objective functions, and the upper-level problem has as a constraint the lower-level problem:

$$\begin{array}{cc}\hfill \underset{\lambda \in \Lambda }{min}& F_1(\lambda ,x^{*})\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& x^{*}\in arg\underset{x\in X}{min}\left\{F_2(\lambda ,x)\right\},\hfill \end{array}$$

(1)

where $\lambda \in \Lambda \subseteq \mathbb{R}$ is the upper-level variable, $x\in X\subseteq {\mathbb{R}}^n$ are the lower-level variables, $F_1(\lambda ,x)$ is the objective function of the leader agent, and $F_2(\lambda ,x)$ is the objective function of the followers. It was assumed that each follower has an objective function $J_i(\lambda ,x)$ with $i=\{1,2,\dots ,n\}$, and then, without loss of generality, we assumed that the global objective function is:

$$F_2(\lambda ,x)=\sum _{i=1}^n{J}_i(\lambda ,x).$$

(2)

Let us recall the main equilibrium concept studied in hierarchical play games: consider one leader agent and one follower agent $x_f$.

Definition 1

(Local Stackelberg equilibrium (SE)). Let Λ be the set of strategies for the leader agent and X_f be the set of strategies for the follower agent. Let BR_f (λ) be the best response function for the follower defined as:

$$B{R}_f\left(\lambda \right)=\{y\in X_f|F_2(\lambda ,y)\le F_2(\lambda ,x_f),\forall x_f\in X_f\}.$$

A leader strategy ${\lambda }^{*}\in \Lambda$ and a follower strategy $x_f^{*}$ are in Stackelberg equilibrium if the following conditions hold:

(a) 

${max}_{x_f\in B{R}_f\left({\lambda }^{*}\right)}{F}_1({\lambda }^{*},x_f)\le {max}_{x_f\in B{R}_f\left(\lambda \right)}{F}_1(\lambda ,x_f)$;

(b) 

$x_f^{*}\in B{R}_f\left({\lambda }^{*}\right)$.

The local notions of the equilibrium concepts are preferred since it is standard in population games where the concavity or convexity is not assumed in the objective functions.

Some assumptions have to be considered to assure the optimality and existence of the solutions to Problem (1) such as the connectivity of the communication graph and the regularity of objective functions.

Assumption 1.

The communication graph connecting each agent of the system such as the leader and follower population agents is connected.

Assumption 2.

Every objective function $F_1(\lambda ,x)$ and $F_2(\lambda ,x)$ is assumed to be differentiable everywhere and with Lipschitz continuous partial derivatives, and the Hessian matrices ${\nabla }_{\lambda \lambda }^2{F}_1(\lambda ,x)$ and ${\nabla }_{xx}^2{F}_2(\lambda ,x)$ are globally invertible.

Local solutions’ concepts and first- and second-order necessary and sufficient conditions for bilevel optimization problems [19] are needed to establish the Stackelberg population flow with predictive-sensitivity conditioning in the next section.

Definition 2.

A point $({\lambda }^{*},x^{*})$ is said to be a local solution to the bilevel optimization problem (1) if:

(a) 

$x^{*}$ is a local minimum of $F_2({\lambda }^{*},x)$;

(b) 

There exists a neighborhood $\Omega \subset \Lambda \times X$ of $({\lambda }^{*},x^{*})$ such that $F_1({\lambda }^{*},x^{*})\le F_1({\lambda }^{*},x)$ for all local solutions $({\lambda }^{*},x^{*})\in \Omega$ such that $x^{*}$ is a local minimum of $F_2(\lambda ,x)$.

Lemma 1

(First-order optimality conditions). If $({\lambda }^{*},x^{*})$ is a local solution of Problem (1), then it is a stationary point satisfying the first-order Karush–Kuhn–Tucker (KKT) conditions:

$$\begin{array}{c}\hfill {\nabla }_{\lambda }{F}_1({\lambda }^{*},x^{*}\left({\lambda }^{*}\right))=0,\\ \hfill {\nabla }_x{F}_2({\lambda }^{*},x^{*})=0.\end{array}$$

(3)

For a given $\lambda$, Assumption 2 guarantees that the lower-level problem in (1) has at most a single optimal solution $x^{*}$ satisfying the first-order optimality conditions in Lemma 1 of the lower-level ${\nabla }_x{F}_2(\lambda ,x^{*})=0$. The implicit function theorem [39] guarantees the local existence of the optimal solution function $x^{*}\left(\lambda \right)$, and its derivative is obtained as:

$${\nabla }_{\lambda }{x}^{*}\left(\lambda \right)=-{\left({\nabla }_{xx}^2{F}_2(\lambda ,x^{*}\left(\lambda \right))\right)}^{-1}{\nabla }_{x\lambda }^2{F}_2(\lambda ,x^{*}\left(\lambda \right)).$$

In addition, this optimal solution function can be used to give an expression for the directional derivative, also known as the total derivative in points $x\left(\lambda \right)$ and defined as:

$$\begin{array}{cc}\hfill {\mathcal{D}}_{\lambda }{F}_1(\lambda ,x\left(\lambda \right))& :={\nabla }_{\lambda }{F}_1(\lambda ,x\left(\lambda \right))\hfill \\ \hfill & +{\nabla }_{\lambda }x{\left(\lambda \right)}^{\top }{\nabla }_x{F}_2(\lambda ,x\left(\lambda \right)).\hfill \end{array}$$

(4)

Here, ${(·)}^{\top }$ stands for the vector or matrix transpose.

Lemma 2

(Second-order optimality conditions). If a stationary point $({\lambda }^{*},x^{*})$ satisfies:

$$\begin{array}{c}\hfill {\nabla }_{\lambda \lambda }^2{F}_1({\lambda }^{*},x^{*}\left({\lambda }^{*}\right))\ge 0,\\ \hfill {\nabla }_{xx}^2{F}_2({\lambda }^{*},x^{*})\ge 0,\end{array}$$

(5)

then $({\lambda }^{*},x^{*})$ is a local solution of the bilevel optimization Problem (1).

The second-order total derivative can be deduced similarly as the first-order (4) as follows,

$$\begin{array}{cc}\hfill {\mathcal{D}}_{\lambda \lambda }^2{F}_1(\lambda ,x\left(\lambda \right)):=& {\nabla }_{\lambda \lambda }^2{F}_1(\lambda ,x\left(\lambda \right))+{\nabla }_{\lambda }x{\left(\lambda \right)}^{\top }{\nabla }_{x\lambda }^2{F}_1(\lambda ,x\left(\lambda \right))\hfill \\ \hfill & +{\nabla }_{\lambda ,x}^2{F}_1(\lambda ,x\left(\lambda \right)){\nabla }_{\lambda }x\left(\lambda \right)+{\nabla }_{\lambda }x{\left(\lambda \right)}^{\top }{\nabla }_{xx}^2{F}_1(\lambda ,x\left(\lambda \right)){\nabla }_{\lambda }x\left(\lambda \right)\hfill \\ \hfill & +{\nabla }_x{F}_1{(\lambda ,x\left(\lambda \right))}^{\top }({\nabla }_{\lambda \lambda }^{2}x\left(\lambda \right)+{\nabla }_{xx}^{2}x\left(\lambda \right){\nabla }_{\lambda }x\left(\lambda \right)).\hfill \end{array}$$

The following definition introduces the differential versions of the Nash and Stackelberg equilibria [7].

Definition 3

(Differential Nash equilibrium (DNE)). The joint strategy $({\lambda }^{*},x^{*})\in \Lambda \times X$ is a differential Nash equilibrium if ${\nabla }_{\lambda }{F}_1({\lambda }^{*},x^{*})=0$, ${\nabla }_x{F}_2({\lambda }^{*},x^{*})=0$, ${\nabla }_{\lambda \lambda }^2{F}_1({\lambda }^{*},x^{*})>0$, and ${\nabla }_{xx}{F}_2({\lambda }^{*},x^{*})>0$.

Definition 4

(Differential Stackelberg equilibrium (DSE)). The joint strategy $({\lambda }^{*},x^{*})\in \Lambda \times X$ is a differential Stackelberg equilibrium if the total derivative ${\mathcal{D}}_{\lambda }{F}_1({\lambda }^{*},x^{*})=0$, ${\nabla }_x{F}_2({\lambda }^{*},x^{*})=0$, ${\mathcal{D}}_{\lambda \lambda }^2{F}_1({\lambda }^{*},x^{*})>0$, and ${\nabla }_{xx}^2{F}_2({\lambda }^{*},x^{*})>0$.

In the next section, we present the main contribution of this work using the concept introduced above.

3. Stackelberg Population Games with Predictive-Sensitivity Conditioning

In this section, we show how the bilevel optimization problem in two time scales can be solved in a single time scale using an interconnection based on a predictive-sensitivity matrix of two dynamical systems in continuous-time, namely a gradient descent and population dynamics. First, the basic concepts of population games are introduced. Thus, the definitions and results for Stackelberg population games are derived. Finally, the interconnection using the predictive-sensitivity matrix for Stackelberg population games is shown.

3.1. Population Games’ Essentials

Population games have been proposed as a multiagent model to find the Nash equilibrium for a large set of agents or a population. The main dynamical model in population games is the replicator dynamics. Replicator dynamics has been successfully implemented in various engineering applications where real-time adaptation and robustness to dynamic environmental uncertainties is of vital importance [35]. Replicator dynamics represents a mass M of players choosing strategies evolving in time. It is assumed that a finite set of pure strategies $\mathcal{S}=\{1,2,\dots ,n\}$, and the analysis is based on a payoff function associated with the selected strategy. The population states are denoted by the vector $x=[x_1,x_2,\dots ,x_n]$, and the population states are constrained by all possible distributions of individuals among the strategies given by the simplex:

$$\Delta =\left\{x\in {\mathbb{R}}_{\ge 0}^n|\sum _{i=1}^n{x}_i=M\right\}$$

(6)

The agents playing strategy i obtain a reward depending on the population state. A fitness function representing the reward for agents playing strategy i is defined as $f_i:\Delta \mapsto \mathbb{R}$ for $i\in \mathcal{S}$. The replicator dynamics is interpreted as an evolutionary game where the proportion of the population of agents playing the most successful strategy (higher payoff than the average) is increasing. Agents dynamically compare their fitness function with the other agents’ performance through the average fitness function of the whole population $\overline{f}={\sum }_i^n{x}_i{f}_i$, also understood as the expected average payoff of the population. The replicator dynamics associated with each population of agents playing the i-th strategy is given by:

$${\dot{x}}_i=x_i(f_i\left(x\right)-\overline{f}\left(x\right)),\mathrm{for}\mathrm{all}i\in \mathcal{S}.$$

(7)

As mentioned, the solution concept of the replicator dynamics is an equilibrium, and the Nash equilibrium is the preferred solution concept in population games. Let F be a population game: the Nash equilibrium set is defined as:

$$PNE\left(F\right)=\{x^{*}\in \Delta :x_i^{*}>0\Rightarrow f_i\left(x^{*}\right)\ge f_j\left(x^{*}\right), \forall i,j\in \mathcal{S}\}.$$

All agents obtain the same profit in a Nash equilibrium. An important type of population games is potential games. Potential games can be defined as follows.

Definition 5.

Let $F:{\mathbb{R}}_{+}^n\mapsto {\mathbb{R}}^n$ be a vector of fitness functions with positive population game payoffs. Let $V:{\mathbb{R}}_{+}^n\mapsto \mathbb{R}$ be a continuously differentiable function and the following condition hold,

$${\nabla }_{x}V\left(x\right)=F\left(x\right),\forall x\in {\mathbb{R}}_{+}^n.$$

Then, F is a full potential game.

In potential games, a single scalar-valued function is associated with the game, called the potential function, which retains key information about the payoffs of the agents. If there exists a continuously differentiable potential function $V:{\mathbb{R}}_{+}^n\mapsto \mathbb{R}$, then a potential game satisfies:

$$\begin{array}{c}\hfill \frac{\partial V\left(x\right)}{\partial x_i}=f_i\left(x\right)\mathrm{for}\mathrm{all}i\in \mathcal{S},\end{array}$$

(8)

and (8) also implies that F must satisfy the external symmetry:

$$\begin{array}{c}\hfill \frac{\partial f_i}{\partial x_j}=\frac{\partial f_j}{\partial x_i}\mathrm{for}\mathrm{all}i,j\in \mathcal{S}.\end{array}$$

(9)

In potential games, the Nash equilibrium is related to local maximizers of potential functions [34]; this is stated in the following lemma.

Lemma 3.

In a full potential game, the Nash equilibrium is equal to the solution of the first-order KKT conditions of the maximizing $V\left(x\right)$ subject to a feasible set Δ.

Finally, an important result on the stability of population games claims that a population game satisfying (8) is a stable game if the potential function V is concave [34].

3.2. Stackelberg Population Games

One of the main concepts in the design of population games is the selection of the fitness functions such that the replicator dynamics (7) can be used as a distributed optimization algorithm. In order to obtain fitness functions to accomplish a stable and convergent solution, we use the following Lemma 4, which characterizes an optimal solution for the lower-level optimization problem (1) satisfying Assumption 2 [44].

Lemma 4.

A solution of the bilevel optimization problem (1) $({\lambda }^{*},x^{*})$, with ${\lambda }^{*}\in \Lambda$ and $x^{*}$ belonging to the feasible set Δ, is an optimal solution for the population game if and only if ${\nabla }_{x_i}{F}_2\left(x_i^{*}\right)={\nabla }_{x_j}{F}_2\left(x_j^{*}\right)$ for all $i,j\in \mathcal{S}$.

Definition 6.

Let Λ be the set of strategies for the leader agent and $X\in {\mathbb{R}}^n$ be the set of strategies for the follower population. Let the best response function for the follower population $b{r}_f\left(\lambda \right)$ be defined as:

$$b{r}_f\left(\lambda \right)=\{x\in \Delta |x_i>0\Rightarrow f_i(\lambda ,x)\ge f_j(\lambda ,x),\forall i,j\in S\}.$$

A leader strategy ${\lambda }^{*}\in \Lambda$ and a follower population strategy vector $x^{*}$ are in a Stackelberg population equilibrium if the following conditions hold:

(a) 

${max}_{x\in b{r}_f\left({\lambda }^{*}\right)}{F}_1({\lambda }^{*},x)\le {max}_{x\in b{r}_f\left(\lambda \right)}{F}_1(\lambda ,x)$;

(b) 

$x^{*}\in b{r}_f\left({\lambda }^{*}\right)$.

Definition 7

(Differential Stackelberg population equilibrium (DSPE)). The joint strategy $({\lambda }^{*},x^{*})\in \Lambda \times \Delta$ is a differential Stackelberg population equilibrium if the following KKT conditions hold:

(a) 

First-order: ${\mathcal{D}}_{\lambda }{F}_1({\lambda }^{*},x^{*})=0$, ${\nabla }_x{F}_2({\lambda }^{*},x^{*})=0$;

(b) 

Second-order: ${\mathcal{D}}_{\lambda \lambda }^2{F}_1({\lambda }^{*},x^{*})>0$, ${\nabla }_{xx}^2{F}_2({\lambda }^{*},x^{*})>0$.

For the leader upper-level optimization problem (1), we propose a continuous-time gradient descent flow using the concept of the directional derivative, also known as total derivative obtained following the traditional ideas of gradient dynamics for convex optimization problems [45] as follows:

$$\begin{array}{c}\hfill {\Sigma }_1:\dot{\lambda }=-{\mathcal{D}}_{\lambda }{F}_1(\lambda ,x).\end{array}$$

(10)

For the lower-level optimization problem, we propose the follower population game dynamics using the replicator dynamics (7). In order to guarantee the convergence properties of the population games, a full potential game introduced in Definition 5 is chosen using the objective function $F_2(\lambda ,x)$ as the potential function, i.e., $V(\lambda ,x)=-F_2(\lambda ,x)$. Notice that the potential games are related to maximizing the potential function $V\left(x\right)$, and in this case, we seek to minimize, so we convert the problem. With this potential function, we can obtain the fitness function using (8), yielding for each agent i:

$$f_i(\lambda ,x)=\frac{\partial V(\lambda ,x)}{\partial x_i}=-\frac{\partial F_2(\lambda ,x)}{\partial x_i},$$

and since $F_2(\lambda ,x)$ is separable, then the fitness function for each agent i is:

$$f_i(\lambda ,x)=-\frac{\partial }{\partial x_i}\left(\sum _{i=1}^n{J}_i(\lambda ,x)\right)=-\frac{\partial J_i(\lambda ,x)}{\partial x_i}.$$

(11)

The average fitness function $\overline{f}(\lambda ,x)$ is then:

$$\overline{f}(\lambda ,x)=-\sum _{i=1}^n{x}_i\frac{\partial J_i(\lambda ,x)}{\partial x_i}.$$

(12)

With fitness functions for each agent (11) and the average fitness function (12), the replicator dynamics is the lower-level flow as follows:

$${\dot{x}}_i=x_i\left(f_i(\lambda ,x)-\overline{f}(\lambda ,x)\right),\forall i\in \mathcal{S}.$$

In compact form, we can define the vector of fitness functions $f(\lambda ,x)=[f_1,f_2,\dots ,f_n]$. We obtain the replicator dynamics in compact form as:

$${\Sigma }_2:\dot{x}=x·\left(f(\lambda ,x)-\overline{f}(\lambda ,x)\right).$$

(13)

We used the notation “·” to indicate an elementwise product between vectors. In this leader–follower population dynamics, we have a nested interaction model in two time scales, which could slow down the convergence of the dynamics. In order to solve these limitations in two time scales, we propose to use the prediction-sensitivity conditioning concept presented in the next section.

3.3. Predictive-Sensitivity for Stackelberg Population Games

In this section, we introduce the predictive-sensitivity conditioning approach to deal with the bilevel optimization problem (1). The regularity introduced in Assumption 2 guarantees that the bilevel problem has at most one solution, and this also ensures that the problem is well posed. Then, it is possible to define the sensitivity of $x^{*}$. The main idea is represented in Figure 1, where it is shown that the sensitivity matrix is used to integrate in one single level the dynamics associated with the upper-level by the gradient dynamics (10) and the lower-level by the replicator dynamics (13), respectively.

Considering Assumption 2, the analytic expression of (4) is well defined at any point $(\lambda ,x)$, so defining an extended sensitivity for any point $(\lambda ,x)$ is possible as:

$$S_{\lambda }^x(\lambda ,x):=-{\left({\nabla }_{xx}^2{F}_2(\lambda ,x)\right)}^{-1}{\nabla }_{x\lambda }^2{F}_2(\lambda ,x),$$

(14)

while:

$$S_{\lambda }^x{(\lambda ,x)|}_{(\lambda ,x^{*}\left(\lambda \right))}={\nabla }_{\lambda }{x}^{*}\left(\lambda \right).$$

is satisfied. The total derivative using the sensitivity matrix can be extended to any point $(\lambda ,x)$ as:

$$\begin{array}{cc}\hfill {\mathcal{D}}_{\lambda }{F}_1(\lambda ,x)& :={\nabla }_{\lambda }{F}_1(\lambda ,x)\hfill \\ \hfill & +S_{\lambda }^x{(\lambda ,x)}^{\top }{\nabla }_x{F}_2(\lambda ,x).\hfill \end{array}$$

(15)

The two-level interconnected dynamical system represented by ${\Sigma }_1$ and ${\Sigma }_2$ can be integrated into a single time scale using the sensitivity matrix (14) and the total derivative (15). A feed-forward term is added as the predictive-sensitivity conditioning based on the optimization sensitivity to the follower population dynamics ${\Sigma }_2$ (13) to predict the dynamics of the leader gradient descent dynamics ${\Sigma }_1$ (10).

Based on a recent work [37], we used a predictive-sensitivity conditioning interconnection between the two-time-scale systems without the need for the time scale separation. This interconnection preserves the optimal solution $({\lambda }^{*},x^{*}\left({\lambda }^{*}\right))$ and its convergence properties. A sensitivity-based conditioning matrix $S_{\lambda }^x$ is used as the interconnection between both subsystems as shown in Figure 1, and it is defined as follows.

Definition 8.

Consider the extended sensitivity $S_{\lambda }^x(\lambda ,x)$ defined in (14): the conditioning matrix is defined as:

$$\mathbf{S}=\left[\begin{array}{cc}I& 0\\ -S_{\lambda }^x(\lambda ,x)& I\end{array}\right]$$

(16)

where I is an identity matrix of the appropriate dimension.

Using Definition 8, we obtain the predictive-sensitivity Stackelberg learning dynamics:

$$\mathbf{S}\left[\begin{array}{c}\dot{\lambda }\\ \dot{x}\end{array}\right]=\left[\begin{array}{cc}I& 0\\ -S_{\lambda }^x(\lambda ,x)& I\end{array}\right]\left[\begin{array}{c}\dot{\lambda }\\ \dot{x}\end{array}\right]=\left[\begin{array}{c}-{\mathcal{D}}_{\lambda }{F}_1(\lambda ,x)\\ x·\left(f(\lambda ,x)-\overline{f}(\lambda ,x)\right)\end{array}\right]$$

(17)

It is observed that the extended sensitivity term modifies the dynamics of the follower population as:

$$\dot{x}=x·\left(f(\lambda ,x)-\overline{f}(\lambda ,x)\right)-S_{\lambda }^x(\lambda ,x){\mathcal{D}}_{\lambda }{F}_1(\lambda ,x),$$

where the first term drives the population state x to the optimal solution $x^{*}\left(\lambda \right)$, while the predictive-sensitivity feed-forward term anticipates the variation of $x^{*}\left(\lambda \right)$ due to the dynamics $\dot{\lambda }$. The complete predictive-sensitivity Stackelberg population flow is given by:

$$\begin{array}{cc}\hfill \dot{\lambda }& =-{\mathcal{D}}_{\lambda }{F}_1(\lambda ,x)\hfill \\ \hfill \dot{x}& =x·\left(f(\lambda ,x)-\overline{f}(\lambda ,x)\right)-S_{\lambda }^x(\lambda ,x){\mathcal{D}}_{\lambda }{F}_1(\lambda ,x)\hfill \end{array}$$

(18)

In the next result, the relation between the differential Stackelberg population equilibrium concepts defined in Definition 7 and the stability of the predictive-sensitivity Stackelberg population dynamics is established (18).

Theorem 1.

The joint strategy $({\lambda }^{*},x^{*})$ is a DSPE of the Stackelberg population game (1) satisfying the condition in Definition 7 if and only if it is a locally exponentially stable point of the predictive-sensitivity Stackelberg learning dynamics (18).

Proof. 

Consider that (18) satisfies Assumptions 1 and 2. Since we are proving local stability, it is sufficient to prove that the eigenvalues of the Jacobian of the system at the joint strategy $({\lambda }^{*},x^{*})$ have a strictly negative real part [36]. For this, observe that by Definition 7, if the point $({\lambda }^{*},x^{*})$ is a DSPE, then the second-order conditions are satisfied at this point, and in addition, it has been proven that if the population dynamics are a full potential game, this implies that the Stackelberg follower population (see Definition 6) is asymptotically stable [35], then it is guaranteed that the Jacobian would have a strictly negative part at point $({\lambda }^{*},x^{*})$. Conversely, if a point $({\lambda }^{*},x^{*})$ of the system (18) is locally exponentially stable, then the corresponding Jacobian must have eigenvalues with a negative real part; thus, it is a strictly local solution satisfying the conditions in Definition 7. □

In the next section, an interesting application to the coordination of distributed energy resources via pricing dynamics is presented to illustrate the theoretical results of the proposed framework.

4. DER Coordination via Pricing Dynamics

A distributed energy resource (DER) coordination problem via pricing dynamics [6] is presented to illustrate the applicability of the proposed predictive-sensitivity Stackelberg learning dynamics. A dynamic transactive control involving a feedback loop is implemented to solve a bilevel optimization problem via a pricing dynamics. The DERs consist of a set of distributed generators (follower population) interacting with a distribution system operator (DSO) (leader) to reach a Stackelberg equilibrium.

Consider a set of distributed generators $G=\{1,2,\dots ,M\}$: each distributed generator has a payoff function:

$$U_i(\lambda ,p_i)=\lambda p_i-C_i\left(p_j\right)$$

(19)

where $p_i$ is the amount of power produced by each generator i (this power is stacked in a vector as $p={[p_1,p_2,\dots ,p_M]}^{\top }$), $\lambda$ is the energy price defined by the DSO, and $C_i\left(p_i\right)$ is the cost function for producing energy for each generator. The cost function is traditionally chosen as a convex quadratic function such as $C_i\left(p_i\right)=1/2{\alpha }_i{p}_i^2+{\beta }_i{p}_i$, where ${\alpha }_i$, ${\beta }_i$ are the cost coefficients. The power resource allocation is constrained by the power balance equation ${\sum }_{i=1}^M{p}_i=P_L$, where $P_L$ is the total power demand of the distribution network.

On the other hand, the main goal of the pricing problem is to determine the market clearing price (MCP) for the energy producer. The DSO solves an optimization problem that maximize the welfare of the population over a market cycle. It is observed that the coordination scheme is cast as a Stackelberg game, which is a challenging bilevel optimization problem. In the objective function, it is assumed that the DERs will have chosen the optimal generation profile for a particular price $\lambda$. The optimal market clearing price is then obtained by solving the following bilevel optimization problem:

$$\begin{array}{cc}\hfill \underset{\lambda }{max}& U(\lambda ,p)=\sum _{i=1}^M{U}_i(\lambda ,p_i)\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& p=arg\underset{p}{max}\{U(\lambda ,p):\sum _{i=1}^M{p}_i=P_L\}.\hfill \end{array}$$

(20)

Since we posed the proposed dynamics for minimization problems, we define the objective functions as $F_1(\lambda ,p)=F_2(\lambda ,p)=-U(\lambda ,p)={\sum }_{i=1}^M(C_i\left(p_j\right)-\lambda p_i)$. Following the population game’s description, the fitness function for each generator is given by:

$$f_i(\lambda ,\mathit{p})=\frac{dU}{d{p}_i}=\frac{d{U}_i}{d{p}_i}=\lambda -({\alpha }_i{p}_i+{\beta }_i).$$

In order to obtain the pricing dynamics, we need the sensitivity matrix (14):

$$\begin{array}{cc}\hfill S_{\lambda }^x(\lambda ,x)& =-{\left({\nabla }_{xx}^2{F}_2(\lambda ,x)\right)}^{-1}{\nabla }_{x\lambda }^2{F}_2(\lambda ,x)\hfill \\ \hfill & =-{\left[\begin{array}{cccc}-{\alpha }_1& 0& \cdots & 0\\ 0& -{\alpha }_2& \cdots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \cdots & -{\alpha }_M\end{array}\right]}^{-1}\left[\begin{array}{c}1\\ 1\\ ⋮\\ 1\end{array}\right]\hfill \\ \hfill & =\left[\begin{array}{c}1/{\alpha }_1\\ 1/{\alpha }_2\\ ⋮\\ 1/{\alpha }_M\end{array}\right]\hfill \end{array}$$

With this sensitivity matrix, we can obtain the total derivative (15) for the gradient descent for the upper-level pricing variable $\lambda$. Now, we are ready to introduce the predictive-sensitivity Stackelberg learning dynamics (18) for the DER coordination problem:

$$\begin{array}{cc}\hfill \dot{\lambda }& =\sum _{i=1}^M{p}_i+\sum _{i=1}^M\left(p_i+\frac{{\beta }_i}{{\alpha }_i}-\frac{\lambda }{{\alpha }_i}\right)=\sum _{i=1}^M\left(2p_i+\frac{({\beta }_i-\lambda )}{{\alpha }_i}\right),\hfill \\ \hfill \dot{p}& =(1/P_L)p·\left(f(\lambda ,p)-\overline{f}(\lambda ,p)\right)-\frac{1}{\alpha }\left(\sum _{i=1}^M{p}_i+\sum _{i=1}^M\left(p_i+\frac{{\beta }_i}{{\alpha }_i}-\frac{\lambda }{{\alpha }_i}\right)\right),\hfill \end{array}$$

(21)

where $\frac{1}{\alpha }$ is the vector with elements $1/{\alpha }_i$.

The distributed generators’ (DGs) capacities and coefficients for the simulation experiments are presented in

Table 1. System parameters’ simulation.

Distributed Generators
i	$\overline{p_i}$	$\underline{p_i}$	${\alpha }_i$	${\beta }_i$
1	6000 (W)	200 (W)	2.5	3.5
2	5500 (W)	200 (W)	2.5	3.6
3	4000 (W)	200 (W)	1.9	2.5

. The numerical experiments were simulated in a IEEE 9-bus test feeder adapted from [41], where each DG was modeled using a voltage–source inverter with local controllers. In Figure 2 is presented the distribution network with three distributed generators and three loads. The test case scenario is as follows. Initially, the system has to supply a total load of $P_L=4950$ W, corresponding to three loads as $L_1=1500$ W, $L_2=1250$ W, and $L_3=2200$ W in the distribution network. The distributed generators in the distribution network dynamically interact with each other and with the DSO to reach the optimal solution via a pricing dynamics that determines the optimal price and the optimal energy allocation for each $DG$ based on the proposed predictive-sensitivity Stackelberg learning dynamics (18). Then, at $t=0.8$ s of simulation, an additional load is added to the grid, summing up the total to $P_L=5850$ W.

In Figure 3 is presented the frequency response. It is shown that the system maintains the frequency stable around the reference of 60 Hz. The variations of the load are observed as small overshoots, but the system returns to the equilibrium point with a rapid response.

In Figure 4 is presented the power behavior of each DG. It is shown that the less expensive DG3 (with small $\alpha$) dispatches more power to the system, while the more expensive DG2 (with the bigger $\alpha$) dispatches less power. The response to variations of the load (see Figure 5) at $t=0.8$ s is observed, and the generators dynamically adapt to the new demand with a small overshoot.

Finally, Figure 6 presents the price dynamics $\lambda \left(t\right)$. Behavior towards the stable point is observed and, then, the dynamic response to the variation of the load at $t=0.8$ s, as expected.

5. Conclusions

To solve bilevel optimization problems between a leader and a follower population, we proposed two dynamical systems interconnected by a predictive-sensitivity conditioning matrix in a single time scale. For the leader optimization problem, we developed a gradient descent algorithm based on the total derivative, and for the followers’ optimization problem, we used the population dynamics framework to model a population of interacting strategic agents. We extended the concept of the Stackelberg population equilibrium to the differential Stackelberg population equilibrium for population dynamics. Theoretical guarantees for the stability of the proposed Stackelberg population learning dynamics were presented. A distributed energy resource coordination problem was solved via a pricing dynamics based on the proposed approach. Some simulation experiments were presented to illustrate the effectiveness of the framework. As future work, several research avenues are available. For instance, extensions of this work to include uncertainty would be very useful for real-life problems.