Where Freshness Matters in the Control Loop: Mixed Age-of-Information and Event-based Co-design for Multi-loop Networked Control Systems

: In the design of multi-loop Networked Control Systems (NCSs) wherein each control system is characterized by heterogeneous dynamics and associated with certain set of timing speciﬁcations and constraints, appropriate metrics need to be employed for the synthesis of control and networking policies to efﬁciently respond to the requirements of each control loop. Majority of the design approaches for sampling, scheduling and control policies include either time-based or event-based metrics to perform pertinent actions in response to the changes of the parameters of interest. We speciﬁcally focus in this article on Age-of-Information (AoI) as a recently-developed time-based metric and threshold-based triggering function as a generic event-based metric. As the NCS model, we consider multiple heterogeneous stochastic linear control systems that close their feedback loops over a shared-resource communication network. We investigate the co-design across the NCS, and discuss the pros and cons with AoI and ET approaches in terms of asymptotic control performance measured by linear-quadratic Gaussian (LQG) cost functions. In particular, sampling and scheduling policies combining AoI and stochastic event-triggered metrics are proposed. It is argued that pure AoI functions that generate decision variables solely upon minimizing the average age irrespective of control systems dynamics may not be able to improve the overall NCS performance even compared with pure randomized policies. Our theoretical analyses are successfully validated through several simulation scenarios.

supporting information exchange required to make decisions at the right component and at the right time. This is the basic motivation behind employing appropriate utility functions to coordinate the process of data exchange in a network of many dynamical users. Over the last two decades, there have been many attempts from the control and the communication communities to develop, evaluate and improve such utility functions compared to the conventional fixed-period and randomized data coordination approaches. Notions such as Value-of-Information (VoI), [7,8], Age-of-Information (AoI), [9,10], and Event-Triggered (ET) [11,12], are metrics that have been separately shown to be capable of coordinating information distribution taking into account the integrated and coupled context of NCSs. Traditionally, however, two rather distinct paths on addressing the NCS design have been followed: from the communication perspective, the focus mainly resulting in the design approaches that maximize the network throughput or minimize the end-to-end latency and jitter often ignoring the dynamics, requirements and characteristics of the sending and receiving entities and the specific data that are being transmitted [13][14][15]. From the control perspective, on the other hand, the major goal has been to maximize quality-of-control (QoC), and the communication network is usually abstracted as one or more maximum-rate and delay-negligible transmission channels with enough computation and functional capability to resolve contentions [16,17]. Hence, to fill this research void, it is essential to develop systematic and applicable co-design principles for NCSs that bring both QoC and QoS together by studying novel architectures that take into account the requirements, limitations, and tolerances of both network and control systems.

Contributions
In this article, the goal is to propose an efficient co-design architecture for heterogeneous NCSs where the influence of both control and network systems are taken into account. Specifically, we study a sampling-scheduling-control co-design problem for stochastic NCSs comprised of multiple heterogeneous linear time-invariant (LTI) control systems. The sampling and control units reside at the control system layer and are designed distributedly, i.e., they are locally installed in every control loop and generate decision variables for their corresponding local control systems. The scheduling unit resides at the network layer and arbitrates the channel access in a centralized fashion, i.e., a unique scheduler coordinates the allocation process of the limited resources among the control loops to avoid contention and consequently data loss. We consider a realistic communication model in that the data packets that are not scheduled for immediate transmissions, if not updated by a newer data sample, are stored in a buffer for possible transmissions in future time instances. If a current sample is not successfully transmitted due to resource limitations, it is not discarded, and remains in the buffer to be either replaced by a newer sample, or transmitted with some delay whenever the communication resource is assigned to it. Therefore, end-to-end delay in our formulation is comprised of an inter-sampling duration induced by the local samplers and a network-induced delay due to the resource limitations. Performance of each local control system is asymptotically measured by the local linear-quadratic Gaussian (LQG) cost function and the overall asymptotic NCS performance is determined by the average sum of their local LQG costs. Note that the performance influenced by the resource constraints and the end-to-end transmission delays.
Motivated by the existing results for the design of control and communication systems, in this article we focus on two celebrated notions of utility metrics: AoI-and ET-based functions. We first discuss if these two design concepts may properly co-exist in a networked control scenario and study where each of them excels in terms of decision making efficiency. We evaluate them based on two crucial aspects: first, which class of policies result in lower local and overall cost values, and second, how much information is required for a policy maker to generate appropriate decisions. The first one, as explained earlier, is evaluated based on asymptotic LQG cost functions, while the second is basically judged based on that a policy maker needs less information, and distributed parts of networked system may not be willing to disclose too much information. Therefore, a desirable and applicable co-design architecture would result in sampling, scheduling and control depending on the model of sampling and communication network might possess different dynamic characteristics. In fact, delay affects the states of the control systems, and the states themselves affect the decision outcomes of the event-based policies, and those decisions also affect the network-induced delays [44]. Therefore, an optimal co-design needs to keep track of the network-induced delays which might not be feasible for stochastic networked systems.
The AoI metric, proposed in [45], has emerged to quantify the freshness of the received status updates at the estimator and has attracted significant attention from communication and networking communities. The AoI is defined as the time elapsed since the generation of the latest successfully received status update at the estimator. Several authors have studied the problem of minimizing some functions of AoI under different queuing and communication models [46][47][48][49][50]. While the works in [46][47][48] consider time averaged AoI, the authors in [49] consider minimizing the tail of the AoI, and the authors in [50] consider any non-decreasing and measurable function of AoI. Apart from studying the effects of communication scheduling on AoI, none of the above works consider estimation or control objectives for networked systems. Nonetheless, a general consensus is that, a lower AoI in an NCS may result in a lower estimation and control cost, because having access to fresher state information often improves the performance. However, only a handful of works considered the performance of the solutions proposed for AoI with respect to such costs. The authors in [51] have studied the minimum mean squared error problem with independent and identically distributed (i.i.d.) transmission delays for Wiener process estimation. They have shown that the estimation error is a function of AoI if the sampling decisions are independent of the observed Weiner process; otherwise, the estimation error is not a function of AoI. In [52], we studied a state estimator of a single-loop stochastic LTI system with i.i.d. transmission delays and derived the relation between AoI and the estimation error, assuming that the sampling decisions are independent of the observed states.
There has been an increasing interest recently from the control community to consider AoI utility functions due to their simpler evolution and characteristics compared to ET or VoI metrics. Despite some progress, however, there exist results suggesting that AoI-based approaches with the original linear formulation of AoI, may not be sensitive enough to dynamic changes of control systems and their QoC requirements [53,54]. In [55], various nonlinear functions of AoI are considered to be minimized instead of the conventional average linear AoI and it is shown that these variations of AoI utility functions can be beneficial to improve the control performance. The authors in [56] showed in a recent work that a discounted AoI-dependent monotonic function can be employed to optimally govern wireless network scheduling to maximize control performance over infinite horizon. Despite recent efforts reflected in the literature, there are still many challenges. Specifically, there is no result, to the best of our knowledge, on combined ET and AoI-based co-design across control systems and communication network layers.

Outline
In the remainder of this article, the NCS model and the problem statement and are described in Section 1. The co-design architecture with CE controllers, sampling and scheduling policies is presented in Section 2. Performance analysis and comparisons with other co-design architectures are presented in Section 3. Simulation results are demonstrated in Section 4 and the concluding remarks are summarized in Section 5.

Buffer input
Centralized scheduler local samplers  includes an array of variables belonging to the sub-system indexed by i at time t, while we define For constant matrices, a subscript indicates the corresponding sub-system, and a superscript denotes matrix power. An optimal decision variable/policy X is represented by X * . The set of natural, real, non-negative integer, and non-negative real numbers are denoted by N, R, N 0 , and R ≥0 , respectively. For n-by-m-dimensional real space, we use the notation R n×m .

NCS Model
We consider an NCS consisting of N heterogeneous stochastic linear time-invariant (LTI) controlled dynamical processes that synchronously exchange their sensory information with their corresponding controllers via a common resource-limited communication network, see Figure 1. Each process i ∈ N {1, . . . , N} comprises of a plant P i , a noisy sensor S i , and a feedback control unit including a feedback controller C i and an estimator E i . Each process i ∈ N is described as follows: where x i k ∈ R n i , u i k ∈ R m i and y i k ∈ R n i represent the state vector, control input and sensor measurement of the process i at a time-step k ∈ N 0 , respectively. Constant matrices A i ∈ R n i ×n i and B i ∈ R n i ×m i describe the system matrix and input matrix, respectively, and we assume that each pair (A i , B i ) is controllable. To allow for heterogeneity, A i and B i matrices may differ for different processes and may also adopt different dimensions. The random processes w i k ∈ R n i and v i k ∈ R n i are, respectively, the exogenous disturbance acting on the process dynamics and the measurement noise. They are assumed to be Gaussian distributed independent random sequences with mutually i.i.d. realizations The initial states x i 0 's, i ∈ N, are also presumed to be randomly selected from an arbitrary finite-moment distribution with mean µ x i 0 and variance Σ x i 0 ≻ 0. At every time-step k, the decision on whether the state measurement y i k of sub-system i is sent for transmission is taken by a local sampler S i located at the sensor station. The sampling decision is assumed to be the outcome of a local sampling policy ξ i : I i k → {0, 1}, where I i k represents the information available at S i at time-step k and will be formally defined later. The sampling decision outcome, denoted by the binary-valued variable δ i k , is as follows: At every time-step k, those sub-systems which locally decided to update their corresponding controllers will forward their sensor measurements to the communication network. We assume that the communication network has capacity limitations such that not all N sub-systems can simultaneously close their sensor-to-controller links at a time instance, i.e., if the network capacity at every single time-step is denoted by the constant c ∈ N, the following resource constraint holds The communication network is assumed to be consisting of a queue to store the received data packets and a scheduling unit that decides which data packets are to be transmitted at each time-step. It should be mentioned that, transmissions of data from sensors to the buffer and from the buffer to the controllers are not subject to communication delay, i.e., if the sampler or scheduler decides on a sample being sent to the buffer or a buffered data sent to the controller, the transmissions are completed instantaneously. The scheduling decision at every time-step k is assumed to be the outcome of a centralized resource allocation policy π : where I s k denotes the information available at the network scheduling unit at time-step k which will be formally defined later, and c is the constant capacity constraint. The scheduling decision associated with sub-system i at time-step k is denoted by the binary variable φ i k and is defined as send the latest measurement of sub-system i in the buffer to E i , 0, send nothing from sub-system i to E i .
The network queue buffers at most one data packet from each sub-system at every time instance. Hence, in case a new measurement belonging to a certain sub-system arrives at the queue, the fresher data packet replaces the formerly buffered data of that sub-system. The older data packet will be discarded. Therefore, for each sub-system, there is either no buffered data packet in the queue or there is one which is the latest measurement sent to the network by the local sampler. This means even the freshest data packet of a sub-system in the queue might contain the measurement that corresponds to a previous time-step.
When bandwidth is assigned to a certain sub-system, its freshest measurement in the queue will be forwarded to the corresponding control unit. The received state measurement by the control unit of a sub-system i at a time-step k, denoted by z i k , might belong to a previous timek < k due to the communication delay imposed by the scheduling unit. Therefore, z i k is determined as a function of the scheduling variable, as discussed in the following. Before that, we define the notion of AoI at the control unit in our NCS model, as follows: Definition 1. AoI at the control side of a sub-system i ∈ N, at time-step k ∈ N 0 , is defined as ∆ i k = k −k i , where y ik i is the latest received measurement by the estimator E i up to time k, which confirms δ ik i = 1.
Assume that at a time-step k, y ik i = y i . . = φ i k−1 = 0, since otherwise, no data belonging to sub-system i would be in the queue at time-step k. To conveniently denote this, we use the notation φ i k (k − ∆ i k ) = 1 to express the time index of the freshest buffered measurement belonging to sub-system i at time-step k that is scheduled to be transmitted to the estimator E i . Hence, by φ i k (k − ∆ i k ) = 1, we denote that y i k−∆ i k will be received by E i at k. If no measurement of sub-system i is scheduled to be transmitted at k, we simply write φ i k = 0. With this notation we declare two essential aspects of the information structure: 1) if a sample is scheduled for transmission, then the estimator knows which time instance the received measurement belongs to, and 2) receiving no measurement update might correspond to having no measurement sample of sub-system i in the queue and not necessarily to resource limitations. It should be noted that if there is no data belonging to a sub-system i buffered at a time-step k, then we certainly have φ i k = 0. In the other words, if the scheduler decides for φ i k = 1, then there exists exactly one buffered data packet of sub-system i to be sent to its corresponding control unit. Therefore, φ i k = 0 might correspond to either having no measurement sample of sub-system i in the buffer to forward or having not enough resources to schedule the available sample at that specific time. In the latter case y i k−∆ i k remains in the queue to be either serviced in future time-steps or replaced by a fresher sampled measurement. Finally, according to Definition 1, the information update at an estimator E i can be stated as Note that the estimator E i receives the current measurement sample y i k , only if φ i k (k) = 1, which ensures δ i k = 1 and ∆ i k = 0. Depending on the information received at the estimator and the state estimate computed, the control input u i k is assumed to be generated as the outcome of a causal mapping γ i :Ĩ i k → R m i , whereĨ i k represents the set of available information at the controller and will be formally defined later.

Remark 1.
In the absence of a measurement sample at the control side at a certain time k, i.e., if φ i k = 0, the estimator E i may use the information contained in the sampling variable, i.e., knowing the outcome of δ i k , and incorporate it in computingx i k . This extra knowledge is known as the side-information contained in the sampling variable. In this article, we do not investigate the impact of the side-information when no measurement update is received by an estimator. As we will see later when we introduce the information structures, we assume that the control unit of a sub-system keeps the history of the sampling variables δ i [0,k] , however, does not incorporate this side-information in computingx i k in the absence of a measurement sample. Incorporating side-information results in a nonlinear estimator and possibly non-tractable state estimator design problem, especially for threshold-based sampling policies in the presence of resource limitations. We assume that if no update is received at the estimator at some certain time-steps, then the estimator constructsx i k in a model-based fashion using the previous estimatex i k−1 .
Depending on the sampling and scheduling decision variables {δ i 0 , . . . , δ i k } and {φ i 0 , . . . , φ i k }, we can derive the dynamics of the AoI at the estimator E i . It is straightforward to derive the dynamics of ∆ i k , as functions of the sampling and scheduling variables: It can be seen from (7) that the AoI at the estimator depends on both sampling and scheduling decision outcomes.
Having the outcomes of the sampling and scheduling policies determined in (3) and (5), we can introduce the information sets I i k and I s k , available, respectively, for the local sampler of sub-system i and the centralized scheduling unit, as follows: where, I i , Σ x i 0 }, and N b k denotes the set of buffered state measurements at time-step k. Additionally, we introduce the set of available information for the estimator and controller of sub-system i at time-step k: Note that, with the information about sampling and scheduling variables in (8)-(10) and the expression for the AoI in (7), the sampler S i is aware of the sequence , and the centralized sampler has the knowledge of ∪ i∈N {∆ i [0,k−1] }. Having the information setĨ i k introduced, we can construct the state estimate and compute the estimation error at the estimator of sub-system i. We denote the state estimate at the estimator of , and define the corresponding estimation error as The dynamics of the estimation errorẽ i k can be obtained as Note that, we can write , which confirms (12). Assume now that the decision variables δ i k and φ i k are generated and y i k−∆ i k , for any Note that the realization of ∆ i k is determined by the sampling and scheduling variables δ i We can compute the state estimate as where, P i − t and P i t denote, respectively, the a priori and the a posteriori estimation error covariances. Therefore, from (11), and using the equivalent expression where,ẽ i k−∆ i k is the MMSE error due to having access to y i Otherwise, if φ i k = 0, we use the model-based estimation error as in (12), whereinẽ i k−1 is not necessarily MMSE error.

Problem Description
As discussed above, the time of generating a measurement sample and injecting it to the queue is determined by the sampler while the time of delivering that generated sample, if not discarded due to the arrival of a new sample, to the corresponding controller is determined by the network scheduler. Hence, the source-to-destination delay, i.e., the gap between the current time until the time a generated sample is received by the controller, depends on how the local samplers and the centralized scheduler policies are designed. The problem we tackle in this article is the co-design of sampling, scheduling and control policies {ξ i , π, γ i }. We discuss the optimal control policy, and then consider ET and AoI-based policies for the design of sampling and scheduling policies and study the effects of the combined architecture on the control performance which is correlated with the end-to-end delay. Performance comparisons are made according to the LQG index functions as the asymptotic cost metrics for each local sub-system, denoted by J i : where Q 1 i , Q 2 i 0 and R i ≻ 0 are, respectively, the state and control input weight matrices of appropriate dimensions, and we assume each pair The overall asymptotic NCS performance is measured by the average cost

NCS Design
In this section, we first study the structural properties of the feedback controllers C i , i ∈ N, and show that local control law γ i (Ĩ i k ) can be designed separately from the local sampling law ξ i (I i k ) and the scheduling law π(I s k ). Afterwards, we discuss the combined design of the local sampling law and the network scheduling law and discuss which class of ET or AoI-based policies match the corresponding decision maker.

CE Control Law
Let us first make a crucial assumption on the sampling policy ξ i (I i k ): Assumption 1 does not result in a loss of generality w.r.t. the introduced information structure at the sampler, see (8) that indicated I i k does not contain any knowledge of control inputs {u i 0 , . . . , u i k−1 }. This is crucial for the derivation of the optimal control policies, as will be discussed in Theorem 1. (1)-(6), where each control system is steered at every time-step k ∈ N 0 by a local sampler ξ i (I i k ) and a local plant controller γ i (Ĩ i k ) with I i k andĨ i k given in (8) and (10), respectively. If the local sampling policies are selected according to the Assumption (1), then the optimal control policy in sense of LQG given in (14) is CE, i.e.,

Theorem 1. Consider an NCS as described in
Proof. See Appendix A.

Remark 2.
Showing that the optimal control law exists over the time horizon [0, T], we can take the limit as T → ∞ which results in having the asymptotic control gain L i  [22,57] that in the presence of control-input-dependent sampling policies, the separation between the sampling and control policies cannot generally be achieved. As it is shown in (18) and (19), the estimation error evolves independent of the control inputs, therefore, the sampling policies are allowed to be function of the estimation error without violating the results of Theorem 1.

Remark 4.
Theorem 1 states that the optimal control law is of certainty equivalence form, however, the optimal control inputs u i, * k are still computed based on the state estimate E x i k |Ĩ i k . As shown before, the estimation process depends on the sampling and scheduling policies ξ i (I i k ) and π(I s k ), hence the sequence of control inputs is only optimal w.r.t. the given sampling and scheduling policies, and the control inputs are globally optimal only if sampling/scheduling policies are optimal. However, under any sampling policy that satisfies Assumption 1 and any scheduling policy, the optimal control law (16) remains CE. Now that the control law is characterized, we can derive the dynamics of the estimation error at the sampler, assuming that the local samplers are aware of the control law form in (16). This assumption is essential in the sense that the samplers do not need to have the knowledge of the control inputs {u i 0 , . . . , u i k−1 } to compute the estimation error, and this coincides with the information structure (8). The estimation error at the sampler is defined as From (8), and at time-step k, the sampler has the knowledge of the latest controller measurement update z i k−1 . Let for any arbitrary Then, similar to (13), we can compute the estimation error e i k as If φ i k−1 = 0, the estimation error at the sampler is, similar to (12), computed based on the model parameters, i.e., Note the difference betweenẽ i (18) and (19), where the former is the MMSE error due to having the measurement sample y i , while the latter is not MMSE as the estimator does not have access to y i k−1 at time-step k − 1. (12) and (19), we conclude that if the estimator E i does not receive any state measurement update at time k − 1, i.e., φ i k−1 = 0, then e i k =ẽ i k . It should, however, be noted that this equality is valid under the assumption that the estimator does not incorporate side information contained in the sampling variables to compute the state estimate.

Co-design of Sampling and Scheduling Laws
As the optimal control policy is shown to be CE, we now propose the sampling/scheduling co-design. We specifically focus on two common classes of policies, the ET and AoI utility functions, and study which class of policies is more suitable for sampling and which fits better to govern the scheduling process. Remind that the sampling is performed locally within each sub-system while the scheduler resides in the network layer and is performed in centralized fashion, see Figure 1.
We now introduce the ET and VoI functions used in the rest of this article. For the sampling policy, if the ET threshold-based approach is employed, then a sample of a local sub-system i ∈ N is generated and forwarded to the network buffer whenever the square norm of the corresponding sub-system's estimation error exceeds a positive random threshold r i k , i.e., where, the binary-valued δ i, ET k indicates if a sample is forwarded for transmission or not based on the ET policy. The sequence of i.i.d. real-valued random thresholds r i k ∼ exp(µ i r ), k ∈ N 0 are assumed to be exponentially distributed, with µ i r ∈ R ≥0 being the rate parameter of the exponential distribution. Random threshold policy is a more general form of the threshold-based policies, hence the presented results in this article are easily extendable for ET deterministic threshold-based approach. Note that, the sampling policy (20) is in accordance with the Assumption 1. Remind that e i k denotes the estimation error computed at time k at the sampler side S i (not at the controller side C i ).
When AoI policy is employed for sampling, a state sample of a sub-system i is sent to the communication network for transmission whenever the age of the latest received state information at the controller C i exceeds a given threshold λ i ∈ N 0 , i.e., Since age is a discrete variable taking only non-negative integer value, without loss of any generality, the threshold λ i is also assumed to be selected from non-negative integers.
As a comparative scenario, we also consider the periodic sampling, in which each sensor sample is sent for transmission at pre-defined instances of time and the inter-transmission time is determined by the constant time period T p ∈ N. Therefore, we have As noticed in the expressions (20)- (22), we use the superscripts "ET", "AoI" and "P" to indicate that the sampling policies are ET, AoI-based, and periodic, respectively. For the purpose of illustrations and ease of analysis, let us set the communication channel capacity to c = 1, i.e., at every time-step k the scheduler allows only one state information to be forwarded to the corresponding controller, (see (4)). We already introduced N b k as the set of all sub-systems that have a state sample in the network buffer at time-step k. Note that, this state information might belong to the current time k or to a previous time, hence, the buffered state measurements are not necessarily time-synchronized. For the AoI scheduling, we introduce the highest-age-first policy that in fact minimizes the average age of all sub-systems in N b k . For a sub-system i ∈ N b k , this can be expressed as where, η k denotes the number of sub-systems in N b k with the highest age at time-step k. We also express that if i / ∈ N b k , then P[φ i, AoI k = 1] = 0. For pure random scheduling, we employ the common uniform randomization and we, therefore, have for all i ∈ N where, | · | represents the set cardinality operator and the superscript "R" in (24) stands for random scheduling policy. In the following, we analytically compare AoI-based vs. ET design for the decentralized sampling and will show (Section 3.3) that ET threshold-based sampling policy outperforms AoI-based counterpart if thresholds are appropriately designed. We, moreover, show that AoI sampling is in fact a more general form of periodic sampling with two differences, first, the transmission pattern may contain more than one fixed period, and second, the period(s) is a function of the number of sub-systems and the AoI thresholds. For the centralized scheduling process, we employ the AoI-based prioritizing policy of highest-age-first. In comparison with the pure random scheduling policy (Section 3.3), we show that the highest-age-first policy is not necessarily outperforming the pure random scheduling, if heterogeneity of sub-systems is not taken into account. We then propose the highest-age-first prioritization for the unstable sub-systems and show that this AoI-based policy is indeed capable of coordinating the communication resources more efficiently compared to the random scheduling, in sense of lower average sum of estimation errors of all sub-systems. We do not investigate the ET design as an applicable architecture for the scheduling policy since scheduling is a centralized process and decision making based on ET policies requires knowledge of real-time state information from all sub-systems which might not be preferred. It is, however, conjectured that if for certain small-size networked control scenarios ET policy might be favorable to be employed as the centralized scheduler, then it would even outperform AoI-based prioritizing scheduling due to its powerful capability of real-time prioritizing based on the current state of each single sub-system. The sampling/scheduling policy combinations that we address in this article, either analytically or in the simulation results, are summarized in Table 1.

Performance Analysis of the Joint Design
In this section, we propose two major co-design methodologies for the sampling and scheduling, where in the first method the sampling process is governed by an AoI threshold-based policy introduced in (21) and the scheduling is performed based on highest-age-first policy introduced in (23). In the second co-design the scheduling will be performed similarly based on the highest-age-first policy law in (23), while sampling process is controlled by the ET threshold-based policy shown in (20). We additionally consider periodic sampling policy and random scheduling, introduced in (22) and (24), respectively, as two conventional models for sampling and scheduling and provide comparisons, theoretically or numerically, with the proposed co-designs. For the purpose of brevity, we use the abbreviations "AoI/AoI", "ET/AoI", "ET/R", "AoI/R", "P/R", and "P/AoI" to denote the combined "sampling/scheduling" policy, see Table 1. To avoid confusion, it is worth reminding the difference between the AoI policies for decentralized sampling and centralized scheduling, see (21) and (23).

AoI Sampling and Scheduling Co-design
In the AoI/AoI co-design architecture, the AoI sampling is performed locally at every sub-system's sensor station according to the threshold-based policy (21), while the AoI scheduling is done in centralized fashion according to the highest-age-first prioritizing policy (23). Assume a NCS is comprised of a set of stable and a set of unstable sub-systems, denoted respectively by N s and N u , where N s ∪ N u = N, and N s = |N s | and N u = |N u | indicate the number of stable and unstable sub-systems, respectively. Here, we study the asymptotic sampling and transmission patterns for the AoI/AoI co-design for different values of the deterministic thresholds λ i , i ∈ N s and λ j , j ∈ N u . Let λ i < N and λ j < N. It is straightforward to conclude that each sub-system, either stable or unstable, will be scheduled for transmission once in every N time-steps with a fixed unique pattern. Moreover, the sampler of each stable sub-system will send N − λ i number of samples to the buffer in the same cycle of N time-steps, while unstable sub-systems send each N − λ j samples. We demonstrate this pattern for an illustrative example in below, and then summarize the concluding statements in the Proposition 1.
Illustrative example: Assume N s = N u = 3, c = 1, λ i = 5 and λ j = 2. Fig. 2 shows the sampling and transmission patterns of each sub-system, wherein, each circle (square) shows that a new measurement sample from a stable (unstable) sub-system is sent to the buffer. The red-bordered ones are the scheduled data packets and the numbers inside circles and squares denote the age of that corresponding sub-system at that time-step. According to (21), every unstable system j (denoted by US1, US2, US3 in Fig. 2) sends a fresh sample to the buffer at any time-step k at which ∆ j k−1 > 2. Hence, no data packet is injected to the buffer before time-step k = 4, at which all three unstable sub-systems will send a measurement sample to the buffer (see Fig. 2). Note that, at time-step k = 4, the samplers decide based on ∆ j 3 = 3 > 2. The same occurs for the stable sub-systems (denoted by SS1, SS2, SS3 in Fig. 2), hence, they all send their first measurement samples to the buffer at time-step k = 7, knowing that ∆ i 6 = 6 > 5. Since at time-step k = 4, there are three data packets all with identical highest ages, the AoI scheduler selects one of the three randomly, i.e., η 4 = 3 (see the second argument of(23)). This randomization is repeated again at the next time-step k = 5 now with only two data packets with similar ages belonging to US2 and US3 (US1 remains silent for the next two time-steps). At k = 6, there is only one data packet in the buffer and it is certainly scheduled as there is no competition for the single transmission resource. At time-step k = 7, there are 4 data packets belonging to SS1, SS2, SS3, US1. The data packet belonging to US1 will not be scheduled for transmission because it has a lower age compared to the other three. For the remaining ones with the same ages, one will be scheduled for transmission randomly (e.g., SS2 as in Fig. 2). At k = 8, random selection is done between only SS1 and SS3 since the existing data packets of US1 and US2 entail lower ages. Finally, at k = 9, SS1 is certainly scheduled for transmission as it has the highest age among all the data packets in the buffer. From this time-step forward, the same pattern of transmissions is repeated without any randomization.
As it is also illustrated by the above example, we state the following proposition for which we omit the lengthy but straightforward proof: Proposition 1. For the sketched heterogeneous NCS scenario, if c = 1, λ i , λ j < N, ∀i ∈ N s and ∀j ∈ N u , then the following statements hold, asymptotically: 1. each sub-system is scheduled a transmission once every N time-steps. 2. stable and unstable sub-systems send, respectively, N − λ i and N − λ j fresh samples to the buffer during every N time-steps. 3. if λ i = λ j = N − 1, then the AoI sampling is equivalent with the time-triggered sampling. Now assume that λ i , λ j ≥ N. We can express similar statements as in Proposition 1 and conclude that both stable and unstable sub-systems successfully transmit in asymptotic regime, respectively, every λ i + 1 and λ j + 1 time-steps, and they send only one sample to the buffer per each successful transmission. This is then clear that this scenario is also equivalent with the periodic transmission with periods of λ i + 1 and λ j + 1 for stable and unstable sub-systems, respectively.
If λ i ≥ N and λ j < N, the transmission pattern for each sub-system i ∈ N s is similarly periodic with time period of λ i + 1, and only one measurement sample is sent to the buffer per each transmission. For sub-systems j ∈ N u , however, the transmission pattern is not periodic with a unique period, i.e., the inter-transmission times vary between every two consecutive successful transmissions, if N ≤ λ i < 2N. In fact it changes between λ j + 1 and N for each j ∈ N u . When the inter-transmission time is λ j + 1, no data sample is discarded in between, while, when it is N, each sub-system j sends N − λ j number of samples per each transmission. In addition, if λ i → ∞, then every j ∈ N u successfully transmits every λ j + 1 time-steps during which each sub-system j sends max(1, N u − λ j ) number of samples to the buffer. The same can be said for λ j → ∞. These statements can be numerically tested by, for example, setting λ i = 7 in the depicted illustrative example in Fig. 2.
From the above discussions, we can make two crucial conclusions. First, AoI/AoI co-design policy governed by the AoI threshold-based sampling law (21) and AoI-based highest-age-first scheduling law (23) is not equivalent to the unique fixed periodic transmission policy, although, for some specific parameters, e.g., λ i = λ j = N − 1, they coincide. Second, all the possible transmission patterns are determined by the capacity constraint (4), the AoI thresholds λ i and λ j and the number of network sharing sub-systems, that are all constants. Hence, the resulting transmission patterns are insensitive w.r.t. the dynamics of stable or unstable sub-systems. We may design the AoI thresholds λ i and λ j differently for stable and unstable sets of sub-systems, however, they are assumed to be constant parameters and not adjusted by changing the dynamics 1 .

ET Sampling and AoI Scheduling Co-design
In this section, we study the ET/AoI co-design architecture, where the sampling is locally performed according to the ET law (20) and the scheduling is centrally governed by the highest-age-first law in (23). For the clarity of analysis and illustrative purposes, we first assume that the ET thresholds in (20) are deterministic and constant, i.e., r i k = r i ∈ R ≥0 . We discuss in the next section how to extend the performance results to the ET sampling with stochastic thresholds.
Since the estimation error e i k is a random variable (see (18) and (19)), there is generally no fixed pattern for transmission of each sub-system when sampling is controlled by the ET law in (20). Hence, 1 The discussions of the Section 3.1 can be extended to cover the scenarios that the AoI thresholds are not identical within the set of stable or unstable sub-systems, i.e., if i, l ∈ N u , then λ i = λ l . Although this leads to more complex transmission patterns, it does not contradict the crucial conclusions of this section, as summarized in the last part of the Section 3.1.
we study the asymptotic transmission rate for which we try to find mathematical expressions or bounds. To do that, we first compute the asymptotic sampling rate for an arbitrary sub-system i ∈ N, as follows: where the inequality in (25) is obtained using Markov's inequality knowing e i k 2 2 is a non-negative random variable and r i is a non-negative constant. To provide more meaningful bound, we first state the following Lemma which essentially states that the dynamics of a stable sub-system's estimation error variance becomes insensitive to closing the feedback loop, asymptotically.

Lemma 1. For any stable LTI stochastic control system modeled by (1), the estimation error variance is asymptotically bounded regardless of how often the feedback loop is closed.
Proof. From (19), we can express the estimation error at the sampler's side, assuming that no transmission has taken place from the initial time until the current time k, i.e., φ i where the second equality holds sinceẽ i According to the last expression, e i k is zero-mean, hence, we can compute the asymptotic estimation error variance as where, to obtain (26), we used the mutual statistical independence of x i 0 , v i k and {w i 0 , w i 1 , . . . , w i k }, and the third expression is derived using the sub-multiplicative property of vector norms and also knowing that lim k→∞ A k i = 0, since A i is Hurwitz. The final bound (27) is obtained knowing that the infinite series lim k→∞ ∑ k r=1 A i Having Lemma 1, we can then re-express (25) for all stable sub-systems i ∈ N s , as follows: It should be noted that the expression (25) holds for both stable and unstable sub-systems, while (28) is valid only for the former ones. Moreover, the bound (28) becomes trivial if the right hand side of the inequality is bigger than one, so this might also be seen as a rule to design the threshold r i . It is clear that the higher r i is, the lower the transmission rate of stable sub-systems becomes, which is expected.
Lemma 1 does not apply to unstable sub-systems, hence, to derive similar upper-bound for sub-systems j ∈ N u , we compute lim k→∞ E[ e j k 2 2 ] according to the estimation error expression (18), as in the following: where, P j is the asymptotic estimation error covanrince and will be obtained from the following algebraic Riccati equation Finally, having (30), we can express the upper-bound in (25) for the unstable sub-systems j ∈ N u as Note that, if lim k→∞ ∆ j k−1 ≤ M j holds for finite M j , P j ∞ will also be finite as the Kalman filter receives state measurements asymptotically to compute the MMSE error. Even though the received measurements might not be fresh, the delay is finite and the Kalman filter algorithm converges.

Performance Comparisons
To conduct asymptotic performance analysis, we consider a heterogeneous NCS comprised of multiple stable and unstable sub-systems. Note that, for a NCS including all stable sub-systems asymptotic performance becomes independent of the sampling, scheduling and control policies due to the natural convergence of states. In fact, for a stochastic system of the form (1), the system states are expected to converge asymptotically to a bounded set around the origin where the set boundary is characterized by the moments of the primitive random variables, as we will show later in this section. Hence, to study the properties of the co-design we focus on the heterogeneous NCS scenario.
The overall asymptotic performance of the NCS is measured by the average cost functions J introduced in (15). It can be seen from (A3) that the local LQG cost function J i can be minimized by the control law γ i in an inner optimization problem and then the residual cost becomes a function of the sampling law ξ i . From (A8), we know that the residual local cost is a function of E[x i k |Ĩ i k ] and ψ i k where the latter is shown in (A15) to be a function of the estimation errorẽ i t and its variance P i t , t ≥ k. This should be remembered, however, that the resource constraint (4) does not allow the transmissions to be solely determined by the sampling law ξ i , and therefore, the local cost J i , from the overall perspective of control and network layers, becomes dependent also on the scheduling law π. The dependency appears inĨ i k which eventually affects both the estimate E[x i k |Ĩ i k ] and ψ i k . Hence, the optimal asymptotic cost function (15) should be minimized by both sampling and scheduling policies, which can be written as wherein, E[x i 0 ] is known a priori, and ψ i 0 is a function of the estimation errorsẽ i k and their variances P i k , k ∈ {0, 1, 2, . . .}, according to (A15).
Solving the optimization problem (32) is very challenging due to the coupling of the decision variables with ψ i 0 through the end-to-end delay ∆ i t and also the non-linear nature of the ET and AoI functions. The aim of this article is, therefore, to identify the appropriate class of policies for the sampling and scheduling that jointly result in an improved overall performance. Since the overall performance is a convex function of the estimation error according to (32), it is easier to study the asymptotic behavior of the estimation error of all sub-systems. In fact, if a certain co-design of sampling and scheduling policies results in a lower asymptotic average sum of estimation errors of all sub-systems, compared to another co-design, then it certainly results in a lower asymptotic overall cost (32) as well. Hence, for the performance analysis and make comparisons between different sampling/scheduling co-designs, we consider the following performance metric: Theorem 2 summarizes the second main result of this article on the appropriate sampling/scheduling co-design architecture. First, we define "non-trivial threshold-based functions", as follows:

Theorem 2. Consider a heterogeneous NCS comprised of N LTI stochastic sub-systems modeled as (1)-(2)
from which N s sub-systems are stable and N u sub-systems are unstable. Let the network scheduler select only one sub-system per time-step to transmit its freshest state measurement in the buffer to the controller, i.e., c = 1 in (4). Then, for any non-trivial AoI sampling policy given in (21), there exists a non-trivial constant threshold ET sampling policy in form of (20) that asymptotically outperforms, in terms of (33), the AoI sampling policy, given that network scheduling policy obeys the AoI-based highest-age-first law in (24).

Remark 6.
Reminding the evolution of e j k in (18) and (19), it is clear that the estimation error at the sampler has a zero-mean but not normal distribution. For general square matrix A j ∈ R n j ×n j , the asymptotic CDF of In the following, we discuss that the pure AoI scheduling policy (23) may outperform the pure random transmission policy (24), but not always. In fact, we discuss that if the scheduler's highest-age-first prioritizing feature is applied first to the set of unstable sub-systems, then the AoI scheduling policy (23) certainly outperforms the pure random transmission policy. We define the highest-age-first policy for the unstable sub-systems similar to (23), with the exception that the law is applied, asymptotically, first on the set of unstable sub-systems and the resource is assigned to the unstable sub-system with the highest age, even if there are stable ones with higher age than the unstable ones.

Corollary 1.
For a fixed sampling policy, the highest-age-first threshold-based scheduling law (23) does not necessarily outperform the pure random scheduling policy (24), asymptotically, in an NCS of heterogeneous stable and unstable control sub-systems sharing a capacity limited communication network. The AoI-based highest-age-first policy for unstable sub-systems, however, asymptotically outperforms the pure random scheduling policy (24).
Proof. As discussed before, the average sum of the estimation error variance of the set of stable sub-systems do not asymptotically change. Reminding (13), we see that the higher the age ∆ i k is for unstable sub-systems, the larger the estimation error becomes. This is also true for the variance of the estimation error. Therefore, if a scheduling policy results in a higher transmission probability for the unstable sub-systems with the highest age, then the average sum of the estimation error variance will also be more reduced. According to (24), all sub-systems that have a data packet in the buffer are assigned identical probabilities of transmission 1 , irrespective of their age or stability properties.
For the same set of sub-systems with a packet in the buffer, the probability that the sub-system with the highest age, stable or unstable, successfully transmits is, according to (23), 1 η k , if there are η k ≤ |N b k | number of sub-systems all with identical highest age, which leads to 1 The equality occurs only if all sub-systems in the buffer have the similar age which is also the highest age. It should, however, be noted that if all the sub-systems that have the highest age are stable, then the unstable sub-systems in the buffer that may have relatively large age but not the highest are assigned with probability zero for successful transmissions, while this probability is 1 for the pure random scheduling policy that leads to a lower average sum of estimation error variance. With the modified prioritized highest-age-first policy for unstable sub-systems, however, the described problem can be easily considered in scheduling, and therefore, this policy always outperforms the pure random scheduling, for any fixed sampling policy.

Numerical Evaluations
We consider different NCS setups with different number of stable and unstable sub-systems to numerically test the co-design architectures and compare with the common approaches. Number of sub-systems N is chosen from the set {2, 4, 6, 8, 10} with equal number of stable and unstable sub-systems. For the ease of interpretation, we choose scalar LTI sub-systems. The system matrices for stable and unstable sub-systems are selected to be 0.5 and 1.05, respectively. The system disturbance is modeled as w i k ∼ N (0, 1), for all i ∈ N, and k ∈ N 0 , and for the ease of illustrations we assume that measurements are noiseless. Each data point in the plots is generated by running the simulative setup for 10 6 iterations. In the following we define the sampling and network scheduling strategies and the parameter values chosen for each scenario.  [2,30] that are divisible by 2.
We use a default setting where each sampler uses the same mean threshold. 2. Period-n sampling: Each sampler samples the plant periodically with period n. 3. AoI sampling: Each sampler samples the plant whenever the AoI at the sampler exceeds N − 1.
The AoI at the sampler is equal to the AoI at the respective estimator from the previous time-step plus one.

Network scheduling
1. Max AoI: Under this scheduling policy, the network chooses the plant which has a packet in the queue and maximum AoI at the estimator, which is the highest-age-first policy. 2. Pure randomized: The network scheduler chooses a packet uniformly randomly from the queue.
In Figure 3, we plot the average estimation error variance (across all sub-systems) by varying the mean threshold. We observe that the estimation error variance is minimized for certain mean threshold values which increase with the number of sub-systems N. To understand this, note that when the thresholds are small, all the samplers will place a packet in the queue in almost every time-step. In this case, Max AoI does close to round-robin scheduling for the plants. Thus, plants with high or low estimation errors are treated rather indifferently leading to relatively high estimation error variance. The threshold values that attain minimum estimation error variances are such that the sub-systems with low estimation errors (usually the stable ones) do not contend for the network frequently as they do not exceed the thresholds frequently. This results in more often transmission of packets from sub-systems with high estimation errors, thus lowering the overall estimation error variance. In the following figures, we present the statistics of the event triggered sampling at mean thresholds that minimize the estimation error variance, that are marked by black circles in Figure 3.  In Figure 4, we compare the estimation error variance achieved under different schemes. For event triggered sampling, we plot the minimum estimation error variance achieved over different thresholds for each N. We observe that event triggered sampling strategy obtains the lower estimation error variance, 30 − 40% lower than that of AoI sampling when N = 10. While AoI sampling and period-1 sampling results in same variance, the sampling frequency of AoI sampling is much lower and equals 1 N−1 . Also, it can be observed that, in general, using Max AoI scheduling results in lower estimation error variance especially as the number of sub-systems grows.
In Figure 5, we compare the normalized total number of network transmissions that occur under different schemes. While period-1 sampling and AoI sampling result in a transmission in each time-step, event triggered results in transmissions 80% of the time for varying number of sub-systems. This is because, the queue remains empty 20% (on average) under event triggered sampling since only sub-systems with estimation errors greater than the threshold are allowed to place a packet in the queue. Therefore, event triggering not only provides lower estimation error variance, but also reduces the number of network transmissions.  In Figure 6, we compare the average AoI (averaged over all the estimators), achieved under different schemes. Since AoI sampling samples a plant based on its AoI at the estimator, this strategy results in the lowest average AoI. On the other hand, event triggering results in higher average AoI, as it samples based on estimation error, which increases non-linearly with AoI. Also, since AoI Max scheduling picks the plant with highest AoI and transmits its packet, this strategy results in lower average AoI across different sampling strategies. The main conclusion is, although ET sampling policy does not result in the lowest average AoI across the NCS, it results in the lowest achieved estimation error variance.

Conclusions
In this article, the major goal is to propose a co-design networked control architecture of sampling, scheduling and control for NCSs comprised of multiple heterogeneous LTI stochastic control systems that close their sensor-to-controller loops over a shared capacity-limited communication network. We first show that under mild assumptions on the information structure of each policy maker, the optimal control law is of certainty equivalence form. We then investigate various combinations of decentralized sampling and centralized scheduling architectures employing the applicable concepts of event-triggered and AoI utility functions. We analytically show that the event-triggered sampling is capable of asymptotically outperforming AoI sampling policy when the communication resources are limited, while we demonstrate AoI-based prioritizing scheduling may outperform the pure random scheduling policy under appropriate prioritization metric. To discuss the effectiveness of each co-design, we measure the overall NCS performance by the average sum of local LQG cost functions. Our theoretical analyses are successfully validated for the proposed co-designs and comparisons are made with conventional periodic and pure random access approaches through simulations on different NCS scenarios.
From (8) and (10), we know that I i k ⊂Ĩ i k , therefore, we can re-write (A1) by employing the general law of total expectation 3 , as Define the LQG cost-to-go at time-step k as We then have from (A2) We, moreover, define the optimal stage cost J * i (k) as follows: which results in the compact form J * i = E[J * i (0)]. The LQG optimal cost-to-go at time-step k + 1 has the following form: Knowing thatĨ i k ⊂Ĩ i k+1 , we have from the law of total expectation that Having (A6), we obtain Let us assume that V i, * k can be expressed in the following form: where ψ i k is a control-input-independent expression. We will show later in this proof that (A8) is indeed an authentic assumption. According to (A8), we can re-express (A7) as follows: We have for the a priori state estimate where, for the first equality we use the conditional expectation property of E[XY|H] = X E[Y|H] if X is H-measurable, and the second equality holds according to the law of total expectation knowing , together with knowing ǫ i k+1 and ψ i k+1 are independent of control inputs, we can re-write (A9) as Since the last term after the equality above is u i k -independent, finding the optimal control u i * k is straightforward and can be obtained by setting the derivative of the first two terms in (A11) w.r.t. u i k to zero, which results in (16) will be obtained. We still need to show that ψ i k+1 is indeed independent of control inputs. By plugging in (16) in the optimal cost-to-go (A11), and also using x i k =ẽ i k +x i k − v i k (see (11)), we have Now, comparing (A12) with (A8), we conclude the two following statements: From the definitions ofẽ i k , and ǫ i k , and using u i k = L i k E[x i k |Ĩ i k ], we obtain the following: From the above expression, therefore, we obtain According to Lemma 1, for stable sub-systems, we know that the expression (26) converges to a constant value that depends only on A i , Σ v i , Σ w i . The exact constant expression equality for non-scalar systems, however, is non-trivial to derive. For scalar systems, i.e., if 0 < A i < 1, then the inequality in (27) becomes equality. For non-scalar case, though, we can use Cauchy-Schwarz inequality to find a constant upper-bound. What we need for the proof of Theorem 2 is not the exact expression for lim k→∞ E ẽ i k 2 2 , but only knowing that lim k→∞ E ẽ i k 2 This equality is clear from (26) since A i 2 2 < 1 and this diminishes the role of time k in the expression for the estimation error variance. This essentially concludes that, for the set of stable sub-systems, transmissions in asymptotic regime do not influence the estimation error variance, and hence, (A16) can be rewritten as The problematic term in the above expression that may lead to increase the asymptotic average estimation error variance is E ẽ j k 2 2 φ j k = 0 while ẽ j k 2 2 > r i . Hence, the aim of the co-design policy is to increase lim k→∞ P(φ j k = 1) which consequently leads to a decrease in lim k→∞ P(φ i k = 1). Simply, we would like to assign the transmission opportunities more often to the unstable sub-systems, asymptotically. In addition, we are interested in not only a successful transmission, but a successful transmission of a low-age state measurement. This means more frequent sampling and more frequent scheduling of unstable sub-systems, in probabilistic sense. To achieve this, we should first notice from the statements of the Proposition 1 and the discussions afterwards in Section 3.1 that, in the non-trivial AoI/AoI co-design architecture, the minimum sampling rate of stable or unstable sub-systems with the AoI threshold λ i is lim k→∞ P(δ i k = 1) = 1 max{N,λ i +1} = 1 M i . To have a higher sampling rate for the ET sampling law compared to the AoI sampling law, we need to show lim k→∞ P(δ i k = 1|i ∈ N s , ET/AoI) < lim k→∞ P(δ i k = 1|i ∈ N s , AoI/AoI).
Hence, using (28), the inequality (A18) is satisfied if which results in the following lower-bound for the ET thresholds for stable sub-systems: For unstable sub-systems, we need to show lim k→∞ P(δ j k = 1|j ∈ N u , ET/AoI) > lim k→∞ P(δ j k = 1|j ∈ N u , AoI/AoI).
We know lim k→∞ P( e j k 2 2 > r j ) = 1 − F j e j (r j ), where F j e j is the asymptotic cumulative distribution function (CDF) of the random process e j k 2 2 and F j e j (r j ) is the value of the asymptotic CDF at r j . Hence, (A21) is satisfied if The CDF F j e j is a monotonically non-decreasing function w.r.t. r j , hence, the lower M j is (i.e., either lower N or lower λ j ), the ET thresholds for unstable sub-systems should also be decreased to asymptotically out-sample the AoI/AoI architecture, and vice-versa, which is an expected conclusion. Having (A19) and (A22) satisfied, it is ensured that, first, the asymptotic sampling rate of stable sub-systems is lower in the ET/AoI co-design compared to the AoI/AoI, and second, the sampling rate of unstable sub-systems is higher for the former approach. Hence, not only the probability that the unstable sub-systems transmit is higher for the ET/AoI compared to the AoI/AoI policy, but also the scheduled transmissions that are determined by the AoI-based highest-age-first policy in (23) have lower average age for the ET/AoI co-design. This means, lim k→∞ P(φ j k = 1|j ∈ N u , ET/AoI) > lim k→∞ P(φ j k = 1|j ∈ N u , AoI/AoI) and therefore, lower asymptotic average estimation error variance in (A17). Finally, knowing that the asymptotic behavior of the stable set of sub-systems are independent of the sampling and scheduling policies, the asymptotic average estimation error variance in (A17) can be upper-bounded as follows: The bound is trivial if A i 2 2 = 1 or P j ∞ → ∞. The first one is avoided due to assuming A i is Hurwitz, and P j ∞ is bounded due to the fact that if lim k→∞ e j k 2 2 > r j asymptotically, then ∆ j k ≤ M j which means, in the worst case, there is one state information with bounded delay to construct the Kalman estimate. This completes the proof.