MASA: Multi-Application Scheduling Algorithm for Heterogeneous Resource Platform

Abstract

Heterogeneous architecture-based systems-on-chip enable the development of flexible and powerful multifunctional RF systems. In complex and dynamic environments where applications arrive continuously and stochastically, real-time scheduling of multiple applications to appropriate processor resources is crucial for fully utilizing the heterogeneous SoC’s resource potential. However, heterogeneous resource-scheduling algorithms still face many problems in practical situations, including generalized abstraction of applications and heterogeneous resources, resource allocation, efficient scheduling of multiple applications in complex mission scenarios, and how to ensure the effectiveness combining with real-world applications of scheduling algorithms. Therefore, in this paper, we design the Multi-Application Scheduling Algorithm, named MASA, which is a two-phase scheduler architecture based on Deep Reinforcement Learning. The algorithm is made up of neural network scheduler-based task prioritization for dynamic encoding of applications and heuristic scheduler-based task mapping for solving the processor resource allocation problem. In order to achieve stable and fast training of the network scheduler based on the actor–critic strategy, we propose optimization methods for the training of MASA: reward dynamic alignment (RDA), earlier termination of the initial episodes, and asynchronous multi-agent training. The performance of the MASA is tested with classic directed acyclic graph and six real-world application datasets, respectively. Experimental results show that MASA outperforms other neural scheduling algorithms and heuristics, and ablation experiments illustrate how these training optimizations improve the network’s capacity.

1. Introduction

With the rapid development of software-defined technologies and hardware architectures, next-generation terminal systems are possible that can simultaneously implement communication, radar, timing, and other radio-frequency (RF) functions [1]. Heterogeneous System-on-Chip (SoC) integrates General Purpose, Special Purpose, and other types of processor resources (FPGAs, Accelerators, etc.) on a single chip [2], realizing a balance between performance and energy and enabling the development of flexible and powerful multifunction RF systems. As the complexity of RF functions increases, heterogeneous SoCs that power multifunctional RF systems must meet stringent performance requirements before deployment. For example, RF systems in autonomous vehicles must simultaneously process multiple applications under highly dynamic environmental conditions to accomplish their missions safely [3,4]. Therefore, how to efficiently schedule multiple strong real-time applications onto a limited heterogeneous resource system is the key to fully utilize the potential of heterogeneous SoC, as well as to enhance the flexibility and scalability of the terminal’s RF system.

Efficient scheduling of applications on heterogeneous resource requires complex combinatorial optimization algorithms. The prime objective of scheduling algorithms is to optimize the deployment of structurally different RF applications, such as communications and radar, by assigning the application’s different functional modules to the appropriate Processing Elements (PEs) in the heterogeneous resources. Meanwhile, intra-application dependencies are generally described by the form of a directed acyclic graph (DAG) that illustrates the relationship between tasks in a point-to-point format, and these tasks can organize fine-grained parallel computation on different PEs. Tasks and PEs should be matched as closely as possible. If a processor with low computing capability is assigned to a task with high computational demand, it could easily cause the task to become a computational bottleneck for the system. Conversely, if a fast processor is assigned to a task with low computational demand, the processor’s efficiency cannot be guaranteed. Furthermore, in comparison to scheduling a single application, the task of runtime scheduling for multiple applications becomes significantly more complicated. To address this issue, this paper to study a multi-application scheduling algorithm designed for heterogeneous resources.

Static scheduling algorithms generally generate optimal schedules for individual applications on a resource system. However, compared to the scheduling of a single application, the processing of multiple applications for heterogeneous runtime scheduling is more complex. Currently, the neural network approach has proved its advancement in resource management in cloud computing, edge computing and other environments [5,6,7,8,9], so this paper draws on its ideas to apply neural network scheduling algorithms to the resource management problem of terminal heterogeneous resources. Therefore, for the characteristics of continuous stochastic arrival of applications, interdependence among application and terminal resource heterogeneity in multifunctional RF systems, this paper designs a Multi-Application Scheduling Algorithm (MASA) based on Deep Reinforcement Learning (DRL) to minimize the average application execution time (denoted as avg. JET) as the optimization global objective.

The main contributions of this paper are summarized as follows:

• Designing a DRL-based multi-application scheduling algorithm-MASA, which consists of two parts: a neural network scheduler (NN-Scheduler) and a Heuristic scheduler, to solve heterogeneous resource management in a dynamic environment.
• Adopting a self-attention mechanism [10] in NN-Scheduler to realize feature vector extraction for multiple applications, and a hybrid allocation algorithm DEFT in Heuristic scheduler to reduce the idle time of PE and improve the actual performance of scheduling.
• Proposing optimizing methods for network training with respect to the practical problems of terminal heterogeneous resources. The Reward Dynamic Alignment (RDA) method is proposed to obtain the correct empirical data, initial episodes early termination method and asynchronous multi-agents joint training method to improve the speed and utility of network training.

The rest of the paper is organized as follows. Section 2 presents research related to the task scheduling algorithm. Section 3 introduces the multi-application scheduling scenario and defines the average application execution time minimization problem. Section 4 proposes the multi-application scheduling algorithm-MASA and presents RDA, early termination of the initial episodes and asynchronous multi-agents joint training optimization methods. Simulation results of the proposed algorithm are discussed in Section 5 to validate its performance. Finally, the paper is summarized in Section 6.

2. Related Work

The complexity of DAG scheduling has been demonstrated to be NP-complete [11], and its complexity is greatly increased when PE resources are heterogeneous. Scheduling strategies can broadly be classified into two types: dynamic and static DAG scheduling. In static DAG scheduling, the DAG information is known in advance (at the beginning of the scheduling process). Conversely, the DAG topology in dynamic scheduling is unknown in advance, and all configuration information is obtained at runtime. Scheduling decisions are then made in real time.

Most current static scheduling algorithms on heterogeneous resources aim to optimize the time required to execute a single DAG. For example, the Mixed Integer Programming approach uses objective functions and constraints to formulate the task scheduling problem, exchanging computational and time complexity for an optimal solution [12]. Genetic algorithms and particle swarm algorithms based on heuristic stochastic search algorithms can find the optimal value of the parameter set through a large number of iterative experiments, and such algorithms can be applied to multi-objective optimization problems [13,14,15,16]. Heuristic-based list task scheduling algorithms demonstrate their high execution capabilities in a heterogeneous processor resource. HEFT (Heterogeneous Earliest Finish Time) is the dominant algorithm for heterogeneous static list scheduling [17]. HEFT assigns priority to each task based on the critical path length, and later assigns PEs to tasks with the highest priority on the list based on the earliest finish time (EFT),mapping the task to the active PE with the minimum EFT value; in addition, HEFT uses a scheme that tries to insert tasks into the PE’s idle time slots, and CPATH is a critical path aware dynamic scheduler for heterogeneous systems that prioritizes tasks in the DAG based on the lowest-cost longest-path approach, submitting high priority tasks to high frequency PEs and low frequency PEs when work is enabled. This scheduling approach does not consider communication costs for tasks [18]. These algorithms have been further expanded and extended in recent years and applied to the scheduling of real hardware platforms, such as FPGA [19].

Dynamic scheduling algorithms for multiple DAGs are also investigated, and many machine learning based scheduling algorithms for dynamic scheduling scenarios are described below. Xie G et al. proposed an adaptive scheduler (ADS) based on RL with the aim of improving reliability and minimizing completion time. However, ADS dynamically schedules DAGs with higher-level DAGs from the priority level of DAGs alone, and the large computational granularity is not suitable for radio application streaming analysis [3,20]. Biao H et al. investigated how to dynamically schedule dynamic applications on heterogeneous embedded systems and proposed energy-efficient scheduling algorithms for the optimization of the system energy consumption problem. However, this paper uses an objective function and a constraint formulation to design the scheduling problem, which has a relatively high time complexity [21]. Tegg T et al. proposed SoCRATE, a task scheduling scheme for the domain of System-on-a-Chip based on the DRL algorithm. SoCRATE employs an integrated network structure to generate scheduling decisions. This results in a higher level of complexity in the neural network’s decision space, which prevents the neural network from converging to a steady state more quickly [22]. Amarnath A et al. designed a heterogeneous-aware multilevel scheduler, HetSched, which utilizes information about the runtime state of the underlying heterogeneous SoCs as well as the real-time nature of the application to meet the growing throughput requirements of self-driving cars. However, it takes into account the task security level and the granularity of task division is large, which is not adapted to the RF application environment [4]. Mao H et al. proposed the Decima algorithm to solve the task scheduling problem in the field of cloud computing with a combination of graph neural network and actor–critic training strategy, but Decima is used to solve the isomorphic resource-scheduling problem and is not applicable to the scheduling problem of heterogeneous resource systems [23].

3. A Multi-Application Scheduling Formal Definition

Resource management for terminal heterogeneous SoCs is a domain-specific resource management problem, and the following are some of the definitions that will be used later.

Definition 1. 

An RF application consists of multiple specific function modules, such as Fast Fourier Transforms (FFTs), encoders and decoders, and other computationally intensive software components. In this paper, an application (WiFi-TX/RX, Range Detection, etc.) is defined as a job, and a function module is defined as a task. The terms task and node are used interchangeably throughout this paper.

Definition 2. 

Task scheduling includes task prioritization and task mapping, where task mapping describes the process of assigning PEs to tasks.

3.1. Scheduling Scenario Description

RF signal-processing applications are mostly computationally intensive tasks, and assigning the task to a suitable PE is an essential step, e.g., the FFT can be processed 10 times faster on a hardware accelerator than on a GPP. If directly using the neural network to process both job DAGs and PE information and generate executable tasks and available PE pairs, the complexity of the neural network’s decision space is exponentially increased, preventing the neural network from converging to a steady state. Therefore, in this paper, we design a two-phase scheduling algorithm MASA to reduce the complexity of the decision space, i.e., task prioritization and task-mapping phases, as shown in Figure 1. Firstly, in the task prioritization phase, the NN-Scheduler mainly processes the information of job DAG without considering the information of PE. Second, in the task-mapping phase, the Heuristic Scheduler makes decisions based only on the identified high-priority tasks and PE information.

The NN-Scheduler sorts the tasks in the multi-DAG, and the high priority tasks are moved to the ready queue to wait for execution, while the low priority tasks waiting for pretasks to be completed are left in the outstanding queue. The hardware database includes static profiles (PE type, estimated computational execution time for each task, maximum number of tasks to be processed by the PE) and dynamic profiles (PE utilization, status of the PE as busy or idle) of the PE. In each mapping phase, ready tasks are deployed to available PEs according to the Heuristic Scheduler and assigned to executable tasks after waiting for the PEs to become idle. After PEs have processed the executable tasks, the SCHEDULER removes the dependencies of the tasks and updates the status information of the tasks and the PEs. After the above is completed, a new scheduling decision phase is triggered and the NN-Scheduler moves the outstanding tasks to the ready queue if all of their predecessors have been deleted. After each scheduling decision is completed, SCHEDULER updates the environment state to continue in subsequent decision phases until all waiting tasks in the outstanding queue are removed.

3.2. Optimization Objective

The simulation environment for heterogeneous resources includes $\mathrm{m}$ independent jobs, referred to as $\mathrm{J}=\left[{\mathrm{j}}_1,{\mathrm{j}}_2,\dots ,{\mathrm{j}}_{\mathrm{m}}\right]$, and $\mathrm{K}$ heterogeneous processing elements (PE), referred to as $\mathrm{P}=\left[{\mathrm{p}}_1,{\mathrm{p}}_2,\dots ,{\mathrm{p}}_{\mathrm{K}}\right]$. It is necessary to investigate the impact of randomly arriving jobs (streaming mode) on the scheduling algorithm in different mission scenarios of varying scales, thus the $scale$ shown in Algorithm 1 is an adjustable parameter value.

Algorithm 1. Flowchart of Heterogeneous Resource Simulation Environment

Input: Job’s profile $\mathrm{J}=\left[{\mathrm{j}}_1,\dots ,{\mathrm{j}}_m\right]$, PE’s profile $\mathrm{P}=\left[{\mathrm{p}}_1,\dots ,{\mathrm{p}}_K\right]$, Maximum simulation length CLK, Job Queue capacity maximum C, job inject interval’ expectation $Scale$ Output: Average execution time for multi-application (avg.JET) for each episode do   clk=0   ${\mathrm{clk}}_{\mathrm{inj}}~\exp \left(\frac{1}{Scale}\right)$   repeat     if Job Queue capacity maximum not reached       Inject job into the Job Queue every ${\mathrm{clk}}_{\mathrm{inj}}$     end if     for all tasks for each job in the job Queue do       Task coding based on feature networks       Task ordering based on policy network       for Ready Task do         Mapping task to PE based on heuristic algorithm       end for     end for     $\mathrm{clk}\leftarrow \mathrm{clk}+1$   until clk→CLK   Computer avg.JET using Equation (5) end for

Each job is modeled as a DAG, denoting the job DAG as $\mathrm{G}\left\{\mathrm{N},\mathrm{E}\right\}$, in which each task ${\mathrm{n}}_{\mathrm{i}}\in \mathrm{N}$ represents the software components that composed the application, and the directed edge ${\mathrm{e}}_{\mathrm{i},\mathrm{j}}\in \mathrm{E}$ is the link from task ${\mathrm{n}}_{\mathrm{i}}$ to ${\mathrm{n}}_{\mathrm{j}}$. For any ${\mathrm{n}}_{\mathrm{i}}\in \mathrm{N}$, the parent node has a higher priority than the child nodes. $\mathrm{comp}\left({\mathrm{n}}_{\mathrm{i}},{\mathrm{p}}_{\mathrm{m}}\right)$ represents the computational execution time of task ${\mathrm{n}}_{\mathrm{i}}$ on ${\mathrm{p}}_{\mathrm{m}}$, as provided by the profile of the heterogeneous resource PE. $\mathrm{comm}\left({\mathrm{n}}_{\mathrm{i}},{\mathrm{n}}_{\mathrm{j}}\right)$ represents the communication delay from ${\mathrm{n}}_{\mathrm{i}}$ to ${\mathrm{n}}_{\mathrm{j}}$, as shown in Equation (1). ${\mathrm{w}}_{\mathrm{i},\mathrm{j}}$ is the transmission data volume between ${\mathrm{n}}_{\mathrm{i}}$ and ${\mathrm{n}}_{\mathrm{j}}$, task ${\mathrm{n}}_{\mathrm{i}}$ is mapped to processor ${\mathrm{p}}_{\mathrm{m}}$, denoted ${\mathrm{p}}_{\mathrm{m}\left({\mathrm{n}}_{\mathrm{i}}\right)}$. $\mathrm{B}\left({\mathrm{p}}_{\mathrm{m} \left({\mathrm{n}}_{\mathrm{i}}\right)},{\mathrm{p}}_{\mathrm{n} \left({\mathrm{n}}_{\mathrm{j}}\right)}\right)$ denotes the bandwidth between ${\mathrm{p}}_{\mathrm{m}}$ and ${\mathrm{p}}_{\mathrm{n}}$. For task ${\mathrm{n}}_{\mathrm{i}}$ and its parent node $\mathrm{pred}\left({\mathrm{n}}_{\mathrm{i}}\right)$, assignment to different PEs incurs communication overhead, and frequent switching of PEs leads to an increase in task execution time.

$$\mathrm{comm}\left({\mathrm{n}}_{\mathrm{i}},{\mathrm{n}}_{\mathrm{j}}\right)=\underset{{\mathrm{n}}_{\mathrm{j}}\in \mathrm{pred}\left({\mathrm{n}}_{\mathrm{i}}\right)}{\max }\frac{{\mathrm{w}}_{\mathrm{i},\mathrm{j}}}{\mathrm{B}\left({\mathrm{p}}_{\mathrm{m}\left({\mathrm{n}}_{\mathrm{i}}\right)},{\mathrm{p}}_{\mathrm{n}\left({\mathrm{n}}_{\mathrm{j}}\right)}\right)}$$

(1)

The earliest start time (EST) and earliest finish time (EFT) of task ${\mathrm{n}}_{\mathrm{i}}$ on processor ${\mathrm{p}}_{\mathrm{m}}$ are given by Equations (2) and (3), where $\mathrm{pred}\left({\mathrm{n}}_{\mathrm{i}}\right)$ is the parent node of task ${\mathrm{n}}_{\mathrm{i}}$, $\mathrm{avail} \left[{\mathrm{p}}_{\mathrm{m}}\right]$ denotes the earliest time that ${\mathrm{p}}_{\mathrm{m}}$ can be used to perform the task, and $\mathrm{AFT} \left({\mathrm{n}}_{\mathrm{j}}\right)$ is the time that ${\mathrm{n}}_{\mathrm{j}}$ actually finishes.

$$\mathrm{EST}({\mathrm{n}}_{\mathrm{i}},{\mathrm{p}}_{\mathrm{m}})=\max \left\{\mathrm{avail}\left[{\mathrm{p}}_{\mathrm{m}}\right],\underset{{\mathrm{n}}_{\mathrm{j}}\in \mathrm{pred}\left({\mathrm{n}}_{\mathrm{i}}\right)}{\max }\left(\mathrm{AFT}\left({\mathrm{n}}_{\mathrm{j}}\right)+\mathrm{comm}({\mathrm{n}}_{\mathrm{j}},{\mathrm{n}}_{\mathrm{i}})\right)\right\}$$

(2)

$$\mathrm{EFT}({\mathrm{n}}_{\mathrm{i}},{\mathrm{p}}_{\mathrm{m}})=\mathrm{comp}({\mathrm{n}}_{\mathrm{i}},{\mathrm{p}}_{\mathrm{m}})+\mathrm{EST}({\mathrm{n}}_{\mathrm{i}},{\mathrm{p}}_{\mathrm{m}})$$

(3)

The true execution time of a completed application $\mathrm{j}\in {\mathrm{J}}_{\mathrm{comp}}$ is $\mathrm{exec} \left(\mathrm{j}\right)$, given by Equation (4), where ${\mathrm{n}}_{\mathrm{exit}}$ is the exit node of application and $\mathrm{AFT} \left({\mathrm{n}}_{\mathrm{exit}}\right)$ is the actual completion time of the exit node.

$$\mathrm{exec}\left(\mathrm{j}\right)=\max \left\{\mathrm{AFT}({\mathrm{n}}_{\mathrm{exit}})\right\}$$

(4)

In this paper, the final objective of the scheduling algorithm is to minimize the average execution time of multiple applications (avg.JET) as shown in Equation (5). Where ${\mathrm{J}}_{\mathrm{comp}}$ is the set of completed job DAGs and ${\mathrm{J}}_{\mathrm{comp}}\in \mathrm{J}$, $\left|{\mathrm{J}}_{\mathrm{comp}}\right|$ is the number of completed job DAGs.

$$\min \hspace{0.33em}\mathrm{avg}.\mathrm{JET}=\frac{{\displaystyle {\sum }_{\mathrm{j}\in {\mathrm{J}}_{\mathrm{comp}}}\mathrm{exec}\left(\mathrm{j}\right)}}{\left|{\mathrm{J}}_{\mathrm{comp}}\right|}$$

(5)

4. Multi-Application Scheduling Algorithm (MASA) Based on DRL

The MASA proposed in this paper mainly addresses two key issues of multi-application scheduling in dynamic environments: (1) scheduling decisions for multiple job DAGs in streaming mode, and (2) neural network training methods in dynamic mission environments.

Task scheduling is essentially a sequential decision optimization problem, as shown in Figure 2. This section represents the multi-application scheduling scenario as a Markov Decision Making (MDP) process [24], when the ready task completes (releasing PEs) or a new application arrives (adding a job DAG), SCHEDULER takes the information of the job in the current job Queue as the input state, outputs the scheduling action, and after the action completes and returns a reward to SCHEDULER to determine whether the action is good or bad. The reward is designed according to the framework’s final optimization objective.

4.1. Framework of MASA

4.1.1. NN-Scheduler

NN-Scheduler solves the multitask prioritization problem, which contains state encoding network and policy network. As shown in Figure 3, the state encoding network is used to encode the job DAG into feature vectors, and the policy network is used to prioritize the tasks.

• State encoding network

The state encoding network design references graph neural networks and self-attention mechanisms [10] to achieve scalable network input dimensions. The input states contain the jobs DAG’s attributes (e.g., number of remaining tasks, examined task computational execution time, the PE types to which the tasks can be mapped, and DAG dependency structure), and the outputs are feature vectors at the task-level, the job-level, and the global-level, respectively.

The task-level feature vector is performed by the ${\mathrm{g}}_1$-network, which transforms the attribute information of each job DAG through a nonlinear transformation into a task-level feature vector ${\mathrm{e}}_{\mathrm{v}}^{\mathrm{i}}$. ${\mathrm{e}}_{\mathrm{v}}^{\mathrm{i}}$ captures the dependencies of the front and back nodes, as shown in Equation (6):

$${\mathrm{e}}_{\mathrm{v}}^{\mathrm{i}}={\mathrm{g}}_{ }[{\displaystyle \sum _{\mathrm{u}\in \mathrm{sub}\left({\mathrm{n}}_{\mathrm{v}}^{\mathrm{i}}\right)}\mathrm{f}(\mathrm{u}})]+{\mathrm{x}}_{\mathrm{v}}^{\mathrm{i}}$$

(6)

where ${ \mathrm{x}}_{\mathrm{v}}^{\mathrm{i}}$ represents the attribute information of each node ${\mathrm{n}}_{\mathrm{v}}^{\mathrm{i}}$ and $\mathrm{sub} \left({\mathrm{n}}_{\mathrm{v}}^{\mathrm{i}}\right)$ is the set of child nodes of ${\mathrm{n}}_{\mathrm{v}}^{\mathrm{i}}$. ${\mathrm{g}}_1 \left(\ast \right)$ contains the nonlinear transformation, where $\mathrm{f}~\log \left(·/\mathrm{n}\right)$, $\mathrm{g}~\exp \left(\mathrm{n}\times ·\right)$, $\mathrm{n}\to \infty$.

The ${\mathrm{g}}_2$-network’s input vectors, ${\mathrm{e}}_{\mathrm{v}}^{\mathrm{i}}$, derived from $\left|\mathrm{J}\right|\times \left|\mathrm{j}\right|$ space. We adopt the attention mechanism to represent the complex relationship between ${\mathrm{e}}_{\mathrm{v}}^{\mathrm{i}}$, transforming the raw data into scalable states to capture the feature vectors ${\mathrm{y}}^{\mathrm{i}}$ of the entire job DAG. The job-level feature vector ${\mathrm{y}}^{\mathrm{i}}$ is obtained by summarizing all the feature vectors ${\mathrm{e}}_{\mathrm{v}}^{\mathrm{i}}$ in the DAG, as indicated in the blue box in Figure 3. Equation (7) calculates the weighted vector ${\mathsf{\phi }}_{\mathrm{score}}^{\mathrm{i}}$, while Equation (8) represents the attention function that computes the weighted sum. This sum indicates the correlation between the weighted vector ${\mathsf{\phi }}_{\mathrm{score}}^{\mathrm{i}}$ and the vector sequence ${\mathrm{e}}^{\mathrm{i}}=\left\{{\mathrm{e}}_1^{\mathrm{i}},{\mathrm{e}}_2^{\mathrm{i}} \dots ,{\mathrm{e}}_{\left|\mathrm{j}\right|}^{\mathrm{i}}\right\}$, where $\left|\mathrm{j}\right|$ is the number of tasks in job ${\mathrm{j}}_{\mathrm{i}}$.

$${\phi }_{\mathrm{score} }^{\mathrm{i}}=\mathrm{softmax}\left(\frac{{\phi }_{\mathrm{q}}\cdot {\phi }_{\mathrm{k}}}{\sqrt{{\mathrm{d}}_{\mathrm{k}}}}\right)\cdot {\phi }_{\mathrm{v}}=\mathrm{softmax}\left(\frac{{\left({\mathrm{w}}_{\mathrm{q}}\cdot {\mathrm{e}}_{\mathrm{v}}^{\mathrm{i}}\right)}^{\mathrm{T}}\left({\mathrm{w}}_{\mathrm{k}}\cdot {\mathrm{e}}_{\mathrm{v}}^{\mathrm{i}}\right)}{\sqrt{{\mathrm{d}}_{\mathrm{k}}}}\right)\left({\mathrm{w}}_{\mathrm{v}}\cdot {\mathrm{e}}_{\mathrm{v}}^{\mathrm{i}}\right)$$

(7)

$${\mathrm{y}}^{\mathrm{i}}=\mathrm{att}\left({\phi }_{\mathrm{score} }^{\mathrm{i}},{\mathrm{e}}^{\mathrm{i}}\right)={\displaystyle \sum _{\mathrm{v}=1}^{\left|\mathrm{j}\right|}{\phi }_{\mathrm{score} }^{\mathrm{i}}}\cdot {\mathrm{e}}_{\mathrm{v}}^{\mathrm{i}}$$

(8)

In order to be able to better represent the connection between all job DAGs, a global feature vector $\mathrm{z}$, $\mathrm{z}=\psi \left({\mathrm{y}}^1,{\mathrm{y}}^2\dots {\mathrm{y}}^{\mathrm{n}}\right)$ represents a nonlinear function.

2.

Policy network for task prioritization

The prioritization of tasks is achieved using the policy network after obtaining the feature vectors that correspond to the tasks in the job DAG. For each node n in the job DAG, its priority value ${\mathrm{q}}_{\mathrm{v}}^{\mathrm{i}}$ is calculated according to Equation (9), as indicated by the red box in Figure 3.

$${\mathrm{q}}_{\mathrm{v}}^{\mathrm{i}}=\mathrm{q}\left({\mathrm{e}}_{\mathrm{v}}^{\mathrm{i}},{\mathrm{y}}^{\mathrm{i}},\mathrm{z}\right)$$

(9)

It can be seen from the formula that the ${\mathrm{q}}_{\mathrm{v}}^{\mathrm{i}}$ of each node in each DAG is related to three elements, i.e., the task-level vector ${\mathrm{e}}_{\mathrm{v}}^{\mathrm{i}}$ containing the attribute information and dependencies of the node itself; the information ${\mathrm{y}}^{\mathrm{i}}$ contained in the DAG where each node is located; and all the global vectors $\mathrm{z}$ representing the connections between job DAGs.

Tasks ${\mathrm{n}}_{\mathrm{v}}^{\mathrm{i}}$ with high priority values ${\mathrm{q}}_{\mathrm{v}}^{\mathrm{i}}$ in the state space are transferred to the set of ready tasks ${\mathcal{A}}_{\mathrm{t}}$. Each episode updates ${\mathcal{A}}_{\mathrm{t}}$. Subsequently, the sofmax function is used to calculate the mapping probability $\mathrm{P} \left({\mathrm{n}}_{\mathrm{v}}^{\mathrm{i}}\in {\mathcal{A}}_{\mathrm{t}}\right)$ for each ready task in ${\mathcal{A}}_{\mathrm{t}}$, as shown in Equation (10).

$$\mathrm{P}({\mathrm{n}}_{\mathrm{v}}^{\mathrm{i}}\in {\mathcal{A}}_{\mathrm{t}})=\frac{\exp \left({\mathrm{q}}_{\mathrm{v}}^{\mathrm{i}}\right)}{{\displaystyle {\sum }_{{\mathrm{n}}_{\mathrm{v}}^{\mathrm{i}}\in {\mathcal{A}}_{\mathrm{t}}}\exp }\left({\mathrm{q}}_{\mathrm{v}}^{\mathrm{i}}\right)}$$

(10)

4.1.2. Heuristic Scheduler

Because it involves the selection of PE type, the heterogeneous computing environment makes the assignment of PEs more difficult than the problem of assigning PEs in a homogeneous environment. Therefore, after obtaining the mapping probabilities corresponding to high-priority tasks, the task-mapping problem in the second scheduling phase is realized based on a heuristic algorithm. There are some implicit constraints in the resource allocation process, such as the sum of the computational requirements of multiple tasks deployed on a PE must not exceed the upper limit of this PE’s computational resources, and the sum of the task data volume overlays must not exceed the physical bandwidth between the PEs.

This paper aims to improve execution efficiency in a continuous dynamic environment through the inclusion of a heuristic algorithm that duplicates critical parent nodes, reducing application execution time. The mapping scheme of the selected node in this study is based on the DEFT algorithm [25]. Comparison of the computational results of the EFT algorithm and the Copy Parent Earliest Finish Time (CPEFT) algorithm helps select the smallest result as the most optimal value. Equations (11) and (12) detail how the CPEFT algorithm tries to replicate each parent node of the selected node in order to compare and arrive at the earliest possible completion time. Equation (13) explains how the DEFT algorithm selects the minimum result of the EFT and CPEFT algorithms, regardless of whether it involves duplicated parent nodes or not. This ensures that computations are performed with the minimum result.

$$\mathrm{CPEST}\left(\mathrm{perd}\left({\mathrm{n}}_{\mathrm{i}}\right),{\mathrm{n}}_{\mathrm{i}},{\mathrm{p}}_{\mathrm{m}}\right)={\min }_{{\mathrm{n}}_{\mathrm{j}}\in \mathrm{pred}\left({\mathrm{n}}_{\mathrm{i}}\right)}\left(\mathrm{AFT}\left({\mathrm{n}}_{\mathrm{j}}\right)+\mathrm{comm}\left({\mathrm{n}}_{\mathrm{j}},{\mathrm{n}}_{\mathrm{i}}\right)\right)$$

(11)

$$\mathrm{CPEFT}\left({\mathrm{n}}_{\mathrm{i}},{\mathrm{p}}_{\mathrm{m}}\right)={\max }_{{\mathrm{n}}_{\mathrm{a}},{\mathrm{n}}_{\mathrm{b}}\in \mathrm{perd}\left({\mathrm{n}}_{\mathrm{i}}\right),\mathrm{a}\ne \mathrm{b}}\left\{\mathrm{CPEST}\left({\mathrm{n}}_{\mathrm{a}},{\mathrm{n}}_{\mathrm{i}},{\mathrm{p}}_{\mathrm{m}}\right),\mathrm{CPEST}\left({\mathrm{n}}_{\mathrm{b}},{\mathrm{n}}_{\mathrm{i}},{\mathrm{p}}_{\mathrm{m}}\right)\right\}+\mathrm{comp}\left({\mathrm{n}}_{\mathrm{i}},{\mathrm{p}}_{\mathrm{m}}\right)$$

(12)

$$\mathrm{DEFT}\left({\mathrm{n}}_{\mathrm{i}}\right)=\min \left\{\mathrm{CPEFT}\left({\mathrm{n}}_{\mathrm{i}},{\mathrm{p}}_{\mathrm{m}}\right),\mathrm{EFT}\left({\mathrm{n}}_{\mathrm{i}},{\mathrm{p}}_{\mathrm{m}}\right)\right\}$$

(13)

4.1.3. Description of the Empirical Data

The NN-Scheduler obtains state information ${\mathrm{s}}_{\mathrm{t}}$ and executes action ${\left\{{\mathrm{a}}_{\mathrm{n},\mathrm{t}}\right\}}_{\mathrm{n} =1}^{\mathrm{n}\prime }$ at each interaction time step t. This paper differs from standard MDP processes in that the moment when the environment state changes is uncertain. The policy network generates multiple task decisions in time step t. When all the actions in time slot t (time step [t, t + 1)) are completed, rewards are generated and the environment state is updated. The environment state changes if a new job DAG is reached during action execution.

1.

State space

The NN-Scheduler receives an input state represented by $\mathrm{S}=\left[{\mathrm{s}}_1,{\mathrm{s}}_2,\dots ,{\mathrm{s}}_{\mathrm{o}}\right]$. ${\mathrm{s}}_{\mathrm{n}}$ represents the state of the nth job, and $\mathrm{o}$ is the number of concurrent jobs in the job queue at the current time, where $\mathrm{o}$ is a variable and its value cannot exceed the maximum job queue capacity C. Concurrent jobs include previously unfinished jobs and newly arrived jobs, and the NN-Scheduler can learn about the potential interference between concurrent jobs through training. The state ${\mathrm{s}}_{\mathrm{n}}$ of the nth job can be represented by the following equation:

$${\mathrm{s}}_{\mathrm{n}}=\left({\left(\overline{{\mathrm{p}}_{\mathrm{v}}},\overline{{\mathrm{EET}}_{\mathrm{v}}},\overline{{\mathrm{status}}_{\mathrm{v}}}\right)}_{\mathrm{v}=1}^{|\mathrm{j}|},|\mathrm{OT}|,|\mathrm{RT}|,\mathrm{JWT}\right)$$

(14)

The task ${\mathrm{n}}_{\mathrm{v}}^{ }$’s assignable PE number is ${\mathrm{p}}_{\mathrm{v}}^{ }$; ${\mathrm{EET}}_{\mathrm{v}}^{ }$ is the estimated execution time corresponding to the task; ${\mathrm{status}}_{\mathrm{v}}^{ }$ is the current state of the task, classified by the labels ready, completed, or outstanding; $\left|\mathrm{j}\right|$ is the number of task nodes in job j. $\left|\mathrm{OT}\right|$ refers to the number of outstanding tasks that are waiting for the parent node to finish its execution. Similarly, $\left|\mathrm{RT}\right|$ refers to the number of ready tasks in the set of ready tasks ${\mathcal{A}}_{\mathrm{t}}$, and JWT is the time that job ${\mathrm{j}}_{\mathrm{n}}$ has been running from the time it arrived on the system to the time it is currently running.

2.

Action space

At each time step, the SCHEDULER observes the state ${\mathrm{s}}_{\mathrm{t}}$ of the environment and performs the number of $\mathrm{n}\prime$ actions, ${\left\{{\mathrm{a}}_{\mathrm{n},\mathrm{t}}~{\pi }_{\mathsf{\theta }}\left({\mathrm{a}|\mathrm{s}}_{\mathrm{t}}\right)\right\}}_{\mathrm{n}=1}^{\mathrm{n}\prime }$. The action space is shown in the red box in Figure 3, and the NN-Scheduler outputs the set of selected high-priority tasks ${\mathrm{n}}_{\mathrm{v}}^{\mathrm{i}}$.

3.

Reward

The correct reward is computed according to the RDA method (Section 4.2.1), taking into account the inconsistency between the time step of the scheduler’s interaction with the environment and the time step of the environment’s interaction with the scheduler. ${\mathrm{r}}_{\mathrm{t}}$ is computed by constructing the reward under each clock signal as ${\mathrm{r}}_{\mathrm{clk}}^{\prime }$, and then calculating ${\mathrm{r}}_{\mathrm{t}}$ from it. ${\mathrm{r}}_{\mathrm{clk}}^{\prime }$ is given by Equation (15), where $\left|{\mathrm{n}}_{\mathrm{comp}}\right|$ is the number of newly completed tasks under each clock.

$${\mathrm{r}}_{\mathrm{clk}}^{\prime }=\left|{\mathrm{n}}_{\mathrm{comp}}\right|$$

(15)

4.2. Network Training

This paper addresses the problem of training neural networks based on the policy gradient algorithm, Actor Critic (AC) [26,27]. AC evaluates the merits of an action by using a dominance function, and if the action value function $\mathrm{Q} \left(\mathrm{S},\mathrm{A}\right)$ is better than the state value function $\mathrm{V} \left(\mathrm{S}\right)$, the corresponding action is improved. At each time step t, the actor network observes the environment state ${\mathrm{s}}_{\mathrm{t}}$ and chooses to execute action ${\left\{{\mathrm{a}}_{\mathrm{n},\mathrm{t}}\right\}}_{\mathrm{n} =1}^{\mathrm{n}\prime }~{\pi }_{\mathsf{\theta },\mathrm{n}}\left({\mathrm{s}}_{\mathrm{t}}\right)$. $\mathrm{n}\prime$ denotes the number of tasks with the same priority.

The Actor network’s loss function is

$${\mathrm{Loss}}_{\mathrm{actor}}={\displaystyle {\sum }_{\mathrm{t}=0}^{\mathrm{T}}\log {\pi }_{\theta }\left({\mathrm{a}}_{\mathrm{n},\mathrm{t}}|{\mathrm{s}}_{\mathrm{t}}\right)\left({\mathrm{Q}}_{{\pi }_{\theta }}({\mathrm{s}}_{\mathrm{t}},{\mathrm{a}}_{\mathrm{n},\mathrm{t}})-{\mathrm{V}}_{{\pi }_{\theta }}({\mathrm{s}}_{\mathrm{t}})\right)}$$

(16)

The Critic network assesses the state value function ${\mathrm{V}}_{{\pi }_{\mathsf{\theta }}} \left({\mathrm{s}}_{\mathrm{t}}\right)$ and computes its loss function as shown in Equation (17).

$${\mathrm{Loss}}_{\mathrm{critic}}=\frac{1}{2}{\left({\mathrm{Q}}_{{\pi }_{\theta }}({\mathrm{s}}_{\mathrm{t}},{\mathrm{a}}_{\mathrm{n},\mathrm{t}})-{\mathrm{V}}_{{\pi }_{\theta }}({\mathrm{s}}_{\mathrm{t}})\right)}^2$$

(17)

$${\nabla }_{\theta }\mathrm{J}(\theta )\triangleq {\nabla }_{\theta }\left[{\mathrm{Loss}}_{\mathrm{actor} }+{\mathrm{Loss}}_{\mathrm{critic} }+\eta \cdot {\mathrm{E}}_{\mathrm{S}}\left[\mathrm{H}\left({\pi }_{\theta }\left(·|{\mathrm{s}}_{\mathrm{t}}\right)\right)\right]\right]$$

(18)

The training policy learns the optimal strategy using the regular entropy to make the output probability distribution of actions more uniform and to prevent convergence to a single output action. This paper considers introducing entropy regularity $\eta ·{\mathrm{E}}_{\mathrm{S}}\left[\mathrm{H}\left({\pi }_{\mathsf{\theta }}\left(·|{\mathrm{s}}_{\mathrm{t}}\right)\right)\right]$ into the loss function, represented with entropy H (·) and weight $\eta$. In summary, the gradient strategy is as follows.

To achieve stable and fast training convergence, this paper also employs the following methods to optimize the training process: (1) Reward Dynamic Alignment (RDA) is used to address the problem of incorrect reward propagation. This problem arises due to inconsistent interaction time steps between SCHEDULER and the environment. (2) Gradually increasing the length of episodes can improve the initial training phase’s inference ability for neural networks and help address the challenge of making multiple job DAG scheduling decisions. (3) Asynchronous multi-agent joint training is another method used to speed up the neural network’s training process.

Algorithm 2 shows the pseudocode of the network training process in MASA. Throughout the training process, asynchronous multi-agent joint training is performed. In order to avoid wasting training time, the initial events are terminated early, and Line 4 samples the event length T from an exponential distribution with a small initial expectation ${\mathrm{T}}_{\mathrm{mean}}$. (See Training Optimization Method 2 for more information). In order to avoid errors caused by randomness during job arrivals and inconsistent task execution times, Line 11 calculates the correct time-step reward based on the clock reward (see Training Optimization Method 1). Additionally, Line 22 implements the increment of the average length of episodes ${\mathrm{T}}_{\mathrm{mean}}$. Lastly, the strategy parameter θ of MASA is updated in Line 23.

Algorithm 2. Training Algorithm

//Assume global shared parameter vector ${\theta }_1$ and ${\theta }_2$, one-thread parameter ${\theta }_1^{\prime }$ and ${\theta }_2^{\prime }$, globally shared iteration counter M, globally shared iteration max.counter $M_{max}$, maximum episode length mean $T_{mean}$, decay factor γ, Initial thread time step counter $t\leftarrow 1$ repeat    sample episode length $T$    Reset actor/critic network’s policy gradient: ${\theta }_1\leftarrow 0$ and ${\theta }_2\leftarrow 0$    Synchronize global parameter to one-thread parameters ${\theta }_1^{\prime }={\theta }_1$ and ${\theta }_2^{\prime }={\theta }_2$    $t_{\mathrm{start}}=t,$ get state $s_t$    repeat       Perform $a_t$ according to policy ${\pi }_{{\theta }_1^{\prime }}\left(a_t|s_t\right)$       Assign PE for task       Calculate the reward by clock’s reward according to Equation (19)       Receive reward $r_t$ and new state $s_{t+1}$          $t\leftarrow t+1$          $M\leftarrow M+1$     until Job Queue is empty (terminal $s_t$) or $t-t_{\mathrm{start}}==T$     $Q\left(s,t\right)=\left\{\begin{array}{cc}0& for terminal \\ V_{{\pi }_{{\theta }_2^{\prime }}}\left(s_t\right)\hfill & for non-terminal s_t\end{array}\right.$     for $j\in \left\{t-1,\dots ,t_{\mathrm{start}}\right\}$ do        $Q\left(s,j\right)\leftarrow r_j+\gamma Q\left(s,j+1\right)$        Accumulate actor’s local gradient updates: $$d{\theta }_1\leftarrow d{\theta }_1+{𝛻}_{{\theta }_1^{\prime }}\mathit{log}{\pi }_{{\theta }_1^{\prime }}\left(a_j,s_j\right)\left(Q\left(s,j\right)-V_{{\pi }_{{\theta }_2^{\prime }}}\left(s_j\right)\right)+\eta {𝛻}_{{\theta }_1^{\prime }}H\left({\pi }_{{\theta }_1^{\prime }}\left(·|s_j\right)\right)$$        Accumulate critic’s local gradient updates: $d{\theta }_2\leftarrow d{\theta }_2+{𝛻}_{{\theta }_2^{\prime }}\frac{1}{2}{\left(Q\left(s,j\right)-V_{{\pi }_{{\theta }_2^{\prime }}}\left(s_j\right)\right)}^2$     end for     $T_{mean}\leftarrow T_{mean}+\epsilon$     Update global network parameters ${\theta }_1={\theta }_1+\alpha d{\theta }_1$ and ${\theta }_2={\theta }_2+\beta d{\theta }_2$ Until $M>M_{max}$ Output the global network parameters ${\theta }_1$ and ${\theta }_2$

4.2.1. Reward Dynamic Alignment Method

The standard MDP process is suitable for round-based decision problems, where the change in the state of the environment occurs after the agent has executed an action [28]. In heterogeneous resource systems, external factors such as the random arrival of jobs and internal factors such as the different completion times of multiple actions at the same timestep can cause the state of the environment to change while the SCHEDULER is executing an action. This paper proposes the RDA method as a solution to incorrect reward propagation during finite MDP.

1.

incorrect rewards

Scenario 1. 

SCHEDULER can achieve real-time task scheduling by observing the real-time environment state and taking immediate actions while making decisions in the current time step. For instance, as depicted on the Figure 4a, when the action $a_{n,t}$ ends at time step, denoted by t’, the immediate reward for the action is also granted at t’. While the action $a_{i,t}$ is being executed, the random arrival of a job can modify the priority relation between tasks, and SCHEDULER observes S_{t + 1} when t′ ∈ (t + 1, t + 2]. This circumstance leads to the incidence of the incorrect reward for the action at time t + 1.

Scenario 2. 

Scheduling decisions of tasks with the same priority are formed at the same timestep. However, due to the differences in the performance of heterogeneous PE as well as the task requirements, each decision process completes at a different time, resulting in an inconsistency between the immediate rewards and the observed rewards, a situation that also leads to the occurrence of incorrect rewards. For example, as shown on the Figure 4b, although task T₁ has been accomplished earlier than expected, the scheduling policies for the subsequent tasks T₃, T₄ and T₅ are applied after the completion of task T₂. As a result, there is a delay in the reward for task T₁.

2.

Implementation of RDA

The RL process of an MDP can be summarized as a ${\left\{{\mathrm{s}}_{\mathrm{t}},{\left\{{\mathrm{a}}_{\mathrm{n},\mathrm{t}}\right\}}_{\mathrm{n} =1}^{ \mathrm{n} \prime },{\left\{{\mathrm{r}}_{\mathrm{n},\mathrm{t}}\right\}}_{\mathrm{n} =1}^{ \mathrm{n} \prime }\right\}}_{\mathrm{t} =1}^{\mathrm{T}}$ sequence. The reward ${\mathrm{r}}_{\mathrm{t}}$ corresponding to the previous decision affects the SCHEDULER’s action decision ${\mathrm{a}}_{\mathrm{t} +1}$ at time step t + 1, so how to obtain the correct ${\mathrm{r}}_{\mathrm{t}}$. The essential reason for generating incorrect rewards is the inconsistency of the time step at which the SCHEDULER interacts with the environment. Therefore, this paper proposes to compute the reward ${\mathrm{r}}_{\mathrm{clk}}^{\prime }$ on each clock signal so that the computation of ${\mathrm{r}}_{\mathrm{clk}}^{\prime }$ is independent of the interaction time step. The time-step reward ${\mathrm{r}}_{\mathrm{t}}$ consists of the clock reward ${\mathrm{r}}_{\mathrm{clk}}^{\prime }$, i.e., the value of ${\mathrm{r}}_{\mathrm{t}}$ consists of a superposition of $\left\{{\mathrm{r}}_{\mathrm{clk}}^{\prime }\right\}$. The RDA method is specifically described below.

• Step 1: Decouple the empirical data associated with the time step to obtain the sequence ${\left\{{\mathrm{s}}_{\mathrm{t}},{\mathrm{a}}_{\mathrm{t}}\right\}}_{\mathrm{t} =1}^{\mathrm{T}}$ as well as ${\left\{{\mathrm{r}}_{\mathrm{t}}\right\}}_{\mathrm{t} =1}^{\mathrm{T}}$. T is the last interaction time step in the episodes. To obtain the sequences ${\left\{{\mathrm{r}}_{\mathrm{clk}}^{\prime }\right\}}_{\mathrm{clk} =1}^{\mathrm{CLK}}$ and ${\left\{{{\displaystyle \sum }}_{\mathrm{clk} =1}^{{\mathrm{clk}}_{\mathrm{now}}}{\mathrm{r}}_{\mathrm{clk}}^{\prime }\right\}}_{{\mathrm{clk}}_{\mathrm{now}}=1}^{\mathrm{CLK}}$, compute the ${\mathrm{r}}_{\mathrm{clk}}^{\prime }$ on each clock signal and perform the ${{\displaystyle \sum }}_{\mathrm{clk} =1}^{{\mathrm{clk}}_{\mathrm{now}}}{\mathrm{r}}_{\mathrm{clk}}^{\prime }$ computation independently of the interaction timestep. Here, CLK refers to the clock length of the simulation.
• Step 2: At the beginning of the action, memorize the clock signal ${\mathrm{clk}}_{\mathrm{start}}$, and at the end, memorize the clock signal ${\mathrm{clk}}_{\mathrm{end}}$. Use Equation (19) to calculate ${\mathrm{r}}_{\mathrm{t}}$.

$${\mathrm{r}}_{\mathrm{t}}={\displaystyle {\sum }_{\mathrm{clk}=1}^{{\mathrm{clk}}_{\mathrm{end}}}{\mathrm{r}}_{\mathrm{clk},\mathrm{t}}^{\prime }}-{\displaystyle {\sum }_{\mathrm{clk}=1}^{{\mathrm{clk}}_{\mathrm{start}}}{\mathrm{r}}_{\mathrm{clk},\mathrm{t}}^{\prime }}={\displaystyle {\sum }_{{\mathrm{clk}}_{\mathrm{start}}}^{{\mathrm{clk}}_{\mathrm{end}}}{\mathrm{r}}_{\mathrm{clk},\mathrm{t}}^{\prime }}$$

(19)

In summary, the RL process based on the RDA method can be summarized in Equation (20).

$${\left\{{\mathrm{s}}_{\mathrm{t}},{\left\{{\mathrm{a}}_{\mathrm{n},\mathrm{t}}\right\}}_{\mathrm{n}=1}^{{\mathrm{n}}^{\prime }},{\left\{{\mathrm{r}}_{\mathrm{n},\mathrm{t}}\right\}}_{\mathrm{n}=1}^{{\mathrm{n}}^{\prime }}\right\}}_{\mathrm{t}=1}^{\mathrm{T}}\to {\left\{{\mathrm{s}}_{\mathrm{t}},{\left\{{\mathrm{a}}_{\mathrm{n},\mathrm{t}}\right\}}_{\mathrm{n}=1}^{{\mathrm{n}}^{\prime }},{\left\{{\displaystyle {\sum }_{{\mathrm{clk}}_{\mathrm{start}}}^{{\mathrm{clk}}_{\mathrm{end}}}{\mathrm{r}}_{\mathrm{clk},\mathrm{n},\mathrm{t}}^{\prime }}\right\}}_{\mathrm{n}=1}^{{\mathrm{n}}^{\prime }}\right\}}_{\mathrm{t}=1}^{\mathrm{T}}$$

(20)

In this way, the correct empirical data can be obtained using the RDA method during the interaction between SCHEDULER and the environment, no matter how dynamically the set of actions is changing.

4.2.2. Early Termination of the Initial Episode Method

During the initial phase of neural network training, network parameters are randomly set, resulting in a weak initial scheduling decision for SCHEDULER. As jobs randomly and successively arrive during the training process, the initial job DAG may not receive a reasonable scheduling decision. This may result in a negative impact on the final job execution time, and this process is irreversible. Furthermore, in most of the initial episodes, the outstanding queue may accumulate a large number of waiting tasks. If too much empirical data are sampled during these episodes, it will lead to increased duration of network training. To prevent such a situation, this paper limits the initial episode length and increases it gradually during the training process. The memoryless termination process is employed for achieving incrementally longer episodes. Timestep $\mathrm{T}$ is randomly sampled from an exponential distribution, causing the episode length T to obey an exponential distribution as well $\mathrm{T}~\mathrm{exponential}1/{\mathrm{T}}_{\mathrm{mean}}$.

4.2.3. Asynchronous Multi-Agent Joint Training Method

Training RL models using continuous sampling is difficult to converge due to the correlation between empirical data. Using empirical replay, the correlation of the data can be broken. However, there is another asynchronous multi-agent joint training method that can break the correlation of empirical data by creating multiple Agents [29,30], starting multiple training environments for sampling and using the collected empirical data for training, accumulating the gradients obtained from training in multiple processes and updating the shared parameters together after a certain number of steps. Compared to the experience playback method that requires an experience pool to store historical samples and randomly selects training samples, independent and identically distributed samples produced with asynchronous training not only reduce storage space but also greatly accelerate the sampling speed and consequently improve the training speed. Additionally, different strategies used in distinct training environments create a more uniform distribution of empirical data, facilitating the stable training of neural networks.

5. Experiments

This section aims to verify the performance of our proposed MASA algorithm. Firstly, we compare the performance of the proposed algorithm with that of current resource-scheduling algorithms. Next, ablation experiments are conducted to verify the effectiveness of the proposed training optimization method. To demonstrate the scalability of the proposed algorithm for complex mission environments, the experiments simulate the task-scheduling problem in streaming mode using six real-world RF applications in the field of wireless communication and radar processing.

5.1. Experimental Environment Setting

During simulation execution, a job randomly arrives every ${\mathrm{clk}}_{\mathrm{inj}}$ clock signal in the job queue to simulate the dynamic task environment. ${\mathrm{clk}}_{\mathrm{inj}}$’s expectations are $\mathrm{scale}$, $scale=\mathrm{E} \left({\mathrm{clk}}_{\mathrm{inj}}\right)$. The inter-arrival time of the ${\mathrm{clk}}_{\mathrm{inj}}$ follows the exponential distribution, ${\mathrm{clk}}_{\mathrm{inj}}~\mathrm{exponential}\left(\frac{1}{\mathrm{scale}}\right)$.

Assuming the multifunction RF system has the ability to handle different RF applications, real-world application configurations are more complex than classic application configurations.

Table 1. Benchmark of applications.

Application Type	Application	Number of Tasks	Max Width, Depth
Real-world’ Communication Application	WiFi-TX	27	5, 7
	WiFi-RX	34	5, 10
	Single-carrier Tx	10	1, 10
	Single-carrier Rx	10	1, 10
Real-world’ Radar Application	range detection	7	2, 6
	Temporal Mitigation	10	2, 6
Classic DAG		10	random

shows the classic DAG comprising of 10 randomly generated tasks. WiFi-TX, WiFi-RX, Single-carrier Tx (SCT), Single-carrier Rx (SCR), Range Detection, and Temporal Mitigation are the six common RF applications. Their corresponding hardware resource profiles are obtained from engineering data. Figure 5a displays the respective DAGs for WiFi-TX [31]. Here, Inverse Fast Fourier Transform (FFT) tasks are computationally intensive, taking 10 times longer to process on a CPU than on a HW accelerator. Figure 5b shows the computational execution time of each task on the corresponding PE that we obtained from statistics in the actual project; in Figure 5a, the value inside the circle represents the task ID, and the value beside the connecting line represents the data volume between the WiFi-Tx tasks. The communication bandwidth between PEs in the hardware configuration file is set as follows: the bandwidth value between HW Acc2 and the other PEs (cpu A15, cpu A15 and HW Acc1) is set to 10, and the bandwidth value be-tween all other PEs is set to 1000. The data volume between tasks and the communication bandwidth between PEs are known, and the communication delay can be derived based on Equation (2). Based on the above profiles in the simulation platform, MASA determines which PE to assign to the current highest priority task in the task-mapping phase based on the constraints of equations such as EST and EFT. Network structure and training parameters are listed in

Table 2. Algorithm parameter setting.

Parameter	Value	Parameter	Value
Job queue capacity	3	Optimizer	Adam
Node input dimensions	6	Scale (job inject interval’ expectation)	20/40/60/80/100
Output dimensions	8	CLK (simulation length)	10,000
Maximum depth	3	${\mathrm{T}}_{\mathrm{mean}}$ (episode length’ expectation)	10,000
Hidden dimensions	[16, 8]	${\mathrm{M}}_{\max }$ (number of Agent)	1/2/4/6/8/10
$\mathsf{\beta }$ (learning rate)	0.0001	$\eta$ (entropy coefficient)	0.1
$\mathsf{\alpha }$(learning rate)	0.0003	$\mathrm{y}$ (discount factor)	0.98

.

These applications show different workload characteristics due to differences in DAG structure and computational requirements. For instance, WiFi-TX demonstrates a considerable level of parallelism compared to SCT and SCR, which contain sequential links among nodes, thus consuming less energy during low bandwidth conditions.

For the purpose of comparison, two types of representative scheduling algorithms have been considered: The heuristic algorithms MET [32] and HEFT [17], and the neural scheduling algorithms SCARL [33], decima [23], and SoCRATES [22]. This paper incorporates the mainstream homogeneous resource-scheduling algorithm decima into the comparative experiments to emphasize the necessity of designing heterogeneous scheduling algorithms. Decima is dedicated to resource scheduling in homogeneous environments, and SCARL only supports DAG for link structure, so its configuration must be modified in simulation slightly, and each experiment is repeated five times.

5.2. Algorithm Performance Analysis

We present the performance results of the neural network algorithm and heuristic algorithm for different application configurations in Figure 6, Figure 7 and Figure 8. The scheduling algorithm is employed to schedule and execute both classical and real-world applications’ DAGs. We plot the trend of the average application execution time with the job DAG arrival rate. The horizontal axis depicts different scale values, the smaller the scale, the faster the application arrival rate; the vertical axis depicts the average application execution time.

Figure 6. (a) Results of scheduling algorithms in different numbers of classical DAGs: (a) the DAG pool consists of one classical DAG; (b) the DAG pool consists of 2 random classical DAGs; (c) 4; (d) 6; (e) 8; (f) 10.

Figure 6. (a) Results of scheduling algorithms in different numbers of classical DAGs: (a) the DAG pool consists of one classical DAG; (b) the DAG pool consists of 2 random classical DAGs; (c) 4; (d) 6; (e) 8; (f) 10.

Figure 7. Scheduling results of scheduling algorithms for different real RF applications (a) WiFi-TX; (b) WiFi-RX; (c) Single-carrier Tx; (d) Single-carrier Rx; (e) Ranging detection; (f) Temporal mitigation.

Figure 7. Scheduling results of scheduling algorithms for different real RF applications (a) WiFi-TX; (b) WiFi-RX; (c) Single-carrier Tx; (d) Single-carrier Rx; (e) Ranging detection; (f) Temporal mitigation.

Figure 8. Algorithm performance results in mixed real-world environments (consisting of 6 real-world applications with the same probability).

Figure 8. Algorithm performance results in mixed real-world environments (consisting of 6 real-world applications with the same probability).

5.2.1. MASA in Classic DAG Environment

In Figure 6, we show the performance results of different scheduling algorithms under the classical DAG environment configuration. We also plot the algorithms’ performance results under different numbers of applications, respectively, as in Figure 6a–f. As the arrival rate of applications increases, the average application execution time of each algorithm gradually increases. However, the performance of the MASA algorithm proposed in this paper consistently outperforms the other algorithms. The graph shown in Figure 6a represents the execution results of a classical DAG with successive arrivals to the scheduler, under the condition of following an exponential distribution. To comprehend the performance of the scheduling algorithm, an analysis is conducted on the average application execution time at different scales. At a low arrival rate (scale = 100), the heuristic-based algorithm’s scheduling performance is clearly weaker than the neural network-based algorithm’s scheduling performance. When the arrival rate increases (scale = 20), the MASA algorithm proposed in this paper also outperforms the other neural scheduling algorithms, with an improvement of about 9% over the performance of SCRAL and about 18% over the performance of decima and SoCRATES. Figure 6f illustrates an application pool of 10 classical DAGs, each with the same arrival probability. As shown by the figure, the algorithm presented in this paper offers superior performance compared to other algorithms in all application arrival rates, while also improving with an increase in the number of applications.

The data presented in Figure 6 show that the heuristic algorithm can yield results comparable to other algorithms when the number of tasks is small and the state environment is relatively uncomplicated. However, as the task injection rate increases, the complexity of the state space faced by the scheduler based on the heuristic algorithm increases dramatically, and the heuristic algorithm is unable to solve the situation very well. Decima is based on a graph neural network for encoding the state space. Compared to the heuristic algorithm, it can provide a more detailed explanation of the complex state space, leading to significantly improved performance, especially when dealing with large task injection rate, but it should be noted that Decima is mainly proposed to solve the isomorphic resource-scheduling problem. Based on this foundation, SCARL and SoCRATES introduce the fundamental concept of RL to improve the results when the model is trained on the dynamically transformed heterogeneous resource platform through the interaction between the environment and the Agent, and the experimental results also prove the effectiveness of RL in the problem setting. To further encode the complex and dynamic state space of the heterogeneous resource platform, our proposed MASA introduces the mechanism of attention, which further improves its encoding ability. The final outcomes demonstrate that the attention mechanism can significantly enhance the encoding capacity of the network model as the task volume increases and the encoding environment becomes more complex.

5.2.2. MASA in Real-World Environment

To further illustrate the superiority of our proposed algorithms, we test the performance of each algorithm in a job profile in a real mission scenario, and the experimental results are shown in Figure 7a–f and Figure 8. The configuration for WiFi-RX job is the most complex, consisting of 34 tasks with complicated dependencies. However, its application execution time is approximately three times longer than that of the latter.

The WiFi-TX job includes 27 tasks, which is more than a typical task configuration environment. However, its execution time does not increase much compared to a classical environment because the dependencies between its tasks are mostly linear, and it can be observed that MASA outperforms the other compared algorithms in every application arrival rate ($\mathrm{scale}$). The configuration of WiFi-RX is the most complicated, which contains the largest number of tasks and complex dependencies between tasks, so the execution time of this task is much larger than all the other application configurations in this paper. It can be seen that the algorithm proposed in this paper, MASA, has a significant improvement over the other heuristics and neural scheduling algorithms, which is mainly due to the fact that MASA adopts the attention mechanism, which can effectively analyze the topology between complex tasks to improve the performance of the neural scheduling network. The SCT task configuration includes only 8 tasks and straightforward dependencies, so its execution time is relatively low, and when the task injection rate is low, there is not much difference in the performance of the neural scheduling algorithms (better than the heuristic algorithms), but when the task injection rate reaches the maximum, the performance of the neural scheduling algorithms is better than the heuristic algorithms. Nevertheless, when the task injection rate is maximum, MASA’s algorithm surpasses the performance of other comparable algorithms. The task environment configuration of SCR is similar to that of SCR, except it demands a higher data volume between tasks. According to the experimental results, both SoCRATES and MASA’s scheduling schemes yield the lowest task execution time, and MASA slightly outperforms SoCRATES. The configurations of the range detection and Temporal Mitigation environments show that MASA performs better than the other algorithms at all task arrival rates. In conclusion, the study demonstrates that the simulation environment is scalable by validating the proposed algorithm under different real profiles.

This paper includes a simulation of six applications (WiFi-TX/RX, SCT, SCR, range detection, and Temporal Mitigation) concurrently running in a real environment to analyze algorithm performance under varying task arrival rates. Figure 8 shows that the heuristic algorithm does not dominate the overall task scheduling due to the extended execution time of the WiFi-TX. The complexity of the task environment results in higher demands on the performance of the scheduling algorithms as the task injection rate increases, and the MASA algorithm gradually demonstrates its dynamic scheduling capability in complex task environments. The real-world job profiles are more complex, so the scheduler’s ability to encode the state space puts forward higher requirements, and its final average task execution time depends largely on each scheduler’s ability to interpret the state space. MASA’s excellent dynamic scheduling ability is due to the fact that the introduced attention mechanism, which eliminates the need to adjust the scale of the network model based on the specific task environment and provides a better interpretation of the dynamic environment.

Figure 9a illustrates the convergence process of actor network during training. It can be observed that the actor–critic strategy results in convergence around 100 episodes. The fast convergence of ${\mathrm{LOSS}}_{\mathrm{Actor}}$ in MASA proves its stronger learning ability in dynamic environments.

Figure 9b presents the change of entropy $\mathrm{H}\left({\pi }_{\mathsf{\theta }}\left(·|{\mathrm{s}}_{\mathrm{t}}\right)\right)$ in Equation (18), describing the task environment faced by the scheduler. As training progresses, the MASA network effectively resolves the job DAGs, leading to a gradual reduction of uncertainty in the environment. Lower values of $\mathrm{H}\left({\pi }_{\mathsf{\theta }}\left(·|{\mathrm{s}}_{\mathrm{t}}\right)\right)$ are indicative of the fact that MASA effectively encodes the complex environment so that its complex state space can be mapped into a relatively stable encoded vector space.

5.3. Ablation Experiment

This paper proposes several training optimization methods to stabilize neural network training in dynamic environments and improve its convergence speed. This section analyzes the utility of these methods. The effect of training optimization methods on scheduling performance is observed in both large-scale and small-scale complex environments. The simulation parameter settings in this section are consistent with those in Figure 6a.

5.3.1. Reward Dynamic Alignment Analysis

In Figure 10, we plot the convergence of the training results of the MASA algorithm with and without the RDA optimization method during training at different application arrival rates. The introduction of the RDA method makes it clear that the network converges to a stable state earlier and the final training results in better performance. The RDA optimization method maintains the average application execution time in a more stable state, even with changes in the arrival rate of applications. RDA solves the reward inaccuracy problem by buffering the rewards under clock signals, thus improving the performance of network training. RDA is a lightweight optimization method that can be directly applied to similar application scenarios, as it only memorizes the decision start and end times of the task in comparison to the traditional RL method.

5.3.2. Asynchronous Multi-Agent Joint Training Analysis

The direct training of neural networks using a single agent is unstable and has poor performance, as the correlation of the empirical data is high. To improve the convergence speed of network training, we consider the method of asynchronous multi-agent joint training in this paper. Figure 11 shows the effect of different numbers of agents on network training. When the number of agents is one, no multi-asynchronous action joint training is used. The experimental results indicate that asynchronous multi-agent joint training can effectively improve the convergence speed of network training. The greater the number of agents, the shorter the time for the network to converge to a stable state. When the number of agents is 10, the training curve of the network is the smoothest and can converge to a stable value in a shorter period of time, resulting in optimal performance of the network.

6. Conclusions

This paper proposes a neural scheduling algorithm, called MASA, to address the heterogeneous resource-scheduling problem in dynamic environments, and the attention mechanism is adopted to represent complex relations between job DAGs. To train networks fast and stable in a dynamic environment, this paper proposes the RDA, early termination of initial episodes, and asynchronous multi-agent joint training methods. Experimental results demonstrate that the MASA algorithm proposed in this paper outperforms both neural scheduler algorithms and heuristic algorithms in task configurations under hypothetical and real applications. Additionally, ablation experiments confirm the effectiveness of the RDA algorithm and the asynchronous multi-agent training strategy proposed in this paper in enhancing the ability of the network to be trained in dynamic environments. In the current simulation environment, in order to enhance the high fidelity of the heterogeneous scheduling algorithms, we have included the inter-PE communication overhead into the application execution time overhead. However, considering that the communication bandwidth resources between processing elements are relatively abundant compared to the volume of intertask data transmission, we have not scrutinized the magnitude of the communication transmission overhead on the total execution time in our current work, and we will further explore this issue.

Future work will further explore the application of MASA algorithms in large-scale task scheduling and test their reliability in more radio signal processing. It is also crucial to verify whether the scheduling algorithms proposed in this paper have high fidelity and high reliability.