Real-Time Scheduling with Independent Evaluators: Explainable Multi-Agent Approach

Abstract

This study introduces a multi-agent reinforcement learning approach to address the challenges of real-time scheduling in dynamic environments, with a specific focus on healthcare operations. The proposed system integrates the Human-in-the-Loop (HITL) paradigm, providing continuous feedback from human evaluators, and it employs a sophisticated reward function to attenuate the effects of human-driven events. Novel mapping between reinforcement learning (RL) concepts and the Belief–Desire–Intention (BDI) framework is developed to enhance the explainability of the agent’s decision-making. A system is designed to adapt to changes in patient conditions and preferences while minimizing disruptions to existing schedules. Experimental results show a notable decrease in patient waiting times compared to conventional methods while adhering to operator-induced constraints. This approach offers a robust, explainable, and adaptable solution for the challenging tasks of scheduling in the environments that require human-centered decision-making.

1. Introduction

Scheduling workflow has always been highly challenging, and it has recently become increasingly complicated with the growing range of diversified jobs and with the dynamic demands that may require rescheduling. This task has been shown to be an NP-hard combinatorial problem [1] that cannot be solved in polynomial time with conventional game theoretic or linear programming algorithms. Therefore, a plethora of approximate gradient-based optimization algorithms can be used to address computational complexity. Among the family of neural network algorithms, the RL including Multi-Agent Reinforcement Learning (MARL) approach [2] stands out as the most promising one. Namely, Proximal Policy Optimization (PPO) [3] and its extension, Multi-Agent Proximal Policy Optimization (MAPPO) [4], are prevalent as stable baselines to numerical experiments due to their ability to work with time and other continuous spaces. MAPPO requires a number of entities called agents to learn to optimally interact in a shared environment during the simulation process, referred to as a game. The game represents a simplified working process to be optimized, whereas the agents represent the subprocesses within the working pipeline. As explicit computation for each working scenario is no longer required, MAPPO can effectively take into account the uncertainties related to the dynamic changes in the schedule.

In terms of dynamic changes, a human operator appears to be an active component of the workflow to be optimized, featuring a reactive behavior in response to the changes caused by the algorithm. This theoretical concept is referred to as HITL and has been thoroughly discussed in [5]. To summarize, as soon as the operator has the opportunity to actively interfere with the running algorithm, the scheduling problem becomes part of a wider class of systemically complex problems. The NP-complexity of the challenging task of scheduling is aggravated by a number of implicit factors related to human orchestrating.

Another concept, called Human Feedback Reinforcement Learning (RLHF) [6], provides a general approach aimed at attenuating inherent uncertainty that results from a human in the loop. It uses various sophisticated reward function designs related to the preferences an expert expresses over time.

Reward-free approaches [7,8,9] aimed to address the same challenge all have the same limitations as the reward-based methods described above. They require the consent of several independent evaluators to proceed with a decision proposed by a model. This is aggravated with increases in the number of clients acting as those evaluators. One of the ways to mitigate the effects of the above limitation is to address the leading expert who communicates with the evaluators, following them and acting on their behalf, which results in successive rescheduling until one fit for all decisions is reached.

A typical example of such a scenario is arranging high-tech medical intervention flow in a specialized clinic, including surgery, dentistry, and high-tech medical examinations, to name a few. The degree of disease severity determines the acceptable waiting time for the intervention, which can also depend on personal preferences regarding the appointment timing. The clinic, on the other hand, can provide a certain number of interventions each day. A corresponding schedule is based on the above data and is partially observable for patients as required. However, during the waiting time, both the patient’s condition and their preferences regarding the appointment time may change, which urges an ad hoc reorganization of the entire schedule. This may result in the waiting time of other clients increasing or expensive workstations being eventually underloaded.

To address these challenges, we propose an algorithm for real-time rescheduling of medical interventions in response to the eventual changes in the medical state and/or preferences of individual patients while minimizing the deviations in the already established schedules. In terms of the proposed algorithm, each human actor has two roles: a participant in the game process through the intellectual agent representing them, and a direct independent evaluator.

The main contributions of this paper are as follows:

• We devised a novel multi-agent environment to simulate resource allocation in the varying of the planning horizon within dynamic scheduling;
• We designed a sophisticated reward function with an adjustable $\alpha$ level to attenuate the effects of human-driven events;
• We suggested a technique to refine the decision-making process during the sequential rescheduling phase with active human agents present;
• We established mapping between RL nomenclature and a BDI cognitive framework.
• We proposed Large Language Model (LLM)-based tools to elucidate the intricacies of cooperative–competitive game interactions between agents for a supervising human expert;
• We applied our findings to the task of allocating operating rooms (OR).

The rest of this paper is organized as follows. In Section 2, we present a review of the recent studies on the main approaches related to our work. Particular attention is paid to the methods for specifying the parameters of human activity and behavior applied to address the challenge of real-time rescheduling. Section 3 covers the theoretical foundations common to most RL algorithms and to actor–critic architecture in particular. In Section 4, we present a custom gaming environment with an emphasis on the mapping between the environment elements, RL algorithm, and BDI framework to use the above relationship later when discussing explainability and robustness. Section 5 presents a comparative analysis of the proposed algorithm and conventional human heuristic. The further discussion covers current limitations and prospects.

2. Related Works

A survey by Abdalkarim et al [10] offers a comprehensive overview of the possible problem statements of healthcare scheduling and considers a number of classical solutions, including ant colony optimization [11], mixed-integer programming [12], genetic algorithms [13], and Monte Carlo methods [14]. Our problem statement closely relates to their survey’s discussions on dynamic patient admission scheduling with operating room constraints, flexible planning horizons, and patient delay, as well as advanced operating room scheduling. Similarly, our approaches account for constraints such as the total number of available ORs, the availability of qualified personnel, and necessary equipment. However, the following difference is crucial. Another important constraint is introduced: the patient’s consent. Hence, the introduction of human feedback implies a shift from classical planning tools to multi-agent reinforcement learning, as the latter can work in non-stationary environments and adapt to rescheduling during online learning.

The MARL approach and its various representations, including Deep MARL (DMARL), Multi-Agent Actor–Critic (MAAC), and MAPPO, have been widely discussed in recent years. Their advantages and applicability to workflow scheduling have been summarized in reviews [15,16,17,18].

Papers [19,20,21,22] refer to the issue of workflow scheduling under the conditions of dynamic changes in the states of the environment and agents. In [21], task priorities are calculated for individual users and workstations and then fed to a scheduler that dynamically forms the tasks using a deep Q-network model and a swarm algorithm. In [19], each workpiece is treated as an intelligent agent, and the representation of state, action, observation, and reward is introduced based on the Markov decision-making formula. A heterogeneous graph neural network based on graph node embedding is used to compute policies. In [20,21,22], the actor–critic RL method is used to solve the flexible job shop scheduling problem. The actor network is responsible for choosing the most suitable scheduling rule in different states, while the critic network is responsible for outputting the value function of the actions and providing feedback to the actor network to better adjust the scheduling strategy. Papers [20,21,22], among others, have shown that the actor–critic approach is computationally efficient and flexible in terms of the number of jobs, agents, and workstations compared to genetic [21] and graph [19] approaches.

A mandatory condition for the application of the MAAC [18] and MAPPO [4] approaches is the possibility of agent typification [23,24], which may seem to contradict the principles of personalized medicine. However, a review [25] confirmed the credibility and relevance of such typification for the patients of medical institutions. On the other hand, none of the above works appears to suggest the possibility of flexible changes in the preferences of individual agents in the course of the workflow.

The HITL paradigm aims to introduce weakly formalized parameters of human activity and behavior into the models for workflow scheduling. The large amount of recent studies summarized in [2,6,26,27], testifies to the demand for this paradigm in real-life applications. Various approaches have been proposed to meet the main goal of using the HITL paradigm, enhancing the formalization of the description of human activity.

Inverse capturing of domain-expert knowledge based on demonstration is described in [7]. It derives an unknown reward function from the expert’s observed behavior. However, as the number and complexity of the tasks to be scheduled increases, the size of the state space grows dramatically, and scheduling can quickly become computationally intractable. Study [28] uses action-driven learning to extract the scheduling strategies of domain experts. Training examples are based on pairwise comparisons between the scheduled and unscheduled tasks. A pairwise approach describes the behavior of experts in a more natural way and is less resource-intensive. A similar approach is presented in [6]. To support reward learning from inconsistent and diverse human preferences, ref. [29] proposes to stabilize it by regularizing and correcting in a latent space. Other options for reward functions directly learning from human feedback are presented in [30,31,32], to name a few.

In general, this group of approaches reproduces with a certain degree of accuracy the decisions made by an experienced dispatcher; however, they cannot improve or explain them.

Another approach to the formalization of human activity can be experts’ typification, which aims to classify them into statistically homogeneous cohorts. In [33,34], the parameter of typification is represented by the decision-making scheme. Namely, ref. [34] introduces five key features to consider about human feedback when launching HITL RL systems: binary, delay, stochasticity, unsustainability, and natural reaction. In [33], human decision-making strategies have been shown to be clustered as analytic or heuristic, both of which demonstrated similar degrees of success as measured by task performance. This clustering can provide software agents that more accurately represent human behavior. In [35], the humans involved in the scheduling by their place in the process timeline were classified as follows: (i) human in the loop provides sole or collaborative decision-making, (ii) human on the loop refers to supervisory oversight, (iii) human above the loop is related to strategic governance, and (iv) human behind the loop represents output analysis and improvement. However, such typification of experts takes no account of their implicit and spontaneous decisions, which essentially limits the opportunities for generalizing the approach.

Distributing the process participants according to their individual parameters is essential when making a decision. Taking account of this distribution can be considered an advantage compared to the approaches described above. For the scenario of appointments in a medical clinic described in the Introduction, this parameter is participants’ availability time slots. In [36], this distribution is considered as Poisson: users initially propose a single starting time for each of their jobs, then scheduling based on integer linear programming proposes to each user a small number of alternative time intervals, which the user may accept or reject. The authors argue that the initial schedule can usually be quickly improved over a few interaction rounds. Meanwhile, previous investigations [37,38] showed that when individuals execute the tasks based on some perceived priority (which is the case in decision-making), the task timing will follow heavy-tailed distributions. Accordingly, for the processes involving a human decision-maker with spontaneous and implicitly inspired decisions, it seems natural to model the reward distributions using asymmetric functions, including heavy-tailed ones. In [39,40,41], RL algorithms are presented with a non-Gaussian return function; however, it is always considered as being the same for all actors, and no parameterization for different actors seems to be provided.

Since artificially crafted rewards may fail to accurately mirror human intentions, reward-free RL models have become very popular. The Direct Preference Optimization (DPO) approach [42] organizes RL without explicitly setting the reward function, instead using a change in variables to directly define the preference loss as a function of the policy. Preference-based RL [43] directly learns from preference without any reward modeling. To achieve this, it adopts a contrastive learning framework to design a policy scoring metric, assigning a high score to the policies that align with the given preferences. However, direct preference optimization uses LLM as a domain model, and preference-based RL is designed for offline RL tasks. To the best of our knowledge, no adaptation for real-time scheduling has yet been suggested.

AI systems for safety-critical applications are supposed to explain their decisions, actions, or predictions to ensure transparency, explainability, and accountability. This requirement, enshrined in governmental documents of some countries, has given rise to a vast stream of research in this area. Reviews [44,45] present a range of classification features for explainable AI (XAI) systems, which identified the most relevant areas of research for this study.

XAI systems are classified [45] as data-driven and goal-driven. Data-driven XAI algorithms aim to determine the input attributes that account for the output predictions [46]. For example, the SHAP [47] and GradCAM [48] algorithms highlight the features of the input that have had the greatest impact on the classification result performed by the neural network, respectively. Goal-driven XAI [49] aims to create explainable robots or agents that can justify their own behaviors to a user. Although goal-driven XAI is increasingly used in our current AI-dependent world [50], the choice of algorithm type depends primarily on the target person requiring an explanation [51,52].

In [52], the XAI problem is suggested to be considered for a specific agent system from a multi-level perspective: the implementation level is targeted for developers, the knowledge level is for designers, and the domain level is for users. The authors emphasize that explanations for different addresses, even when based on a single source of information, cannot not coincide.

The most commonly used sources of information for constructing explanations in the agent systems are records of important system events, or characteristic trajectories, as well as the complete system log [53,54]. In the latter case, two modes of audit logging have been specified: behavior logs and belief logs [54].

Various agent models have been proposed to describe the agents’ behavioral manifestations with varying degrees of completeness. For example, reactive RL agents [55,56] rely on a simple behavioral policy scheme; that is, state-to-action mapping via trial-and-error interactions with the environment. This agent model provides explainable RL in terms of ongoing evaluation of individual agent parameters; most importantly, the reward functions [57]. However, the most complete understanding of an agent’s behavior is provided by cognitive models, primarily the BDI model [58], which uses folk psychology ideas about human mental attitudes to implement rational agents. Here, belief is the agent’s ideas about the current state of the environment, desire is its ideas about the target state of the environment, and intention is the agent’s actions for the current and following game steps. In terms of multi-agent systems (MAS), explaining emergent behaviors and complex agent interactions present some new challenges [59], and cognitive models work best to address them [60]. Review [61] justified the ample use of the BDI model for constructing agent-based XAI.

Visual and iconic forms are widely used to represent explanations; for example, by highlighting salient features in the input image [48], graphic representation of the relative importance of features [46], and salient steps in agent trajectories with Atari games [57]. On the other hand, in recent years, natural language explanations for RL agents have been gaining attention [62,63]. The application of LLM in the field of XAI has provided new opportunities for generating more nuanced and context-aware explanations [64]. Some researchers [65] have explored the use of LLMs for dynamic role assignment in generating explanations, providing adaptive and flexible interpretation of agent behaviors.

As stated in review [45], the development of metrics for the explainable behavior of MAS provided by XAI systems appears to remain an issue. The authors of [51] provide a well-grounded explanation of the need for their development, whereas [66] discusses some requirements they have to meet. The comprehensive approach (presented in [52]) to developing a system of metrics, taking into account different levels and different addressees of explanation, seems valid as a concept; however, it has not been followed by any practical implementation and has not been developed in any other works. No studies available to date seem to include any examples of the practical implementation of metric complexes in terms of textual representations of explanations.To summarize, in view of the aim of the present work, the main challenge remains to reconcile the spontaneous decisions of individual users regarding the proposed schedule options. Implementation and metric support of a complex explainable MAS may be a viable option.

3. Preliminary

3.1. Reinforcement Learning

Standard single-agent RL settings is a stochastic environment in which the agent acts in discrete time steps and obtains rewards. These settings are targeted at maximizing the expected cumulative rewards over the episodes of the agent interaction with the environment.

We refer to each RL environment as defining state S and action spaces A. While operating in the environment, the intelligent agent observes the current state of the environment or a subset of it, denoted O, and selects an action. In response to this action, the environment changes its state and provides a feedback signal referred to as reward R. Formally, S is the set of states, A is the set of actions available to the agent (which depends on the current state), and $p\left(s_{t+1}\right|a_t,s_t,\dots ,a_1,s_1)$ is the transition probability function. Reward is defined as the function $R(s_{t+1},s_t,a_t)$, which rewards the agent for the transition between states $s_t$ and $s_{t+1}$ via the action $a_t$. For an efficient learning process, an assumption is normally made that the transition probability function represents a Markov process [67]:

$$p\left(s_{t+1}\right|a_t,s_t,\dots ,a_1,s_1)=p\left(s_{t+1}\right|a_t,s_t)$$

(1)

The transition probability depends only on the current state $s_t$ and chosen action $a_t$. The objective is then formalized as learning the probability distribution of different actions in different states, referred to as transition policy:

$$\pi (s,a)=p(a_t=a|s_t=s)$$

(2)

This probability distribution is optimized to maximize the expected reward for moving on starting from state s according to it, which is expressed as the following function:

$$V_{\pi }\left(s\right)=E[\sum _{t=0}^{\infty }{\gamma }^t{R}_{t+1}|s_0=s]$$

(3)

where $\gamma$ is the discount factor, typically chosen within the range $(0,1)$.

To achieve this goal, there are different methods such as value-based, policy-based, and actor–critic methods.

Value-based methods refer to learning a value function:

$$V^{\pi }\left(s_t\right)=E[\sum _{i=0}^{\infty }{\gamma }^i{R}_{t+i}|s_t,\pi ]$$

(4)

where the expected reward ${\sum }_{i=0}^{\infty }{\gamma }^i{R}_{t+i}$ is estimated by the trajectories generated by the transition policy mentioned earlier.

This method defines the optimal transition policy as the one that maximizes $V^{\pi }$. However, it is an epsilon-greedy strategy that is amply used to add exploration ability to agents. It tends to choose the optimal action; however, it saves some exploration space for the remaining options, and it selects the action uniformly at random. Instead of value function $V^{\pi }\left(s_t\right)$, a Q-function is often used according to the Bellman optimality criterion. The Q-function estimates the expected discounted reward after performing action $a_t$ in state $s_t$. It is defined as follows:

$$Q^{\pi }(s_t,a_t)=E[\sum _{i=0}^{\infty }{\gamma }^i{R}_{t+i}|s_t,a_t,\pi ]$$

(5)

Methods that utilize this function are commonly referred to as Q-learning [68].

Transition policy ${\pi }_{\theta }(s,a)$ with parameters $\theta$ can also be directly optimized instead of using $V^{\pi }\left(s\right)$ or $Q^{\pi }(s,a)$ functions. With this approach, the transition policy is often parametrized by some known algorithm. Meanwhile, the goal remains to maximize the expected reward; however, in this case, this is achieved by other methods, including policy gradient, to give an example.

3.2. Non-Stationary

In multi-agent settings, the goal is still to optimize the expected reward. However, here it is performed simultaneously for several single agents. This may seem a minor increment in complexity; however, in practice, the family of methods relying on the Markov property become suboptimal. The transitions and rewards of every single agent no longer depend solely on the current state of the agent, as there are also direct or indirect interactions with other agents.

As most of the conventional methods tend to depend on the stationarity assumption, MARL methods often are focused on reducing non-stationarity that may occur. This can be achieved, for example, by introducing a global coordinator in the game or by aiding the agents with some additional small models that will reduce the uncertainty about other agent actions or consequences [69].

3.3. Multi-Agent Actor–Critic Methods

Both Q-learning and policy gradient methods appear to have some challenges when working in non-stationary environments [70]. Thus, Q-learning has an issue with the violation of Markov property, while policy gradient features very high variance, which grows with the number of agents. On the other hand, application of the actor–critic architecture to multi-agent settings to coordinate the policies of agents ensures learning complex multi-agent strategies.

The actor–critic algorithm combines policy- and value-based methods by introducing a pair of models [71]. The first of them is referred to as the actor; it learns the transition policy (2) with parameters $\theta$ to make decisions. The second one is the critic; it learns (3) or (5) to properly evaluate the actions taken by the agent. This approach provides an algorithm that would be flexible and robust to non-stationarity, as it features a good adaptive model for the transition policy and is also capable of interpolating the value functions for the unvisited states.

One of the most efficient algorithms utilizing the actor–critic approach is PPO. It uses independent actor and shared critic neural networks with some variations possible. Its loss function is designed to (i) minimize the mean squared difference between the estimate of the value function $V^{{\pi }_{\theta }}$ and its factual value $V^{targ}$; (ii) regularize the actor to have a lower entropy of ${\pi }_{\theta }$; and (iii) tune the transition policy ${\pi }_{\theta }$ for maximizing the expected value of the advantage function:

$$A_t(s,a)=Q(s,a)-V\left(s\right)$$

(6)

The advantage function quantifies how much better it is to take action a in state s compared to the average actions in this state.

The learning process of the PPO includes an iterative evaluation of the current transition policy in an environment for T timesteps in N parallel processes.

PPO is straightforwardly generalized to multi-agent setting with the algorithm MAPPO [4], with separate actor networks and a centralized critic neural network evaluating $V^{\pi }\left(s\right)$ based on access to global information to reduce variance from non-stationarity. Actor networks for each agent parameterize ${\pi }_{\theta }(a_i,o_i)$, where $a_i$ is the action of the i-th agent and $o_i$ are its observations.

4. Methods and Materials

4.1. Environment Design

For the experiments to be consistent and reproducible, we chose the Python 3.11 programming language and PettingZoo 1.24.3 framework featuring Agent Environment Cycle (AEC) and Parallel as standardized application program interfaces (APIs) to build a custom environment. Despite AEC being most widely used, in our study, we applied Parallel. On the one hand, the Parallel environment API is based on the highly appealing paradigm of Partially Observable Stochastic Games (POSGs), while on the other hand, it provides agents with simultaneous actions, which meets the intricacies of the real-world resource allocation tasks.

We defined the environment state space as a timeline of a certain length, referred to as the planning horizon, with a limited throughput capacity $\lambda$ on each time slot (Figure 1).

When it comes to real-world applications, it is convenient to start with a uniform target distribution while training and to fine tune with different target $\lambda$ levels later.

A linear design allows us to shift the planning horizon over time, providing a flexible framework for continuous planning models (Figure 2).

Left as is, this state representation is prone to overfitting, which in terms of RL would mean that the algorithm would become perfectly adapted to a certain variant of the gaming scenario rather than learning meaningful insights. To mitigate the effects of overfitting, it has to be trained on multiple instances of the environment in parallel with a limited size of the agent’s observation space and freedom of action.

The action space is designed so that no agent can learn to place its bid on a certain day (Figure 3). It can rather learn its relative position compared to others. At each stage of the game, the agents can move their bids one day further from or back to the starting point of a planning horizon.

Collisions in RL can be addressed in many ways; for instance, we can mask the agent action space when staying on boundaries, define a penalty for choosing an action that causes a collision, or even go further to eliminate the rigid boundaries of a planning horizon.

In our work, we implemented a cyclic action space instead of the above options. This means that if we start from Monday and choose to move left in the planning horizon, we will find ourselves on Sunday, and following the same logic, moving right from Sunday will bring us to Monday. By applying this rule, the agent can reach any day within the planning horizon in no more than three steps, regardless of its initial starting position. Given the limited waiting time, we thus eliminate the unnecessary steps before the agents reach the optimal solution.

The observation space encompasses the agent’s knowledge about itself and about its environment and depends on the window parameter $w\in \{3,5,7\}$, which limits the agent’s vision (Figure 4).

To maintain the size of the observation space throughout the training, we use the full size planning horizon and set an unknown number of bids per day to a negative value. This provides the RL algorithm with a clear distinction between the unknown number of bids, marked as negative one, zero, or a positive non-zero value on a given day.

As stated above, in addition to the knowledge about their surroundings, the agents perceive themselves. In the specific field of our research, this means that the agents assess the health status of the patients they represent. In a broader sense, this self-assessment could apply to any subject or entity.

The patient’s health status (Figure 5) involves a combination of data completeness, difficulty, and the urgency of the medical intervention.

Data completeness evaluates whether the data available are sufficient to provide the appropriate treatment, $c\in \{0,1\}$. Difficulty provides an overall assessment of medical intervention complexity, $d\in \{0,1\}$, whereas urgency limits the agent’s ability to wait, with higher values indicating a more immediate need to receive medical care, $u\in \{1,2,3\}$.

The relationship between the day when the agent decided to bid and the patient’s health status is shown in Figure 4. The mutation rate is demonstrated to slowly increase from the second half of the planning horizon at a rate of 5% per day. This mutation coefficient determines the probability of a change in the patient’s health status and thereby affects the agent’s observations.

The reward function directly depends on the selection of the termination rule. In a highly complex gaming scenario, the termination rule is supplemented with a truncation rule. The former delineates the optimal outcome anticipated, while the latter ensures that the game does not exceed the preset number of the episodes.

We set the termination rule so that the game ends when more than 80% of the agents take the “stay” action. The truncation rule suggests that the game ends when the number of steps taken is equal to the length of the planning horizon.

Therefore, we propose to provide a reward for the agents after each step until one of the above rules is met:

$$r=-(D\left(w\right)+{\beta }_{1}D\left(H\right)){\beta }_2{r}_b+{\gamma }_1{p}_u+{\gamma }_2{p}_o+{\gamma }_3{p}_s$$

(7)

where $r_b$ denotes the basic reward, and the terms $D\left(w\right)$ and $D\left(H\right)$ refer to the discrepancies between the observed and the target states measured within a window of length w and across the entire planning horizon of the length H. The terms $p_u$ and $p_o$ refer to penalties for the agent settling itself in an underutilized or overutilized day, respectively, and the term $p_s$ is the operator set penalty for the agents representing the clients within a group of risk. Parameters $\beta$ and $\gamma$ represent the factors that balance the contributions of the components.

Let us formalize the measurement of the discrepancy D described above:

$$D\left(L\right)=\sum _{d\in L}f\left(d\right)\Delta =f\left(d\right)|s_o\left(d\right)-s_t\left(d\right)|$$

(8)

where $f\left(d\right)$ is the value of the Lévy alpha-stable probability density function on d, the day within the observation window of length L. The terms $s_o\left(d\right)$ and $s_t\left(d\right)$ are the observed and target states of the environment, respectively, on the same day d.

The term $p_s$ is used to refer to clients with a severe lesion and is calculated as follows:

$$p_s=-(d-1)\mathrm{if}k>3$$

(9)

where d is the current position of the agent within the planning horizon and k is the scaling factor.

The scaling factor k attenuates the penalties for the agents representing moderate to high-risk patients and is calculated as follows:

$$argmax(1,((d+(1-c\left)\right)·u\left)\right)$$

(10)

where c is the data completeness, d is the difficulty, and u is the urgency of medical intervention.

As shown in Figure 6, with the observation window broadening, the agents receive more freedom and more ability of making their informed choice. On the other hand, by lowering the adjustable $\alpha$ level, they are more penalized for deviating from the target state at the boundaries of the observation window. The fundamental tenet is that as the $\alpha$ level declines, the value of $f\left(d\right)$ will increase at the extremes of the probability distribution density function, resulting in a proportional rise in the penalty value.

4.2. Rescheduling with Human Feedback

To reduce the computational costs related to rescheduling, we suggest training a limited set of intelligent agents to represent the categories of end-users; in our case, patients. To facilitate efficient operation, we pre-trained the model with twelve agents with twelve health states, shown above in Figure 5. During the inference stage, each patient was embodied with one of these intelligent agents, which would represent their interests.

Figure 7 shows an n-step re-scheduling process, which is designed as follows. At the first stage, an RL algorithm proposes a draft of the future schedule. Multiple human evaluators assess this proposal, and the lead expert receives their affirmative or negative answers. Based on the human feedback provided, the lead expert refines the schedule, and the process is run again. The patients who accepted the slot selected are removed from the game. This results in the agent representing them being deactivated, and the game continues as if they have consistently chosen the “stay” action. Once all the patients have accepted the slots, the game is completed, followed by the evaluation of the quality of the solution provided.

4.3. Evaluating the Schedule Quality

To evaluate the quality of the schedule generated, we used the Kullback–Leibler (KL) divergence measure combined with some less-rigorous, empirically derived ad hoc metrics. These supplementary metrics assess the schedule from the standpoint of the end-user; namely, the operator of the multi-aggregate system.

To apply the KL divergence, the obtained request frequencies had to be converted into pseudo-probabilities. This was achieved by dividing the frequency of requests on the selected day by the sum of the frequencies over the entire planning horizon.

$$D_{KL}(P\Vert Q)=\sum _{d\in H}P\left(d\right){log}_2{\displaystyle \frac{P\left(d\right)}{Q\left(d\right)}}$$

(11)

where $P\left(d\right)$ represents the relative frequency of requests on day d within the simulated planning horizon of the length H, and similarly, $Q\left(d\right)$ denotes the relative frequency of requests corresponding to the constraints set by the operator.

To assess the anticipated violations of the preferences imposed by the operator, the following expression is applied:

$$V_P={\displaystyle \frac{1}{H}}\sum _{d\in H}{\displaystyle \frac{s_o\left(d\right)}{s_t\left(d\right)}}$$

(12)

where $s_o\left(d\right)$ and $s_t\left(d\right)$ represent the mean observed state assessed over multiple episodes and the target state of the environment, respectively, on day d within the planning horizon of the length H.

Furthermore, to evaluate the efficiency of the scheduler in view of the risk-group clients, we need to calculate the mean position occupied by the agents representing the interests of patients; in other words, the average waiting time (AWT) of the patients with a condition classified as severe:

$$AWT(k>3)={\displaystyle \frac{1}{n}}\sum _{agent\in Agents}{p}_{agent}(k>3)$$

(13)

where $p_{agent}(k>3)$ represents the position of the agent with a scaling factor of $k>3$ obtained over n episodes.

4.4. Mapping Between Belief–Desire–Intention Structure and Reinforcement Learning Terminology

Figure 8 illustrates the conceptual transitions from classical RL paradigms to the constituent elements of the BDI framework.

Beliefs are constructed from the intelligent agent internal representations of the current and target states of its environment. In terms of RL, beliefs are similar to the state space. The agent desires encapsulate the priorities established by the MAS engineer or acquired through iterative interactions with the environment. The reward function is the closest representation of the short-term desires. Intentions delineate the actions that the agent plans to execute to bring the observed state of the environment to the desired target state.

At the level of the conceptual architecture of the Actor–critic model, all the three components—beliefs, desires, and intentions—are collected within the replay buffer and are used in critic network training. The values derived from these components subsequently contribute to actor network training.

Mapping RL concepts into the BDI paradigm facilitates the integration of BDI goal-setting methodologies and develops explainable RL-agents, thereby fostering the interpretability and transparency of autonomous decision-making.

4.5. Multi-Level Explainability

In view of our implementation of MAS, its elements have a single global goal but are aimed to meet the interests of multiple human representatives who have their own local interests that are not necessarily strictly aligned. Therefore, following [72], we need a different explainability for each end-user. The resulting MAS has three levels of explainability (Figure A1).

At the first level, we have ample information that requires the end-user to have specialized engineering training in terms of the structure of the neural networks constituting the RL-agents, as well as the environment states, agent actions, and rewards at each step of the training. The above information is stored in the replay buffer.

The second level is a more generalized representation of the beliefs, goals, and intentions of the RL-agents. It also documents the patterns between the customized operator, the reward accrual process, the observation window parameters, and the level of $\alpha$.

The first and second levels are connected by the work of a large language model; therefore, a sensitivity analysis of prompts to the unit-words highlighted can be useful and interpretable by both the engineer and the operator.

The third level of explainability implies an explanation provided for the client. Since they have normally not had the required training, the interpretation at this level is from the operator to the client.

4.6. Natural Language Bridging

Language models can be used to generalize the concepts of beliefs, desires, and intentions to provide a bidirectional interface between the explainability levels of the engineer and the operator of a MAS.

For interpreting the system logs, we developed a series of prompts guided by the Chain of Thoughts (CoT) methodology [73] (Figure A2).

Generally, CoT refers to a technique where a sequence of reasoning steps and intermediate thoughts is explicitly represented in successive prompts. This method leverages the capabilities of language models to follow structured thinking patterns, breaking down the tasks down into manageable subtasks and thereby improving the quality of the responses.

Appendix B presents a complete list of the prompts devised from the Claude Sonnet 3.5 language model. We chose it for our experiments for its excellent reasoning over the text.

4.7. Prompt Sensitivity Analysis

To ensure the reliability of the outputs produced by a language model, it is crucial to identify the unit-words to which the model assigns greater significance when generating responses.

Our hypothesis was that the model assigns greater significance to those unit-words whose alterations result in a significant variation in the model’s output. It was therefore essential to evaluate the extent to which the model’s response is sensitive to the input prompt.

To achieve the above, we proposed using the same Claude Sonnet 3.5 to mask the key concepts and related unit-words in the prompt, as illustrated in Figure A3. Appendix C presents a complete list of the prompts devised.

We suggest evaluating the difference in responses by inputting the masked prompt into the model. To this end, we use cosine similarity provided by the scikit-learn 1.5.1 scientific computing framework. The above measure is appropriate for comparing texts in terms of their overall meaning or their semantic similarity. The calculation is as follows:

$$cos\left(\theta \right)={\displaystyle \frac{a·b}{\parallel a\parallel \parallel b\parallel }}$$

(14)

where a and b are the responses to masked and source prompts, and $\theta$ is the angle between the representations of responses in a vector space.

The Jaccard distance measure is a more straightforward approach that circumvents the translation of the source language model responses into a numerical form, which would otherwise require TF-IDF tokenization. Moreover, the Jaccard measure is more granular in its approach, focusing on the individual words as the fundamental units for comparison. The calculation is as follows:

$$J(a,b)={\displaystyle \frac{|a\cap b|}{|a\cup b|}}$$

(15)

where a and b are the responses to masked and source prompts, respectively.

To delve further in terms of granularity. In comparing two texts, we suggest considering the Levenshtein distance measure. This measure was designed with the specific purpose of detecting the smallest character-by-character differences and is therefore particularly appropriate for the situations where the phrase is relatively short and structured. The calculation is as follows:

$$\begin{array}{c}\hfill L(a_i,b_j)=\left\{\begin{array}{cc}max(i,j)\hfill & \mathrm{if}min(i,j)=0,\hfill \\ min\left\{\begin{array}{c}L(a_{i-1},b_j)+1\hfill \\ L(a_i,b_{j-1})+1\hfill \\ L(a_{i-1},b_{j-1})+1_{(a_i\ne b_j)}\hfill \end{array}\right.\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(16)

where a and b are the responses to masked and source prompts, respectively, and i and j are the current indices of the two strings under comparison.

4.8. Experiment Design

The unique nature of the proposed game environment precludes the use of the conventional benchmarks for a direct comparison. Therefore, we conducted a theoretical experiment instead.

First, we randomly set the environment, its initial parameters, and constraints. Then, we created twelve unique patients, with their internal parameters of each randomly selected in accordance with the constraints of the queue data structure.

To demonstrate how the system functions, we implemented a vanilla version of the MAPPO algorithm and the simplest form of sorting first-in-first-out (FIFO) logic applied in the operational block.

The first step was to allocate patients using the FIFO method. Once the first day was full, the scheduling horizon was shifted one day forward, in accordance with the constraints set by the operator on the $\lambda$ levels for each day.

To utilize the MAPPO, we trained one RL agent for each health category. During training, each of the agents had an independent actor network and an access to the centralized critic network. After training, only the weights for the actor networks remained, and these were used in the time allocation process.

All measurements were conducted following the two scenarios: zero-shot and few-shot. In the zero-shot scenario, the compared algorithms were executed once, and the metrics calculated from the experimental results (as described in Section 4.3 and Section 4.7) were recorded without any modification. The few-shot scenario involved multiple algorithm execution, and the resulting metrics were averaged to mitigate the effects of the uncertainty related to the probabilistic nature of the experiment.

5. Results and Discussion

Figure 9 illustrates a gradual decline in actor and critic network errors starting from approximately the 2000 steps point. The mean episodic reward increases rapidly, approaching the zero value.

Table 1. Experiment results.

Algorithm	Metrics	Zero-Shot	Few-Shot Average (n = 10,000)
FIFO	$D_{KL}$	0.0	0.0
	P, %	0.0	0.0
	$AWT(k>3)$, days	2.0	2.22
	$AWT(k\le 3)$, days	2.0	2.25
MAPPO	$D_{KL}$	0.0025	0.0062
	$V_P$, %	2.86	9.96
	$AWT(k>3)$, days	1.5	1.75
	$AWT(k\le 3)$, days	2.5	2.05

Note: Values in bold indicate the most satisfactory results for each algorithm.

shows the results achieved in our experiments in zero- and few-shots scenarios.

When FIFO is used, the results demonstrate strict adherence to the constraints imposed by the operator on the maximum lambda throughput of each day, which was in accordance with our hypothesis. However, the mean waiting time for the patients with the scaling factor $k>3$, which refers to the patients with severe health conditions, is close to the weighted mean position of the agent in the planning horizon. Conversely, MAPPO performance was noticeably higher than basic heuristic: for a minor concession on the part of the operator (five-shot average deviation from the operator preferences $<10\%$), the waiting time for a patient with a severe health condition to have a scheduled surgery conducted was reduced by ∼20%.

Figure 10 illustrates a four-step game episode. The game commenced with twelve agents (A1–A12), each of them situated at a randomly determined initial position.

Subsequently, each agent took a step. The actions selected by the agents within a single step were considered as parallel. At the outset of the game, agents A3 and A11, representing the patients with the most severe health conditions (scaling factor = 6), were located at the far end of the planning horizon ($AWT=4.5$). By the time the game episode was over, the agents were shifted closer to the beginning of the planning horizon ($AWT=1.0$).

To assess the scalability of the system, we increased the number of patients to one hundred. In this case, each patient was assigned one of the twelve health categories and was treated using the same pre-trained models (Figure 11). To accommodate the inherent unpredictability of real-world scenarios, the patients were designed to decline the operator’s proposed schedule with a probability of 30 to 70%. In such instances, the intelligent agents that were not aligned with their interests was randomly reinitialized within the environment, and the game resumed as if other agents were frozen.

As illustrated in Figure 11, the agents were randomly initialized. Starting with the fourth iteration of the replanning, (see Figure 11b), the schedule began to manifest improvement. In the final iteration of the planning (see Figure 11c), the schedule exhibited a minimal movement, approaching the outcome anticipated theoretically.

Since the replanning uses the same models as those used to estimate the behavior of the twelve pre-trained agents, the $D_{KL}$, $V_P$, and $AWT(k>3)$ measures remain close to those presented in Table 1.

As the number of patients and the service efficiency increase, the issue of induced demand arises in the development of resource allocation systems [74,75]. In [76], and several potential strategies for mitigating the effects of imposed demand in hospitals are outlined. Our system supports two of these strategies, namely: (i) medical decision-making by the highest-ranked individual in a medical team, and (ii) raising awareness among healthcare recipients. Thus, we used the 12 health categories described in Figure 5 and a scaling factor k (see expression 10) to favor the highest-ranked individual and inform the patients by collecting their consent during the iterative selection of the optimal schedule.

The explainability of the system considered was achieved through log analysis by large language models. The following example presents a summary of the behavior of agent number 10 during an experiment:

Agent 10 starts with a moderately urgent and complex task scheduled for Monday. Initially, with incomplete information, the agent assesses its situation as moderately unfavorable and decides to move forward in the week (from Monday to Tuesday). Upon moving to Tuesday, the agent reevaluates its beliefs, showing an updated but still incomplete understanding of the situation. Despite a slight improvement in its assessment (desire becoming less negative), the agent chooses to move backward, returning to Monday.

A comprehensive list of all responses generated by the large language model is given in Appendix D.

To enhance the reliability of responses from large language models and to streamline the prompt engineering task, we devised the technique described above (Method, Section 4.7) to assess the sensitivity of prompts to arbitrary alterations in the input tokens.

Figure 12 provides an example of the sensitivity analysis of a large language model to the alterations in the source prompt.

In this case, the words that are part of the same concept were replaced with the [MASK]. The resulting prompt was then provided as the input to the large language model. The resulting cosine distances were averaged, with the values presented in

Table 2. Similarity measures for different prompts and concepts.

Prompt	Concepts	Cosine Similarity		Jaccard Distance		Levenshtein Distance
		Hard	Soft	Hard	Soft	Hard	Soft
Appendix B.1	Intelligent agents	0.3214	0.6237	0.3630	0.6207	0.3832	0.6771
	Reinforcement learning	0.3361	0.6237	0.3143	0.6207	0.5689	0.6771
	Simulation	0.5473	0.5840	0.5333	0.5938	0.5689	0.6911
	State transitions	0.2748	0.8149	0.2647	0.8148	0.4192	0.7869
	Resource optimization	0.3641	0.3150	0.3429	0.3077	0.3643	0.4792
Appendix B.2	Belief–desire–intention model	0.2841	0.5722	0.3243	0.5000	0.3321	0.4542
	Agent beliefs	0.4513	0.6708	0.4146	0.6111	0.5214	0.6044
	Agent desires	0.4885	0.8622	0.4103	0.8571	0.4786	0.9000
	Agent intentions	0.6294	0.6852	0.5455	0.7105	0.4643	0.8071
	Planning horizon	0.4138	0.5179	0.4000	0.4634	0.3643	0.5018
Appendix B.3	Simulation	0.2617	0.2369	0.2813	0.2727	0.3698	0.3708
	Reinforcement learning	0.2826	0.2470	0.2500	0.2727	0.4688	0.4944
	Beliefs	0.5121	0.3298	0.4828	0.4138	0.5885	0.6461
	Intentions	0.2989	0.2462	0.3429	0.2432	0.5156	0.3539
	Rewards	0.3063	0.1858	0.3125	0.2105	0.5052	0.3708
Appendix B.4	System logs	0.5612	0.2060	0.4348	0.0893	0.6111	0.2310
	Log analysis	0.2049	0.3584	0.0656	0.3043	0.1941	0.4196
Appendix B.5	Belief–desire–intention model	0.2950	0.7727	0.2237	0.6034	0.2978	0.5559
	Agent analysis	0.9780	0.8950	0.9608	0.8113	0.9801	0.7224
	Reasoning reconstruction	0.3038	0.7133	0.1719	0.6071	0.3052	0.7093

Note: values in bold indicate the highest similarity scores for each measure and concept.

(line Appendix B.1, cosine similarity, few-shots, hard).

Instead of completely replacing the term with [MASK], we found it was equally viable to substitute it with a less-specific synonym. This is shown in the soft cosine similarity column in Table 2.

6. Limitations

6.1. Direct Preference Optimization

Once trained, a MAS should be able to adapt over time to new preferences set by the operator. To achieve this, we suggest following the direct preference optimization cycle (Figure 13).

To this end, we suggested assigning an additional penalty $p_d$ which refers to the discrepancy between the RL proposed schedule and the updated operator preferences. The critic network, which is normally eliminated after the model has been trained, could be reused in a fine-tuning process with the episodic reward r, exponentially smoothed as a weighted sum of r and $p_d$.

6.2. Day and Time Management

In the present study, we considered only planning the day of the patient appointment. However, for more complex scenarios, it would be necessary to take into account both the day and the time of the appointment (Figure 14).

In this case, the game takes a form similar to classic maze pathfinding problems, where the free cells refer to the paths, and the occupied ones are the walls. The observation window in this case becomes a 3 × 3 square, and the number of actions increases to 5.

6.3. Linked Entities

In some exceptional cases, it may be necessary to consider the possibility of placing two patients in strict succession. In such instances, swarming tactics may be a viable solution. One potential approach would be to incorporate an additional action that copies the final step of a given agent. This would result in both agents moving in the same direction while remaining separated by a single temporal step.

7. Conclusions and Future Work

This study presents a novel MARL approach designed to address the challenges of real-time scheduling in dynamic environments, particularly within the context of healthcare operations. The system proposed enhances both the efficiency and the adaptability of scheduling processes by integrating human feedback through the HITL paradigm and leveraging sophisticated rewards with adjustable $\alpha$ levels. The integration of the BDI framework enhances the explainability of agent behaviors, thereby resulting in a transparent and user-friendly system. The experiment outcomes illustrate the advantages of this approach over conventional methods, particularly in reducing patient waiting times and aligning with operator preferences. Future work would focus on refining the explainability mechanisms and extending the applicability of the model to other domains requiring dynamic and human-centered scheduling solutions.