Multi-Agent Reinforcement Learning Model Simulation for Attention-Deficit Hyperactivity Disorder Children

Abstract

Background: A child with Attention-Deficit Hyperactivity Disorder (ADHD) faces two issues: inattention and hyperactivity/impulsivity. These two symptoms make the child’s life more challenging compared to non-ADHD individuals. Therefore, one of the steps toward better quality of life involves cooperation with and contact with the environment to better address this condition. Thanks to Artificial Intelligence (AI), doctors, caregivers, and parents are increasingly better able to understand the hardships these children face. One AI technique is Reinforcement Learning (RL). Methods: We propose an RL model simulation with 44 child agents with or without ADHD, using the Independent Deep Q Network (IDQN), Value Decomposition Network (VDN), and QMIX algorithms. Results: Comparing the results obtained with these three algorithms, children with ADHD find it more challenging to choose the maximum rewards than neurotypical children (395 at episode 300 for non-ADHD compared to 340 at episode 120 for ADHD using IDQN, 69 from episode 90 for ADHD compared to 82 for non-ADHD via VDN, and 31 at episode 110 for ADHD versus 28 at episode 110 for non-ADHD with QMIX). Conclusions: The simulated ADHD agents struggle to aim for the maximum rewards as much as neurotypical children.

1. Introduction

Attention-Deficit Hyperactivity Disorder, or ADHD, is a neurodevelopmental disorder that affects children more frequently than adults [1]. Two main symptoms characterize this condition: inattention and hyperactivity/impulsivity. People with this disorder have difficulty staying focused on a task for as long as neurotypical individuals (TD), are easily distracted, quickly lose track of the main idea, move in all directions, have trouble waiting their turn, and exhibit many other behaviors [1]. ADHD affects boys more frequently than girls (15% versus 8%) [2]. Some researchers and scientists confirm that this disorder can be associated with other illnesses. For example, in China, 17% of children with ADHD suffer from anxiety disorders [3]. Moreover, since COVID-19, people affected by this disorder have been feeling more isolated than those without ADHD [4]. Furthermore, a meta-analysis has confirmed that children with ADHD are 17.3 times more likely to be depressed [5]. According to [6], 70% of children with ADHD suffer from sleep disorders. In addition, 40–50% of children with ADHD have speech and/or language impairments [7]. According to a systematic review, approximately 30% of children with ADHD are epileptic [8]. Furthermore, according to a study evaluating dysgraphia in Arabic, 50% of children with ADHD have writing problems [9]. On the other hand, 20% of children with ADHD also have eating disorders [10]. Another study showed that 17 to 48% of children with ADHD injure themselves and fall more often than those without ADHD [11]. On the other hand, several researchers have noted the difficulties these children experience. For example, Ref. [12] noted that children with ADHD have a lower quality of life, lower levels of happiness, and higher levels of sibling bullying compared to children without ADHD. Ref. [13] found that this group is more prone to academic failure, social rejection, low self-esteem, emotional dysregulation, risks of delinquency in adulthood, and many other challenges. Ref. [14] demonstrated, using a 2-year follow-up, that one of the reasons why children and adolescents with ADHD have a more complicated school life is due to an alteration in a brain region called Executive Function (EF). On this basis, numerous Artificial Intelligence (AI) techniques have been employed to support these children. Among these techniques, we can mention Machine Learning (ML), Deep Learning (DL), Reinforcement Learning (RL), and many others. In this work, we propose a Multi-Agent Reinforcement Learning simulation in an ADHD context and adapt and compare three value-based algorithms in an inspired but not clinical environment. The remainder of the paper is organized as follows: Section 2 presents tutorials that use AI in ADHD research. Section 3 discusses the principles of Reinforcement Learning and its extension to multiple agents (MARL). Section 4 proposes a methodology based on Reinforcement Learning techniques for ADHD. Section 5 opens discussion and results, and Section 6 concludes the paper.

2. Related Works

Several Artificial Intelligence (AI) techniques have contributed to supporting this research.

Monitoring and understanding cognitive and behavioral characteristics: Ref. [15] proposed an extended reality solution to understand and support children with ADHD through 3D graphics, immersive games, and various other tools, utilizing multifaceted data such as eye movements, voice, and other factors. Ref. [16] proposed a simple fixation test using a tablet and the GraphPad Prism AI software to measure attention and gaze toward and away from a target. They found that, during a simple fixation, children with ADHD had more saccades, meaning they moved more abruptly and more frequently than children without ADHD, and that, during anti-saccade tests, they required more correction time to orient towards their target than non-ADHD children. As for [17], authors employed techniques from Machine Learning (ML), Deep Learning (DL), Random Forest, Temporal Convolutional Network (TCN), and Adaptive Control of Thought-Rational (ACT-R) to better understand the cognitive processes of individuals with ADHD, ultimately aiming to improve diagnosis and treatment. Their model achieved 98.21% accuracy and 93.86% recall. Other researchers have focused on the reasons why creativity in children with ADHD is higher than in typically developing children. For example, Ref. [18] conducted a pilot test using four stimulating environments, varying colors (from lighter to darker) and sounds (from instrumental to background). The results show that changes in these two factors significantly affect the children’s creativity.

Other researchers have focused more on Reinforcement Learning (RL) techniques in the context of ADHD.

Assisted treatment and intervention with Reinforcement Learning: Ref. [19] proposed a framework that first applied the principles of Universal Design for Learning (UDL) and the RL technique Q-Learning to teach skills to a student with ADHD, and then recommended learning tools and stimuli based on the student’s development. To achieve this, they treated the student’s behavior as the environment, the state as the schema presented to the student, and the action as the agent’s response to the environment, along with a Q-table to store rewards. They then combined RL with UDL, which involves implementing a range of means for representation, engagement, and expression. Representation is divided into four levels: reading and writing, visual, auditory, and kinesthetic. Expression, linked to representation, includes two stages: evaluation and interaction, each with four sub-stages. For evaluation, there is a final reading and writing exam, a quiz-style test for the visual and auditory parts, and embedded questions and quizzes for the kinesthetic part. Regarding interaction, there are links in the reading–writing and visual sections, forums and debates for the auditory section, and debates and interactions between teachers and students for the kinesthetic section. Engagement has two levels: the number of course units and their navigation, which is divided into four sub-steps: basic chapters and courses, horizontal sections, horizontal and vertical subsections, short clips, and customized programs. The results show improved student performance when combining RL with UDL. Ref. [20] developed an application and a toy using Q-learning to better regulate the emotions of children with ADHD. Using a heart rate frequency bracelet, several assessments were conducted, and data were collected to evaluate the impact on stress reduction and user engagement. The methodology was as follows: first, to better understand ADHD, the authors conducted qualitative and quantitative research. Next, they developed a framework comprising three fundamental components of ADHD emotional regulation: regulation methods, children, and ADHD itself. Then, they integrated Q-Learning to store rewards in a Q-table, along with heart rate data collected by the heart rate monitor. They also designed a small toy with sensory sensors. Finally, they developed an application that provides access to heart rate data, meditation sessions, and other features. The objective of [21] was to improve health interventions for individuals with ADHD by using a Reinforcement Learning (RL) technique, SARSA, and computational persuasion (CP) to develop a tool for adaptive, personalized computerized cognitive training. They set three missions: to verify the effectiveness of RL-based cognitive training, to explore advanced RL and CP techniques, and to conduct a clinical trial to estimate the impact of AI-driven cognitive training (CCT) on cognitive functions. Ref. [22] aimed to create a multimodal framework with Deep Learning (DL) and RL to improve accuracy and decision-making. To achieve this, they used a multimodal approach that integrated data types such as images, text, and sound. Next, they applied Transfer Learning to enhance object recognition performance. Subsequently, they employed RL techniques such as DQN, PPO, and A3C to enhance decision-making. They concluded that PPO and A3C achieved 90.2% accuracy. Ref. [23] used value-based Reinforcement Learning techniques to obtain the internal value and decision policy for 54 participants (28 with ADHD and 26 without) who were presented with a challenge comprising three tasks. For example, the task was to find an image of a puppy. In this task, there are two houses; each house contains two rooms, and each room contains two chests, with a dog image in one of them. The researchers observed that individuals with ADHD had more difficulty choosing the correct option and took longer to reflect than neurotypical individuals during the initial task.

Personalization and decision-making with Machine Learning and Reinforcement Learning: Ref. [24] proposed a combination of Machine Learning and Reinforcement Learning to better differentiate between Autism Spectrum Disorder (ASD), the combined ASD and Attention-Deficit Hyperactivity Disorder (ADHD), and neurotypicals, as well as to enhance and optimize feature extraction in the model. For the Machine Learning part, they used a multi-task convolutional neural network (CNN) model that takes as input a dataset from two sources: (1) auxiliary tasks distinguishing between ASD and non-ASD and (2) the target task differentiating ASD from ADHD. The features from these two steps are extracted separately, with some shared and some private, and then combined according to the CNN model’s requirements. Regarding the Reinforcement Learning component, Q-Learning was used to optimize the model. In this step, an agent selects an action, which is a layer of the network, by consulting the Q-table according to the current state or by selecting a random action and adding the action to the model. Then, the model is trained and validated based on the number of layers, and the reward is returned to the Q-table for updating. When the number of layers reaches its limit, the model restarts. The results showed that combining these two techniques (ML and RL) increased performance by 11.07%.

3. Background

Reinforcement Learning (RL) is a subfield of Machine Learning (ML) [25] that, unlike this discipline, learns from a sequence of actions and rewards/punishments in an environment [26]. The two main pillars of RL are Professors Richard Sutton and Andrew Barto [27]. In RL, the most essential components are the agent, which represents any entity in an RL problem, the environment, the setting where the agent operates, the states, which represent the agent’s situations, the actions, which are what the agent can perform as tasks in its environment, the rewards or penalties, which the agent receives as feedback when performing a positive or negative action, and the policy, which helps the agent choose the best actions to maximize its rewards [28,29].

Single-agent Reinforcement Learning (RL) can be subdivided into K-Armed Bandit methods [30] and Markov Decision Processes (MDPs). MDP is a category of RL divided into two types: Model-Free (e.g., Monte Carlo methods or Temporal Difference methods) and Model-Based (e.g., Dynamic Programming) [28,30]. The Model-Free category itself is subdivided into two subcategories: value-based (Q-Learning, SARSA, or DQN [26,28,31,32,33,34,35]) and policy-based (REINFORCE [36,37] and actor–critic [38,39,40,41]). The Multi-Agent approach (MARL), on the other hand, involves multiple agents that act on environmental states and receive individual or collective rewards or penalties. MARL requires more complex models [42]. Figure 1 below shows a diagram related to MARL.

MARL models can be classified as normal or stochastic games. The Normal-Form Game, like the K-Armed Bandit, involves multiple agents but only one state. It is classified by reward type as zero-sum, common-goal, or general-sum. The Stochastic Game, by contrast, uses an environment composed of multiple states that evolve according to the agents’ actions and transition functions [42]. However, MARL algorithms face some obstacles, for example, non-stationarity, in which the environment changes with each agent action, and scalability, in which the joint action space grows with the number of agents. Another challenge is credit assignment, which involves the difficulty of distributing rewards fairly and equitably [43,44], especially when one agent contributes the most or the least. Below is a figure listing most of the challenges encountered by MARL.

There are several ways to categorize MARL techniques, such as by interaction mode or training mode. There are three modes of interaction: fully cooperative (Team-Q-Learning [45], OPTQTRAN [45], QTRAN or QTRAN++ [46], or Comm-MAPPO [45]), fully competitive (minmax-Q or DP-MA2C [45]), and mixed [45]. A fundamental concept to understand in this category is the Nash equilibrium. A Nash equilibrium involves an agent believing that their strategy or policy is optimal and that they will not change it if the other agent does not alter their policy [45,47,48,49]. Regarding the mode of training, there are three categories: Decentralized Training and Decentralized Execution (DTDE) [50,51] (such as IPPO and IQL [49]), Centralized Training and Centralized Execution (CTCE) [50,51] (such as CommNet or BicNet [49]), and the hybrid category, which is Centralized Training and Decentralized Execution (CTDE) [50,51] (VDN, MAPPO, QMIX, and so on [49]). It is also possible to categorize MARL algorithms into value-based (G2Anet [52], Primary–Secondary Multi-Agent Policy Gradient [53]), policy-based (G2Anet [52], Primary–Secondary Multi-Agent Policy Gradient [53]) or both (MADDPG-M [54], MAGIC [55], DCSS [56]), or based on the reward mode (VDN [57], QMIX [58], COMA [59]), or based on communication (DIAL [60], ATOC [61], IC3Net [62], IMAC [63], LSC [64], NeurComm [65], ETCNet [66], IP [67], IS [68]), and so forth.

4. Materials and Methods

As mentioned in Section 3, algorithms can be categorized as value-based, policy-based, or both. In our case, individuals with ADHD tend to explore various options and hesitate frequently. In other words, if they are not well guided, they may make poor decisions, which can affect their future [69,70,71,72,73]. Therefore, we adopt a value-based, deterministic approach with complete observation. Among the algorithms that meet these criteria, we can mention Independent Deep Q Network (IDQN), Value Decomposition Network (VDN), and QMIX. Based on this, we can model our multi-agent Markov Decision Process (MDP) as (S, A, P, R, γ). However, it is important to note that handling an individual with ADHD requires going through multiple actions, not just a single action. Based on this, we use a deterministic Markov game approach, as shown in Figure 2.

4.1. Objective Function

For a child/adolescent with ADHD, it is tough to address both symptoms at the same time; it is better to focus on one symptom at a time, which is the goal of our objective function, written below. The weighting factor $\alpha$ represents the importance of choosing between the two symptoms. The threshold is 0.5 [74,75,76] to better balance hyperactivity and inattention, promote stable learning, and avoid bias. If alpha is less than or equal to 0.5, we focus on inattention; otherwise, on hyperactivity. The $R_{Inattention}$ and $R_{Hyperactiviy}$, respectively, represent the penalties for inattention and hyperactivity, which can be calculated as 1—the attention reward and 1—the activity reward (the attention and activity rewards are calculated in Section 4.6).

$$J=min(\alpha \sum R_{Inattention},(1-\alpha )\sum R_{Hyperactivity})$$

(1)

4.2. Environment

The environment is designed to capture multiple behavioral and contextual factors related to ADHD. For clarity and interpretability, these factors are illustrated in four conceptual feature groupings: the first focuses on sweet foods, the second on healthy foods, the third on physical activity, and the fourth on writing. This grouping is introduced solely as an explanatory abstraction to reflect domain knowledge and does not correspond to separate vector structures in the implementation. The sweet and healthy foods in the first and second vectors are Boolean variables.

The variables are used as follows: 1 if one or many agents consumed the food and 0 otherwise. Regarding the third vector, the field for the frequency of physical activity is 0 if the individual has not performed any physical activity, 1 if the individual has performed physical activity for 1 to 3 days, 2 if the individual has performed physical activity for 4 to 6 days, and 3 if the individual has performed physical activity every day. The physical pain field is a Boolean variable equal to 0 if the individual has not been injured and 1 otherwise. For the fourth vector, the writing tilt field takes the value 0 if the tilt is straight, 1 if it is ascending, 2 if it is wavy, and 3 if it is descending. The letter deviation field has a value of 0 if the letters are written in a proper oblique manner, 1 if they are written vertically, 2 if they are written obliquely and variably, and 3 if they are written in a left oblique manner. The font size value is 0 if the font size is between 3 and 4 mm, 1 if it exceeds 5 mm, 2 if it is less than 2 mm, and 3 if the size is variable. The values of these four vector variables vary with the agents’ actions. In practice, all features are combined into a unified representation used internally by the environment. To train this environment, we use the XGBoost machine learning algorithm with a seed of 42 and an 80–20% train–test split, given the number of samples and the number of individuals per ADHD category. Then, we trained the agents using three MARL algorithms and established a benchmark in Section 4.8.

4.3. Agents

In our MARL model, we have 90 agents, some with and some without ADHD. Among them, 44 are children, and 46 are adolescents. Among these children, 14 have both symptoms of ADHD, 8 have hyperactive ADHD, 5 have inattentive ADHD, and 17 do not have ADHD. Among these 46 adolescents, 7 are both hyperactive and inattentive, 9 are hyperactive, 6 are inattentive, and 24 do not have ADHD. In our study, we focus only on children.

4.4. Actions

In our modeling, we chose actions related to food, sports, or writing that can be classified as hyperactivity, activity, inattention, or attention. For example, the first, second, fifth, and sixth actions are hyperactivity actions. The third, fourth, and seventh actions are related to activity. The eighth and ninth actions are inattention actions, and the eleventh and twelfth actions are about attention.

Table 1. Agents’ actions.

Indices	Action	Explanation
1	Eating sugar food	According to [77], ADHD children eat more sugary food than non-ADHD children.
2	Eat less proteins	[77] found that children with ADHD consumed less protein-based food.
3	Eat more protein	[77] concluded that increased protein intake reduces ADHD symptoms.
4	Eat less sugary food	In agreement with [77], a decrease in sugar consumption could alleviate ADHD symptoms.
5	Fall	A systematic review mentioned by [78] indicates that children with ADHD have more fractures than those without ADHD.
6	Move clumsily	According to a study mentioned by [79], children with ADHD have widespread subtle motor activities and greater motor overflow than non-ADHD children.
7	Walk more slowly	Studies cited by [80] indicate that a slower walking pace could alleviate ADHD symptoms.
8	Making graphic design mistakes	According to [81], children with ADHD make more errors in drawing than neurotypical children.
9	Write an incoherent text.	[82] mention that children with ADHD write more incoherent texts than children without ADHD.
10	Write more often	The findings by [83] highlight that more frequent writing practice may increase attention in individuals with ADHD.
11	Highlight the words	A study conducted by [84] suggests that highlighting words can improve attention in individuals with ADHD.
12	Summarize the paragraphs	[82] propose paragraph brainstorming as an intervention to reduce inattention in individuals with ADHD.

below describes and references these actions.

4.5. States

Since we are working with a deterministic Markovian game model, we represent the states as scalars ranging from 0 to 3. Four states are possible: a scalar value of 0 for ADHD agents with both symptoms, a value of 1 for hyperactive ADHD agents, a value of 2 for inattentive ADHD agents, or a value of 3 for agents without ADHD:

$$S_{state}\in [S_0,S_1,S_2,S_3]$$

(2)

$$\{\begin{array}{l}If S_{state}=S_0, then the agents are ADHD-C \\ If S_{state}=S_1, then the agents are ADHD - H\\ If S_{state}=S_2, then the agents are ADHD - I\\ If S_{state}=S_3, then the agents are not ADHD\end{array}$$

(3)

4.6. Rewards

Rewards are also subdivided into two sub-rewards: one for attention and the other for activity.

Table 2. Reward Variables.

Variables	Meaning	Margin
DFT	Duration of focus during a task	0: Less than 10 min 1: Between 10 and 15 min 2: Between 15 and 20 min 3: Between 20 and 30 min
WSpe	Writing speed	0: Normal speed 1: High speed 2: Low speed
LS	Letter spacing	0: Normal (1–2 mm) 1: Large 2: Narrow 3: Irregularly spaced
WSpa	Word spacing	0: Normal 1: Large 2: Narrow 3: Irregularly spaced
Pro	Proteins	0: Between 0 and 20 1: Between 21 and 40 2: Between 41 and 60
Kcal	Kilocalories	0: Between 0 and 200 1: Between 201 and 400 2: Between 401 and 600
WC	Walking cadence	0: Very slow 1: Slow 2: Normal 3: Fast 4: Very fast

below and the Equation (4) [85] detail the variables of these two sub-rewards.

$$\{\begin{array}{c}R(Att)=DFT+({\displaystyle \frac{WSpe\times LS}{WSpa+1}})\\ R(Act)={\displaystyle \frac{Pro-Kcal/100}{WC+1}}\end{array}$$

(4)

4.7. Reward Aggregation Function

$$G = \alpha \sum R_{Attention}+(1-\alpha )\sum R_{Activity}$$

(5)

An aggregation function is a function that aggregates all rewards collected from the initial state to the final state. In our case, when our ADHD agents move from a non-final state to a final state, their rewards are accumulated.

4.8. Algorithms

There are numerous MARL algorithms, some of which are listed in

Table 3. Benchmarking of a few Multi-Agent Reinforcement Learning (MARL) Algorithms.

Algorithm	Value-Based or Policy-Based	Credit Assignment
MAPPO [50]	Policy-Based	✕
MADDPG [50]	Policy-Based	✕
IDQN [86]	Value-Based	✕
CommNet [87]	Policy-Based	✕
BicNet [87]	Policy-Based/Value-Based	✕
DIAL [50]	Value-Based	✕
COMA [50]	Policy-Based	✓
VDN [50]	Value-Based	✓
QMIX [50]	Value-Based	✓

below. In our modeling, we aim to guide our agents (children) as effectively as possible to prevent confusion during decision-making. We also aim to distribute rewards fairly among the agents. These are the reasons why we chose the IDQN, VDN, and QMIX algorithms.

IDQN: Independent Deep Q Network is a MARL algorithm in which each agent learns its own Q-function independently and treats other agents as part of the environment [86]. The pseudocode of Algorithm 1 is described below.

Algorithm 1: Independent Deep Q Network (IDQN) [42]

Inputs: values networks with parameters, target networks with parameters, replay buffers $D_i$, time steps T, agent number n, observation o, action a, reward r, next observation $o_{i+1}$, transitions B, state s, loss L.

Outputs: updated target network parameters ${\theta }_i$ and ${\overline{\theta }}_i$, global reward y.

Initialize n value networks with random parameters ${\theta }_1,{\theta }_2,\dots {\theta }_n$;

Initialize n target networks with parameters ${\overline{\theta }}_1={\theta }_1,\dots ,{\overline{\theta }}_n={\theta }_n$;

Initialize a replay buffer for each agent $D_1,D_2,\dots ,D_n$.

For time step t = 0, 1, 2,…T do

Collect current observations $o_t^1,o_t^2,\dots ,o_t^n$.

For agent i = 1 to n do

Choose an action (ex, $ϵ$-greedy).

End For

Apply actions $(a_1^t,a_2^t,\dots ,a_n^t)$, get rewards $r_1^t,r_2^t,\dots r_n^t$ and next observations $o_1^{t+1},o_2^{t+2},\dots .,o_n^{t+n}$.

For agent i = 1 to n do

Store transition $(h_i^t,a_i^t,r_i^t,h_i^{t+1})$ in replay buffers $D_i$;

Sample random mini batch of B transitions $(h_i^k,a_i^k,r_i^k,h_i^{k+1})$ from $D_i$.

If $s^{k+1}$ is terminal, then

$y_i^k\leftarrow r_i^k$

else

$y_i^k\leftarrow r_i^k+\gamma {max}_{{a^{\prime }}_i\in A_i}Q(h_i^{k+1},{a^{\prime }}_i;{\overline{\theta }}_i).$

End If

$L({\theta }_i)\leftarrow {\displaystyle \frac{1}{B}}{\sum }_{i=1}^B(y_i^k-{Q(h_i^k,a_i^k;{\theta }_i))}^2$

Update parameters ${\theta }_i$ by minimizing the loss $L({\theta }_i)$;

Update target network parameters ${\overline{\theta }}_i$.

End For

End For

VDN: Value Decomposition Network (VDN) is a MARL algorithm that, unlike IQL, is decentralized only in execution. Here, the joint value function is decomposed into multiple value functions depending on the number of agents. Its equation and Figure 3 are below [42,86,88].

$$Q_{tot}(h,a)=\sum _{i=1}^n{Q}_i(h_i,a_i;{\theta }_i)$$

(6)

The pseudocode of Algorithm 2 is shown below.

Algorithm 2: Value Decomposition Network (VDN) [42]

Inputs: values networks with parameters, target networks with parameters, shared replay buffers D, time steps T, agent number n, observation o, action a, reward r, next observation $o_{i+1}$, transitions B, state s, loss L

Outputs: updated target network parameters ${\theta }_i$ and ${\overline{\theta }}_i$, global reward y.

Initialize n value networks with random parameters ${\theta }_1,{\theta }_2,\dots {\theta }_n$;

Initialize n target networks with parameters ${\overline{\theta }}_1={\theta }_1,\dots ,{\overline{\theta }}_n={\theta }_n$;

Initialize a shared replay buffer for all the agents D.

For time step t = 0, 1, 2,…T do

Collect current observations $o_t^1,o_t^2,\dots ,o_t^n$.

For agent i = 1 to n do

Choose an action (ex: $ϵ$-greedy).

End for

Apply actions $(a_1^t,a_2^t,\dots ,a_n^t)$, get the shared reward $r^t$ and next observations $o_1^{t+1},o_2^{t+2},\dots .,o_n^{t+n}$.

For agent i = 1 to n do

Store transition $(h^t,a^t,r^t,h^{t+1})$ in the shared replay buffer D;

Sample random mini batch of B transitions $(h^k,a^k,r^t,h^{k+1})$ from D.

If $s^{k+1}$ is terminal, then

$y^k\leftarrow r^k$

else

$y^k\leftarrow r^k+\gamma \sum _{i\in I}{max}_{{a^{\prime }}_i\in A_i}Q(h_i^{k+1},{a^{\prime }}_i;{\overline{\theta }}_i)$

End If

$L(\theta )\leftarrow {\displaystyle \frac{1}{B}}\sum _{i=1}^B(y_i^k-{\displaystyle \sum _{i\in I}}{Q(h_i^k,a_i^k;{\theta }_i))}^2$

Update parameters ${\theta }_i$ by minimizing the loss $L(\theta )$;

Update target network parameters ${\overline{\theta }}_i$ for each agent i.

End For

End For

QMIX is a MARL algorithm that, like VDN, uses a shared reward and only its execution is decentralized. One of its improvements is that it does not need to decompose everything but only to verify that $Q_{tot}$ and $Q_a$ have the maximum value simultaneously. Furthermore, the value function $Q_{tot}$ not only meets these criteria but also ensures that it is not fixed and can become complex. It is enough to verify the monotonicity relation between $Q_{tot}$ and $Q_a$ (7). Like VDN, QMIX represents each agent’s utility function via a deep neural network Q (DQN). To better capture the monotonicity of the action-value function across individual value functions, QMIX uses a neural network function $f_{mix}$ and a feedforward neural network that combines the individual value functions to approximate the centralized action-value function (8). Equations (7) and (8) and Figure 4 are written below [42,89].

$${\displaystyle \frac{\mathsf{\partial }{Q}_{tot}}{\mathsf{\partial }{Q}_a}}\ge 0,\forall a\in A$$

(7)

$$Q_{tot}(h,a)=f_{mix}(Q(h_1,a_1,{\theta }_1),Q(h_2,a_2,{\theta }_2),\dots \dots ,Q(h_n,a_n,{\theta }_{,});{\theta }_{mix})$$

(8)

The pseudocode of Algorithm 3 is as follows.

Algorithm 3: QMIX [42]

Inputs: values networks with parameters, target networks with parameters, shared replay buffers D, time steps T, agent number n, observation o, action a, reward r, next observation $o_{i+1}$, transitions B, state s, loss L.

Outputs: updated target network parameters ${\theta }_i$ and ${\overline{\theta }}_i$, global reward y.

Initialize n value networks with random parameters ${\theta }_1,{\theta }_2,\dots {\theta }_n$;

Initialize n target networks with parameters ${\overline{\theta }}_1={\theta }_1,\dots ,{\overline{\theta }}_n={\theta }_n$;

Initialize a shared replay buffer for all the agents D; Initialize hypernetwork with random parameters ${\theta }_{hyper}$.

For time step t = 0, 1, 2,…T do

Collect current observations $o_t^1,o_t^2,\dots ,o_t^n$.

For agent i = 1 to n do

Choose an action (ex: $ϵ$-greedy).

End for

Apply actions $(a_1^t,a_2^t,\dots ,a_n^t)$, get the shared reward $r^t$, next observations $o_1^{t+1},o_2^{t+2},\dots .,o_n^{t+n}$ and next centralized information $z^{t+1}$.

For agent i = 1 to n do

Store transition $(h^t, z^t,a^t,r^t,h^{t+1},z^{t+1})$ in the shared replay buffer D;

Sample random mini batch of B transitions $(h^k, z^k,a^k,r^k,h^{k+1},z^{k+1})$ from D.

If $s^{k+1}$ is terminal, then

$y^k\leftarrow r^k$

else

Mixing parameters ${\theta }_{mix}^{k+1}\leftarrow f_{hyper}(z^{k+1};{\theta }_{hyper})$

$y^k\leftarrow r^k\gamma f_{mix}\left(\begin{array}{c}{max}_{{a^{\prime }}_1}Q(h_1^{k+1},{a^{\prime }}_1;{\overline{\theta }}_1),\\ \begin{array}{c}\dots \dots ;{\theta }_{mix}^{k+1}\\ {max}_{{a^{\prime }}_n}Q(h_n^{k+1},{a^{\prime }}_n;{\overline{\theta }}_n),\end{array}\end{array}\right).$

End If           Mixing parameters ${\theta }_{mix}^k\leftarrow f_{hyper}(z^k;{\theta }_{hyper})$;           Value estimates $Q_{tot}(h^k,z^k,a^k;\theta )=f_{mix}(Q(h_1^k,a_1^k;{\theta }_1),Q(h_2^k,a_2^k;{\theta }_2),\dots \dots ,Q(h_n^k,a_n^k;{\theta }_n);{\theta }_{mix}^k)$.

$L(\theta )\leftarrow {\displaystyle \frac{1}{B}}\sum _{i=1}^B(y_i^k-{Q(h^k,z^k,a^k;\theta ))}^2$

Update parameters ${\theta }_i$ by minimizing the loss $L(\theta )$;

Update target network parameters ${\overline{\theta }}_i$ for each agent i.

End For

End For

5. Discussion and Results

ADHD is a neurodevelopmental disorder that affects a patient’s behavior from an early age and can worsen their condition if the patient is not diagnosed and treated as early as possible. Fortunately, many AI techniques have helped doctors and caregivers better treat this disease and support patients with it. One of these methods is Reinforcement Learning (RL). With this technique, it is possible to model a patient’s behavior and thus better analyze their symptoms. However, in life, a child never learns alone. They need contact and communication with their environment, including friends, parents, teachers, doctors, or children who can understand them or are in the same situation. The same applies to the RL mechanism: a single agent cannot learn in isolation; without cooperation, coordination, or communication, it would face significant difficulties adapting to a constantly changing environment. This is one of the reasons why it would be beneficial to have multiple ADHD agents to generalize their behavior better and thus provide better treatments.

The innovation of this article lies in formulating a Multi-Agent Reinforcement Learning problem and applying practical algorithms in the context of Attention-Deficit Hyperactivity Disorder.

In our modeling, we began by developing a questionnaire on the daily habits of 106 individuals, some with ADHD and some without, and distributed it to the TDAH Maroc Enfants association and local schools. We noticed that, among these individuals, 44 were children and 46 were adolescents. Since our paper mentions only children, we focused on them. We stored their responses in a dataset and, to further enrich it with attributes not directly captured in the questionnaire, we incorporated two additional datasets: one provides kilocalorie and protein information for each food item and the other contains features related to dysgraphia [90,91]. After integrating the three datasets, we observed that the first dataset has more rows than the other two. To address this issue, we trained a Generative Adversarial Network (GAN) on the second and third datasets. For data preprocessing, we employed the Synthetic Minority Over-sampling Technique (SMOTE) to balance the ADHD and non-ADHD classes. Then, with a seed of 42 to ensure fully deterministic, reproducible learning trajectories and to facilitate controlled analysis of agents, we split the dataset into a training set of 80% and a test set of 20%. We then used the XGBoost algorithm for a prediction test of the states, next states, and reward. As for the MARL algorithm hyperparameters, we chose the number of states as the input dimension, the number of actions as the output dim, a hidden dimension of either 64 or 128, a gamma value of 0.99, a start epsilon of 1, a decay epsilon of 0.995, an ending epsilon of 0.05, an episode number of 300, a maximum number of 50 steps, and a Boolean variable called done. The loop ends when the 50 steps of all the episodes are completed. Then, we represented an office environment as our setting through four vectors: one for sweet foods, one for healthy foods, one for physical activity, and one for writing. Then we identified specific actions that increase or decrease the symptoms of this disorder. When the agents perform these actions, they modify the fields of these vectors. The states are divided into four categories in scalar format: 0 for ADHD-C, 1 for ADHD-H, 2 for ADHD-I, and 3 for non-ADHD.

Regarding rewards, we divided them into two sub-rewards: one for activity and the other for attention. We then selected three value-based MARL algorithms: IQL, VDN, and QMIX. However, we limited to 44 agents as increasing the agent number would impact train stability. Although DIAL is a value-based MARL technique, it was excluded due to computational and memory constraints, as it supports multiple communication channels, which increase resource consumption. The results of these three algorithms showed that children with ADHD aim for fewer maximum rewards than children without ADHD.

Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9 show that ADHD children consume pastries, soft drinks, and other sugary foods, resulting in an unbalanced diet. Additionally, ADHD children are more dysgraphic, may have an unbalanced speed of writing, and tend to be more aggressive than neurotypical children. Because the VDN and QMIX algorithms both employ centralized learning, they mitigate non-stationarity, thereby improving agent stability. The figures below show the rest of the results.

Figure 5 shows the percentages of sweet foods consumed by children with ADHD and children without ADHD. For example, 2.3% of inattentive children with ADHD eat ice cream, or 4.6% of children with ADHD and two predominant symptoms eat biscuits.

Figure 6, on the other hand, shows the percentages of consumption of certain healthy foods by children with ADHD. For example, 4.5% of children with ADHD who have both symptoms dominant eat fish, and 1.4% of inattentive children with ADHD consume dried fruits.

Figure 7 highlights the percentages of dysgraphia among children with ADHD compared to those without ADHD. We can infer from this pie chart that 7.8% of children with ADHD have both symptoms, as well as dysgraphia.

Figure 8 demonstrates the percentage of children with or without ADHD by walking speed. For example, 13.3% of children with ADHD and the two prominent symptoms walk and/or run quickly, whereas only 6.7% of hyperactive children with ADHD walk and/or run more slowly.

Figure 9 shows the categories of aggressiveness according to the type of ADHD and non-ADHD children. We can conclude that 7.8% of ADHD children with both symptoms are very aggressive, while 4.4% of inattentive ADHD children are not.

Figure 10 shows the average rewards for children with ADHD and non-ADHD using the IDQN, where those of non-ADHD children surpass those of children with ADHD (the peak for non-ADHD is 395 at episode 300, compared to 340 at episode 120 for ADHD). However, the results also show some instability. The reason for this is that when children act alone, especially children with ADHD, they are not necessarily oriented towards the maximum rewards.

Figure 11 displays the average rewards for children with and without ADHD using VDN, showing that non-ADHD children have higher average rewards than those with ADHD (the ADHD peak is 69 starting from episode 90, compared to 82 for non-ADHD). Additionally, the results show some stability. One reason is that when children coordinate, synchronize, and share a reward, their performance is more stable than when they act independently.

Figure 12 presents the average rewards of children with ADHD and without ADHD using the QMIX algorithm, where, similar to the previous figure, the average rewards of children without ADHD exceed those of children with ADHD (the peak for non-ADHD is 31 at episode 110, and for ADHD it is 28 at episode 110), except that the results demonstrate some instability. In addition to this, ADHD agents tend to be more explorative, and so their behavior is more stochastic. The use of the QMIX algorithm, which employs a nonlinear, monotonic function, may explain the variance and behavior.

6. Conclusions and Future Work

In conclusion, this paper listed some of the difficulties faced by children with ADHD, outlined guided work using Artificial Intelligence (AI) techniques to help these children, introduced the concept of Reinforcement Learning (RL) for an agent and subsequently for multiple agents (MARL). We then proposed a methodology that used three algorithms from this discipline, with our agents being children with or without ADHD. We can infer that using Multi-Agent Reinforcement Learning (MARL) algorithms in a cooperative framework can improve agents’ performance, particularly for agents with ADHD. Moreover, this demonstrates that ADHD agents can understand one another and be better treated when grouped. Other Artificial Intelligence tools may be used in the future to better support children with this neurodevelopmental disorder, such as large language models (LLMs).