Mathematical Perspectives of a Coupled System of Nonlinear Hybrid Stochastic Fractional Differential Equations

Abstract

This research develops a novel coupled system of nonlinear hybrid stochastic fractional differential equations that integrates neutral effects, stochastic perturbations, and hybrid switching mechanisms. The system is formulated using the Atangana–Baleanu–Caputo fractional operator with a non-singular Mittag–Leffler kernel, which enables accurate representation of memory effects without singularities. Unlike existing approaches, which are limited to either neutral or hybrid stochastic structures, the proposed framework unifies both features within a fractional setting, capturing the joint influence of randomness, history, and abrupt transitions in real-world processes. We establish the existence and uniqueness of mild solutions via the Picard approximation method under generalized Carathéodory-type conditions, allowing for non-Lipschitz nonlinearities. In addition, mean-square Mittag–Leffler stability is analyzed to characterize the boundedness and decay properties of solutions under stochastic fluctuations. Several illustrative examples are provided to validate the theoretical findings and demonstrate their applicability.

1. Introduction

This study focuses on the qualitative analysis of a coupled system of fractional hybrid neutral stochastic differential equations. In this section, we presented some fundamentals of fractional calculus (FC), coupled systems, hybrid differential equations, and literature concerning these significant aspects of FC.

1.1. Fractional Calculus

In the modern era, FC plays a vital role due to its potential applications in epidemic modeling, viscoelastic material control, population growth, and smart engineering systems. FC is the study of non-integer order derivatives and integrals, evolving from purely theoretical aspects into a powerful tool for modeling real-world phenomena characterized by memory, nonlocality, and hereditary effects [1,2,3,4]. Classical differential equations (DEs) often fall short when describing materials or processes where the present state depends not only on the current conditions but also on historical behavior. These significant aspects of FC have led to the development of various fractional derivatives (FDs), including the Riemann–Liouville, Hosmanard, Grünwald–Letnikov, Caputo, and, more recently, the Atangana–Baleanu (AB) derivatives [5]. Among the different types of FDs, each has its own advantages and limitations under certain circumstances. One of the most important classes is the Atangana–Baleanu–Caputo (ABC) derivative, defined with a non-singular Mittag-Leffler kernel, which has received significant attention due to its ability to preserve memory without introducing singularities [6]. The advantages of the ABC derivative allow for better investigation of stability analysis and physical interpretation, especially in modeling viscoelastic materials, anomalous diffusion, biological systems, and control processes [7].

An important aspect of FC is the study of fractional differential equations (FDEs) from different points of view, such as theoretical insight and numerical approximations. Researchers employ FDEs for accurate and reliable modeling of dynamical systems involving sudden fluctuations, long-term memory, and non-local effects. One notable class of these DEs is stochastic differential equations (SDEs). SDEs introduce random perturbations into dynamical systems, often represented through Wiener processes or white noise [8]. They are essential for understanding systems affected by uncertainty, noise, or incomplete information, such as financial models (e.g., the Black–Scholes equation), control systems, and gene expression under stochastic regulation in noisy environments. SDEs have become a fundamental framework in fields such as statistical physics, quantitative finance, epidemiology, and machine learning [9,10,11,12]. Moreover, SDEs allow analysis not only of mean trajectories but also of variances and probabilistic bounds.

1.2. Coupled System and Hybrid Differential Equations

A system of multi-differential equations containing interdependent variables and their derivatives, where the evolution of each dependent variable depends on the others, is called a coupled system of differential equations (CSDE). In certain circumstances, real-world problems arise in the form of CSDEs. Such systems describe multiple interacting subsystems that evolve together through interdependence. They can capture a wide range of phenomena, including cooperative dynamics, synchronized behaviors, and feedback regulation. Common examples of CSDEs include multi-agent robotics, reaction–diffusion systems in chemistry, electrical oscillator networks, and interconnected ecological models such as predator–prey dynamics [13,14]. This type of coupling structure allows for rich dynamical behavior, such as phase locking or multi-stability, that cannot be modeled by isolated systems. These systems play a crucial role in both the modeling and control of distributed systems. Recently, a class of CSDEs, i.e., coupled systems of hybrid neutral stochastic fractional differential equations (HNSFDEs), has gained more attention from researchers around the globe. HNSFDEs represent systems that exhibit both continuous evolution and discrete transitions [15]. Neutral differential equations (NDEs) extend the concepts of delay differential equations (DDEs) by incorporating dependence on past derivatives, making them particularly useful for modeling systems with memory-dependent dynamics, such as circuit networks or viscoelastic materials with transmission delays [16]. These models are essential for accurately describing processes where abrupt changes, switching dynamics, or threshold-based decisions coexist with smooth temporal evolution. Hybrid systems naturally arise in power electronics (e.g., switching converters), robotics (e.g., legged locomotion), biochemical networks (e.g., gene regulation), and cyber–physical systems where digital logic interacts with analog physical processes [17,18]. By blending discrete and continuous behaviors, hybrid models offer a more flexible and realistic description of systems with regime-switching dynamics and logic-driven feedback.

Hybrid differential equations combined with neutral stochastic differential equations (NSDEs) model complex systems where continuous random fluctuations interact with sudden discrete transitions [19]. When hybrid behavior and stochastic effects are embedded within a fractional framework, we obtain a powerful unified tool for modeling complex phenomena in neural networks, fractional order sliding mode control multiagent system with time-varying delays [20], epidemic propagation [21], and industrial automation [22], finite time stability results were deduced for fractional-order hydraulic turbine regulating system [23], regime switching [24], and Mittag-Leffler stability for Hopfield neural networks [25]. These systems appear in financial markets experiencing both Brownian-motion-driven price changes and abrupt regime shifts and in biological systems where stochastic chemical reactions coexist with discrete cellular events [26]. This mathematical framework typically involves an SDE governing the continuous state dynamics, coupled with a Markov chain or deterministic switching mechanism that triggers instantaneous jumps in system parameters [27]. Such models require specialized analytical techniques, including extended versions of Itô’s calculus and piecewise deterministic Markov process theory. The applications of coupled systems of HNSDEs range from robotics to power systems, where purely continuous or purely discrete models prove inadequate. The study draws on tools from stochastic analysis, dynamical systems theory, and control engineering to address these hybrid stochastic dynamics [28]. Likewise, HNSDEs are effective tools for modeling smart energy grids, where the neutral nature is used for voltage regulation, stochasticity arises from renewable energy output, and switching between power sources ensures stability despite unsteady supply and demand [29]. HNSDEs also play a vital role in biomedical systems, optimizing drug delivery by addressing delayed metabolic reactions, adaptive dosage adjustments, and random patient responses, leading to more effective and precise treatments [30]. To handle such significant and complex problems, specialized techniques are required, whether analyzing almost-sure stability in stochastic cases or developing adaptive solvers for their deterministic counterparts [31].

1.3. Related Literature to the Study

The existence and uniqueness (EU) of solutions to SDEs driven by Brownian motion are widely examined in the literature [32]. Additionally, significant attention has been given to extending the qualitative theory of SDEs to infinite-dimensional settings [33,34]. Among these developments, Taniguchi [35] demonstrates EU results for SDEs under generalized non-Lipschitz conditions on the drift and diffusion terms, which encompass earlier findings of Yamada as a special case. Further contributions by Yamada [36], Rodkina [37], and Taniguchi [35] provided analogous EU theorems for ordinary SDEs with non-Lipschitz coefficients. In addition, Vinayagam and Balasubramaniam [38] explored the existence of mild solutions (MSs) for a specific category of non-linear neutral SDEs formulated in Hilbert spaces. Meanwhile, Jiang and Shen [39] advanced the theory by analyzing EU for neutral stochastic partial functional DEs under certain Carathéodory-type coefficient conditions. Their approach relied on successive approximation methods and led to generalizations of earlier work, including results from [40]. Arzu and Nazim [41] extended the EU of MSs to general fractional neutral SDEs under Carathéodory-type conditions (CTCs) on the coefficients using the well-known Picard iterative method (PIM).

1.4. New Findings

After reviewing the existing literature on the existence theory of FSDEs and considering the aforementioned research, this study uses the PIM to generalize the results concerning EU of mild solutions and mean square stability analysis for the proposed novel coupled system of HNSFDEs (1) under CTC on the coefficients. The proposed generalized coupled system of HNSFDEs with ($\nu \in ({\displaystyle \frac{1}{2}},1)$) is given by

$$\left\{\begin{array}{c}{}^{\mathbf{ABC}}\mathbb{D}_{0+}^{\nu }(\mathrm{\Psi }\left(t\right)+g_1(t,\mathrm{\Psi }\left(t\right)))={\mathbf{A}}_1\mathrm{\Psi }\left(t\right)+{\mathsf{F}}_1(t,\mathrm{{\rm Y}}\left(t\right))+{\sigma }_1(t,\mathrm{\Psi }\left(t\right)){\displaystyle \frac{d\mathbb{W}\left(t\right)}{dt}},\hfill \\ {}^{\mathbf{ABC}}\mathbb{D}_{0+}^{\nu }(\mathrm{{\rm Y}}\left(t\right)+g_2(t,\mathrm{{\rm Y}}\left(t\right)))={\mathbf{A}}_2\mathrm{{\rm Y}}\left(t\right)+{\mathsf{F}}_2(t,\mathrm{\Psi }\left(t\right))+{\sigma }_2(t,\mathrm{{\rm Y}}\left(t\right)){\displaystyle \frac{d\mathbb{W}\left(t\right)}{dt}},\hfill \\ \mathrm{\Psi }\left(0\right)={\alpha }_1,\mathrm{{\rm Y}}\left(0\right)={\alpha }_2,\hfill \end{array}\right.$$

(1)

where ${}^{\mathbf{ABC}}\mathbb{D}_{0+}^{\nu }$ is the ABC fractional derivative, ${\left(\mathbb{W}\left(t\right)\right)}_{t\ge 0}$ is a standard Brownian motion on a complete probability space $(\mathrm{\Omega },{\mathbb{F}}_{\mathbb{T}},P)$ for $T\ge 0$, with some filtration ${\mathcal{F}}_T:={\left\{{\mathcal{F}}_t\right\}}_{t\in [0,t]}$ satisfying the usual conditions (i.e., ${\mathcal{F}}_0$ consists of all $\mathbb{P}$-null sets, while it is right-continuous and increasing), where $g_1, g_2:[0,t]\times {\mathbb{R}}^d\to {\mathbb{R}}^d$ are the nonlinear functions defining the neutral terms. The functions ${\mathsf{F}}_1, {\mathsf{F}}_2$ represent the nonlinear drift terms, and ${\sigma }_1, {\sigma }_2$ are the diffusion coefficients governing the stochastic perturbations. The initial conditions ${\alpha }_1, {\alpha }_2$ are ${\mathcal{F}}_0$-measurable H-valued random variables, and ${\mathbf{A}}_1,{\mathbf{A}}_2\in {\mathbb{R}}^{d\times d}$.

Previous studies on neutral FSDEs have mainly focused on simple forms using classical derivatives, and primarily from the existence perspective. This work addresses the following novel aspects, which were not yet been analyzed to the best of our knowledge, particularly in terms of stability:

• We introduce the idea of coupling neutral DEs with hybrid stochasticity, thereby advancing research in new directions by formulating a more realistic and mathematically enriched model.
• By incorporating hybrid differential structures, our model accounts for both continuous dynamics and discrete regime-switching behavior.
• Instead of relying on traditional fractional derivatives, which involve singular kernels and associated limitations, we employ the ABC derivative with a non-singular Mittag-Leffler kernel.
• We establish the mean-square Mittag-Leffler stability of the proposed system, ensuring that solutions remain bounded and decay predictably under stochastic fluctuations.
• To the best of our knowledge, mean-square stability analysis for both simple and coupled systems of FSDEs has not yet been investigated.

We address the theoretical gap by formulating a nonlinear coupled system of hybrid stochastic differential equations using the ABC derivative, establishing existence and uniqueness under CTC via the Picard approximation and proving mean-square Mittag-Leffler stability for reliable applications in engineering, neuroscience, epidemiology, and control systems. The introduction of these aspects into existing models enhances theoretical and mathematical understanding. In particular, by extending a single SDE to a coupled system of FSDEs with hybrid concepts, which allow for mutual interactions between subsystems, an essential feature in modeling real-world phenomena such as interconnected biological systems and cyber–physical infrastructures. The stability property reinforces the robustness and applicability of our framework, making it a powerful generalization of existing models and highly suitable for simulating complex, real-world dynamical systems, which have not yet been studied. The investigation of the proposed coupled system of HNSDEs also provides new directions for researchers to establish further results regarding the existence of mild solutions and stability analysis.

2. Preliminaries

We present essential structures of fractional calculus and necessary facts about fractional stochastic neutral differential systems.

Definition 1 

([5]). For a function $f\in AC\left(\right[a,b\left]\right)$ and $0<\nu <1$, the ABC fractional derivative is defined as

$${}^{\mathbf{ABC}}D_a^{\nu }f\left(t\right)={\displaystyle \frac{\mathcal{C}\left(\nu \right)}{1-\nu }}{\mathit{\int }}_a^t{f}^{\prime }\left(\delta \right){\mathbf{E}}_{\nu }\left[{\displaystyle \frac{\nu {(t-\delta )}^{\nu }}{\nu -1}}\right]d\delta .$$

(2)

Definition 2

([42]). The associated ABC fractional integral is given by

$${}^{\mathbf{ABC}}I_a^{\nu }f\left(t\right)={\displaystyle \frac{1-\nu }{\mathcal{C}\left(\nu \right)}}f\left(t\right)+{\displaystyle \frac{\nu }{\mathcal{C}\left(\nu \right)\Gamma \left(\nu \right)}}{\int }_a^t{(t-\delta )}^{\nu -1}f\left(\delta \right)d\delta .$$

(3)

where $\Gamma (·)$ is the Gamma function:

$$\Gamma \left(\nu \right):={\int }_0^{\infty }{\delta }^{\nu -1}{e}^{-\delta }d\delta ,$$

(4)

and $\mathcal{C}\left(\nu \right)$ is a normalization function with $\mathcal{C}\left(0\right)=\mathcal{C}\left(1\right)=1$, and ${\mathbf{E}}_{\nu }(·)$ is the Mittag-Leffler function (MLF).

Definition 3 

([43]). MLF, in two parameters,

$${\mathbf{E}}_{\nu ,\beta }\left(z\right)=\sum _{k=0}^{\infty }{\displaystyle \frac{z^k}{\Gamma (ku+\beta )}},\nu ,\beta >0,$$

(5)

with its matrix version,

$${\mathbf{E}}_{\nu ,\beta }\left(t^{\nu }A\right)=\sum _{k=0}^{\infty }{A}^k{\displaystyle \frac{t^{ku}}{\Gamma (ku+\beta )}},A\in {\mathbb{R}}^{n\times n},$$

(6)

satisfying:

$${}^{C}D_{0+}^{\nu }{\mathbf{E}}_{\nu }\left(t^{\nu }A\right)=A{\mathbf{E}}_{\nu }\left(t^{\nu }A\right).$$

(7)

Definition 4 

([44]). Let $\left(\mathrm{\Psi }\right(t),\mathrm{{\rm Y}}(t\left)\right)$ be a mild solution of the coupled system on $[0,t]$. The system is said to be mean-square Mittag-Leffler-stable, if there exist constants $C>0$, $\lambda >0$, and $0<\nu <1$ such that

$${\mathbb{E}\parallel \mathrm{{\rm Y}}\left(t\right)\parallel }^2+\mathbb{E}{\parallel \mathrm{\Psi }\left(t\right)\parallel }^2\le C{\mathbf{E}}_{\nu }(-\lambda t^{\nu })(\mathbb{E}\parallel {\alpha }_1{\parallel }^2+\mathbb{E}{\parallel {\alpha }_2\parallel }^2).$$

Let ${\mathbb{R}}^{\mathfrak{d}}$ be endowed with the standard Euclidean norm, and let $H^2([0,t],{\mathbb{R}}^{\mathfrak{d}})$ denote the space of all $F_T$-measurable processes $\xi$ satisfying:

$$H^2([0,t],{\mathbb{R}}^{\mathfrak{d}}):=\left\{{\xi :\parallel \xi \parallel }_{H^2}^2=\underset{t\in [0,t]}{sup}\mathbb{E}{\parallel \xi \left(t\right)\parallel }^2<\infty \right\}.$$

(8)

Lemma 1. 

Assume that $\mathrm{\Psi }\left(\delta \right)$ is an H-valued process which satisfies $\mathbb{E}\left[{\int }_0^T{\parallel \mathrm{\Psi }\left(\delta \right)\parallel }^{2}d\delta \right]<\infty$. Then, for every $0<\mathbf{T}$ and $2\nu >1$, there exists a constant $C_T={\displaystyle \frac{T^{2\nu -1}}{2\nu -1}}$ such that

$$\underset{0\le t\le T}{sup}\mathbb{E}{∥{\int }_0^t{(t-\delta )}^{\nu -1}\mathrm{\Psi }\left(\delta \right)dW\left(\delta \right)∥}^2\le C_T\mathbb{E}\left[{\int }_0^T{\parallel \mathrm{\Psi }\left(\delta \right)\parallel }^{2}d\delta \right].$$

(9)

Apparently, (8) is a Banach space, and $g_i, {\mathsf{F}}_i, {\sigma }_i:[0,t]\times {\mathbb{R}}^{\mathfrak{d}}⟶{\mathbb{R}}^{\mathfrak{d}}$ are bounded and measurable functions that satisfy the following assumptions:

Assumption 1.

$\left({\mathbf{A}}_{\mathbf{1}}\right):$

The functions $g_i$ satisfy $\mathbb{E}\parallel g_i(t,\mathrm{\Psi })-g_i{(t,\mathrm{Y})\parallel }^2\le {\mathfrak{L}}_i^2\mathbb{E}{\parallel \mathrm{\Psi }-\mathrm{Y}\parallel }^2$ with $0<{\mathfrak{L}}_i<{\displaystyle \frac{1}{2}}$.

$\left({\mathbf{A}}_{\mathbf{2}}\right):$

The drift and diffusion terms satisfy

$$\begin{array}{cc}\hfill \parallel {\mathsf{F}}_1(t,{\mathrm{\Psi }}_1,{\mathrm{{\rm Y}}}_1)-{\mathsf{F}}_1(t,{\mathrm{\Psi }}_2,{\mathrm{{\rm Y}}}_2){\parallel }^2& \le {\mathfrak{L}}_{{\mathsf{F}}_1}(\parallel {\mathrm{\Psi }}_1-{\mathrm{\Psi }}_2{\parallel }^2+ \parallel {\mathrm{{\rm Y}}}_1-{\mathrm{{\rm Y}}}_2{\parallel }^2),\hfill \\ \hfill \parallel {\sigma }_1(t,\mathrm{\Psi })-{\sigma }_1{(t,\mathrm{{\rm Y}})\parallel }^2& \le {\mathfrak{L}}_{\sigma 1}{\parallel \mathrm{\Psi }-\mathrm{{\rm Y}}\parallel }^2,\hfill \\ \hfill \parallel {\mathsf{F}}_2(t,{\mathrm{\Psi }}_1,{\mathrm{{\rm Y}}}_1)-{\mathsf{F}}_2(t,{\mathrm{\Psi }}_2,{\mathrm{{\rm Y}}}_2){\parallel }^2& \le {\mathfrak{L}}_{{\mathsf{F}}_2}(\parallel {\mathrm{\Psi }}_1-{\mathrm{\Psi }}_2{\parallel }^2+ \parallel {\mathrm{{\rm Y}}}_1-{\mathrm{{\rm Y}}}_2{\parallel }^2),\hfill \\ \hfill \parallel {\sigma }_2(t,{\mathrm{{\rm Y}}}_1)-{\sigma }_2(t,{\mathrm{{\rm Y}}}_2){\parallel }^2& \le {\mathfrak{L}}_{\sigma 2}{\parallel {\mathrm{{\rm Y}}}_1-{\mathrm{{\rm Y}}}_2\parallel }^2.\hfill \end{array}$$

$\left({\mathbf{A}}_{\mathbf{3}}\right):$

Linear growth conditions hold: $\parallel {\mathsf{F}}_i{(t,0,0)\parallel }^2+{\parallel {\sigma }_i(t,0)\parallel }^2\le K_i<\infty$.

3. Results of Existence

Lemma 2. 

The mild solution to the coupled system of NHSFDEs (1) with order $\nu \in (0.5,1)$, is given by

$$\begin{array}{cc}\hfill \mathrm{\Psi }\left(t\right)& ={\alpha }_1+g_1(0,{\alpha }_1)-g_1(t,\mathrm{\Psi }\left(t\right))\hfill \\ \hfill & +{\displaystyle \frac{1-\nu }{\mathcal{C}\left(\nu \right)}}\left[{\mathsf{F}}_1(t,\mathrm{{\rm Y}}\left(t\right))+{\mathbf{A}}_1\mathrm{\Psi }\left(t\right)\right]\hfill \\ \hfill & +{\displaystyle \frac{\nu }{\mathcal{C}\left(\nu \right)\Gamma \left(\nu \right)}}{\int }_0^t{(t-\delta )}^{\nu -1}\left[{\mathsf{F}}_1(\delta ,\mathrm{{\rm Y}}\left(\delta \right))+{\mathbf{A}}_1\mathrm{\Psi }\left(\delta \right)\right]d\delta \hfill \\ \hfill & +{\displaystyle \frac{1-\nu }{\mathcal{C}\left(\nu \right)}}{\sigma }_1(t,\mathrm{\Psi }\left(t\right))\dot{\mathbb{W}}\left(t\right)+{\displaystyle \frac{\nu }{\mathcal{C}\left(\nu \right)\Gamma \left(\nu \right)}}{\int }_0^t{(t-\delta )}^{\nu -1}{\sigma }_1(\delta ,\mathrm{\Psi }\left(\delta \right))d\mathbb{W}\left(\delta \right),\hfill \\ \end{array}$$

(10)

$$\begin{array}{cc}\hfill \mathrm{{\rm Y}}\left(t\right)& ={\alpha }_2+g_2(0,{\alpha }_2)-g_2(t,\mathrm{{\rm Y}}\left(t\right))\hfill \\ \hfill & +{\displaystyle \frac{1-\nu }{\mathcal{C}\left(\nu \right)}}\left[{\mathsf{F}}_2(t,\mathrm{\Psi }\left(t\right))+{\mathbf{A}}_2\mathrm{{\rm Y}}\left(t\right)\right]\hfill \\ \hfill & +{\displaystyle \frac{\nu }{\mathcal{C}\left(\nu \right)\Gamma \left(\nu \right)}}{\int }_0^t{(t-\delta )}^{\nu -1}\left[{\mathsf{F}}_2(\delta ,\mathrm{\Psi }\left(\delta \right))+{\mathbf{A}}_2\mathrm{{\rm Y}}\left(t\right)\right]d\delta \hfill \\ \hfill & +{\displaystyle \frac{1-\nu }{\mathcal{C}\left(\nu \right)}}{\sigma }_2(t,\mathrm{{\rm Y}}\left(t\right))\dot{\mathbb{W}}\left(t\right)+{\displaystyle \frac{\nu }{\mathcal{C}\left(\nu \right)\Gamma \left(\nu \right)}}{\int }_0^t{(t-\delta )}^{\nu -1}{\sigma }_2(\delta ,\mathrm{{\rm Y}}\left(\delta \right))d\mathbb{W}\left(\delta \right).\hfill \end{array}$$

(11)

Proof. 

We consider the coupled system of NHSFDEs (1) involving the ABC fractional derivative with $\nu \in ({\displaystyle \frac{1}{2}},1)$. Applying ${}^{\mathbf{ABC}}I_0^{\nu }$ on both sides of the system and using the identity

$${}^{\mathbf{ABC}}I_0^{\nu }\left({}^{\mathbf{ABC}}D_0^{\nu }f\left(t\right)\right)=f\left(t\right)-f\left(0\right),$$

we obtain

$$\begin{array}{cc}\hfill \mathrm{\Psi }\left(t\right)+g_1(t,\mathrm{\Psi }\left(t\right))& ={\alpha }_1+g_1(0,{\alpha }_1)+{}^{\mathbf{ABC}}I_0^{\nu }\left[{\mathbf{A}}_1\mathrm{\Psi }\left(t\right)+{\mathsf{F}}_1(t,\mathrm{{\rm Y}}\left(t\right))+{\sigma }_1(t,\mathrm{\Psi }\left(t\right)){\displaystyle \frac{d\mathbb{W}\left(t\right)}{dt}}\right],\hfill \\ \hfill \mathrm{{\rm Y}}\left(t\right)+g_2(t,\mathrm{{\rm Y}}\left(t\right))& ={\alpha }_2+g_2(0,{\alpha }_2)+{}^{\mathbf{ABC}}I_0^{\nu }\left[{\mathbf{A}}_2\mathrm{{\rm Y}}\left(t\right)+{\mathsf{F}}_2(t,\mathrm{\Psi }\left(t\right))+{\sigma }_2(t,\mathrm{{\rm Y}}\left(t\right)){\displaystyle \frac{d\mathbb{W}\left(t\right)}{dt}}\right].\hfill \end{array}$$

Subtracting the nonlinear terms $g_1(t,\mathrm{\Psi }\left(t\right))$ and $g_2(t,\mathrm{{\rm Y}}\left(t\right))$, we have

$$\begin{array}{cc}\hfill \mathrm{\Psi }\left(t\right)& ={\alpha }_1+g_1(0,{\alpha }_1)-g_1(t,\mathrm{\Psi }\left(t\right))\hfill \\ \hfill & +{}^{\mathbf{ABC}}I_0^{\nu }\left[{\mathbf{A}}_1\mathrm{\Psi }\left(t\right)+{\mathsf{F}}_1(t,\mathrm{{\rm Y}}\left(t\right))+{\sigma }_1(t,\mathrm{\Psi }\left(t\right)){\displaystyle \frac{d\mathbb{W}\left(t\right)}{dt}}\right],\hfill \\ \hfill \mathrm{{\rm Y}}\left(t\right)& ={\alpha }_2+g_2(0,{\alpha }_2)-g_2(t,\mathrm{{\rm Y}}\left(t\right))\hfill \\ \hfill & +{}^{\mathbf{ABC}}I_0^{\nu }\left[{\mathbf{A}}_2\mathrm{{\rm Y}}\left(t\right)+{\mathsf{F}}_2(t,\mathrm{\Psi }\left(t\right))+{\sigma }_2(t,\mathrm{{\rm Y}}\left(t\right)){\displaystyle \frac{d\mathbb{W}\left(t\right)}{dt}}\right].\hfill \end{array}$$

Applying the ABC Integral to each term,

• For ${}^{\mathbf{ABC}}I_{0+}^{\nu }\left[A_1\mathrm{\Psi }\left(t\right)\right]$

  $${}^{\mathbf{ABC}}I_{0+}^{\nu }\left[A_1\mathrm{\Psi }\left(t\right)\right]={\displaystyle \frac{1-\nu }{\mathcal{C}\left(\nu \right)}}{A}_1\mathrm{\Psi }\left(t\right)+{\displaystyle \frac{\nu }{\mathcal{C}\left(\nu \right)\Gamma \left(\nu \right)}}{\int }_0^t{(t-\delta )}^{\nu -1}{A}_1\mathrm{\Psi }\left(\delta \right)d\delta .$$
• For ${}^{\mathbf{ABC}}I_{0+}^{\nu }\left[{\mathsf{F}}_1(t,\mathrm{{\rm Y}}\left(t\right))\right]$:

  $${}^{\mathbf{ABC}}I_{0+}^{\nu }\left[{\mathsf{F}}_1(t,\mathrm{{\rm Y}}\left(t\right))\right]={\displaystyle \frac{\nu }{\mathcal{C}\left(\nu \right)\Gamma \left(\nu \right)}}{\int }_0^t{(t-\delta )}^{\nu -1}{\mathsf{F}}_1(\delta ,\mathrm{{\rm Y}}\left(\delta \right))d\delta +{\displaystyle \frac{1-\nu }{\mathcal{C}\left(\nu \right)}}{\mathsf{F}}_1(t,\mathrm{{\rm Y}}\left(t\right)).$$
• For the stochastic term,

  $${}^{\mathbf{ABC}}I_{0+}^{\nu }[{\sigma }_1(t,\mathrm{\Psi }\left(t\right)){\displaystyle \frac{dW\left(t\right)}{dt}}]\approx {\displaystyle \frac{\nu }{\mathcal{C}\left(\nu \right)\Gamma \left(\nu \right)}}{\int }_0^t{(t-\delta )}^{\nu -1}{\sigma }_1(\delta ,\mathrm{\Psi }\left(\delta \right))d\mathbb{W}\left(\delta \right).$$

  (12)

• We use the term ${\displaystyle \frac{\nu }{\mathcal{C}\left(\nu \right)\Gamma \left(\nu \right)}}{\int }_0^t{(t-s)}^{\nu -1}{\sigma }_1(\delta ,\mathrm{\Psi }\left(\delta \right))d\mathbb{W}\left(\delta \right)$ from (12) instead of ${}^{\mathbf{ABC}}I_{0+}^{\nu }[{\sigma }_1(t,\mathrm{\Psi }\left(t\right)){\displaystyle \frac{dW\left(t\right)}{dt}}]$ in the Itô calculus for simplicity.

• Putting these in the original equation, we get

$$\begin{array}{cc}\hfill \mathrm{\Psi }\left(t\right)& ={\alpha }_1+g_1(0,{\alpha }_1)-g_1(t,\mathrm{\Psi }\left(t\right))\hfill \\ \hfill & +{\displaystyle \frac{1-\nu }{\mathcal{C}\left(\nu \right)}}\left[{\mathsf{F}}_1(t,\mathrm{{\rm Y}}\left(t\right))+{\mathbf{A}}_1\mathrm{\Psi }\left(t\right)\right]\hfill \\ \hfill & +{\displaystyle \frac{\nu }{\mathcal{C}\left(\nu \right)\Gamma \left(\nu \right)}}{\int }_0^t{(t-\delta )}^{\nu -1}\left[{\mathsf{F}}_1(\delta ,\mathrm{{\rm Y}}\left(\delta \right))+{\mathbf{A}}_1\mathrm{\Psi }\left(t\right)\right]d\delta \hfill \\ \hfill & +{\displaystyle \frac{1-\nu }{\mathcal{C}\left(\nu \right)}}{\sigma }_1(t,\mathrm{\Psi }\left(t\right))\dot{\mathbb{W}}\left(t\right)+{\displaystyle \frac{\nu }{\mathcal{C}\left(\nu \right)\Gamma \left(\nu \right)}}{\int }_0^t{(t-\delta )}^{\nu -1}{\sigma }_1(\delta ,\mathrm{\Psi }\left(\delta \right))d\mathbb{W}\left(\delta \right).\hfill \end{array}$$

(13)

A similar procedure follows for the mild solution of $\mathrm{{\rm Y}}\left(t\right)$,

$$\begin{array}{cc}\hfill \mathrm{{\rm Y}}\left(t\right)& ={\alpha }_2+g_2(0,{\alpha }_2)-g_2(t,\mathrm{{\rm Y}}\left(t\right))\hfill \\ \hfill & +{\displaystyle \frac{1-\nu }{\mathcal{C}\left(\nu \right)}}\left[{\mathbf{A}}_2\mathrm{{\rm Y}}\left(t\right)+{\mathsf{F}}_2(t,\mathrm{\Psi }\left(t\right))\right]\hfill \\ \hfill & +{\displaystyle \frac{\nu }{\mathcal{C}\left(\nu \right)\Gamma \left(\nu \right)}}{\int }_0^t{(t-\delta )}^{\nu -1}\left[{\mathbf{A}}_2\mathrm{{\rm Y}}\left(\delta \right)+{\mathsf{F}}_2(\delta ,\mathrm{\Psi }\left(\delta \right))\right]d\delta \hfill \\ \hfill & +{\displaystyle \frac{1-\nu }{\mathcal{C}\left(\nu \right)}}{\sigma }_2(t,\mathrm{{\rm Y}}\left(t\right))\dot{\mathbb{W}}\left(t\right)+{\displaystyle \frac{\nu }{\mathcal{C}\left(\nu \right)\Gamma \left(\nu \right)}}{\int }_0^t{(t-\delta )}^{\nu -1}{\sigma }_2(\delta ,\mathrm{{\rm Y}}\left(\delta \right))d\mathbb{W}\left(\delta \right).\hfill \end{array}$$

(14)

These represent the required mild solutions. □

3.1. Results with Lipschitz Coefficients

Here, we analyze the ES of solutions to a coupled system of NHSFDEs. The primary focus is on applying a weighted maximum norm to demonstrate that the integral Equation (10) coincides with the mild solution (15).

Theorem 1. 

Under the conditions ($A_1$) to ($A_3$), there exists a mild solution $\left(\mathrm{\Psi }\right(t)$$whichisunique,\mathrm{{\rm Y}}\left(t\right))$ to the coupled system of NHSFDEs (1), satisfying $\mathrm{\Psi }\left(0\right)={\alpha }_1,\mathrm{{\rm Y}}\left(0\right)={\alpha }_2$, which can be written in the following way:

$$\begin{array}{cc}\hfill \mathrm{\Psi }\left(t\right)& ={\mathbf{E}}_{\nu }\left(\frac{\mathcal{C}\left(\nu \right)}{1-\nu }{\mathbf{A}}_1{t}^{\nu }\right)[{\alpha }_1+g_1(0,{\alpha }_1)]-g_1(t,\mathrm{\Psi }\left(t\right))\hfill \\ \hfill & +\frac{\nu }{\mathcal{C}\left(\nu \right)\Gamma \left(\nu \right)}{\int }_0^t{(t-\delta )}^{\nu -1}{\mathsf{F}}_1(\delta ,\mathrm{{\rm Y}}\left(\delta \right)){\mathbf{E}}_{\nu ,\nu }\left(\frac{\mathcal{C}\left(\nu \right)}{1-\nu }{\mathbf{A}}_1{(t-\delta )}^{\nu }\right)d\delta \hfill \\ \hfill & +\frac{1-\nu }{\mathcal{C}\left(\nu \right)}{\int }_0^t{\mathbf{E}}_{\nu }\left(\frac{\mathcal{C}\left(\nu \right)}{1-\nu }{\mathbf{A}}_1{(t-\delta )}^{\nu }\right){\sigma }_1(\delta ,\mathrm{\Psi }\left(\delta \right))d\mathbb{W}\left(\delta \right)\hfill \\ \hfill & +\frac{\nu }{\mathcal{C}\left(\nu \right)\Gamma \left(\nu \right)}{\int }_0^t{(t-\delta )}^{\nu -1}{\mathbf{E}}_{\nu ,\nu }\left(\frac{\mathcal{C}\left(\nu \right)}{1-\nu }{\mathbf{A}}_1{(t-\delta )}^{\nu }\right){\sigma }_1(\delta ,\mathrm{\Psi }\left(\delta \right))d\mathbb{W}\left(\delta \right),\hfill \end{array}$$

(15)

and

$$\begin{array}{cc}\hfill \mathrm{{\rm Y}}\left(t\right)& ={\mathbf{E}}_{\nu }\left(\frac{\mathcal{C}\left(\nu \right)}{1-\nu }{\mathbf{A}}_2{t}^{\nu }\right)[{\alpha }_2+g_2(0,{\alpha }_2)]-g_2(t,\mathrm{{\rm Y}}\left(t\right))\hfill \\ \hfill & +\frac{\nu }{\mathcal{C}\left(\nu \right)\Gamma \left(\nu \right)}{\int }_0^t{(t-\delta )}^{\nu -1}{\mathbf{E}}_{\nu ,\nu }\left(\frac{\mathcal{C}\left(\nu \right)}{1-\nu }{\mathbf{A}}_2{(t-\delta )}^{\nu }\right){\mathsf{F}}_2(\delta ,\mathrm{\Psi }\left(\delta \right))d\delta \hfill \\ \hfill & +\frac{1-\nu }{\mathcal{C}\left(\nu \right)}{\int }_0^t{\mathbf{E}}_{\nu }\left(\frac{\mathcal{C}\left(\nu \right)}{1-\nu }{\mathbf{A}}_2{(t-\delta )}^{\nu }\right){\sigma }_2(\delta ,\mathrm{{\rm Y}}\left(\delta \right))d\mathbb{W}\left(\delta \right)\hfill \\ \hfill & +\frac{\nu }{\mathcal{C}\left(\nu \right)\Gamma \left(\nu \right)}{\int }_0^t{(t-\delta )}^{\nu -1}{\mathbf{E}}_{\nu ,\nu }\left(\frac{\mathcal{C}\left(\nu \right)}{1-\nu }{\mathbf{A}}_2{(t-\delta )}^{\nu }\right){\sigma }_2(\delta ,\mathrm{{\rm Y}}\left(\delta \right))d\mathbb{W}\left(\delta \right).\hfill \end{array}$$

(16)

Proof: 

We establish the result using Banach’s fixed point theorem on the product space as the extension of b-matric space defined in [45],

$$\mathcal{H}:=H_{\alpha }^2([0,t],{\mathbb{R}}^{\mathfrak{d}})\times H_{\alpha }^2([0,t],{\mathbb{R}}^{\mathfrak{d}}),$$

equipped with the weighted norm:

$${\parallel (\mathrm{\Psi },\mathrm{{\rm Y}})\parallel }_{\beta }^2:=\underset{t\in [0,t]}{sup}{\displaystyle \frac{{\mathbb{E}\parallel \mathrm{\Psi }\left(t\right)\parallel }^2+\mathbb{E}{\parallel \mathrm{{\rm Y}}\left(t\right)\parallel }^2}{{\mathbf{E}}_{\nu }2\nu -1\left(\beta t^{2\nu -1}\right)}}.$$

Define $\mathcal{T}=({\mathcal{T}}_1,{\mathcal{T}}_2):\mathcal{H}\to \mathcal{H}$ where

$$\begin{array}{cc}\hfill {\mathcal{T}}_1(\mathrm{\Psi },\mathrm{{\rm Y}})\left(t\right)& ={\mathbf{E}}_{\nu }\left(\frac{\mathcal{C}\left(\nu \right)}{1-\nu }{\mathbf{A}}_1{t}^{\nu }\right)[{\alpha }_1+g_1(0,{\alpha }_1)]-g_1(t,\mathrm{\Psi }\left(t\right))\hfill \\ \hfill & +\frac{\nu }{\mathcal{C}\left(\nu \right)\Gamma \left(\nu \right)}{\int }_0^t{(t-\delta )}^{\nu -1}{\mathbf{E}}_{\nu ,\nu }\left(\frac{\mathcal{C}\left(\nu \right)}{1-\nu }{\mathbf{A}}_1{(t-\delta )}^{\nu }\right){\mathsf{F}}_1(\delta ,\mathrm{{\rm Y}}\left(\delta \right))d\delta \hfill \\ \hfill & +\frac{1-\nu }{\mathcal{C}\left(\nu \right)}{\int }_0^t{\mathbf{E}}_{\nu }\left(\frac{\mathcal{C}\left(\nu \right)}{1-\nu }{\mathbf{A}}_1{(t-\delta )}^{\nu }\right){\sigma }_1(\delta ,\mathrm{\Psi }\left(\delta \right))d\mathbb{W}\left(\delta \right)\hfill \\ \hfill & +\frac{\nu }{\mathcal{C}\left(\nu \right)\Gamma \left(\nu \right)}{\int }_0^t{(t-\delta )}^{\nu -1}{\mathbf{E}}_{\nu ,\nu }\left(\frac{\mathcal{C}\left(\nu \right)}{1-\nu }{\mathbf{A}}_1{(t-\delta )}^{\nu }\right){\sigma }_1(\delta ,\mathrm{\Psi }\left(\delta \right))d\mathbb{W}\left(\delta \right),\hfill \end{array}$$

and

$$\begin{array}{cc}\hfill {\mathcal{T}}_2(\mathrm{\Psi },\mathrm{{\rm Y}})\left(t\right)& ={\mathbf{E}}_{\nu }\left(\frac{\mathcal{C}\left(\nu \right)}{1-\nu }{\mathbf{A}}_2{t}^{\nu }\right)[{\alpha }_2+g_2(0,{\alpha }_2)]-g_2(t,\mathrm{{\rm Y}}\left(t\right))\hfill \\ \hfill & +\frac{\nu }{\mathcal{C}\left(\nu \right)\Gamma \left(\nu \right)}{\int }_0^t{(t-\delta )}^{\nu -1}{\mathbf{E}}_{\nu ,\nu }\left(\frac{\mathcal{C}\left(\nu \right)}{1-\nu }{\mathbf{A}}_2{(t-\delta )}^{\nu }\right){\mathsf{F}}_2(\delta ,\mathrm{\Psi }\left(\delta \right))d\delta \hfill \\ \hfill & +\frac{1-\nu }{\mathcal{C}\left(\nu \right)}{\int }_0^t{\mathbf{E}}_{\nu }\left(\frac{\mathcal{C}\left(\nu \right)}{1-\nu }{\mathbf{A}}_2{(t-\delta )}^{\nu }\right){\sigma }_2(\delta ,\mathrm{{\rm Y}}\left(\delta \right))d\mathbb{W}\left(\delta \right)\hfill \\ \hfill & +\frac{\nu }{\mathcal{C}\left(\nu \right)\Gamma \left(\nu \right)}{\int }_0^t{(t-\delta )}^{\nu -1}{\mathbf{E}}_{\nu ,\nu }\left(\frac{\mathcal{C}\left(\nu \right)}{1-\nu }{\mathbf{A}}_2{(t-\delta )}^{\nu }\right){\sigma }_2(\delta ,\mathrm{{\rm Y}}\left(\delta \right))d\mathbb{W}\left(\delta \right).\hfill \end{array}$$

For ${\mathcal{T}}_1$: Using assumptions $\left(A_1\right)$–$\left(A_3\right)$ and Hölder’s inequality,

$$\begin{array}{cc}\hfill \mathbb{E}\parallel {\mathcal{T}}_1({\mathrm{\Psi }}_1,{\mathrm{{\rm Y}}}_1)\left(t\right)-{\mathcal{T}}_1({\mathrm{\Psi }}_2,{\mathrm{{\rm Y}}}_2)\left(t\right){\parallel }^2& \le 4{\mathfrak{L}}_1^2\mathbb{E}{\parallel {\mathrm{\Psi }}_1\left(t\right)-{\mathrm{\Psi }}_2\left(t\right)\parallel }^2\hfill \\ \hfill & +4M_1^2\parallel {\mathbf{A}}_1{\parallel }^2{\displaystyle \frac{T^{2\nu -1}}{2\nu -1}}{\mathfrak{L}}_1^2{\int }_0^t{(t-\delta )}^{2\nu -2}\mathbb{E}{\parallel {\mathrm{\Psi }}_1\left(\delta \right)-{\mathrm{\Psi }}_2\left(\delta \right)\parallel }^{2}d\delta \hfill \\ \hfill & +4M_1^2{\displaystyle \frac{T^{2\nu -1}}{2\nu -1}}{\mathfrak{L}}_{{\mathsf{F}}_1}{\int }_0^t{(t-\delta )}^{2\nu -2}(\mathbb{E}\parallel {\mathrm{\Psi }}_1\left(\delta \right)-{\mathrm{\Psi }}_2{\left(\delta \right)\parallel }^2\hfill \\ & +\mathbb{E}\parallel {\mathrm{{\rm Y}}}_1\left(\delta \right)-{\mathrm{{\rm Y}}}_2\left(\delta \right){\parallel }^2)d\delta \hfill \\ \hfill & +4M_1^2{\mathfrak{L}}_{\sigma 1}{\int }_0^t{(t-\delta )}^{2\nu -2}\mathbb{E}{\parallel {\mathrm{\Psi }}_1\left(\delta \right)-{\mathrm{\Psi }}_2\left(\delta \right)\parallel }^{2}d\delta ,\hfill \\ \hfill ⟹\mathbb{E}\parallel {\mathcal{T}}_1({\mathrm{\Psi }}_1,{\mathrm{{\rm Y}}}_1)\left(t\right)-{\mathcal{T}}_1({\mathrm{\Psi }}_2,{\mathrm{{\rm Y}}}_2)\left(t\right){\parallel }^2& \le 4{\mathfrak{L}}_1^2\mathbb{E}{\parallel {\mathrm{\Psi }}_1\left(t\right)-{\mathrm{\Psi }}_2\left(t\right)\parallel }^2\hfill \\ & +\frac{4u^2{M}_1^2{\parallel {\mathbf{A}}_1\parallel }^2{\mathfrak{L}}_1^2}{\mathcal{C}{\left(\nu \right)}^2\Gamma {\left(\nu \right)}^2(2\nu -1)}{\int }_0^t{(t-\delta )}^{2\nu -2}\mathbb{E}{\parallel {\mathrm{\Psi }}_1\left(\delta \right)-{\mathrm{\Psi }}_2\left(\delta \right)\parallel }^{2}d\delta \hfill \\ \hfill & +\frac{4u^2{M}_1^2{\mathfrak{L}}_{{\mathsf{F}}_1}{T}^{2\nu -1}}{\mathcal{C}{\left(\nu \right)}^2\Gamma {\left(\nu \right)}^2(2\nu -1)}{\int }_0^t{(t-\delta )}^{2\nu -2}(\mathbb{E}\parallel {\mathrm{\Psi }}_1\left(\delta \right)-{\mathrm{\Psi }}_2{\left(\delta \right)\parallel }^2\hfill \\ & +\mathbb{E}\parallel {\mathrm{{\rm Y}}}_1\left(\delta \right)-{\mathrm{{\rm Y}}}_2{\left(\delta \right)\parallel }^2)d\delta +\frac{8{(1-\nu )}^2{M}_1^2{\mathfrak{L}}_{\sigma 1}}{\mathcal{C}{\left(\nu \right)}^2}\mathbb{E}{\parallel {\mathrm{\Psi }}_1\left(t\right)-{\mathrm{\Psi }}_2\left(t\right)\parallel }^2\hfill \\ \hfill & +\frac{8u^2{M}_1^2{\mathfrak{L}}_{\sigma 1}}{\mathcal{C}{\left(\nu \right)}^2\Gamma {\left(\nu \right)}^2(2\nu -1)}{\int }_0^t{(t-\delta )}^{2\nu -2}\mathbb{E}{\parallel {\mathrm{\Psi }}_1\left(\delta \right)-{\mathrm{\Psi }}_2\left(\delta \right)\parallel }^{2}d\delta .\hfill \end{array}$$

For ${\mathcal{T}}_2$,

$$\begin{array}{cc}\hfill \mathbb{E}\parallel T_2({\mathrm{\Psi }}_1,{\mathrm{{\rm Y}}}_1)\left(t\right)-T_2({\mathrm{\Psi }}_2,{\mathrm{{\rm Y}}}_2)\left(t\right){\parallel }^2& \le 4{\mathfrak{L}}_2^2\mathbb{E}{\parallel {\mathrm{{\rm Y}}}_1\left(t\right)-{\mathrm{{\rm Y}}}_2\left(t\right)\parallel }^2\hfill \\ \hfill & +4M_2^2\parallel {\mathbf{A}}_2{\parallel }^2{\displaystyle \frac{T^{2\nu -1}}{2\nu -1}}{\mathfrak{L}}_2^2{\int }_0^t{(t-\delta )}^{2\nu -2}\mathbb{E}{\parallel {\mathrm{{\rm Y}}}_1\left(\delta \right)-{\mathrm{{\rm Y}}}_2\left(\delta \right)\parallel }^{2}d\delta \hfill \\ \hfill & +4M_2^2{\displaystyle \frac{T^{2\nu -1}}{2\nu -1}}{\mathfrak{L}}_{{\mathsf{F}}_2}{\int }_0^t{(t-\delta )}^{2\nu -2}(\mathbb{E}\parallel {\mathrm{\Psi }}_1\left(\delta \right)-{\mathrm{\Psi }}_2{\left(\delta \right)\parallel }^2\hfill \\ & +\mathbb{E}\parallel {\mathrm{{\rm Y}}}_1\left(\delta \right)-{\mathrm{{\rm Y}}}_2\left(\delta \right){\parallel }^2)d\delta \hfill \\ \hfill & +4M_2^2{\mathfrak{L}}_{\sigma 2}{\int }_0^t{(t-\delta )}^{2\nu -2}\mathbb{E}{\parallel {\mathrm{{\rm Y}}}_1\left(\delta \right)-{\mathrm{{\rm Y}}}_2\left(\delta \right)\parallel }^{2}d\delta ,\hfill \\ \hfill & \le 4{\mathfrak{L}}_2^2\mathbb{E}{\parallel {\mathrm{{\rm Y}}}_1\left(t\right)-{\mathrm{{\rm Y}}}_2\left(t\right)\parallel }^2\hfill \\ \hfill & +\frac{4u^2{M}_2^2{\parallel {\mathbf{A}}_2\parallel }^2{\mathfrak{L}}_2^2}{\mathcal{C}{\left(\nu \right)}^2\Gamma {\left(\nu \right)}^2(2\nu -1)}{\int }_0^t{(t-\delta )}^{2\nu -2}\mathbb{E}{\parallel {\mathrm{{\rm Y}}}_1\left(\delta \right)-{\mathrm{{\rm Y}}}_2\left(\delta \right)\parallel }^{2}d\delta \hfill \\ \hfill & +\frac{4u^2{M}_2^2{\mathfrak{L}}_{{\mathsf{F}}_2}{T}^{2\nu -1}}{\mathcal{C}{\left(\nu \right)}^2\Gamma {\left(\nu \right)}^2(2\nu -1)}{\int }_0^t{(t-\delta )}^{2\nu -2}(\mathbb{E}\parallel {\mathrm{\Psi }}_1\left(\delta \right)-{\mathrm{\Psi }}_2{\left(\delta \right)\parallel }^2\hfill \\ & +\mathbb{E}\parallel {\mathrm{{\rm Y}}}_1\left(\delta \right)-{\mathrm{{\rm Y}}}_2{\left(\delta \right)\parallel }^2)d\delta +\frac{8{(1-\nu )}^2{M}_2^2{\mathfrak{L}}_{\sigma 2}}{\mathcal{C}{\left(\nu \right)}^2}\mathbb{E}{\parallel {\mathrm{{\rm Y}}}_1\left(t\right)-{\mathrm{{\rm Y}}}_2\left(t\right)\parallel }^2\hfill \\ \hfill & +\frac{8u^2{M}_2^2{\mathfrak{L}}_{\sigma 2}}{\mathcal{C}{\left(\nu \right)}^2\Gamma {\left(\nu \right)}^2(2\nu -1)}{\int }_0^t{(t-\delta )}^{2\nu -2}\mathbb{E}{\parallel {\mathrm{{\rm Y}}}_1\left(\delta \right)-{\mathrm{{\rm Y}}}_2\left(\delta \right)\parallel }^{2}d\delta .\hfill \end{array}$$

Dividing both estimates by ${\mathbf{E}}_{2\nu -1}\left(\beta t^{2\nu -1}\right)$ and combining

$$\begin{array}{c}\parallel \mathcal{T}({\mathrm{\Psi }}_1,{\mathrm{{\rm Y}}}_1)-\mathcal{T}({\mathrm{\Psi }}_2,{\mathrm{{\rm Y}}}_2){\parallel }_{\beta }^2\le \left(max\left(\frac{4{\mathfrak{L}}_1^2+8{(1-\nu )}^2{M}_1^2{\mathfrak{L}}_{\sigma 1}/\mathcal{C}{\left(\nu \right)}^2}{1-4{\mathfrak{L}}_1^2},\right.\right.\hfill \\ \hfill \left.\left.\frac{4{\mathfrak{L}}_2^2+8{(1-\nu )}^2{M}_2^2{\mathfrak{L}}_{\sigma 2}/\mathcal{C}{\left(\nu \right)}^2}{1-4{\mathfrak{L}}_2^2}\right)+\frac{C}{\beta }\right){\parallel ({\mathrm{\Psi }}_1,{\mathrm{{\rm Y}}}_1)-({\mathrm{\Psi }}_2,{\mathrm{{\rm Y}}}_2)\parallel }_{\beta }^2,\end{array}$$

which implies that

$$\parallel \mathcal{T}({\mathrm{\Psi }}_1,{\mathrm{{\rm Y}}}_1)-\mathcal{T}({\mathrm{\Psi }}_2,{\mathrm{{\rm Y}}}_2){\parallel }_{\beta }^2\le \rho {\parallel ({\mathrm{\Psi }}_1,{\mathrm{{\rm Y}}}_1)-({\mathrm{\Psi }}_2,{\mathrm{{\rm Y}}}_2)\parallel }_{\beta }^2,$$

where C collects all the constants from the estimates. Choose a $\beta$ that is sufficiently large such that

$$\rho :={\displaystyle \frac{4max({\mathfrak{L}}_1^2,{\mathfrak{L}}_2^2)}{1-4max({\mathfrak{L}}_1^2,{\mathfrak{L}}_2^2)}}+{\displaystyle \frac{C}{\beta }}<1,$$

This is possible since $0<{\mathfrak{L}}_i<\frac{1}{2}$ by assumption $\left(A_1\right)$. Thus, $\mathcal{T}$ is a contraction on $(\mathcal{H},\parallel ·{\parallel }_{\beta },{\mathbb{R}}^{\mathfrak{d}})$. Hence, by the contraction principle, $\mathcal{T}$ has a unique fixed point. □

Further, we require the following results in our work. Using the representation theorem for martingales, i.e., for any function $h\in L^2(\mathrm{\Omega },{\mathcal{F}}_T,{\mathbb{R}}^{\mathfrak{d}})$, there exist unique adapted processes ${\theta }_1,{\theta }_2\in H_{\alpha }^2([0,t],{\mathbb{R}}^{\mathfrak{d}})$ such that

$$h=\mathbb{E}h+{\int }_0^T{\theta }_1\left(\delta \right)d\mathbb{W}\left(\delta \right)+{\int }_0^T{\theta }_2\left(\delta \right)d\mathbb{W}\left(\delta \right).$$

It is clear that

$$h=\sum _{n=1}^d{h}_n{e}_n=\sum _{n=1}^d\left(\mathbb{E}{h}_n+{\int }_0^T{\theta }_{1,n}\left(\delta \right)d\mathbb{W}\left(\delta \right)+{\int }_0^T{\theta }_{2,n}\left(\delta \right)d\mathbb{W}\left(\delta \right)\right)e_n,$$

where

$$h_n=\mathbb{E}{h}_n+{\int }_0^T{\theta }_{1,n}\left(\delta \right)d\mathbb{W}\left(\delta \right)+{\int }_0^T{\theta }_{2,n}\left(\delta \right)d\mathbb{W}\left(\delta \right),$$

$$h_n\in L^2(\mathrm{\Omega },{\mathcal{F}}_T,{\mathbb{R}}^{\mathfrak{d}})$$

To show uniqueness for the coupled system of NHSFDEs (1), it is important to prove that

$$\mathrm{{\rm Y}}(t,\alpha )=\mathrm{\Psi }(t,\alpha ).$$

For any $h\in L^2(\mathrm{\Omega },{\mathcal{F}}_T,{\mathbb{R}}^{\mathfrak{d}})$, we must verify

$$\mathbb{E}⟨\mathrm{\Psi }(t,\alpha ),h⟩=\mathbb{E}⟨\mathrm{{\rm Y}}(t,\alpha ),h⟩.$$

Equivalently,

$$\mathbb{E}⟨\mathrm{\Psi }(t,\alpha )-\mathrm{{\rm Y}}(t,\alpha ),h⟩=\sum _{n=1}^d\mathbb{E}⟨({\mathrm{\Psi }}_n(t,\alpha )-{\mathrm{{\rm Y}}}_n(t,\alpha ))h_n,e_n⟩.$$

Since ${\mathbb{R}}^{\mathfrak{d}}$ has the Euclidean norm,

$$\begin{array}{cc}\hfill {\left|\mathbb{E}⟨\mathrm{\Psi }(t,\alpha )-\mathrm{{\rm Y}}(t,\alpha ),h⟩\right|}^2& \le \sum _{n=1}^d{\left|\mathbb{E}\left({\mathrm{\Psi }}_n(t,\alpha )-{\mathrm{{\rm Y}}}_n(t,\alpha )\right)h_n\right|}^2,\hfill \\ \hfill & \le d\sum _{n=1}^d{\left|\mathbb{E}\left({\mathrm{\Psi }}_n(t,\alpha )-{\mathrm{{\rm Y}}}_n(t,\alpha )\right)h_n\right|}^2.\hfill \end{array}$$

Before finding $\parallel E⟨\mathrm{\Psi }(t,n)-\mathrm{{\rm Y}}(t,n),h⟩\parallel$. Define the following measurable and bounded functions:

$$\begin{array}{cc}\hfill {\mathrm{\Psi }}_n\left(t\right)& =\mathbb{E}\left[{\mathrm{\Psi }}_n(t,\alpha )h_n\right],\hfill \\ \hfill {\nu }_n\left(t\right)& =\mathbb{E}\left[{\mathsf{ϝ}}_{1,n}(t,\mathrm{{\rm Y}}(t,\alpha ))h_n\right],\hfill \\ \hfill {\omega }_n\left(t\right)& =\mathbb{E}\left[g_{1,n}(t,\mathrm{\Psi }(t,\alpha ))h_n\right],\hfill \\ \hfill {\tilde{\mathrm{\Psi }}}_n\left(t\right)& =\mathbb{E}\left[{\mathrm{{\rm Y}}}_n(t,\alpha )h_n\right],\hfill \\ \hfill {\tilde{\nu }}_n\left(t\right)& =\mathbb{E}\left[{\mathsf{F}}_{2,n}(t,\mathrm{\Psi }(t,\alpha ))h_n\right],\hfill \\ \hfill {\tilde{\omega }}_n\left(t\right)& =\mathbb{E}\left[g_{2,n}(t,\mathrm{{\rm Y}}(t,\alpha ))h_n\right].\hfill \end{array}$$

Remark 1. 

Since $\mathrm{\Psi }(t,\alpha ),\mathrm{{\rm Y}}(t,\alpha )\in H_{\alpha }^2([0,t],{\mathbb{R}}^{\mathfrak{d}})$, the functions ${\mathrm{\Psi }}_n, {\nu }_n, {\omega }_n, {\tilde{\mathrm{\Psi }}}_n, {\tilde{\nu }}_n, {\tilde{\omega }}_n$ are bounded and measurable on $[0,t]$.

Lemma 3. 

Let $t\in [0,t]$ and $c\in {\mathbb{R}}^{\mathfrak{d}}$; then, for the hybrid coupled stochastic neutral system involving the ABC derivative, the following hold:

$$\begin{array}{cc}\hfill {\mathrm{\Psi }}_n\left(t\right)& ={\displaystyle \frac{1-\nu }{B\left(\nu \right)}}c\mathbb{E}\left(\alpha +g(0,\alpha )\right)-{\omega }_n\left(t\right)+{\displaystyle \frac{\nu }{B\left(\nu \right)}}{\int }_0^t{\mathbf{E}}_{\nu }\left({\displaystyle \frac{\nu }{\nu -1}}{(t-\delta )}^{\nu }\right)\hfill \\ & \times \left[A{\mathrm{\Psi }}_n\left(\delta \right)+u_n\left(\delta \right)+\mathbb{E}{\theta }_n\left(\delta \right)\sigma (\delta ,\mathrm{\Psi }(\delta ,{\alpha }_1))+\mathbb{E}J(\delta ,\mathrm{\Psi }\left({\delta }^{-}\right))\right]d\delta ,\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill {\tilde{\mathrm{\Psi }}}_n\left(t\right)& ={\displaystyle \frac{1-\nu }{B\left(\nu \right)}}c{\mathbf{E}}_{\nu }\left(t^{\nu }A\right)\mathbb{E}\left({\alpha }_1+g(0,\alpha )\right)-{\tilde{\omega }}_n\left(t\right)+{\displaystyle \frac{\nu }{B\left(\nu \right)}}{\int }_0^t{\mathbf{E}}_{\nu }\left({\displaystyle \frac{\nu }{\nu -1}}{(t-\delta )}^{\nu }\right){(t-\delta )}^{\nu -1}\hfill \\ & \times {\mathbf{E}}_{\nu ,\nu }\left({(t-\delta )}^{\nu }A\right)\times \left({\tilde{\nu }}_n\left(\delta \right)+\mathbb{E}{\theta }_n\left(\delta \right)\sigma (\delta ,\mathrm{{\rm Y}}(\delta ,{\alpha }_2))+\mathbb{E}J(\delta ,\mathrm{{\rm Y}}\left({\delta }^{-}\right))\right)d\delta .\hfill \end{array}$$

(18)

where the hybrid jump function $J:[0,t]\times {\mathbb{R}}^{\mathfrak{d}}\to {\mathbb{R}}^{\mathfrak{d}}$ is defined as

$$J(\delta ,\mathrm{\Psi }\left({\delta }^{-}\right))=\underset{ϵ\to 0^{+}}{lim}\left(\mathrm{\Psi }(\delta +ϵ)-\mathrm{\Psi }\left(\delta \right)\right),$$

where $\mathrm{\Psi }\left({\delta }^{-}\right)$ denotes the left-limit of the process X at time s. The inclusion of the hybrid jump function $f(\delta ,X\left({\delta }^{-}\right))$ in Lemma 3 is a critical feature that significantly expands the modeling capability of the proposed framework beyond conventional stochastic differential equations. While the subsequent stability analysis (Theorem 4) and examples focus on the continuous dynamics for clarity and conciseness, the mathematical framework is explicitly designed to incorporate discrete events. Here, we discuss the role and potential impact of this function.

That is, J captures possible discrete jumps or switching effects in the hybrid stochastic system.

Proof: 

Since $\mathrm{\Psi }(t,{\alpha }_1)$ is a mild solution for the hybrid coupled ABC stochastic neutral differential system, it satisfies the stochastic ABC integral equation:

$$\begin{array}{cc}\hfill & {\displaystyle \frac{B\left(\nu \right)}{1-\nu }}{\int }_0^t{\displaystyle \frac{d}{d\delta }}\left(\mathrm{\Psi }(\delta ,{\alpha }_1)+g(\delta ,\mathrm{\Psi }(\delta ,{\alpha }_1))\right){\mathbf{E}}_{\nu }\left({\displaystyle \frac{\nu }{\nu -1}}{(t-\delta )}^{\nu }\right)d\delta \hfill \\ & =A\mathrm{\Psi }(t,{\alpha }_1)+b(t,\mathrm{\Psi }(t,{\alpha }_1))+\sigma (t,\mathrm{\Psi }(t,{\alpha }_1)){\displaystyle \frac{dW\left(t\right)}{dt}}+J(t,\mathrm{\Psi }\left(t^{-}\right)).\hfill \end{array}$$

(19)

Applying the inverse ABC fractional integral to both sides gives

$$\begin{array}{cc}\hfill \mathrm{\Psi }(t,{\alpha }_1)& ={\displaystyle \frac{1-\nu }{B\left(\nu \right)}}\left(g(0,\alpha )\right)+\alpha -g(t,\mathrm{\Psi }(t,{\alpha }_1))\hfill \\ \hfill & +{\displaystyle \frac{\nu }{B\left(\nu \right)}}{\int }_0^t{\mathbf{E}}_{\nu }\left({\displaystyle \frac{\nu }{\nu -1}}{(t-\delta )}^{\nu }\right)\left(A\mathrm{\Psi }\left(\delta \right)+b(\delta ,\mathrm{\Psi }(\delta \left)\right)\right)d\delta \hfill \\ \hfill & +{\displaystyle \frac{\nu }{B\left(\nu \right)}}{\int }_0^t{\mathbf{E}}_{\nu }\left({\displaystyle \frac{\nu }{\nu -1}}{(t-\delta )}^{\nu }\right)\sigma (\delta ,\mathrm{\Psi }\left(\delta \right))dW\left(\delta \right)\hfill \\ \hfill & +{\displaystyle \frac{\nu }{B\left(\nu \right)}}{\int }_0^t{\mathbf{E}}_{\nu }\left({\displaystyle \frac{\nu }{\nu -1}}{(t-\delta )}^{\nu }\right)J(\delta ,\mathrm{\Psi }\left({\delta }^{-}\right))d\delta .\hfill \end{array}$$

Taking the inner product with $h_n$ and the expectation $\mathbb{E}$ yields

$$\begin{array}{cc}\hfill {\mathrm{\Psi }}_n\left(t\right)& ={\displaystyle \frac{1-\nu }{B\left(\nu \right)}}c\mathbb{E}\left(g(0,\alpha +\alpha )\right)-{\omega }_n\left(t\right)+{\displaystyle \frac{\nu }{B\left(\nu \right)}}{\int }_0^t{\mathbf{E}}_{\nu }\left({\displaystyle \frac{\nu }{\nu -1}}{(t-\delta )}^{\nu }\right)\hfill \\ & \times \left[A{\mathrm{\Psi }}_n\left(\delta \right)+{\nu }_n\left(\delta \right)+\mathbb{E}{\theta }_n\left(\delta \right)\sigma (\delta ,\mathrm{\Psi }(\delta ,{\alpha }_1))+\mathbb{E}J(\delta ,\mathrm{\Psi }\left({\delta }^{-}\right))\right]d\delta ,\hfill \end{array}$$

from which we proved the result (17).

• Similarly, for the mild solution $\mathrm{\Psi }(t,{\alpha }_2)$,

$$\begin{array}{cc}\hfill \mathrm{\Psi }(t,{\alpha }_2)& ={\mathbf{E}}_{\nu }\left(t^{\nu }A\right)(g(0,\alpha )+\alpha -g(t,\mathrm{\Psi }(t,{\alpha }_2)))\hfill \\ \hfill & -{\int }_0^{t}A{(t-\delta )}^{\nu -1}{\mathbf{E}}_{\nu ,\nu }\left({(t-\delta )}^{\nu }A\right)g(\delta ,\mathrm{{\rm Y}}\left(\delta \right))d\delta \hfill \\ \hfill & +{\int }_0^t{(t-\delta )}^{\nu -1}{\mathbf{E}}_{\nu ,\nu }\left({(t-\delta )}^{\nu }A\right)b(\delta ,\mathrm{{\rm Y}}\left(\delta \right))d\delta \hfill \\ \hfill & +{\int }_0^t{(t-\delta )}^{\nu -1}{\mathbf{E}}_{\nu ,\nu }\left({(t-\delta )}^{\nu }A\right)\sigma (\delta ,\mathrm{{\rm Y}}\left(\delta \right))dW\left(\delta \right)\hfill \\ \hfill & +{\int }_0^t{(t-\delta )}^{\nu -1}{\mathbf{E}}_{\nu ,\nu }\left({(t-\delta )}^{\nu }A\right)J(\delta ,\mathrm{{\rm Y}}\left({\delta }^{-}\right))d\delta .\hfill \end{array}$$

Taking inner product with $h_n$ and expectation $\mathbb{E}$, using Itô’s isometry theorem, we derive

$$\begin{array}{cc}\hfill {\tilde{\mathrm{\Psi }}}_n\left(t\right)& ={\displaystyle \frac{1-\nu }{B\left(\nu \right)}}c{\mathbf{E}}_{\nu }\left(t^{\nu }A\right)\mathbb{E}\left(g(0,\alpha )\right)+\alpha -{\tilde{\omega }}_n\left(t\right)\hfill \\ \hfill & +{\displaystyle \frac{\nu }{B\left(\nu \right)}}{\int }_0^t{\mathbf{E}}_{\nu }\left({\displaystyle \frac{\nu }{\nu -1}}{(t-\delta )}^{\nu }\right){(t-\delta )}^{\nu -1}{\mathbf{E}}_{\nu ,\nu }\left({(t-\delta )}^{\nu }A\right)\hfill \\ \hfill & \times \left({\tilde{\nu }}_n\left(\delta \right)+\mathbb{E}{\theta }_n\left(\delta \right)\sigma (\delta ,\mathrm{{\rm Y}}(\delta ,{\alpha }_2))+\mathbb{E}J(\delta ,\mathrm{{\rm Y}}\left({\delta }^{-}\right))\right)d\delta ,\hfill \end{array}$$

from which we proved the result (18). □

Remark 2. 

Let $h\in L^2(\mathrm{\Omega },{\mathcal{F}}_T,{\mathbb{R}}^{\mathfrak{d}})$. Then, for any $t\in [0,t]$, we have

$$\begin{array}{cc}\hfill {\left|\mathbb{E}⟨\mathrm{\Psi }(t,{\alpha }_1)-\mathrm{\Psi }(t,{\alpha }_2),h⟩\right|}^2& \le 4d{\mathfrak{L}}_g^2\mathbb{E}\parallel \mathrm{\Psi }(t,{\alpha }_1)-\mathrm{\Psi }(t,{\alpha }_2){\parallel }^2\mathbb{E}{\parallel h\parallel }^2\hfill \\ \hfill & +4d{M}^2{\mathfrak{L}}_g^2{\parallel A\parallel }^2{\displaystyle \frac{T^{2\nu -1}}{2\nu -1}}\left({\int }_0^t\mathbb{E}{\parallel \mathrm{\Psi }(\delta ,{\alpha }_1)-\mathrm{{\rm Y}}(\delta ,{\alpha }_2)\parallel }^{2}d\delta \right)\mathbb{E}{\parallel h\parallel }^2\hfill \\ \hfill & +4d{M}^2{\mathfrak{L}}_b^2{\displaystyle \frac{T^{2\nu -1}}{2\nu -1}}\left({\int }_0^t\mathbb{E}{\parallel \mathrm{\Psi }(\delta ,{\alpha }_1)-\mathrm{{\rm Y}}(\delta ,{\alpha }_2)\parallel }^{2}d\delta \right)\mathbb{E}{\parallel h\parallel }^2\hfill \\ \hfill & +4d{M}^2{\mathfrak{L}}_{\sigma }^2\left({\int }_0^t{(t-\delta )}^{2\nu -2}\mathbb{E}{\parallel \mathrm{\Psi }(\delta ,{\alpha }_1)-\mathrm{{\rm Y}}(\delta ,{\alpha }_2)\parallel }^{2}d\delta \right)\mathbb{E}{\parallel h\parallel }^2.\hfill \end{array}$$

Proof: 

We begin by expanding the left-hand side using the orthonormal basis ${\left\{h_n\right\}}_{n=1}^d$ of ${\mathbb{R}}^{\mathfrak{d}}$:

$$\begin{array}{cc}\hfill |⟨\mathrm{\Psi }(t,{\alpha }_1)-\mathrm{\Psi }(t,{\alpha }_2),h⟩{|}^2& ={\left|\sum _{n=1}^d⟨\mathrm{\Psi }(t,{\alpha }_1)-\mathrm{\Psi }(t,{\alpha }_2),h_n⟩\right|}^2,\hfill \\ \hfill & \le \left(\sum _{n=1}^d{1}^2\right)\left(\sum _{n=1}^d{\left|⟨\mathrm{\Psi }(t,{\alpha }_1)-\mathrm{\Psi }(t,{\alpha }_2),h_n⟩\right|}^2\right),\hfill \\ \hfill & =d\sum _{n=1}^d{\left|⟨\mathrm{\Psi }(t,{\alpha }_1)-\mathrm{\Psi }(t,{\alpha }_2),h_n⟩\right|}^2.\hfill \end{array}$$

Taking the expectation

$${\left|\mathbb{E}⟨\mathrm{\Psi }(t,{\alpha }_1)-\mathrm{\Psi }(t,{\alpha }_2),h⟩\right|}^2\le d\sum _{n=1}^d{\left|\mathbb{E}⟨\mathrm{\Psi }(t,{\alpha }_1)-\mathrm{\Psi }(t,{\alpha }_2),h_n⟩\right|}^2.$$

Since ${\mathrm{\Psi }}_n\left(t\right)$ and ${\tilde{\mathrm{\Psi }}}_n\left(t\right)$ are projections, we have

$${\left|\mathbb{E}⟨\mathrm{\Psi }(t,{\alpha }_1)-\mathrm{\Psi }(t,{\alpha }_2),h_n⟩\right|}^2\le {\parallel {\mathrm{\Psi }}_n\left(t\right)-{\tilde{\mathrm{\Psi }}}_n\left(t\right)\parallel }^2,$$

and thus,

$${\left|\mathbb{E}⟨\mathrm{\Psi }(t,{\alpha }_1)-\mathrm{\Psi }(t,{\alpha }_2),h⟩\right|}^2\le d\sum _{n=1}^d{\parallel {\mathrm{\Psi }}_n\left(t\right)-{\tilde{\mathrm{\Psi }}}_n\left(t\right)\parallel }^2.$$

Now, estimate $\parallel {\mathrm{\Psi }}_n\left(t\right)-{\tilde{\mathrm{\Psi }}}_n\left(t\right)\parallel$ carefully.

By the mild solution with ABC derivative:

$$\begin{array}{cc}\hfill \parallel {\mathrm{\Psi }}_n\left(t\right)-{\tilde{\mathrm{\Psi }}}_n\left(t\right)\parallel & \le \parallel {\omega }_n\left(t\right)-{\tilde{\omega }}_n\left(t\right)\parallel +M\parallel A\parallel {\int }_0^t{(t-\delta )}^{\nu -1}\parallel {\omega }_n\left(\delta \right)-{\tilde{\omega }}_n\left(\delta \right)\parallel d\delta \hfill \\ & +M{\int }_0^t{(t-\delta )}^{\nu -1}\parallel {\nu }_n\left(\delta \right)-{\tilde{\nu }}_n\left(\delta \right)\parallel d\delta \hfill \\ & +M{\mathfrak{L}}_{\sigma }{\int }_0^t{(t-\delta )}^{\nu -1}\mathbb{E}(\parallel {\theta }_n\left(\delta \right)\parallel \parallel \mathrm{\Psi }(\delta ,{\alpha }_1)-\mathrm{{\rm Y}}(\delta ,{\alpha }_2)\parallel )d\delta .\hfill \end{array}$$

Applying Cauchy-Schwarz inequality on each integral:

First integral:

$${\int }_0^t{(t-\delta )}^{\nu -1}\parallel {\omega }_n\left(\delta \right)-{\tilde{\omega }}_n\left(\delta \right)\parallel d\delta \le {\left({\int }_0^t{(t-\delta )}^{2\nu -2}d\delta \right)}^{{\displaystyle \frac{1}{2}}}{\left({\int }_0^t{\parallel {\omega }_n\left(\delta \right)-{\tilde{\omega }}_n\left(\delta \right)\parallel }^{2}d\delta \right)}^{{\displaystyle \frac{1}{2}}}.$$

Second integral:

$${\int }_0^t{(t-\delta )}^{\nu -1}\parallel {\nu }_n\left(\delta \right)-{\tilde{\nu }}_n\left(\delta \right)\parallel d\delta \le {\left({\int }_0^t{(t-\delta )}^{2\nu -2}d\delta \right)}^{{\displaystyle \frac{1}{2}}}{\left({\int }_0^t{\parallel {\nu }_n\left(\delta \right)-{\tilde{\nu }}_n\left(\delta \right)\parallel }^{2}d\delta \right)}^{{\displaystyle \frac{1}{2}}}.$$

Third integral:

$$\begin{array}{cc}\hfill & {\int }_0^t{(t-\delta )}^{\nu -1}\mathbb{E}(\parallel {\theta }_n\left(\delta \right)\parallel \parallel \mathrm{\Psi }(\delta ,{\alpha }_1)-\mathrm{{\rm Y}}(\delta ,{\alpha }_2)\parallel )d\delta \hfill \\ & \le {\left({\int }_0^t\mathbb{E}{\parallel {\theta }_n\left(\delta \right)\parallel }^{2}d\delta \right)}^{{\displaystyle \frac{1}{2}}}{\left({\int }_0^t{(t-\delta )}^{2\nu -2}\mathbb{E}{\parallel \mathrm{\Psi }(\delta ,{\alpha }_1)-\mathrm{{\rm Y}}(\delta ,{\alpha }_2)\parallel }^{2}d\delta \right)}^{{\displaystyle \frac{1}{2}}}.\parallel {\mathrm{\Psi }}_n\left(t\right)-{\tilde{\mathrm{\Psi }}}_n\left(t\right)\parallel .\hfill \end{array}$$

Bounded by terms involving integrals of differences,

Thus,

$$\begin{array}{cc}\hfill \parallel {\mathrm{\Psi }}_n\left(t\right)-{\tilde{\mathrm{\Psi }}}_n\left(t\right)\parallel & \le {\mathfrak{L}}_g{\left(\mathbb{E}\parallel \mathrm{\Psi }(t,{\alpha }_1)-\mathrm{\Psi }(t,{\alpha }_2){\parallel }^2\right)}^{{\displaystyle \frac{1}{2}}}{\left(\mathbb{E}\parallel h_n{\parallel }^2\right)}^{{\displaystyle \frac{1}{2}}}\hfill \\ & +M{\mathfrak{L}}_g\parallel A\parallel \sqrt{{\displaystyle \frac{T^{2\nu -1}}{2\nu -1}}}{\left({\int }_0^t\mathbb{E}{\parallel \mathrm{\Psi }(\delta ,{\alpha }_1)-\mathrm{{\rm Y}}(\delta ,{\alpha }_2)\parallel }^{2}d\delta \right)}^{{\displaystyle \frac{1}{2}}}{\left(\mathbb{E}\parallel h_n{\parallel }^2\right)}^{{\displaystyle \frac{1}{2}}}\hfill \\ & +M{\mathfrak{L}}_b\sqrt{{\displaystyle \frac{T^{2\nu -1}}{2\nu -1}}}{\left({\int }_0^t\mathbb{E}{\parallel \mathrm{\Psi }(\delta ,{\alpha }_1)-\mathrm{{\rm Y}}(\delta ,{\alpha }_2)\parallel }^{2}d\delta \right)}^{{\displaystyle \frac{1}{2}}}{\left(\mathbb{E}\parallel h_n{\parallel }^2\right)}^{{\displaystyle \frac{1}{2}}}\hfill \\ & +M{\mathfrak{L}}_{\sigma }{\left({\int }_0^t{(t-\delta )}^{2\nu -2}\mathbb{E}{\parallel \mathrm{\Psi }(\delta ,{\alpha }_1)-\mathrm{{\rm Y}}(\delta ,{\alpha }_2)\parallel }^{2}d\delta \right)}^{{\displaystyle \frac{1}{2}}}{\left(\mathbb{E}\parallel h_n{\parallel }^2\right)}^{{\displaystyle \frac{1}{2}}}.\hfill \end{array}$$

Now, summing over $n=1,\dots ,d$ and multiplying by d leads exactly to

$$\begin{array}{cc}\hfill {\left|\mathbb{E}⟨\mathrm{\Psi }(t,{\alpha }_1)-\mathrm{\Psi }(t,{\alpha }_2),h⟩\right|}^2& \le 4d{\mathfrak{L}}_g^2\mathbb{E}\parallel \mathrm{\Psi }(t,{\alpha }_1)-\mathrm{\Psi }(t,{\alpha }_2){\parallel }^2\mathbb{E}{\parallel h\parallel }^2\hfill \\ & +4d{M}^2{\mathfrak{L}}_g^2{\parallel A\parallel }^2{\displaystyle \frac{T^{2\nu -1}}{2\nu -1}}\left({\int }_0^t\mathbb{E}{\parallel \mathrm{\Psi }(\delta ,{\alpha }_1)-\mathrm{{\rm Y}}(\delta ,{\alpha }_2)\parallel }^{2}d\delta \right)\mathbb{E}{\parallel h\parallel }^2\hfill \\ & +4d{M}^2{\mathfrak{L}}_b^2{\displaystyle \frac{T^{2\nu -1}}{2\nu -1}}\left({\int }_0^t\mathbb{E}{\parallel \mathrm{\Psi }(\delta ,{\alpha }_1)-\mathrm{{\rm Y}}(\delta ,{\alpha }_2)\parallel }^{2}d\delta \right)\mathbb{E}{\parallel h\parallel }^2\hfill \\ & +4d{M}^2{\mathfrak{L}}_{\sigma }^2\left({\int }_0^t{(t-\delta )}^{2\nu -2}\mathbb{E}{\parallel \mathrm{\Psi }(\delta ,{\alpha }_1)-\mathrm{{\rm Y}}(\delta ,{\alpha }_2)\parallel }^{2}d\delta \right)\mathbb{E}{\parallel h\parallel }^2.\hfill \end{array}$$

This concludes the proof. □

Theorem 2. 

The mild solution for the coupled system of NHSFDEs (1) is unique, if for any $\delta >0$ satisfying

$${\mathfrak{L}}_g^2+{\displaystyle \frac{{\nu }^2{M}^2}{\mathcal{C}{\left(\nu \right)}^2}}\left({\displaystyle \frac{{\mathfrak{L}}_g^2{\parallel A\parallel }^2+{\mathfrak{L}}_b^2}{\Gamma {\left(\nu \right)}^2}}{\displaystyle \frac{T^{2\nu -1}}{2\nu -1}}+{\displaystyle \frac{{\mathfrak{L}}_{\sigma }^2}{(2\nu -1){\delta }^{1-2\nu }}}\right)<{\displaystyle \frac{1}{4\delta d}},$$

(20)

where ${\mathfrak{L}}_g, {\mathfrak{L}}_b, {\mathfrak{L}}_{\sigma }$ are Lipschitz constants for $g_i, {\mathsf{F}}_i, {\sigma }_i$ respectively, and M bounds the MLFs.

Proof. 

Let $T^{*}=inf\{t\in [0,t]:\mathrm{\Psi }(t,{\alpha }_1)\ne \mathrm{{\rm Y}}(t,{\alpha }_2)\mathrm{or}\mathrm{{\rm Y}}(t,{\alpha }_2)\ne \mathrm{\Psi }(t,{\alpha }_1)\}$. We prove $T^{*}=T$ by contradiction. Suppose $T^{*}<T$. For $t\in [T^{*},T^{*}+\delta ]$, using the ABC integral form,

$$\begin{array}{cc}\hfill {\mathbb{E}\parallel \mathrm{\Psi }\left(t\right)-\mathrm{{\rm Y}}\left(t\right)\parallel }^2& \le 4d\mathbb{E}\parallel g_1(t,\mathrm{\Psi })-g_2{(t,\mathrm{{\rm Y}})\parallel }^2+4d{\displaystyle \frac{{\nu }^2}{\mathcal{C}{\left(\nu \right)}^2\Gamma {\left(\nu \right)}^2}}\hfill \\ & \times \mathbb{E}{∥{\int }_{T^{*}}^t{(t-\delta )}^{\nu -1}[{\mathbf{A}}_1\mathrm{\Psi }-{\mathbf{A}}_2\mathrm{{\rm Y}}+{\mathsf{F}}_1(\delta ,\mathrm{{\rm Y}})-{\mathsf{F}}_2(\delta ,\mathrm{\Psi })]d\delta ∥}^2\hfill \\ \hfill & +4d{\displaystyle \frac{{(1-\nu )}^2}{\mathcal{C}{\left(\nu \right)}^2}}\mathbb{E}{\parallel {\sigma }_1(t,\mathrm{\Psi })-{\sigma }_2(t,\mathrm{{\rm Y}})\parallel }^2+4d{\displaystyle \frac{{\nu }^2}{\mathcal{C}{\left(\nu \right)}^2\Gamma {\left(\nu \right)}^2}}\hfill \\ & \times \mathbb{E}{∥{\int }_{T^{*}}^t{(t-\delta )}^{\nu -1}[{\sigma }_1(\delta ,\mathrm{\Psi })-{\sigma }_2(\delta ,\mathrm{{\rm Y}})]dW\left(\delta \right)∥}^2.\hfill \end{array}$$

For the deterministic integrals,

$$\begin{array}{cc}\hfill {\int }_{T^{*}}^t{(t-\delta )}^{2\nu -2}d\delta & ={\displaystyle \frac{{\delta }^{2\nu -1}}{2\nu -1}},\hfill \end{array}$$

and

$$\begin{array}{cc}\hfill {\int }_{T^{*}}^t\mathbb{E}{\parallel \mathrm{\Psi }-\mathrm{{\rm Y}}\parallel }^{2}d\delta & \le \underset{s\in [T^{*},T^{*}+\delta ]}{sup}\delta \mathbb{E}{\parallel \mathrm{\Psi }\left(\delta \right)-\mathrm{{\rm Y}}\left(\delta \right)\parallel }^2.\hfill \end{array}$$

For the stochastic integral,

$$\mathbb{E}{∥{\int }_{T^{*}}^t{(t-\delta )}^{\nu -1}[{\sigma }_1-{\sigma }_2]dW\left(\delta \right)∥}^2\le {\mathfrak{L}}_{\sigma }^2{\displaystyle \frac{{\delta }^{2\nu -1}}{2\nu -1}}\underset{s\in [T^{*},T^{*}+\delta ]}{sup}\mathbb{E}{\parallel \mathrm{\Psi }-\mathrm{{\rm Y}}\parallel }^2.$$

Using the Lipschitz conditions and Itô’s isometry, and by combining all estimates:

$$\begin{array}{cc}\hfill \underset{t\in [T^{*},T^{*}+\delta ]}{sup}\mathbb{E}{\parallel \mathrm{\Psi }-\mathrm{{\rm Y}}\parallel }^2& \le 4d{\mathfrak{L}}_g^{2}sup\mathbb{E}{\parallel \mathrm{\Psi }-\mathrm{{\rm Y}}\parallel }^2\hfill \\ & +{\displaystyle \frac{4d{\nu }^2{M}^2}{\mathcal{C}{\left(\nu \right)}^2\Gamma {\left(\nu \right)}^2}}\left({\displaystyle \frac{2\parallel {\mathbf{A}}_1{\parallel }^2+2{\mathfrak{L}}_b^2}{2\nu -1}}\delta +{\displaystyle \frac{{\mathfrak{L}}_{\sigma }^2{\delta }^{2\nu -1}}{2\nu -1}}\right)sup\mathbb{E}{\parallel \mathrm{\Psi }-\mathrm{{\rm Y}}\parallel }^2.\hfill \end{array}$$

Under condition (20), with a Grönwall-type Argument, we obtain,

$$\underset{t\in [T^{*},T^{*}+\delta ]}{sup}{\mathbb{E}\parallel \mathrm{\Psi }\left(t\right)-\mathrm{{\rm Y}}\left(t\right)\parallel }^2\le {C\left(\delta \right)sup\mathbb{E}\parallel \mathrm{\Psi }-\mathrm{{\rm Y}}\parallel }^2<sup\mathbb{E}{\parallel \mathrm{\Psi }-\mathrm{{\rm Y}}\parallel }^2.$$

where $C\left(\nu \right)\le 1$, which forces ${sup\mathbb{E}\parallel \mathrm{\Psi }-\mathrm{{\rm Y}}\parallel }^2=0$, contradicting $T^{*}<T$. Thus $T^{*}=T$. □

3.2. Results Without Lipschitz Conditions

Here, we examine the ES of mild solutions to the coupled system (1) by applying Picard’s successive approximation method. The analysis relies on certain CTCs imposed on the coefficients. Specifically, we make the following five key assumptions:

Hypothesis 1. 

The functions ${\mathsf{F}}_i(t,·)$ and ${\sigma }_i(t,·)$ ($i=1,2$) satisfy the non-Lipschitz condition:

for any $t\in [0,t]$ and $\tilde{\mathrm{\Psi }},\widetilde{\mathrm{{\rm Y}}},\mathrm{\Psi },\mathrm{{\rm Y}}\in H^2([0,t],{\mathbb{R}}^{\mathfrak{d}})$,

$$\begin{array}{cc}\hfill & \parallel {\mathsf{F}}_1(t,\mathrm{{\rm Y}})-{\mathsf{F}}_1(t,\widetilde{\mathrm{{\rm Y}}}){\parallel }^2+\parallel {\sigma }_1(t,\mathrm{\Psi })-{\sigma }_1(t,\tilde{\mathrm{\Psi }}){\parallel }^2+\parallel {\mathsf{F}}_2(t,\mathrm{\Psi })-{\mathsf{F}}_2(t,\tilde{\mathrm{\Psi }}){\parallel }^2+{\parallel {\sigma }_2(t,\mathrm{{\rm Y}})-{\sigma }_2(t,\widetilde{\mathrm{{\rm Y}}})\parallel }^2\hfill \\ & \le \nu \left(\parallel \mathrm{\Psi }-\tilde{\mathrm{\Psi }}{\parallel }^2+{\parallel \mathrm{{\rm Y}}-\widetilde{\mathrm{{\rm Y}}}\parallel }^2\right),\hfill \end{array}$$

where $\nu (·):{\mathbb{R}}^{+}\to {\mathbb{R}}^{+}$ is a concave, non-decreasing function.

Hypothesis 2. 

There exists a constant $N>0$ such that for all $t\in [0,t]$,

$$\parallel {\mathsf{F}}_1{(t,0)\parallel }^2 \vee \parallel {\sigma }_1{(t,0)\parallel }^2 \vee \parallel {\mathsf{F}}_2{(t,0)\parallel }^2 \vee {\parallel {\sigma }_2(t,0)\parallel }^2\le N.$$

Hypothesis 3. 

The mappings $g_1$ and $g_2$ satisfy Lipschitz conditions: there exists $L\in (0,1)$ such that

$$\mathbb{E}\parallel g_i(t,{\mathrm{\Psi }}_i)-g_i(t,{\tilde{\mathrm{\Psi }}}_i){\parallel }^2 \le L^2\mathbb{E}{\parallel {\mathrm{\Psi }}_i-{\tilde{\mathrm{\Psi }}}_i\parallel }^2,\mathit{for}\mathit{all}t\in [0,t],\mathit{where}i=1,2.$$

Hypothesis 4. 

There exists a non-decreasing continuous function $G:[0,t]\times {\mathbb{R}}^{+}\to {\mathbb{R}}^{+}$ such that for some $D>0$, if a non-negative continuous function $\nu \left(t\right)$ satisfies

$$\nu \left(t\right)\le {\int }_0^{t}DG(\delta ,\nu \left(\delta \right))d\delta ,t\in [0,t],$$

then $\nu \left(t\right)\equiv 0$ on $[0,t]$.

Hypothesis 5. 

For all $\beta >0$ and $0\le \nu \left(0\right)$,

$$\nu \left(t\right)=\beta {\int }_0^{t}G(\delta ,\nu \left(\delta \right))d\delta +\nu \left(0\right),$$

has a solution on $[0,t]$. By considering these assumptions, we will find the ES results for the mild solutions to (1).

Theorem 3. 

Assume that H1–H5 hold. Then, the CSNHSDE (1) admits a mild solution, which is unique on $[0,t]$. We check the existence by the Picard successive approximation method. Define the sequence for the stochastic processes ${\left\{({\mathrm{\Psi }}_n\left(t\right),{\mathrm{{\rm Y}}}_n\left(t\right))\right\}}_{n\ge 0}$ as given:

$$\left\{\begin{array}{c}{\mathrm{\Psi }}_0\left(t\right)={\mathbf{E}}_{\nu }\left(t^{\nu }{\mathbf{A}}_1\right)[{\alpha }_1+g_1(0,{\alpha }_1)],\hfill \\ {\mathrm{{\rm Y}}}_0\left(t\right)={\mathbf{E}}_{\nu }\left(t^{\nu }{\mathbf{A}}_2\right)[{\alpha }_2+g_2(0,{\alpha }_2)],\hfill \end{array}\right.$$

(21)

and for $n\ge 0$,

$$\left\{\begin{array}{c}{\mathrm{\Psi }}_{n+1}\left(t\right)+g_1(t,{\mathrm{\Psi }}_{n+1}\left(t\right))={\mathrm{\Psi }}_0\left(t\right)+{\mathcal{G}}_1\left({\mathrm{\Psi }}_{n+1}\right)\left(t\right)+{\mathcal{C}}_1\left({\mathrm{{\rm Y}}}_n\right)\left(t\right)+{\mathrm{\Sigma }}_1\left({\mathrm{\Psi }}_n\right)\left(t\right),\hfill \\ {\mathrm{{\rm Y}}}_{n+1}\left(t\right)+g_2(t,{\mathrm{{\rm Y}}}_{n+1}\left(t\right))={\mathrm{{\rm Y}}}_0\left(t\right)+{\mathcal{G}}_2\left({\mathrm{{\rm Y}}}_{n+1}\right)\left(t\right)+{\mathcal{C}}_2\left({\mathrm{\Psi }}_n\right)\left(t\right)+{\mathrm{\Sigma }}_2\left({\mathrm{{\rm Y}}}_n\right)\left(t\right),\hfill \end{array}\right.$$

(22)

where the operators are defined by

$$\begin{array}{c}\hfill {\mathcal{G}}_1\left({\mathrm{\Psi }}_{n+1}\right)\left(t\right)=-{\int }_0^t{\mathbf{A}}_1{(t-\delta )}^{\nu -1}{\mathbf{E}}_{\nu ,\nu }\left({(t-\delta )}^{\nu }{\mathbf{A}}_1\right)g_1(\delta ,{\mathrm{\Psi }}_{n+1}\left(\delta \right))d\delta ,\\ \hfill {\mathcal{C}}_1\left({\mathrm{{\rm Y}}}_n\right)\left(t\right)={\int }_0^t{(t-\delta )}^{\nu -1}{\mathbf{E}}_{\nu ,\nu }\left({(t-\delta )}^{\nu }{\mathbf{A}}_1\right){\mathsf{F}}_1(\delta ,{\mathrm{{\rm Y}}}_n\left(\delta \right))d\delta ,\\ \hfill {\mathrm{\Sigma }}_1\left({\mathrm{\Psi }}_n\right)\left(t\right)={\int }_0^t{(t-\delta )}^{\nu -1}{\mathbf{E}}_{\nu ,\nu }\left({(t-\delta )}^{\nu }{\mathbf{A}}_1\right){\sigma }_1(\delta ,{\mathrm{\Psi }}_n\left(\delta \right))d\mathbb{W}\left(\delta \right),\end{array}$$

and similarly for Y:

$$\begin{array}{c}\hfill {\mathcal{G}}_2\left({\mathrm{{\rm Y}}}_{n+1}\right)\left(t\right)=-{\int }_0^t{\mathbf{A}}_2{(t-\delta )}^{\nu -1}{\mathbf{E}}_{\nu ,\nu }\left({(t-\delta )}^{\nu }{\mathbf{A}}_2\right)g_2(\delta ,{\mathrm{{\rm Y}}}_{n+1}\left(\delta \right))d\delta ,\\ \hfill {\mathcal{C}}_2\left({\mathrm{\Psi }}_n\right)\left(t\right)={\int }_0^t{(t-\delta )}^{\nu -1}{\mathbf{E}}_{\nu ,\nu }\left({(t-\delta )}^{\nu }{\mathbf{A}}_2\right){\mathsf{F}}_2(\delta ,{\mathrm{\Psi }}_n\left(\delta \right))d\delta ,\\ \hfill {\mathrm{\Sigma }}_2\left({\mathrm{{\rm Y}}}_n\right)\left(t\right)={\int }_0^t{(t-\delta )}^{\nu -1}{\mathbf{E}}_{\nu ,\nu }\left({(t-\delta )}^{\nu }{\mathbf{A}}_2\right){\sigma }_2(\delta ,{\mathrm{{\rm Y}}}_n\left(\delta \right))d\mathbb{W}\left(\delta \right).\end{array}$$

To prove the existence results of this theorem, we consider the following lemma.

Lemma 4. 

Using the assumptions H1–H3, the sequence ${\left\{({\mathrm{\Psi }}_n\left(t\right),{\mathrm{{\rm Y}}}_n\left(t\right))\right\}}_{n\ge 0}$ defined by the Picard iteration for the coupled ABC system is well-defined and uniformly bounded in $H^2([0,t],{\mathbb{R}}^{\mathfrak{d}})\times H^2([0,t],{\mathbb{R}}^{\mathfrak{d}})$. Specifically, there exists a constant $C_T>0$ such that

$$\underset{n\ge 0}{sup}\left(\underset{0\le t\le T}{sup}\mathbb{E}\parallel {\mathrm{\Psi }}_n{\left(t\right)\parallel }^2+\underset{0\le t\le T}{sup}\mathbb{E}{\parallel {\mathrm{{\rm Y}}}_n\left(t\right)\parallel }^2\right)\le C_T.$$

Proof: 

For the coupled ABC fractional system with derivatives, we define the iterative scheme as follows:

$$\begin{array}{cc}\hfill {\mathrm{\Psi }}_{n+1}\left(t\right)& ={\displaystyle \frac{1-\nu }{\mathcal{C}\left(\nu \right)}}({\mathbf{A}}_1{\alpha }_1+{\mathsf{F}}_1(0,{\alpha }_2))+{\displaystyle \frac{\nu }{\mathcal{C}\left(\nu \right)\Gamma \left(\nu \right)}}{\int }_0^t{(t-\delta )}^{\nu -1}({\mathbf{A}}_1{\alpha }_1+{\mathsf{F}}_1(0,{\alpha }_2))d\delta \hfill \\ \hfill & -g_1(t,{\mathrm{\Psi }}_{n+1}\left(t\right))+{\displaystyle \frac{1-\nu }{\mathcal{C}\left(\nu \right)}}{\mathsf{F}}_1(t,{\mathrm{{\rm Y}}}_n\left(t\right))+{\displaystyle \frac{\nu }{\mathcal{C}\left(\nu \right)\Gamma \left(\nu \right)}}{\int }_0^t{(t-\delta )}^{\nu -1}{\mathsf{F}}_1(\delta ,{\mathrm{{\rm Y}}}_n\left(\delta \right))d\delta \hfill \\ \hfill & +{\displaystyle \frac{1-\nu }{\mathcal{C}\left(\nu \right)}}{\sigma }_1(t,{\mathrm{\Psi }}_n\left(t\right))\mathbb{W}\left(t\right)+{\displaystyle \frac{\nu }{\mathcal{C}\left(\nu \right)\Gamma \left(\nu \right)}}{\int }_0^t{(t-\delta )}^{\nu -1}{\sigma }_1(\delta ,{\mathrm{\Psi }}_n\left(\delta \right))d\mathbb{W}\left(\delta \right),\hfill \end{array}$$

and symmetrically for ${\mathrm{{\rm Y}}}_{n+1}\left(t\right)$ with ${\mathbf{A}}_2$, ${\mathsf{F}}_2$, ${\sigma }_2$, and $\mathbb{W}$. First, consider the neutral-term structure. From $\left(H_3\right)$, for $0<L<1$,

$$\begin{array}{cc}\hfill \mathbb{E}\parallel {\mathrm{\Psi }}_{n+1}\left(t\right)+g_1(t,{\mathrm{\Psi }}_{n+1}\left(t\right)){\parallel }^2& \ge {(1-L)}^2\mathbb{E}{\parallel {\mathrm{\Psi }}_{n+1}\left(t\right)\parallel }^2,\hfill \\ \hfill \mathbb{E}\parallel g_1(t,{\mathrm{\Psi }}_{n+1}\left(t\right)){\parallel }^2& \le L^2\mathbb{E}{\parallel {\mathrm{\Psi }}_{n+1}\left(t\right)\parallel }^2.\hfill \end{array}$$

This implies

$$\begin{array}{cc}\hfill \mathbb{E}\parallel {\mathrm{\Psi }}_{n+1}{\left(t\right)\parallel }^2& \le \mathbb{E}\parallel {\mathrm{\Psi }}_{n+1}\left(t\right)+g_1(t,{\mathrm{\Psi }}_{n+1}\left(t\right)){\parallel }^2{\displaystyle \frac{1}{{(1-L)}^2}}+{\displaystyle \frac{L}{{(1-L)}^2}}\mathbb{E}{\parallel {\alpha }_1\parallel }^2.\hfill \end{array}$$

We analyze the norm of the Picard iteration by decomposing it into deterministic and stochastic components:

$$\begin{array}{cc}\hfill \mathbb{E}\parallel {\mathrm{\Psi }}_{n+1}\left(t\right)+g_1(t,{\mathrm{\Psi }}_{n+1}\left(t\right)){\parallel }^2& \le 5(\mathbb{E}\parallel {\mathrm{\Psi }}_0\left(t\right){\parallel }^2+\mathbb{E}\parallel g_1(t,{\mathrm{\Psi }}_{n+1}\left(t\right)){\parallel }^2+\mathbb{E}{∥{\displaystyle \frac{1-\nu }{\mathcal{C}\left(\nu \right)}}{\mathsf{F}}_1(t,{\mathrm{{\rm Y}}}_n\left(t\right))∥}^2\hfill \\ \hfill & +\mathbb{E}{∥{\displaystyle \frac{\nu }{\mathcal{C}\left(\nu \right)\Gamma \left(\nu \right)}}{\int }_0^t{(t-\delta )}^{\nu -1}{\mathsf{F}}_1(\delta ,{\mathrm{{\rm Y}}}_n\left(\delta \right))d\delta ∥}^2+\mathbb{E}{∥\mathrm{stochastic}\mathrm{terms}∥}^2),\hfill \end{array}$$

and

$$\begin{array}{cc}\hfill \mathbb{E}\parallel {\mathrm{\Psi }}_0{\left(t\right)\parallel }^2& \le 2{∥{\displaystyle \frac{1-\nu }{\mathcal{C}\left(\nu \right)}}({\mathbf{A}}_1{\alpha }_1+{\mathsf{F}}_1(0,{\alpha }_2))∥}^2+2{∥{\displaystyle \frac{\nu }{\mathcal{C}\left(\nu \right)\Gamma \left(\nu \right)}}{\int }_0^t{(t-\delta )}^{\nu -1}({\mathbf{A}}_1{\alpha }_1+{\mathsf{F}}_1(0,{\alpha }_2))d\delta ∥}^2,\hfill \\ \hfill & \le C_1(\nu ,\mathcal{C}\left(\nu \right),T)(\mathbb{E}\parallel {\alpha }_1{\parallel }^2+\mathbb{E}\parallel {\alpha }_2{\parallel }^2).\hfill \end{array}$$

Using the hypothesis $H1$ and $H2$ with the concavity of $\nu$:

$$\begin{array}{cc}\hfill \mathbb{E}\parallel {\mathsf{F}}_1(t,{\mathrm{{\rm Y}}}_n\left(t\right)){\parallel }^2& \le 2\nu (\mathbb{E}\parallel {\mathrm{{\rm Y}}}_n\left(t\right){\parallel }^2)+2N,\hfill \\ \hfill \mathbb{E}{∥{\int }_0^t{(t-\delta )}^{\nu -1}{\mathsf{F}}_1(\delta ,{\mathrm{{\rm Y}}}_n\left(\delta \right))d\delta ∥}^2& \le {\displaystyle \frac{t^{2\nu -1}}{2\nu -1}}{\int }_0^t\nu (\mathbb{E}\parallel {\mathrm{{\rm Y}}}_n\left(\delta \right){\parallel }^2)d\delta .\hfill \end{array}$$

By Itô isometry and hypothesis $H1$ with $H2$:

$$\begin{array}{cc}\hfill \mathbb{E}{∥{\int }_0^t{(t-\delta )}^{\nu -1}{\sigma }_1(\delta ,{\mathrm{\Psi }}_n\left(\delta \right))d\mathbb{W}\left(\delta \right)∥}^2& ={\int }_0^t{(t-\delta )}^{2\nu -2}\mathbb{E}{\parallel {\sigma }_1(\delta ,{\mathrm{\Psi }}_n\left(\delta \right))\parallel }^{2}d\delta \hfill \\ \hfill & \le {\displaystyle \frac{t^{2\nu -1}}{2\nu -1}}\left(N+{\int }_0^t\nu (\mathbb{E}\parallel {\mathrm{\Psi }}_n\left(\delta \right){\parallel }^2)d\delta \right).\hfill \end{array}$$

$$\begin{array}{cc}\hfill \mathbb{E}\parallel {\mathrm{\Psi }}_{n+1}{\left(t\right)\parallel }^2& \le {\displaystyle \frac{C_1}{{(1-L)}^2}}+{\displaystyle \frac{C_2(\nu ,\mathcal{C}\left(\nu \right))}{{(1-L)}^2}}{\int }_0^t\nu (\mathbb{E}\parallel {\mathrm{{\rm Y}}}_n{\left(\delta \right)\parallel }^2)d\delta +{\displaystyle \frac{C_3(\nu ,\mathcal{C}\left(\nu \right))}{{(1-L)}^2}}{\int }_0^t\nu (\mathbb{E}\parallel {\mathrm{\Psi }}_n\left(\delta \right){\parallel }^2)d\delta \hfill \\ & +{\displaystyle \frac{L}{{(1-L)}^2}}\mathbb{E}{\parallel {\alpha }_1\parallel }^2.\hfill \end{array}$$

The same estimate holds for $\mathbb{E}\parallel {\mathrm{{\rm Y}}}_{n+1}{\left(t\right)\parallel }^2$ with ${\mathrm{\Psi }}_n$ and ${\mathrm{{\rm Y}}}_n$ interchanged. Define ${\nu }_n\left(t\right):=\mathbb{E}\parallel {\mathrm{\Psi }}_n{\left(t\right)\parallel }^2+\mathbb{E}{\parallel {\mathrm{{\rm Y}}}_n\left(t\right)\parallel }^2$. Then,

$$\begin{array}{cc}\hfill {\nu }_{n+1}\left(t\right)& \le C_4+C_5{\int }_0^t\nu \left({\nu }_n\left(\delta \right)\right)d\delta +C_6{\int }_0^t\nu \left({\nu }_{n+1}\left(\delta \right)\right)d\delta ,\hfill \\ \hfill & \le C_4+(C_5+C_6){\int }_0^t\nu (max\{{\nu }_n\left(\delta \right),{\nu }_{n+1}\left(\delta \right)\})d\delta .\hfill \end{array}$$

By the comparison principle for integral equations and the concavity of $\nu$, there exists a global bound ${\nu }_n\left(t\right)\le C_T$ for all n and $t\in [0,t]$. □

Lemma 5. 

The sequence ${\left\{({\mathrm{\Psi }}_n,{\mathrm{{\rm Y}}}_n)\right\}}_{n\ge 0}$ is Cauchy under assumptions H1–H5, in $H^2([0,t],{\mathbb{R}}^{\mathfrak{d}})\times H^2([0,t],{\mathbb{R}}^{\mathfrak{d}})$.

Proof: 

For the proposed coupled system in the sense of ABC derivatives, we analyze the difference between successive iterates:

$$\begin{array}{cc}\hfill {\mathrm{\Psi }}_{n+1}\left(t\right)-{\mathrm{\Psi }}_n\left(t\right)& =-\left[g_1(t,{\mathrm{\Psi }}_{n+1}\left(t\right))-g_1(t,{\mathrm{\Psi }}_n\left(t\right))\right]+{\displaystyle \frac{1-\nu }{\mathcal{C}\left(\nu \right)}}\left[{\mathsf{F}}_1(t,{\mathrm{{\rm Y}}}_n\left(t\right))-{\mathsf{F}}_1(t,{\mathrm{{\rm Y}}}_{n-1}\left(t\right))\right]\hfill \\ \hfill & +{\displaystyle \frac{\nu }{\mathcal{C}\left(\nu \right)\Gamma \left(\nu \right)}}{\int }_0^t{(t-\delta )}^{\nu -1}\left[{\mathsf{F}}_1(\delta ,{\mathrm{{\rm Y}}}_n\left(\delta \right))-{\mathsf{F}}_1(\delta ,{\mathrm{{\rm Y}}}_{n-1}\left(\delta \right))\right]d\delta \hfill \\ \hfill & +{\displaystyle \frac{1-\nu }{\mathcal{C}\left(\nu \right)}}\left[{\sigma }_1(t,{\mathrm{\Psi }}_n\left(t\right))-{\sigma }_1(t,{\mathrm{\Psi }}_{n-1}\left(t\right))\right]\mathbb{W}\left(t\right)\hfill \\ \hfill & +{\displaystyle \frac{\nu }{\mathcal{C}\left(\nu \right)\Gamma \left(\nu \right)}}{\int }_0^t{(t-\delta )}^{\nu -1}\left[{\sigma }_1(\delta ,{\mathrm{\Psi }}_n\left(\delta \right))-{\sigma }_1(\delta ,{\mathrm{\Psi }}_{n-1}\left(\delta \right))\right]d\mathbb{W}\left(\delta \right).\hfill \end{array}$$

Taking expectations and applying Itô isometry,

$$\begin{array}{cc}\hfill \mathbb{E}\parallel {\mathrm{\Psi }}_{n+1}\left(t\right)-{\mathrm{\Psi }}_n{\left(t\right)\parallel }^2& \le {\displaystyle \frac{1}{{(1-L)}^2}}(\mathbb{E}\parallel {\displaystyle \frac{1-\nu }{\mathcal{C}\left(\nu \right)}}\left[{\mathsf{F}}_1(t,{\mathrm{{\rm Y}}}_n\left(t\right))-{\mathsf{F}}_1(t,{\mathrm{{\rm Y}}}_{n-1}\left(t\right))\right]{\parallel }^2\hfill \\ \hfill & +\mathbb{E}\parallel {\displaystyle \frac{\nu }{\mathcal{C}\left(\nu \right)\Gamma \left(\nu \right)}}{\int }_0^t{(t-\delta )}^{\nu -1}\left[{\mathsf{F}}_1(\delta ,{\mathrm{{\rm Y}}}_n\left(\delta \right))-{\mathsf{F}}_1(\delta ,{\mathrm{{\rm Y}}}_{n-1}\left(\delta \right))\right]d\delta {\parallel }^2\hfill \\ \hfill & +{\displaystyle \frac{{(1-\nu )}^2}{\mathcal{C}{\left(\nu \right)}^2}}\mathbb{E}\parallel {\sigma }_1(t,{\mathrm{\Psi }}_n\left(t\right))-{\sigma }_1(t,{\mathrm{\Psi }}_{n-1}\left(t\right)){\parallel }^2\hfill \\ \hfill & +{\displaystyle \frac{{\nu }^2}{\mathcal{C}{\left(\nu \right)}^2\Gamma {\left(\nu \right)}^2}}\mathbb{E}\parallel {\int }_0^t{(t-\delta )}^{\nu -1}\left[{\sigma }_1(\delta ,{\mathrm{\Psi }}_n\left(\delta \right))-{\sigma }_1(\delta ,{\mathrm{\Psi }}_{n-1}\left(\delta \right))\right]d\mathbb{W}\left(\delta \right){\parallel }^2),\hfill \end{array}$$

Applying the martingale property of Itô integrals,

$$\begin{array}{cc}\hfill \mathbb{E}\parallel {\mathrm{\Psi }}_{n+1}\left(t\right)-{\mathrm{\Psi }}_n{\left(t\right)\parallel }^2& ={\displaystyle \frac{1}{{(1-L)}^2}}({\displaystyle \frac{{(1-\nu )}^2}{\mathcal{C}{\left(\nu \right)}^2}}\mathbb{E}\parallel {\mathsf{F}}_1(t,{\mathrm{{\rm Y}}}_n)-{\mathsf{F}}_1(t,{\mathrm{{\rm Y}}}_{n-1}){\parallel }^2\hfill \\ \hfill & +{\displaystyle \frac{{\nu }^2}{\mathcal{C}{\left(\nu \right)}^2\Gamma {\left(\nu \right)}^2}}\mathbb{E}\parallel {\int }_0^t{(t-\delta )}^{\nu -1}\left[{\mathsf{F}}_1(\delta ,{\mathrm{{\rm Y}}}_n)-{\mathsf{F}}_1(\delta ,{\mathrm{{\rm Y}}}_{n-1})\right]d\delta {\parallel }^2\hfill \\ \hfill & +{\displaystyle \frac{{(1-\nu )}^2}{\mathcal{C}{\left(\nu \right)}^2}}\mathbb{E}\parallel {\sigma }_1(t,{\mathrm{\Psi }}_n)-{\sigma }_1(t,{\mathrm{\Psi }}_{n-1}){\parallel }^2\hfill \\ & +{\displaystyle \frac{{\nu }^2}{\mathcal{C}{\left(\nu \right)}^2\Gamma {\left(\nu \right)}^2}}{\int }_0^t{(t-\delta )}^{2\nu -2}\mathbb{E}\parallel {\sigma }_1(\delta ,{\mathrm{\Psi }}_n)-{\sigma }_1(\delta ,{\mathrm{\Psi }}_{n-1}){\parallel }^{2}d\delta ).\hfill \end{array}$$

Using $\left(H_1\right)$ with $\nu (·)$ concavity,

$$\begin{array}{cc}\hfill \mathbb{E}\parallel {\mathrm{\Psi }}_{n+1}\left(t\right)-{\mathrm{\Psi }}_n{\left(t\right)\parallel }^2& \le {\displaystyle \frac{1}{{(1-L)}^2}}({\displaystyle \frac{2{(1-\nu )}^2}{\mathcal{C}{\left(\nu \right)}^2}}\nu \left(\mathbb{E}\parallel {\mathrm{{\rm Y}}}_n-{\mathrm{{\rm Y}}}_{n-1}{\parallel }^2\right)\hfill \\ \hfill & +{\displaystyle \frac{2{\nu }^2{T}^{2\nu -1}}{\mathcal{C}{\left(\nu \right)}^2\Gamma {\left(\nu \right)}^2(2\nu -1)}}{\int }_0^t\nu \left(\mathbb{E}\parallel {\mathrm{{\rm Y}}}_n-{\mathrm{{\rm Y}}}_{n-1}{\parallel }^2\right)d\delta \hfill \\ \hfill & +{\displaystyle \frac{2{(1-\nu )}^2}{\mathcal{C}{\left(\nu \right)}^2}}\nu \left(\mathbb{E}\parallel {\mathrm{\Psi }}_n-{\mathrm{\Psi }}_{n-1}{\parallel }^2\right)\hfill \\ \hfill & +{\displaystyle \frac{2{\nu }^2}{\mathcal{C}{\left(\nu \right)}^2\Gamma {\left(\nu \right)}^2(2\nu -1)}}{\int }_0^t\nu \left(\mathbb{E}\parallel {\mathrm{\Psi }}_n-{\mathrm{\Psi }}_{n-1}{\parallel }^2\right)d\delta ).\hfill \end{array}$$

The symmetric estimate holds for $\mathbb{E}\parallel {\mathrm{{\rm Y}}}_{n+1}\left(t\right)-{\mathrm{{\rm Y}}}_n{\left(t\right)\parallel }^2$. Defining ${\lambda }_n\left(t\right):={sup}_{m\ge n}\left(\mathbb{E}\parallel {\mathrm{\Psi }}_m-{\mathrm{\Psi }}_n{\parallel }^2+\mathbb{E}{\parallel {\mathrm{{\rm Y}}}_m-{\mathrm{{\rm Y}}}_n\parallel }^2\right)$, we obtain

$${\lambda }_n\left(t\right)\le D{\int }_0^{t}G(\delta ,{\lambda }_{n-1}\left(\delta \right))d\delta ,$$

where $D=max\left\{{\displaystyle \frac{4{(1-\nu )}^2}{\mathcal{C}{\left(\nu \right)}^2{(1-L)}^2}},{\displaystyle \frac{4u^2{T}^{2\nu -1}}{\mathcal{C}{\left(\nu \right)}^2\Gamma {\left(\nu \right)}^2{(1-L)}^2(2\nu -1)}}\right\}$. That is, $\lambda =0$ for all $t\in [0,t]$, via $H4$, which implies that $\lambda$ is continuous and ${lim}_{n\to \infty }{\lambda }_n\left(T\right)=\lambda \left(T\right)=0$; as a result, ${\left\{{\mathrm{\Psi }}_n\right\}}_{n\ge 0}$ is a Cauchy sequence in $H^2([0,t],{\mathbb{R}}^{\mathfrak{d}})$. □

Proof of Theorem 3. 

• Existence: Using Lemma (4), we represent $\mathrm{\Psi }$ and $\mathrm{{\rm Y}}$ as the limit of the sequences ${\left\{{\mathrm{\Psi }}_n\right\}}_{n\ge 0}$ and ${\left\{{\mathrm{{\rm Y}}}_n\right\}}_{n\ge 0}$. Now, in Lemma (4), as $n\to \infty$, the RHS of (21) becomes

$$\begin{array}{cc}\hfill \mathrm{\Psi }\left(t\right)& ={\displaystyle \frac{1-\nu }{\mathcal{C}\left(\nu \right)}}({\mathbf{A}}_1{\alpha }_1+{\mathsf{F}}_1(0,{\alpha }_2))\hfill \\ & +{\displaystyle \frac{\nu }{\mathcal{C}\left(\nu \right)\Gamma \left(\nu \right)}}{\int }_0^t{(t-\delta )}^{\nu -1}({\mathbf{A}}_1{\alpha }_1+{\mathsf{F}}_1(0,{\alpha }_2))d\delta -g_1(t,\mathrm{\Psi }\left(t\right))+{\displaystyle \frac{1-\nu }{\mathcal{C}\left(\nu \right)}}{\mathsf{F}}_1(t,\mathrm{{\rm Y}}\left(t\right))\hfill \\ \hfill & +{\displaystyle \frac{\nu }{\mathcal{C}\left(\nu \right)\Gamma \left(\nu \right)}}{\int }_0^t{(t-\delta )}^{\nu -1}{\mathsf{F}}_1(\delta ,\mathrm{{\rm Y}}\left(\delta \right))d\delta +{\displaystyle \frac{1-\nu }{\mathcal{C}\left(\nu \right)}}{\sigma }_1(t,\mathrm{\Psi }\left(t\right))\mathbb{W}\left(t\right)\hfill \\ \hfill & +{\displaystyle \frac{\nu }{\mathcal{C}\left(\nu \right)\Gamma \left(\nu \right)}}{\int }_0^t{(t-\delta )}^{\nu -1}{\sigma }_1(\delta ,\mathrm{\Psi }\left(\delta \right))d\mathbb{W}\left(\delta \right),\hfill \end{array}$$

and symmetrically for $\mathrm{{\rm Y}}\left(t\right)$.

• Uniqueness: Assume that $({\mathrm{\Psi }}_1,{\mathrm{{\rm Y}}}_1),({\mathrm{\Psi }}_2,{\mathrm{{\rm Y}}}_2)\in ([0,t],{\mathbb{R}}^{\mathfrak{d}})$ are two mild solutions of Equation (6). Using Lemma (4), we obtain the estimate

$$\parallel ({\mathrm{\Psi }}_1,{\mathrm{{\rm Y}}}_1)-({\mathrm{\Psi }}_2,{\mathrm{{\rm Y}}}_2){\parallel }_{H^2}^2\le D{\int }_0^{t}G\left(\delta ,\parallel ({\mathrm{\Psi }}_1,{\mathrm{{\rm Y}}}_1)-({\mathrm{\Psi }}_2,{\mathrm{{\rm Y}}}_2){\parallel }_{H^2}^2\right)d\delta .$$

Applying assumption $\left(H_4\right)$, we conclude that $\parallel ({\mathrm{\Psi }}_1,{\mathrm{{\rm Y}}}_1)-({\mathrm{\Psi }}_2,{\mathrm{{\rm Y}}}_2){\parallel }_{H^2}^2=0$ for all $t\in [0,t]$, which implies that $({\mathrm{\Psi }}_1,{\mathrm{{\rm Y}}}_1)\equiv ({\mathrm{\Psi }}_2,{\mathrm{{\rm Y}}}_2)$. This completes the proof. □

Corollary 1. 

Suppose that the hypotheses (H₁)–(H₅) are satisfied. Let $\left(\mathrm{\Psi }\right(t),\mathrm{{\rm Y}}(t\left)\right)$ and $(\tilde{\mathrm{\Psi }}\left(t\right),\widetilde{\mathrm{{\rm Y}}}\left(t\right))$ be two mild solutions of system (1) corresponding to initial values $({\alpha }_1,{\alpha }_2)$ and $({\tilde{\alpha }}_1,{\tilde{\alpha }}_2)$, respectively.

Then, there exists a constant $C>0$ such that

$$\underset{t\in [0,t]}{sup}\mathbb{E}\left(\parallel \mathrm{\Psi }\left(t\right)-\tilde{\mathrm{\Psi }}{\left(t\right)\parallel }^2+{\parallel \mathrm{{\rm Y}}\left(t\right)-\widetilde{\mathrm{{\rm Y}}}\left(t\right)\parallel }^2\right)\le C\mathbb{E}\left(\parallel {\alpha }_1-{\tilde{\alpha }}_1{\parallel }^2+{\parallel {\alpha }_2-{\tilde{\alpha }}_2\parallel }^2\right).$$

Proof. 

Let $\left(\mathrm{\Psi }\right(t),\mathrm{{\rm Y}}(t\left)\right)$ and $(\tilde{\mathrm{\Psi }}\left(t\right),\widetilde{\mathrm{{\rm Y}}}\left(t\right))$ be the mild solutions with respective initial conditions.

Taking the difference of the two mild solution expressions and applying Itô’s isometry, together with the assumptions $H1–H3,$ we obtain

$$\begin{array}{cc}\hfill & \mathbb{E}\parallel \mathrm{\Psi }\left(t\right)-\tilde{\mathrm{\Psi }}{\left(t\right)\parallel }^2+\mathbb{E}{\parallel \mathrm{{\rm Y}}\left(t\right)-{\displaystyle \widetilde{\mathrm{{\rm Y}}}}\left(t\right)\parallel }^2\le C_1\mathbb{E}\left(\parallel {\alpha }_1-{\tilde{\alpha }}_1{\parallel }^2+{\parallel {\alpha }_2-{\tilde{\alpha }}_2\parallel }^2\right)\hfill \\ \hfill & +C_2{\int }_0^t\nu \left(\mathbb{E}\parallel \mathrm{\Psi }\left(\delta \right)-\tilde{\mathrm{\Psi }}{\left(\delta \right)\parallel }^2+\mathbb{E}{\parallel \mathrm{{\rm Y}}\left(\delta \right)-\widetilde{\mathrm{{\rm Y}}}\left(\delta \right)\parallel }^2\right)d\delta ,\hfill \end{array}$$

for some positive constants $C_1$ and $C_2$.

Applying the assumption $H5$ and the structure of $G(t,\nu )$, and using standard integral inequalities (e.g., a generalized Gronwall-type argument under non-Lipschitz conditions), we deduce that

$$\underset{t\in [0,t]}{sup}\mathbb{E}\left(\parallel \mathrm{\Psi }\left(t\right)-\tilde{\mathrm{\Psi }}{\left(t\right)\parallel }^2+{\parallel \mathrm{{\rm Y}}\left(t\right)-\widetilde{\mathrm{{\rm Y}}}\left(t\right)\parallel }^2\right)\le C\mathbb{E}\left(\parallel {\alpha }_1-{\tilde{\alpha }}_1{\parallel }^2+{\parallel {\alpha }_2-{\tilde{\alpha }}_2\parallel }^2\right),$$

for some constant $C>0$ depending on T and the data of the problem. □

4. Mean-Square Mittag-Leffler Stability Analysis

The study of stability is fundamental for understanding the long-term behavior of dynamical systems, ensuring that solutions remain bounded and converge to an equilibrium over time. For the coupled system of HNSDEs (6), establishing a stability result is particularly crucial. The system’s complexity rising from memory effects (via the ABC derivative), stochastic disturbances, nonlinearities, and coupling between subsystems raises natural questions about its robustness and reliability. To address this, we adopt the concept of mean-square Mittag-Leffler stability. This notion is the natural generalization of exponential stability to the fractional-order stochastic setting. The Mittag-Leffler function captures the characteristic power-law decay and long-range memory effects inherent to fractional systems, while the mean-square criterion ensures that not only the average trajectory but also its variance (i.e., the spread of all possible sample paths) decays reliably. This provides a strong guarantee of performance under uncertainty. The following theorem establishes the sufficient conditions under which our coupled system exhibits this desirable form of stability, ensuring that the complex interactions between noise, memory, and coupling do not lead to divergent behavior.

Assumption  2. 

The following assumptions holds for the mean square stability of the coupled system (1).

Hypothesis  6. 

The nonlinear functions $g_i$, ${\mathsf{F}}_i$, and ${\sigma }_i$$(i=1,2)$are Lipschitz-continuous in the spatial variable with a common constant $L>0$:

$$\parallel g_i(t,\mathrm{\Psi })-g_i(t,y)\parallel \le L\parallel x-y\parallel ,\mathit{for}\mathit{all}x,y\in {\mathbb{R}}^{\mathfrak{d}}.$$

The same is true for ${\mathsf{F}}_i$ and ${\sigma }_i$.

Hypothesis  7. 

The matrices ${\mathbf{A}}_1$ and ${\mathbf{A}}_2$ are negative definite: there exist constants ${\mu }_1,{\mu }_2>0$ such that

$$⟨{\mathbf{A}}_{i}x,x⟩\le -{\mu }_i{\parallel x\parallel }^2,\mathit{for}\mathit{all}x\in {\mathbb{R}}^{\mathfrak{d}},i=1,2.$$

Hypothesis  8.

The initial data ${\alpha }_1$, ${\alpha }_2$ are square-integrable, i.e.,

$$\mathbb{E}\parallel {\alpha }_1{\parallel }^2<\infty ,\mathbb{E}{\parallel {\alpha }_2\parallel }^2<\infty .$$

Remark 3 

(Remark on Assumptions). The conditions $\left(A_1\right)–\left(A_3\right)$ and H1–H8 are technical but have practical interpretations. In particular, (H1) requires that the drift and diffusion satisfy a non-Lipschitz growth bound through a concave modulus $\nu (·)$, which covers many realistic cases such as Hölder-type nonlinearities ($\nu \left(\delta \right)=C{s}^{\alpha }$, $0<\alpha <1$), saturating functions, or logarithmic growth. For a given physical system, this condition can be verified either analytically by bounding the increments of the nonlinear terms or numerically by fitting a concave function ν to finite-difference tests or simulation data. Thus, while abstract, these assumptions provide verifiable criteria that link the mathematical theory to practical models.

Theorem 4. 

Consider the coupled system of NHSDEs (1) governed by the ABC derivative. Then, under assumptions H6–H8, the mild solution $\left(\mathrm{\Psi }\right(t),\mathrm{{\rm Y}}(t\left)\right)$ of the system (1) is mean-square Mittag-Leffler-stable. Thus, ∃ constants $0<C$ and $0<\lambda$, such that

$${\mathbb{E}\parallel \mathrm{\Psi }\left(t\right)\parallel }^2+\mathbb{E}{\parallel \mathrm{{\rm Y}}\left(t\right)\parallel }^2\le C{\mathbf{E}}_{\nu }(-\lambda t^{\nu })(\mathbb{E}\parallel {\alpha }_1{\parallel }^2+\mathbb{E}{\parallel {\alpha }_2\parallel }^2),\forall t\in [0,t].$$

Proof. 

Let $\left(\mathrm{\Psi }\right(t),\mathrm{{\rm Y}}(t\left)\right)$ be the mild solution to the system (1). From the mild solution representation using the ABC derivative, we have

$$\begin{array}{cc}\hfill \mathrm{\Psi }\left(t\right)& ={\mathbf{E}}_{\nu }\left(t^{\nu }{\mathbf{A}}_1\right)({\alpha }_1+g_1(0,{\alpha }_1))-g_1(t,\mathrm{\Psi }\left(t\right))\hfill \\ \hfill & -{\int }_0^t{\mathbf{A}}_1{(t-\delta )}^{\nu -1}{\mathbf{E}}_{\nu ,\nu }\left({(t-s)}^{\nu }{\mathbf{A}}_1\right)g_1(\delta ,\mathrm{\Psi }\left(\delta \right))d\delta \hfill \\ \hfill & +{\int }_0^t{(t-\delta )}^{\nu -1}{\mathbf{E}}_{\nu ,\nu }\left({(t-s)}^{\nu }{\mathbf{A}}_1\right){\mathsf{F}}_1(\delta ,\mathrm{{\rm Y}}\left(\delta \right))d\delta \hfill \\ \hfill & +{\int }_0^t{(t-\delta )}^{\nu -1}{\mathbf{E}}_{\nu ,\nu }\left({(t-s)}^{\nu }{\mathbf{A}}_1\right){\sigma }_1(\delta ,\mathrm{\Psi }\left(\delta \right))d\mathbb{W}\left(\delta \right),\hfill \end{array}$$

$$\begin{array}{cc}\hfill \mathrm{{\rm Y}}\left(t\right)& ={\mathbf{E}}_{\nu }\left(t^{\nu }{\mathbf{A}}_2\right)({\alpha }_2+g_2(0,{\alpha }_2))-g_2(t,\mathrm{{\rm Y}}\left(t\right))\hfill \\ \hfill & -{\int }_0^t{\mathbf{A}}_2{(t-\delta )}^{\nu -1}{\mathbf{E}}_{\nu ,\nu }\left({(t-s)}^{\nu }{\mathbf{A}}_2\right)g_2(\delta ,\mathrm{{\rm Y}}\left(\delta \right))d\delta \hfill \\ \hfill & +{\int }_0^t{(t-\delta )}^{\nu -1}{\mathbf{E}}_{\nu ,\nu }\left({(t-s)}^{\nu }{\mathbf{A}}_2\right){\mathsf{F}}_2(\delta ,\mathrm{\Psi }\left(\delta \right))d\delta \hfill \\ \hfill & +{\int }_0^t{(t-\delta )}^{\nu -1}{\mathbf{E}}_{\nu ,\nu }\left({(t-s)}^{\nu }{\mathbf{A}}_2\right){\sigma }_2(\delta ,\mathrm{{\rm Y}}\left(\delta \right))d\mathbb{W}\left(\delta \right).\hfill \end{array}$$

We now estimate ${\mathbb{E}\parallel \mathrm{\Psi }\left(t\right)\parallel }^2$ and ${\mathbb{E}\parallel \mathrm{{\rm Y}}\left(t\right)\parallel }^2$ separately. Before estimating the mean-square norms of the mild solution, we apply the inequality for the square of the norm of a sum of vectors. Let $x_1,x_2,\dots ,x_5\in {\mathbb{R}}^{\mathfrak{d}}$; then,

$${∥\sum _{i=1}^5{x}_i∥}^2\le 5\sum _{i=1}^5{\parallel x_i\parallel }^2.$$

Taking expectations preserves the inequality due to linearity. We apply this bound to both $\mathrm{\Psi }\left(t\right)$ and $\mathrm{{\rm Y}}\left(t\right)$, whose mild solutions consist of five additive terms, allowing us to estimate

$${\mathbb{E}\parallel \mathrm{\Psi }\left(t\right)\parallel }^2\le 5\sum _{i=1}^5\mathbb{E}\parallel {\mathrm{\Psi }}_i{\left(t\right)\parallel }^2{,\mathbb{E}\parallel \mathrm{{\rm Y}}\left(t\right)\parallel }^2\le 5\sum _{i=1}^5\mathbb{E}{\parallel {\mathrm{{\rm Y}}}_i\left(t\right)\parallel }^2,$$

where each ${\mathrm{\Psi }}_i\left(t\right)$, ${\mathrm{{\rm Y}}}_i\left(t\right)$ corresponds to a term in the mild solution expressions of $\mathrm{\Psi }\left(t\right)$ and $\mathrm{{\rm Y}}\left(t\right)$, respectively. Using the triangle inequality and Jensen’s inequality, we obtain

$$\begin{array}{cc}\hfill {\mathbb{E}\parallel \mathrm{\Psi }\left(t\right)\parallel }^2& \le 5[\parallel {\mathbf{E}}_{\nu }\left(t^{\nu }{\mathbf{A}}_1\right){\parallel }^2\mathbb{E}\parallel {\alpha }_1+g_1(0,{\alpha }_1){\parallel }^2+\mathbb{E}{\parallel g_1(t,\mathrm{\Psi }\left(t\right))\parallel }^2\hfill \\ \hfill & +{∥{\int }_0^t{\mathbf{A}}_1{(t-\delta )}^{\nu -1}{\mathbf{E}}_{\nu ,\nu }\left({(t-\delta )}^{\nu }{\mathbf{A}}_1\right)g_1(\delta ,\mathrm{\Psi }\left(\delta \right))d\delta ∥}^2\hfill \\ \hfill & +{∥{\int }_0^t{(t-\delta )}^{\nu -1}{\mathbf{E}}_{\nu ,\nu }\left({(t-\delta )}^{\nu }{\mathbf{A}}_1\right){\mathsf{F}}_1(\delta ,\mathrm{{\rm Y}}\left(\delta \right))d\delta ∥}^2\hfill \\ \hfill & +\mathbb{E}{∥{\int }_0^t{(t-\delta )}^{\nu -1}{\mathbf{E}}_{\nu ,\nu }\left({(t-\delta )}^{\nu }{\mathbf{A}}_1\right){\sigma }_1(\delta ,\mathrm{\Psi }\left(\delta \right))d\mathbb{W}\left(\delta \right)∥}^2].\hfill \end{array}$$

Using Lipschitz and boundedness assumptions, we derive

$${\mathbb{E}\parallel \mathrm{\Psi }\left(t\right)\parallel }^2\le C_1\mathbb{E}{\parallel {\alpha }_1\parallel }^2+C_2{\int }_0^t{(t-\delta )}^{2\nu -2}\left({\mathbb{E}\parallel \mathrm{\Psi }\left(\delta \right)\parallel }^2+\mathbb{E}{\parallel \mathrm{{\rm Y}}\left(\delta \right)\parallel }^2\right)d\delta .$$

Similarly, we bound

$$\begin{array}{cc}\hfill {\mathbb{E}\parallel \mathrm{{\rm Y}}\left(t\right)\parallel }^2& \le 5[\parallel {\mathbf{E}}_{\nu }\left(t^{\nu }{\mathbf{A}}_2\right){\parallel }^2\mathbb{E}\parallel {\alpha }_2+g_2(0,{\alpha }_2){\parallel }^2+\mathbb{E}{\parallel g_2(t,\mathrm{{\rm Y}}\left(t\right))\parallel }^2\hfill \\ \hfill & +\mathbb{E}{∥{\int }_0^t{\mathbf{A}}_2{(t-\delta )}^{\nu -1}{\mathbf{E}}_{\nu ,\nu }\left({(t-\delta )}^{\nu }{\mathbf{A}}_2\right)g_2(\delta ,\mathrm{{\rm Y}}\left(\delta \right))d\delta ∥}^2\hfill \\ \hfill & +\mathbb{E}{∥{\int }_0^t{(t-\delta )}^{\nu -1}{\mathbf{E}}_{\nu ,\nu }\left({(t-\delta )}^{\nu }{\mathbf{A}}_2\right){\mathsf{F}}_2(\delta ,\mathrm{\Psi }\left(\delta \right))d\delta ∥}^2\hfill \\ \hfill & +\mathbb{E}{∥{\int }_0^t{(t-\delta )}^{\nu -1}{\mathbf{E}}_{\nu ,\nu }\left({(t-\delta )}^{\nu }{\mathbf{A}}_2\right){\sigma }_2(\delta ,\mathrm{{\rm Y}}\left(\delta \right))d\mathbb{W}\left(\delta \right)∥}^2].\hfill \end{array}$$

which leads to

$${\mathbb{E}\parallel \mathrm{{\rm Y}}\left(t\right)\parallel }^2\le C_3\mathbb{E}{\parallel {\alpha }_2\parallel }^2+C_4{\int }_0^t{(t-\delta )}^{2\nu -2}\left({\mathbb{E}\parallel \mathrm{{\rm Y}}\left(\delta \right)\parallel }^2+\mathbb{E}{\parallel \mathrm{\Psi }\left(\delta \right)\parallel }^2\right)d\delta .$$

Define

$$\nu \left(t\right):={\mathbb{E}\parallel \mathrm{\Psi }\left(t\right)\parallel }^2+{\mathbb{E}\parallel \mathrm{{\rm Y}}\left(t\right)\parallel }^2,\nu \left(0\right)=\mathbb{E}\parallel {\alpha }_1{\parallel }^2+\mathbb{E}{\parallel {\alpha }_2\parallel }^2.$$

Then,

$$\nu \left(t\right)\le C_0\nu \left(0\right)+C{\int }_0^t{(t-\delta )}^{2\nu -2}\nu \left(\delta \right)d\delta ,$$

where $C_0,C>0$ depend on Lipschitz constants and operator norms. From the inequality

$$\nu \left(t\right)\le C_0\nu \left(0\right)+C{\int }_0^t{(t-s)}^{2\nu -2}\nu \left(\delta \right)d\delta ,$$

we apply the fractional Gronwall lemma (see e.g., Diethelm, 2010) to obtain

$$\nu \left(t\right)\le C\nu \left(0\right){\mathbf{E}}_{2\nu -1}\left(C{t}^{2\nu -1}\right),$$

and since $2\nu -1\in (0,1)$, we use the inequality

$${\mathbf{E}}_{2\nu -1}\left(C{t}^{2\nu -1}\right)\le C^{\prime }{\mathbf{E}}_{\nu }(-\lambda t^{\nu }),$$

for suitable $\lambda >0$ to conclude

$${\mathbb{E}\parallel \mathrm{\Psi }\left(t\right)\parallel }^2+{\mathbb{E}\parallel \mathrm{{\rm Y}}\left(t\right)\parallel }^2\le C{\mathbf{E}}_{\nu }(-\lambda t^{\nu })(\mathbb{E}\parallel {\alpha }_1{\parallel }^2+\mathbb{E}\parallel {\alpha }_2{\parallel }^2).$$

This proves mean-square Mittag-Leffler stability. □

Remark 4. 

The stability result is not merely a theoretical exercise. It provides a crucial design criterion for applications in smart grids, neural networks, and epidemiological modeling. By ensuring the system parameters (matrices ${\mathbf{A}}_1, {\mathbf{A}}_2$ and Lipschitz constants ${\mathfrak{L}}_F, {\mathfrak{L}}_{\sigma }$) satisfy the conditions of Theorem 4, an engineer or scientist can be confident that the underlying system will remain stable despite stochastic environmental fluctuations and complex subsystem interactions. The Mittag-Leffler function provides the precise rate at which the system returns to equilibrium after a disturbance.

5. Examples

This section of the research is devoted to the numerical applicability of the derived results concerning the existence of mild solutions and the mean-square stability analysis. We present some examples to demonstrate the validity of the obtained results.

Example 1 

Consider the following fractional neural hybrid stochastic coupled system with $\nu ={\displaystyle \frac{3}{4}}$:

$$\left\{\begin{array}{c}{}^{\mathbf{ABC}}\mathbb{D}_{0+}^{2/3}\left(\mathrm{\Psi }\left(t\right)+C\mathrm{\Psi }\left(t\right)\right)={\mathbf{A}}_1\mathrm{\Psi }\left(t\right)+B_1\mathrm{{\rm Y}}\left(t\right)+{\sigma }_1{\displaystyle \frac{d\mathbb{W}\left(t\right)}{dt}},\hfill \\ {}^{\mathbf{ABC}}\mathbb{D}_{0+}^{2/3}\left(\mathrm{{\rm Y}}\left(t\right)+D\mathrm{{\rm Y}}\left(t\right)\right)={\mathbf{A}}_2\mathrm{{\rm Y}}\left(t\right)+B_2\mathrm{\Psi }\left(t\right)+{\sigma }_2{\displaystyle \frac{d\mathbb{W}\left(t\right)}{dt}},\hfill \\ \mathrm{\Psi }\left(0\right)=x_0,\mathrm{{\rm Y}}\left(0\right)=y_0,\hfill \end{array}\right.$$

(23)

where

$$C=\left(\begin{array}{cc}0.5& 0\\ 0& 0.5\end{array}\right),D=\left(\begin{array}{cc}0.2& 0\\ 0& 0.2\end{array}\right),{\mathbf{A}}_1=\left(\begin{array}{cc}-2& 0\\ 0& -1\end{array}\right),{\mathbf{A}}_2=\left(\begin{array}{cc}-1& 0\\ 0& -3\end{array}\right),$$

$$B_1=B_2=\left(\begin{array}{cc}1& 0\\ 0& 1\end{array}\right),{\sigma }_1={\sigma }_2=\left(\begin{array}{cc}1& 0\\ 0& 1\end{array}\right).$$

We define the mappings:

$$g_1(t,\mathrm{\Psi })=C\mathrm{\Psi },g_2(t,\mathrm{{\rm Y}})=D\mathrm{{\rm Y}},{\mathsf{F}}_1(t,\mathrm{{\rm Y}})=B_1\mathrm{{\rm Y}},{\mathsf{F}}_2(t,\mathrm{\Psi })=B_2\mathrm{\Psi },{\sigma }_1(t,\mathrm{\Psi })={\sigma }_1,{\sigma }_2(t,\mathrm{{\rm Y}})={\sigma }_2.$$

The mild solutions of the system (23), based on the ABC derivative are given by

$$\begin{array}{cc}\hfill \mathrm{\Psi }\left(t\right)+C\mathrm{\Psi }\left(t\right)& ={\mathbf{E}}_{3/4}\left(t^{3/4}{\mathbf{A}}_1\right)(x_0+C{x}_0)-{\int }_0^t{\mathbf{A}}_1{(t-\delta )}^{-1/4}{\mathbf{E}}_{3/4,3/4}\left({(t-\delta )}^{3/4}{\mathbf{A}}_1\right)C\mathrm{\Psi }\left(\delta \right)d\delta \hfill \\ \hfill & +{\int }_0^t{(t-\delta )}^{-1/4}{\mathbf{E}}_{3/4,3/4}\left({(t-\delta )}^{3/4}{\mathbf{A}}_1\right)B_1\mathrm{{\rm Y}}\left(\delta \right)d\delta \hfill \\ \hfill & +{\int }_0^t{(t-\delta )}^{-1/4}{\mathbf{E}}_{3/4,3/4}\left({(t-\delta )}^{3/4}{\mathbf{A}}_1\right){\sigma }_{1}d\mathbb{W}\left(\delta \right),\hfill \\ \hfill \mathrm{{\rm Y}}\left(t\right)+D\mathrm{{\rm Y}}\left(t\right)& ={\mathbf{E}}_{3/4}\left(t^{3/4}{\mathbf{A}}_2\right)(y_0+D{y}_0)-{\int }_0^t{\mathbf{A}}_2{(t-\delta )}^{-1/4}{\mathbf{E}}_{3/4,3/4}\left({(t-\delta )}^{3/4}{\mathbf{A}}_2\right)D\mathrm{{\rm Y}}\left(\delta \right)d\delta \hfill \\ \hfill & +{\int }_0^t{(t-\delta )}^{-1/4}{\mathbf{E}}_{3/4,3/4}\left({(t-\delta )}^{3/4}{\mathbf{A}}_2\right)B_2\mathrm{\Psi }\left(\delta \right)d\delta \hfill \\ \hfill & +{\int }_0^t{(t-\delta )}^{-1/4}{\mathbf{E}}_{3/4,3/4}\left({(t-\delta )}^{3/4}{\mathbf{A}}_2\right){\sigma }_{2}d\mathbb{W}\left(\delta \right).\hfill \end{array}$$

All operators ${\mathsf{F}}_i$ and ${\sigma }_i$ are linear and therefore Lipschitz:

$$\parallel {\mathsf{F}}_1\left({\mathrm{{\rm Y}}}_1\right)-{\mathsf{F}}_1\left({\mathrm{{\rm Y}}}_2\right)\parallel =\parallel B_1({\mathrm{{\rm Y}}}_1-{\mathrm{{\rm Y}}}_2)\parallel \le \parallel B_1\parallel ·\parallel {\mathrm{{\rm Y}}}_1-{\mathrm{{\rm Y}}}_2\parallel ,$$

so condition $H1$ is satisfied with $\nu \left(r\right)=L^{2}r$. At the origin,

$${\mathsf{F}}_i(t,0)=0,{\sigma }_i(t,0)={\sigma }_i\Rightarrow \parallel {\mathsf{F}}_i{\left(0\right)\parallel }^2\vee {\parallel {\sigma }_i\left(0\right)\parallel }^2<\infty ,$$

so $H2$ holds.

The neutral terms $g_1\left(\mathrm{\Psi }\right)=C\mathrm{\Psi }$ and $g_2\left(\mathrm{{\rm Y}}\right)=D\mathrm{{\rm Y}}$ are Lipschitz with $\parallel C\parallel ,\parallel D\parallel <1$, satisfying $H3$:

$$\parallel g_i\left(Z_1\right)-g_i\left(Z_2\right)\parallel \le \parallel C\parallel ·\parallel Z_1-Z_2\parallel .$$

For the function $\nu \left(r\right)=L^{2}r$, any inequality of the form

$$\nu \left(t\right)\le D{\int }_0^t{L}^2\nu \left(\delta \right)d\delta ,$$

implies $\nu \left(t\right)\equiv 0$ by Gronwall’s inequality, satisfying $H4$.

The following integral equation

$$\nu \left(t\right)=\nu \left(0\right)+\beta {\int }_0^t{L}^2\nu \left(\delta \right)d\delta ,$$

has the explicit solution $\nu \left(t\right)=\nu \left(0\right)e^{\beta L^{2}t}$, which is globally defined on $[0,t]$, satisfying $\left(H_5\right)$. Since the system is linear with matrices ${\mathbf{A}}_1,{\mathbf{A}}_2$, i.e.,

$${\mathbb{E}\parallel \mathrm{{\rm Y}}\left(t\right)\parallel }^2+\mathbb{E}{\parallel \mathrm{\Psi }\left(t\right)\parallel }^2\le C{\mathbf{E}}_{3/4}(-\lambda t^{3/4}).$$

Therefore, for constants $C,\lambda >0$, the mild solution $\left(\mathrm{\Psi }\right(t),\mathrm{{\rm Y}}(t\left)\right)$ is mean-square Mittag-Leffler-stable and admits a unique mild solution on $[{\displaystyle \frac{1}{2}},1]$.

The Figure 1 and Figure 2 for the $\mathrm{\Psi }\left(t\right)$ and $\mathrm{{\rm Y}}\left(t\right)$ trajectories demonstrate the behavior of the linear coupled fractional stochastic system. Both variables exhibit smooth exponential decay due to negative eigenvalues $(A_1=-2,A_2=-3)$, with $\mathrm{\Psi }\left(t\right)$ starting at 2 and $\mathrm{{\rm Y}}\left(t\right)$ at 1. While $\mathrm{\Psi }\left(t\right)$ shows slightly wider variations due to noise, $\mathrm{{\rm Y}}$ decays faster and remains more tightly clustered, reflecting lower stochastic sensitivity. The blue $\mathrm{\Psi }\left(t\right)$ and red $\mathrm{{\rm Y}}\left(t\right)$ gradients highlight initial conditions fading over time, confirming the system’s mean-square Mittag-Leffler stability despite random perturbations.

Figure 1. Three-dimensional plot for dynamic of $\mathrm{\Psi }\left(t\right)$.

Figure 2. Three-dimensional plot for dynamics of $\mathrm{{\rm Y}}\left(t\right)$.

Example 2. 

Consider the following nonlinear fractional neural hybrid stochastic system with $\nu ={\displaystyle \frac{2}{3}}$:

$$\left\{\begin{array}{c}{}^{\mathbf{ABC}}\mathbb{D}_{0+}^{2/3}\left(\mathrm{\Psi }\left(t\right)+tanh\left(\mathrm{\Psi }\right(t\left)\right)\right)=-\mathrm{\Psi }\left(t\right)+\sqrt[3]{\left|\mathrm{{\rm Y}}\right(t\left)\right|}+{\displaystyle \frac{d\mathbb{W}\left(t\right)}{dt}},\hfill \\ {}^{\mathbf{ABC}}\mathbb{D}_{0+}^{2/3}\left(\mathrm{{\rm Y}}\left(t\right)+tanh\left(\mathrm{{\rm Y}}\right(t\left)\right)\right)=-\mathrm{{\rm Y}}\left(t\right)+\sqrt[3]{\left|\mathrm{\Psi }\right(t\left)\right|}+{\displaystyle \frac{d\mathbb{W}\left(t\right)}{dt}},\hfill \\ \mathrm{\Psi }\left(0\right)=2,\mathrm{{\rm Y}}\left(0\right)=1,\hfill \end{array}\right.$$

(24)

where $\mathbb{W}\left(t\right)$ are independent scalar Wiener processes.

We identify the system components as

$${\mathbf{A}}_1={\mathbf{A}}_2=-1,g_1(t,\mathrm{\Psi })=tanh\left(\mathrm{\Psi }\right),g_2(t,\mathrm{{\rm Y}})=tanh\left(\mathrm{{\rm Y}}\right),$$

$${\mathsf{F}}_1(t,\mathrm{{\rm Y}})=\sqrt[3]{\left|\mathrm{{\rm Y}}\right|},{\mathsf{F}}_2(t,\mathrm{\Psi })=\sqrt[3]{\left|\mathrm{\Psi }\right|},{\sigma }_1(t,\mathrm{\Psi })={\sigma }_2(t,\mathrm{{\rm Y}})=1.$$

The mild solution of system (24) is given by

$$\begin{array}{cc}\hfill \mathrm{\Psi }\left(t\right)+tanh\left(\mathrm{\Psi }\right(t\left)\right)& ={\mathbf{E}}_{2/3}\left(t^{2/3}{\mathbf{A}}_1\right)\left[2+tanh\left(2\right)\right]-{\int }_0^t{\mathbf{A}}_1{(t-\delta )}^{-1/3}\hfill \\ \hfill & \times {\mathbf{E}}_{2/3,2/3}\left({(t-\delta )}^{2/3}{\mathbf{A}}_1\right)tanh\left(\mathrm{\Psi }\left(\delta \right)\right)d\delta \hfill \\ \hfill & +{\int }_0^t{(t-\delta )}^{-1/3}{\mathbf{E}}_{2/3,2/3}\left({(t-\delta )}^{2/3}{\mathbf{A}}_1\right)\sqrt[3]{\left|\mathrm{{\rm Y}}\right(\delta \left)\right|}d\delta \hfill \\ \hfill & +{\int }_0^t{(t-\delta )}^{-1/3}{\mathbf{E}}_{2/3,2/3}\left({(t-\delta )}^{2/3}{\mathbf{A}}_1\right)d\mathbb{W}\left(\delta \right),\hfill \\ \hfill \mathrm{{\rm Y}}\left(t\right)+tanh\left(\mathrm{{\rm Y}}\right(t\left)\right)& ={\mathbf{E}}_{2/3}\left(t^{2/3}{\mathbf{A}}_2\right)\left[1+tanh\left(1\right)\right]-{\int }_0^t{\mathbf{A}}_2{(t-\delta )}^{-1/3}\hfill \\ \hfill & \times {\mathbf{E}}_{2/3,2/3}\left({(t-\delta )}^{2/3}{\mathbf{A}}_2\right)tanh\left(\mathrm{{\rm Y}}\left(\delta \right)\right)d\delta \hfill \\ \hfill & +{\int }_0^t{(t-\delta )}^{-1/3}{\mathbf{E}}_{2/3,2/3}\left({(t-\delta )}^{2/3}{\mathbf{A}}_2\right)\sqrt[3]{\left|\mathrm{\Psi }\right(\delta \left)\right|}d\delta \hfill \\ \hfill & +{\int }_0^t{(t-\delta )}^{-1/3}{\mathbf{E}}_{2/3,2/3}\left({(t-\delta )}^{2/3}{\mathbf{A}}_2\right)d\mathbb{W}\left(\delta \right).\hfill \end{array}$$

The nonlinear functions $g_1,g_2$ and ${\mathsf{F}}_1,{\mathsf{F}}_2$ satisfy CTC. They are measurable in t, continuous in state variables, and satisfy growth conditions via

$$|tanh\left(\mathrm{\Psi }\right)|\le 1,\left|\sqrt[3]{\left|x\right|}\right|\le C(1+|x\left|\right),$$

ensuring non-Lipschitz but controlled growth.

The nonlinear integral equation

$$\nu \left(t\right)=\beta {\int }_0^t\left(\nu \left(\delta \right)\nu \right)d\delta +\nu \left(0\right),$$

admits a global solution by Peano-type existence theorems, as $\nu \left(\nu \right)=C(1+|\nu \left|\right)$ is continuous and sublinear.

All hypotheses H1–H5 in Theorem (3) are thus satisfied, and the system admits a unique mild solution on $\left[{\displaystyle \frac{1}{2}},1\right]$. The linear operators ${\mathbf{A}}_1={\mathbf{A}}_2=-1$ are negative definite. The nonlinearities satisfy growth conditions allowing for integral bounds in the stability estimate. The Lipschitz constants of ${\sigma }_i$ are bounded and continuous, i.e.,

$${\mathbb{E}\parallel \mathrm{\Psi }\left(t\right)\parallel }^2+\mathbb{E}{\parallel \mathrm{{\rm Y}}\left(t\right)\parallel }^2\le C{\mathbf{E}}_{2/3}(-\lambda t^{2/3}),$$

for constants $C,\lambda >0$. Hence, by the stability result in Theorem (4), the solution $\left(\mathrm{\Psi }\right(t),\mathrm{{\rm Y}}(t\left)\right)$ is mean-square Mittag-Leffler stable. This guarantees long-term boundedness under stochastic perturbations, confirming the robustness of the solution.

The nonlinear coupled system displays more complex dynamics, with Figure 3 and Figure 4 for $\mathrm{\Psi }$ and $\mathrm{{\rm Y}}$ trajectories showing irregular decay patterns influenced by tanh and cube root terms. Unlike the linear case, the paths exhibit intermittent plateaus and wider dispersion due to noise, particularly for $\mathrm{{\rm Y}}$. The fractional-order memory effects are visible in the persistent correlations between states, while the ABC derivative ensures Mittag-Leffler-type stability. The plots illustrate how nonlinearities introduce richer behavior while maintaining overall stochastic stability.

Figure 3. Three-dimensional plot for dynamics of $\mathrm{\Psi }\left(t\right)$.

Figure 4. Three-dimensional plot for dynamics of $\mathrm{{\rm Y}}\left(t\right)$.

Example 3. 

Consider the following nonlinear fractional hybrid stochastic coupled system with $\nu ={\displaystyle \frac{4}{5}}$:

$$\left\{\begin{array}{c}{}^{\mathbf{ABC}}\mathbb{D}_{0+}^{4/5}\left(\mathrm{\Psi }\left(t\right)+0.4\mathrm{\Psi }\left(t\right)cos\left(\mathrm{\Psi }\right(t\left)\right)\right)={\mathbf{A}}_1\left(r\left(t\right)\right)\mathrm{\Psi }\left(t\right)+B_{1}tanh\left(\mathrm{{\rm Y}}\left(t\right)\right)+{\sigma }_1\left(r\left(t\right)\right){\displaystyle \frac{d{\mathbb{W}}_1\left(t\right)}{dt}},\hfill \\ {}^{\mathbf{ABC}}\mathbb{D}_{0+}^{4/5}\left(\mathrm{{\rm Y}}\left(t\right)+0.3\mathrm{{\rm Y}}\left(t\right)sin\left(\mathrm{{\rm Y}}\right(t\left)\right)\right)={\mathbf{A}}_2\left(r\left(t\right)\right)\mathrm{{\rm Y}}\left(t\right)+B_2\mathrm{\Psi }\left(t\right)e^{-{\mathrm{\Psi }}^2\left(t\right)}+{\sigma }_2\left(r\left(t\right)\right){\displaystyle \frac{d{\mathbb{W}}_2\left(t\right)}{dt}},\hfill \\ \mathrm{\Psi }\left(0\right)=2.0,\mathrm{{\rm Y}}\left(0\right)=1.0,\hfill \end{array}\right.$$

where $r\left(t\right)$ is a continuous-time Markov chain taking values in $\{1,2\}$ the with generator matrix

$$Q=\left(\begin{array}{cc}-0.6& 0.6\\ 0.8& -0.8\end{array}\right).$$

The mode-dependent parameters are

$$\begin{array}{cc}\hfill & {\mathbf{A}}_1\left(1\right)=-3.0,{\mathbf{A}}_2\left(1\right)=-2.5,{\sigma }_1\left(1\right)=0.4,{\sigma }_2\left(1\right)=0.3,\hfill \\ \hfill & {\mathbf{A}}_1\left(2\right)=-1.2,{\mathbf{A}}_2\left(2\right)=-1.0,{\sigma }_1\left(2\right)=0.9,{\sigma }_2\left(2\right)=0.7,\hfill \end{array}$$

with coupling coefficients $B_1=1.2$, $B_2=0.9$. The mild solutions of the system are given by

$$\begin{array}{cc}\hfill \mathrm{\Psi }\left(t\right)& ={\mathbf{E}}_{4/5}\left(t^{4/5}{\mathbf{A}}_1\left(r\left(t\right)\right)\right)[2.0+0.4·2.0cos\left(2.0\right)]-0.4\mathrm{\Psi }\left(t\right)cos\left(\mathrm{\Psi }\left(t\right)\right)\hfill \\ \hfill & +{\displaystyle \frac{4/5}{B(4/5)\Gamma (4/5)}}{\int }_0^t{(t-\delta )}^{-1/5}{\mathbf{E}}_{4/5,4/5}\left({(t-\delta )}^{4/5}{\mathbf{A}}_1\left(r\left(\delta \right)\right)\right)\hfill \\ \hfill & \times \left[B_{1}tanh\left(\mathrm{{\rm Y}}\left(\delta \right)\right)+{\sigma }_1\left(r\left(\delta \right)\right){\dot{\mathbb{W}}}_1\left(\delta \right)\right]d\delta ,\hfill \end{array}$$

$$\begin{array}{cc}\hfill \mathrm{{\rm Y}}\left(t\right)& ={\mathbf{E}}_{4/5}\left(t^{4/5}{\mathbf{A}}_2\left(r\left(t\right)\right)\right)[1.0+0.3·1.0sin\left(1.0\right)]-0.3\mathrm{{\rm Y}}\left(t\right)sin\left(\mathrm{{\rm Y}}\left(t\right)\right)\hfill \\ \hfill & +{\displaystyle \frac{4/5}{B(4/5)\Gamma (4/5)}}{\int }_0^t{(t-\delta )}^{-1/5}{\mathbf{E}}_{4/5,4/5}\left({(t-\delta )}^{4/5}{\mathbf{A}}_2\left(r\left(\delta \right)\right)\right)\hfill \\ \hfill & \times \left[B_2\mathrm{\Psi }\left(\delta \right)e^{-{\mathrm{\Psi }}^2\left(\delta \right)}+{\sigma }_2\left(r\left(\delta \right)\right){\dot{\mathbb{W}}}_2\left(\delta \right)\right]d\delta .\hfill \end{array}$$

$$\begin{array}{cc}\hfill \parallel g_1(t,\mathrm{\Psi })-g_1(t,\tilde{\mathrm{\Psi }})\parallel & =0.4\parallel \mathrm{\Psi }cos\mathrm{\Psi }-\tilde{\mathrm{\Psi }}cos\tilde{\mathrm{\Psi }}\parallel \le 0.8\parallel \mathrm{\Psi }-\tilde{\mathrm{\Psi }}\parallel ,\hfill \\ \hfill \parallel g_2(t,\mathrm{{\rm Y}})-g_2(t,\widetilde{\mathrm{{\rm Y}}})\parallel & =0.3\parallel \mathrm{{\rm Y}}sin\mathrm{{\rm Y}}-\widetilde{\mathrm{{\rm Y}}}sin\widetilde{\mathrm{{\rm Y}}}\parallel \le 0.6\parallel \mathrm{{\rm Y}}-\widetilde{\mathrm{{\rm Y}}}\parallel ,\hfill \end{array}$$

with $0<{\mathfrak{L}}_1=0.8<{\displaystyle \frac{1}{2}}$, $0<{\mathfrak{L}}_2=0.6<{\displaystyle \frac{1}{2}}$.

$$\begin{array}{cc}\hfill \parallel F_1(t,\mathrm{{\rm Y}})-F_1(t,\widetilde{\mathrm{{\rm Y}}})\parallel & =1.2\parallel tanh\mathrm{{\rm Y}}-tanh\widetilde{\mathrm{{\rm Y}}}\parallel \le 1.2\parallel \mathrm{{\rm Y}}-\widetilde{\mathrm{{\rm Y}}}\parallel ,\hfill \\ \hfill \parallel F_2(t,\mathrm{\Psi })-F_2(t,\tilde{\mathrm{\Psi }})\parallel & =0.9\parallel \mathrm{\Psi }{e}^{-{\mathrm{\Psi }}^2}-\tilde{\mathrm{\Psi }}{e}^{-{\tilde{\mathrm{\Psi }}}^2}\parallel \le 0.9\parallel \mathrm{\Psi }-\tilde{\mathrm{\Psi }}\parallel ,\hfill \\ \hfill \parallel {\sigma }_i(t,\mathrm{\Psi })-{\sigma }_i(t,\mathrm{{\rm Y}})\parallel & \le {\mathfrak{L}}_{\sigma i}\parallel \mathrm{\Psi }-\mathrm{{\rm Y}}\parallel ,\hfill \end{array}$$

satisfying the Lipschitz conditions.

$$\begin{array}{cc}\hfill \parallel F_1{(t,0)\parallel }^2+{\parallel {\sigma }_1(t,0)\parallel }^2& \le (0+0.4^2)=0.16<\infty ,\hfill \\ \hfill \parallel F_2{(t,0)\parallel }^2+{\parallel {\sigma }_2(t,0)\parallel }^2& \le (0+0.3^2)=0.09<\infty .\hfill \end{array}$$

The matrices ${\mathbf{A}}_1$ and ${\mathbf{A}}_2$ are negative definite in both modes:

$$\begin{array}{cc}\hfill ⟨{\mathbf{A}}_1\left(1\right)x,x⟩& \le -{3.0\parallel x\parallel }^2,⟨{\mathbf{A}}_2\left(1\right)x,x⟩\le -2.5{\parallel x\parallel }^2,\hfill \\ \hfill ⟨{\mathbf{A}}_1\left(2\right)x,x⟩& \le -{1.2\parallel x\parallel }^2,⟨{\mathbf{A}}_2\left(2\right)x,x⟩\le -1.0{\parallel x\parallel }^2.\hfill \end{array}$$

The initial conditions are square-integrable: ${\mathbb{E}\parallel 2.0\parallel }^2<\infty$, ${\mathbb{E}\parallel 1.0\parallel }^2<\infty$. By Theorem 4, the mild solution $\left(\mathrm{\Psi }\right(t),\mathrm{{\rm Y}}(t\left)\right)$ is mean-square Mittag-Leffler-stable:

$${\mathbb{E}\parallel \mathrm{\Psi }\left(t\right)\parallel }^2+{\mathbb{E}\parallel \mathrm{{\rm Y}}\left(t\right)\parallel }^2\le C{\mathbf{E}}_{4/5}(-\lambda t^{4/5}){(\mathbb{E}\parallel 2.0\parallel }^2+{\mathbb{E}\parallel 1.0\parallel }^2),\forall t\in [0,t].$$

All functions $g_i$, $F_i$, and ${\sigma }_i$ are measurable in t and continuous in the state variables. The drift terms satisfy

$$\parallel F_1(t,\mathrm{{\rm Y}})\parallel \le 1.2,\parallel F_2(t,\mathrm{\Psi })\parallel \le 0.9e^{-1/2}\approx 0.55,$$

ensuring controlled growth. The neutral terms satisfy $\parallel g_i(t,x)\parallel \le 0.4\parallel x\parallel$ and $\parallel g_i(t,x)\parallel \le 0.3\parallel x\parallel$, respectively. The integral equation

$$\nu \left(t\right)=\beta {\int }_0^{t}G(\delta ,\nu \left(\delta \right))d\delta +\nu \left(0\right),$$

admits a global solution by Peano’s existence theorem, as the nonlinearities are continuous and sublinear. Therefore, by Theorem 3, the system admits a unique mild solution on $\left[{\displaystyle \frac{1}{2}},1\right]$. In Figure 5 and Figure 6, we have presented three-dimensional plots for the concerned example.

Figure 5. Three-dimensional plot. for dynamics of $\mathrm{\Psi }\left(t\right)$.

Figure 6. Three-dimensional plot. for dynamics of $\mathrm{{\rm Y}}\left(t\right)$.

6. Numerical Scheme: Adams–Bashforth Method for the ABC System

Consider the coupled stochastic fractional system in the Atangana–Baleanu–Caputo (ABC) sense, written in mild form:

$$\begin{array}{c}\hfill X\left(t\right)=X_0+{\displaystyle \frac{1-\nu }{C\left(\nu \right)}}F(t,X\left(t\right))+{\displaystyle \frac{\nu }{C\left(\nu \right)\Gamma \left(\nu \right)}}{\int }_0^t{(t-s)}^{\nu -1}F(\delta ,X\left(\delta \right))ds\\ & +{\displaystyle \frac{\nu }{C\left(\nu \right)\Gamma \left(\nu \right)}}{\int }_0^t{(t-s)}^{\nu -1}\mathrm{\Sigma }(\delta ,X\left(\delta \right))dW\left(\delta \right),\hfill \end{array}$$

(25)

where $X={(\mathrm{\Psi },Y)}^{\top }$, F is the drift vector, $\mathrm{\Sigma }$ is the diffusion vector, $0.5<\nu <1$, and $C\left(\nu \right)$ is the ABC normalization constant.

Let the uniform grid be $t_n=n∆t$ for $n=0,1,\dots ,N$ with $∆t=T/N$. Define

$$F_j:=F(t_j,X_j),{\mathrm{\Sigma }}_j:=\mathrm{\Sigma }(t_j,X_j),∆W_j=W\left(t_{j+1}\right)-W\left(t_j\right).$$

(26)

The kernel weights for the fractional integral at $t_{n+1}$ are

$$w_{n+1,j}={(t_{n+1}-t_j)}^{\nu -1},j=0,1,\dots ,n.$$

(27)

The fractional Adams–Bashforth (order–2) predictor is given by

$$X_{n+1}^{\mathrm{pred}}=X_0+{\displaystyle \frac{1-\nu }{C\left(\nu \right)}}{F}_n+{\displaystyle \frac{\nu }{C\left(\nu \right)\Gamma \left(\nu \right)}}\sum _{j=0}^n{w}_{n+1,j}{\tilde{F}}_j∆t+{\displaystyle \frac{\nu }{C\left(\nu \right)\Gamma \left(\nu \right)}}\sum _{j=0}^n{w}_{n+1,j}{\mathrm{\Sigma }}_j∆W_j,$$

(28)

where the modified values ${\tilde{F}}_j$ are defined by

$${\tilde{F}}_0=F_0,{\tilde{F}}_j=\frac{3}{2}{F}_j-\frac{1}{2}{F}_{j-1},j\ge 1.$$

(29)

To improve stability and accuracy, a one–step Adams–Moulton corrector is applied using the predicted value:

$$\begin{array}{c}\hfill X_{n+1}=X_0+{\displaystyle \frac{1-\nu }{C\left(\nu \right)}}{F}_{n+1}^{\mathrm{pred}}+{\displaystyle \frac{\nu }{C\left(\nu \right)\Gamma \left(\nu \right)}}\left[\sum _{j=0}^n{w}_{n+1,j}{F}_j∆t+\frac{1}{2}{w}_{n+1,n}{F}_{n+1}^{\mathrm{pred}}∆t\right]\\ & +{\displaystyle \frac{\nu }{C\left(\nu \right)\Gamma \left(\nu \right)}}\left[\sum _{j=0}^n{w}_{n+1,j}{\mathrm{\Sigma }}_j∆W_j+\frac{1}{2}{w}_{n+1,n}{\mathrm{\Sigma }}_{n+1}^{\mathrm{pred}}∆W_n\right].\hfill \end{array}$$

(30)

Figure 7 shows the trajectories of $\mathrm{\Psi }\left(t\right)$, where the mean path (blue) remains stable with a narrow confidence band despite random fluctuations. Figure 8 presents the dynamics of $\mathrm{{\rm Y}}\left(t\right)$, which exhibits higher variability but still converges around its mean (red). Both results confirm that the coupled system remains stable under stochastic and fractional effects.

Figure 7. Three-dimensional plot for dynamics of $\mathrm{\Psi }\left(t\right)$.

Figure 8. Three-dimensional plot for dynamics of $\mathrm{{\rm Y}}\left(t\right)$.

7. Conclusions

In this article, we have proposed and investigated a novel class of nonlinear FSNHDE coupled systems based on the ABC derivative. The model accurately incorporates nonlocal memory effects, stochastic disturbances, and intercoupled subsystem interactions within a comprehensive framework. Based on the use of the non-singular Mittag-Leffler kernel, the ABC operator provides more accurate modeling of hereditary dynamics than traditional fractional models. The coupled structure also enhances modeling capability by capturing mutual influence and feedback among interdependent subsystems, such as neuron–glia interactions in neuroscience, temperature–humidity feedback cycles in climate models, and bidirectional actuator–sensor systems in engineering controls. By applying rigorous analysis, we proved the existence and uniqueness of mild solutions under Carathéodory-type assumptions, without imposing global Lipschitz continuity. The Picard successive approximation approach, in conjunction with integral kernel estimation, provides such systems with a strong theoretical basis. In addition to modeling flexibility, hybrid coupled systems offer significant benefits for real-world applications by allowing the concurrent representation of multiple dynamic agents that exhibit noise, memory, and switching behavior. The examples demonstrate that the theoretical findings are not only mathematically valid but also applicable to intricate dynamical systems found in bioengineering, smart grid control, epidemiological modeling, and viscoelastic material dynamics.

In summary, this study contributes to the growing body of literature on fractional stochastic dynamics and provides a flexible platform for further research in stability analysis, optimal control, numerical simulations, and delay-induced phenomena in practical hybrid systems.