Coefficient Estimates, the Fekete–Szegö Inequality, and Hankel Determinants for Universally Prestarlike Functions Defined by Fractional Derivative in a Shell-Shaped Region

Abstract

In this paper, we introduce and investigate a new subclass ${\left({\mathcal{R}}_{\varsigma }^u\right)}^{\mathfrak{g}}\left(\varphi \right)$ of universally prestarlike generalized functions of order $\varsigma$, where $\varsigma \le 1$, associated with a shell-shaped region defined by $\Lambda =\mathrm{C}\setminus [1,\infty )$ for the present investigation, by utilizing the Srivastava–Owa fractional derivative of order $\delta$. Coefficient inequalities for $|{\mathfrak{a}}_2|$ and $|{\mathfrak{a}}_3|$ for functions belonging to the newly introduced class are obtained. Additionally, the Fekete–Szegö inequality is investigated for this class of functions. In order to enhance the coefficient studies for this class, the second Hankel determinant is also evaluated.

1. Introduction

The study of analytic functions, particularly those exhibiting geometric properties such as univalence and starlikeness, continues to attract significant attention in complex analysis and geometric function theory. Earlier, starlikeness and convexity were intensely investigated using integral transforms and the duality principle, as seen, for example, in [1,2]. Recent developments have shown increasing interest in subclasses of such functions defined through fractional calculus operators, which offer enhanced flexibility in analyzing their structural characteristics. Among these, universally prestarlike functions represent a robust class with intriguing geometric behavior and have been the subject of active research in recent years. In addition, investigations of analytic functions defined in particularly shaped domains are presented in recent publications. A new subclass of bi-univalent functions related to the leaf-like domain is presented and examined with regard to coefficient problems in [3], while the Fekete–Szegö inequality is examined in the context of the leaf-like domain using the Hohlov operator in [4] and using the Sălăgean-difference operator in [5]. A modified Caputo’s fractional operator is involved in the definition of two new Ozaki type subclasses of bi-close-to-convex functions and bi-concave functions associated with three-leaf functions in [6] for which coefficient estimates are investigated, including the Fekete–Szegö inequality. The Fekete–Szegö problem as well as the second-order Hankel determinant are investigated for a new class of bi-univalent functions in a three-leaf domain in [7]. A class of functions associated with a Limacon domain is introduced and investigated in [8]. Since no such study has been proposed before, we have chosen to carry out similar investigations using the prolific Srivastava–Owa fractional derivative of order $\delta$ in a shell-shaped domain, in light of the numerous recent studies that have used various types of fractional derivatives considering geometrically significant domains as shown above. It can also be seen that recently published research has established coefficient problems pertaining to new classes of analytic functions defined in domains of significant geometric importance. Therefore, the current study tackles this topic that is of relevance to scholars today.

This paper is motivated by the need to explore deeper coefficient properties for certain function classes under the influence of fractional differentiation, particularly within geometrically meaningful regions such as shell-shaped domains. The shell-shaped region, defined between two concentric disks, provides a rich setting for extending classical results and understanding function behavior beyond the unit disk.

The main objective of the present study is to establish new results concerning coefficient bounds, Fekete–Szegö inequalities, and Hankel determinants for a newly introduced class of universally prestarlike functions defined by the Srivastava–Owa fractional derivative, which is defined in the next section. Several existing results from the literature appear as special cases of the more general outcomes presented here and are explicitly indicated through corollaries and remarks.

This paper is structured as follows: Section 2 introduces the new class under investigation, the fundamental definitions, and the essential preliminary results. In Section 3, precise coefficient estimates for the initial terms of the functions are established, along with a generalization of the classical Fekete–Szegö inequality. Section 4 focuses on deriving bounds for the second-order Hankel determinants. Finally, the paper concludes with remarks on the significance of the results.

Let $\mathcal{A}(\Delta )$ represent the class of analytic functions on a domain $\Delta$. When $\Delta$ includes the origin, we define ${\mathcal{A}}_0(\Delta )$ as the collection of functions $\mathfrak{f}\in \mathcal{A}(\Delta )$ satisfying $\mathfrak{f}\left(0\right)=1$. We also define ${\mathcal{A}}_1(\Delta ):=\left\{\zeta \mathfrak{f}:\mathfrak{f}\in {\mathcal{A}}_0(\Delta )\right\}$. In the particular case where $\Delta$ is the unit disc $\mathfrak{U}:=\left\{\zeta \in \mathbb{C}:\left|\zeta \right|<1\right\},$ we abbreviate the notations as $\mathcal{A}$, ${\mathcal{A}}_0$, and ${\mathcal{A}}_1$, respectively.

Consider $\Xi$ to denote the class of functions $\omega \left(\zeta \right)$ that are analytic in the domain $\mathfrak{U}$, defined by

$$\Xi =\{\omega \in \mathcal{A}:\omega (0)=0\mathrm{and}|\omega \left(\zeta \right)|<1,\zeta \in \mathfrak{U}\}.$$

If $\mathfrak{f}\left(\zeta \right)$ and $\mathfrak{g}\left(\zeta \right)$ are functions in $\mathcal{A}$, we say that $\mathfrak{f}$ is subordinate to $\mathfrak{g}$, if there exists a function $\omega \left(\zeta \right)\in \Omega$, such that $\mathfrak{f}\left(\zeta \right)=\mathfrak{g}\left(\omega \right(\zeta \left)\right)$ for all $(\zeta \in \mathfrak{U})$, written as $\mathfrak{f}\left(\mathfrak{z}\right)\prec \mathfrak{g}\left(\zeta \right)$. It is well known that if $\mathfrak{g}$ is univalent in $\mathfrak{U}$, then $\mathfrak{f}\left(\mathfrak{U}\right)\subset \mathfrak{g}\left(\mathfrak{U}\right)$.

Paprocki and Sokół [9] introduced the concept of shell-like domains, showing that the function

$$\varphi \left(\zeta \right):=\zeta +\sqrt{{\zeta }^2+1},$$

maps the unit disc $\mathfrak{U}$ onto a shell-shaped region in the right half-plane. This function is analytic and univalent within $\mathfrak{U}$.

The class of starlike function of order $\varsigma$ (where $0\le \varsigma <1)$ is defined as follows:

$${\mathcal{S}}_{\varsigma }=\left\{\mathfrak{f}\in {\mathcal{A}}_1:Re\left({\displaystyle \frac{\zeta {\mathfrak{f}}^{\prime }\left(\zeta \right)}{\mathfrak{f}\left(\zeta \right)}}\right)>\varsigma ,\zeta \in \mathfrak{U}\right\}.$$

In 1992, Ma Minda [10] introduced a creative approach to constructing subclasses of starlike and convex functions by leveraging the concept of subordination. Another interesting approach on subordination is seen in [11]. Subordination also plays a central role in the investigation presented in [8].

Definition 1

([12]). Let ${\mathcal{S}}^{*}\left(\varphi \right)$ represent the class of analytic functions $\mathfrak{f}$, defined on the unit disc $\mathfrak{U}$, which are normalized such that $\mathfrak{f}\left(0\right)=0$ and ${\mathfrak{f}}^{\prime }\left(0\right)=1$. Additionally, these functions must satisfy the following condition:

$${\displaystyle \frac{\zeta {\mathfrak{f}}^{\prime }\left(\zeta \right)}{\mathfrak{f}\left(\zeta \right)}}\prec \varphi \left(\zeta \right):=\zeta +\sqrt{1+{\zeta }^2},\zeta \in \mathfrak{U}$$

(1)

where the square root is taken with the branch satisfying $\varphi \left(0\right)=1$, and ≺ denotes subordination.

It is evident from Equation (1) in Definition 1 that the image set $\varphi \left(\mathfrak{U}\right)$ lies entirely within the right half-plane, but it is not starlike with respect to the origin.

Recently, Raina and Sokol [12] investigated and established several coefficient inequalities for the class ${\mathcal{S}}^{*}\left(\varphi \right)$. These results were later extended by Sokol and Thomas [13] to the class $\mathcal{C}\left(\varphi \right)$, utilizing the Alexander relation, which states that a function $\mathfrak{f}\in \mathcal{C}\left(\varphi \right)$ if and only if $\zeta {\mathfrak{f}}^{\prime }\left(\zeta \right)\in {\mathcal{S}}^{*}\left(\varphi \right)$. Additionally, the Fekete–Szegö inequality for functions in ${\mathcal{S}}^{*}\left(\varphi \right)$ was also established.

The convolution (also known as the Hadamard product) of two analytic functions $\mathfrak{f}\left(\zeta \right)$ and $\mathfrak{g}\left(\zeta \right)$, where $\mathfrak{g}\left(\zeta \right)={\sum }_{\mathfrak{j}=0}^{\infty }{\mathfrak{g}}_{\mathfrak{j}}{\zeta }^{\mathfrak{j}}$ and $\mathfrak{f}\left(\zeta \right)={\sum }_{\mathfrak{j}=0}^{\infty }{\mathfrak{a}}_{\mathfrak{j}}{\zeta }^{\mathfrak{j}}$, is defined as follows:

$$(\mathfrak{f}\ast \mathfrak{g})\left(\zeta \right)=\mathfrak{f}\left(\zeta \right)\ast \mathfrak{g}\left(\zeta \right):=\sum _{\mathfrak{j}=0}^{\infty }{\mathfrak{a}}_{\mathfrak{j}}{\mathfrak{g}}_{\mathfrak{j}}{\zeta }^{\mathfrak{j}},\zeta \in \mathfrak{U}.$$

According to S. Ruscheweyh [14], for $\mathfrak{f}\in {\mathcal{A}}_1$, let ${\mathcal{R}}_{\varsigma }$ denote the class of all prestarlike functions of order $\varsigma$, (where $\varsigma \le 1)$ in $\mathfrak{U}$, which satisfy the following conditions:

$$\left\{\begin{array}{cc}{\mathfrak{K}}_{2-2\varsigma }\ast \mathfrak{f}\in {\mathcal{S}}_{\varsigma },\hfill & \varsigma <1\hfill \\ Re\left({\displaystyle \frac{\mathfrak{f}\left(\zeta \right)}{\zeta }}\right)>{\displaystyle \frac{1}{2}},\hfill & \varsigma =1,\hfill \end{array}\right.$$

$\zeta \in \mathfrak{U}$, with

$${\mathfrak{K}}_{2-2\varsigma }\left(\zeta \right)={\displaystyle \frac{\zeta }{{(1-\zeta )}^{2-2\varsigma }}}=\zeta +\sum _{\mathfrak{j}=2}^{\infty }\Theta (\varsigma ,\mathfrak{j}){\zeta }^{\mathfrak{j}},$$

commonly known as the extremal function within ${\mathcal{S}}_{\varsigma }$, and

$$\Theta (\varsigma ,\mathfrak{j})={\displaystyle \frac{{\prod }_{\iota =2}^{\mathfrak{j}}(\iota -2\varsigma )}{(\mathfrak{j}-1)!}},\left(\mathfrak{j}\in \mathbb{N}\setminus \left\{1\right\},\mathbb{N}:=\{1,2,3,\cdots \}\right).$$

It is important to note that $\Theta (\varsigma ,\mathfrak{j})$ is a decreasing function of $\varsigma$, with the following limits:

$$\underset{\mathfrak{j}\to \infty }{lim}\Theta (\varsigma ,\mathfrak{j})=\left\{\begin{array}{cc}\infty \hfill & \mathrm{if}\varsigma <{\displaystyle \frac{1}{2}},\hfill \\ 1\hfill & \mathrm{if}\varsigma ={\displaystyle \frac{1}{2}},\hfill \\ 0\hfill & \mathrm{if}\varsigma >{\displaystyle \frac{1}{2}}.\hfill \end{array}\right.$$

When studying prestarlike functions and convolutions, the following notation is useful:

$$\left({\mathfrak{D}}^{\mathfrak{j}}\mathfrak{f}\right)\left(\zeta \right)=\left({\mathfrak{K}}_{\mathfrak{j}}\ast \mathfrak{f}\right)\left(\zeta \right),{\mathfrak{K}}_{\mathfrak{j}}\left(\zeta \right)={\displaystyle \frac{\zeta }{{(1-\zeta )}^{\mathfrak{j}}}},$$

where $\mathfrak{j}\in {\mathbb{N}}_0=\{0,1,2,3,\dots \}$. Therefore, we also have

$${\mathfrak{D}}^{\mathfrak{j}+1}\mathfrak{f}={\displaystyle \frac{\zeta }{\mathfrak{j}!}}{\left({\zeta }^{\mathfrak{j}-1}\mathfrak{f}\right)}^{\left(\mathfrak{j}\right)},\mathrm{for}\mathfrak{j}\in {\mathbb{N}}_0.$$

Using this operator, we find that a function $\mathfrak{f}\in {\mathcal{A}}_1$ is prestarlike of order $\varsigma \le 1$ if and only if the following condition holds:

$${\displaystyle \frac{{\mathfrak{D}}^{3-2\varsigma }\mathfrak{f}}{{\mathfrak{D}}^{2-2\varsigma }\mathfrak{f}}}\in \mathcal{P},$$

where $\mathcal{P}$ is the class defined as

$$\mathcal{P}=\left\{\mathfrak{f}\in {\mathcal{A}}_0:Re\left(\mathfrak{f}\left(\zeta \right)\right)>{\displaystyle \frac{1}{2}},\zeta \in \mathfrak{U}\right\}.$$

Equivalently, using the Herglotz formula, we have the following characterization:

$$\mathfrak{f}\in \mathcal{P}\iff \mathfrak{f}\left(\zeta \right)={\int }_0^{2\pi }{\displaystyle \frac{d\mu \left(\tau \right)}{1-e^{-i\tau }\zeta }},$$

where $\mu$ is a probability measure on $[0,2\pi ]$. Note that for any function $\mathfrak{f}\in T$ the restriction of $\mathfrak{f}$ to $\mathfrak{U}$ is in $\mathcal{P}$.

Consider T to denote the set of such functions $\mathfrak{f}$, that are analytic in the slit domain $\Lambda =\mathrm{C}\setminus [1,\infty )$ such that

$$\begin{array}{c}\hfill T:=\left\{\mathfrak{f}:\mathfrak{f}\left(\zeta \right)=\sum _{\mathfrak{j}=0}^{\infty }{\mathfrak{a}}_{\mathfrak{j}}{\zeta }^{\mathfrak{j}}={\int }_0^1{\displaystyle \frac{d\mu \left(\tau \right)}{1-t}}\zeta \right\}.\end{array}$$

The concept of prestarlike functions of order $\varsigma$ has recently been generalized beyond the unit disc $\mathfrak{U}$, extending to other discs and half-planes that include the origin (see [15,16,17]). One such disc, denoted by ${\Delta }_{\gamma ,\rho }$, is defined as

$${\Delta }_{\gamma ,\rho }=\left\{{\varpi }_{\gamma ,\rho }\left(\zeta \right):\zeta \in \mathfrak{U}\right\},$$

where $\gamma \in \mathbb{C}\setminus \left\{0\right\}$ and $\rho \in [0,1]$ are two unique parameters, and ${\varpi }_{\gamma ,\rho }\left(\zeta \right)={\displaystyle \frac{\gamma \zeta }{1-{\rho }_{\zeta }}}$. It is important to note that $1\notin {\Delta }_{\gamma ,\rho }$ if and only if $|\gamma +\rho |\le 1$.

Definition 2

(see [15,16,17]). A function $\mathfrak{f}\in {\mathcal{R}}_{\varsigma }\left({\Delta }_{\gamma ,\rho }\right)$ is said to be prestarlike of order ς within the parameters γ and ρ. For $\varsigma \le 1$, and for an admissible pair $(\gamma ,\rho )$, the following class is defined as

$${\mathcal{R}}_{\varsigma }\left({\Delta }_{\gamma ,\rho }\right)=\left\{\mathfrak{f}\in {\mathcal{A}}_1\left({\Delta }_{\gamma ,\rho }\right):{\displaystyle \frac{1}{\gamma }}\mathfrak{f}\left({\varpi }_{\gamma ,\rho }\left(\zeta \right)\right)\in {\mathcal{R}}_{\varsigma }\right\},$$

where ${\mathcal{A}}_1\left({\Delta }_{\gamma ,\rho }\right)=\left\{\zeta \mathfrak{f}:\mathfrak{f}\in {\mathcal{A}}_0\left({\Delta }_{\gamma ,\rho }\right)\right.$, $\left.\mathfrak{f}\left(0\right)=1\right\}$.

Definition 3

([17]). Let $\varsigma \le 1$. A function $\mathfrak{f}\in {\mathcal{A}}_1(\Lambda )$ is called universally prestarlike of order ς in Λ if and only if it is prestarlike of order ς in every domain ${\varpi }_{\gamma ,\rho }$ satisfying $|\gamma +\rho |\le 1$. The set of all such functions is denoted by ${\mathcal{R}}_{\varsigma }^u$.

Definition 4

(see [17] Theorem 1.1). Let $\varsigma \le 1$ and $\mathfrak{f}\in {\mathcal{A}}_1(\Lambda )$. Then the function $\mathfrak{f}\in {\mathcal{R}}_{\varsigma }^u$ if and only if

$${\displaystyle \frac{{\mathfrak{D}}^{3-2\varsigma }\mathfrak{f}}{{\mathfrak{D}}^{2-2\varsigma }\mathfrak{f}}}\in T.$$

Remark 1

(see [17]). For $\varsigma \le 1$, the function ${\displaystyle \frac{\zeta }{1-t\zeta }}\in {\mathcal{R}}_{\varsigma }^u,\mathit{where}t\in [0,1].$

Definition 5

([17]). Let $\varphi \left(\zeta \right)$ be an analytic function with positive real part on $\mathfrak{U}$, which satisfies $\varphi \left(0\right)=1,{\varphi }^{\prime }\left(0\right)>0$ and which maps the unit disc $\mathfrak{U}$ onto a region starlike with respect to 1 and symmetric with respect to the real axis. Then the class ${\mathcal{R}}_{\alpha }^u\left(\varphi \right)$ consists of all analytic functions $\mathfrak{f}\in {\mathcal{A}}_1(\Lambda )$ satisfying

$${\displaystyle \frac{{\mathfrak{D}}^{3-2\varsigma }\mathfrak{f}}{{\mathfrak{D}}^{2-2\varsigma }\mathfrak{f}}}\prec \varphi \left(\zeta \right).$$

where ≺ denotes the subordination.

Remark 2.

In [18], Shanmugam and Mary introduced the class ${\mathcal{R}}_{\varsigma }^u(A,B)$, which is defined as the subclass ${\mathcal{R}}_{\varsigma }^u\left(\varphi \right)$, where

$$\varphi \left(\zeta \right)={\displaystyle \frac{1+A\zeta }{1+B\zeta }}(-1\le B<A\le 1)$$

For suitable choices of $A,B,\varsigma$ the class ${\mathcal{R}}_{\varsigma }^u(A,B)$ reduces to several well-known classes of functions.

Definition 6

([19]). Let $\mathfrak{f}\left(\zeta \right)$ be an analytic function defined in a simply connected region of the ζ-plane that includes the origin. The fractional derivative of order δ, where $0\le \delta <1$, is given by

$${\mathfrak{D}}_{\zeta }^{\delta }\mathfrak{f}\left(\zeta \right):={\displaystyle \frac{1}{\Gamma (1-\delta )}}{\displaystyle \frac{d}{d\zeta }}{\int }_0^{\zeta }{\displaystyle \frac{\mathfrak{f}\left(\mathfrak{z}\right)}{{(\zeta -\mathfrak{z})}^{\delta }}}d\mathfrak{z}.$$

Here, the expression ${(\zeta -\mathfrak{z})}^{\delta }$ is made single-valued by taking the branch of the logarithm such that $log(\zeta -\mathfrak{z})$ is real whenever $(\zeta -\mathfrak{z})>0$.

Based on Definition 6, Owa and Srivastava (see [20,21,22]) introduced a fractional derivative operator ${\Omega }^{\delta }:{\mathcal{A}}_1\to {\mathcal{A}}_1$, defined by

$$\left({\Omega }^{\delta }\mathfrak{f}\right)\left(\zeta \right)=\Gamma (2-\delta ){\zeta }^{\delta }{\mathfrak{D}}_{\zeta }^{\delta }\mathfrak{f}\left(\zeta \right),(\delta \ne 2,3,4,\dots ).$$

The class ${\left({\mathcal{R}}_{\varsigma }^u\right)}^{\delta }\left(\varphi \right)$ consists of all functions $\mathfrak{f}\in {\mathcal{A}}_1$ for which the fractional derivative ${\Omega }^{\delta }\mathfrak{f}$ belongs to the class $\left({\mathcal{R}}_{\varsigma }^u\right)\left(\varphi \right)$. This class can be viewed as a particular case of the more general class ${\left({\mathcal{R}}_{\varsigma }^u\right)}^{\mathfrak{g}}\left(\varphi \right)$, where $\mathfrak{g}$ is analytic in the unit disc $\mathfrak{U}$, and the convolution $\mathfrak{f}\ast \mathfrak{g}\ne 0$. The function $\mathfrak{g}$ is given by

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \mathfrak{g}\left(\zeta \right)=\zeta +\sum _{\mathfrak{j}=2}^{\infty }{\displaystyle \frac{\Gamma (\mathfrak{j}+1)\Gamma (2-\delta )}{\Gamma (\mathfrak{j}+1-\delta )}}{\zeta }^{\mathfrak{j}}=\zeta +\sum _{\mathfrak{j}=2}^{\infty }{\mathfrak{g}}_{\mathfrak{j}}{\zeta }^{\mathfrak{j}}(\zeta \in \mathfrak{U}),\end{array}\end{array}$$

(2)

with coefficients, ${\mathfrak{g}}_{\mathfrak{j}}={\displaystyle \frac{\Gamma (\mathfrak{j}+1)\Gamma (2-\delta )}{\Gamma (\mathfrak{j}+1-\delta )}},$

$${\mathfrak{g}}_{\mathfrak{2}}={\displaystyle \frac{2}{(2-\delta )}},{\mathfrak{g}}_{\mathfrak{3}}={\displaystyle \frac{6}{(2-\delta )(3-\delta )}},\mathrm{and}{\mathfrak{g}}_{\mathfrak{4}}={\displaystyle \frac{24}{(2-\delta )(3-\delta )(4-\delta )}}.$$

Definition 7

([18]). A function of the form $\mathfrak{f}\left(\zeta \right)=\zeta +{\sum }_{\mathfrak{j}=2}^{\infty }{\mathfrak{a}}_{\mathfrak{j}}{\zeta }^{\mathfrak{j}}$ is said to be in the class ${\left({\mathcal{R}}_{\varsigma }^u\right)}^{\mathfrak{g}}\left(\varphi \right)$ if and only if its convolution with $\mathfrak{g},$ that is $(\mathfrak{f}\ast \mathfrak{g})\left(\zeta \right)$, lies in the class $\left({\mathcal{R}}_{\varsigma }^u\right)\left(\varphi \right)$ such that

$$\begin{array}{c}\hfill {\displaystyle \frac{{\mathfrak{D}}^{3-2\varsigma }(\mathfrak{f}\ast \mathfrak{g})\left(\zeta \right)}{{\mathfrak{D}}^{2-2\varsigma }(\mathfrak{f}\ast \mathfrak{g})\left(\zeta \right)}}\in {\mathcal{R}}_{\varsigma }^u.\end{array}$$

In 1976 [23], Noonan and Thomas introduced a definition for the ${\mathfrak{k}}^{th}$ Hankel determinant of a function $\mathfrak{f}$, denoted by ${\mathfrak{H}}_{\mathfrak{m}}$, for integers $\mathfrak{m}\ge 1$ and $\mathfrak{k}\ge 1$, as follows:

$${\mathfrak{H}}_{\mathfrak{m}}\left(\mathfrak{k}\right)=\left|\begin{array}{ccccc}{\mathfrak{a}}_{\mathfrak{k}}& {\mathfrak{a}}_{\mathfrak{k}+1}& {\mathfrak{a}}_{\mathfrak{k}+2}& \dots & {\mathfrak{a}}_{\mathfrak{k}+\mathfrak{m}-1}\\ {\mathfrak{a}}_{\mathfrak{k}+1}& {\mathfrak{a}}_{\mathfrak{k}+2}& {\mathfrak{a}}_{\mathfrak{k}+3}& \dots & {\mathfrak{a}}_{\mathfrak{k}+\mathfrak{m}}\\ ·& ·& ·& \dots & ·\\ :& :& :& \dots & :\\ {\mathfrak{a}}_{\mathfrak{m}+\mathfrak{k}-1}& {\mathfrak{a}}_{\mathfrak{m}+\mathfrak{k}}& {\mathfrak{a}}_{\mathfrak{m}+\mathfrak{k}+1}& \dots & {\mathfrak{a}}_{\mathfrak{k}+2\mathfrak{m}-2}\end{array}\right|,\mathrm{with}{\mathfrak{a}}_1=1.$$

Noor [24] studied the growth behavior of ${\mathfrak{H}}_{\mathfrak{m}}\left(\mathfrak{k}\right)$ as $\mathfrak{m}$ tends to infinity under bounded boundary conditions. For the special case where $\mathfrak{m}=2$ and $\mathfrak{k}=1$, the Hankel determinant reduces to the well-known expression

$${\mathfrak{H}}_2\left(1\right)={\mathfrak{a}}_3-{\mathfrak{a}}_2^2,$$

which has been generalized to the form $|{\mathfrak{a}}_3-\aleph {\mathfrak{a}}_2^2|$ for $\aleph \in \mathbb{C}$; see, for example, [25,26,27].

Moreover, setting $\mathfrak{m}=2,$ and $\mathfrak{k}=2$, the second Hankel determinant takes the form

$${\mathfrak{H}}_2\left(2\right)=\left|\begin{array}{cc}{\mathfrak{a}}_2& {\mathfrak{a}}_3\\ {\mathfrak{a}}_3& {\mathfrak{a}}_4\end{array}\right|={\mathfrak{a}}_2{\mathfrak{a}}_4-{\mathfrak{a}}_3^2.$$

Similarly, for $\mathfrak{m}=2$ and $\mathfrak{k}=3,$ one obtains the third Hankel determinant ${\mathfrak{H}}_2\left(3\right)$.

2. Preliminaries

Inspired by the class seen in Definition 7, we introduce a new subclass ${\left({\mathcal{R}}_{\varsigma }^u\right)}^{\mathfrak{g}}\left(\varphi \right)$, defined as follows:

Definition 8.

Let $\varsigma \le 1$. The generalized class of universally prestarlike functions, denoted by ${\left({\mathcal{R}}_{\varsigma }^u\right)}^{\mathfrak{g}}\left(\varphi \right)$, is to consist of all functions $\mathfrak{f}\in {\mathcal{A}}_1(\Lambda )$ satisfying the subordination condition:

$$\begin{array}{c}\hfill {\displaystyle \frac{{\mathfrak{D}}^{3-2\varsigma }(\mathfrak{f}\ast \mathfrak{g})\left(\zeta \right)}{{\mathfrak{D}}^{2-2\varsigma }(\mathfrak{f}\ast \mathfrak{g})\left(\zeta \right)}}\prec \varphi \left(\zeta \right)=\zeta +\sqrt{1+{\zeta }^2},\end{array}$$

where ≺ denotes subordination and ϕ is the analytic function defined in Equation (1).

From Remark 1, it follows that the class ${\left({\mathcal{R}}_{\varsigma }^u\right)}^{\mathfrak{g}}\left(\varphi \right)$ is non-empty.

In this paper, we employ the same methodology as in [18,28,29], while also utilizing the definition of the Srivastava–Owa fractional derivative (see [20,21,22]), along with the concept of subordination, to obtain the estimates for the coefficients ${\mathfrak{a}}_2$, ${\mathfrak{a}}_3$, and ${\mathfrak{a}}_4$ of the subclass of analytic functions of universally prestarlike functions ${\left({\mathcal{R}}_{\varsigma }^u\right)}^{\mathfrak{g}}\left(\varphi \right)$ introduced in Definition 8. Furthermore, we establish the Fekete–Szegö inequality of the form $\left|{\mathfrak{a}}_3-\aleph {\mathfrak{a}}_2^2\right|$ and determine the second Hankel determinant $\left|{\mathfrak{a}}_2{\mathfrak{a}}_4-{\mathfrak{a}}_3^2\right|$.

To proceed with the proof of our main results, we introduce the following lemmas. The initial lemma is the well-known Carathéodory’s lemma (see [30]).

Let $\mathbb{P}$ be the class of all analytic functions $\mathrm{{\rm Y}}$ in $\mathfrak{U}$ with positive real part, represented as

$$\mathrm{{\rm Y}}\left(\zeta \right)=1+\sum _{\mathfrak{j}=1}^{\infty }{\mathfrak{q}}_{\mathfrak{j}}{\zeta }^{\mathfrak{j}},(\mathfrak{Re}\left\{{\mathfrak{q}}_{\mathfrak{j}}\right\}>0,\zeta \in \mathfrak{U}).$$

(3)

Lemma 1

([31]). Assume $\mathrm{{\rm Y}}\in \mathbb{P}$ given by (3), therefore $\left|{\mathfrak{q}}_{\mathfrak{j}}\right|\le 2$ for all $\mathfrak{j}\ge 1$.

The following lemma provides an upper bound for the coefficients of functions in the class $\mathbb{P}$. Further information can be found in [10] (Lemma 1).

Lemma 2

([32]). Let $\mathrm{{\rm Y}}\in \mathbb{P}$ be given as in Equation (3). Then the following inequality holds:

$$\left|{\mathfrak{q}}_2-\varsigma {\mathfrak{q}}_1^2\right|\le 2max\{1,|2\varsigma -1\left|\right\}.$$

This bound is sharp, and equality is attained by the functions

$$\mathrm{{\rm Y}}\left(\zeta \right)={\displaystyle \frac{1+{\zeta }^2}{1-{\zeta }^2}},\mathit{and}\mathrm{{\rm Y}}\left(\zeta \right)={\displaystyle \frac{1+\zeta }{1-\zeta }}.$$

Lemma 3

([33]). Let $\mathrm{{\rm Y}}\in \mathbb{P}$ be as defined in Equation (3). Then, there exist $\mathfrak{x}$ and ζ such that $\left|\mathfrak{x}\right|\le 1,\left|\zeta \right|\le 1,$ and ${\mathfrak{q}}_{\mathfrak{1}}\in [0,2]$, for which the following representations hold:

$$2{\mathfrak{q}}_2={\mathfrak{q}}_1^2+\mathfrak{x}\left(4-{\mathfrak{q}}_1^2\right),$$

and

$$4{\mathfrak{q}}_3={\mathfrak{q}}_1^3+2\mathfrak{x}\left(4-{\mathfrak{q}}_1^2\right){\mathfrak{q}}_1-{\mathfrak{q}}_1\left(4-{\mathfrak{q}}_1^2\right){\mathfrak{x}}^2+2\left(4-{\mathfrak{q}}_1^2\right)\left(1-{\left|\mathfrak{x}\right|}^2\right)\zeta .$$

Lemma 4

([34]). The power series expansion of a function Υ, defined as in (3), converges within the unit disk $\mathfrak{U}$ to a function belonging to the class $\mathbb{P}$ if and only if the associated Toeplitz determinants,

$${\mathfrak{D}}_{\mathfrak{j}}=\left|\begin{array}{ccccc}2& {\mathfrak{q}}_1& {\mathfrak{q}}_2& \dots & {\mathfrak{q}}_{\mathfrak{j}}\\ {\mathfrak{q}}_{-1}& 2& {\mathfrak{q}}_1& \dots & {\mathfrak{q}}_{\mathfrak{j}-1}\\ ⋮& ⋮& ⋮& ⋮& ⋮\\ {\mathfrak{q}}_{-\mathfrak{j}}& {\mathfrak{q}}_{-\mathfrak{j}+1}& {\mathfrak{q}}_{-\mathfrak{j}+2}& \dots & 2\end{array}\right|,\mathfrak{j}=1,2,3,\dots ,$$

are all non-negative, where ${\mathfrak{q}}_{-\iota }=\overline{{\mathfrak{q}}_{\iota }}$. These determinants are strictly positive unless the function $\mathfrak{q}\left(\zeta \right)$ has the form

$$\mathfrak{q}\left(\zeta \right)=\sum _{\iota =1}^m{\rho }_{\iota }{\mathfrak{q}}_0\left(e^{i{\tau }_{\iota }\zeta }\right),$$

where ${\rho }_{\iota }>0,{\tau }_{\iota }\in \mathbb{R}$, and the values ${\tau }_{\mathfrak{j}}$ are pairwise distinct for $\iota \ne \mathfrak{j}$. In such cases, the determinants ${\mathfrak{D}}_{\mathfrak{j}}>0$ remain positive for $\mathfrak{j}<m-1$, but vanish for all $\mathfrak{j}\ge m$.

This result represents a necessary and sufficient condition derived by Carathéodory and Toeplitz (see [34]).

3. Estimates for the Coefficients and Fekete–Szegö Inequality

In this section, we derive bounds for the coefficients of functions $\mathfrak{f}$ belonging to the class ${\left({\mathcal{R}}_{\varsigma }^u\right)}^{\mathfrak{g}}\left(\varphi \right)$, which contributes to resolving the Fekete–Szegö problem within this subclass. Consider

$$\mathfrak{f}\left(\zeta \right)=\sum _{\mathfrak{j}=0}^{\infty }{\mathfrak{a}}_{\mathfrak{j}}{\zeta }^{\mathfrak{j}}={\int }_0^1{\displaystyle \frac{d\mu \left(\tau \right)}{1-\tau \zeta }},$$

where the coefficients are given by ${\mathfrak{a}}_{\mathfrak{j}}={\int }_0^1{\tau }^{\mathfrak{j}}d\mu \left(\tau \right)$, and $\mu \left(\tau \right)$ represents a probability measure on the interval $[0,1]$.

Theorem 1.

Let $\mathfrak{f}\in {\left({\mathcal{R}}_{\varsigma }^u\right)}^{\mathfrak{g}}\left(\varphi \right)$ be expressed as

$$\mathfrak{f}\left(\zeta \right)=\sum _{\mathfrak{j}=0}^{\infty }{\mathfrak{a}}_{\mathfrak{j}}{\zeta }^{\mathfrak{j}},\mathit{with}{\mathfrak{a}}_0=0\mathit{and}{\mathfrak{a}}_1=1.$$

Assume that ϕ is given by Equation (1). Then the following coefficient bounds hold:

$$\left|{\mathfrak{a}}_2\right|\le {\displaystyle \frac{2-\delta }{2}},\left|{\mathfrak{a}}_3\right|\le {\displaystyle \frac{(2-\delta )(3-\delta )}{6(3-2\varsigma )}}max\left\{1,\left|2\varsigma -{\displaystyle \frac{5}{2}}\right|\right\}.$$

Moreover, the Fekete–Szegö type inequality is given by

$$\left|{\mathfrak{a}}_3-\aleph {\mathfrak{a}}_2^2\right|\le {\displaystyle \frac{(2-\delta )(3-\delta )}{6(3-2\varsigma )}}max\left\{1,\left|2\varsigma -{\displaystyle \frac{5}{2}}+{\displaystyle \frac{3(2-\delta )(3-2\varsigma )}{2(3-\delta )}}\aleph \right|\right\}.$$

Proof. 

Given that $\mathfrak{f}\in {\left({\mathcal{R}}_{\varsigma }^u\right)}^g\left(\varphi \right)$, there exists a Schwarz function $\omega$ that is analytic in $\mathfrak{U}$ and satisfies $\left|\omega \right(\zeta \left)\right|<1$ for all $\zeta \in \mathfrak{U}$, such that

$${\displaystyle \frac{{\mathfrak{D}}^{3-2\varsigma }(\mathfrak{f}\ast \mathfrak{g})\left(\zeta \right)}{{\mathfrak{D}}^{2-2\varsigma }\left(\mathfrak{f}\ast \mathfrak{g}\right)\left(\zeta \right)}}=\varphi \left(\omega \left(\zeta \right)\right)$$

(4)

Define the function $\mathrm{{\rm Y}}$ by

$$\mathrm{{\rm Y}}\left(\zeta \right)={\displaystyle \frac{1+\omega \left(\zeta \right)}{1-\omega \left(\zeta \right)}}=1+{\mathfrak{q}}_1\zeta +{\mathfrak{q}}_2{\zeta }^2+{\mathfrak{q}}_3{\zeta }^3+\cdots$$

Because $\omega$ is a Schwarz function, it follows that $Re\left(\mathrm{{\rm Y}}\right(\zeta \left)\right)\ge 0$ and $\mathrm{{\rm Y}}\left(0\right)=1$, implying that $\mathrm{{\rm Y}}$ belongs to the class $\mathbb{P}$. Consequently,

$$\begin{array}{c}\hfill \omega \left(\zeta \right)={\displaystyle \frac{\mathrm{{\rm Y}}\left(\zeta \right)-1}{\mathrm{{\rm Y}}\left(\zeta \right)+1}}={\displaystyle \frac{1}{2}}\left[{\mathfrak{q}}_1\zeta +\left({\mathfrak{q}}_2-{\displaystyle \frac{{\mathfrak{q}}_1^2}{2}}\right){\zeta }^2+\left({\mathfrak{q}}_3-{\mathfrak{q}}_1{\mathfrak{q}}_2+{\displaystyle \frac{{\mathfrak{q}}_1^3}{4}}\right){\zeta }^3+\cdots \right].\end{array}$$

(5)

Considering Equation (5), it follows that

$$\begin{array}{ccc}\hfill \varphi \left(\omega \right(\zeta \left)\right)& =& \varphi \left({\displaystyle \frac{\mathrm{{\rm Y}}\left(\zeta \right)-1}{\mathrm{{\rm Y}}\left(\zeta \right)+1}}\right)=\omega \left(\zeta \right)+\sqrt{{\left(\omega \left(\zeta \right)\right)}^2+1}\hfill \\ & =& 1+{\displaystyle \frac{{\mathfrak{q}}_1}{2}}\zeta +\left({\displaystyle \frac{{\mathfrak{q}}_2}{2}}-{\displaystyle \frac{{\mathfrak{q}}_1^2}{8}}\right){\zeta }^2+\left({\displaystyle \frac{{\mathfrak{q}}_3}{2}}-{\displaystyle \frac{{\mathfrak{q}}_1{\mathfrak{q}}_2}{4}}\right){\zeta }^3+\dots .\hfill \end{array}$$

(6)

Now by (6),

$${\mathfrak{c}}_1={\displaystyle \frac{{\mathfrak{q}}_1}{2}},{\mathfrak{c}}_2={\displaystyle \frac{{\mathfrak{q}}_2}{2}}-{\displaystyle \frac{{\mathfrak{q}}_1^2}{8}},{\mathfrak{c}}_3={\displaystyle \frac{{\mathfrak{q}}_3}{2}}-{\displaystyle \frac{{\mathfrak{q}}_1{\mathfrak{q}}_2}{4}}$$

(7)

On the other hand, in view of (4) and (6), we have

$$\begin{array}{c}\hfill 1+\sum _{\mathfrak{j}=1}^{\infty }{\mathfrak{c}}_{\mathfrak{j}}{\zeta }^{\mathfrak{j}}={\displaystyle \frac{{\mathfrak{D}}^{3-2\varsigma }(\mathfrak{f}\ast \mathfrak{g})\left(\zeta \right)}{{\mathfrak{D}}^{2-2\varsigma }(\mathfrak{f}\ast \mathfrak{g})\left(\zeta \right)}}={\displaystyle \frac{\zeta +{\sum }_{\mathfrak{j}=2}^{\infty }\widehat{\Theta }(\varsigma ,\mathfrak{j}){\mathfrak{a}}_{\mathfrak{j}}{\mathfrak{g}}_{\mathfrak{j}}{\zeta }^{\mathfrak{j}}}{\zeta +{\sum }_{\mathfrak{j}=2}^{\infty }\Theta (\varsigma ,\mathfrak{j}){\mathfrak{a}}_{\mathfrak{j}}{\mathfrak{g}}_{\mathfrak{j}}{\zeta }^{\mathfrak{j}}}}\end{array}$$

(8)

where

$$\Theta (\varsigma ,\mathfrak{j})={\displaystyle \frac{{\prod }_{\iota =2}^{\mathfrak{j}}(\iota -2\varsigma )}{(\mathfrak{j}-1)!}},\mathrm{and}\widehat{\Theta }(\varsigma ,\mathfrak{j})={\displaystyle \frac{{\prod }_{\iota =2}^{\mathfrak{j}}(\iota +1-2\varsigma )}{(\mathfrak{j}-1)!}}.$$

Equating the coefficients of $\zeta ,{\zeta }^2$, and ${\zeta }^3$ in (8), we obtain

$$\begin{array}{c}{\mathfrak{c}}_1=\left[\widehat{\Theta }(\varsigma ,2)-\Theta (\varsigma ,2)\right]{\mathfrak{a}}_2{\mathfrak{g}}_2,\hfill \end{array}$$

(9)

$$\begin{array}{c}\hfill {\mathfrak{c}}_2=\left[\widehat{\Theta }(\varsigma ,3)-\Theta (\varsigma ,3)\right]{\mathfrak{a}}_3{\mathfrak{g}}_3+{\left[\Theta (\varsigma ,2){\mathfrak{a}}_2{\mathfrak{g}}_2\right]}^2-\left[\Theta (\varsigma ,2)\widehat{\Theta }(\varsigma ,2)\right]{\left({\mathfrak{a}}_2{\mathfrak{g}}_2\right)}^2,\end{array}$$

(10)

and

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathfrak{c}}_3& =\left[\widehat{\Theta }(\varsigma ,4)-\Theta (\varsigma ,4)\right]{\mathfrak{a}}_4{\mathfrak{g}}_4+\left[2\Theta (\varsigma ,2)\Theta (\varsigma ,3)-\Theta (\varsigma ,2)\widehat{\Theta }(\varsigma ,3)-\Theta (\varsigma ,3)\widehat{\Theta }(\varsigma ,2)\right]{\mathfrak{a}}_2{\mathfrak{a}}_3{\mathfrak{g}}_2{\mathfrak{g}}_3\hfill \\ & +\left[{\left(\Theta (\varsigma ,2)\right)}^2\widehat{\Theta }(\varsigma ,2)-{\left(\Theta (\varsigma ,2)\right)}^3\right]{\left({\mathfrak{a}}_2{\mathfrak{g}}_2\right)}^3.\hfill \end{array}\end{array}$$

(11)

Simplifying (9)–(11) we have

$$\begin{array}{c}\hfill {\mathfrak{a}}_2={\displaystyle \frac{{\mathfrak{c}}_1}{{\mathfrak{g}}_2}},{\mathfrak{a}}_3={\displaystyle \frac{{\mathfrak{c}}_2+2(1-\varsigma ){\mathfrak{c}}_1^2}{(3-2\varsigma ){\mathfrak{g}}_3}}\end{array}$$

(12)

and

$${\mathfrak{a}}_4={\displaystyle \frac{{\mathfrak{c}}_3+3(1-\varsigma ){\mathfrak{c}}_1{\mathfrak{c}}_2+2{(1-\varsigma )}^2{\mathfrak{c}}_1^3}{(3-2\varsigma )(2-\varsigma ){\mathfrak{g}}_4}}$$

(13)

Using the equalities (7) in (12), it follows that

$$\begin{array}{c}\hfill {\mathfrak{a}}_2={\displaystyle \frac{{\mathfrak{q}}_1}{2{\mathfrak{g}}_2}}={\displaystyle \frac{(2-\delta ){\mathfrak{q}}_1}{4}},{\mathfrak{a}}_3={\displaystyle \frac{(2-\delta )(3-\delta )}{6(3-2\varsigma )}}\left[{\displaystyle \frac{1}{2}}\left({\mathfrak{q}}_2-{\displaystyle \frac{(2\varsigma -\frac{3}{2}){\mathfrak{q}}_1^2}{2}}\right)\right].\end{array}$$

(14)

Taking the modulus and applying Lemma 1, we obtain

$$\begin{array}{c}\hfill \left|{\mathfrak{a}}_2\right|\le {\displaystyle \frac{2-\delta }{2}},\left|{\mathfrak{a}}_3\right|={\displaystyle \frac{(2-\delta )(3-\delta )}{6(3-2\varsigma )}}\left[{\displaystyle \frac{1}{2}}\left|\left({\mathfrak{q}}_2-{\displaystyle \frac{(2\varsigma -\frac{3}{2}){\mathfrak{q}}_1^2}{2}}\right)\right|\right].\end{array}$$

(15)

Then, by employing Lemma 2, it follows that

$$\left|{\mathfrak{a}}_3\right|\le {\displaystyle \frac{(2-\delta )(3-\delta )}{6(3-2\vartheta )}}max\left\{1,\left|2\varsigma -{\displaystyle \frac{5}{2}}\right|\right\}.$$

Now, for any $\aleph \in \mathbb{C}$, and by using Equations (14) and (15), we get

$$\begin{array}{cc}\hfill {\mathfrak{a}}_3-\aleph {\mathfrak{a}}_2^2& ={\displaystyle \frac{(2-\delta )(3-\delta )}{6(3-2\varsigma )}}\left[{\displaystyle \frac{1}{2}}\left({\mathfrak{q}}_2-{\mathfrak{q}}_1^2\left({\displaystyle \frac{2\varsigma -\frac{3}{2}}{2}}+{\displaystyle \frac{3(2-\delta )(3-2\varsigma )}{4(3-\delta )}}\aleph \right)\right)\right]\\ & ={\displaystyle \frac{(2-\delta )(3-\delta )}{12(3-2\varsigma )}}\left|{\mathfrak{q}}_2-\nu {\mathfrak{q}}_1^2\right|,\end{array}$$

where $\nu =\left({\displaystyle \frac{2\varsigma -\frac{3}{2}}{2}}+{\displaystyle \frac{3(2-\delta )(3-2\varsigma )}{4(3-\delta )}}\aleph \right)$. Thus, by applying Lemma 2, we get

$$\begin{array}{c}\hfill \left|{\mathfrak{a}}_3-\aleph {\mathfrak{a}}_2^2\right|\le {\displaystyle \frac{(2-\delta )(3-\delta )}{6(3-2\varsigma )}}max\left\{1,\left|2\varsigma -{\displaystyle \frac{5}{2}}+{\displaystyle \frac{3(2-\delta )(3-2\varsigma )}{2(3-\delta )}}\aleph \right|\right\}.\end{array}$$

□

Corollary 1.

By setting $\varsigma ={\displaystyle \frac{1}{2}}$ in Theorem 1, the following inequality holds.

$$\begin{array}{c}\hfill \left|{\mathfrak{a}}_3-\aleph {\mathfrak{a}}_2^2\right|\le {\displaystyle \frac{(2-\delta )(3-\delta )}{12}}max\left\{1,\left|-{\displaystyle \frac{3}{2}}+{\displaystyle \frac{3(2-\delta )}{(3-\delta )}}\aleph \right|\right\}.\end{array}$$

Remark 3

([28]). In particular, by setting $\aleph =1$ and $\delta =0$ in Theorem 1, we obtain $\left|{\mathfrak{a}}_3-{\mathfrak{a}}_2^2\right|\le {\displaystyle \frac{1}{(3-2\varsigma )}}$.

4. The Hankel Inequality

This section is devoted to deriving upper bounds for the Hankel determinant $\left|{\mathfrak{a}}_2{\mathfrak{a}}_4-{\mathfrak{a}}_3^2\right|$ for functions $\mathfrak{f}\in {\left({\mathcal{R}}_{\varsigma }^u\right)}^{\mathfrak{g}}\left(\varphi \right)$.

Theorem 2.

Let $\mathfrak{f}\in {\left({\mathcal{R}}_{\varsigma }^u\right)}^{\mathfrak{g}}\left(\varphi \right)$, with ϕ specified by (1). Then, we have the inequality $\left|{\mathfrak{a}}_2{\mathfrak{a}}_4-{\mathfrak{a}}_3^2\right|\le \mathcal{G}\left(0\right)={\left({\displaystyle \frac{(3-\delta )(2-\delta )}{6(3-2\varsigma )}}\right)}^2$.

Proof. 

As $\mathfrak{f}\in {\left({\mathcal{R}}_{\varsigma }^u\right)}^{\mathfrak{g}}\left(\varphi \right)$, there exists a Schwarz function $\omega$ that is analytic in $\mathfrak{U}$, satisfying $\omega \left(0\right)=0$, and $\left|\omega \right(\zeta \left)\right|<1$ for all $\zeta \in \mathfrak{U}$, such that

$${\displaystyle \frac{{\mathfrak{D}}^{3-2\varsigma }\left(\mathfrak{f}\ast \mathfrak{g}\right)\left(\zeta \right)}{{\mathfrak{D}}^{2-2\varsigma }\left(\mathfrak{f}\ast \mathfrak{g}\right)\left(\zeta \right)}}=\varphi \left(\omega \left(\zeta \right)\right).$$

(16)

Using the equalities (7) in (12) and (13), it follows that

$$\begin{array}{c}\hfill {\mathfrak{a}}_2={\displaystyle \frac{{\mathfrak{q}}_1}{2{\mathfrak{g}}_2}}={\displaystyle \frac{(2-\delta ){\mathfrak{q}}_1}{4}},{\mathfrak{a}}_3={\displaystyle \frac{(2-\delta )(3-\delta )}{6(3-2\varsigma )}}\left[{\displaystyle \frac{1}{2}}\left({\mathfrak{q}}_2-{\displaystyle \frac{(2\varsigma -\frac{3}{2}){\mathfrak{q}}_1^2}{2}}\right)\right],\end{array}$$

and

$$\begin{array}{c}\hfill {\mathfrak{a}}_4={\displaystyle \frac{(4-\delta )(3-\delta )(2-\delta )}{{\left(4\right)}^2(4!)(3-2\varsigma )(2-\varsigma )}}\left(8{\mathfrak{q}}_3+4(2-3\varsigma ){\mathfrak{q}}_1{\mathfrak{q}}_2+(1-\varsigma )(1-4\varsigma ){\mathfrak{q}}_1^3\right)\end{array}$$

(17)

Therefore, we conclude that the estimate for the second Hankel determinant is provided by

$$\begin{array}{c}\hfill \begin{array}{cc}& {\mathfrak{a}}_2{\mathfrak{a}}_4-{\mathfrak{a}}_3^2={\displaystyle \frac{1}{\mathcal{H}(\varsigma ,\delta )}}\left[{\mathfrak{d}}_{\mathfrak{1}}{\mathfrak{q}}_1^4-{\mathfrak{d}}_{\mathfrak{2}}{\mathfrak{q}}_1^2{\mathfrak{q}}_2-{\mathfrak{d}}_{\mathfrak{3}}{\mathfrak{q}}_2^2+{\mathfrak{d}}_{\mathfrak{4}}{\mathfrak{q}}_1{\mathfrak{q}}_3\right]\hfill \end{array}\end{array}$$

(18)

where

$$\begin{array}{ccc}\hfill \mathcal{H}(\varsigma ,\delta )& =& {\displaystyle \frac{3{\left(4\right)}^4(3!){(3-2\varsigma )}^2(2-\varsigma )}{(4-\delta ){(3-\delta )}^2{(2-\delta )}^2}},\hfill \\ \hfill {\mathfrak{d}}_{\mathfrak{1}}& =& {\displaystyle \frac{3(3-2\varsigma )(1-4\varsigma )(1-\varsigma )}{(3-\delta )}}-{\displaystyle \frac{2^3(2-\varsigma ){(2\varsigma -\frac{3}{2})}^2}{(4-\delta )}},\hfill \\ \hfill {\mathfrak{d}}_{\mathfrak{2}}& =& -\left[{\displaystyle \frac{12(3-2\varsigma )(2-3\varsigma )}{(3-\delta )}}+{\displaystyle \frac{2^5(2-\varsigma )(2\varsigma -\frac{3}{2})}{(4-\delta )}}\right],\hfill \\ \hfill {\mathfrak{d}}_{\mathfrak{3}}& =& {\displaystyle \frac{2^5(2-\varsigma )}{(4-\delta )}},\mathrm{and}{\mathfrak{d}}_{\mathfrak{4}}={\displaystyle \frac{(4!)(3-2\varsigma )}{(3-\delta )}}.\hfill \end{array}$$

Using Lemma 3 in (18), we have

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \left|{\mathfrak{a}}_2{\mathfrak{a}}_4-{\mathfrak{a}}_3^2\right|& ={\displaystyle \frac{1}{\mathcal{H}(\varsigma ,\delta )}}\left|{\displaystyle \frac{-1}{4}}\left[2{\mathfrak{d}}_{\mathfrak{2}}-4{\mathfrak{d}}_{\mathfrak{1}}+{\mathfrak{d}}_{\mathfrak{3}}-{\mathfrak{d}}_{\mathfrak{4}}\right]{\mathfrak{q}}_1^4+\left[{\displaystyle \frac{{\mathfrak{d}}_{\mathfrak{4}}-{\mathfrak{d}}_{\mathfrak{2}}-{\mathfrak{d}}_{\mathfrak{3}}}{2}}\right]\left(4-{\mathfrak{q}}_1^2\right){\mathfrak{q}}_1^2\mathfrak{x}-\right.\hfill \\ & \left[{\displaystyle \frac{{\mathfrak{d}}_{\mathfrak{4}}-{\mathfrak{d}}_{\mathfrak{3}}}{4}}{\mathfrak{q}}_1^2+{\mathfrak{d}}_{\mathfrak{3}}\right]\left(4-{\mathfrak{q}}_1^2\right){\mathfrak{x}}^2+{\displaystyle \frac{{\mathfrak{d}}_{\mathfrak{4}}}{2}}\left(4-{\mathfrak{q}}_1^2\right){\mathfrak{q}}_1\left(1-{\left|\mathfrak{x}\right|}^2\right)\zeta \mid .\hfill \end{array}\end{array}$$

(19)

Letting $\left|{\mathfrak{q}}_1\right|=\eta$ and in view of Lemma 1, we may assume without restriction that $\eta \in [0,2]$. Thus, applying the triangle inequality in (19) with $\epsilon =\left|\mathfrak{x}\right|\le 1$ and $\left|\zeta \right|\le 1$, we derive

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \left|{\mathfrak{a}}_2{\mathfrak{a}}_4-{\mathfrak{a}}_3^2\right|& \le {\displaystyle \frac{1}{\mathcal{H}(\varsigma ,\delta )}}\left|{\displaystyle \frac{1}{4}}\left|2{\mathfrak{d}}_{\mathfrak{2}}-4{\mathfrak{d}}_{\mathfrak{1}}+{\mathfrak{d}}_{\mathfrak{3}}-{\mathfrak{d}}_{\mathfrak{4}}\right|{\eta }^4+\left[{\displaystyle \frac{{\mathfrak{d}}_{\mathfrak{4}}-{\mathfrak{d}}_{\mathfrak{2}}-{\mathfrak{d}}_{\mathfrak{3}}}{2}}\right]\left(4-{\eta }^2\right){\eta }^2\epsilon +\right.\hfill \\ & \left[{\displaystyle \frac{{\mathfrak{d}}_{\mathfrak{4}}-{\mathfrak{d}}_{\mathfrak{3}}}{4}}{\eta }^2+{\mathfrak{d}}_{\mathfrak{3}}\right]\left(4-{\eta }^2\right){\epsilon }^2+{\displaystyle \frac{{\mathfrak{d}}_{\mathfrak{4}}}{2}}\left(4-{\eta }^2\right)\eta \left(1-{\epsilon }^2\right)\mid \hfill \\ & ={\displaystyle \frac{1}{\mathcal{H}(\varsigma ,\delta )}}\left|{\displaystyle \frac{1}{4}}\left|2{\mathfrak{d}}_{\mathfrak{2}}-4{\mathfrak{d}}_{\mathfrak{1}}+{\mathfrak{d}}_{\mathfrak{3}}-{\mathfrak{d}}_{\mathfrak{4}}\right|{\eta }^4+\left[{\displaystyle \frac{{\mathfrak{d}}_{\mathfrak{4}}-{\mathfrak{d}}_{\mathfrak{2}}-{\mathfrak{d}}_{\mathfrak{3}}}{2}}\right]\left(4-{\eta }^2\right){\eta }^2\epsilon +\right.\hfill \\ & \left[{\displaystyle \frac{{\mathfrak{d}}_{\mathfrak{4}}-{\mathfrak{d}}_{\mathfrak{3}}}{4}}{\eta }^2-{\displaystyle \frac{{\mathfrak{d}}_{\mathfrak{4}}}{2}}\eta +{\mathfrak{d}}_{\mathfrak{3}}\right]\left(4-{\eta }^2\right){\epsilon }^2+{\displaystyle \frac{{\mathfrak{d}}_{\mathfrak{4}}}{2}}\left(4-{\eta }^2\right)\eta \mid =\mathcal{V}(\eta ,\epsilon ).\hfill \end{array}\end{array}$$

(20)

Observe that for $(\eta ,\epsilon )\in [0,2)\times [0,1]$, taking the partial derivative of $\mathcal{V}(\eta ,\epsilon )$ with respect to $\epsilon$ gives

$$\begin{array}{c}\hfill {\displaystyle \frac{\partial \mathcal{V}}{\partial \epsilon }}={\displaystyle \frac{1}{\mathcal{H}(\varsigma ,\delta )}}\left\{\left[{\displaystyle \frac{{\mathfrak{d}}_{\mathfrak{4}}-{\mathfrak{d}}_{\mathfrak{2}}-{\mathfrak{d}}_{\mathfrak{3}}}{2}}\right]\left(4-{\eta }^2\right){\eta }^2+2\left[{\displaystyle \frac{{\mathfrak{d}}_{\mathfrak{4}}-{\mathfrak{d}}_{\mathfrak{3}}}{4}}{\eta }^2-{\displaystyle \frac{{\mathfrak{d}}_{\mathfrak{4}}}{2}}\eta +{\mathfrak{d}}_{\mathfrak{3}}\right]\left(4-{\eta }^2\right)\epsilon \right\}.\end{array}$$

It is clear that the coefficient of $\epsilon$ in Equation (20) remains a positive real number for all $(\eta ,\epsilon )\in [0,2)\times [0,1]$. Consequently, expression (20) is always positive for $\varsigma \le 1$, indicating that $\mathcal{V}(\eta ,\epsilon )$ is an increasing function with respect to $\epsilon$. Therefore, the maximum value cannot occur in the interior of the closed region $[0,2)\times [0,1]$. Furthermore, since $\eta \in [0,2)$, it follows that $max\mathcal{V}(\eta ,\epsilon )=\mathcal{V}(\eta ,1)=\mathcal{G}\left(\eta \right)$. Upon simplification, we obtain

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \mathcal{V}(\eta ,1)& =\mathcal{G}\left(\eta \right)={\displaystyle \frac{1}{\mathcal{H}(\varsigma ,\delta )}}\left|{\displaystyle \frac{1}{4}}\left|2{\mathfrak{d}}_{\mathfrak{2}}-4{\mathfrak{d}}_{\mathfrak{1}}+{\mathfrak{d}}_{\mathfrak{3}}-{\mathfrak{d}}_{\mathfrak{4}}\right|{\eta }^4+\left[{\displaystyle \frac{{\mathfrak{d}}_{\mathfrak{4}}-{\mathfrak{d}}_{\mathfrak{2}}-{\mathfrak{d}}_{\mathfrak{3}}}{2}}\right]\left(4-{\eta }^2\right){\eta }^2+\right.\hfill \\ & \left[{\displaystyle \frac{{\mathfrak{d}}_{\mathfrak{4}}-{\mathfrak{d}}_{\mathfrak{3}}}{4}}{\eta }^2-{\displaystyle \frac{{\mathfrak{d}}_{\mathfrak{4}}}{2}}\eta +{\mathfrak{d}}_{\mathfrak{3}}\right]\left(4-{\eta }^2\right)+{\displaystyle \frac{{\mathfrak{d}}_{\mathfrak{4}}}{2}}\left(4-{\eta }^2\right)\eta \mid .\hfill \end{array}\end{array}$$

Differentiating $\mathcal{G}\left(\eta \right)$ with respect to $\eta$ yields:

$$\begin{array}{c}\hfill \begin{array}{cc}& {\mathcal{G}}^{\prime }\left(\eta \right)={\displaystyle \frac{1}{\mathcal{H}(\varsigma ,\delta )}}\left|\left|2{\mathfrak{d}}_{\mathfrak{2}}-4{\mathfrak{d}}_{\mathfrak{1}}+{\mathfrak{d}}_{\mathfrak{3}}-{\mathfrak{d}}_{\mathfrak{4}}\right|{\eta }^3+\left[{\displaystyle \frac{{\mathfrak{d}}_{\mathfrak{4}}-{\mathfrak{d}}_{\mathfrak{2}}-{\mathfrak{d}}_{\mathfrak{3}}}{2}}\right]\left(-4{\eta }^3+8\eta \right)+\right.\hfill \\ & \left[{\displaystyle \frac{{\mathfrak{d}}_{\mathfrak{4}}-{\mathfrak{d}}_{\mathfrak{3}}}{4}}{\eta }^2-{\displaystyle \frac{{\mathfrak{d}}_{\mathfrak{4}}}{2}}\eta +{\mathfrak{d}}_{\mathfrak{3}}\right]\left(-2\eta \right)+\left[{\displaystyle \frac{{\mathfrak{d}}_{\mathfrak{4}}-{\mathfrak{d}}_{\mathfrak{3}}}{2}}\eta -{\displaystyle \frac{{\mathfrak{d}}_{\mathfrak{4}}}{2}}\right]\left(4-{\eta }^2\right)+{\displaystyle \frac{{\mathfrak{d}}_{\mathfrak{4}}}{2}}\left(4-3{\eta }^2\right)\mid .\hfill \end{array}\end{array}$$

This can be simplified to the following form:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\mathcal{G}}^{\prime }\left(\eta \right)={\displaystyle \frac{1}{\mathcal{H}(\varsigma ,\delta )}}\left|\left\{\left|2{\mathfrak{d}}_{\mathfrak{2}}-4{\mathfrak{d}}_{\mathfrak{1}}+{\mathfrak{d}}_{\mathfrak{3}}-{\mathfrak{d}}_{\mathfrak{4}}\right|+2{\mathfrak{d}}_{\mathfrak{2}}+3{\mathfrak{d}}_{\mathfrak{3}}-3{\mathfrak{d}}_{\mathfrak{4}}\right\}{\eta }^3+\left\{-4{\mathfrak{d}}_{\mathfrak{2}}-8{\mathfrak{d}}_{\mathfrak{3}}+6{\mathfrak{d}}_{\mathfrak{4}}\right\}\eta \right|.\end{array}\end{array}$$

Evaluating at $\eta =0$, we find that ${\mathcal{G}}^{\prime }\left(0\right)$ is negative, which implies that $\mathcal{G}\left(\eta \right)$ attains a local maximum at $\eta =0$. Thus, the following inequality holds:

$$\begin{array}{c}\hfill \left|{\mathfrak{a}}_2{\mathfrak{a}}_4-{\mathfrak{a}}_3^2\right|\le \mathcal{G}\left(0\right)={\left({\displaystyle \frac{(3-\delta )(2-\delta )}{6(3-2\varsigma )}}\right)}^2.\end{array}$$

If ${\mathcal{G}}^{\prime }\left(0\right)=0$, then the root of the derivative still occurs at $\eta =0$. Furthermore, the second derivative is given by

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\mathcal{G}}^{″}\left(\eta \right)={\displaystyle \frac{1}{\mathcal{H}(\varsigma ,\delta )}}\left|3\left\{\left|2{\mathfrak{d}}_{\mathfrak{2}}-4{\mathfrak{d}}_{\mathfrak{1}}+{\mathfrak{d}}_{\mathfrak{3}}-{\mathfrak{d}}_{\mathfrak{4}}\right|+2{\mathfrak{d}}_{\mathfrak{2}}+3{\mathfrak{d}}_{\mathfrak{3}}-3{\mathfrak{d}}_{\mathfrak{4}}\right\}{\eta }^2+\left\{-4{\mathfrak{d}}_{\mathfrak{2}}-8{\mathfrak{d}}_{\mathfrak{3}}+6{\mathfrak{d}}_{\mathfrak{4}}\right\}\right|.\end{array}\end{array}$$

It is negative at $\eta =0$, indicating that the function $\mathcal{G}\left(\eta \right)$ attains its maximum at $\eta =0$. Consequently, we obtain the bound

$$|{\mathfrak{a}}_2{\mathfrak{a}}_4-{\mathfrak{a}}_3^2|\le \mathcal{G}\left(0\right)={\left({\displaystyle \frac{(3-\delta )(2-\delta )}{(18-12\varsigma )}}\right)}^2.$$

□

Corollary 2.

By setting $\varsigma ={\displaystyle \frac{1}{2}}$ in Theorem 2, the following inequality holds.

$$\left|{\mathfrak{a}}_2{\mathfrak{a}}_4-{\mathfrak{a}}_3^2\right|\le {\left({\displaystyle \frac{(3-\delta )(2-\delta )}{12}}\right)}^2.$$

Remark 4

([28]). In particular, choosing $\varsigma ={\displaystyle \frac{1}{2}}$ and $\delta =0$ in Theorem 2 yields $\left|{\mathfrak{a}}_2{\mathfrak{a}}_4-{\mathfrak{a}}_3^2\right|\le {\displaystyle \frac{1}{4}}.$

5. Conclusions

In this paper, a new subclass of universally prestarlike functions ${\left({\mathcal{R}}_{\varsigma }^u\right)}^{\mathfrak{g}}\left(\varphi \right)$ of $\mathcal{A}$ was introduced in Definition 8 and investigated concerning coefficient studies. By employing the subordination technique, as used in [28], together with the definition of the Srivastava–Owa fractional derivative, we obtained coefficient estimates for ${\mathfrak{a}}_2$, ${\mathfrak{a}}_3$, and ${\mathfrak{a}}_4$. Furthermore, the Fekete–Szegö inequality of the form $\left|{\mathfrak{a}}_3-\aleph {\mathfrak{a}}_2^2\right|$ was evaluated. The second Hankel determinant $\left|{\mathfrak{a}}_2{\mathfrak{a}}_4-{\mathfrak{a}}_3^2\right|$ was also obtained. Some corollaries and remarks are also provided to illustrate and support the main results. The present results are an extension of the class previously investigated given by Definition 7.

The results obtained in this paper are not only of theoretical interest but also have potential applications in mathematical physics and engineering. In particular, fractional derivatives play a significant role in the modeling of fractional differential equations that arise in viscoelasticity, anomalous diffusion, and fluid dynamics. For example, the applications of the artificial-intelligence-based deep neural networks combined with fractional calculus tools can be seen in [35], fractional order particular partial differential equations have been investigated in [36], while fractal-fractional derivatives with a power law kernel are used in the investigation presented in [37]. Moreover, the study of analytic functions in shell-shaped domains may provide tools for solving boundary value problems in complex geometries. Thus, the introduced class can contribute to both pure and applied aspects of fractional calculus and geometric function theory. Furthermore, considering the results obtained in shell-shaped domains related to bi-starlike functions seen in [38], the present investigation could inspire future investigations using the same pattern and including bi-starlike functions.