Using the operator defined by Carlson and Shaffer, we defined a new subclass of analytic functions defined by a subordination relation to the shell shaped function. We determined estimate bounds of the four coefficients of the power series expansions, we gave upper bound for the Fekete–SzegőSzegő functional and for the Hankel determinant of order two for .
Let be the class of functions which are analytic in the open unit disk , and also let be the subset of comprising of functions
Let which are analytic in , then the well-known Hadamard (or convolution) product of and is given by
For two functions , we say that f is subordinate to g, denoted by , if there exists a Schwarz function with , , and , such that for all . In particular, if g is univalent in , then the following equivalence relationship holds true:
Let be the well-known class of Carathéodory functions that is a set of functions with the power series expansion
and such that for all .
For the function of the form (1), Noonan and Thomas  defined q-th Hankel determinant as
It is well-known (see Duren ) that, if f is given by (1) and is univalent in , then occurs, and this result is sharp. The determinant has also been measured by many authors. For example, the rate of growth of as for functions with bounded boundary was determined. In , it has been shown, a fraction of two bounded analytic functions with its Laurent series around the origin having integral coefficients, is rational. The Hankel determinant of meromorphic functions, (see ), and various properties of these determinants can be found in ). In 1966, the Hankel determinant of areally mean p-valent functions, univalent functions, and starlike functions were extensively studied by Pommerenke . Lately, several authors have investigated of innumerable subclasses of univalent and multivalent functions and, for more details on Hankel determinants, one may refer [1,6,7,8,9,10,11,12,13,14]. For , a problem of finding a sharp (best possible) upper bound of for the subclass is generally called Fekete–Szegő problem for the subclass , where is a real or a complex number. There are some well known subclasses of univalent functions, such that the starlike functions, convex functions, and close-to-convex functions, for which the problem of finding sharp upper bounds for the functional was completely solved (see [15,16,17,18]). For the family of analytic functions , Janteng et al.  have found the sharp upper bound to . For initial work on the class , one may refer to the article of MacGregor .
The concept of shell-like domains gained importance in the recent times and it was introduced by Sokół and Paprocki . Recently, for , Raina and Sokół  have widely studied and found some coefficient inequalities for if it satisfies the subordination condition that , and these results are further improved by Sokół and Thomas , the Fekete–Szegő inequality for were obtained and, in view of the Alexander result between the class and , the Fekete–Szegő inequality for functions in were also obtained. The function maps the unit disc onto a shell shaped region on the right half plane, and it is analytic and univalent on . The range is symmetric respecting the real axis and is a function with positive real part in , with . Moreover, it is a starlike domain with respect to the point (see ), such as Figure 1 shows.
 Let be normalized by in the unit disc . We denote by the class of analytic functions and satisfying the condition that
where the branch of the square root is chosen to be the principal one that is .
Now, we recall the Carlson–Shaffer operator  defined by
is the incomplete beta function, and denotes the Pochhammer symbol (or the shifted factorial) defined in terms of the Gamma function by
For is given by (1) and by (3), one can get the Carlson and Shaffer operator
Next, we will emphasize a few special cases of the operator , as follows:
(iv), , is the well-known Ruscheweyh derivative of f ;
(v), is the well-known Owa-Srivastava fractional differential operator of f .
Motivated by the articles of Raina and Sokół , Sokół and Thomas , Dziok and Raina , and Raina et al. , using the concept of subordination and the linear operator , we define a new subclass of denoted by . For this subclass, we obtained coefficient inequalities, Fekete–Szegő inequality, and upper bound for the Hankel determinant .
We define a new subclass of as below:
For , let , with and , denote the subclass of functions that satisfies the subordination condition
where the branch of the square root is chosen to be the principal one that is .
In the following remark, we prove that is non-empty.
If we define the function by , , a simple computation yields to
Considering the circular transformation
with , and assuming that , we obtain that maps the unit disc onto the open disc that is symmetric respecting the real axes connecting the points and .
If , then , and for , , and , using the MAPLE™ software we get the next images of by like in the Figure 2:
These show that , which is for some values of that is , whenever , for , , and . It follows that there exist values of the parameters , , and , such that .
Now, by suitably specializing the parameter , we define the new subclasses of as remarked below:
(i) For , let denote the subclass of , the members of which are given by (1) and satisfy the subordination condition
(ii) For , let denote the subclass of , members of which are of the form (1) and if it satisfy the condition
(iii) For the special case for , let , members of which are given by (1) and satisfy the subordination
In the all of the above subordinations, and throughout the whole paper, the branch of the square root is chosen at the principal one, which is , and , .
Using the techniques of Libera and Zlotkiewicz  and Koepf , combined with the help of MAPLE™ software, we find Fekete–Szegő inequality and Hankel determinant for the function of the class .
To establish our main results, we recall the followings lemmas. The first lemma is the well-known Carathéodory’s lemma (see also  Corollary 2.3):
 If and given by (2), then , for all , and the result is best possible for , .
The next lemma gives us a majorant for the coefficients of the functions of the class , and more details may be found in  (Lemma 1):
To prove that the bounds are sharp, we define the functions and , , respectively, with and by
respectively. Clearly, . In addition, we write .
If or , then the equality holds if and only if f is or one of its rotations. When , then the equality holds if and only if f is or one of its rotations. If , then the equality holds if and only if f is or one of its rotations. If , then the equality holds if and only if f is or one of its rotations. □
4. Hankel Determinant Result for
The next result deals with an upper bound of for the subclass :
If , using a similar proof like in the proof of Theorem 1, from (17), (18), and (19), we get
Using the relations (11) and (12) of Lemma 4, we get
with , , and
where . Since , it follows that , hence we may assume without loss of generality that , and, according to Lemma 1, it follows that . Now, using the triangle’s inequality in (26) and substituting , we get
Next, we will find maximum of on the closed rectangle . Using the MAPLE™ software for the following code, where we denoted and ,
We will prove that, under our assumption we have , and therefore
Letting and , from (24), it follows that . A simple computation shows that
then if and only if the inequality holds for all . This last inequality is equivalent to
and a simple computation shows that for all . Therefore, the above inequality holds whenever the assumption (24) is satisfied, hence . Since , we have
we have .
If , then , and using the inequality , we get . If , then , and, because , , it follows that .
Therefore, for all , we have . Since (27) was proved, the upper bound of on the closed rectangle is attained at and , which implies the inequality (25). □
By suitably specializing the parameter λ, one can deduce the above results for the subclasses of , and , which are defined, respectively, in Remark 3 (i) and (ii). Furthermore, by taking , we can easily state the result for the function class given in Remark 3 (iii). The details involved may be left as an exercise for the interested reader.
Conceptualization, G.M. and T.B.; methodology, G.M. and T.B.; investigation, G.M. and T.B.; resources, G.M. and T.B.; writing—original draft preparation, G.M. and T.B.; writing—review and editing, G.M. and T.B.; supervision, G.M. and T.B.; project administration, G.M. and T.B.. The authors contributed equally to this work. All authors have read and agreed to the published version of the manuscript.
This research received no external funding.
The authors are grateful to the reviewers of this article who gave valuable remarks, comments, and advice, in order to revise and improve the results of the paper.
Conflicts of Interest
The authors declare no conflict of interest.
Noonan, J.W.; Thomas, D.K. On the second Hankel determinant of areally mean p-valent functions. Trans. Am. Math. Soc.1976, 223, 337–346. [Google Scholar]
Duren, P.L. Univalent Functions; Grundlehren der Mathematischen Wissenschaften, 259; Springer: New York, NY, USA, 1983. [Google Scholar]
Cantor, D.G. Power series with integral coefficients. Bull. Amer. Math. Soc.1963, 69, 362–366. [Google Scholar] [CrossRef]
Wilson, R. Determinantal criteria for meromorphic functions. Proc. Lond. Math. Soc.1954, 4, 357–374. [Google Scholar] [CrossRef]
Vein, R.; Dale, P. Determinants and Their Applicatons im Mathematical Physics; Applied Mathematical Sciences; Springer: New York, NY, USA, 1999; Volume 134. [Google Scholar]
Pommerenke, C. On the coefficients and Hankel determinant of univalent functions. J. Lond. Math. Soc.1966, 41, 111–122. [Google Scholar] [CrossRef]
Ehrenborg, R. The Hankel determinant of exponential polynomials. Am. Math. Mon.2000, 107, 557–560. [Google Scholar] [CrossRef]
Hayami, T.; Owa, S. Generalized Hankel determinant for certain classes. Int. J. Math. Anal.2010, 4, 2573–2585. [Google Scholar]
Janteng, A.; Halim, S.A.; Darus, M. Hankel determinant for starlike and convex functions. Int. J. Math. Anal.2007, 1, 619–625. [Google Scholar]
Layman, J.W. The Hankel transform and some of its properties. J. Integer Seq.2001, 4, 1–11. [Google Scholar]
Libera, R.J.; Zlotkiewicz, E.J. Early coefficients of the inverse of a regular convex function. Proc. Am. Math. Soc.1982, 85, 225–230. [Google Scholar] [CrossRef]
Mishra, A.K.; Gochhayat, P. Second Hankel determinant for a classes of analytic functions defined by fractional derivatives. Int. J. Math. Math. Sci.2008, 2008, 153280. [Google Scholar] [CrossRef]
Mohamed, N.; Mohamed, D.; Cik Soh, S. Second Hankel determinant for certain generalized classes of analytic functions. Int. J. Math. Anal.2012, 6, 807–812. [Google Scholar]
Noor, K.I. Hankel determinant problem for the class of functions with bounded boundary rotation. Rev. Roumaine Math. Pures Appl.1983, 28, 731–739. [Google Scholar]
Fekete, M.; Szegő, G. Eine Bemerkung über ungerade schlichte Funktionen. J. Lond. Math. Soc.1933, 8, 85–89. [Google Scholar] [CrossRef]
Keogh, F.R.; Merkes, E.P. A coefficient inequality for certain classes of analytic functions. Proc. Am. Math. Soc.1969, 20, 8–12. [Google Scholar] [CrossRef]
Koepf, W. On the Fekete–Szegő problem for close-to-convex functions. Proc. Am. Math. Soc.1987, 101, 89–95. [Google Scholar]
Koepf, W. On the Fekete–Szegő problem for close-to-convex functions-II. Arch. Math.1987, 49, 420–433. [Google Scholar] [CrossRef]
Janteng, A.; Halim, S.A.; Darus, M. Coefficient inequality for a function whose derivative has a positive real part. JIPAM. J. Inequal. Pure Appl. Math.2006, 7, 50. [Google Scholar]
MacGregor, T.H. Functions whose derivative have a positive real part. Trans. Am. Math. Soc.1962, 104, 532–537. [Google Scholar] [CrossRef]
Paprocki, E.; Sokół, J. The extremal problems in some subclass of strongly starlike functions. Zeszyty Nauk. Politech. Rzeszowskiej Mat.1996, 20, 89–94. [Google Scholar]
Raina, R.K.; Sokół, J. On Coefficient estimates for a certain class of starlike functions. Hacet. J. Math. Stat.2015, 44, 1427–1433. [Google Scholar] [CrossRef]
Sokół, J.; Thomas, D.K. Further results on a class of starlike functions related to the Bernoulli lemniscate. Houston J. Math.2018, 44, 83–95. [Google Scholar]
Raina, R.K.; Sokół, J. On a class of analytic functions governed by subordination. Acta Univ. Sapientiae Math.2019, 11, 144–155. [Google Scholar] [CrossRef]
Carlson, B.C.; Shaffer, D.B. Starlike and prestarlike hypergeometric functions. SIAM J. Math. Anal.1984, 15, 737–745. [Google Scholar] [CrossRef]
Ruscheweyh, S. New criteria for univalent functions. Proc. Am. Math. Soc.1975, 49, 109–115. [Google Scholar] [CrossRef]
Owa, S.; Srivastava, H.M. Univalent and starlike generalized hypergeometric functions. Can. J. Math.1987, 39, 1057–1077. [Google Scholar] [CrossRef]
Dziok, J.; Raina, R.K.; Sokół, J. Differential subordinations and alpha-convex functions. Acta Math. Sci. Ser. B Engl. Ed.2013, 33 B, 609–620. [Google Scholar] [CrossRef]
Raina, R.K.; Sharma, P.; Sokół, J. Certain classes of analytic functions related to the crescent-shaped regions. J. Contemp. Math. Anal.2018, 53, 355–362. [Google Scholar]
Carathéodory, C. Über den Variabilitätsbereich der Fourier’schen Konstanten von positiven harmonischen Funktionen. Rend. Circ. Mat. Palermo1911, 32, 193–217. [Google Scholar] [CrossRef]
Ma, W.C.; Minda, D. A unified treatment of some special classes of univalent functions. In Proceedings of the Conference on Complex Analysis (Tianjin, 1992); Li, Z., Ren, F., Yang, L., Zhang, S., Eds.; International Press: Cambridge, MA, USA, 1994; pp. 157–169. [Google Scholar]
Libera, R.J.; Zlotkiewicz, E.J. Coefficient bounds for the inverse of a function with derivative in P. Proc. Am. Math. Soc.1983, 87, 251–257. [Google Scholar] [CrossRef]