On the Composition Structures of Certain Fractional Integral Operators

Abstract

This paper investigates the composition structures of certain fractional integral operators whose kernels are certain types of generalized hypergeometric functions. It is shown how composition formulas of these operators can be closely related to the various Erdélyi-type hypergeometric integrals. We also derive a derivative formula for the fractional integral operator and some applications of the operator are considered for a certain Volterra-type integral equation, which provide two generalizations to Khudozhnikov’s integral equation (see below). Some specific relationships, examples, and some future research problems are also discussed.

1. Introduction

In 1978, Saigo [1] introduced his widely used fractional integral operators $I^{\alpha ,\beta ,\eta }$ and $J^{\alpha ,\beta ,\eta }$ (see Equations (16) and (17) below). Saigo’s operators involve the Gauss hypergeometric functions ${}_2{F}_1$ as kernels and possess many properties (see, for example, Refs. [1,2,3,4,5]). Over the past few decades, Saigo’s operators have been applied in various branches of mathematics, especially in the Geometric Function Theory (see Refs. [6,7,8]). The symmetry of parameters of various hypergeometric functions injects more choice and flexibility into the theory of Generalized Fractional Calculus.

A natural question that arises is: Can an operator involving a generalized hypergeometric function ${}_p{F}_q$ as kernel have such properties as Saigo’s operators? In this direction, some efforts have been made by some authors to find particular forms of operators. In 1987, Goyal and Jain [9] introduced two fractional integral operators $I_{\alpha }^h$ and $K_{\beta }^{\lambda }$, which involve the generalized hypergeometric functions ${}_p{F}_q$ as kernels. Later, Goyal et al. [10,11] introduced two more general fractional integral operators involving the generalized hypergeometric function ${}_p{F}_q$ and Srivastava’s polynomial $S_n^m$.

Although very general in form, the properties of the operators $I_{\alpha }^h$ and $K_{\beta }^{\lambda }$ introduced by Goyal et al. are far less succinct than those of Saigo’s operators. For Saigo’s operators $I^{\alpha ,\beta ,\eta }$ and $J^{\alpha ,\beta ,\eta }$, we have the following useful properties (see Refs. [12,13]):

$$I^{\alpha ,\beta ,\eta }{x}^{\lambda }=\frac{\mathsf{\Gamma }\left(\lambda \right)\mathsf{\Gamma }(\lambda -\beta +\eta +1)}{\mathsf{\Gamma }(\lambda -\beta +1)\mathsf{\Gamma }(\lambda +\alpha +\eta +1)}{x}^{\lambda -\beta }$$

(1)

$$\left(\Re \left(\alpha \right)>0,\Re \left(\lambda \right)>max\left\{0,\Re (\beta -\eta )\right\}-1\right)$$

and

$$J^{\alpha ,\beta ,\eta }{x}^{\lambda }=\frac{\mathsf{\Gamma }(\beta -\lambda )\mathsf{\Gamma }(\eta -\lambda )}{\mathsf{\Gamma }(-\lambda )\mathsf{\Gamma }(\alpha +\beta +\eta -\lambda )}{x}^{\lambda -\beta }$$

(2)

$$\left(\Re \left(\alpha \right)>0,\Re \left(\lambda \right)<max\left\{\Re \left(\beta \right),\Re \left(\eta \right)\right\}\right).$$

Under certain conditions, we also have the following composition properties (see Ref. [1], p. 140, Equations (2.22) and (2.23), see also Ref. [3]):

$$\begin{array}{cc}\hfill I^{\alpha ,\beta ,\eta }{I}^{\gamma ,\delta ,\alpha +\eta }f& =I^{\alpha +\gamma ,\beta +\delta ,\eta }f,\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill I^{\alpha ,\beta ,\eta }{I}^{\gamma ,\delta ,\eta -\beta -\gamma -\delta }f& =I^{\alpha +\gamma ,\beta +\delta ,\eta -\gamma -\delta }f,\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill J^{\gamma ,\delta ,\alpha +\eta }{J}^{\alpha ,\beta ,\eta }f& =J^{\alpha +\gamma ,\beta +\delta ,\eta }f\hfill \end{array}$$

(5)

and

$$J^{\gamma ,\delta ,\eta -\beta -\gamma -\delta }{J}^{\alpha ,\beta ,\eta }f=J^{\alpha +\gamma ,\beta +\delta ,\eta -\gamma -\delta }f.$$

(6)

However, it seems rather difficult to find properties for the operators $I_{\alpha }^h$ and $K_{\beta }^{\lambda }$ similar to those given above by (1)–(6). Moreover, it is still unknown whether the corresponding generalized fractional derivatives of the forms (see Ref. [3], Equations (3.2) and (3.4))

$$I^{\alpha ,\beta ,\eta }f=\frac{{\mathrm{d}}^n}{\mathrm{d}{x}^n}{I}^{\alpha +n,\beta -n,\eta -n}f\mathrm{and}{J}^{\alpha ,\beta ,\eta }f={\left(-1\right)}^n\frac{{\mathrm{d}}^n}{\mathrm{d}{x}^n}{J}^{\alpha +n,\beta -n,\eta -n}f$$

(7)

can be defined for the operators $I_{\alpha }^h$ and $K_{\beta }^{\lambda }$.

Very recently, the authors [14] introduced two fractional integral operators $\mathcal{I}$ and $\mathcal{J}$ (see below Equations (12) and (13)) whose kernels involve a very special class of generalized hypergeometric function. The authors have to some extent overcome the limitations of the operators $I_{\alpha }^h$ and $K_{\beta }^{\lambda }$ and obtained results similar to (1) and (2). Subsequently, some further results and applications related to $\mathcal{I}$ and $\mathcal{J}$ were discovered in the papers [15,16].

The aim of the present paper is to first establish for the operators $\mathcal{I}$ and $\mathcal{J}$ some results relating to the composition structures of the defined operators analogous to Formulas (3)–(7). We also consider defining the corresponding fractional derivative operators of these operators $\mathcal{I}$ and $\mathcal{J}$. Finally, we shall consider some connections of our work with Khudozhnikov’s work [17] on Volterra-type integral equations.

2. Preliminaries

In this paper, the symbols $\mathbb{N}$, ${\mathbb{R}}_{+}$, and $\mathbb{C}$ denote the set of natural, positive real, and complex numbers, respectively. The Pochhammer symbol ${\left(a\right)}_k$ is defined by

$${\left(a\right)}_k:=\frac{\mathsf{\Gamma }\left(a+k\right)}{\mathsf{\Gamma }\left(a\right)}=\left\{\begin{array}{cc}1\hfill & (k=0;a\in \mathbb{C}\backslash \{0\left\}\right)\hfill \\ a\left(a+1\right)\cdots \left(a+k-1\right)\hfill & (k\in \mathbb{N};a\in \mathbb{C}).\hfill \end{array}\right.$$

In addition, we shall use the convention of writing the finite sequence of parameters $a_1,\cdots ,a_p$ by $\left(a_p\right)$ and the product of p Pochhammer symbols by ${\left(\left(a_p\right)\right)}_k\equiv {\left(a_1\right)}_k\cdots {\left(a_p\right)}_k$, where an empty product $p=0$ is treated as unity.

We are particularly interested in the generalized hypergeometric function ${}_{r+p}{F}_{r+q}$ of the form

$${}_{r+p}{F}_{r+q}\left[\begin{array}{cc}\left(a_p\right),& (f_r+m_r)\\ \left(b_q\right),& \left(f_r\right)\end{array};z\right]:=\sum _{k=0}^{\infty }\frac{{\left(\left(a_p\right)\right)}_k}{{\left(\left(b_q\right)\right)}_k}\frac{{\left((f_r+m_r)\right)}_k}{{\left(\left(f_r\right)\right)}_k}\frac{z^k}{k!},$$

(8)

where $m_1,\cdots ,m_r\in \mathbb{N}$. The conditions of convergence of (8) follow easily from the usual definition of the generalized hypergeometric function; see Ref. [18], p. 62 and Ref. [19], p. 30. Several recent results concerning this particular type of generalized hypergeometric function have been obtained in Ref. [20] (see also Ref. [21]).

For convenience, we put

$$m:=m_1+\cdots +m_r,$$

(9)

and let ${\sigma }_j$ ($0\le j\le m$) be determined by the generating relation

$$\prod _{j=1}^r{(f_j+x)}_{m_j}=\sum _{j=0}^m{\sigma }_{m-j}{x}^j.$$

(10)

Obviously, ${\sigma }_j$’s depend only on $f_j$ ($1\le j\le r$). Additionally, we define $A_k$ ($0\le k\le m$) by

$$A_k=\sum _{j=k}^m\left\{\begin{array}{c}j\\ k\end{array}\right\}{\sigma }_{m-j},A_0={\left(f_1\right)}_{m_1}\cdots {\left(f_r\right)}_{m_r},A_m=1,$$

(11)

where the notation $\left\{\begin{array}{c}j\\ k\end{array}\right\}$ denotes the Stirling number of the second kind.

Definition 1

([14], p. 423, Definition 1.1). Let $x,h,\nu \in {\mathbb{R}}_{+}$, $\delta ,a,b,f_1,\cdots ,f_r\in \mathbb{C}$ and $m_1,\cdots ,m_r\in \mathbb{N}$. Also, let $\Re \left(\mu \right)>0$ and φ be a suitable complex-valued function defined on ${\mathbb{R}}_{+}$. Then the fractional integral of the first kind of a function φ is defined by

$$\begin{array}{cc}\hfill \left(\mathcal{I}\phi \right)\left(x\right)& \equiv \left(\mathcal{I}\begin{array}{cc}\mu ;a,b:& (f_r+m_r)\\ h;\nu ,\delta :& \left(f_r\right)\end{array}\phi \right)\left(x\right)\hfill \\ \hfill & :=\frac{\nu x^{-\delta -\nu (\mu +h)}}{\mathsf{\Gamma }\left(\mu \right)}{\int }_0^x{\left(x^{\nu }-s^{\nu }\right)}^{\mu -1}{}_{r+2}{F}_{r+1}\left[\begin{array}{c}a,b,\\ \mu ,\end{array}\begin{array}{c}(f_r+m_r)\\ \left(f_r\right)\end{array};1-\frac{s^{\nu }}{x^{\nu }}\right]\phi \left(s\right)s^{\nu h+\nu -1}\mathrm{d}s,\hfill \end{array}$$

(12)

and the fractional integral of the second kind of a function $\phi \left(x\right)$ is defined by

$$\begin{array}{cc}\hfill \left(\mathcal{J}\phi \right)\left(x\right)& \equiv \left(\mathcal{J}\begin{array}{cc}\mu ;a,b:& (f_r+m_r)\\ h;\nu ,\delta :& \left(f_r\right)\end{array}\phi \right)\left(x\right)\hfill \\ \hfill & :=\frac{\nu x^{\nu h+\nu -1}}{\mathsf{\Gamma }\left(\mu \right)}{\int }_x^{\infty }{\left(s^{\nu }-x^{\nu }\right)}^{\mu -1}{}_{r+2}{F}_{r+1}\left[\begin{array}{c}a,b,\\ \mu ,\end{array}\begin{array}{c}(f_r+m_r)\\ \left(f_r\right)\end{array};1-\frac{x^{\nu }}{s^{\nu }}\right]\phi \left(s\right)s^{-\delta -\nu (\mu +h)}\mathrm{d}s.\hfill \end{array}$$

(13)

When $r=0$, we obtain

$$\left(\mathcal{I}\begin{array}{c}\mu ;a,b\\ h;\nu ,\delta \end{array}\phi \right)\left(x\right)=\frac{\nu x^{-\delta -\nu (\mu +h)}}{\mathsf{\Gamma }\left(\mu \right)}{\int }_0^x{\left(x^{\nu }-s^{\nu }\right)}^{\mu -1}{}_2{F}_1\left[\begin{array}{c}a,b\\ \mu \end{array};1-\frac{s^{\nu }}{x^{\nu }}\right]\phi \left(s\right)s^{\nu h+\nu -1}\mathrm{d}s$$

(14)

and

$$\left(\mathcal{J}\begin{array}{c}\mu ;a,b\\ h;\nu ,\delta \end{array}\phi \right)\left(x\right)=\frac{\nu x^{\nu h+\nu -1}}{\mathsf{\Gamma }\left(\mu \right)}{\int }_x^{\infty }{\left(s^{\nu }-x^{\nu }\right)}^{\mu -1}{}_2{F}_1\left[\begin{array}{c}a,b\\ \mu \end{array};1-\frac{x^{\nu }}{s^{\nu }}\right]\phi \left(s\right)s^{-\delta -\nu (\mu +h)}\mathrm{d}s.$$

(15)

Some properties of the operators (12) and (13) have been presented in Refs. [14,16]. Further, the operators $\mathcal{I}\begin{array}{c}\mu ;a,b\\ h;\nu ,\delta \end{array}$ and $\mathcal{J}\begin{array}{c}\mu ;a,b\\ h;\nu ,\delta \end{array}$ have the following special cases:

(a)

For $h=0$, $\nu =1$ and $\delta =0$ in (14) and (15), we obtain

$$\left(\mathcal{I}\begin{array}{c}\mu ;a,b\\ 0;1,0\end{array}\phi \right)\left(x\right)={}_2{I}_{0+}^{\mu }\left(a,b\right)\phi \left(x\right)\mathrm{and}\left(\mathcal{I}\begin{array}{c}\mu ;a,b\\ 0;1,0\end{array}\phi \right)\left(x\right)={}_4{I}_{-}^{\mu }\left(a,b\right)\phi \left(x\right),$$

where ${}_2{I}_{0+}^{\mu }\left(a,b\right)$ and ${}_4{I}_{-}^{\mu }\left(a,b\right)$ are two of the four operators introduced by Grinko and Kilbas [22].

(b)

When $h=0$, $\nu =1$, $\delta =\beta$, $\mu =\alpha$, $a=\alpha +\beta$ and $b=-\eta$ in (14) and (15), then we obtain Saigo’s fractional integral operators

$$\begin{array}{cc}\hfill \left(I^{\alpha ,\beta ,\eta }\phi \right)\left(x\right)& =\left(\mathcal{I}\begin{array}{c}\alpha ;\alpha +\beta ,-\eta \\ 0;1,\beta \end{array}\phi \right)\left(x\right)\hfill \\ \hfill & =\frac{x^{-\beta -\alpha }}{\mathsf{\Gamma }\left(\alpha \right)}{\int }_0^x{\left(x-s\right)}^{\alpha -1}{}_2{F}_1\left[\begin{array}{c}\alpha +\beta ,-\eta \\ \alpha \end{array};1-\frac{s}{x}\right]\phi \left(s\right)\mathrm{d}s\left(\Re \left(\alpha \right)>0\right)\hfill \end{array}$$

(16)

and

$$\begin{array}{cc}\hfill \left(J^{\alpha ,\beta ,\eta }\phi \right)\left(x\right)& =\left(\mathcal{J}\begin{array}{c}\alpha ;\alpha +\beta ,-\eta \\ 0;1,\beta \end{array}\phi \right)\left(x\right)\hfill \\ \hfill & =\frac{1}{\mathsf{\Gamma }\left(\alpha \right)}{\int }_x^{\infty }{\left(s-x\right)}^{\alpha -1}{}_2{F}_1\left[\begin{array}{c}\alpha +\beta ,-\eta \\ \alpha \end{array};1-\frac{x}{s}\right]\phi \left(s\right)s^{-\beta -\alpha }\mathrm{d}s\left(\Re \left(\alpha \right)>0\right).\hfill \end{array}$$

(17)

(c)

When $a=b=0$, it is not difficult to observe that $\mathcal{I}\begin{array}{c}\mu ;a,b\\ h;\nu ,\delta \end{array}$ and $\mathcal{J}\begin{array}{c}\mu ;a,b\\ h;\nu ,\delta \end{array}$ contain the Erdélyi–Kober operators (see Ref. [19], p. 105 and Ref. [23], p. 322)

$$\begin{array}{cc}\hfill \left(I_{+;\nu ,h}^{\mu }f\right)\left(x\right)& =\left(\mathcal{I}\begin{array}{c}\mu ;0,0\\ h;\nu ,0\end{array}f\right)\left(x\right)\hfill \\ \hfill & =\frac{\nu x^{-\nu (\mu +h)}}{\mathsf{\Gamma }\left(\mu \right)}{\int }_0^x{\left(x^{\nu }-s^{\nu }\right)}^{\mu -1}f\left(s\right)s^{\nu h+\nu -1}\mathrm{d}s\hfill \\ \hfill & =\frac{1}{\mathsf{\Gamma }\left(\mu \right)}{\int }_0^1{\left(1-u\right)}^{\mu -1}f\left(x{u}^{1/\nu }\right)u^h\mathrm{d}u\left(\Re \left(\mu \right)>0,\nu ,h\in {\mathbb{R}}_{+}\right)\hfill \end{array}$$

(18)

and

$$\begin{array}{cc}\hfill \left(I_{-;\nu ,h}^{\mu }\right)f\left(x\right)& =\left(\mathcal{J}\begin{array}{c}\mu ;0,0\\ h-1+1/\nu ;\nu ,0\end{array}f\right)\left(x\right)\hfill \\ \hfill & =\frac{\nu x^{\nu h}}{\mathsf{\Gamma }\left(\mu \right)}{\int }_x^{\infty }{\left(s^{\nu }-x^{\nu }\right)}^{\mu -1}f\left(s\right)s^{\nu (1-\mu -h)-1}\mathrm{d}s\hfill \\ \hfill & =\frac{1}{\mathsf{\Gamma }\left(\mu \right)}{\int }_1^{\infty }{\left(u-1\right)}^{\mu -1}f\left(x{u}^{1/\nu }\right)u^{-\mu -h}\mathrm{d}u\left(\Re \left(\mu \right)>0,\nu ,h\in {\mathbb{R}}_{+}\right)\hfill \end{array}$$

(19)

as special cases. The operators obtained by letting $\nu =1$ in (18) and (19) are usually denoted by $I_{\eta ,\alpha }^{+}$ and $K_{\eta ,\alpha }^{-}$, respectively (see Ref. [19], p. 106).

The operators defined above by (12) and (13) were previously studied in Refs. [14,16] in the space $X_c^p$ ($c\in \mathbb{R}$, $1\le p\le \infty$) of those complex-valued Lebesgue measurable functions $\phi$ on ${\mathbb{R}}_{+}$ for which ${\parallel \phi \parallel }_{X_c^p}<\infty$, where

$${\parallel \phi \parallel }_{X_c^p}:={\left({\int }_0^{\infty }{\left|u^c\phi \left(u\right)\right|}^p\frac{\mathrm{d}u}{u}\right)}^{1/p}.$$

(20)

It follows at once that $X_{1/p}^p=L^p\left({\mathbb{R}}_{+}\right)$. For convenience, we define

$${\mathfrak{c}}_1\left(t\right):=1+h+\frac{t}{\nu }\mathrm{and}{\mathfrak{c}}_2\left(t\right):={\mathfrak{c}}_1\left(\delta -1\right)-\frac{t}{\nu }.$$

The following lemma gives some useful properties of the operators $\mathcal{I}$ and $\mathcal{J}$ relating to the norm defined in (20).

Lemma 1.

Let $\phi \in X_c^p$.

(i) 

If $\Re \left(\mu \right)>0$ and ${\mathfrak{c}}_1\left(-c\right)+min\left\{0,\Re (\mu -a-b-m)\right\}>0$, then the operator $\mathcal{I}$ is bounded from $X_c^p$ into $X_{c+\Re \left(\delta \right)}$, and

$${\parallel \mathcal{I}\phi \parallel }_{X_{c+\Re \left(\delta \right)}^p}\le C_1{\parallel \phi \parallel }_{X_c^p}.$$

(ii) 

If $\Re \left(\mu \right)>0$ and $\Re \left({\mathfrak{c}}_2\left(-c\right)\right)+min\left\{0,\Re (\mu -a-b-m)\right\}>0$, then the operator $\mathcal{J}$ is bounded from $X_c^p$ into $X_{c+\Re \left(\delta \right)}$, and

$${\parallel \mathcal{J}\phi \parallel }_{X_{c+\Re \left(\delta \right)}^p}\le C_2{\parallel \phi \parallel }_{X_c^p}.$$

(iii) 

If $\Re \left(\mu \right)>0$ and ${\mathfrak{c}}_1\left(-c\right)+min\left\{0,\Re (\mu -a-b)\right\}>0$, then the operator $\mathcal{I}\begin{array}{c}\mu ;a,b\\ h;\nu ,\delta \end{array}$ is bounded from $X_c^p$ into $X_c^p$, and

$$\parallel x^{\delta }\mathcal{I}\begin{array}{c}\mu ;a,b\\ h;\nu ,\delta \end{array}{\phi \parallel }_{X_c^p}\le C_1^{*}{\parallel \phi \parallel }_{X_c^p}.$$

(iv) 

If $\Re \left(\mu \right)>0$ and $\Re \left({\mathfrak{c}}_2\left(-c\right)\right)+min\left\{0,\Re (\mu -a-b)\right\}>0$, then the operator $\mathcal{J}\begin{array}{c}\mu ;a,b\\ h;\nu ,\delta \end{array}$ is bounded from $X_c^p$ into $X_c^p$, and

$$\parallel x^{\delta }\mathcal{J}\begin{array}{c}\mu ;a,b\\ h;\nu ,\delta \end{array}{\phi \parallel }_{X_c^p}\le C_2^{*}{\parallel \phi \parallel }_{X_c^p}.$$

(v) 

If $\Re \left(\mu \right)>0$ and ${\mathfrak{c}}_1\left(-c\right)>0$, then the operator $I_{+;\nu ,h}^{\mu }$ is bounded from $X_c^p$ into $X_c^p$, and

$$\parallel I_{+;\nu ,h}^{\mu }{\phi \parallel }_{X_c^p}\le C_1^{**}{\parallel \phi \parallel }_{X_c^p}.$$

(vi) 

If $\Re \left(\mu \right)>0$ and $\nu h+c>0$, then the operator $I_{-;\nu ,h}^{\mu }$ is bounded from $X_c^p$ into $X_c^p$, and

$$\parallel I_{-;\nu ,h}^{\mu }{\phi \parallel }_{X_c^p}\le C_2^{**}{\parallel \phi \parallel }_{X_c^p}.$$

Proof. 

The results (i) and (ii) are established in Ref. [14], p. 437, Theorem 3.1.

On the other hand, the results (iii) and (iv) are the corollaries of (i) and (ii) (see also Ref. [16], p. 614).

Finally, the results (v) and (vi) follow immediately from (iii) and (iv). These results are consistent with the classical ones. It may be noted that if we set $c=1/p$ in (v) and (vi), then the operator $I_{+;\nu ,h}^{\mu }$ is bounded in $L_p\left({\mathbb{R}}_{+}\right)$ provided that $\Re \left(\mu \right)>0$ and $h>-1+1/p\nu$, and the operator $I_{-;\nu ,h}^{\mu }$ is bounded in $L_p\left({\mathbb{R}}_{+}\right)$ provided that $\Re \left(\mu \right)>0$ and $h>-1/p\nu$ (see Ref. [19], p. 107, Lemma 2.28 and Ref. [23], p. 323).    □

It should be particularly emphasized here that the operators $\mathcal{I}$ and $\mathcal{J}$ are quite different from the multiple Erdélyi–Kober fractional integral operators (see Ref. [4], p. 11, see also Refs. [24,25]), though some special cases of $\mathcal{I}$ and $\mathcal{J}$ when $r=0$ (e.g., Saigo’s operators) can be expressed as multiple Erdélyi–Kober fractional integral operators. The cases that $r=0$ are very special because Meijer’s G-function $G_{2,2}^{2,0}\left[\sigma \right]$ and ${}_2{F}_1[1-\sigma ]$ have the following relationship (see [4], p. 18, Equation (1.1.18))

$$G_{2,2}^{2,0}\left[\sigma \left|\begin{array}{c}{\gamma }_1+{\delta }_1,{\gamma }_2+{\delta }_2\\ {\gamma }_1,{\gamma }_2\end{array}\right.\right]=\frac{{\sigma }^{{\gamma }_2}{\left(1-\sigma \right)}^{{\delta }_1+{\delta }_2-1}}{\mathsf{\Gamma }({\delta }_1+{\delta }_2)}{}_2{F}_1\left[\begin{array}{c}{\gamma }_2+{\delta }_2-{\gamma }_1,{\delta }_1\\ {\delta }_1+{\delta }_2\end{array};1-\sigma \right]$$

(21)

for $\sigma <1$. However, there is no such relationship between $G_{m,m}^{m,0}\left[\sigma \right]$ and ${}_{r+2}{F}_{r+1}[1-\sigma ]$. A slightly more general case than (21) will lead us to the Marichev–Saigo–Maeda fractional integral operators (see Refs. [26,27]), which are also very different from our operators $\mathcal{I}$ and $\mathcal{J}$. In addition, the operators $\mathcal{I}$ and $\mathcal{J}$cannot be regarded as special cases of G-transform studied in Ref. [28]. Since the kernels of $\mathcal{I}$ and $\mathcal{J}$ are not of Sonine’s type, they cannot be included in the theory developed very recently by Luchko (see Ref. [29]).

3. The Main Results

3.1. Composition Formulas

Theorem 1.

Assume that $\phi \in X_c^p$. Let

$${\lambda }_1\equiv \lambda -a-m,{\lambda }_2\equiv \lambda -b-m\mathrm{and}{\mathfrak{p}}_m\equiv \lambda -a-b-m,$$

(22)

where m is given by (9). Let $\left({\vartheta }_m\right)$ be the nonvanishing zeros of the parametric polynomial ${\mathfrak{Q}}_m\left(t\right)$ defined by

$$\begin{array}{cc}\hfill {\mathfrak{Q}}_m\left(t\right)& =\sum _{k=0}^m{\left(-1\right)}^k{A}_k{\left({\lambda }_1\right)}_k{\left({\lambda }_2\right)}_k{\left(t\right)}_k{\left(a+k\right)}_{m-k}{\left(b+k\right)}_{m-k}\hfill \\ \hfill & ·{}_3{F}_2\left[\begin{array}{c}k-m,k+t,-{\mathfrak{p}}_m\\ a+k,b+k\end{array};1\right],\hfill \end{array}$$

(23)

where $A_k$$(0\le k\le m)$ is defined in (11). Then for $\Re \left(\gamma \right)>0$, $\Re \left(\mu \right)>1/p>0$,

$$h+min\left\{0,\Re (\gamma +\mu -a-b-m)\right\}>\Re \left(\gamma +{\mathfrak{p}}_m+(\rho -c)/\nu \right)$$

and $h+1+min\left\{0,\Re (\mu -\lambda -{\mathfrak{p}}_m)\right\}>\Re \left((c+\rho )/\nu \right)$, we have

$$\left(\mathcal{I}\begin{array}{cc}\mu ;{\lambda }_1,{\lambda }_2:& (f_r+m_r)\\ h;\nu ,\delta :& \left(f_r\right)\end{array}\left(\mathcal{I}\begin{array}{c}\gamma ;-{\mathfrak{p}}_m,\lambda -\mu \\ h-\gamma -{\mathfrak{p}}_m-\rho /\nu ;\nu ,\rho \end{array}\phi \right)\right)\left(x\right)=\left(\mathcal{I}\begin{array}{cc}\gamma +\mu ;a,b:& ({\vartheta }_m+1)\\ h-\gamma -{\mathfrak{p}}_m-\rho /\nu ;\nu ,\delta +\rho :& \left({\vartheta }_m\right)\end{array}\phi \right)\left(x\right)$$

(24)

where $\mathcal{I}\begin{array}{cc}\mu ;a,b:& (f_r+m_r)\\ h;\nu ,\delta :& \left(f_r\right)\end{array}$ and $\mathcal{I}\begin{array}{c}\mu ;a,b\\ h;\nu ,\delta \end{array}$ are defined by (12) and (14), respectively.

Proof. 

Denote the left-hand side of (24) by $\mathsf{\Phi }\left(x\right)$. Then by interchanging the order of integration, we obtain

$$\begin{array}{cc}\hfill \mathsf{\Phi }\left(x\right)& =\frac{\nu x^{-\delta -\nu (\mu +h)}}{\mathsf{\Gamma }\left(\mu \right)}{\int }_0^x{\left(x^{\nu }-s^{\nu }\right)}^{\mu -1}{}_{r+2}{F}_{r+1}\left[\begin{array}{c}{\lambda }_1,{\lambda }_2,\\ \mu ,\end{array}\begin{array}{c}(f_r+m_r)\\ \left(f_r\right)\end{array};1-\frac{s^{\nu }}{x^{\nu }}\right]s^{\nu h+\nu -1}\hfill \\ \hfill & ·\left\{\frac{\nu s^{-\nu (h-{\mathfrak{p}}_m)}}{\mathsf{\Gamma }\left(\gamma \right)}{\int }_0^s{\left(s^{\nu }-t^{\nu }\right)}^{\gamma -1}{}_2{F}_1\left[\begin{array}{c}-{\mathfrak{p}}_m,\lambda -\mu \\ \gamma \end{array};1-\frac{t^{\nu }}{s^{\nu }}\right]\phi \left(t\right)t^{\nu (h-\gamma -{\mathfrak{p}}_m)-\rho +\nu -1}\mathrm{d}t\right\}\mathrm{d}s\hfill \\ \hfill & =\frac{{\nu }^2{x}^{-\delta -\nu (\mu +h)}}{\mathsf{\Gamma }\left(\mu \right)\mathsf{\Gamma }\left(\gamma \right)}{\int }_0^x\phi \left(t\right)t^{\nu (h-\gamma -{\mathfrak{p}}_m)-\rho +\nu -1}{\Delta }_1\left(t\right)\mathrm{d}t,\hfill \end{array}$$

(25)

where

$$\begin{array}{cc}\hfill {\Delta }_1\left(t\right)& :={\int }_t^x{s}^{\nu h+\nu -1-\nu h+\nu {\mathfrak{p}}_m}{\left(x^{\nu }-s^{\nu }\right)}^{\mu -1}{\left(s^{\nu }-t^{\nu }\right)}^{\gamma -1}\hfill \\ \hfill & ·{}_{r+2}{F}_{r+1}\left[\begin{array}{c}{\lambda }_1,{\lambda }_2,\\ \mu ,\end{array}\begin{array}{c}(f_r+m_r)\\ \left(f_r\right)\end{array};1-\frac{s^{\nu }}{x^{\nu }}\right]{}_2{F}_1\left[\begin{array}{c}-{\mathfrak{p}}_m,\lambda -\mu \\ \gamma \end{array};1-\frac{t^{\nu }}{s^{\nu }}\right]\mathrm{d}s.\hfill \end{array}$$

(26)

We shall tackle Equation (24) and leave the verification of the validity of interchanging the order of integration in (25) at the end of the proof.

Letting $s^{\nu }=x^{\nu }-u\left(x^{\nu }-t^{\nu }\right)$ in (26), we have

$$\begin{array}{cc}\hfill & {\Delta }_1\left(t\right)=\frac{1}{\nu }{x}^{\nu {\mathfrak{p}}_m}{\left(x^{\nu }-t^{\nu }\right)}^{\mu +\gamma -1}{\int }_0^1{u}^{\mu -1}{\left(1-u\right)}^{\gamma -1}{\left(1-\left(1-\frac{t^{\nu }}{x^{\nu }}\right)u\right)}^{{\mathfrak{p}}_m}\hfill \\ \hfill & ·{}_{r+2}{F}_{r+1}\left[\begin{array}{c}{\lambda }_1,{\lambda }_2,\\ \mu ,\end{array}\begin{array}{c}(f_r+m_r)\\ \left(f_r\right)\end{array};\left(1-\frac{t^{\nu }}{x^{\nu }}\right)u\right]{}_2{F}_1\left[\begin{array}{c}-{\mathfrak{p}}_m,\lambda -\mu \\ \gamma \end{array};\frac{(1-u)(1-t^{\nu }/x^{\nu })}{1-u(1-t^{\nu }/x^{\nu })}\right]\mathrm{d}u.\hfill \end{array}$$

(27)

The right-hand side of (27) can be evaluated by using an Erdélyi-type integral established by Luo and Raina [21]. For $\Re \left(\gamma \right)>\Re \left(\mu \right)>0$ and $z\in \mathbb{C}\setminus [1,\infty )$, Luo and Raina proved that (Ref. [21], p. 482, Theorem 3.2)

$$\begin{array}{cc}\hfill & {}_{m+2}{F}_{m+1}\left[\begin{array}{cc}a,b,& \left({\vartheta }_m+1\right)\\ \gamma ,& \left({\vartheta }_m\right)\end{array};z\right]=\frac{\mathsf{\Gamma }\left(\gamma \right)}{\mathsf{\Gamma }\left(\mu \right)\mathsf{\Gamma }(\gamma -\mu )}{\int }_0^1{t}^{\mu -1}{\left(1-t\right)}^{\gamma -\mu -1}{\left(1-tz\right)}^{{\mathfrak{p}}_m}\hfill \\ \hfill & ·{}_{r+2}{F}_{r+1}\left[\begin{array}{cc}{\lambda }_1,{\lambda }_2,& (f_r+m_r)\\ \mu ,& \left(f_r\right)\end{array};zt\right]{}_2{F}_1\left[\begin{array}{c}-{\mathfrak{p}}_m,\lambda -\mu \\ \gamma -\mu \end{array};\frac{\left(1-t\right)z}{1-tz}\right]\mathrm{d}t,\hfill \end{array}$$

(28)

where ${\lambda }_1$, ${\lambda }_2$ and ${\mathfrak{p}}_m$ are given by (22) and $\left({\vartheta }_m\right)$ are the nonvanishing zeros of the parametric polynomial defined in (23). We note that the parametric polynomial is independent of parameter $\gamma$, and thus we may replace $\gamma$ by $\gamma +\mu$ (without changing the values of ${\lambda }_1$, ${\lambda }_2$, ${\mathfrak{p}}_m$ and ${\mathfrak{Q}}_m\left(t\right)$) in (28) to get

$$\begin{array}{cc}\hfill & {}_{m+2}{F}_{m+1}\left[\begin{array}{cc}a,b,& \left({\vartheta }_m+1\right)\\ \gamma +\mu ,& \left({\vartheta }_m\right)\end{array};z\right]=\frac{\mathsf{\Gamma }(\gamma +\mu )}{\mathsf{\Gamma }\left(\mu \right)\mathsf{\Gamma }\left(\gamma \right)}{\int }_0^1{t}^{\mu -1}{\left(1-t\right)}^{\gamma -1}{\left(1-tz\right)}^{{\mathfrak{p}}_m}\hfill \\ \hfill & ·{}_{r+2}{F}_{r+1}\left[\begin{array}{cc}{\lambda }_1,{\lambda }_2,& (f_r+m_r)\\ \mu ,& \left(f_r\right)\end{array};zt\right]{}_2{F}_1\left[\begin{array}{c}-{\mathfrak{p}}_m,\lambda -\mu \\ \gamma \end{array};\frac{\left(1-t\right)z}{1-tz}\right]\mathrm{d}t,\hfill \end{array}$$

(29)

where $min\{\Re (\gamma ),\Re (\mu \left)\right\}>0$.

Using the Erdélyi-type integral (29) in (27), we obtain

$${\Delta }_1\left(u\right)=\frac{\mathsf{\Gamma }\left(\mu \right)\mathsf{\Gamma }\left(\gamma \right)}{\mathsf{\Gamma }(\gamma +\mu )}{x}^{\nu {\mathfrak{p}}_m}{\left(x^{\nu }-t^{\nu }\right)}^{\mu +\gamma -1}{}_{m+2}{F}_{m+1}\left[\begin{array}{c}a,b,\\ \gamma +\mu ,\end{array}\begin{array}{c}\left({\vartheta }_m+1\right)\\ \left({\vartheta }_m\right)\end{array};1-\frac{t^{\nu }}{x^{\nu }}\right].$$

(30)

Finally, substituting (30) into (25), we get

$$\begin{array}{cc}\hfill \mathsf{\Phi }\left(x\right)& =\frac{\nu x^{-\delta -\rho }}{\mathsf{\Gamma }(\gamma +\mu )}{x}^{-\nu \left(\mu +\gamma +(h-\gamma -{\mathfrak{p}}_m-\rho /\nu )\right)}{\int }_0^x{\left(x^{\nu }-t^{\nu }\right)}^{\mu +\gamma -1}\hfill \\ \hfill & ·{}_{m+2}{F}_{m+1}\left[\begin{array}{c}a,b,\\ \gamma +\mu ,\end{array}\begin{array}{c}\left({\vartheta }_m+1\right)\\ \left({\vartheta }_m\right)\end{array};1-\frac{t^{\nu }}{x^{\nu }}\right]\phi \left(t\right)t^{\nu (h-\gamma -{\mathfrak{p}}_m-\rho /\nu )+\nu -1}\mathrm{d}t\hfill \\ \hfill & =\left(\mathcal{I}\begin{array}{cc}\gamma +\mu ;a,b:& ({\vartheta }_m+1)\\ h-\gamma -{\mathfrak{p}}_m-\rho /\nu ;\nu ,\delta +\rho :& \left({\vartheta }_m\right)\end{array}\phi \right)\left(x\right),\hfill \end{array}$$

which is the desired right-hand side of (24).

Now, we validate the interchanging of the integration. It is sufficient to show that

$$I={\int }_0^x{\left(x^{\nu }-s^{\nu }\right)}^{\Re \left(\mu \right)-1}\left|{}_{r+2}{F}_{r+1}\left[\begin{array}{c}{\lambda }_1,{\lambda }_2,\\ \mu ,\end{array}\begin{array}{c}(f_r+m_r)\\ \left(f_r\right)\end{array};1-\frac{s^{\nu }}{x^{\nu }}\right]\right|s^{\nu +\Re \left(\nu {\mathfrak{p}}_m\right)-1}{\Delta }_2\left(s\right)\mathrm{d}s<\infty ,$$

where

$$\begin{array}{cc}\hfill {\Delta }_2\left(s\right)& ={\int }_0^s{\left(s^{\nu }-t^{\nu }\right)}^{\Re \left(\gamma \right)-1}\left|{}_2{F}_1\left[\begin{array}{c}-{\mathfrak{p}}_m,\lambda -\mu \\ \gamma \end{array};1-\frac{t^{\nu }}{s^{\nu }}\right]\right|\left|\phi \left(t\right)\right|t^{\nu h-\Re (\nu \gamma +\nu {\mathfrak{p}}_m+\rho )+\nu -1}\mathrm{d}t\hfill \\ \hfill & =\frac{1}{\nu }{s}^{\nu h-\Re (\nu {\mathfrak{p}}_m+\rho )}{\int }_0^1{\left(1-u\right)}^{\Re \left(\gamma \right)-1}\left|{}_2{F}_1\left[\begin{array}{c}-{\mathfrak{p}}_m,\lambda -\mu \\ \gamma \end{array};1-u\right]\right|\hfill \\ \hfill & ·\left|\phi \left(s{u}^{1/\nu }\right)\right|u^{h-\Re (\gamma +{\mathfrak{p}}_m+\rho /\nu )-1}\mathrm{d}u.\hfill \end{array}$$

Note that (see Ref. [18], p. 63, Theorem 2.1.3 and [30], p. 387)

$${}_2{F}_1\left[\begin{array}{c}a,b\\ c\end{array};1-z\right]=\left\{\begin{array}{cc}\mathcal{O}\left(1\right),\hfill & \Re \left(c-a-b\right)>0;\hfill \\ \mathcal{O}\left(z^{\Re (c-a-b)}\right),\hfill & \Re \left(c-a-b\right)<0;\hfill \\ \mathcal{O}\left(logz\right),\hfill & a+b=c;\hfill \\ \mathcal{O}\left(z^{\Re (c-a-b)}\right)+\mathcal{O}\left(1\right),\hfill & \Re \left(c-a-b\right)=0,c\ne a+b\hfill \end{array}\right.$$

(31)

as $z\to 0^{+}$, so for each s, we have

$$\begin{array}{cc}\hfill {\Delta }_2\left(s\right)& \le D_1·s^{\nu h-\Re (\nu {\mathfrak{p}}_m+\rho )}{\int }_0^1{\left(1-u\right)}^{\Re \left(\gamma \right)-1}\hfill \\ \hfill & ·u^{h-\Re (\gamma +{\mathfrak{p}}_m+\rho /\nu )+min\{0,\Re (\gamma +{\mathfrak{p}}_m-\lambda +\mu )\}-1}\left|\phi \left(s{u}^{1/\nu }\right)\right|\mathrm{d}u,\hfill \end{array}$$

where $D_1$ is a positive number. In view of the definition of the Erdélyi–Kober operator (18), we have

$${\Delta }_2\left(s\right)\le D_2·s^{\nu h-\Re (\nu {\mathfrak{p}}_m+\rho )}F\left(s\right),$$

where $D_2:=D_1\mathsf{\Gamma }\left(\Re \left(\gamma \right)\right)$ ($\Re \left(\gamma \right)>0$) and

$$F\left(s\right):=\left(I_{+;\nu ,h-\Re (\gamma +{\mathfrak{p}}_m+\rho /\nu )+min\{0,\Re (\gamma +{\mathfrak{p}}_m-\lambda +\mu )\}-1}^{\Re \left(\gamma \right)}\left|\phi \right|\right)\left(s\right).$$

From Lemma 1, we have $F\in X_c^p$, since $\phi \in X_c^p$ and

$$h+min\left\{0,\Re (\gamma +{\mathfrak{p}}_m-\lambda +\mu )\right\}>\Re \left(\gamma +{\mathfrak{p}}_m+(\rho -c)/\nu \right).$$

For the generalized hypergeometric function ${}_{p+1}{F}_p\left[z\right]$, we have (see, for example Ref. [31], p. 149)

$${}_{p+1}{F}_p\left[\begin{array}{c}{a}_1,\cdots ,a_{p+1}\\ b_1,\cdots ,b_p\end{array};1-z\right]=\left\{\begin{array}{cc}\mathcal{O}\left(1\right),\hfill & \Re \left({\psi }_p\right)>0;\hfill \\ \mathcal{O}\left(z^{\Re \left({\psi }_p\right)}\right),\hfill & \Re \left({\psi }_p\right)<0;\hfill \\ \mathcal{O}\left(logz\right),\hfill & {\psi }_p=0,\hfill \end{array}\right.$$

(32)

as $z\to 0^{+}$, where ${\psi }_p:={\sum }_{\ell =1}^p{b}_{\ell }-{\sum }_{\ell =1}^{p+1}{a}_{\ell }$. Therefore, for each $x\in {\mathbb{R}}_{+}$, we find that

$$\begin{array}{cc}\hfill I& \le D_2{D}_3{x}^{-\nu min\{0,\Re (\mu -{\lambda }_1-{\lambda }_2)-m\}}\hfill \\ \hfill & ·{\int }_0^x{\left(x^{\nu }-s^{\nu }\right)}^{\Re \left(\mu \right)-1}{s}^{\nu min\{0,\Re (\mu -{\lambda }_1-{\lambda }_2)-m\}+\nu h+\nu -\Re \left(\rho \right)}F\left(s\right)\frac{\mathrm{d}s}{s}\hfill \\ \hfill & \le D_2{D}_3{x}^{-\nu min\{0,\Re (\mu -{\lambda }_1-{\lambda }_2)-m\}}{\parallel F\parallel }_{X_c^p}\hfill \\ \hfill & ·{\left({\int }_0^x{\left(x^{\nu }-s^{\nu }\right)}^{p^{\prime }\Re \left(\mu \right)-p^{\prime }}{s}^{p^{\prime }\nu min\{0,\Re (\mu -{\lambda }_1-{\lambda }_2)-m\}+p^{\prime }\nu (h+1)-p^{\prime }\Re \left(\rho \right)-p^{\prime }c-1}\mathrm{d}s\right)}^{1/p^{\prime }}\hfill \\ \hfill & \le D_2{D}_3{v}^{-1/p^{\prime }}{x}^{\nu \Re (\mu -\rho /\nu )+\nu h-c}{\parallel F\parallel }_{X_c^p}\hfill \\ \hfill & ·{\left({\int }_0^1{\left(1-u\right)}^{p^{\prime }\Re \left(\mu \right)-p^{\prime }}{u}^{p^{\prime }min\{0,\Re (\mu -{\lambda }_1-{\lambda }_2)-m\}+p^{\prime }(h+1)-p^{\prime }\Re \left(\rho \right)/\nu -p^{\prime }c/\nu -1}\mathrm{d}s\right)}^{1/p^{\prime }}\hfill \\ \hfill & <\infty ,\hfill \end{array}$$

where $D_3$ is a positive number, $1/p+1/p^{\prime }=1$, $p^{\prime }\Re \left(\mu \right)-p^{\prime }+1>0$ and

$$min\left\{0,\Re (\mu -{\lambda }_1-{\lambda }_2)-m\right\}+h+1>\Re \left((c+\rho )/\nu \right).$$

Thus, Fubini’s theorem is applicable and the proof is complete.    □

Remark 1.

When $r=0$, we can set $h=0$, $\nu =1$, $\mu =\alpha$, $\delta =\lambda -b-\alpha$ and $\rho =a+b-\lambda -\gamma$ in (24) to get

$$\begin{array}{cc}\hfill \left(\mathcal{I}\begin{array}{c}\alpha ;\lambda -a,\lambda -b\\ 0;1,\lambda -b-\alpha \end{array}\left(\mathcal{I}\begin{array}{c}\gamma ;a+b-\lambda ,\lambda -\alpha \\ 0;1,a+b-\lambda -\gamma \end{array}\phi \right)\right)\left(x\right)& =\left(\mathcal{I}\begin{array}{c}\alpha ;\lambda -b,\lambda -a\\ 0;1,\lambda -b-\alpha \end{array}\left(\mathcal{I}\begin{array}{c}\gamma ;a+b-\lambda ,\lambda -\alpha \\ 0;1,a+b-\lambda -\gamma \end{array}\phi \right)\right)\left(x\right)\hfill \\ \hfill & =\left(\mathcal{I}\begin{array}{c}\gamma +\alpha ;a,b\\ 0;1,a-\alpha -\gamma \end{array}\phi \right)\left(x\right).\hfill \end{array}$$

(33)

By comparing it with (16), we find that (33) is equivalent to the identity

$$\left(I^{\alpha ,\lambda -b-\alpha ,a-\lambda }\left(I^{\gamma ,a+b-\lambda -\gamma ,\alpha -\lambda }\phi \right)\right)\left(x\right)=\left(I^{\gamma +\alpha ,a-\gamma -\alpha ,-b}\phi \right)\left(x\right).$$

(34)

If we let further $a=\beta +\gamma +\delta +\alpha$, $b=\gamma +\delta -\eta$ and $\lambda =\beta -\eta +\gamma +\delta +\alpha$, then (34) reduces to (4).

Theorem 2.

Assume that $\phi \in X_c^p$. Let ${\lambda }_1$, ${\lambda }_2$, and ${\mathfrak{p}}_m$ be defined in (22). Let $\left({\vartheta }_m\right)$ be the nonvanishing zeros of the parametric polynomial ${\mathfrak{Q}}_m\left(t\right)$ defined in (23). Then for $\Re \left(\gamma \right)>0$, $\Re \left(\mu \right)>1/p>0$,

$$h+1+\Re \left((\rho +\delta )/\nu -{\mathfrak{p}}_m-\gamma \right)+min\left\{0,\Re (\gamma +{\mathfrak{p}}_m-\lambda +\mu )\right\}+(c-1)/\nu >0$$

and $1+h+(1+c)/\nu +min\left\{0,\Re (\mu -\lambda -{\mathfrak{p}}_m)\right\}+\Re \left((\rho +\delta )/\nu \right)>0$, we have

$$\left(\mathcal{J}\begin{array}{cc}\mu ;{\lambda }_1,{\lambda }_2:& (f_r+m_r)\\ h;\nu ,\delta :& \left(f_r\right)\end{array}\left(\mathcal{J}\begin{array}{c}\gamma ;-{\mathfrak{p}}_m,\lambda -\mu \\ h-\gamma -{\mathfrak{p}}_m+\delta /\nu ;\nu ,\rho \end{array}\phi \right)\right)\left(x\right)=\left(\mathcal{J}\begin{array}{cc}\gamma +\mu ;a,b:& ({\vartheta }_m+1)\\ h-{\mathfrak{p}}_m-\gamma ;\nu ,\delta +\rho :& \left({\vartheta }_m\right)\end{array}\phi \right)\left(x\right),$$

(35)

where $\mathcal{J}\begin{array}{cc}\mu ;a,b:& (f_r+m_r)\\ h;\nu ,\delta :& \left(f_r\right)\end{array}$ and $\mathcal{J}\begin{array}{c}\mu ;a,b\\ h;\nu ,\delta \end{array}$ are defined by (13) and (15), respectively.

Proof. 

Denote the left-hand side of (35) by $\mathsf{\Psi }\left(s\right)$. Then, following a similar procedure as described in the proof of Theorem 1, we have

$$\begin{array}{cc}\hfill \mathsf{\Psi }\left(s\right)& =\frac{\nu x^{\nu h+\nu -1}}{\mathsf{\Gamma }\left(\mu \right)}{\int }_x^{\infty }{\left(s^{\nu }-x^{\nu }\right)}^{\mu -1}{}_{r+2}{F}_{r+1}\left[\begin{array}{c}{\lambda }_1,{\lambda }_2,\\ \mu ,\end{array}\begin{array}{c}(f_r+m_r)\\ \left(f_r\right)\end{array};1-\frac{x^{\nu }}{s^{\nu }}\right]s^{-\delta -\nu (\mu +h)}\hfill \\ \hfill & ·\{\frac{\nu s^{\nu h+\nu -1}}{\mathsf{\Gamma }\left(\gamma \right)}{s}^{-\nu (\gamma +{\mathfrak{p}}_m)+\delta }{\int }_s^{\infty }{\left(t^{\nu }-s^{\nu }\right)}^{\gamma -1}{}_2{F}_1\left[\begin{array}{c}-{\mathfrak{p}}_m,\lambda -\mu \\ \gamma \end{array};1-\frac{s^{\nu }}{t^{\nu }}\right]\hfill \\ \hfill & ·\phi \left(t\right)t^{-\rho -\nu (h-{\mathfrak{p}}_m)-\delta }\mathrm{d}t\}\mathrm{d}s\hfill \\ \hfill & =\frac{{\nu }^2{x}^{\nu h+\nu -1}}{\mathsf{\Gamma }\left(\mu \right)\mathsf{\Gamma }\left(\gamma \right)}{\int }_x^{\infty }\phi \left(t\right)t^{-\rho -\nu (h-{\mathfrak{p}}_m)-\delta }{\Delta }_3\left(t\right)\mathrm{d}t,\hfill \end{array}$$

(36)

where

$$\begin{array}{cc}\hfill {\Delta }_3\left(t\right)& ={\int }_x^t{s}^{\nu -1-\nu (\mu +\gamma +{\mathfrak{p}}_m)}{\left(s^{\nu }-x^{\nu }\right)}^{\mu -1}{\left(t^{\nu }-s^{\nu }\right)}^{\gamma -1}\hfill \\ \hfill & ·{}_{r+2}{F}_{r+1}\left[\begin{array}{c}{\lambda }_1,{\lambda }_2,\\ \mu ,\end{array}\begin{array}{c}(f_r+m_r)\\ \left(f_r\right)\end{array};1-\frac{x^{\nu }}{s^{\nu }}\right]{}_2{F}_1\left[\begin{array}{c}-{\mathfrak{p}}_m,\lambda -\mu \\ \gamma \end{array};1-\frac{s^{\nu }}{t^{\nu }}\right]\mathrm{d}s.\hfill \end{array}$$

Letting

$$s=\frac{tx}{{\left(t^{\nu }+(x^{\nu }-t^{\nu })u\right)}^{1/\nu }},$$

so that

$$\mathrm{d}s=\frac{1}{\nu }tx\left(t^{\nu }-x^{\nu }\right){\left(t^{\nu }+(x^{\nu }-t^{\nu })u\right)}^{-1-1/\nu }\mathrm{d}u\mathrm{and}u=\frac{t^{\nu }(x^{\nu }-s^{\nu })}{s^{\nu }(x^{\nu }-t^{\nu })}\in \left(0,1\right),$$

we have

$$\begin{array}{cc}\hfill {\Delta }_3\left(t\right)& =\frac{1}{\nu }{\left(t^{\nu }-x^{\nu }\right)}^{\mu +\gamma -1}{t}^{-\nu \mu }{x}^{-\nu (\gamma +{\mathfrak{p}}_m)}{\int }_0^1{u}^{\mu -1}{\left(1-u\right)}^{\gamma -1}{\left(1-\left(1-\frac{x^{\nu }}{t^{\nu }}\right)u\right)}^{{\mathfrak{p}}_m}\hfill \\ \hfill & ·{}_{r+2}{F}_{r+1}\left[\begin{array}{c}{\lambda }_1,{\lambda }_2,\\ \mu ,\end{array}\begin{array}{c}(f_r+m_r)\\ \left(f_r\right)\end{array};\left(1-\frac{x^{\nu }}{t^{\nu }}\right)u\right]{}_2{F}_1\left[\begin{array}{c}-{\mathfrak{p}}_m,\lambda -\mu \\ \gamma \end{array};\frac{(1-u)(1-x^{\nu }/t^{\nu })}{1-u(1-x^{\nu }/t^{\nu })}\right]\mathrm{d}u.\hfill \end{array}$$

The use of Erdélyi-type integral (29) gives

$${\Delta }_3\left(t\right)=\frac{1}{\nu }\frac{\mathsf{\Gamma }\left(\mu \right)\mathsf{\Gamma }\left(\gamma \right)}{\mathsf{\Gamma }(\gamma +\mu )}{\left(t^{\nu }-x^{\nu }\right)}^{\mu +\gamma -1}{t}^{-\nu \mu }{x}^{-\nu (\gamma +{\mathfrak{p}}_m)}{}_{m+2}{F}_{m+1}\left[\begin{array}{cc}a,b,& \left({\vartheta }_m+1\right)\\ \gamma +\mu ,& \left({\vartheta }_m\right)\end{array};1-\frac{x^{\nu }}{t^{\nu }}\right],$$

and thus (36) becomes

$$\begin{array}{cc}\hfill \mathsf{\Psi }\left(s\right)& =\frac{\nu x^{\nu (h-\gamma -{\mathfrak{p}}_m)+\nu -1}}{\mathsf{\Gamma }(\mu +\gamma )}{\int }_x^{\infty }{\left(t^{\nu }-x^{\nu }\right)}^{\mu +\gamma -1}{}_{m+2}{F}_{m+1}\left[\begin{array}{cc}a,b,& \left({\vartheta }_m+1\right)\\ \gamma +\mu ,& \left({\vartheta }_m\right)\end{array};1-\frac{x^{\nu }}{t^{\nu }}\right]\hfill \\ \hfill & ·\phi \left(t\right)t^{-(\rho +\delta )-\nu (\mu +\gamma +h-\gamma -{\mathfrak{p}}_m)}\mathrm{d}t\hfill \\ \hfill & =\left(\mathcal{J}\begin{array}{cc}\gamma +\mu ;a,b:& ({\vartheta }_m+1)\\ h-{\mathfrak{p}}_m-\gamma ;\nu ,\delta +\rho :& \left({\vartheta }_m\right)\end{array}\phi \right)\left(x\right),\hfill \end{array}$$

where $\left({\vartheta }_m\right)$ are the nonvanishing zeros of the parametric polynomial (23).

As in the proof of Theorem 1, we verify the validity of interchanging the order of integration by checking the finiteness of the integral

$$I={\int }_x^{\infty }{\left(s^{\nu }-x^{\nu }\right)}^{\Re \left(\mu \right)-1}\left|{}_{r+2}{F}_{r+1}\left[\begin{array}{c}{\lambda }_1,{\lambda }_2,\\ \mu ,\end{array}\begin{array}{c}(f_r+m_r)\\ \left(f_r\right)\end{array};1-\frac{x^{\nu }}{s^{\nu }}\right]\right|s^{\nu -1-\Re (\nu \mu +\nu \gamma +\nu {\mathfrak{p}}_m)}{\Delta }_4\left(s\right)\mathrm{d}s,$$

where

$$\begin{array}{cc}\hfill {\Delta }_4\left(s\right)& ={\int }_s^{\infty }{\left(t^{\nu }-s^{\nu }\right)}^{\Re \left(\gamma \right)-1}\left|{}_2{F}_1\left[\begin{array}{c}-{\mathfrak{p}}_m,\lambda -\mu \\ \gamma \end{array};1-\frac{s^{\nu }}{t^{\nu }}\right]\right|\left|\phi \left(t\right)\right|t^{-\Re (\rho +\delta -\nu {\mathfrak{p}}_m)-\nu h}\mathrm{d}t\hfill \\ \hfill & =\frac{1}{\nu }{s}^{1-\nu -\Re (\rho +\delta -\nu \gamma -\nu {\mathfrak{p}}_m)-\nu h}{\int }_1^{\infty }{\left(u-1\right)}^{\Re \left(\gamma \right)-1}\left|{}_2{F}_1\left[\begin{array}{c}-{\mathfrak{p}}_m,\lambda -\mu \\ \gamma \end{array};1-\frac{1}{u}\right]\right|\hfill \\ \hfill & ·|\phi \left(s{u}^{1/\nu }\right)|u^{1/\nu -1-\Re (\rho /\nu +\delta /\nu -{\mathfrak{p}}_m)-h}\mathrm{d}u.\hfill \end{array}$$

Using (31) gives

$$\begin{array}{cc}\hfill {\Delta }_4\left(s\right)& \le D_4·s^{1-\nu -\Re (\rho +\delta -\nu \gamma -\nu {\mathfrak{p}}_m)-\nu h}{\int }_1^{\infty }{\left(u-1\right)}^{\Re \left(\gamma \right)-1}\hfill \\ \hfill & ·\left|\phi \left(s{u}^{1/\nu }\right)\right|u^{-\Re \left(\gamma \right)-h+\Re \left(\gamma \right)+1/\nu -1-\Re (\rho /\nu +\delta /\nu -{\mathfrak{p}}_m)-min\{0,\Re (\gamma +{\mathfrak{p}}_m-\lambda +\mu )\}}\mathrm{d}u,\hfill \end{array}$$

where $D_4$ is a positive number. Thus we have

$${\Delta }_4\left(s\right)\le D_4·s^{1-\nu -\Re (\rho +\delta -\nu \gamma -\nu {\mathfrak{p}}_m)-\nu h}G\left(s\right),$$

where $D_5:=D_4\mathsf{\Gamma }\left(\Re \left(\gamma \right)\right)$ ($\Re \left(\gamma \right)>0$) and

$$G\left(s\right):=\left(I_{-;\nu ,h-1/\nu +1+\Re (\rho /\nu +\delta /\nu -{\mathfrak{p}}_m-\gamma )+min\{0,\Re (\gamma +{\mathfrak{p}}_m-\lambda +\mu )\}}^{\Re \left(\gamma \right)}\left|\phi \right|\right)\left(s\right)\in X_c^p.$$

Then from (32) we have

$$\begin{array}{cc}\hfill I& \le D_5{D}_6{x}^{\nu min\{0,\Re (\mu -{\lambda }_1-{\lambda }_2)-m\}}\hfill \\ \hfill & ·{\int }_x^{\infty }{\left(s^{\nu }-x^{\nu }\right)}^{\Re \left(\mu \right)-1}{s}^{-\nu min\{0,\Re (\mu -{\lambda }_1-{\lambda }_2)-m\}-\nu h-\Re (\rho +\delta +\nu \mu )-1}G\left(s\right)\frac{\mathrm{d}s}{s}\hfill \\ \hfill & \le D_5{D}_6{x}^{\nu min\{0,\Re (\mu -{\lambda }_1-{\lambda }_2)-m\}}{\parallel G\parallel }_{X_c^p}\hfill \\ \hfill & ·{\left({\int }_x^{\infty }{\left(s^{\nu }-x^{\nu }\right)}^{p^{\prime }\Re \left(\mu \right)-p^{\prime }}{s}^{-p^{\prime }\nu min\{0,\Re (\mu -{\lambda }_1-{\lambda }_2)-m\}-p^{\prime }\nu h-p^{\prime }\Re (\rho +\delta +\nu \mu )-p^{\prime }-p^{\prime }c-1}\mathrm{d}s\right)}^{1/p^{\prime }}\hfill \\ \hfill & \le D_5{D}_6{x}^{-\nu -\nu h-\Re (\rho +\delta )-1-c-1/p^{\prime }}{\parallel G\parallel }_{X_c^p}\hfill \\ \hfill & ·{\left({\int }_1^{\infty }{\left(u-1\right)}^{p^{\prime }\Re \left(\mu \right)-p^{\prime }}{u}^{-p^{\prime }min\{0,\Re (\mu -{\lambda }_1-{\lambda }_2)-m\}-p^{\prime }h-p^{\prime }\Re ((\rho +\delta )/\nu +\mu )-p^{\prime }(1+c)/\nu -1}\mathrm{d}s\right)}^{1/p^{\prime }}\hfill \\ \hfill & <\infty .\hfill \end{array}$$

This completes the proof.   □

Remark 2.

When $r=0$, we can set $h=0$, $\nu =1$, $\delta =\gamma +\lambda -a-b$ and $\rho =a+b-\lambda -\gamma$ in (35) to get

$$\left(\mathcal{J}\begin{array}{c}\mu ;\lambda -b,\lambda -a\\ 0;1,\gamma +\lambda -a-b\end{array}\left(I^{\gamma ,a+b-\lambda -\gamma ,\mu -\lambda }\phi \right)\right)\left(x\right)=\left(\mathcal{J}\begin{array}{c}\gamma +\mu ;a,b\\ a+b-\lambda -\gamma ;1,0\end{array}\phi \right)\left(x\right).$$

(37)

Letting further $a=\mu +\gamma$ in (37), we have

$$\begin{array}{cc}\hfill \left(I^{\mu ,\lambda -\mu -b,\gamma +\mu -\lambda }\left(I^{\gamma ,\mu +b-\lambda ,\mu -\lambda }\phi \right)\right)\left(x\right)& =\left(\mathcal{J}\begin{array}{c}\gamma +\mu ;\gamma +\mu ,b\\ \mu +b-\lambda ;1,0\end{array}\phi \right)\left(x\right)\hfill \\ \hfill & =\left(I_{-;1,\mu -\lambda }^{\gamma +\mu }\phi \right)\left(x\right)=\left(K_{\mu -\lambda ,\gamma +\mu }^{-}\phi \right)\left(x\right).\hfill \end{array}$$

(38)

Additionally, by putting $b=\beta +\lambda -\mu$ in (38) and then letting $\lambda =\mu -\eta$ in the resulting equation we get the following clearer form

$$\left(I^{\mu ,-\beta ,\gamma +\eta }\left(I^{\gamma ,\beta ,\eta }\phi \right)\right)\left(x\right)=\left(K_{\eta ,\gamma +\mu }^{-}\phi \right)\left(x\right),$$

which is a special case of (5) when $\delta =-\beta$. It does not seem possible to deduce (5) by merely specializing the parameters in (35). Therefore, it should be interesting to find a composition formula from (35) which may include (5) or (6) as particular cases.

As depicted in Theorems 1 and 2, the study of the composition structure of the operators $\mathcal{I}$ and $\mathcal{J}$ rests heavily on the existence of a suitable Erdélyi-type integral, because we derive (24) and (35) from the Erdélyi-type integral (29). However, there may possibly be an alternative approach by which the Erdelyi-type integral may be obtained from a known composition structure [1] (see also Refs. [22,32]). Such an approach may be of special interest since our operators involve the generalized hypergeometric function ${}_{r+2}{F}_{r+1}$ and the methodology may lead to some new results.

3.2. Derivative Formula

In this section we derive a derivative formula involving the fractional integral operator (12).

We introduce here some notations describing necessary concepts that would be used in the sequel. Let $\left({\xi }_m\right)$ be the nonvanishing zeros of the parametric polynomial $Q_m\left(t\right)$ of degree m defined by

$$Q_m\left(t\right)=\sum _{j=0}^m{\sigma }_{m-j}\sum _{k=0}^j\left\{\begin{array}{c}j\\ k\end{array}\right\}{\left(b\right)}_k{\left(t\right)}_k{(\mu -b-t)}_{m-k},$$

(39)

where the ${\sigma }_j$ ($0\le j\le m$) are determined by the generating relation (10). We define the parametric polynomial ${\tilde{Q}}_m\left(t\right)$ by

$${\tilde{Q}}_m\left(t\right)=\sum _{j=0}^m{\tilde{\sigma }}_{m-j}\sum _{k=0}^j\left\{\begin{array}{c}j\\ k\end{array}\right\}{(\mu -b-m)}_k{\left(t\right)}_k{(b+m-n-t)}_{m-k},$$

(40)

where ${\tilde{\sigma }}_j$$(0\le j\le m)$ are determined by the generating relation

$$\prod _{j=1}^m({\xi }_j+x)=\sum _{j=0}^m{\tilde{\sigma }}_{m-j}{x}^j.$$

(41)

Theorem 3.

For $\Re \left(\mu \right)>n$$(n\in \mathbb{N})$, we have

$$\begin{array}{cc}\hfill & \frac{{\partial }^n}{\partial x^n}\left\{x^{\delta +\nu (\mu -a+h)}\left(\mathcal{I}\begin{array}{cc}\mu ;a,b:& (f_r+m_r)\\ h;\nu ,\delta :& \left(f_r\right)\end{array}\phi \right)\left(x\right)\right\}\hfill \\ \hfill & ={\nu }^n{x}^{\delta +\nu (\mu -n-a+h)}\left(\mathcal{I}\begin{array}{cc}\mu -n;a,b-n:& ({\eta }_m+1)\\ h;\nu ,\delta :& \left({\eta }_m\right)\end{array}\phi \right)\left(x\right),\hfill \end{array}$$

(42)

where $\left({\eta }_m\right)$ are the nonvanishing zeros of the parametric polynomial ${\tilde{Q}}_m\left(t\right)$ given by (40).

Proof. 

Using the Euler-type transformation due to Miller and Paris [20], p. 305, Theorem 3

$$\begin{array}{c}\hfill {}_{r+2}{F}_{r+1}\left[\begin{array}{c}a,b,\\ \mu ,\end{array}\begin{array}{c}(f_r+m_r)\\ \left(f_r\right)\end{array};x\right]={\left(1-x\right)}^{-a}{}_{m+2}{F}_{m+1}\left[\begin{array}{c}a,\mu -b-m,\\ \mu ,\end{array}\begin{array}{c}({\xi }_m+1)\\ \left({\xi }_m\right)\end{array};\frac{x}{x-1}\right],\end{array}$$

(43)

we have

$$\begin{array}{cc}\hfill & x^{\delta +\nu (\mu -a+h)}\left(\mathcal{I}\begin{array}{cc}\mu ;a,b:& (f_r+m_r)\\ h;\nu ,\delta :& \left(f_r\right)\end{array}\phi \right)\left(x\right)\hfill \\ \hfill & =\frac{\nu }{\mathsf{\Gamma }\left(\mu \right)}{\int }_0^x{\left(x^{\nu }-s^{\nu }\right)}^{\mu -1}{}_{m+2}{F}_{m+1}\left[\begin{array}{c}a,\mu -b-m,\\ \mu ,\end{array}\begin{array}{c}({\xi }_m+1)\\ \left({\xi }_m\right)\end{array};1-\frac{x^{\nu }}{s^{\nu }}\right]\phi \left(s\right)s^{\nu (h-a)+\nu -1}\mathrm{d}s,\hfill \end{array}$$

where $\left({\xi }_m\right)$ are the nonvanishing zeros of the parametric polynomial $Q_m\left(t\right)$ defined by (39). By making use of the Leibniz integral rule, we obtain

$$\begin{array}{cc}\hfill & \frac{\partial }{\partial x}\left\{x^{\delta +\nu (\mu -a+h)}\left(\mathcal{I}\begin{array}{cc}\mu ;a,b:& (f_r+m_r)\\ h;\nu ,\delta :& \left(f_r\right)\end{array}\phi \right)\left(x\right)\right\}\hfill \\ \hfill & =\frac{\nu }{\mathsf{\Gamma }\left(\mu \right)}\frac{\partial }{\partial x}{\int }_0^x{\left(x^{\nu }-s^{\nu }\right)}^{\mu -1}{}_{m+2}{F}_{m+1}\left[\begin{array}{c}a,\mu -b-m,\\ \mu ,\end{array}\begin{array}{c}({\xi }_m+1)\\ \left({\xi }_m\right)\end{array};1-\frac{x^{\nu }}{s^{\nu }}\right]\phi \left(s\right)s^{\nu (h-a)+\nu -1}\mathrm{d}s\hfill \\ \hfill & =\frac{\nu }{\mathsf{\Gamma }\left(\mu \right)}{\int }_0^x\frac{\partial }{\partial x}\left\{{\left(x^{\nu }-s^{\nu }\right)}^{\mu -1}{}_{m+2}{F}_{m+1}\left[\begin{array}{c}a,\mu -b-m,\\ \mu ,\end{array}\begin{array}{c}({\xi }_m+1)\\ \left({\xi }_m\right)\end{array};1-\frac{s^{\nu }}{x^{\nu }}\right]\right\}\hfill \\ \hfill & ·\phi \left(s\right)s^{\nu (h-a)+\nu -1}\mathrm{d}s.\hfill \end{array}$$

Taking into account the formula [33], p. 442, Equation (51)

$$\frac{{\partial }^n}{\partial z^n}\left\{z^{\sigma -1}{}_p{F}_q\left[\begin{array}{c}\left(a_p\right)\\ \left(b_{q-1}\right),\sigma \end{array};z\right]\right\}={(\sigma -n)}_n{z}^{\sigma -n-1}{}_p{F}_q\left[\begin{array}{c}\left(a_p\right)\\ \left(b_{q-1}\right),\sigma -n\end{array};z\right],$$

(44)

we have

$$\begin{array}{cc}\hfill & \frac{\partial }{\partial x}\left\{x^{\delta +\nu (\mu -a+h)}\left(\mathcal{I}\begin{array}{cc}\mu ;a,b:& (f_r+m_r)\\ h;\nu ,\delta :& \left(f_r\right)\end{array}\phi \right)\left(x\right)\right\}=\frac{{\nu }^2}{\mathsf{\Gamma }(\mu -1)}{\int }_0^x{\left(x^{\nu }-s^{\nu }\right)}^{(\mu -1)-1}\hfill \\ \hfill & ·{}_{m+2}{F}_{m+1}\left[\begin{array}{c}a,\mu -b-m,\\ \mu -1,\end{array}\begin{array}{c}({\xi }_m+1)\\ \left({\xi }_m\right)\end{array};1-\frac{x^{\nu }}{s^{\nu }}\right]\phi \left(s\right)s^{\nu (h-a)+\nu -1}\mathrm{d}s.\hfill \end{array}$$

Next, differentiating n times, we obtain

$$\begin{array}{cc}\hfill & \frac{{\partial }^n}{\partial x^n}\left\{x^{\delta +\nu (\mu -a+h)}\left(\mathcal{I}\begin{array}{cc}\mu ;a,b:& (f_r+m_r)\\ h;\nu ,\delta :& \left(f_r\right)\end{array}\phi \right)\left(x\right)\right\}=\frac{{\nu }^{1+n}}{\mathsf{\Gamma }(\mu -n)}{\int }_0^x{\left(x^{\nu }-s^{\nu }\right)}^{\mu -n-1}\hfill \\ \hfill & ·{}_{m+2}{F}_{m+1}\left[\begin{array}{c}a,\mu -b-m,\\ \mu -n,\end{array}\begin{array}{c}({\xi }_m+1)\\ \left({\xi }_m\right)\end{array};1-\frac{x^{\nu }}{s^{\nu }}\right]\phi \left(s\right)s^{\nu (h-a)+\nu -1}\mathrm{d}s.\hfill \end{array}$$

By applying the Euler-type transformation (43) again, we get

$$\begin{array}{cc}\hfill & \frac{{\partial }^n}{\partial x^n}\left\{x^{\delta +\nu (\mu -a+h)}\left(\mathcal{I}\begin{array}{cc}\mu ;a,b:& (f_r+m_r)\\ h;\nu ,\delta :& \left(f_r\right)\end{array}\phi \right)\left(x\right)\right\}\hfill \\ \hfill & =\frac{{\nu }^{1+n}{x}^{-\nu a}}{\mathsf{\Gamma }(\mu -n)}{\int }_0^x{\left(x^{\nu }-s^{\nu }\right)}^{\mu -n-1}{}_{m+2}{F}_{m+1}\left[\begin{array}{c}a,b-n,\\ \mu -n,\end{array}\begin{array}{c}({\eta }_m+1)\\ \left({\eta }_m\right)\end{array};1-\frac{s^{\nu }}{x^{\nu }}\right]\phi \left(s\right)s^{\nu h+\nu -1}\mathrm{d}s\hfill \\ \hfill & ={\nu }^n{x}^{\delta +\nu (\mu -n-a+h)}\left(\mathcal{I}\begin{array}{cc}\mu -n;a,b-n:& ({\eta }_m+1)\\ h;\nu ,\delta :& \left({\eta }_m\right)\end{array}\phi \right)\left(x\right),\hfill \end{array}$$

(45)

where the sequence of parameters $\left({\eta }_m\right)$ are the nonvanishing zeros of the parametric polynomial ${\tilde{Q}}_m\left(t\right)$ of degree m given by (40). This completes the proof of (42).    □

Before proceeding further, we consider here a simple example.

Example 1.

When $r=1$ and $m=m_1=1$, $f_1=f$ and ${\eta }_1=\eta$ in (42), we get

$$\frac{{\partial }^n}{\partial x^n}\left\{x^{\delta +\nu (\mu -a+h)}\left(\mathcal{I}\begin{array}{cc}\mu ;a,b:& f+1\\ h;\nu ,\delta :& f\end{array}\phi \right)\left(x\right)\right\}={\nu }^n{x}^{\delta +\nu (\mu -n-a+h)}\left(\mathcal{I}\begin{array}{cc}\mu -n;a,b-n:& \eta +1\\ h;\nu ,\delta :& \eta \end{array}\phi \right)\left(x\right),$$

(46)

where η is the nonvanishing zero of the parametric polynomial

$$\begin{array}{cc}\hfill {\tilde{Q}}_1\left(t\right)& =\sum _{j=0}^1{\tilde{\sigma }}_{1-j}\sum _{k=0}^j\left\{\begin{array}{c}j\\ k\end{array}\right\}{(\mu -b-1)}_k{\left(t\right)}_k{(b+1-n-t)}_{1-k}\hfill \\ \hfill & ={\tilde{\sigma }}_1\left\{\begin{array}{c}0\\ 0\end{array}\right\}(b+1-n-t)+{\tilde{\sigma }}_0\left\{\begin{array}{c}1\\ 0\end{array}\right\}(b+1-n-t)+{\tilde{\sigma }}_0\left\{\begin{array}{c}1\\ 1\end{array}\right\}(\mu -b-1)t\hfill \\ \hfill & ={\tilde{\sigma }}_1(b+1-n)+[{\tilde{\sigma }}_0(\mu -b-1)-{\tilde{\sigma }}_1]t.\hfill \end{array}$$

Therefore, η can be expressed as

$$\eta =\frac{{\tilde{\sigma }}_1(b+1-n)}{{\tilde{\sigma }}_1-{\tilde{\sigma }}_0(\mu -b-1)}.$$

It follows from (41) that ${\tilde{\sigma }}_0=1$ and ${\tilde{\sigma }}_1=\xi$, where ξ is the nonvanishing zero of the parametric polynomial

$$Q_1\left(t\right)={\sigma }_1(\mu -b)+[{\sigma }_{0}b-{\sigma }_1]t.$$

From (10), we have ${\sigma }_0=1$ and ${\sigma }_1=f$ and thus ξ can be written as $\xi =f(\mu -b)/(f-b)$. Hence,

$$\eta =\frac{f(\mu -b)(b+1-n)}{f+b(\mu -b-1)},$$

wherein we note that η depends on n.

It may be observed that the Euler-type transformation (43) is used twice, so we need to be careful while finding special cases of Theorem 3.

(i)

By letting $b=n$ ($n\in \mathbb{N}$) in (42) and noting that ${}_{m+2}{F}_{m+1}$-function in (42) reduces to 1, we get

$$\begin{array}{cc}\hfill & \frac{{\partial }^n}{\partial x^n}\left\{x^{\delta +\nu (\mu -a+h)}\left(\mathcal{I}\begin{array}{cc}\mu ;a,n:& (f_r+m_r)\\ h;\nu ,\delta :& \left(f_r\right)\end{array}\phi \right)\left(x\right)\right\}\hfill \\ \hfill & =\frac{{\nu }^{1+n}{x}^{-\nu a}}{\mathsf{\Gamma }(\mu -n)}{\int }_0^x{\left(x^{\nu }-s^{\nu }\right)}^{\mu -n-1}\phi \left(s\right)s^{\nu h+\nu -1}\mathrm{d}s\hfill \\ \hfill & ={\nu }^n{x}^{\nu (\mu -n-a+h)}\left(I_{+;\nu ,h}^{\mu -n}\phi \right)\left(x\right),\hfill \end{array}$$

(47)

where $I_{+;\nu ,h}^{\mu -n}$ denotes the Erdélyi–Kober type fractional integral defined by (18).

In fact, letting $b=n$ changes the parametric polynomials $Q_m\left(t\right)$ and ${\tilde{Q}}_m\left(t\right)$ defined by (39) and (40), respectively. However, if the new polynomials, say $Q_m^{*}\left(t\right)$ and ${\tilde{Q}}_m^{*}\left(t\right)$, also have nonvanishing zeros, denoted by $\left({\xi }_m^{*}\right)$ and $\left({\eta }_m^{*}\right)$ respectively, then (47) holds true. To illustrate here, let us set $b=n$ in Example 1, then $Q_1\left(t\right)$ becomes $Q_1^{*}\left(t\right)=f(\mu -n)+(n-f)t$ with ${\xi }^{*}=f(\mu -n)/(f-n)$ its nonvanishing zero and ${\tilde{Q}}_1\left(t\right)$ becomes ${\tilde{Q}}_1^{*}\left(t\right)={\xi }^{*}+(\mu -n-1-{\xi }^{*})t$. The nonvanishing zero of ${\tilde{Q}}_1^{*}\left(t\right)$ is

$${\eta }^{*}=\frac{f(\mu -n)}{f+n(\mu -n-1)}(f\ne 0,\mu \ne n).$$

Therefore, we obtain from (46) that

$$\begin{array}{cc}\hfill \frac{{\partial }^n}{\partial x^n}\left\{x^{\delta +\nu (\mu -a+h)}\left(\mathcal{I}\begin{array}{cc}\mu ;a,n:& f+1\\ h;\nu ,\delta :& f\end{array}\phi \right)\left(x\right)\right\}& ={\nu }^n{x}^{\delta +\nu (\mu -n-a+h)}\left(\mathcal{I}\begin{array}{cc}\mu -n;a,0:& {\eta }^{*}+1\\ h;\nu ,\delta :& {\eta }^{*}\end{array}\phi \right)\left(x\right)\hfill \\ \hfill & ={\nu }^n{x}^{\nu (\mu -n-a+h)}\left(I_{+;\nu ,h}^{\mu -n}\phi \right)\left(x\right).\hfill \end{array}$$

We also observe that the subsitution $b=n$ may always reduce the right-hand side of (42) to a Erdélyi–Kober type integral.

(ii)

When $r=0$, then in view of (14) and (42), we simply obtain

$$\frac{{\partial }^n}{\partial x^n}\left\{x^{\delta +\nu (\mu -a+h)}\left(\mathcal{I}\begin{array}{c}\mu ;a,b\\ h;\nu ,\delta \end{array}\phi \right)\left(x\right)\right\}={\nu }^n{x}^{\delta +\nu (\mu -n-a+h)}\left(\mathcal{I}\begin{array}{c}\mu -n;a,b-n\\ h;\nu ,\delta \end{array}\phi \right)\left(x\right).$$

(48)

Further, if $h=0$, $\nu =1$, $\delta =\beta$, $a=\alpha +\beta$, $b=-\eta +n$ and $\mu =\alpha +n$ in (48), we then have

$$\frac{{\partial }^n}{\partial x^n}\left\{x^n\left(\mathcal{I}\begin{array}{c}\alpha +n;\alpha +\beta ,-\eta +n\\ 0;1,\beta \end{array}\phi \right)\left(x\right)\right\}=\left(\mathcal{I}\begin{array}{c}\alpha ;\alpha +\beta ,-\eta \\ 0;1,\beta \end{array}\phi \right)\left(x\right).$$

In addition, in view of (16) and the relation

$$\begin{array}{cc}\hfill x^n\left(\mathcal{I}\begin{array}{c}\alpha +n;\alpha +\beta ,-\eta +n\\ 0;1,\beta \end{array}\phi \right)\left(x\right)& =\frac{x^{-\beta -\alpha }}{\mathsf{\Gamma }\left(\alpha +n\right)}{\int }_0^x{\left(x-s\right)}^{\alpha +n-1}{}_2{F}_1\left[\begin{array}{c}\alpha +\beta ,-\eta +n\\ \alpha +n\end{array};1-\frac{s}{x}\right]\phi \left(s\right)\mathrm{d}s\hfill \\ \hfill & =\left(I_{0,x}^{\alpha +n,\beta -n,\eta -n}\phi \right)\left(x\right),\hfill \end{array}$$

we note the following interesting and remarkable relation:

$$\frac{{\partial }^n}{\partial x^n}\left(I_{0,x}^{\alpha +n,\beta -n,\eta -n}\phi \right)\left(x\right)=\left(I_{0,x}^{\alpha ,\beta ,\eta }\phi \right)\left(x\right),$$

which serves as the definition of Saigo’s generalized fractional derivative (see Ref. [3], p. 8, Equation (3.2)).

4. Relationship with Khudozhnikov’s Work

In a very short paper, Khudozhnikov [17] considered in a certain class of integrable functions the following Volterra-type integral equation

$${\int }_a^x\frac{{\left(x-s\right)}^{\gamma -1}}{\mathsf{\Gamma }\left(\gamma \right)}{}_3{F}_2\left[\begin{array}{c}\alpha ,\beta ,\\ \gamma ,\end{array}\begin{array}{c}\epsilon +m\\ \epsilon \end{array};1-\frac{x}{s}\right]\phi \left(s\right)\mathrm{d}s=g\left(x\right),$$

(49)

where $0<\Re \left(\gamma \right)<1$, $m\in \mathbb{N}$ and $0<a\le x\le b<+\infty$. By using some known formulas from Ref. [33], Khudozhnikov obtained the following result [17], p. 79, Equation (2).

Theorem 4

(Khudozhnikov). The Volterra-type integral Equation (49) can be reduced to the following system of differential and integral equations:

$$\left\{\begin{array}{c}{\displaystyle \sum _{k=0}^m\left(\begin{array}{c}m\\ k\end{array}\right)\frac{{\left(\alpha \right)}_k{\left(\beta \right)}_k}{{\left(\epsilon \right)}_k{(-x)}^k}{y}^{(m-k)}\left(x\right)=g\left(x\right)x^{\alpha +\beta -\gamma },}\hfill \\ {\displaystyle {\int }_a^x\frac{{(x-s)}^{\gamma +m-1}}{\mathsf{\Gamma }(\gamma +m)}{}_2{F}_1\left[\begin{array}{c}\gamma -\alpha ,\gamma -\beta \\ \gamma +m\end{array};1-\frac{x}{s}\right]s^{\alpha +\beta -\gamma }\phi \left(s\right)\mathrm{d}s=y\left(x\right),}\hfill \end{array}\right.$$

with initial conditions $y\left(a\right)=y^{\prime }\left(a\right)=\cdots =y^{(m-1)}\left(a\right)=0$.

In Ref. [17], Khudozhnikov briefly mentioned that the result can be generalized to those equations involving the generalized hypergeometric functions ${}_{p+1}{F}_p$, ${}_p{F}_p$ and ${}_{p-1}{F}_p$. However, he did not give possible forms of the generalizations or the formulas to be used. In fact, the most likely generalization requires use of a generalized Euler-type transformation, which is not included in Ref. [33]. Therefore, we think that the question of finding a generalization of Theorem 4 is still open.

In this section, we first propose a generalization of Theorem 4. We then consider a Volterra-type integral equation generated by the operator $\mathcal{I}$ defined by (12) and obtain an analogue of Khudozhnikov’s theorem.

4.1. A Generalization of Khudozhnikov’s Theorem

Let us consider the Volterra-type integral equation

$${\int }_a^x\frac{{\left(x-s\right)}^{\gamma -1}}{\mathsf{\Gamma }\left(\gamma \right)}{}_{r+2}{F}_{r+1}\left[\begin{array}{c}\alpha ,\beta ,\\ \gamma ,\end{array}\begin{array}{c}(f_r+m_r)\\ \left(f_r\right)\end{array};1-\frac{x}{s}\right]\phi \left(s\right)\mathrm{d}s=g\left(x\right),$$

(50)

where $0<\Re \left(\gamma \right)<1$, $m\in \mathbb{N}$ and $0<a\le x\le b<+\infty$. Obviously, (50) reduces to (49) when $r=1$, $f_1=\epsilon$ and $m_1=m$.

By using a lemma due to Miller and Paris [20], p. 298, Lemma 4, and the classical Euler transformation [18], p. 68, Equation (2.2.7), we can express the ${}_{r+2}{F}_{r+1}$-function as a finite sum of ${}_2{F}_1$-functions given by

$$\begin{array}{cc}\hfill {}_{r+2}{F}_{r+1}\left[\begin{array}{c}\alpha ,\beta ,\\ \gamma ,\end{array}\begin{array}{c}(f_r+m_r)\\ \left(f_r\right)\end{array};x\right]& =\sum _{k=0}^m\frac{A_k}{A_0}\frac{{\left(\alpha \right)}_k{\left(\beta \right)}_k}{{\left(\gamma \right)}_k}{}_2{F}_1\left[\begin{array}{c}\alpha +k,\beta +k\\ \gamma +k,\end{array};x\right]x^k\hfill \\ \hfill & ={\left(1-x\right)}^{\gamma -\alpha -\beta }\sum _{k=0}^m\frac{A_k}{A_0}\frac{{\left(\alpha \right)}_k{\left(\beta \right)}_k}{{\left(\gamma \right)}_k}{}_2{F}_1\left[\begin{array}{c}\gamma -\alpha ,\gamma -\beta \\ \gamma +k,\end{array};x\right]{\left(\frac{x}{1-x}\right)}^k.\hfill \end{array}$$

(51)

Then (50) can be written as

$$\begin{array}{cc}\hfill g\left(x\right)& ={\int }_a^x\frac{{\left(x-s\right)}^{\gamma -1}}{\mathsf{\Gamma }\left(\gamma \right)}{}_{r+2}{F}_{r+1}\left[\begin{array}{c}\alpha ,\beta ,\\ \gamma ,\end{array}\begin{array}{c}(f_r+m_r)\\ \left(f_r\right)\end{array};1-\frac{x}{s}\right]\phi \left(s\right)\mathrm{d}s\hfill \\ \hfill & =\sum _{k=0}^m\frac{A_k}{A_0}\frac{{\left(\alpha \right)}_k{\left(\beta \right)}_k}{{\left(-x\right)}^k}{x}^{\gamma -\alpha -\beta }{\int }_a^x\frac{{\left(x-s\right)}^{\gamma +k-1}}{\mathsf{\Gamma }(\gamma +k)}{}_2{F}_1\left[\begin{array}{c}\gamma -\alpha ,\gamma -\beta \\ \gamma +k\end{array};1-\frac{x}{s}\right]s^{\alpha +\beta -\gamma }\phi \left(s\right)\mathrm{d}s.\hfill \end{array}$$

(52)

Let

$$y\left(x\right):={\int }_a^x\frac{{\left(x-s\right)}^{\gamma +m-1}}{\mathsf{\Gamma }(\gamma +m)}{}_2{F}_1\left[\begin{array}{c}\gamma -\alpha ,\gamma -\beta \\ \gamma +m\end{array};1-\frac{x}{s}\right]s^{\alpha +\beta -\gamma }\phi \left(s\right)\mathrm{d}s.$$

In view of the derivative Formula (44), we have

$$\frac{{\partial }^{m-k}}{\partial x^{m-k}}\left\{\frac{{\left(x-s\right)}^{\gamma +m-1}}{\mathsf{\Gamma }(\gamma +m)}{}_2{F}_1\left[\begin{array}{c}\gamma -\alpha ,\gamma -\beta \\ \gamma +m\end{array};1-\frac{x}{s}\right]\right\}=\frac{{\left(x-s\right)}^{\gamma +k-1}}{\mathsf{\Gamma }(\gamma +k)}{}_2{F}_1\left[\begin{array}{c}\gamma -\alpha ,\gamma -\beta \\ \gamma +k\end{array};1-\frac{x}{s}\right],$$

and therefore

$$y^{(m-k)}\left(x\right)={\int }_a^x\frac{{\left(x-s\right)}^{\gamma +k-1}}{\mathsf{\Gamma }(\gamma +k)}{}_2{F}_1\left[\begin{array}{c}\gamma -\alpha ,\gamma -\beta \\ \gamma +k\end{array};1-\frac{x}{s}\right]s^{\alpha +\beta -\gamma }\phi \left(s\right)\mathrm{d}s$$

and $y^{(m-k)}\left(a\right)=0$ for $k=1,\cdots ,m-1$. Now (52) can be expressed as

$$\sum _{k=0}^m\frac{A_k}{A_0}\frac{{\left(\alpha \right)}_k{\left(\beta \right)}_k}{{\left(-x\right)}^k}{y}^{(m-k)}\left(x\right)=x^{\alpha +\beta -\gamma }g\left(x\right).$$

The above steps concerning the integral Equation (50) therefore yield the following theorem.

Theorem 5.

The Volterra-type integral Equation (50) can be reduced to the following system of differential and integral equations:

$$\left\{\begin{array}{c}{\displaystyle \sum _{k=0}^m\frac{A_k}{A_0}\frac{{\left(\alpha \right)}_k{\left(\beta \right)}_k}{{\left(-x\right)}^k}{y}^{(m-k)}\left(x\right)=g\left(x\right)x^{\alpha +\beta -\gamma },}\hfill \\ {\displaystyle {\int }_a^x\frac{{(x-s)}^{\gamma +m-1}}{\mathsf{\Gamma }(\gamma +m)}{}_2{F}_1\left[\begin{array}{c}\gamma -\alpha ,\gamma -\beta \\ \gamma +m\end{array};1-\frac{x}{s}\right]s^{\alpha +\beta -\gamma }\phi \left(s\right)\mathrm{d}s=y\left(x\right),}\hfill \end{array}\right.$$

with initial conditions $y\left(a\right)=y^{\prime }\left(a\right)=\cdots =y^{(m-1)}\left(a\right)=0$, where $A_k$$(0\le k\le m)$ is defined in (11).

To show that Theorem 5 contains Khudozhnikov’s result as a special case, we only need to prove that

$$\frac{A_k}{A_0}=\left(\begin{array}{c}m\\ k\end{array}\right)\frac{1}{{\left(\epsilon \right)}_k}.$$

(53)

Our calculations require some basics on the theory of combinatorics.

When $r=1$, $f_1=\epsilon$ and $m_1=m$, we get

$$A_0={\left(\epsilon \right)}_m\mathrm{and}{A}_k=\sum _{j=k}^m\left\{\begin{array}{c}j\\ k\end{array}\right\}{\widehat{\sigma }}_{m-j},$$

(54)

where ${\widehat{\sigma }}_{m-j}$ is generated by

$${(\epsilon +x)}_m=\sum _{j=0}^m{\widehat{\sigma }}_{m-j}{x}^j.$$

(55)

We need in fact to find an explicit expression for ${\widehat{\sigma }}_{m-j}$. By using the Chu–Vandermonde identity [18], p. 70, we have

$${(\epsilon +x)}_m=\sum _{k=0}^m\left(\begin{array}{c}m\\ k\end{array}\right){\left(\epsilon \right)}_{m-k}{\left(x\right)}_k.$$

(56)

Recall that

$${\left(x\right)}_k=\sum _{j=0}^k{(-1)}^{k-j}s(k,j)x^j=\sum _{j=0}^k\left[\begin{array}{c}k\\ j\end{array}\right]x^j,$$

(57)

where $s(k,j)$ is the Stirling number of the first kind and the symbol $\left[\begin{array}{c}k\\ j\end{array}\right]$ is usually used to denote the unsigned Stirling number of the first kind (see Ref. [34], p. 239). Substituting (57) into (56) and then interchanging the order of summation, we obtain

$${(\epsilon +x)}_m=\sum _{k=0}^m\left(\begin{array}{c}m\\ k\end{array}\right){\left(\epsilon \right)}_{m-k}\sum _{j=0}^k\left[\begin{array}{c}k\\ j\end{array}\right]x^j=\sum _{j=0}^m\sum _{k=j}^m\left(\begin{array}{c}m\\ k\end{array}\right){\left(\epsilon \right)}_{m-k}\left[\begin{array}{c}k\\ j\end{array}\right]x^j.$$

(58)

Comparing (58) with (55), it follows that

$${\widehat{\sigma }}_{m-j}=\sum _{k=j}^m\left(\begin{array}{c}m\\ k\end{array}\right){\left(\epsilon \right)}_{m-k}\left[\begin{array}{c}k\\ j\end{array}\right],$$

(59)

and combining (54) with (59) and taking into account the index factorization

$$[k\le j\le m][j\le \ell \le m]=[k\le j\le \ell \le m]=[k\le \ell \le m][k\le j\le \ell ],$$

we obtain

$$\begin{array}{cc}\hfill A_k& =\sum _{j=k}^m\left\{\begin{array}{c}j\\ k\end{array}\right\}\sum _{\ell =j}^m\left(\begin{array}{c}m\\ \ell \end{array}\right){\left(\epsilon \right)}_{m-\ell }\left[\begin{array}{c}\ell \\ j\end{array}\right]=\sum _{\ell =k}^m\left(\begin{array}{c}m\\ \ell \end{array}\right){\left(\epsilon \right)}_{m-\ell }\sum _{j=k}^{\ell }\left\{\begin{array}{c}j\\ k\end{array}\right\}\left[\begin{array}{c}\ell \\ j\end{array}\right]\hfill \\ \hfill & =\frac{1}{\left(k-1\right)!}\sum _{\ell =k}^m\left(\begin{array}{c}m\\ \ell \end{array}\right)\left(\begin{array}{c}\ell \\ k\end{array}\right){\left(\epsilon \right)}_{m-\ell }{\left(1\right)}_{\ell -1}=\frac{1}{\left(k-1\right)!}\sum _{\ell =0}^{m-k}\left(\begin{array}{c}m\\ \ell +k\end{array}\right)\left(\begin{array}{c}\ell +k\\ k\end{array}\right){\left(\epsilon \right)}_{m-\ell -k}{\left(1\right)}_{\ell +k-1}\hfill \\ \hfill & =\left(\begin{array}{c}m\\ k\end{array}\right)\sum _{\ell =0}^{m-k}\left(\begin{array}{c}m-k\\ \ell \end{array}\right){\left(\epsilon \right)}_{m-k-\ell }{\left(k\right)}_{\ell }=\left(\begin{array}{c}m\\ k\end{array}\right)\frac{{\left(\epsilon \right)}_m}{{\left(\epsilon \right)}_k},\hfill \end{array}$$

(60)

where we have used the familiar convoluation identity (see, for example Ref. [34], p. 240)

$$\sum _{j=k}^{\ell }\left\{\begin{array}{c}j\\ k\end{array}\right\}\left[\begin{array}{c}\ell \\ j\end{array}\right]=\left(\begin{array}{c}\ell \\ k\end{array}\right)\frac{\left(\ell -1\right)!}{\left(k-1\right)!}(\ell \ge k\ge 1).$$

Evidently, (60) is equivalent to (53).

4.2. A Variant of Khudozhnikov’s Theorem

A comparison of the fractional integral operator $\mathcal{I}$ with Equations (49) and (50) inspire us to consider the following integral equation

$${\int }_{\rho }^x\frac{{\left(x^{\nu }-s^{\nu }\right)}^{\mu -1}}{\mathsf{\Gamma }\left(\mu \right)}{}_{r+2}{F}_{r+1}\left[\begin{array}{c}a,b,\\ \mu ,\end{array}\begin{array}{c}(f_r+m_r)\\ \left(f_r\right)\end{array};1-\frac{s^{\nu }}{x^{\nu }}\right]\phi \left(s\right)s^{\nu h+\nu -1}\mathrm{d}s=g\left(x\right),$$

(61)

where $0<\Re \left(\mu \right)<1$ and $0<\rho \le s\le x<\infty$.

Using the Euler-type transformation (43), then Equation (61) can be converted into

$$\begin{array}{cc}\hfill {\int }_{\rho }^x\frac{{\left(x^{\nu }-s^{\nu }\right)}^{\mu -1}}{\mathsf{\Gamma }\left(\mu \right)}{}_{m+2}{F}_{m+1}\left[\begin{array}{c}a,\mu -b-m,\\ \mu ,\end{array}\begin{array}{c}({\xi }_m+1)\\ \left({\xi }_m\right)\end{array};1-\frac{x^{\nu }}{s^{\nu }}\right]& \phi \left(s\right)s^{\nu (h-a)+\nu -1}\mathrm{d}s\hfill \\ \hfill & =x^{-\nu a}g\left(x\right),\hfill \end{array}$$

(62)

where $\left({\xi }_m\right)$ are nonvanishing zeros of the parametric polynomial $Q_m\left(t\right)$ of degree m given by (39).

By using the same lemma of Miller and Paris [20], p. 298, Lemma 4 and the Euler transformation [18], p. 68, Equation (2.2.7) or else using Equation (51), we can express (as in the proof of Theorem 5) the ${}_{m+2}{F}_{m+1}$-function as a finite sum of ${}_2{F}_1$-functions given by

$$\begin{array}{cc}\hfill & {}_{m+2}{F}_{m+1}\left[\begin{array}{c}a,\mu -b-m,\\ \mu ,\end{array}\begin{array}{c}({\xi }_m+1)\\ \left({\xi }_m\right)\end{array};x\right]\hfill \\ \hfill & =\sum _{k=0}^m\frac{{\mathcal{A}}_k}{{\left(\mu \right)}_k}{}_2{F}_1\left[\begin{array}{c}a+k,\mu -b-m+k,\\ \mu +k,\end{array};x\right]x^k\hfill \\ \hfill & ={\left(1-x\right)}^{b-a+m}\sum _{k=0}^m\frac{{\mathcal{A}}_k}{{\left(\mu \right)}_k}{}_2{F}_1\left[\begin{array}{c}\mu -a,b+m,\\ \mu +k,\end{array};x\right]{\left(\frac{x}{1-x}\right)}^k,\hfill \end{array}$$

(63)

where

$${\mathcal{A}}_k:=\frac{{\left(a\right)}_k{(\mu -b-m)}_k}{{\xi }_1\cdots {\xi }_m}\sum _{j=k}^m\left\{\begin{array}{c}j\\ k\end{array}\right\}{\tilde{\sigma }}_{m-j}$$

(64)

and ${\tilde{\sigma }}_j$ ($0\le j\le m$) are generated by (41). With the help of (63), the integral Equation (62) can then be written as

$$\begin{array}{cc}\hfill \sum _{k=0}^m\frac{{\mathcal{A}}_k}{{(-x^{\nu })}^k}{\int }_{\rho }^x\frac{{\left(x^{\nu }-s^{\nu }\right)}^{\mu +k-1}}{\mathsf{\Gamma }(\mu +k)}{}_2{F}_1\left[\begin{array}{c}\mu -a,b+m,\\ \mu +k,\end{array};1-\frac{x^{\nu }}{s^{\nu }}\right]& \phi \left(s\right)s^{\nu (h-b-m)+\nu -1}\mathrm{d}s\hfill \\ \hfill & =x^{-\nu (b+m)}g\left(x\right).\hfill \end{array}$$

(65)

By making use of (44), we obtain

$$\frac{{\partial }^{m-k}}{\partial z^{m-k}}\left\{z^{\mu +m-1}{}_2{F}_1\left[\begin{array}{c}\mu -a,b+m\\ \mu +m\end{array};z\right]\right\}=\frac{{\left(\mu \right)}_m}{{\left(\mu \right)}_k}{z}^{\mu +k-1}{}_2{F}_1\left[\begin{array}{c}\mu -a,b+m\\ \mu +k\end{array};z\right],$$

and thus

$$\begin{array}{cc}\hfill & \frac{{\partial }^{m-k}}{\partial x^{m-k}}\left\{\frac{{(x^{\nu }-s^{\nu })}^{\mu +m-1}}{\mathsf{\Gamma }(\mu +m)}{}_2{F}_1\left[\begin{array}{c}\mu -a,b+m\\ \mu +m\end{array};1-\frac{x^{\nu }}{s^{\nu }}\right]\right\}\hfill \\ \hfill & ={\nu }^{m-k}\frac{{(x^{\nu }-s^{\nu })}^{\mu +k-1}}{\mathsf{\Gamma }(\mu +k)}{}_2{F}_1\left[\begin{array}{c}\mu -a,b+m\\ \mu +k\end{array};1-\frac{x^{\nu }}{s^{\nu }}\right].\hfill \end{array}$$

(66)

Substituting (66) into (65), we get

$$\begin{array}{cc}\hfill \sum _{k=0}^m\frac{{\mathcal{A}}_k{\nu }^k}{{(-x^{\nu })}^k}{\int }_{\rho }^x\frac{{\partial }^{m-k}}{\partial x^{m-k}}\left\{\frac{{\left(x^{\nu }-s^{\nu }\right)}^{\mu +m-1}}{\mathsf{\Gamma }(\mu +m)}{}_2{F}_1\left[\begin{array}{c}\mu -a,b+m,\\ \mu +m,\end{array};1-\frac{x^{\nu }}{s^{\nu }}\right]\right\}& \phi \left(s\right)s^{\nu (h-b-m)+\nu -1}\mathrm{d}s\hfill \\ \hfill & ={\nu }^m{x}^{-\nu (b+m)}g\left(x\right).\hfill \end{array}$$

Finally, using the Leibniz integral rule and simplifying the resulting formula by the Pfaff transformation [18], p. 68, Equation (2.2.6), we obtain

$$\begin{array}{cc}\hfill \sum _{k=0}^m\frac{{\mathcal{A}}_k{\nu }^k}{{(-x^{\nu })}^k}\frac{{\partial }^{m-k}}{\partial x^{m-k}}& \{x^{\nu (a-\mu )}{\int }_{\rho }^x\frac{{\left(x^{\nu }-s^{\nu }\right)}^{\mu +m-1}}{\mathsf{\Gamma }(\mu +m)}{}_2{F}_1\left[\begin{array}{c}\mu -a,\mu -b,\\ \mu +m,\end{array};1-\frac{s^{\nu }}{x^{\nu }}\right]\hfill \\ \hfill & \phi \left(s\right)s^{\nu (h+\mu -b-a-m)+\nu -1}\mathrm{d}s\}={\nu }^m{x}^{-\nu (b+m)}g\left(x\right).\hfill \end{array}$$

If

$$y\left(x\right)=x^{\nu (a-\mu )}{\int }_{\rho }^x\frac{{\left(x^{\nu }-s^{\nu }\right)}^{\mu +m-1}}{\mathsf{\Gamma }(\mu +m)}{}_2{F}_1\left[\begin{array}{c}\mu -a,\mu -b,\\ \mu +m,\end{array};1-\frac{s^{\nu }}{x^{\nu }}\right]\phi \left(s\right)s^{\nu (h+\mu -b-a-m)+\nu -1}\mathrm{d}s,$$

then the above details concerning the integral equation (61) may be put in the following theorem.

Theorem 6.

The Volerra-type integral Equation (61) can be reduced to the following system of differential and integral equations:

$$\left\{\begin{array}{c}{\displaystyle \sum _{k=0}^m\frac{{\mathcal{A}}_k{\nu }^k}{{(-x^{\nu })}^k}{y}^{(m-k)}\left(x\right)={\nu }^m{x}^{-\nu (b+m)}g\left(x\right),}\hfill \\ {\displaystyle x^{\nu (a-\mu )}{\int }_{\rho }^x\frac{{\left(x^{\nu }-s^{\nu }\right)}^{\mu +m-1}}{\mathsf{\Gamma }(\mu +m)}{}_2{F}_1\left[\begin{array}{c}\mu -a,\mu -b,\\ \mu +m,\end{array};1-\frac{s^{\nu }}{x^{\nu }}\right]\phi \left(s\right)s^{\nu (h+\mu -b-a-m)+\nu -1}\mathrm{d}s=y\left(x\right),}\hfill \end{array}\right.$$

with initial conditions $y\left(\rho \right)=y^{\prime }\left(\rho \right)=\cdots =y^{(m-1)}\left(\rho \right)=0$, where ${\mathcal{A}}_k$$(0\le k\le m)$ is given by (64).

5. Conclusions

In this paper, some composition formulas of $\mathcal{I}$ and $\mathcal{J}$ defined by (12) and (13) are obtained by making use of a Erdélyi-type integral. We find a derivative formula, which in the future may enable us to define a new fractional derivative operator. Finally, we generalize Khudozhnikov’s work on Volterra-type integral equation and find its relationship with our operator $\mathcal{I}$.

Considering the obtained properties of the operators $\mathcal{I}$ and $\mathcal{J}$, we briefly mention here some problems that deserve further study.

(i)

Since only two composition formulas for $\mathcal{I}$ and $\mathcal{J}$ are found in the present work, which is still a very small number compared to the number of the composition formulas of Saigo’s operators $I^{\alpha ,\beta ,\eta }$ and $J^{\alpha ,\beta ,\eta }$, it may be worthwhile if additional composition structures can be discovered for the operators $\mathcal{I}$ and $\mathcal{J}$. The exploration in this direction may also lead us to new discoveries related to the Erdélyi-type integrals;

(ii)

The present work together with our previous papers [14,16] have established many fundamental properties of $\mathcal{I}$ and $\mathcal{J}$. For further possible work, some new properties and problems may be worthy of attention in view of the classical books [4,23] on the subject and some recent review articles contained, for example, in Ref. [35]. In particular, it may be worthwhile to first focus on the problem of finding a reasonable analogue of the well known limit case formula, viz. ${\displaystyle \underset{\alpha \to 0}{lim}\left(I_{a+}^{\alpha }\phi \right)\left(x\right)=\phi \left(x\right)}$ concerning the Riemann–Liouville fractional integral operator (see Ref. [23], p. 51, Theorem 2.7).