Gabor Transform Associated with the Dunkl–Bessel Transform and Spectrograms

Abstract

Time–frequency (or space–phase) analysis plays a key role in signal analysis. In particular, signals that have a very concentrated time–frequency content are of great importance. However, the uncertainty principle sets a limitation to the possible simultaneous concentration of a function and its Dunkl–Bessel transform. For this purpose, we introduce and study a new transformation called Dunkl–Bessel Gabor transform. For this transformation, we define the Toeplitz-type (or time–frequency localization) operators, in order to localize signals on the time–frequency plane. We study these operators; in particular, we give criteria for their boundedness and Schatten class properties. Then, using the special class of concentration operators, which are compact and self-adjoint, we show that their eigenfunctions are maximally time–frequency-concentrated in the region of interest.

1. Introduction and Preliminaries

The Gabor theory has gained respectable status in the field of time–frequency analysis in a short period of time, since the Gabor transform is superior to the Fourier transform, because of its ability to measure the time–frequency variations of a signal at different time–frequency resolutions.

However, the uncertainty principles in Fourier analysis set a limit to the maximal time–frequency resolution. To overcome this problem, Daubechies [1] introduced the concept of localization operators, which have been shown to be effective in localizing signals on the time–frequency plane. These operators are also called Toeplitz operators or short-time Fourier multipliers.

Extensive research on localization operators has been conducted, notably by Slepian and Pollak [2] in the classical setting, with further contributions by Landau and Pollak [3,4], as well as by Slepian [5,6]. Many generalizations and extensions can be found in the literature, such as for the Dunkl transform, the Weinstein transform [7,8,9,10] and the multivariable Bessel transform [11].

To be more precise, we first recall some results related to the Dunkl theory (see [12,13,14]). Let ${\mathbb{R}}^d$ be the Euclidean space equipped with a scalar product $\langle ,\rangle$ and let $\parallel x\parallel =\sqrt{\langle x,x\rangle }$. For $\alpha$ in ${\mathbb{R}}^d\backslash \left\{0\right\}$, let ${\varsigma }_{\alpha }$ be the reflection in the hyperplane $H_{\alpha }\subset {\mathbb{R}}^d$ orthogonal to $\alpha$, i.e., for $x\in {\mathbb{R}}^d$,

$${\varsigma }_{\alpha }\left(x\right)=x-2{\displaystyle \frac{\langle \alpha ,x\rangle }{{\parallel \alpha \parallel }^2}}\alpha .$$

A finite set $R\subset {\mathbb{R}}^d\backslash \left\{0\right\}$ is called a root system if $R\cap \mathbb{R}\alpha =\{\alpha ,-\alpha \}$ and ${\varsigma }_{\alpha }R=R$ for all $\alpha \in R$. For a given root system R, reflections ${\varsigma }_{\alpha },\alpha \in R$, generate a finite group $W\subset O(d-1)$, called the reflection group associated with R. We fix ${\displaystyle \beta \in {\mathbb{R}}^d\backslash {\cup }_{\alpha \in R}{H}_{\alpha }}$ and define a positive root system $R_{+}=\left\{\alpha \in R:\langle \alpha ,\beta \rangle >0\right\}$. We normalize each $\alpha \in R_{+}$ as $\langle \alpha ,\alpha \rangle =2$.

A function $k:R⟶\mathbb{C}$ is called a multiplicity function if it is invariant under the action of W, and we introduce the index $\gamma$ as

$${\displaystyle \gamma =\gamma \left(k\right)=\sum _{\alpha \in R_{+}}k\left(\alpha \right).}$$

Throughout this paper, we will assume that $k\left(\alpha \right)\ge 0$ for all $\alpha \in R$, and we will denote by ${\omega }_k$ the weight function on ${\mathbb{R}}^d$ given by

$${\omega }_k\left(x\right)=\prod _{\alpha \in R_{+}}{\left|\langle \alpha ,x\rangle \right|}^{2k\left(\alpha \right)},$$

which is invariant and homogeneous of degree $2\gamma$.

The Dunkl operators $T_i$, $i=1,2,\cdots ,d$, on ${\mathbb{R}}^d$ related to the multiplicity function k and the positive root system $R_{+}$ are as follows:

$${\displaystyle T_{i}g\left(t\right)=\frac{\partial g}{\partial t_i}\left(t\right)+\sum _{\alpha \in R_{+}}k\left(\alpha \right){\alpha }_i\frac{g\left(t\right)-g\left({\varsigma }_{\alpha }\left(t\right)\right)}{\langle \alpha ,t\rangle },g\in C^1\left({\mathbb{R}}^d\right).}$$

Specifically, we see that the Dunkl operators $T_i$ commute pairwise and are skew-symmetric with respect to the W-invariant measure $w_k\left(x\right)dx$.

If $g\in C^2\left({\mathbb{R}}^d\right)$, then the Dunkl–Laplace operator ${\Delta }_k$ is defined by:

$${\Delta }_{k}g\left(t\right)=\sum _{i=1}^d{T}_i^{2}g\left(t\right)=\Delta g\left(t\right)+2\sum _{\alpha \in R^{+}}k\left(\alpha \right)\left({\displaystyle \frac{\langle \nabla g\left(t\right),\alpha \rangle }{\langle \alpha ,t\rangle }}-{\displaystyle \frac{g\left(t\right)-g\left({\varsigma }_{\alpha }\left(t\right)\right)}{{\langle \alpha ,t\rangle }^2}}\right),t\in {\mathbb{R}}^d,$$

where ∇ and Δ are, respectively, the nabla and Euclidean Laplacian operators.

Like spherical functions on a Riemannian symmetric space, for all $t\in {\mathbb{R}}^d$, the following system

$$\left\{\begin{array}{ccc}{T}_{i}v(x,t)\hfill & =\hfill & t_{i}v(x,t),i=1,\dots ,d,\hfill \\ v(0,t)\hfill & =\hfill & 1\hfill \end{array}\right.$$

has a unique analytic solution $K(·,t)$. This solution is known as the Dunkl kernel (also called nonsymmetric generalized Bessel function). Observe, moreover, that the Dunkl kernel has symmetric arguments and is holomorphic on ${\mathbb{C}}^d\times {\mathbb{C}}^d$. This kernel is explicitly known only in very few cases, that include, for example, the rank-one case and the symmetric group ${\mathbb{S}}_3$.

The Dunkl–Bessel transform [15] is defined by

$${\displaystyle {\mathcal{F}}_{k,\beta }\left(f\right)\left(y\right)={\int }_{{\mathbb{R}}_{+}^{d+1}}f\left(x\right){\Lambda }_{k,\beta }(-x,y)d{\mu }_{k,\beta }\left(x\right),y=(y^{\prime },y_{d+1})\in {\mathbb{R}}_{+}^{d+1}={\mathbb{R}}^d\times {\mathbb{R}}_{+},}$$

(1)

where

$${\Lambda }_{k,\beta }(x,z)=K(i{x}^{\prime },z^{\prime })j_{\beta }\left(x_{d+1}{z}_{d+1}\right).$$

(2)

and ${\mu }_{k,\beta }$ is the measure on ${\mathbb{R}}_{+}^{d+1}$ defined by

$${\displaystyle d{\mu }_{k,\beta }(x^{\prime },x_{d+1})=\frac{1}{m_{k,\beta }}{\omega }_k\left(x^{\prime }\right)x_{d+1}^{2\beta +1}d{x}^{\prime }d{x}_{d+1},m_{k,\beta }={\int }_{{\mathbb{R}}^{d+1}}{e}^{-\frac{{\parallel x\parallel }^2}{2}}{\omega }_k\left(x^{\prime }\right)x_{d+1}^{2\beta +1}d{x}^{\prime }d{x}_{d+1}.}$$

Here, $j_{\beta }$ is the normalized Bessel function and K is the Dunkl kernel.

This transformation generalizes the Fourier transform, the Weinstein transform and the multivariable Bessel transform, and has been studied in several mathematical problems, such as the mean value theorem [15], uncertainty principles [16], wavelet theory [17] and so on.

As for the Fourier transform, the Dunkl–Bessel transform has shown some weakness in the study of signals on the time–frequency plane, and this fact has been illustrated by the various formulations of the uncertainty principle in this setting.

The aim of this paper is to follow Daubechies’ approach, by studying first the Dunkl–Bessel Gabor transform, and then introducing the Toeplitz operator in this context. We will show some harmonic analysis results for the Dunkl–Bessel Gabor transform, such as a Parseval-type equality and an inversion formula. Then, we will give criteria on the compactness and boundedness of the introduced Toeplitz operators. As a special case of these operators, we will focus on concentration operators, which are compact and self-adjoint and even trace-class. As in the case of the prolate spheroidal wave functions, we will show that their eigenfunctions are maximally time–frequency-concentrated on the region of interest from the time–frequency plane.

Typical examples of higher dimensional signals include image and video signals. Indeed, a video signal is a sequence of individual video frames, or images. Unlike one-dimensional signals, there are special problems that arise in higher dimensional signals, such as edge detection, target detection and tracking, and special techniques are required to deal with such problems. It is therefore worthwhile to delve into the field of higher dimensional time–frequency analysis both from theoretical and applied perspectives. The theoretical considerations in the proposal shall encompass the formulation of novel tools of time–frequency analysis, particularly well suited for higher dimensional signals arising in numerous scientific pursuits. The application areas shall be primarily within the field of signal and image processing, with special emphasis on distinct sectors such as energy and economics.

The structure of this manuscript is as follow: In Section 2, we define the Dunkl–Bessel Gabor transform and study its harmonic analysis. Then, we introduce and study the Toeplitz operators in Section 3. Next, we examine the $L^p$-boundedness and compactness of these operators, under some appropriate conditions on the symbol and the window function. Finally, in Section 4, we investigate some spectral analysis issues related to concentration operators.

2. The Dunkl–Bessel Gabor Transform

We will first recall the basic results on the Dunkl–Bessel transform (cf. [15]). To do this, let ${\mathbb{R}}_{+}^{d+1}={\mathbb{R}}^d\times [0,\infty )$ and let $x=(x_1,\dots ,x_d,x_{d+1})=(x^{\prime },x_{d+1})\in {\mathbb{R}}_{+}^{d+1}$.

If $x=(x^{\prime },x_{d+1})\in {\mathbb{R}}^d\times [0,\infty )$, then for all $f\in C_{*}^2\left({\mathbb{R}}^{d+1}\right)$ we define:

$${\Delta }_{k,\beta }f\left(x\right)={\Delta }_{k,x^{\prime }}f(x^{\prime },x_{d+1})+{\mathcal{L}}_{\beta ,x_{d+1}}f(x^{\prime },x_{d+1}),$$

(3)

where $C_{*}^p\left({\mathbb{R}}^{d+1}\right)$ is the space of functions of class $C^p$ on ${\mathbb{R}}^{d+1}$, even with respect to the last variable. This operator is known as the Dunkl–Bessel–Laplace operator, where ${\Delta }_{k,x^{\prime }}$ is the Dunkl–Laplace operator on ${\mathbb{R}}^d$ and ${\mathcal{L}}_{\beta ,x_{d+1}}$ the Bessel operator on $[0,\infty )$ given by

$${\mathcal{L}}_{\beta ,x_{d+1}}={\displaystyle \frac{d^2}{d{x}_{d+1}^2}}+{\displaystyle \frac{2\beta +1}{x_{d+1}}}{\displaystyle \frac{d}{d{x}_{d+1}}},\beta >-{\displaystyle \frac{1}{2}}.$$

The Dunkl–Bessel kernel ${\Lambda }_{k,\beta }$ is given by

$${\Lambda }_{k,\beta }(x,z)=K(i{x}^{\prime },z^{\prime })j_{\beta }\left(x_{d+1}{z}_{d+1}\right),(x,z)\in {\mathbb{R}}_{+}^{d+1}\times {\mathbb{C}}^{d+1},$$

(4)

where $j_{\beta }$ is the normalized Bessel function and K is the Dunkl kernel.

This kernel satisfies the following properties:

• For all $z,t\in {\mathbb{C}}^{d+1}$ and for all $\lambda \in \mathbb{C}$, we have

  $${\Lambda }_{k,\beta }(z,t)={\Lambda }_{k,\beta }(t,z);{\Lambda }_{k,\beta }(z,0)=1\mathrm{and}{\Lambda }_{k,\beta }(\lambda z,t)={\Lambda }_{k,\beta }(z,\lambda t).$$

  (5)
• For all $\nu \in {\mathbb{N}}^{d+1},x\in {\mathbb{R}}_{+}^{d+1}$ and $z\in {\mathbb{C}}^{d+1}$, we have

  $$|D_z^{\nu }{\Lambda }_{k,\beta }{(x,z)|\le \parallel x\parallel }^{\left|\nu \right|}exp\left(\parallel x\parallel \parallel \mathrm{Im}z\parallel \right),$$

  (6)

  where $D_z^{\nu }={\displaystyle \frac{{\partial }^{\nu }}{\partial z_1^{{\nu }_1}...\partial z_{d+1}^{{\nu }_{d+1}}}}$ and $\left|\nu \right|={\nu }_1+\cdots +{\nu }_{d+1}.$

In particular, for all $x,y\in {\mathbb{R}}_{+}^{d+1}$,

$$|{\Lambda }_{k,\beta }(x,y)|\le 1.$$

(7)

We denote by $L_{k,\beta }^p\left({\mathbb{R}}_{+}^{d+1}\right)$ the space of measurable functions on ${\mathbb{R}}_{+}^{d+1}$ such that:

$$\begin{array}{ccc}{\parallel g\parallel }_{L_{k,\beta }^p}\hfill & :=\hfill & {\displaystyle {\left({\int }_{{\mathbb{R}}_{+}^{d+1}}{\left|g\left(t\right)\right|}^{p}d{\mu }_{k,\beta }\left(t\right)\right)}^{\frac{1}{p}}<\infty ,\mathrm{if}1\le p<\infty ,}\hfill \\ \\ {\parallel g\parallel }_{L_{k,\beta }^{\infty }}\hfill & :=\hfill & {\displaystyle \mathrm{ess}\underset{t\in {\mathbb{R}}_{+}^{d+1}}{sup}\left|g\left(t\right)\right|<\infty ,}\hfill \end{array}$$

where ${\mu }_{k,\beta }$ is the measure on ${\mathbb{R}}_{+}^{d+1}$ defined by

$$d{\mu }_{k,\beta }(x^{\prime },x_{d+1})={\displaystyle \frac{1}{m_{k,\beta }}}{\omega }_k\left(x^{\prime }\right)x_{d+1}^{2\beta +1}d{x}^{\prime }d{x}_{d+1},$$

and

$${\displaystyle m_{k,\beta }={\int }_{{\mathbb{R}}^{d+1}}{e}^{-\frac{{\parallel x\parallel }^2}{2}}{\omega }_k\left(x^{\prime }\right)x_{d+1}^{2\beta +1}d{x}^{\prime }d{x}_{d+1}.}$$

(8)

If $p=2$, this space is provided with the following scalar product:

$${\displaystyle {\langle f,g\rangle }_{L_{k,\beta }^2}:={\int }_{{\mathbb{R}}_{+}^{d+1}}f\left(x\right)\overline{g\left(x\right)}d{\mu }_{k,\beta }\left(x\right).}$$

For $f\in L_{k,\beta }^1\left({\mathbb{R}}_{+}^{d+1}\right)$, the Dunkl–Bessel transform is defined by:

$${\displaystyle {\mathcal{F}}_{k,\beta }\left(f\right)\left(y\right)={\int }_{{\mathbb{R}}_{+}^{d+1}}f\left(x\right){\Lambda }_{k,\beta }(-x,y)d{\mu }_{k,\beta }\left(x\right),y=(y^{\prime },y_{d+1})\in {\mathbb{R}}_{+}^{d+1}.}$$

(9)

This transformation satisfies the following properties:

• For any $f\in L_{k,\beta }^1\left({\mathbb{R}}_{+}^{d+1}\right)$,

  $$\parallel {\mathcal{F}}_{k,\beta }{\left(f\right)\parallel }_{L_{k,\beta }^{\infty }}\le {\parallel f\parallel }_{L_{k,\beta }^1}.$$

  (10)
• For all $f\in L_{k,\beta }^1\left({\mathbb{R}}_{+}^{d+1}\right)$ such that ${\mathcal{F}}_{k,\beta }\left(f\right)$ belongs to $L_{k,\beta }^1\left({\mathbb{R}}_{+}^{d+1}\right)$, we have

  $${\displaystyle f\left(y\right)={\int }_{{\mathbb{R}}_{+}^{d+1}}{\mathcal{F}}_{k,\beta }\left(f\right)\left(x\right){\Lambda }_{k,\beta }(x,y)d{\mu }_{k,\beta }\left(x\right).a.e.}$$

  (11)
• If we define

  $$\overline{{\mathcal{F}}_{k,\beta }}\left(f\right)\left(y\right)={\mathcal{F}}_{k,\beta }\left(f\right)(-y),f\in {\mathcal{S}}_{*}\left({\mathbb{R}}^{d+1}\right),$$

  then

  $${\mathcal{F}}_{k,\beta }\overline{{\mathcal{F}}_{k,\beta }}=\overline{{\mathcal{F}}_{k,\beta }}{\mathcal{F}}_{k,\beta }=Id,$$

  (12)

where ${\mathcal{S}}_{*}\left({\mathbb{R}}^{d+1}\right)$ is the Schwartz space of rapidly decreasing functions on ${\mathbb{R}}^{d+1}$, even with respect to the last variable.

Moreover we have the following Plancherel-type and Parseval-type formulas for the Dunkl–Bessel transform.

Proposition 1.

 1. 

Plancherel’s formula: The Dunkl–Bessel transform ${\mathcal{F}}_{k,\beta }$ is a topological isomorphism from ${\mathcal{S}}_{*}\left({\mathbb{R}}^{d+1}\right)$ onto itself and for all f in ${\mathcal{S}}_{*}\left({\mathbb{R}}^{d+1}\right)$,

$${\displaystyle {\int }_{{\mathbb{R}}_{+}^{d+1}}{\left|f\left(x\right)\right|}^{2}d{\mu }_{k,\beta }\left(x\right)={\int }_{{\mathbb{R}}_{+}^{d+1}}{\left|{\mathcal{F}}_{k,\beta }\left(f\right)\left(\xi \right)\right|}^{2}d{\mu }_{k,\beta }\left(\xi \right),}$$

(13)

In particular, the Dunkl–Bessel transform $f\mapsto {\mathcal{F}}_{k,\beta }\left(f\right)$ can be uniquely extended to an isometric isomorphism on $L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$.

 2. 

Parseval’s formula: For all $f,g\in L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$, we have

$${\displaystyle {\int }_{{\mathbb{R}}_{+}^{d+1}}f\left(x\right)\overline{g\left(x\right)}d{\mu }_{k,\beta }\left(x\right)={\int }_{{\mathbb{R}}_{+}^{d+1}}{\mathcal{F}}_{k,\beta }\left(f\right)\left(\xi \right)\overline{{\mathcal{F}}_{k,\beta }\left(g\right)\left(\xi \right)}d{\mu }_{k,\beta }\left(\xi \right).}$$

(14)

Next, we will introduce the generalized translation and the convolution operator by the use of the Dunkl–Bessel kernel. That is, the Dunkl–Bessel translation ${\tau }_y^{k,\beta }$ is defined by:

$${\mathcal{F}}_{k,\beta }\left({\tau }_y^{k,\beta }f\right)(·)={\Lambda }_{k,\beta }(·,y){\mathcal{F}}_{k,\beta }\left(f\right)(·),f\in {\mathcal{S}}_{*}\left({\mathbb{R}}^{d+1}\right),y\in {\mathbb{R}}_{+}^{d+1}.$$

(15)

In the following theorem, we recall the main properties of ${\tau }_y^{k,\beta }$ established on the subspace $L_{k,\beta ,rad}^p\left({\mathbb{R}}_{+}^{d+1}\right)$ of radial functions in $L_{k,\beta }^p\left({\mathbb{R}}_{+}^{d+1}\right)$, $1\le p\le \infty$ (see [15]).

Theorem 1.

 1. 

Let $f\in L_{k,\beta ,rad}^1\left({\mathbb{R}}_{+}^{d+1}\right)$ be a nonnegative function. Then, we have

$$\forall y\in {\mathbb{R}}_{+}^{d+1},{\tau }_y^{k,\beta }f\ge 0,{\tau }_y^{k,\beta }f\in L_{k,\beta }^1\left({\mathbb{R}}_{+}^{d+1}\right)$$

and

$${\displaystyle {\int }_{{\mathbb{R}}_{+}^{d+1}}{\tau }_y^{k,\beta }f\left(x\right)d{\mu }_{k,\beta }\left(x\right)={\int }_{{\mathbb{R}}_{+}^{d+1}}f\left(x\right)d{\mu }_{k,\beta }\left(x\right).}$$

(16)

 2. 

For all $f\in L_{k,\beta ,rad}^p\left({\mathbb{R}}_{+}^{d+1}\right)$, $1\le p\le \infty$, we have

$$\forall y\in {\mathbb{R}}_{+}^{d+1},\parallel {\tau }_y^{k,\beta }{f\parallel }_{L_{k,\beta }^p}\le {\parallel f\parallel }_{L_{k,\beta }^p}.$$

(17)

The Dunkl–Bessel convolution product $f{*}_{k,\beta }g$ of the two functions $f,g\in {\mathcal{S}}_{*}\left({\mathbb{R}}^{d+1}\right)$ is defined by:

$${\displaystyle f{*}_{k,\beta }g\left(x\right)={\int }_{{\mathbb{R}}_{+}^{d+1}}{\tau }_x^{k,\beta }f(-y)g\left(y\right)d{\mu }_{k,\beta }\left(y\right).}$$

(18)

This convolution is commutative, associative and satisfies the following properties.

Proposition 2.

Let $1\le p,q,r\le \infty$ such that ${\displaystyle \frac{1}{p}}+{\displaystyle \frac{1}{q}}-{\displaystyle \frac{1}{r}}=1.$

 1. 

For all $f,g\in {\mathcal{S}}_{*}\left({\mathbb{R}}_{+}^{d+1}\right)$, $f{*}_{k,\beta }g$ belongs to ${\mathcal{S}}_{*}\left({\mathbb{R}}_{+}^{d+1}\right)$ and

$${\mathcal{F}}_{k,\beta }\left(f{*}_{k,\beta }g\right)={\mathcal{F}}_{k,\beta }\left(f\right){\mathcal{F}}_{k,\beta }\left(g\right).$$

(19)

 2. 

If $f\in L_{k,\beta }^p\left({\mathbb{R}}_{+}^{d+1}\right)$ and $g\in L_{k,\beta }^q\left({\mathbb{R}}_{+}^{d+1}\right)$ is radial, then $f{*}_{k,\beta }g\in L_{k,\beta }^r\left({\mathbb{R}}_{+}^{d+1}\right)$ such that

$${∥f{*}_{k,\beta }g∥}_{L_{k,\beta }^r}\le {∥f∥}_{L_{k,\beta }^p}{∥g∥}_{L_{k,\beta }^q}.$$

(20)

 3. 

If $f,g\in L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$, then the function $f{*}_{k,\beta }g$ belongs to $L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$ if and only if the function ${\mathcal{F}}_{k,\beta }\left(f\right){\mathcal{F}}_{k,\beta }\left(g\right)$ belongs to $L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$ and (19) holds.

 4. 

If $f,g\in L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$, then

$${\displaystyle {\int }_{{\mathbb{R}}_{+}^{d+1}}|f{*}_{k,\beta }{g\left(x\right)|}^{2}d{\mu }_{k,\beta }\left(x\right)={\int }_{{\mathbb{R}}_{+}^{d+1}}|{\mathcal{F}}_{k,\beta }{\left(f\right)\left(\xi \right)|}^2{\left|{\mathcal{F}}_{k,\beta }\left(g\right)\left(\xi \right)\right|}^{2}d{\mu }_{k,\beta }\left(\xi \right),}$$

(21)

where both members are finite or infinite.

Now, to introduce the Dunkl–Bessel Gabor transform, we will fix some notation: Let $X_{2d+2}={\mathbb{R}}_{+}^{d+1}\times {\mathbb{R}}_{+}^{d+1},$ and we will denote by $L_{{\gamma }_{k,\beta }}^p\left(X_{2d+2}\right)$, $1\le p\le \infty$, the space of measurable functions f on $X_{2d+2}$ satisfying

$$\begin{array}{ccc}\hfill {\parallel f\parallel }_{L_{{\gamma }_{k,\beta }}^p}& :=& {\left({\int }_{X_{2d+2}}{\left|f(x,\lambda )\right|}^{p}d{\gamma }_{k,\beta }(x,\lambda )\right)}^{1/p}<\infty ,1\le p<\infty ,\hfill \\ \hfill {\parallel f\parallel }_{L_{{\gamma }_{k,\beta }}^{\infty }}& :=& {\displaystyle \mathrm{ess}\underset{(x,\lambda )\in X_{2d+2}}{sup}\left|f(x,\lambda )\right|<\infty ,p=\infty ,}\hfill \end{array}$$

where $d{\gamma }_{k,\beta }(x,\lambda )=d{\mu }_{k,\beta }\left(x\right)d{\mu }_{k,\beta }\left(\lambda \right)$.

Definition 1.

For $\nu \in {\mathbb{R}}_{+}^{d+1}$, we define the modulation of any function $h\in L_{k,\beta ,rad}^2\left({\mathbb{R}}_{+}^{d+1}\right)$ by

$$h_{\nu }:={\mathcal{F}}_{k,\beta }\left(\sqrt{{\tau }_{\nu }^{k,\beta }\left({\left|h\right|}^2\right)}\right),$$

(22)

where ${\tau }_{\nu }^{k,\beta }$ is the generalized translation operator.

Equality (22) may be easily confirmed by the positivity of the translation operator ${\tau }_{\nu }^{k,\beta }$ on radial functions, as shown by Theorem 1. Furthermore, based on (13) and (16), we have for all $f\in L_{k,\beta ,rad}^2\left({\mathbb{R}}_{+}^{d+1}\right)$

$${\parallel f_{\nu }\parallel }_{L_{k,\beta }^2}=\parallel f{\parallel }_{L_{k,\beta }^2}.$$

(23)

For $h\in L_{k,\beta ,rad}^2\left({\mathbb{R}}_{+}^{d+1}\right)$ and $\nu ,y\in {\mathbb{R}}_{+}^{d+1}$, we define a new family of functions $h_{\nu ,y}$ as follows:

$$\begin{array}{c}\hfill h_{\nu ,y}\left(x\right)={\tau }_{-y^{\prime },y_{d+1}}^{k,\beta }{h}_{\nu }\left(x\right),x\in {\mathbb{R}}_{+}^{d+1}.\end{array}$$

(24)

Then,

$$\parallel h_{\nu ,y}{\parallel }_{L_{k,\beta }^2}\le \parallel h{\parallel }_{L_{k,\beta }^2}.$$

(25)

Definition 2.

Let $h\in L_{k,\beta ,rad}^2\left({\mathbb{R}}_{+}^{d+1}\right)$. Then, the Dunkl–Bessel Gabor transform ${\mathcal{G}}_h^{k,\beta }$, with respect to h, is defined by

$${\displaystyle {\mathcal{G}}_h^{k,\beta }\left(f\right)(y,\nu ):={\int }_{{\mathbb{R}}_{+}^{d+1}}f\left(x\right)\overline{h_{\nu ,y}\left(x\right)}d{\mu }_{k,\beta }\left(x\right),\nu ,y\in {\mathbb{R}}_{+}^{d+1},}$$

(26)

for any function $f\in L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$.

It can also be expressed via the convolution product (18) as

$${\mathcal{G}}_h^{k,\beta }\left(f\right)(y,\nu )=f{*}_{k,\beta }\overline{\check{h_{\nu }}}\left(y\right),$$

(27)

where for $t=(t^{\prime },t_{d+1})\in {\mathbb{R}}_{+}^{d+1}$, $\check{g}\left(t\right)=g(-t^{\prime },t_{d+1})$.

One can easily see that for any $\lambda >0$ and $(y,\nu )\in X_{2d+2}$, we have

$${\mathcal{G}}_{h_{1/\lambda }}^{k,\beta }\left(f_{\lambda }\right)(y,\nu )={\mathcal{G}}_h^{k,\beta }\left(f\right)\left({\displaystyle \frac{y}{\lambda }},\lambda \nu \right),$$

(28)

where

$$f_{\lambda }\left(x\right):={\displaystyle \frac{1}{{\lambda }^{\gamma +\beta +1+\frac{d}{2}}}}f\left({\displaystyle \frac{x}{\lambda }}\right),x\in {\mathbb{R}}_{+}^{d+1}.$$

Theorem 2.

Let $h\in L_{k,\beta ,rad}^2\left({\mathbb{R}}_{+}^{d+1}\right)$. Then, for all $f,g\in L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$ we have:

 1. 

The function ${\mathcal{G}}_h^{k,\beta }\left(f\right)$ belongs to $L_{{\gamma }_{k,\beta }}^{\infty }\left(X_{2d+2}\right)$ such that

$${\parallel {\mathcal{G}}_h^{k,\beta }\left(f\right)\parallel }_{L_{{\gamma }_{k,\beta }}^{\infty }}\le {\parallel f\parallel }_{L_{k,\beta }^2}{\parallel h\parallel }_{L_{k,\beta }^2}.$$

(29)

 2. 

A Plancherel-type formula:

$${∥{\mathcal{G}}_h^{k,\beta }\left(f\right)∥}_{L_{{\gamma }_{k,\beta }}^2}={\parallel h\parallel }_{L_{k,\beta }^2}{\parallel f\parallel }_{L_{k,\beta }^2}.$$

(30)

 3. 

A Parseval-type formula:

$${\displaystyle {\displaystyle {\int }_{X_{2d+2}}{\mathcal{G}}_h^{k,\beta }\left(f\right)(y,\nu )\overline{{\mathcal{G}}_h^{k,\beta }\left(g\right)(y,\nu )}d{\gamma }_{k,\beta }(y,\nu )}=\parallel h{\parallel }_{L_{k,\beta }^2}^2{\int }_{{\mathbb{R}}_{+}^{d+1}}f\left(x\right)\overline{g\left(x\right)}d{\mu }_{k,\beta }\left(x\right).}$$

(31)

 4. 

For any $p\in [2,\infty )$,

$${∥{\mathcal{G}}_h^{k,\beta }\left(f\right)∥}_{L_{{\gamma }_{k,\beta }}^p}\le \parallel f{\parallel }_{L_{k,\beta }^2}\parallel h{\parallel }_{L_{k,\beta }^2}.$$

(32)

Proof. 

The first statement can easily obtained from the Cauchy–Schwarz inequality and Inequality (25). Then, by (21), (27), (12), (22) and Plancherel’s formula (13), we have

$$\begin{array}{ccc}\hfill {∥{\mathcal{G}}_h^{k,\beta }\left(f\right)∥}_{L_{{\gamma }_{k,\beta }}^2}^2& =& {\displaystyle {\int }_{{\mathbb{R}}_{+}^{d+1}}{\int }_{{\mathbb{R}}_{+}^{d+1}}\left|{\mathcal{F}}^{k,\beta }\left(f\right)\left(\xi \right){|}^2\right|{\mathcal{F}}^{k,\beta }\left(h_{\nu }\right)\left(\xi \right){|}^{2}d{\mu }_{k,\beta }\left(\xi \right)d{\mu }_{k,\beta }\left(\nu \right)}\hfill \\ & =& {\displaystyle {\int }_{{\mathbb{R}}_{+}^{d+1}}{\int }_{{\mathbb{R}}_{+}^{d+1}}|{\mathcal{F}}^{k,\beta }\left(f\right)\left(\xi \right){|}^2{\tau }_{\nu }^{k,\beta }{\left(\right|h|}^2)(-{\xi }^{\prime },{\xi }_{d+1})d{\mu }_{k,\beta }\left(\xi \right)d{\mu }_{k,\beta }\left(\nu \right)}\hfill \\ & =& {\displaystyle {\int }_{{\mathbb{R}}_{+}^{d+1}}{\int }_{{\mathbb{R}}_{+}^{d+1}}|{\mathcal{F}}^{k,\beta }\left(f\right)\left(\xi \right){|}^2{\tau }_{-{\xi }^{\prime },{\xi }_{d+1}}^{k,\beta }{\left(\right|h|}^2)\left(\nu \right)d{\mu }_{k,\beta }\left(\nu \right)d{\mu }_{k,\beta }\left(\xi \right)}\hfill \\ & =& {\parallel f\parallel }_{L_{k,\beta }^2}{\parallel h\parallel }_{L_{k,\beta }^2}.\hfill \end{array}$$

This implies (31). Finally, Inequality (32) follows by applying the Riesz–Thorin interpolation theorem and the relations (29) and (30). □

Proposition 3.

Let $h\in L_{k,\beta ,rad}^2\left({\mathbb{R}}_{+}^{d+1}\right)\cap L_{k,\beta }^{\infty }\left({\mathbb{R}}_{+}^{d+1}\right)$. Then, ${\mathcal{G}}_h^{k,\beta }\left(L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)\right)$ is a reproducing kernel Hilbert space embedded as a subspace of $L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$, with kernel

$${\displaystyle {\mathcal{W}}_h({\nu }^{\prime },x^{\prime };\nu ,x):=\frac{1}{{\parallel h\parallel }_{L_{k,\beta }^2}^2}{\int }_{{\mathbb{R}}_{+}^{d+1}}{h}_{x^{\prime },{\nu }^{\prime }}\left(y\right)\overline{h_{x,\nu }\left(y\right)}d{\mu }_{k,\beta }\left(y\right).}$$

(33)

It is pointwise bounded, that is:

$$\left|{\mathcal{W}}_h({\nu }^{\prime },x^{\prime };\nu ,x)\right|\le 1,\forall (\nu ,x),({\nu }^{\prime },x^{\prime })\in X_{2d+2}.$$

(34)

Proof. 

By Parseval’s formula (31), we obtain

$${\displaystyle {\mathcal{G}}_h^{k,\beta }\left(f\right)(x,\nu )=\frac{1}{{\parallel h\parallel }_{L_{k,\beta }^2}^2}{\int }_{X_{2d+2}}{\mathcal{G}}_h^{k,\beta }\left(f\right)(x^{\prime },{\nu }^{\prime })\overline{{\mathcal{G}}_h^{k,\beta }\left(h_{x,\nu }\right)(x^{\prime },{\nu }^{\prime })}d{\gamma }_{k,\beta }(x^{\prime },{\nu }^{\prime }).}$$

Then, by Proposition 2, we can easily deduce that for every $(x,\nu )\in X_{2d+2}$, the function

$${\displaystyle x^{\prime }\mapsto \frac{1}{{\parallel h\parallel }_{L_{k,\beta }^2}^2}\overline{{\mathcal{G}}_h^{k,\beta }\left(h_{x,\nu }\right)(x^{\prime },{\nu }^{\prime })}=\frac{1}{\parallel h{\parallel }_{L_{k,\beta }^2}^2}{\int }_{{\mathbb{R}}_{+}^{d+1}}{h}_{x^{\prime },{\nu }^{\prime }}\left(y\right)\overline{h_{x,\nu }\left(y\right)}d{\mu }_{k,\beta }\left(y\right)}$$

belongs to $L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$. This achieves the proof. □

Now, in order to state an inversion formula for the Dunkl–Bessel Gabor transform, we need to establish a few lemmas.

Lemma 1.

For $h\in L_{k,\beta ,rad}^2\left({\mathbb{R}}_{+}^{d+1}\right)$ a nonzero function and N a positive integer, we define the functions $G_N$ and $H_N$ by:

$${\displaystyle G_N\left(x\right)={\int }_{B(0,N)}{\int }_{{\mathbb{R}}_{+}^{d+1}}{\Lambda }_{k,\beta }\left(-\xi ,x\right)|{\mathcal{F}}_{k,\beta }\left(h_{\nu }\right)\left(\xi \right){|}^{2}d{\mu }_{k,\beta }\left(\nu \right)d{\mu }_{k,\beta }\left(\xi \right),x\in {\mathbb{R}}_{+}^{d+1},}$$

(35)

and

$${\displaystyle H_N\left(\xi \right)={\int }_{B(0,N)}{\left|{\mathcal{F}}_{k,\beta }\left(h_{\nu }\right)\left(\xi \right)\right|}^{2}d{\mu }_{k,\beta }\left(\nu \right),\xi \in {\mathbb{R}}_{+}^{d+1},}$$

(36)

where

$$B(0,N):=\left\{\nu \in {\mathbb{R}}_{+}^{d+1}:\parallel \nu \parallel <N\right\}.$$

Then, we have

$$\begin{array}{c}\hfill G_N\in L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right),H_N\in L_{k,\beta }^1\left({\mathbb{R}}_{+}^{d+1}\right)\cap L_{k,\beta }^{\infty }\left({\mathbb{R}}_{+}^{d+1}\right),{\mathcal{F}}_{k,\beta }\left(G_N\right)=H_N.\end{array}$$

(37)

Proof. 

By Cauchy–Schwarz’s inequality, the sequence $G_N$ satisfies

$$\begin{array}{ccc}\hfill |G_N\left(x\right){|}^2& \le & {\displaystyle \left({\displaystyle {\int }_{B(0,N)}d{\mu }_{k,\beta }\left(\nu \right)}\right){\int }_{B(0,N)}|{\int }_{{\mathbb{R}}_{+}^{d+1}}{\Lambda }_{k,\beta }(-\xi ,x){\left|{\mathcal{F}}_{k,\beta }\left(h_{\nu }\right)\left(\xi \right)\right|}^{2}d{\mu }_{k,\beta }\left(\xi \right){|}^{2}d{\mu }_{k,\beta }\left(\nu \right)}\hfill \\ & \le & {\displaystyle C{\int }_{B(0,N)}|{\int }_{{\mathbb{R}}_{+}^{d+1}}{\Lambda }_{k,\beta }(-\xi ,x){\left|{\mathcal{F}}_{k,\beta }\left(h_{\nu }\right)\left(\xi \right)\right|}^{2}d{\mu }_{k,\beta }\left(\xi \right){|}^{2}d{\mu }_{k,\beta }\left(\nu \right).}\hfill \end{array}$$

On the other hand, by Proposition 2 and relations (7), (13), (12) and (22), we obtain:

$$\begin{array}{ccc}& & {\displaystyle {\int }_{{\mathbb{R}}_{+}^{d+1}}|G_N\left(x\right){|}^{2}d{\mu }_{k,\beta }\left(x\right)}\hfill \\ & \le & {\displaystyle C{\int }_{B(0,N)}{\int }_{{\mathbb{R}}_{+}^{d+1}}{\left|{\displaystyle {\int }_{{\mathbb{R}}_{+}^{d+1}}{\Lambda }_{k,\beta }(-\xi ,x){\left|{\mathcal{F}}_{k,\beta }\left(h_{\nu }\right)\left(\xi \right)\right|}^{2}d{\mu }_{k,\beta }\left(\xi \right)}\right|}^{2}d{\mu }_{k,\beta }\left(\nu \right)d{\mu }_{k,\beta }\left(x\right)}\hfill \\ & \le & {\displaystyle C{\int }_{B(0,N)}{\int }_{{\mathbb{R}}_{+}^{d+1}}{\left|{\mathcal{F}}_{k,\beta }^{-1}\left(\right|{\mathcal{F}}_{k,\beta }\left(h_{\nu }\right){|}^2)\left(x\right)\right|}^{2}d{\mu }_{k,\beta }\left(x\right)d{\mu }_{k,\beta }\left(\nu \right)}\hfill \\ & \le & {\displaystyle C{\int }_{B(0,N)}{\int }_{{\mathbb{R}}_{+}^{d+1}}|{\tau }_{\nu }^{k,\beta }{\left|h\right|}^2(-\xi ){|}^{2}d{\mu }_{k,\beta }\left(\nu \right)d{\mu }_{k,\beta }\left(\xi \right)}\hfill \\ & \le & {\displaystyle C{\int }_{B(0,N)}\parallel {\tau }_{\nu }^{k,\beta }{\left|h\right|}^2{\parallel }_{L_{k,\beta }^1\left({\mathbb{R}}_{+}^{d+1}\right)}\parallel {\tau }_{\nu }^{k,\beta }{\left|h\right|}^2{\parallel }_{L_{k,\beta }^{\infty }}d{\mu }_{k,\beta }\left(\nu \right)}\hfill \\ & \le & {\displaystyle C{\int }_{B(0,N)}\parallel {\tau }_{\nu }^{k,\beta }{\left|h\right|}^2{\parallel }_{L_{k,\beta }^{\infty }}d{\mu }_{k,\beta }\left(\nu \right)<\infty .}\hfill \end{array}$$

Then, by (22),

$$\begin{array}{ccc}\hfill \left|H_N\left(\xi \right)\right|& =& \left|{\displaystyle {\int }_{B(0,N)}|{\mathcal{F}}_{k,\beta }\left(h_{\nu }\right)\left(\xi \right){|}^{2}d{\mu }_{k,\beta }\left(\nu \right)}\right|\hfill \\ & =& {\displaystyle {\int }_{B(0,N)}{\tau }_{\nu }^{k,\beta }{\left|h\right|}^2(-{\xi }^{\prime },{\xi }_{d+1})d{\mu }_{k,\beta }\left(\nu \right)}\hfill \\ & \le & {\displaystyle {\int }_{{\mathbb{R}}_{+}^{d+1}}{\tau }_{\nu }^{k,\beta }{\left|h\right|}^2(-{\xi }^{\prime },{\xi }_{d+1})d{\mu }_{k,\beta }\left(\nu \right)}\hfill \\ & =& {\displaystyle {\int }_{{\mathbb{R}}_{+}^{d+1}}{\tau }_{-{\xi }^{\prime },{\xi }_{d+1}}^{k,\beta }{\left|h\right|}^2\left(\nu \right)d{\mu }_{k,\beta }\left(\nu \right)=\parallel h{\parallel }_{L_{k,\beta }^2}^2<\infty .}\hfill \end{array}$$

This implies that $H_N\in L_{k,\beta }^{\infty }\left({\mathbb{R}}_{+}^{d+1}\right)$. Therefore, by (16) we obtain

$$\begin{array}{ccc}\hfill \parallel H_N{\parallel }_{L_{k,\beta }^1}& =& {\displaystyle {\int }_{{\mathbb{R}}_{+}^{d+1}}\left|H_N\left(\xi \right)\right|d{\mu }_{k,\beta }\left(\xi \right)}\hfill \\ & =& {\displaystyle {\int }_{{\mathbb{R}}_{+}^{d+1}}\left|{\displaystyle {\int }_{B(0,N)}|{\mathcal{F}}_{k,\beta }\left(h_{\nu }\right)\left(\xi \right){|}^{2}d{\mu }_{k,\beta }\left(\nu \right)}\right|d{\mu }_{k,\beta }\left(\xi \right)}\hfill \\ & =& {\displaystyle {\int }_{B(0,N)}\left({\displaystyle {\int }_{{\mathbb{R}}_{+}^{d+1}}{\tau }_{\nu }^{k,\beta }{\left|h\right|}^2(-{\xi }^{\prime },{\xi }_{d+1})d{\mu }_{k,\beta }\left(\xi \right)}\right)d{\mu }_{k,\beta }\left(\nu \right)}\hfill \\ & \le & {\displaystyle {\parallel h\parallel }_{L_{k,\beta }^2}^2{\int }_{B(0,N)}d{\mu }_{k,\beta }\left(\nu \right)<\infty ,}\hfill \end{array}$$

which means that $H_N\in L_{k,\beta }^1\left({\mathbb{R}}_{+}^{d+1}\right)$. Finally, for all $x\in {\mathbb{R}}_{+}^{d+1}$,

$$\begin{array}{ccc}\hfill {\mathcal{F}}_{k,\beta }^{-1}\left(H_N\right)\left(x\right)& =& {\displaystyle {\int }_{{\mathbb{R}}_{+}^{d+1}}{H}_N\left(\xi \right){\Lambda }_{k,\beta }\left(-\xi ,x\right)d{\mu }_{k,\beta }\left(\xi \right)}\hfill \\ & =& {\displaystyle {\int }_{{\mathbb{R}}_{+}^{d+1}}{\Lambda }_{k,\beta }\left(-\xi ,x\right){\int }_{B(0,N)}|{\mathcal{F}}_{k,\beta }\left(h_{\nu }\right)\left(\xi \right){|}^{2}d{\mu }_{k,\beta }\left(\nu \right)d{\mu }_{k,\beta }\left(\xi \right)}\hfill \\ & =& {\displaystyle {\int }_{B(0,N)}{\int }_{{\mathbb{R}}_{+}^{d+1}}{\Lambda }_{k,\beta }\left(-\xi ,x\right)|{\mathcal{F}}_{k,\beta }\left(h_{\nu }\right)\left(\xi \right){|}^{2}d{\mu }_{k,\beta }\left(\nu \right)d{\mu }_{k,\beta }\left(\xi \right)=G_N\left(x\right).}\hfill \end{array}$$

This achieves the proof. □

Lemma 2.

The function $G_N$ can be written as

$$\begin{array}{c}\hfill {\displaystyle G_N\left(x\right)={\int }_{B(0,N)}{h}_{\nu }{*}_{k,\beta }\overline{\left(\check{h_{\nu }}\right)}\left(x\right)d{\mu }_{k,\beta }\left(\nu \right),x\in {\mathbb{R}}_{+}^{d+1}.}\end{array}$$

(38)

Proof. 

By Proposition 2, we have

$$\begin{array}{ccc}\hfill G_N\left(x\right)& =& {\displaystyle {\int }_{B(0,N)}{\int }_{{\mathbb{R}}_{+}^{d+1}}{\Lambda }_{k,\beta }\left(-\xi ,x\right)|{\mathcal{F}}_{k,\beta }\left(h_{\nu }\right)\left(\xi \right){|}^{2}d{\mu }_{k,\beta }\left(\nu \right)d{\mu }_{k,\beta }\left(\xi \right)}\hfill \\ & =& {\displaystyle {\int }_{B(0,N)}{\mathcal{F}}_{k,\beta }^{-1}\left(|{\mathcal{F}}_{k,\beta }\left(h_{\nu }\right){|}^2\right)\left(x\right)d{\mu }_{k,\beta }\left(\nu \right)}\hfill \\ & =& {\displaystyle {\int }_{B(0,N)}{h}_{\nu }{*}_{k,\beta }\overline{\left(\check{h_{\nu }}\right)}\left(x\right)d{\mu }_{k,\beta }\left(\nu \right).}\hfill \end{array}$$

This gives the desired result. □

Lemma 3.

Let h be in $L_{k,\beta ,rad}^2\left({\mathbb{R}}_{+}^{d+1}\right)\cap L_{k,\beta }^{\infty }\left({\mathbb{R}}_{+}^{d+1}\right)$ and N be any positive integer. Then, for all $f\in L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$, we have

$$\begin{array}{c}\hfill f_N=G_N{*}_{k,\beta }f,\end{array}$$

(39)

where $f_N$ is defined by

$$\begin{array}{c}\hfill {\displaystyle f_N\left(x\right)={\int }_{B(0,N)}{\int }_{{\mathbb{R}}_{+}^{d+1}}{\mathcal{G}}_h^{k,\beta }\left(f\right)(y,\nu )h_{y,\nu }\left(x\right)d{\gamma }_{k,\beta }(y,\nu ).}\end{array}$$

(40)

Proof. 

For all $x\in {\mathbb{R}}_{+}^{d+1}$,

$$\begin{array}{ccc}\hfill f_N\left(x\right)& =& {\displaystyle {\int }_{B(0,N)}{\int }_{{\mathbb{R}}_{+}^{d+1}}{\mathcal{G}}_h^{k,\beta }\left(f\right)(y,\nu ){\tau }_{-y^{\prime },y_{d+1}}^{k,\beta }{h}_{\nu }\left(x\right)d{\gamma }_{k,\beta }(y,\nu )}\hfill \\ & =& {\displaystyle {\int }_{B(0,N)}\left({\mathcal{G}}_h^{k,\beta }\left(f\right)(.,\nu ){*}_{k,\beta }{h}_{\nu }\right)\left(x\right)d{\mu }_{k,\beta }\left(\nu \right)}\hfill \\ & =& {\displaystyle {\int }_{B(0,N)}f{*}_{k,\beta }\overline{\left(\check{h_{\nu }}\right)}{*}_{k,\beta }{h}_{\nu }\left(x\right)d{\mu }_{k,\beta }\left(\nu \right)}\hfill \\ & =& {\displaystyle {\int }_{B(0,N)}{\int }_{{\mathbb{R}}_{+}^{d+1}}{\tau }_x^{k,\beta }f\left(y\right)\left(\overline{\left(\check{h_{\nu }}\right)}{*}_{k,\beta }{h}_{\nu }(-y^{\prime },y_{d+1})\right)d{\gamma }_{k,\beta }(y,\nu )}\hfill \\ & =& {\displaystyle {\int }_{{\mathbb{R}}_{+}^{d+1}}{\tau }_x^{k,\beta }f\left(y\right)\left({\displaystyle {\int }_{B(0,N)}\overline{\left(\check{h_{\nu }}\right)}{*}_{k,\beta }{h}_{\nu }(-y^{\prime },y_{d+1})d{\mu }_{k,\beta }\left(\nu \right)}\right)d{\mu }_{k,\beta }\left(y\right)}\hfill \\ & =& {\displaystyle {\int }_{{\mathbb{R}}_{+}^{d+1}}{\tau }_x^{k,\beta }f\left(y\right)G_N(-y^{\prime },y_{d+1})d{\mu }_{k,\beta }\left(y\right)}\hfill \\ & =& f{*}_{k,\beta }{G}_N\left(x\right).\hfill \end{array}$$

Since by Proposition 2 we have $\overline{\left(\check{h_{\nu }}\right)}{*}_{k,\beta }{h}_{\nu }\in L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$, then by using Young’s inequality and Parseval’s formula, we obtain

$$\begin{array}{ccc}\hfill {∥f{*}_{k,\beta }\overline{\left(\check{h_{\nu }}\right)}{*}_{k,\beta }{h}_{\nu }∥}_{L_{k,\beta }^{\infty }}& \le & \parallel f{\parallel }_{L_{k,\beta }^2}{∥\overline{\left(\check{h_{\nu }}\right)}{*}_{k,\beta }{h}_{\nu }∥}_{L_{k,\beta }^2}\hfill \\ & \le & C\parallel f{\parallel }_{L_{k,\beta }^2}\parallel h{\parallel }_{L_{k,\beta }^2}\parallel h{\parallel }_{L_{k,\beta }^{\infty }},\hfill \end{array}$$

and

$${\displaystyle {\int }_{B(0,N)}\left|f{*}_{k,\beta }\overline{\left(\check{h_{\nu }}\right)}{*}_{k,\beta }{h}_{\nu }\left(x\right)\right|d{\mu }_{k,\beta }\left(\nu \right)\le C\left({\displaystyle {\int }_{B(0,N)}d{\mu }_{k,\beta }\left(\nu \right)}\right)\parallel f{\parallel }_{L_{k,\beta }^2}\parallel h{\parallel }_{L_{k,\beta }^2}\parallel h{\parallel }_{L_{k,\beta }^{\infty }}.}$$

The proof is complete. □

We will now state the main result of this section.

Theorem 3

(inversion formula). Let $h\in L_{k,\beta ,rad}^2\left({\mathbb{R}}_{+}^{d+1}\right)\cap L_{k,\beta }^{\infty }\left({\mathbb{R}}_{+}^{d+1}\right)$ such that ${\parallel h\parallel }_{L_{k,\beta }^2}=1$. Then, for all $f\in L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$, the function $f_N$ belongs to $L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$ and satisfies

$$\begin{array}{c}\hfill \underset{N\to \infty }{lim}\parallel f-f_N{\parallel }_{L_{k,\beta }^2}=0.\end{array}$$

Proof. 

By Proposition 2, Lemma 1 and Equality (39), we derive that $f_N\in L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$ and

$${\mathcal{F}}_{k,\beta }\left(f_N\right)\left(\xi \right)=H_N\left(\xi \right){\mathcal{F}}_{k,\beta }\left(f\right)\left(\xi \right),\xi \in {\mathbb{R}}_{+}^{d+1}.$$

On the other hand, by (13) we obtain that $H_N\to 1$ pointwise as $N\to \infty$. Thus,

$$\begin{array}{ccc}\hfill \parallel f-f_N{\parallel }_{L_{k,\beta }^2}^2& =& {\displaystyle {\int }_{{\mathbb{R}}_{+}^{d+1}}|{\mathcal{F}}_{k,\beta }\left(f\right)\left(\xi \right)-H_N\left(\xi \right){\mathcal{F}}_{k,\beta }\left(f\right)\left(\xi \right){|}^{2}d{\mu }_{k,\beta }\left(\xi \right)}\hfill \\ & =& {\displaystyle {\int }_{{\mathbb{R}}_{+}^{d+1}}|{\mathcal{F}}_{k,\beta }\left(f\right)\left(\xi \right)(1-H_N\left(\xi \right)){|}^{2}d{\mu }_{k,\beta }\left(\xi \right),}\hfill \end{array}$$

which gives the desired result. □

3. Toeplitz Operators for the Dunkl-Bessel Gabor Transform

We begin this section by recalling some notions on Schatten–von Neumann classes. We will denote by ${\ell }^p\left(\mathbb{N}\right)$ the set of all infinite sequences of real (or complex) numbers $x:={\left(x_j\right)}_{j\in \mathbb{N}}$, such that

$$\begin{array}{ccc}{\parallel x\parallel }_p\hfill & :=\hfill & {\displaystyle {\left(\sum _{j=1}^{\infty }{\left|x_j\right|}^p\right)}^{\frac{1}{p}}<\infty ,if1\le p<\infty ,}\hfill \\ {\parallel x\parallel }_{\infty }\hfill & :=\hfill & {\displaystyle \underset{j\in \mathbb{N}}{sup}\left|x_j\right|<\infty .}\hfill \end{array}$$

For $p=2$, we provide the space ${\ell }^2\left(\mathbb{N}\right)$ with the scalar product

$${\displaystyle {\langle x,y\rangle }_2:=\sum _{j=1}^{\infty }{x}_j\overline{y_j}.}$$

If we denote by $B\left(L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)\right)$ the space of bounded operators from $L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$ into itself, then the singular values ${\left(s_n\left(T\right)\right)}_{n\in \mathbb{N}}$ of the compact operator $T\in B\left(L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)\right)$ are the eigenvalues of the positive self-adjoint operator $\left|T\right|=\sqrt{T^{*}T}$. In particular, for $1\le p<\infty$, the Schatten class $S_p$ is the space of all compact operators whose singular values lie in ${\ell }^p\left(\mathbb{N}\right)$. This space is equipped with the norm

$${\parallel T\parallel }_{S_p}:={\left(\sum _{n=1}^{\infty }{\left(s_n\left(T\right)\right)}^p\right)}^{{\displaystyle \frac{1}{p}}}.$$

(41)

Notice that $S_1$ is the space of trace-class operators, and $S_2$ is the space of Hilbert–Schmidt operators. Thus, if T is an operator in $S_1$, and ${\left(v_n\right)}_n$ is an orthonormal basis of $L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$, then the trace of T is defined by:

$$\mathrm{tr}\left(T\right)=\sum _{n=1}^{\infty }{\langle T{v}_n,v_n\rangle }_{L_{k,\beta }^2}.$$

(42)

In particular, if T is positive, then its trace satisfies

$$\mathrm{tr}\left(T\right)={\parallel T\parallel }_{S_1}.$$

(43)

Furthermore, if the positive operator $T^{*}T$ is in $S_1$, then any compact operator T on $L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$ is Hilbert–Schmidt. Following that, for any orthonormal basis ${\left(v_n\right)}_n$ of $L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$, we have

$${\parallel T\parallel }_{HS}^2:={\parallel T\parallel }_{S_2}^2=\parallel T^{*}{T\parallel }_{S_1}=\mathrm{tr}\left(T^{*}T\right)=\sum _{n=1}^{\infty }{\parallel T{v}_n\parallel }_{L_{k,\beta }^2}^2<\infty .$$

(44)

We set $S_{\infty }:=B\left(L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)\right)$, the space of bounded operators equipped with the standard norm:

$${\parallel T\parallel }_{S_{\infty }}:=\underset{u\in L_{k,\beta }^2:{\parallel u\parallel }_{L_{k,\beta }^2}=1}{sup}{\parallel Tu\parallel }_{L_{k,\beta }^2}.$$

(45)

Now, in order the prove the main results of this section, we will introduce the Toeplitz (or time–frequency) operator into our setting. To do this, let $\varsigma$ be a measurable function on $X_{2d+2}$ and h be a radial measurable function on ${\mathbb{R}}_{+}^{d+1}$; then, the Toeplitz operator ${\mathcal{L}}_h\left(\varsigma \right)$ related to the Dunkl–Bessel Gabor transform on $L_{k,\beta }^p\left({\mathbb{R}}_{+}^{d+1}\right)$, $1\le p\le \infty$ is defined by

$${\displaystyle {\mathcal{L}}_h\left(\varsigma \right)\left(f\right)\left(y\right)={\int }_{X_{2d+2}}\varsigma (x,\nu ){\mathcal{G}}_h^{k,\beta }\left(f\right)(x,\nu )h_{\nu ,x}\left(y\right)d{\gamma }_{k,\beta }(x,\nu ),y\in {\mathbb{R}}_{+}^{d+1}.}$$

(46)

The last relation is often better understood in a weak sense, that is, for any $f\in L_{k,\beta }^p\left({\mathbb{R}}_{+}^{d+1}\right)$ and $g\in L_{k,\beta }^{p^{\prime }}\left({\mathbb{R}}_{+}^{d+1}\right)$, $p\in [1,\infty ]$,

$$\begin{array}{c}\hfill {\displaystyle {〈{\mathcal{L}}_h\left(\varsigma \right)\left(f\right),g〉}_{L_{k,\beta }^2}={\int }_{X_{2d+2}}\varsigma (x,\nu ){\mathcal{G}}_h^{k,\beta }\left(f\right)(x,\nu )\overline{{\mathcal{G}}_h^{k,\beta }\left(g\right)(x,\nu )}d{\gamma }_{k,\beta }(x,\nu ).}\end{array}$$

(47)

For $p\in [1,\infty )$, the adjoint of the operator ${\mathcal{L}}_h\left(\varsigma \right):L_{k,\beta }^p\left({\mathbb{R}}_{+}^{d+1}\right)\to L_{k,\beta }^p\left({\mathbb{R}}_{+}^{d+1}\right)$ is ${\mathcal{L}}_h\left(\overline{\varsigma }\right):L_{k,\beta }^{p^{\prime }}\left({\mathbb{R}}_{+}^{d+1}\right)\to L_{k,\beta }^{p^{\prime }}\left({\mathbb{R}}_{+}^{d+1}\right)$ such that

$${\mathcal{L}}_h^{*}\left(\varsigma \right)={\mathcal{L}}_h\left(\overline{\varsigma }\right).$$

(48)

In the remaining part of this section, h is a function in $L_{k,\beta ,rad}^2\left({\mathbb{R}}_{+}^{d+1}\right)$ such that ${\parallel h\parallel }_{L_{k,\beta }^2}=1.$

Proposition 4.

If $\varsigma \in L_{{\gamma }_{k,\beta }}^1\left(X_{2d+2}\right)$, then ${\mathcal{L}}_h\left(\varsigma \right)$ is in $S_{\infty }$ with

$$\parallel {\mathcal{L}}_h\left(\varsigma \right){\parallel }_{S_{\infty }}\le \parallel \varsigma {\parallel }_{L_{{\gamma }_{k,\beta }}^1}.$$

(49)

Proof. 

From (47) and (29), we have for all $f,g\in L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$,

$$\begin{array}{ccc}\hfill \left|{〈{\mathcal{L}}_h\left(\varsigma \right)\left(f\right),g〉}_{L_{k,\beta }^2}\right|& \le & {\displaystyle {\int }_{X_{2d+2}}\left|\varsigma (x,\nu )\right|\left|{\mathcal{G}}_h^{k,\beta }\left(f\right)(x,\nu )\right|\left|\overline{{\mathcal{G}}_h^{k,\beta }\left(g\right)(x,\nu )}\right|d{\gamma }_{k,\beta }(x,\nu )}\hfill \\ & \le & {∥{\mathcal{G}}_h^{k,\beta }\left(f\right)∥}_{L_{{\gamma }_{k,\beta }}^{\infty }}{∥{\mathcal{G}}_h^{k,\beta }\left(g\right)∥}_{L_{{\gamma }_{k,\beta }}^{\infty }}\parallel \varsigma {\parallel }_{L_{{\gamma }_{k,\beta }}^1}\hfill \\ & \le & \parallel f{\parallel }_{L_{k,\beta }^2}\parallel g{\parallel }_{L_{k,\beta }^2}\parallel \varsigma {\parallel }_{L_{{\gamma }_{k,\beta }}^1}.\hfill \end{array}$$

Then, by (45), we obtain the desired result. □

Proposition 5.

If $\varsigma \in L_{{\gamma }_{k,\beta }}^{\infty }\left(X_{2d+2}\right)$, then the Toeplitz operator ${\mathcal{L}}_h\left(\varsigma \right)$ is in $S_{\infty }$ and we have

$$\begin{array}{c}\hfill \parallel {\mathcal{L}}_h\left(\varsigma \right){\parallel }_{S_{\infty }}\le \parallel \varsigma {\parallel }_{L_{{\gamma }_{k,\beta }}^{\infty }}.\end{array}$$

Proof. 

For $f,g\in L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$,

$$\begin{array}{ccc}\hfill \left|{〈{\mathcal{L}}_h\left(\varsigma \right)\left(f\right),g〉}_{L_{k,\beta }^2}\right|& \le & {\displaystyle {\int }_{X_{2d+2}}\left|\varsigma (x,\nu )\right|\left|{\mathcal{G}}_h^{k,\beta }\left(f\right)(x,\nu )\right|\left|\overline{{\mathcal{G}}_h^{k,\beta }\left(g\right)(x,\nu )}\right|d{\gamma }_{k,\beta }(x,\nu )}\hfill \\ & \le & \parallel \varsigma {\parallel }_{L_{{\gamma }_{k,\beta }}^{\infty }}{∥{\mathcal{G}}_h^{k,\beta }\left(f\right)∥}_{L_{{\gamma }_{k,\beta }}^2}{∥{\mathcal{G}}_h^{k,\beta }\left(g\right)∥}_{L_{{\gamma }_{k,\beta }}^2}.\hfill \end{array}$$

By (30), we derive that

$$\begin{array}{c}\hfill \left|{〈{\mathcal{L}}_h\left(\varsigma \right)\left(f\right),g〉}_{L_{k,\beta }^2}\right|\le \parallel f{\parallel }_{L_{k,\beta }^2}\parallel g{\parallel }_{L_{k,\beta }^2}\parallel \varsigma {\parallel }_{L_{{\gamma }_{k,\beta }}^{\infty }}.\end{array}$$

Thus, by (45) we obtain the desired result. □

Theorem 4.

Let $1\le p\le \infty$ and let $\varsigma \in L_{{\gamma }_{k,\beta }}^p\left(X_{2d+2}\right)$. Then, there is a unique bounded linear operator ${\mathcal{L}}_h\left(\varsigma \right):L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)\to L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$ such that

$$\begin{array}{c}\hfill \parallel {\mathcal{L}}_h\left(\varsigma \right){\parallel }_{S_{\infty }}\le \parallel \varsigma {\parallel }_{L_{{\gamma }_{k,\beta }}^p}.\end{array}$$

Proof. 

For $f\in L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$, we consider the operator $\mathcal{T}:L_{{\gamma }_{k,\beta }}^1\left(X_{2d+2}\right)\cap L_{{\gamma }_{k,\beta }}^{\infty }\left(X_{2d+2}\right)⟶L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$ defined by

$$\begin{array}{c}\hfill \mathcal{T}\left(\varsigma \right):={\mathcal{L}}_h\left(\varsigma \right)\left(f\right).\end{array}$$

Then, by Propositions 4 and 5, we obtain

$$\parallel \mathcal{T}\left(\varsigma \right){\parallel }_{L_{k,\beta }^2}\le \parallel f{\parallel }_{L_{k,\beta }^2}\parallel \varsigma {\parallel }_{L_{{\gamma }_{k,\beta }}^1}$$

(50)

and

$$\parallel \mathcal{T}\left(\varsigma \right){\parallel }_{L_{k,\beta }^2}\le \parallel f{\parallel }_{L_{k,\beta }^2}\parallel \varsigma {\parallel }_{L_{{\gamma }_{k,\beta }}^{\infty }}.$$

(51)

Thus, by the Riesz–Thorin interpolation theorem (see [18] [Theorem 2] and [19] [Theorem 2.11]),

$$\parallel {\mathcal{L}}_h\left(\varsigma \right)\left(f\right){\parallel }_{L_{k,\beta }^2}=\parallel \mathcal{T}\left(\varsigma \right){\parallel }_{L_{k,\beta }^2}\le \parallel f{\parallel }_{L_{k,\beta }^2}\parallel \varsigma {\parallel }_{L_{{\gamma }_{k,\beta }}^p}.$$

(52)

Since (52) is true for arbitrary function f in $L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$, then we obtain the desired result. □

The second aim of this section is to prove that the Toeplitz operator

$${\mathcal{L}}_h\left(\varsigma \right):L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)\to L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$$

is in the Schatten class $S_p$.

Proposition 6.

If $\varsigma \in L_{{\gamma }_{k,\beta }}^1\left(X_{2d+2}\right)$, then ${\mathcal{L}}_h\left(\varsigma \right)$ is a Hilbert–Schmidt operator such that

$$\begin{array}{c}\hfill \parallel {\mathcal{L}}_h\left(\varsigma \right){\parallel }_{S_2}\le \parallel \varsigma {\parallel }_{L_{{\gamma }_{k,\beta }}^1}.\end{array}$$

Proof. 

Assume that $\{{\varphi }_j,j=1,2,\cdots \}$ is an orthonormal basis for $L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$. Next, by using Parseval’s identity and relations (26), (47) and (48), we obtain

$$\begin{array}{ccc}\hfill \sum _{j=1}^{\infty }\parallel {\mathcal{L}}_h\left(\varsigma \right)\left({\varphi }_j\right){\parallel }_{L_{k,\beta }^2}^2& =& \sum _{j=1}^{\infty }{〈{\mathcal{L}}_h\left(\varsigma \right)\left({\varphi }_j\right),{\mathcal{L}}_h\left(\varsigma \right)\left({\varphi }_j\right)〉}_{L_{k,\beta }^2}\hfill \\ & =& {\displaystyle \sum _{j=1}^{\infty }{\int }_{X_{2d+2}}\varsigma (x,\nu ){〈{\varphi }_j,h_{\nu ,x}〉}_{L_{k,\beta }^2}{\overline{〈{\mathcal{L}}_h\left(\varsigma \right)\left({\varphi }_j\right),h_{\nu ,x}〉}}_{L_{k,\beta }^2}d{\gamma }_{k,\beta }(x,\nu )}\hfill \\ & =& {\displaystyle {\int }_{X_{2d+2}}\varsigma (x,\nu )\sum _{j=1}^{\infty }{〈{\varphi }_j,h_{\nu ,x}〉}_{L_{k,\beta }^2}{〈{\mathcal{L}}_h^{*}\left(\varsigma \right)\left(h_{\nu ,x}\right),{\varphi }_j〉}_{L_{k,\beta }^2}d{\gamma }_{k,\beta }(x,\nu )}\hfill \\ & =& {\displaystyle {\int }_{X_{2d+2}}\varsigma (x,\nu ){〈{\mathcal{L}}_h^{*}\left(\varsigma \right)\left(h_{\nu ,x}\right),h_{\nu ,x}〉}_{L_{k,\beta }^2}d{\gamma }_{k,\beta }(x,\nu ).}\hfill \end{array}$$

Therefore, from (48), (49) and (25), we derive

$${\displaystyle \sum _{j=1}^{\infty }\parallel {\mathcal{L}}_h\left(\varsigma \right)\left({\varphi }_j\right){\parallel }_{L_{k,\beta }^2}^2\le {\int }_{X_{2d+2}}\left|\varsigma (x,\nu )\right|\parallel {\mathcal{L}}_h^{*}\left(\varsigma \right){\parallel }_{S_{\infty }}d{\gamma }_{k,\beta }(x,\nu )\le \parallel \varsigma {\parallel }_{L_{{\gamma }_{k,\beta }}^1}^2<\infty .}$$

(53)

Hence, according to (53) and [19] [Proposition 2.8], the operator ${\mathcal{L}}_h\left(\varsigma \right):L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)\to L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$ belongs to $S_2$ and is then compact. □

Proposition 7.

Let $1⩽p<\infty$ and let $\varsigma \in L_{{\gamma }_{k,\beta }}^p\left(X_{2d+2}\right).$ Then, the operator ${\mathcal{L}}_h\left(\varsigma \right)$ is compact.

Proof. 

Let ${\left({\varsigma }_j\right)}_{j\in \mathbb{N}}$ be a sequence in $L_{{\gamma }_{k,\beta }}^1\left(X_{2d+2}\right)\cap L_{{\gamma }_{k,\beta }}^{\infty }\left(X_{2d+2}\right)$ that converges to $\varsigma$ in $L_{{\gamma }_{k,\beta }}^p\left(X_{2d+2}\right)$. Then, Theorem 4 implies

$$\begin{array}{c}\hfill \parallel {\mathcal{L}}_{u,v}\left({\varsigma }_j\right)-{\mathcal{L}}_h\left(\varsigma \right){\parallel }_{S_{\infty }}\le \parallel {\varsigma }_j-\varsigma {\parallel }_{L_{{\gamma }_{k,\beta }}^p}.\end{array}$$

Therefore, in $S_{\infty }$, we have ${\mathcal{L}}_h\left({\varsigma }_j\right)\to {\mathcal{L}}_h\left(\varsigma \right)$ as $j\to \infty$. Thus, for all j, the operator ${\mathcal{L}}_h\left({\varsigma }_j\right)$ is compact by Proposition 6. Given that compact operators form a closed subspace of $S_{\infty }$, it follows that ${\mathcal{L}}_h\left(\varsigma \right)$ is also compact. □

Theorem 5.

Let $\varsigma \in L_{{\gamma }_{k,\beta }}^1\left(X_{2d+2}\right)$. Then,

 1. 

The operator ${\mathcal{L}}_h\left(\varsigma \right):L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)\to L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$ belongs to $S_1$ and

$$\parallel \tilde{\varsigma }{\parallel }_{L_{{\gamma }_{k,\beta }}^1}\le \parallel {\mathcal{L}}_h\left(\varsigma \right){\parallel }_{S_1}\le \parallel \varsigma {\parallel }_{L_{{\gamma }_{k,\beta }}^1},$$

(54)

where $\tilde{\varsigma }$ is given by

$$\begin{array}{c}\hfill \tilde{\varsigma }(x,\nu )={〈{\mathcal{L}}_h\left(\varsigma \right)h_{\nu ,x},h_{\nu ,x}〉}_{L_{k,\beta }^2},(x,\nu )\in X_{2d+2}.\end{array}$$

 2. 

This trace formula occurs:

$${\displaystyle \mathrm{tr}\left({\mathcal{L}}_h\left(\varsigma \right)\right)={\int }_{X_{2d+2}}\varsigma (x,\nu )\parallel h_{x,\nu }{\parallel }_{L_{k,\beta }^2}^{2}d{\gamma }_{k,\beta }(x,\nu ).}$$

(55)

Proof. 

If $\varsigma \in L_{{\gamma }_{k,\beta }}^1\left(X_{2d+2}\right)$, then, according to Proposition 6, we have ${\mathcal{L}}_h\left(\varsigma \right)\in S_2$. Moreover, by [19] [Theorem 2.2]), the kernel’s orthogonal complement of ${\mathcal{L}}_h\left(\varsigma \right)$ has an orthonormal basis $\{{\varphi }_j,j=1,2,\dots \}$, which is made up of eigenvectors of $|{\mathcal{L}}_h\left(\varsigma \right)|$ and $\{{\phi }_j,j=1,2,\dots \}$ an orthonormal set in $L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$, such that

$$\begin{array}{c}\hfill {\mathcal{L}}_h\left(\varsigma \right)\left(f\right)=\sum _{j=1}^{\infty }{s}_j{〈f,{\varphi }_j〉}_{L_{k,\beta }^2}{\phi }_j,\end{array}$$

(56)

where the positive singular values of ${\mathcal{L}}_h\left(\varsigma \right)$ corresponding to ${\varphi }_j$ are $s_j,j=1,2,\cdots$. Next, we obtain

$$\begin{array}{c}\hfill \parallel {\mathcal{L}}_h\left(\varsigma \right){\parallel }_{S_1}=\sum _{j=1}^{\infty }{s}_j=\sum _{j=1}^{\infty }{〈{\mathcal{L}}_h\left(\varsigma \right)\left({\varphi }_j\right),{\phi }_j〉}_{L_{k,\beta }^2}.\end{array}$$

Therefore, using Bessel and Cauchy–Schwarz’s inequalities together with relations (25) and (26), we obtain

$$\begin{array}{ccc}\hfill \parallel {\mathcal{L}}_h\left(\varsigma \right){\parallel }_{S_1}& =& \sum _{j=1}^{\infty }{〈{\mathcal{L}}_h\left(\varsigma \right)\left({\varphi }_j\right),{\phi }_j〉}_{L_{k,\beta }^2}\hfill \\ & =& {\displaystyle \sum _{j=1}^{\infty }{\int }_{X_{2d+2}}\varsigma (x,\nu ){\mathcal{G}}_h^{k,\beta }\left({\varphi }_j\right)(x,\nu )\overline{{\mathcal{G}}_h^{k,\beta }\left({\phi }_j\right)(x,\nu )}d{\gamma }_{k,\beta }(x,\nu )}\hfill \\ & \le & {\displaystyle {\int }_{X_{2d+2}}\left|\varsigma (x,\nu )\right|{\left({\displaystyle \sum _{j=1}^{\infty }{\left|{\mathcal{G}}_h^{k,\beta }\left({\varphi }_j\right)(x,\nu )\right|}^2}\right)}^{1/2}{\left({\displaystyle \sum _{j=1}^{\infty }{\left|{\mathcal{G}}_h^{k,\beta }\left({\phi }_j\right)(x,\nu )\right|}^2}\right)}^{1/2}d{\gamma }_{k,\beta }(x,\nu )}\hfill \\ & \le & {\displaystyle {\int }_{X_{2d+2}}\left|\varsigma (x,\nu )\right|\parallel h_{\nu ,x}{\parallel }_{L_{k,\beta }^2}^{2}d{\gamma }_{k,\beta }(x,\nu )}\hfill \\ & \le & \parallel \varsigma {\parallel }_{L_{{\gamma }_{k,\beta }}^1}.\hfill \end{array}$$

Thus, $\parallel {\mathcal{L}}_h\left(\varsigma \right){\parallel }_{S_1}\le \parallel \varsigma {\parallel }_{L_{{\gamma }_{k,\beta }}^1}.$

On the other hand, we have that $\tilde{\varsigma }$ belongs to $L_{{\gamma }_{k,\beta }}^1\left(X_{2d+2}\right)$, and by (56), we have

$$\begin{array}{ccc}\hfill \left|\tilde{\varsigma }(x,\nu )\right|& =& \left|{〈{\mathcal{L}}_h\left(\varsigma \right)\left(h_{\nu ,x}\right),h_{\nu ,x}〉}_{L_{k,\beta }^2}\right|\hfill \\ & =& \left|\sum _{j=1}^{\infty }{s}_j{〈h_{\nu ,x},{\varphi }_j〉}_{L_{k,\beta }^2}{〈{\phi }_j,h_{\nu ,x}〉}_{L_{k,\beta }^2}\right|\hfill \\ & \le & {\displaystyle {\displaystyle \frac{1}{2}}\sum _{j=1}^{\infty }{s}_j\left(|{〈h_{\nu ,x},{\varphi }_j〉}_{L_{k,\beta }^2}{|}^2+|{〈h_{\nu ,x},{\phi }_j〉}_{L_{k,\beta }^2}{|}^2\right).}\hfill \end{array}$$

Then, using Fubini’s theorem

$$\begin{array}{ccc}\hfill {\displaystyle {\int }_{X_{2d+2}}\left|\tilde{\varsigma }(x,\nu )\right|d{\gamma }_{k,\beta }(x,\nu )}& \le & {\displaystyle {\displaystyle \frac{1}{2}}\sum _{j=1}^{\infty }{s}_j({\int }_{X_{2d+2}}{\left|{〈h_{\nu ,x},{\varphi }_j〉}_{L_{k,\beta }^2}\right|}^{2}d{\gamma }_{k,\beta }(x,\nu )}\hfill \\ & +& {\displaystyle {\int }_{X_{2d+2}}{\left|{〈h_{\nu ,x},{\phi }_j〉}_{L_{k,\beta }^2}\right|}^{2}d{\gamma }_{k,\beta }(x,\nu )).}\hfill \end{array}$$

Thus, by (30),

$$\begin{array}{c}\hfill {\displaystyle {\int }_{X_{2d+2}}\left|\tilde{\varsigma }(x,\nu )\right|d{\gamma }_{k,\beta }(x,\nu )\le \sum _{j=1}^{\infty }{s}_j=\parallel {\mathcal{L}}_h\left(\varsigma \right){\parallel }_{S_1}.}\end{array}$$

This allows us to conclude.

Now, assume that $\{{\varphi }_j,j=1,2,\cdots \}$ is an orthonormal basis for $L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$. Then, the operator ${\mathcal{L}}_h\left(\varsigma \right)\in S_1$. Therefore, using (42) and Parseval’s identity,

$$\begin{array}{ccc}\hfill \mathrm{tr}\left({\mathcal{L}}_h\left(\varsigma \right)\right)& =& \sum _{j=1}^{\infty }{〈{\mathcal{L}}_h\left(\varsigma \right)\left({\varphi }_j\right),{\varphi }_j〉}_{L_{k,\beta }^2}\hfill \\ & =& {\displaystyle \sum _{j=1}^{\infty }{\int }_{X_{2d+2}}\varsigma (x,\nu ){〈{\varphi }_j,h_{\nu ,x}〉}_{L_{k,\beta }^2}{\overline{〈{\varphi }_j,h_{\nu ,x}〉}}_{L_{k,\beta }^2}d{\gamma }_{k,\beta }(x,\nu )}\hfill \\ & =& {\displaystyle {\int }_{X_{2d+2}}\varsigma (x,\nu )\sum _{j=1}^{\infty }{〈{\varphi }_j,h_{\nu ,x}〉}_{L_{k,\beta }^2}{\overline{〈{\varphi }_j,h_{\nu ,x}〉}}_{L_{k,\beta }^2}d{\gamma }_{k,\beta }(x,\nu )}\hfill \\ & =& {\displaystyle {\int }_{X_{2d+2}}\varsigma (x,\nu )\parallel h_{\nu ,x}{\parallel }_{L_{k,\beta }^2}^{2}d{\gamma }_{k,\beta }(x,\nu ).}\hfill \end{array}$$

This achieves the proof. □

Involving Theorem 5 and Proposition 5 and using an interpolation argument (see [19] [Theorems 2.10 and 2.11]), we deduce the following result.

Corollary 1.

If $\varsigma \in L_{{\gamma }_{k,\beta }}^p\left(X_{2d+2}\right),1⩽p⩽\infty$, then the Toeplitz operator ${\mathcal{L}}_h\left(\varsigma \right):L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)⟶L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$ belongs to $S_p$ and

$$\begin{array}{c}\hfill \parallel {\mathcal{L}}_h\left(\varsigma \right){\parallel }_{S_p}\le \parallel \varsigma {\parallel }_{L_{{\gamma }_{k,\beta }}^p}.\end{array}$$

Remark 1.

If $\varsigma \in L_{{\gamma }_{k,\beta }}^1\left(X_{2d+2}\right)$ is positive and real-valued, then ${\mathcal{L}}_h\left(\varsigma \right):L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)\to L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$ is a positive operator. Moreover, using (43) and (55), we obtain

$${\displaystyle \parallel {\mathcal{L}}_h\left(\varsigma \right){|}_{S_1}={\int }_{X_{2d+2}}\varsigma (x,\nu )\parallel h_{\nu ,x}{\parallel }_{L_{k,\beta }^2}^{2}d{\gamma }_{k,\beta }(x,\nu ).}$$

(57)

Corollary 2.

If ${\varsigma }_1,{\varsigma }_2\in L_{{\gamma }_{k,\beta }}^1\left(X_{2d+2}\right)$ are positive and real-valued, then the trace-class operators ${\mathcal{L}}_h\left({\varsigma }_1\right),{\mathcal{L}}_h\left({\varsigma }_2\right)$ are positive, such that for all $j\in \mathbb{N}$:

$$\begin{array}{c}\hfill \parallel {\left({\mathcal{L}}_h\left({\varsigma }_1\right){\mathcal{L}}_h\left({\varsigma }_2\right)\right)}^j{\parallel }_{S_1}\le \parallel {\mathcal{L}}_h\left({\varsigma }_1\right){\parallel }_{S_1}^j\parallel {\mathcal{L}}_h\left({\varsigma }_2\right){\parallel }_{S_1}^j.\end{array}$$

Proof. 

If $A,B\in S_1$ are positive, then (see [20] [Theorem 1]),

$$\begin{array}{c}\hfill \mathrm{tr}{\left(AB\right)}^j\le {\left(\mathrm{tr}\left(A\right)\right)}^j{\left(\mathrm{tr}\left(B\right)\right)}^j,\forall j\in \mathbb{N}.\end{array}$$

Let $A={\mathcal{L}}_h\left({\varsigma }_1\right)$ and $B={\mathcal{L}}_h\left({\varsigma }_2\right)$. Then, by Remark 1, we obtain the desired result. □

4. Spectral Analysis for the Generalized Concentration Operator

In this section, h will be any function of $L_{k,\beta ,rad}^2\left({\mathbb{R}}_{+}^{d+1}\right)\cap L_{k,\beta }^{\infty }\left({\mathbb{R}}_{+}^{d+1}\right)$ such that ${\parallel h\parallel }_{L_{k,\beta }^2}=1$. We define ${\left({\mathcal{G}}_h^{k,\beta }\right)}^{*}:L_{{\gamma }_{k,\beta }}^2\left(X_{2d+2}\right)\to L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$ the adjoint of ${\mathcal{G}}_h^{k,\beta }$ by

$${\langle {\mathcal{G}}_h^{k,\beta }\left(f\right),g\rangle }_{L_{{\gamma }_{k,\beta }}^2}={〈f,{\left({\mathcal{G}}_h^{k,\beta }\right)}^{*}\left(g\right)〉}_{L_{k,\beta }^2},f\in L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right),g\in L_{{\gamma }_{k,\beta }}^2\left(X_{2d+2}\right).$$

We denote by $P_h:L_{{\gamma }_{k,\beta }}^2\left(X_{2d+2}\right)\to L_{{\gamma }_{k,\beta }}^2\left(X_{2d+2}\right)$ the orthogonal projection from $L_{{\gamma }_{k,\beta }}^2\left(X_{2d+2}\right)$ onto ${\mathcal{G}}_h^{k,\beta }\left(L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)\right)$ and $P_U:L_{{\gamma }_{k,\beta }}^2\left(X_{2d+2}\right)\to L_{{\gamma }_{k,\beta }}^2\left(X_{2d+2}\right)$ the orthogonal projection from $L_{{\gamma }_{k,\beta }}^2\left(X_{2d+2}\right)$ onto the subspace of functions supported in the subset $U\subset X_{2d+2}$, that is,

$$P_{U}F={\chi }_{U}F,F\in L_{{\gamma }_{k,\beta }}^2\left(X_{2d+2}\right),$$

where ${\chi }_U$ denotes the characteristic function of U.

We define the concentration operator ${\mathcal{L}}_h\left(U\right)$ by

$${\mathcal{L}}_h\left(U\right):={\mathcal{L}}_h\left(\varsigma \right)$$

where $\varsigma ={\chi }_U$, and U is a subset of $X_{2d+2}$ with finite measure $0<{\gamma }_{k,\beta }\left(U\right)<\infty$.

We have $P_h={\mathcal{G}}_h^{k,\beta }{\left({\mathcal{G}}_h^{k,\beta }\right)}^{*}$, and by (33), the operator $P_h$ can be written as follows

$${\displaystyle P_{h}F\left(z\right)={\int }_{X_{2d+2}}F(x,\nu ){\mathcal{W}}_h(z;x,\nu )d{\gamma }_{k,\beta }(x,\nu ),z=(x^{\prime },{\nu }^{\prime })\in X_{2d+2}.}$$

On the other hand, since ${\mathcal{W}}_h$ is the integral kernel of an orthogonal projection, then we have ${\mathcal{W}}_h(z;z^{\prime })=\overline{{\mathcal{W}}_h(z^{\prime };z)}$. Therefore,

$${\displaystyle {\mathcal{W}}_h(z;z^{\prime })={\int }_{X_{2d+2}}{\mathcal{W}}_h(z;z^{\prime \prime }){\mathcal{W}}_h(z^{\prime \prime };z^{\prime })d{\gamma }_{k,\beta }\left(z^{\prime \prime }\right),z,z^{\prime }\in X_{2d+2}.}$$

(58)

Moreover, for any orthonormal basis $\{v_j:j\in \mathbb{N}\}$ of ${\mathcal{G}}_h^{k,\beta }\left(L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)\right)$, the kernel ${\mathcal{W}}_h$ can be represented as follows

$${\mathcal{W}}_h(z;z^{\prime })=\sum _{j=1}^{\infty }{v}_j\left(z\right)\overline{v_j\left(z^{\prime }\right)},z,z^{\prime }\in X_{2d+2}.$$

(59)

4.1. Dunkl–Bessel Gabor Toeplitz Operator

Definition 3.

If ${\mathcal{G}}_h^{k,\beta }\left(f\right)$ is the Dunkl–Bessel Gabor transform of the function $f\in L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$ with respect to the window function $h\in L_{k,\beta ,rad}^2\left({\mathbb{R}}_{+}^{d+1}\right)$, then the spectrogram of f is defined by

$${\mathbf{G}}_h^{k,\beta }\left(f\right)={\left|{\mathcal{G}}_h^{k,\beta }\left(f\right)\right|}^2.$$

(60)

From the Plancherel-type formula for ${\mathcal{G}}_h^{k,\beta }$, we have

$${\displaystyle {\int }_{X_{2d+2}}{\mathbf{G}}_h^{k,\beta }\left(f\right)(x,\nu )d{\gamma }_{k,\beta }(x,\nu )=\parallel f{\parallel }_{L_{k,\beta }^2}^2.}$$

(61)

Spectrogram (60) is interpreted as a time–frequency energy density of the function $f\in L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$, and by (47), we have

$${\displaystyle {〈{\mathcal{L}}_h\left(U\right)f,f〉}_{L_{k,\beta }^2}={\int }_{X_{2d+2}}\varsigma (x,\nu ){\mathbf{G}}_h^{k,\beta }\left(f\right)(x,\nu )d{\gamma }_{k,\beta }(x,\nu ).}$$

(62)

Definition 4.

We define the Dunkl–Bessel Gabor Toeplitz operator

$$T_{h,U}:{\mathcal{G}}_h^{k,\beta }\left(L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)\right)\to {\mathcal{G}}_h^{k,\beta }\left(L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)\right)$$

by

$$T_{h,U}F=P_h{P}_{U}F.$$

(63)

Proposition 8.

The operator $T_{h,U}$ is bounded and positive, satisfying

$$0\le T_{h,U}\le P_U\le I,$$

(64)

and

$$T_{h,U}={\mathcal{G}}_h^{k,\beta }{\mathcal{L}}_h\left(U\right){\left({\mathcal{G}}_h^{k,\beta }\right)}^{*}.$$

(65)

Proof. 

For any F in ${\mathcal{G}}_h^{k,\beta }\left(L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)\right)$, we have

$${\displaystyle {〈T_{h,U}F,F〉}_{L_{{\gamma }_{k,\beta }}^2}={〈P_h\left(P_{U}F\right),F〉}_{L_{{\gamma }_{k,\beta }}^2}={〈P_{U}F,F〉}_{L_{{\gamma }_{k,\beta }}^2}={\int }_U|F(x,\nu ){|}^{2}d{\gamma }_{k,\beta }(x,\nu ).}$$

(66)

Then, we obtain (64). On the other hand, by (64), it follows that $T_{h,U}$ is positive and bounded.

Since the operator ${\mathcal{L}}_h\left(U\right):L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)\to L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$ can be written as

$${\mathcal{L}}_h\left(U\right)\left(f\right)={\left({\mathcal{G}}_h^{k,\beta }\right)}^{*}\left(P_U{\mathcal{G}}_h^{k,\beta }f\right),f\in L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right),$$

then we obtain

$$\left({\mathcal{G}}_h^{k,\beta }{\mathcal{L}}_h\left(U\right){\left({\mathcal{G}}_h^{k,\beta }\right)}^{*}\right)F=P_h{P}_{U}F=T_{h,U}F,F\in {\mathcal{G}}_h^{k,\beta }\left(L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)\right).$$

(67)

Thus,

$$T_{h,U}={\mathcal{G}}_h^{k,\beta }{\mathcal{L}}_h\left(U\right){\left({\mathcal{G}}_h^{k,\beta }\right)}^{*}.$$

The proof is complete. □

Proposition 9.

The operator $T_{h,U}$ is compact and trace-class, satisfying

$$\mathrm{tr}\left(T_{h,U}\right)=\mathrm{tr}\left({\mathcal{L}}_h\left(U\right)\right)={\mathcal{M}}_{k,\beta }(h,U),$$

(68)

where

$${\displaystyle {\mathcal{M}}_{k,\beta }(h,U):={\int }_U\parallel h_{\nu ,x}{\parallel }_{L_{k,\beta }^2}^{2}d{\gamma }_{k,\beta }(x,\nu ).}$$

(69)

Proof. 

We know that $T_{h,U}$ is a positive bounded operator. If ${\left\{v_j\right\}}_{j=1}^{\infty }$ is any orthonormal basis for ${\mathcal{G}}_h^{k,\beta }\left(L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)\right)$, then ${\left\{u_j\right\}}_{j=1}^{\infty }$ is an orthonormal basis for $L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$, where $u_j={\left({\mathcal{G}}_h^{k,\beta }\right)}^{*}\left(v_j\right)$. Thus, by (47) and the Fubini’s theorem, we obtain

$$\begin{array}{ccc}\hfill {\displaystyle \sum _{j=1}^{\infty }{〈T_{h,U}\left(v_j\right),v_j〉}_{L_{{\gamma }_{k,\beta }}^2}}& =& {\displaystyle \sum _{j=1}^{\infty }{〈{\mathcal{L}}_h\left(U\right){\left({\mathcal{G}}_h^{k,\beta }\right)}^{*}\left(v_j\right),{\left({\mathcal{G}}_h^{k,\beta }\right)}^{*}\left(v_j\right)〉}_{L_{k,\beta }^2}}\hfill \\ & =& {\displaystyle \sum _{j=1}^{\infty }{\int }_U{\left|{\mathcal{G}}_h^{k,\beta }\left(u_j\right)(x,\nu )\right|}^{2}d{\gamma }_{k,\beta }(x,\nu )}\hfill \\ & =& {\displaystyle {\int }_U\sum _{j=1}^{\infty }{\left|{\mathcal{G}}_h^{k,\beta }\left(u_j\right)(x,\nu )\right|}^{2}d{\gamma }_{k,\beta }(x,\nu )}\hfill \\ & =& {\displaystyle {\int }_U\sum _{j=1}^{\infty }{\left|{〈u_j,h_{\nu ,x}〉}_{L_{k,\beta }^2}\right|}^{2}d{\gamma }_{k,\beta }(x,\nu )}\hfill \\ & =& {\displaystyle {\int }_U\parallel h_{\nu ,x}{\parallel }_{L_{k,\beta }^2}^{2}d{\gamma }_{k,\beta }(x,\nu )}\hfill \\ & =& {\mathcal{M}}_{k,\beta }(h,U).\hfill \end{array}$$

Hence, by (41), (42) and (44), the operator $T_{h,U}$ is trace-class such that

$$\begin{array}{c}\hfill \parallel T_{h,U}{\parallel }_{S_1}=\mathrm{tr}\left(T_{h,U}\right)={\mathcal{M}}_{k,\beta }(h,U).\end{array}$$

This gives the desired result. □

Let ${\mathbf{V}}_{h,U}:L_{{\gamma }_{k,\beta }}^2\left(X_{2d+2}\right)\to L_{{\gamma }_{k,\beta }}^2\left(X_{2d+2}\right)$ be the operator defined by ${\mathbf{V}}_{h,U}=P_h{P}_U{P}_h$. This operator is more convenient than the operator $T_{h,U}$, since its domain is the whole space $L_{{\gamma }_{k,\beta }}^2\left(X_{2d+2}\right)$, and then its spectral properties can be obtained from its integral kernel.

As $T_{h,U}$ is positive and trace-class, then by the following decomposition:

$$L_{{\gamma }_{k,\beta }}^2\left(X_{2d+2}\right)={\mathcal{G}}_h^{k,\beta }\left(L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)\right)\oplus {\left({\mathcal{G}}_h^{k,\beta }\left(L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)\right)\right)}^{\perp },$$

it follows that ${\mathbf{V}}_{h,U}$ is also positive and trace-class, satisfying

$$\mathrm{tr}\left({\mathbf{V}}_{h,U}\right)=\mathrm{tr}\left(T_{h,U}\right)={\mathcal{M}}_{k,\beta }(h,U).$$

(70)

Proposition 10.

The trace of $T_{h,U}^2$ is given by the following relation

$${\displaystyle \mathrm{tr}\left(T_{h,U}^2\right)={\int }_U{\int }_U{\left|{\mathcal{W}}_h(x,\nu ;x^{\prime },{\nu }^{\prime })\right|}^{2}d{\gamma }_{k,\beta }(x,\nu )d{\gamma }_{k,\beta }(x^{\prime },{\nu }^{\prime }).}$$

(71)

Proof. 

Since ${\mathbf{V}}_{h,U}$ is positive, then

$$\mathrm{tr}\left(T_{h,U}^2\right)=\mathrm{tr}\left({\mathbf{V}}_{h,U}^2\right).$$

(72)

Moreover, as ${\mathcal{G}}_h^{k,\beta }\left(L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)\right)$ is a reproducing kernel Hilbert space with kernel ${\mathcal{W}}_h$, then for $F\in L_{{\gamma }_{k,\beta }}^2\left(X_{2d+2}\right)$,

$${\displaystyle {\mathbf{V}}_{h,U}F(x,\nu )={\int }_{X_{2d+2}}F(x^{\prime },{\nu }^{\prime }){\int }_{X_{2d+2}}{\chi }_U(b,y){\mathcal{W}}_h(x,\nu ;b,y){\mathcal{W}}_h(b,y;x^{\prime },{\nu }^{\prime })d{\gamma }_{k,\beta }(b,y)d{\gamma }_{k,\beta }(x^{\prime },{\nu }^{\prime }).}$$

Therefore, ${\mathbf{V}}_{h,U}$ has the integral kernel:

$${\displaystyle {\mathbf{N}}_{h,U}(x,\nu ;x^{\prime },{\nu }^{\prime })={\int }_{X_{2d+2}}{\chi }_U(b,y){\mathcal{W}}_h(x,\nu ;b,y){\mathcal{W}}_h(b,y;x^{\prime },{\nu }^{\prime })d{\gamma }_{k,\beta }(b,y).}$$

(73)

Thus,

$$\begin{array}{ccc}\hfill \mathrm{tr}\left({\mathbf{V}}_{h,U}^2\right)& =& {\displaystyle {\int }_{X_{2d+2}}{\int }_{X_{2d+2}}|{\mathbf{N}}_{h,U}(x,\nu ;x^{\prime },{\nu }^{\prime }){|}^{2}d{\gamma }_{k,\beta }(x,\nu )d{\gamma }_{k,\beta }(x^{\prime },{\nu }^{\prime })}\hfill \\ & =& {\displaystyle {\int }_{X_{2d+2}}{\int }_{X_{2d+2}}{\mathbf{N}}_{h,U}(x,\nu ;x^{\prime },{\nu }^{\prime })\overline{{\mathbf{N}}_{h,U}(x,\nu ;x^{\prime },{\nu }^{\prime })}d{\gamma }_{k,\beta }(x,\nu )d{\gamma }_{k,\beta }(x^{\prime },{\nu }^{\prime })}\hfill \\ & =& {\displaystyle {\int }_{X_{2d+2}}{\int }_{X_{2d+2}}{\chi }_U\left(z_1\right){\chi }_U\left(z_2\right){\mathbf{W}}_h(z_1;z_2)d{\gamma }_{k,\beta }\left(z_1\right)d{\gamma }_{k,\beta }\left(z_2\right),}\hfill \end{array}$$

where

$$\begin{array}{ccc}\hfill {\mathbf{W}}_h(z_1;z_2)& =& {\displaystyle {\int }_{X_{2d+2}}{\int }_{X_{2d+2}}{\mathcal{W}}_h(z_2;x,\nu ){\mathcal{W}}_h(x,\nu ;z_1){\mathcal{W}}_h(z_1;x^{\prime },{\nu }^{\prime })}\hfill \\ & \times & {\mathcal{W}}_h(x^{\prime },{\nu }^{\prime };z_2)d{\gamma }_{k,\beta }(x,\nu )d{\gamma }_{k,\beta }(x^{\prime },{\nu }^{\prime })\hfill \\ & =& {\mathcal{W}}_h(z_2;z_1){\mathcal{W}}_h(z_1;z_2).\hfill \end{array}$$

Hence, by (58), we have

$${\mathbf{W}}_h(z_1;z_2)=|{\mathcal{W}}_h(z_1;z_2){|}^2.$$

(74)

This completes the proof. □

4.2. Eigenvalues and Eigenfunctions

Since the concentration operator ${\mathcal{L}}_h\left(U\right)$ is compact and self-adjoint, then the spectral theorem gives the following spectral representation:

$${\displaystyle {\mathcal{L}}_h\left(U\right)\left(f\right)=\sum _{j=1}^{\infty }{s}_j\left(U\right){〈f,{\phi }_j^U〉}_{L_{k,\beta }^2}{\phi }_j^U,f\in L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right),}$$

(75)

where ${\left\{s_j\left(U\right)\right\}}_{j=1}^{\infty }$ denotes the positive eigenvalues arranged in a non-increasing way and ${\left\{{\phi }_j^U\right\}}_{j=1}^{\infty }$ its corresponding orthonormal set of eigenfunctions. As $s_j\left(U\right)\searrow 0$, then by (49) we obtain

$$s_j\left(U\right)\le s_1\left(U\right)\le 1,\forall j\ge 1.$$

(76)

Then, by (65), we obtain that the operator $T_{h,U}:{\mathcal{G}}_h^{k,\beta }\left(L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)\right)\to {\mathcal{G}}_h^{k,\beta }\left(L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)\right)$ can be diagonalized as

$${\displaystyle T_{h,U}F=\sum _{j=1}^{\infty }{s}_j\left(U\right){〈F,t_j^U〉}_{L_{{\gamma }_{k,\beta }}^2}{t}_j^U,F\in {\mathcal{G}}_h^{k,\beta }\left(L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)\right),}$$

(77)

where $t_j^U={\mathcal{G}}_h^{k,\beta }\left({\phi }_j^U\right)$.

Lemma 4.

For all $z=(x,\nu )\in X_{2d+2},$ we have

$${\displaystyle \Theta \left(z\right):={\int }_{X_{2d+2}}{\chi }_U\left(\omega \right)|{\mathcal{W}}_h(\omega ;z){|}^{2}d{\gamma }_{k,\beta }\left(\omega \right)=\sum _{j=1}^{\infty }{s}_j\left(U\right){\mathbf{G}}_h^{k,\beta }\left({\phi }_j^U\right)\left(z\right).}$$

(78)

Proof. 

By Proposition 3, we have that for any $z=(x,\nu )\in X_{2d+2}$, the function ${\mathcal{W}}_h(·;z)$ belongs to ${\mathcal{G}}_h^{k,\beta }\left(L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)\right)$. Therefore,

$$\begin{array}{ccc}\hfill {〈T_{h,U}{\mathcal{W}}_h(·;z),{\mathcal{W}}_h(·;z)〉}_{L_{{\gamma }_{k,\beta }}^2}& =& {〈P_U{\mathcal{W}}_h(·;z),{\mathcal{W}}_h(·;z)〉}_{L_{{\gamma }_{k,\beta }}^2}\hfill \\ & =& {\displaystyle {\int }_{X_{2d+2}}{\chi }_U\left(\omega \right){\mathcal{W}}_h(\omega ;z)\overline{{\mathcal{W}}_h(\omega ;z)}d{\gamma }_{k,\beta }\left(\omega \right)}\hfill \\ & =& {\displaystyle {\int }_{X_{2d+2}}{\chi }_U\left(\omega \right)|{\mathcal{W}}_h(\omega ;z){|}^{2}d{\gamma }_{k,\beta }\left(\omega \right).}\hfill \end{array}$$

If ${\left\{w_j^U\right\}}_{j=1}^{\infty }\subset {\mathcal{G}}_h^{k,\beta }\left(L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)\right)$ is an orthonormal basis for $Ker\left(T_{h,U}\right)$. Then, ${\left\{t_j^U\right\}}_{j=1}^{\infty }\cup {\left\{w_j^U\right\}}_{j=1}^{\infty }$ is an orthonormal basis of ${\mathcal{G}}_h^{k,\beta }\left(L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)\right)$, and then the reproducing kernel ${\mathcal{W}}_h$ can be written as follows

$${\displaystyle {\mathcal{W}}_h(x,\nu ;x^{\prime },{\nu }^{\prime })=\overline{{\mathcal{W}}_h(x^{\prime },{\nu }^{\prime };z)}=\sum _{j=1}^{\infty }{t}_j^U\left(z\right)\overline{t_j^U(x^{\prime },{\nu }^{\prime })}+\sum _{j=1}^{\infty }{w}_j^U\left(z\right)\overline{w_j^U(x^{\prime },{\nu }^{\prime })}.}$$

(79)

Thus,

$$\begin{array}{ccc}\hfill {〈T_{h,U}{\mathcal{W}}_h(·;z),{\mathcal{W}}_h(·;z)〉}_{L_{{\gamma }_{k,\beta }}^2}& =& {〈{\displaystyle T_{h,U}\sum _{j=1}^{\infty }\overline{t_j^U\left(z\right)}{t}_j^U,\sum _{m=1}^{\infty }\overline{t_m^U\left(z\right)}{t}_m^U}〉}_{L_{{\gamma }_{k,\beta }}^2}\hfill \\ & =& {\displaystyle \sum _{j,m}\overline{t_j^U\left(z\right)}{t}_m^U\left(z\right){〈T_{h,U}{t}_j^U,t_m^U〉}_{L_{{\gamma }_{k,\beta }}^2}}\hfill \\ & =& {\displaystyle \sum _{j=1}^{\infty }{s}_j\left(U\right)|t_j^U\left(z\right){|}^2.}\hfill \end{array}$$

The proof is complete. □

Proposition 11.

Let $0<\epsilon <1$. Define

$$N(\epsilon ,U):=card\left\{j:s_j\left(U\right)\ge 1-\epsilon \right\}.$$

(80)

Then, the eigenvalue distribution has the following estimate:

$$\begin{array}{ccc}\hfill |N(\epsilon ,U)-{\mathcal{M}}_{k,\beta }(h,U)|& \le & max\left\{{\displaystyle \frac{1}{\epsilon }},{\displaystyle \frac{1}{1-\epsilon }}\right\}\hfill \\ & \times & \left|{\displaystyle {\int }_U{\int }_U|{\mathcal{W}}_h(x^{\prime },{\nu }^{\prime };x,\nu ){|}^{2}d{\gamma }_{k,\beta }(x,\nu )d{\gamma }_{k,\beta }(x^{\prime },{\nu }^{\prime })-{\mathcal{M}}_{k,\beta }(h,U)}\right|.\hfill \end{array}$$

Proof. 

The proof is obtained directly by adaptation of the proof of Lemma 3.3 in [21]. □

4.3. Spectrogram of a Subspace

Given V as a N-dimensional subspace of $L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$, then the orthogonal projection $P_V$ onto V, with projection kernel ${\mathfrak{G}}_V$ is given by:

$${\displaystyle P_{V}f(·)={\int }_{{\mathbb{R}}_{+}^{d+1}}{\mathfrak{G}}_V(·,t)f\left(t\right)d{\mu }_{k,\beta }\left(t\right),}$$

(81)

where the kernel ${\mathfrak{G}}_V$ is defined by

$${\displaystyle {\mathfrak{G}}_V(x,t)=\sum _{j=1}^N{v}_j\left(x\right)\overline{v_j\left(t\right)},}$$

(82)

for any orthonormal basis ${\left\{v_j\right\}}_{j=1}^N$ of V.

Definition 5.

The spectrogram of the subspace V of $L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$, with respect to the window function h, is denoted by ${\mathbf{SPEC}}_h^{k,\beta }V$, and is defined by

$${\displaystyle {\mathbf{SPEC}}_h^{k,\beta }V(x,\nu ):={\int }_{{\mathbb{R}}_{+}^{d+1}}{\int }_{{\mathbb{R}}_{+}^{d+1}}{\mathfrak{G}}_V(t,y)\overline{h_{x,\nu }\left(t\right)}{h}_{x,\nu }\left(y\right)d{\mu }_{k,\beta }\left(t\right)d{\mu }_{k,\beta }\left(y\right).}$$

(83)

Lemma 5.

If V is a subspace of $L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$, then the spectrogram ${\mathbf{SPEC}}_h^{k,\beta }V$ can be written as

$${\displaystyle {\mathbf{SPEC}}_h^{k,\beta }V:=\sum _{j=1}^N{\mathbf{G}}_h^{k,\beta }\left(v_j\right).}$$

(84)

Proof. 

We have

$$\begin{array}{ccc}\hfill {\mathbf{SPEC}}_h^{k,\beta }V(x,\nu )& =& {\displaystyle {\int }_{{\mathbb{R}}_{+}^{d+1}}{\int }_{{\mathbb{R}}_{+}^{d+1}}\sum _{j=1}^N{v}_j\left(t\right)\overline{v_j\left(y\right)h_{x,\nu }\left(t\right)}{h}_{x,\nu }\left(y\right)d{\mu }_{k,\beta }\left(t\right)d{\mu }_{k,\beta }\left(y\right)}\hfill \\ & =& {\displaystyle \sum _{j=1}^N{〈v_j,h_{x,\nu }〉}_{L_{k,\beta }^2}{\overline{〈v_j,h_{x,\nu }〉}}_{L_{k,\beta }^2}}\hfill \\ & =& {\displaystyle \sum _{j=1}^N{\mathcal{G}}_h^{k,\beta }\left(v_j\right)(x,\nu )\overline{{\mathcal{G}}_h^{k,\beta }\left(v_j\right)(x,\nu )}}\hfill \\ & =& {\displaystyle \sum _{j=1}^N{\left|{\mathcal{G}}_h^{k,\beta }\left(v_j\right)(x,\nu )\right|}^2.}\hfill \end{array}$$

That achieves the proof. □

Let ${\xi }_{U,h}\left(V\right)$ be the time–frequency regional concentration of the subspace V in U. Then, it is defined by

$${\displaystyle {\xi }_{U,h}\left(V\right):=\frac{1}{N}{\int }_U{\mathbf{SPEC}}_h^{k,\beta }V(x,\nu )d{\gamma }_{k,\beta }(x,\nu ).}$$

(85)

By Lemma 5, it can be written as follows

$${\displaystyle {\xi }_{U,h}\left(V\right):=\frac{1}{N}\sum _{j=1}^N{\int }_U{\mathbf{G}}_h^{k,\beta }\left(v_j\right)(x,\nu )d{\gamma }_{k,\beta }(x,\nu ).}$$

(86)

Theorem 6.

The regional concentration ${\xi }_{U,h}\left(V\right)$ is maximized by the N-dimensional signal space $v_N=span{\left\{{\phi }_j^U\right\}}_{j=1}^N$, which is composed of the first N eigenfunctions of ${\mathcal{L}}_h\left(U\right)$ corresponding to the N greatest eigenvalues ${\left\{s_j\left(U\right)\right\}}_{j=1}^N$, and we have

$${\displaystyle {\xi }_{U,h}\left(v_N\right):=\frac{1}{N}\sum _{j=1}^N{s}_j\left(U\right).}$$

(87)

Proof. 

Using the min-max lemma for self-adjoint operators (see [22] [Section 95]), we obtain

$$\begin{array}{ccc}\hfill s_j\left(U\right)& =& {\displaystyle {\int }_U{\mathbf{G}}_h^{k,\beta }\left({\phi }_j^U\right)(x,\nu )d{\gamma }_{k,\beta }(x,\nu )}\hfill \\ & =& max\left\{{〈{\mathcal{L}}_h\left(U\right)\left(h\right),h〉}_{L_{k,\beta }^2}:\parallel h{\parallel }_{L_{k,\beta }^2}=1,h\perp {\phi }_1^U,\cdots ,{\phi }_{j-1}^U\right\}.\hfill \end{array}$$

Therefore, the number of orthogonal functions with a well-concentrated spectrogram in U is determined by the eigenvalues of ${\mathcal{L}}_h\left(U\right)$. Then, we obtain

$${\displaystyle {\xi }_{U,h}\left(v_N\right)=\frac{1}{N}\sum _{j=1}^N{s}_j\left(U\right).}$$

(88)

The first N eigenfunctions of the time–frequency operator ${\mathcal{L}}_h\left(U\right)$ have the best cumulative time–frequency concentration inside U, according to the min-max characterisation of the eigenvalues of compact operators. That is,

$${\displaystyle \sum _{j=1}^N{〈{\mathcal{L}}_h\left(U\right)\left({\phi }_j^U\right),{\phi }_j^U〉}_{L_{k,\beta }^2}=max\left\{\sum _{j=1}^N{〈{\mathcal{L}}_h\left(U\right)v_j,v_j〉}_{L_{k,\beta }^2}:{\left\{v_j\right\}}_{j=1}^N\mathrm{orthonormal}\right\}.}$$

(89)

Consequently, $v_N$ can be more concentrated in U than any N-dimensional subset V of $L_{k,\beta }^2\left({\mathbb{R}}_{+}^{d+1}\right)$; that is,

$${\xi }_{U,h}\left(V\right)\le {\xi }_{U,h}\left(v_N\right).$$

(90)

The proof is complete. □

Remark 2.

The time–frequency concentration of a subspace $v_N$ in U satisfies

$$s_N\left(U\right)\le {\xi }_{U,h}\left(v_N\right)\le s_1\left(U\right)\le 1.$$

(91)

4.4. Accumulated Spectrogram

We define the accumulated spectrogram by

$${\rho }_{(h,U)}^{k,\beta }:={\mathbf{SPEC}}_h^{k,\beta }{V}_{N_{k,\beta }(h,U)},$$

where $N_{k,\beta }(h,U)=\left[{\mathcal{M}}_{k,\beta }(h,U)\right]$ is the smallest integer greater than or equal to ${\mathcal{M}}_{k,\beta }(h,U)$ and

$$V_{N_{k,\beta }(h,U)}=span{\left\{v_j^U\right\}}_{j=1}^{N_{k,\beta }(h,U)}.$$

Then, we obtain

$${\displaystyle {\rho }_{(h,U)}^{k,\beta }(x,\nu )=\sum _{j=1}^{N_{k,\beta }(h,U)}{\left|{\mathcal{G}}_h^{k,\beta }\left(v_j^U\right)(x,\nu )\right|}^2=\sum _{j=1}^{N_{k,\beta }(h,U)}{\left|{\varphi }_j^U(x,\nu )\right|}^2.}$$

(92)

Therefore,

$${∥{\rho }_{(h,U)}^{k,\beta }∥}_{L_{{\gamma }_{k,\beta }}^1}=N_{k,\beta }(h,U)={\mathcal{M}}_{k,\beta }(h,U)+O\left(1\right).$$

Since

$$\begin{array}{c}\hfill {\displaystyle \sum _{j=1}^{N_{k,\beta }(h,U)}{s}_j\left(U\right)\le \mathrm{tr}\left({\mathcal{L}}_h\left(U\right)\right)={\mathcal{M}}_{k,\beta }(h,U),}\end{array}$$

then

$$E_{k,\beta }(h,U):=1-{\displaystyle \frac{{\sum }_{j=1}^{N_{k,\beta }(h,U)}{s}_j\left(U\right)}{{\mathcal{M}}_{k,\beta }(h,U)}}$$

(93)

satisfies

$$0\le E_{k,\beta }(h,U)\le 1.$$

(94)

Consequently, we obtain the following result.

Lemma 6.

If $\epsilon \in (0,1)$, then we have

$$0\le E_{k,\beta }(h,U)\le 1-(1-\epsilon )min\left\{1,{\displaystyle \frac{N(\epsilon ,U)}{{\mathcal{M}}_{k,\beta }(h,U)}}\right\}.$$

(95)

Proof. 

For $\epsilon \in (0,1)$, we define $l_{k,\beta }(\epsilon ,U)=min(N_{k,\beta }(h,U),N(\epsilon ,U))$. Then, it follows that

$$s_j\left(U\right)\ge 1-\epsilon ,1\le j\le l_{k,\beta }(\epsilon ,U).$$

(96)

Now, since $N_{k,\beta }(h,U)\ge l_{k,\beta }(\epsilon ,U)$, then

$${\displaystyle \sum _{j=1}^{N_{k,\beta }(h,U)}{s}_j\left(U\right)\ge \sum _{j=1}^{l_{k,\beta }(\epsilon ,U)}{s}_j\left(U\right)\ge (1-\epsilon )l_{k,\beta }(\epsilon ,U).}$$

(97)

Thus, it follows that

$$0\le E_{k,\beta }(h,U)\le 1-(1-\epsilon ){\displaystyle \frac{l_{k,\beta }(\epsilon ,U)}{{\mathcal{M}}_{k,\beta }(h,U)}}.$$

(98)

Hence, since $N_{k,\beta }(\epsilon ,U)\ge {\mathcal{M}}_{k,\beta }(h,U)$, we derive the desired result. □

Notice that $E_{k,\beta }(h,U)\to 0$ when the eigenvalues ${\left\{s_j\left(U\right)\right\}}_{j=0}^{N(\epsilon ,U)}$ are around 1. Furthermore, the error bounding the difference between ${\rho }_{(h,U)}^{k,\beta }$ and $\Theta$ is given as follows.

Proposition 12.

The following error estimate holds:

$${\displaystyle \frac{1}{{\mathcal{M}}_{k,\beta }(h,U)}}\parallel {\rho }_{(h,U)}^{k,\beta }-\Theta {\parallel }_{L_{{\gamma }_{k,\beta }}^1}\le {\displaystyle \frac{1}{{\mathcal{M}}_{k,\beta }(h,U)}}+2E_{k,\beta }(h,U).$$

(99)

Proof. 

For any $z=(x,\nu )\in U$, Lemma 4 implies

$${\displaystyle {\rho }_{(h,U)}^{k,\beta }\left(z\right)-\Theta \left(z\right)=\sum _{j=1}^{\infty }\left(t_j-s_j\left(U\right)\right)|{\varphi }_n^U\left(z\right){|}^2,}$$

(100)

where $t_j$ is given by

$$\begin{array}{c}\hfill t_j=\left\{\begin{array}{cc}1,\hfill & j\le N_{k,\beta }(h,U)\hfill \\ 0,\hfill & otherwise.\hfill \end{array}\right.\end{array}$$

(101)

Since

$$\begin{array}{c}\hfill {\displaystyle \parallel |{\varphi }_n^U{|}^2{\parallel }_{L_{{\gamma }_{k,\beta }}^1}=1\mathrm{and}\sum _{j=1}^{\infty }{s}_j\left(U\right)={\mathcal{M}}_{k,\beta }(h,U),}\end{array}$$

then

$$\begin{array}{ccc}\hfill \parallel {\rho }_{(h,U)}^{k,\beta }-\Theta {\parallel }_{L_{{\gamma }_{k,\beta }}^1}& \le & \sum _{j=1}^{\infty }|t_j-s_j\left(U\right)|\hfill \\ & =& {\displaystyle \sum _{j=1}^{N_{k,\beta }(h,U)}\left(1-s_j\left(U\right)\right)+\sum _{j>N_{k,\beta }(h,U)}{s}_j\left(U\right)}\hfill \\ & =& {\displaystyle N_{k,\beta }(h,U)+\sum _{j=1}^{\infty }{s}_j\left(U\right)-2\sum _{j=1}^{N_{k,\beta }(h,U)}{s}_j\left(U\right)}\hfill \\ & =& \left(N_{k,\beta }(h,U)-{\mathcal{M}}_{k,\beta }(h,U)\right)+2\left({\mathcal{M}}_{k,\beta }(h,U)-\sum _{j=1}^{N_{k,\beta }(h,U)}{s}_j\left(U\right)\right)\hfill \\ & \le & 1+2\left({\mathcal{M}}_{k,\beta }(h,U)-\sum _{j=1}^{N_{k,\beta }(h,U)}{s}_j\left(U\right)\right),\hfill \end{array}$$

that achieves the proof. □

5. Conclusions and Perspectives

In the present paper, we studied the harmonic analysis related to the Dunkl–Bessel Gabor transform and proved in particular a Plancherel-type formula and an inversion formula. Next, we introduced the Toeplitz operator in the Dunkl–Bessel Gabor setting, and studied its boundedness, compactness and Schatten class properties. Finally, we studied the spectral analysis of the generalized concentration operator, which is compact and self-adjoint. This allowed us to study the optimization problem, which aims to look for functions that have a well-concentrated spectrogram in a subset of the time–frequency plane. Then, we showed that the N-dimensional signal space, consisting of the first N eigenfunctions of the generalized concentration operator corresponding to the N largest eigenvalues, maximizes the concentration of the region of interest.

In this paper, we showed that there is a limitation of maximal time–frequency resolution. As a perspective, one can prove some qualitative or quantitative uncertainty principles for the Dunkl–Bessel Gabor transform, that set restrictions to the possible concentration of a function on the time–frequency plane.