Recasting the (Synchrosqueezed) Short-Time Fourier Transform as an Instantaneous Spectrum

Abstract

In a previous work, we proposed a time-frequency analysis called instantaneous spectral analysis (ISA), which generalizes the notion of the Fourier spectrum and in which instantaneous frequency is utilized to the fullest extent. In this paper, we recast both the Fourier transform (FT) and filterbank (FB) interpretations of the short-time Fourier transform (STFT) as instantaneous spectra. We show that to recast the FB interpretation of STFT as an instantaneous spectrum with valid structure, frequency reassignment is a fundamental necessity, thus demonstrating that this IS is closely related to the synchrosqueezed STFT. This result provides a new theoretical motivation for the synchrosqueezed STFT. Finally, we illustrate through example the instantaneous spectra corresponding to the FT and FB interpretations of STFT using two closed-form examples.

1. Introduction

In Gabor’s seminal work [1], the notion of joint time-frequency analysis was proposed and led to the development of the short-time Fourier transform (STFT) [2,3]. Even today, STFT is the most well-known and utilized method in time frequency analysis [4,5,6,7,8,9] and extensions such as the synchrosqueezing transform (SST) and synchrosqueezed STFT are still actively being investigated [10]. Synchrosqueezing and reassignment are typically motivated as post-processing techniques in order to improve readability in the time-frequency plane [11,12] or as refinements based on phase information.

In [13], we developed a generalized framework for time-frequency analysis which we termed instantaneous spectral analysis (ISA) and proved that the instantaneous spectrum (IS) exactly localizes signal components in an instantaneous bandwidth sense. Using ISA, we are able to define more general time-frequency spectra than is possible using STFT. This is a result of the basic time-frequency atom or component utilized in these methodologies. More specifically, ISA theory allows the use of an AM–FM component whereas STFT uses a far more restrictive component. Moreover, IS theory allows the unambiguous specification of an IS $S(t,\omega )$ and a related complex-valued signal $z\left(t\right)$ corresponding to a component set $\mathcal{S}$

Although this may be considered as an ideal synthesis model for instantaneous spectra and AM–FM models, the many-to-one mappings are sources of information loss which allow an infinite number of instantaneous spectra and component sets to map to the same signal. As a result, the reverse process is under determined and no unique analysis model exists.

One approach which can be taken for the analysis stage is to consider two stages: (1) signal decomposition and (2) component demodulation

where all ambiguity lies in how the signal $z\left(t\right)$ is decomposed into a set of components—there exist an infinite number of ways to express a whole as a sum of parts. However, every decomposition has the advantage that it can be associated with an instantaneous spectrum in which each component is exactly localized. Thus, ISA can be immediately used to enhance existing decomposition methods with an associated instantaneous spectrum with exact time-frequency localization. However, while the distinct separation of the analysis stage into decomposition and demodulation is well suited for pairing ISA with decomposition methods and AM–FM models, alternate approaches to the analysis stage exist.

For example, there exists a great body of literature devoted to the study of time-frequency distributions (TFDs), and although TFDs and ISs are both mathematical objects which describe signals in time-frequency spaces, they describe different types of spaces. Thus, it is natural to seek to establish the connection of particular TFDs as special cases of ISs, if and when this is possible. Let $\mathcal{T}$ denote an integral transformation relating a complex-valued signal $z\left(t\right)$ and a TFD $Z(t,\omega )$, and ${\mathcal{T}}^{-1}$ denote its inverse, assuming it exists

$$\begin{array}{c}z\left(t\right)\underset{{\mathcal{T}}^{-1}}{\overset{\mathcal{T}}{{\displaystyle \rightleftharpoons }}}Z(t,\omega ).\end{array}$$

The focus of traditional TFD analysis may be considered as the theory regarding the existence and mathematical properties for different choices of the transformation $\mathcal{T}$. ISA theory can be used in conjunction with other time-frequency methods as a framework to pose and address other important questions. For example, suppose we wish to study the performance of a particular integral transformation $\mathcal{T}$ on a particular class of signals. If we represent the particular class of signals by constraints on the form of the components in $\mathcal{S}$, then we can compare the IS obtained by recasting the time-frequency distribution $Z(t,\omega )$, denoted by $\widehat{S}(t,\omega )$, by considering

and comparing $\widehat{S}(t,\omega )$ to $S(t,\omega )$.

The purpose of this work is to show how to enforce the structure necessary to recast the time-frequency distribution $Z(t,\omega )$ to an instantaneous spectrum $\widehat{S}(t,\omega )$, if we choose the integral transformation $\mathcal{T}$ as the STFT. We show that there are two ways to do the recasting, each corresponding to one of the two classic interpretations of STFT. Our contributions are as follows:

• We show that the two equivalent STFT interpretations lead to different ISs, and thus provide new insights into STFT. In particular, we show that the IS corresponding to the FT interpretation of STFT corresponds to an IS for each window grain, while on the other hand, a single IS corresponding to the FB interpretation of STFT exists when STFT is synchrosqueezed [14,15]. As a result, in the FT interpretation, the components have a restrictive fixed amplitude and fixed frequency, while in the FB interpretation, the components are AM–FM in nature. This results in significant conceptual and practical differences between the two interpretations.
• We contribute a new theoretical motivation for synchrosqueezing. In particular, in order to recast the FB interpretation as an IS, we show that reassignment in frequency is a fundamental requirement. This is in contrast to the view of synchrosqueezing largely as a heuristic approach to improve energy concentration in the time-frequency plane.
• By recasting the two STFT interpretations as an IS, we can leverage the 3D IS visualization [13] to contribute a novel visualization of STFT. This is advantageous because the 3D IS allows visualization of multiple aspects of the signal decomposition simultaneously, including both magnitude and phase of each signal component. While it may take the reader some time to become comfortable with the 3D visualization, we believe it has significant advantages in terms of interpretability and note that the STFT phase spectrum is almost never considered or visualized.

For the benefit of the reader, this paper reviews key concepts of STFT and synchrosqueezed STFT, however, this work is not intended to be a review paper. For such reviews, we refer the reader to [2,3] for STFT and [14,16] for reassignment and synchrosqueezing. Rather, this paper recasts STFT as an IS, which we believe to be a more powerful signal analysis framework. The remainder of this paper is organized as follows. In Section 2, we introduce our notation for the continuous STFT and provide the expressions specifically related to the FT and FB interpretations. In Section 3, we provide a brief history of the development of reassignment techniques leading to the synchrosqueezed STFT. In Section 4, we provide a brief summary of ISA theory. However, as this work is an extension of our prior work, it is strongly recommended that [13] be read in advance. In Section 5, we give our main contribution by recasting the FT and FB interpretations of STFT as an IS. In Section 6, we provide illustrations and discussion on the relationships of the FT and FB interpretations of STFT to IS for example signals. Finally, in Section 7, we provide concluding remarks.

2. The Short-Time Fourier Transform

In this section, we review the (continuous) STFT following the development of the discrete STFT by Allen and Rabiner [2]. We begin by choosing a real, even-symmetric window function $\mathrm{w}(·)$ such that

$${\int }_{-\infty }^{\infty }\mathrm{w}\left(\tau \right)\mathrm{d}\tau =1.$$

(1)

This ensures that

$$\mathrm{w}(t-\tau )=\mathrm{w}(\tau -t)$$

(2)

and superimposing all “window grains” $z\left(t\right)\mathrm{w}(\tau -t)$ over $\tau$ gives

$$z\left(t\right)={\int }_{-\infty }^{\infty }z\left(t\right)\mathrm{w}(\tau -t)\mathrm{d}\tau .$$

(3)

Next, we review the two interpretations of STFT [2,3].

2.1. Fourier Transform Interpretation of STFT

The FT of (3) yields

$$\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} Z\left(\mathrm{j}\omega \right)={\int }_{-\infty }^{\infty }\left[{\int }_{-\infty }^{\infty }z\left(t\right)\mathrm{w}(\tau -t)\mathrm{d}\tau \right]{\mathrm{e}}^{-\mathrm{j}\omega t}\mathrm{d}t$$

(4a)

$$\begin{array}{ccc}& \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} =\hfill & \hfill {\int }_{-\infty }^{\infty }\left[{\int }_{-\infty }^{\infty }z\left(t\right)\mathrm{w}(\tau -t){\mathrm{e}}^{-\mathrm{j}\omega t}\mathrm{d}t\right]\mathrm{d}\tau \end{array}$$

(4b)

$$\begin{array}{ccc}& \hspace{1em}\hspace{1em}\hspace{1em}=\hfill & \hfill {\int }_{-\infty }^{\infty }\mathcal{F}\left\{z\left(t\right)\mathrm{w}(\tau -t)\phantom{0^0}\right\}\mathrm{d}\tau \end{array}$$

(4c)

$$\begin{array}{ccc}& =\hfill & \hfill {\int }_{-\infty }^{\infty }{Z}_{\mathrm{w}}(\omega ;\tau )\mathrm{d}\tau \end{array}$$

(4d)

where $Z_{\mathrm{w}}(\omega ;\tau )$ denotes the classical (the use of classical and modified for describing STFT is based on the current literature). See for example [4,17,18,19]) STFT. Equating the expressions inside the integrals of (4c) and (4d), shows that STFT may be considered as the FT of all window grains

$$Z_{\mathrm{w}}(\omega ;\tau )\triangleq \mathcal{F}\left\{z\left(t\right)\mathrm{w}(\tau -t)\phantom{0^0}\right\}.$$

(5)

This may be viewed as a function of $\omega$ at a fixed value of time shift $\tau$. The signal may be recovered by means of the overlap-add (OLA) method for short-time synthesis

$$z\left(t\right)\triangleq {\int }_{-\infty }^{\infty }{\mathcal{F}}^{-1}\left\{Z_{\mathrm{w}}(\omega ;\tau )\phantom{0^0}\right\}\mathrm{d}\tau .$$

(6)

2.2. Filterbank Interpretation of STFT

The FB interpretation of the modified STFT may be developed by considering $\mathrm{w}\left(t\right)e^{\mathrm{j}\nu t}$ as a channelizer with center frequency $\nu$

$$Z_{\mathrm{w}}(t;\nu )=z\left(t\right)\ast \mathrm{w}\left(t\right){\mathrm{e}}^{\mathrm{j}\nu t}$$

(7)

where ∗ denotes convolution. Equivalently, the classical STFT

$$Z_{\mathrm{w}}(t;\nu ){\mathrm{e}}^{-\mathrm{j}\nu t}\triangleq \left[z\left(t\right){\mathrm{e}}^{-\mathrm{j}\nu t}\phantom{0^0}\right]\ast \mathrm{w}\left(t\right)$$

(8)

which may be viewed as a function of t at a fixed value of $\nu$, i.e., the signal is frequency shifted and filtered with the impulse response $\mathrm{w}\left(t\right)$ on a continuum of frequency shifts $\nu$. The signal may be recovered by means of the filterbank summation (FBS) method for short-time synthesis

$$\begin{array}{ccc}z\left(t\right)& \triangleq & {\displaystyle \frac{1}{2\pi \mathrm{w}\left(0\right)}}{\int }_{-\infty }^{\infty }{Z}_{\mathrm{w}}(t;\nu )\mathrm{d}\nu \hfill \\ & =& {\displaystyle \frac{1}{2\pi \mathrm{w}\left(0\right)}}{\int }_{-\infty }^{\infty }\left[z\left(t\right)\ast \mathrm{w}\left(t\right){\mathrm{e}}^{\mathrm{j}\nu t}\right]\mathrm{d}\nu \hfill \\ & =& {\displaystyle \frac{1}{2\pi \mathrm{w}\left(0\right)}}{\int }_{-\infty }^{\infty }z\left(\tau \right)\mathrm{w}(t-\tau )\left[{\int }_{-\infty }^{\infty }{\mathrm{e}}^{\mathrm{j}\nu (t-\tau )}\mathrm{d}\nu \right]\mathrm{d}\tau \hfill \\ & =& {\displaystyle \frac{1}{2\pi \mathrm{w}\left(0\right)}}{\int }_{-\infty }^{\infty }z\left(\tau \right)\mathrm{w}(t-\tau )2\pi \delta (t-\tau )\mathrm{d}\tau \hfill \\ & =& {\displaystyle \frac{1}{\mathrm{w}\left(0\right)}}z\left(t\right)\mathrm{w}\left(0\right).\hfill \end{array}$$

(9)

2.3. Complimentary Interpretations

We point out to the reader a crucial difference between the meaning of the independent variables in (5) and (8) even though these equations are equivalent

$$Z_{\mathrm{w}}(t;\nu ){\mathrm{e}}^{-\mathrm{j}\nu t}\equiv Z_{\mathrm{w}}(\omega ;\tau ).$$

(10)

In (5), $\tau$ is a time shift variable and $\omega$ is instantaneous frequency (IF), while in (8), t is time instant and $\nu$ is a frequency shift variable. While this difference is well known and insignificant in STFT theory, it results in major differences in the relationship of STFT to IS based on the interpretation taken. Note that regardless of how STFT is computed, we may switch interpretations by interchanging variables $t\leftrightarrow \tau$ and variables $\omega \leftrightarrow \nu$ in (10).

3. Synchrosqueezed Short-Time Fourier Transform

In the 1970s, Kodera et al. [11,20] proposed to modify the spectrogram by taking into account the phase information that is usually discarded. The basic idea is to reassign energy in the spectrogram to a new time and frequency location by utilizing phase derivatives. Kodera’s work received little attention for decades [14], and in the 1980s Friedman also proposed spectrogram reassignment [21], apparently without knowledge of the work by Kodera. In Friedman’s approach, reassignment occurs in frequency but not in time. In both approaches, phase is used to perform reassignment but is subsequently discarded, thus preventing reconstruction. Slow adoption of these methods is often attributed to the inability to reconstruct the signal and the numerical problems associated with derivative approximations [12,14,16]. Auger and Flandrin showed that the numerical problems associated with the phase derivative could be avoided by computing three STFTs with related window functions, which led to a more efficient implementation.

In the 1990s, reassignment resurfaced when two independent groups developed the reassignment method (RM) [12,14,16] and the SST [22,23]. The RM developed by Auger and Flandrin is similar to Kodera’s work in that reassignments occur in both time and frequency. Additionally, they showed that the reassignment concept could be generalized to work for a broader class of time-frequency representations, e.g.,  in the Wigner–Ville distribution by recasting the problem in terms of centroids instead of phase derivatives. The SST developed by Maes and Daubechies is similar to Friedman’s work in that reassignments occur only in frequency using phase derivatives. However, differences from Friedman’s work include the use of a complex wavelet transform instead of the STFT and a reassignment of a complex value rather than a real value.

In [14,24], the SST is computed using an STFT rather than a wavelet transform, leading to the synchrosqueezed STFT

$$\begin{array}{c}\mathrm{SST}(t,\omega )=\frac{1}{2\pi \mathrm{w}\left(0\right)}{\int }_{-\infty }^{\infty }{Z}_{\mathrm{w}}(t;\nu ){\mathrm{e}}^{\mathrm{j}\omega t}\delta \left(\omega -\phantom{0^0}\widehat{\omega }(t;\nu )\phantom{0^0}\right) \mathrm{d}\nu \end{array}$$

where

$$\begin{array}{c}\hat{\omega }(t;\nu )=\frac{\mathrm{d}}{\mathrm{d}t}arg\left\{Z_{\mathrm{w}}(t;\nu )\right\}.\end{array}$$

4. Instantaneous Spectral Analysis

In [13], we introduced ISA as a general framework for time-frequency analysis consisting of three parts: (1) a parameter set, (2) an IS, and (3) a complex AM–FM signal model. More specifically, in this framework: (1) a signal is represented by a set of canonical triplets $\mathcal{S}\triangleq \left\{{\mathcal{C}}_0,{\mathcal{C}}_1,\dots ,{\mathcal{C}}_{K-1}\right\}$, (2) each set has a single-valued mapping to an IS $\mathcal{S}\mapsto S(t,\omega )$, and (3) each IS has a single-valued mapping to a signal $S(t,\omega )\mapsto z\left(t\right)$

$$\mathcal{S}=\left\{{\mathcal{C}}_k\right\}\stackrel{\mathrm{Equation}\left(14\right)}{⟼}S(t,\omega )\stackrel{\mathrm{Equation}\left(15\right)}{⟼}z\left(t\right).$$

(11)

The canonical triplet for the kth AM–FM component is

$${\mathcal{C}}_k\triangleq \left(a_k\left(t\right),{\omega }_k\left(t\right),{\varphi }_k\phantom{0^0}\right)$$

(12)

where $a_k\left(t\right)$ is the instantaneous amplitude (IA), ${\omega }_k\left(t\right)$ is the IF, and ${\varphi }_k$ is the phase reference. The kth complex AM–FM component is then given by

$$\begin{array}{ccc}\hfill \phantom{}{\psi }_k\left(t;{\mathcal{C}}_k\phantom{0^0}\right)& \triangleq \hfill & \hfill a_k\left(t\right){\mathrm{e}}^{\mathrm{j}\left[{\int }_{-\infty }^t{\omega }_k\left(\tau \right) \mathrm{d}\tau +{\varphi }_k\right]}\end{array}$$

(13a)

$$\begin{array}{ccc}& =\hfill & \hfill a_k\left(t\right){\mathrm{e}}^{\mathrm{j}{\theta }_k\left(t\right)}\end{array}$$

(13b)

$$\begin{array}{ccc}& \hspace{1em}=\hfill & \hfill s_k\left(t\right)+\mathrm{j}{\sigma }_k\left(t\right)\end{array}$$

(13c)

where ${\theta }_k\left(t\right)$ is the phase function, $s_k\left(t\right)$ is the real part, and ${\sigma }_k\left(t\right)$ is the imaginary part. With (12) and (13), the IS is defined as

$$\begin{array}{ccc}S(t,\omega ;\mathcal{S})& \triangleq & 2\pi {\displaystyle \sum _{k=0}^{K-1}}{\int }_{-\infty }^{\infty }{\psi }_k\left(\tau ;{\mathcal{C}}_k\phantom{0^0}\right){}^2\delta \left(t-\tau ,\phantom{0^0}\omega -{\omega }_k\left(\tau \right)\phantom{0^0}\right) \mathrm{d}\tau \hfill \\ & =& 2\pi {\displaystyle \sum _{k=0}^{K-1}}{\psi }_k\left(t;{\mathcal{C}}_k\phantom{0^0}\right)\delta \left(\phantom{0^0}\omega -{\omega }_k\left(t\right)\phantom{0^0}\right).\hfill \end{array}$$

(14)

The IS $S(t,\omega ;\mathcal{S})$ maps to the complex signal $z(t;\mathcal{S})$ with

$$\frac{1}{2\pi }{\int }_{-\infty }^{\infty }S(t,\omega ;\mathcal{S})\mathrm{d}\omega =z(t;\mathcal{S}).$$

(15)

Finally, the complex signal $z(t;\mathcal{S})$ is represented as a superposition of K (possibly infinite) complex AM–FM components

$$\begin{array}{ccc}\hfill z\left(t;\mathcal{S}\phantom{0^0}\right)& \triangleq \hfill & \hfill \sum _{k=0}^{K-1}{\psi }_k\left(t;{\mathcal{C}}_k\phantom{0^0}\right)\end{array}$$

(16a)

$$\begin{array}{ccc}& \hspace{1em}\hspace{1em} =\hfill & \hfill x\left(t\right)+\mathrm{j}y\left(t\right).\end{array}$$

(16b)

We refer the reader to [13] for additional details.

We emphasize to the reader that although IS is expressed by a (complex-valued) function of t and $\omega$, not every function of t and $\omega$ has the necessary structural requirements to be a valid IS. This is not unlike STFT which is also a (complex-valued) function of time and frequency and where not every function of time and frequency has the necessary structural requirements to be a valid STFT. For example, it is well understood that when modifying the STFT magnitude there is the distinct possibility that modification may lead to an invalid STFT. In this case, inversion requires algorithms such as least-squared error inverse STFT (LSE-ISTFT) which inverts the invalid STFT to the signal which has an STFT closest (in an LSE sense) to the invalid STFT [25,26,27].

Moreover, although both the STFT and IS provide spectral representations in time and frequency, they have different structural requirements. This is due to the fact that the requirements are imposed by the analysis equations in (5) and (8), whereas the requirements of IS are imposed by the definition in (14). One implication of this is that one cannot assume that an STFT has the necessary structure to be a valid IS. On one hand, we show that although the FT interpretation of STFT does not possess the necessary structure to be a valid IS, it may be interpreted as a continuum of ISs. On the other hand, we show that while the FB interpretation of STFT does not possess the necessary structure to be a valid IS, the structure necessary to utilize ISA theory may be imposed by synchrosqueezing the STFT.

Relation to Frequency Domain Analysis

In [13], we gave proof that frequency domain analysis corresponds to a specialized (and restricted) form of an IS when $a_k\left(t\right)=a_k{\omega }_0$, ${\omega }_k\left(t\right)=k{\omega }_0$, and the discrete set takes on a continuum, i.e., ${\omega }_0\to 0$,

$$\begin{array}{ccc}Z\left(\mathrm{j}\omega \right){\mathrm{e}}^{\mathrm{j}\omega t}& =& \underset{{\omega }_0\to 0}{lim}S(t,\omega ){\left.\phantom{0^{0^0}}\right|}_{\begin{array}{c}{a}_k\left(t\right)=a_k{\omega }_0\\ {\omega }_k\left(t\right)=k{\omega }_0\end{array}}\hfill \\ & =& S^{\mathrm{FD}}(t,\omega ).\hfill \end{array}$$

(17)

Finally, evaluating (17) at $t=0$ yields

$$Z\left(\mathrm{j}\omega \right)=S^{\mathrm{FD}}(0,\omega ).$$

(18)

We refer the reader to [13] for additional details.

5. Recasting the Short-Time Fourier Transform as an IS

IS provides a signal analysis which is both instantaneous in t and $\omega$. From Section 2, we see that STFT allows for instantaneous analysis in only one of the variables, i.e., the FB interpretation is instantaneous in time while the FT interpretation is instantaneous in frequency (albeit constant frequency). In this section, we recast each STFT interpretation as an IS. While the two interpretations are conceptual in nature, the ISs corresponding to these interpretations take on different mathematical forms. As we show, the IS corresponding to the FB interpretation $S^{\mathrm{FB}}(t,\omega )$ makes use of AM–FM components, and thus is easily understood in terms of a single IS. On the other hand, the IS corresponding to the FT interpretation $S_{\tau }^{\mathrm{FT}}(t,\omega )$ uses a more restrictive component and is best understood using a continuum of ISs. In this section, we continue from (11) and develop (19), (20), and (24), whose context in the overall theory is illustrated below

$$\begin{array}{ccccc}z\left(t\right)& \stackrel{\mathcal{F}}{⟼}& Z\left(\mathrm{j}\omega \right)& \stackrel{\mathrm{Equation}\left(17\right)}{⟼}& S^{\mathrm{FD}}(t,\omega )\\ z\left(t\right)& \stackrel{\mathrm{Equation}\left(5\right)}{⟼}& Z_{\mathrm{w}}(\omega ;\tau )& \stackrel{\mathrm{Equation}\left(19\right)}{⟼}& S_{\tau }^{\mathrm{FT}}(t,\omega )& \stackrel{\mathrm{Equation}\left(20\right)}{⟼}& S^{\mathrm{FD}}(t,\omega ).\\ z\left(t\right)& \stackrel{\mathrm{Equation}\left(7\right)}{⟼}& Z_{\mathrm{w}}(t;v)& \stackrel{\mathrm{Equation}\left(24\right)}{⟼}& S^{\mathrm{FB}}(t,\omega )\end{array}$$

5.1. IS Corresponding to the FT Interpretation of STFT

The FT interpretation of the classical STFT in (5) is that of a continuum of FTs indexed by $\tau$. Thus, from the relation of the FT to the IS in (17), the IS corresponding to the window grain at $t=\tau$ is

$$S_{\tau }^{\mathrm{FT}}(t,\omega )=Z_{\mathrm{w}}(\omega ;\tau ){\mathrm{e}}^{\mathrm{j}\omega t}.$$

(19)

Superimposing the ISs in (19) gives the IS corresponding to the FT interpretation of STFT

$$S^{\mathrm{FD}}(t,\omega )={\int }_{-\infty }^{\infty }{S}_{\tau }^{\mathrm{FT}}(t,\omega )\mathrm{d}\tau .$$

(20)

Equation (4d) shows that STFT is a decomposition of FT. Likewise, the continuum of ISs corresponding to the window grains decomposes the IS corresponding to the FT interpretation

$$\begin{array}{ccc}\hfill {\int }_{-\infty }^{\infty }{S}_{\tau }^{\mathrm{FD}}(t,\omega )\mathrm{d}\tau & =\hfill & \hfill {\int }_{-\infty }^{\infty }{Z}_{\mathrm{w}}(\omega ;\tau ){\mathrm{e}}^{\mathrm{j}\omega t}\mathrm{d}\tau \end{array}$$

(21a)

$$\begin{array}{cc}& \hfill \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}={\mathrm{e}}^{\mathrm{j}\omega t}{\int }_{-\infty }^{\infty }{Z}_{\mathrm{w}}(\omega ;\tau )\mathrm{d}\tau \end{array}$$

(21b)

$$\begin{array}{ccc}& \hspace{1em}\hspace{1em} =\hfill & \hfill {\mathrm{e}}^{\mathrm{j}\omega t}Z\left(\mathrm{j}\omega \right)\end{array}$$

(21c)

$$\begin{array}{ccc}& \hspace{1em}\hspace{1em}\hspace{1em}=\hfill & \hfill S^{\mathrm{FD}}(t,\omega ).\end{array}$$

(21d)

In other words, superimposing the ISs corresponding to the FT interpretation of STFT yields the IS corresponding to frequency domain analysis (Fourier transform) $S^{\mathrm{FD}}(t,\omega )$, and as a result does not provide a new IS to study. Rather, when taking the FT interpretation of STFT, we only gain new insights by studying the ISs corresponding to the window grains.

5.2. IS Corresponding to the FB Interpretation of STFT

The FB interpretation of the modified STFT in (7) is that of an infinite number of signal components, each corresponding to frequency shift $-\nu$ followed by filtering (convolution) with $\mathrm{w}(·)$. Naively comparing (15) with (9) one might be tempted to assume that a corresponding IS may be formed with

$$S^{\mathrm{FB}}(t,\omega )=\frac{1}{\mathrm{w}\left(0\right)}{Z}_{\mathrm{w}}(t;\nu ){\left.\phantom{0^0}\right|}_{\nu \to \omega }.$$

(22)

However, this would be incorrect because it does not provide the structure necessary to be a valid IS. This is further illustrated and discussed below. On the other hand, we can construct an IS with valid structure from $Z_{\mathrm{w}}(t;\nu )$ by reassigning the component associated with frequency shift $\nu$ to the appropriate IF. We begin by writing the modified STFT in polar form as

$$Z_{\mathrm{w}}(t;\nu )=a_{\mathrm{w}}(t;\nu )exp\left[\mathrm{j}{\theta }_{\mathrm{w}}(t;\nu )\phantom{0^0}\right].$$

(23)

Using the IS definition in (14) we have

$$S^{\mathrm{FB}}(t,\omega )={\int }_{-\infty }^{\infty }{a}_{\mathrm{w}}(t;\nu ){\mathrm{e}}^{\mathrm{j}{\theta }_{\mathrm{w}}(t;\nu )}\delta \left(\omega -\phantom{0^0}\frac{\mathrm{d}}{\mathrm{d}t}{\theta }_{\mathrm{w}}(t;\nu )\right) \mathrm{d}\nu$$

(24)

which is immediately recognized as a synchrosqueezed STFT [14,15]. We note that while most developments of reassignment/synchrosqueezing are motivated as post-processing techniques in order to improve readability of spectrograms, in our development, reassignment in frequency is a fundamental necessity to ensure a valid IS structure. Furthermore, techniques which reassign in time are not compatible with a valid IS structure.

5.3. Discussion

A critical difference between the ISs corresponding to FT and FB interpretations of STFT is the form of the components utilized. With the FT interpretation, the individual components are obtained from the classical STFT $Z_{\mathrm{w}}(\omega ;\tau )$, as follows. For each point $(\tau ,\omega )$, the component is formed as

$${\psi }_{\tau ,\omega }\left(t\right)=Z_{\mathrm{w}}(\omega ;\tau ){\mathrm{e}}^{\mathrm{j}\omega t}.$$

(25)

Here, $Z_{\mathrm{w}}(\omega ;\tau )$ acts as an initial condition, and multiplication with ${\mathrm{e}}^{\mathrm{j}\omega t}$ projects the component forward and backward in time. With the FB interpretation, the individual components are obtained from the modified STFT $Z_{\mathrm{w}}(t;\nu )$, as follows. For each frequency shift $\nu$, the component is formed as

$${\psi }_{\nu }\left(t\right)=Z_{\mathrm{w}}(t;\nu ).$$

(26)

While (25) and (26) have similar mathematical forms, they are very different because $Z_{\mathrm{w}}(\omega ;\tau )$ is independent of t while $Z_{\mathrm{w}}(t;\nu )$ is dependent on t. As a result, the components in (25) have a fixed amplitude $|Z_{\mathrm{w}}(\omega ;\tau )|$ and fixed frequency $\omega$, while the components in (26) have, in general, a time-varying amplitude $|Z_{\mathrm{w}}(t;\nu )|$ and time-varying frequency $\frac{\mathrm{d}}{\mathrm{d}t}arg\left\{Z_{\mathrm{w}}(t;\nu )\right\}$. Thus, the former component is very restrictive, whereas the latter component is AM–FM in nature. This results in significant conceptual and practical differences between $S_{\tau }^{\mathrm{FT}}(t,\omega )$ in (19) and $S^{\mathrm{FB}}(t,\omega )$ in (24), even though mathematically there is little practical difference between $Z_{\mathrm{w}}(\omega ;\tau )$ and $Z_{\mathrm{w}}(t;\nu )$. Moreover, (26) along with (23) explains why $Z_{\mathrm{w}}(t;\nu )$ does not have the necessary structure to be interpreted as an IS: the energy associated with the component ${\psi }_{{\nu }_0}\left(t\right)$ is located at channelizer frequency $\nu ={\nu }_0$ in $Z_{\mathrm{w}}(t;{\nu }_0)$, rather than at the appropriate IF location $\frac{\mathrm{d}}{\mathrm{d}t}arg\left\{Z_{\mathrm{w}}(t;{\nu }_0)\right)\}$.

6. Instantaneous Spectra Corresponding to STFT Interpretations for Example Signals

In this section, we illustrate through example the IS corresponding to the two interpretations of STFT. Information regarding visualization of the IS can be found in [13] and software for IS visualization at [28,29]. The examples shown below consist of two signals which can be expressed and analyzed in a closed-form as well as a real world signal, i.e., acoustic recording. In order to develop closed-form expressions for STFT, we choose to analyze the complex exponential and linear FM chirp with a Gaussian window. Our analysis uses the following FT pairs. First, the FT of a quadratic chirplet is given by

$$\begin{array}{c}z\left(t\right)=exp\left(-p_1{t}^2\right)\\ \updownarrow \mathcal{F}\\ Z\left(\mathrm{j}\omega \right)=\sqrt{{\displaystyle \frac{\pi }{p_1}}}exp\left[-{\displaystyle \frac{1}{4p_1}}{\omega }^2\right]\end{array}$$

(27)

where $p_1\in \mathbb{C}$ and $\mathrm{Re}\left\{p_1\right\}>0$. Second, it can be shown by completing the square and using (27), that the FT of a product of time-shifted quadratic chirplets is given by

$$\begin{array}{c}z\left(t\right)=\sqrt{{\displaystyle \frac{p_2}{\pi }}}exp\left[-p_1{(t-t_1)}^2\right]exp\left[-p_2{(t-t_2)}^2\right]\\ \updownarrow \mathcal{F}\\ Z\left(\mathrm{j}\omega \right)=cexp\left(-{\displaystyle \frac{1}{4p_3}}{\omega }^2\right)exp\left(-\mathrm{j}T\omega \right)\end{array}$$

(28)

with the chirp parameters $p_1\in \mathbb{C}$, $p_2\in \mathbb{C}$, $\mathrm{Re}\left\{p_1\right\}>0$, $\mathrm{Re}\left\{p_2\right\}>0$, $p_3=p_1+p_2$, $T=\left(t_1{p}_1+t_2{p}_2\right)/p_3$, and $c=\sqrt{p_2/p_3}exp\left[-(t_1^2{p}_1+t_2^2{p}_2-T^2{p}_3)\right]$.

6.1. Complex Exponential

In the first example, consider the canonical triplet

$${\mathcal{C}}_0=\left(1,{\omega }_0,0\right)$$

(29)

which using (16a) gives the complex exponential signal

$$z\left(t\right)=exp\left(\mathrm{j}{\omega }_{0}t\right)$$

(30)

and with (14) gives the IS

$$S(t,\omega )=2\pi \delta (\omega -{\omega }_0){\mathrm{e}}^{\mathrm{j}\omega t}.$$

(31)

Choosing a Gaussian window

$$\mathrm{w}\left(t\right)=\sqrt{{\displaystyle \frac{{\beta }^2}{2\pi }}}exp\left[-{\displaystyle \frac{{\beta }^2{t}^2}{2}}\right]$$

(32)

we compute the STFT corresponding to the FT interpretation by using (5), choosing $p_1={\beta }^2/2$ in (27), and using time- and frequency-shift properties of the FT

$$Z_{\mathrm{w}}(\omega ;\tau )=exp\left[\frac{-{(\omega -{\omega }_0)}^2}{2{\beta }^2}-\mathrm{j}\tau (\omega -{\omega }_0)\phantom{0^0}\right].$$

(33)

With (33) and (10) we then form the IS corresponding to the FT interpretation

$$S_{\tau }^{\mathrm{FT}}(t,\omega )=exp\left[\frac{-{(\omega -{\omega }_0)}^2}{2{\beta }^2}-\mathrm{j}\tau (\omega -{\omega }_0)\phantom{0^0}\right]{\mathrm{e}}^{\mathrm{j}\omega t}.$$

(34)

Next, we compute the STFT corresponding to the FB interpretation using (33) and (10)

$$Z_{\mathrm{w}}(t;\nu )=exp\left[\frac{-{(\nu -{\omega }_0)}^2}{2{\beta }^2}+\mathrm{j}{\theta }_{\mathrm{w}}(t;\nu )\phantom{0^0}\right]$$

(35)

where

$${\theta }_{\mathrm{w}}(t;\nu )={\omega }_{0}t.$$

(36)

Substituting into (24), we form the IS corresponding to the FB interpretation

$$\begin{array}{ccc}{S}^{\mathrm{FB}}(t,\omega )& =& {\int }_{-\infty }^{\infty }exp\left[\frac{-{(\nu -{\omega }_0)}^2}{2{\beta }^2}\right]{\mathrm{e}}^{\mathrm{j}t{\omega }_0}\delta \left(\omega -\phantom{0^0}{\omega }_0\right) \mathrm{d}\nu \hfill \\ & =& \beta \sqrt{2\pi }\delta \left(\omega -\phantom{0^0}{\omega }_0\phantom{0^0}\right){\mathrm{e}}^{\mathrm{j}{\omega }_{0}t}.\hfill \end{array}$$

(37)

Comparing (37) with (31), we see that after reassignment $S^{\mathrm{FB}}(t,\omega )$ yields the correct IS (to within a scale factor) for this signal. On the other hand, direct comparison of (34) with (31) is not possible. While one could superimpose (34) on the continuum $\tau$ as described in (20), this would only lead to $S^{\mathrm{FD}}(t,\omega )$. Although this does lead to a valid IS, it is not useful for time-frequency analysis because it provides the same information as FT.

The ISs corresponding to the FT interpretation of the STFT in (34) are shown in the left column of Figure 1. The top plot shows $S^{\mathrm{FD}}(t,\omega )$ for the complex exponential while the lower three plots show $S_{\tau }^{\mathrm{FT}}(t,\omega )$ for three different $\tau$ [see (19)]. The fact that the FT interpretation yields a representation that may be considered as a decomposition into a continuum of ISs of the window grains at each $\tau$ [see (21d)] is visually demonstrated by the “+” notation used in the figure. From the figure, it is apparent that the frequency spectrum which results from taking the FT of any window grain has components that extend beyond the time support of the window. Thus, even if a window grain has finite time support, the associated frequency spectrum has infinite extent and is not simply limited to the local vicinity of the window. This demonstrates that while the STFT is mathematically correct, there is a conceptual flaw in the IS obtained by recasting the FT interpretation of STFT.

The FB interpretation of the STFT in (35) is shown in Figure 2a. As discussed in Section 5.2, this STFT is not a valid IS and the structural problem may be seen in (36) and observed in the figure. In particular, the IF of the component corresponding to the channelizer with center frequency $\nu$ is given by $\frac{\mathrm{d}}{\mathrm{d}t}{\theta }_{\mathrm{w}}(t;\nu )={\omega }_0$, i.e., the IF has constant value ${\omega }_0$ and is thus independent of the value $\nu$. This can be seen in the figure by observing that the oscillation rate of components is fixed and does not change along the frequency axis. On the other hand, reassigning this IS using (24) gives (37), which is illustrated in Figure 2b.

6.2. Linear FM Chirp

In the second example, consider the canonical triplet

$${\mathcal{C}}_0=\left(1,{\omega }_0+{\omega }_{c}t,0\right)$$

(38)

which using (16a) gives the linear FM chirp signal

$$z\left(t\right)=exp\left[\mathrm{j}({\omega }_{0}t+\frac{{\omega }_c}{2}{t}^2)\right]$$

(39)

and with (14) gives the IS

$$S(t,\omega )=2\pi \delta \left(\omega -({\omega }_0+{\omega }_{c}t)\phantom{0^0}\right){\mathrm{e}}^{\mathrm{j}({\omega }_{0}t+\frac{{\omega }_c}{2}{t}^2)}.$$

(40)

Choosing the Gaussian window in (32), we compute the STFT corresponding to the FT interpretation by using (5); choosing $p_1=-\mathrm{j}{\omega }_c/2$, $p_2={\beta }^2/2$, $t_1=0$, and $t_2=\tau$ in (28) and using time- and frequency-shift properties of the FT

$$\begin{array}{c}{Z}_{\mathrm{w}}(\omega ;\tau )=\\ \left|\lambda \right|exp\left[\frac{{\beta }^2({\beta }^4-\gamma ){\tau }^2+{\beta }^2{\omega }_c\tau (\omega -{\omega }_0)-{\beta }^2{(\omega -{\omega }_0)}^2}{2\gamma }\right]\\ \times exp\left[\mathrm{j}\frac{{\beta }^4{\omega }_c{\tau }^2-2{\beta }^4\tau (\omega -{\omega }_0)-{\omega }_c{(\omega -{\omega }_0)}^2}{2\gamma }+\mathrm{j}arg\left\{\lambda \right\}\right]\end{array}$$

(41)

where $\lambda =\beta /\sqrt{{\beta }^2-\mathrm{j}{\omega }_c}$. The IS corresponding to the FT interpretation $S_{\tau }^{\mathrm{FT}}(t,\omega )$ is then formed by substituting the above into (19).

Next, we compute the STFT corresponding to the FB interpretation using (41) and (10), then, in polar form (see (23)) we have

$$\begin{array}{ccc}\hfill a_{\mathrm{w}}(t;\nu )& =\hfill & \left|\lambda \right|exp\left[\frac{({\beta }^4-\gamma )t^2+{\omega }_{c}t(\nu -{\omega }_0)-{(\nu -{\omega }_0)}^2}{2\gamma {\beta }^{-2}}\right]\end{array}$$

and

$$\begin{array}{ccc}\hfill {\theta }_{\mathrm{w}}(t;\nu )& =\hfill & \frac{{\beta }^4{\omega }_c}{2\gamma }{t}^2-\frac{{\beta }^4(\nu -{\omega }_0)}{\gamma }t-\frac{{\omega }_c{(\nu -{\omega }_0)}^2}{2\gamma }+\nu t+arg\left\{\lambda \right\}.\end{array}$$

Finally, the IF as a function of $\nu$ is given by

$$\begin{array}{ccc}\hfill \frac{\mathrm{d}}{\mathrm{d}t}{\theta }_{\mathrm{w}}(t;\nu )& =\hfill & \frac{{\beta }^4{\omega }_c}{\gamma }t-\frac{{\beta }^4(\nu -{\omega }_0)}{\gamma }+\nu .\end{array}$$

The IS corresponding to the FB interpretation of the STFT $S^{\mathrm{FB}}(t,\omega )$ is readily obtained from the equations above together with (24).

As in the previous example, $S_{\tau }^{\mathrm{FT}}(t,\omega )$ does not allow direct comparison with (40) because it yields a continuum of ISs. Thus, superposition of the IS continuum across $\tau$ would only lead to $S^{\mathrm{FD}}(t,\omega )$. However, unlike in the previous example, reassignment of $Z_{\mathrm{w}}(t;\nu )$ to form $S^{\mathrm{FB}}(t,\omega )$ does not yield the correct IS given in (40). Although reassignment improves energy concentration in time-frequency representations, it is unlikely to lead to the correct IS in general. We note that other variations of syncrosqueezing methods (e.g., higher order and adaptive methods) exist [18,19,30,31,32] that may perform well for specific signals, but in general there exists no single method that is well suited for all signals.

The ISs corresponding to the FT interpretation of the STFT in (41) are shown in the right column of Figure 1. The top plot shows $S^{\mathrm{FD}}(t,\omega )$ for the linear FM chirp while the lower three plots show $S_{\tau }^{\mathrm{FT}}(t,\omega )$ for three different $\tau$ [see (19)]. As before, it is apparent that the frequency spectrum which results from taking the FT of any window grain has components that extend beyond the time support of the window, further demonstrating the conceptual flaw in the FT interpretation of the STFT.

The FB interpretation of the STFT in (41) is shown in Figure 3a. Again, we see in the figure that the oscillation rate of components is fixed and does not change along the frequency axis. With reassignment, the resulting IS shown in Figure 3b [15] has improved energy concentration but does not give the correct IS provided in (40) and shown in Figure 3c.

6.3. Bat Vocalization

Finally, we illustrate the recasting of a synchrosqueezed STFT as an IS (i.e., the IS corresponding to the FB interpretation of the STFT) using a bat vocalization signal which is popular in the time-frequency literature [18,33,34,35]. The acoustic recording features a ∼2.5 ms pulse emitted by the Large Brown Bat Eptesicus Fuscus. The original recording consists of 400 samples captured with a sampling period of 7 $\mathsf{\mu }$s. In order to alleviate issues associated with numerical derivatives when recasting as an IS, we up-sampled the signal by $4\times$. Finally, a 128 point Hann window was used in the analysis.

The synchrosqueezed STFT, shown in Figure 4a, was computed using the fsst() function in Matlab and plotted using the default Matlab visualization (with a perceptual colormap). For comparison, the 2D IS corresponding to the FB interpretation of the STFT is shown in Figure 4b. Finally, the 3D IS corresponding to the FB interpretation of the STFT is shown in Figure 4c. Broadly speaking, the energy in Figure 4a,b are in general agreement, however, the IS provides more precision that allows the display of finer details. Moreover, while both Figure 4a,b provide information about the magnitude in the time-frequency plane, by leveraging the 3D IS visualization, we are able to additionally illustrate the spectral phase.

7. Conclusions

In this paper, we used the ISA framework to recast the FT and FB interpretations of STFT in terms of an IS. We showed that these two equivalent STFT interpretations lead to different ISs, and thus provide new insights into STFT: the FT interpretation of STFT corresponds to an IS for each window grain, while the FB interpretation of STFT is a valid IS if the STFT is synchrosqueezed. Thus, we provided a new theoretical motivation for synchrosqueezing, which is a fundamental necessity in order to cast the FB interpretation of STFT as a valid IS. We also highlighted the differences in the components for these interpretations, which have significant conceptual and practical differences. Specifically, in the FT interpretation, the components have a restrictive fixed amplitude and fixed frequency, while in the FB interpretation the components are AM–FM in nature. We leveraged the 3D IS visualization to provide a novel visualization of an STFT in which multiple aspects, i.e., magnitude and phase of each signal component, can be viewed simultaneously. Moreover, the phase is visualized in a way that is easily interpreted—this is in stark contrast with typical STFT analysis where phase is rarely visualized because it is not easily interpreted. Finally, in order to demonstrate these relations and results, we provided examples and illustrations.