A Fast and Robust Spectrogram Reassignment Method

Abstract

The improvement of the readability of time-frequency transforms is an important topic in the field of fast-oscillating signal processing. The reassignment method is often used due to its adaptivity to different transforms and nice formal properties. However, it strongly depends on the selection of the analysis window and it requires the computation of the same transform using three different but well-defined windows. The aim of this work is to provide a simple method for spectrogram reassignment, named FIRST (Fast Iterative and Robust Reassignment Thinning), with comparable or better precision than classical reassignment method, a reduced computational effort, and a near independence of the adopted analysis window. To this aim, the time-frequency evolution of a multicomponent signal is formally provided and, based on this law, only a subset of time-frequency points is used to improve spectrogram readability. Those points are the ones less influenced by interfering components. Preliminary results show that the proposed method can efficiently reassign spectrograms more accurately than the classical method in the case of interfering signal components, with a significant gain in terms of required computational effort.

1. Introduction

A sparse representation of a function strongly depends on its properties. This task is made more difficult in the case of non-stationary time-frequency functions [1,2,3,4], as a single linear transform is not able to provide a really sparse representation of the function. That is why either nonlinear transforms or nonlinear operations in the transformed domain are necessary.

In this paper, time-varying oscillatory functions have been considered. More precisely, we focus on amplitude and frequency-modulated multicomponent signals which are defined as [1,4]

$$f\left(t\right)=\sum _{k=1}^M{f}_k\left(t\right)=\sum _{k=1}^M{a}_k\left(t\right)e^{i{\varphi }_k\left(t\right)},$$

(1)

where $f_k$ is the k-th mode, $a_k$ and ${\varphi }_k$ respectively are its amplitude and phase functions of the time variable t while M is the number of modes. Equation (1) represents chirp-like signals, as for example radar, audio, speech and seismic signals, gravitational waves, and so on. In several applications it is required to separate each single mode of such kind of signals to retrieve its phase. Specifically, the main goal is to estimate the instantaneous frequency (IF) of each mode, i.e., the positive value of the derivative with respect to the time variable of the phase ${\varphi }_k\left(t\right)$ [5,6,7]. As a result, it is desirable to have a transform which can well separate the modes of the signal to allow a straightforward estimation of the corresponding instantaneous frequencies.

The literature concerning linear and quadratic time-frequency analysis [2,4,8,9,10,11,12,13] has proved that localized time-frequency analysis is able to provide interesting and quite efficient solutions to the problem. Some examples are the Short-Term Fourier Transform (STFT), Gabor Transform, Wavelet Transform, etc. All these transforms are based on the selection of a proper analysis window with time-frequency compaction properties whose limit is subjected to the Heisenberg principle. Unfortunately, although window-dependent, time-frequency transforms are not able to sparsify and separate each single mode with an acceptable precision due to interference patterns in the time-frequency plane. Additional operators have to be applied to the adopted time-frequency transform in order to further compact information; some interesting examples are the reassignment method [14,15,16,17,18] and the more recent synchrosqueezing transform [19,20,21,22,23,24,25] that can be applied to a wide class of transforms, hold for discrete signals and can be extended to filter banks [26,27]. Their main idea is to move each point toward the center of mass of the energy distribution. The latter is evaluated in an interval whose amplitude is consistent with the support of the analysis window—see Figure 1. Unfortunately, modes separation is possible whenever the separability condition is satisfied, i.e., whenever the distance between the instantaneous frequencies of each pair of modes is at least equal to the bandwidth of the analysis window. If the separability condition is not satisfied, the reassignment method can fail since centroids cannot be correctly estimated (see Figure 2); for similar reasons, the synchrosqueezing does not separate the modes. Even though the use of a different window can sometimes help in better separating signal components in the time-frequency plane, it is not guaranteed the existence of an optimal window for separating all signal components; in addition, for those signals whose instantaneous frequency curves intersect, it does not exist a window able to separate them at each point. An accurate separation of signal modes at each point of the time-frequency plane allows for signal identification even from a limited region of the plane, resulting useful in the case of signals with close IFs in the observation time interval or finite-length signals.

This paper aims at giving a step forward in this sense by providing a different pointwise reassignment strategy which is more robust to the lack of separability of signal modes, without using a priori estimation of corresponding IFs or the identification of non-separable regions. To this aim, FM signals with constant amplitude have been considered and FIRST (fast iterative and robust reassignment thinning) has been presented. FIRST mainly relies on the description of the modulus of signal STFT through a proper time-frequency evolution law. The latter allows us to characterize a limited set of points which represent good information to reassign even in the interference region. A local approximation of square root of the spectrogram is then employed to derive a fast algorithm for estimating the amount of reassignment for each selected point. As a result, for each point in the time-frequency plane, reassignment consists of an iterative procedure which gradually converges to the centroid of the energy distribution of the corresponding mode. Preliminary experimental results, achieved on multicomponent signals with constant amplitude, show that FIRST is able to: (i) provide comparable results with the reassignment method in regions where the separability condition is satisfied (non-interfering regions); (ii) improve reassignment result in correspondence to the non-separability region (i.e., the interference region depicted in Figure 2d); (iii) be computationally efficient.

The remainder of the paper is the following. Next section presents a brief formal description of standard reassignment method. Section 3 presents the proposed method and the details concerning the corresponding reassignment procedure. Section 4 contains some numerical results and comparative studies. Finally, the last section draws the conclusions.

2. The Reassignment Method: Theoretical Model

Definition 1.

The STFT at time $u\in \mathbf{R}$ and frequency $\xi \in {\mathbf{R}}^{+}$ of a function $f\left(t\right)\in L^1\left(\mathbf{R}\right)$ is

$$S_f^g(u,\xi )={\int }_{-\infty }^{+\infty }f\left(t\right)g^{*}(u-t)e^{-i\xi t}dt$$

(2)

where g belongs to the Schwartz class (i.e., the space of smooth functions with fast decaying derivatives of any order) and represents the analysis window—* denotes the complex conjugate.

The spectrogram is then defined as the square modulus of $S_f^g(u,\xi )$, i.e.,

$$P_{S_f^g}(u,\xi )={\left|S_f^g(u,\xi )\right|}^2.$$

It is an element of the Cohen’s class (i.e., the family of time-frequency energy distributions covariant by translations in time and frequency), since it is quadratic, time and frequency covariant, and preserves energy.

The spectrogram provides a sparse representation in the time-frequency plane of a multicomponent signal, as in Equation (1), if it consists of M distinct curves, each of them corresponding to the IF of each signal single mode. Unfortunately, this requirement cannot always be satisfied, as it happens for the signal shown in Figure 3; in this case the spectrogram spreads, providing a not enough compact signal representation in the time-frequency plane.

Spectrogram spreading is evident if one considers the following equivalent definition [4,14]

$$P_{S_f^g}(u,\xi )=\frac{1}{2\pi }\underset{-\infty }{\overset{+\infty }{\int }}\underset{-\infty }{\overset{+\infty }{\int }}W{V}_f(u^{{}^{\prime }},{\xi }^{{}^{\prime }})W{V}_g(u^{{}^{\prime }}-u,{\xi }^{{}^{\prime }}-\xi )d{u}^{{}^{\prime }}d{\xi }^{{}^{\prime }}$$

(3)

where

$$W{V}_f(u,\xi )={\int }_{-\infty }^{+\infty }f(u+\tau /2)f^{*}(u-\tau /2)e^{-i\xi \tau }d\tau$$

is the Wigner-Ville [4] distribution of the function f.

The spectrogram of f is then the smoothed version of the Wigner-Ville distribution of f while the smoothing kernel is in turn the Wigner-Ville distribution of the analysis window. The Wigner-Ville distribution is a measure of signal energy density which derives from the correlation of the signal with itself. This is the reason why it guarantees high compaction properties which strictly depend on the time-frequency properties of the function itself. In particular, the following result holds [4,14].

Proposition 1.

If $f\left(t\right)=a\left(t\right)e^{i\varphi \left(t\right)}$, then

$${\varphi }^{\prime }\left(u\right)=\frac{{\int }_{-\infty }^{+\infty }\xi W{V}_f(u,\xi )d\xi }{{\int }_{-\infty }^{+\infty }W{V}_f(u,\xi )d\xi }.$$

Hence, the center of mass along the frequency direction of the Wigner-Ville distribution of f is the instantaneous frequency of f.

Unfortunately, for a multicomponent signal the quadratic terms of the Wigner-Ville distribution cause some oscillatory interference patterns that alter the localization of each single IF in the time-frequency plane. As a result, the smoothing introduced by the spectrogram contributes to reduce the oscillatory interference between individual components introduced by the same $W{V}_f$ but, at the same time, it spreads energy density distribution.

The aim of reassignment is to compensate the spreading effects caused by the 2D smoothing introduced by the spectrogram. The main idea is to move the local energy to the centroid of the distribution, whose coordinates are

$${\widehat{u}}_f(u,\xi )=\frac{1}{P_{S_f^g}(u,\xi )}\underset{-\infty }{\overset{+\infty }{\int }}\underset{-\infty }{\overset{+\infty }{\int }}\frac{1}{2\pi }tW{V}_f(t,\omega )W{V}_g(t-u,\omega -\xi )dtd\omega ,$$

(4)

$${\widehat{\xi }}_f(u,\xi )=\frac{1}{P_{S_f^g}(u,\xi )}\underset{-\infty }{\overset{+\infty }{\int }}\underset{-\infty }{\overset{+\infty }{\int }}\frac{1}{2\pi }\omega W{V}_f(t,\omega )W{V}_g(t-u,\omega -\xi )dtd\omega .$$

(5)

The mean of the reassignment defines the reassigned spectrogram, i.e.,

$${\widehat{P}}_{S_f^g}(u,\xi )=\frac{1}{2\pi }\underset{-\infty }{\overset{+\infty }{\int }}\underset{-\infty }{\overset{+\infty }{\int }}{P}_{S_f^g}(u^{{}^{\prime }},{\xi }^{{}^{\prime }})\delta (u^{{}^{\prime }}-{\widehat{u}}_f(u,\xi ),{\xi }^{{}^{\prime }}-{\widehat{\xi }}_f(u,\xi ))d{u}^{{}^{\prime }}d{\xi }^{{}^{\prime }}.$$

(6)

Reassignment holds in a region which depends on the support of the analysis window.

Proposition 2.

Let $\phi (u,\xi )$ denote the phase of the STFT $S_f^g(u,\xi )$ of a function f, computed through the analysis windows $g\left(t\right)$, and let $S_f^{g^{\prime }}(u,\xi )$ and $S_f^{tg}(u,\xi )$ be the STFT of f computed respectively through $g^{\prime }\left(t\right)$, $tg\left(t\right)$. Let us set

$${△}_u=Re\left(\frac{S_f^{tg}(u,\xi )}{S_f^g(u,\xi )}\right)$$

(7)

$${△}_{\xi }=Im\left(\frac{S_f^{g^{\prime }}(u,\xi )}{S_f^g(u,\xi )}\right)$$

(8)

where $Re(*)$ and $Im(*)$ respectively denote the real and the imaginary part of *; then

• Equations (4) and (5) are equivalent to

  $${\widehat{u}}_f(u,\xi )=\frac{u}{2}-\frac{\partial \phi (u,\xi )}{\partial \xi };$$

  (9)

  $${\widehat{\xi }}_f(u,\xi )=\frac{\xi }{2}+\frac{\partial \phi (u,\xi )}{\partial u};$$

  (10)
• Equations (9) and (10) are equivalent to

  $${\widehat{u}}_f(u,\xi )=u+{△}_u;$$

  (11)

  $${\widehat{\xi }}_f(u,\xi )=\xi -{△}_{\xi }.$$

  (12)

The proof of Proposition 2 can be found in [14,15,28].

It is worth observing that Equations (11) and (12) provide a straightforward procedure for the estimation of the centroid of the distribution. In fact, they consist of a proper combination of the STFTs of the analyzed function using three different but specific analysis windows, i.e., $g,g^{\prime },tg$. On the other hand, the properties of the analysis windows strongly influence reassignment result, especially in the case of multicomponent signals.

One of the main constraints in a reassignment-like method is the separability condition on the individual signal components [15]: time-frequency points are correctly reassigned to their corresponding mode if the following condition is satisfied

$$|{\varphi }_k^{\prime }\left(t\right)-{\varphi }_j^{\prime }\left(t\right)|\ge \Delta \omega ,j,k\in [1,N],k\ne j$$

(13)

where $\Delta \omega$ is the frequency bandwidth of the analysis window. If this property is not met, it is not possible to achieve the desired multicomponents decomposition, independently of the ”effectiveness” of the adopted transform. For example, the synchrosqueezing transform is a frequency analysis method that can decompose complex signals into time-varying oscillatory components. It is a form of time-frequency reassignment that is both sparse and invertible, allowing for the recovery of the signal—see [15] for details. Unfortunately, the properties of such transform are guaranteed only if the separability condition is satisfied.

The aim of this paper is to define a reassignment method that can characterize signal components with high precision even when the adopted time-frequency transform is not able to resolve signal components. The main idea mainly consists of weakening the separability condition. The goal is to describe the relation between points in the time-frequency plane and to use this relation for selecting those points where the separability condition is nearly verified; as a result, reassignment is only applied to those subsets of points. In other words, the separability condition is translated into the concept of interference between modes in the time-frequency plane; hence, only those points less influenced by interference terms are used for spectrogram reassignment.

3. Time-Frequency Evolution Law

Let $f\left(t\right)=acos\left(\varphi \right(t\left)\right)$ be a frequency-modulated signal with constant amplitude and let $S_f^g(u,\xi )$ be the Short-Term Fourier Transform of f (STFT) computed through a real and symmetric window g, with support $[-\frac{1}{2},\frac{1}{2}]$ and such that $\widehat{g}\left(0\right)=1$. Since a is constant, then [4]

$$S_f^g(u,\xi )=\frac{\sqrt{s}}{2}a{e}^{i\left(\varphi \right(u)-\xi u)}(\widehat{g}\left(s[\xi -{\varphi }^{{}^{\prime }}\left(u\right)]\right)+\epsilon (u,\xi )),$$

(14)

where s is a fixed scaling factor while $\epsilon (u,\xi )$ is almost negligible.

For the sake of simplicity, in the sequel superscripts will be omitted and, without loss of generality, the scaling parameter s will be set equal to 1; hence,

$$Sf(u,\xi )=S_f^g(u,\xi )\mathrm{and}P(u,\xi )=P_{S_f^g}(u,\xi ).$$

Using the methodologies in [6,29,30], the following Proposition can be proved.

Proposition 3.

Let $f\left(t\right)=acos\left(\varphi \right(t\left)\right)$ be a monocomponent signal and let $P(u,\xi )={\left|Sf(u,\xi )\right|}^2$, then

$$P_u(u,\xi )=-{\varphi }^{″}\left(u\right)P_{\xi }(u,\xi ),$$

(15)

where $P_u$ and $P_{\xi }$ are the partial derivatives of P with respect to u and ξ respectively.

Proof of Proposition 3.

From Equation (14), it follows $P(u,\xi )=\frac{a^2}{4}{\widehat{g}}^2(\xi -{\varphi }^{\prime }\left(u\right))$ and then

$$P_{\xi }(u,\xi )=\frac{a^2}{2}\widehat{g}(\xi -{\varphi }^{\prime }\left(u\right)){\widehat{g}}^{\prime }(\xi -{\varphi }^{\prime }\left(u\right))$$

(16)

$$P_u(u,\xi )=-{\varphi }^{″}\left(u\right)\frac{a^2}{2}\widehat{g}(\xi -{\varphi }^{\prime }\left(u\right)){\widehat{g}}^{\prime }(\xi -{\varphi }^{\prime }\left(u\right)).$$

(17)

By comparing the partial derivatives of P with respect to $\xi$ and u in Equations (16) and (17) we get the proof. ☐

Equation (15) exactly describes the relation between spectrogram points in the time-frequency plane. In particular, these points belong to specific curves in the time-frequency plane, as proved by the following proposition.

Proposition 4.

The characteristic curves ${\mathcal{C}}_{c,\varphi }$ of the partial differential equation in Equation (15) are

$$\xi \left(u\right)={\varphi }^{\prime }\left(u\right)+c$$

(18)

with $c={\xi }_0-{\varphi }^{\prime }\left(u_0\right)$, where $({\xi }_0,u_0)$ is a point in the time-frequency plane.

Proof of Proposition 4.

Let us consider the parametric curve

$$\left\{\begin{array}{c}u=u\left(\tau \right)\hfill \\ \xi =\xi \left(\tau \right),\hfill \end{array}\right.$$

(19)

with $\tau \in \mathbf{R}$, and $P\left(u\right(\tau ),\xi (\tau \left)\right)$. The derivative of $P\left(u\right(\tau ),\xi (\tau \left)\right)$ with respect to $\tau$ is

$$\frac{d}{d\tau }P(u\left(\tau \right),\xi \left(\tau \right))=\frac{\partial P}{\partial u}\dot{u}+\frac{\partial P}{\partial \xi }\dot{\xi },$$

(20)

where $\dot{u}=\frac{du}{d\tau }$ and $\dot{\xi }=\frac{d\xi }{d\tau }$. Using Equation (15), we have that

$$\frac{d}{d\tau }P(u\left(\tau \right),\xi \left(\tau \right))=0$$

(21)

if

$$\left\{\begin{array}{c}\dot{u}=\frac{1}{{\varphi }^{″}\left(u\right)}\hfill \\ \dot{\xi }=1.\hfill \end{array}\right.$$

(22)

Finally, the proof derives from the solution of the system of ordinary differential equations in Equation (22). □

It is worth observing that the characteristic curves in Equation (18) are parallel to the instantaneous frequency curve, which in turn is a particular characteristic curve ($c=0$)—see Figure 4. The following property for the spectrogram can be then derived.

Proposition 5.

The spectrogram is a constant function along each characteristic curve.

Proof of Proposition 5.

The proof straightforwardly follows from Equation (21). □

Proposition 5 implies that points with the same amplitude can be moved in the same way toward their center of mass.

Unfortunately, in case of multicomponent signals, Equation (15) does not hold $\forall (u,\xi )\in {\mathbf{R}}^2$; it turns out that we cannot derive characteristic curves by writing the evolution law in the time-frequency plane for a multicomponent signal. The main motivation is exactly the separability condition in Equation (13) that cannot be verified for all $(u,\xi )\in {\mathbf{R}}^2$. However, Equation (18) associates a group of curves ${\mathcal{C}}_{c,\varphi }$, depending on the parameter c, to the same component (mode); in particular, the curve with $c=0$ is the ridge curve [4]. As a consequence, if the ridge curve of a mode does not satisfy the separability condition with respect to another mode, it does not imply that any characteristic curve does—see Figure 5. More precisely, given two modes with IF respectively ${\varphi }_1^{\prime }\left(u\right)$ and ${\varphi }_2^{\prime }\left(u\right)$, the separability condition can be weakened as follows

Definition 2.

Two modes with instantaneous frequencies ${\varphi }_1^{\prime }\left(u\right)$ and ${\varphi }_2^{\prime }\left(u\right)$ are separated at time location u if there exists at least one curve in ${\mathcal{C}}_{c_1,{\varphi }_1}$, i.e., ${\xi }_1\left(u\right)={\varphi }_1^{\prime }\left(u\right)+c_1$, such that $|{\xi }_1\left(u\right)-{\varphi }_2^{\prime }\left(u\right)|\ge \Delta \omega$; or viceversa.

Remark 1.

It is worth observing that, for example, if $\xi <{\varphi }_1^{\prime }\left(u\right)<{\varphi }_2^{\prime }\left(u\right)$, $|{\varphi }_1^{\prime }\left(u\right)-{\varphi }_2^{\prime }\left(u\right)|=\epsilon \le \Delta \omega$, and $\xi \in {\xi }_1\left(u\right)\in {\mathcal{C}}_{c_1,{\varphi }_1}$, then $\exists c_1\in \mathbf{R},c_1<0suchthat\xi ={\varphi }_1^{\prime }\left(u\right)+c_1$. As a consequence, $|\xi -{\varphi }_2^{\prime }\left(u\right)|=|{\varphi }_1^{\prime }\left(u\right)+c_1-{\varphi }_2^{\prime }\left(u\right)|=|-\epsilon +c_1|\ge \Delta \omega \iff -c_1\ge \Delta \omega -\epsilon$.

Hence, the farther the two modes, the more the points satisfying the separability condition in Definition 2. Curves satisfying constraints in Definition 2 can be characterized by the following proposition.

Proposition 6.

If ${\xi }_1\left(u\right)\in {\mathcal{C}}_{c_1,{\varphi }_1}$ and if it satisfies the separability condition with respect to ${\varphi }_2^{\prime }$, as in Definition 2, then $\widehat{g}\left(\right|{\xi }_1-{\varphi }_2^{\prime }\left(u\right)\left|\right)<<1$.

Proof of Proposition 6.

By setting $\omega ={\xi }_1-{\varphi }_2^{\prime }\left(u\right)$, the proof directly derives from the definition of window bandwidth [4], i.e., $\widehat{g}\left(\omega \right)<<1for\left|\omega \right|\ge \Delta \omega$, when applied to the mode with ${\varphi }_2^{\prime }\left(u\right)$ as IF. ☐

Remark 2.

It is worth observing that the condition $\widehat{g}(\xi -{\varphi }_2^{\prime }\left(u\right))<<1$ is equivalent to $\widehat{g}(\xi -{\varphi }_2^{\prime }\left(u\right))\approx 0$. By observing that the spectrogram of a two-component signal is

$$P(u,\xi )=|S{f}_1(u,\xi )+S{f}_2{(u,\xi )|}^2=P_1(u,\xi )+P_2(u,\xi )+2\sqrt{P_1{P}_2}cos({\varphi }_2\left(u\right)-{\varphi }_1\left(u\right)),$$

(23)

where $P_1(u,\xi )=a_1^2{\widehat{g}}^2(\xi -{\varphi }_1^{\prime }\left(u\right))$ and $P_2(u,\xi )=a_2^2{\widehat{g}}^2(\xi -{\varphi }_2^{\prime }\left(u\right))$, then for a fixed u

$$P(u,\xi )\approx |S{f}_1{(u,\xi )|}^2,\forall \xi :|\xi -{\varphi }_2^{\prime }\left(u\right)|\ge \Delta \omega .$$

More in general, for fixed u the following inequality holds:

$$\begin{array}{c}\hfill 0\le \frac{P(u,\xi )}{ma{x}_{\xi }P(u,\xi )}=\frac{P(u,\xi )}{{\widehat{g}}^2\left(0\right)(a_1^2+a_2^2)}\le \frac{{(\sqrt{P_1(u,\xi )}+\sqrt{P_2(u,\xi )})}^2}{{\widehat{g}}^2\left(0\right)(a_1^2+a_2^2)}=\\ \hfill =\frac{{(a_1\widehat{g}\left([\xi -{\varphi }_1^{\prime }\left(u\right)]\right)+a_2\widehat{g}\left([\xi -{\varphi }_2^{\prime }\left(u\right)]\right))}^2}{{\widehat{g}}^2\left(0\right)(a_1^2+a_2^2)}.\end{array}$$

(24)

Let ${\xi }_1:\frac{\widehat{g}({\xi }_1-{\varphi }_1^{\prime }\left(u\right))}{\widehat{g}\left(0\right)}=\frac{1}{H},$ and $\frac{\widehat{g}({\xi }_1-{\varphi }_2^{\prime }\left(u\right))}{\widehat{g}\left(0\right)}=\frac{1}{K}$, with $\widehat{g}$ symmetric and monotonically decreasing window function; if $\widehat{g}\left(\right|{\xi }_1-{\varphi }_2^{\prime }\left(u\right)\left|\right)<<1$ and ${\xi }_1<{\varphi }_1^{\prime }\left(u\right)<{\varphi }_2^{\prime }\left(u\right)$, then $\widehat{g}\left(\right|{\xi }_1-{\varphi }_2^{\prime }\left(u\right)\left|\right)<\widehat{g}\left(\right|{\xi }_1-{\varphi }_1^{\prime }\left(u\right)\left|\right)$, i.e., $H<K$, and then inequality in Equation (24) becomes

$$0\le \frac{P(u,\xi )}{ma{x}_{\xi }P(u,\xi )}\le \frac{{\left(\frac{a_1}{H}+\frac{a_2}{K}\right)}^2}{(a_1^2+a_2^2)}<\frac{{(a_1+a_2)}^2}{H^2(a_1^2+a_2^2)}<\frac{2}{H^2}.$$

By accounting for the definition of window bandwidth [4], previous inequality is not trivial for $H<\sqrt{2}$, i.e., for $|{\xi }_1-{\varphi }_1^{\prime }\left(u\right)|<\frac{\Delta \omega }{2}$. Hence, time-frequency plane points $(u,\xi )$ such that the ratio

$$\frac{P(u,\xi )}{ma{x}_{\xi }P(u,\xi )}<<\rho =\frac{1}{2}$$

(25)

with high probability satisfy the separability condition, as in Definition 2, and then we expect that reassignment can be more correct if it is applied only to those points in the interference region. In our simulations we set $\rho =\frac{1}{4}$.

3.1. Fast Iterative and Robust Spectrogram Thinning (FIRST)

The profile of the window function can be used for a direct estimation of the reassignment shift. By considering the regularity of the window function, for a fixed u, we can compute the Taylor expansion around the ridge point, i.e.,

$$P(u,\xi )=P(u,{\varphi }^{\prime }\left(u\right))+\frac{P^{″}(u,{\varphi }^{\prime }\left(u\right))}{2}{(\xi -{\varphi }^{\prime }\left(u\right))}^2+o\left({(\xi -{\varphi }^{\prime }\left(u\right))}^4\right).$$

(26)

By neglecting the error term, we get

$${(\xi -{\varphi }^{\prime }\left(u\right))}^2\approx 2\frac{P(u,\xi )-P(u,{\varphi }^{\prime }\left(u\right))}{P^{″}(u,{\varphi }^{\prime }\left(u\right))}$$

(27)

and, by multiplying the second member by $\frac{P(u,{\varphi }^{\prime }\left(u\right))}{P(u,{\varphi }^{\prime }\left(u\right))}$, we have

$${(\xi -{\varphi }^{\prime }\left(u\right))}^2\approx 2\left(\frac{P(u,\xi )}{P(u,{\varphi }^{\prime }\left(u\right))}-1\right)\frac{P(u,{\varphi }^{\prime }\left(u\right))}{P^{″}(u,{\varphi }^{\prime }\left(u\right))}.$$

(28)

The term $\frac{P(u,{\varphi }^{\prime }\left(u\right))}{P^{″}(u,{\varphi }^{\prime }\left(u\right))}$ only depends on the window function and its second derivative computed at the ridge point, i.e., $\xi ={\varphi }^{\prime }\left(u\right)$, while $\frac{P(u,\xi )}{P(u,{\varphi }^{\prime }\left(u\right))}$ is the value of the normalized spectrogram at point $(u,\xi )$.

Even though previous estimation is almost acceptable whenever $\xi$ approaches ${\varphi }^{\prime }\left(u\right)$, for points far from ${\varphi }^{\prime }\left(u\right)$, i.e., $\frac{P(u,\xi )}{P(u,{\varphi }^{\prime }\left(u\right))}<<1$, it does not provide a good approximation of the distance from the ridge.

As a matter of fact, by setting ${\tau }_n={\xi }_n-{\varphi }^{\prime }\left(u\right)$ we can construct the following sequence

$${\tau }_{n+1}={\tau }_n-sign\left({\tau }_n\right)\Delta {\tau }_n,\mathrm{with}\Delta {\tau }_n=\sqrt{\alpha \left(\frac{Q(u,{\tau }_n+{\varphi }^{\prime }\left(u\right))}{Q(u,{\varphi }^{\prime }\left(u\right))}-1\right)\frac{Q(u,{\varphi }^{\prime }\left(u\right))}{Q^{″}(u,{\varphi }^{\prime }\left(u\right))}},$$

(29)

that directly derives from a generalization of Equations (26)–(28) written for the function $Q(u,\xi )=\sqrt{P(u,\xi )}$ for simplicity, where $\alpha \le 2$ is a positive real parameter.

As proved in the following proposition, the sequence ${\tau }_n$ monotonically converges to 0, i.e., ${\xi }_n\to {\varphi }^{\prime }\left(u\right)$, and then it provides a reassignment method.

Proposition 7.

Let $\widehat{g}\left(\omega \right)$ denote the modulus of the Fourier transform of a real, positive and symmetric analysis window and let $Q(u,\xi )$ denote the square root of the spectrogram of a monocomponent function; then, for fixed u the sequence

$${\tau }_{n+1}=\phi \left({\tau }_n\right),n\ge 0$$

where $\phi \left(\tau \right)=\tau -sign\left(\tau \right)\Delta \tau$, with $\Delta \tau =\sqrt{C(1-R(\tau \left)\right)}$, $C=\alpha \frac{Q(u,{\varphi }^{\prime }\left(u\right))}{|Q^{\prime \prime }(u,{\varphi }^{\prime }\left(u\right))|}$ and $R\left(\tau \right)=\frac{Q(u,\tau +{\varphi }^{\prime }\left(u\right))}{Q(u,{\varphi }^{\prime }\left(u\right))}$, monotonically converges to 0.

Proof of Proposition 7.

Reminding that $Q(u,\tau +{\varphi }^{\prime }\left(u\right))=a\widehat{g}\left(\tau \right)$, then $\tau =0$ is a fixed point of the function $\phi \left(\tau \right)$. In addition, it is unique whenever $\widehat{g}$ is positive, symmetric with respect to 0 and monotonically decreasing away from 0, as it is the case of the analysis window used for the computation of the spectrogram. Hence, the convergence of ${\tau }_n$ to 0 can be proved using the fixed-point convergence theorems.

$\phi \left(\tau \right)$ is a continuous function as well as ${\phi }^{\prime }\left(\tau \right)$. In fact,

$${\phi }^{\prime }\left(\tau \right)=1+\left\{\begin{array}{cc}\frac{d\Delta \tau }{d\tau },\hfill & \tau <0\hfill \\ -\frac{d\Delta \tau }{d\tau },\hfill & \tau >0\hfill \end{array}\right.$$

(30)

where $\frac{d\Delta \tau }{d\tau }=-\frac{\sqrt{C}}{2\sqrt{1-R\left(\tau \right)}}\frac{{\widehat{g}}^{\prime }\left(\tau \right)}{\widehat{g}\left(0\right)}$ and ${\phi }^{\prime }\left(0\right)=1-\frac{\sqrt{\alpha }}{\sqrt{2}}.$ In fact, reminding that $sign\left(\tau {\widehat{g}}^{\prime }\left(\tau \right)\right)<0$ and ${\widehat{g}}^{″}\left(0\right)<0$,

$$\underset{\tau \to 0^{-}}{lim}\frac{d\Delta \tau }{d\tau }=\underset{\tau \to 0^{-}}{lim}-\frac{\sqrt{C}}{2\widehat{g}\left(0\right)}\sqrt{\frac{{\widehat{g}}^{\prime 2}\left(\tau \right)}{1-R\left(\tau \right)}}=-\frac{\sqrt{\alpha }}{2\sqrt{|{\widehat{g}}^{″}\left(0\right)|}}\sqrt{\underset{\tau \to 0^{-}}{lim}\frac{2{\widehat{g}}^{\prime }\left(\tau \right){\widehat{g}}^{″}\left(\tau \right)}{-{\widehat{g}}^{\prime }\left(\tau \right)}}=-\frac{\sqrt{\alpha |{\widehat{g}}^{″}\left(0\right)|}}{\sqrt{2|{\widehat{g}}^{″}\left(0\right)|}}=-\frac{\sqrt{\alpha }}{\sqrt{2}},$$

and similarly, ${lim}_{\tau \to 0^{+}}\frac{d\Delta \tau }{d\tau }=\frac{\sqrt{\alpha }}{\sqrt{2}}.$

In addition, since $0\le {\phi }^{\prime }\left(0\right)<1$, $\exists I_0=[-\delta ,\delta ],with\delta >0$, such that $\forall {\tau }_0\in I_0,$ the sequence ${\tau }_{n+1}$ monotonically converges to 0. The convergence is linear for $\alpha <2$.

It is worth observing that

$${\phi }^{″}\left(\tau \right)=\left\{\begin{array}{cc}\frac{d^2\Delta \tau }{d{\tau }^2},\hfill & \tau <0\hfill \\ -\frac{d^2\Delta \tau }{d{\tau }^2},\hfill & \tau >0\hfill \end{array}\right.$$

(31)

with

$$\frac{d^2\Delta \tau }{d{\tau }^2}=-\frac{\sqrt{C}}{2\widehat{g}\left(0\right)}\frac{2{\widehat{g}}^{″}\left(\tau \right)(1-R\left(\tau \right))+{\widehat{g}}^{\prime }\left(\tau \right)R^{\prime }\left(\tau \right)}{{(1-R\left(\tau \right))}^{\frac{3}{2}}}.$$

and ${lim}_{\tau \to 0^{-}}\frac{d^2\Delta \tau }{d{\tau }^2}={lim}_{\tau \to 0^{+}}\frac{d^2\Delta \tau }{d{\tau }^2}=0.$

However,

$${\phi }^{″}\left(\tau \right)=0\iff {\left({\widehat{g}}^{\prime }\left(\tau \right)\right)}^2=2\widehat{g}\left(0\right){\widehat{g}}^{″}\left(\tau \right)(1-R\left(\tau \right)).$$

(32)

Hence, $\tau =\overline{\tau }$ satisfies previous equation if $\overline{\tau }=0$ or ${\widehat{g}}^{″}\left(\overline{\tau }\right)>0$.

It turns out that if $\overline{\tau }$ is a zero for ${\phi }^{″}\left(\tau \right)$ and $\overline{\tau }\ne 0$, then ${\phi }^{\prime }\left(\overline{\tau }\right)=1+sign\left(\overline{\tau }\right)\frac{\sqrt{C}}{2\widehat{g}\left(0\right)}\sqrt{2\widehat{g}\left(0\right)\left|{\widehat{g}}^{″}\left(\tau \right)\right|}=1+sign\left(\overline{\tau }{\widehat{g}}^{\prime }\left(\tau \right)\right)\sqrt{\frac{\alpha {\widehat{g}}^{″}\left(\tau \right)}{2|{\widehat{g}}^{″}\left(0\right)|}}.$ By observing that $0=argma{x}_{\tau }|{\widehat{g}}^{″}\left(\tau \right)|$ for the considered analysis windows and $\overline{\tau }\ne 0$, we have

$$0<\sqrt{\alpha \frac{{\widehat{g}}^{″}\left(\overline{\tau }\right)}{2|{\widehat{g}}^{″}\left(0\right)|}}<1$$

and then, since $sign\left(\tau {\widehat{g}}^{\prime }\left(\tau \right)\right)<0$,

$$|{\phi }^{\prime }\left(\overline{\tau }\right)|<1,\forall \overline{\tau }\ne 0.$$

Finally, since $\overline{\tau }$ is a relative extreme point for ${\phi }^{\prime }\left(\tau \right)$, then

$$|{\phi }^{\prime }\left(\tau \right)|\le ma{x}_{\overline{\tau }}\left|{\phi }^{\prime }\left(\overline{\tau }\right)\right|<1.$$

On the other hand, for $\overline{\tau }=0$, ${\phi }^{\prime }\left(\overline{\tau }\right)=1-\sqrt{\frac{\alpha }{2}}<1.$ Hence, the convergence of ${\tau }_n$ is guaranteed $\forall {\tau }_0\in supp\left\{\widehat{g}\right\}$, i.e., ${\tau }_0\in \left[-\frac{\Delta \omega }{2},\frac{\Delta \omega }{2}\right].$ ☐

Equation (29) defines an iterative spectrogram reassignment method and it will be denoted as FIRST in the sequel. For monocomponent signals it represents a different way of reassigning time-frequency points and it exactly reassigns on the ridge curve as the standard reassignment does. On the contrary, in case of multicomponent signals it allows us to correctly reassign points in the interference region without changing the analysis window; hence, it contributes to reduce the region of incorrectly reassigned points thanks to the use of a weak separability condition. As a result, only a limited set of points is reassigned and each of them is forced to converge by sequential decreasing shifts to the ridge point of the mode it belongs to, i.e., the one with a dominant contribution at the considered location.

Although the better robustness to the non-separability condition, there exists a region where the weak separability condition in Definition 2 is not satisfied too. As a consequence, the shift cannot be correctly reassigned as it depends on the value of the spectrogram $P(u,{\xi }_n)$ of the single mode which is unknown whenever the point ${\xi }_n$ does not satisfy the weak separability condition. However, the lower $P(u,{\xi }_n)$ the higher the shift and the higher the probability to be far from the interference region. As a result, the larger n the higher the eventual reassignment error. Hence, in case of multicomponent signals, in correspondence to the interference region, we expect a correct reassignment for the very first iterations while convergence is not assured as n increases.

To force the convergence to the ridge point of each component, we can proceed as follows. From the contraction property it holds

$$|{\tau }_{n+1}-{\tau }_n|=|{\xi }_{n+1}-{\xi }_n|=|{\phi }^{\prime }\left({\theta }_n\right)\left|\right|{\xi }_n-{\xi }_{n-1}|=|{\phi }^{\prime }\left({\theta }_n\right)\left|\right|{\tau }_n-{\tau }_{n-1}|,{\theta }_n\in [{\tau }_{n-1},{\tau }_n],$$

By denoting with $K=ma{x}_{\tau }\left|{\phi }^{\prime }\left(\tau \right)\right|=max\left\{1-\sqrt{\frac{\alpha {\widehat{g}}^{″}\left(\overline{\tau }\right)}{2|{\widehat{g}}^{″}\left(0\right)|}},1-\sqrt{\frac{\alpha }{2}}\right\}$ the contraction constant of $\phi$ we have

$$|{\xi }_{n+1}-{\xi }_n|\le K|{\xi }_n-{\xi }_{n-1}|,\forall n\ge 1.$$

As a result, in case of multicomponent signals, $\forall n\ge 1$ we can modify Equation (29) as follows

$$\left\{\begin{array}{cc}{\tau }_1={\tau }_0-sign\left({\tau }_0\right)\Delta {\tau }_0\hfill & n=0\hfill \\ {\tau }_{n+1}={\tau }_n+K({\tau }_n-{\tau }_{n-1})\hfill & n>0\hfill \end{array}\right.$$

(33)

with ${\tau }_0$ the initial point. The use of K instead of the spectrogram value for determining the elements of the sequence, except for the first one, makes the shift toward the ridge more accurate—as it will be shown in the next section (each point is forced to move as if it is a point satisfying the separability condition with respect to another mode). The reassignment as in Equation (33) will be denoted as FIRST_K.

Remark 3.

K characterizes the analysis window ($\overline{\tau }$ satisfies Equation (32)) but its computation could be not straightforward—nonlinear iterative methods should be required. Hence, by considering that the closer ${\tau }_n$ to 0 the more $|{\phi }^{\prime }\left({\theta }_n\right)|\approx |{\phi }^{\prime }\left(0\right)|=1-\sqrt{\frac{\alpha }{2}}$, a more simple reassignment can be performed by simply setting $K=1-\sqrt{\frac{\alpha }{2}}$ and using the first few iterations of the resulting sequence, as it will be shown in the experimental results.

3.2. Computational Complexity

This section aims at estimating the number of operations per pixel required by the proposed reassignment method. Only multiplications, divisions, and comparisons will be considered. The estimated complexity is then compared with the one required by the standard reassignment. To this aim we will focus on the number of operations required by the computation of the centroid for each point of the time-frequency plane as it represents the actual difference with respect to the standard method. Equation (33) (FIRST_K) will be considered.

L denotes the signal length, $L\times F$ is the time-frequency dimension of the STFT, N is the number of adopted iterations and

$$\Omega =\bigcup _{0.01\le \rho \le 0.25}{\Omega }_{\rho },\mathrm{with}{\Omega }_{\rho }=\left\{(u,\xi ):\xi =argmi{n}_{\xi }\left(\frac{P(u,\xi )}{maxP(u,\xi )}=\rho \right)\right\}.$$

(34)

The estimation of the centroid using FIRST_K requires:

• the computation of one STFT, i.e., $(2Flo{g}_2\left(F\right)+F)$ for each signal sample;
• $Flo{g}_2\left(F\right)$ comparisons of each u for the construction of the set $\Omega$;
• 1 multiplication for each iteration and each point in the time-frequency domain belonging to the set $\Omega$ and 1 square root for the first iteration.

Hence, the proposed reassignment method requires $C_{proposed}=|\Omega |N+|\Omega |\beta +(2Flo{g}_2\left(F\right)+F)L+LFlo{g}_2\left(F\right)$ operations, where $\beta$ is the number of operations required for the computation of the square root.

In the standard reassignment, the estimation of the centroid requires:

• 2 multiplications and 1 division for each equation in (11) and (12) and each point in the time-frequency domain;
• the computation of three STFTs, i.e., $3(2Flo{g}_2\left(F\right)+F)$ for each signal sample.

Hence, the standard reassignment requires $C_{stdreass}=3LF+3(2Flo{g}_2\left(F\right)+F)L$ operations.

As a result, FIRST_K is computationally more convenient than the standard reassignment if

$$|\Omega |N+|\Omega |\beta +(2Flo{g}_2\left(F\right)+F)L+LFlo{g}_2\left(F\right)<3LF+3(2Flo{g}_2\left(F\right)+F)L.$$

By dividing both members by $LF$, we get the relation in terms of operations per pixel (spectrogram image), i.e.,

$$\frac{(N+\beta )|\Omega |}{LF}+(3lo{g}_2\left(F\right)+1)<3+3(2lo{g}_2\left(F\right)+1),$$

and then

$$\frac{|\Omega |}{F}<\frac{5+3lo{g}_2\left(F\right)}{N+\beta }L.$$

It is worth observing that $|\Omega |\le |\Delta \omega |\le F$, where $|\Delta \omega |$ is the support of the analysis window in the frequency domain. It turns out that FIRST_K is computationally convenient if

$$\frac{5+3lo{g}_2\left(F\right)}{N+\beta }L>1\iff (N+\beta )<(5+3lo{g}_2\left(F\right))L.$$

By considering that $F\le L$, $\beta =16$ and $N\le 4$, previous equation always holds for non-trivial signal lengths. In addition, $(N+\beta )<<(5+3lo{g}_2\left(F\right))L,$ for $L\ge 512$, which represents a standard signal length.

4. Results

The proposed iterative reassignment method has been tested on several synthetic signals with more than one component and different instantaneous frequencies functions. A Gaussian window function has been used in all tests. The results have been directly compared with the standard reassignment method to appreciate and evaluate the step forward provided by the proposed method in terms of improved spectrogram readability and computational gain. In all tests, time-frequency points belonging to the set $\Omega$ defined in Equation (34) have been considered—the left bound has been used to prevent eventual distortions due to numerical approximations in the computation of the spectrogram; the right bound has been fixed according to Remark 2 in Section 3.

The first test is simply oriented to evaluate the reassignment result in the case of a two components signal, the one shown in Figure 2, where only points satisfying condition in Equation (25) are considered. Results are shown in Figure 6. As it can be observed, the two results are the same whenever the two modes satisfy the separability condition; on the contrary, in the interference region, the selection of a reduced number of points, i.e., the ones satisfying the weak separability condition in Definition 2, allows slight reduction of the region where reassignment fails in the separation of the two modes.

The second test aims at evaluating the reassignment result obtained using FIRST. Figure 7 and Figure 8 depict the results for the two signals in Figure 1 and Figure 2. Reassignment results using an increasing number of iterations have been shown using $\alpha =2$. As it can be observed, the method approaches the solution after very few iterations. The same figures show the reassignment result which has been achieved using two iterations of FIRST, where the value of Q is used at each iteration. As it can be observed, FIRST has a fast convergence to the ridge curve in correspondence to separable regions; in addition, the method is able to reassign correctly even in the interference region, except for a small region close to the intersection point, i.e., the one where the weak separability condition is not satisfied. The same considerations are valid whenever FIRST_K is used. In this case, one iteration allows us to reach the final solution and to provide better results in the interference region and for points very close to modes intersection. Even though reassignment is not correct for these points, estimated reassignment locations are closer to the correct ones with respect to the ones provided by FIRST. The main reason is the fact that FIRST_K is not influenced by incorrect spectrogram values since it does not employ those values except for the initial point. It is also worth observing that FIRST_K is less accurate than FIRST in correspondence to interference region since the sequence in (33) converges but its limit is not the ridge point of the signal mode. Results do not change if FIRST_K is used by setting $K=1-\frac{1}{\sqrt{2}}$, independently of the adopted analysis window—as discussed in Remark 3. However, in this case the sequence converges to a point which can be closer to the ridge point.

Figure 9 shows reassignment results provided by the proposed iterative procedure for the signals in Figure 1 and Figure 2 where the number of iterations N has been adaptively estimated for each ratio $\frac{Q(u,\xi )}{Q(u,{\varphi }^{\prime }\left(u\right))}$ and requiring a tolerance $\epsilon$ corresponding to the discretization step of the frequency axis adopted in the numerical computation of the STFT, i.e., $N=⌈\frac{log\left(\frac{(1-K)\epsilon }{\Delta {\tau }_1}\right)}{log\left(K\right)}⌉.$ As it can be observed, the use of a different number of iterations for points with a different spectrogram value, allows us to correctly reassign points outside and inside the interference region. The same Figure shows the reassignment result which has been obtained using one iteration of the sequence Equation (33); results are almost equivalent, making FIRST_K a robust and computable version of FIRST in the case of lack of both weak and strong separability condition.

Figure 10 depicts the convergence rate for two different points in the spectrogram ($\rho =0.10$ and $\rho =0.50$—see Equation (34)). As it can be observed, for the signal with two separated modes, the proposed method is able to correctly reassign each point since the value of the spectrogram at ${\tau }_0$ is exact. For the signal with two non-separated modes, the convergence is slightly altered (especially for points closer to the ridge, i.e., the ones for which the separability condition does not hold). It is due to the error on the spectrogram value caused by the interference of the two modes. In this case, FIRST_K provides a more robust solution since it only depends on the analysis window while it does not depend on the value of the spectrogram except for the initial point.

Finally, it is worth observing that the proposed method results more computationally efficient than the standard reassignment as it just requires the computation of one spectrogram; in addition, for points with the same spectrogram value, the sequence ${\tau }_n$ in Equation (33) is the same—this allows us to drastically reduce the number of operations and the computing time of reassignment.

5. Conclusions

In this paper, an iterative spectrogram reassignment method for multicomponent signals has been presented. The method relies on a generalization of the separability condition in the case of multicomponent signals and then it mainly consists of successive shifts of selected points in the spectrogram. These points are those satisfying the separability condition with respect to signal modes, i.e., the ones less influenced by the interaction of signal modes in the time-frequency plane. As a result these points are moved as if the spectrogram is the one of a monocomponent signal. The sequential shifts are the ones estimated from a local Taylor approximation of the analysis window. Convergence of the method has been proved and discussions concerning its conditioning have been included. The latter allows us to define a robust and faster reassignment method which only requires the computation of one spectrogram. Extensive numerical studies show that the proposed method is more robust than standard reassignment method in the case of multicomponent signals with non-separable modes with constant amplitudes, especially in regions of the time-frequency plane where the modes strongly interfere. The convergence is achieved after two or three iterations making the method computationally advantageous. Future research will be devoted to an in-depth numerical study for further improving reassignment in the interference region. Particular attention will be devoted to the robustness to incorrect initial values and/or noisy data as well as to the extension to non-constant amplitude modes.