Sequential Learning of Principal Curves: Summarizing Data Streams on the Fly

Abstract

When confronted with massive data streams, summarizing data with dimension reduction methods such as PCA raises theoretical and algorithmic pitfalls. A principal curve acts as a nonlinear generalization of PCA, and the present paper proposes a novel algorithm to automatically and sequentially learn principal curves from data streams. We show that our procedure is supported by regret bounds with optimal sublinear remainder terms. A greedy local search implementation (called slpc, for sequential learning principal curves) that incorporates both sleeping experts and multi-armed bandit ingredients is presented, along with its regret computation and performance on synthetic and real-life data.

1. Introduction

Numerous methods have been proposed in the statistics and machine learning literature to sum up information and represent data by condensed and simpler-to-understand quantities. Among those methods, principal component analysis (PCA) aims at identifying the maximal variance axes of data. This serves as a way to represent data in a more compact fashion and hopefully reveal as well as possible their variability. PCA was introduced by [1,2] and further developed by [3]. This is one of the most widely used procedures in multivariate exploratory analysis targeting dimension reduction or feature extraction. Nonetheless, PCA is a linear procedure and the need for more sophisticated nonlinear techniques has led to the notion of principal curve. Principal curves may be seen as a nonlinear generalization of the first principal component. The goal is to obtain a curve which passes “in the middle” of data, as illustrated by Figure 1. This notion of skeletonization of data clouds has been at the heart of numerous applications in many different domains, such as physics [4,5], character and speech recognition [6,7], mapping and geology [5,8,9], to name but a few.

1.1. Earlier Works on Principal Curves

The original definition of principal curve dates back to [10]. A principal curve is a smooth ($C^{\infty }$) parameterized curve $\mathbf{f}\left(s\right)=\left(f_1\left(s\right),\dots ,f_d\left(s\right)\right)$ in ${\mathbb{R}}^d$ which does not intersect itself, has finite length inside any bounded subset of ${\mathbb{R}}^d$ and is self-consistent. This last requirement means that $\mathbf{f}\left(s\right)=\mathbb{E}\left[X\right|s_{\mathbf{f}}\left(X\right)=s]$, where $X\in {\mathbb{R}}^d$ is a random vector and the so-called projection index $s_{\mathbf{f}}\left(x\right)$ is the largest real number s minimizing the squared Euclidean distance between $\mathbf{f}\left(s\right)$ and x, defined by

$$s_{\mathbf{f}}\left(x\right)=sup\left\{s:{∥x-\mathbf{f}\left(s\right)∥}_2^2=\underset{\tau }{inf}{∥x-\mathbf{f}\left(\tau \right)∥}_2^2\right\}.$$

Self-consistency means that each point of $\mathbf{f}$ is the average (under the distribution of X) of all data points projected on $\mathbf{f}$, as illustrated by Figure 2.

However, an unfortunate consequence of this definition is that the existence is not guaranteed in general for a particular distribution, let alone for an online sequence for which no probabilistic assumption is made. In order to handle complex data structures, Ref. [11] proposed principal curves (PCOP) of principal oriented points (POPs) which are defined as the fixed points of an expectation function of points projected to a hyperplane minimising the total variance. To obtain POPs, a cluster analysis is performed on the hyperplane and only data in the local cluster are considered. Ref. [12] introduced the local principal curve (LPC), whose concept is similar to that of [11], but accelerates the computation of POPs by calculating local centers of mass instead of performing cluster analysis, and local principal component instead of principal direction. Later, Ref. [13] also considered LPC in data compression and regression to reduce the dimension of predictors space to low-dimension manifold. Ref. [14] extended the idea of localization to independent component analysis (ICA) by proposing a local-to-global non-linear ICA framework for visual and auditory signal. Ref. [15] considered principal curves from a different perspective: as the ridge of a smooth probability density function (PDF) generating dataset, where the ridges are collections of all points; the local gradient of a PDF is an eigenvector of the local Hessian, and the eigenvalues corresponding to the remaining eigenvectors are negative. To estimate principal curves based on this definition, the subspace constrained mean shift (SCMS) algorithm was proposed. All the local methods above require strong assumptions on the PDF, such as twice continuous differentiability, which may prove challenging to be satisfied in the settings of online sequential data. Ref. [16] proposed a new concept of principal curves which ensures its existence for a large class of distributions. Principal curves ${\mathbf{f}}^{\star }$ are defined as the curves minimizing the expected squared distance over a class ${\mathcal{F}}_L$ of curves whose length is smaller than $L>0$; namely,

$${\mathbf{f}}^{\star }\in \underset{\mathbf{f}\in {\mathcal{F}}_L}{arginf}\Delta \left(\mathbf{f}\right),$$

where

$$\Delta \left(\mathbf{f}\right)=\mathbb{E}\left[\Delta \left(\mathbf{f},X\right)\right]=\mathbb{E}\left[\underset{s}{inf}{∥\mathbf{f}\left(s\right)-X∥}_2^2\right].$$

If ${\mathbb{E}\parallel X\parallel }_2^2<\infty$, ${\mathbf{f}}^{\star }$ always exists but may not be unique. In practical situations where only i.i.d. copies $X_1,\dots ,X_n$ of X are observed, the method of [16] considers classes ${\mathcal{F}}_{k,L}$ of all polygonal lines with k segments and length not exceeding L, and chooses an estimator ${\widehat{\mathbf{f}}}_{k,n}$ of ${\mathbf{f}}^{\star }$ as the one within ${\mathcal{F}}_{k,L}$, which minimizes the empirical counterpart

$${\Delta }_n\left(\mathbf{f}\right)=\frac{1}{n}\sum _{i=1}^n\Delta \left(\mathbf{f},X_i\right)$$

of $\Delta \left(\mathbf{f}\right)$. It is proved in [17] that if X is almost surely bounded and $k\propto n^{1/3}$, then

$$\Delta \left({\widehat{\mathbf{f}}}_{k,n}\right)-\Delta \left({\mathbf{f}}^{\star }\right)=\mathcal{O}\left(n^{-1/3}\right).$$

As the task of finding a polygonal line with k segments and length of at most L that minimizes ${\Delta }_n\left(\mathbf{f}\right)$ is computationally costly, Ref. [17] proposed a polygonal line algorithm. This iterative algorithm proceeds by fitting a polygonal line with k segments and considerably speeds up the exploration part by resorting to gradient descent. The two steps (projection and optimization) are similar to what is done by the k-means algorithm. However, the polygonal line algorithm is not supported by theoretical bounds and leads to variable performance depending on the distribution of the observations.

As the number of segments, k, plays a crucial role (a too small a k value leads to a poor summary of data, whereas a too-large k yields overfitting; see Figure 3), Ref. [18] aimed to fill the gap by selecting an optimal k from both theoretical and practical perspectives.

Their approach relies strongly on the theory of model selection by penalization introduced by [19] and further developed by [20]. By considering countable classes ${\left\{{\mathcal{F}}_{k,\ell }\right\}}_{k,\ell }$ of polygonal lines with k segments and total length $\ell \le L$, and whose vertices are on a lattice, the optimal $(\widehat{k},\widehat{\ell })$ is obtained as the minimizer of the criterion

$$\mathrm{crit}(k,\ell )={\Delta }_n\left({\widehat{\mathbf{f}}}_{k,\ell }\right)+\mathrm{pen}(k,\ell ),$$

where

$$\mathrm{pen}(k,\ell )=c_0\sqrt{\frac{k}{n}}+c_1\frac{\ell }{n}+c_2\frac{1}{\sqrt{n}}+{\delta }^2\sqrt{\frac{w_{k,\ell }}{2n}}$$

is a penalty function where $\delta$ stands for the diameter of observations and $w_{k,\ell }$ denotes the weight attached to class ${\mathcal{F}}_{k,\ell }$; and it has constants $c_0,c_1,c_2$ depending on $\delta$, maximum length L and a certain number of dimensions of observations. Ref.  [18] then proved that

$$\mathbb{E}\left[\Delta \left({\widehat{\mathbf{f}}}_{\widehat{k},\widehat{\ell }}\right)-\Delta \left({\mathbf{f}}^{\star }\right)\right]\le \underset{k,\ell }{inf}\left\{\mathbb{E}\left[\Delta \left({\widehat{\mathbf{f}}}_{k,\ell }\right)-\Delta \left({\mathbf{f}}^{\star }\right)\right]+\mathrm{pen}(k,\ell )\right\}+\frac{{\delta }^2\Sigma }{2^{3/2}}\sqrt{\frac{\pi }{n}},$$

(1)

where $\Sigma$ is a numerical constant. The expected loss of the final polygonal line ${\widehat{\mathbf{f}}}_{\widehat{k},\widehat{\ell }}$ is close to the minimal loss achievable over ${\mathcal{F}}_{k,\ell }$ up to a remainder term decaying as $1/\sqrt{n}$.

1.2. Motivation

The big data paradigm—where collecting, storing and analyzing massive amounts of large and complex data becomes the new standard—commands one to revisit some of the classical statistical and machine learning techniques. The tremendous improvements of data acquisition infrastructures generates new continuous streams of data, rather than batch datasets. This has drawn great interest to sequential learning. Extending the notion of principal curves to the sequential settings opens up immediate practical application possibilities. As an example, path planning for passengers’ locations can help taxi companies to better optimize their fleet. Online algorithms that could yield instantaneous path summarization would be adapted to the sequential nature of geolocalized data. Existing theoretical works and practical implementations of principal curves are designed for the batch setting [7,16,17,18,21] and their adaptation to the sequential setting is not a smooth process. As an example, consider the algorithm in [18]. It is assumed that vertices of principal curves are located on a lattice, and its computational complexity is of order $\mathcal{O}\left(n{N}^p\right)$ where n is the number of observations, N the number of points on the lattice and p the maximum number of vertices. When p is large, running this algorithm at each epoch yields a monumental computational cost. In general, if data are not identically distributed or even adversary, algorithms that originally worked well in the batch setting may not be ideal when cast onto the online setting (see [22], Chapter 4). To the best of our knowledge, little effort has been put so far into extending principal curves algorithms to the sequential context.

Ref. [23] provided an incremental version of the SCMS algorithm [15] which is based on a definition of a principal curve as the ridge of a smooth probability density function generating observations. They applied the SCMS algorithm to the input points that are associated with the output points which are close to the new incoming sample and leave the remaining outputs unchanged. Hence, this algorithm can be used to deal with sequential data. As presented in the next section, our algorithm for sequentially updating principal curve vertices that are close to new data is similar in spirit to that of incremental SCMS. However, a difference is that our algorithm outputs polygonal lines. In addition, the computation complexity of our method is $\mathcal{O}\left(n^2\right)$, and incremental SCMS has $\mathcal{O}\left(n^3\right)$ complexity, where n is the number of observations. Ref. [24] considered sequential principal curves analysis in a fairly different setting in which the goal was to derive in an adaptive fashion a set of nonlinear sensors by using a set of preliminary principal curves. Unfolding sequentially principal curves and a sequential path for Jacobian integration were considered. The “sequential” in this setting represented the generalization of principal curves to principal surfaces or even a principal manifold of higher dimensions. This way of sequentially exploiting principal curves was firstly proposed by [11] and later extended by [14,25,26] to give curvilinear representations using sequence of local-to-global curves. In addition, Refs. [15,27,28] presented, respectively, principal polynomial and non-parametric regressions to capture the nonlinear nature of data. However, these methods are not originally designed for treating sequential data. The present paper aims at filling this gap: our goal was to propose an online perspective to principal curves by automatically and sequentially learning the best principal curve summarizing a data stream. Sequential learning takes advantage of the latest collected (set of) observations and therefore suffers a much smaller computational cost.

Sequential learning operates as follows: a blackbox reveals at each time t some deterministic value $x_t,t=1,2,\dots$, and a forecaster attempts to predict sequentially the next value based on past observations (and possibly other available information). The performance of the forecaster is no longer evaluated by its generalization error (as in the batch setting) but rather by a regret bound which quantifies the cumulative loss of a forecaster in the first T rounds with respect to some reference minimal loss. In sequential learning, the velocity of algorithms may be favored over statistical precision. An immediate use of aforecited techniques [17,18,21] at each time round t (treating data collected until t as a batch dataset) would result in a monumental algorithmic cost. Rather, we propose a novel algorithm which adapts to the sequential nature of data, i.e., which takes advantage of previous computations.

The contributions of the present paper are twofold. We first propose a sequential principal curve algorithm, for which we derived regret bounds. We then present an implementation, illustrated on a toy dataset and a real-life dataset (seismic data). The sketch of our algorithm’s procedure is as follows. At each time round t, the number of segments of $k_t$ is chosen automatically and the number of segments $k_{t+1}$ in the next round is obtained by only using information about $k_t$ and a small number of past observations. The core of our procedure relies on computing a quantity which is linked to the mode of the so-called Gibbs quasi-posterior and is inspired by quasi-Bayesian learning. The use of quasi-Bayesian estimators is especially advocated by the PAC-Bayesian theory, which originated in the machine learning community in the late 1990s, in the seminal works of [29] and McAllester [30,31]. The PAC-Bayesian theory has been successfully adapted to sequential learning problems; see, for example, Ref. [32] for online clustering. We refer to [33,34] for a recent overview of the field.

The paper is organized as follows. Section 2 presents our notation and our online principal curve algorithm, for which we provide regret bounds with sublinear remainder terms in Section 3. A practical implementation was proposed in Section 4, and we illustrate its performance on synthetic and real-life datasets in Section 5. Proofs of all original results claimed in the paper are collected in Section 6.

2. Notation

A parameterized curve in ${\mathbb{R}}^d$ is a continuous function $\mathbf{f}:I⟶{\mathbb{R}}^d$ where $I=[a,b]$ is a closed interval of the real line. The length of $\mathbf{f}$ is given by

$$\mathcal{L}\left(\mathbf{f}\right)=\underset{M\to \infty }{lim}\left\{\underset{a=s_0<s_1<\cdots <s_M=b}{sup}\sum _{i=1}^M{\parallel \mathbf{f}\left(s_i\right)-\mathbf{f}\left(s_{i-1}\right)\parallel }_2\right\}.$$

Let $x_1,x_2,\dots ,x_T\in B(0,\sqrt{d}R)\subset {\mathbb{R}}^d$ be a sequence of data, where $B(\mathbf{c},R)$ stands for the ${\ell }_2$-ball centered in $\mathbf{c}\in {\mathbb{R}}^d$ with radius $R>0$. Let ${\mathcal{Q}}_{\delta }$ be a grid over $B(0,\sqrt{d}R)$, i.e., ${\mathcal{Q}}_{\delta }=B(0,\sqrt{d}R)\cap {\Gamma }_{\delta }$ where ${\Gamma }_{\delta }$ is a lattice in ${\mathbb{R}}^d$ with spacing $\delta >0$. Let $L>0$ and define for each $k\in ⟦1,p⟧$ the collection ${\mathcal{F}}_{k,L}$ of polygonal lines $\mathbf{f}$ with k segments whose vertices are in ${\mathcal{Q}}_{\delta }$ and such that $\mathcal{L}\left(\mathbf{f}\right)\le L$. Denote by ${\mathcal{F}}_p={\cup }_{k=1}^p{\mathcal{F}}_{k,L}$ all polygonal lines with a number of segments $\le p$, whose vertices are in ${\mathcal{Q}}_{\delta }$ and whose length is at most L. Finally, let $\mathcal{K}\left(\mathbf{f}\right)$ denote the number of segments of $\mathbf{f}\in {\mathcal{F}}_p$. This strategy is illustrated by Figure 4.

Our goal is to learn a time-dependent polygonal line which passes through the “middle” of data and gives a summary of all available observations $x_1,\dots ,x_{t-1}$ (denoted by ${\left(x_s\right)}_{1:(t-1)}$ hereafter) before time t. Our output at time t is a polygonal line ${\widehat{\mathbf{f}}}_t\in {\mathcal{F}}_p$ depending on past information ${\left(x_s\right)}_{1:(t-1)}$ and past predictions ${\left({\widehat{\mathbf{f}}}_s\right)}_{1:(t-1)}$. When $x_t$ is revealed, the instantaneous loss at time t is computed as

$$\Delta \left({\widehat{\mathbf{f}}}_t,x_t\right)=\underset{s\in I}{inf}{\parallel {\widehat{\mathbf{f}}}_t\left(s\right)-x_t\parallel }_2^2.$$

(2)

In what follows, we investigate regret bounds for the cumulative loss based on (2). Given a measurable space $\Theta$ (embedded with its Borel $\sigma$-algebra), we let $\mathcal{P}(\Theta )$ denote the set of probability distributions on $\Theta$, and for some reference measure $\pi$, we let ${\mathcal{P}}_{\pi }(\Theta )$ be the set of probability distributions absolutely continuous with respect to $\pi$.

For any $k\in ⟦1,p⟧$, let ${\pi }_k$ denote a probability distribution on ${\mathcal{F}}_{k,L}$. We define the prior$\pi$ on ${\mathcal{F}}_p={\cup }_{k=1}^p{\mathcal{F}}_{k,L}$ as

$$\pi \left(\mathbf{f}\right)=\sum _{k\in ⟦1,p⟧}{w}_k{\pi }_k\left(\mathbf{f}\right){𝟙}_{\left\{\mathbf{f}\in {\mathcal{F}}_{k,L}\right\}},\mathbf{f}\in {\mathcal{F}}_p,$$

where $w_1,\dots ,w_p\ge 0$ and ${\sum }_{k\in ⟦1,p⟧}{w}_k=1$.

We adopt a quasi-Bayesian-flavored procedure: consider the Gibbs quasi-posterior (note that this is not a proper posterior in all generality, hence the term “quasi”):

$${\widehat{\rho }}_t(·)\propto exp(-\lambda S_t(·))\pi (·),$$

where

$$S_t\left(\mathbf{f}\right)=S_{t-1}\left(\mathbf{f}\right)+\Delta (\mathbf{f},x_t)+\frac{\lambda }{2}{\left(\Delta (\mathbf{f},x_t)-\Delta ({\widehat{\mathbf{f}}}_t,x_t)\right)}^2,$$

as advocated by [32,35] who then considered realizations from this quasi-posterior. In the present paper, we will rather focus on a quantity linked to the mode of this quasi-posterior. Indeed, the mode of the quasi-posterior ${\widehat{\rho }}_{t+1}$ is

$$arg\underset{\mathbf{f}\in {\mathcal{F}}_p}{min}\left\{\underset{\left(i\right)}{\underbrace{\sum _{s=1}^t\Delta (\mathbf{f},x_s)}}+\underset{\left(ii\right)}{\underbrace{\frac{\lambda }{2}\sum _{s=1}^t{\left(\Delta (\mathbf{f},x_t)-\Delta ({\widehat{\mathbf{f}}}_t,x_t)\right)}^2}}+\underset{\left(iii\right)}{\underbrace{\frac{ln\pi \left(\mathbf{f}\right)}{\lambda }}}\right\},$$

where (i) is a cumulative loss term, (ii) is a term controlling the variance of the prediction $\mathbf{f}$ to past predictions ${\widehat{\mathbf{f}}}_s,s\le t$, and (iii) can be regarded as a penalty function on the complexity of $\mathbf{f}$ if $\pi$ is well chosen. This mode hence has a similar flavor to follow the best expert or follow the perturbed leader in the setting of prediction with experts (see [22,36], Chapters 3 and 4) if we consider each $\mathbf{f}\in {\mathcal{F}}_p$ as an expert which always delivers constant advice. These remarks yield Algorithm 1.

Algorithm 1 Sequentially learning principal curves.

1: Input parameters: $p>0,\eta >0,\pi \left(z\right)={\mathrm{e}}^{-z}{\mathbb{1}}_{\{z>0\}}$ and penalty function $h:{\mathcal{F}}_p\to {\mathbb{R}}^{+}$    2: Initialization: For each $\mathbf{f}\in {\mathcal{F}}_p$, draw $z_{\mathbf{f}}\sim \pi$ and ${\Delta }_{\mathbf{f},0}=\frac{1}{\eta }(h\left(\mathbf{f}\right)-z_{\mathbf{f}})$    3: For $t=1,\dots ,T$    4:       Get the data $x_t$    5:       Obtain $${\widehat{\mathbf{f}}}_t=\underset{\mathbf{f}\in {\mathcal{F}}_p}{arginf}\left\{\sum _{s=0}^{t-1}{\Delta }_{\mathbf{f},s}\right\},$$       where ${\Delta }_{\mathbf{f},s}=\Delta (\mathbf{f},x_s)$, $s\ge 1$.    6: End for

3. Regret Bounds for Sequential Learning of Principal Curves

We now present our main theoretical results.

Theorem 1.

For any sequence ${\left(x_t\right)}_{1:T}\in B(0,\sqrt{d}R)$, $R\ge 0$ and any penalty function $h:{\mathcal{F}}_p\to {\mathbb{R}}^{+}$, let $\pi \left(z\right)={\mathrm{e}}^{-z}{\mathbb{1}}_{\{z>0\}}$. Let $0<\eta \le \frac{1}{d{(2R+\delta )}^2}$; then the procedure described in Algorithm 1 satisfies

$$\sum _{t=1}^T{\mathbb{E}}_{\pi }\left[\Delta ({\widehat{\mathbf{f}}}_t,x_t)\right]\le \left(1+c_0(\mathrm{e}-1)\eta \right)S_{T,h,\eta }+\frac{1}{\eta }\left(1+ln\sum _{\mathbf{f}\in {\mathcal{F}}_p}{\mathrm{e}}^{-h\left(\mathbf{f}\right)}\right),$$

where $c_0=d{(2R+\delta )}^2$ and

$$S_{T,h,\eta }=\underset{k\in ⟦1,p⟧}{inf}\left\{\underset{\begin{array}{c}\mathbf{f}\in {\mathcal{F}}_p\\ \mathcal{K}\left(\mathbf{f}\right)=k\end{array}}{inf}\left\{\sum _{t=1}^T\Delta (\mathbf{f},x_t)+\frac{h\left(\mathbf{f}\right)}{\eta }\right\}\right\}.$$

The expectation of the cumulative loss of polygonal lines ${\widehat{\mathbf{f}}}_1,\dots ,{\widehat{\mathbf{f}}}_T$ is upper-bounded by the smallest penalized cumulative loss over all $k\in \{1,\dots ,p\}$ up to a multiplicative term $(1+c_0(\mathrm{e}-1)\eta )$, which can be made arbitrarily close to 1 by choosing a small enough $\eta$. However, this will lead to both a large $h\left(\mathbf{f}\right)/\eta$ in $S_{T,h,\eta }$ and a large $\frac{1}{\eta }(1+ln{\sum }_{\mathbf{f}\in {\mathcal{F}}_p}{\mathrm{e}}^{-h\left(\mathbf{f}\right)})$. In addition, another important issue is the choice of the penalty function h. For each $\mathbf{f}\in {\mathcal{F}}_p$, $h\left(\mathbf{f}\right)$ should be large enough to ensure a small ${\sum }_{\mathbf{f}\in {\mathcal{F}}_p}{\mathrm{e}}^{-h\left(\mathbf{f}\right)}$, but not too large to avoid overpenalization and a larger value for $S_{T,h,\eta }$. We therefore set

$$h\left(\mathbf{f}\right)\ge ln\left(p\mathrm{e}\right)+ln\left|\right\{\mathbf{f}\in {\mathcal{F}}_p,\mathcal{K}\left(\mathbf{f}\right)=k\left\}\right|$$

(3)

for each $\mathbf{f}$ with k segments (where $\left|M\right|$ denotes the cardinality of a set M) since it leads to

$$\sum _{\mathbf{f}\in {\mathcal{F}}_p}{\mathrm{e}}^{-h\left(\mathbf{f}\right)}=\sum _{k\in ⟦1,p⟧}\sum _{\begin{array}{c}\mathbf{f}\in {\mathcal{F}}_p\\ \mathcal{K}\left(\mathbf{f}\right)=k\end{array}}{\mathrm{e}}^{-h\left(\mathbf{f}\right)}\le \sum _{k\in ⟦1,p⟧}\frac{1}{p\mathrm{e}}\le \frac{1}{\mathrm{e}}.$$

The penalty function $h\left(\mathbf{f}\right)=c_1\mathcal{K}\left(\mathbf{f}\right)+c_{2}L+c_3$ satisfies (3), where $c_1,c_2,c_3$ are constants depending on R, d, $\delta$, p (this is proven in Lemma 3, in Section 6). We therefore obtain the following corollary.

Corollary 1.

Under the assumptions of Theorem 1, let

$$\eta =min\left\{\frac{1}{d{(2R+\delta )}^2},\sqrt{\frac{c_{1}p+c_{2}L+c_3}{c_0(\mathrm{e}-1){inf}_{\mathbf{f}\in {\mathcal{F}}_p}{\sum }_{t=1}^T\Delta (\mathbf{f},x_t)}}\right\}.$$

Then

$$\begin{array}{c}\sum _{t=1}^T\mathbb{E}\left[\Delta ({\widehat{\mathbf{f}}}_t,x_t)\right]\le \underset{k\in ⟦1,p⟧}{inf}\left\{\underset{\begin{array}{c}\mathbf{f}\in {\mathcal{F}}_p\\ \mathcal{K}\left(\mathbf{f}\right)=k\end{array}}{inf}\left\{\sum _{t=1}^T\Delta (\mathbf{f},x_t)+\sqrt{c_0(\mathrm{e}-1)r_{T,k,L}}\right\}\right\}\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+\sqrt{c_0(\mathrm{e}-1)r_{T,p,L}}+c_0(\mathrm{e}-1)(c_{1}p+c_{2}L+c_3),\hfill \end{array}$$

where $r_{T,k,L}={inf}_{\mathbf{f}\in {\mathcal{F}}_p}{\sum }_{t=1}^T\Delta (\mathbf{f},x_t)(c_{1}k+c_{2}L+c_3)$.

Proof. 

Note that

$$\sum _{t=1}^T\mathbb{E}\left[\Delta ({\widehat{\mathbf{f}}}_t,x_t)\right]\le S_{T,h,\eta }+\eta c_0(\mathrm{e}-1)\underset{\mathbf{f}\in {\mathcal{F}}_p}{inf}\sum _{t=1}^T\Delta (\mathbf{f},x_t)+c_0(\mathrm{e}-1)(c_{0}p+c_{2}L+c_3),$$

and we conclude by setting

$$\eta =\sqrt{\frac{c_{1}p+c_{2}L+c_3}{c_0(\mathrm{e}-1){inf}_{\mathbf{f}\in {\mathcal{F}}_p}{\sum }_{t=1}^T\Delta (\mathbf{f},x_t)}}.$$

 □

Sadly, Corollary 1 is not of much practical use since the optimal value for $\eta$ depends on ${inf}_{\mathbf{f}\in {\mathcal{F}}_p}{\sum }_{t=1}^T\Delta (\mathbf{f},x_t)$ which is obviously unknown, even more so at time $t=0$. We therefore provide an adaptive refinement of Algorithm 1 in the following Algorithm 2.

Algorithm 2 Sequentially and adaptively learning principal curves.

1: Input parameters: $p>0$, $L>0$, $\pi$, h and ${\eta }_0=\frac{\sqrt{c_{1}p+c_{2}L+c_3}}{c_0\sqrt{\mathrm{e}-1}}$    2: Initialization: For each $\mathbf{f}\in {\mathcal{F}}_p$, draw $z_{\mathbf{f}}\sim \pi$, ${\Delta }_{\mathbf{f},0}=\frac{1}{{\eta }_0}(h\left(\mathbf{f}\right)-z_{\mathbf{f}})$ and ${\widehat{\mathbf{f}}}_0=\underset{\mathbf{f}\in {\mathcal{F}}_p}{arginf}{\Delta }_{\mathbf{f},0}$    3: For $t=1,\dots ,T$    4:       Compute ${\eta }_t=\frac{\sqrt{c_{1}p+c_{2}L+c_3}}{c_0\sqrt{(\mathrm{e}-1)t}}$    5:       Get data $x_t$ and compute ${\Delta }_{\mathbf{f},t}=\Delta (\mathbf{f},x_t)+\left(\frac{1}{{\eta }_t}-\frac{1}{{\eta }_{t-1}}\right)\left(h\left(\mathbf{f}\right)-z_{\mathbf{f}}\right)$    6:       Obtain $${\widehat{\mathbf{f}}}_t=\underset{\mathbf{f}\in {\mathcal{F}}_p}{arginf}\left\{\sum _{s=0}^{t-1}{\Delta }_{\mathbf{f},s}\right\}.$$ (4)    7: End for

Theorem 2.

For any sequence ${\left(x_t\right)}_{1:T}\in B(0,\sqrt{d}R),R\ge 0$, let $h\left(\mathbf{f}\right)=c_1\mathcal{K}\left(\mathbf{f}\right)+c_{2}L+c_3$ where $c_1$, $c_2$, $c_3$ are constants depending on $R,d,\delta ,lnp$. Let $\pi \left(z\right)={\mathrm{e}}^{-z}{\mathbb{1}}_{\{z>0\}}$ and

$${\eta }_0=\frac{\sqrt{c_{1}p+c_{2}L+c_3}}{c_0\sqrt{\mathrm{e}-1}},{\eta }_t=\frac{\sqrt{c_{1}p+c_{2}L+c_3}}{c_0\sqrt{(\mathrm{e}-1)t}},$$

where $t\ge 1$ and $c_0=d{(2R+\delta )}^2$. Then the procedure described in Algorithm 2 satisfies

$$\begin{array}{c}\sum _{t=1}^T\mathbb{E}\left[\Delta ({\widehat{\mathbf{f}}}_t,x_t)\right]\le \underset{k\in ⟦1,p⟧}{inf}\left\{\underset{\begin{array}{c}\mathbf{f}\in {\mathcal{F}}_p\\ \mathcal{K}\left(\mathbf{f}\right)=k\end{array}}{inf}\left\{\sum _{t=1}^T\Delta (\mathbf{f},x_t)+c_0\sqrt{(\mathrm{e}-1)T(c_{1}k+c_{2}L+c_3)}\right\}\right\}\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+2c_0\sqrt{(\mathrm{e}-1)T(c_{1}p+c_{2}L+c_3)}.\hfill \end{array}$$

The message of this regret bound is that the expected cumulative loss of polygonal lines ${\widehat{\mathbf{f}}}_1,\dots ,{\widehat{\mathbf{f}}}_T$ is upper-bounded by the minimal cumulative loss over all $k\in \{1,\dots ,p\}$, up to an additive term which is sublinear in T. The actual magnitude of this remainder term is $\sqrt{kT}$. When L is fixed, the number k of segments is a measure of complexity of the retained polygonal line. This bound therefore yields the same magnitude as (1), which is the most refined bound in the literature so far ([18] where the optimal values for k and L were obtained in a model selection fashion).

4. Implementation

The argument of the infimum in Algorithm 2 is taken over ${\mathcal{F}}_p={\cup }_{k=1}^p{\mathcal{F}}_{k,L}$ which has a cardinality of order ${\left|{\mathcal{Q}}_{\delta }\right|}^p$, making any greedy search largely time-consuming. We instead turn to the following strategy: Given a polygonal line ${\widehat{\mathbf{f}}}_t\in {\mathcal{F}}_{k_t,L}$ with $k_t$ segments, we consider, with a certain proportion, the availability of ${\widehat{\mathbf{f}}}_{t+1}$ within a neighborhood $\mathcal{U}\left({\widehat{\mathbf{f}}}_t\right)$ (see the formal definition below) of ${\widehat{\mathbf{f}}}_t$. This consideration is well suited for the principal curves setting, since if observation $x_t$ is close to ${\widehat{\mathbf{f}}}_t$, one can expect that the polygonal line which well fits observations $x_s,s=1,\dots ,t$ lies in a neighborhood of ${\widehat{\mathbf{f}}}_t$. In addition, if each polygonal line $\mathbf{f}$ is regarded as an action, we no longer assume that all actions are available at all times, and allow the set of available actions to vary at each time. This is a model known as “sleeping experts (or actions)” in prior work [37,38]. In this setting, defining the regret with respect to the best action in the whole set of actions in hindsight remains difficult, since that action might sometimes be unavailable. Hence, it is natural to define the regret with respect to the best ranking of all actions in the hindsight according to their losses or rewards, and at each round one chooses among the available actions by selecting the one which ranks the highest. Ref. [38] introduced this notion of regret and studied both the full-information (best action) and partial-information (multi-armed bandit) settings with stochastic and adversarial rewards and adversarial action availability. They pointed out that the EXP4 algorithm [37] attains the optimal regret in the adversarial rewards case but has a runtime exponential in the number of all actions. Ref. [39] considered full and partial information with stochastic action availability and proposed an algorithm that runs in polynomial time. In what follows, we materialize our implementation by resorting to “sleeping experts”, i.e., a special set of available actions that adapts to the setting of principal curves.

Let $\sigma$ denote an ordering of $|{\mathcal{F}}_p|$ actions, and ${\mathcal{A}}_t$ a subset of the available actions at round t. We let $\sigma \left({\mathcal{A}}_t\right)$ denote the highest ranked action in ${\mathcal{A}}_t$. In addition, for any action $\mathbf{f}\in {\mathcal{F}}_p$ we define the reward $r_{\mathbf{f},t}$ of $\mathbf{f}$ at round $t,t\ge 0$ by

$$r_{\mathbf{f},t}=c_0-\Delta (\mathbf{f},x_t).$$

It is clear that $r_{\mathbf{f},t}\in (0,c_0)$. The convention from losses to gains is done in order to facilitate the subsequent performance analysis. The reward of an ordering $\sigma$ is the cumulative reward of the selected action at each time:

$$\sum _{t=1}^T{r}_{\sigma \left({\mathcal{A}}_t\right),t},$$

and the reward of the best ordering is ${max}_{\sigma }{\sum }_{t=0}^T{r}_{\sigma \left({\mathcal{A}}_t\right),t}$ (respectively, $\mathbb{E}\left[{max}_{\sigma }{\sum }_{t=1}^T{r}_{\sigma \left({\mathcal{A}}_t\right),t}\right]$ when ${\mathcal{A}}_t$ is stochastic).

Our procedure starts with a partition step which aims at identifying the “relevant” neighborhood of an observation $x\in {\mathbb{R}}^d$ with respect to a given polygonal line, and then proceeds with the definition of the neighborhood of an action $\mathbf{f}$. We then provide the full implementation and prove a regret bound.

Partition. For any polygonal line $\mathbf{f}$ with k segments, we denote by $\stackrel{\rightharpoonup }{\mathbf{V}}=\left(v_1,\dots ,v_{k+1}\right)$ its vertices and by $s_i,i=1,\dots ,k$ the line segments connecting $v_i$ and $v_{i+1}$. In the sequel, we use $\mathbf{f}\left(\stackrel{\rightharpoonup }{\mathbf{V}}\right)$ to represent the polygonal line formed by connecting consecutive vertices in $\stackrel{\rightharpoonup }{\mathbf{V}}$ if no confusion arises. Let $V_i,i=1,\dots ,k+1$ and $S_i,i=1,\dots ,k$ be the Voronoi partitions of ${\mathbb{R}}^d$ with respect to $\mathbf{f}$, i.e., regions consisting of all points closer to vertex $v_i$ or segment $s_i$. Figure 5 shows an example of Voronoi partition with respect to $\mathbf{f}$ with three segments.

Neighborhood. For any $x\in {\mathbb{R}}^d$, we define the neighborhood $\mathcal{N}\left(x\right)$ with respect to $\mathbf{f}$ as the union of all Voronoi partitions whose closure intersects with two vertices connecting the projection $\mathbf{f}\left(s_{\mathbf{f}}\left(x\right)\right)$ of x to $\mathbf{f}$. For example, for the point x in Figure 5, its neighborhood $\mathcal{N}\left(x\right)$ is the union of $S_2,V_3,S_3$ and $V_4$. In addition, let ${\mathcal{N}}_t\left(x\right)=\left\{x_s\in \mathcal{N}\left(x\right),s=1,\dots ,t.\right\}$ be the set of observations $x_{1:t}$ belonging to $\mathcal{N}\left(x\right)$ and ${\overline{\mathcal{N}}}_t\left(x\right)$ be its average. Let $\mathcal{D}\left(M\right)={sup}_{x,y\in M}\left|\right|x-{y\left|\right|}_2$ denote the diameter of set $M\subset {\mathbb{R}}^d$. We finally define the local grid ${\mathcal{Q}}_{\delta ,t}\left(x\right)$ of $x\in {\mathbb{R}}^d$ at time t as

$${\mathcal{Q}}_{\delta ,t}\left(x\right)=B\left({\overline{\mathcal{N}}}_t\left(x\right),\mathcal{D}\left({\mathcal{N}}_t(x\right)\right)\cap {\mathcal{Q}}_{\delta }.$$

We can finally proceed to the definition of the neighborhood $\mathcal{U}\left({\widehat{\mathbf{f}}}_t\right)$ of ${\widehat{\mathbf{f}}}_t$. Assume ${\widehat{\mathbf{f}}}_t$ has $k_t+1$ vertices $\stackrel{\rightharpoonup }{\mathbf{V}}=(\underset{\left(i\right)}{\underbrace{v_{1:i_t-1}}},\underset{\left(ii\right)}{\underbrace{v_{i_t:j_t-1}}},\underset{\left(iii\right)}{\underbrace{v_{j_t:k_t+1}}})$, where vertices of $\left(ii\right)$ belong to ${\mathcal{Q}}_{\delta ,t}\left(x_t\right)$ while those of $\left(i\right)$ and $\left(iii\right)$ do not. The neighborhood $\mathcal{U}\left({\widehat{\mathbf{f}}}_t\right)$ consists of $\mathbf{f}$ sharing vertices $\left(i\right)$ and $\left(iii\right)$ with ${\widehat{\mathbf{f}}}_t$, but can be equipped with different vertices $\left(ii\right)$ in ${\mathcal{Q}}_{\delta ,t}\left(x_t\right)$; i.e.,

$$\mathcal{U}\left({\widehat{\mathbf{f}}}_t\right)=\left\{\mathbf{f}\left(\stackrel{\rightharpoonup }{\mathbf{V}}\right),\stackrel{\rightharpoonup }{\mathbf{V}}=\left(v_{1:i_t-1},v_{1:m},v_{j_t:k_t+1}\right)\right\},$$

where $v_{1:m}\in {\mathcal{Q}}_{\delta ,t}\left(x_t\right)$ and m is given by

$$m=\left\{\begin{array}{cc}{j}_t-i_t-1\hfill & \mathrm{reduce\; segments\; by\; 1\; unit},\hfill \\ j_t-i_t\hfill & \mathrm{same\; number\; of\; segments},\hfill \\ j_t-i_t+1\hfill & \mathrm{increase\; segments\; by\; 1\; unit}.\hfill \end{array}\right.$$

In Algorithm 3, we initiate the principal curve ${\widehat{\mathbf{f}}}_1$ as the first component line segment whose vertices are the two farthest projections of data $x_{1:t_0}$ ($t_0$ can be set to 20 in practice) on the first component line. The reward of $\mathbf{f}$ at round t in this setting is therefore $r_{\mathbf{f},t}=c_0-\Delta (\mathbf{f},x_{t_0+t})$. Algorithm 3 has an exploration phase (when $I_t=1$) and an exploitation phase ($I_t=0$). In the exploration phase, it is allowed to observe rewards of all actions and to choose an optimal perturbed action from the set ${\mathcal{F}}_p$ of all actions. In the exploitation phase, only rewards of a part of actions can be accessed and rewards of others are estimated by a constant, and we update our action from the neighborhood $\mathcal{U}\left({\widehat{\mathbf{f}}}_{t-1}\right)$ of the previous action ${\widehat{\mathbf{f}}}_{t-1}$. This local update (or search) greatly reduces computation complexity since $|\mathcal{U}\left({\widehat{\mathbf{f}}}_{t-1}\right)|\ll \left|{\mathcal{F}}_p\right|$ when p is large. In addition, this local search will be enough to account for the case when $x_t$ locates in $\mathcal{U}\left({\widehat{\mathbf{f}}}_{t-1}\right)$. The parameter $\beta$ needs to be carefully calibrated since it should not be too large to ensure that the condition $cond\left(t\right)$ is non-empty; otherwise, all rewards are estimated by the same constant and thus lead to the same descending ordering of tuples for both $\left({\sum }_{s=1}^{t-1}{\widehat{r}}_{\mathbf{f},s},\mathbf{f}\in {\mathcal{F}}_p\right)$ and $\left({\sum }_{s=1}^t{\widehat{r}}_{\mathbf{f},s},\mathbf{f}\in {\mathcal{F}}_p\right)$. Therefore, we may face the risk of having ${\widehat{\mathbf{f}}}_{t+1}$ in the neighborhood of ${\widehat{\mathbf{f}}}_t$ even if we are in the exploration phase at time $t+1$. Conversely, very small $\beta$ could result in large bias for the estimation $\frac{r_{\mathbf{f},t}}{\mathbb{P}\left({\widehat{\mathbf{f}}}_t=\mathbf{f}|{\mathcal{H}}_t\right)}$ of $r_{\mathbf{f},t}$. Note that the exploitation phase is close yet different to the label efficient prediction ([40], Remark 1.1) since we allow an action at time t to be different from the previous one. Ref. [41] proposed the geometric resampling method to estimate the conditional probability $\mathbb{P}\left({\widehat{\mathbf{f}}}_t=\mathbf{f}|{\mathcal{H}}_t\right)$ since this quantity often does not have an explicit form. However, due to the simple exponential distribution of $z_{\mathbf{f}}$ chosen in our case, an explicit form of $\mathbb{P}\left({\widehat{\mathbf{f}}}_t=\mathbf{f}|{\mathcal{H}}_t\right)$ is straightforward.

Algorithm 3 A locally greedy algorithm for sequentially learning principal curves.

1: Input parameters: $p>0$, $R>0$, $L>0$, $ϵ>0$, $\alpha >0$, $1>\beta >0$ and any penalty function h    2: Initialization: Given ${\left(x_t\right)}_{1:t_0}$, obtain ${\widehat{\mathbf{f}}}_1$ as the first principal component    3: For $t=2,\dots ,T$    4:       Draw $I_t\sim Bernoulli\left(ϵ\right)$ and $z_{\mathbf{f}}\sim \pi$.    5:       Let $${\widehat{\sigma }}_t=\mathrm{sort}\left(\mathbf{f},\sum _{s=1}^{t-1}{\widehat{r}}_{\mathbf{f},s}-\frac{1}{{\eta }_{t-1}}h\left(\mathbf{f}\right)+\frac{1}{{\eta }_{t-1}}{z}_{\mathbf{f}}\right),$$ i.e., sorting all $\mathbf{f}\in {\mathcal{F}}_p$ in descending order according to their perturbed cumulative reward till $t-1$.    6:       If $I_t=1$, set ${\mathcal{A}}_t={\mathcal{F}}_p$ and ${\widehat{\mathbf{f}}}_t={\widehat{\sigma }}^t\left({\mathcal{A}}_t\right)$ and observe $r_{{\widehat{\mathbf{f}}}_t,t}$    7:              $${\widehat{r}}_{\mathbf{f},t}=r_{\mathbf{f},t}\mathrm{for}\mathbf{f}\in {\mathcal{F}}_p.$$    8:        If $I_t=0$, set ${\mathcal{A}}_t=\mathcal{U}\left({\widehat{\mathbf{f}}}_{t-1}\right)$, ${\widehat{\mathbf{f}}}_t={\widehat{\sigma }}^t\left({\mathcal{A}}_t\right)$ and observe $r_{{\widehat{\mathbf{f}}}_t,t}$    9:              $${\widehat{r}}_{\mathbf{f},t}=\left\{\begin{array}{ccc}\frac{r_{\mathbf{f},t}}{\mathbb{P}\left({\widehat{\mathbf{f}}}_t=\mathbf{f}|{\mathcal{H}}_t\right)}\hfill & if\mathbf{f}\in \mathcal{U}\left({\widehat{\mathbf{f}}}_{t-1}\right)\cap cond\left(t\right)\mathrm{and}{\widehat{\mathbf{f}}}_t=\mathbf{f},\alpha \hfill & \mathrm{otherwise},\end{array}\right.$$ where ${\mathcal{H}}_t$ denotes all the randomness before time t and $\mathit{cond}\left(t\right)=\left\{\mathbf{f}\in {\mathcal{F}}_p:\mathbb{P}\left({\widehat{\mathbf{f}}}_t=\mathbf{f}|{\mathcal{H}}_t\right)>\beta \right\}$. In particular, when $t=1$, we set ${\widehat{r}}_{\mathbf{f},1}=r_{\mathbf{f},1}$ for all $\mathbf{f}\in {\mathcal{F}}_p$, $\mathcal{U}\left({\widehat{\mathbf{f}}}_0\right)=\varnothing$ and ${\widehat{r}}_{{\widehat{\sigma }}^1\left(\mathcal{U}\left({\widehat{\mathbf{f}}}_0\right)\right),1}\equiv 0$.    10: End for

Theorem 3.

Assume that $p>6$, $T\ge 2|{\mathcal{F}}_p{|}^2$ and let $\beta ={\left|{\mathcal{F}}_p\right|}^{-\frac{1}{2}}{T}^{-\frac{1}{4}}$, $\alpha =\frac{c_0}{\beta }$, ${\widehat{c}}_0=\frac{2c_0}{\beta }$, $ϵ=1-{\left|{\mathcal{F}}_p\right|}^{\frac{1}{2}-\frac{3}{p}}{T}^{-\frac{1}{4}}$ and

$${\eta }_1={\eta }_2=\cdots ={\eta }_T=\frac{\sqrt{c_{1}p+c_{2}L+c_3}}{\sqrt{T(e-1)}{\widehat{c}}_0}.$$

Then the procedure described in Algorithm 3 satisfies the regret bound

$$\sum _{t=1}^T\mathbb{E}\left[\Delta \left({\widehat{\mathbf{f}}}_t,x_t\right)\right]\le \underset{\mathbf{f}\in {\mathcal{F}}_p}{inf}\mathbb{E}\left[\sum _{t=1}^T\Delta \left(\mathbf{f},x_t\right)\right]+\mathcal{O}\left(T^{\frac{3}{4}}\right).$$

The proof of Theorem 3 is presented in Section 6. The regret is upper bounded by a term of order $\left({\left|{\mathcal{F}}_p\right|}^{\frac{1}{2}}{T}^{\frac{3}{4}}\right)$, sublinear in T. The term $(1-ϵ)c_{0}T=c_0{\left|{\mathcal{F}}_p\right|}^{\frac{1}{2}}{T}^{\frac{3}{4}}$ is the price to pay for the local search (with a proportion $1-ϵ$) of polygonal line ${\widehat{\mathbf{f}}}_t$ in the neighborhood of the previous ${\widehat{\mathbf{f}}}_{t-1}$. If $ϵ=1$, we would have that ${\widehat{c}}_0=c_0$, and the last two terms in the first inequality of Theorem 3 would vanish; hence, the upper bound reduces to Theorem 2. In addition, our algorithm achieves an order that is smaller (from the perspective of both the number $\left|{\mathcal{F}}_p\right|$ of all actions and the total rounds T) than [39] since at each time, the availability of actions for our algorithm can be either the whole action set or a neighborhood of the previous action while [39] consider at each time only partial and independent stochastic available set of actions generated from a predefined distribution.

5. Numerical Experiments

We illustrate the performance of Algorithm 3 on synthetic and real-life data. Our implementation (hereafter denoted by slpc—Sequential Learning of Principal Curves) is conducted with the R language and thus our most natural competitors are the R package princurve, which is the algorithm from [10], and incremental, which is the algorithm from SCMS [23]. We let $p=50$, $R={max}_{t=1,\dots ,T}{\left|\right|x\left|\right|}_2/\sqrt{d}$, $L=0.1p\sqrt{d}R$. The spacing $\delta$ of the lattice is adjusted with respect to data scale.

Synthetic data We generate a dataset $\left\{x_t\in {\mathbb{R}}^2,t=1,\dots ,500\right\}$ uniformly along the curve $y=0.05\times {(x-5)}^3$, $x\in [0,10]$.

Table 1. The first line is the regret (cumulative loss) on synthetic data (average over 10 trials, with standard deviation in brackets). Second and third lines are the average computation time for two values of the time horizon T. princurve and incremental SCMS are deterministic, hence the zero standard deviation for regret.

Ground Truth	Princurve	Incremental SCMS	slpc
2.48 (0)	26.02 (0)	19.09 (0)	20.83 (3.23)
T = 500	0.029 s (0.0001 s)	18.79 s (0.007 s)	1.44 s (0.030 s)
T = 5000	0.35 s (0.006 s)	>60 s (NA)	4.13 s (0.807 s)

shows the regret (first row) for

• the ground truth (sum of squared distances of all points to the true curve),
• princurve and incremental SCMS (sum of squared distances between observation $x_{t+1}$ and fitted princurve on observations $x_{1:t}$),
• slpc (regret being equal to ${\sum }_{t=0}^{T-1}\mathbb{E}[\Delta ({\widehat{\mathbf{f}}}_{t+1},x_{t+1})]$ in both cases).

The mean computation time with different values for the time horizons T are also reported.

Table 1 demonstrates the advantages of our method slpc, as it achieved the optimal tradeoff between performance (in terms of regret) and runtime. Although princurve outperformed the other two algorithms in terms of computation time, it yielded the largest regret, since it outputs a curve which does not pass in “the middle of data” but rather bends towards the curvature of the data cloud, as shown in Figure 6 where the predicted principal curves ${\widehat{\mathbf{f}}}_{t+1}$ for princurve, incremental SCMS and slpc are presented. incremental SCMS and slpc both yielded satisfactory results, although the mean computation time of splc was significantly smaller than that of incremental SCMS (the reason being that eigenvectors of the Hessian of PDF need to be computed in incremental SCMS). Figure 7 showed, respectively, the estimation of the regret of slpc and its per-round value (i.e., the cumulative loss divided by the number of rounds) both with respect to the round t. The jumps in the per-round curve occurred at the beginning, due to the initialization from a first principal component and to the collection of new data. When data accumulates, the vanishing pattern of the per-round curve illustrates that the regret is sublinear in t, which matches our aforementioned theoretical results.

In addition, to better illustrate the way slpc works between two epochs, Figure 8 focuses on the impact of collecting a new data point on the principal curve. We see that only a local vertex is impacted, whereas the rest of the principal curve remains unaltered. This cutdown in algorithmic complexity is one the key assets of slpc.

Synthetic data in high dimension. We also apply our algorithm on a dataset $\{x_t\in {\mathbb{R}}^6,$$t=1,2,\dots ,200\}$ in higher dimension. It is generated uniformly along a parametric curve whose coordinates are

$$\left(\begin{array}{c}0.5tcos\left(t\right)\\ 0.5tsin\left(t\right)\\ 0.5t\\ -t\\ \sqrt{t}\\ 2ln(t+1)\end{array}\right)$$

where t takes 100 equidistant values in $[0,2\pi ]$. To the best of our knowledge, [10,16,18] only tested their algorithm on 2-dimensional data. This example aims at illustrating that our algorithm also works on higher dimensional data.

Table 2. Regret (cumulative loss) on synthetic high dimensional data in (average over 10 trials, with standard deviation in brackets). princurve and incremental SCMS are deterministic, hence the zero standard deviation.

Ground Truth	Princurve	Incremental SCMS	slpc
3.290 (0)	14.204 (0)	5.38 (0)	6.797 (0.409)

shows the regret for the ground truth, princurve and slpc.

In addition, Figure 9 shows the behaviour of slpc (green) on each dimension.

Seismic data. Seismic data spanning long periods of time are essential for a thorough understanding of earthquakes. The “Centennial Earthquake Catalog” [42] aims at providing a realistic picture of the seismicity distribution on Earth. It consists in a global catalog of locations and magnitudes of instrumentally recorded earthquakes from 1900 to 2008. We focus on a particularly representative seismic active zone (a lithospheric border close to Australia) whose longitude is between E${130}^{\circ }$ to E${180}^{\circ }$ and latitude between S${70}^{\circ }$ to N${30}^{\circ }$, with $T=218$ seismic recordings. As shown in Figure 10, slpc recovers nicely the tectonic plate boundary, but both princurve and incremental SCMS with well-calibrated bandwidth fail to do so.

Lastly, since no ground truth is available, we used the $R^2$ coefficient to assess the performance (residuals are replaced by the squared distance between data points and their projections onto the principal curve). The average over 10 trials was 0.990.

Back to Seismic Data.Figure 11 was taken from the USGS website (https://earthquake.usgs.gov/data/centennial/) and gives the global locations of earthquakes for the period 1900–1999. The seismic data (latitude, longitude, magnitude of earthquakes, etc.) used in the present paper may be downloaded from this website.

Daily Commute Data. The identification of segments of personal daily commuting trajectories can help taxi or bus companies to optimize their fleets and increase frequencies on segments with high commuting activity. Sequential principal curves appear to be an ideal tool to address this learning problem: we tested our algorithm on trajectory data from the University of Illinois at Chicago (https://www.cs.uic.edu/~boxu/mp2p/gps_data.html). The data were obtained from the GPS reading systems carried by two of the laboratory members during their daily commute for 6 months in the Cook county and the Dupage county of Illinois. Figure 12 presents the learning curves yielded by princurve and slpc on geolocalization data for the first person, on May 30. A particularly remarkable asset of slpc is that abrupt curvature in the data sequence was perfectly captured, whereas princurve does not enjoy the same flexibility. Again, we used the $R^2$ coefficient to assess the performance (where residuals are replaced by the squared distances between data points and their projections onto the principal curve). The average over 10 trials was 0.998.

6. Proofs

This section contains the proof of Theorem 2 (note that Theorem 1 is a straightforward consequence, with ${\eta }_t=\eta$, $t=0,\dots ,T$) and the proof of Theorem 3 (which involves intermediary lemmas). Let us first define for each $t=0,\dots ,T$ the following forecaster sequence ${\left({\widehat{\mathbf{f}}}_t^{\star }\right)}_t$

$$\begin{array}{cc}\hfill & {\widehat{\mathbf{f}}}_0^{\star }=\underset{\mathbf{f}\in {\mathcal{F}}_p}{arginf}\left\{{\Delta }_{\mathbf{f},0}\right\}=\underset{\mathbf{f}\in {\mathcal{F}}_p}{arginf}\left\{\frac{1}{{\eta }_0}h\left(\mathbf{f}\right)-\frac{1}{{\eta }_0}{z}_{\mathbf{f}}\right\},\hfill \\ \hfill & {\widehat{\mathbf{f}}}_t^{\star }=\underset{\mathbf{f}\in {\mathcal{F}}_p}{arginf}\left\{\sum _{s=0}^t{\Delta }_{\mathbf{f},s}\right\}=\underset{\mathbf{f}\in {\mathcal{F}}_p}{arginf}\left\{\sum _{s=1}^t\Delta (\mathbf{f},x_s)+\frac{1}{{\eta }_{t-1}}h\left(\mathbf{f}\right)-\frac{1}{{\eta }_{t-1}}{z}_{\mathbf{f}}\right\},t\ge 1.\hfill \end{array}$$

Note that ${\widehat{\mathbf{f}}}_t^{\star }$ is an “illegal” forecaster since it peeks into the future. In addition, denote by

$${\mathbf{f}}^{\star }=\underset{\mathbf{f}\in {\mathcal{F}}_p}{arginf}\left\{\sum _{t=1}^T\Delta (\mathbf{f},x_t)+\frac{1}{{\eta }_T}h\left(\mathbf{f}\right)\right\}$$

the polygonal line in ${\mathcal{F}}_p$ which minimizes the cumulative loss in the first T rounds plus a penalty term. ${\mathbf{f}}^{\star }$ is deterministic, and ${\widehat{\mathbf{f}}}_t^{\star }$ is a random quantity (since it depends on $z_{\mathbf{f}}$, $\mathbf{f}\in {\mathcal{F}}_p$ drawn from $\pi$). If several $\mathbf{f}$ attain the infimum, we chose ${\mathbf{f}}_T^{\star }$ as the one having the smallest complexity. We now enunciate the first (out of three) intermediary technical result.

Lemma 1.

For any sequence $x_1,\dots ,x_T$ in $B(0,\sqrt{d}R)$,

$$\sum _{t=0}^T{\Delta }_{{\widehat{\mathbf{f}}}_t^{\star },t}\le \sum _{t=0}^T{\Delta }_{{\widehat{\mathbf{f}}}_T^{\star },t},\pi \text{-}\mathit{almost}\mathit{surely}.$$

(5)

Proof. 

Proof by induction on T. Clearly (5) holds for $T=0$. Assume that (5) holds for $T-1$:

$$\sum _{t=0}^{T-1}{\Delta }_{{\widehat{\mathbf{f}}}_t^{\star },t}\le \sum _{t=0}^{T-1}{\Delta }_{{\widehat{\mathbf{f}}}_{T-1}^{\star },t}.$$

Adding ${\Delta }_{{\widehat{\mathbf{f}}}_T^{\star },T}$ to both sides of the above inequality concludes the proof. □

By (5) and the definition of ${\widehat{\mathbf{f}}}_T^{\star }$, for $k\ge 1$, we have $\pi$-almost surely that

$$\begin{array}{cc}\hfill \sum _{t=1}^T\Delta ({\widehat{\mathbf{f}}}_t^{\star },x_t)& \le \sum _{t=1}^T\Delta ({\widehat{\mathbf{f}}}_T^{\star },x_t)+\frac{1}{{\eta }_T}h\left({\widehat{\mathbf{f}}}_T^{\star }\right)-\frac{1}{{\eta }_T}{Z}_{{\widehat{\mathbf{f}}}_T^{\star }}+\sum _{t=0}^T\left(\frac{1}{{\eta }_{t-1}}-\frac{1}{{\eta }_t}\right)\left(h\left({\widehat{\mathbf{f}}}_t^{\star }\right)-Z_{{\widehat{\mathbf{f}}}_t^{\star }}\right)\hfill \\ \hfill & \le \sum _{t=1}^T\Delta ({\mathbf{f}}^{\star },x_t)+\frac{1}{{\eta }_T}h\left({\mathbf{f}}^{\star }\right)-\frac{1}{{\eta }_T}{Z}_{{\mathbf{f}}^{\star }}+\sum _{t=0}^T\left(\frac{1}{{\eta }_{t-1}}-\frac{1}{{\eta }_t}\right)\left(h\left({\widehat{\mathbf{f}}}_t^{\star }\right)-Z_{{\widehat{\mathbf{f}}}_t^{\star }}\right)\hfill \\ \hfill & =\underset{\mathbf{f}\in {\mathcal{F}}_p}{inf}\left\{\sum _{t=1}^T\Delta (\mathbf{f},x_t)+\frac{1}{{\eta }_T}h\left(\mathbf{f}\right)\right\}-\frac{1}{{\eta }_T}{Z}_{{\mathbf{f}}^{\star }}+\sum _{t=0}^T\left(\frac{1}{{\eta }_{t-1}}-\frac{1}{{\eta }_t}\right)\left(h\left({\widehat{\mathbf{f}}}_t^{\star }\right)-Z_{{\widehat{\mathbf{f}}}_t^{\star }}\right),\hfill \end{array}$$

where $1/{\eta }_{-1}=0$ by convention. The second and third inequality is due to respectively the definition of ${\widehat{\mathbf{f}}}_T^{\star }$ and ${\mathbf{f}}_T^{\star }$. Hence

$$\begin{array}{cc}\hfill \mathbb{E}\left[\sum _{t=1}^T\Delta \left({\widehat{\mathbf{f}}}_t^{\star },x_t\right)\right]& \le \underset{\mathbf{f}\in {\mathcal{F}}_p}{inf}\left\{\sum _{t=1}^T\Delta (\mathbf{f},x_t)+\frac{1}{{\eta }_T}h\left(\mathbf{f}\right)\right\}-\frac{1}{{\eta }_T}\mathbb{E}\left[Z_{{\mathbf{f}}_T^{\star }}\right]\hfill \\ & +\sum _{t=0}^T\mathbb{E}\left[\left(\frac{1}{{\eta }_t}-\frac{1}{{\eta }_{t-1}}\right)\left(-h\left({\widehat{\mathbf{f}}}_t^{\star }\right)+Z_{{\widehat{\mathbf{f}}}_t^{\star }}\right)\right]\hfill \\ \hfill & \le \underset{\mathbf{f}\in {\mathcal{F}}_p}{inf}\left\{\sum _{t=1}^T\Delta (\mathbf{f},x_t)+\frac{1}{{\eta }_T}h\left(\mathbf{f}\right)\right\}+\sum _{t=1}^T\left(\frac{1}{{\eta }_t}-\frac{1}{{\eta }_{t-1}}\right)\mathbb{E}\left[\underset{\mathbf{f}\in {\mathcal{F}}_p}{sup}\left(-h\left(\mathbf{f}\right)+Z_{\mathbf{f}}\right)\right]\hfill \\ \hfill & =\underset{\mathbf{f}\in {\mathcal{F}}_p}{inf}\left\{\sum _{t=1}^T\Delta (\mathbf{f},x_t)+\frac{1}{{\eta }_T}h\left(\mathbf{f}\right)\right\}+\frac{1}{{\eta }_T}\mathbb{E}\left[\underset{\mathbf{f}\in {\mathcal{F}}_p}{sup}\left(-h\left(\mathbf{f}\right)+Z_{\mathbf{f}}\right)\right],\hfill \end{array}$$

where the second inequality is due to $\mathbb{E}\left[Z_{{\mathbf{f}}_T^{\star }}\right]=0$ and $\left(\frac{1}{{\eta }_t}-\frac{1}{{\eta }_{t-1}}\right)>0$ for $t=0,1,\dots ,T$ since ${\eta }_t$ is decreasing in t in Theorem 2. In addition, for $y\ge 0$, one has

$$\mathbb{P}\left(-h\left(\mathbf{f}\right)+Z_{\mathbf{f}}>y\right)={\mathrm{e}}^{-h\left(\mathbf{f}\right)-y}.$$

Hence, for any $y\ge 0$

$$\mathbb{P}\left(\underset{\mathbf{f}\in {\mathcal{F}}_p}{sup}\left(-h\left(\mathbf{f}\right)+Z_{\mathbf{f}}\right)>y\right)\le \sum _{\mathbf{f}\in {\mathcal{F}}_p}\mathbb{P}\left(Z_{\mathbf{f}}\ge h\left(\mathbf{f}\right)+y\right)=\sum _{\mathbf{f}\in {\mathcal{F}}_p}{\mathrm{e}}^{-h\left(\mathbf{f}\right)}{\mathrm{e}}^{-y}=u{\mathrm{e}}^{-y},$$

where $u={\sum }_{\mathbf{f}\in {\mathcal{F}}_p}{\mathrm{e}}^{-h\left(\mathbf{f}\right)}$. Therefore, we have

$$\begin{array}{cc}\hfill \mathbb{E}\left[\underset{\mathbf{f}\in {\mathcal{F}}_p}{sup}\left(-h\left(\mathbf{f}\right)+Z_{\mathbf{f}}\right)-lnu\right]& \le \mathbb{E}\left[max\left(0,\underset{\mathbf{f}\in {\mathcal{F}}_p}{sup}\left(-h\left(\mathbf{f}\right)+Z_{\mathbf{f}}-lnu\right)\right)\right]\hfill \\ \hfill & \le {\int }_0^{\infty }\mathbb{P}\left(max\left(0,\underset{\mathbf{f}\in {\mathcal{F}}_p}{sup}\left(-h\left(\mathbf{f}\right)+Z_{\mathbf{f}}-lnu\right)\right)>y\right)\mathrm{d}y\hfill \\ \hfill & \le {\int }_0^{\infty }\mathbb{P}\left(\underset{\mathbf{f}\in {\mathcal{F}}_p}{sup}\left(-h\left(\mathbf{f}\right)+Z_{\mathbf{f}}\right)>y+lnu\right)\mathrm{d}y\hfill \\ \hfill & \le {\int }_0^{\infty }u{\mathrm{e}}^{-(y+lnu)}\mathrm{d}y=1.\hfill \end{array}$$

We thus obtain

$$\mathbb{E}\left[\sum _{t=1}^T\Delta \left({\widehat{\mathbf{f}}}_t^{\star },x_t\right)\right]\le \underset{\mathbf{f}\in {\mathcal{F}}_p}{inf}\left\{\sum _{t=1}^T\Delta (\mathbf{f},x_t)+\frac{1}{{\eta }_T}h\left(\mathbf{f}\right)\right\}+\frac{1}{{\eta }_T}\left(1+ln\sum _{\mathbf{f}\in {\mathcal{F}}_p}{\mathrm{e}}^{-h\left(\mathbf{f}\right)}\right).$$

(6)

Next, we control the regret of Algorithm 2.

Lemma 2.

Assume that $z_{\mathbf{f}}$ is sampled from the symmetric exponential distribution in $\mathbb{R}$, i.e., $\pi \left(z\right)={\mathrm{e}}^{-z}{\mathbb{1}}_{\{z>0\}}$. Assume that ${sup}_{t=1,\dots ,T}{\eta }_{t-1}\le \frac{1}{d{(2R+\delta )}^2}$, and define $c_0=d{(2R+\delta )}^2$. Then for any sequence $\left(x_t\right)\in B(0,\sqrt{d}R)$, $t=1,\dots ,T$,

$$\sum _{t=1}^T\mathbb{E}\left[\Delta \left({\widehat{\mathbf{f}}}_t,x_t\right)\right]\le \sum _{t=1}^T\left(1+{\eta }_{t-1}{c}_0(\mathrm{e}-1)\right)\mathbb{E}\left[\Delta \left({\widehat{\mathbf{f}}}_t^{\star },x_t\right)\right].$$

(7)

Proof. 

Let us denote by

$$F_t\left(Z_{\mathbf{f}}\right)=\Delta \left({\widehat{\mathbf{f}}}_t,x_t\right)=\Delta \left(\underset{\mathbf{f}\in \mathcal{F}}{arginf}\left(\sum _{s=1}^{t-1}\Delta (\mathbf{f},x_s)+\frac{1}{{\eta }_{t-1}}h\left(\mathbf{f}\right)-\frac{1}{{\eta }_{t-1}}{Z}_{\mathbf{f}}\right),x_t\right)$$

the instantaneous loss suffered by the polygonal line ${\widehat{\mathbf{f}}}_t$ when $x_t$ is obtained. We have

$$\begin{array}{cc}\hfill \mathbb{E}[\Delta \left({\widehat{\mathbf{f}}}_t^{\star },x_t\right)]& =\int F_t\left(z-{\eta }_{t-1}\Delta \left(\mathbf{f},x_t\right)\right)\pi \left(z\right)\mathrm{d}z\hfill \\ & =\int F_t\left(z\right)\pi \left(z+{\eta }_{t-1}\Delta (\mathbf{f},x_t)\right)\mathrm{d}z\hfill \\ \hfill & =\int F_t\left(z\right){\mathrm{e}}^{-\left(z+{\eta }_{t-1}\Delta (\mathbf{f},x_t)\right)}\mathrm{d}z\hfill \\ \hfill & \ge {\mathrm{e}}^{-{\eta }_{t-1}d{(2R+\delta )}^2}\int F_t\left(z\right){\mathrm{e}}^{-z}\mathrm{d}z\hfill \\ & ={\mathrm{e}}^{-{\eta }_{t-1}d{(2R+\delta )}^2}\mathbb{E}[\Delta \left({\widehat{\mathbf{f}}}_t,x_t\right)],\hfill \end{array}$$

where the inequality is due to the fact that $\Delta (\mathbf{f},x)\le d{(2R+\delta )}^2$ holds uniformly for any $\mathbf{f}\in {\mathcal{F}}_p$ and $x\in B(0,\sqrt{d}R)$. Finally, summing on t on both sides and using the elementary inequality ${\mathrm{e}}^x\le 1+(\mathrm{e}-1)x$ if $x\in (0,1)$ concludes the proof. □

Lemma 3.

For $k\in ⟦1,p⟧$, we control the cardinality of set $\left\{\mathbf{f}\in {\mathcal{F}}_p,\mathcal{K}\left(\mathbf{f}\right)=k\right\}$ as

$$\begin{array}{cc}\hfill ln\left|\left\{\mathbf{f}\in {\mathcal{F}}_p,\mathcal{K}\left(\mathbf{f}\right)=k\right\}\right|& \le \left(ln\left(8p\mathrm{e}{V}_d\right)+3d^{\frac{3}{2}}-d\right)k+\left(\frac{ln2}{\delta \sqrt{d}}+\frac{d}{\delta }\right)L+dln\left(\frac{\sqrt{d}(2R+\delta )}{\delta }\right)\hfill \\ \hfill & \stackrel{\Delta }{=}{c}_{1}k+c_{2}L+c_3,\hfill \end{array}$$

where $V_d$ denotes the volume of the unit ball in ${\mathbb{R}}^d$.

Proof. 

First, let $N_{k,\delta }$ denote the set of polygonal lines with k segments and whose vertices are in ${\mathcal{Q}}_{\delta }$. Notice that $N_{k,\delta }$ is different from $\{\mathbf{f}\in {\mathcal{F}}_p,\mathcal{K}\left(\mathbf{f}\right)=k\}$ and that

$$\left|\{\mathbf{f}\in {\mathcal{F}}_p,\mathcal{K}\left(\mathbf{f}\right)=k\}\right|\le \left(\genfrac{}{}{0pt}{}{p}{k}\right)\left|N_{k,\delta }\right|.$$

Hence

$$\begin{array}{cc}\hfill ln\left|\{\mathbf{f}\in {\mathcal{F}}_p,\mathcal{K}\left(\mathbf{f}\right)=k\}\right|& \le ln\left(\genfrac{}{}{0pt}{}{p}{k}\right)+ln\left|N_{k,\delta }\right|\hfill \\ \hfill & \le kln\frac{p\mathrm{e}}{k}+k\left(ln8V_d+3d^{\frac{3}{2}}-d\right)+\left(\frac{ln2}{\sqrt{d}\delta }+\frac{d}{\delta }\right)L+dln\left(\frac{\sqrt{d}(2R+\delta )}{\delta }\right)\hfill \\ \hfill & \le kln\left(p\mathrm{e}\right)+k\left(ln8V_d+3d^{\frac{3}{2}}-d\right)+\left(\frac{ln2}{\sqrt{d}\delta }+\frac{d}{\delta }\right)L+dln\left(\frac{\sqrt{d}(2R+\delta )}{\delta }\right),\hfill \end{array}$$

where the second inequality is a consequence to the elementary inequality $\left(\genfrac{}{}{0pt}{}{p}{k}\right)\le {\left(\frac{p\mathrm{e}}{k}\right)}^k$ combined with Lemma 2 in [16]. □

We now have all the ingredients to prove Theorem 1 and Theorem 2.

First, combining (6) and (7) yields that

$$\begin{array}{cc}\hfill \sum _{t=1}^T\mathbb{E}\left[\Delta ({\widehat{\mathbf{f}}}_t,x_t)\right]& \le \underset{\mathbf{f}\in {\mathcal{F}}_p}{inf}\left\{\sum _{t=1}^T\Delta (\mathbf{f},x_t)+\frac{1}{{\eta }_T}h\left(\mathbf{f}\right)\right\}+\frac{1}{{\eta }_T}\left(\frac{1}{2}+ln\sum _{\mathbf{f}\in {\mathcal{F}}_p}{\mathrm{e}}^{-h\left(\mathbf{f}\right)}\right)\hfill \\ & +c_0(\mathrm{e}-1)\sum _{t=1}^T{\eta }_{t-1}\mathbb{E}\left[\Delta ({\widehat{\mathbf{f}}}_t^{\star },x_t)\right]\hfill \\ \hfill & \le \underset{k\in ⟦1,p⟧}{inf}\left\{\underset{\begin{array}{c}\mathbf{f}\in {\mathcal{F}}_p\\ \mathcal{K}\left(\mathbf{f}\right)=k\end{array}}{inf}\left\{\sum _{t=1}^T\Delta (\mathbf{f},x_t)+\frac{h\left(\mathbf{f}\right)}{{\eta }_T}\right\}\right\}+\frac{1}{{\eta }_T}\left(\frac{1}{2}+ln\sum _{\mathbf{f}\in {\mathcal{F}}_p}{\mathrm{e}}^{-h\left(\mathbf{f}\right)}\right)\hfill \\ & +c_0(\mathrm{e}-1)\sum _{t=1}^T{\eta }_{t-1}\mathbb{E}\left[\Delta ({\widehat{\mathbf{f}}}_t^{\star },x_t)\right].\hfill \end{array}$$

Assume that ${\eta }_t=\eta$, $t=0,\dots ,T$ and $h\left(\mathbf{f}\right)=c_1\mathcal{K}\left(\mathbf{f}\right)+c_{2}L+c_3$ for $\mathbf{f}\in {\mathcal{F}}_p$, then $(\frac{1}{2}+{\sum }_{\mathbf{f}\in {\mathcal{F}}_p}{\mathrm{e}}^{-h\left(\mathbf{f}\right)})\le 0$ and moreover

$$\begin{array}{cc}\hfill \sum _{t=1}^T\mathbb{E}\left[\Delta ({\widehat{\mathbf{f}}}_t,x_t)\right]& \le S_{T,h,\eta }+\frac{1}{\eta }\left(\frac{1}{2}+ln\sum _{\mathbf{f}\in {\mathcal{F}}_p}{\mathrm{e}}^{-h\left(\mathbf{f}\right)}\right)+c_0(\mathrm{e}-1)\eta \sum _{t=1}^T\mathbb{E}\left[\Delta ({\widehat{\mathbf{f}}}_t^{\star },x_t)\right]\hfill \\ \hfill & \le S_{T,h,\eta }+c_0(\mathrm{e}-1)\eta S_{T,h,\eta }\hfill \\ \hfill & \le S_{T,h,\eta }+\eta c_0(\mathrm{e}-1)\underset{\mathbf{f}\in {\mathcal{F}}_p}{inf}\sum _{t=1}^T\Delta (\mathbf{f},x_t)+c_0(\mathrm{e}-1)(c_{1}p+c_{2}L+c_3),\hfill \end{array}$$

where

$$S_{T,h,\eta }=\underset{k\in ⟦1,p⟧}{inf}\left\{\underset{\begin{array}{c}\mathbf{f}\in {\mathcal{F}}_p\\ \mathcal{K}\left(\mathbf{f}\right)=k\end{array}}{inf}\left\{\sum _{t=1}^T\Delta (\mathbf{f},x_t)+\frac{h\left(\mathbf{f}\right)}{\eta }\right\}\right\}$$

and the second inequality is obtained with Lemma 1. By setting

$$\eta =\sqrt{\frac{c_{1}p+c_{2}L+c_3}{c_0(\mathrm{e}-1){inf}_{\mathbf{f}\in {\mathcal{F}}_p}{\sum }_{t=1}^T\Delta (\mathbf{f},x_t)}}$$

we obtain

$$\begin{array}{c}\sum _{t=1}^T\mathbb{E}\left[\Delta ({\widehat{\mathbf{f}}}_t,x_t)\right]\le \underset{k\in ⟦1,p⟧}{inf}\left\{\underset{\begin{array}{c}\mathbf{f}\in {\mathcal{F}}_p\\ \mathcal{K}\left(\mathbf{f}\right)=k\end{array}}{inf}\left\{\sum _{t=1}^T\Delta (\mathbf{f},x_t)+\sqrt{c_0(\mathrm{e}-1)r_{T,k,L}}\right\}\right\}\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.17em}+\sqrt{c_0(\mathrm{e}-1)L_{T,p,L}}+c_0(\mathrm{e}-1)c_{1}p+c_{2}L+c_3,\hfill \end{array}$$

where $r_{T,k,L}={inf}_{\mathbf{f}\in {\mathcal{F}}_p}{\sum }_{t=1}^T\Delta (\mathbf{f},x_t)(c_{1}k+c_{2}L+c_3)$. This proves Theorem 1.

Finally, assume that

$${\eta }_0=\frac{\sqrt{c_{1}p+c_{2}L+c_3}}{c_0\sqrt{(\mathrm{e}-1)}}\mathrm{and}{\eta }_t=\frac{\sqrt{c_{1}p+c_{2}L+c_3}}{c_0\sqrt{(\mathrm{e}-1)t}},t=1,\dots ,T.$$

Since $\mathbb{E}\left[\Delta ({\widehat{\mathbf{f}}}_t^{\star },x_t)\right]\le c_0$ for any $t=1,\dots ,T$, we have

$$\begin{array}{cc}\hfill \sum _{t=1}^T\mathbb{E}\left[\Delta ({\widehat{\mathbf{f}}}_t,x_t)\right]& \le \underset{k\in ⟦1,p⟧}{inf}\left\{\underset{\begin{array}{c}\mathbf{f}\in {\mathcal{F}}_p\\ \mathcal{K}\left(\mathbf{f}\right)=k\end{array}}{inf}\left\{\sum _{t=1}^T\Delta (\mathbf{f},x_t)+\frac{h\left(\mathbf{f}\right)}{{\eta }_T}\right\}\right\}+\frac{1}{{\eta }_T}\left(1+ln\sum _{\mathbf{f}\in {\mathcal{F}}_p}{\mathrm{e}}^{-h\left(\mathbf{f}\right)}\right)\hfill \\ & +c_0^2(\mathrm{e}-1)\sum _{t=1}^T{\eta }_{t-1}\hfill \\ \hfill & \le \underset{k\in ⟦1,p⟧}{inf}\left\{\underset{\begin{array}{c}\mathbf{f}\in {\mathcal{F}}_p\\ \mathcal{K}\left(\mathbf{f}\right)=k\end{array}}{inf}\left\{\sum _{t=1}^T\Delta (\mathbf{f},x_t)+c_0\sqrt{(\mathrm{e}-1)T(c_{0}k+c_{2}L+c_3)}\right\}\right\}\hfill \\ & +2c_0\sqrt{(\mathrm{e}-1)T(c_{0}p+c_{2}L+c_3)},\hfill \end{array}$$

which concludes the proof of Theorem 2.

Lemma 4.

Using Algorithm 3, if $0<ϵ\le 1$, $0<\beta <1$, $\alpha \ge \frac{(1-\beta )c_0}{\beta }$ and $\left|\mathcal{U}\left({\widehat{\mathbf{f}}}_{t-1}\right)\right|\ge 2$ for all $t\ge 2$, where $\left|\mathcal{U}\left({\widehat{\mathbf{f}}}_{t-1}\right)\right|$ is the cardinality of $\mathcal{U}\left({\widehat{\mathbf{f}}}_{t-1}\right)$, then we have

$$\sum _{t=1}^T\mathbb{E}\left[r_{{\widehat{\mathbf{f}}}_t,t}\right]\ge \sum _{t=1}^T\mathbb{E}\left[{\widehat{r}}_{{\widehat{\sigma }}^t\left({\mathcal{A}}_t\right),t}\right]-2(1-ϵ)\alpha \beta \sum _{t=1}^T\left|\mathcal{U}\left({\widehat{\mathbf{f}}}_{t-1}\right)\right|.$$

Proof. 

First notice that ${\mathcal{A}}_t=\mathcal{U}\left({\widehat{\mathbf{f}}}_{t-1}\right)$ if $I_t=0$, and that for $t\ge 2$

$$\begin{array}{cc}\hfill \mathbb{E}\left[r_{{\widehat{\mathbf{f}}}_t,t}|{\mathcal{H}}_t,I_t=0\right]=& \mathbb{E}\left[r_{{\widehat{\sigma }}^t\left({\mathcal{A}}_t\right),t}|{\mathcal{H}}_t,I_t=0\right]\hfill \\ \hfill =& \sum _{\mathbf{f}\in {\mathcal{A}}_t\cap cond\left(t\right)}{r}_{\mathbf{f},t}\mathbb{P}\left({\widehat{\sigma }}^t\left({\mathcal{A}}_t\right)=\mathbf{f}|{\mathcal{H}}_t\right)+\sum _{\mathbf{f}\in {\mathcal{A}}_t\cap {cond\left(t\right)}^c}{r}_{\mathbf{f},t}\mathbb{P}\left({\widehat{\sigma }}^t\left({\mathcal{A}}_t\right)=\mathbf{f}|{\mathcal{H}}_t\right)\hfill \\ \hfill \ge & \sum _{\mathbf{f}\in {\mathcal{A}}_t\cap cond\left(t\right)}{r}_{\mathbf{f},t}+\sum _{\mathbf{f}\in {\mathcal{A}}_t\cap {cond\left(t\right)}^c}\alpha \mathbb{P}\left({\widehat{\sigma }}^t\left({\mathcal{A}}_t\right)=\mathbf{f}|{\mathcal{H}}_t\right)\hfill \\ & -(1-\beta )\sum _{\mathbf{f}\in {\mathcal{A}}_t\cap cond\left(t\right)}{r}_{\mathbf{f},t}-\sum _{\mathbf{f}\in {\mathcal{A}}_t\cap {cond\left(t\right)}^c}\left(\alpha -r_{\mathbf{f},t}\right)\mathbb{P}\left({\widehat{\sigma }}^t\left({\mathcal{A}}_t\right)=\mathbf{f}|{\mathcal{H}}_t\right)\hfill \\ \hfill =& \mathbb{E}\left[{\widehat{r}}_{{\widehat{\sigma }}^t\left({\mathcal{A}}_t\right),t}|{\mathcal{H}}_t,I_t=0\right]-(1-\beta )\sum _{\mathbf{f}\in {\mathcal{A}}_t\cap cond\left(t\right)}{r}_{\mathbf{f},t}\hfill \\ & -\sum _{\mathbf{f}\in {\mathcal{A}}_t\cap {cond\left(t\right)}^c}\left(\alpha -r_{\mathbf{f},t}\right)\mathbb{P}\left({\widehat{\sigma }}^t\left({\mathcal{A}}_t\right)=\mathbf{f}|{\mathcal{H}}_t\right)\hfill \\ \hfill \ge & \mathbb{E}\left[{\widehat{r}}_{{\widehat{\sigma }}^t\left({\mathcal{A}}_t\right),t}|{\mathcal{H}}_t,I_t=0\right]-(1-\beta )c_0\left|{\mathcal{A}}_t\right|-\alpha \beta \left|{\mathcal{A}}_t\right|\hfill \\ \hfill \ge & \mathbb{E}\left[{\widehat{r}}_{{\widehat{\sigma }}^t\left({\mathcal{A}}_t\right),t}|{\mathcal{H}}_t,I_t=0\right]-2\alpha \beta \left|{\mathcal{A}}_t\right|,\hfill \end{array}$$

where $cond{\left(t\right)}^c$ denotes the complement of set $cond\left(t\right)$. The first inequality above is due to the assumption that for all $\mathbf{f}\in {\mathcal{A}}_t\cap cond\left(t\right)$, we have $\mathbb{P}\left({\widehat{\sigma }}^t\left({\mathcal{A}}_t\right)=\mathbf{f}|{\mathcal{H}}_t\right)\ge \beta$. For $t=1$, the above inequality is trivial since ${\widehat{r}}_{{\widehat{\sigma }}^1\left(\mathcal{U}\left({\widehat{\mathbf{f}}}_0\right)\right),1}\equiv 0$ by its definition. Hence, for $t\ge 1$, one has

$$\begin{array}{cc}\hfill \mathbb{E}\left[r_{{\widehat{\mathbf{f}}}_t,t}|{\mathcal{H}}_t\right]& =ϵ\mathbb{E}\left[r_{{\widehat{\sigma }}^t\left({\mathcal{F}}_p\right),t}|{\mathcal{H}}_t,I_t=1\right]+(1-ϵ)\mathbb{E}\left[r_{{\widehat{\sigma }}^t\left({\mathcal{A}}_t\right),t}|{\mathcal{H}}_t,I_t=0\right]\hfill \\ \hfill & \ge \mathbb{E}\left[{\widehat{r}}_{{\widehat{\mathbf{f}}}_t,t}|{\mathcal{H}}_t\right]-2\alpha \beta \left|{\mathcal{A}}_t\right|.\hfill \end{array}$$

(8)

Summing on both sides of inequality (8) over t terminates the proof of Lemma 4. □

Lemma 5.

Let ${\widehat{c}}_0=\frac{c_0}{\beta }+\alpha$. If $0<{\eta }_1={\eta }_2=\dots ={\eta }_T=\eta <\frac{1}{{\widehat{c}}_0}$, then we have

$$\begin{array}{c}\mathbb{E}\left[\underset{\widehat{\sigma }}{max}\left\{\sum _{t=1}^T{\widehat{r}}_{\widehat{\sigma }\left({\mathcal{A}}_t\right),t}-\frac{1}{\eta }h\left(\widehat{\sigma }\left({\mathcal{A}}_t\right)\right)\right\}\right]-\sum _{t=1}^T\mathbb{E}\left[{\widehat{r}}_{{\widehat{\sigma }}^t\left({\mathcal{A}}_t\right),t}\right]\le \hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.17em}{\widehat{c}}_0^2(e-1)\eta T+{\widehat{c}}_0(e-1)\left(c_{1}p+c_{2}L+c_3\right).\hfill \end{array}$$

Proof. 

By the definition of ${\widehat{r}}_{\mathbf{f},t}$ in Algorithm 3, for any $\mathbf{f}\in {\mathcal{F}}_p$ and $t\ge 1$, we have

$${\widehat{r}}_{\mathbf{f},t}\le max\left\{\frac{r_{\mathbf{f},t}}{\mathbb{P}\left({\widehat{\mathbf{f}}}_t=\mathbf{f}|{\mathcal{H}}_t\right)},\alpha ,r_{\mathbf{f},t}\right\}\le max\left\{\frac{c_0}{\beta },\alpha \right\}\le {\widehat{c}}_0,$$

where in the second inequality we use that $r_{\mathbf{f},t}\le c_0$ for all $\mathbf{f}$ and t, and that $\mathbb{P}\left({\widehat{\mathbf{f}}}_t=\mathbf{f}|{\mathcal{H}}_t\right)\ge \beta$ when $\mathbf{f}\in \mathcal{U}\left({\widehat{\mathbf{f}}}_{t-1}\right)\cap cond\left(t\right)$. The rest of the proof is similar to those of Lemmas 1 and 2. In fact, if we define by $\widehat{\Delta }\left(\mathbf{f},x_t\right)={\widehat{c}}_0-{\widehat{r}}_{\mathbf{f},t}$, then one can easily observe the following relation when $I_t=1$ (similar relation in the case that $I_t$ = 0)

$$\begin{array}{cc}\hfill {\widehat{\mathbf{f}}}_t={\widehat{\sigma }}^t\left({\mathcal{F}}_p\right)& =\underset{\mathbf{f}\in {\mathcal{F}}_p}{arg\; max}\left\{\sum _{s=1}^{t-1}{\widehat{r}}_{\mathbf{f},s}+\frac{1}{\eta }\left(z_{\mathbf{f}}-h\left(\mathbf{f}\right)\right)\right\}\hfill \\ \hfill & =\underset{\mathbf{f}\in {\mathcal{F}}_p}{arg\; min}\left\{\sum _{s=1}^{t-1}\widehat{\Delta }(\mathbf{f},x_s)+\frac{1}{\eta }\left(h\left(\mathbf{f}\right)-z_{\mathbf{f}}\right)\right\}.\hfill \end{array}$$

Then applying Lemmas 1 and 2 on this newly defined sequence $\widehat{\Delta }\left({\widehat{\mathbf{f}}}_t,x_t\right),t=1,\dots T$ leads to the result of Lemma 5. □

The proof of the upcoming Lemma 6 requires the following submartingale inequality: let $Y_0,\dots Y_T$ be a sequence of random variable adapted to random events ${\mathcal{H}}_0,\dots ,{\mathcal{H}}_T$ such that for $1\le t\le T$, the following three conditions hold:

$$\mathbb{E}\left[Y_t|H_t\right]\le 0,\mathrm{Var}\left(Y_t\right|H_t)\le a^2,Y_t-\mathbb{E}\left[Y_t|H_t\right]\le b.$$

Then for any $\lambda >0$,

$$\mathbb{P}\left(\sum _{t=1}^T{Y}_t>Y_0+\lambda \right)\le exp\left(-\frac{{\lambda }^2}{2T(a^2+b^2)}\right).$$

The proof can be found in Chung and Lu [43] (Theorem 7.3).

Lemma 6.

Assume that $0<\beta <\frac{1}{\left|{\mathcal{F}}_p\right|},\alpha \ge \frac{c_0}{\beta }$ and $\eta >0$, then we have

$$\begin{array}{c}\mathbb{E}\left[\underset{\sigma }{max}\left\{\sum _{t=1}^T{r}_{\sigma \left({\mathcal{A}}_t\right),t}-\frac{1}{\eta }h\left(\sigma \left({\mathcal{A}}_t\right)\right)\right\}\right]-\mathbb{E}\left[\underset{\widehat{\sigma }}{max}\left\{\sum _{t=1}^T{\widehat{r}}_{\widehat{\sigma }\left({\mathcal{A}}_t\right),t}-\frac{1}{\eta }h\left(\widehat{\sigma }\left({\mathcal{A}}_t\right)\right)\right\}\right]\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\le \left(1-\left|{\mathcal{F}}_p\right|\beta \right)\sqrt{2T\left[\frac{c_0^2}{\beta }+{\alpha }^2(1-\beta )+{\left(c_0+2\alpha \right)}^2\right]ln\left(\frac{1}{\beta }\right)}+\left|{\mathcal{F}}_p\right|\beta c_{0}T.\hfill \end{array}$$

Proof. 

First, we have almost surely that

$$\begin{array}{cc}\hfill \underset{\sigma }{max}\left\{\sum _{t=1}^T{r}_{\sigma \left({\mathcal{A}}_t\right),t}-\frac{1}{\eta }h\left(\sigma \left({\mathcal{A}}_t\right)\right)\right\}-\underset{\widehat{\sigma }}{max}\left\{\sum _{t=1}^T{\widehat{r}}_{\widehat{\sigma }\left({\mathcal{A}}_t\right),t}-\frac{1}{\eta }h\left(\widehat{\sigma }\left({\mathcal{A}}_t\right)\right)\right\}& \le \underset{\mathbf{f}\in {\mathcal{F}}_p}{max}\sum _{t=1}^T\left(r_{\mathbf{f},t}-{\widehat{r}}_{\mathbf{f},t}\right).\hfill \end{array}$$

Denote by $Y_{\mathbf{f},t}=r_{\mathbf{f},t}-{\widehat{r}}_{\mathbf{f},t}$. Since

$$\mathbb{E}\left[{\widehat{r}}_{\mathbf{f},t}|{\mathcal{H}}_t\right]=\left\{\begin{array}{cc}{r}_{\mathbf{f},t}+(1-ϵ)\alpha \left(1-\mathbb{P}\left({\widehat{\mathbf{f}}}_t=\mathbf{f}|{\mathcal{H}}_t\right)\right)\hfill & if\mathbf{f}\in \mathcal{U}\left({\widehat{\mathbf{f}}}_{t-1}\right)\cap cond\left(t\right),\hfill \\ ϵr_{\mathbf{f},t}+(1-ϵ)\alpha \hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

and $\alpha >c_0\ge r_{\mathbf{f},t}$ uniformly for any $\mathbf{f}$ and t, we have uniformly that $\mathbb{E}\left[Y_t|{\mathcal{H}}_t\right]\le 0$, satisfying the first condition.

For the second condition, if $\mathbf{f}\in \mathcal{U}\left({\widehat{\mathbf{f}}}_{t-1}\right)\cap \hspace{0.17em}cond\left(t\right)$, then

$$\begin{array}{cc}\hfill \mathrm{Var}\left(Y_t\right|{\mathcal{H}}_t)=& \mathbb{E}\left[{\widehat{r}}_{\mathbf{f},t}^2|{\mathcal{H}}_t\right]-{\left(\mathbb{E}\left[{\widehat{r}}_{\mathbf{f},t}|{\mathcal{H}}_t\right]\right)}^2\hfill \\ \hfill \le & ϵr_{\mathbf{f},t}^2+(1-ϵ)\left[\frac{r_{\mathbf{f},t}^2}{\mathbb{P}\left({\widehat{\mathbf{f}}}_t=\mathbf{f}|{\mathcal{H}}_t\right)}+\alpha \left(1-\mathbb{P}\left({\widehat{\mathbf{f}}}_t=\mathbf{f}|{\mathcal{H}}_t\right)\right)\right]\hfill \\ \hfill & -{\left[r_{\mathbf{f},t}+(1-ϵ)\alpha \left(1-\mathbb{P}\left({\widehat{\mathbf{f}}}_t=\mathbf{f}|{\mathcal{H}}_t\right)\right)\right]}^2\hfill \\ \hfill \le & \frac{r_{\mathbf{f},t}^2}{\beta }+{\alpha }^2(1-\beta )\le \frac{c_0^2}{\beta }+{\alpha }^2(1-\beta ).\hfill \end{array}$$

Similarly, for $\mathbf{f}\notin \mathcal{U}\left({\widehat{\mathbf{f}}}_{t-1}\right)\cap cond\left(t\right)$, one can have $\mathrm{Var}\left(Y_t\right|{\mathcal{H}}_t)\le {\alpha }^2$. Moreover, for the third condition, since

$$\mathbb{E}\left[Y_{\mathbf{f},t}|{\mathcal{H}}_t\right]\ge -2\alpha ,$$

then

$$Y_{\mathbf{f},t}-\mathbb{E}\left[Y_{\mathbf{f},t}|{\mathcal{H}}_t\right]\le r_{\mathbf{f},t}+2\alpha \le c_0+2\alpha .$$

Setting $\lambda =\sqrt{2T\left[\frac{c_0^2}{\beta }+{\alpha }^2(1-\beta )+{\left(c_0+2\alpha \right)}^2\right]ln\left(\frac{1}{\beta }\right)}$ leads to

$$\mathbb{P}\left(\sum _{t=1}^T{Y}_{\mathbf{f},t}\ge \lambda \right)\le \beta .$$

Hence the following inequality holds with probability $1-\left|{\mathcal{F}}_p\right|\beta$

$$\underset{\mathbf{f}\in {\mathcal{F}}_p}{max}\sum _{t=1}^T\left(r_{\mathbf{f},t}-{\widehat{r}}_{\mathbf{f},t}\right)\le \sqrt{2T\left[\frac{c_0^2}{\beta }+{\alpha }^2(1-\beta )+{\left(c_0+2\alpha \right)}^2\right]ln\left(\frac{1}{\beta }\right)}.$$

Finally, noticing that ${max}_{\mathbf{f}\in {\mathcal{F}}_p}{\sum }_{t=1}^T\left(r_{\mathbf{f},t}-{\widehat{r}}_{\mathbf{f},t}\right)\le c_{0}T$ almost surely, we terminate the proof of Lemma 6. □

Proof of Theorem 3.

Assume that $p>6$, $T\ge 2|{\mathcal{F}}_p{|}^2$ and let

$$\begin{array}{c}\beta ={\left|{\mathcal{F}}_p\right|}^{-\frac{1}{2}}{T}^{-\frac{1}{4}},\alpha =\frac{c_0}{\beta },{\widehat{c}}_0=\frac{2c_0}{\beta },\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{\eta }_1={\eta }_2=\dots ={\eta }_T=\frac{\sqrt{c_{1}p+c_{2}L+c_3}}{\sqrt{T(e-1)}{\widehat{c}}_0},ϵ=1-{\left|{\mathcal{F}}_p\right|}^{\frac{1}{2}-\frac{3}{p}}{T}^{-\frac{1}{4}}.\hfill \end{array}$$

With those values, the assumptions of Lemmas 4, 5 and 6 are satisfied. Combining their results lead to the following

$$\begin{array}{cc}\hfill \sum _{t=1}^T\mathbb{E}\left[r_{{\widehat{\mathbf{f}}}_t,t}\right]\ge & \mathbb{E}\left[\underset{\sigma }{max}\left\{\sum _{t=1}^T{r}_{\sigma \left({\mathcal{A}}_t\right),t}-\frac{1}{\eta }h\left(\sigma \left({\mathcal{A}}_t\right)\right)\right\}\right]-2\alpha \beta (1-ϵ)\sum _{t=1}^T\left|\mathcal{U}\left({\widehat{\mathbf{f}}}_{t-1}\right)\right|\hfill \\ \hfill & -{\widehat{c}}_0^2(e-1)\eta T-{\widehat{c}}_0(e-1)\left(c_{1}p+c_{2}L+c_3\right)\hfill \\ \hfill & -\left(1-\left|{\mathcal{F}}_p\right|\beta \right)\sqrt{2T\left[\frac{c_0^2}{\beta }+{\alpha }^2(1-\beta )+{\left(c_0+2\alpha \right)}^2\right]ln\left(\frac{1}{\beta }\right)}-\left|{\mathcal{F}}_p\right|\beta c_{0}T\hfill \\ \hfill \ge & \mathbb{E}\left[\underset{\sigma }{max}\left\{\sum _{t=1}^T{r}_{\sigma \left({\mathcal{A}}_t\right),t}-\frac{1}{\eta }h\left(\sigma \left({\mathcal{A}}_t\right)\right)\right\}\right]-(1-ϵ){\left|{\mathcal{F}}_p\right|}^{\frac{3}{p}}{c}_{0}T\hfill \\ & -{\widehat{c}}_0^2(e-1)\eta T-{\widehat{c}}_0(e-1)\left(c_{1}p+c_{2}L+c_3\right)\hfill \\ \hfill & -\left(1-\left|{\mathcal{F}}_p\right|\beta \right)\sqrt{2T\left[\frac{c_0^2}{\beta }+{\alpha }^2(1-\beta )+{\left(c_0+2\alpha \right)}^2\right]ln\left(\frac{1}{\beta }\right)}-\left|{\mathcal{F}}_p\right|\beta c_{0}T\hfill \\ \hfill \ge & \mathbb{E}\left[\underset{\sigma }{max}\left\{\sum _{t=1}^T{r}_{\sigma \left({\mathcal{A}}_t\right),t}-\frac{1}{\eta }h\left(\sigma \left({\mathcal{A}}_t\right)\right)\right\}\right]-\mathcal{O}\left({\left|{\mathcal{F}}_p\right|}^{\frac{1}{2}}{T}^{\frac{3}{4}}\right),\hfill \end{array}$$

where the second inequality is due to the fact that the cardinality $\left|\mathcal{U}\left({\widehat{\mathbf{f}}}_{t-1}\right)\right|$ is upper bounded by ${\left|{\mathcal{F}}_p\right|}^{\frac{3}{p}}$ for $t\ge 1$. In addition, using the definition of $r_{\mathbf{f},t}$ that $r_{\mathbf{f},t}=c_0-\Delta (\mathbf{f},x_t)$ terminates the proof of Theorem 3. □