Phase Transitions in Transfer Learning for High-Dimensional Perceptrons

Abstract

Transfer learning seeks to improve the generalization performance of a target task by exploiting the knowledge learned from a related source task. Central questions include deciding what information one should transfer and when transfer can be beneficial. The latter question is related to the so-called negative transfer phenomenon, where the transferred source information actually reduces the generalization performance of the target task. This happens when the two tasks are sufficiently dissimilar. In this paper, we present a theoretical analysis of transfer learning by studying a pair of related perceptron learning tasks. Despite the simplicity of our model, it reproduces several key phenomena observed in practice. Specifically, our asymptotic analysis reveals a phase transition from negative transfer to positive transfer as the similarity of the two tasks moves past a well-defined threshold.

1. Introduction

Transfer learning [1,2,3,4,5] is a promising approach to improving the performance of machine learning tasks. It does so by exploiting the knowledge gained from a previously learned model, referred to as the source task, to improve the generalization performance of a related learning problem, referred to as the target task. One particular challenge in transfer learning is to avoid so-called negative transfer [6,7,8,9], where the transferred source information reduces the generalization performance of the target task. Recent literature [6,7,8,9] shows that negative transfer is closely related to the similarity between the source and target tasks. Transfer learning may hurt the generalization performance if the tasks are sufficiently dissimilar.

In this paper, we present a theoretical analysis of transfer learning by studying a pair of related perceptron learning tasks. Despite the simplicity of our model, it reproduces several key phenomena observed in practice. Specifically, the model reveals a sharp phase transition from negative transfer to positive transfer (i.e., when transfer becomes helpful) as a function of the model similarity.

1.1. Models and Learning Formulations

We start by describing the models for our theoretical study. We assume that the source task has a collection of training data ${\left\{({\mathit{a}}_{s,i},y_{s,i})\right\}}_{i=1}^{n_s}$, where ${\mathit{a}}_{s,i}\in {\mathbb{R}}^p$ is the source feature vector and $y_{s,i}\in \mathbb{R}$ denotes the label corresponding to ${\mathit{a}}_{s,i}$. Following the standard teacher–student paradigm, we assume that the labels ${\left\{y_{s,i}\right\}}_{i=1}^{n_s}$ are generated according to the following model:

$$\begin{array}{c}\hfill y_{s,i}=\phi \left({\mathit{a}}_{s,i}^{\top }{\mathit{\xi }}_s\right),\forall i\in \{1,\dots ,n_s\},\end{array}$$

(1)

where $\phi (·)$ is a scalar deterministic or probabilistic function and ${\mathit{\xi }}_s\in {\mathbb{R}}^p$ is an unknown source teacher vector.

Similar to the source task, the target task has access to a different collection of training data ${\left\{({\mathit{a}}_{t,i},y_{t,i})\right\}}_{i=1}^{n_t}$, generated according to

$$\begin{array}{c}\hfill y_{t,i}=\phi \left({\mathit{a}}_{t,i}^{\top }{\mathit{\xi }}_t\right),\forall i\in \{1,\dots ,n_t\},\end{array}$$

(2)

where ${\mathit{\xi }}_t\in {\mathbb{R}}^p$ is an unknown target teacher vector. We measure the similarity of the two tasks using

$$\begin{array}{c}\hfill \rho \stackrel{\mathrm{def}}{=}\frac{{\mathit{\xi }}_t^{\top }{\mathit{\xi }}_s}{\parallel {\mathit{\xi }}_t\parallel \parallel {\mathit{\xi }}_s\parallel },\end{array}$$

(3)

with $\rho =0$ indicating two uncorrelated tasks whereas $\rho =1$ means that the tasks are perfectly aligned.

For the source task, we learn the optimal weight vector ${\widehat{\mathit{w}}}_s$ by solving a convex optimization problem:

$$\begin{array}{c}\hfill {\widehat{\mathit{w}}}_s=\underset{\mathit{w}\in {\mathbb{R}}^p}{argmin}\frac{1}{p}\sum _{i=1}^{n_s}\ell \left(y_{s,i};{\mathit{a}}_{s,i}^{\top }\mathit{w}\right)+\frac{\lambda }{2}{\parallel \mathit{w}\parallel }^2,\end{array}$$

(4)

where $\lambda \ge 0$ is a regularization parameter and $\ell (.;.)$ denotes some general loss function that can take one of the following two forms:

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}\ell \left(y;x\right)=\widehat{\ell }(y-x),\hfill & \mathrm{for}\mathrm{regression}\mathrm{task}\hfill \\ \ell \left(y;x\right)=\widehat{\ell }\left(yx\right),\hfill & \mathrm{for}\mathrm{classification}\mathrm{task},\hfill \end{array}\right.\end{array}$$

(5)

where $\widehat{\ell }(.)$ is a convex function.

In this paper, we consider a common strategy in transfer learning [4], which consists of transferring the optimal source vector, i.e., ${\widehat{\mathit{w}}}_s$, to the target task. One popular approach is to fix a (random) subset of the target weights to values of the corresponding optimal weights learned during the source training process [10]. In our learning model, this amounts to the following target learning formulation:

$$\begin{array}{cc}\hfill {\widehat{\mathit{w}}}_t=& \underset{\mathit{w}\in {\mathbb{R}}^p}{argmin}\frac{1}{p}\sum _{i=1}^{n_t}\ell \left(y_{t,i};{\mathit{a}}_{t,i}^{\top }\mathit{w}\right)+\frac{\lambda }{2}{\parallel \mathit{w}\parallel }^2\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill & \mathrm{s}.\mathrm{t}.\mathit{Q}\mathit{w}=\mathit{Q}{\widehat{\mathit{w}}}_s.\hfill \end{array}$$

(7)

The vector ${\widehat{\mathit{w}}}_s$ is the optimal solution of the source learning problem, and $\mathit{Q}\in {\mathbb{R}}^{p\times p}$ is a diagonal matrix with diagonal entries drawn independently from a Bernoulli distribution with probability $\delta =m/p\le 1$. Here, m denotes the number of transferred components. Thus, on average, we retain $\delta p$ number of entries from the source optimal vector ${\widehat{\mathit{w}}}_s$. In addition to a possible improvement in the generalization performance, this approach can considerably lower the computational complexity of the target learning task by reducing the number of free optimization variables. In what follows, we refer to $\delta$ as the transfer rate and call (6) the hard transfer formulation.

Another popular approach in transfer learning is to search for target weight vectors in the vicinity of the optimal source weight vector ${\widehat{\mathit{w}}}_s$. This can be achieved by adding a regularization term to the target formulation [11,12], which in our model becomes

$${\widehat{\mathit{w}}}_t=\underset{\mathit{w}\in {\mathbb{R}}^p}{argmin}\frac{1}{p}\sum _{i=1}^{n_t}\ell \left(y_{t,i};{\mathit{a}}_{t,i}^{\top }\mathit{w}\right)+\frac{\lambda }{2}{\parallel \mathit{w}\parallel }^2+\frac{1}{2}{\parallel \mathsf{\Sigma }(\mathit{w}-{\widehat{\mathit{w}}}_s)\parallel }^2,$$

(8)

with $\mathsf{\Sigma }\in {\mathbb{R}}^{p\times p}$ denoting some weighting matrix. In what follows, we refer to (8) as the soft transfer formulation, since it relaxes the strict equality in (6). In fact, the hard transfer in (6) is just a special case of the soft transfer formulation, if we set $\mathsf{\Sigma }$ to be a diagonal matrix in which the diagonal entries are either $+\infty$ (with probability $\delta$) or 0 (with probability $1-\delta$).

To measure the performance of the transfer learning methods, we use the generalization error of the target task. Given a new data sample $({\mathit{a}}_{t,\mathrm{new}},y_{t,\mathrm{new}})$ with $y_{t,\mathrm{new}}=\phi \left({\mathit{\xi }}_t^{\top }{\mathit{a}}_{t,\mathrm{new}}\right)$, we assume that the target task predicts the corresponding label as

$${\widehat{y}}_{t,\mathrm{new}}=\widehat{\phi }\left[{\widehat{\mathit{w}}}_t^{\top }{\mathit{a}}_{t,\mathrm{new}}\right],$$

(9)

where $\widehat{\phi }(·)$ is a predefined scalar function that might be different from $\phi (·)$. We then calculate the generalization error of the target task as

$$\begin{array}{c}\hfill {\mathcal{E}}_{\mathrm{test}}=\frac{1}{4^{\upsilon }}\mathbb{E}\left[{\left(y_{t,\mathrm{new}}-\widehat{\phi }\left({\widehat{\mathit{w}}}_t^{\top }{\mathit{a}}_{t,\mathrm{new}}\right)\right)}^2\right],\end{array}$$

(10)

where the expectation is taken with respect to the new data $({\mathit{a}}_{t,\mathrm{new}},y_{t,\mathrm{new}})$. The variable $\upsilon$ allows us to write a more compact formula: $\upsilon$ is taken to be 0 for a regression problem and $\upsilon =1$ for a binary classification problem. Finally, we use the training error

$$\begin{array}{c}\hfill {\mathcal{E}}_{\mathrm{train}}=\frac{1}{p}\sum _{i=1}^{n_t}\ell \left(y_{t,i};{\mathit{a}}_{t,i}^{\top }{\widehat{\mathit{w}}}_t\right)+\frac{1}{2}{\parallel \mathsf{\Sigma }({\widehat{\mathit{w}}}_t-{\widehat{\mathit{w}}}_s)\parallel }^2,\end{array}$$

to quantify the performance of the training process. Here, we measure the training error on the training data without regularization.

1.2. Main Contributions

The main contributions of this paper are two-fold, as summarized below:

1.2.1. Precise Asymptotic Analysis

We present a precise asymptotic analysis of the transfer learning approaches introduced in (6) and (8) for Gaussian feature vectors and under regularity conditions on the eigenvalue distribution of the weighting matrix $\mathsf{\Sigma }$. Specifically, we show that, as the dimensions $p,n_s,n_t$ grow to infinity with the ratios ${\alpha }_s=n_s/p,{\alpha }_t=n_t/p$ fixed, the generalization errors of the hard and soft formulations can be exactly characterized by the solutions of two low-dimensional deterministic optimization problems. (See Theorem 1 and Corollary 1 for details.) Our asymptotic predictions hold for any convex loss functions used in the training process, including the squared loss for regression problems and logistic loss commonly used for binary classification problems.

As illustrated in Figure 1, our theoretical predictions (drawn as solid lines in the figures) reach excellent agreement with the actual performance (shown as circles) of the transfer learning problem. Figure 1a considers a binary classification setting with logistic loss, and we plot the generalization errors of different transfer approaches as a function of the target data/dimension ratio ${\alpha }_t=n_t/p$. We can see that the hard transfer formulation (6) is only useful when ${\alpha }_t$ is small. In fact, we encounter negative transfer (i.e., hard transfer performing worse than no transfer) when ${\alpha }_t$ becomes sufficiently large. Moreover, the soft transfer formulation (8) seems to achieve more favorable generalization errors compared to the hard formulation. In Figure 1b, we consider a regression setting with a squared loss and explore the impact of different weighting schemes on the performance of the soft formulation. We can see that the soft formulation indeed considerably improves the generalization performance of the standard learning method (i.e., learning the target task without any knowledge transfer).

Figure 1. Theoretical predictions v.s. numerical simulations obtained by averaging over 100 independent Monte Carlo trials with dimension $p=2500$. (a) Binary classification with logistic loss. We take ${\alpha }_s=10{\alpha }_t$, $\lambda =0.3$, $\mathsf{\Sigma }={\mathit{I}}_p/\sqrt{5}$, and $\rho =0.85$, where ${\alpha }_s=n_s/p$ and ${\alpha }_t=n_t/p$. The functions $\phi (·)$ and $\widehat{\phi }(·)$ are both the sign function. For hard transfer, we set the transfer rate to be $\delta =0.5$. Full source transfer corresponds to $\delta =1.0$, whereas no transfer corresponds to $\delta =0$. (b) Nonlinear regression using quadratic loss, where $\phi (·)$ is the ReLu function and $\widehat{\phi }(·)$ is the identity function. Soft identity, beta, and uniform matrices refer to different choices of the weighting matrix in (8). Soft Identity Matrix: $\mathsf{\Sigma }$ is an identity matrix. Soft Uniform Matrix: $\mathsf{\Sigma }$ is a random matrix with diagonal elements drawn from the uniform distribution. Soft Beta Matrix: $\mathsf{\Sigma }$ is a random matrix with diagonal elements drawn from the beta distribution. We scale all diagonal elements of $\mathsf{\Sigma }$ to have the same mean. We also take ${\alpha }_s=10{\alpha }_t$, $\lambda =0.1$, and $\rho =0.8$.

1.2.2. Phase Transitions

Our asymptotic characterizations reveal a phase transition phenomenon in the hard transfer formulation. Let

$${\delta }^{★}=\underset{0\le \delta \le 1}{argmin}{\mathcal{E}}_{\mathrm{test}}\left(\delta \right),$$

be the optimal transfer rate that minimizes the generalization error of the target task. Clearly, ${\delta }^{★}=0$ corresponds to the negative transfer regime, where transferring the knowledge of the source task will actually hurt the performance of the target task. In contract, ${\delta }^{★}>0$ signifies that we have entered the positive transfer regime, where transfer becomes helpful.

Figure 2a illustrates the phase transition from negative to positive transfer regimes in a binary classification setting, as the similarity $\rho$ between the two tasks moves past a critical threshold. Similar phase transition phenomena also appear in nonlinear regression, as shown in Figure 2b. Interestingly, for this setting, the optimal transfer rate jumps from ${\delta }^{★}=0$ to ${\delta }^{★}=1$ at the transition threshold.

Figure 2. Phase transitions of the hard transfer formulation. When the similarity $\rho$ between the two tasks is small, we are in the negative transfer regime, where we should not transfer the knowledge from the source task. However, as $\rho$ moves past a critical threshold, we enter the positive transfer regime. (a) Binary classification with squared loss, with parameters ${\alpha }_t=2$, ${\alpha }_s=2{\alpha }_t$, and $\lambda =0$. Both $\phi (·)$ and $\widehat{\phi }(·)$ are the sign function. (b) Nonlinear regression with squared loss, with parameters ${\alpha }_t=2$, ${\alpha }_s=2{\alpha }_t$, and $\lambda =0$. $\phi (.)$ is the ReLu function and $\widehat{\phi }(.)$ is the identity function.

For general loss functions, the exact locations of the phase transitions can only be found numerically by solving the deterministic optimization problems in our asymptotic characterizations. For the special case of squared loss with no regularization, however, we are able to obtain the following simple analytical characterization for the phase transition threshold: We are in the positive transfer regime if and only if

$$\rho >{\rho }_c({\alpha }_s,{\alpha }_t)=1-\frac{\mathbb{E}\left[{\phi }^2\left(z\right)\right]-{\mathbb{E}}^2\left[z\phi \left(z\right)\right]}{2{\mathbb{E}}^2\left[z\phi \left(z\right)\right]}\left(\frac{1}{{\alpha }_t-1}-\frac{1}{{\alpha }_s-1}\right),$$

(11)

where z is a standard Gaussian random variable. This result is shown in Proposition 1.

By the Cauchy–Schwarz inequality, $\mathbb{E}\left[{\phi }^2\left(z\right)\right]\ge {\mathbb{E}}^2\left[z\phi \left(z\right)\right]$. It follows that ${\rho }_c({\alpha }_s,{\alpha }_t)$ is an increasing function of ${\alpha }_t$ and a decreasing function of ${\alpha }_s$. This property is consistent with our intuition: As we increase ${\alpha }_t$, the target task has more training data to work with, and thus, we should set a higher bar in terms of when to transfer knowledge. As we increase ${\alpha }_s$, the quality of the optimal source vector becomes better, in which case, we can start the transfer at a lower similarity level. In particular, when ${\alpha }_t>{\alpha }_s$, we have ${\rho }_c({\alpha }_s,{\alpha }_t)>1$ and, thus, the inequality in (11) is never satisfied (because $\left|\rho \right|\le 1$ by definition). This indicates that no transfer should be done when the target task has more training data than the source task.

1.3. Related Work

The idea of transferring informaton between different domains or different tasks was first proposed in [1] and further developed in [2]. It has been attracting significant interest in recent literature [4,5,6,7,8,9,11,12]. While most work focuses on the practical aspects of transfer learning, there have been several studies (e.g., [13,14]) that seek to provide analytical understandings of transfer learning in simplified models. Our work is particularly related to [14], which considers a transfer learning model similar to ours but for the special case of linear regression. The analysis in this paper is more general as it considers arbitrary convex loss functions. We would also like to mention an interesting recent work that studies a different but related setting referred to as knowledge distillation [15].

In term of technical tools, our asymptotic predictions are derived using the convex Gaussian min–max theorem (CGMT). The CGMT was first introduced in [16] and further developed in [17]. It extends a Gaussian comparison inequality first introduced in [18]. It particularly uses convexity properties to show the equivalence between two Gaussian processes. The CGMT has been successfully used to analyze convex regression formulations [17,19,20] and convex classification formulations [21,22,23,24].

1.4. Organization

The rest of this paper is organized as follows. Section 2 states the technical assumptions under which our results are obtained. Section 3 provides an asymptotic characterization of the soft transfer formulation. Precise analysis of the hard transfer formulation is presented in Section 4. We provide remarks about our approach in Section 5. Our theoretical predictions hold for general convex loss functions. We specialize these results to the settings of nonlinear regression and binary classification in Section 6, where we also provide additional numerical results to validate our predictions. Section 7 provides detailed proof of the technical statements introduced in Section 3 and Section 4. Section 8 concludes the paper. The Appendix provides additional technical details.

2. Technical Assumptions

The theoretical analysis of this paper is carried out under the following assumptions.

Assumption 1

(Gaussian Feature Vectors). The feature vectors ${\left\{{\mathit{a}}_{s,i}\right\}}_{i=1}^{n_s}$ and ${\left\{{\mathit{a}}_{t,i}\right\}}_{i=1}^{n_t}$ are drawn independently from a standard Gaussian distribution. The vector ${\mathit{\xi }}_s\in {\mathbb{R}}^p$ can be expressed as ${\mathit{\xi }}_s=\rho {\mathit{\xi }}_t+\sqrt{1-{\rho }^2}{\mathit{\xi }}_r$, where the vectors ${\mathit{\xi }}_t\in {\mathbb{R}}^p$ and ${\mathit{\xi }}_r\in {\mathbb{R}}^p$ are independent from the feature vectors, and they are generated independently from a uniform distribution on the unit sphere.

Moreover, our results are valid in a high-dimensional asymptotic setting, where the dimensions p, $n_s$, $n_t$, and m grow to infinity at fixed ratios.

Assumption 2

(High-dimensional Asymptotic). The number of samples and the number of transferred components in hard transfer satisfy $n_s=n_s\left(p\right)$, $n_t=n_t\left(p\right)$, and $m=m\left(p\right)$, with ${\alpha }_{s,p}=n_s\left(p\right)/p\to {\alpha }_s>0$, ${\alpha }_{t,p}=n_t\left(p\right)/p\to {\alpha }_t>0$, and ${\delta }_p=m\left(p\right)/p\to \delta >0$ as $p\to \infty$.

The CGMT framework makes specific assumptions about the loss function and the feasibility sets. To guarantee these assumptions, this paper considers a family of loss functions that satisfy the following conditions. Note that the assumption is stated for the target task, but we assume that it is also valid for the source task.

Assumption 3

(Loss Function). If $\lambda >0$, the loss function $\ell (y;.)$ defined in (5) is a proper convex function in $\mathbb{R}$. If $\lambda =0$, the loss function $\ell (y;.)$ defined in (5) is a proper strongly convex function in $\mathbb{R}$, where the constant $S>0$ is a strong convexity parameter. In this case, we only consider the case when ${\alpha }_t>1$. Define a random function $\mathcal{L}\left(\mathit{x}\right)={\sum }_{i=1}^{n_t}\ell (y_i;x_i)$, where $y_i\sim \phi \left(z_i\right)$, with $\left\{z_i\right\}$ being a collection of independent standard normal random variables and ∼ denoting equality in distribution. Denote by $\partial \mathcal{L}$ the sub-differential set of $\mathcal{L}\left(\mathit{x}\right)$. Then, for any constant $C>0$, there exists a constant $R>0$ such that

$$\begin{array}{c}\hfill \left\{\begin{array}{c}\mathbb{P}\left(\underset{\parallel \mathit{v}\parallel \le C\sqrt{n_t}}{sup}\underset{\mathit{s}\in \partial \mathcal{L}\left(\mathit{v}\right)}{sup}\parallel \mathit{s}\parallel \le R\sqrt{n_t}\right)\stackrel{p\to \infty }{\to }1.\hfill \\ \mathbb{P}\left(\underset{\parallel \mathit{v}\parallel \le C\sqrt{n_t}}{sup}\left|\mathcal{L}\left(\mathit{v}\right)\right|\le R{n}_t\right)\stackrel{p\to \infty }{\to }1.\hfill \end{array}\right.\end{array}$$

(12)

Furthermore, we consider the following assumption to guarantee that the generalization error defined in (10) concentrates in the large system limit.

Assumption 4

(Regularity Conditions). The data-generating function $\phi (·)$ is independent from the feature vectors. Moreover, the following conditions are satisfied.

• $\phi (·)$ and $\widehat{\phi }(·)$ are continuous almost everywhere in $\mathbb{R}$. For every $h>0$ and $z\sim \mathcal{N}(0,h)$, we have $0<\mathbb{E}\left[{\phi }^2\left(z\right)\right]<+\infty$ and $0<\mathbb{E}\left[{\widehat{\phi }}^2\left(z\right)\right]<+\infty$.
• For any compact interval $[c,C]$, there exists a function $g(·)$ such that

  $$\underset{h\in [c,C]}{sup}{\left|\widehat{\phi }\left(hx\right)\right|}^2\le g\left(x\right)\mathit{for}\mathit{all}x\in \mathbb{R}.$$

  Additionally, the function $g(·)$ satisfies $\mathbb{E}\left[g^2\left(z\right)\right]<+\infty$, where $z\sim \mathcal{N}(0,1)$.

Finally, we introduce the following assumption to guarantee that the training and generalization errors of the soft formulation can be asymptotically characterized by deterministic optimization problems.

Assumption 5

(Weighting Matrix). Let $\mathsf{\Lambda }={\mathsf{\Sigma }}^{\top }\mathsf{\Sigma }$, where Σ is the weighting matrix in the soft transfer formulation. Let ${\sigma }_{\mathit{min},1}\left(\mathsf{\Lambda }\right)$ and ${\sigma }_{\mathit{min},2}\left(\mathsf{\Lambda }\right)$ denote its two smallest eigenvalues. There exists a constant ${\mu }_{\mathit{min}}\ge 0$ such that

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{\sigma }_{\mathit{min},1}\left(\mathsf{\Lambda }\right)\stackrel{p}{\to }{\mu }_{\mathit{min}}\hfill \\ |{\sigma }_{\mathit{min},1}\left(\mathsf{\Lambda }\right)-{\sigma }_{\mathit{min},2}\left(\mathsf{\Lambda }\right)|\stackrel{p}{\to }0.\hfill \end{array}\right.\end{array}$$

(13)

Moreover, we assume that empirical distribution of the eigenvalues of the matrix Λ converges weakly to a probability distribution ${\mathbb{P}}_{\mu }(·)$.

The above assumptions are essential to show that the soft formulation in (8) concentrates in the large system limit. We provide more details about these assumptions in Appendix A.

3. Sharp Asymptotic Analysis of Soft Transfer Formulation

In this section, we study the asymptotic properties of soft transfer formulation. Specifically, we provide a precise characterization of the training and generalization errors corresponding to (8).

The asymptotic performance of the source formulation defined in (4) has been studied in the literature [24]. In particular, it has been shown that the asymptotic limit of the source formulation in (4) can be quantified by the following deterministic optimization problem:

$$\underset{\begin{array}{c}{q}_s,r_s\ge 0\end{array}}{min}\underset{\begin{array}{c}{\sigma }_s>0\end{array}}{sup}{\alpha }_s\mathbb{E}\left[{\mathcal{M}}_{\ell (Y_s,.)}\left(r_s{H}_s+q_s{S}_s;\frac{r_s}{{\sigma }_s}\right)\right]-\frac{r_s{\sigma }_s}{2}+\frac{\lambda }{2}(q_s^2+r_s^2),$$

(14)

where $Y_s=\phi \left(S_s\right)$ and $H_s$ and $S_s$ are two independent standard Gaussian random variables. Furthermore, the function ${\mathcal{M}}_{\ell (Y_s,.)}$ introduced in the scalar optimization problem (14) is the Moreau envelope function defined as

$$\begin{array}{c}\hfill {\mathcal{M}}_{\ell (y,.)}(a;b)=\underset{c\in \mathbb{R}}{min}\ell (y;c)+\frac{1}{2b}{(c-a)}^2.\end{array}$$

(15)

The expectation in (14) is taken over the random variables $H_s$ and $S_s$.

In our work, we focus on the target problem with soft transfer, as formulated in (8). It turns out that the asymptotic performance of the target problem can also be characterized by a deterministic optimization problem:

$$\begin{array}{cc}\hfill \underset{\begin{array}{c}{q}_t,r_t\ge 0\end{array}}{min}& \underset{{\sigma }_t>-{\mu }_{\min }}{sup}-\frac{{\sigma }_t{r}_t^2}{2}+\frac{1}{2}\left((1-{\rho }^2){\left(q_s^{★}\right)}^2+{\left(r_s^{★}\right)}^2\right)T_2\left({\sigma }_t\right)\hfill \\ \hfill & +{\alpha }_t\mathbb{E}\left[{\mathcal{M}}_{\ell (Y_t,.)}\left(r_t{H}_t+q_t{S}_t;T_1\left({\sigma }_t\right)\right)\right]+\frac{\lambda }{2}(q_t^2+r_t^2)\hfill \\ \hfill & -\frac{1}{2}{\left(q_t-\rho q_s^{★}\right)}^2\left({\sigma }_t-1/T_1\left({\sigma }_t\right)\right),\hfill \end{array}$$

(16)

where $Y_t=\phi \left(S_t\right)$, and $H_t$ and $S_t$ are independent standard Gaussian random variables. Additionally, ${\mu }_{\min }$ represents the minimum value of the random variable with distribution ${\mathbb{P}}_{\mu }(.)$ as defined in Assumption 5. In the formulation (16), the constants $q_s^{★}$ and $r_s^{★}$ are optimal solutions of the asymptotic formulation given in (14). Moreover, the functions $T_1(.)$ and $T_2(.)$ are defined as follows:

$$\begin{array}{c}\hfill T_1\left({\sigma }_t\right)={\mathbb{E}}_{\mu }[1/(\mu +{\sigma }_t)],T_2\left({\sigma }_t\right)={\mathbb{E}}_{\mu }\left[\mu {\sigma }_t/(\mu +{\sigma }_t)\right],\end{array}$$

where the expectations are taken over the probability distribution ${\mathbb{P}}_{\mu }(.)$ defined in Assumption 5.

Theorem 1

(Precise Analysis of the Soft Transfer). Suppose that Assumptions 1–5 are satisfied. Then, the training error corresponding to the soft transfer formulation in (8) converges in probability as follows:

$$\begin{array}{c}\hfill {\mathcal{E}}_{\mathit{train}}\stackrel{p\to \infty }{\to }{C}_t^{★}-\frac{\lambda }{2}\left({\left(q_t^{★}\right)}^2+{\left(r_t^{★}\right)}^2\right),\end{array}$$

(17)

where $C_t^{★}$ denotes the minimum value achieved by the scalar formulation introduced in (16), and $q_t^{★}$ and $r_t^{★}$ are optimal solutions of the scalar formulation in (16). Moreover, the generalization error introduced in (10) corresponding to soft transfer formulation converges in probability as follows:

$$\begin{array}{c}\hfill {\mathcal{E}}_{\mathrm{test}}\stackrel{p\to \infty }{\to }\frac{1}{4^{\upsilon }}\mathbb{E}\left[{\left(\phi \left({\nu }_1\right)-\widehat{\phi }\left({\nu }_2\right)\right)}^2\right],\end{array}$$

(18)

where ${\nu }_1$ and ${\nu }_2$ are two jointly Gaussian random variables with zero mean and a covariance matrix given by

$$\begin{array}{c}\hfill \left[\begin{array}{cc}1& q_t^{★}\\ q_t^{★}& {\left(q_t^{★}\right)}^2+{\left(r_t^{★}\right)}^2\end{array}\right].\end{array}$$

The proof of Theorem 1 is based on the CGMT framework [17] (Theorem 6.1). A detailed proof is provided in Section 7.3. The statements in Theorem 1 are valid for a general convex loss function and general learning models that can be expressed as in (1) and (2). The analysis in Section 7.3 shows that the deterministic problems in (14) and (16) are the asymptotic limits of the source and target formulations given in (4) and (8), respectively. Moreover, it shows that the deterministic problems (14) and (16) are strictly convex in the minimization variables. This implies the uniqueness of the optimal solutions of the minimization problems.

Remark 1.

The results of the theorem show that the training and generalization errors corresponding to soft transfer formulation can be fully characterized using the optimal solutions of scalar formulation in (16). Moreover, from its definition, (16) depends on the optimal solutions of the scalar formulation in (14) of the source task. This shows that the precise asymptotic performance of the soft transfer formulation can be characterized after solving two scalar deterministic problems.

4. Sharp Asymptotic Analysis of Hard Transfer Formulation

In this section, we study the asymptotic properties of hard transfer formulation. We then use these predictions to rigorously prove the existence of phase transitions from negative to positive transfer.

4.1. Asymptotic Predictions

As mentioned earlier, the hard transfer formulation can be recovered from (8) as a special case where the eigenvalues of the matrix $\mathsf{\Lambda }$ are $+\infty$ with probability $\delta$ and 0 otherwise. Thus, we obtain the following result as a simple consequence of Theorem 1.

Corollary 1.

Suppose that Assumptions 1–4 are satisfied. Then, the asymptotic limit of the hard formulation defined in (6) is given by the following deterministic formulation:

$$\begin{array}{cc}\hfill \underset{\begin{array}{c}{q}_t,r_t\ge 0\end{array}}{min}\underset{\sigma >0}{sup}& \frac{\lambda }{2}(q_t^2+r_t^2)+\frac{\sigma \delta }{2}\left[(1-{\rho }^2){\left(q_s^{★}\right)}^2+{\left(r_s^{★}\right)}^2\right]\hfill \\ \hfill & +{\alpha }_t\mathbb{E}\left[{\mathcal{M}}_{\ell (Y_t,.)}\left(r_t{H}_t+q_t{S}_t;\frac{1-\delta }{\sigma }\right)\right]-\frac{\sigma r_t^2}{2}\hfill \\ \hfill & +\frac{\sigma \delta }{2(1-\delta )}{\left(q_t-\rho q_s^{★}\right)}^2.\hfill \end{array}$$

(19)

Additionally, the training and generalization errors associated with the hard formulation converge in probability to the limits given in (17) and (18), respectively.

4.2. Phase Transitions

As illustrated in Figure 2, there is a phase transition phenomenon in the hard transfer formulation, where the problem moves from negative transfer to positive transfer as the similarity of the source and target tasks increases. For general loss functions, the exact location of the phase transition boundary can only be determined by numerically solving the scalar optimization problem in (19).

For the special case of squared loss, however, we are able to obtain analytical expressions. For the rest of this section, we restrict our discussions to the following special settings:

(a)

The loss function $\ell (·,·)$ in (4) and (6) is the squared loss, i.e., $\ell (y,x)=\frac{1}{2}{(y-x)}^2$.

(b)

The regularization strength $\lambda =0$ in the source and target formulations (4) and (6).

(c)

The data/dimension ratios ${\alpha }_s$ and ${\alpha }_t$ satisfy ${\alpha }_s>1$ and ${\alpha }_t>1$.

We first consider a nonlinear regression task, where the function $\phi (·)$ in the generative models (1) and (2) can be arbitrary and where the function $\widehat{\phi }(·)$ in (9) is the identity function.

Proposition 1

(Regression Phase Transition). In addition to conditions (a)–(c) introduced above, assume that the predefined function $\widehat{\phi }(·)$ in (9) is the identity function. Let ${\delta }^{★}$ be the optimal transfer rate that leads to the lowest generalization error in the hard formulation (6). Then,

$$\begin{array}{c}\hfill {\delta }^{★}=\left\{\begin{array}{cc}0\hfill & \mathit{if}\rho <{\rho }_c({\alpha }_s,{\alpha }_t)\hfill \\ 1\hfill & \mathit{if}\rho >{\rho }_c({\alpha }_s,{\alpha }_t),\hfill \end{array}\right.\end{array}$$

(20)

where ${\rho }_c({\alpha }_s,{\alpha }_t)$ is defined in (11).

The result of Proposition 1, for which the proof can be found in Section 7.4, shows that ${\rho }_c({\alpha }_s,{\alpha }_t)$ is the phase transition boundary separating the negative transfer regime from the positive transfer regime. When the similarity metric is $\rho <{\rho }_c({\alpha }_s,{\alpha }_t)$, the optimal transfer ratio is ${\delta }^{★}=0$, indicating that we should not transfer any source knowledge. Transfer becomes helpful only when $\rho$ moves past the threshold. Note that, for this particular model, there is also an interesting feature that the optimal ${\delta }^{★}$ jumps to 1 in the positive transfer phase, meaning that we should fully copy the source weight vector.

4.3. Sufficient Condition

Next, we consider a binary classification task, where the nonlinear functions $\phi (·)$ and $\widehat{\phi }(·)$ are both the sign function. In this part, we provide a sufficient condition for when the hard transfer is beneficial. Before stating our predictions, we need a few definitions related to the Moreau envelope function defined in (15). For simplicity of notation, we refer to the Moreau envelope function as ${\mathcal{M}}_{\ell }(·,·)$. Based on [25], ${\mathcal{M}}_{\ell }(·,·)$ is differentiable in $\mathbb{R}\times {\mathbb{R}}^{+}$. We refer to its derivatives with respect to the first and second arguments as ${\mathcal{M}}_{\ell ,1}^{\prime }(·,·)$ and ${\mathcal{M}}_{\ell ,2}^{\prime }(·,·)$, respectively. If ${\mathcal{M}}_{\ell }(·,·)$ is twice differentiable, we refer to its second derivative with respect to the first and second arguments as ${\mathcal{M}}_{\ell ,1}^{″}(·,·)$ and ${\mathcal{M}}_{\ell ,2}^{″}(·,·)$, respectively. Additionally, we refer to its second derivative with respect to the first then the second arguments as ${\mathcal{M}}_{\ell ,12}^{″}(·,·)$.

We define $q_0$, $r_0$, and ${\sigma }_0$ as the optimal solutions of the standard learning formulation (i.e., $\delta =0$ in (19)). Moreover, we define the constants ${\beta }_1$ and ${\beta }_2$ as follows:

$$\begin{array}{c}\hfill {\beta }_1=(1-{\rho }^2){\left(q_s^{★}\right)}^2+{\left(r_s^{★}\right)}^2,{\beta }_2=\rho q_s^{★},\end{array}$$

(21)

where $q_s^{★}$ and $r_s^{★}$ are optimal solutions of the deterministic source formulation given in (14). Define the constants $I_{11}$, $I_{12}$, $I_{13}$, and $I_{14}$ as follows:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{I}_{11}={\alpha }_t\mathbb{E}\left(SH{\mathcal{M}}_{\ell ,1}^{″}[r_{0}H+q_{0}S;\frac{1}{{\sigma }_0}]\right),I_{12}={\alpha }_t\mathbb{E}\left(S^2{\mathcal{M}}_{\ell ,1}^{″}[r_{0}H+q_{0}S;\frac{1}{{\sigma }_0}]\right)+\lambda \hfill \\ I_{13}=-\frac{{\alpha }_t}{{\sigma }_0^2}\mathbb{E}\left(S{\mathcal{M}}_{\ell ,12}^{″}[r_{0}H+q_{0}S;\frac{1}{{\sigma }_0}]\right);I_{14}=-\frac{{\alpha }_t}{{\sigma }_0}\mathbb{E}\left(S{\mathcal{M}}_{\ell ,12}^{″}[r_{0}H+q_{0}S;\frac{1}{{\sigma }_0}]\right)+{\sigma }_0(q_0-{\beta }_2),\hfill \end{array}\right.\end{array}$$

where $Y=\phi \left(S\right)$, and H and S are two independent standard Gaussian random variables. Now, define the constants $I_{21}$, $I_{22}$, $I_{23}$, and $I_{24}$ as follows:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{I}_{21}={\alpha }_t\mathbb{E}\left(H^2{\mathcal{M}}_{\ell ,1}^{″}[r_{0}H+q_{0}S;\frac{1}{{\sigma }_0}]\right)-{\sigma }_0+\lambda ,I_{22}={\alpha }_t\mathbb{E}\left(HS{\mathcal{M}}_{\ell ,1}^{″}[r_{0}H+q_{0}S;\frac{1}{{\sigma }_0}]\right)\hfill \\ I_{23}=-\frac{{\alpha }_t}{{\sigma }_0^2}\mathbb{E}\left(H{\mathcal{M}}_{\ell ,12}^{″}[r_{0}H+q_{0}S;\frac{1}{{\sigma }_0}]\right)-r_0,I_{24}=-\frac{{\alpha }_t}{{\sigma }_0}\mathbb{E}\left(H{\mathcal{M}}_{\ell ,12}^{″}[r_{0}H+q_{0}S;\frac{1}{{\sigma }_0}]\right).\hfill \end{array}\right.\end{array}$$

Finally, define the constants $I_{31}$, $I_{32}$, $I_{33}$, and $I_{34}$ as follows:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{I}_{31}=-\frac{{\alpha }_t}{{\sigma }_0^2}\mathbb{E}\left(H{\mathcal{M}}_{\ell ,21}^{″}[r_{0}H+q_{0}S;\frac{1}{{\sigma }_0}]\right)-r_0,I_{32}=-\frac{{\alpha }_t}{{\sigma }_0^2}\mathbb{E}\left(S{\mathcal{M}}_{\ell ,12}^{″}[r_{0}H+q_{0}S;\frac{1}{{\sigma }_0}]\right)\hfill \\ I_{33}=\frac{2{\alpha }_t}{{\sigma }_0^3}\mathbb{E}\left({\mathcal{M}}_{\ell ,2}^{\prime }[r_{0}H+q_{0}S;\frac{1}{{\sigma }_0}]\right)+\frac{{\alpha }_t}{{\sigma }_0^4}\mathbb{E}\left({\mathcal{M}}_{\ell ,2}^{″}[r_{0}H+q_{0}S;\frac{1}{{\sigma }_0}]\right)\hfill \\ I_{34}=\frac{{\alpha }_t}{{\sigma }_0^2}\mathbb{E}\left({\mathcal{M}}_{\ell ,2}^{\prime }[r_{0}H+q_{0}S;\frac{1}{{\sigma }_0}]\right)+\frac{{\alpha }_t}{{\sigma }_0^3}\mathbb{E}\left({\mathcal{M}}_{\ell ,2}^{″}[r_{0}H+q_{0}S;\frac{1}{{\sigma }_0}]\right)+\frac{1}{2}{(q_0-{\beta }_2)}^2+\frac{{\beta }_1}{2}.\hfill \end{array}\right.\end{array}$$

Now, we are ready to state our sufficient condition.

Proposition 2

(Classification). Assume that the Moreau envelope function is twice continuously differentiable almost everywhere in $\mathbb{R}\times {\mathbb{R}}^{+}$ and that the above expectations are all well-defined. Moreover, assume that both $\phi (·)$ and $\widehat{\phi }(·)$ are the sign function. Then,

$$\begin{array}{c}\hfill {\delta }^{★}>0\mathit{if}{q}_0^{\prime }{r}_0-q_0{r}_0^{\prime }>0,\end{array}$$

(22)

where $q_0^{\prime }$ and $r_0^{\prime }$ are solutions to the following linear system of equations:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{I}_{11}{r}_0^{\prime }+I_{12}{q}_0^{\prime }+I_{13}{\sigma }_0^{\prime }+I_{14}=0\hfill \\ I_{21}{r}_0^{\prime }+I_{22}{q}_0^{\prime }+I_{23}{\sigma }_0^{\prime }+I_{24}=0\hfill \\ I_{31}{r}_0^{\prime }+I_{32}{q}_0^{\prime }+I_{33}{\sigma }_0^{\prime }+I_{34}=0.\hfill \end{array}\right.\end{array}$$

(23)

We prove this result at the end of Section 7. Note that Proposition 2 is valid for a general family of loss functions and general regularization strength $\lambda \ge 0$. For instance, we can see that the results stated in Proposition 2 are valid for the squared loss and the least absolute deviation (LAD) loss, i.e.,

$$\begin{array}{c}\hfill \left\{\begin{array}{c}\ell (y;x)=\frac{1}{2}{(1-yx)}^2\hfill \\ \ell (y;x)=\left|1-yx\right|.\hfill \end{array}\right.\end{array}$$

(24)

Unlike (20), the result in (22) only provides a sufficient condition for when the hard transfer is beneficial. Nevertheless, our numerical simulations show that the sufficient condition in (22) provides a good prediction of the phase transition boundary for the majority of parameter settings.

5. Remarks

5.1. Learning Formulations

Given that the target task predicts the new label with $\widehat{\phi }(·)$, it is more natural to consider loss functions satisfying the following form:

$$\ell (y_i;\widehat{\phi }\left({\mathit{a}}_i^{\top }\mathit{w}\right)).$$

In this case, the convexity assumption is not necessarily satisfied since the loss function can be viewed as the composition of a convex function with a nonlinear function. To guarantee the convexity, we need additional assumptions on the function $\widehat{\phi }(·)$. Moreover, note that, once the convexity is guaranteed, the function $\widehat{\phi }(·)$ can be absorbed by the loss function $\ell (·;·)$.

5.2. Transition from Negative to Positive Transfer

Our first simulation example in Figure 2 shows that the optimal transfer rate ${\delta }^{★}$ can be 1 while the similarity $\rho$ is still less than 1. Here, we provide an intuitive explanation of this behavior.

Given that the source and target feature vectors are generated from the same distribution, one can see that the source labels can be equivalently expressed as follows:

$$\begin{array}{c}\hfill y_{s,i}=\phi (\rho {\mathit{a}}_{t,i}^{\top }{\mathit{\xi }}_t+z_i),\forall i\in \{1,\dots ,n_s\},\end{array}$$

(25)

where ${\left\{z_i\right\}}_{i=1}^{n_s}$ is an additive noise caused by the mismatch between the source and target hidden vectors. Moreover, note that the noise strength depends on the similarity measure $\rho$.

First, consider the case when the number of source samples is bigger than the number of target samples (i.e., ${\alpha }_s>{\alpha }_t$). We can see that a large value of $\rho$ means that the source and target models are very closely related. Then, one can expect that the additional available data in the source task will be capable of defeating the effects of noise in (25) for large values of $\rho$. Specifically, it is expected in this regime that the source model will perform better than the standard learning formulation for values of $\rho$ close to 1. However, as we decrease the similarity $\rho$, the source model will have a small information about the target data. Then, the performance of the hard formulation is expected to be lower than the standard formulation for small values of $\rho$. In this regime, the source information may hurt the generalization performance of the target task. Then, we need to only transfer a portion of the source information (see Figure 2a). In some settings, the transition is sharp, which means that the source information is irrelevant for the target task when $\rho$ is smaller than a threshold (see Figure 2b).

Second, consider the case when the number of source samples is smaller than the number of target samples (i.e., ${\alpha }_s<{\alpha }_t$). Given the observation in (25), the performance of the standard method is expected to be better than the hard formulation for all possible values of $\rho$ in this regime (see Figure 7).

6. Additional Simulation Results

In this section, we provide additional simulation examples to confirm our asymptotic analysis and illustrate the phase transition phenomenon. In our experiments, we focus on the regression and classification models.

6.1. Model Assumptions

For the regression model, we assume that the source, target, and test data are generated according to

$$\begin{array}{c}\hfill y_i=max({\mathit{a}}_i^{\top }\mathit{\xi },0),\forall i\in \{1,\dots ,n\}.\end{array}$$

(26)

The data ${\left\{({\mathit{a}}_i,y_i)\right\}}_{i=1}^n$ can be the training data of the source or target tasks. In this regression model, we assume that the function $\widehat{\phi }(.)$ is the identity function, i.e., $\widehat{\phi }\left(x\right)=x$. Then, the generalization error corresponding to the soft formulation converges in probability as follows:

$$\begin{array}{c}\hfill {\mathcal{E}}_{\mathrm{test}}\stackrel{p\to \infty }{\to }v-2c{q}_t^{★}+({\left(q_t^{★}\right)}^2+{\left(r_t^{★}\right)}^2),\end{array}$$

where c and v are defined as follows

$$\begin{array}{cc}\hfill & c=\mathbb{E}[zmax(z,0)],v=\mathbb{E}[max{(z,0)}^2],\hfill \end{array}$$

where z is a standard Gaussian random variable and $q_t^{★}$ and $r_t^{★}$ are defined in Theorem 1. Additionally, the asymptotic limit of the generalization error corresponding to the hard formulation can be expressed in a similar fashion.

For the binary classification model, we assume that the source, target, and test data labels are binary and generated as follows:

$$\begin{array}{c}\hfill y_i=\mathrm{sign}\left({\mathit{a}}_i^{\top }\mathit{\xi }\right),\forall i\in \{1,\dots ,n\},\end{array}$$

(27)

where the data ${\left\{({\mathit{a}}_i,y_i)\right\}}_{i=1}^n$ can be the training data of the source and target tasks. In this classification model, the objective is to predict the correct sign of any unseen sample $y_{\mathrm{new}}$. Then, we fix the function $\widehat{\phi }(.)$ to be the sign function. Following Theorem 1, it can be easily shown that the generalization error corresponding to the soft formulation given in (8) converges in probability as follows:

$$\begin{array}{c}\hfill {\mathcal{E}}_{\mathrm{test}}\stackrel{p\to \infty }{\to }\frac{1}{\pi }{cos}^{-1}\left(\frac{q_t^{★}}{\sqrt{{\left(q_t^{★}\right)}^2+{\left(r_t^{★}\right)}^2}}\right).\end{array}$$

Here, $q_t^{★}$ and $r_t^{★}$ are optimal solutions of the target scalar formulation given in (16). The generalization error corresponding to the hard formulation given in (6) can be expressed in a similar fashion.

6.2. Phase Transitions in the Hard Formulation

In Section 4, we presented analytical formulas for the phase transition phenomenon but only for the special case of squared loss with no regularization. The main purpose of this experiment, shown in Figure 3, is to demonstrate that the phase transition phenomenon still takes place in more general settings with different loss functions and regularization strengths.

Figure 3. Additional illustrations of the phase transition phenomenon. (a) Regression (squared loss, ${\alpha }_t=0.5$, and ${\alpha }_s=3{\alpha }_t$) (b) Regression (squared loss, ${\alpha }_t=2$, and ${\alpha }_s=2{\alpha }_t$) (c) Binary classification (squared loss, ${\alpha }_t=1.5$, and ${\alpha }_s=3{\alpha }_t$) (d) Binary classification (hinge loss, ${\alpha }_t=1.5$, and ${\alpha }_s=3{\alpha }_t$). In all the experiments, we set the regularization strength to be $\lambda =0.1$. The blue line represents our theoretical predictions of the optimal transfer rate obtained by solving our asymptotic results in Section 4 for multiple values of $\delta$. The empirical results are averaged over 100 independent Monte Carlo trials with $p=2500$.

In all the cases shown in Figure 3, the transition from negative to positive transfer is a discontinuous jump from standard learning (i.e., no transfer) to full source transfer. Additionally, Figure 3c,d show that the loss function has a small effect on the phase transition boundary.

6.3. Sufficient Condition for the Hard Formulation

In Section 4, we presented a sufficient condition for positive transfer. This sufficient condition is valid for a general family of loss functions and a general regularization strength. The main purpose of this experiment, shown in Figure 4, is to illustrate the precision of the sufficient condition for two particular loss functions, i.e., the squared loss and LAD loss.

Figure 4. Illustrations of the sufficient condition in Proposition 2. (a) Classification (squared loss, ${\alpha }_t=1.5$, and ${\alpha }_s=8{\alpha }_t$) (b) Classification (LAD loss, ${\alpha }_t=1.5$, and ${\alpha }_s=8{\alpha }_t$). In all the experiments, we set the regularization strength to be $\lambda =0.1$. The blue line represents our theoretical predictions of the optimal transfer rate obtained by solving our asymptotic results in Section 4 for multiple values of $\delta$. The green line represents our sufficient condition for positive transfer stated in Proposition 2.

In all the cases shown in Figure 4, we can see that the transition from negative to positive transfer is a discontinuous jump from standard learning to full source transfer. Additionally, Figure 4a,b show that the sufficient condition summarized in Proposition 2 provides a good prediction of the phase transition boundary for the considered setting.

6.4. Soft Transfer: Impact of the Weighting Matrix and Regularization Strength

In this experiment, we empirically explore the impact of the weighting matrix $\mathsf{\Sigma }$ on the generalization error corresponding to the soft formulation. We focus on the binary classification problem with logistic loss. The weighting matrix in (8) takes the following form:

$$\mathsf{\Sigma }=\sqrt{{\beta }_t}\mathit{V},$$

(28)

where $\mathit{V}$ is a diagonal matrix generated in three different ways. (1) Soft Identity: $\mathit{V}$ is an identity matrix; (2) Soft Uniform: the diagonal entries of $\mathit{V}$ are drawn independently from the uniform distribution and then scaled to have their mean equal to 1; and (3): Soft Beta: similar to (2), but with the diagonal entries drawn from the beta distribution, followed by rescaling to the unit mean.

Figure 5a shows that the considered weighting matrix choices have similar generalization performances, with the identity matrix being slightly better than the other alternatives. Moreover, Figure 5b illustrates the effects of the parameter ${\beta }_t$ in (28) on the generalization performance. It points to the interesting possibility of “designing” the optimal weight matrix to minimize the generalization error.

Figure 5. Continuous line: theoretical predictions. Circles: numerical simulations. (a) ${\alpha }_s=6{\alpha }_t$, $\lambda =0.1$, ${\beta }_t=1/10$, and $\rho =0.9$. (b) ${\alpha }_t=1$, ${\alpha }_s=5{\alpha }_t$, $\lambda =0.3$, and $\rho =0.75$. In all the experiments, we consider the binary classification problem with the logistic loss function. The empirical results are averaged over 50 independent Monte Carlo trials, and we set $p=1000$.

6.5. Soft and Hard Transfer Comparison

In this simulation example, we consider the regression model and compare the performances of the hard and soft transfer formulations as functions of ${\alpha }_t$ and $\rho$.

Figure 6a shows that the soft formulation provides the best generalization performance for all values of ${\alpha }_t$. Moreover, we can see that the hard transfer formulation is only useful for small values ${\alpha }_t$. Figure 6b shows that the performance of the soft and hard transfer formulations depend on the similarity between the source and target tasks. Specifically, the generalization performances of different transfer approaches all improve as we increase the similarity measure $\rho$. We can also see that the full source transfer approach provides the lowest generalization error when the similarity measure is close to 1, while the soft transfer method leads to the best generalization performance at moderate values of the similarity measure. At very small values of $\rho$, which means that the two tasks share little resemblance, the standard learning method (i.e., no transfer) is the best scheme one should use.

Figure 6. Continuous line: theoretical predictions. Circles: numerical simulations. (a) ${\alpha }_s=12{\alpha }_t$, $\lambda =0.2$, and $\rho =0.75$. (b) ${\alpha }_t=1.5$, ${\alpha }_s=8{\alpha }_t$, and $\lambda =0.4$. In all the experiments, we consider the regression setting with a squared loss. The hard transfer formulation uses $\delta =0.5$, and the soft transfer formulation uses an identity weighting matrix. The empirical results are averaged over 50 independent Monte Carlo trials and we set $p=1000$.

6.6. Effects of the Source Parameters

In the last simulation example, we consider the regression and classification models. We study the performance of the hard and soft transfer formulations when ${\alpha }_s<{\alpha }_t$.

Figure 7a considers the regression model. It first shows that the soft transfer formulation provides a slightly better generalization performance compared to the standard method. This behavior can be explained by the fact that the soft formulation requires the target weight vector to be close and not necessarily equal to the source weight vector. Additionally, the source model carries some information about the target task.

Figure 7. Continuous line: theoretical predictions. Circles: numerical simulations. (a) ${\alpha }_s=0.5{\alpha }_t$, $\lambda =0.6$, and $\rho =0.7$. We consider the regression setting with a squared loss. (b) ${\alpha }_s=0.5{\alpha }_t$, $\lambda =0.3$, and $\rho =0.8$. We consider the classification setting with a logistic loss. The hard transfer formulation uses $\delta =0.5$, and the soft transfer formulation uses an identity weighting matrix. The empirical results are averaged over 60 independent Monte Carlo trials, and we set $p=1000$.

We can also see that the hard transfer approach is not beneficial when the number of source samples is smaller than the number of target samples. This result can be explained by the fact that the hard formulation restricts some entries in the target weight vector to be exactly equal to the corresponding entries in the source weight vector. Moreover, the source model is not perfectly aligned with the target model and has smaller data than the target model (see Section 5.2).

The same behavior can be observed in Figure 7b, which considers the classification model.

7. Technical Details

In this section, we provide a detailed proof of Theorem 1, and Proportions 1 and 2. Specifically, we focus on analyzing the generalized formulation in (8) using the CGMT framework introduced in the following part.

7.1. Technical Tool: Convex Gaussian Min–Max Theorem

The CGMT provides an asymptotic equivalent formulation of primary optimization (PO) problems of the following form:

$${\mathsf{\Phi }}_p\left(\mathit{G}\right)=\underset{\mathit{w}\in {\mathcal{S}}_{\mathit{w}}}{min}\underset{\mathit{u}\in {\mathcal{S}}_{\mathit{u}}}{max}{\mathit{u}}^{\top }\mathit{G}\mathit{w}+\psi (\mathit{w},\mathit{u}).$$

(29)

Specifically, the CGMT shows that the PO given in (29) is asymptotically equivalent to the following formulation:

$${\varphi }_p(\mathit{g},\mathit{h})=\underset{\mathit{w}\in {\mathcal{S}}_{\mathit{w}}}{min}\underset{\mathit{u}\in {\mathcal{S}}_{\mathit{u}}}{max}\parallel \mathit{u}\parallel {\mathit{g}}^{\top }\mathit{w}+\parallel \mathit{w}\parallel {\mathit{h}}^{\top }\mathit{u}+\psi (\mathit{w},\mathit{u}),$$

(30)

referred to as the auxiliary optimization (AO) problem. Before showing the equivalence between PO and AO, the CGMT assumes that $\mathit{G}\in {\mathbb{R}}^{n\times p}$, $\mathit{g}\in {\mathbb{R}}^p$, and $\mathit{h}\in {\mathbb{R}}^n$; that all have independent and identically distributed standard normal entries; that the feasibility sets ${\mathcal{S}}_{\mathit{w}}\subset {\mathbb{R}}^p$ and ${\mathcal{S}}_{\mathit{u}}\subset {\mathbb{R}}^n$ are convex and compact; and that the function $\psi (.,.):{\mathbb{R}}^p\times {\mathbb{R}}^n\to \mathbb{R}$ is continuous convex-concave on ${\mathcal{S}}_{\mathit{w}}\times {\mathcal{S}}_{\mathit{u}}$. Moreover, the function $\psi (.,.)$ is independent of the matrix $\mathit{G}$. Under these assumptions, the CGMT [17] (Theorem 6.1) shows that, for any $\chi \in \mathbb{R}$ and $\zeta >0$, the following holds:

$$\mathbb{P}\left(|{\mathsf{\Phi }}_p\left(\mathit{G}\right)-\chi |>\zeta \right)\le 2\mathbb{P}\left(|{\varphi }_p(\mathit{g},\mathit{h})-\chi |>\zeta \right).$$

(31)

Additionally, the CGMT [17] (Theorem 6.1) provides the following conditions under which the optimal solutions of the PO and AO concentrates around the same set.

Theorem 2

(CGMT Framework). Consider an open set ${\mathcal{S}}_p$. Moreover, define the set ${\mathcal{S}}_p^c={\mathcal{S}}_{\mathit{w}}\backslash {\mathcal{S}}_p$. Let ${\varphi }_p$ and ${\varphi }_p^c$ be the optimal cost values of AO formulation in (30) with feasibility sets ${\mathcal{S}}_{\mathit{w}}$ and ${\mathcal{S}}_p^c$, respectively. Assume that the following properties are all satisfied:

(1) 

There exists a constant ϕ such that the optimal cost ${\varphi }_p$ converges in probability to ϕ as p goes to $+\infty$.

(2) 

There exists a positive constant $\zeta >0$ such that ${\varphi }_p^c\ge \varphi +\zeta$ with probability going to 1 as $p\to +\infty$.

Then, the following convergence in probability holds:

$$|{\mathsf{\Phi }}_p-{\varphi }_p|\stackrel{p\to +\infty }{⟶}0,\mathit{and}\mathbb{P}({\widehat{\mathit{w}}}_p\in {\mathcal{S}}_p)\stackrel{p\to +\infty }{⟶}1,,$$

where ${\mathsf{\Phi }}_p$ and ${\widehat{\mathit{w}}}_p$ are the optimal cost and the optimal solution of the PO formulation in (29).

Theorem 2 allows us to analyze the generally easy AO problem to infer the asymptotic properties of the generally hard PO problem. Next, we use the CGMT to rigorously prove the technical results presented in Theorem 1.

7.2. Precise Analysis of the Source Formulation

The source formulation defined in (4) is well-studied in recent literature [26]. Specifically, it has been rigorously proven that the performance of the source formulation can be fully characterized after solving the following scalar formulation:

$$\begin{array}{cc}\hfill \underset{\begin{array}{c}{q}_s,r_s\ge 0\end{array}}{min}\underset{\begin{array}{c}{\sigma }_s>0\end{array}}{sup}& {\alpha }_s\mathbb{E}\left[{\mathcal{M}}_{\ell (Y_s,.)}\left(r_s{H}_s+q_s{S}_s;\frac{r_s}{{\sigma }_s}\right)\right]\hfill \\ \hfill & -\frac{r_s{\sigma }_s}{2}+\frac{\lambda }{2}(q_s^2+r_s^2),\hfill \end{array}$$

(32)

where $Y_s=\phi \left(S_s\right)$, and $H_s$ and $S_s$ are two independent standard Gaussian random variables. The expectation in (32) is taken over the random variables $H_s$ and $S_s$. Furthermore, the function ${\mathcal{M}}_{\ell (Y_s,.)}$ introduced in the scalar optimization problem (32) is the Moreau envelope function defined in (15).

7.3. Precise Analysis of the Soft Transfer Approach

In this part, we provide a precise asymptotic analysis of the generalized transfer formulation given in (8). Specifically, we focus on analyzing the following formulation:

$$\begin{array}{cc}\hfill \underset{\mathit{w}\in {\mathbb{R}}^p}{min}& \frac{1}{p}\sum _{i=1}^{n_t}\ell \left(y_i;{\mathit{a}}_i^{\top }\mathit{w}\right)+\frac{\lambda }{2}{\parallel \mathit{w}\parallel }^2+\frac{1}{2}{\parallel \mathsf{\Sigma }(\mathit{w}-{\widehat{\mathit{w}}}_s)\parallel }^2,\hfill \end{array}$$

(33)

where ${\widehat{\mathit{w}}}_s$ is the optimal solution of the source formulation given in (4). Note that the vector ${\widehat{\mathit{w}}}_s$ is independent of the training data of the target task. For simplicity of notation, we denote by ${\left\{({\mathit{a}}_i,y_i)\right\}}_{i=1}^{n_t}$ the training data of the target task. Here, we use the CGMT framework introduced in Section 7.1 to precisely analyze the above formulation.

7.3.1. Formulating the Auxiliary Optimization Problem

Our first objective is to rewrite the generalized formulation in the form of the PO problem given in (29). To this end, we introduce additional optimization variables. Specifically, the generalized formulation can be equivalently formulated as follows:

$$\begin{array}{cc}\hfill \underset{\mathit{w}\in {\mathbb{R}}^p}{min}\underset{\mathit{u}\in {\mathbb{R}}^{n_t}}{max}& \frac{1}{p}{\mathit{u}}^{\top }\mathit{A}\mathit{w}-\frac{1}{p}\sum _{i=1}^{n_t}{\ell }^{★}\left(y_i;u_i\right)+\frac{\lambda }{2}{\parallel \mathit{w}\parallel }^2\hfill \\ \hfill & +\frac{1}{2}{\parallel \mathsf{\Sigma }(\mathit{w}-{\widehat{\mathit{w}}}_s)\parallel }^2,\hfill \end{array}$$

(34)

where the optimization vector $\mathit{u}\in {\mathbb{R}}^{n_t}$ is formed as $\mathit{u}={[u_1,\dots ,u_{n_t}]}^{\top }$ and the data matrix $\mathit{A}\in {\mathbb{R}}^{n_t\times p}$ is given by $\mathit{A}={[{\mathit{a}}_1,\dots ,{\mathit{a}}_{n_t}]}^{\top }$. Additionally, the function ${\ell }^{★}(y;.)$ denotes the convex conjugate function of the loss function $\ell (y;.)$. First, observe that the CGMT framework assumes that the feasibility sets of the minimization and maximization problems are compact. Then, our next step is to show that the formulation given in (34) satisfies this assumption.

Lemma 1

(Primal-Dual Compactness). Assume that $\widehat{\mathit{w}}$ and $\widehat{\mathit{u}}$ are optimal solutions of the optimization problem in (34). Then, there exist two constants $C_w>0$ and $C_u>0$ such that the following convergence in probability holds:

$$\begin{array}{c}\hfill \mathbb{P}(\parallel \widehat{\mathit{w}}\parallel \le C_w)\stackrel{p\to +\infty }{\to }1,\mathbb{P}(\parallel \widehat{\mathit{u}}\parallel /\sqrt{n_t}\le C_u)\stackrel{p\to +\infty }{\to }1.\end{array}$$

(35)

A detailed proof of Lemma 1 is provided in Appendix B. The proof of the above result follows using Assumption 3 to prove the compactness of the optimal solution $\widehat{\mathit{w}}$. Moreover, it uses the asymptotic results in [27] (Theorem 2.1), which provides the concentration properties of the minimum and maximum eigenvalues of random matrices. To show the compactness of the optimal dual vector $\widehat{\mathit{u}}$, we use Assumption 3 and the result in [25] (Proposition 11.3), which provides the inversion rules for subgradient relations.

The theoretical result in Lemma 1 shows that the optimization problem in (34) can be equivalently formulated with compact feasibility sets on events with probability going to one. Then, it suffices to study the constrained version of (34). Note that the data labels ${\left\{y_i\right\}}_{i=1}^{n_t}$ depend on the data matrix $\mathit{A}$. Then, one can decompose the matrix $\mathit{A}$ as follows:

$$\begin{array}{cc}\hfill \mathit{A}& =\mathit{A}{\mathit{P}}_{{\mathit{\xi }}_t}+\mathit{A}{\mathit{P}}_{\mathit{\xi }}^{\perp }=\mathit{A}{\mathit{\xi }}_t{\mathit{\xi }}_t^{\top }+\mathit{A}{\mathit{P}}_{\mathit{\xi }}^{\perp },\hfill \end{array}$$

where the matrix ${\mathit{P}}_{{\mathit{\xi }}_t}\in {\mathbb{R}}^{p\times p}$ denotes the projection matrix onto the space spanned by the vector ${\mathit{\xi }}_t$ and the matrix ${\mathit{P}}_{\mathit{\xi }}^{\perp }={\mathit{I}}_p-{\mathit{\xi }}_t{\mathit{\xi }}_t^{\top }$ denotes the projection matrix onto the orthogonal complement of the space spanned by the vector ${\mathit{\xi }}_t$. Note that we can express $\mathit{A}$ as follows without changing its statistics:

$$\begin{array}{c}\hfill \mathit{A}={\mathit{s}}_t{\mathit{\xi }}_t^{\top }+\mathit{G}{\mathit{P}}_{\mathit{\xi }}^{\perp },\end{array}$$

(36)

where ${\mathit{s}}_t\sim \mathcal{N}(0,{\mathit{I}}_{n_t})$ and the components of the matrix $\mathit{G}\in {\mathbb{R}}^{n_t\times p}$ are drawn independently from a standard Gaussian distribution and where ${\mathit{s}}_t$ and $\mathit{G}$ are independent. Here, (36) represents an equality in distribution. This means that the formulation in (34) can be expressed as follows:

$$\begin{array}{cc}\hfill & \underset{\parallel \mathit{w}\parallel \le C_w}{min}\underset{\mathit{u}\in {\mathcal{C}}_t}{max}\frac{1}{p}{\mathit{u}}^{\top }\mathit{G}{\mathit{P}}_{\mathit{\xi }}^{\perp }\mathit{w}+\frac{1}{p}{\mathit{u}}^{\top }{\mathit{s}}_t{\mathit{\xi }}_t^{\top }\mathit{w}+\frac{\lambda }{2}{\parallel \mathit{w}\parallel }^2\hfill \\ \hfill & -\frac{1}{p}\sum _{i=1}^{n_t}{\ell }^{★}\left(y_i;u_i\right)+\frac{1}{2}{\parallel \mathsf{\Sigma }(\mathit{w}-{\widehat{\mathit{w}}}_s)\parallel }^2,\hfill \end{array}$$

(37)

where the set ${\mathcal{C}}_t$ is defined as ${\mathcal{C}}_t=\{\mathit{u}:\parallel \mathit{u}\parallel /\sqrt{n_t}\le C_u\}$. Note that the formulation in (37) is in the form of the primary formulation given in (29). Here, the function $\psi (.,.)$ is defined as follows:

$$\begin{array}{cc}\hfill \psi (\mathit{w},\mathit{u})& =\frac{1}{p}{\mathit{u}}^{\top }{\mathit{s}}_t{\mathit{\xi }}_t^{\top }\mathit{w}+\frac{\lambda }{2}{\parallel \mathit{w}\parallel }^2-\frac{1}{p}\sum _{i=1}^{n_t}{\ell }^{★}\left(y_i;u_i\right)\hfill \\ \hfill & +\frac{1}{2}{\parallel \mathsf{\Sigma }(\mathit{w}-{\widehat{\mathit{w}}}_s)\parallel }^2.\hfill \end{array}$$

(38)

One can easily see that the optimization problem in (37) has compact convex feasibility sets. Moreover, the function $\psi (.,.)$ is continuous, convex–concave, and independent of the Gaussian matrix $\mathit{G}$. This shows that the assumptions of the CGMT are all satisfied by the primary formulation in (37). Then, following the CGMT framework, the auxiliary formulation corresponding to our primary problem in (37) can be expressed as follows:

$$\begin{array}{cc}\hfill & \underset{\parallel \mathit{w}\parallel \le C_w}{min}\underset{\mathit{u}\in {\mathcal{C}}_t}{max}\frac{\parallel \mathit{u}\parallel }{p}{\mathit{g}}^{\top }{\mathit{P}}_{\mathit{\xi }}^{\perp }\mathit{w}+\frac{1}{p}{\mathit{u}}^{\top }{\mathit{s}}_t{\mathit{\xi }}_t^{\top }\mathit{w}+\frac{{\mathit{h}}^{\top }\mathit{u}}{p}\parallel {\mathit{P}}_{\mathit{\xi }}^{\perp }\mathit{w}\parallel \hfill \\ \hfill & +\frac{\lambda }{2}{\parallel \mathit{w}\parallel }^2-\frac{1}{p}\sum _{i=1}^{n_t}{\ell }^{★}\left(y_i;u_i\right)+\frac{1}{2}{\parallel \mathsf{\Sigma }(\mathit{w}-{\widehat{\mathit{w}}}_s)\parallel }^2,\hfill \end{array}$$

(39)

where $\mathit{g}\in {\mathbb{R}}^p$ and $\mathit{h}\in {\mathbb{R}}^{n_t}$ are two independent standard Gaussian vectors. The rest of the proof focuses on simplifying the obtained AO formulation and on studying its asymptotic properties.

7.3.2. Simplifying the AO Problem of the Target Task

Here, we focus on simplifying the auxiliary formulation corresponding to the target task. We start our analysis by decomposing the target optimization vector $\mathit{w}\in {\mathbb{R}}^p$ as follows:

$$\begin{array}{c}\hfill \mathit{w}=\left({\mathit{\xi }}_t^{\top }\mathit{w}\right){\mathit{\xi }}_t+{\mathit{B}}_{{\mathit{\xi }}_t}^{\perp }{\mathit{r}}_t,\end{array}$$

(40)

where ${\mathit{r}}_t\in {\mathbb{R}}^{p-1}$ is a free vector and ${\mathit{B}}_{{\mathit{\xi }}_t}^{\perp }\in {\mathbb{R}}^{p\times (p-1)}$ is formed by an orthonormal basis orthogonal to the vector ${\mathit{\xi }}_t$. Now, define the variable $q_t$ as follows: $q_t={\mathit{\xi }}_t^{\top }\mathit{w}$. Based on the result in Lemma 1 and the decomposition in (40), there exist $C_{q_t}>0$, $C_r>0$, and $C_u>0$ such that our auxiliary formulation can be asymptotically expressed in terms of the variables $q_t$ and ${\mathit{r}}_t$ as follows:

$$\begin{array}{cc}\hfill & \underset{\begin{array}{c}(q_t,{\mathit{r}}_t)\in {\mathcal{T}}_1\end{array}}{min}\underset{\begin{array}{c}\mathit{u}\in {\mathcal{C}}_t\end{array}}{max}\frac{\parallel \mathit{u}\parallel }{p}{\mathit{g}}^{\top }{\mathit{B}}_{{\mathit{\xi }}_t}^{\perp }{\mathit{r}}_t+\frac{\parallel {\mathit{r}}_t\parallel }{p}{\mathit{h}}^{\top }\mathit{u}+\frac{q_t}{p}{\mathit{u}}^{\top }{\mathit{s}}_t+\frac{\lambda }{2}{q}_t^2\hfill \\ \hfill & +\frac{\lambda }{2}{\parallel {\mathit{r}}_t\parallel }^2-\frac{1}{p}\sum _{i=1}^{n_t}{\ell }^{★}\left(y_i;u_i\right)+\frac{1}{2}{q}_t^2{V}_{p,t}-q_t{V}_{p,ts}\hfill \\ \hfill & +\frac{1}{2}{\mathit{r}}_t^{\top }{\left({\mathit{B}}_{{\mathit{\xi }}_t}^{\perp }\right)}^{\top }\mathsf{\Lambda }{\mathit{B}}_{{\mathit{\xi }}_t}^{\perp }{\mathit{r}}_t+q_t{\mathit{\xi }}_t^{\top }\mathsf{\Lambda }{\mathit{B}}_{{\mathit{\xi }}_t}^{\perp }{\mathit{r}}_t-{\mathit{r}}_t^{\top }{\left({\mathit{B}}_{{\mathit{\xi }}_t}^{\perp }\right)}^{\top }\mathsf{\Lambda }{\widehat{\mathit{w}}}_s.\hfill \end{array}$$

Here, we drop terms independent of the optimization variables and the matrix $\mathsf{\Lambda }\in {\mathbb{R}}^{p\times p}$ is defined as $\mathsf{\Lambda }={\mathsf{\Sigma }}^{\top }\mathsf{\Sigma }$. Additionally, the feasibility set ${\mathcal{T}}_1$ is defined as follows:

$$\begin{array}{c}\hfill {\mathcal{T}}_1=\left\{(q_t,{\mathit{r}}_t):\left|q_t\right|\le C_{q_t},\parallel {\mathit{r}}_t\parallel \le C_r\right\}.\end{array}$$

(41)

Here, the sequence of random variables $V_{p,t}$ and $V_{p,ts}$ are defined as follows:

$$\begin{array}{cc}\hfill & V_{p,t}={\mathit{\xi }}_t^{\top }\mathsf{\Lambda }{\mathit{\xi }}_t,V_{p,ts}={\mathit{\xi }}_t^{\top }\mathsf{\Lambda }{\widehat{\mathit{w}}}_s.\hfill \end{array}$$

(42)

Next, we focus on simplifying the obtained auxiliary formulation. Our strategy is to solve over the direction of the optimization vector $\mathit{r}\in {\mathbb{R}}^{p-1}$. This step requires an interchange between non-convex minimization and non-concave maximization. We can justify the interchange using the theoretical result in [17] (Lemma A.3). The main argument in [17] (Lemma A.3) is that the strong convexity of the primary formulation in (37) allows us to perform such an interchange in the corresponding auxiliary formulation. The optimization problem over the vector ${\mathit{r}}_t$ with fixed norm, i.e., $\parallel {\mathit{r}}_t\parallel =r_t$, can be formulated as follows:

$$\begin{array}{cc}\hfill C_p^{★}=& \underset{{\mathit{r}}_t\in {\mathbb{R}}^{p-1}}{min}{\mathbf{b}}_p^{\top }{\mathit{r}}_t+\frac{1}{2}{\mathit{r}}_t^{\top }{\mathsf{\Lambda }}^{\perp }{\mathit{r}}_t,\mathrm{s}.\mathrm{t}.\parallel {\mathit{r}}_t\parallel =r_t.\hfill \end{array}$$

(43)

Here, we ignore constant terms independent of ${\mathit{r}}_t$, and the matrix ${\mathsf{\Lambda }}^{\perp }\in {\mathbb{R}}^{(p-1)\times (p-1)}$ and the vector ${\mathbf{b}}_p\in {\mathbb{R}}^{p-1}$ can be expressed as follows:

$$\begin{array}{cc}\hfill {\mathsf{\Lambda }}^{\perp }={\left({\mathit{B}}_{{\mathit{\xi }}_t}^{\perp }\right)}^{\top }\mathsf{\Lambda }{\mathit{B}}_{{\mathit{\xi }}_t}^{\perp },{\mathbf{b}}_p& =\frac{\parallel \mathit{u}\parallel }{p}{\left({\mathit{B}}_{{\mathit{\xi }}_t}^{\perp }\right)}^{\top }\mathit{g}+q_t{\left({\mathit{B}}_{{\mathit{\xi }}_t}^{\perp }\right)}^{\top }\mathsf{\Lambda }{\mathit{\xi }}_t^{\top }\hfill \\ \hfill & -{\left({\mathit{B}}_{{\mathit{\xi }}_t}^{\perp }\right)}^{\top }\mathsf{\Lambda }{\widehat{\mathit{w}}}_s.\hfill \end{array}$$

The optimization problem in (43) is non-convex given the norm equality constraint. It is well-studied in the literature [28] and is known as the trust region subproblem. Using the same analysis as in [20], the optimal cost value of the optimization problem (43) can be expressed in terms of a one-dimensional optimization problem as follows:

$$\begin{array}{c}\hfill C_p^{★}=\underset{{\sigma }_t>-{\mu }_p}{sup}\left\{-\frac{1}{2}{\mathbf{b}}_p^{\top }{[{\mathsf{\Lambda }}^{\perp }+{\sigma }_t{\mathit{I}}_{p-1}]}^{-1}{\mathbf{b}}_p-\frac{{\sigma }_t{r}_t^2}{2}\right\},\end{array}$$

(44)

where ${\mu }_p$ is the minimum eigenvalue of the matrix ${\mathsf{\Lambda }}^{\perp }$, denoted by ${\sigma }_{\min }\left({\mathsf{\Lambda }}^{\perp }\right)$. This result can be seen by equivalently formulating the non-convex problem in (43) as follows:

$$\begin{array}{cc}\hfill C_p^{★}=& \underset{{\mathit{r}}_t\in {\mathbb{R}}^{p-1}}{min}\underset{{\sigma }_t\in \mathbb{R}}{max}{\mathbf{b}}_p^{\top }{\mathit{r}}_t+\frac{1}{2}{\mathit{r}}_t^{\top }{\mathsf{\Lambda }}^{\perp }{\mathit{r}}_t+\frac{{\sigma }_t}{2}\left({\parallel {\mathit{r}}_t\parallel }^2-r_t^2\right).\hfill \end{array}$$

Then, we show that the optimal ${\sigma }_t$ satisfies a constraint that preserves the convexity over ${\mathit{r}}_t$. This allows us to interchange the maximization and minimization and to solve over the vector ${\mathit{r}}_t$. The above analysis shows that the AO formulation corresponding to our primary problem can be expressed as follows:

$$\begin{array}{cc}\hfill & \underset{\begin{array}{c}(q_t,r_t)\in {\mathcal{T}}_2\end{array}}{min}\underset{\begin{array}{c}\mathit{u}\in {\mathcal{C}}_t\end{array}}{max}\underset{{\sigma }_t>-{\mu }_p}{sup}\frac{r_t}{p}{\mathit{h}}^{\top }\mathit{u}+\frac{q_t}{p}{\mathit{u}}^{\top }{\mathit{s}}_t+\frac{\lambda }{2}{q}_t^2+\frac{\lambda }{2}{r}_t^2\hfill \\ \hfill & -\frac{1}{p}\sum _{i=1}^{n_t}{\ell }^{★}\left(y_i;u_i\right)+\frac{1}{2}{q}_t^2{V}_{p,t}-q_t{V}_{p,ts}-\frac{{\parallel \mathit{u}\parallel }^2}{2p}{T}_{p,g}\left({\sigma }_t\right)\hfill \\ \hfill & -\frac{{\sigma }_t{r}_t^2}{2}-\frac{1}{2}{q}_t^2{T}_{p,t}\left({\sigma }_t\right)-\frac{1}{2}{T}_{p,s}\left({\sigma }_t\right)+q_t{T}_{p,ts}\left({\sigma }_t\right),\hfill \end{array}$$

(45)

where the set ${\mathcal{T}}_2$ has the same definition as the set ${\mathcal{T}}_1$ except that we replace $\parallel {\mathit{r}}_t\parallel$ with $r_t$. Here, the sequence of random functions $T_{p,g}(.)$, $T_{p,t}(.)$, $T_{p,s}(.)$, and $T_{p,ts}(.)$ can be expressed as follows:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{T}_{p,g}\left({\sigma }_t\right)=\frac{1}{p}{\mathit{g}}^{\top }{\mathit{B}}_{{\mathit{\xi }}_t}^{\perp }{[{\mathsf{\Lambda }}^{\perp }+{\sigma }_t{\mathit{I}}_{p-1}]}^{-1}{\left({\mathit{B}}_{{\mathit{\xi }}_t}^{\perp }\right)}^{\top }\mathit{g}\hfill \\ T_{p,t}\left({\sigma }_t\right)={\mathit{\xi }}_t^{\top }\mathsf{\Lambda }{\mathit{B}}_{{\mathit{\xi }}_t}^{\perp }{[{\mathsf{\Lambda }}^{\perp }+{\sigma }_t{\mathit{I}}_{p-1}]}^{-1}{\left({\mathit{B}}_{{\mathit{\xi }}_t}^{\perp }\right)}^{\top }\mathsf{\Lambda }{\mathit{\xi }}_t\hfill \\ T_{p,s}\left({\sigma }_t\right)={\widehat{\mathit{w}}}_s^{\top }\mathsf{\Lambda }{\mathit{B}}_{{\mathit{\xi }}_t}^{\perp }{[{\mathsf{\Lambda }}^{\perp }+{\sigma }_t{\mathit{I}}_{p-1}]}^{-1}{\left({\mathit{B}}_{{\mathit{\xi }}_t}^{\perp }\right)}^{\top }\mathsf{\Lambda }{\widehat{\mathit{w}}}_s\hfill \\ T_{p,ts}\left({\sigma }_t\right)={\mathit{\xi }}_t^{\top }\mathsf{\Lambda }{\mathit{B}}_{{\mathit{\xi }}_t}^{\perp }{[{\mathsf{\Lambda }}^{\perp }+{\sigma }_t{\mathit{I}}_{p-1}]}^{-1}{\left({\mathit{B}}_{{\mathit{\xi }}_t}^{\perp }\right)}^{\top }\mathsf{\Lambda }{\widehat{\mathit{w}}}_s.\hfill \end{array}\right.\end{array}$$

Note that the formulation in (45) is obtained after dropping terms that converge in probability to zero. This simplification can be justified using a similar analysis to that in [20] (Lemma 3). The main idea in [20] (Lemma 3) is to show that both loss functions converge uniformly to the same limit.

Next, the objective is to simplify the obtained AO formulation over the optimization vector $\mathit{u}\in {\mathbb{R}}^{n_t}$. Based on the property stated in [20] (Lemma 4), the optimization over the vector $\mathit{u}$ can be expressed as follows:

$$\begin{array}{cc}\hfill I_p^{★}& =\underset{\begin{array}{c}\mathit{u}\in {\mathcal{C}}_t\end{array}}{max}{r}_t{\mathit{h}}^{\top }\mathit{u}+q_t{\mathit{u}}^{\top }{\mathit{s}}_t-\sum _{i=1}^{n_t}{\ell }^{★}\left(y_i;u_i\right)-\frac{{\parallel \mathit{u}\parallel }^2}{2}{T}_{p,g}\left({\sigma }_t\right)\hfill \\ \hfill & =\sum _{i=1}^{n_t}{\mathcal{M}}_{\ell (y_i,.)}\left(r_t{h}_i+q_t{s}_{t,i};T_{p,g}\left({\sigma }_t\right)\right).\hfill \end{array}$$

This result is valid on events with probability going to one as p goes to $+\infty$. Here, the function ${\mathcal{M}}_{\ell (y_i,.)}$ is the Moreau envelope function defined in (15). The proof of this property is omitted since it follows the same ideas as [20] (Lemma 4). The main idea in [20] (Lemma 4) is to use Assumption 3 to show that the optimal solution of the unconstrained version of the maximization problem is bounded asymptotically and then to use the property introduced in [25] (Example 11.26) to complete the proof. Now, our auxiliary formulation can be asymptotically simplified to a scalar optimization problem as follows:

$$\begin{array}{cc}\hfill & \underset{\begin{array}{c}(q_t,r_t)\in {\mathcal{T}}_2\end{array}}{min}\underset{{\sigma }_t>-{\mu }_p}{sup}\frac{\lambda }{2}(q_t^2+r_t^2)-\frac{{\sigma }_t{r}_t^2}{2}-\frac{1}{2}{q}_t^2{Z}_{p,t}\left({\sigma }_t\right)-\frac{1}{2}{Z}_{p,s}\left({\sigma }_t\right)\hfill \\ \hfill & +\frac{1}{p}\sum _{i=1}^{n_t}{\mathcal{M}}_{\ell (y_i,.)}\left(r_t{h}_i+q_t{s}_{t,i};T_{p,g}\left({\sigma }_t\right)\right)+q_t{Z}_{p,ts}\left({\sigma }_t\right),\hfill \end{array}$$

(46)

where the functions $Z_{p,t}(·)$, $Z_{p,ts}(·)$, and $Z_{p,s}(·)$ are defined as follows:

$$\begin{array}{cc}\hfill & Z_{p,t}\left({\sigma }_t\right)=T_{p,t}\left({\sigma }_t\right)-V_{p,t},Z_{p,ts}\left({\sigma }_t\right)=T_{p,ts}\left({\sigma }_t\right)-V_{p,ts}\hfill \end{array}$$

(47)

$$\begin{array}{cc}\hfill & Z_{p,s}\left({\sigma }_t\right)=T_{p,s}\left({\sigma }_t\right)-V_{p,s},\mathrm{where}{V}_{p,s}={\widehat{\mathit{w}}}_s^{\top }\mathsf{\Lambda }{\widehat{\mathit{w}}}_s.\hfill \end{array}$$

(48)

Note that the auxiliary formulation in (46) now has scalar optimization variables. Then, it remains to study its asymptotic properties. We refer to this problem as the target scalar formulation.

7.3.3. Asymptotic Analysis of the Target Scalar Formulation

In this part, we study the asymptotic properties of the target scalar formulation expressed in (46). We start our analysis by studying the asymptotic properties of the sequence of random functions $T_{p,g}(.)$, $Z_{p,t}(.)$, $Z_{p,s}(.)$, and $Z_{p,ts}(.)$ as given in the following lemma.

Lemma 2

(Asymptotic Properties). First, the random variable ${\mu }_p$ converges in probability to ${\mu }_{\mathit{min}}$, where ${\mu }_{\mathit{min}}$ is defined in Assumption 5. For any fixed $\sigma >0$, the following convergence in probability holds true:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{Z}_{p,t}(\sigma -{\mu }_p)\stackrel{p\to +\infty }{\to }{Z}_t(\sigma -{\mu }_{\min })\hfill \\ Z_{p,ts}(\sigma -{\mu }_p)\stackrel{p\to +\infty }{\to }{Z}_{ts}(\sigma -{\mu }_{\min })\hfill \\ Z_{p,s}(\sigma -{\mu }_p)\stackrel{p\to +\infty }{\to }{Z}_s(\sigma -{\mu }_{\min })\hfill \\ T_{p,g}(\sigma -{\mu }_p)\stackrel{p\to +\infty }{\to }{T}_g(\sigma -{\mu }_{\min })=T_1(\sigma -{\mu }_{\min }).\hfill \end{array}\right.\end{array}$$

Here, the deterministic functions $Z_t(.)$, $Z_{ts}(.)$, $Z_s(.)$, $T_1(.)$, and $T_3(.)$ are defined as follows:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{Z}_t\left(\sigma \right)=\sigma -1/T_1\left(\sigma \right),Z_{ts}\left(\sigma \right)=\rho q_s^{★}{Z}_t\left(\sigma \right)\hfill \\ Z_s\left(\sigma \right)=((1-{\rho }^2){\left(q_s^{★}\right)}^2+{\left(r_s^{★}\right)}^2)T_3\left(\sigma \right)+{\left(\rho q_s^{★}\right)}^2{Z}_t\left(\sigma \right)\hfill \\ T_1\left(\sigma \right)={\mathbb{E}}_{\mu }\left[1/(\mu +\sigma )\right],T_3\left(\sigma \right)=-{\mathbb{E}}_{\mu }\left[\mu \sigma /(\mu +\sigma )\right].\hfill \end{array}\right.\end{array}$$

Moreover, the constants $q_s^{★}$ and $r_s^{★}$ are optimal solutions of the source asymptotic formulation defined in (32).

A detailed proof of Lemma 2 is provided in Appendix C. Now that we obtained the asymptotic properties of the sequence of random variables, it remains to study the asymptotic properties of the optimal cost and optimal solution set of the scalar formulation in (46). To state our first asymptotic result, we define the following deterministic optimization problem:

$$\begin{array}{cc}\hfill & \underset{\begin{array}{c}(q_t,r_t)\in {\mathcal{T}}_2\end{array}}{min}\underset{{\sigma }_t>-{\mu }_{\min }}{sup}\frac{\lambda }{2}(q_t^2+r_t^2)-\frac{{\sigma }_t{r}_t^2}{2}-\frac{1}{2}{Z}_s\left({\sigma }_t\right)-\frac{1}{2}{q}_t^2{Z}_t\left({\sigma }_t\right)\hfill \\ \hfill & +{\alpha }_t\mathbb{E}\left[{\mathcal{M}}_{\ell (Y_t,.)}\left(r_t{H}_t+q_t{S}_t;T_g\left({\sigma }_t\right)\right)\right]+q_t{Z}_{ts}\left({\sigma }_t\right),\hfill \end{array}$$

(49)

where $H_t$ and $S_t$ are two independent standard Gaussian random variables and $Y_t=\phi \left(S_t\right)$. Here, the function ${\mathcal{M}}_{\ell (Y_t,.)}$ denotes the Moreau envelope function defined in (15) and the expectation is taken over the random variables $H_t$ and $S_t$, and the possibly random function $\phi (.)$. Now, we are ready to state our asymptotic property of the cost function of (46).

Lemma 3

(Cost Function of the Traget AO Formulation). Define ${\mathcal{O}}_{p,t}(.)$ as the loss function of the target scalar optimization problem given in (46). Additionally, define ${\mathcal{O}}_t(.)$ as the cost function of the deterministic formulation in (49). Then, the following convergence in probability holds true:

$$\begin{array}{c}\hfill {\mathcal{O}}_{p,t}(q_t,r_t,{\sigma }_t-{\mu }_p)\stackrel{p\to +\infty }{\to }{\mathcal{O}}_t(q_t,r_t,{\sigma }_t-{\mu }_{\mathit{min}}),\end{array}$$

(50)

for any fixed feasible $q_t$, $r_t$, and ${\sigma }_t>0$.

The proof of the asymptotic property stated in Lemma 3 uses the asymptotic results stated in Lemma 2. Moreover, it uses the weak law of large numbers to show that the empirical mean of the Moreau envelope concentrates around its expected value. Based on Assumption 3, one can see that the following pointwise convergence is valid:

$$\begin{array}{c}\hfill \frac{1}{n_t}\sum _{i=1}^{n_t}{\mathcal{M}}_{\ell (y_i,.)}\left(r_t{h}_i+q_t{s}_{t,i};x\right)\stackrel{p\to +\infty }{\to }\mathbb{E}\left[{\mathcal{M}}_{\ell (Y,.)}\left(r_{t}H+q_{t}S;x\right)\right],\end{array}$$

where H and S are independent standard Gaussian random variables and $Y=\phi \left(S\right)$. The above property is valid for any $x>0$, $r_t\ge 0$, and $q_t$. Based on [25] (Theorem 2.26), the Moreau envelope function is convex and continuously differentiable with respect to $x>0$. Combining this with [29] (Theorem 7.46), the above asymptotic function is continuous in $x>0$. Then, using Lemma 2, the uniform convergence, and the continuity property, we conclude that the empirical average of the Moreau envelope converges in probability to the following function:

$$\begin{array}{c}\hfill \mathbb{E}\left[{\mathcal{M}}_{\ell (Y,.)}\left(r_{t}H+q_{t}S;T_g({\sigma }_t-{\mu }_{\min })\right)\right],\end{array}$$

(51)

for any fixed feasible $q_t$, $r_t$, and ${\sigma }_t>0$. This completes the proof of Lemma 3.

Before continuing our analysis, we provide the convexity properties of the cost function of the deterministic problem in (49) in the following lemma.

Lemma 4

(Strong Conexity). Define ${\mathcal{O}}_t(·,·,·)$ as the cost function of the optimization problem in (49). Then, ${\mathcal{O}}_t(·,·,·)$ is concave in the maximization variable ${\sigma }_t$ for any fixed feasible $(q_t,r_t)$. Moreover, define the function ${\mathcal{O}}_t(·,·)$ as follows:

$$\begin{array}{c}\hfill {\mathcal{O}}_t(q_t,r_t)=\underset{{\sigma }_t>-{\mu }_{\mathit{min}}}{sup}f(q_t,r_t,{\sigma }_t).\end{array}$$

(52)

Then, the function ${\mathcal{O}}_t(·,·)$ is strongly convex in the minimization variables $(q_t,r_t)$.

The proof of Lemma 4 is provided in Appendix D. Now, we use these properties to show that the optimal solution set of the formulation in (46) converges in probability to the optimal solution set of the formulation in (49).

Lemma 5

(Consistency of the Target AO Formulation). Define ${\mathcal{P}}_{p,t}$ and ${\mathcal{P}}_t$ as the optimal set of $(q_t,r_t)$ of the optimization problems formulated in (46) and (49). Moreover, define ${\mathcal{O}}_{p,t}^{★}$ and ${\mathcal{O}}_t^{★}$ as the optimal cost values of the optimization problems formulated in (46) and (49). Then, the following converges in probability holds true:

$$\begin{array}{c}\hfill {\mathcal{O}}_{p,t}^{★}\stackrel{p\to +\infty }{\to }{\mathcal{O}}_t^{★},\mathbb{D}({\mathcal{P}}_{p,t},{\mathcal{P}}_t)\stackrel{p\to +\infty }{\to }0,\end{array}$$

(53)

where $\mathbb{D}(\mathcal{A},\mathcal{B})$ denotes the deviation between the sets $\mathcal{A}$ and $\mathcal{B}$ and is defined as $\mathbb{D}(\mathcal{A},\mathcal{B})={sup}_{{\mathit{c}}_1\in \mathcal{A}}{inf}_{{\mathit{c}}_2\in \mathcal{B}}\parallel {\mathit{c}}_1-{\mathit{c}}_2\parallel$.

The stated result can be proven by first observing that the loss function ${\mathcal{O}}_t(.)$ corresponding to the deterministic formulation in (49) satisfies the following:

$$\begin{array}{c}\hfill \underset{{\sigma }_t\to +\infty }{lim}{\mathcal{O}}_t(q_t,r_t,{\sigma }_t-{\mu }_{\min })=-\infty \end{array}$$

(54)

for any $r_t>0$ and any fixed $q_t$. Combining this with the convergence result in Lemma 3, ref. [17] (Lemma B.1), and [17] (Lemma B.2), we obtain the following asymptotic result:

$$\begin{array}{c}\hfill \underset{{\sigma }_t>0}{sup}{\mathcal{O}}_{p,t}(q_t,r_t,{\sigma }_t-{\mu }_p)\stackrel{p\to +\infty }{\to }\underset{{\sigma }_t>0}{sup}{\mathcal{O}}_t(q_t,r_t,{\sigma }_t-{\mu }_{\min }).\end{array}$$

Here, the results in [17] (Lemma B.1) and [17] (Lemma B.2) provide convergence properties of minimization problems over open sets. Note that, if $r_t=0$, the supremum in the above convergence result occurs at ${\sigma }_t\to +\infty$. However, it can be checked that the above convergence result still holds. Based on Lemma 4, the cost function of the minimization problem in (49) is strongly convex in $(q_t,r_t)$. Moreover, the feasibility set of the minimization problem is convex and compact. Additionally, the cost function of the minimization problem in (49) is continuous in the feasibility set. Then, using the results in [30] (Theorem II.1) and [31] (Theorem 2.1), we obtain the convergence properties stated in Lemma 5. Here, the results in [30] (Theorem II.1) and [31] (Theorem 2.1) provide uniform convergence and consistency properties of convex optimization problems.

Now that we obtained the asymptotic problem, it remains to study the asymptotic properties of the training and generalization errors corresponding to the target formulation in (8).

7.3.4. Specialization to Hard Formulation

Before starting the analysis of the generalization error, we specialize our general analysis to the hard transfer formulation. First, note that $\delta =1$ implies that the hard transfer formulation is equivalent to the source formulation. Next, we assume that $\delta <1$. To obtain the asymptotic limit of the hard formulation, we specialize the general results in (49) to the following probability distribution:

$$\begin{array}{c}\hfill {\mathbb{P}}_p\left(\mu \right)=\left\{\begin{array}{cc}0\hfill & \mathrm{with}\mathrm{probability}(1-\delta )\hfill \\ +\infty \hfill & \mathrm{with}\mathrm{probability}\delta .\hfill \end{array}\right.\end{array}$$

(55)

Note that the probability distribution in (55) satisfies Assumption 5. Then, the asymptotic limit of the soft formulation corresponding to the probability distribution ${\mathbb{P}}_{\mu }(.)$, defined in (55), can be expressed as follows:

$$\begin{array}{cc}\hfill \underset{\begin{array}{c}(q_t,r_t)\in {\mathcal{T}}_2\end{array}}{min}\underset{{\sigma }_t>0}{sup}& \frac{\lambda }{2}(q_t^2+r_t^2)+\frac{{\sigma }_t\delta }{2}\left((1-{\rho }^2){\left(q_s^{★}\right)}^2+{\left(r_s^{★}\right)}^2\right)\hfill \\ \hfill & +{\alpha }_t\mathbb{E}\left[{\mathcal{M}}_{\ell (Y_t,.)}\left(r_t{H}_t+q_t{S}_t;\frac{1-\delta }{{\sigma }_t}\right)\right]-\frac{{\sigma }_t{r}_t^2}{2}\hfill \\ \hfill & +\frac{{\sigma }_t\delta }{2(1-\delta )}{\left(q_t-\rho q_s^{★}\right)}^2.\hfill \end{array}$$

(56)

This shows that the asymptotic limit of the hard formulation is the deterministic problem (56).

7.3.5. Asymptotic Analysis of the Training and Generalization Errors

First, the generalization error corresponding to the target task is given by

$$\begin{array}{cc}\hfill {\mathcal{E}}_{\mathrm{test}}& =\frac{1}{4^{\upsilon }}\mathbb{E}\left[{\left(\phi \left({\mathit{a}}_{t,\mathrm{new}}^{\top }{\mathit{\xi }}_t\right)-\widehat{\phi }\left({\widehat{\mathit{w}}}_t^{\top }{\mathit{a}}_{t,\mathrm{new}}\right)\right)}^2\right],\hfill \end{array}$$

(57)

where ${\mathit{a}}_{t,\mathrm{new}}$ is an unseen target feature vector. Now, consider the following two random variables

$$\begin{array}{c}\hfill {\nu }_1={\mathit{a}}_{t,\mathrm{new}}^{\top }{\mathit{\xi }}_t,\mathrm{and}{\nu }_2={\widehat{\mathit{w}}}_t^{\top }{\mathit{a}}_{t,\mathrm{new}}.\end{array}$$

Given ${\widehat{\mathit{w}}}_t$ and ${\mathit{\xi }}_t$, the random variables ${\nu }_1$ and ${\nu }_2$ have a bivaraite Gaussian distribution with zero mean vector and covariance matrix given as follows:

$$\begin{array}{c}\hfill {\mathit{C}}_p=\left[\begin{array}{cc}{\parallel {\mathit{\xi }}_t\parallel }^2& {\mathit{\xi }}_t^{\top }{\widehat{\mathit{w}}}_t\\ {\mathit{\xi }}_t^{\top }{\widehat{\mathit{w}}}_t& {\parallel {\widehat{\mathit{w}}}_t\parallel }^2\end{array}\right].\end{array}$$

(58)

To precisely analyze the asymptotic behavior of the generalization error, it suffices to analyze the properties of the covariance matrix ${\mathit{C}}_p$. Define the random variables ${\widehat{q}}_{p,t}^{★}$ and ${\widehat{r}}_{p,t}^{★}$ for the target task as follows:

$$\begin{array}{cc}\hfill & {\widehat{q}}_{p,t}^{★}={\mathit{\xi }}_t^{\top }{\widehat{\mathit{w}}}_t,\mathrm{and}{\widehat{r}}_{p,t}^{★}=\parallel {\left({\mathit{B}}_{{\mathit{\xi }}_t}^{\perp }\right)}^{\top }{\widehat{\mathit{w}}}_t\parallel ,\hfill \end{array}$$

(59)

where ${\mathit{B}}_{{\mathit{\xi }}_t}^{\perp }$ is defined in Section 7.3.2. Then, the covariance matrix ${\mathit{C}}_p$ given in (58) can be expressed as follows:

$$\begin{array}{c}\hfill \left[\begin{array}{cc}1& {\widehat{q}}_{p,t}^{★}\\ {\widehat{q}}_{p,t}^{★}& {\left({\widehat{q}}_{p,t}^{★}\right)}^2+{\left({\widehat{r}}_{p,t}^{★}\right)}^2\end{array}\right].\end{array}$$

Hence, to study the asymptotic properties of the generalization error, it suffices to study the asymptotic properties of the random quantities ${\widehat{q}}_{p,t}^{★}$ and ${\widehat{r}}_{p,t}^{★}$.

Lemma 6

(Consistency of the Target Formulation). The random quantities ${\widehat{q}}_{p,t}^{★}$ and ${\widehat{r}}_{p,t}^{★}$ satisfy the following asymptotic properties:

$$\begin{array}{c}\hfill {\widehat{q}}_{p,t}^{★}\stackrel{p\to +\infty }{\to }{q}_t^{★},\mathit{and}{\widehat{r}}_{p,t}^{★}\stackrel{p\to +\infty }{\to }{r}_t^{★},\end{array}$$

where $q_t^{★}$ and $r_t^{★}$ are the optimal solutions of the deterministic formulation stated in (49).

To prove the above asymptotic result, we define ${\tilde{q}}_{p,t}^{★}$ and ${\tilde{r}}_{p,t}^{★}$ as follows:

$$\begin{array}{cc}\hfill & {\tilde{q}}_{p,t}^{★}={\mathit{\xi }}_t^{\top }{\tilde{\mathit{w}}}_t,\mathrm{and}{\tilde{r}}_{p,t}^{★}=\parallel {\left({\mathit{B}}_{{\mathit{\xi }}_t}^{\perp }\right)}^{\top }{\tilde{\mathit{w}}}_t\parallel ,\hfill \end{array}$$

(60)

where ${\tilde{\mathit{w}}}_t$ is the optimal solution of the auxiliary formulation in (39). Given the result in Lemma 5 and the analysis in Section 7.3.2 and Section 7.3.3, the convergence result in Lemma 5 is also satisfied by our auxiliary formulation in (39), i.e.,

$$\begin{array}{c}\hfill {\tilde{q}}_{p,t}^{★}\stackrel{p\to +\infty }{\to }{q}_t^{★},\mathrm{and}{\tilde{r}}_{p,t}^{★}\stackrel{p\to +\infty }{\to }{r}_t^{★}.\end{array}$$

The rest of the proof of the convergence result stated in Lemma 6 is based on the CGMT framework, i.e., Theorem 2. Specifically, it follows after showing that the assumptions in Theorem 2 are all satisfied. First, we define the set ${\mathcal{S}}_p$ in Theorem 2 as follows:

$$\begin{array}{c}\hfill {\mathcal{S}}_p=\{\mathit{w}\in {\mathbb{R}}^p:|{\mathit{\xi }}_t^{\top }\mathit{w}-q_t^{★}|<ϵ\}\cup \{\mathit{w}\in {\mathbb{R}}^p:|\parallel {\left({\mathit{B}}_{{\mathit{\xi }}_t}^{\perp }\right)}^{\top }\mathit{w}\parallel -r_t^{★}|<ϵ\},\end{array}$$

(61)

where $q_t^{★}$ and $r_t^{★}$ are the optimal solutions of the deterministic formulation stated in (49). Note that the cost function of the problem (49) is strongly convex in the minimization variables. Based on the analysis in the previous sections, note that the feasibility sets of the problems defined in Theorem 2 are compact asymptotically. Moreover, the analysis in the previous sections shows that there exists a constant $\varphi$ such that the optimal cost ${\varphi }_p$ defined in Theorem 2 converges in probability to $\varphi$ as p goes to $+\infty$. Additionally, the same analysis in the previous sections shows that there exists a constant ${\varphi }^c$ such that the optimal cost ${\varphi }_p^c$ defined in Theorem 2 converges in probability to ${\varphi }^c$ as p goes to $+\infty$. The strong convexity property of the cost function of the optimization problem in (49) can then be used to show that there exists $\zeta >0$ such that ${\varphi }^c>\varphi +\zeta$. This implies that the second assumption in Theorem 2 is satisfied for the considered set ${\mathcal{S}}_p$ and any fixed $ϵ>0$. This then shows that the convergence results in Lemma 6 are all satisfied.

Note that the CGMT framework applied to prove Lemma 6 also shows that the optimal cost value of the soft target formulation in (8) converges in probability to the optimal cost value of the deterministic formulation given in (49). Combining this with the result in Lemma 6 shows the convergence property of the training error stated in (17). Now, it remains to show the convergence of the generalization error. It suffices to show that the generalization error defined in (57) is continuous in the quantities ${\widehat{q}}_{p,t}^{★}$ and ${\widehat{r}}_{p,t}^{★}$. This follows based on Assumption 4 and the continuity under integral sign property [32]. This shows the convergence result in (18), which completes the proof of Theorem 1. Note that the above analysis of the soft target formulation in (8) is valid for any choice of $C_{q_t}$ and $C_r$ that satisfy the result in Lemma 1. One can ignore these bounds given the convexity properties of the deterministic formulation in (49). This leads to the scalar formulations introduced in (16) and (19).

7.4. Phase Transitions in Hard Formulation

In this part, we provide a rigorous proof of Proposition 1. Here, we consider the squared loss function. In this case, the deterministic source formulation given in (14) can be simplified as follows:

$$\begin{array}{cc}\hfill \underset{\begin{array}{c}{q}_s,r_s\ge 0\end{array}}{min}& \frac{1}{2}max{\left\{-r_s+\sqrt{{\alpha }_s}{(q_s^2+r_s^2+v_s-2q_s{c}_s)}^{\frac{1}{2}},0\right\}}^2\hfill \\ \hfill & +\frac{\lambda }{2}(q_s^2+r_s^2),\hfill \end{array}$$

(62)

where the constants $v_s$ and $c_s$ are defined as $v_s=\mathbb{E}\left[Y_s^2\right]$ and $c_s=\mathbb{E}\left[S_s{Y}_s\right]$, $Y_s=\phi \left(S_s\right)$, and $S_s$ is a standard Gaussian random variable. Additionally, the target scalar formulation given in (16) can be simplified as follows:

$$\begin{array}{cc}\hfill \underset{\begin{array}{c}{q}_t,r_t\ge 0\end{array}}{min}\underset{{\sigma }_t>0}{sup}& \frac{\lambda }{2}(q_t^2+r_t^2)+\frac{{\sigma }_t\delta }{2}\left((1-{\rho }^2){\left(q_s^{★}\right)}^2+{\left(r_s^{★}\right)}^2\right)\hfill \\ \hfill & +\frac{{\alpha }_t{\sigma }_t}{2(1-\delta )+2{\sigma }_t}(r_t^2+q_t^2+v_t-2q_t{c}_t)-\frac{{\sigma }_t{r}_t^2}{2}\hfill \\ \hfill & +\frac{{\sigma }_t\delta }{2(1-\delta )}{\left(q_t-\rho q_s^{★}\right)}^2,\hfill \end{array}$$

(63)

where the constants $v_t$ and $c_t$ are defined as $v_t=\mathbb{E}\left[Y_t^2\right]$ and $c_t=\mathbb{E}\left[Y_t{S}_t\right]$, $Y_t=\phi \left(S_t\right)$, and $S_t$ is a standard Gaussian random variable. Under the conditions stated in Proposition 1, the source deterministic formulation given in (62) can be simplified as follows:

$$\begin{array}{cc}\hfill \underset{\begin{array}{c}{q}_s,r_s\ge 0\end{array}}{min}& -r_s+\sqrt{{\alpha }_s}{(q_s^2+r_s^2+v_s-2q_s{c}_s)}^{\frac{1}{2}}.\hfill \end{array}$$

(64)

Note that one can easily solve the variables $q_s$ and $r_s$. Specifically, the optimal solutions of (64) can be expressed as follows:

$$\begin{array}{c}\hfill q_s^{★}=c_s,\mathrm{and}{r}_s^{★}=\sqrt{v_s-c_s^2}/\sqrt{{\alpha }_s-1}.\end{array}$$

(65)

Moreover, the target deterministic formulation given in (63) can be expressed as follows:

$$\begin{array}{cc}\hfill \underset{\begin{array}{c}{q}_t,r_t\ge 0\end{array}}{min}\underset{{\sigma }_t>0}{sup}& \frac{{\sigma }_t\delta }{2}{\beta }_2+\frac{{\alpha }_t{\sigma }_t}{2(1-\delta )+2{\sigma }_t}(r_t^2+q_t^2+v_t-2q_t{c}_t)-\frac{{\sigma }_t{r}_t^2}{2}\hfill \\ \hfill & +\frac{{\sigma }_t\delta }{2(1-\delta )}{\left(q_t-{\beta }_1\right)}^2,\hfill \end{array}$$

(66)

where ${\beta }_1$ and ${\beta }_2$ are given by

$$\begin{array}{c}\hfill {\beta }_1=\rho q_s^{★},{\beta }_2=\left((1-{\rho }^2){\left(q_s^{★}\right)}^2+{\left(r_s^{★}\right)}^2\right).\end{array}$$

(67)

Before solving the optimization problem in (66), we consider the following change in variable:

$$\begin{array}{c}\hfill x_t^2+r_t^2-\delta {\beta }_2-\frac{\delta }{1-\delta }{(q_t-{\beta }_1)}^2.\end{array}$$

(68)

Note that the above change in variable is valid since the formulation in (66) requires the left-hand side of (68) to be positive. Therefore, the formulation in (66) can be expressed in terms of $x_t$ instead of $r_t$ as follows:

$$\begin{array}{cc}\hfill \underset{\begin{array}{c}{q}_t,x_t\ge 0\end{array}}{min}\underset{{\sigma }_t>0}{sup}& \frac{{\alpha }_t{\sigma }_t}{2(1-\delta )+2{\sigma }_t}\left(x_t^2+\delta {\beta }_2+\frac{\delta }{1-\delta }{(q_t-{\beta }_1)}^2+q_t^2+v_t-2q_t{c}_t\right)-\frac{{\sigma }_t{x}_t^2}{2}.\hfill \end{array}$$

(69)

Now, it can be easily checked that the above optimization problem can be solved over the variable ${\sigma }_t$ to give the following formulation:

$$\begin{array}{cc}\hfill \underset{\begin{array}{c}{q}_t,x_t\ge 0\end{array}}{min}& \frac{1}{2}max{\left\{-x_t\sqrt{1-\delta }+\sqrt{{\alpha }_t}{\left(x_t^2+\delta {\beta }_2+\frac{\delta }{1-\delta }{(q_t-{\beta }_1)}^2+q_t^2+v_t-2q_t{c}_t\right)}^{\frac{1}{2}},0\right\}}^2.\hfill \end{array}$$

It is now clear that one can solve the problem in (69) in closed form. Moreover, it can be easily checked that the optimal solutions of the optimization problem (66) can be expressed as follows:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{q}_t^{★}=(1-\delta )c_t+\delta {\beta }_1\hfill \\ {\left(r_t^{★}\right)}^2=\frac{1-\delta }{{\alpha }_t+\delta -1}\left((\delta -1)c_t^2+\delta {\beta }_1^2+\delta {\beta }_2+v_t-2\delta {\beta }_1{c}_t\right)+\delta {\beta }_2+\delta (1-\delta ){(c_t-{\beta }_1)}^2.\hfill \end{array}\right.\end{array}$$

Then, the asymptotic limit of the generalization error corresponding to the hard formulation can be determined in closed-form. Given that the source and target models given in (1) and (2) use the same data-generating function, the constants $v_t$, $c_t$, $v_s$, and $c_s$ are all equal. We express them as v and c in the rest of the proof.

Next, we assume that the function $\widehat{\phi }(.)$ is the identity function. Based on the asymptotic result stated in Corollary 1, the asymptotic limit of the generalization error corresponding to the hard formulation can be expressed as follows:

$$\begin{array}{cc}\hfill {\mathcal{E}}_{\mathrm{test}}& =v-2c{q}_t^{★}+{\left(q_t^{★}\right)}^2+{\left(r_t^{★}\right)}^2.\hfill \end{array}$$

It can be easily checked that the generalization error can be express as follows:

$$\begin{array}{cc}\hfill {\mathcal{E}}_{\mathrm{test}}& =\frac{{\alpha }_t}{{\alpha }_t+\delta -1}\left(\delta \{{(c-{\beta }_1)}^2+{\beta }_2\}+(v-c^2)\right).\hfill \end{array}$$

(70)

Note that the generalization error obtained above depends explicitly on $\delta$. Now, it suffices to study the derivative of ${\mathcal{E}}_{\mathrm{test}}$ to find the properties of the optimal transfer rate $\delta$ that minimizes the generalization error. Note that the derivative can be expressed as follows:

$$\begin{array}{c}\hfill {\mathcal{E}}_{\mathrm{test}}^{\prime }\left(\delta \right)=\frac{({\alpha }_t-1)\{{(c-{\beta }_1)}^2+{\beta }_2\}-(v-c^2)}{{({\alpha }_t+\delta -1)}^2}.\end{array}$$

(71)

This shows that the derivative of the generalization error has the same sign as the numerator. This means that the optimal transfer rate satisfies the following:

$$\begin{array}{c}\hfill {\delta }^{★}=\left\{\begin{array}{cc}1\hfill & \mathrm{if}{Z}_t<0\hfill \\ 0\hfill & \mathrm{if}{Z}_t>0\hfill \\ [0 1]\hfill & \mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(72)

where $Z_t$ is given by

$$\begin{array}{c}\hfill Z_t=({\alpha }_t-1)\{{(c-{\beta }_1)}^2+{\beta }_2\}-(v-c^2).\end{array}$$

(73)

It can be easily shown that the condition in (72) can be expressed as the one given in (20). This completes the proof of Proposition 1.

7.5. Sufficient Condition for the Hard Formulation

In this part, we provide a rigorous proof to Proposition 2. Suppose that the assumptions in Proposition 2 are all satisfied. Additionally, we assume that the function $\widehat{\phi }(.)$ is the sign function. Based on the asymptotic result stated in Corollary 1, the asymptotic limit of the generalization error corresponding to the hard formulation can be expressed as follows:

$$\begin{array}{cc}\hfill {\mathcal{E}}_{\mathrm{test}}\left(\delta \right)& =\frac{1}{\pi }\mathrm{acos}\left(\frac{q_t\left(\delta \right)}{\sqrt{{\left(q_t\left(\delta \right)\right)}^2+{\left(r_t\left(\delta \right)\right)}^2}}\right),\hfill \end{array}$$

(74)

where $q_t\left(\delta \right)$ and $r_t\left(\delta \right)$ are optimal solutions to the deterministic problem in (19) for fixed $\delta$. A simple sufficient condition for positive transfer is when ${\mathcal{E}}_{\mathrm{test}}\left(\delta \right)$ is decreasing at $\delta =0$. This means that there exists some $\delta >0$ such that the transfer learning method introduced in (6) is better than the standard method when the following function increases at $\delta =0$:

$$\begin{array}{c}\hfill g\left(\delta \right)=\frac{q_t\left(\delta \right)}{\sqrt{{\left(q_t\left(\delta \right)\right)}^2+{\left(r_t\left(\delta \right)\right)}^2}}.\end{array}$$

(75)

After computing the derivative of the function $g(·)$ at zero, one can see that the transfer learning method introduced in (6) is better than the standard method when the following condition is true:

$$\begin{array}{c}\hfill q_t^{\prime }\left(0\right)r_t\left(0\right)-q_t\left(0\right)r_t^{\prime }\left(0\right)>0,\end{array}$$

(76)

where $q_t\left(0\right)$ and $r_t\left(0\right)$ denote the optimal solutions of the standard learning formulation (i.e., $\delta =0$ in (19)). Additionally, $q_t^{\prime }\left(0\right)$ and $r_t^{\prime }\left(0\right)$ denote the derivative of the functions $q_t\left(\delta \right)$ and $r_t\left(\delta \right)$ at $\delta =0$. The above analysis shows that it suffices to find the values of $q_t^{\prime }\left(0\right)$ and $r_t^{\prime }\left(0\right)$ to fully characterize the sufficient condition in (76). Before stating our analysis, we define ${\beta }_1$ and ${\beta }_2$ as follows:

$$\begin{array}{c}\hfill {\beta }_1=\left((1-{\rho }^2){\left(q_s^{★}\right)}^2+{\left(r_s^{★}\right)}^2\right),{\beta }_2=\rho q_s^{★},\end{array}$$

(77)

where $q_s^{★}$ and $r_s^{★}$ are the optimal solutions of the deterministic source formulation given in (14).

Note that the optimal solution of the deterministic formulation in (19) satisfy the following system of equations:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{\alpha }_t\mathbb{E}\left(S{\mathcal{M}}_{\ell ,1}^{\prime }[r_t\left(\delta \right)H+q_t\left(\delta \right)S;\frac{1-\delta }{{\sigma }_t\left(\delta \right)}]\right)+\frac{\delta {\sigma }_t\left(\delta \right)}{1-\delta }(q_t\left(\delta \right)-{\beta }_2)+\lambda q_t\left(\delta \right)=0\hfill \\ {\alpha }_t\mathbb{E}\left(H{\mathcal{M}}_{\ell ,1}^{\prime }[r_t\left(\delta \right)H+q_t\left(\delta \right)S;\frac{1-\delta }{{\sigma }_t\left(\delta \right)}]\right)-{\sigma }_t\left(\delta \right)r_t\left(\delta \right)+\lambda r_t\left(\delta \right)=0\hfill \\ \frac{\delta }{2}{\beta }_1-\frac{{\alpha }_t(1-\delta )}{{\sigma }_t{\left(\delta \right)}^2}\mathbb{E}\left({\mathcal{M}}_{\ell ,2}^{\prime }[r_t\left(\delta \right)H+q_t\left(\delta \right)S;\frac{1-\delta }{{\sigma }_t\left(\delta \right)}]\right)-\frac{r_t{\left(\delta \right)}^2}{2}+\frac{\delta }{2(1-\delta )}{(q_t\left(\delta \right)-{\beta }_2)}^2=0.\hfill \end{array}\right.\end{array}$$

The derivative of the first equation at $\delta =0$ can be expressed as follows:

$$\begin{array}{cc}\hfill & {\alpha }_t\mathbb{E}\left((SH{r}^{\prime }\left(0\right)+S^2{q}^{\prime }\left(0\right)){\mathcal{M}}_{\ell ,1}^{″}[r_t\left(0\right)H+q_t\left(0\right)S;\frac{1}{{\sigma }_t\left(0\right)}]\right)+{\sigma }_t\left(0\right)(q_t\left(0\right)-{\beta }_2)\hfill \\ \hfill & -\frac{{\alpha }_t}{{\sigma }_t{\left(0\right)}^2}({\sigma }_t\left(0\right)+{\sigma }_t^{\prime }\left(0\right))\mathbb{E}\left(S{\mathcal{M}}_{\ell ,12}^{″}[r_t\left(0\right)H+q_t\left(0\right)S;\frac{1}{{\sigma }_t\left(0\right)}]\right)+\lambda q_t^{\prime }\left(0\right)=0,\hfill \end{array}$$

(78)

where $q_t\left(0\right)$, $r_t\left(0\right)$, and ${\sigma }_t\left(0\right)$ denote optimal solutions of the standard learning formulation (i.e., $\delta =0$ in (19)). This means that they are known. Moreover, $q_t^{\prime }\left(0\right)$, $r_t^{\prime }\left(0\right)$ and ${\sigma }_t^{\prime }\left(0\right)$ are unknown and denote the derivative of the functions $q_t\left(\delta \right)$, $r_t\left(\delta \right)$, and ${\sigma }_t\left(\delta \right)$ at $\delta =0$. Now, define the constants $I_{11}$, $I_{12}$, $I_{13}$, and $I_{14}$ as follows:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{I}_{11}={\alpha }_t\mathbb{E}\left(SH{\mathcal{M}}_{\ell ,1}^{″}[r_t\left(0\right)H+q_t\left(0\right)S;\frac{1}{{\sigma }_t\left(0\right)}]\right)\hfill \\ I_{12}={\alpha }_t\mathbb{E}\left(S^2{\mathcal{M}}_{\ell ,1}^{″}[r_t\left(0\right)H+q_t\left(0\right)S;\frac{1}{{\sigma }_t\left(0\right)}]\right)+\lambda \hfill \\ I_{13}=-\frac{{\alpha }_t}{{\sigma }_t{\left(0\right)}^2}\mathbb{E}\left(S{\mathcal{M}}_{\ell ,12}^{″}[r_t\left(0\right)H+q_t\left(0\right)S;\frac{1}{{\sigma }_t\left(0\right)}]\right)\hfill \\ I_{14}=-\frac{{\alpha }_t}{{\sigma }_t\left(0\right)}\mathbb{E}\left(S{\mathcal{M}}_{\ell ,12}^{″}[r_t\left(0\right)H+q_t\left(0\right)S;\frac{1}{{\sigma }_t\left(0\right)}]\right)+{\sigma }_t\left(0\right)(q_t\left(0\right)-{\beta }_2).\hfill \end{array}\right.\end{array}$$

(79)

This means that the equation in (78) can be expressed as follows:

$$\begin{array}{c}\hfill I_{11}{r}_t^{\prime }\left(0\right)+I_{12}{q}_t^{\prime }\left(0\right)+I_{13}{\sigma }_t^{\prime }\left(0\right)+I_{14}=0.\end{array}$$

(80)

Similarly, the derivative of the second equation at $\delta =0$ can be expressed as follows:

$$\begin{array}{cc}\hfill & {\alpha }_t\mathbb{E}\left((H^2{r}^{\prime }\left(0\right)+HS{q}^{\prime }\left(0\right)){\mathcal{M}}_{\ell ,1}^{″}[r_t\left(0\right)H+q_t\left(0\right)S;\frac{1}{{\sigma }_t\left(0\right)}]\right)-{\sigma }_t^{\prime }\left(0\right)r_t\left(0\right)-{\sigma }_t\left(0\right)r_t^{\prime }\left(0\right)\hfill \\ \hfill & -\frac{{\alpha }_t}{{\sigma }_t{\left(0\right)}^2}({\sigma }_t\left(0\right)+{\sigma }_t^{\prime }\left(0\right))\mathbb{E}\left(H{\mathcal{M}}_{\ell ,12}^{″}[r_t\left(0\right)H+q_t\left(0\right)S;\frac{1}{{\sigma }_t\left(0\right)}]\right)+\lambda r_t^{\prime }\left(0\right)=0.\hfill \end{array}$$

(81)

Now, define the constants $I_{21}$, $I_{22}$, $I_{23}$, and $I_{24}$ as follows:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{I}_{21}={\alpha }_t\mathbb{E}\left(H^2{\mathcal{M}}_{\ell ,1}^{″}[r_t\left(0\right)H+q_t\left(0\right)S;\frac{1}{{\sigma }_t\left(0\right)}]\right)-{\sigma }_t\left(0\right)+\lambda \hfill \\ I_{22}={\alpha }_t\mathbb{E}\left(HS{\mathcal{M}}_{\ell ,1}^{″}[r_t\left(0\right)H+q_t\left(0\right)S;\frac{1}{{\sigma }_t\left(0\right)}]\right)\hfill \\ I_{23}=-\frac{{\alpha }_t}{{\sigma }_t{\left(0\right)}^2}\mathbb{E}\left(H{\mathcal{M}}_{\ell ,12}^{″}[r_t\left(0\right)H+q_t\left(0\right)S;\frac{1}{{\sigma }_t\left(0\right)}]\right)-r_t\left(0\right)\hfill \\ I_{24}=-\frac{{\alpha }_t}{{\sigma }_t\left(0\right)}\mathbb{E}\left(H{\mathcal{M}}_{\ell ,12}^{″}[r_t\left(0\right)H+q_t\left(0\right)S;\frac{1}{{\sigma }_t\left(0\right)}]\right).\hfill \end{array}\right.\end{array}$$

(82)

This means that the equation in (81) can be expressed as follows:

$$\begin{array}{c}\hfill I_{21}{r}_t^{\prime }\left(0\right)+I_{22}{q}_t^{\prime }\left(0\right)+I_{23}{\sigma }_t^{\prime }\left(0\right)+I_{24}=0.\end{array}$$

(83)

Moreover, the derivative of the third equation at $\delta =0$ can be expressed as follows:

$$\begin{array}{cc}\hfill & \frac{{\beta }_1}{2}+\frac{{\alpha }_t}{{\sigma }_t{\left(0\right)}^3}({\sigma }_t\left(0\right)+2{\sigma }_t^{\prime }\left(0\right))\mathbb{E}\left({\mathcal{M}}_{\ell ,2}^{\prime }[r_t\left(0\right)H+q_t\left(0\right)S;\frac{1}{{\sigma }_t\left(0\right)}]\right)-r_t^{\prime }\left(0\right)r_t\left(0\right)\hfill \\ \hfill & -\frac{{\alpha }_t}{{\sigma }_t{\left(0\right)}^2}\mathbb{E}\left((H{r}^{\prime }\left(0\right)+S{q}^{\prime }\left(0\right)){\mathcal{M}}_{\ell ,21}^{″}[r_t\left(0\right)H+q_t\left(0\right)S;\frac{1}{{\sigma }_t\left(0\right)}]\right)+\frac{1}{2}{(q_t\left(0\right)-{\beta }_2)}^2\hfill \\ \hfill & +\frac{{\alpha }_t}{{\sigma }_t{\left(0\right)}^4}({\sigma }_t\left(0\right)+{\sigma }_t^{\prime }\left(0\right))\mathbb{E}\left({\mathcal{M}}_{\ell ,2}^{″}[r_t\left(0\right)H+q_t\left(0\right)S;\frac{1}{{\sigma }_t\left(0\right)}]\right)=0.\hfill \end{array}$$

(84)

We define the constants $I_{31}$, $I_{32}$, $I_{33}$, and $I_{34}$ as follows:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{I}_{31}=-\frac{{\alpha }_t}{{\sigma }_t{\left(0\right)}^2}\mathbb{E}\left(H{\mathcal{M}}_{\ell ,21}^{″}[r_t\left(0\right)H+q_t\left(0\right)S;\frac{1}{{\sigma }_t\left(0\right)}]\right)-r_t\left(0\right)\hfill \\ I_{32}=-\frac{{\alpha }_t}{{\sigma }_t{\left(0\right)}^2}\mathbb{E}\left(S{\mathcal{M}}_{\ell ,12}^{″}[r_t\left(0\right)H+q_t\left(0\right)S;\frac{1}{{\sigma }_t\left(0\right)}]\right)\hfill \\ I_{33}=\frac{2{\alpha }_t}{{\sigma }_t{\left(0\right)}^3}\mathbb{E}\left({\mathcal{M}}_{\ell ,2}^{\prime }[r_t\left(0\right)H+q_t\left(0\right)S;\frac{1}{{\sigma }_t\left(0\right)}]\right)+\frac{{\alpha }_t}{{\sigma }_t{\left(0\right)}^4}\mathbb{E}\left({\mathcal{M}}_{\ell ,2}^{″}[r_t\left(0\right)H+q_t\left(0\right)S;\frac{1}{{\sigma }_t\left(0\right)}]\right)\hfill \\ I_{34}=\frac{{\alpha }_t}{{\sigma }_t{\left(0\right)}^2}\mathbb{E}\left({\mathcal{M}}_{\ell ,2}^{\prime }[r_t\left(0\right)H+q_t\left(0\right)S;\frac{1}{{\sigma }_t\left(0\right)}]\right)+\frac{{\alpha }_t}{{\sigma }_t{\left(0\right)}^3}\mathbb{E}\left({\mathcal{M}}_{\ell ,2}^{″}[r_t\left(0\right)H+q_t\left(0\right)S;\frac{1}{{\sigma }_t\left(0\right)}]\right)\hfill \\ +\frac{1}{2}{(q_t\left(0\right)-{\beta }_2)}^2+\frac{{\beta }_1}{2}.\hfill \end{array}\right.\end{array}$$

Therefore, the equation in (84) can be expressed as follows:

$$\begin{array}{c}\hfill I_{31}{r}_t^{\prime }\left(0\right)+I_{32}{q}_t^{\prime }\left(0\right)+I_{33}{\sigma }_t^{\prime }\left(0\right)+I_{34}=0.\end{array}$$

(85)

The above analysis shows that the values of $q_t^{\prime }\left(0\right)$ and $r_t^{\prime }\left(0\right)$ can be determined after solving the following system of linear equations:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{I}_{11}{r}_t^{\prime }\left(0\right)+I_{12}{q}_t^{\prime }\left(0\right)+I_{13}{\sigma }_t^{\prime }\left(0\right)+I_{14}=0\hfill \\ I_{21}{r}_t^{\prime }\left(0\right)+I_{22}{q}_t^{\prime }\left(0\right)+I_{23}{\sigma }_t^{\prime }\left(0\right)+I_{24}=0\hfill \\ I_{31}{r}_t^{\prime }\left(0\right)+I_{32}{q}_t^{\prime }\left(0\right)+I_{33}{\sigma }_t^{\prime }\left(0\right)+I_{34}=0,\hfill \end{array}\right.\end{array}$$

(86)

over the three unknowns $q_t^{\prime }\left(0\right)$, $r_t^{\prime }\left(0\right)$, and ${\sigma }_t^{\prime }\left(0\right)$. This completes the proof of Proposition 2.

8. Conclusions

In this paper, we presented a precise characterization of the asymptotic properties of two simple transfer learning formulations. Specifically, our results show that the training and generalization errors corresponding to the considered transfer formulations converge to deterministic functions. These functions can be explicitly found by combining the solutions of two deterministic scalar optimization problems. Our simulation results validate our theoretical predictions and reveal the existence of a phase transition phenomenon in the hard transfer formulation. Specifically, it shows that the hard transfer formulation moves from negative transfer to positive transfer when the similarity of the source and target tasks move past a well-defined critical threshold.

Appendix B.1. Primal Compactness

We start our analysis by assuming that $\lambda >0$. We first consider the compactness of the source problem given in (4). Note that the formulation in (4) has a unique optimal solution. Assume that ${\widehat{\mathit{w}}}_{s,p}\in {\mathbb{R}}^p$ is the unique optimal solution of the optimization problem given in (4). The analysis in [20] (Lemma 1) can be used to prove that there exists $C_1>0$ such that the following inequality is valid:

$$\begin{array}{c}\hfill {\parallel {\widehat{\mathit{w}}}_{s,p}\parallel }^2\le C_1,\end{array}$$

(A13)

with probability going to one as $p\to \infty$. Moreover, observe that the formulation in (33) has a unique optimal solution. Assume that ${\widehat{\mathit{w}}}_{t,p}\in {\mathbb{R}}^p$ is the unique optimal solution of the optimization problem given in (33). Assumption 3 supposes that the loss function is proper. Then, we can conclude that there exists $C_2>0$ such that

$$\begin{array}{c}\hfill \ell (y,z)\ge -C_2,\forall z\in \mathbb{R}.\end{array}$$

(A14)

Now, we define $O_{t,p}^{★}$ as the optimal objective value of the formulation in (33). Then, we can see that there exists $C_3>0$ such that

$$\begin{array}{c}\hfill \frac{\lambda }{2}{\parallel {\widehat{\mathit{w}}}_{t,p}\parallel }^2\le O_{t,p}^{★}+C_3.\end{array}$$

(A15)

Given that ${\widehat{\mathit{w}}}_{s,p}$ is a feasible solution in the formulation given in (33), we obtain the following inequality:

$$\begin{array}{c}\hfill \frac{\lambda }{2}{\parallel {\widehat{\mathit{w}}}_{t,p}\parallel }^2\le \frac{1}{p}\sum _{i=1}^p\ell (y_i;{\mathit{a}}_i^{\top }{\widehat{\mathit{w}}}_{s,p})+\frac{\lambda }{2}{\parallel {\widehat{\mathit{w}}}_{s,p}\parallel }^2+C_3.\end{array}$$

(A16)

Based on [27] (Theorem 2.1), the following convergence in probability holds:

$$\begin{array}{c}\hfill \frac{\parallel \mathit{A}\parallel }{\sqrt{n_t}}\stackrel{p\to +\infty }{\to }\frac{\sqrt{{\alpha }_t}+1}{\sqrt{{\alpha }_t}},\end{array}$$

(A17)

where the matrix $\mathit{A}\in {\mathbb{R}}^{n_t\times p}$ is formed by the concatenation of vectors ${\left\{{\mathit{a}}_i\right\}}_{i=1}^{n_t}$. Then, there exists $C_4>0$ such that the following inequality is valid:

$$\begin{array}{c}\hfill \parallel \mathit{A}{\widehat{\mathit{w}}}_{s,p}\parallel \le \parallel \mathit{A}\parallel \parallel {\widehat{\mathit{w}}}_{s,p}\parallel \le C_4\sqrt{n_t},\end{array}$$

(A18)

with probability going to one as $p\to \infty$. Combining this with the assumption in (12), we see that there exists $C_5>0$ such that the following inequality is valid:

$$\begin{array}{c}\hfill \frac{1}{n_t}\left|\sum _{i=1}^p\ell (y_i;{\mathit{a}}_i^{\top }{\widehat{\mathit{w}}}_{s,p})\right|\le C_5.\end{array}$$

(A19)

Given that $\lambda >0$ and the result in (A13), we conclude that there exists $C_6>0$ such that the following holds:

$$\begin{array}{c}\hfill {\parallel {\widehat{\mathit{w}}}_{t,p}\parallel }^2\le C_6,\end{array}$$

(A20)

with probability going to one as $p\to \infty$.

Now, we consider the case when $\lambda =0$. Define $g_{s,p}(·)$, $O_{s,p}^{★}$, and ${\widehat{\mathit{w}}}_{s,p}$ as the cost function, the optimal cost value, and the optimal solution of the formulation in (4). Moreover, define $g_{t,p}(·)$, $O_{t,p}^{★}$, and ${\widehat{\mathit{w}}}_{t,p}$ as the cost function, the optimal cost value, and the optimal solution of the formulation in (33). Note that the loss function $\ell (y,·)$ is strongly convex with a strong convexity parameter $S>0$. Then, for any $x_1,x_2\in \mathbb{R}$, the following property is valid:

$$\begin{array}{c}\hfill \ell \left(y,\frac{x_1+x_2}{2}\right)\le \frac{1}{2}\ell (y,x_1)+\frac{1}{2}\ell (y,x_2)-\frac{S}{8}{\left|x_1-x_2\right|}^2.\end{array}$$

(A21)

This means that, for any $i\in \{1,\dots ,n\}$, the following property is valid:

$$\begin{array}{c}\hfill \ell \left(y_i,\frac{{\mathit{a}}_i^{\top }{\mathit{w}}_1+{\mathit{a}}_i^{\top }{\mathit{w}}_2}{2}\right)\le \frac{1}{2}\ell (y_i,{\mathit{a}}_i^{\top }{\mathit{w}}_1)+\frac{1}{2}\ell (y_i,{\mathit{a}}_i^{\top }{\mathit{w}}_2)-\frac{S}{8}{\left|{\mathit{a}}_i^{\top }{\mathit{w}}_1-{\mathit{a}}_i^{\top }{\mathit{w}}_2\right|}^2,\end{array}$$

(A22)

where n can be the number of samples of the source task or target task and ${\left\{y_i\right\}}_{i=1}^n$ are the labels of the source task or target task. Given the convexity of the norm, we obtain the following inequality:

$$\begin{array}{c}\hfill g_p\left(\frac{{\mathit{w}}_1+{\mathit{w}}_2}{2}\right)\le \frac{1}{2}{g}_p\left({\mathit{w}}_1\right)+\frac{1}{2}{g}_p\left({\mathit{w}}_2\right)-\frac{S}{8p}{\parallel \mathit{A}({\mathit{w}}_1-{\mathit{w}}_2)\parallel }^2,\end{array}$$

(A23)

where $g_p(·)$ can be the cost value of the source task or target task formulations. Now, we focus on the source formulation. Take ${\mathit{w}}_1={\widehat{\mathit{w}}}_{s,p}$ and ${\mathit{w}}_2=0$. Moreover, see that the loss function is proper. Then, there exists $C_7>0$ such that

$$\begin{array}{c}\hfill \frac{S}{8p}{\parallel \mathit{A}{\widehat{\mathit{w}}}_{s,p}\parallel }^2\le C_7+\frac{1}{2}{g}_{s,p}\left(0\right).\end{array}$$

(A24)

Given the assumption in (12), $S>0$, ${\alpha }_s>1$, and the analysis in [27] (Theorem 2.1), there exists $C_8>0$ such that

$$\begin{array}{c}\hfill {\parallel {\widehat{\mathit{w}}}_{s,p}\parallel }^2\le C_8.\end{array}$$

(A25)

Now, we focus on the target task. Take ${\mathit{w}}_1={\widehat{\mathit{w}}}_{t,p}$ and ${\mathit{w}}_2={\widehat{\mathit{w}}}_{s,p}$. Moreover, see that the loss function is proper. Then, there exists $C_9>0$ such that

$$\begin{array}{c}\hfill \frac{S}{8p}{\parallel \mathit{A}({\widehat{\mathit{w}}}_{t,p}-{\widehat{\mathit{w}}}_{s,p})\parallel }^2\le C_9+\frac{1}{2}{g}_{t,p}\left({\widehat{\mathit{w}}}_{s,p}\right).\end{array}$$

(A26)

Given the assumption in (12), the result in (A25), $S>0$, ${\alpha }_t>1$, and the analysis in [27] (Theorem 2.1), there exists $C_{10}>0$ such that

$$\begin{array}{c}\hfill {\parallel {\widehat{\mathit{w}}}_{t,p}\parallel }^2\le C_{10}.\end{array}$$

(A27)

This completes the first part of the proof of Lemma 1.

Appendix B.2. Dual Compactness

The analysis in Appendix B.1 shows that the formulation in (34) can be equivalently formulated, where the primal feasibility set is given by

$$\begin{array}{c}\hfill {\parallel \mathit{w}\parallel }^2\le C,\end{array}$$

(A28)

where $C>0$ is a sufficiently large constant that satisfies the analysis in Appendix B.1. Now, define ${\widehat{\mathit{u}}}_p$ as the optimal solution of the formulation in (34). Additionally, define the function ${\mathcal{L}}^{★}(·)$ as ${\mathcal{L}}^{★}\left(\mathit{u}\right)={\sum }_{i=1}^{n_t}{\ell }^{★}\left(y_i;u_i\right)$. We can see that the optimal vector ${\widehat{\mathit{u}}}_p$ solves the following maximization problem:

$$\begin{array}{c}\hfill {\widehat{\mathit{u}}}_p=\underset{\mathit{u}\in {\mathbb{R}}^{n_t}}{argmax}{\mathit{u}}^{\top }\mathit{A}\mathit{w}-{\mathcal{L}}^{★}\left(\mathit{u}\right),\end{array}$$

where the data matrix $\mathit{A}={[{\mathit{a}}_1,\dots ,{\mathit{a}}_{n_t}]}^{\top }\in {\mathbb{R}}^{n_t\times p}$. Now, we denote by $\partial {\mathcal{L}}^{★}\left(\mathit{u}\right)$ the sub-differential set of the function ${\mathcal{L}}^{★}(·)$ evaluated at $\mathit{u}$. Therefore, the solution of the above maximization problem satisfies the following condition:

$$\begin{array}{c}\hfill \mathit{A}\mathit{w}\in \partial {\mathcal{L}}^{★}\left({\widehat{\mathit{u}}}_p\right).\end{array}$$

(A29)

Now, we use the result in [25] (Proposition 11.3) to show that the condition in (A29) can be equivalently expressed as follows:

$$\begin{array}{c}\hfill {\widehat{\mathit{u}}}_p\in \partial \mathcal{L}\left(\mathit{A}\mathit{w}\right),\end{array}$$

(A30)

where the loss function $\mathcal{L}\left(\mathit{w}\right)={\sum }_{i=1}^{n_t}\ell \left(y_i;w_i\right)$ based on [25] (Proposition 11.22). Note that the introduced constraint in (A28) is satisfied. Moreover, the analysis presented in (A17) shows that there exists $C_1>0$ such that the following inequality holds:

$$\begin{array}{c}\hfill {\parallel \mathit{A}\mathit{w}\parallel }^2\le C_1{n}_t,\end{array}$$

(A31)

with probability going to one as p goes to $+\infty$. Now, we use the assumption in (12) to conclude that there exists $C_2>0$ such that the following inequality holds:

$$\begin{array}{c}\hfill {\parallel {\widehat{\mathit{u}}}_p\parallel }^2\le C_2{n}_t,\end{array}$$

(A32)

with probability going to one as p goes to $+\infty$. This completes the proof of Lemma 1.

Appendix C.1. Auxiliary Convergence

Note that the analysis present in Section 7 is also valid for the source problem. This is because the formulation in (8) is equivalent to the source problem in (4) if $\mathsf{\Sigma }$ is the all zero matrix and we use the source training data. Then, we can see that the optimal solution of the auxiliary formulation corresponding to the source problem, denoted by ${\tilde{\mathit{w}}}_s$, can be expressed as follows:

$$\begin{array}{c}\hfill {\tilde{\mathit{w}}}_s=q_{p,s}^{★}{\mathit{\xi }}_s-\frac{r_{p,s}^{★}}{\parallel {\tilde{\mathit{g}}}_s\parallel }{\mathit{B}}_{{\mathit{\xi }}_s}^{\perp }{\tilde{\mathit{g}}}_s,\end{array}$$

(A33)

where ${\tilde{\mathit{g}}}_s={\left({\mathit{B}}_{{\mathit{\xi }}_s}^{\perp }\right)}^{\top }{\mathit{g}}_s$ and ${\mathit{g}}_s$ has independent standard Gaussian components. Here, ${\mathit{B}}_{{\mathit{\xi }}_s}^{\perp }\in {\mathbb{R}}^{p\times (p-1)}$ is formed by an orthonormal basis orthogonal to the vector ${\mathit{\xi }}_s$. Additionally, our analysis in Section 7 shows that the following convergence in probability holds:

$$\begin{array}{c}\hfill q_{p,s}\stackrel{p\to +\infty }{\to }{q}_s^{★}\mathrm{and}{r}_{p,s}^{★}\stackrel{p\to +\infty }{\to }{r}_s^{★}.\end{array}$$

(A34)

Here, $q_s^{★}$ and $r_s^{★}$ are the optimal solutions of the asymptotic limit of the source formulation defined in (14).

Note that ${\mu }_p$ can be expressed as follows:

$$\begin{array}{c}\hfill {\mu }_p={\sigma }_{\min }\left({\left({\mathit{B}}_{{\mathit{\xi }}_t}^{\perp }\right)}^{\top }\mathsf{\Lambda }{\mathit{B}}_{{\mathit{\xi }}_t}^{\perp }\right).\end{array}$$

(A35)

Using the eigenvalue interlacing theorem, one can see that

$$\begin{array}{c}\hfill {\sigma }_{\min ,1}\left(\mathsf{\Lambda }\right)\le {\mu }_p\le {\sigma }_{\min ,2}\left(\mathsf{\Lambda }\right).\end{array}$$

(A36)

Then, using the assumption in (13), we can see that the random variable ${\mu }_p$ converges in probability to ${\mu }_{\min }$, where ${\mu }_{\min }$ is defined in Assumption 5. Now, we study the properties of the remaining functions using the optimal solution of the auxiliary formulation defined in (A33), i.e., ${\tilde{\mathit{w}}}_s$, instead of ${\widehat{\mathit{w}}}_s$. For instance, we first study the random sequence ${\tilde{V}}_{p,ts}={\mathit{\xi }}_t^{\top }\mathsf{\Lambda }{\tilde{\mathit{w}}}_s$ to infer the asymptotic properties of $V_{p,ts}$.

First, fix $\sigma >-{\mu }_{\min }$. Then, based on the convergence of ${\mu }_p$ and [34] (Proposition 3), the sequence of random functions $T_{p,g}(.)$ converges in probability as follows:

$$\begin{array}{c}\hfill T_{p,g}\left(\sigma \right)\stackrel{p\to +\infty }{\to }{T}_g\left(\sigma \right)={\mathbb{E}}_{\mu }\left[1/(\mu +\sigma )\right].\end{array}$$

(A37)

Now, we express $\sigma$ as $\sigma ={\sigma }^{\prime }-x$, where ${\sigma }^{\prime }>0$. This means that the following convergence in probability holds true:

$$\begin{array}{c}\hfill T_{p,g}({\sigma }^{\prime }-x)\stackrel{p\to +\infty }{\to }{T}_g({\sigma }^{\prime }-x),\end{array}$$

(A38)

for any $x<{\sigma }^{\prime }+{\mu }_{\min }$. Note that the functions $T_{p,g}(.)$ and $T_g(.)$ are both convex and continuous in the variable x in the set $[0,{\sigma }^{\prime }+{\mu }_{\min }[$. Then, based on [30] (Theorem II.1), the convergence in (A38) is uniform in the variable x in the compact set $[0,{\sigma }^{\prime }/2+{\mu }_{\min }]$. Now, note that ${\mu }_p$ converges in probability to ${\mu }_{\min }$. Therefore, we obtain the following convergence in probability:

$$\begin{array}{c}\hfill T_{p,g}({\sigma }^{\prime }-{\mu }_p)\stackrel{p\to +\infty }{\to }{T}_g({\sigma }^{\prime }-{\mu }_{\min }),\end{array}$$

(A39)

valid for any fixed ${\sigma }^{\prime }>0$. Using the block matrix inversion lemma, the function $T_{p,t}(.)$ can be expressed as follows:

$$\begin{array}{cc}\hfill T_{p,t}\left(\sigma \right)& ={\mathit{\xi }}_t^{\top }\mathsf{\Lambda }{\mathit{B}}_{{\mathit{\xi }}_t}^{\perp }{[{\left({\mathit{B}}_{{\mathit{\xi }}_t}^{\perp }\right)}^{\top }\mathsf{\Lambda }{\mathit{B}}_{{\mathit{\xi }}_t}^{\perp }+\sigma {\mathit{I}}_{p-1}]}^{-1}{\left({\mathit{B}}_{{\mathit{\xi }}_t}^{\perp }\right)}^{\top }\mathsf{\Lambda }{\mathit{\xi }}_t\hfill \\ \hfill & ={\mathit{\xi }}_t^{\top }\mathsf{\Lambda }{\mathit{\xi }}_t+\sigma -\frac{1}{{\mathit{\xi }}_t^{\top }{[\mathsf{\Lambda }+\sigma {\mathit{I}}_p]}^{-1}{\mathit{\xi }}_t}.\hfill \end{array}$$

(A40)

Therefore, we obtain the following expression:

$$\begin{array}{c}\hfill Z_{p,t}\left(\sigma \right)=\sigma -\frac{1}{{\mathit{\xi }}_t^{\top }{[\mathsf{\Lambda }+\sigma {\mathit{I}}_p]}^{-1}{\mathit{\xi }}_t}.\end{array}$$

(A41)

Then, using the theoretical results stated in [34] (Proposition 3), the functions $Z_{p,t}(.)$ converges in probability as follows:

$$\begin{array}{c}\hfill Z_{p,t}\left(\sigma \right)\stackrel{p\to +\infty }{\to }{Z}_t\left(\sigma \right)=\sigma -\frac{1}{{\mathbb{E}}_{\mu }\left[1/(\mu +\sigma )\right]}.\end{array}$$

(A42)

Combine this with the above analysis to obtain the following convergence in probability:

$$\begin{array}{c}\hfill Z_{p,t}({\sigma }^{\prime }-{\mu }_p)\stackrel{p\to +\infty }{\to }{Z}_t({\sigma }^{\prime }-{\mu }_{\min }),\end{array}$$

(A43)

valid for any ${\sigma }^{\prime }>0$. Based on the result in (A33), the sequence of random functions ${\tilde{Z}}_{p,ts}(.)$ converges in probability to the following function:

$$\begin{array}{c}\hfill Z_{ts}\left(\sigma \right)=q_s^{★}\rho Z_t\left(\sigma \right).\end{array}$$

(A44)

Combine this with the above analysis to obtain the following convergence in probability:

$$\begin{array}{c}\hfill {\tilde{Z}}_{p,ts}({\sigma }^{\prime }-{\mu }_p)\stackrel{p\to +\infty }{\to }{Z}_{ts}({\sigma }^{\prime }-{\mu }_{\min }),\end{array}$$

(A45)

valid for any ${\sigma }^{\prime }>0$. Using the same analysis and based on (A33) and (A34), one can see that the sequence of random functions ${\tilde{Z}}_{p,s}(.)$ converges in probability to the following function:

$$\begin{array}{cc}\hfill {\tilde{Z}}_{p,s}\left(\sigma \right)& \stackrel{p\to +\infty }{\to }{Z}_s\left(\sigma \right)={\left(\rho q_s^{★}\right)}^2{Z}_t\left(\sigma \right)\hfill \\ \hfill & -\left((1-{\rho }^2){\left(q_s^{★}\right)}^2+{\left(r_s^{★}\right)}^2\right){\mathbb{E}}_{\mu }\left[\mu \sigma /(\mu +\sigma )\right].\hfill \end{array}$$

(A46)

Combine this with the above analysis to obtain the following convergence in probability:

$$\begin{array}{c}\hfill {\tilde{Z}}_{p,s}({\sigma }^{\prime }-{\mu }_p)\stackrel{p\to +\infty }{\to }{Z}_s({\sigma }^{\prime }-{\mu }_{\min }),\end{array}$$

(A47)

valid for any ${\sigma }^{\prime }>0$. The above analysis shows that the asymptotic properties stated in Lemma 2 are valid for the AO formulation corresponding to the source problem. Now, it remains to show that these properties also hold for the primary formulation.

Appendix C.2. Primary Convergence

Here, we assume that $\lambda >0$. The case when $\lambda =0$ can be conducted similarly. Now, we show that the convergence properties proved above are also valid for the primary problem. To this end, we show that all the assumptions in Theorem 2 are satisfied. We start our proof by defining the following open set:

$$\begin{array}{c}\hfill {\mathcal{T}}_{ϵ}=\{\mathit{w}\in {\mathbb{R}}^p:|{\mathit{\xi }}_t^{\top }{[\mathsf{\Lambda }+\sigma {\mathit{I}}_p]}^{-1}\mathit{w}-\rho q_s^{★}K|<ϵ\},\end{array}$$

where K is defined as follows:

$$\begin{array}{c}\hfill K={\mathbb{E}}_{\mu }\left[1/(\mu +\sigma )\right].\end{array}$$

(A48)

Now, we consider the feasibility set ${\mathcal{D}}_{ϵ}={\mathcal{T}}_1/{\mathcal{S}}_{ϵ}$, where ${\mathcal{T}}_1$ is defined in (41). Based on the analysis of the generalized target formulation in Section 7.3.2, one can see that the AO formulation corresponding to the source formulation with the set ${\mathcal{D}}_{ϵ}$ can be asymptotically expressed as follows:

$$\begin{array}{cc}\hfill & {\mathfrak{V}}_p:\underset{\begin{array}{c}(q_s,r_s)\in {\mathcal{T}}_2\end{array}}{min}\underset{\begin{array}{c}{\mathit{r}}_s\in {\tilde{\mathcal{D}}}_{ϵ}\end{array}}{min}\underset{\mathit{u}\in {\mathcal{C}}_s}{max}\frac{\parallel \mathit{u}\parallel }{p}{\mathit{g}}_s^{\top }{\mathit{B}}_{{\mathit{\xi }}_s}^{\perp }{\mathit{r}}_s+\frac{q_s}{p}{\mathit{u}}^{\top }{\mathit{s}}_s\hfill \\ \hfill & +\frac{\lambda }{2}(q_s^2+{\parallel {\mathit{r}}_s\parallel }^2)+\frac{1}{p}\parallel {\mathit{r}}_s\parallel {\mathit{h}}_s^{\top }\mathit{u}-\frac{1}{p}\sum _{i=1}^{n_s}{\ell }^{★}\left(y_{s,i};u_i\right).\hfill \end{array}$$

Here, the feasibility set ${\mathcal{T}}_2$ is defined in Section 7.3.2 and the feasibility set ${\tilde{\mathcal{D}}}_{ϵ}$ is given by

$$\begin{array}{cc}\hfill & \{{\mathit{r}}_s:\left|q_s\rho K_{p,t}+q_s\sqrt{1-{\rho }^2}{K}_{p,r}+{\mathit{\xi }}_t^{\top }{[\mathsf{\Lambda }+\sigma {\mathit{I}}_p]}^{-1}{\mathit{B}}_{{\mathit{\xi }}_s}^{\perp }{\mathit{r}}_s-\rho q_s^{★}K\right|\ge ϵ\hfill \\ \hfill & ,\parallel {\mathit{r}}_s\parallel =r_s\}.\hfill \end{array}$$

This follows based on the decomposition in (40) and where $K_{p,t}={\mathit{\xi }}_t^{\top }{[\mathsf{\Lambda }+\sigma {\mathit{I}}_p]}^{-1}{\mathit{\xi }}_t$ and $K_{p,r}={\mathit{\xi }}_t^{\top }{[\mathsf{\Lambda }+\sigma {\mathit{I}}_p]}^{-1}{\mathit{\xi }}_r$. Note that the optimization problem given in ${\mathfrak{V}}_p$ can be equivalently formulated as follows:

$$\begin{array}{cc}\hfill & {\mathfrak{V}}_p:\underset{\begin{array}{c}(q_s,r_s)\in {\widehat{\mathcal{S}}}_{ϵ}\end{array}}{min}\underset{\begin{array}{c}{\mathit{r}}_s\in {\tilde{\mathcal{D}}}_{ϵ}\end{array}}{min}\underset{\mathit{u}\in {\mathcal{C}}_s}{max}\frac{\parallel \mathit{u}\parallel }{p}{\mathit{g}}_s^{\top }{\mathit{B}}_{\mathit{\xi }}^{\perp }{\mathit{r}}_s+\frac{q_s}{p}{\mathit{u}}^{\top }{\mathit{s}}_s\hfill \\ \hfill & +\frac{\lambda }{2}(q_s^2+{\parallel {\mathit{r}}_s\parallel }^2)+\frac{1}{p}\parallel {\mathit{r}}_s\parallel {\mathit{h}}_s^{\top }\mathit{u}-\frac{1}{p}\sum _{i=1}^{n_s}{\ell }^{★}\left(y_{s,i};u_i\right).\hfill \end{array}$$

Here, we replace the feasibility set ${\mathcal{T}}_2$ by the feasibility set ${\widehat{\mathcal{S}}}_{ϵ}$ defined as follows:

$$\begin{array}{cc}\hfill & \left\{\right|q_s\rho K_{p,t}+q_s\sqrt{1-{\rho }^2}{K}_{p,r}-r_s{\mathit{\xi }}_t^{\top }{[\mathsf{\Lambda }+\sigma {\mathit{I}}_p]}^{-1}{\mathit{B}}_{{\mathit{\xi }}_s}^{\perp }\frac{{\tilde{\mathit{g}}}_s}{\parallel {\tilde{\mathit{g}}}_s\parallel }-\rho q_s^{★}K|\hfill \\ \hfill & \ge ϵ\}\cap {\mathcal{T}}_2,\hfill \end{array}$$

where ${\tilde{\mathit{g}}}_s={\left({\mathit{B}}_{{\mathit{\xi }}_s}^{\perp }\right)}^{\top }{\mathit{g}}_s$. This follows since the first set in ${\widehat{\mathcal{S}}}_{ϵ}$ satisfies the condition in the set ${\tilde{\mathcal{D}}}_{ϵ}$. Now, assume that ${\widehat{\varphi }}_p^{★}$ is the optimal cost value of the optimization problem ${\mathfrak{V}}_p$ and define the function ${\widehat{h}}_p(.)$ as follows:

$$\begin{array}{cc}\hfill & {\widehat{h}}_p(q_s,r_s)=\underset{\begin{array}{c}{\mathit{r}}_s\in {\tilde{\mathcal{D}}}_{ϵ}\end{array}}{min}\underset{\mathit{u}\in {\mathcal{C}}_s}{max}\frac{\parallel \mathit{u}\parallel }{p}{\mathit{g}}_s^{\top }{\mathit{B}}_{{\mathit{\xi }}_s}^{\perp }{\mathit{r}}_s+\frac{q_s{\mathit{u}}^{\top }{\mathit{s}}_s}{p}\hfill \\ \hfill & +\frac{\lambda }{2}(q_s^2+r_s^2)+\frac{r_s}{p}{\mathit{h}}_s^{\top }\mathit{u}-\frac{1}{p}\sum _{i=1}^{n_s}{\ell }^{★}\left(y_{s,i};u_i\right),\hfill \end{array}$$

in the set ${\widehat{\mathcal{S}}}_{ϵ}$. Based on the max–min inequality [35], the function ${\widehat{h}}_p(.)$ can be lower bounded by the following function:

$$\begin{array}{cc}\hfill & {\tilde{h}}_p(q_s,r_s)=\underset{\mathit{u}\in {\mathcal{C}}_s}{max}\underset{\begin{array}{c}{\mathit{r}}_s\in {\tilde{\mathcal{D}}}_{ϵ}\end{array}}{min}\frac{\parallel \mathit{u}\parallel }{p}{\mathit{g}}_s^{\top }{\mathit{B}}_{{\mathit{\xi }}_s}^{\perp }{\mathit{r}}_s+\frac{q_s{\mathit{u}}^{\top }{\mathit{s}}_s}{p}\hfill \\ \hfill & +\frac{\lambda }{2}(q_s^2+r_s^2)+\frac{r_s}{p}{\mathit{h}}_s^{\top }\mathit{u}-\frac{1}{p}\sum _{i=1}^{n_s}{\ell }^{★}\left(y_{s,i};u_i\right).\hfill \end{array}$$

This is valid for any $(q_s,r_s)\in {\widehat{\mathcal{S}}}_{ϵ}$. Moreover, note that the following inequality holds true:

$$\begin{array}{c}\hfill \underset{\begin{array}{c}{\mathit{r}}_s\in {\tilde{\mathcal{D}}}_{ϵ}\end{array}}{min}\frac{\parallel \mathit{u}\parallel }{p}{\mathit{g}}_s^{\top }{\mathit{B}}_{{\mathit{\xi }}_s}^{\perp }{\mathit{r}}_s\ge -\frac{\parallel \mathit{u}\parallel }{p}\parallel {\left({\mathit{B}}_{{\mathit{\xi }}_s}^{\perp }\right)}^{\top }{\mathit{g}}_s\parallel r_s,\end{array}$$

(A49)

for any $(q_s,r_s)\in {\widehat{\mathcal{S}}}_{ϵ}$. Following the generalized analysis in Section 7.3.2, one can see that the auxiliary problem corresponding to the source formulation can be expressed as follows:

$$\begin{array}{cc}\hfill & \underset{\begin{array}{c}(q_s,r_s)\in {\mathcal{T}}_2\end{array}}{min}\underset{{\sigma }_s>0}{sup}\frac{1}{n_s}\sum _{i=1}^{n_s}{\mathcal{M}}_{\ell (y_{s,i},.)}\left(r_s{h}_{s,i}+q_s{s}_{s,i};\frac{r_s\parallel {\tilde{\mathit{g}}}_s\parallel }{\sqrt{n_s}{\sigma }_s}\right)\hfill \\ \hfill & -\frac{r_s{\sigma }_s}{2}\frac{\parallel {\tilde{\mathit{g}}}_s\parallel }{\sqrt{n_s}}+\frac{\lambda }{2}(q_s^2+r_s^2).\hfill \end{array}$$

(A50)

This means that the function ${\tilde{h}}_p(.)$ can be lower bounded by the cost function of the minimization problem formulated in (A50) denoted by ${\widehat{g}}_p(.)$, i.e.,

$$\begin{array}{c}\hfill {\widehat{g}}_p(q_s,r_s)\le {\tilde{h}}_p(q_s,r_s).\end{array}$$

(A51)

Here, both functions are defined in the feasibility set ${\widehat{\mathcal{S}}}_{ϵ}$. Now, define ${\varphi }_p^{★}$ as the optimal cost value of the auxiliary optimization problem corresponding to the source formulation defined in Section 7.3.1. Note that the loss function ${\widehat{g}}_p(.)$ is strongly convex in the variables $(q_s,r_s)$ with strong convexity parameter $\lambda >0$. This means that, for any $\beta \in [0,1]$, $(q_{s,1},r_{s,1})\in {\mathcal{T}}_2$ and $(q_{s,2},r_{s,2})\in {\mathcal{T}}_2$, we have the following inequality:

$$\begin{array}{cc}\hfill & {\widehat{g}}_p(\beta {\mathit{v}}_1+(1-\beta ){\mathit{v}}_2)\le \beta {\widehat{g}}_p\left({\mathit{v}}_1\right)\hfill \\ \hfill & +(1-\beta ){\widehat{g}}_p\left({\mathit{v}}_2\right)-\frac{\lambda }{2}\beta (1-\beta ){\parallel {\mathit{v}}_1-{\mathit{v}}_2\parallel }^2,\hfill \end{array}$$

(A52)

where ${\mathit{v}}_1=[q_{s,1},r_{s,1}]$ and ${\mathit{v}}_2=[q_{s,2},r_{s,2}]$. Take ${\mathit{v}}_1$ as ${\mathit{v}}_p^{★}$, which represents the optimal solution of the optimization problem (A50). Then, the inequality in (A52) implies the following inequality:

$$\begin{array}{cc}\hfill & {\varphi }_p^{★}\le {\widehat{g}}_p\left({\mathit{v}}_2\right)-\frac{\lambda }{2}\beta {\parallel {\mathit{v}}_p^{★}-{\mathit{v}}_2\parallel }^2.\hfill \end{array}$$

(A53)

This is valid for any ${\mathit{v}}_2$ in the set ${\mathcal{T}}_2$. Now, taking $\beta =1/2$ and the minimum over ${\mathit{v}}_2$ in the set ${\widehat{\mathcal{S}}}_{ϵ}$ in both sides, we obtain the following inequality:

$$\begin{array}{c}\hfill {\varphi }_p^{★}+\frac{\lambda }{4}\underset{\mathit{v}\in {\widehat{\mathcal{S}}}_{ϵ}}{min}{\parallel {\mathit{v}}_p^{★}-\mathit{v}\parallel }^2\le \underset{\mathit{v}\in {\widehat{\mathcal{S}}}_{ϵ}}{min}{\widehat{g}}_p\left(\mathit{v}\right).\end{array}$$

Based on the above analysis, note that the following inequality also holds true:

$$\begin{array}{c}\hfill \underset{\mathit{v}\in {\widehat{\mathcal{S}}}_{ϵ}}{min}{\widehat{g}}_p\left(\mathit{v}\right)\le {\widehat{\varphi }}_p^{★}.\end{array}$$

(A54)

Then, to verify the assumption of [17] (Theorem 6.1), it remains to show that there exists ${ϵ}^{\prime }>0$ such that, the following inequality holds:

$$\begin{array}{c}\hfill \frac{\lambda }{4}\underset{\mathit{v}\in {\widehat{\mathcal{S}}}_{ϵ}}{min}{\parallel {\mathit{v}}_p^{★}-\mathit{v}\parallel }^2\ge {ϵ}^{\prime },\end{array}$$

(A55)

with probability going to 1 as $p\to \infty$. Note that any element in the set ${\widehat{\mathcal{S}}}_{ϵ}$ satisfies the following inequality:

$$\begin{array}{cc}\hfill & ϵ\le |q_s\rho K_{p,t}+q_s\sqrt{1-{\rho }^2}{K}_{p,r}-r_s{\mathit{\xi }}_t^{\top }{[\mathsf{\Lambda }+\sigma {\mathit{I}}_p]}^{-1}{\mathit{B}}_{\mathit{\xi }}^{\perp }\frac{{\tilde{\mathit{g}}}_s}{\parallel {\tilde{\mathit{g}}}_s\parallel }-\rho q_s^{★}K|\le \hfill \\ \hfill & |q_s\rho K_{p,t}-\rho q_s^{★}K|+|q_s\sqrt{1-{\rho }^2}\left|\right|K_{p,r}|+|r_s\left|\right|{\mathit{\xi }}_t^{\top }{[\mathsf{\Lambda }+\sigma {\mathit{I}}_p]}^{-1}{\mathit{B}}_{\mathit{\xi }}^{\perp }\frac{{\tilde{\mathit{g}}}_s}{\parallel {\tilde{\mathit{g}}}_s\parallel }|.\hfill \end{array}$$

Based on the analysis in Appendix C.1, we have the following convergence in probability:

$$\begin{array}{c}\hfill |q_s\rho K_{p,t}-\rho q_s^{★}K|\stackrel{p\to +\infty }{\to }\left|q_s-q_s^{★}\right|\rho K\\ \hfill |q_s|\sqrt{1-{\rho }^2}|K_{p,r}|\stackrel{p\to +\infty }{\to }0,|r_s\left|\right|{\mathit{\xi }}_t^{\top }{[\mathsf{\Lambda }+\sigma {\mathit{I}}_p]}^{-1}{\mathit{B}}_{\mathit{\xi }}^{\perp }\frac{{\tilde{\mathit{g}}}_s}{\parallel {\tilde{\mathit{g}}}_s\parallel }|\stackrel{p\to +\infty }{\to }0.\end{array}$$

(A56)

This means that there exists ${ϵ}^{″}>0$ such that any elements in the set ${\widehat{\mathcal{S}}}_{ϵ}$ satisfies the following inequality:

$$\begin{array}{c}\hfill |q_s-q_s^{★}|\rho K\ge {ϵ}^{″},\end{array}$$

(A57)

with probability going to 1 as $p\to \infty$. Combining this with Assumption 5 and the consistency result stated in (A34) shows that there exists ${ϵ}^{\prime }>0$ such that the following inequality holds:

$$\begin{array}{c}\hfill \frac{\lambda }{4}\underset{\mathit{v}\in {\widehat{\mathcal{D}}}_{ϵ}}{min}{\parallel {\mathit{v}}_p^{★}-\mathit{v}\parallel }^2\ge {ϵ}^{\prime },\end{array}$$

(A58)

with probability going to 1 as $p\to \infty$. This also proves that there exists ${ϵ}^{\prime }>0$ such that the following inequality holds:

$$\begin{array}{c}\hfill {\widehat{\varphi }}_p^{★}\ge {\varphi }_p^{★}+{ϵ}^{\prime },\end{array}$$

(A59)

with probability going to 1 as $p\to \infty$. This completes the verification of the assumptions in Theorem 2. This means that the optimal solution of the primary problem belongs to the set ${\mathcal{S}}_{ϵ}$ on events with probability going to 1 as $p\to \infty$. Since the choice of $ϵ$ is arbitrary, we obtain the following asymptotic result:

$$\begin{array}{c}\hfill {\mathit{\xi }}_t^{\top }{[\mathsf{\Lambda }+\sigma {\mathit{I}}_p]}^{-1}{\widehat{\mathit{w}}}_s\stackrel{p\to +\infty }{\to }{q}_s^{★}\rho K,\end{array}$$

(A60)

where ${\widehat{\mathit{w}}}_s$ is the optimal solution of the source problem (4). Following the same analysis, one can also show the convergence properties stated in Lemma 2.