On Data-Enriched Logistic Regression

Abstract

Biomedical researchers typically investigate the effects of specific exposures on disease risks within a well-defined population. The gold standard for such studies is to design a trial with an appropriately sampled cohort. However, due to the high cost of such trials, the collected sample sizes are often limited, making it difficult to accurately estimate the effects of certain exposures. In this paper, we discuss how to leverage the information from external “big data” (datasets with significantly larger sample sizes) to improve the estimation accuracy at the risk of introducing a small amount of bias. We propose a family of weighted estimators to balance bias increase and variance reduction when incorporating the big data. We establish a connection between our proposed estimator and the well-known penalized regression estimators. We derive optimal weights using both second-order and higher-order asymptotic expansions. Through extensive simulation studies, we demonstrate that the improvement in mean square error (MSE) for the regression coefficient can be substantial even with finite sample sizes, and our weighted method outperformed existing approaches such as penalized regression and James–Stein estimator. Additionally, we provide a theoretical guarantee that the proposed estimators will never yield an asymptotic MSE larger than the maximum likelihood estimator using small data only in general. Finally, we apply our proposed methods to the Asia Cohort Consortium China cohort data to estimate the relationships between age, BMI, smoking, alcohol use, and mortality.

1. Introduction

In most research settings in medicine, we aim to know the effect of a specific exposure to the risk of some specific disease among a well-defined targeted population. To achieve this goal, well-designed trials are usually used. However, the sample sizes for such studies are usually limited due to the high cost of recruitment and thus the sample size usually just has the power to detect the effect for the primary exposure of interest. In the meantime, the number of available observational studies or trial studies from other populations is accumulating quickly nowadays. Can we use information from these data to improve our inference on the population where small data are drawn from? Here, we refer to the randomized clinical trial data that have a clearly defined target population by the design and sampling scheme as “small data” and refer to other external data as “big data”. Our goal is to efficiently combine the information from these two types of data to obtain a more accurate estimation of the association between some readily available quantities that are presented in both data (e.g., age and gender) and the risk of diseases among the population where small data are drawn from. Since the distributions of predictors, as well as the relationship among these predictors and the event of interest, are likely to be different between “big data” and “small data”, we are at risk of introducing some bias when using the information from big data. However, given the size of the big data, they can still provide insightful information about how we predict risk among the targeted population, assuming that the two populations share a certain degree of similarity in their prediction model forms. This motivates us to find an estimator that is better than using “small data” only with a bias–variance trade-off. Directly pooling the two data together can lead to substantial bias and increase mean square error (MSE) when the difference between the two sources is large. This motivates us to find estimators that always do not lead to increased MSE and lead to decreased MSE under certain situations when compared with estimator that using “small data” only.

Previous studies have shown the plausibility of this type of idea. In the simple mean estimation, ref. [1] showed that the simple sample mean is inadmissible. Ref. [2] studied the combining of regression results from small and big data in a linear regression setting and showed Stein-type results for Gaussian responses, i.e., the use of small data only is inadmissible when $p\ge 5$ and degree of freedom is more than 10. Ref. [3] proposed to use shared Lasso to achieve this in a linear regression setting. However, the similar enhanced regression approach for non-Gaussian outcomes has not been fully studied. To our knowledge, risk prediction work is mostly dependent on the assumption that certain reduced marginal models or marginal information from the big data are accurate [4,5]. In this work, we propose to fill the gap of risk prediction by combining information from small and big data for binary outcomes that rely on an alternative structural assumption, where we assume the effect structure rather than the effect magnitude is the same. Specifically, we proposed two new estimators that can incorporate information from big data to improve efficiency in estimating parameters related to small data. We showed that these estimators are better than using small data both theoretically and via extensive numerical studies. In addition, we compared our proposed estimators with several existing alternative estimators (pooled estimator, penalized regression estimator, and James–Stein estimator).

The structure of this paper is as follows: In Section 2, we introduce the notation and models we used followed by our proposed estimators and their connection to other existing estimators. In Section 3, we study the performance of different estimators for their finite sample properties and show the improvement using our proposed estimator. In Section 4, we provide theoretical results that guarantee our proposed estimator to be no worse than the small data-only analysis in terms of MSE. In Section 5, we applied our method to analyze the Asia Cohort Consortium data with sensitivity for potential violation of model assumptions. In Section 6, we discuss the potential extension of our proposed estimator to a more general setting.

2. Methods

2.1. Notation and Model

We denote our outcome of interest as $Y\in {\{0,1\}}^{n_S+n_B}$ and denote the design matrices by $X_S\in R^{n_S\times p}$ and $X_B\in R^{n_B\times p}$, where $n_S$ and $n_B$ represent the sample sizes for the small data and the big data, respectively. In general, we reserve the subscripts B and S to denote quantities related to the big data and the small data, respectively. Since the outcomes of interest are binary (disease occurrence), denoting ${\mu }_i=E\left(Y_i\right|X_i)=Pr\left(Y_i\right|X_i)$, we assume logistic regression models for both the small data and the large data and write them as

$$\begin{array}{ccc}\hfill & & log\left({\displaystyle \frac{{\mu }_i}{1-{\mu }_i}}\right)=X_i\left\{\beta +\gamma I(i\in B)\right\}\hfill \end{array}$$

(1)

where $\beta \in R^{p\times 1}$ and $\gamma \in R^{p\times 1}$ are unknown regression parameters, and $I(·)$ is the indicator function. Our goal is to obtain an accurate estimation of $\beta$ while treating $\gamma$ as a nuisance parameter using information from both the small and big data. In this project, we propose novel weighted shrinkage estimators and relate them to the penalized regression-based estimators. We compare the performance of weighted estimators, penalized estimators, and the James–Stein-type shrinkage estimator [6,7]. In this paper, we adopt the following big O and small O notations: A sequence $A_n$ is said to be $o\left(n^k\right)$ if ${lim}_{n\to \infty }{\displaystyle \frac{A}{n_k}}\to 0$ and is said to be $O\left(n^k\right)$ if ${lim\; sup}_{n\to \infty }|{\displaystyle \frac{A}{n_k}}|<\infty$; a random variable sequence $A_n$ is said to be $o_p\left(n^k\right)$ if ${\displaystyle \frac{A}{n_k}}{\to }_{p}0$ for $n\to \infty$ and is said to be $O_p\left(n^k\right)$ if ${sup}_{n}Pr({\displaystyle \frac{A_n}{n^k}}>M)\to 0$ for arbitrary $M\le \infty$ when $n\to \infty$.

2.2. Penalized Regression-Based Estimators

Here, we first introduce the existing penalized regression-based estimators that can be used to integrate the information from the two data sets. We consider minimizing the following object function to obtain an estimate of $\beta$, where a penalty is put on $\gamma$ only:

$$\begin{array}{ccc}\hfill L_{\lambda }(\beta ,\gamma )& =& -\sum _{i=1}^{n_S+n_B}\{y_{i}log{\displaystyle \frac{{\mu }_i}{1-{\mu }_i}}+log(1-{\mu }_i)\}+p_{\lambda }\left(\gamma \right)\hfill \\ & =& -\sum _{i=1}^{n_S+n_B}{y}_i{x}_i\left\{\beta +\gamma I(i\in B)\right\}-log\left\{1+exp\{\beta +\gamma I(i\in B\left)\right\}\right\}+p_{\lambda }\left(\gamma \right),\hfill \end{array}$$

where $p_{\lambda }\left(\gamma \right)$ is the penalty term. The estimators from the above optimization problem can be implemented using a penalized logistic regression such as glmnet in R 4.0 [8] with a design matrix that has a row like $(x_i,x_{i}I(i\in B))$ and the penalty factor $(0_p,1_p)$. Here, the form of the penalty term can be flexible, for example, $p_{\lambda }\left(\gamma \right)=\lambda {\sum }_{j=1}^p\left|{\gamma }_j\right|$ for LASSO $L_1$ penalty [9,10], $p_{\lambda }\left(\gamma \right)=\lambda {\sum }_{j=1}^p{\gamma }_j^2$ for ridge regression $L_2$ penalty. Other penalties like elastic net [11], SCAD [12], and MCP [13] can also be used, but for comparison purposes, we use $L_1$ and $L_2$ penalty to represent the performance of this class of estimators.

When the prediction in the small dataset is more important than the estimation of regression parameter $\beta$, instead of penalizing based on the parameter, we might penalize on the extra linear predictor $X_s\gamma$ and use $p_{\lambda }\left(X_s\gamma \right)$ to replace $p_{\lambda }\left(\gamma \right)$. The tuning parameter $\lambda$ could be determined via $K-$fold cross-validation [14] in the small data set.

2.3. Weighted Shrinkage Estimator

We propose an alternative approach to the penalized regression method via the weighted shrinkage method. This method has been shown to be useful under a linear regression model [2]; however, as we see in this section, the application of it to this nonlinear model is not straightforward.

The basic idea of this kind of weighted shrinkage estimator is to first fit the logistic regression model among the small data and the big data separately to obtain ${\widehat{\beta }}_S$ and ${\widehat{\beta }}_B$ as the estimator for $\beta$ and $\beta +\gamma$ and then combine the two estimators through a weighted average ${\widehat{\beta }}_W=W{\widehat{\beta }}_S+(I-W){\widehat{\beta }}_B$ for a specific weight matrix $W\in R^{p\times p}$. Specifically, when $W=I$, this is just the estimator of using small data only, and when $W={\displaystyle \frac{n_S}{n_S+n_B}}I$, this can be approximately viewed as a pooled estimator of the small and the big dataset assuming $\gamma =0$.

It is obvious that the performance of ${\widehat{\beta }}_W$ highly depends on the choice of weight matrix W. The major goal here is to find the optimal weight matrix as a function of $\beta$, $\gamma ,$ and data $X_S$, $X_B$, where $X_S\in R^{n_S\times p}$ and $X_B\in R^{n_B\times p}$ are design matrices for small and large data, respectively. Here, we define optimal weight by the weight that minimizes the coefficient estimation error $E\left|\right|{\widehat{\beta }}_W-\beta {\left|\right|}_2^2$.

To find the form for the optimal weight, we use the asymptotic expansion of ${\widehat{\beta }}_S$ and ${\widehat{\beta }}_B$. The optimal weight obtained via second-order approximation is denoted as $W_2(\beta ,\gamma ,X_S,X_B)$, and the optimal weight obtained via higher-order Edgeworth expansion [15] is denoted as $W_h(\beta ,\gamma ,X_S,X_B)$. For all these weights, we could plug in ${\widehat{\beta }}_S$ and ${\widehat{\beta }}_B-{\widehat{\beta }}_S$ for $\beta$ and $\gamma$ to obtain estimated version of these optimal weights.

Another existing estimator that we want to compare our newly proposed estimator with is the James–Stein estimator. For the James–Stein estimator, we use the form from [16], i.e, $W_{JS}=diag\left\{(1-c/F_i)\right\}$ or $W_{JS+}=diag\left\{{(1-c/F_i)}_{+}\right\}$, where $x_{+}=max\{0,x\}$ and $F_i$, $i=1,\cdots ,p$ is the test statistic for ${\widehat{\beta }}_{Si}={\widehat{\beta }}_{Bi}$. Here, we have $c={\displaystyle \frac{(p-2)(n-2)}{pn}}$.

Now, we provide more details on how to obtain these weights. We begin with the second-order approximation. Using the expansion of the logistic regression estimator, we have

$$\begin{array}{ccc}\hfill {\widehat{\beta }}_S& =& \beta +{\left(X_S^T{V}_S{X}_S\right)}^{-1}{X}_S^T(Y_S-{\mu }_S)+O_p\left(n_S^{-1}\right),\hfill \\ \hfill {\widehat{\beta }}_B& =& \beta +\gamma +{\left(X_B^T{V}_B{X}_B\right)}^{-1}{X}_B^T(Y_B-{\mu }_B)+O_p\left(n_B^{-1}\right),\hfill \end{array}$$

where $V_S={\mu }_S(1-{\mu }_S)$ are the variance of $Y_S$, and $V_B={\mu }_B(1-{\mu }_B)$ are the variance of $Y_B$. Ignoring the $O_p\left(n_S^{-1}\right)$ and $O_p\left(n_B^{-1}\right)$ terms in the above expansion lead to second-order optimal weight

$$W_2(\beta ,\gamma ,X_S,X_B)={(\gamma {\gamma }^T+{\Sigma }_B^{-1}+{\Sigma }_S^{-1})}^{-1}(\gamma {\gamma }^T+{\Sigma }_B^{-1}),$$

(2)

where ${\Sigma }_S=X_S^T{V}_S{X}_S$, ${\Sigma }_B=X_B^T{V}_B{X}_B$ and corresponding estimated second-order optimal weight ${\widehat{W}}_2(\widehat{\beta },\widehat{\gamma },X_S,X_B)$, where ${\widehat{W}}_2$ means the terms ${\Sigma }_S$ and ${\Sigma }_B$ are replaced by their consistent estimated version ${\widehat{\Sigma }}_S=X_S^{\top }{\widehat{V}}_S{X}_S$ and ${\widehat{\Sigma }}_B=X_B^{\top }{\widehat{V}}_B{X}_B$, where ${\widehat{V}}_S={\widehat{\mu }}_S(1-{\widehat{\mu }}_S)$ and ${\widehat{V}}_B={\widehat{\mu }}_B(1-{\widehat{\mu }}_B)$ with ${\widehat{\mu }}_S={\displaystyle \frac{exp\left\{X_S\widehat{\beta }\right\}}{1+exp\left\{X_S\widehat{\beta }\right\}}}$ and ${\widehat{\mu }}_B={\displaystyle \frac{exp\left\{X_B(\widehat{\beta }+\widehat{\gamma })\right\}}{1+exp\left\{X_B(\widehat{\beta }+\widehat{\gamma })\right\}}}$.

For the higher-order approximation, we use the approximation

$$\begin{array}{ccc}\hfill {\widehat{\beta }}_S& =& \beta +n_S^{-1/2}{B}_S+n_S^{-1}{C}_S+n_S^{-3/2}{D}_S+n_S^{-2}{E}_S+o_p\left(n_S^{-2}\right),\hfill \end{array}$$

(3)

$$\begin{array}{ccc}\hfill {\widehat{\beta }}_B& =& \beta +\gamma +n_B^{-1/2}{B}_B+n_B^{-1}{C}_B+n_B^{-3/2}{D}_B+n_B^{-2}{E}_B+o_p\left(n_B^{-2}\right),\hfill \end{array}$$

(4)

where $B_S=n_S^{1/2}{\left(X_S^T{V}_S{X}_S\right)}^{-1}{X}_S^T(Y_S-{\mu }_S)$, $B_B=n_B^{1/2}{\left(X_B^T{V}_B{X}_B\right)}^{-1}{X}_B^T(Y_B-{\mu }_B)$, and the expression of higher-order terms can be found in Appendix A. We have $E{B}_S=0$ and $Var\left(B_S\right)=n_S{\Sigma }_S^{-1}$, $E{B}_B=0$, and $Var\left(B_B\right)=n_B{\Sigma }_B^{-1}$. Denote $c_S=E{C}_S$, $v_S=Var\left(C_S\right)$, $d_S=E{D}_S$, $e_S=E{E}_S$, and ${\rho }_S=Cov(B_S,C_S)=<B_S,C_S>$, ${\nu }_S=Cov(B_S,D_S)=<B_S,D_S>$, $c_B=E{C}_B$, $v_B=Var\left(C_B\right)$, $d_B=E{D}_B$, $e_B=E{E}_B$ and ${\rho }_B=Cov(B_B,C_B)=<B_B,C_B>$, ${\nu }_B=Cov(B_B,D_B)=<B_B,D_B>$. Ignore $o_p\left(n_S^{-2}\right)$, $o_p\left(n_B^{-2}\right)$ terms; we have $E\left|\right|{\widehat{\beta }}_W-\beta {\left|\right|}_2^2$ is minimized at

$$W_h(\beta ,\gamma ,X_S,X_B)=(A_{11}-A_{10}){(A_{00}+A_{11}-A_{01}-A_{10})}^{-1},$$

(5)

where

$$\begin{array}{ccc}\hfill A_{00}& =& {\Sigma }_S^{-1}+n_S^{-2}(c_S{c}_S^T+v_S)+n_S^{-3/2}{\rho }_S+n_S^{-3/2}{\rho }_S^T+n_S^{-2}{\nu }_S+n_S^{-2}{\nu }_S^T,\hfill \\ \hfill A_{01}& =& n_S^{-1}{c}_S{\gamma }^T+n_S^{-3/2}{d}_S{\gamma }^T+n_S^{-2}{e}_S{\gamma }^T+n_S^{-1}{n}_B^{-1}{c}_S{c}_B^T,\hfill \\ \hfill A_{10}& =& n_S^{-1}\gamma c_S^T+n_S^{-3/2}\gamma d_S^T+n_S^{-2}\gamma e_S^T+n_S^{-1}{n}_B^{-1}{c}_B{c}_S^T,\hfill \\ \hfill A_{11}& =& \gamma {\gamma }^T+{\Sigma }_B^{-1}+n_B^{-2}(c_B{c}_B^T+v_B)+n_B^{-1}\gamma c_B^T+n_B^{-1}{c}_B{\gamma }^T+n_B^{-3/2}\gamma d_B^T+n_B^{-3/2}{d}_B{\gamma }^T\hfill \\ & & +n_B^{-2}\gamma e_B^T+n_B^{-2}{e}_B{\gamma }^T+n_B^{-3/2}{\rho }_B+n_B^{-3/2}{\rho }_B^T+n_B^{-2}{\nu }_B+n_B^{-2}{\nu }_B^T.\hfill \end{array}$$

The estimated version can be denoted as ${\widehat{W}}_h(\widehat{\beta },\widehat{\gamma },X_S,X_B)$, where ${\widehat{W}}_h$ means $c_S$, $c_B$, $d_S$, $d_B$, $e_S$, $e_B$, $v_S$, $v_B$, ${\rho }_S$, ${\rho }_B$, ${\nu }_S$, ${\nu }_B$ in $W_h$ are replaced by their consistent estimators (i.e., $\beta$ and $\gamma$ replaced by $\widehat{\beta }$ and $\widehat{\gamma }$ and expectation and variance replaced by sample mean and sample variance-covariance within the expression of $c_S$, $c_B$, $d_S$, $d_B$, $e_S$, $e_B$, $v_S$, $v_B$, ${\rho }_S$, ${\rho }_B$, ${\nu }_S$, ${\nu }_B$ terms).

2.4. Relationship Between Two Types of Estimators

The shrinkage estimator defined above is closely related to the penalized estimator. W can be written as $W_{\lambda }$ based on the $L_2$ penalty using the asymptotic linear expansion of GLM as in Equations (2) and (5) are the $O_p\left(1\right)$ term, whose form can be obtained from Edgeworth expansion [17]. To relate this to the penalized regression method, we consider the following penalized version and find ${\widehat{\beta }}_{\lambda }$ and ${\widehat{\gamma }}_{\lambda }$ minimize

$$\begin{array}{c}\hfill \left|\right|V_S^{-1/2}(Y_S-{\mu }_S){\left|\right|}_2^2+\left|\right|V_B^{-1/2}(Y_B-{\mu }_B){\left|\right|}_2^2+\left|\right|{\lambda }^{1/2}{X}_T\gamma {\left|\right|}_2^2.\end{array}$$

Denote $\mathcal{X}=\left(\begin{array}{cc}{X}_S& 0\\ X_B& X_B\\ 0& X_T\end{array}\right)$ and $\mathcal{Y}=\left(\begin{array}{c}{Y}_S-{\mu }_S\\ Y_B-{\mu }_B\\ 0\end{array}\right)$ and $\mathcal{V}=\left(\begin{array}{ccc}{V}_S& 0& 0\\ 0& V_B& 0\\ 0& 0& \lambda I\end{array}\right)$, the score function will be

$$\begin{array}{c}\hfill U={\mathcal{X}}^T\mathcal{Y},\end{array}$$

and the information matrix

$$\begin{array}{c}\hfill \mathcal{I}={\mathcal{X}}^T\mathcal{V}\mathcal{X}.\end{array}$$

So, we have the approximation

$$\begin{array}{ccc}\hfill \left(\begin{array}{c}{\widehat{\beta }}_{\lambda }\\ {\widehat{\gamma }}_{\lambda }\end{array}\right)& =& \left(\begin{array}{c}\beta \\ \gamma \end{array}\right)+{\left({\mathcal{X}}^T\mathcal{V}\mathcal{X}\right)}^{-1}{\mathcal{X}}^T\mathcal{Y}+O_p\left(n_S^{-1}\right).\hfill \end{array}$$

Define ${\Sigma }_T=X_T^T{X}_T$ and

$$W_{\lambda }={({\Sigma }_S+\lambda {\Sigma }_T+\lambda {\Sigma }_T{\Sigma }_B^{-1}{\Sigma }_S)}^{-1}({\Sigma }_S+\lambda {\Sigma }_T{\Sigma }_B^{-1}{\Sigma }_S).$$

Then, we have

$$\begin{array}{c}\hfill {\widehat{\beta }}_{\lambda }=W_{\lambda }{\widehat{\beta }}_S+(I-W_{\lambda }){\widehat{\beta }}_B+O_p\left(n_S^{-1}\right).\end{array}$$

So, we can see that with different choices of $X_T$ and $\lambda$, the $L_2$ penalized estimator is asymptotically equivalent to the weighted estimator.

3. Simulations

To see how much efficiency we can gain using our proposed estimator under a finite sample setting, we use a detailed simulation exercise, as described below, to compare the different methods with each other. In these simulations, we consider different sizes for p, the dimension for $\beta$ (including intercept), such that $p\in \{3,6,11\}$, but we set the $L_2$ norm of $\beta$, i.e., ${\left|\right|\beta \left|\right|}_2$ to be fixed at $plog\left(1.1\right)$. We also vary $\gamma$, the amount of bias in the big data such that ${\left|\right|\gamma \left|\right|}_2/{\left|\right|\beta \left|\right|}_2$ takes the following values $\{0.5,1,2\}$. We generate the small data $X_S$ and the big data $X_B$ from the same Gaussian distribution, and the covariates are assumed to be uncorrelated with each other. We also vary $n_S$, the size of the small data $X_S$, between 100 and 500 in increments of 50 (thus $n_S\in \{100,150,200,250,300,350,400,450,500\}$), while we consider two fixed sizes for $n_B$, namely {1000, 10,000}. For each simulation, we generate Y based on our assumed logistic models for the small and the big data, given in Equation (1). In addition, we also consider the setting where $\gamma$ is 0 and the setting where the bias of $\gamma$ is due to covariate $X_6$ missing in the big data.

For each simulation scenario, we perform 100 simulations to compute the mean squared error (MSE) in estimation, $E\left|\right|\widehat{\beta }-\beta {\left|\right|}_2^2$. We obtain estimates for $\beta$ by (1) using small data only (Small), (2) pooling big and small data (Pool), (3) weighted with optimal weight from second-order approximation ($W_2$), (4) weighted with optimal weight from higher-order approximation ($W_h$), (5) $L_1$ penalized regression ($L_1$), (6) $L_2$ penalized regression ($L_2$), or (7) JS+ weighted estimator (JSP).

Table 1 and Table 2 provides the mean square error (MSE) of the 7 estimators listed above under various small data sample sizes ($n_B$), number of covariates (p), and the magnitude of bias in the big data (${\left|\right|\gamma \left|\right|}_2/{\left|\right|\beta \left|\right|}_2$). The simulation results are further summarized in Figure 1 and Figure 2 to compare the efficiency gain between different methods and to show the trend of efficiency gain over the sample size of the small data. Specifically, Table 1 and Figure 1 provide results for different simulation settings when $n_B=1000$ while Table 2 and Figure 2 provide results for different simulation settings when $n_B$ = 10,000. In each figure, nine plots are presented in a grid of three rows and three columns, where the columns show plots for a particular ratio of ${\left|\right|\gamma \left|\right|}_2/{\left|\right|\beta \left|\right|}_2$ (in increasing order of magnitude from left to right), while the rows show plots for different dimension size p (in increasing order of magnitude from top to bottom). Figure 3 shows the setting without bias (i.e., $\gamma =0$), and Figure 4 shows the setting where the bias in the big data is due to missing of $X_6$. Each plot in the grid presents the graphs of the log-transformed ratio of the MSE of $\widehat{\beta }$, when we use each of the procedures (Small, Pool, $W_2$, $W_h$, $L_1$, $L_2$, JSP), versus that when we only use the small data (Small), as a function of the varying sizes for $n_S$. A specific curve below the horizontal zero line indicates that the estimator is consistently more efficient than the estimator from using small data only. When the curve for an estimator A is lower than the curve for another estimator B, it shows that estimator A is consistently more efficient than estimator B. From the plots, we can make the following observations:

Table 1. Mean square errors (MSE) of $\widehat{\beta }$ when we use each of the procedures (Small, Pool, $W_2$, $W_h$, $L_1$, $L_2$, JSP) for varying sizes of $n_S$, number of covariates p, and magnitude of bias ${\left|\right|\gamma \left|\right|}_2/{\left|\right|\beta \left|\right|}_2$ when $n_B$ is 1000.

${\mathit{n}}_{\mathit{S}}$	p	${\left|\right|\mathit{\gamma }\left|\right|}_2/{\left|\right|\mathit{\beta }\left|\right|}_2$	Small	Pool	${\mathit{W}}_2$	${\mathit{W}}_{\mathit{h}}$	${\mathit{L}}_1$	${\mathit{L}}_2$	JSP
100	3	0.5	0.125	0.007	0.008	0.008	0.006	0.050	0.084
100	3	1.0	0.125	0.022	0.020	0.020	0.016	0.048	0.084
100	3	2.0	0.125	0.085	0.044	0.044	0.051	0.067	0.105
100	6	0.5	0.329	0.050	0.039	0.039	0.042	0.113	0.191
100	6	1.0	0.329	0.140	0.049	0.049	0.090	0.126	0.228
100	6	2.0	0.329	0.443	0.054	0.054	0.171	0.159	0.276
100	11	0.5	0.673	0.161	0.075	0.075	0.124	0.197	0.364
100	11	1.0	0.673	0.506	0.081	0.080	0.286	0.253	0.491
100	11	2.0	0.673	1.630	0.087	0.086	0.477	0.375	0.680
200	3	0.5	0.068	0.010	0.008	0.008	0.008	0.043	0.046
200	3	1.0	0.068	0.030	0.017	0.017	0.019	0.043	0.050
200	3	2.0	0.068	0.098	0.025	0.025	0.044	0.050	0.065
200	6	0.5	0.141	0.054	0.035	0.035	0.041	0.088	0.086
200	6	1.0	0.141	0.141	0.037	0.037	0.075	0.094	0.119
200	6	2.0	0.141	0.463	0.040	0.040	0.113	0.107	0.155
200	11	0.5	0.294	0.146	0.057	0.057	0.108	0.160	0.196
200	11	1.0	0.294	0.445	0.058	0.058	0.182	0.178	0.265
200	11	2.0	0.294	1.332	0.064	0.064	0.249	0.221	0.342
300	3	0.5	0.041	0.009	0.007	0.007	0.008	0.031	0.029
300	3	1.0	0.041	0.027	0.014	0.014	0.016	0.030	0.032
300	3	2.0	0.041	0.083	0.019	0.019	0.027	0.033	0.040
300	6	0.5	0.095	0.050	0.031	0.031	0.039	0.070	0.064
300	6	1.0	0.095	0.135	0.034	0.034	0.064	0.073	0.089
300	6	2.0	0.095	0.418	0.036	0.036	0.081	0.077	0.114
300	11	0.5	0.170	0.122	0.045	0.045	0.086	0.114	0.131
300	11	1.0	0.170	0.334	0.049	0.049	0.130	0.123	0.179
300	11	2.0	0.170	0.935	0.054	0.054	0.154	0.141	0.195
400	3	0.5	0.033	0.008	0.006	0.006	0.007	0.026	0.023
400	3	1.0	0.033	0.022	0.012	0.012	0.013	0.025	0.026
400	3	2.0	0.033	0.068	0.016	0.016	0.023	0.026	0.034
400	6	0.5	0.064	0.041	0.024	0.024	0.032	0.052	0.051
400	6	1.0	0.064	0.102	0.025	0.025	0.046	0.052	0.066
400	6	2.0	0.064	0.306	0.028	0.028	0.057	0.054	0.078
400	11	0.5	0.131	0.122	0.046	0.046	0.085	0.104	0.116
400	11	1.0	0.131	0.347	0.051	0.051	0.113	0.112	0.153
400	11	2.0	0.131	0.967	0.057	0.057	0.129	0.122	0.159
500	3	0.5	0.026	0.009	0.007	0.007	0.008	0.022	0.020
500	3	1.0	0.026	0.022	0.012	0.012	0.013	0.021	0.023
500	3	2.0	0.026	0.061	0.014	0.014	0.019	0.022	0.027
500	6	0.5	0.052	0.036	0.023	0.023	0.028	0.043	0.042
500	6	1.0	0.052	0.094	0.023	0.023	0.041	0.045	0.058
500	6	2.0	0.052	0.284	0.025	0.025	0.047	0.045	0.065
500	11	0.5	0.101	0.099	0.037	0.037	0.065	0.083	0.092
500	11	1.0	0.101	0.288	0.038	0.038	0.084	0.086	0.119
500	11	2.0	0.101	0.794	0.042	0.042	0.097	0.090	0.120

Table 2. Mean square errors (MSE) of $\widehat{\beta }$ when we use each of the procedures (Small, Pool, $W_2$, $W_h$, $L_1$, $L_2$, JSP) for varying sizes of $n_S$, number of covariates p, and magnitude of bias ${\left|\right|\gamma \left|\right|}_2/{\left|\right|\beta \left|\right|}_2$ when $n_B$ is 10,000.

${\mathit{n}}_{\mathit{S}}$	p	${\left|\right|\mathit{\gamma }\left|\right|}_2/{\left|\right|\mathit{\beta }\left|\right|}_2$	Small	Pool	${\mathit{W}}_2$	${\mathit{W}}_{\mathit{h}}$	${\mathit{L}}_1$	${\mathit{L}}_2$	JSP
100	3	0.5	0.125	0.006	0.007	0.007	0.006	0.050	0.083
100	3	1.0	0.125	0.022	0.019	0.019	0.017	0.054	0.087
100	3	2.0	0.125	0.085	0.039	0.039	0.054	0.068	0.103
100	6	0.5	0.329	0.040	0.022	0.022	0.029	0.108	0.189
100	6	1.0	0.329	0.152	0.033	0.033	0.080	0.122	0.227
100	6	2.0	0.329	0.594	0.038	0.038	0.177	0.161	0.280
100	11	0.5	0.673	0.148	0.037	0.037	0.101	0.184	0.350
100	11	1.0	0.673	0.585	0.045	0.045	0.276	0.240	0.493
100	11	2.0	0.673	2.307	0.046	0.047	0.479	0.388	0.669
200	3	0.5	0.068	0.009	0.008	0.008	0.007	0.043	0.046
200	3	1.0	0.068	0.031	0.015	0.015	0.020	0.045	0.053
200	3	2.0	0.068	0.115	0.020	0.020	0.044	0.050	0.063
200	6	0.5	0.141	0.046	0.017	0.017	0.030	0.086	0.086
200	6	1.0	0.141	0.179	0.020	0.020	0.072	0.094	0.117
200	6	2.0	0.141	0.699	0.021	0.021	0.116	0.109	0.156
200	11	0.5	0.294	0.156	0.023	0.023	0.090	0.152	0.190
200	11	1.0	0.294	0.632	0.026	0.026	0.176	0.181	0.264
200	11	2.0	0.294	2.379	0.026	0.026	0.265	0.232	0.327
300	3	0.5	0.041	0.008	0.006	0.006	0.006	0.031	0.029
300	3	1.0	0.041	0.029	0.010	0.010	0.015	0.031	0.033
300	3	2.0	0.041	0.110	0.013	0.013	0.027	0.032	0.040
300	6	0.5	0.095	0.047	0.015	0.015	0.030	0.069	0.064
300	6	1.0	0.095	0.178	0.018	0.018	0.062	0.072	0.088
300	6	2.0	0.095	0.702	0.019	0.019	0.084	0.077	0.112
300	11	0.5	0.170	0.150	0.012	0.012	0.069	0.113	0.122
300	11	1.0	0.170	0.594	0.013	0.013	0.132	0.123	0.174
300	11	2.0	0.170	2.343	0.014	0.014	0.155	0.143	0.185
400	3	0.5	0.033	0.007	0.005	0.005	0.006	0.026	0.024
400	3	1.0	0.033	0.026	0.009	0.009	0.013	0.026	0.027
400	3	2.0	0.033	0.105	0.011	0.011	0.023	0.027	0.032
400	6	0.5	0.064	0.045	0.010	0.010	0.024	0.051	0.046
400	6	1.0	0.064	0.172	0.011	0.011	0.044	0.052	0.067
400	6	2.0	0.064	0.665	0.012	0.012	0.057	0.054	0.079
400	11	0.5	0.131	0.157	0.017	0.017	0.076	0.101	0.109
400	11	1.0	0.131	0.625	0.019	0.019	0.115	0.116	0.142
400	11	2.0	0.131	2.349	0.019	0.019	0.130	0.123	0.147
500	3	0.5	0.026	0.009	0.005	0.005	0.007	0.022	0.020
500	3	1.0	0.026	0.029	0.008	0.008	0.012	0.022	0.024
500	3	2.0	0.026	0.113	0.009	0.009	0.018	0.022	0.027
500	6	0.5	0.052	0.046	0.009	0.009	0.023	0.043	0.040
500	6	1.0	0.052	0.175	0.011	0.011	0.039	0.044	0.057
500	6	2.0	0.052	0.697	0.011	0.011	0.048	0.045	0.065
500	11	0.5	0.101	0.153	0.012	0.012	0.061	0.083	0.086
500	11	1.0	0.101	0.562	0.013	0.013	0.086	0.088	0.111
500	11	2.0	0.101	2.259	0.014	0.014	0.099	0.095	0.113

Figure 1. Plot for the log-transformed ratios of the mean squared error of $\widehat{\beta }$, when we use each of the procedures (Small, Pool, $W_2$, $W_h$, $L_1$, $L_2$, JSP), versus that when we only use the small data (Small), for varying sizes of $n_S$ and when $n_B$ is 1000.

Figure 2. Plot for the log-transformed ratios of the mean squared error of $\widehat{\beta }$, when we use each of the procedures (Small, Pool, $W_2$, $W_h$, $L_1$, $L_2$, JSP), versus that when we only use the small data (Small), for varying sizes of $n_S$ and when $n_B$ is 10,000.

Figure 3. Plot for the log-transformed ratios of the mean squared error of $\widehat{\beta }$, when we use each of the procedures (Small, Pool, $W_2$, $W_h$, $L_1$, $L_2$, JSP), versus that when we only use the small data (Small), for varying sizes of $n_S$ and when $\gamma =0$.

Figure 4. Plot for the log-transformed ratios of the mean squared error of $\widehat{\beta }$, when we use each of the procedures (Small, Pool, $W_2$, $W_h$, $L_1$, $L_2$, JSP), versus that when we only use the small data (Small), for varying sizes of $n_S$ and when the bias term $\gamma$ is caused by missing the covariate $X_6$ in big data.

• The performance of $W_2$ and $W_h$ procedures are very close to each other in every setting, pointing to the fact that probably the optimal second-order approximation weights $W_2$ suffice for our problem (in fact it is difficult to visually observe the graph for $W_2$ as it is exactly overlaid by the graph for $W_h$).
• In every simulation setting, the $W_2$ and the $W_h$ procedures outperform every other method. The gain in performance of $W_2$ and $W_h$ over the next best-performing method increases with increasing dimension size p, and increasing ratio of ${\left|\right|\gamma \left|\right|}_2/{\left|\right|\beta \left|\right|}_2$. The same trend is observed for both sizes for $n_B$ (1000/10,000).
• The $L_1$ penalized procedure is the third best-performing method overall (after $W_2$ and $W_h$), and its performance is similar to the $W_2$/$W_h$ procedures when dimension size is small ($p=3$), and the relative bias is low (${\left|\right|\gamma \left|\right|}_2/{\left|\right|\beta \left|\right|}_2=0.5$).
• The pooled procedure is the worst-performing method overall and is quite sensitive to the bias ${\left|\right|\gamma \left|\right|}_2$. Although it shows relatively good performance when dimension size is small ($p=3$), and when the relative bias is low (${\left|\right|\gamma \left|\right|}_2/{\left|\right|\beta \left|\right|}_2=0.5$), with increasing dimensions, and especially with increasing bias, its performance becomes very poor. Apart from JSPs in some scenarios, it is the only procedure that shows extremely elevated MSEs in comparison to the small data.
• The performance of the $L_2$ procedure is similar to JSP in some settings but is better than it in others. For example, with increasing dimensions, and with increasing bias, the JSP procedure sometimes tends to have higher MSE than those obtained from the small data (Small), especially when the size of the small data is on the higher end, but the $L_2$ procedure always performs better than Small.
• All methods (except for Pool and in some instances JSP) show lower MSE than the estimates obtained from the small data themselves, and the gain in efficiency is most pronounced when the size of the small data is small.

4. Theoretical Results

The simulation results indicate that our proposed estimator always outperforms the small-data-only analysis in terms of MSE and such improvement is substantial sometimes. However, to apply our proposed method in general, a natural question is whether there exists a certain scenario under which the proposed estimator will underperform the small-data-only analysis, especially when the difference between the two sources of data is large. To answer this question, here we summarize the theoretical guarantee of our proposed weighted estimators in the following theorems with brief proof idea and the detailed expression and proof can be found in the Appendix A.

Theorem 1.

The second-order optimal weight $W_2(\beta ,\gamma ,X_S,X_B)$ and its estimated version $W_2(\widehat{\beta },\widehat{\gamma },X_S,X_B)$ approximately minimize $E\left|\right|{\widehat{\beta }}_W-\beta {\left|\right|}_2^2$ at $O\left(n_S^{-1}\right)$ level in the sense that

$$\begin{array}{cc}& E\left|\right|{\widehat{\beta }}_{W_2(\beta ,\gamma ,X_S,X_B)}-{\beta \left|\right|}_2^2\le {\inf }_{W}E\left|\right|{\widehat{\beta }}_W-\beta {\left|\right|}_2^2+O\left(n_S^{-1}\right)\\ & E\left|\right|{\widehat{\beta }}_{{\widehat{W}}_2(\widehat{\beta },\widehat{\gamma },X_S,X_B)}-{\beta \left|\right|}_2^2\le {\inf }_{W}E\left|\right|{\widehat{\beta }}_W-\beta {\left|\right|}_2^2+O\left(n_S^{-1}\right),\end{array}$$

where the infimum is taken over all $p\times p$ random matrix that are measurable given $X_S,X_B,Y$.

$W_2(\beta ,\gamma ,X_S,X_B)$ and ${\widehat{W}}_2(\widehat{\beta },\widehat{\gamma },X_S,X_B)$ are also approximate optimal weights for prediction purpose in the sense that

$$\begin{array}{cc}& n_S^{-1}E\left|\right|X_S({\widehat{\beta }}_{W_2(\beta ,\gamma ,X_S,X_B)}-\beta ){\left|\right|}_2^2\le {\inf }_W{n}_S^{-1}E\left|\right|X_S({\widehat{\beta }}_W-\beta ){\left|\right|}_2^2+O\left(n_S^{-1}\right),\\ & n_S^{-1}E\left|\right|X_S({\widehat{\beta }}_{{\widehat{W}}_2(\widehat{\beta },\widehat{\gamma },X_S,X_B)}-\beta ){\left|\right|}_2^2\le {\inf }_W{n}_S^{-1}E\left|\right|X_S({\widehat{\beta }}_W-\beta ){\left|\right|}_2^2+O\left(n_S^{-1}\right).\end{array}$$

For the proof of Theorem 1, we can use the second order expansion of $\widehat{\beta }$ around $\beta$ to obtain the optimal weight as a function of $\beta$ and $\gamma$. Since the approximation error for $\widehat{\beta }$ will be at the level of $O_p\left(n_S^{-1}\right)$, the mean square error for $\beta$ and mean square prediction error can be approximated with error at level $O\left(n_S^{-1}\right)$. Replacing optimal weight with estimated optimal weight will further introduce an error term of $O\left(n_s^{-2}\right)$, but the term $O\left(n_S^{-1}\right)$ will still be the dominant term. Details can be found in Appendix A.

Theorem 2.

The higher-order optimal weight $W_h(\beta ,\gamma ,X_S,X_B)$ and its estimated version ${\widehat{\beta }}_{{\widehat{W}}_h(\widehat{\beta },\widehat{\gamma },X_S,X_B)}$ approximately minimize $E\left|\right|{\widehat{\beta }}_W-\beta {\left|\right|}_2^2$ at $o\left(n_S^{-2}\right)$ and $O\left(n_S^{-2}\right)$ level in the sense that

$$\begin{array}{cc}& E\left|\right|{\widehat{\beta }}_{W_h(\beta ,\gamma ,X_S,X_B)}{\left|\right|}_2^2\le {\inf }_{W}E\left|\right|{\widehat{\beta }}_W-\beta {\left|\right|}_2^2+o\left(n_S^{-2}\right),\\ & E\left|\right|{\widehat{\beta }}_{{\widehat{W}}_h(\widehat{\beta },\widehat{\gamma },X_S,X_B)}{\left|\right|}_2^2\le {\inf }_{W}E\left|\right|{\widehat{\beta }}_W-\beta {\left|\right|}_2^2+O\left(n_S^{-2}\right).\end{array}$$

Similarly, for Theorem 2, we can use the high order expansion of $\widehat{\beta }$ around $\beta$ with approximation error at the level of $o_p\left(n_S^{-3/2}\right)$, then the approximation error for the mean square error for $\beta$ and mean square prediction error will be at the level of $o\left(n_S^{-2}\right)$. In addition, we can show that the difference between MSE based on this estimated optimal weight and the optimal oracle weight is with an approximation error of the order of $o\left(n_S^{-2}\right)$ so that the final approximation error will be at the level of $O\left(n_S^{-2}\right)$.

Theorem 3.

Assuming $n_B/n_S\to r\in (0,\infty )$ and $n_S\to \infty$, the weighted estimator based on estimated higher-order optimal weights is more efficient than using small data only, i.e., $E\left|\right|{\widehat{\beta }}_{{\widehat{W}}_h(\widehat{\beta },\widehat{\gamma },X_S,X_B)}-{\beta \left|\right|}_2^2\le E\left|\right|{\widehat{\beta }}_I-\beta {\left|\right|}_2^2$ hold asymptotically when $n_S\to \infty$.

For Theorem 3, since using small data only is equivalent to using weight matrix I, we can show that the improvement from the estimated higher-order optimal weight is always larger than the estimated second-order optimal weight which is in the order of $O\left(n_S^{-2}\right)$ and thus approximation error $o\left(n_S^{-2}\right)$ can be ignored when comparing $E\left|\right|{\widehat{\beta }}_{{\widehat{W}}_h(\widehat{\beta },\widehat{\gamma },X_S,X_B)}-\beta {\left|\right|}_2^2$ and $E\left|\right|{\widehat{\beta }}_I-\beta {\left|\right|}_2^2$.

5. Analysis of ACC Data

The Asia Cohort Consortium (ACC) is a collaborative effort born out of the need to study the Asian population, seeking to understand the relationship between genetics, environmental exposures, and the etiology of a disease through the establishment of a cohort of at least one million healthy people around different countries in Asia, followed over time to various disease endpoints and death. This pooling project, with its huge sample size across 29 subcohorts from 10 Asian countries (https://www.asiacohort.org/ParticipatingCohorts/index.html (accessed on 23 March 2017)), provides the perfect opportunity to explore informative relationships (association of exposure with disease, genome variability with disease, etc.) among major Asian ethnic groups.

Over the last few decades, obesity has become an important health issue in many countries. According to World Health Organization estimates, more than a billion adults around the world are overweight, and at least 300 million of them are obese (see [18]). Many epidemiological studies have found an association between the body mass index (BMI) and a variety of health outcomes, including mortality (see [19]). However, most of these inferences have been drawn from studies in populations of European origins, and very little focus has been given to the relationship between BMI and the overall risk of death among Asians, who account for more than 60% of the world population (see [20]). The data collected as part of the ACC can be used to answer these important questions.

To show the usefulness of our proposed methodology in a practical setting, we use data from the ACC to explore the relationship between BMI and mortality. In particular, we concentrate only on the cohorts from China—data from the Shanghai Cohort Study (SCS) are used to form our small data—while data from the rest of the Chinese subcohorts—China Hypertension Survey Epidemiology Follow-up Study (CHEFS), Linxian General Population Trial Cohort, Shanghai Men’s Health Study (SMHS), and Shanghai Women’s Health Study (SWHS)—are pooled together to form the initial big data. Since the SCS cohort only included males, we decided to restrict the big data to include only male participants from the other subcohorts (which completely excluded the SWHS). For individuals in the small data, enrollment started in 1986 and the study continued till 2007, while for the pooled large data, enrollment started in 1985, and the last year of follow-up was 2011. Missingness in covariates is not a big concern (no missingness in the small data and only 0.79% missingness in the large data). The baseline age distribution of the individuals is found to be different in the small and the large data, and since mortality is a definite function of age, for better comparability, we decided to restrict the two datasets such that they contain individuals whose baseline age varied between 50 and 60. And because methods described in this paper pertain to binary outcomes only, and time to follow-up varies for different individuals in the two datasets, we decided to only consider the first year of follow-up for each individual. Firstly, this makes the binary statuses of mortality comparable for individuals in the two datasets, and secondly, the short period of follow-up ensures that we do not lose too many individuals who are lost to follow-up. Such individuals form only 0.06% of the small data and 2.88% of the large data and are removed from the analysis. After performing all these data management steps, the small data are found to contain 10,675 individuals with 40 mortality events, while the large data are found to contain 46,779 individuals with 206 events. Apart from BMI, baseline age is also included as a covariate in the model, as well as indicators for each individual’s smoking and drinking habits, as these covariates have been proven to be important predictors of mortality in many settings.

We start off by analyzing the small and the big data separately first, using the standard logistic regression model, and then pooling them together. We then estimate the regression coefficients using the proposed weighted shrinkage methods, namely, with the optimal second-order weights ($W_2$), the optimal higher-order Edgeworth weights ($W_h$), and for comparison, the optimal James–Stein weights (JSPs). We also obtained the penalized estimates, using the $L_1$ and $L_2$ procedures.

The estimates, their standard errors, and estimated mean square errors for the various procedures are presented below in Table 3. Unlike the simulation data, we cannot know the true bias, and thus, we cannot obtain the true MSE. To illustrate the efficiency gain of the proposed estimators, we estimated MSE for each method by approximating the bias by the difference between the point estimate of each method and the small data estimator. From the table, we can see that the two proposed weighting estimators $W_2$ and $W_h$ provide smaller estimated MSEs than the estimator using small data only. These numerical results match the theoretical results from Theorem 3 as expected, since we have a large sample size here for $n_B$ and $n_S$ to make the asymptotic results hold. In addition, we noticed that except for the intercept, the two proposed weighting estimators $W_2$ and $W_h$ also provide smaller estimated MSE than other alternative estimators which is consistent with the simulation results. As can be seen, the pooled procedure obtains the lowest standard errors as expected, because it uses the entirety of the big and the small data, but it also means that the estimates for this procedure are inherently biased towards the ones that we obtain from the big data themselves, as it contains a lot more information (than the small data) because of its size, so naive pooling inappropriately shifts most of the focus to the big data themselves. The weighted shrinkage procedures seem to be better adjusted in this respect, with estimates shrunk somewhat but much closer to the ones that we see from the small data themselves but with much lower standard errors than the small data estimates. The optimal second-order and higher-order Edgeworth weights ($W_2$ and $W_h$) we proposed perform similarly in this regard and have lower standard errors than the estimates from the James–Stein-adjusted weights (JSPs), except for BMI, in which case the JSP procedure obtains a lower standard error than $W_2$ or $W_h$; however, the estimate for BMI obtained by JSP is shrunk completely to that obtained from the big data themselves, which implies potential large bias. Similarly, for other variables, the JSP estimate is far from the small data estimator compared with our two proposed weighted estimators which suggests a potential larger bias. Among the penalized procedures, $L_1$ seems to borrow more strength from the big data and thus has lower standard errors and a higher amount of shrinkage, which leads to potential higher bias, while the estimates for the $L_2$ procedure seem to be closer to the small data estimates and thus have higher standard errors compared with the proposed weighted estimators. In general, the two proposed weighted estimators show a better balance of bias and SD in this example.

Table 3. Estimates (Est), their standard errors (Std Err), and estimated mean square error (Est MSE) from ACC data analysis.

		Small	Big	Pool	${\mathit{W}}_2$	${\mathit{W}}_{\mathit{h}}$	JSP	${\mathit{L}}_1$	${\mathit{L}}_2$
(Intercept)	Est	−4.80	−9.89	−9.08	−5.25	−5.26	−6.27	−9.04	−4.85
	Std Err	3.21	1.48	1.34	1.88	1.88	2.32	1.42	1.77
	Est MSE	10.30	28.10	20.11	3.74	3.75	7.54	20.00	3.14
Age at Baseline	Est	0.03	0.12	0.10	0.04	0.04	0.05	0.10	0.03
	Std Err	0.05	0.02	0.02	0.03	0.03	0.04	0.02	0.04
	Est MSE	0.003	0.009	0.005	0.001	0.001	0.002	0.005	0.002
Body Mass Index	Est	−0.14	−0.09	−0.09	−0.13	−0.13	−0.09	−0.12	−0.14
	Std Err	0.06	0.02	0.02	0.03	0.03	0.02	0.03	0.05
	Est MSE	0.004	0.003	0.003	0.001	0.001	0.003	0.001	0.003
Ever Smoked	Est	0.71	0.08	0.20	0.65	0.65	0.55	0.62	0.71
	Std Err	0.38	0.16	0.14	0.22	0.22	0.29	0.30	0.40
	Est MSE	0.14	0.42	0.28	0.05	0.05	0.11	0.10	0.16
Ever Used Alcohol	Est	0.06	−0.37	−0.30	0.02	0.02	−0.13	−0.02	0.06
	Std Err	0.33	0.16	0.14	0.18	0.18	0.20	0.27	0.31
	Est MSE	0.11	0.21	0.15	0.03	0.03	0.08	0.08	0.10

6. Discussion

In this paper, we proposed better estimators that allow more accurate estimation of the regression coefficient and the risk prediction for our target population using information from another different population with more observations. Although the expansion and detailed form of the weight we provided are specifically for logistic regression, the optimal weight formula is general in terms of the expansion formula C, D, Es. So the framework we proposed here could be extended to a generalized linear model and estimating equation models straightforwardly, though the more complicated computation of Edgeworth expansion for these estimating equation-based estimators needs to be derived for the optimal estimation weight.

To utilize big data, although we do not need to know the exact relationship between the small and big data in terms of association strength, we need the model form in big data to be correctly specified. In our setting, the same logistic form needs to hold for both the big and the small data. When the covariate is limited and categorical, this assumption is weak and easy to satisfy. When there is a continuous covariate, we can apply existing model-checking tools to the big data to check whether our model assumption holds.

In our analysis of the ACC data, we only concentrated on the first year of follow-up, because the methods presented in this paper are only relevant for binary outcomes, and the short period of follow-up ensured that we did not lose too many individuals to loss to follow-up, which would have otherwise introduced unforeseen sources of bias in our analysis. However, in doing so, we lost a lot of rich information that is contained in the time-to-follow-up data. This shows the need to extend our methods to the case when we have time-to-event data, and this indeed is one of our future research goals.

Appendix A.1. High-Order Expansion for Logistic Regression

In this section, we derive the high-order asymptotic expansion for the maximum likelihood estimator $\widehat{\beta }$ based on n i.i.d. data $(x_i,y_i)$ sampled from the logistic regression model $log{\displaystyle \frac{P(Y=1)}{P(Y=0)}}=X\beta$. The small and big data expansion can be obtained by replacing X by $X_S$ and $X_B$, replacing $\beta$ by $\beta$ and $\beta +\gamma$ and replacing n by $n_S$ and $n_B$.

For logistic regression, we have the score function written as ${\dot{l}}_n\left(\beta \right)=X^T\left\{y-{\displaystyle \frac{exp\left(X^T\beta \right)}{1+exp\left(X^T\beta \right)}}\right\}$. Let $Z_n={\displaystyle \frac{1}{\sqrt{n}}}{\dot{l}}_n\left({\beta }_0\right)$ and $\Sigma =Var\left({\dot{l}}_n\right)$, then $(\Sigma /n)Z_n$ converge to standard normal distribution. So, following Sun, Loader, McCormick [17], we have that $\left|\right|{\widehat{\beta }}_2-\widehat{\beta }{\left|\right|}_2=o_p\left(n^{-1}\right)$ where

$$\begin{array}{c}\hfill {\widehat{\beta }}_2={\beta }_0+{(\Sigma /n)}^{-1}\left\{n^{-1/2}{Z}_n+{\displaystyle \frac{1}{2n}}\left(\begin{array}{c}{b}_1\\ \cdots \\ b_p\end{array}\right)\right\}\end{array}$$

with

$$\begin{array}{ccc}\hfill b_k& =& {\left[{\Sigma }^{-1}{X}^T\left\{y-{\displaystyle \frac{exp\left(X^T\beta \right)}{1+exp\left(X^T\beta \right)}}\right\}\right]}^T{\stackrel{⃛}{l}}_k\left[{\Sigma }^{-1}{X}^T\left\{y-{\displaystyle \frac{exp\left(X^T\beta \right)}{1+exp\left(X^T\beta \right)}}\right\}\right]\hfill \\ & =& {\left\{{(\Sigma /n)}^{-1}{Z}_n\right\}}^T{\displaystyle \frac{{\stackrel{⃛}{l}}_k}{n}}\left\{{(\Sigma /n)}^{-1}{Z}_n\right\}+o_p\left(1\right),\hfill \end{array}$$

where ${\stackrel{⃛}{l}}_k$ is a $p\times p$ matrix with the $(i,j)$th element equal to ${\stackrel{⃛}{l}}_{ijk}$.

Similarly, we have $\left|\right|{\widehat{\beta }}_3-\widehat{\beta }{\left|\right|}_2=o_p\left(n^{-3/2}\right)$, where

$$\begin{array}{ccc}\hfill {\widehat{\beta }}_3& =& {\beta }_0+{(\Sigma /n)}^{-1}\left\{n^{-1/2}{Z}_n+{\displaystyle \frac{1}{2n}}\left(\begin{array}{c}{\tilde{b}}_1\\ \cdots \\ {\tilde{b}}_p\end{array}\right)+{\displaystyle \frac{1}{6n^{3/2}}}\left(\begin{array}{c}{d}_1\\ \cdots \\ d_p\end{array}\right)\right\}\hfill \\ & =& {\beta }_0+{(\Sigma /n)}^{-1}\left\{n^{-1/2}{Z}_n+{\displaystyle \frac{1}{2n}}\left(\begin{array}{c}{b}_1\\ \cdots \\ b_p\end{array}\right)+{\displaystyle \frac{1}{6n^{3/2}}}\left(\begin{array}{c}{\tilde{d}}_1\\ \cdots \\ {\tilde{d}}_p\end{array}\right)\right\}+o_p\left(n^{-3/2}\right)\hfill \end{array}$$

where

$${\tilde{b}}_k={\left[{(\Sigma /n)}^{-1}\left\{Z_n+{\displaystyle \frac{1}{2\sqrt{n}}}\left(\begin{array}{c}{b}_1\\ \cdots \\ b_p\end{array}\right)\right\}\right]}^T{\displaystyle \frac{{\stackrel{⃛}{l}}_k}{n}}\left[{(\Sigma /n)}^{-1}\left\{Z_n+{\displaystyle \frac{1}{2\sqrt{n}}}\left(\begin{array}{c}{b}_1\\ \cdots \\ b_p\end{array}\right)\right\}\right]$$

and

$$d_k={\left[{(\Sigma /n)}^{-1}\left\{Z_n+{\displaystyle \frac{1}{2\sqrt{n}}}\left(\begin{array}{c}{b}_1\\ \cdots \\ b_p\end{array}\right)\right\}\right]}^T{m}_k\left[{(\Sigma /n)}^{-1}\left\{Z_n+{\displaystyle \frac{1}{2\sqrt{n}}}\left(\begin{array}{c}{b}_1\\ \cdots \\ b_p\end{array}\right)\right\}\right]$$

where

$$m_k=\sum _j{e}_j{\left\{{(\Sigma /n)}^{-1}\left[Z_n+{\displaystyle \frac{1}{2\sqrt{n}}}\left(\begin{array}{c}{b}_1\\ \cdots \\ b_p\end{array}\right)\right]\right\}}^T{\displaystyle \frac{{\stackrel{⃜}{l}}_{kj}}{n}}$$

where ${\stackrel{⃜}{l}}_{kj}$ is a $p\times p$ matrix with the $(i,l)$th element equal to ${\stackrel{⃜}{l}}_{ilkj}$ and

$$\begin{array}{ccc}\hfill {\tilde{d}}_k& =& d_k+3Z_n^T{\displaystyle \frac{{\stackrel{⃛}{l}}_k}{n}}b\hfill \\ & \approx & {(\Sigma /n)}^{-1}{\left\{{(\Sigma /n)}^{-1}{Z}_n\right\}}^T\left[\sum _j{\left\{e_j{(\Sigma /n)}^{-1}{Z}_n\right\}}^T{\displaystyle \frac{{\stackrel{⃜}{l}}_{kj}}{n}}\right]\left\{{(\Sigma /n)}^{-1}{Z}_n\right\}\hfill \\ & & +6Z_n^T{\displaystyle \frac{{\stackrel{⃛}{l}}_k}{n}}\left(\begin{array}{c}{Z}_n^T{\displaystyle \frac{{\stackrel{⃛}{l}}_1}{n}}{Z}_n\\ \cdots \\ Z_n^T{\displaystyle \frac{{\stackrel{⃛}{l}}_p}{n}}{Z}_n\end{array}\right).\hfill \end{array}$$

Now, we have $\left|\right|{\widehat{\beta }}_4-\widehat{\beta }{\left|\right|}_2=o_p\left(n^{-2}\right)$, where

$$\begin{array}{ccc}\hfill {\widehat{\beta }}_4& =& {\beta }_0+{(\Sigma /n)}^{-1}\left\{n^{-1/2}{Z}_n+{\displaystyle \frac{1}{2n}}\left(\begin{array}{c}{\tilde{b}}_1^{\left(2\right)}\\ \cdots \\ {\tilde{b}}_p^{\left(2\right)}\end{array}\right)+{\displaystyle \frac{1}{6n^{3/2}}}\left(\begin{array}{c}{\tilde{d}}_1^{\left(2\right)}\\ \cdots \\ {\tilde{d}}_p^{\left(2\right)}\end{array}\right)+{\displaystyle \frac{1}{24n^2}}\left(\begin{array}{c}{\tilde{E}}_1^{\left(2\right)}\\ \cdots \\ {\tilde{E}}_p^{\left(2\right)}\end{array}\right)\right\}\hfill \\ & \approx & {\beta }_0+{(\Sigma /n)}^{-1}\left\{n^{-1/2}{Z}_n+{\displaystyle \frac{1}{2n}}\left(\begin{array}{c}{b}_1\\ \cdots \\ b_p\end{array}\right)+{\displaystyle \frac{1}{6n^{3/2}}}\left(\begin{array}{c}{\tilde{d}}_1\\ \cdots \\ {\tilde{d}}_p\end{array}\right)+{\displaystyle \frac{1}{24n^2}}\left(\begin{array}{c}{\tilde{E}}_1\\ \cdots \\ {\tilde{E}}_p\end{array}\right)\right\}\hfill \end{array}$$

where

$$\begin{array}{ccc}\hfill {\tilde{b}}_k^{\left(2\right)}& =& {\left[{(\Sigma /n)}^{-1}\left\{Z_n+{\displaystyle \frac{1}{2\sqrt{n}}}\left(\begin{array}{c}{b}_1\\ \cdots \\ b_p\end{array}\right)+{\displaystyle \frac{1}{6n}}\left(\begin{array}{c}{\tilde{d}}_1\\ \cdots \\ {\tilde{d}}_p\end{array}\right)\right\}\right]}^T{\displaystyle \frac{{\stackrel{⃛}{l}}_k}{n}}\hfill \\ & & \left[{(\Sigma /n)}^{-1}\left\{Z_n+{\displaystyle \frac{1}{2\sqrt{n}}}\left(\begin{array}{c}{b}_1\\ \cdots \\ b_p\end{array}\right)+{\displaystyle \frac{1}{6n}}\left(\begin{array}{c}{\tilde{d}}_1\\ \cdots \\ {\tilde{d}}_p\end{array}\right)\right\}\right]\hfill \end{array}$$

and

$$\begin{array}{ccc}\hfill {\tilde{d}}_k^{\left(2\right)}& =& {\left[{(\Sigma /n)}^{-1}\left\{Z_n+{\displaystyle \frac{1}{2\sqrt{n}}}\left(\begin{array}{c}{b}_1\\ \cdots \\ b_p\end{array}\right)+{\displaystyle \frac{1}{6n}}\left(\begin{array}{c}{\tilde{d}}_1\\ \cdots \\ {\tilde{d}}_p\end{array}\right)\right\}\right]}^T{m}_k\hfill \\ & & \left[{(\Sigma /n)}^{-1}\left\{Z_n+{\displaystyle \frac{1}{2\sqrt{n}}}\left(\begin{array}{c}{b}_1\\ \cdots \\ b_p\end{array}\right)+{\displaystyle \frac{1}{6n}}\left(\begin{array}{c}{\tilde{d}}_1\\ \cdots \\ {\tilde{d}}_p\end{array}\right)\right\}\right]\hfill \end{array}$$

where

$$m_k=\sum _j{\left[e_j{(\Sigma /n)}^{-1}\left\{Z_n+{\displaystyle \frac{1}{2\sqrt{n}}}\left(\begin{array}{c}{b}_1\\ \cdots \\ b_p\end{array}\right)+{\displaystyle \frac{1}{6n}}\left(\begin{array}{c}{\tilde{d}}_1\\ \cdots \\ {\tilde{d}}_p\end{array}\right)\right\}\right]}^T{\displaystyle \frac{{\stackrel{⃜}{l}}_{kj}}{n}}$$

$$\begin{array}{ccc}\hfill {\tilde{E}}_k^{\left(2\right)}& =& {\left[{(\Sigma /n)}^{-1}\left\{Z_n+{\displaystyle \frac{1}{2\sqrt{n}}}\left(\begin{array}{c}{b}_1\\ \cdots \\ b_p\end{array}\right)+{\displaystyle \frac{1}{6n}}\left(\begin{array}{c}{\tilde{d}}_1\\ \cdots \\ {\tilde{d}}_p\end{array}\right)\right\}\right]}^T{\zeta }_k\hfill \\ & & \left[{(\Sigma /n)}^{-1}\left\{Z_n+{\displaystyle \frac{1}{2\sqrt{n}}}\left(\begin{array}{c}{b}_1\\ \cdots \\ b_p\end{array}\right)+{\displaystyle \frac{1}{6n}}\left(\begin{array}{c}{\tilde{d}}_1\\ \cdots \\ {\tilde{d}}_p\end{array}\right)\right\}\right]\hfill \end{array}$$

where

$$\begin{array}{ccc}\hfill {\zeta }_k& =& \sum _{j,l}{\left[e_l{(\Sigma /n)}^{-1}\left\{Z_n+{\displaystyle \frac{1}{2\sqrt{n}}}\left(\begin{array}{c}{b}_1\\ \cdots \\ b_p\end{array}\right)+{\displaystyle \frac{1}{6n}}\left(\begin{array}{c}{\tilde{d}}_1\\ \cdots \\ {\tilde{d}}_p\end{array}\right)\right\}\right]}^T\hfill \\ & & {\left[e_j{(\Sigma /n)}^{-1}\left\{Z_n+{\displaystyle \frac{1}{2\sqrt{n}}}\left(\begin{array}{c}{b}_1\\ \cdots \\ b_p\end{array}\right)+{\displaystyle \frac{1}{6n}}\left(\begin{array}{c}{\tilde{d}}_1\\ \cdots \\ {\tilde{d}}_p\end{array}\right)\right\}\right]}^T{\displaystyle \frac{l_{kjl}^{\left(5\right)}}{n}}\hfill \end{array}$$

So, we have

$$\begin{array}{ccc}\hfill {\tilde{E}}_k& =& {\tilde{E}}_k^{\left(2\right)}+4Z_n^T{\displaystyle \frac{{\stackrel{⃛}{l}}_k}{n}}\tilde{d}+3b^T{\displaystyle \frac{{\stackrel{⃛}{l}}_k}{n}}b+4Z_n^T{m}_{k}b\hfill \end{array}$$

and

$$\begin{array}{ccc}\hfill C& =& {\Sigma }^{-1}{\displaystyle \frac{1}{2}}\left(\begin{array}{c}{b}_1\\ \cdots \\ b_p\end{array}\right),\hfill \\ \hfill D& =& {\Sigma }^{-1}{\displaystyle \frac{1}{6}}\left(\begin{array}{c}{\tilde{d}}_1\\ \cdots \\ {\tilde{d}}_p\end{array}\right),\hfill \\ \hfill E& =& {\Sigma }^{-1}{\displaystyle \frac{1}{24}}\left(\begin{array}{c}{\tilde{E}}_1\\ \cdots \\ {\tilde{E}}_p\end{array}\right).\hfill \end{array}$$

Using subindex S and B, we can obtain $C_S$, $D_S$, $C_B$, and $D_B$, and we can estimate the mean and variance and the covariance between these terms using an empirical version of these terms.

Appendix A.2. Proof of Theorem 1

Proof of Theorem 1.

Under second-order approximation, we have

$$\begin{array}{ccc}& & E\left\{({\widehat{\beta }}_W-\beta ){({\widehat{\beta }}_W-\beta )}^T\right\}\hfill \\ & =& (I-W)\gamma {\gamma }^T{(I-W)}^T+n_S^{-1}W{\Sigma }_S^T{W}^T+n_B^{-1}(I-W){\Sigma }_B^T{(I-W)}^T+O\left(n_S^{-1}\right)\hfill \\ & =& (I-W)(\gamma {\gamma }^T+n_B^{-1}{\Sigma }_B){(I-W)}^T+n_S^{-1}W{\Sigma }_S{W}^T+O\left(n_S^{-1}\right).\hfill \end{array}$$

$W_2(\beta ,\gamma ,X_S,X_B)$ minimize the quadratic form when ignoring the $O\left(n_S^{-1}\right)$ term. Denote the real optimal weight as $W^{*}$, and the optimal value is $L^{*}$; then, we have

$$\begin{array}{ccc}\hfill L^{*}& \ge & E\left\{({\widehat{\beta }}_{W_2(\beta ,\gamma ,X_S,X_B)}-\beta ){({\widehat{\beta }}_{W_2(\beta ,\gamma ,X_S,X_B)}-\beta )}^T\right\}\hfill \\ & =& \left\{I-W_2(\beta ,\gamma ,X_S,X_B)\right\}(\gamma {\gamma }^T+n_B^{-1}{\Sigma }_B){\left\{I-W_2(\beta ,\gamma ,X_S,X_B)\right\}}^T\hfill \\ & & +n_S^{-1}{W}_2(\beta ,\gamma ,X_S,X_B){\Sigma }_S^T{W}_2{(\beta ,\gamma ,X_S,X_B)}^T+O\left(n_S^{-1}\right)\hfill \\ & \ge & (I-W^{*})(\gamma {\gamma }^T+n_B^{-1}{\Sigma }_B){(I-W^{*})}^T\hfill \\ & & +n_S^{-1}{W}^{*}{\Sigma }_S^T{W}^{*T}+O\left(n_S^{-1}\right)\hfill \\ & =& L^{*}-O\left(n_S^{-1}\right)+O\left(n_S^{-1}\right)\hfill \\ & =& L^{*}+O\left(n_S^{-1}\right)\hfill \end{array}$$

For the main part, we have that $W_2(\beta ,\gamma ,X_S,X_B)$ minimize both the estimation MSE and prediction MSE with the same approximation error rate. So, we finish the first part of the proof.

Note that both $\widehat{\beta }$ and $\widehat{\gamma }$ are $\sqrt{n}$ consistent, so we have $\left|\right|W_2(\widehat{\beta },\widehat{\gamma },X_S,X_B)-W_2(\beta ,\gamma ,X_S,X_B){\left|\right|}_F=O_p\left(n_S^{-1}\right)$, so $E\left|\right|{\widehat{\beta }}_{W_2(\widehat{\beta },\widehat{\gamma },X_S,X_B)}-{\beta \left|\right|}_2^2-E\left|\right|{\widehat{\beta }}_{W_2(\beta ,\gamma ,X_S,X_B)}-\beta {\left|\right|}_2^2=O\left(n_S^{-2}\right)$. So, this leads to the conclusion

$$E\left|\right|{\widehat{\beta }}_{{\widehat{W}}_2(\widehat{\beta },\widehat{\gamma },X_S,X_B)}-{\beta \left|\right|}_2^2\le \underset{W}{inf}E\left|\right|{\widehat{\beta }}_W-\beta {\left|\right|}_2^2+O\left(n_S^{-1}\right)$$

and assume $X_S$ is bounded; then, we have

$$n_S^{-1}E\left|\right|X_S({\widehat{\beta }}_{{\widehat{W}}_2(\widehat{\beta },\widehat{\gamma },X_S,X_B)}-\beta ){\left|\right|}_2^2\le \underset{W}{inf}{n}_S^{-1}E\left|\right|X_S({\widehat{\beta }}_W-\beta ){\left|\right|}_2^2+O\left(n_S^{-1}\right)$$

 □

Appendix A.3. Proof of Theorem 2

Proof of Theorem 2.

We have $E{B}_S=0$ and $Var\left(B_S\right)=n_S{\Sigma }_S^{-1}$, $E{B}_B=0$ and $Var\left(B_B\right)=n_B{\Sigma }_B^{-1}$. Denote $c_S=E{C}_S$, $v_S=Var\left(C_S\right)$, $d_S=E{D}_S$, $e_S=E{E}_S$ and ${\rho }_S=Cov(B_S,C_S)=<B_S,C_S>$, ${\nu }_S=Cov(B_S,D_S)=<B_S,D_S>$, $c_B=E{C}_B$, $v_B=Var\left(C_B\right)$, $d_B=E{D}_B$, $e_B=E{E}_B$ and ${\rho }_B=Cov(B_B,C_B)=<B_B,C_B>$, ${\nu }_B=Cov(B_B,D_B)=<B_B,D_B>$. So, the mean square error for a weight matrix W will be

$$\begin{array}{ccc}& & E\left\{({\widehat{\beta }}_W-\beta ){({\widehat{\beta }}_W-\beta )}^T\right\}\hfill \\ & =& E\{(I-W)\gamma {\gamma }^T{(I-W)}^T+n_S^{-1}W{B}_S{B}_S^T{W}^T+n_B^{-1}(I-W)B_B{B}_B^T{(I-W)}^T\hfill \\ & & +n_S^{-2}W{C}_S{C}_S^T{W}^T+n_B^{-2}(I-W)C_B{C}_B^T{(I-W)}^T\hfill \\ & & +n_S^{-1/2}(I-W)\gamma B_S^T{W}^T+n_S^{-1/2}W{B}_S{\gamma }^T{(I-W)}^T+n_B^{-1/2}(I-W)\gamma B_B^T{(I-W)}^T\hfill \\ & & +n_B^{-1/2}(I-W)B_B{\gamma }^T{(I-W)}^T+n_S^{-1}(I-W)\gamma C_S^T{W}^T+n_S^{-1}W{C}_S{\gamma }^T{(I-W)}^T\hfill \\ & & +n_B^{-1}(I-W)\gamma C_B^T{(I-W)}^T+n_B^{-1}(I-W)C_B{\gamma }^T{(I-W)}^T+n_S^{-3/2}(I-W)\gamma D_S^T{W}^T\hfill \\ & & +n_S^{-3/2}W{D}_S{\gamma }^T{(I-W)}^T+n_B^{-3/2}(I-W)\gamma D_B^T{(I-W)}^T+n_B^{-3/2}(I-W)D_B{\gamma }^T{(I-W)}^T\hfill \\ & & +n_S^{-2}(I-W)\gamma E_S^T{W}^T+n_S^{-2}W{E}_S{\gamma }^T{(I-W)}^T+n_B^{-2}(I-W)\gamma E_B^T{(I-W)}^T\hfill \\ & & +n_B^{-2}(I-W)E_B{\gamma }^T{(I-W)}^T+n_S^{-1/2}{n}_B^{-1/2}W{B}_S{B}_B^T{(I-W)}^T+n_S^{-1/2}{n}_B^{-1/2}(I-W)B_B{B}_S^T{W}^T\hfill \\ & & +n_S^{-3/2}W{B}_S{C}_S^T{W}^T+n_S^{-3/2}W{C}_S{B}_S^T{W}^T+n_S^{-1/2}{n}_B^{-1}W{B}_S{C}_B^T{(I-W)}^T\hfill \\ & & +n_S^{-1/2}{n}_B^{-1}(I-W)C_B{B}_S^T{W}^T+n_S^{-2}W{B}_S{D}_S^T{W}^T+n_S^{-2}W{D}_S{B}_S^T{W}^T\hfill \\ & & +n_S^{-1/2}{n}_B^{-3/2}W{B}_S{D}_B^T{(I-W)}^T+n_S^{-1/2}{n}_B^{-3/2}(I-W)D_B{B}_S^T{W}^T+n_B^{-1/2}{n}_S^{-1}(I-W)B_B{C}_S^T{W}^T\hfill \\ & & +n_B^{-1/2}{n}_S^{-1}W{C}_S{B}_B^T{(I-W)}^T+n_B^{-3/2}(I-W)B_B{C}_B^T{(I-W)}^T+n_B^{-3/2}(I-W)C_B{B}_B^T{(I-W)}^T\hfill \\ & & +n_B^{-1/2}{n}_S^{-3/2}W{D}_S{B}_B^T{(I-W)}^T+n_B^{-1/2}{n}_S^{-3/2}(I-W)B_B{D}_S^T{W}^T+n_B^{-2}(I-W)D_B{B}_B^T{(I-W)}^T\hfill \\ & & +n_B^{-2}(I-W)B_B{D}_B^T{(I-W)}^T+n_S^{-1}{n}_B^{-1}W{C}_S{C}_B^T{(I-W)}^T+n_S^{-1}{n}_B^{-1}(I-W)C_B{C}_S^T{W}^T\}+o_p\left(n_S^{-2}\right)\hfill \\ & =& (I-W)\gamma {\gamma }^T{(I-W)}^T+n_S^{-1}W{n}_S{\Sigma }_S^{-1}{W}^T+n_B^{-1}(I-W)n_B{\Sigma }_B^{-1}{(I-W)}^T\hfill \\ & & +n_S^{-2}W(c_S{c}_S^T+v_S)W^T+n_B^{-2}(I-W)(c_B{c}_B^T+v_B){(I-W)}^T\hfill \\ & & +n_S^{-1}(I-W)\gamma c_S^T{W}^T+n_S^{-1}W{c}_S{\gamma }^T{(I-W)}^T\hfill \\ & & +n_B^{-1}(I-W)\gamma c_B^T{(I-W)}^T+n_B^{-1}(I-W)c_B{\gamma }^T{(I-W)}^T+n_S^{-3/2}(I-W)\gamma d_S^T{W}^T\hfill \\ & & +n_S^{-3/2}W{d}_S{\gamma }^T{(I-W)}^T+n_B^{-3/2}(I-W)\gamma d_B^T{(I-W)}^T+n_B^{-3/2}(I-W)d_B{\gamma }^T{(I-W)}^T\hfill \\ & & +n_S^{-2}(I-W)\gamma e_S^T{W}^T+n_S^{-2}W{e}_S{\gamma }^T{(I-W)}^T+n_B^{-2}(I-W)\gamma e_B^T{(I-W)}^T\hfill \\ & & +n_B^{-2}(I-W)e_B{\gamma }^T{(I-W)}^T+n_S^{-3/2}W{\rho }_S{W}^T+n_S^{-3/2}W{\rho }_S^T{W}^T+n_S^{-2}W{\nu }_S{W}^T+n_S^{-2}W{\nu }_S^T{W}^T\hfill \\ & & +n_B^{-3/2}(I-W){\rho }_B{(I-W)}^T+n_B^{-3/2}(I-W){\rho }_B^T{(I-W)}^T\hfill \\ & & +n_B^{-2}(I-W){\nu }_B^T{(I-W)}^T+n_B^{-2}(I-W){\nu }_B{(I-W)}^T+n_S^{-1}{n}_B^{-1}W{c}_S{c}_B^T{(I-W)}^T+n_S^{-1}{n}_B^{-1}(I-W)c_B{c}_S^T{W}^T+o_p\left(n_S^{-2}\right)\hfill \end{array}$$

This can be simplified as

$$\begin{array}{ccc}& & E\left\{({\widehat{\beta }}_W-\beta ){({\widehat{\beta }}_W-\beta )}^T\right\}\hfill \\ & =& W{A}_{00}{W}^T+(I-W)A_{11}{(I-W)}^T+W{A}_{01}{(I-W)}^T+(I-W)A_{10}{W}^T+o_p\left(n_S^{-2}\right)\hfill \\ & =& W(A_{00}+A_{11}-A_{01}-A_{10})W^T+W(A_{01}-A_{11})+(A_{10}-A_{11})W^T+A_{11}+o\left(n_S^{-2}\right)\hfill \\ & =& W{B}_{11}{W}^T+W{B}_{10}+B_{01}{W}^T+B_{00}+o\left(n_S^{-2}\right)\hfill \\ & =& f\left(W\right)+o\left(n_S^{-2}\right)\hfill \end{array}$$

where

$$\begin{array}{ccc}\hfill A_{00}& =& {\Sigma }_S^{-1}+n_S^{-2}(c_S{c}_S^T+v_S)+n_S^{-3/2}{\rho }_S+n_S^{-3/2}{\rho }_S^T+n_S^{-2}{\nu }_S+n_S^{-2}{\nu }_S^T\hfill \\ \hfill A_{01}& =& n_S^{-1}{c}_S{\gamma }^T+n_S^{-3/2}{d}_S{\gamma }^T+n_S^{-2}{e}_S{\gamma }^T+n_S^{-1}{n}_B^{-1}{c}_S{c}_B^T\hfill \\ \hfill A_{10}& =& n_S^{-1}\gamma c_S^T+n_S^{-3/2}\gamma d_S^T+n_S^{-2}\gamma e_S^T+n_S^{-1}{n}_B^{-1}{c}_B{c}_S^T\hfill \\ \hfill A_{11}& =& \gamma {\gamma }^T+{\Sigma }_B^{-1}+n_B^{-2}(c_B{c}_B^T+v_B)+n_B^{-1}\gamma c_B^T+n_B^{-1}{c}_B{\gamma }^T+n_B^{-3/2}\gamma d_B^T+n_B^{-3/2}{d}_B{\gamma }^T\hfill \\ & & +n_B^{-2}\gamma e_B^T+n_B^{-2}{e}_B{\gamma }^T+n_B^{-3/2}{\rho }_B+n_B^{-3/2}{\rho }_B^T+n_B^{-2}{\nu }_B+n_B^{-2}{\nu }_B^T\hfill \end{array}$$

So we have $W_h(\beta ,\gamma ,X_S,X_B)$ minimize the $f\left(W\right)$. Denote the real optimal weight as $W^{*}$ and the optimal value is $L^{*}$, then we have

$$\begin{array}{ccc}\hfill L^{*}& \ge & E\left[\left\{{\widehat{\beta }}_{W_2(\beta ,\gamma ,X_S,X_B)}-\beta \right\}{\left\{{\widehat{\beta }}_{W_2(\beta ,\gamma ,X_S,X_B)}-\beta \right\}}^T\right]\hfill \\ & =& f\left(W\right)+o\left(n_S^{-2}\right)\hfill \\ & \ge & f\left(W^{*}\right)+o\left(n_S^{-2}\right)\hfill \\ & =& L^{*}-o\left(n_S^{-2}\right)+o\left(n_S^{-2}\right)\hfill \\ & =& L^{*}+o\left(n_S^{-2}\right)\hfill \end{array}$$

So, we finish the first part of the proof.

Following the proof of Theorem 1, we have $\left|\right|W_h(\widehat{\beta },\widehat{\gamma },X_S,X_B)-W_h(\beta ,\gamma ,X_S,X_B){\left|\right|}_F=O_p\left(n_S^{-1}\right)$ and $E\left|\right|{\widehat{\beta }}_{W_h(\widehat{\beta },\widehat{\gamma },X_S,X_B)}-{\beta \left|\right|}_2^2-E\left|\right|{\widehat{\beta }}_{W_h(\beta ,\gamma ,X_S,X_B)}-\beta {\left|\right|}_2^2=O\left(n_S^{-2}\right)$, which will lead to the result

$$E\left|\right|{\widehat{\beta }}_{{\widehat{W}}_h(\widehat{\beta },\widehat{\gamma },X_S,X_B)}-{\beta \left|\right|}_2^2\le \underset{W}{inf}E\left|\right|{\widehat{\beta }}_W-\beta {\left|\right|}_2^2+O\left(n_S^{-2}\right)$$

 □

Appendix A.4. Proof of Theorem 3

Proof of Theorem 3.

When $\gamma =0$, or if we allow $\gamma$ to be a function of sample size $n_S$ such that $\gamma =O\left(n_S^{-1/2}\right)$, then we have $I-W=O\left(1\right)$, and the improvement in MSE by using oracle high-order weight will be on the order of $O\left(n_S^{-1}\right)$, so combined with Theorem 2, we conclude that the estimated high-order weight will lead to smaller MSE compared with using small data only.

When $\gamma \ne 0$, we have $I-W=O\left(n_S^{-1}\right)$, so the improvement is on the order of $O\left(n_S^{-2}\right)$ from Taylor expansion $f\left(I\right)=f\left(W\right)+\dot{f}\left(W\right)(I-W)+(I-W){\displaystyle \frac{\ddot{f}\left(W\right)}{2}}(I-W)$. Thus, we cannot directly apply Theorem 2 to obtain the conclusion that the estimated high-order weight is always superior to using small data only.

To obtain the conclusion we want, we need to show that ignoring the $o\left(n_S^{-2}\right)$ approximation error term, the estimated weight leads to a smaller main part compared with using small data only, i.e.,

$$f\left(W_h(\widehat{\beta },\widehat{\gamma },X_S,X_B)\right)<f\left(I\right).$$

It is obvious that the main estimation error contributes to the estimated weight that comes from the $\widehat{\gamma }$ term, while the estimation error of other parts, c, d, e, $\rho$, $\nu$ are asymptotically ignorable. The improvement from the high-order approximation is asymptotically larger than the second-order approximation, and the improvement of the second-order approximation part is shown to always be positive and is on the order of $O\left(n_S^{-2}\right)$ with large $n_S$ (see Section 9.7 of [2]). This proves that $f\left(W_h(\widehat{\beta },\widehat{\gamma },X_S,X_B)\right)<f\left(I\right)$, and thus, we have

$$\begin{array}{ccc}& & E[{({\widehat{\beta }}_{W^{*}}-\beta )}^T({\widehat{\beta }}_{W^{*}}-\beta )-{({\widehat{\beta }}_S-\beta )}^T({\widehat{\beta }}_S-\beta )]\hfill \\ & \le & O\left(n_S^{-2}\right)-o\left(n_S^{-2}\right)<0\hfill \end{array}$$

when $n_S$ is large. □