Change-Point Estimation and Detection for Mixture of Linear Regression Models

Abstract

This paper studies the estimation and detection problems in the mixture of linear regression models with change point. An improved Expectation–Maximization (EM) algorithm is devised specifically for multi-classified mixture data with change points. Under appropriate conditions, the large-sample properties of the estimator are rigorously proven. This improved EM algorithm not only precisely locates the change points but also yields accurate parameter estimates for each class. Additionally, a detector grounded in the score function is proposed to identify the presence of change points in mixture data. The limiting distributions of the test statistics under both the null and alternative hypotheses are systematically derived. Extensive simulation experiments are conducted to assess the effectiveness of the proposed method, and comparative analyses with the conventional EM algorithm are performed. The results clearly demonstrate that the EM algorithm without considering change points exhibits poor performance in classifying data, often resulting in the misclassification or even omission of certain classes. In contrast, the estimation method introduced in this study showcases remarkable accuracy and robustness, with favorable empirical sizes and powers.

1. Introduction

Mixture models have found extensive applications in econometrics and social sciences, and the associated theoretical frameworks have been thoroughly investigated. Among them, finite mixtures of linear regression models, a particularly valuable class of mixture models, have been widely employed across diverse fields, including econometrics, marketing, epidemiology, and biology. Simultaneously, the statistical inference and parameter estimation based on mixture models play a pivotal role in practical applications. On another front, the change point problem remains a focal point in statistical research. Conceptually, a change point denotes the specific location or instant at which a sudden alteration in a statistical property occurs. Change point phenomena are pervasive in both natural and social systems, and have enabled crucial advancements in numerous domains such as industrial control, financial economics, biomedicine, and signal processing. The detection and estimation of change points within mixture datasets are essential for discerning underlying patterns and making informed decisions, thereby holding significant practical implications.

In recent decades, remarkable progress has been achieved in the analysis of mixture models. Goldfeld and Quandt provided an outstanding introduction to mixtures of linear regression models [1]. They delved into the parameter estimation problem of such mixtures and applied it to analyze the imbalance in the housing price market. Frühwirth and Schnatter summarized the Bayesian approach for mixtures of linear regression models in their paper [2]. Hurn and Justel proposed a generalized linear finite mixture model and overcame the label switching issue by normalizing the loss function [3]. Li, Chen, and their collaborators (Chen, Li, and Fu) concentrated on the hypothesis testing problem regarding the order of the mixture model and derived the limiting distribution of the test statistics [4,5]. Huang and Yao introduced a semiparametric mixture regression model, where the mixing proportion is a smooth function of covariates [6]. Huang, Li, and Wang developed a non-parametric finite mixture method for regression models and demonstrated the estimation procedure through an analysis of U.S. Housing Price Index (HPI) data [7].

The change point problem has attracted extensive research attention, with a vast body of literature accumulated over the years. It was first introduced by Page during his study of continuous inspection schemes, who proposed the Cumulative Sum (CUSUM) test method [8]. Subsequently, scholars like Sen and Srivastava, Hawkins, James et al., and Srivastava and Worsley investigated the detection of change points in the mean of normal random variable sequences [9,10,11,12,13]. Basseville and Nikiforov systematically expounded the theoretical foundation of change point detection and estimation algorithms [14]. Wang, Zou, and Yin delved into change point detection in multinomial data as the number of categories approaches infinity, rigorously proving the statistical properties of relevant test statistics [15]. Xia and Qiu proposed the JIC criterion to tackle the multi-change point problem in non-parametric regression models [16]. Bai employed the least squares method to estimate the mean change point of linear processes, deriving the consistency and convergence rate of the change point estimator [17]. Baranowski et al. utilized grid search algorithms based on binary segmentation to accurately determine the number and positions of change points [18]. Follain et al. developed a novel approach for estimating change points in partially observed, high-dimensional time series with simultaneous mean shifts in sparse coordinate subsets [19]. Gong et al. redefined the graph-based change point detection problem as a prediction task and proposed an innovative change point detection (CPD) method for dynamic graphs through a latent evolution model. Drabech et al. introduced a Markov Random Field (MRF) model to detect slope changes [20], while Ratnasingam et al. proposed an empirical likelihood-based non-parametric procedure for identifying structural changes in quantile regression models [21]. Notably, despite these abundant studies, the statistical analysis of change points within finite mixtures of linear models remains a relatively overlooked area [22].

This paper delves into the estimation and detection problems of the finite mixture of linear regression models with change points. Specifically, for multi-classified mixture data incorporating change points, we ingeniously propose an improved Expectation–Maximization (EM) algorithm. This algorithm is meticulously designed to accurately compute parameter estimators, such as the location of change points and regression coefficients. Through rigorous mathematical proofs, we firmly establish the consistency and asymptotic normality of these parameter estimators, providing a solid theoretical foundation. Moreover, to effectively detect the presence of change points in the mixture of linear regression models, we innovatively develop a detector based on the score function. By conducting in-depth analyses, we systematically derive the limiting distributions under both the null hypothesis and the alternative hypothesis, which significantly contributes to the theoretical framework of this study. To thoroughly verify the practical effectiveness of the proposed method, we carefully design and execute a series of simulation experiments.

The remaining sections of this paper are structured as follows. In Section 2, we introduce the estimation and detection methods, statistical models, along with their associated assumptions. In Section 3 is dedicated to the development of the estimation and detection techniques for change points within the mixture of linear regression models. Additionally, we obtain the large sample properties of the estimators and test statistics. In Section 4, the simulation results are presented. Section 5 provides some concluding remarks and discussions. In Section 6, the limitations and future research directions are presented. The proofs of the theorems are included in the appendix.

2. Statistical Model and Assumptions

Assume that $\{(X_i,Y_i),i=1,\cdots ,n\}$  are independent random samples from the following data generating process:

$$Y_i\sim \left\{\begin{array}{ccc}\hfill {\displaystyle \sum _{c=1}^C}{\pi }_c^{\ast -}N\left(X_i^T{\mathit{\beta }}_c^{\ast -},{\sigma }_c^{2\ast -}\right),& & 1\le i\le k^{\ast },\hfill \\ \hfill {\displaystyle \sum _{c=1}^C}{\pi }_c^{\ast +}N\left(X_i^T{\mathit{\beta }}_c^{\ast +},{\sigma }_c^{2\ast +}\right),& & k^{\ast }+1\le i\le n,k^{\ast }\in [1,n-1],\hfill \end{array}\right.$$

(1)

where $k^{\ast }$ is a change point, $X_i$ is a p-dimensional random covariate, C is the number of components, ${\mathit{\beta }}_c^{\ast -},{\pi }_c^{\ast -},{\sigma }_c^{\ast -},$  ${\mathit{\beta }}_c^{\ast +},{\pi }_c^{\ast +},{\sigma }_c^{\ast +}$ are mixture coefficients for the cth component, $1\ge {\pi }_c^{\ast -}\ge 0,$${\pi }_c^{\ast +}\ge 0,{\displaystyle \sum _{c=1}^C}{\pi }_c^{\ast -}={\displaystyle \sum _{c=1}^C}{\pi }_c^{\ast +}=1$. ${\pi }_c^{\ast -}$ and ${\pi }_c^{\ast +}$ are called mixing proportions or weights. ${\mathit{\beta }}_c^{\ast -},{\mathit{\beta }}_c^{\ast +}\in R^p$, ${\sigma }_c^{2\ast -},{\sigma }_c^{2\ast +}\in R^{+}$ and the parameters are unknown. Equivalently speaking, $\{Y_i,i=1,\cdots ,n\}$ follow a finite mixture of normals,

$$Y_i\sim \sum _{c=1}^C{\pi }_c^{\ast -}N(X_i^T{\mathit{\beta }}_c^{\ast -},{\sigma }_c^{2\ast -})·I_{\{i\le k^{\ast }\}}+\sum _{c=1}^C{\pi }_c^{\ast +}N(X_i^T{\mathit{\beta }}_c^{\ast +},{\sigma }_c^{2\ast +})·I_{\{i\ge k^{\ast }+1\}}.$$

Naturally, the log-likelihood function corresponding to the dataset $\{(X_i,Y_i),i=1,\cdots ,n\}$ can be expressed as

$$\begin{array}{cc}\hfill & l(k,{\pi }_c^{-},{\mathit{\beta }}_c^{-},{\sigma }_c^{2-},{\pi }_c^{+},{\mathit{\beta }}_c^{+},{\sigma }_c^{2+})=\hfill \\ \hfill & \sum _{i=1}^n\log \left[\sum _{c=1}^C\left\{{\pi }_c^{-}\varphi \left(Y_i|X_i^T{\mathit{\beta }}_c^{-},{\sigma }_c^{2-}\right)·I_{\{i\le k\}}+{\pi }_c^{+}\varphi \left(Y_i|X_i^T{\mathit{\beta }}_c^{+},{\sigma }_c^{2+}\right)·I_{\{i\ge k+1\}}\right\}\right]\hfill \end{array}$$

(2)

where $\varphi \left(y\right|\mu ,{\sigma }^2)$ denotes the normal density.

For later use, we introduce some notations. Let ${\mathit{\theta }}^{\ast }={\left({\left({\mathit{\pi }}^{\ast }\right)}^T,{\left({\mathit{\beta }}^{\ast }\right)}^T,{\left({\mathit{\sigma }}^{2\ast }\right)}^T\right)}^T$ denote the true parameter, where ${\mathit{\pi }}^{\ast }={({\pi }_1^{\ast },\cdots ,{\pi }_{C-1}^{\ast })}^T,$ ${\mathit{\beta }}^{\ast }={({\mathit{\beta }}_1^{\ast T},\cdots ,{\mathit{\beta }}_C^{\ast T})}^T,{\mathit{\sigma }}^{2^{\ast }}={({\sigma }_1^{2^{\ast }},\cdots ,{\sigma }_C^{2^{\ast }})}^T$ and ${\mathit{\beta }}_i^{\ast }\in R^P,i=1,\cdots ,C.$ Let ${\mathit{\theta }}^{\ast -},{\mathit{\theta }}^{\ast +}$ represent the parameters on the left and right sides of the change point, respectively. Similarly, we can define $\mathit{\theta }$, ${\mathit{\theta }}^{-}$, ${\mathit{\theta }}^{+}.$

Let ${\widehat{k}}_n$ and ${\widehat{\mathit{\theta }}}_n$ be the maximum likelihood estimator of $k^{\ast }$ and $\mathit{\theta }$, which can be obtained by maximizing (2). In Section 3.1, an improved EM algorithm is proposed to carry out the estimation. The following assumptions are imposed.

Assumption 1.

${\displaystyle \frac{k^{\ast }}{n}}\to {\tau }^{\ast }$, $n\to \infty$ and ${\tau }^{\ast }\in (0,1)$.

The above assumption requires that $k^{\ast }$ depends on the sample size n and is approximately proportional to n.

Assumption 2.

We suppose that $\mathit{\theta }\in \mathrm{\Theta }$ and parameter space $\mathrm{\Theta }\subset R^q$ is a compact set, $q=(p+2)·C-1.$

Assumption 2 restricts the parameter space to be compact, which means any open cover of $\mathrm{\Theta }$ has a finite subcover.

Assumption 3.

$\{(X_i,Y_i),i=1,2,\cdots ,n\}$ are independent.

3. Parameter Inference

3.1. Estimation Procedure

In this subsection, we propose an effective EM algorithm to deal with the parameter estimation problem. We take $i\ge k+1$ as an example to illustrate the derivation of the iterative formula ( $i<k$ can be obtained in the same way). At this time, the log-likelihood function corresponding to the data is

$$\sum _{i=k+1}^n\log \left[\sum _{c=1}^C{\pi }_c^{+}\varphi \left\{Y_i\right|X_i^T{\mathit{\beta }}_c^{+},{\sigma }_c^{2+}\}\right].$$

(3)

In the EM framework, the mixture problem is described as an incomplete-data problem. We view the observed data $(X_i,Y_i,i=1,2,\cdots ,n)$ as being incomplete, and then introduce unobserved $Bernoulli$ random variables

$$z_{ic}=\left\{\begin{array}{c}1,if(X_i,Y_i)isinthecthgroup,\hfill \\ 0,otherwise,\hfill \end{array}\right.$$

$$i=1,2,\cdots ,n,c=1,2,\cdots ,C.$$

and ${\mathbf{z}}_i={(z_{i1},\cdots ,z_{iC})}^T$, which is the component label of $(X_i,Y_i)$. Therefore, the complete data are ${\left\{(X_i,Y_i,{\mathbf{z}}_i)\right\}}_{i=1}^n$, and the complete log-likelihood function corresponding to (3) is

$$\sum _{i=k+1}^n\sum _{c=1}^C{z}_{ic}\left[\log {\pi }_c^{+}+\log \varphi \left\{Y_i\right|X_i^T{\mathit{\beta }}_c^{+},{\sigma }_c^{2+}\}\right].$$

Supposing in the lth cycle of the EM algorithm iteration, we have ${\pi }_c^{+\left(l\right)},{\mathit{\beta }}_c^{+\left(l\right)},{\sigma }_c^{2+\left(l\right)}$; then, in the E-step of $(l+1)$th cycle, the expectation of the latent variable $z_{ic}$ can be calculated by

$$r_{ic}^{(l+1)}={\displaystyle \frac{{\pi }_c^{+\left(l\right)}\varphi \left\{Y_i\right|X_i^T{\mathit{\beta }}_c^{+\left(l\right)},{\sigma }_c^{2+\left(l\right)}\}}{{\sum }_{c=1}^C{\pi }_c^{+\left(l\right)}\varphi \left\{Y_i\right|X_i^T{\mathit{\beta }}_c^{+\left(l\right)},{\sigma }_c^{2+\left(l\right)}\}}}.$$

In the M-step of the $(l+1)$th cycle, we maximize

$$\sum _{i=k+1}^n\sum _{c=1}^C{r}_{ic}^{(l+1)}\left[\log {\pi }_c^{+}+\log \varphi \left\{Y_i\right|X_i^T{\mathit{\beta }}_c^{+},{\sigma }_c^{2+}\}\right].$$

(4)

The maximization of Equation (4) is equivalent to separately maximize  ${\displaystyle \sum _{i=k+1}^n}{\displaystyle \sum _{c=1}^C}{r}_{ic}^{(l+1)}\log {\pi }_c^{+}$ and ${\displaystyle \sum _{i=k+1}^n}{\displaystyle \sum _{c=1}^C}{r}_{ic}^{(l+1)}\log \varphi \left\{Y_i\right|X_i^T{\mathit{\beta }}_c^{+},{\sigma }_c^{2+}\}$. Furthermore, the iterative formula can be easily derived as follows:

$${\pi }_c^{+(l+1)}={\displaystyle \frac{{\sum }_{i=k+1}^n{r}_{ic}^{(l+1)}}{n}},$$

$${\mathit{\beta }}_c^{+(l+1)}={\left(\sum _{i=k+1}^n{X}_i{r}_{ic}^{(l+1)}{X}_i^T\right)}^{-1}\left(\sum _{i=k+1}^n{X}_i{r}_{ic}^{(l+1)}{Y}_i\right),$$

$${\sigma }_c^{2+(l+1)}={\displaystyle \frac{{\sum }_{i=k+1}^n{r}_{ic}^{(l+1)}(Y_i-X_i^T{\mathit{\beta }}_c^{+\left(l\right)}){(Y_i-X_i^T{\mathit{\beta }}_c^{+\left(l\right)})}^T}{{\sum }_{i=k+1}^n{r}_{ic}^{(l+1)}}},c=1,2,\cdots ,C$$

Let the initial value be ${\pi }_c^{+\left(0\right)},{\mathit{\beta }}_c^{+\left(0\right)},{\sigma }_c^{+\left(0\right)},c=1,\cdots ,C$, repeat the above iteration algorithm, and when the iteration converges, we obtain the estimator of mixture parameters, denoted as ${\pi }_c^{+}\left(k\right),{\mathit{\beta }}_c^{+}\left(k\right),{\sigma }_c^{2+}\left(k\right),c=1,\cdots ,C$. At this time, the log-likelihood function corresponding to the data ${\left\{(X_i,Y_i)\right\}}_{i=k+1}^n$ is

$$l^{+}(k,{\mathit{\pi }}_c^{+},{\mathit{\beta }}_c^{+},{\mathit{\sigma }}_c^{2+})=\sum _{i=k+1}^n\log \left[\sum _{c=1}^C{\pi }_c^{+}\left(k\right)\varphi \left\{Y_i\right|X_i^T{\mathit{\beta }}_c^{+}\left(k\right),{\sigma }_c^{2+}\left(k\right)\}\right].$$

Whenever a value of k is determined, the mixture data are divided into two parts: ${\left\{(X_i,Y_i)\right\}}_{i=1}^k$ and ${\left\{(X_i,Y_i)\right\}}_{i=k+1}^n$. Let the initial value be ${\pi }_c^{\pm \left(0\right)},{\mathit{\beta }}_c^{\pm \left(0\right)},{\sigma }_c^{\pm \left(0\right)},c=1,\cdots ,C$, and we can obtain the estimations of change point and mixture parameter according to Algorithm 1:

$$(\widehat{k},\widehat{{\mathit{\theta }}^{-}},\widehat{{\mathit{\theta }}^{+}})=\underset{k,{\mathit{\pi }}_c^{\pm },{\mathit{\beta }}_c^{\pm },{\mathit{\sigma }}_c^{\pm }}{\arg \max }l(k,{\mathit{\pi }}_c^{-},{\mathit{\beta }}_c^{-},{\mathit{\sigma }}_c^{2-},{\mathit{\pi }}_c^{+},{\mathit{\beta }}_c^{+},{\mathit{\sigma }}_c^{2+}),$$

where ${\widehat{\mathit{\theta }}}^{\pm }={({\left({\widehat{\mathit{\pi }}}^{\pm }\right)}^T,{\left({\widehat{\mathit{\beta }}}^{\pm }\right)}^T,{\left({\widehat{\mathit{\sigma }}}^{2\pm }\right)}^T)}^T$, ${\widehat{\mathit{\pi }}}^{\pm }={({\widehat{\pi }}_1^{\pm },\cdots ,{\widehat{\pi }}_C^{\pm })}^T$ and ${\widehat{\mathit{\beta }}}^{\pm }={({\widehat{\beta }}_1^{\pm },\cdots ,{\widehat{\beta }}_C^{\pm })}^T$, ${\widehat{\mathit{\sigma }}}^{2\pm }={({\widehat{\sigma }}_1^{2\pm },\cdots ,{\widehat{\sigma }}_C^{2\pm })}^T$, $l(k,{\mathit{\pi }}_c^{-},{\mathit{\beta }}_c^{-},{\mathit{\sigma }}_c^{2-},{\mathit{\pi }}_c^{+},{\mathit{\beta }}_c^{+},{\mathit{\sigma }}_c^{2+})=l^{+}(k,{\mathit{\pi }}_c^{+},{\mathit{\beta }}_c^{+},{\mathit{\sigma }}_c^{2+})+l^{-}(k,{\mathit{\pi }}_c^{-},{\mathit{\beta }}_c^{-},{\mathit{\sigma }}_c^{2-}).$

Algorithm 1 EM algorithm considering change point

Intput: ${\{X_i,Y_i\}}_{i=1}^n$, the number of mixture component C, the maximum iterations T.

Output: $\widehat{k},{\widehat{\mathit{\theta }}}^{-},{\widehat{\mathit{\theta }}}^{+}$   1: for $k=1,2,\cdots ,n-1$ do   2:      For any given k, the mixture data are divided into two parts: ${\left\{(X_i,Y_i)\right\}}_{i=1}^k$ and ${\left\{(X_i,Y_i)\right\}}_{i=k+1}^n$. Using EM algorithm separately for two parts of data. For ${\left\{(X_i,Y_i)\right\}}_{i=1}^k$, let the initial value be ${\pi }_c^{-\left(0\right)},{\mathit{\beta }}_c^{-\left(0\right)},{\sigma }_c^{-\left(0\right)},c=1,\cdots ,C$.   3:      for $t=1,2,\cdots ,T$ do   4:           E-step: calculate $r_{ic}^{(l+1)}\left(k\right)$, $$r_{ic}^{(l+1)}\left(k\right)={\displaystyle \frac{{\pi }_c^{-\left(l\right)}\varphi \left\{Y_i\right|X_i^T{\mathit{\beta }}_c^{-\left(l\right)},{\sigma }_c^{2-\left(l\right)}\}}{{\sum }_{c=1}^C{\pi }_c^{-\left(l\right)}\varphi \left\{Y_i\right|X_i^T{\mathit{\beta }}_c^{-\left(l\right)},{\sigma }_c^{2-\left(l\right)}\}}},i=1,\cdots ,k,c=1,\cdots ,C$$   5:           M-step: calculate $${\pi }_c^{-(l+1)}\left(k\right)={\displaystyle \frac{{\sum }_{i=1}^n{r}_{ic}^{(l+1)}}{n}},{\mathit{\beta }}_c^{-(l+1)}\left(k\right)={\left(\sum _{i=1}^n{X}_i{r}_{ic}^{(l+1)}{X}_i^T\right)}^{-1}\left(\sum _{i=1}^n{X}_i{r}_{ic}^{(l+1)}{Y}_i\right)$$ $${\sigma }_c^{2-(l+1)}\left(k\right)={\displaystyle \frac{{\sum }_{i=1}^n{r}_{ic}^{(l+1)}(Y_i-X_i^T{\mathit{\beta }}_c^{-\left(l\right)}){(Y_i-X_i^T{\mathit{\beta }}_c^{-\left(l\right)})}^T}{{\sum }_{i=1}^n{r}_{ic}^{(l+1)}}}$$   6:      end for   7:      When the iteration reaches convergence, we obtain the estimator ${\pi }_c^{-}\left(k\right),{\mathit{\beta }}_c^{-}\left(k\right),{\sigma }_c^{2-}\left(k\right),c=1,\cdots ,C$ and the log-likelihood function value $l^{-}\left(k\right)$   8:      For ${\left\{(X_i,Y_i)\right\}}_{i=k+1}^n$, repeat steps 3–7 and obtain ${\pi }_c^{+}\left(k\right),{\mathit{\beta }}_c^{+}\left(k\right),{\sigma }_c^{2+}\left(k\right),c=1,\cdots ,C$ and $l^{+}\left(k\right)$    9: end for 10: $\widehat{k}=\underset{1\le k\le n-1}{\arg \max }(l^{-}\left(k\right)+l^{+}\left(k\right)),{\widehat{\mathit{\theta }}}^{-}=\widehat{{\mathit{\theta }}^{-}}\left(\widehat{k}\right),{\widehat{\mathit{\theta }}}^{+}=\widehat{{\mathit{\theta }}^{+}}\left(\widehat{k}\right),$

3.2. Hypothesis Test

Define the following notations for later use:

$$\eta \left(Y\right|\mathit{\theta })=\sum _{c=1}^C{\pi }_c\varphi \{X,Y|{\beta }_c,{\sigma }_c^2\},\ell (\mathit{\theta },Y)=\log \eta \left(Y\right|\mathit{\theta }),$$

$$q_1\{\mathit{\theta },Y\}={\displaystyle \frac{\partial \ell \{\mathit{\theta },Y\}}{\partial \mathit{\theta }}},q_2\{\mathit{\theta },Y\}={\displaystyle \frac{{\partial }^2\ell \{\mathit{\theta },Y\}}{\partial \mathit{\theta }\partial {\mathit{\theta }}^T}},$$

$$\mathcal{I}\left(\mathit{\theta }\right)=-E\left[q_2\{\mathit{\theta },Y\}\right],\mathcal{B}\left(\mathit{\theta }\right)=Var\left[q_1\{\mathit{\theta },Y\}\right]=E\left[q_1\{\mathit{\theta },Y\}{q}_1^T\{\mathit{\theta },Y\}\right].$$

In the following, we are interested in testing the null hypothesis:

$$H_0:{\mathit{\theta }}_i={\mathit{\theta }}^{\ast }(i=1,\cdots ,n)$$

against the local alternative hypothesis:

$$H_A:{\mathit{\theta }}_i={\mathit{\theta }}^{\ast }+{\displaystyle \frac{1}{\sqrt{n}}}·\mathit{g}\left({\displaystyle \frac{i}{n}}\right),$$

where $\mathit{g}={(g_1,g_2,\cdots ,g_q)}^T$ is a function of bounded variation on [0,1] which describes the patter of departure from stability of the parameter ${\mathit{\theta }}^{\ast }$.

Firstly, we give the following lemmas and then we build an empirical process and infer its limiting process under the $H_0$ and $H_1$ hypothesis, respectively, as described by Theorem 1.

Theorem 1.

Under $H_0$ and Lemma A3 in the Appendix A, given by

$${\Delta }_n(t,{\widehat{\mathit{\theta }}}_n)={\displaystyle \frac{1}{\sqrt{n}}}\sum _{i=1}^{\left[nt\right]}{q}_1({\widehat{\mathit{\theta }}}_n,Y_i),t\in (0,1),$$

(5)

then, we have

(1). under $H_0$,

$${\widehat{\mathcal{B}}}_n^{-1/2}{\Delta }_n(t,{\widehat{\mathit{\theta }}}_n)\stackrel{d}{\to }{\mathit{W}}^0\left(t\right),n\to \infty ,$$

where ${\mathit{W}}^0\left(t\right)$ is a $\mathit{\text{q-dimensional}}$ standard Brownian bridge with ${\mathit{W}}^0\left(t\right)=\mathit{W}\left(t\right)-t\mathit{W}\left(1\right)$ and ${\widehat{\mathcal{B}}}_n$ some consistent covariance matrix estimation, such as ${\widehat{\mathcal{B}}}_n={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{q}_1({\widehat{\mathit{\theta }}}_n,Y_i)q_1{({\widehat{\mathit{\theta }}}_n,Y_i)}^T.$

(2). Under $H_A$,

$${\widehat{\mathcal{B}}}_n^{-1/2}{\Delta }_n(t,{\widehat{\mathit{\theta }}}_n)\stackrel{d}{\to }{W}^0\left(t\right)+\mathcal{B}{\left({\mathit{\theta }}^{\ast }\right)}^{1/2}{\mathit{G}}^0\left(t\right),n\to \infty ,$$

where ${\mathit{G}}^0\left(t\right)=\mathit{G}\left(t\right)-t\mathit{G}\left(1\right)$ with $\mathit{G}\left(t\right)={\int }_0^t\mathit{g}\left(y\right)dy$.

Next, we construct test statistics according to the empirical process (5) stated above. The key idea is that the empirical process reflects some symptoms of structural changes, and its performance under $H_0$ and $H_A$ is significantly different; then, the test statistics can be constructed as follows:

$$T_n=\underset{1\le k\le n}{\sup }{∥{\widehat{\mathcal{B}}}_n^{-1/2}{\Delta }_n(k/n,{\widehat{\mathit{\theta }}}_n)∥}^2.$$

Further from Theorem 1 and continuous mapping theorem, the following corollary can be obtained:

Corollary 1.

Given the empirical process (5) in Theorem 1, we have

(1). under $H_0$,

$$T_n\stackrel{d}{\to }\underset{t\in [0,1]}{\sup }{∥{\mathit{W}}^0\left(t\right)∥}^2=\underset{t\in [0,1]}{\sup }\left\{\sum _{i=1}^q{W}_i^0{\left(t\right)}^2\right\},n\to \infty .$$

(6)

(2). Under $H_A$,

$$T_n\stackrel{d}{\to }\underset{t\in [0,1]}{\sup }{∥{\mathit{W}}^0\left(t\right)+\mathcal{B}{\left({\mathit{\theta }}^{\ast }\right)}^{1/2}{\mathit{G}}^0\left(t\right)∥}^2,n\to \infty$$

The above results show that the limiting distributions of the test statistics are different under the null and alternative hypothesis; thus, we can judge the existence of the change point according to it.

3.3. Consistency

In order to prove the consistency of the estimator, we first construct a function ${\mathcal{L}}_n$ as follows:

$${\mathcal{L}}_n(\tau ,\mathit{\theta })={\displaystyle \frac{1}{n}}\sum _{i=1}^n\tilde{f}(X_i,Y_i|t,\mathit{\theta })={\displaystyle \frac{1}{n}}\sum _{i=1}^{\left[n\tau \right]}\tilde{f}(X_i,Y_i|t,\mathit{\theta })+{\displaystyle \frac{1}{n}}\sum _{i=\left[n\tau \right]+1}^n\tilde{f}(X_i,Y_i|t,\mathit{\theta }),$$

(7)

where $\left[n\tau \right]=k,\left[a\right]$ means the integer part of a and

$$\begin{array}{cc}\hfill \tilde{f}(X_i,Y_i|t,\mathit{\theta })=\log [{\displaystyle \sum _{c=1}^C}\{& {\pi }_c^{-}\varphi \left(Y_i\right|X_i^T{\beta }_c^{-},{\sigma }_c^{2-})I_{\{i\le [nt\left]\right\}}+\hfill \\ \hfill & {\pi }_c^{+}\varphi \left(Y_i\right|X_i^T{\beta }_c^{+},{\sigma }_c^{2+})I_{\{i\ge [nt]+1\}}\}].\hfill \end{array}$$

Without loss of generality, we only consider the case $k\le k^{\ast }$ for brevity; then, by $Kolmogorov$’s Law of Large Numbers and $Jessen$ inequality, the Formula (7) can be rewritten as

$$\begin{array}{cc}\hfill {\mathcal{L}}_n(\tau ,\mathit{\theta })& ={\displaystyle \frac{1}{n}}\sum _{i=1}^{\left[n\tau \right]}f(X_i,Y_i|\tau ,{\mathit{\theta }}^{-})+{\displaystyle \frac{1}{n}}\sum _{i=\left[n\tau \right]+1}^{\left[n{\tau }^{\ast }\right]}f(X_i,Y_i|\tau ,{\mathit{\theta }}^{-})+{\displaystyle \frac{1}{n}}\sum _{i=\left[n{\tau }^{\ast }\right]+1}^{n}f(X_i,Y_i|\tau ,{\mathit{\theta }}^{+})\hfill \\ \hfill & ={\displaystyle \frac{\left[n\tau \right]}{n}}·{\displaystyle \frac{1}{\left[n\tau \right]}}\sum _{i=1}^{\left[n\tau \right]}f(X_i,Y_i|\tau ,{\mathit{\theta }}^{-})+{\displaystyle \frac{\left[n{\tau }^{\ast }\right]-\left[n\tau \right]}{n}}·{\displaystyle \frac{1}{\left[n{\tau }^{\ast }\right]-\left[n\tau \right]}}\sum _{i=\left[n\tau \right]+1}^{\left[n{\tau }^{\ast }\right]}f(X_i,Y_i|\tau ,{\mathit{\theta }}^{-})\hfill \\ \hfill & +{\displaystyle \frac{n-\left[n{\tau }^{\ast }\right]}{n}}·{\displaystyle \frac{1}{n-\left[n{\tau }^{\ast }\right]}}\sum _{i=\left[n{\tau }^{\ast }\right]+1}^{n}f(X_i,Y_i|\tau ,{\mathit{\theta }}^{+})\hfill \\ \hfill & \stackrel{p}{\to }\tau ·E_{({\tau }^{\ast },{\mathit{\theta }}^{\ast -})}f(X,Y|\tau ,{\mathit{\theta }}^{-})+({\tau }^{\ast }-\tau )·E_{({\tau }^{\ast },{\mathit{\theta }}^{\ast -})}f(X,Y|\tau ,{\mathit{\theta }}^{-})\hfill \\ \hfill & +(1-{\tau }^{\ast })·E_{({\tau }^{\ast },{\mathit{\theta }}^{\ast +})}f(X,Y|\tau ,{\mathit{\theta }}^{+})\hfill \\ \hfill & \triangleq \mathcal{L}(\tau ,\mathit{\theta }|{\tau }^{\ast },{\mathit{\theta }}^{\ast })\hfill \end{array}$$

where

$$f(X_i,Y_i|\tau ,{\mathit{\theta }}^{-})=\tilde{f}(X_i,Y_i|\tau ,\mathit{\theta }){{|}_{\mathit{\theta }={\mathit{\theta }}^{-}},f(X_i,Y_i|\tau ,{\mathit{\theta }}^{+})=\tilde{f}(X_i,Y_i|\tau ,\mathit{\theta })|}_{\mathit{\theta }={\mathit{\theta }}^{+}}.$$

Now, we state our second theoretical result, which is the consistency of the obtained estimator.

Theorem 2.

Let $\widehat{\tau }={\displaystyle \frac{{\widehat{k}}_n}{n}}$ and under the conditions of the Lemmas A1 and A2 in Appendix A; then, we have

$$(\widehat{\tau },{\widehat{\mathit{\theta }}}_n)\stackrel{p}{\to }({\tau }^{\ast },{\mathit{\theta }}^{\ast }).$$

4. Simulation

To assess the performance of our proposed method, we shall consider two simulation experiments as follows.

4.1. Estimation Experiment

In Experiment 1, we focus on the scenario where the vector $\mathit{\beta }$ is two-dimensional and the value of C is set to 2. Additionally, when dealing with the parameter estimation problem of mixed models that incorporate change points, the conventional approach is to overlook the presence of these change points. Instead, the Expectation–Maximization (EM) algorithm is directly employed to estimate the unknown parameters, as if the data were generated from a mixed model consisting of $2C$ components. Consequently, we conduct a comparison between the estimation results obtained by the EM method that does not take change points into account (referred to as “NCPEM”) and those derived from our improved EM method (“CPEM”). We utilize the data generation mechanism of Model (1) to generate the data. Subsequently, we construct a two-component mixed model with a change point and proceed to specify the parameters as follows:

$$\begin{array}{cc}\hfill {\left({\mathit{\beta }}_1^{\ast -}\right)}^T& =(1,2),{\sigma }_1^{2\ast -}=0.1,{\pi }_1^{\ast -}=0.4,{\left({\mathit{\beta }}_2^{\ast -}\right)}^T=(1,3),{\sigma }_2^{2\ast -}=0.2,{\pi }_2^{\ast -}=0.6.\hfill \\ \hfill {\left({\mathit{\beta }}_1^{\ast +}\right)}^T& =(3,4),{\sigma }_1^{2\ast +}=0.2,{\pi }_1^{\ast +}=0.7,{\left({\mathit{\beta }}_2^{\ast +}\right)}^T=(3,5),{\sigma }_2^{2\ast +}=0.2,{\pi }_2^{\ast +}=0.3.\hfill \end{array}$$

${\mathit{X}}_{\mathit{i}}^{\mathit{T}}=(1,X_i)$ and $X_i\sim U(0,1)$ for $i=1,2,\cdots ,n.$ Let the total sample size n from $\{500,1000,2000\}$ and the change point $k^{\ast }$ be $\{250,500,1000\}$, respectively. The change point estimators and the corresponding standard errors are $\widehat{k}=250.45\left(0.730\right),500.18\left(0.458\right), 1000.5\left(0.759\right)$ over 100 simulations. It can be seen that the change point estimator is close to the true value. The mixture parameter estimators and the corresponding standard errors are presented in

Table 1. Estimated value and standard error of ${\mathit{\beta }}^{\ast }$.

	${\mathit{\beta }}^{\ast }$	${\left({{\mathit{\beta }}_1}^{\ast -}\right)}^{\mathit{T}}=(1,2)$	${\left({{\mathit{\beta }}_2}^{\ast -}\right)}^{\mathit{T}}=(1,3)$	${\left({{\mathit{\beta }}_1}^{\ast +}\right)}^{\mathit{T}}=(3,4)$	${\left({{\mathit{\beta }}_2}^{\ast +}\right)}^{\mathit{T}}=(3,5)$
n = 500	CPEM	(0.994, 2.004)	(0.975, 3.023)	(2.993, 4.004)	(2.999, 4.999)
		(0.045, 0.029)	(0.150, 0.129)	(0.064, 0.041)	(0.085, 0.057)
	NCPEM	(2.235, 1.911)	(0.988, 3.048)	(3.086, 3.874)	(3.022, 4.941)
		(9.906, 1.624)	(7.933, 1.290)	(9.285, 1.609)	(0.511, 0.278)
n = 1000	CPEM	(0.982, 2.054)	(1.006, 2.993)	(2.992, 4.005)	(2.996, 5.001)
		(0.042, 0.017)	(0.104, 0.077)	(0.039, 0.024)	(0.058, 0.038)
	NCPEM	(0.452, 2.220)	(0.884, 3.044)	(1.550, 4.192)	(3.187, 4.939)
		(8.490, 1.664)	(3.710, 0.678)	(8.211, 1.367)	(2.540, 0.484)
n = 2000	CPEM	(1.000, 1.999)	(1.014, 2.995)	(2.930, 4.017)	(3.001, 4.997)
		(0.022, 0.014)	(0.069, 0.046)	(0.030, 0.019)	(0.041, 0.035)
	NCPEM	(1.225, 2.054)	(1.318, 2.975)	(2.880, 3.921)	(3.009, 4.975)
		(4.212, 0.624)	(2.383, 0.505)	(0.557, 0.342)	(0.182, 0.107)

,

Table 2. Estimated value and standard error of ${\sigma }^{2\ast }$.

	${\mathit{\sigma }}^{2\ast }$	${\mathit{\sigma }}_1^{2\ast -}=0.1$	${\mathit{\sigma }}_2^{2\ast -}=0.2$	${\mathit{\sigma }}_1^{2\ast +}=0.1$	${\mathit{\sigma }}_2^{2\ast +}=0.2$
n = 500	CPEM	0.099 (0.006)	0.199 (0.012)	0.100 (0.008)	0.197 (0.011)
	NCPEM	0.193 (0.466)	0.346 (0.626)	0.171 (0.436)	0.441 (0.786)
n = 1000	CPEM	0.098 (0.011)	0.200 (0.005)	0.099 (0.006)	0.199 (0.008)
	NCPEM	0.143 (0.287)	0.375 (0.686)	0.147 (0.338)	0.376 (0.661)
n = 2000	CPEM	0.100 (0.003)	0.203 (0.015)	0.099 (0.011)	0.204 (0.005)
	NCPEM	0.153 (0.339)	0.333 (0.616)	0.096 (0.022)	0.376 (0.687)

and

Table 3. Estimated value and standard error of ${\pi }^{\ast }$.

	${\mathit{\pi }}^{\ast }$	${\mathit{\pi }}_1^{\ast -}=0.6$	${\mathit{\pi }}_2^{\ast -}=0.4$	${\mathit{\pi }}_1^{\ast +}=0.3$	${\mathit{\pi }}_2^{\ast +}=0.7$
n = 500	CPEM	0.601 (0.031)	0.399 (0.031)	0.301 (0.030)	0.699 (0.030)
	NCPEM	0.182 (0.053)	0.310 (0.050)	0.189 (0.045)	0.320 (0.062)
n = 1000	CPEM	0.594 (0.063)	0.406 (0.063)	0.295 (0.019)	0.705 (0.019)
	NCPEM	0.189 (0.060)	0.310 (0.044)	0.193 (0.039)	0.308 (0.056)
n = 2000	CPEM	0.599 (0.014)	0.401 (0.013)	0.298 (0.033)	0.702 (0.033)
	NCPEM	0.194 (0.035)	0.311 (0.058)	0.186 (0.049)	0.310 (0.045)

.

As is evident from Table 1, the estimation results obtained by the Change Point Expectation–Maximization (CPEM) method are much closer to the true parameter values compared to those derived from the Non-Change Point Expectation–Maximization (NCPEM) method. Moreover, the CPEM method yields a significantly smaller standard error. Based on the estimations of ${\sigma }^{2\ast }$ and ${\pi }^{\ast }$ presented in Table 2 and Table 3, it is clear that the estimation results of the NCPEM method deviate substantially from the true parameter values. Additionally, as the sample size increases, the majority of the estimation results tend to converge more closely to the true parameter values, accompanied by a smaller standard error. These findings strongly indicate that our improved CPEM method exhibits superior accuracy and enhanced robustness.

4.2. Detection Experiment

In experiment 2, we explore the performance of the detection procedure. Let $p=1$ and consider the following mixture model:

$$Y_i\sim \left\{\begin{array}{ccc}\hfill {\displaystyle \sum _{c=1}^2}{\pi }_c^{\ast -}N\left({\mu }_c^{\ast }-,{\sigma }_c^{2\ast -}\right),& & 1\le i\le k^{\ast },\hfill \\ \hfill {\displaystyle \sum _{c=1}^2}{\pi }_c^{\ast +}N\left({\mu }_c^{\ast +},{\sigma }_c^{2\ast +}\right),& & k^{\ast }+1\le i\le n,k^{\ast }\in [1,n],\hfill \end{array}\right.$$

(8)

where

$${\mu }_1^{\ast -}=1,{\sigma }_1^{2\ast -}=0.1,{\pi }_1^{\ast -}=0.6,{\mu }_2^{\ast -}=2,{\sigma }_2^{2\ast -}=0.2,{\pi }_2^{\ast -}=0.4,$$

$${\mu }_1^{\ast +}=3,{\sigma }_1^{2\ast +}=0.2,{\pi }_1^{\ast +}=0.3,{\mu }_2^{\ast +}=4,{\sigma }_2^{2\ast +}=0.3,{\pi }_2^{\ast +}=0.7.$$

The limiting distribution function in (6) is well known and tabulated by Kiefer [23]. In model (8), $q=5$, at the significance level of 0.05, the critical value is 2.0005. We set the values of n to be 500, 1000, 2000, k to be $n/4$, $n/2$, and $3n/4$ to represent the case where the change point is located in the front, middle, and back end of the data. The empirical sizes and powers are shown in

Table 4. Empirical sizes.

n	500	1000	2000
size	0.07	0.06	0.05

and

Table 5. Empirical powers.

	${\mathit{k}}^{\ast }$	$\frac{\mathit{n}}{4}$	$\frac{\mathit{n}}{2}$	$\frac{3\mathit{n}}{4}$
n
500		0.60	0.83	0.59
1000		0.96	0.88	0.67
2000		1	0.98	0.94

under 100 simulations.

Based on the analysis of the data in the tables, the performance of the method proposed in this paper under large-sample scenarios can be summarized as follows:

(1) Empirical Level Performance: As clearly shown in Table 4, with the continuous increase in the sample size n, the empirical level gradually approaches the significance level of 0.05. This indicates that, under large sample conditions, the actual test level of this method is in good agreement with the preset theoretical level.

(2) Empirical Power Performance: According to the data results in Table 5, there is a significant positive correlation between the increase in sample size and the improvement in empirical power. That is, the larger the sample size, the closer the value of empirical power is to 1, suggesting that the method can more effectively detect true change point situations in large samples.

(3) Influence of Change Point Location: The position of the change point in the data sequence has an impact on the detection effect. When the change point is located in the middle or front part of the data, the detection effect of this method is significantly better than when the change point is located at the end of the data.

(4) Comprehensive Performance Advantages: In large sample scenarios, the method proposed in this paper demonstrates good empirical size and power.

5. Concluding Remarks

This paper proposes an improved EM method to solve the parameter estimation problem with a mixture of the linear regression model and change point. This method is more accurate and robust against usual EM algorithms. Furthermore, a detection method based on score function is obtained to test the change point in mixture data. Simulations demonstrate the effectiveness of the proposed methodology.

6. Limitations and Future Research Directions

The method presented in this article also has certain limitations and challenges. Firstly, there is the issue of robustness in complex noisy scenarios. The method proposed in this article performs well in dealing with normal noise, but the detection accuracy remains to be studied under complex noise interference. The algorithm mentioned in the article still needs further research in dealing with high-dimensional data point detection and estimation problems. Finally, the computational complexity of the method proposed in this article is relatively high, and it is worth studying whether there is a simpler and faster processing method. The above issues are also possible research directions for the future.