Scalable Model Selection for Spatial Additive Mixed Modeling: Application to Crime Analysis

Abstract

A rapid growth in spatial open datasets has led to a huge demand for regression approaches accommodating spatial and non-spatial effects in big data. Regression model selection is particularly important to stably estimate flexible regression models. However, conventional methods can be slow for large samples. Hence, we develop a fast and practical model-selection approach for spatial regression models, focusing on the selection of coefficient types that include constant, spatially varying, and non-spatially varying coefficients. A pre-processing approach, which replaces data matrices with small inner products through dimension reduction, dramatically accelerates the computation speed of model selection. Numerical experiments show that our approach selects a model accurately and computationally efficiently, highlighting the importance of model selection in the spatial regression context. Then, the present approach is applied to open data to investigate local factors affecting crime in Japan. The results suggest that our approach is useful not only for selecting factors influencing crime risk but also for predicting crime events. This scalable model selection will be key to appropriately specifying flexible and large-scale spatial regression models in the era of big data. The developed model selection approach was implemented in the R package spmoran.

1. Introduction

Regression modeling is widely used to investigate the factors of geographical phenomena, such as plague spread, species distribution, economic agglomeration, and crime rates. For instance, [1,2,3] apply regression models to study the influence of neighborhood affluence, race, unemployment rate, and facilities, such as liquor stores and stations, and other covariates on crime risk. Nowadays, an increasing number of open datasets of crime statistics are available [4]. For example, the Tokyo Metropolitan Government (https://www.bouhan.metro.tokyo.lg.jp/opendata/index.html), has made crime statistics (2014–present), classified by crime type and minor municipal districts, publicly available. Moreover, many of the crime statistics are of good geographical resolutions at district-level or other spatial fine scales, and they record thousands to tens of thousands of data entries in each time period. For such large spatiotemporal data, a computationally efficient regression approach is very important.

In applied spatial analysis, estimation and identification of linear effects and spatially effects to objective variables are actively studied. For example, spatial econometric models are used to estimate linear effects in the presence of spatial dependence [5]. Gaussian process models have been extended to accommodate a wide variety of spatial effects in geostatistics [6]. Geographically weighted regression (GWR) [7] is used to estimate spatially varying coefficients (SVCs) on covariates [8]. Among spatial effects, we especially focus on SVC modeling that allows for estimating the local determinants of crimes [2,9,10]. For instance, [2] found that affluence reduces crime in a suburban area of Portland, Oregon, whereas it increases crime in the city center. Understanding such local differences in crime occurrence is important when considering security measures against crimes.

Apart from the spatial effects, influence from covariates can also vary non-spatially depending on time, quantile, or other variables or events [11]. Unfortunately, when all possible effects are included in the model, over-parametrization occurs and the model becomes unstable. To balance model accuracy and complexity, model selection is crucially important. For example, in crime analysis, it is needed to appropriately specify the key factors behind crimes.

There are many model selection methods for SVC models [12,13,14] and other additive models that accommodate spatial and/or non-spatial effects [15,16]. Model selection is typically performed through iterations of model updating through inclusion/exclusion of effects (e.g., SVC) until convergence. In the case of large samples, however, SVC model selection by iterative fitting is computationally demanding. For example, mixed/semiparametric GWR [12,17], which selects constant coefficients or SVC, requires a computational time complexity of O(N²) per iteration [18], where N is the sample size and O($·$) denotes the order. Although there are fast approaches for selecting spatial and/or non-spatial effects, they still iterate model-fitting steps with a computational complexity of O(N).

Given this background, this study develops a computationally efficient approach for selecting spatial and/or non-spatial effects under the framework of spatial additive mixed modeling [19,20]. It employs a pre-conditioning treatment to reduce the computation cost of parameter estimation but is not applied to model/effects selection. By extending the idea of [19,20], we develop a scalable approach for selecting both spatial/non-spatial effects. This method significantly reduces the computational time complexity of the iterative fitting steps such that the cost is independent of the sample size N.

The remainder of this paper is organized as follows. Section 2 introduces our model. Section 3 develops our model selection procedures, and Section 4 examines its performance through Monte Carlo simulation experiments. Section 5 applies the developed approach to crime modeling and forecasting and Section 6 concludes our discussion.

2. Spatial Additive Mixed Model

2.1. Model

We consider the following spatial additive mixed model:

$$y={\displaystyle \sum }_{p=1}^P{x}_p°f_p+\epsilon , \epsilon ~N\left(0,{\sigma }^{2}I\right)$$

(1)

where $°$ is the element-wise product operator, $y$ is a vector of response variables ($N\times 1$), N is the sample size, $x_p$ is a vector of the p-th covariate, $\epsilon$ is a vector of disturbances with variance ${\sigma }^2$, 0 is a vector of zeros, and I is an identity matrix. $f_p$ is a vector of coefficients describing the influence of the p-th covariate.

There are many specifications for $f_p$. The most basic specification is the constant $f_p=b_p\mathbf{1}$, where $b_p$ is a coefficient and $\mathbf{1}$ is a vector of ones, which is assumed in common linear regression models.

One key idea used in this study is to specify $f_p$ SVCs. For instance, [21] adopted the following specification:

$$f_p=b_p{\mathbf{1}}_p+\frac{{\tau }_p^{\left(s\right)}}{\sigma }{E}^{\left(s\right)}{\Lambda }^{{\alpha }_p}{u}_p^{\left(s\right)}, u_p^{\left(s\right)}~N\left(0_p,{\sigma }^2{I}_p\right)$$

(2)

$E^{\left(s\right)}$ is a $\left(N\times L_p\right)$ matrix of $L_p$ eigenvectors corresponding to positive eigenvalues, which are called Moran eigenvectors (ME); they are extracted from a doubly-centered spatial proximity matrix [22]. $\Lambda$ is a $L_p\times L_p$ diagonal matrix whose elements are positive eigenvalues. $u_p^{\left(s\right)}$ is a $\left(L_p\times 1\right)$ vector of random variables that acts as a Gaussian prior to stabilizing the SVC estimations. The ${\alpha }_p$ and ${\tau }_p^{\left(s\right)}$ parameters determine the scale and standard error of the spatial process. One characteristic advantage of the ME approach is that the resulting SVC ($f_p$) is interpretable through the Moran coefficient, which is a diagonal statistic of spatial dependence and its value can be positive (negative) in the presence of positive (negative) spatial dependence. Specifically, when considering all the MEs that have positive eigenvalues, $\frac{{\tau }_p^{\left(s\right)}}{\sigma }{E}^{\left(s\right)}{\Lambda }^{{\alpha }_p}{u}_p$ describes a positively dependent map pattern. In other words, Equation (2) furnishes a positively dependent spatial process, which is dominant in many real-world cases [23], efficiently. The Moran coefficient value increases as ${\alpha }_p$ grows.

The $f_p$ function can also be determined in terms of a non-spatially varying coefficient (NVC). This specification captures the influence varying with respect to the covariate $x_p$ as follows:

$$f_p=b_p{\mathbf{1}}_p+\frac{{\tau }_p^{\left(n\right)}}{\sigma }{E}_p^{\left(n\right)}{u}_p^{\left(n\right)}, u_p^{\left(n\right)}~N\left(0_p,{\sigma }^2{I}_p\right)$$

(3)

where $E_p^{\left(n\right)}$ is a $N\times L_p$ matrix of $L_p,$ and the basis function generated from $x_p$. ${\tau }_p^{\left(n\right)}$ denotes the variance of non-spatial effects.

The following specification, which assumes both spatially and non-spatially varying coefficients (S&NVC) on all the covariates is also possible:

$$f_p=b_p{1}_p+\frac{{\tau }_p^{\left(s\right)}}{\sigma }{E}^{\left(s\right)}{\Lambda }^{{\alpha }_p}{u}_p^{\left(s\right)}+\frac{{\tau }_p^{\left(n\right)}}{\sigma }{E}_p^{\left(n\right)}{u}_p^{\left(n\right)}, u_p^{\left(s\right)}~N\left(0_p,{\sigma }^2{I}_p\right), u_p^{\left(n\right)}~N\left(0_p,{\sigma }^2{I}_p\right)$$

(4)

Ref. [24] showed that the S&NVC model is robust against spurious correlations, whereas naive SVC models tend to have spurious correlations [25]. Finally, Figure 1 illustrates the coefficients we will consider in this study.

In summary, $f_p$, the constant is given by a fixed coefficient ($b_p$), while SVC, NVC, and S&NVC are specified by sums (linear combinations) of fixed and random effects. By substituting these values, the model Equation (1) is formulated as follows:

$$y=Xb+\tilde{E}\left(\Theta \right)U+\epsilon , U~N\left(0, {\sigma }^{2}I\right), \epsilon ~N\left(0,{\sigma }^{2}I\right)$$

(5)

$X=\left[x_1,\dots ,x_P\right]$, $b=\left[b_1,\dots ,b_P\right]{}^{\prime }$, $U=\left[u_1,\dots ,u_P\right]{}^{\prime }$, and $\Theta \in \left\{{\mathsf{\theta }}_1,\dots ,{\mathsf{\theta }}_P\right\},$ where “ ′ ” represents the matrix transpose. $\tilde{E}\left(\Theta \right)=\left[x_1°E_1{V}_1\left({\mathsf{\theta }}_1\right),\dots ,x_P°E_P{V}_P\left({\mathsf{\theta }}_P\right)\right]$, where “$a°B$” is an operator multiplying a column vector $a$ element-wise with each column of $B$. The matrices $E_p$, $V_p\left({\mathsf{\theta }}_p\right)$, and parameter ${\mathsf{\theta }}_p$ are defined in

Table 1. Specifications for $\left\{E_p, V_p\left({\mathsf{\theta }}_p\right),{\mathsf{\theta }}_p\right\}$.

Model Type	$${\mathrm{E}}_p$$	$${\mathrm{V}}_P\left({\mathrm{\theta }}_P\right)$$	$${\mathrm{\theta }}_P$$
Constant	0	0	N.A.
SVC	$$E^{\left(s\right)}$$	$$\frac{{\tau }_p^{\left(s\right)}}{\sigma }{\Lambda }^{{\alpha }_p}$$	$$\left\{\frac{{\tau }_p^{\left(s\right)}}{\sigma },{\alpha }_p\right\}$$
NVC	$$E^{\left(n\right)}$$	$$\frac{{\tau }_p^{\left(n\right)}}{\sigma }{I}_p$$	$$\frac{{\tau }_p^{\left(n\right)}}{\sigma }$$
S&NVC	$$[E^{\left(s\right)}, E^{\left(n\right)}]$$	$$\left[\begin{array}{cc}\frac{{\tau }_p^{\left(s\right)}}{\sigma }{\Lambda }^{{\alpha }_p}& ⬚\\ ⬚& \frac{{\tau }_p^{\left(n\right)}}{\sigma }{I}_p\end{array}\right]$$	$$\left\{\frac{{\tau }_p^{\left(s\right)}}{\sigma },{\alpha }_p\right\}$$

. Equation (5) suggests that our model is formulated as a linear mixed-effects model [26] with fixed effects $Xb$, random effects, $\tilde{E}\left(\Theta \right)U,$ and $\epsilon$.

Although this study focuses on four specifications, other $f_p$ functions have been proposed to represent linear effects, non-linear effects, group effects, and other effects, as outlined in [11]. Due to its flexibility, the additive mixed model is now used in many applied studies [27,28].

2.2. Estimation

Among the estimation algorithms for spatial additive mixed models, we focus on the fast restricted maximum likelihood (REML) estimation of [19], which is scalable for both sample size N and the number of effects P. The computational bottleneck here is an iterative evaluation of the restricted log-likelihood $logli{k}_R\left(\Theta \right)$ of Equation (1) (or Equation (5)) to numerically estimate the variance parameters $\Theta \in \left\{{\mathsf{\theta }}_1,\cdots {\mathsf{\theta }}_P\right\}$ (see Appendix A). To reduce cost, [19] developed a sequential estimation of $\left\{{\mathsf{\theta }}_1,\cdots {\mathsf{\theta }}_P\right\}$ parameters by applying Equation (6) until convergence.

$${\widehat{\mathsf{\theta }}}_p= \arg \underset{{\mathsf{\theta }}_p}{\max } logli{k}_R({\mathsf{\theta }}_p|{\Theta }_{-p})$$

(6)

While the direct maximization of Equation (6) still has the same computational burden, [19] developed the following procedure for the fast REML:

(I)

Replace the data matrices {$y,X,E_1,\dots ,E_P$}, whose dimensions are dependent on N, with their inner products, whose dimensions are independent of N.

(II)

Using the inner products, iterate the following calculations sequentially for $p\in \left\{1,\dots ,P\right\}$:

(II–1)

Estimate ${\widehat{\mathsf{\theta }}}_p$ by maximizing $loglik({\mathsf{\theta }}_p|{\widehat{\Theta }}_{-p})$ with ${\Theta }_{-p}\in \left\{{\mathsf{\theta }}_1,\cdots {\mathsf{\theta }}_{p-1},{\mathsf{\theta }}_{p+1},\cdots {\mathsf{\theta }}_P\right\}$.

(II–2)

Go to (III) if the likelihood value converges. Else, return to (II-1).

(III)

Output the final model.

In this algorithm, the data matrices are replaced with their inner products before the iterative likelihood evaluation step. After all, the computational complexity of the iterative likelihood evaluation to find ${\widehat{\mathsf{\theta }}}_p$ in step (II–1) reduces to $O\left(L_p^3\right)$ [19]. In other words, after the pre-conditioning step (I), the computational complexity of the fast REML is highly scalable for both the sample size N and the number of effects P. Owing to this property, the spatial additive mixed model Equation (1) can be estimated efficiently even when N and P are very large.

3. Model Selection

3.1. Introduction

As explained in Section 2, the coefficients in Equation (1) can be constant, SVC, NVC, or S&NVC. The complexity of the model depends considerably on the selection of the coefficient type. For instance, Equation (1) has only P coefficients if all the coefficients are assumed to be constant. In the case of the SVC-based model, on the other hand, the number of coefficient parameters is ${\displaystyle \sum }_{p=1}^P{L}_p,$ since the model utilizes $L_p$-dimension Moran eigenvectors for its P coefficient representations. Too many parameters can lead to overfitting and overestimation of statistical significance, whereas too few parameters can cause underfitting. This issue can be remedied by choosing an optimal model that provides an appropriate balance of the sizes of parameters and datasets with respect to model accuracy.

There is one difficulty in the selection of a large number of candidate models; there are 4^P model specifications for Equation (1). For example, if P = 9, which we will assume later, there are 4⁹ = 262,144 models. However, in practice, it is desirable to find or approximate the best model within seconds or minutes. Hence, this study develops a computationally efficient model selection approach. We attempt to search the model by minimizing the cost function, which can be defined by the Akaike information criterion (AIC) or Bayesian information criterion (BIC).

It is important to note that constant, SVC, NVC, and S&NVC share the same fixed effect ($b_{p}1$), whereas their random effects differ from each other. In other words, we select random effects. In such a case, REML-based AIC and BIC are available for the model selection of linear additive mixed models [29]. While there are marginal and conditional AIC/BIC specifications, we focus on marginal BIC for the following reasons:

• It is the most common specification for linear mixed effects models [30], including spatial additive mixed models.
• Poor performance of conditional AIC/BIC-based model selection was reported when considering two or more random effects (see [31,32]), whereas [33] showed that conditional AIC/BIC outperforms when comparing models with/without one random effect.
• Although the marginal specification suffers from a theoretical bias, [32] showed that the influence of the bias on model selection result is quite small.

Based on a preliminary analysis, we decided to use the REML-based marginal BIC, defined by $-2logli{k}_R\left(\Theta \right)-2Q\log \left(N\right)$, where Q is the number of fixed coefficients (P) and variance parameters in $\Theta$ in the crime analysis in Section 4.

3.2. Model Selection Procedures

This section proposes novel practical methods for model selection. The first incorporates a model selection into the sequential REML estimation (see Section 2.2). To reduce the chance of trapping to local optima, the second approach relies on a Monte Carlo (MC) simulation iterating the sequential REML estimation. We call the former a simple selection method, which emphasizes simplicity and practicality, and the latter a MC selection method (see Appendix B). Section 3.2.1 and Section 3.2.2 explain these methods.

3.2.1. Simple Selection Method

The simple selection method consists of model selection steps in sequential REML estimation. The procedure of this method is as follows:

(a)

Replace the data matrices {$y,X,E_1,\dots ,E_P$} with the inner products as processed in step (II) in Section 2.2.

(b)

Perform the following calculation sequentially for each $p\in \left\{1,\dots ,P\right\}$:

(b–1)

Estimate the p-th SVC by maximizing $loglik({\mathsf{\theta }}_{p\left(s\right)}|{\widehat{\Theta }}_{-p\left(s\right)})$ with respect to ${\mathsf{\theta }}_{p\left(s\right)}$, which is a subset of ${\mathsf{\theta }}_p$ characterizing the SVC, and ${\widehat{\Theta }}_{-p\left(s\right)}$ represents the set of variance parameters excluding ${\widehat{\mathsf{\theta }}}_p$ from $\widehat{\Theta }$.

(b–2)

Select the SVC if it improves the cost function value (e.g., BIC). Otherwise, replace it with a constant.

(b–3)

Estimate the p-th NVC by maximizing $loglik({\mathsf{\theta }}_{p\left(n\right)}|{\widehat{\Theta }}_{-p\left(n\right)})$ with respect to ${\mathsf{\theta }}_{p\left(n\right)}$, which is a subset of ${\mathsf{\theta }}_p$ characterizing the NVC, and ${\widehat{\Theta }}_{-p\left(s\right)}$ represents the set of variance parameters excluding ${\widehat{\mathsf{\theta }}}_{p\left(n\right)}$ from $\widehat{\Theta }$.

(b–4)

The NVC is selected if it improves the cost function value (e.g., BIC). Otherwise, it is replaced with a constant.

(c)

Go to (d) if the cost function converges. Otherwise, go back to (b).

(d)

Output the final model.

While there are similar selection approaches [34], ours is distinctive because its computational complexity for model selection is independent of the sample size, owing to step (a) that renders all the data matrices into their inner products. Due to the drastic dimension reduction, this simple method is suitable for very large samples. Specifically, the computational complexity of the iterative part in the model selection procedure equals $O\left(L_p^3\right)$, which is required to evaluate the log likelihood value in step (b–1) and (b–3). Because the computational complexity is very small and independent of the sample size N, our model selection procedure is extremely fast even for large samples.

One problem is the large number of combinations of pre-determined sequence $p\in \left\{1,\dots ,P\right\}$. For example, if P = 9, there are $P!=362,880$ sequences; some of them might result in poor model selection result (i.e., local optimum). At least, unlike the maximum likelihood estimation or cross-validation, REML tends to not have local optima [35,36]. Section 4 examines if this simple approach accurately approximates/selects the true model through Monte Carlo experiments.

3.2.2. Monte Carlo (MC) Selection Method

To reduce the risk of falling into local optima, we randomly sample sequences $p\in \left\{1,\dots ,P\right\}$ and iterate the REML-based estimation given the sequence. Specifically, we propose the following model selection approach:

(A)

Replace the data matrices {$y,X,E_1,\dots ,E_P$} with the inner products.

(B)

Iterate the following calculation G times using the inner products:

(B–1)

Randomly sample the g-th sequence $\left\{1_g,\dots ,P_g\right\}$ without replacement.

(B–2)

Perform the following calculation sequentially for each $p_g\in \left\{1_g,\dots ,P_g\right\}$:

(B–2a)

Estimate the $p_g$-th SVC by maximizing $loglik({\mathsf{\theta }}_{p_g\left(s\right)}|{\widehat{\Theta }}_{-p_g\left(s\right)})$, where $\left\{ {\mathsf{\theta }}_{p_g\left(s\right)},{\widehat{\Theta }}_{-p_g\left(s\right)}\right\}$ are defined similarly as $\left\{{\mathsf{\theta }}_{p\left(s\right)},{\widehat{\Theta }}_{-p\left(s\right)}\right\}$.

(B–2b)

Select the SVC if it improves the cost function value (e.g., BIC). Otherwise, replace it with a constant.

(B–2c)

Estimate the $p_g$-th NVC by maximizing $loglik({\mathsf{\theta }}_{p_g\left(n\right)}|{\widehat{\Theta }}_{-p_g\left(n\right)})$, where $\left\{ {\mathsf{\theta }}_{p_g\left(n\right)},{\widehat{\Theta }}_{-p_g\left(n\right)}\right\}$ are defined similarly as $\left\{{\mathsf{\theta }}_{p\left(n\right)},{\widehat{\Theta }}_{-p\left(n\right)}\right\}$.

(B–2d)

The NVC is selected if it improves the cost function value (e.g., BIC). Otherwise, it is replaced with a constant.

(B–3)

Go to (B–4) if the cost function converges. Otherwise, go back to (B–2).

(B–4)

Calculate the cost function value of the selected model.

(C)

Output the best model in the selected G models in terms of the lowest cost function.

As with the simple selection approach, the computational cost for iterative step (B) is independent of the sample size. In addition, the iterative step is easily parallelized. Thus, this is a computationally efficient model selection procedure. Step (B) performs a MC simulation to marginalize G and obtain the distribution of the cost value. We call this approach the MC selection approach.

4. Numerical Experiments

4.1. Computational Details

Here, we examine whether this simple selection method accurately approximates the true model, which is the model with all the coefficient types (i.e., constant, SVC, NVC, or S&NVC) being defined correctly, or if the MC selection method is needed to achieve accurate model selection through comparative Monte Carlo experiments. We compare with/without the effects selection model by fitting these models to the synthetic data generated from

$$y={\mathsf{\beta }}_0+{\displaystyle \sum }_{p=1}^P{x}_p{b}_p+{\displaystyle \sum }_{p=1}^P{x}_{1,p}{\mathsf{\beta }}_{1,p}+{\displaystyle \sum }_{p=1}^P{x}_{2,p}{\mathsf{\beta }}_{2,p}+\epsilon , \epsilon ~N\left(0,I\right)$$

(7)

$${\mathsf{\beta }}_0=1+\tilde{C}{u}_0, u_0~N\left(0,I\right),$$

(8)

$${\mathsf{\beta }}_{1,p}=1+\tilde{C}{u}_{1,p}, u_{1,p}~N\left(0,{\tau }_{1,p}^{2}I\right),$$

(9)

$${\mathsf{\beta }}_{2,p}=1+E_{2,p}{u}_{2,p}, u_{2,p}~N\left(0,{\tau }_{2,p}^{2}I\right),$$

(10)

where the covariates $\left\{x_1,\dots ,x_1\right\},\left\{x_{1,p},\dots ,x_{1,p}\right\},\left\{x_{2,p},\dots ,x_{2,p}\right\}$ are generated from independent standard normal distributions. The matrix $\tilde{C}$ is constructed from row standardization of spatial connectivity matrix $C$, whose (i, j)-th element equals $\exp \left(-d_{i,j}\right)$, where $d_{i,j}$ is the Euclidean distance between sample sites i and j. The sample sites were generated from two independent standard normal distributions. The SVCs ${\mathsf{\beta }}_0$ and ${\mathsf{\beta }}_{1,p}$ are defined by spatial moving average processes. ${\mathsf{\beta }}_{2,p}$ is a NVC that varies with respect to $x_{2,p}$, in which $E_{2,p}$ is a matrix of 10 polynomial basis functions generated from $x_{2,p}$. Figure 2 illustrates the coefficients obtained from these generating processes.

The main objective is to compare the coefficient estimation accuracy and computational efficiency of simple and MC-based S&NVC model selections with alternatives. For MC model selection, we assumed 30 replicates. These models were fitted 200 times while varying $P\in \left\{1, 2, 3\right\}$, and sample size $N\in \left\{50, 200, 1000\right\}$.

The coefficient estimation accuracy is evaluated using the root mean squared error (RMSE), which is defined as

$$RMSE\left({\mathsf{\beta }}_{i,p}\right)=\sqrt{\frac{1}{200N}{\displaystyle \sum }_{iter=1}^{200}{\displaystyle \sum }_{i=1}^N{\left({\widehat{\beta }}_{i,p}^{\left(iter\right)}-{\beta }_{i,p}\right)}^2}$$

(11)

where iter represents the iteration number, ${\beta }_{i,p}$ is the i-th element of ${\mathsf{\beta }}_p$, and ${\widehat{\beta }}_{i,p}^{\left(iter\right)}$ is the estimate given in the iter-th iteration. The bias of the estimates is evaluated using

$$Bias\left({\mathsf{\beta }}_{i,p}\right)=\frac{1}{200N}{\displaystyle \sum }_{iter=1}^{200}{\displaystyle \sum }_{i=1}^N\left({\widehat{\beta }}_{i,p}^{\left(iter\right)}-{\beta }_{i,p}\right)$$

(12)

Under these settings, Section 4.2 examines if our approaches accurately select the best model, while Section 4.3 compares our approaches with other approaches.

4.2. Performance of Model Selection

In this experiment, we compare the following six models. As baseline models, the linear regression model (LM) and the SVCs across coefficients (SVC model) are used. As another baseline, we use the model assuming true coefficients types as known (i.e., constant coefficient on $x_p$, SVC on $x_{1,p}$, and NVC on $x_{2,p}$ (true model)). Note that the true model assumes coefficients values as unknown (i.e., only coefficients types are known), and the values are estimated from samples as with the other models. The estimates will have an estimation error. We construct a full model that assumes S&NVC across coefficients (S&NVC model). In addition, we prepare simple and MC-based S&NVC model selection approaches that select their coefficients among constant, SVC, NVC, and S&NVC using the simple approach and the Monte Carlo approach, respectively.

Figure 3 summarizes the RMSEs of the estimated coefficients. As expected, LM has higher estimation errors because of ignoring the spatial and non-spatial variations in regression coefficients. Although the SVC model is popular in spatial statistics, the RMSEs for the NVCs are considerably high. In addition, probably due to error, the estimation accuracy for the constant coefficients by the SVC model is worse than that of LM model. These results suggest that SVC-based models become unstable in the presence of constant or non-spatial coefficients.

Regarding the S&NVC model without model selection, the estimated SVCs and NVCs are as accurate as the true model. However, its RMSE values for the constants are the highest across the models, probably because of over-parameterization. On the other hand, the RMSEs of the simple and MC-based S&NVC model selections are close to the true model across all coefficients. These results demonstrate the importance of effect selection in spatial regression modeling.

Figure 4 plots the biases of the coefficient standard errors, which are used to evaluate statistical significance; the downward bias of the standard errors yields an overestimation of the statistical significance, whereas the opposite is true for the upward bias. Here, the standard errors estimated from the true model are regarded as the true values. Standard errors of the constant coefficients estimated from LM, SVC, and S&NVC models are downwardly biased. For SVC, those estimated from LM are upwardly biased while the SVC and SNVC models are downwardly biased. For NVC, the standard errors estimated from the LM and SVC models are upwardly biased. The RMSEs of the standard errors obtained by LM, SVC and S&NVC models are also high as shown in Figure 5. Based on the results, the models without effect selection (LM, SVC, and S&NVC models) suffer from an overestimation or underestimation of statistical significance.

Conversely, the standard errors estimated from the simple and MC-based model selection approaches are almost the same as those of the true model, except for the case with P = 3 and N = 100, which is the most severe case, estimating 10 effects from 100 samples. Our effects selection approaches are useful for improving estimation accuracy for both the coefficients and their standard errors. Surprisingly, it was also found that results from the simple model selection approach are almost the same as the MC-based approach, in spite of the fact that the simple approach relies on a pre-determined sequence $p\in \left\{1,\dots ,P\right\}$ for model selection, whereas the MC-based method implicitly optimizes it. This is attributable to the stability of REML (see Section 3.2.1).

4.3. Benchmark Comparison of Model Selection Methods

Here, we compare our model selection approach with another commonly used model selection approach. Among the existing approaches, we selected the one implemented in the mgcv package in R (https://cran.r-project.org/web/packages/mgcv/index.html) because of the following reasons: (i) mgcv is one of the most popular packages for additive mixed modeling; (ii) the effects selection procedure in mgcv is computationally highly scalable [37], and it is a sensible benchmark for testing both the accuracy and computational efficiency of our approach.

The following models are compared: LM, S&NVC model, simple S&NVC model selection, another S&NVC model estimated from mgcv (Mgcv), and double-penalty-based Mgcv model selection [37]. The double-penalty approach is the default model selection method in the mgcv package. Roughly speaking, the double-penalty approach imposes penalty parameters to select effects in addition to the usual penalty parameters in additive models determining the smoothness of each effect. [37] showed the superior accuracy of this approach over alternatives. For faster computation, we estimated Mgcv and Mgcv with the double-penalty model selection using the bam function in the mgcv package, using fast REML [38]. As in the previous section, we assumed Equations (7)–(10) as the true model and each model was iteratively fitted 200 times.

Figure 6 summaries the RMSEs for the estimated coefficients when P = 3 and $N\in \left\{400,1000\right\}$. For SVC and NVC, the RMSE values obtained from all the models except for LM are quite similar. Regarding the constant effect, our simple S&NVC model selection yields the lowest estimation error. Therefore, our approach is a promising alternative to Mgcv with/without the double-penalty model selection, which is now widely used.

Finally, Figure 7 compares the computational time in the cases of P = 10 and $N\in \left\{1000,10,000, 30,000,50,000, 100,000\right\}$. Here, the estimations were performed five times in each case, and the resulting computation times were averaged. Mgcv with double-penalty-based model selection is slightly slower than Mgcv because the former additionally estimates the penalty parameters of the selection effects. As explained in Section 1, such additional computational time for model/effect selection is taken for granted. On the other hand, the simple S&NVC model selection approach has a considerably shorter computational time than the naïve S&NVC model because the former treats only selected effects when evaluating likelihood in each iteration, while the latter includes all the effects when evaluating likelihood. In addition, owing to the pre-conditioning procedure, the increase in computational time with respect to N is suppressed with respect to the Mgcv-based alternatives. In contrast, the MC-based S&NVC model selection is slower than the alternatives, as shown in Figure 7. Based on estimation accuracy and computational efficiency, we recommend the simple S&NVC model selection as the default choice.

Theoretically, the simple S&NVC model selection is the fastest because of the following reasons: Monte Carlo simulation is not needed unlike the MC-based approach; it requires a smaller number of penalty parameters than Mgcv using K penalty parameters for smoothness selection while using other K parameters for model selection. The results here confirm the computational efficiency of our simple selection approach experimentally.

In summary, the analysis results demonstrate that our approach selects effects accurately and computationally efficiently, even in comparisons with state-of-the-art approaches.

5. Application to Crime Modeling

5.1. Outline

This section applies a simple approach to the Dai-Tokyo Bouhan network database (https://www.bouhan.metro.tokyo.lg.jp/) provided by the Office for Promotion of Citizen Safety, Tokyo Metropolitan Government (https://www.tomin-anzen.metro.tokyo.lg.jp/english/). This database records crime statistics categorized by crime type and 1529 minor municipal districts in Tokyo, Japan. The district zones are delineated to be homogeneous within each zone. Districts in urban areas are quite small while some districts in western mountain areas are large. The minimum area equals 0.003 km², the 1st quarter 0.301 km², the median 0.657 km², the 3rd quarter 1.102 km², and the maximum 68.714 km².

This study focuses on bicycle theft and shoplifting, the two most frequent non-burglary crimes reported between 2017 and 2018. The sample size was 12,232. Figure 8 plots the crime densities by minor municipal districts (number of incidences/km²) (first quarter of 2017). The eastern area within the looping railway (Yamanote line) is the central area, and other railways extend in all directions from the looping railway, as shown in the figure. The Chuo line is one such line.

We model the logged number of bicycle theft and shoplifting cases per area by districts by quarter. Based on Figure 9, which displays the histograms of the crime densities, data distributions are nearly Gaussian. A (Gaussian) linear model will be acceptable in our case. This study applies the following linear model:

$$y_{i,t}^c={\beta }_{i,0}+{\beta }_{i,1}{y}_{i,t-1}^c+{\beta }_{i,2}{Y}_{i,t-1}^{-c}+{\displaystyle \sum }_{k=3}^6{\beta }_{i,k}{x}_{i,k}+g_{d\left(i\right)}^{\left(s\right)}+g_{q\left(i\right)}^{\left(t\right)}+{\epsilon }_i, {\epsilon }_i~N\left(0,{\sigma }^2\right)$$

(13)

where $y_{i,t}^c$ is the number of c-th crime per area in the i-th district in the t-th quarter (log-scale).

We then consider explanatory variables. Based on the routine activity theory [39], the following three are the triggers for crimes: (i) potential offender, (ii) suitable target, and (iii) absence of guardian. While the crime prevention level explaining (iii) is roughly equal across Tokyo, (i) and (ii) considerably change depending on the district. Regarding bicycle theft, we consider nighttime population density (Popden) as an explanatory variable approximating (i) the number of potential attackers. Since bicycles are widely used for shopping in the study area, we consider Popden and the number of retailers (Retail) as explanatory variables describing (ii) the number of targets or bicycles. Regarding shoplifting, we consider daytime population density (Dpopden) as a variable explaining (i) the number of attackers, while considering Retail as another explanatory variable explaining (ii) the number of targets.

In crime geography, near repeat victimization [40], explaining the tendency that crimes tend to repeat in similar region, is known as a common phenomenon. Such repetitive tendency occurs because of the regional heterogeneity, such as resident characteristics (risk heterogeneity hypothesis) or repetitive crimes by the same group that have knowledge about the crime area (event dependency hypothesis) (see [41]). To consider such local repetitiveness, we include crime density in the previous quarter $y_{i,t-1}^c$ as an explanatory variable and call this variable Repeat. As such repetitive tendencies can occur across crime types, we also include the logged density of non-burglary crime $Y_{i,t-1}^{-c}$ (RepOther) apart from the c-th crime in the previous quarter. For the bicycle theft model, $Y_{i,t-1}^{-c}$ is given by log(the number of non-burglary crimes except for bicycle theft/area (km²)), while $Y_{i,t-1}^{-c}$ for the shoplifting model is similarly defined.

In addition, we include the following explanatory variables $\left\{x_{i,3},x_{i,4},x_{i,5},x_{i,6}\right\}$ describing local environmental factors: ratio of foreigners among residents (Fpopden); unemployment ratio (UnEmp); ratio of residents who graduated from university (Univ). These three are proxy variables of race, economic deprivation, and education. Strong influence from such local environmental factors has been demonstrated in studies relating to risk terrain modeling (RTM; [42]). These data were collected from the National Census Statistics by minor municipal districts in 2015.

As in the previous section, we assume a spatially varying intercept ${\beta }_{i,0}$ to eliminate residual spatial dependence and select coefficient type {${\beta }_{i,1},\dots ,{\beta }_{i,6}$} among {constant, SVC, NVC, S&NVC} using a simple approach. Furthermore, to capture the heterogeneity among individual districts and time periods, we consider $g_{d\left(i\right)}^{\left(s\right)}~N\left(0,{\tau }_{\left(s\right)}^2\right)$ and $g_{q\left(i\right)}^{\left(t\right)}~N\left(0,{\tau }_{\left(t\right)}^2\right)$, respectively, where ${\tau }_{\left(s\right)}^2$ and ${\tau }_{\left(t\right)}^2$ are variance parameters. These terms represent the group effects of district $d\left(i\right)$ and quarter $q\left(i\right)$ in which the i-th sample is observed. They are constant coefficients by districts and quarter, respectively (e.g., $g_{d\left(i\right)}^{\left(s\right)}$ takes larger positive values in districts with larger number of crimes). The inclusion or exclusion of $\left\{g_{d\left(i\right)}^{\left(s\right)},g_{q\left(i\right)}^{\left(t\right)}\right\}$ is also automatically selected by the simple approach.

5.2. Coefficient Estimation Results

The (conditional) adjusted R-squared value is 0.914 for the bicycle theft model and 0.928 for the shoplifting model. The accuracy of these models was verified. The estimation of these models took 67.3 s and 52.0 s, respectively, confirming the computational efficiency of our approach.

Table 2. Summary of coefficient estimates.

	Bicycle Theft
Coefficients	Intercept	Repeat	RepOther	Popden	Retail	Fpopden	UnEmp	Univ
Minimum	−0.469	0.702	0.144	0.008	0.014	−0.015	0.028	−0.035
1st quantile	−0.387	0.765	0.160	0.017
Median	−0.313	0.787	0.188	0.021
3rd quantile	−0.283	0.805	0.208	0.025
Maximum	−0.208	0.872	0.279	0.031
	Shoplifting
Coefficients	Intercept	Repeat	RepOther	Dpopden	Retail	Fpopden	UnEmp	Univ
Minimum	−0.312	0.854	0.125	8.18×10−5	5.83×10−4	−0.027	0.135	0.035
1st quantile	−0.295	0.886
Median	−0.286	0.894
3rd quantile	−0.261	0.903
Maximum	−0.226	0.934

and

Table 3. Proportion of statistical significance levels in each of the coefficients.

	Bicycle theft
Significance	Intercept	Repeat	RepOther	Popden	Retail	Fpopden	UnEmp	Univ
10% level	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000
5% level	0.000	0.000	0.000	0.000
1% level	1.000	1.000	1.000	1.000
	Shoplifting
Significance	Intercept	Repeat	RepOther	Dpopden	Retail	Fpopden	UnEmp	Univ
10% level	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000
5% level	0.000	0.000	0.000	0.000
1% level	1.000	1.000	1.000	1.000

summarize the estimated coefficients and their statistical significance. For bicycle theft, the variables Repeat, RepOther, and Popden are positively statistically significant at the 1% level across districts. The results suggest the strong tendency of near repeat victimization not only within the same crime type, but also across crime types. Given that a higher population density implies larger number of potential offenders and targets (see Section 5.1), the positive sign of Popden is intuitively reasonable. The selected coefficient type for Repeat was S&NVC, while those for RepOther and Popden were NVCs. Although NVCs have rarely been considered in spatial modeling, these results indicate the importance of considering NVCs. Fpopden, UnEmp, and Univ, whose coefficients are estimated to be constant, are statistically insignificant (Table 3). Among the group effects, $g_{q\left(i\right)}^{\left(t\right)}$ was selected. In short, near repeat victimization, population, and season are the dominant determinants, whereas the number of foreign people, economic status, and education are not. Later, we will look at the estimated coefficients in more detail.

For shoplifting, Repeat and RepOther are, again, positively significant, suggesting that not only shoplifting but also other non-burglary crimes in a quarter increases shoplifting in the next quarter. Dpopden is also positively significant. This suggests that shoplifting is increased in central areas where people concentrate during daytime. Retail, Fpopden, UnEnp, Univ, and the two group effects are insignificant. For the significant variables, the selected type of coefficients are as follows: S&NVC for Repeat; NVC for RepOther and Dpopden; constant for the others. In sum, repeat tendency and daytime population are shown to be significant determinants for shoplifting.

Figure 10 plots the estimated S&NVC on Repeat. The coefficients for bicycle theft decrease in the central area and south-west of the center. These areas are affluent areas. Bicycle theft is expected to be less repeated in these areas. Conversely, in the middle area, the coefficients increase along the Chuo line, which is a major railway (Figure 8) line. The repetitive tendency is especially strong near major stations on the line. Because subways and bus routes are densely spread in the affluent area (although not shown in the figure), bicycle users are relatively limited in this area. In addition, most residents in this area can afford to buy a bicycle. These properties might lower the number of potential offenders for bicycle theft. By contrast, in the middle area, railways and bus routes are relatively limited, and many people use bicycles for shopping or going to railway stations; bicycles are really needed in this area. In addition, the income level in this area is low relative to the affluent area. These properties might increase potential offenders.

As for shoplifting, the coefficients on Repeat increase in the central area where the Yamanote line runs (see Figure 8). This area includes central commercial districts near Shinjuku station and Tokyo station. A wide variety of products are sold in these districts. Besides, luxury stores are concentrated in these areas. Such properties might attract potential offenders and increase near repeat victimization.

Figure 11 displays the estimated NVCs. For both bicycle theft and shoplifting, the NVC on Repeat, which is in S&NVC, takes a high value if Repeat is high (see the left two panels of Figure 11). This means that repetitive tendency becomes strong if there were many crimes in the previous quarter. In contrast, a smaller RepOther has higher coefficients. This means that the number of bicycle thefts and shoplifting are correlated with other crimes in low-risk areas (i.e., low RepOther values). For bicycle theft, higher Popden and Dpopden have lower coefficients. This means that bicycle thefts increase as the population grows, but the rate of increase declines as population increases.

Finally,

Table 4. Estimated group effects.

	Bicycle Theft
	Estimate	Standard Error	t-Value
January–March	−0.104	0.058	−1.812
April–June	0.075	0.058	1.292
July–September	0.028	0.057	0.493
October–December	0.001	NA	NA

summarizes the estimated group effects on bicycle thefts by quarter. The results on shoplifting are not shown because both group effects were not selected. One interesting finding is that bicycle theft cases decrease in the first quarter (January to March), while they increase in the second quarter (April to June). This suggests that bicycle theft cases increase in the second quarter (April–June) probably because of the warm climate increasing bicycle users and/or changes in lifestyle and routine activities in the spring season (see [43]).

While spatial regression models have been used for crime modeling, their results strongly depend on model assumptions. For example, studies using GWR always discuss coefficient estimates smoothly varying over geographical space while spatial econometric studies discuss constant coefficient estimates in the presence of spatial dependence. In contrast, our results, which selected coefficients types, are less dependent on such modeling assumptions. Our results are more reliable. In addition, our results demonstrate the strong impact from NVCs, which have been overlooked in spatial statistics, rather than SVCs, suggests the dangerousness of using a spatial regression model without model selection. Our model selection approach will be useful to increase reliability of spatial regression analysis.

5.3. Application to Crime Prediction

Crime prediction for the near future is important for preventing crimes. Here, we apply our model (SNM_Sel) to predict the number of bicycle theft and shoplifting cases in minor municipal districts in the first quarter of 2019 by employing the model trained with the data from between 2017 and 2018. The accuracy is compared with the kernel density estimation (KDE), which is a popular crime prediction method in this area [44]. The ks package (https://cran.r-project.org/web/packages/ks/index.html) in R was used for the KDE model estimation and prediction. The plug-in selector of [45] is used to optimize the kernel bandwidth.

Figure 12 and Figure 13 compare the predicted and actual number of bicycle theft and shoplifting cases per area, respectively. For both cases, the KDE results are oversmoothed. The RMSE values are 5.77 and 6.93, respectively. Conversely, owing to the inclusion of spatial and non-spatial effects, our approach appropriately detects local hot spots. The resulting map pattern is quite similar to the observed numbers. The RMSEs are 3.53 for bicycle theft and 1.62 for shoplifting, which are considerably lower than those of KDE. This suggests that our approach is useful for crime prevention, such as the design of efficient patrol routes.

In short, we have demonstrated the usefulness of our approach, taking crime modeling as an example. While we considered data noise by introducing a noise term, we could not consider potential data bias attributable to under-recording/reporting, survey questions for crime surveys, whose minor change can dramatically change answers, and other reasons (e.g., see [46,47]). Consideration of bias in crime data will be an important next step for more reliable crime modeling. In addition, we considered the repetitiveness at a district level. However, based on studies on repeat victimization [48,49], such repetitiveness is more prominent in a finer spatial scale, such as individual properties, street segments, and 250-m grids. It is another important future task to consider more disaggregated spatial crime data rather than district-level data. Given that PredPol (https://www.predpol.com/) and many other crime prediction systems operate in a finer spatial scale, such as 250-m grids, extension of our crime model for spatially finer prediction will also be a remaining task toward social implementation.

6. Concluding Remarks

This study develops a model or coefficient type selection approach for spatial additive mixed models. The simulation experiments suggest that the present simple approach accurately selects the true model. Moreover, even though model selections usually increase computational time, our model selection dramatically reduces it. This property will be valuable for estimating a complex model that involves many candidate effects from large samples. The criminal analysis demonstrates that our approach provides intuitively reasonable results. Our approach highlights and quantifies numerous hidden effects behind geographic phenomena. These prominent features will be useful for a wide spectrum of analyses, such as crime analysis, ecological studies, and environmental studies.

However, there are many issues to be addressed. First, we need to incorporate spatio-temporal variations in regression coefficients or residuals. Spatially and temporally varying coefficients (STVC) have been studied by [50,51,52], among others. However, the discussion on selecting SVC, STVC, or other coefficients is still limited in spatial statistics. The spatio-temporal process, which comprises pure-spatial, pure-temporal, and spatio-temporal interaction processes, is much more complex than purely spatial variation. A well-manipulated selection of SVC, STVC, or other coefficient types will be very important in improving the accuracy and stability of spatio-temporal modeling. The consideration of dynamic temporal behavior is important therein [53,54]. Second, it is important to consider a larger number of coefficients, such as over 100. Sparse modeling approaches, such as LASSO [55] and smoothly cropped absolute deviation (SCAD) penalty [56], will be helpful in estimating SVC, NVC, or other varying coefficients while avoiding overfitting. Third, we need to extend our approach for non-Gaussian data modeling while maintaining the computational efficiency. For example, extension for Poisson or negative binomial regression will be an important future task to model crime count rather than crime density (e.g., see [1]), while regression with skewed distribution will be useful for modeling maximum temperature or other data relating extreme values (e.g., see [57]).

The developed model selection procedure was implemented in an R package spmoran (https://cran.r-project.org/web/packages/spmoran/index.html).