A New Algorithm for Large-Scale Geographically Weighted Regression with K-Nearest Neighbors

Abstract

Geographically weighted regression (GWR) is a classical method for estimating nonstationary relationships. Notwithstanding the great potential of the model for processing geographic data, its large-scale application still faces the challenge of high computational costs. To solve this problem, we proposed a computationally efficient GWR method, called K-Nearest Neighbors Geographically weighted regression (KNN-GWR). First, it utilizes a k-dimensional tree (KD tree) strategy to improve the speed of finding observations around the regression points, and, to optimize the memory complexity, the submatrices of neighbors are extracted from the matrix of the sample dataset. Next, the optimal bandwidth is found by referring to the spatial clustering relationship explained by K-means. Finally, the performance and accuracy of the proposed KNN-GWR method was evaluated using a simulated dataset and a Chinese house price dataset. The results demonstrated that the KNN-GWR method achieved computational efficiency thousands of times faster than existing GWR algorithms, while ensuring accuracy and significantly improving memory optimization. To the best of our knowledge, this method was able to run hundreds of thousands or millions of data on a standard computer, which can inform improvement in the efficiency of local regression models.

1. Introduction

Owing to the influence of geographical location, changes in the relationship or structure between variables is an important issue in spatial data analysis. The geographically weighted regression (GWR) model, originally developed by Fotheringham and Brunsdon [1,2], is a common tool for exploring spatial non-smoothness. Currently, GWR is widely used in a variety of fields, such as geology [3,4,5], ecology [6,7], house price modeling [8,9,10,11], epidemiology [12,13,14], and environmental science [7,15]. The popular GWR packages are Spgwr [16], MGWR (PySAL) [17], GWmodel [18], and FastGWR [19]. With the expansion of demand for geographic information technology and resources in various industries [20,21], geographic data with high spatial and temporal resolution have seen explosive growth while promoting innovation in GIS methods [22,23,24,25,26]. Many studies have explicitly reported the computational limitations of GWR when oriented to large-scale geographic data [19,27] in order to fully utilize the geographical information.

Each regression point in GWR is regressed individually based on the distance matrix, and the optimal bandwidth selection is performed before that, which makes it computationally intensive and requires a large amount of memory usage. However, the application of GWR to extract information from large-scale geographic data is difficult. Harris estimated that it would take two or more weeks to complete the experiment using the Spgwr package for a dataset of one hundred thousand points (and five predictor variables) [28]. Li pointed out that the maximum number of records that can be handled by the current open-source GWR software is approximately fifteen thousand observations on a standard desktop [19]. In Yu’s experiment [29], random sampling of a house price dataset was forced to reduce the calculation cost owing to the computational demand of building a GWR model for the 68,906 house price dataset; therefore, 3437 data were selected for the GWR calculation. Although this method addresses the problem of computational volume, it can lead to incomplete exploitation of data information. Feuillet provided a method to deal with a large sample dataset by building GWR sub-models [30]. Wang proposed an improved method based on a computational unified device architecture (CUDA) parallel architecture that can handle GWR corrections for millions of data points [31]. Tasyurek reverse-nearest-neighbor geographically weighted regression (RNN-GWR) only calculates updated data points when dealing with frequently updated datasets, which can result in a higher computational efficiency with guaranteed calculation results [32]. Although many efforts have been made, there is still a lack of an effective GWR algorithm that can analyze large numbers of geographic data in a limited timeframe. Currently, data sets with millions of observations are becoming increasingly common. During the analysis of the GWR regression process, we found that for each regression point, a certain distance (bandwidth) radius range of observation points is analyzed, and therefore observations outside of this range are not considered in the calculation. Additionally, the current optimal bandwidth search range is typically global, and reducing unnecessary calculations can alleviate the computational burden of GWR. Thus, implementing new improvements to the GWR algorithm are necessary to address computational bottlenecks, enable its application in extremely large datasets, and fully exploit data information.

Currently, the nearest-neighbor indexing method demonstrates better performance in the computational optimization of some models, with the k-dimensional tree (KD tree) being the most commonly used. Meenakshi suggested a new index structure KD tree with a linked list (k-dLst tree) for retrieving spatial records with duplicate keys [33]. To improve the computationally expensive state of performing repeated distance evaluations in the search space, a special tree-based structure (called a KD tree) was used to speed up the nearest-neighbor search [34]. Chi-Ren Shyu developed a web server (named ProteinDBS) for the life science community to search for similar protein tertiary structures in real time, which returned search results in hierarchical order from a database of over 46,000 chains in a few seconds and showed considerable accuracy [35]. Böhm performed a range search by indexing K-Nearest-Neighbor join queries [36]. Muja proposed the Fast Library for Approximate Nearest Neighbors (FLANN) library, which reduced the time for nearest-neighbor search by an order of magnitude [37,38]. The FLANN library has been applied in many studies to improve computational efficiency [37,39]. The advantages of nearest-neighbor indexing for optimal computation have been demonstrated in these studies.

When dealing with large amounts of data, clustering methods can help to better understand the distribution patterns and structure of the data, thus revealing hidden information in the data [40,41]. K-means, a widely-used clustering algorithm, partitions a dataset into K clusters and assigns each data point to the nearest cluster. This method has been extensively implemented to analyze data in various domains [40,42]. Macqueen argued that spatial clustering uses spatial location and relationships as feature terms to discover spatial clustering relationships [43,44]. Li used K-means clustering to analyze the impact of building environmental factors on the variation of rail ridership in the study area and proposed differentiated planning guidance for different regions [45]. Hernández employed the K-means clustering algorithm to identify distinct clusters of tourism types within large geographic areas [46]. Deng applied a combination approach of geographically weighted regression and K-means clustering to partition the study area into distinct regions and devise regional policies to mitigate PM2.5 concentrations [47]. The aforementioned studies all indicate that K-means clustering has significant advantages in discovering the spatial distribution of geographic data.

The main objective of this study is to develop a method that can quickly calibrate GWR to overcome the challenges of processing large-scale geographic data. This paper contributes to the previous literature as follows. (1) Using a K-D tree to accelerate the speed of searching for observation points around the regression point. In the GWR model, searching for neighboring points usually requires traversing the entire dataset, leading to significant time consumption as the dataset size increases. Therefore, this paper attempts to incorporate the KD tree into the GWR model to take advantage of its fast search capabilities to quickly identification of surrounding observation points in each local regression process. (2) Transforming the large matrix into a small matrix in the regression process. When using the Bi-square kernel function to calculate the weight matrix, the full sample matrix can cause significant memory and time consumption. Therefore, the matrices of independent variables, dependent variables, and weight matrices involved in the regression are transformed into corresponding small matrices according to the bandwidth. (3) Optimizing the search range of optimal bandwidth. The optimal bandwidth in GWR is extracted in the process of global traversal, which is time-consuming. This may be caused by considering only the local scale of operation of the model represented by the bandwidth and ignoring the spatial relationships implied by the bandwidth at local spatial locations. Therefore, this paper attempts to use K-means to cluster the geographic location data and refer to the obtained clustering results to limit the search range of the optimal bandwidth. Finally, this paper incorporates the KD tree index and K-means clustering into the GWR model, restructuring the matrices of independent variables, dependent variables and weight matrices involved in the calculation, and proposes a new model called KNN-GWR. The potential of using KNN-GWR models for geographically weighted regression on large-scale geographic data was explored in this paper, using simulated data and a dataset of house prices in selected regions of China.

The remainder of this paper is organized as follows. In Section 2, we introduce the proposed algorithm. In Section 3, we compare the performance of KNN-GWR with other GWRs using simulated datasets and house price datasets. In Section 4, conclusions and discussions are presented.

2. Method

2.1. Geographically Weighted Regression

GWR is used as a local fitting technique, and its regression coefficient varies with the geographical location. The mathematical expression is as follows:

$$y_i={\beta }_0\left(u_i,v_i\right)+{\displaystyle \sum _{j=1}^p}{\beta }_j\left(u_i,v_i\right)x_{ij}+{\epsilon }_i$$

(1)

where $\left(u_i,v_i\right)$ represent the coordinates of the $i\mathrm{t}\mathrm{h}$ point in space, ${\beta }_0\left(u_i,v_i\right)$ denotes the intercept value, and ${\beta }_j\left(u_i,v_i\right)$ is the spatial variation coefficient of the $i\mathrm{t}\mathrm{h}$ independent variable. $i ϵ \left\{\mathrm{1,2},\cdots ,n\right\}, jϵ\left\{\mathrm{1,2},\cdots ,p\right\}$, $y_i$ is the response variable at location $i$, $x_{ij}$ is the $j\mathrm{t}\mathrm{h}$ predictor variable, and ${\epsilon }_i$ is the error term. $n$ is the size of the sample dataset and $p$ is the number of independent variables. The GWR calibration in matrix form is given by:

$${\beta }_i=(X^T{W}_i X{)}^{-1}{X}^T{W}_{i}Y$$

(2)

where $X$ is a $n\times (p+1$) matrix of the independent variables, $Y$ is a $n\times 1$ matrix of the response variable, and the matrix of $X$ and $Y$ can be calculated, respectively, by:

$$X=\left[\begin{array}{ccccc}1& x_{11}& x_{12}& \cdots & x_{1p}\\ 1& x_{21}& x_{22}& \cdots & x_{2p}\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ 1& x_{n1}& x_{n2}& \cdots & x_{np}\end{array}\right]$$

(3)

And

$$Y=\left[\begin{array}{c}\begin{array}{c}{y}_1\\ y_2\\ ⋮\end{array}\\ y_n\end{array}\right]$$

(4)

where $W_i$ is a space weight matrix and is expressed as:

$$W_i=\left[\begin{array}{cccc}{w}_{i1}& 0& \cdots & 0\\ 0& w_{i2}& \cdots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \cdots & w_{in}\end{array}\right]$$

(5)

where $W_i$ is an n × n diagonal matrix. It is calculated using a specified kernel function and bandwidth ($bw$). For example, the adaptive bi-square weighting function is as follows:

$$w_{ij}=\left\{\begin{array}{l}{\left[1-{\left(\frac{d_{ij}}{bw}\right)}^2\right]}^2 d_{ij}\le bw\\ 0 d_{ij}>bw\end{array}\right.$$

(6)

where $d_{ij}$ represents the spatial distance between points i and j. $bw$ represents the adaptive bandwidth, and means that the number of neighbors around regression point i is constant, but the distance is variable.

Where the coefficients ${\beta }_i$ of the model can be expressed in matrix form:

$${\beta }_i=\left[\begin{array}{c}{\beta }_{i0}\\ {\beta }_{i1}\\ ⋮\\ {\beta }_{ip}\end{array}\right]$$

(7)

To calibrate the GWR model, a cross-validation (CV) approach is typically used to iterate the bandwidth. In other words, the optimal bandwidth is selected by minimizing the following CV scores:

$$CV={\displaystyle \sum _{i=1}^n}{\left[y_i-{\widehat{y}}_{\ne i}\left(bw\right)\right]}^2, bw\in \left(1,n\right)$$

(8)

In general, the traversal of $bw$ ranges from one to n. Because each regression point in the GWR does not participate in its own local regression calculation, $bw\ne 1$. When $bw=n$, the operation scale of the GWR model is global, so $bw\ne n$.

2.2. Geographically Weighted Regression with K-Nearest Neighbors

To speed up the computation and optimize the storage of the weight matrix in the GWR, the KD tree was adopted to find the observation points around the regression point. In this manner, hundreds of thousands or even millions of calculations are filtered. Thus, the large-scale regression problem can be transformed into an acceptable calc value. This method allows GWR to run hundreds of thousands or even millions of data records on an ordinary computer. The flow of the algorithm is illustrated in Figure 1.

The algorithm can be divided into four parts. Part a: Optimal bandwidth selection reference K-means. K-means clustering is performed on the incoming data to explain the spatial relationships in the dataset. Part b: KD tree construction and search. Establishing the KD tree for the incoming data and searching the observation points around the regression point according to the GWR bandwidth optimization rule based on the results obtained in Part a. Part c: Restructured ${\tilde{W}}_i$, ${\tilde{X}}_i$, ${\tilde{Y}}_i$. The weight matrix ${\tilde{W}}_i$, the independent variable matrix ${\tilde{X}}_i$ and the dependent variable matrix ${\tilde{Y}}_i$ of the GWR regression process are reconstructed according to the requirements of each regression point based on the findings in Part b. The reconstructed results are ${\tilde{X}}_i$, ${\tilde{Y}}_i$, and ${\tilde{W}}_i$, respectively. Part d: KNN-GWR in calibration. Running the model based on the results of Part c and outputting the results of the optimal bandwidth run to complete the model diagnosis.

2.2.1. Optimal Bandwidth Selection Reference K-Means

We counted the optimal bandwidth in some studies, which was usually within a smaller range [19,48,49]; the specific range of values depended on the research field, specific dataset, the researcher’s experience and judgment, and in general was usually ≤200. However, GWR usually searches for the optimal bandwidth on a global scale, which is less necessary in local regression. Murakami pointed out the use of rank reduction and pre-compression to eliminate the effect of data size in the regression of large datasets [49]. Geographically adjacent observation points were considered to have highly similar characteristics; therefore, in the KNN-GWR, they should be grouped into the same category or group as much as possible when performing geographic analysis. In this study, K-means clustering was utilized to analyze the latitude and longitude coordinates of geographic data, thereby facilitating a more comprehensive examination of the distribution patterns and spatial relationships of the data. For K-means clustering of big datasets, a set of values is defined and referred to as KC values, where the optimal number of clusters will be generated from this set of KC values, with $KC\in \left\{k_{min}\right.,k_{sec},k_{mean},k_{four},\left.k_{max}\right\}$. Using the heuristic approach, the square root value of half the data size is $k_{mean}$ [50,51,52], where $k_{mean}$ is obtained by the following formula:

$$k_{mean}=\sqrt{\frac{n}{2}}$$

(9)

where n denotes the number of observation points.$k_{sec}$ and $k_{four}$ are the results of moving $k_{mean}$ up or down by one number. $k_{min}$ and $k_{max}$ are the results of moving $k_{mean}$ up or down by two different numbers.

The sum of squared errors was used to calculate the clustering error of the sample to select the optimal KC value [53,54].

The formula is as follows.

$$SSE={\sum }_{g=1}^k{\sum }_{sϵc_g}\left|s-m_i\right|$$

(10)

where $C_g$ is the g-th cluster, s is a sample point in $C_g$, and $m_i$ is the center of $C_g$.

The pseudocode of the Determine Optimal Bandwidth Range algorithm is given in Algorithm 1. The clusters obtained by the K-means clustering method are labeled as $A_1,A_2,\dots ,A_{k_{optimal}}$, respectively. The number of observation points in each cluster was counted using the Num() function to represent the cluster size. For example, Num($A_1$) represents the number of observation points in Cluster $A_1$. The Num() function helps find the largest and smallest clusters, denoted as $A_{max}$ and $A_{min}$. $A_{max}$ and $A_{min}$ represent the upper and lower limits of the spatial distribution of all observation points based on spatial location features. The cross-validation (CV) method was chosen to determine the optimal bandwidth. The formula is as follows.

$$CV={\sum }_{i=1}^n{\left[y_i-{\widehat{y}}_{\ne i}\left(bw\right)\right]}^2, bw\in [Num(A_{min}), Num(A_{max})]$$

(11)

Algorithm 1: Optimal Bandwidth Selection Reference K-means

According to the number of data n, determine KC $\in \left\{k_{min}\right., k_{sec},k_{mean},k_{four}, \left.k_{max}\right\}$ Perform K-means clustering analysis based on KC Find the optimal KC value $k_{optimal}$ depending on the $SSE$ Obtain clustering results $A_1,A_2,\dots ,A_{k_{optimal}}$ Defined $\left(Num\left(A_{min}\right),Num\left(A_{max}\right)\right)$ = ${Num(A}_1,A_2,\dots ,A_{k_{optimal}})$ Function ${Num(A}_1,A_2,\dots ,A_{k_{optimal}})$ Count the number of data points in each cluster of $A_1,A_2,\dots ,A_{k_{optimal}}$, denoted as $Num\left(A_1\right),Num\left(A_2\right),\cdots ,Num\left(A_{k_{optimal}}\right)$ Compare the size of $Num\left(A_1\right),Num\left(A_2\right),\cdots ,Num\left(A_{k_{optimal}}\right)$ and return the maximum value $\left(Num\left(A_{max}\right)\right)$ and minimum value $\left(Num\left(A_{min}\right)\right)$ Output $Num\left(A_{min}\right)$ and $Num\left(A_{max}\right)$ to provide a reference for the optimal bandwidth

2.2.2. KD Tree Construction and Search

Using the bi-square kernel function in GWR, the surrounding observations need to be found to participate in the calculation during the local regression of regression point i. For two-dimensional geographic data, traditional linear search methods involve brute-force computation of the distances between all pairs of points in the dataset. Assuming a sample size of n, the time complexity of this method is $O\left(n^2\right)$. Since there is no adjacency between point i and point (5, 6, 7, 8, 9) in the limit range of R, as shown in Figure 2. Therefore, the distance calculation between point i and point (5, 6, 7, 8, 9) is not necessary. Further, by reducing the number of visited nodes and the corresponding distance calculation, the speed of finding surrounding points can be improved. In this study, a KD tree was created for all regression points based on spatial coordinates in order to effectively filter invalid calculations when searching for observations around regression points.

A K-dimensional tree (KD tree) built based on distance is an extended version of a binary tree for speeding up the dataset search. If the points include the $\left(u,v\right)$ dimension, they are first sorted according to the value of a dimension, such as u. Then, the middle point is selected as the split (parent) node, and all points are divided into two parts according to the coordinates on the u axis. The u value of the left subspace is smaller than that of the parent node, and the u value of the right subspace must be greater than or equal to the u value of the parent node. This process continues repeatedly until both subregions have no sample points (the terminating node is a leaf node), and finally, the region division of the KD tree is formed. Using this approach, sample points are saved on the appropriate nodes. In this case, the time complexity for performing a perimeter point lookup is $O\left(n\log \left(n\right)\right)$. After the tree structure is established, this algorithm starts from the root node and recursively searches the left and right child nodes to find all points within the radial range of the regression point. The lookup of the observation points around the regression point was completed quickly after the KD tree was constructed, which accelerated the process of reconstructing the matrix in the KNN-GWR method.

2.2.3. Geographically Weighted Regression with K-Nearest Neighbors and Model Evaluation

Press stated that the desirable characteristics of an algorithm are to be fast, computationally inexpensive, and to use as little memory as possible with high computational accuracy [55]. When the bi-square kernel function is selected as the kernel function for GWR, only the observation points within the bandwidth participate in the regression calculation. Here, the zero rows and columns of the weight matrix were removed to minimize the computational cost. In the local regression calculation of regression point i, the construction of the KD tree speeds up the search for the points surrounding regression point i. Considering the above, the KNN-GWR algorithm was set up in this study. The expression of the KNN-GWR algorithm is expressed as follows:

$$y_i={\tilde{\beta }}_0\left(u_i,v_i\right)+{\displaystyle \sum _{j=1}^p}{\tilde{\beta }}_j\left(u_i,v_i\right)x_{ij}+{\epsilon }_i$$

(12)

The formula for calculating ${\tilde{\beta }}_i$ is shown below.

$${\tilde{\beta }}_i={\left({\tilde{X}}_i^T{\tilde{W}}_i{\tilde{X}}_i\right)}^{-1}{\tilde{X}}_i^T{\tilde{W}}_i{\tilde{Y}}_i$$

(13)

The corresponding ${\tilde{X}}_i$ and ${\tilde{Y}}_i$ matrices are determined based on the bandwidth of the regression point and KD tree data structure. The matrices of ${\tilde{X}}_i$ and ${\tilde{Y}}_i$ can be calculated by

$${\tilde{X}}_i=\left[\begin{array}{cccc}1& x_{d_{1}1}& \cdots & x_{d_{1}p}\\ 1& x_{d_{2}1}& \cdots & x_{d_{2}p}\\ ⋮& ⋮& \ddots & ⋮\\ 1& x_{d_{K}1}& \cdots & x_{d_{K}p}\end{array}\right]$$

(14)

And

$${\tilde{Y}}_i=\left[\begin{array}{c}\begin{array}{c}{y}_1\\ y_2\\ ⋮\end{array}\\ y_K\end{array}\right]$$

(15)

where ${\tilde{X}}_i$ is composed of independent variables from the observation points within the bandwidth around the regression point i (${\tilde{X}}_i$ is a $K\times (p+1)$ matrix). ${\tilde{Y}}_i$ is composed of the dependent variable of the observation points within the bandwidth around the regression point i (${\tilde{Y}}_i$ is a $K\times 1$ matrix). K is a constant equal to the number of observation points around the regression point. [1, 2, …, K] is the point number in the bandwidth range after sorting by distance. ${\tilde{W}}_i$ is a space weight matrix and is expressed as:

$${\tilde{W}}_i=\left[\begin{array}{cccc}{w}_{i1}& 0& \cdots & 0\\ 0& w_{i2}& \cdots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \cdots & w_{iK}\end{array}\right]$$

(16)

The equation for the weight matrix $w_{ij}$ is expressed as

$$w_{ij}={\left[1-{\left(\frac{d_{ij}}{d_{iK}}\right)}^2\right]}^2$$

(17)

where $d_{ij}$ represents the spatial distance between points i and j and $d_{iK}$ represents the spatial distance between points i and K.

The hat matrix $\tilde{S}$ is the projection matrix from the observed $y$ to the fitted $\widehat{y}$, where each row ${\tilde{S}}_i$ of the hat matrix is:

$${\tilde{S}}_i=X_i{\left({\tilde{X}}_i^T{\tilde{W}}_i{\tilde{X}}_i\right)}^{-1}{\tilde{X}}_i^T{\tilde{W}}_i$$

(18)

where

$${\widehat{y}}_i\left(bw\right)={\tilde{S}}_i{\tilde{Y}}_i$$

(19)

where ${\widehat{y}}_i\left(bw\right)$ is the fitted value of $y_i$ using a bandwidth. The residual vector is then

$$e=y_i-{\widehat{y}}_i\left(bw\right)$$

(20)

and the residual sum of squares is

$$RSS={\sum }_{i=1}^n{e}^2={\sum }_{i=1}^n{\left(y_i-{\widehat{y}}_i\right)}^2$$

(21)

The estimation accuracy of the GWR model was evaluated by comparing the fitted dependent variable values with the dependent variable values of the sample data, as $R^2$.

$$R^2=1-\frac{{\sum }_{i=1}^n{\left(y_i-{\widehat{y}}_i\right)}^2}{{\sum }_{i=1}^n{\left(y_i-{\overline{y}}_i\right)}^2}$$

(22)

The calculated runtime ratio of the KNN-GWR algorithm to other GWR packages is described by the runtime increase rate (RIR).

$$RIR=\frac{{Runtime}_{others}}{{Runtime}_{KNN-GWR}}$$

(23)

The pseudo-code of KNN-GWR algorithm is given in Algorithm 2.

Algorithm 2: KNN-GWR Algorithm

I  Optimizing bandwidth search by minimizing CV scores

1.  Given the initial data ${bw}_1$, $X$, $Y$, $U$ and $V$

2.  Start the loop $bw={bw}_1,{bw}_2,\cdots ,{bw}_o,\cdots ,{bw}_t$:

3.     Build KD tree spatial index

4.     Starting loop with $i=\mathrm{1,2},\cdots ,n$:

5.        Take the data from the result of 3 steps:

6.        Reconfiguration ${\tilde{W}}_i$, ${\tilde{X}}_i$, and ${\tilde{Y}}_i$

7.        Calculate ${\tilde{\beta }}_i$, ${\widehat{y}}_i$

8.     End of loop

9.     Calculate $\mathrm{C}\mathrm{V}\mathrm{S}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}\mathrm{s}$

10.  End of loop

II   Optimal bandwidth selection based on minimum CV criterion

11.  Start the loop $i=\mathrm{1,2},\cdots ,n$:

12.     Reconfiguration ${\tilde{W}}_i$, ${\tilde{X}}_i$, and ${\tilde{Y}}_i$

13.     Calculate ${\tilde{\beta }}_i$, ${\widehat{y}}_i$

14.  End of loop

15.  Return $\tilde{\beta }$, $\widehat{y}$

16.  Spatial analysis and model diagnosis using $\tilde{\beta }$, $\widehat{y}$

2.2.4. Computational Complexity of GWR and KNN-GWR in Calibration

Time Complexity

In the GWR regression calculation, the time complexity of $X^T{W}_{i}X$ matrix is $O\left({\left(p+1\right)}^{2}n\right)$ and the big $O$ is an asymptotic notation used to describe the upper bound of an algorithm’s efficiency. Calculating its inverse, $({X^T{W}_{i}X)}^{-1}$, requires $O\left({\left(p+1\right)}^3\right)$ [19,30,56] ($p$ is the number of independent variables, (p + 1) ≤ 10 in common). Therefore, the time complexity of computing ${\beta }_i$ is $O\left({\left(p+1\right)}^{2}n\right)$ [19,31]. The calculation of ${\beta }_i$ is repeated at each regression point location n times; therefore, the total time complexity of the GWR calibration for a known bandwidth is $O\left({\left(p+1\right)}^2{n}^2\right)$. If the golden partition is used to find the optimal bandwidth, the time complexity is close to $O\left(logn\right)$, and the total time complexity of the bandwidth selection and model calibration is $O\left({\left(p+1\right)}^2{n}^{2}logn\right)$ [19]. The time complexity of KNN-GWR is explained in the same manner as above. In the KNN-GWR model, ${\tilde{X}}_i$ is a $K\times (p+1)$ matrix and ${\tilde{W}}_i$ is a $K\times K$ matrix. The time complexity of the ${\tilde{X}}_i^T{\tilde{W}}_i$ matrix is $O\left(\left(p+1\right)K\right)$, and the time complexity of the ${\tilde{X}}_i^T{\tilde{W}}_i{\tilde{X}}_i$ matrix is $O\left({\left(p+1\right)}^{2}K\right)$. Calculating its inverse, ${\left({\tilde{X}}_i^T{\tilde{W}}_i{\tilde{X}}_i\right)}^{-1}$, takes $O\left({\left(p+1\right)}^3\right)$, which can be neglected. ${\tilde{\beta }}_i$ is computed at each regression point n times; therefore, the total time complexity of the KNN-GWR calibration for a known bandwidth is $O\left({\left(p+1\right)}^{2}n\right)$. If the golden partition is used to find the optimal bandwidth, the time complexity is close to $O\left(\log (Num\left(A_{max}\right)-Num\left(A_{min}\right))\right)$, and the total time complexity of the bandwidth selection and model calibration is $O\left({\left(p+1\right)}^{2}Knlog\right(Num\left(A_{max}\right)-Num\left(A_{min}\right)\left)\right)$. Therefore, in theory, the time complexity of KNN-GWR is much lower than that of GWR. The time complexity of each algorithm is listed in

Table 1. Comparison of time complexity among KNN-GWR, FastGWR, MGWR (PySAL), GWmodel, and Spgwr.

Name of Algorithm	Time Complexity
KNN-GWR	$O\left({(p+1)}^2\mathrm{K}\mathrm{n}\mathrm{l}\mathrm{o}\mathrm{g}\right(Num(A\_max)-Num(A-min)\left)\right)$
FastGWR	$O\left({\left(p+1\right)}^2{n}^{2}logn\right)$ [19]
MGWR(PySAL), GWmodel and Spgwr	$O\left({\left(p+1\right)}^2{n}^{2}logn\right)$ [19,31]

Note: K is the data of the observation points around the regression point in the bandwidth, n is the number of observation points, and p is the number of independent variables.

.

Memory Complexity

The KNN-GWR algorithm improves the matrix and dataset storage model of the GWR operation by adopting distance sorting and a data structure. The weight matrix $W_i$ of the GWR method is stored as an $n\times n$ diagonal matrix, which requires $n^2$ storage spaces during the operation, and the memory complexity is $O\left(n^2\right)$. In the process of GWR regression calculation, $X^T{W}_i X$, $(X^T{W}_i X{)}^{-1}$, and $(X^T{W}_i X{)}^{-1}{X}^T{W}_i$ inherit the memory complexity of the weight matrix $W_i$. The KNN-GWR algorithm avoids this situation during the calculation. The weight matrix ${\tilde{W}}_i$ is calculated by selecting only the points within the GWR bandwidth (${\tilde{W}}_i$ is a $K\times K$ matrix), and the ${\tilde{X}}_i$ and ${\tilde{Y}}_i$ matrices are reconstructed accordingly (${\tilde{X}}_i$ is a $K\times (p+1)$ matrix; ${\tilde{Y}}_i$ is a $K\times 1$ matrix). The memory complexity of the KNN-GWR algorithm is affected by K, and the memory complexity is $O\left(K^2\right)$. (typically, $K<<n$). The memory complexity of each algorithm is presented in

Table 2. Comparison of memory complexity between KNN-GWR and other GWR packages.

Number of Data Points	KNN-GWR $\mathit{O}\left({\mathit{K}}^{\mathbf{2}}\right)$	FastGWR $\mathit{O}\left(\mathit{n}\left(\mathit{p}+\mathbf{1}\right)\right)$	MGWR(PySAL), GWmodel, and Spgwr $\mathit{O}\left({\mathit{n}}^{\mathbf{2}}\right)$
1000	33.8 KB	19.5 KB	3.8 MB
10,000	33.8 KB	195.3 KB	380 MB
100,000	33.8 KB	1.9 MB	37.25 GB
1,000,000	33.8 KB	19.1 MB	3.8 TB
10,000,000	33.8 KB	190.7 MB	364 TB

Note: The data are 32–bit floating points, where K is numerically equal to 93, n is the number of observation points, and p is the number of independent variables.

.

3. Experiment

3.1. Data Source

To explore the actual performance of KNN-GWR, two datasets were used in the experiment—the simulated dataset [31] and the house price dataset in China from the Anjuke.com website (accessed on 25 March 2022). A simulated dataset was used to verify the viability of the model. The Chinese house price dataset was used to test the scalability of the KNN-GWR model.

3.1.1. Simulated Dataset

The test data were distributed in a square area with a side length of $l$ units and the data sample points were evenly distributed in this area. The sample size of each line was m, the total number of samples was $n=m\times m$, and the distance between adjacent sample points was $\mathsf{\Delta }l=l/(m-1)$. Considering the lower left corner of the square area as the origin of the coordinate system, the calculation formula for the sample point position is as follows:

$$\left(u_i,v_i\right)=\left(∆l\times mod\left(i-1,m\right),∆l\times int\left(i-1,m\right)\right)$$

(24)

where $mod \left(a, b\right)$ and $int \left(a, b\right)$ are the remainder and integer part of $a$ divided by $b$ [57]. The experimental data samples were generated using a predefined GWR model with the following equation:

$$y_i={\beta }_0\left(u_i,v_i\right)+{\beta }_1\left(u_i,v_i\right)x_{i1}+{\beta }_2\left(u_i,v_i\right)x_{i2}+{\beta }_3\left(u_i,v_i\right)x_{i3}+{\beta }_4\left(u_i,v_i\right)x_{i4}+{\epsilon }_i$$

(25)

To standardize the regression coefficient $\beta$, all regression coefficients $\beta$ were limited to the $\left(0, {\beta }_{max}\right)$ interval (${\beta }_{max}$ was a fixed constant) [31]. The regression coefficient beta of the model follows the following five functions.

$$\left\{\begin{array}{c}{\beta }_0\left(u_i,v_i\right)=\frac{2{\beta }_{max}}{l^2}(\frac{l^2}{2}-{\left(l-u_i\right)}^2-{(l-v_i)}^2)\\ {\beta }_1\left(u_i,v_i\right)=\frac{{\beta }_{max}}{2}({\left(sin\frac{u_i\pi }{l}\right)}^2+{\left(sin\frac{v_i\pi }{l}\right)}^2)\\ {\beta }_2\left(u_i,v_i\right)=\frac{{\beta }_{max}}{2}(2-({\left(tan\right(\frac{u_i\pi }{2l}-\frac{\pi }{4}\left)\right)}^2+{\left(tan\right(\frac{v_i\pi }{2l}-\frac{\pi }{4}\left)\right)}^2\left)\right)\\ {\beta }_3\left(u_i,v_i\right)={\beta }_{max}{e}^{-\frac{1}{2l}({(\frac{l}{2}-u_i)}^2+{(\frac{l}{2}-v_i)}^2)}\\ {\beta }_4\left(u_i,v_i\right)=\frac{16{\beta }_{max}}{l^4}(\frac{l^2}{4}-{(\frac{l}{2}-u_i)}^2)(\frac{l^2}{4}-{(\frac{l}{2}-v_i)}^2)\end{array}\right.$$

(26)

This paper prepared 8 sets of simulated datasets with different numbers: 1000, 5000, 10,000, 15,000, 20,000, 50,000, 100,000, and 1,000,000.

3.1.2. House Price Dataset in China

The study area ranged from 108°21′ to 122°42′ E and 23°33′ to 42°40′ N. Fourteen provinces or municipalities were mainly involved (Anhui, Jiangxi, Henan, Hubei, Hunan, Shanxi, Hebei, Jiangsu, Zhejiang, Fujian, Beijing, Tianjin, and Shanghai). The total area was approximately 17,454,000 square kilometers comprising two world-class urban agglomerations, Beijing–Tianjin–Hebei and Yangtze River Delta, as shown in Figure 3.

The house price dataset used in this study was collected from www.anjuke.com (accessed on 25 March 2022). The experimental dataset was selected from residential land price statistics with geographic location, including the number of bedrooms (NBeds), bathrooms (NBaths), floor area (Area), and age of the house (Age). Detailed information on the experimental dataset is shown in Figure 4. In this study, 123,691 research data were sequentially divided into 7 different datasets by random sampling: 1000, 5000, 10,000, 15,000, 20,000, 50,000, and 100,000. Seven sets of datasets were applied to MGWR (PySAL), GWmodel, Spgwr, and KNN-GWR, which satisfy the following expression.

$$y_i={\beta }_0+{\beta }_1{Area}_i+{\beta }_2{NBaths}_i+{\beta }_3{NBeds}_i+{\beta }_4{Age}_i+{\epsilon }_i$$

(27)

3.2. Testing Specifications and Environment

The differences in runtime between the KNN-GWR and other GWR packages (MGWR(PySAL), GWmodel, and Spgwr) were compared. Details of the equipment used in the experiments are listed in

Table 3. Device parameters for running study GWR models.

	KNN-GWR, MGWR(PySAL), GWmodel, Spgwr
CPU	Intel(R) Core(TM) i7-1065G7 CPU @ 1.30 GHz, 8 cores
Memory	16,384 MB RAM

.

3.3. Results

3.3.1. Case One: Simulated Dataset

In Figure 5, the coefficients $\beta \left({\beta }_0,{\beta }_1,{\beta }_2,{\beta }_3,{\beta }_4\right)$ and the respective dependent variable values $Y$ for the simulated data are presented, along with the coefficients $\tilde{\beta }\left({\tilde{\beta }}_0,{\tilde{\beta }}_1,{\tilde{\beta }}_2,{\tilde{\beta }}_3,{\tilde{\beta }}_4\right)$ and predicted values of the dependent variable $Predicted Y$ obtained by applying the KNN-GWR method. As shown in Figure 5, the five coefficients β selected by the model are closely related to the locations of the sample points and exhibit spatial non-stationarity. The spatial distribution of the coefficients $\tilde{\beta }\left({\tilde{\beta }}_0,{\tilde{\beta }}_1,{\tilde{\beta }}_2,{\tilde{\beta }}_3,{\tilde{\beta }}_4\right)$ obtained by the KNN-GWR method is quite similar to the distribution of the simulated data, indicating that it addresses the spatial non-stationarity.

Due to the transformation of a large matrix into a small matrix, the regression efficiency of KNN-GWR is significantly improved compared with that of GWR. As shown in Figure 6 and

Table 4. Information on the runtime and $R^2$ of four software packages (KNN-GWR, MGWR(PySAL), GWmodel, and Spgwr) in simulated dataset experiments.

Number of Data Points	KNN-GWR		MGWR(PySAL)		GWmodel		Spgwr
	Runtimes (s)	${\mathit{R}}^{\mathbf{2}}$	Runtimes (s)	${\mathit{R}}^{\mathbf{2}}$	Runtimes (s)	${\mathit{R}}^{\mathbf{2}}$	Runtimes (s)	${\mathit{R}}^{\mathbf{2}}$
1000	0.34	0.995	3.71	0.995	1.33	0.994	21.65	0.994
5000	3.11	0.996	43.51	0.995	41.51	0.997	1514.71	0.997
10,000	3.59	0.996	178.52	0.994	211.41	0.998	11,574.78	0.997
15,000	6.56	0.993	342.51	0.996	598.91	0.992	n/a	n/a
20,000	8.52	0.994	758.81	0.996	1234.06	0.993	n/a	n/a
50,000	31.77	0.997	6327.91	0.996	8283.33	0.995	n/a	n/a
100,000	83.06	0.998	n/a	n/a	n/a	n/a	n/a	n/a
10,000,000	865.13	0.999	n/a	n/a	n/a	n/a	n/a	n/a

, the average running times of KNN-GWR at 1000, 5000, and 50,000 points are 0.34, 3.59, and 31.77 s, respectively. However, the average running time of MGWR(PySAL) increased from 3.71 s at 1000 points to 178.52 s at 10,000 points, and then reached 6327.91 s at 50,000 points. Figure 7 showed the comparison of KNN-GWR with other GWR packages in terms of runtime increase ratio. The results indicated that for 10,000 observations, KNN-GWR was approximately 49.7 times faster than MGWR(PySAL), approximately 58.9 times faster than GWmodel, and approximately 3224.2 times faster than Spgwr. Spgwr cannot obtain the results because the data are larger than 10,000 observations. In addition, when the data increased to 50,000 observations, KNN-GWR was approximately 199.2 times faster than MGWR(PySAL) and approximately 260.7 times faster than GWmodel. KNN-GWR implements calibration operations at amounts of data greater than 50,000, whereas in other GWR packages memory bottlenecks are caused by storing redundant computations.

The advantage of geographically weighted regression methods for analyzing geographic data was retained in the KNN-GWR method, and the storage of the full sample matrix by other GWR software models was changed in the KNN-GWR method to store only the matrix consisting of the observations involved in the regression.

3.3.2. Case Two: House Price Dataset in China

The local model can discover the rich information in geographic data and we performed KNN-GWR modeling on the house price data collected from www.anjuke.com. KNN-GWR generated local parameter estimates that reflect information on the spatial heterogeneity affecting house prices. The performance of KNN-GWR and its spatial non-smoothness are explored visually through local coefficient estimation of mapped variables. Figure 8a–d show the spatial patterns of the KNN-GWR model estimated coefficients.

The results indicate that Area is positively correlated with residential land price, and the larger the residential area, the higher the residential land price (Figure 8c). And it seems reasonable that the effects of age are greater in Beijing–Tianjin–Hebei and the Yangtze River Delta. As these cities are better developed, housing construction has slowed down in recent years, resulting in higher house prices in some areas. In addition, the number of bathrooms is negatively correlated with house prices in some coastal cities, possibly because higher housing prices have dampened the greater demand for the number of bathrooms.

Compared with other GWR packages, the KNN-GWR method demonstrated high efficiency in processing housing price datasets in the study area. As shown in Figure 9 and

Table 5. Information on the runtime of four software packages (KNN-GWR, MGWR(PySAL), GWmodel, and Spgwr) in the house price dataset in China.

Number of Data Points	KNN-GWR	MGWR(PySAL)	GWmodel	Spgwr
1000	0.73	3.43	1.45	24.70
5000	4.51	44.92	45.26	1564.89
10,000	8.73	164.20	239.99	14,812.23
15,000	18.32	359.21	646.92	n/a
20,000	25.21	563.13	978.95	n/a
50,000	51.21	4116.12	7445.82	n/a
100,000	138.33	n/a	n/a	n/a

, as the number of data increases, the running time of KNN-GWR remains relatively stable, while the running time of all the other GWR packages increases rapidly. For instance, with 50,000 observations, the KNN-GWR method requires 51.21 s, while MGWR (PySAl) and GWmodel require 4116.12 and 7445.82 s, respectively. This indicates that KNN-GWR is capable of processing large-scale geographic datasets relatively quickly, while other GWR packages struggled to achieve the same level of efficiency. In addition, as shown in Figure 10, KNN-GWR has an efficiency improvement of a thousand times compared to other GWR software packages. This indicates that KNN-GWR overcomes the memory limitations and process larger datasets more efficiently than other GWR packages. Overall, these findings demonstrate the usefulness of the KNN-GWR method in analyzing large-scale geographic data and its potential to outperform in computational efficiency and memory optimization compared to existing GWR algorithms.

4. Discussion

The KNN-GWR proposed in this paper is an optimized version of GWR. Specifically, it optimized the computational efficiency and memory of GWR by strategies such as constructing KD tree, referencing geographical clustering and reconstructing regression matrices. In essence, there is no difference between KNN-GWR and GWR in exploring spatial non-stationarity. However, KNN-GWR is better suited for processing geographical big data applications and address the changes associated with it.

Therefore, in our experimental results, KNN-GWR has a good fitting degree, just like GWR. Compared with GWR, the advantages of KNN-GWR in computational optimization are as follows. First, the KD tree is used to establish a spatial data index structure that can quickly identification the observation points around the regression point. Second, KNN-GWR reconstructed the weight matrix $W_i(n\times n)$, independent variable matrix $X$ (n $\times (p+1)$), and dependent variable matrix $Y(n\times 1)$ in the local regression calculation for each regression point (the reconstructed matrices are ${\tilde{W}}_i$ $(K\times K)$; ${\tilde{X}}_i$ $(K\times (p+1\left)\right)$; ${\tilde{Y}}_i$ $(K\times 1)$). In this process, observation points that are not within the bandwidth in the local regression of each regression point are directly removed according to the spatial data index structure established by the KD tree. Memory complexity is reduced from $O\left(n^2\right)$ to $O\left(K^2\right)$, where n is the number of observation points. Third, KNN-GWR references the spatial clustering relationship obtained by K-means, which in turn helps to find the optimal bandwidth. To optimize GWR performance, increasing the hardware configuration is one approach; however, it is often necessary to optimize performance of GWR without relying on hardware enhancements.

To demonstrate the practicability of KNN-GWR, simulated data and the Chinese house price dataset were used for the experiments. The results show that GWR cannot handle regression tasks of large amount of data; in contrast, KNN-GWR has great potential to handle a large amount of geographic data, which can largely alleviate the dilemma of GWR in terms of data size. In this paper, we collected house price data from 14 provinces and municipalities and applied it to verify the practical significance of the model in a real geographical environment. When analyzing the house price data of 14 provinces and municipalities, it was found that the factors affecting the house price in coastal cities, more developed areas and slow-developing areas differed. For example, the Age variable has strong effects on house prices in Beijing and Shanghai, but relatively weak effects in other provinces. In terms of runtime, taking simulated data as an example, for 50,000 observations, KNN-GWR is several hundred times faster than MGWR(PySAL), and the GWmodel. In terms of the memory required for calculation, the memory requirement of KNN-GWR is affected by the number K of the nearest neighbors. Generally speaking, the required memory is very small. The memory assignment of KNN-GWR $O\left(K^2\right)$ was compared with that of FastGWR $O\left(nk\right)$, MGWR(PySAL) $O\left(n^2\right)$, GWmodel $O\left(n^2\right)$, and Spgwr $O\left(n^2\right)$ (the number of independent variables of the database is four, and the adjusted bandwidth is 93). For 50,000 observations, KNN-GWR required 33.8 KB of memory, FastGWR required 976.6 KB, and MGWR(PySAL), GWmodel, and Spgwr required 9.31 GB of memory. KNN-GWR has the same ability as GWR to explore spatial non-stationarity. However, in terms of memory usage, KNN-GWR stores a small matrix consisting of observations involved in the regression rather than a full sample matrix.

5. Conclusions

Geographically weighted regression has become a classical method for exploring the spatial non-stationarity in geographic data. However, it faces computational challenges when applied to geographical big data. Previous studies have adopted strategies such as parallelism to optimize GWR, which can handle applications with millions of geographical spatial data, but still encounter problems of memory and computational efficiency. This paper aims to address this limitation, and the main research results are summarized as follows:

(1) The KD tree is proposed to organize geographical spatial data, which can quickly identify the observation points around the regression points in local regression calculations. This greatly optimizes the time-consuming search process in the geographically weighted regression model and significantly improves the computational efficiency.

(2) This paper reconstructed the weight matrix, the independent variable matrix, and the dependent variable matrix based on the characteristics of the kernel function in GWR. This achieves the transformation from a large matrix $(n\times n)$ to a small matrix $(K\times K)$, avoiding the large memory consumption caused by the large matrix $(n\times n)$ during regression calculation in classical geographically weighted regression.

(3) The spatial clustering relationships of the geographic data obtained by the K-means clustering method are referenced to help narrow the search for the optimal bandwidth. This reduces the computational waste in the process of determining the optimal bandwidth in GWR.