Detection of Temporal Shifts in Semantics Using Local Graph Clustering

Abstract

Many changes in our digital corpus have been brought about by the interplay between rapid advances in digital communication and the current environment characterized by pandemics, political polarization, and social unrest. One such change is the pace with which new words enter the mass vocabulary and the frequency at which meanings, perceptions, and interpretations of existing expressions change. The current state-of-the-art algorithms do not allow for an intuitive and rigorous detection of these changes in word meanings over time. We propose a dynamic graph-theoretic approach to inferring the semantics of words and phrases (“terms”) and detecting temporal shifts. Our approach represents each term as a stochastic time-evolving set of contextual words and is a count-based distributional semantic model in nature. We use local clustering techniques to assess the structural changes in a given word’s contextual words. We demonstrate the efficacy of our method by investigating the changes in the semantics of the phrase “Chinavirus”. We conclude that the term took on a much more pejorative meaning when the White House used the term in the second half of March 2020, although the effect appears to have been temporary. We make both the dataset and the code used to generate this paper’s results available.

1. Introduction

In most languages, the semantics of words and phrases (“terms”) evolve over time. Often, the change is gradual and hence is hardly noticeable to an ordinary person in their lifetime. However, in today’s day and age, certain terms come and go that reflect fleeting tastes and fads of the times. For example, Merriam-Webster’s Dictionary announced that it had added 520 new words in 2020, including words such as “COVID-19”. In addition, during such terms’ short lives, they sometimes take on various semantic meanings before being discontinued in their use. While definitions of these terms often are not specified, they can be inferred from their dynamic contexts at respective points in time. A plethora of recent studies has explored constructing such corpus-based sets using clustering based on count-based distributional models (for example, see [1,2,3]). As shown in [4], with the proper hyperparameter settings, count models’ performance is on par with more popular neural-net-inspired predictive models. Then, if one were interested in exploring how the semantics of particular words have evolved over time, one approach could be to construct such sets at various points in time and compare the sets of synonyms.

However, the current state-of-the-art methods for inferring sets of words with similar contexts (hereinafter, “thesaurus”) suffer from two main drawbacks. One is the level of subjectivity involved in deciding the composition of the set of synonyms for each, given the word. Algorithms typically construct word vectors and compute cosine similarities to identify potential synonyms (for example, see [1]). Naturally, one has to decide on the threshold value for these sets. Perhaps a more fundamental drawback is that there is no efficient and intuitive method for detecting the so-called “change-points” in a word’s meaning in time. Manually constructing thesauri at various points in time and comparing the composition of the sets of synonyms is belaboring at best and lacks rigor. While there has been a surge in the popularity of neural-network-based predictive models for similarity tasks and some claims of superiority of such methods over the traditional count models (see [5,6]), it has been shown that while the predictive models outperform on analogy tasks, the evidence appears to be more mixed on similarity tasks [4].

To fill these gaps, we propose a dynamic graph-theoretic approach to locally identify a cluster of the most significant contextual words for a target term that collectively determine its semantics. We show that for a given metric, such as term-level sentiment scores, one can use this clustering based on a particular stochastically evolving set process to dynamically detect change-points in term-level semantics. Our primary analytical tool is the theory of stochastic processes on graphs, where the nodes in the graph represent the terms in a corpus, and the edges represent contexts in the sense that the weight of an edge between two nodes denotes the Hamming distance (i.e., the number of intervening terms) between them. We construct graphs from these term-level relationships in time-based corpora and hence allow these graphs to evolve dynamically over time. We demonstrate the efficacy of our method with the tweets from the first five months of 2020 containing the terms “Coronavirus”, “Chinavirus”, and their variants (see Table in Section 2 for a complete list of terms). We analyze ca. 672,000 such tweets in the U.S. in 2020, investigate the contextual backgrounds of phrases of interest, and detect change-points.

1.1. Literature Review

There is a deep body ofliterature on count-based distributional semantics models. The literature is rooted in matrix theory, beginning with the influential paper on singular value decomposition by [7] and numerous extensions [8], and applications in the classic text by [9]. Reference [10] explored its applications to semantics in their seminal paper, where they used SVD to represent a large term-by-document matrix in a much lower dimension. For applications to word-level semantics tasks, reference [11] demonstrated its usefulness in semantic similarity tasks, and [12] for the semantic association.

In graph theory, there is active research on several domains directly relevant to term-level semantics. Reference [13] showed that when a graph is transformed into a specific form of an asymmetric matrix, known as the nonbacktracking operator, certain eigenvectors of the latter contain helpful information about the community structure of the original graph. References [14,15] demonstrated the usefulness of a related but symmetric matrix, known as the Bethe Hessian operator, in performing similar tasks for simple graphs. Reference [16] showed that for simple graphs of sufficient density, the Bethe Hessian operators could directly estimate the number of communities in a graph. Reference [17] extended the utility of the nonbacktracking operator to cover community detection in more complex graphs. Reference [18] developed the dynamic framework for these matrices. Our objective in this paper is to connect this rich graph theory to semantics.

A necessary step to make input data amenable for analysis with graph theory is to transform the data into a matrix. Among the many types of matrix representation that have been proposed in the literature, one canonical approach is the “term-by-document type” used in the latent semantic analysis (LSA), where words constitute the rows, and different documents in which the words appear correspond to the columns [10,19]. Another approach is “term-by-term” representation, where both the rows and columns correspond to terms, and the entries in the matrix represent the count of co-occurrences of a given pair of terms [20]. Several variations of the “term-by-term” method have been proposed, each attempting to make the values of the entries in the matrix more representative of the true degree of similarity between a given pair of words [21,22,23].

When applying community detection methods from graph theory and statistical inference on graphs, one way to describe the problem of finding a set of strongly associated terms could be as follows: after embedding the terms and their contextual words as the nodes and edges of a graph, partition the nodes into K disjoint clusters such that the difference between the number of within-cluster and between-cluster edges is large. Methods that can be used for this task include centroid-based clustering algorithms, such as K-means [24] and K-medians [25], spectral methods [13,14,15,26], and deep learning methods [27,28,29]. K is often estimated separately using one of many procedures proposed in the literature, e.g., the hypothesis testing approach with bootstrapping  [30], cross-validation [31,32], semidefinite programming [33], the BIC-based method [34], binary segmentation [35,36], and spectral method based on a Laplacian of the adjacency matrices [14,15,16,18]. This class of approaches to clustering is broadly called “global clustering” because all of the nodes are partitioned into K disjoint clusters. Applying this approach to textual data would assume that target words partition contextual words into some K disjoint sets and that each term serves as a contextual word for exactly one target word.

Several graph clustering methods have been proposed for word semantic detection tasks, with most being global in scope. Analysis of textual data using its graph representation dates back to the TextRank algorithm in reference [37], which used global features of a word corresponding to a node in a graph to determine its relative importance. The approach in reference [26] was similar to ours, as we will see later, in that it involved a Markov chain clustering (MCL) on graphs. However, a critical difference is that while our method derives a small, compact set of contextual words as an embedding for one target term, the MCL approach clusters all the terms in a corpus, which entails estimating the total number of clusters in the dataset. As we discuss below in Section 1.2, this estimation is computationally costly and the resulting time complexity of the MCL is at least quadratic in the vocabulary size. An approach that is more similar to ours, in that it utilizes local clustering, is BorderFlow, proposed in [38], where the idea is similar to the MinCut problem in that it seeks to identify a set of nodes such that the number of outgoing edges from the border of this set is maximized compared to the edges within the set. However, since the underlying problem is NP-hard, the algorithm crucially relies on two heuristics without theoretical guarantees for correctness or complexity.

Subspace clustering methods are closely related to global community detection methods and have become popular, particularly in image processing [39,40,41] and computer vision domains [42,43,44]. Earlier works in subspace clustering had an overarching challenge of effectively addressing errors and outliers in the data. Recent developments in subspace clustering attempt to simultaneously correct possible errors and cluster input data into appropriate subspaces [45,46,47]. Similar to global community detection methods discussed above, these algorithms first construct an affinity matrix based on input data and attempt to derive an underlying basis matrix that is of lower dimension than the data. For instance, the algorithm proposed in [45] uses the “self-expressive property”, wherein each data point can be expressed as an affine combination of other points to identify the underlying bases of subspaces and chooses the sparsest set of bases to ensure uniqueness and the lowest possible dimension of the subspaces. Reference [46] further develops sparse subspace segmentation for a specific type of sensor image data by proposing a sparse representation of data. Although the research focused on methodologies and applications of subspace clustering methods has been active in image processing and adjacent domains, it has been less so in textual analysis, specifically semantic similarity tasks.

There are several studies in the literature on the effect of the practice among certain conservative political leaders referring to coronavirus as “Chinavirus” on attitudes towards Asians. Based on a survey of 4311 respondents during the early stage of the pandemic in the U.S., the authors of [48] presented evidence that anti-Asian attitudes were associated with xenophobia and concerns about the virus, and that such sentiment was unique towards Asian-Americans. Among the evidence cited was a significant increase in the frequency of terms such as “Chinese virus” and “Kung Flu” as Google search terms and news articles whose keywords included such terms [48] during this period. Authors in [49] surveyed U.S. adults in May 2020 and found that those that rely more heavily on social media and digital news apps were more strongly associated with anti-Asian attitudes. Reference [50] analyzed over 1.2 million Twitter hashtags from the week before and the week after President Trump’s tweet containing “Chinese Virus” and found that over 50% of the hashtags containing “#chinesevirus” had anti-Asian sentiment compared to less than 20% containing “#coronavirus”, representing a significant increase in anti-Asian sentiments compared to the week before President Trump’s tweet.

1.2. Contributions

There are several challenges with the aforementioned global clustering and subspace segmentation in term-level semantics. Foremost, these approaches address a different problem altogether compared to our main stated goal of characterizing the semantics of a specific term, such as “Chinavirus” and “Coronavirus”. While global clustering and subspace segmentation attempt to segment the whole input data space into some number of clusters, term-level semantics needs the identification of only its own cluster. Further, term-level semantics necessarily calls for a local scope in that one needs to identify the most informative contextual words that characterize a specific target term, making considering global structures in corpus networks in subspace clustering superfluous. This calls for an approach that starts with the target term as the center of locality and builds out the cluster from this root node by adding informative contextual words.

Second, if a graph is sparse, then the composition of individual clusters becomes difficult to detect. For instance, there is a general class of random graph models (this class of random graphs is known as the “degree-corrected block models”, which are variants of the Erdos-Rényi random graph model with a built-in cluster structure), for which it has been shown [51] that it is impossible to estimate the underlying clusters reliably when the within-cluster edge probabilities are not sufficiently higher than the between-cluster edge probabilities, i.e., the signal-to-noise ratio is not sufficiently high. More precisely, consider a random graph on N nodes with two clusters, where the within-cluster (respectively, between-cluster edge) edge probabilities are $a/N$ (respectively, $b/N$); then, if ${(a-b)}^2\le 2(a+b)$, it is impossible to detect the clusters reliably. Further, there is no known effective algorithm to estimate K in the regimes where the average degree of the input graph (represented by the associated unweighted adjacency matrix) is $o\left(\sqrt{N}\right)$ [16]. This presents a problem in many tasks involving term-level semantics, since a corpus in the form of a graph, on average, has the average degree in the order of $o\left(\sqrt{N}\right)$. For instance, in a corpus with a vocabulary of size 40,000, the average degree of an unweighted graph constructed with the “Word2Vec” representation for each term would fall far short of 200. For subspace segmentation methods, sparse data present exogenous constraints in the “self-expressive property” since sufficiently dense data are needed to represent a given data point.

Third, global and subspace clustering methods partition the nodes of the input graph into a few large clusters whose sizes are proportional to the size of the input graph. However, in the case of many tasks involving term-level semantics, obtaining a fixed number of global clusters, even as the size of the input data grows, would not be useful as they will be too coarse. Instead, one is often interested in a more granular identification of a small set of core contextual words that determine the semantics of a target term, where the number of such sets is allowed to grow proportionally to the total number of terms in the input corpus.

Fourth, most of the global clustering requires constructing a matrix of “term-by-term” type of dimension equal to the vocabulary size on which certain matrix operations are performed. However, even a moderate vocabulary having 40,000 terms involves a very large matrix (a matrix of size 40,000 × 40,000, which is too big to fit into a 16 GB RAM on a standard personal computer), so the computational complexities (e.g., for obtaining the eigenvalues and eigenvectors) would make most algorithms impractical in most workstations (the computing time needed to obtain the eigendecomposition is known to be at least quadratic in the size of the input. More precisely, for an input of size $M\times N$, the computational time is $O\left(N^{\delta }\right)$ for some $\delta \in (2,3)$).

In light of these challenges, we propose a local clustering method based on a specific stochastically evolving set process to investigate the semantics of target terms, as it offers several appealing advantages. Foremost, the local cluster sizes for a target term are much smaller (and are often a small constant) than those expected from the global clustering methods. The local clustering algorithms build out the cluster starting from a singleton set; hence, given multiple possible clusters with identical conductance, the smallest one is always chosen. Consequently, local clustering is more informative and tractable as it outputs a statistically significant set of contextual terms for a given target word based on the desired sparsity (measured by conductance) and density (measured by the volume) of the resulting set.

Second, our approach is unsupervised and automatically constructs word embeddings for target terms. Hence, it is tractable for nonexperts to use on new corpora, such as digital text on social media platforms, without the need for training.

Third, compared to the time complexity of many global clustering algorithms (e.g., those involving some eigendecomposition step and hence requiring $O\left(N^{\delta }\right)$ computing time for some $\delta \in (2,3)$), the local clustering algorithms are computationally much more efficient whose computational time complexity is determined by the size of the output set, which can be predetermined as a fixed quantity. Although the problem of finding the sparsest cut is known to be NP-hard, the local clustering algorithm based on [52] that we employ makes use of Markov chains to explore various cuts and chooses a set that is close to the optimal with high probability. As a result, our local clustering algorithm does not need to compute the eigendecomposition of a large matrix and can achieve a constant computational complexity in the size of the corpus by fixing the size of the output set, which directly determines the dimension of the word embedding.

To demonstrate the efficacy of our approach, we apply the local clustering method to the Twitter dataset and identify a cluster of contextual words for each target term at different time periods. We then investigate progressions of the changes in the semantics of a target term by observing the changes in the composition of its local clusters. To detect whether there has been a change-point, we compare the target term semantics inferred from its local cluster to its (1) “cross-sectional” control, i.e., the local cluster of another target term for a given time period; and (2) “temporal” control, i.e., the target term’s clusters in previous time periods. The technical details of the local graph clustering algorithm we propose to adapt for use in semantic shift detection here are developed in detail in [52], and we refer to that resource for a complete treatment of the subject.

Specifically, we study the changes in the composition of semantic clusters of our phrases of interest in and around the time when the White House used the phrase “Chinavirus” and variants thereof (see

Table 1. Equivalence classes of target terms.

“Chinavirus”		“Coronavirus”
Chinavirus	Chinacovid	Coronavirus	Coronaviruses
Chinaviruses	Chinesecovid	Covidvirus	Covidviruses
Chineseviruses	Chinesevirus	Caronavirus	Caronaviruses
Chinacorona	Wuhanviruses	Virus-corona	Virus-covid
Wuhancovid	Wuhancorona	Coronaflu	Coronoviruses
CCPVirus	CCPCoronavirus	Coronacovid	Covidcorona
Wuhanchinavirus		Coronaoutbreak	Coronovirus
Wuhanchinaviruses
Chinawuhanvirus
Chinesecoronavirus
Chinesecoronaviruses
ChineseCommunistPartyvirus
ChineseCommunistPartyviruses

) on at least 20 occasions from 15 March through 31 March 2020 [53]. Similar to previous instances of naming large-scale pandemics using geographical identifiers, such as Middle East Respiratory Syndrome (MERS), Ebola, and Spanish flu, the term “Chinavirus” drew widespread concerns due to the potential harm it may cause the Asian-American community in the form of discrimination, while the administration maintained that the phrase was neutral and synonymous with the more mainstream “Coronavirus”. We investigate whether there was a shift in the meaning and sentiment of the phrase “Chinavirus” around the 2-week period it was used.

In summary, the main contributions of this paper are as follows.

• An unsupervised algorithm based on local graph clustering to automatically characterize and detect shifts in term semantics:

  (a)

  Clusters are incrementally built out, starting with the target term as the center of locality and adding to the cluster contextual words that meet the user-defined thresholds for informativeness and the word embedding dimension;

  (b)

  It has constant time complexity, where the constant is the user-defined description length for the target term, and hence is scalable in the size of the corpus;

  (c)

  The resulting “soft clusters” allow clusters of different terms to overlap to varying degrees.

• A novel empirical analysis of the semantics of the term “Chinavirus”:

  (a)

  Along the time dimension, the term took on significantly, albeit temporarily, more negative sentiment soon after its use by the White House in March 2020;

  (b)

  Compared to the control term “Coronavirus”, the semantics of “Chinavirus” diverged significantly in March 2020.

2. Materials and Methods

2.1. Notation and Terminology

We use $\left[N\right]$ to denote the set of natural numbers $\{1,2,\dots ,N\}$. For any nonnegative functions $f(·)$ and $g(·)$ on the set of natural numbers, we write

$$\begin{array}{cc}\hfill f\left(n\right)\in & \left\{\begin{array}{cc}O\left(g\right(n\left)\right)\hfill & \mathrm{if}lim\underset{n\to \infty }{sup}\frac{f\left(n\right)}{g\left(n\right)}<\infty \hfill \\ o\left(g\right(n\left)\right)\hfill & \mathrm{if}lim\underset{n\to \infty }{sup}\frac{f\left(n\right)}{g\left(n\right)}=0\hfill \end{array}\right.\hfill \end{array}$$

For any given set $\mathbf{S}$ of words and phrases (collectively referred to as “terms” hereinafter), ${\mathbf{S}}^c$ refers to the complement of $\mathbf{S}$ in some corpus. We denote a single-layer and undirected graph by $\mathbf{G}:=(\mathbf{V},\mathbf{E})$ having $\left|\mathbf{V}\right|=:N$ terms (nodes) denoted $\{1,2,\dots ,N\}$ and $\left|\mathbf{E}\right|=:m$ edges. ${\mathbf{A}}^{\prime }$ denotes the unweighted adjacency matrix of $\mathbf{G}$, so the rows and columns of ${\mathbf{A}}^{\prime }$ are indexed by the set of terms $\left[N\right]$ and ${\mathbf{A}}_{ij}^{\prime }$ equals 1 (respectively, 0) if there is an (respectively, no) edge between the nodes i and j, or in other words, the terms i and j appeared in a tweet together. For two subsets of nodes ${\mathbf{S}}_a,{\mathbf{S}}_b\subset \left[N\right]$, the total number of edges between the nodes in ${\mathbf{S}}_a$ and those in ${\mathbf{S}}_b$ is denoted by $e({\mathbf{S}}_a,{\mathbf{S}}_b)$. Given $k\ge 1$ and a pair of terms i and j, let $w_{ij,k}$ be the reciprocal (respectively, 0) of the number of intervening terms between i and j in the k-th tweet if i and j were (respectively, not) present in the k-th tweet. Let $w_{ij}:={\sum }_{k\in \left[\right|C\left|\right]}{w}_{ij,k}$, where $\left|C\right|$ denotes the size of the corpus in terms of the number of tweets. Let $\mathbf{W}$ denote the $N\times N$ weighted matrix whose $(i,j)$-th element is $w_{ij}{[{\sum }_{j^{\prime }\in \left[N\right]}{w}_{i{j}^{\prime }}]}^{-1}$, which is the row-normalized transform of $w_{ij}$. Then, the weighted adjacency matrix $\mathbf{A}$ is defined by the elementwise product of ${\mathbf{A}}^{\prime }$ and $\mathbf{W}$, i.e., ${\mathbf{A}}_{ij}:={\mathbf{A}}_{ij}^{\prime }{\mathbf{W}}_{ij}$ for all $i,j\in \left[N\right]$. $\mathbf{A}$ is the main matrix that we work with in this paper. The weighted matrix is created with the notion that the closer the terms are in a text, the stronger their semantic relationship. The degree of node $i\in \left[N\right]$ is the sum of the i-th row of $\mathbf{A}$ and is denoted by $d\left(v_i\right)$ or, simply, $d_i$.

For a subset of nodes $\mathbf{S}\subset \mathbf{V}$, its volume $vol\left(\mathbf{S}\right)$ is defined to be the sum of the degrees of all the nodes in $\mathbf{S}$, i.e., $vol\left(\mathbf{S}\right):={\sum }_{i\in \mathbf{S}}{d}_i$. We denote the number of outgoing edges from a set of nodes $\mathbf{S}$ by $\partial \left(\mathbf{S}\right):=\left|\right\{(i,j):{\mathbf{A}}_{ij}>0,i\in \mathbf{S},j\in {\mathbf{S}}^c\left\}\right|$. The conductance $\varphi \left(\mathbf{S}\right)$ of a set of nodes $\mathbf{S}\subset \mathbf{V}$ is defined as $\varphi \left(\mathbf{S}\right):=\partial \left(\mathbf{S}\right)/vol\left(\mathbf{S}\right)$. The conductance $\varphi \left(\mathbf{G}\right)$ of a graph $\mathbf{G}$ denotes the minimum among all $\varphi \left(\mathbf{S}\right)$ where $\mathbf{S}\subseteq \mathbf{V}$ satisfies $vol\left(\mathbf{S}\right)\le vol\left(\mathbf{V}\right)/2$. Clearly, $\varphi \left(V\right)=0$. The two-sided vertex boundary $\delta \left(\mathbf{S}\right)$ of a set of nodes $\mathbf{S}$ is defined as $\delta \left(\mathbf{S}\right):=\{v:v\in \mathbf{S}\mathrm{and}e(v,{\mathbf{S}}^c)>0\}\cup \{v:v\in {\mathbf{S}}^c\mathrm{and}e(v,\mathbf{S})>0\}.$

Following [52], we define the cost of a finite path $({\mathbf{S}}_0,{\mathbf{S}}_1,\dots ,{\mathbf{S}}_t)$ associated with the set process $({\mathbf{S}}_t,t\ge 0)$ to be

$$\cos \mathrm{t}({\mathbf{S}}_0,{\mathbf{S}}_1,\dots ,{\mathbf{S}}_t):=vol\left({\mathbf{S}}_0\right)+\sum _{i=1}^t\left(vol({\mathbf{S}}_i\Delta {\mathbf{S}}_{i-1}\right)+\partial \left({\mathbf{S}}_{i-1}\right)),$$

where $\Delta$ denotes the symmetric difference of sets.

For a given target term $v_0$, the contextual words for $v_0$ are those that are more strongly associated with $v_0$ than any other target word, where the strength of this association is measured by the edge weights. The contextual words for $v_0$ are identified by its local cluster, which we discuss next.

2.2. Local Graph Clustering Algorithm in [52]

Our approach adapts a general-purpose local clustering algorithm from the applied probability theory literature [52]. It is a variation of the evolving set process (ESP), where a set evolves depending on the transition probability to other sets defined by their constituent vertices. In summary, the algorithm simulates a Markov chain (called Vb-ESP) on a subset of nodes of a graph until a specific stopping time that is defined in terms of target conductance, set volume, and cost is reached. The end result is a nonexpanding set of a fixed, user-specified volume that constitutes the dimension of the target term embedding. For a detailed treatment of the technique and the technical details of its development, we refer to reference [52]. For the sake of clarity, we restate the local clustering algorithm as proposed in [52] as Algorithms 1 and 2. In Section 2.3, we describe how we adapt Algorithms 1 and 2 to semantic term embedding and change detection tasks.

2.3. Adapting the Local Graph Clustering Algorithm for Semantic Analysis

A necessary step to precede Algorithms 1 and 2 is the construction of a “term-by-term” matrix to which the algorithms are applied. The entries in the matrix correspond to the Hamming distance between a given pair of terms. Details of the preprocessing step are given in Section 3. The inputs into Algorithm 1 are integers T (the maximum number of algorithm iterations), B (total budget for costs incurred in terms of incremental changes made in obtaining $\mathbf{S}$), $\kappa$ (the threshold volume of $\mathbf{S}$), and $\mathsf{\Phi }$ (target conductance). ${\mathbf{S}}_t\left(v_0\right)$ denotes the subset of $\mathbf{V}$ that contains the target term $v_0$ at time t. When the target term is obvious in the context or is irrelevant, ${\mathbf{S}}_t$ is used. The finite sequence $({\mathbf{S}}_t,t\in \left[T\right])$ denotes a stochastically evolving set process (ESP) and $(v_t,t\in \left[T\right])$ denotes a Markov chain over the nodes of $\mathbf{V}$. The desired cluster of contextual terms $\mathbf{S}$ for the target term $v_0$ results from an ESP $\left\{{\mathbf{S}}_t\right\}$ by adding and removing terms chosen via a lazy Markov chain $\left\{v_t\right\}$ on V.

Next, we describe the adaptation of Algorithm 1 to the task of semantic analysis.

Line 1: $\mathbf{S}$ is initially set to ${\mathbf{S}}_0:=\left\{v_0\right\}$. For the purposes of our empirical demonstration, $v_0$ here is set to either of $\{“Chinavirus”,“Coronavirus”\}$ and their equivalent terms as shown in Table 1.

Lines 2–4: Given $\{v_{t-1}=u\}$ at step $t-1$, we sample (in Lines 3 and 4) $v_t$ at step t based on the probability kernel $p(u,·)$, where the transition probability kernel $p(·,·)$ for the Markov chain $v_t$ is given by

$$\begin{array}{c}\hfill p(u,w):=\mathbb{P}(v_t=w|v_{t-1}=u)=\left\{\begin{array}{cc}1/\left[2d\right(u\left)\right]\hfill & \mathrm{if}(u,w)\in E\hfill \\ 1/2\hfill & \mathrm{if}u=w\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.\end{array}$$

Define $p(u,A):={\sum }_{w\in A}p(u,w)$ for $A\subseteq V$. Line 5: Having chosen $v_t$ at step t, we sample a threshold Z uniformly between 0 and $p(v_t,{\mathbf{S}}_{t-1})$, where ${\mathbf{S}}_{t-1}$ is the value of the ESP at step $t-1$.

Algorithm 1 GenerateSamples [52]

Input: Target term $v_0$, nonzero integers $T,B,\kappa$, target conductance $\mathsf{\Phi }\in [0,1]$.      Output: A set ${\mathbf{S}}_{\tau }$ from the volume-biased ESP with the stopping time $\tau$ depending on input parameters $\tau =\tau (T,B,\mathsf{\Phi },\kappa )$.      Internal State: A set ${\mathbf{S}}_{\tau }$ from the volume-biased ESP with $\tau =\tau (T,B,\mathsf{\Phi },\kappa )$; the current location $v_t$ of the random walk; $\partial \left({\mathbf{S}}_t\right),vol\left({\mathbf{S}}_t\right),$ and $cost({\mathbf{S}}_0,\dots ,{\mathbf{S}}_t)$ for the current set ${\mathbf{S}}_t$. 1: Initialize$\mathbf{S}={\mathbf{S}}_0=\left\{v_0\right\}$ 2: for$t=1$ to T do 3:     Given $v_{t-1}$, select $v_t$ with probability 4:        $p(v_{t-1},v_t)$ and update $X\leftarrow v_t$ 5:     Lookup $p(v_t,{\mathbf{S}}_{t-1})$ and pick $Z\sim Unif[0,p(v_t,{\mathbf{S}}_{t-1})]$. 6:     Define ${\mathbf{S}}_t=\left\{u\right|p(u,{\mathbf{S}}_{t-1})\ge Z\},\mathbf{D}=\varnothing$. 7:     for all $u\in \delta \left({\mathbf{S}}_{t-1}\right)$ do 8:         if $p(u,{\mathbf{S}}_{t-1})\le Z$ and $u\in {\mathbf{S}}_{t-1}\Delta {\mathbf{S}}_t$ then 9:            Add u to $\mathbf{D}$ 10:         end if 11:     end for 12:     Update $vol\left({\mathbf{S}}_t\right)$ and cost(${\mathbf{S}}_0,\dots ,{\mathbf{S}}_t)$ 13:     if $t=T$ or cost$({\mathbf{S}}_0,\dots ,{\mathbf{S}}_t)>B$ then 14:         return $\mathbf{S}={\mathbf{S}}_{t-1}\Delta \mathbf{D}$ 15:     end if 16:     Update $\mathbf{S}\leftarrow {\mathbf{S}}_t={\mathbf{S}}_{t-1}\Delta \mathbf{D}$ by adding or removing vertices in $\mathbf{D}$ from $\mathbf{S}$ 17:     Update $\partial \left({\mathbf{S}}_t\right),\varphi \left({\mathbf{S}}_t\right)=\partial \left({\mathbf{S}}_t\right)/vol\left({\mathbf{S}}_t\right)$ 18:     if $\varphi \left({\mathbf{S}}_t\right)\le \mathsf{\Phi }$ and $vol\left({\mathbf{S}}_t\right)\le \kappa$ then 19:         return $\mathbf{S}$ 20:     end if 21: end for

Algorithm 2 EvoPar($v,k,\varphi ,ϵ$) [52]

Input: Target term $v_0$, target volume k, target conductance $\varphi \in [0,1]$, a constant $ϵ\in (0,1)$. Output: A set $\mathbf{S}$ of vertices. 1: $T\leftarrow ϵlogk/6\varphi$ 2: Run in parallel $k^{ϵ/2}$ independent copies of GenerateSample($v_0,T,\infty ,{\kappa }_{ϵ/2}\left(k\right),{\mathsf{\Phi }}_{ϵ/2}\left(\varphi \right)$) 3: if Any of the copies finds $\mathbf{S}$ satisfying $vol\left(\mathbf{S}\right)\le {\kappa }_{ϵ/2}\left(k\right)$ and $\varphi \left(\mathbf{S}\right)\le {\mathsf{\Phi }}_{ϵ/2}\left(\varphi \right)$then 4:     return $\mathbf{S}$ 5: end if

Line 6: ${\mathbf{S}}_t$ is obtained at step t from ${\mathbf{S}}_{t-1}$ using the threshold Z and the kernel $p(·,·)$. The threshold Z is needed to ensure that $\left\{{\mathbf{S}}_t\right\}$ defined in Line 6 does not vanish. This is because $\left\{{\mathbf{S}}_t\right\}$ can vanish if the threshold probability Z, which is used in adding nodes, can be arbitrarily high. This variant of ESP is coupled with the random walk $\left\{v_t\right\}$ and is called Vb-ESP. The set transition probability kernel $K(·,·)$, where $\mathbf{K}(A,B):=\mathbb{P}({\mathbf{S}}_t=B|{\mathbf{S}}_{t-1}=A)$, is adjusted by the product of the ratio of the volumes of the two arguments of $K(·,·)$. It is well known that the mixing time of the underlying Markov chain $\left\{v_t\right\}$ in Vb-ESP is bounded by the Cheeger expansion constant. With constant probability, this process results in set $\mathbf{S}$ with conductance and volume close (to be more precise, $\varphi \left(\mathbf{S}\right)=\mathcal{O}\left(\sqrt{\varphi /ϵ}\right)$ for target conductance $\varphi$ and parameter $ϵ\in (0,1)$, and vol$\left(\mathbf{S}\right)$ = $\mathcal{O}\left(k^{1+ϵ}\right)$ for target volume k) to the desired quantities in $\mathcal{O}(logN)$ time complexity. We set the target conductance to $0.3$ and the volume threshold to $0.01N$. Multiple iterations of it are run to ensure the resulting $\mathbf{S}$ meets the specific thresholds for conductance and volume.

Lines 7–11: For a set $\mathbf{S}$, $p(u,\mathbf{S})$ denotes the transition probability from the current term u to any $w\in \mathbf{S}$:

$$p(u,\mathbf{S}):=\sum _{w\in \mathbf{S}}p(u,w)=\frac{1}{2}\left(\frac{e(u,\mathbf{S})}{d\left(u\right)}+{\mathbf{1}}_{\{u\in S\}}\right)$$

where $e(u,\mathbf{S}):=\left|\right\{(u,w):(u,w)\in E,u\in \mathbf{S}\left\}\right|$. The stationary probability distribution for the nodes in the graph is given by

$$\begin{array}{c}\hfill {\pi }_S\left(u\right)=\left\{\begin{array}{cc}d\left(u\right)/vol\left(\mathbf{S}\right)\hfill & \mathrm{if}u\in \mathbf{S}\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.\end{array}$$

The transition probability matrix for the random walk described here is expressed as $\mathbf{P}:=\frac{1}{2}({\mathbf{M}}^{-1}\mathbf{A}+\mathbf{I})$, where $\mathbf{M}$ is the diagonal matrix whose element ${\mathbf{M}}_{ii}$ is the sum of the i-th row of $\mathbf{A}$, i.e., degree of the i-th node in $\mathbf{A}$.

Line 12: We update the values for $vol\left({\mathbf{S}}_t\right)$ and cost$({\mathbf{S}}_0,\dots ,{\mathbf{S}}_t)$, which are tested against the stopping conditions in Lines 13 and 18. Line 13: A stopping condition based on the number of iterations T and total cost B for the value in Line 12. Line 17: We update the values for the number of outgoing edges from ${\mathbf{S}}_t$ and the conductance to test against the stopping conditions in Line 18. Line 18: A stopping condition based on conductance $\mathsf{\Phi }$ and volume $\kappa$ for the value in Line 17.

Given current ${\mathbf{S}}_1$, the ESP proceeds to the next state ${\mathbf{S}}_2$ in Algorithm 2. Given the threshold $Z\sim \mathrm{Unif}[0,1]$ and ${\mathbf{S}}_2=\{u:p(u,{\mathbf{S}}_1)\ge Z)$ , the volume-based ESP is an ESP with the transition kernel:

$$\widehat{\mathbf{K}}(\mathbf{S},{\mathbf{S}}^{\prime })=\frac{vol\left({\mathbf{S}}^{\prime }\right)}{vol\left(\mathbf{S}\right)}\mathbf{K}(\mathbf{S},{\mathbf{S}}^{\prime })$$

where $\mathbf{K}(\mathbf{S},{\mathbf{S}}^{\prime })$ is the transition kernel for the ESP. Due to the Diaconis–Fill coupling, the transition probability from the current state $(v_t,{\mathbf{S}}_t)$ is

$$\begin{array}{cc}\hfill \mathbf{P}[v_{t+1}=w:v_t=u,{\mathbf{S}}_t=\mathbf{S}]& =p(u,w)\hfill \\ \hfill \mathbf{P}[{\mathbf{S}}_{t+1}={\mathbf{S}}^{\prime }:{\mathbf{S}}_t=\mathbf{S},v_{t+1}=w]& =\frac{\mathbf{K}(\mathbf{S},{\mathbf{S}}^{\prime }){\mathbf{1}}_{\{w\in {\mathbf{S}}^{\prime }\}}}{\mathbf{P}[w\in {\mathbf{S}}_{t+1}:{\mathbf{S}}_t=\mathbf{S}]}\hfill \end{array}$$

Hence, a node $v_{t+1}$ is selected from the transition kernel, and ${\mathbf{S}}_{t+1}$ from the ESP transition kernel, from those sets containing $v_{t+1}$.

Below, we describe the adaptation of Algorithm 2 for the purpose of semantic analysis.

Input: The inputs into Algorithm 2 are a starting vertex $v\in V$, a target conductance $\varphi \in (0,1)$, target volume $k>0$, and $ϵ\in (0,1)$.

Line 1: Set the maximum total number of iterations T as given.

Line 2: Run in parallel withAlgorithm 1 with the given set of parameters. Multiple threads of local clustering are run in parallel since there is a chance that a target term $v_0$ exits its cluster ${\mathbf{S}}_t$ at some $t\in \left[T\right]$.

Lines 3 and 4: When the stopping time $\tau$ is reached by the algorithm, the ${\mathbf{S}}_{\tau }$ that is returned is the contextual set $\mathbf{S}$ for the target term $v_0$. With inputs k (target volume), $\mathsf{\Phi }$ (target conductance), and constant $ϵ\in (0,1)$ into the algorithm, the contextual cluster $\mathbf{S}$ output by the algorithm has conductance $O\left(\sqrt{\mathsf{\Phi }/ϵ}\right)$ and volume of at least half of any set with conductance at most $\mathsf{\Phi }$ and volume at most $m^{1-ϵ}/c$ with absolute constant $c>0$.

For any stopping time $\tau$, this procedure has been shown to output at least one incidence of ${\mathbf{S}}_t$ with $\varphi \left({\mathbf{S}}_t\right)\le O\left(\sqrt{logvol\left({\mathbf{S}}_{\tau }\right)/\tau }\right)$ [52].

We apply this algorithm to the corpus comprising tweets containing “Chinavirus”, “Coronavirus”, and similar terms, which we denote by “equivalence classes of terms” for nine 2-week periods surrounding the second half of March 2020. The significance of this time period is due to the White House’s use of the term “Chinavirus” on at least 20 occasions [53] during this period. In Table 1, we list all the target terms by equivalence classes we performed local clustering on. Once local clusters for each target term are identified, we look up term-level sentiment scores for all terms in the clusters using the package AFINN [54] in the Python programming language. It uses a numerical scale from −5 to +5, with −5 denoting the most negative, +5 the most positive terms, and 0 the neutral sentiment. We then compare the sentiment score distribution of the “Chinavirus” cluster in the second half of March to those in other 2-week periods (“temporal control”) as well as “Coronavirus” clusters over the same time periods (“cross-sectional control”).

3. Datasets

Our data are composed of geotagged public tweets containing the target query terms “Chinavirus”, “Coronavirus”, and variants thereof originating from within the United States from the period spanning the second half of January through the second half of May 2020. The tweets were limited to those written in the English language. The tweets and the accompanying metadata were downloaded from the Twitter Server via the Twitter Developer Application Programming Interface (API) Early Access version 2 between 20 April and 11 May 2021.

Several variations of “Chinavirus” and “Coronavirus” were used on Twitter in terms of spelling and concatenations with other words. In our preprocessing step, we grouped these variations together into a set that we call “equivalence class”. As a result, we had two “equivalence classes”, one for each of “Chinavirus” and “Coronavirus”, and treated each term in an equivalence class as the same, in effect making the cluster for each target term a union of clusters of the constituent terms in its equivalence class. The compositions of these equivalence classes for the two target terms are shown below in Table 1.

We used the NLTK library in Python to apply the standard preprocessing step of tokenization and normalization and removal of frequent stop words and rare contexts that appear fewer than 100 times in our Twitter corpus. We applied other common preprocessing steps to remove capitalization and apply lemmatization and stemming.

The resulting dataset consisted of a corpus of 9,275,358 words representing a vocabulary size of 119,614 terms based on 747,737 tweets posted by 204,488 authors.

Table 2. Descriptions of corpora by 2-week intervals.

Period a	Words b	Vocabulary b	Tweets b	AuthorIDs b
Jan2H	123,874	10,985	10,462	8881
Feb1H	129,545	12,286	10,748	7190
Feb2H	363,394	20,113	17,639	10,927
Mar1H	2,261,059	52,724	176,386	93,339
Mar2H	2,745,589	65,138	215,908	99,069
Apr1H	1,322,025	45,444	99,874	47,457
Apr2H	866,074	37,137	65,139	31,404
May1H	593,134	30,621	43,706	21,972
May2H	432,832	25,620	32,425	17,494

^a “1H” represents the first half and “2H” the second half; ^b represents the numbers of unique counts for each 2-week interval.

displays the breakdown of the dataset by 2-week intervals.

In Section 4, we analyze the progression of the semantics of the equivalence classes of terms “Chinavirus” versus “Coronavirus”, where the latter term is used as a control. We then test the efficacy of our method with (1) the benchmark Word2Vec embeddings and (2) studies on the effect of “Chinavirus” on public sentiment as reported in the literature.

4. Results

Soon after the phrase “Chinavirus” started appearing in the administration’s public statements on 16 March 2020, the number of tweets containing the phrase “Chinavirus” sharply increased. As is evident in Figure 1, the daily count of such tweets increased from less than 500 to more than 3500.

To identify the clusters of the two phrases, we perform local clustering, as discussed in Section 2.3, and its algorithm, presented in Section 2 as the Algorithms 1 and 2. We assigned the equivalence classes of “Chinavirus” and “Coronavirus” to be the starting nodes $v_0$, with the latter as a control node to place the temporal progression of the former in perspective. Below is the set of words that are clustered together with “Chinavirus” for the first half of March (

Table 3. Cluster of “Chinavirus”, 1–15 March 2020.

restaurant	livestock	divulge	hotpot	wheel	floor
remainder	initiative	Chinese	gaslit	storm	fight
designation	sacrifice	married	undone	funny	front
inevitable	historical	forward	global	sadly	army
sabotaging	possible	quicker	insult	guess	sight
investment	together	strategy	faster	slump	drug
supervisor	ingenious	suspend	spiral	unite	alive

), followed by the cluster for the second half of March (

Table 4. Cluster of “Chinavirus”, 16–31 March 2020.

overreach	quarantine	cronies	apologizes	exile	scare
morality	panicked	kissing	moronovirus	abject	stud
mourner	sacrifice	laughts	accusation	goofy	dog
antiviral	baselessly	urine	compassion	poked	hoax
pollution	espionage	fucked	overwhelmed	clout	loser
assassin	dripping	evils	prohibition	risking	alien
exposure	inability	outrage	unbecoming	stroke	secrets
derailed	enflames	destroy	denounce	namaste	nutjob
tearful	dumbkirk	debunk	discharges	cooking	chased
butthead	robbing	selfish	dangerous	diarrhea	cheat
debunk	despair	huawei	heartattack	ill	protest
scream	gorillas	thrive	concussion	mystery	alarmed
repellant	distance	chaotic	marauding	follies	bedbug
migraine	prosecute	purifier	exterminate	illicit	racism
explodes	pangolin	lizard	reelection	fuck	kloots

). One can readily see that the sentiment noticeably became negative in the second half.

As a reference, the clusters for the control term “Coronavirus” for the same two periods are shown in

Table 5. Cluster of “Coronavirus”, 1–15 March 2020.

cancellation	robitussin	interfere	greenlit	bravely	nuk
disruptions	humiliates	overcome	prophecy	disobey	ease
animalistic	equanimity	pharmacy	hosepipe	unmask	sigh
moisturized	heatstroke	decouple	cheapgas	service	fiery
envelopment	propagate	guillotine	confront	dirty	gosh
untouchable	retraction	crackhead	helpless	pumped	goofy
overeacting	perpective	concerned	funeral	punitive	piss
perpetuates	inadequate	scramble	ecstatic	midget	cost
breathmints	planetizen	deadlier	colonial	faceass	fuel
precipitate	unfriended	populism	educate

and

Table 6. Cluster of “Coronavirus”, 16–31 March 2020.

hospitable	collapsed	crackpot	pumped	stranger	nuk
celebration	blockhead	feckless	pissing	breach	ease
overshadow	possessive	regarded	sodexo	nutcase	bane
excellence	expensive	rampage	stabbed	reckless	dirty
deprivation	disappear	bungling	unstable	ghetto	revive
panoramic	apologize	evacuate	cringed	fucking	goof
fucknugettes	strangled	punitive	squeak	cleanly	ruined
martyrdom	virulence	prevent	flagging	midget	dirt
determined	reputable	kickback	badass	faceass	sigh
contender	downside	delusional	babble

. Compared to the clusters for “Chinavirus” over the same periods, one can see that there is no noticeable variation in sentiment.

We now look at the distributions of the sentiment scores for both terms over time. Figure 2 displays the temporal progression of the distribution of the contextual words by their sentiment scores for “Chinavirus”, where more negative scores denote relatively more negative words and positive scores positive on the scale of −5 to 5. It is apparent that the distribution is more concentrated around a negative score of −3 on 2 March, when the White House made use of the phrase “Chinavirus”. More formally, using the Kolmogorov–Smirnov test as implemented in the SciPy Python package, we are able to reject, with a p-value of less than 0.001, the null hypothesis that the sentiment scores on 2 March came from the same distribution as those in the other 2-week periods. Using the Mann–Whitney U-test in the SciPy package, we conclude, with a p-value of less than 0.001, in favor of the hypothesis that the sentiment scores on 2 March are lower than those in the other periods and the sentiment scores of the cross-sectional control, “Coronavirus” (see Section 2 for discussion).

As a point of reference, Figure 3 displays a similar chart for the term “Coronavirus”. Compared to “Chinavirus”, there is no visible shift in the distribution of the sentiment scores.

Next, we checked the effectiveness of our method in two ways. First, we note that this shift in the sentiment distribution of “Chinavirus” in March, compared to its distributions in previous periods and those of “Coronavirus”, is consistent with what was reported in references [48,49,50]. Second, we transformed the vocabulary in each 2-week period into Word2Vec representations using the Gensim package in the Python programming language and identified a set of synonyms for each of the two target terms. Synonyms in Word2Vec are defined in terms of cosine similarity, and we classified terms in the vocabulary as synonyms if their cosine similarity measures were at least 0.3. Figure 2 and Figure 4 show that the sentiment of the synonyms of “Chinavirus” clearly shifted more negative compared to surrounding periods in the second half of March 2020.

5. Discussion

As new additions to the English language, terms such as “Chinavirus” present a significant challenge when one tries to ascertain its semantics and perform analogy tasks to other related terms such as “Coronavirus”. In the absence of a definitive source for the semantics of such terms, we propose to characterize the semantics of novel terms by looking at their clusters of strongly associated words and phrases, staying true to John R. Firth’s famous quote that “[y]ou shall know a word by the company it keeps” [55]. To obtainthe granular clustering that allows for a differentiation of two closely related terms, we use “local” clustering, as implemented in [52], to build out an informative cluster of contextual words, starting with the singleton set containing the target term and stopping once a target conductance or cluster size is reached. By comparing the composition of the term’s clusters’ composition over time and the “control” terms, one can reasonably infer if the term’s semantics have changed over time and how it compares to other terms.

Our approach makes the following contributions. Compared to existing graph clustering algorithms, our method identifies a local cluster of contextual words that is tailored only to a specific target term, thus making the cluster more informative and compact for inferring term-level semantics. Because our approach does not make use of the information about global structures of corpus graphs, it avoids performing unnecessary work, such as clustering all words in the input data, and hence is computationally efficient with constant time complexity. Our method allows clusters of two different target terms to overlap, which is more intuitive and consistent with how one thinks about the semantics of words than partitioning contextual words into disjoint clusters, as performed in global clustering and some of the subspace segmentation methods. Lastly, we provide a rigorous confirmation of the results in the existing literature that observe that the use of “Chinavirus” and its close variants by the White House in March of 2020 is associated with a significant increase in negative sentiment in public discourse compared to earlier months and compared to “Coronavirus” as the control.

For future study, one possible application of local clustering is the construction of a thesaurus. The current approach to thesaurus construction does not allow for efficient and intuitive detection of the so-called “change-points” in a word’s meaning in time. Manually constructing thesauri at various points in time and comparing the composition of the sets of synonyms is belaboring at best and lacks rigor. Applying local clustering, one could infer from any statistically significant change in cluster composition, from one period to another, a change-point in the thesaurus, suggesting that semantic changes in certain words took place.