Modeling the Mutual Dynamic Correlations of Words in Written Texts Using Multivariate Hawkes Processes

Abstract

The occurrence patterns of important words found in six texts (one historical pamphlet and five renowned academic books) are analyzed using both univariate and multivariate Hawkes processes. By treating the occurrence patterns as binary time-series data along the texts, we investigate how effectively univariate and multivariate Hawkes processes capture the characteristics of these word occurrence signals. Through maximum likelihood estimation and subsequent simulations, we found that the multivariate Hawkes process clearly outperforms the univariate Hawkes process in modeling word occurrence signals. Moreover, we found that the multivariate Hawkes process can provide a Hawkes graph, which serves as an intuitive representation of the relationships between concepts appearing in the analyzed text. Furthermore, our study demonstrates that the importance of concepts within a given text can be quantitatively estimated based on the optimized parameter values of the multivariate Hawkes process.

1. Introduction

Analyzing document data as a time series has been a commonly employed approach in the past [1,2,3,4,5,6]. One notable advantage of representing documents as time-series data is the ability to quantitatively capture long-range correlations among various components of the documents [7,8,9,10,11,12]. This advantage stems from the application of mathematical tools in time-series analysis, such as autocorrelation functions [13,14,15], waiting time distributions [16,17], and similar methods. For instance, autocorrelation functions can be utilized to determine the significance of words within a given document [13]. Words exhibiting strong long-range autocorrelation across a text are often considered to be closely related to the document’s central theme [13,14,15,16,17].

In a previous study [17], we utilized univariate Hawkes processes to model word occurrence patterns in texts, demonstrating their effectiveness in capturing the dynamic, long-range correlations between different positions in a considered text. Hawkes processes are a type of stochastic process designed to model events occurring over time, influenced by prior occurrences [18]. They are particularly effective at capturing self-exciting phenomena, where the occurrence of one event increases the probability of subsequent events, whether in the short-term or long-term [19,20,21,22,23,24,25,26,27,28]. This characteristic makes Hawkes processes highly suitable for describing the occurrences of significant words in a text that are associated with specific concepts or ideas. Such key words often reappear during the explanation of crucial concepts. In other words, important words related to the subject matter of the text exhibit self-excitatory behavior, correlating with their own past occurrence signals. As a result, their patterns can be effectively modeled using Hawkes processes. This was confirmed in our previous study [17], which showed that many keywords in famous academic books follow such patterns.

However, the methodology proposed in the previous study [17] has certain limitations. Specifically, it employed univariate Hawkes processes, which can capture autocorrelation—measuring how a word’s occurrence signal is correlated with itself over different time lags. However, this approach cannot account for cross-correlation, which quantifies the correlation between the occurrence signals of two different words as a function of a time lag. It is common for significant concepts in documents to be explained from multiple perspectives, using several key terms. In such cases, these key terms should be interrelated or correlated, such that they mutually influence one another’s appearance within the document.

To characterize such documents, the model used should be capable of capturing the mutual correlations among keywords. This requirement underscores the necessity of multivariate Hawkes processes [29,30,31] for text modeling. As will be elaborated later, multivariate Hawkes processes can effectively represent multiple, mutually correlated stochastic processes, such as the occurrence signals of several keywords within a text. The objective of this study is to employ multivariate Hawkes processes to model documents. To the best of our knowledge, only a few studies have explored the application of multivariate Hawkes processes to document analysis [32,33]. However, their primary focus lies in modeling relationships between documents—such as interactions among social media posts or between news articles—rather than analyzing the internal structure of a single document, which is the central theme of the present study. We will demonstrate in this study that employing multivariate Hawkes processes, as opposed to univariate ones, significantly enhances the accuracy in describing word occurrence signals. Moreover, we discovered that Hawkes graphs [34], which are graphical representations of multivariate Hawkes processes, are highly effective in intuitively conveying the contents of analyzed documents. These graphs sensitively depict the interrelationships among multiple keywords appearing in the document. From the graph, it is possible to identify a central keyword that is closely tied to the document’s theme, as well as several other keywords used to describe or introduce the central concept. While Hawkes graphs may appear similar to diagrams depicting word co-occurrence networks [35,36], they are more advantageous as they incorporate information about causal relationships and dependencies among keywords. Thus, another significant benefit of utilizing the multivariate Hawkes process lies in its ability to provide an intuitive representation of document content through Hawkes graphs.

The remainder of this paper is organized as follows. In the next section, we outline the methodology, describing the characteristics of the univariate and multivariate Hawkes processes employed, including their kernel functions and log-likelihood functions. Additionally, we explain how word occurrence signals were extracted from each document and how these signals were modeled using the univariate and multivariate Hawkes processes. This section also details the simulation procedures used to validate our modeling approach with both types of Hawkes processes. The subsequent section presents our results, highlighting the advantages of multivariate Hawkes processes over univariate ones. This section also elaborates on the construction and practical aspects of Hawkes graphs. Finally, in the concluding section, we summarize our findings and propose directions for future research.

2. Methodology

2.1. Converting Text as Time-Series Data

To treat written texts as time-series data, we assign a serial number to every sentence in a considered document and assign a time role to this sentence number [13,14,15,16,17]. The occurrence signal of a considered word is then defined as

$$x\left(t\right)=\left\{\begin{array}{c}0 \left(\mathrm{w}\mathrm{h}\mathrm{e}\mathrm{n}\mathrm{ }\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{ }\mathrm{w}\mathrm{o}\mathrm{r}\mathrm{d}\mathrm{ }\mathrm{d}\mathrm{o}\mathrm{e}\mathrm{s}\mathrm{ }\mathrm{n}\mathrm{o}\mathrm{t}\mathrm{ }\mathrm{a}\mathrm{p}\mathrm{p}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{ }\mathrm{i}\mathrm{n}\mathrm{ }\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{ }t\text{-}\mathrm{t}\mathrm{h}\mathrm{ }\mathrm{s}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{e}\right)\\ 1 \left(\mathrm{w}\mathrm{h}\mathrm{e}\mathrm{n}\mathrm{ }\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{ }\mathrm{w}\mathrm{o}\mathrm{r}\mathrm{d}\mathrm{ }\mathrm{a}\mathrm{p}\mathrm{p}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{s}\mathrm{ }\mathrm{i}\mathrm{n}\mathrm{ }\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{ }t\text{-}\mathrm{t}\mathrm{h}\mathrm{ }\mathrm{s}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{e}\right)\end{array},\right.$$

(1)

which is a binary variable expressing word occurrence event, and accumulated value of $x\left(t\right)$, i.e.,

$$N\left(s\right)=\sum _{t=1}^{s}x\left(t\right)$$

(2)

becomes a counting process for the word occurrence event, i.e., $N\left(s\right)$ represents the number of occurrences of a considered word along the documents. We treat $x\left(t\right)$ and $N\left(s\right)$ as time-series data and seek a stochastic process that can well describe the behavior of $x\left(t\right)$ and $N\left(s\right)$. As already clarified, for words that are not directly connected to the theme of a given document and therefore that do not exhibit any dynamic correlations, their occurrences can be accurately modeled using either a homogeneous Poisson process or an inhomogeneous Poisson process [13,14,15,16]. In this paper, we investigate stochastic processes that characterize $x\left(t\right)$ and $N\left(s\right)$ for significant words which are closely related to the document’s theme and therefore have long-range dynamic correlations.

2.2. Maximum Likelihood Estimation of Hawkes Processes

To illustrate our methodology, we briefly introduce the intensity and likelihood functions of univariate and multivariate Hawkes processes, which we consider to be suitable stochastic models for explaining the observed word occurrence signals, Equation (1), for real words in the analyzed text.

The univariate Hawkes process is mathematically defined by its intensity function, $\lambda \left(t\right)$, representing the conditional event rate (word occurrence rate) at time $t$. For a process with an exponential decay kernel, the intensity function is given as [37,38]:

$$\lambda \left(t\right)=\mu +\sum _{\left\{t_i< t\right\}}\alpha \exp \left\{-\beta \left(t-t_i\right)\right\}$$

(3)

where $\mu$ denotes the baseline intensity, $t_i$ represents the occurrence times of previous events ($t_i < t$), $\alpha$ is a positive parameter quantifying the magnitude of self-enhancement, and $\beta$ is a positive parameter characterizing the rate at which the effect of self-enhancement decays. Given that event times $\{t_1, t_2, t_3,\dots , t_n\}$ are observed within a time interval $[0,T]$, where $T$ denotes the total number of sentences in the text, the log-likelihood function of the univariate Hawkes process is expressed as [37,38]:

$$\begin{array}{ll}l\left(\mu , \alpha , \beta \right)={\displaystyle \sum _{i=1}^n}& \log \left(\mu +{\displaystyle \sum _{\left\{t_j< t_i\right\}}}\alpha \exp \left\{-\beta \left(t_i-t_j\right)\right\}\right)-\mu T\\ & -{\displaystyle \frac{\alpha }{\beta }}{\displaystyle \sum _{i=1}^n}\left(1-\exp \left\{-\beta \left(T-t_i\right)\right\}\right).\end{array}$$

(4)

In the framework of maximum likelihood estimation (MLE), $\mu$, $\alpha$ and $\beta$ are treated as fitting parameters to adapt the univariate Hawkes process to the observed sequence of event times.

The multivariate Hawkes process expands upon the univariate Hawkes process by introducing multiple dimensions, allowing it to capture interactions among various types of events. In our case, we anticipate that the multivariate Hawkes process can effectively model the cross-correlation between word occurrence signals of different words within a given text. If we select $d$ words as the focus of our analysis and consider the $i$-th word, the intensity function of the multivariate Hawkes process for events of type $i$ (i.e., the intensity function for the occurrence of the $i$-th word) is defined as [38,39]:

$${\lambda }_i\left(t\right)={\mu }_i+\sum _{j=1}^d\sum _{\left\{{k:t}_k^j< t\right\}}{\alpha }_{ij}\exp \left\{-{\beta }_{ij}\left(t-t_k^j\right)\right\}$$

(5)

where ${\mu }_i$ is the baseline intensity for events of type $i$, $d$ is the total number of event types (the total number of analyzed words), $t_k^j$ represents the time of the $k$-th event of type $j$ (i.e., the $k$-th occurrence of the $j$-th word), ${\alpha }_{ij}$ denotes the influence strength of events of type $j$ on those of type i, and ${\beta }_{ij}$ denotes the decay rate of excitation induced by the occurrence of the $j$-th word on the subsequent occurrence of the $i$-th word.

The log-likelihood function for the multivariate Hawkes process is expressed as:

$$\begin{array}{ll}l\left(\mathit{\mu }, \mathit{\alpha }, \mathit{\beta }\right)={\displaystyle \sum _{i=1}^d}& {\displaystyle \sum _{k=1}^{n_i}}\log \left[{\mu }_i+{\displaystyle \sum _{j=1}^d}{\displaystyle \sum _{\left\{m:t_m^j< t_k^i\right\}}}{\alpha }_{ij}\exp \left\{-{\beta }_{ij}\left(t_k^i-t_m^j\right)\right\}\right]\\ & -{\displaystyle \sum _{i=1}^d}\left[{\mu }_{i}T-{\displaystyle \sum _{j=1}^d}{\displaystyle \sum _{m=1}^{n_j}}{\displaystyle \frac{{\alpha }_{ij}}{{\beta }_{ij}}}\left(1-\exp \left\{-{\beta }_{ij}\left(T-t_m^j\right)\right\}\right)\right]\end{array}$$

(6)

Here, $n_i$ represents the number of occurrences of events of type $i$, $\mathit{\mu }$ is the baseline intensity vector, defined as $\mathit{\mu }=({\mu }_1, {\mu }_2,\dots , {\mu }_d)$, while $\mathit{\alpha }=\left\{{\alpha }_{ij}\right\}$ and $\mathit{\beta }=\left\{{\beta }_{ij}\right\}$ are matrices comprising the fitting parameters introduced in Equation (5), $T$ is the total number of sentences in the text. Equation (6) can be derived in the following manner. According to the general theory of point processes, the likelihood function of the multivariate Hawkes process is given by [40] (Chapter 7):

$$L(\mathit{\mu }, \mathit{\alpha }, \mathit{\beta })=\prod _{i=1}^d\left[\prod _{k=1}^{n_i}{\lambda }_i\left(t_k^i\right)\right]\exp \left(-{\int }_0^T{\lambda }_i\left(t\right)dt\right)$$

(7)

where ${\lambda }_i\left(t\right)$ is defined by Equation (5). By substituting Equation (5) into Equation (7), taking the logarithm of both sides, and performing further integral calculations, we obtain the log-likelihood function given in Equation (6).

In the framework of MLE for univariate Hawkes processes, the goal is to find optimized parameters $\mu$, $\alpha$ and $\beta$ that maximize the log-likelihood function, Equation (4), given a list of event times $\{t_1, t_2, t_3,\dots , t_n\}$. Similarly, for multivariate Hawkes processes, the objective is to optimize the vector $\mathit{\mu }$ and matrices $\mathit{\alpha }$ and $\mathit{\beta }$ to maximize the log-likelihood function, Equation (6), given lists of event times for each type of event, $\{t_1^i, t_2^i,\dots , t_{n_i}^i\}$ for $1\le i\le d$. In practical MLE procedures, we used the minimize() function from the Python (version 3.11.5) library scipy.optimize (version 1.11.1) to determine the optimal parameter values by minimizing $-l(\mu ,\alpha ,\beta )$ and $-l(\mathit{\mu },\mathit{\alpha },\mathit{\beta })$. The L-BFGS-B (Limited-memory Broyden–Fletcher–Goldfarb–Shanno with Box constraints) method was employed to define upper and lower bounds for the fitting parameters, ensuring that the solutions remained within the specified ranges.

2.3. Selecting Important Words in Used Texts

Six texts employed in this study are listed in Table 1. One of the six documents is a famous historical pamphlet (“Common Sense” by Thomas Paine); the other five are well-known academic books. They are chosen so as to represent wide range of written texts, In the table, short names of each book and some information are also shown. The preface, contents and index pages were deleted before starting the text preprocessing because they may act as noise and may affect the final results.

Table 1. Summary of English texts employed.

Short Name	Title	Author	Vocabulary Size	Length in Sentences
Darwin	On the Origin of Species	Charles Darwin	5728	4036
Einstein	Relativity: The Special and General Theory	Albert Einstein	2222	1107
Faraday	The Chemical History of a Candle	Michael Faraday	1563	2563
Freud	Dream Psychology	Sigmund Freud	4520	1977
Paine	Common Sense	Thomas Paine	637	2558
Plato	The Republic	Plato	5686	5268

We select 20 important words from each of the six texts to analyze their occurrence signals by using the univariate and the multivariate Hawkes processes. This means that 20 pairs of Equations (3) and (4), are considered for modeling the 20 words with univariate Hawkes processes while we set $d=20$ in Equations (5) and (6) for modeling with the multivariate Hawkes process. The selected 20 important words having long-range dynamic correlations are listed in Table 2, Table 3, Table 4, Table 5, Table 6 and Table 7. Note that only nouns are included in the tables, as this study aims to investigate the causal relationships between “concepts” represented by nouns within the document (e.g., word “A” is used to describe word “B,” etc.). This approach is feasible because we primarily analyze academic books, where a word that frequently occurs with strong dynamic correlations is assumed to carry a consistent and well-defined meaning. In this context, a word exhibiting strong and long-range dynamic correlations is regarded as representing a coherent, singular “concept.” The investigation of relationships between “concepts” is based on correlations extracted through multivariate Hawkes process analysis.

Table 2. The top 20 important words for Darwin text.

Word	Number of Occurrences	$\mathit{\tau }$	$\mathit{\beta }$	$⟨\mathit{\tau }⟩$ (※)
formation	134	0.446	0.143	2280.488
hybrid	124	4.971	0.199	617.345
cross	160	0.534	0.171	297.335
class	123	0.010	0.130	220.608
selection	350	0.029	0.148	91.912
instinct	100	4.744	0.261	88.223
group	212	0.140	0.181	41.714
island	138	6.593	0.333	39.887
period	267	0.010	0.146	37.960
world	145	0.010	0.148	31.846
organ	164	1.438	0.260	27.215
production	125	0.010	0.150	26.628
plant	302	0.062	0.178	22.628
structure	222	0.015	0.156	22.617
habit	145	0.020	0.162	19.592
variety	360	1.250	0.291	13.272
part	253	0.013	0.165	11.025
character	242	0.454	0.257	9.247
theory	130	0.010	0.164	8.650
species	1005	0.429	0.261	8.021

※ Each word is sorted in descending order according to the importance metric, $⟨\tau ⟩,$ which is an optimal measure derived from autocorrelation. For Table 3, Table 4, Table 5, Table 6 and Table 7, the listed word order is also based on sorting by $⟨\tau ⟩$.

Table 3. The top 20 important words for Einstein text.

Word	Number of Occurrences	$\mathit{\tau }$	$\mathit{\beta }$	$⟨\mathit{\tau }⟩$
velocity	84	0.143	0.188	30.516
field	78	0.832	0.275	11.681
light	82	0.196	0.239	6.240
body	113	0.189	0.240	5.845
equation	50	0.741	0.322	5.092
motion	98	0.188	0.246	4.932
position	53	0.173	0.245	4.751
theory	142	0.688	0.337	3.958
place	58	0.010	0.181	2.977
co-ordinate	86	0.152	0.264	2.646
point	104	0.115	0.256	2.429
law	118	0.258	0.301	2.341
line	52	0.010	0.186	2.298
relativity	152	0.135	0.268	2.184
reference	72	0.010	0.188	2.067
principle	62	0.375	0.342	2.058
time	98	0.065	0.249	1.601
space	77	0.095	0.268	1.535
distance	53	0.163	0.315	1.221
system	101	0.224	0.345	1.188

Table 4. The top 20 important words for Faraday text.

Word	Number of Occurrences	$\mathit{\tau }$	$\mathit{\beta }$	$⟨\mathit{\tau }⟩$
flame	133	0.396	0.169	247.670
candle	241	0.081	0.157	115.727
jar	78	0.011	0.141	62.389
gas	97	0.010	0.146	37.606
carbon	85	2.335	0.269	37.103
water	193	2.403	0.274	34.772
oxygen	115	3.134	0.306	26.441
heat	95	0.187	0.203	20.215
acid	86	0.252	0.216	16.023
atmosphere	51	0.010	0.156	15.353
hydrogen	78	0.167	0.210	13.548
air	207	0.221	0.223	11.207
iron	59	0.242	0.292	2.556
combustion	98	0.045	0.222	2.357
experiment	100	0.010	0.187	2.184
piece	88	0.035	0.221	1.910
vessel	51	0.145	0.285	1.716
action	80	0.238	0.388	0.862
substance	102	0.054	0.274	0.774
part	73	0.010	0.216	0.639

Table 5. The top 20 important words for Freud text.

Word	Number of Occurrences	$\mathit{\tau }$	$\mathit{\beta }$	$⟨\mathit{\tau }⟩$
psychic	134	0.100	0.136	1081.268
dream	791	0.193	0.183	52.727
process	104	0.010	0.143	48.251
content	110	0.010	0.146	36.565
system	69	0.560	0.228	24.712
wish	178	0.238	0.211	18.195
consciousness	65	0.112	0.196	15.783
thought	162	0.194	0.210	15.365
work	69	0.010	0.160	11.299
analysis	92	0.010	0.163	9.023
sleep	94	0.498	0.263	8.937
activity	60	0.010	0.167	6.947
formation	51	0.010	0.171	5.324
interpretation	54	0.010	0.184	2.549
life	83	0.010	0.185	2.419
day	106	0.054	0.229	2.277
state	66	0.010	0.193	1.640
child	73	0.185	0.303	1.634
form	60	0.010	0.195	1.475
idea	69	0.204	0.322	1.394

Table 6. The top 8 important words for Paine text.

Word	Number of Occurrences	$\mathit{\tau }$	$\mathit{\beta }$	$⟨\mathit{\tau }⟩$
king	74	0.674	0.274	9.774
continent	48	0.035	0.202	3.942
government	63	0.399	0.358	1.866
britain	45	0.031	0.223	1.577
england	50	0.140	0.323	0.944
america	45	0.010	0.220	0.571
time	61	0.382	0.999	0.382
power	45	0.033	0.327	0.209

Table 7. The top 20 important words for Plato text.

Word	Number of Occurrences	$\mathit{\tau }$	$\mathit{\beta }$	$⟨\mathit{\tau }⟩$
justice	191	0.056	0.152	126.257
god	124	0.031	0.151	70.764
pleasure	108	0.041	0.156	64.617
knowledge	133	0.042	0.177	15.639
soul	197	0.013	0.159	15.349
opinion	100	0.086	0.202	9.287
state	359	0.018	0.172	9.145
art	113	0.091	0.221	5.013
life	172	0.073	0.227	3.330
principle	102	0.010	0.183	2.716
men	219	0.010	0.192	1.687
nature	211	0.010	0.195	1.496
city	110	0.010	0.196	1.410
reason	119	0.011	0.201	1.241
man	321	0.010	0.201	1.155
truth	120	0.010	0.203	1.080
friend	112	0.010	0.207	0.891
word	102	0.010	0.227	0.447
question	112	0.010	0.232	0.384
right	131	0.345	0.884	0.367

The accuracy of representing word occurrence signals using the multivariate Hawkes process improves as the number of analyzed words increases. This is because a larger set of words allows for the consideration of all possible event causes that mutually excite one another. In most documents, key concepts are explained from multiple perspectives, utilizing numerous words to convey complex ideas. Such texts often exhibit correlations where many words influence each other.

In a multivariate Hawkes process, each dimension represents the occurrence signal of a single word. Consequently, limiting the process to only 20 dimensions restricts its ability to capture correlations among words, making it insufficient for describing the cross-correlations commonly found in texts. However, the maximum likelihood estimation (MLE) of a multivariate Hawkes process requires extensive convergence calculations, making it computationally intensive and practical only for cases with fewer than 20 dimensions. Therefore, establishing clear selection criteria for the 20 key words, as outlined below, is critical to this study.

First, we calculated the autocorrelation functions (ACFs) for all words that appear more than 50 times in each text, using the formula:

$$\varphi \left(s\right)={\displaystyle \frac{\frac{1}{T-s}\sum _{t=1}^{T-s}\left\{x\left(t\right)-\overline{x}\right\}\left\{x\left(t+s\right)-\overline{x}\right\}}{\frac{1}{T}\sum _{t=1}^T{\left\{x\left(t\right)-\overline{x}\right\}}^2}}$$

(8)

where $\overline{x}$ represents a mean value of $x\left(t\right)$. Then, these ACFs were fitted by a Kohlrausch–Williams–Watts (KWW) function:

$$\varphi \left(t\right)=\exp \left\{-{\left({\displaystyle \frac{t}{\tau }}\right)}^{\beta }\right\}$$

(9)

where $\tau$ (a relaxation time) and $\beta$ (a shape parameter) are fitting parameters that satisfy $\tau >0$ and $0<\beta \le 1$. The KWW function has been demonstrated to effectively represent real ACFs of important words with long-range dynamic correlations [13,14,15,16,17]. Next, we evaluated the mean relaxation times $⟨\tau ⟩$ for each word using the equation:

$$⟨\tau ⟩=\underset{0}{\overset{\infty }{\int }}\exp \left\{-{\left({\displaystyle \frac{t}{\tau }}\right)}^{\beta }\right\}dt={\displaystyle \frac{\tau }{\beta }} \mathrm{\Gamma }\left({\displaystyle \frac{1}{\beta }}\right)$$

(10)

where $\mathrm{\Gamma }\left(x\right)$ denotes the gamma function. Finally, we select top 20 words in terms of $⟨\tau ⟩$ as the key words to be analyzed for each document. A large mean relaxation time, $⟨\tau ⟩$, indicates that a word exhibits both strong dynamic correlations and long-range memory, making it relevant to the key concept or theme of the text. Therefore, we consider words with large $⟨\tau ⟩$ to be significant. In Table 2, Table 3, Table 4, Table 5, Table 6 and Table 7, the values of the fitting parameters, $\tau$ and $\beta$, and the mean relaxation time, $⟨\tau ⟩$, are shown for each of the 20 words. Note that for the Paine text seen in Table 6, the selected words are limited to only 8, as the text is significantly shorter than the others. We relaxed the criteria to 45 or more occurrences instead of 50 or more, but only eight words met that criterion for the Paine text. Figure 1 shows the word occurrence signal $x\left(t\right)$, the calculated ACF and the fitted KWW function for a picked word “formation” in the Darwin text. Optimized values of fitting parameters used in Figure 1b are $\tau =0.4455, \beta =0.1427$ and the resultant value of objective function (sum of squared residuals) is 0.24732. In the fitting of the KWW function to ACFs, we employed a nonlinear least-squares algorithm to optimize the fitting parameters $\tau$ and $\beta$. For practical calculations, we used the curve_fit( ) function from the scipy.optimize library. This function iteratively minimizes the objective function, defined as the sum of squared residuals—namely, the difference between the ACF and the KWW model—by adjusting the fitting parameters. The iterative process terminates when the change in the objective function between two successive steps falls below a predefined threshold of $1.0\times {10}^{-8}$. Upon completion of the iteration, we confirmed that the values of the objective functions (sum of squared residuals) are less than 0.3 for all important words.

Figure 1. (a) The word occurrence signal, $x\left(t\right)$, for the word “formation” in the Darwin text, and (b) the calculated ACFs (circles) along with the fitted KWW function to the ACFs (red curve) for the word “formation” in the Darwin text.

Of course, selecting words solely based on $⟨\tau ⟩$—as evaluated from the autocorrelation function—has its limitations, as it overlooks semantic significance. Nevertheless, we do not apply semantic analysis here, since semantically important words with weak dynamic correlations are not well-suited for analysis via stochastic process modeling used in this study. The relationship between dynamic correlations and semantic significance remains a compelling topic, deserving further exploration in future research.

2.4. Validation of Modeling with Hawkes Processes

In this study, we use univariate and multivariate Hawkes processes to model word occurrence signals in the considered texts. The validity of the modeling is confirmed by following two different procedures.

The first procedure is the quantile–quantile (Q–Q) plot. The Q–Q plot is a widely used diagnostic tool for evaluating the goodness-of-fit of Hawkes process models [41]. Examples are shown in Figure 2 and Figure 3, where transformed inter-event times—obtained via intensity normalization—are plotted against theoretical quantiles. Plots (a) illustrate Q–Q plots for univariate Hawkes processes, while plots (b) correspond to multivariate versions. The univariate models (plots (a)) exhibit pronounced deviations from the reference line, particularly in regions of short inter-event times, indicating inadequate fit in those intervals. In contrast, the multivariate models (plots (b)) display notable improvement, with transformed times aligning more closely with the theoretical distribution. This improvement is consistently observed across the majority of important words identified within each text corpus. A more detailed quantitative analysis of inter-event times is currently underway and will be presented in a separate publication.

Figure 2. The Q-Q plot of the word “formation” picked from Darwin text. (a) The Q-Q plot for univariate Hawkes process and (b) The Q-Q plot for multivariate Hawkes process.

Figure 3. The Q-Q plot of the word “selection” picked from Darwin text. (a) The Q-Q plot for univariate Hawkes process and (b) The Q-Q plot for multivariate Hawkes process.

Another approach for validating the applicability of both univariate and multivariate Hawkes processes involves simulating these models and comparing the simulated word occurrence signals with those observed in actual text data.

In this second approach, we first estimated the optimal values of the parameters $\mu$, $\alpha$ and $\beta$ in Equations (3) and (4) using maximum likelihood estimation (MLE) in order to evaluate the effectiveness of the univariate Hawkes process. According to the principle of maximum likelihood, the fitted univariate Hawkes process is expected to most accurately replicate the occurrence signal of the target word. Using the estimated parameters, we generated simulated word occurrence signals for the top 20 important words by running separate simulations for each corresponding univariate Hawkes process. These simulated signals were then compared with the real-word signals, with a particular focus on autocorrelation functions (ACFs). If the ACF of a simulated signal closely resembles that of the real signal, we deem the univariate Hawkes process effective for modeling that word’s occurrence pattern.

A similar procedure was followed to validate the multivariate Hawkes process. First, we estimated the parameters of a 20-dimensional multivariate Hawkes process using MLE, based on the word occurrence signals of the top 20 important words. Next, we simulated this multivariate model to generate 20 corresponding virtual signals. Finally, we compared the ACFs derived from the simulated and actual signals for each word to assess the adequacy of the multivariate Hawkes process in capturing the dynamics of word occurrences.

As mentioned before, it is evident that using only 20 words is insufficient to simulate a comprehensive representation of the central topics in a large document. Although this limitation may be addressed in future work as will be discussed in the last section, it is accepted in the present study due to the high computational cost associated with the MLE procedures.

3. Results and Discussion

3.1. Validity Confirmation of Hawkes Processes

We describe here the results of the second approach for validating our modeling. As outlined above, we primarily compare the characteristic quantities derived from simulated signals with those obtained from word occurrence signals observed in real texts. The comparisons focused on the following four quantities:

• The total number of occurrences of the word throughout the text.
• The relaxation time τ of the ACF, derived from the fitting parameter of the KWW function (Equation (9)).
• The shape parameter β of the ACF, also obtained from the fitting parameter of the KWW function.
• The Bayesian Information Criterion (BIC), calculated during the fitting of the KWW function to ACFs.

The results of these comparisons are displayed in plots, where the vertical axis represents one of the four characteristic quantities calculated from simulated signals, while the horizontal axis represents the corresponding quantity calculated from observed occurrence signals in the real text. Namely, plots (a)–(d) in Figure 4, Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9 present comparisons between four quantities derived from real word occurrence signals (horizontal axes) and those obtained from simulated univariate Hawkes processes (vertical axes). If linear relationships are observed in these plots, it indicates that the univariate Hawkes processes, as evaluated using Equation (4), are sufficiently valid for representing real word occurrence signals. Similarly, plots (e)–(h) in Figure 4, Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9 display analogous comparisons using simulated multivariate Hawkes processes. Note that we used 20-dimensional Hawkes processes in plots (e)–(h) in Figure 4, Figure 5, Figure 6, Figure 7 and Figure 9, while 8-dimensional Hawkes process was employed in plots (e)–(h) in Figure 8. Then, if linear relationships are well established in these plots, it confirms that the multivariate Hawkes processes evaluated via Equation (6) offer adequate modeling of word occurrences in real texts. Moreover, if the linear relationships are more prominent in plots (e)–(h) than in plots (a)–(d), it can be concluded that modeling with multivariate Hawkes processes provides a more accurate representation than the univariate counterparts.

Figure 4. Characteristics comparing real and simulated word occurrence signals for Darwin text. In all plots, the horizontal axis represents characteristics of the real signals, while the vertical axis represents characteristics derived from the simulated signals. Characteristics derived from simulated signals generated from univariate Hawkes processes are shown in plots (a–d), while those from the 20-dimensional multivariate Hawkes process are shown in plots (e–h). The red lines represent the function $y=x$. Comparisons between plots (a) and (e) for the number of occurrences, between plots (b) and (f) for the relaxation time $\tau$, between plots (c) and (g) for the shape parameter $\beta$, and between plots (d) and (h) for the Bayesian Information Criterion (BIC).

Figure 5. Plots conveying the same interpretation as Figure 4 are presented for the Einstein text. In all plots, the horizontal axis represents characteristics of the real signals, while the vertical axis represents characteristics derived from the simulated signals. Characteristics derived from simulated signals generated from univariate Hawkes processes are shown in plots (a–d), while those from the 20-dimensional multivariate Hawkes process are shown in plots (e–h). The red lines represent the function $y=x$. Comparisons between plots (a) and (e) for the number of occurrences, between plots (b) and (f) for the relaxation time $\tau$, between plots (c) and (g) for the shape parameter $\beta$, and between plots (d) and (h) for the Bayesian Information Criterion (BIC).

Figure 6. Plots conveying the same interpretation as Figure 4 are presented for the Faraday text. In all plots, the horizontal axis represents characteristics of the real signals, while the vertical axis represents characteristics derived from the simulated signals. Characteristics derived from simulated signals generated from univariate Hawkes processes are shown in plots (a–d), while those from the 20-dimensional multivariate Hawkes process are shown in plots (e–h). The red lines represent the function y = x. Comparisons between plots (a) and (e) for the number of occurrences, between plots (b) and (f) for the relaxation time $\tau$, between plots (c) and (g) for the shape parameter $y=x$, and between plots (d) and (h) for the Bayesian Information Criterion (BIC).

Figure 7. Plots conveying the same interpretation as Figure 4 are presented for the Freud text. In all plots, the horizontal axis represents characteristics of the real signals, while the vertical axis represents characteristics derived from the simulated signals. Characteristics derived from simulated signals generated from univariate Hawkes processes are shown in plots (a–d), while those from the 20-dimensional multivariate Hawkes process are shown in plots (e–h). The red lines represent the function $y=x$. Comparisons between plots (a) and (e) for the number of occurrences, between plots (b) and (f) for the relaxation time $\tau$, between plots (c) and (g) for the shape parameter $\beta$, and between plots (d) and (h) for the Bayesian Information Criterion (BIC).

Figure 8. Plots conveying the same interpretation as Figure 4 are presented for the Paine text, except that an 8-dimensional multivariate Hawkes process was used. In all plots, the horizontal axis represents characteristics of the real signals, while the vertical axis represents characteristics derived from the simulated signals. Characteristics derived from simulated signals generated from univariate Hawkes processes are shown in plots (a–d), while those from the 8-dimensional multivariate Hawkes process are shown in plots (e–h). The red lines represent the function $y=x$. Comparisons between plots (a) and (e) for the number of occurrences, between plots (b) and (f) for the relaxation time $\tau$, between plots (c) and (g) for the shape parameter $\beta$, and between plots (d) and (h) for the Bayesian Information Criterion (BIC).

Figure 9. Plots conveying the same interpretation as Figure 4 are presented for the Plato text. In all plots, the horizontal axis represents characteristics of the real signals, while the vertical axis represents characteristics derived from the simulated signals. Characteristics derived from simulated signals generated from univariate Hawkes processes are shown in plots (a–d), while those from the 20-dimensional multivariate Hawkes process are shown in plots (e–h). The red lines represent the function $y=x$. Comparisons between plots (a) and (e) for the number of occurrences, between plots (b) and (f) for the relaxation time $\tau$, between plots (c) and (g) for the shape parameter $\beta$, and between plots (d) and (h) for the Bayesian Information Criterion (BIC).

Word occurrence signals were simulated 20 times for both univariate and multivariate Hawkes processes, resulting in 20 vertical values for each characteristic quantity corresponding to each word. The upper end of the error bars represents the maximum value among these 20 features, while the lower end denotes the minimum value. The vertical positions of the blue circles in Figure 4, Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9 indicate the averages of these 20 values.

If the simulated signal for a given word closely matches the real signal, the four characteristic quantities across plots (a)–(h) should align near or directly along the $y=x$ which is represented in red lines in Figure 4, Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9. Thus, we can conclude that the modeling using univariate or multivariate Hawkes processes is effective if the blue circles are positioned along the straight line indicating direct proportionality. To assess the degree of linear correspondence between the vertical and horizontal quantities, we calculated correlation coefficients, which are displayed in the titles of the plots in Figure 4, Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9. Consequently, the validation criterion is how closely the correlation coefficient approaches 1.

Our primary focus is to evaluate the extent to which the effectiveness of modeling improves when transitioning from the univariate Hawkes process to the multivariate Hawkes process. To assess this, we compared the description accuracies of the univariate Hawkes process and the multivariate Hawkes process across Figure 4, Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9: comparisons between plots (a) and (e) for the number of occurrences, between plots (b) and (f) for the relaxation time $\tau$, between plots (c) and (g) for the shape parameter $\beta$, and between plots (d) and (h) for the Bayesian Information Criterion (BIC).

These comparisons clearly demonstrate that the multivariate Hawkes process significantly enhances modeling effectiveness compared to the univariate Hawkes process. This improvement is evidenced by the scatter plots of the four characteristic quantities in Figure 4, Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9 being distributed closer to the red straight lines $y=x$ in the case of the multivariate Hawkes process, with correlation coefficients approaching 1. Table 8 shows such improvement. In the table, each coefficient indicates the degree of correlation between the real and simulated word occurrence signals for the given quantity. The simulated signals are generated using either univariate or multivariate Hawkes processes, and both cases are presented in the table for comparison. Indeed, Table 8 confirms that the correlation coefficient for each quantity is closer to 1 when modeling with the multivariate Hawkes process is applied.

Table 8. Comparison of the correlation coefficients for the four characteristic quantities.

Text	Number of Occurrences		Relaxation Time τ		Shape Parameter β		BIC
	Univariate	Multivariate	Univariate	Multivariate	Univariate	Multivariate	Univariate	Multivariate
Darwin	0.936	0.983	0.078	0.176	0.117	0.287	−0.395	0.832
Einstein	0.821	0.885	0.237	0.670	0.212	0.329	0.419	0.437
Faraday	0.483	0.501	0.016	0.407	0.033	0.431	−0.245	0.395
Freud	0.840	0.990	0.151	0.439	0.220	0.168	−0.115	0.496
Paine	0.357	0.891	0.253	0.764	0.021	0.621	0.257	0.361
Plato	0.251	0.901	−0.300	0.184	−0.149	0.873	0.207	0.624

Table 9 provides supplementary information on Table 8. That is, Table 9 summarizes the p-values assessing statistical significance in the uncorrelated (independence) test. Cells corresponding to significance levels below 10%, 5%, and 1% are shaded with progressively darker tones to indicate increasing levels of significance. As shown by the increased shading in the lower half of the table, the multivariate Hawkes process yields stronger correlations between the simulated quantities and the observed signal values. This trend suggests that the multivariate Hawkes process more accurately captures and reproduces the real signals of word occurrences.

Table 9. p-value of the test for lack of correlation (independence test).

Process	Book	Number of Occurrences	$\mathit{\tau }$	$\mathit{\beta }$	BIC
univariate Hawkes	Einstein	9.37 $\times {10}^{-6}$	3.15 $\times {10}^{-1}$	3.69 $\times {10}^{-1}$	6.56 $\times {10}^{-2}$
	Darwin	1.43 $\times {10}^{-9}$	7.43 $\times {10}^{-1}$	6.23 $\times {10}^{-1}$	8.44 $\times {10}^{-2}$
	Faraday	3.10 $\times {10}^{-2}$	9.47 $\times {10}^{-1}$	8.91 $\times {10}^{-1}$	2.97 $\times {10}^{-1}$
	Freud	3.52 $\times {10}^{-6}$	5.25 $\times {10}^{-1}$	3.51 $\times {10}^{-1}$	6.29 $\times {10}^{-1}$
	Paine	3.86 $\times {10}^{-1}$	5.46 $\times {10}^{-1}$	9.61 $\times {10}^{-1}$	5.39 $\times {10}^{-1}$
	Plato	2.85 $\times {10}^{-1}$	1.99 $\times {10}^{-1}$	5.30 $\times {10}^{-1}$	3.81 $\times {10}^{-1}$
multivariate Hawkes	Einstein	2.12 $\times {10}^{-7}$	1.22 $\times {10}^{-3}$	1.57 $\times {10}^{-1}$	5.41 $\times {10}^{-2}$
	Darwin	1.24 $\times {10}^{-14}$	4.57 $\times {10}^{-1}$	2.21 $\times {10}^{-1}$	5.36 $\times {10}^{-6}$
	Faraday	2.43 $\times {10}^{-2}$	7.51 $\times {10}^{-2}$	5.81 $\times {10}^{-2}$	8.49 $\times {10}^{-2}$
	Freud	8.21 $\times {10}^{-17}$	5.28 $\times {10}^{-2}$	4.79 $\times {10}^{-1}$	2.60 $\times {10}^{-2}$
	Paine	3.01 $\times {10}^{-3}$	5.11 $\times {10}^{-2}$	1.01 $\times {10}^{-1}$	3.80 $\times {10}^{-1}$
	Plato	6.10 $\times {10}^{-8}$	4.38 $\times {10}^{-1}$	5.17 $\times {10}^{-7}$	3.29 $\times {10}^{-3}$

Cells corresponding to significance levels below 10%, 5%, and 1% are shaded with progressively darker tones to indicate increasing levels of significance.

The results described above are unsurprising, given the nature of the document, which presents concepts each of which is explained by using multiple key terms. These key terms are frequently repeated throughout the explanation, creating inherent correlations among them. As a result, the multivariate Hawkes process effectively captures these relationships. In contrast, the univariate Hawkes process is limited to capturing self-correlation, meaning it only reflects the correlations found within the occurrence signal of a single word.

3.2. Hawkes Graphs

Hawkes graphs are directed graphs that intuitively represent multivariate Hawkes processes. In these graphs, each node corresponds to a specific type of event in the multivariate Hawkes process, while each edge indicates how one type of event (represented at the root of the arrow) influences or enhances another type of event (located at the tip of the arrow). Examples of Hawkes graphs are presented in Figure 10 and Figure 11, which illustrate the multivariate Hawkes processes optimized to model word occurrence signals in the Paine text and the Plato text, respectively. Each node represents the occurrence events of an important word, and its corresponding background intensity, ${\mu }_i$, as defined in Equation (5), is also displayed within the node.

Figure 10. An example of a Hawkes graph representing the Paine text. Values of $a_{ij}$ are shown near the corresponding arrows, and values of ${\mu }_i$ are indicated within their respective nodes. Node and arrow colors are shaded according to the magnitudes of $a_{ij}$ and ${\mu }_i$.

Figure 11. Example of a Hawkes graph representing the Plato text. Values of $a_{ij}$ are shown near the corresponding arrows, and values of ${\mu }_i$ are indicated within their respective nodes. Node and arrow colors are shaded according to the magnitudes of $a_{ij}$ and ${\mu }_i$.

In a multivariate Hawkes process, various types of events mutually excite one another, and the magnitude of their mutual excitation is quantified by the following coefficient [32]:

$$a_{ij}={\displaystyle \frac{{\alpha }_{ji}}{{\beta }_{ji}}}=\underset{0}{\overset{\infty }{\int }}{\alpha }_{ji}\exp \left(-{\beta }_{ji}t\right)dt$$

(11)

Note that ${\alpha }_{ji}$ quantifies the intensity by which the occurrence of the $i$-th word enhances the likelihood of the $j$-th word occurring, ${\beta }_{ji}$ represents a decay rate of this enhancement effect. Thus, the integral of Equation (11) expresses how strong the occurrence of $i$-th word induce the occurrence of $j$-th word in which the duration of enhancement is taken into account. The order of the subscripts $i$ and $j$ is deliberately reversed in $a_{ij}$ (on the left hand side) to make it intuitively clear that they are quantities in the $i$→$j$ direction. This coefficient measures how much the event of type-$i$ enhances the occurrence of the type-$j$ event. In Figure 10 and Figure 11, the values of $a_{ij}$ defined by Equation (11) are displayed alongside the edges, while the numbers within the nodes represent the background intensities ${\mu }_i$. The colors of the nodes and arrows are shaded such that higher values of ${\mu }_i$ and $a_{ij}$ result in darker nodes and arrows. To improve readability, edges with $a_{ij}<0.01$ are not shown in the graphs.

By constructing a Hawkes graph for a given text using the optimized parameter values of the multivariate Hawkes process, it becomes easier to infer the central concepts or notions of the text. For instance, based on Figure 10, we identify “time” as one of the central notions in the Paine text. This conclusion is drawn from observing that five edges associated with the “time” node are incoming arrows, while only one edge flows outward from this node. The predominance of incoming arrows indicates that the concepts represented by the source nodes are predominantly used to describe the notion of “time.”

3.3. Finding Important Notions in Texts

The idea for inferring the central notions of a text from the Hawkes graph, as mentioned above, can be refined as follows. The difference between the inflows into the $i$-th node and the outflows from the same node is defined as:

$$\Delta a_i=\sum _{j=1}^d{a}_{ji}-\sum _{j=1}^d{a}_{ij}.$$

(12)

This quantity intuitively represents the net extent to which the notion corresponding to the $i$-th node is explained by other important notions.

The values of $\Delta a_i$ are displayed in Table 10, which ranks the importance of each notion in descending order for each text. The most important notion for each text—repress ented by the top-ranked word in Table 10—can be considered appropriately identified based on the content of each text, as will be discussed below.

Table 10. The difference between inflows and outflows, Δa_i, for each important notion.

Darwin				Einstein				Freud
Node	Δai	TF-IDF	Degree Centrality	Node	Δai	TF-IDF	Degree Centrality	Node	Δai	TF-IDF	Degree Centrality
species	1.1250	0.02013	0.14127	relativity	0.8213	0.03277	0.06735	dream	2.0732	0.04786	0.204243
period	0.7721	0.00393	0.03561	law	0.7765	0.00911	0.05204	life	0.7262	0.00357	0.030397
selection	0.2300	0.00726	0.04724	light	0.7715	0.00749	0.05230	form	0.4541	0.00267	0.016945
cross	0.2029	0.00407	0.00763	body	0.6530	0.00873	0.05689	analysis	0.3601	0.00556	0.028069
plant	0.1817	0.00627	0.01453	system	0.5132	0.00780	0.06582	thought	0.3168	0.00537	0.025352
theory	0.1802	0.00270	0.02078	time	0.4872	0.00757	0.06505	process	0.2235	0.00559	0.022377
part	0.1603	0.00374	0.02609	point	0.2187	0.00803	0.03903	state	0.1755	0.00288	0.022377
class	0.0453	0.00182	0.01148	reference	0.2175	0.00657	0.03929	day	0.1222	0.00456	0.03583
habit	−0.0095	0.00213	0.01148	place	−0.0674	0.00456	0.03265	consciousness	0.1075	0.00780	0.022248
world	−0.0652	0.00211	0.02115	space	−0.0780	0.00595	0.05306	idea	0.0348	0.00292	0.015522
structure	−0.0659	0.00461	0.03357	motion	−0.2615	0.00895	0.06429	sleep	−0.0057	0.00556	0.03389
character	−0.0694	0.00357	0.02275	co-ordinate	−0.3237	0.00324	0.00536	wish	−0.0366	0.00778	0.051093
island	−0.1285	0.00270	0.01214	principle	−0.3451	0.00471	0.03648	formation	−0.1510	0.00308	0.012676
group	−0.1385	0.00537	0.01788	field	−0.3476	0.00712	0.04388	activity	−0.2573	0.00437	0.020437
variety	−0.1548	0.00616	0.01403	line	−0.3938	0.00402	0.02041	system	−0.3008	0.00297	0.018238
instinct	−0.1672	0.00240	0.01170	distance	−0.4206	0.00417	0.03495	psychic	−0.3025	0.01609	0.048118
organ	−0.2482	0.00322	0.01577	theory	−0.4460	0.01541	0.07985	interpretation	−0.3127	0.00393	0.017074
hybrid	−0.4656	0.00503	0.00756	position	−0.4774	0.00409	0.02934	content	−0.5062	0.00473	0.030268
production	−0.5281	0.00218	0.00545	velocity	−0.5738	0.01345	0.05281	work	−0.7205	0.00310	0.023412
formation	−0.8565	0.00278	0.01366	equation	−0.7238	0.01056	0.01250	child	−2.0006	0.00371	0.021731
Faraday				Paine				Plato
Node	Δai	TF-IDF	Degree Centrality	Node	Δai	TF-IDF	Degree Centrality	Node	Δai	TF-IDF	Degree Centrality
substance	0.5210	0.00919	0.04986	time	0.6041	0.00736	0.04880	man	0.4548	0.00692	0.07140
combustion	0.4490	0.01754	0.05252	continent	0.2181	0.01216	0.03944	state	0.4078	0.00779	0.06930
experiment	0.3492	0.00901	0.04410	government	0.1954	0.01069	0.04425	truth	0.2285	0.00263	0.02842
2action	0.3126	0.00607	0.04454	england	0.0994	0.01022	0.03944	word	0.1753	0.00221	0.01009
air	0.3018	0.01569	0.12076	britain	0.0010	0.00920	0.03110	nature	0.1652	0.00453	0.04351
piece	0.2421	0.00564	0.04675	king	−0.2363	0.01255	0.04703	opinion	0.0921	0.00257	0.01930
part	0.1817	0.00468	0.03989	power	−0.3051	0.00543	0.03843	right	0.0841	0.00284	0.02632
water	0.1041	0.01463	0.11700	america	−0.5766	0.01140	0.03540	men	0.0721	0.00562	0.04877
candle	0.0081	0.02617	0.10193					justice	0.0658	0.00870	0.03895
carbon	−0.0058	0.01522	0.04675					pleasure	0.0433	0.00323	0.01579
gas	−0.0113	0.01305	0.05894					art	0.0269	0.00285	0.02105
vessel	−0.0677	0.00686	0.03302					knowledge	−0.0127	0.00289	0.03105
atmosphere	−0.0687	0.00554	0.03412					city	−0.0275	0.00404	0.02605
jar	−0.0965	0.01050	0.04387					question	−0.0759	0.00243	0.02018
oxygen	−0.1220	0.02059	0.06160					reason	−0.0761	0.00258	0.02597
flame	−0.1878	0.01444	0.07755					friend	−0.1050	0.00239	0.01746
hydrogen	−0.2463	0.01396	0.04188					principle	−0.1391	0.00221	0.01947
iron	−0.4329	0.00532	0.04033					soul	−0.2651	0.00595	0.04825
heat	−0.5906	0.00856	0.05096					god	−0.3241	0.00378	0.01263
acid	−0.6399	0.01540	0.04742					life	−0.7905	0.00371	0.03886

Clearly, the notion of “species” in the Darwin text, “relativity” in the Einstein text, and “dream” in the Freud text are all closely tied to the main themes of the respective documents, serving as the central concepts each text seeks to explain. In the Faraday text, the principles of chemistry and physics are explored through the fascinating subject of candles. The most significant topic in the Faraday text is the investigation of the substances that constitute candles and how they transform during combustion. Thus, it is reasonable to identify “substance” as the most critical concept in the text. The Paine text is a groundbreaking pamphlet published during the American Revolution, emphasizing the urgency of declaring independence. In this text, the author employs the notion of “time” as a motivational framework for driving transformative action. Hence, “time” is regarded as central because the author explains the nature of the current “time” within this context. In the Plato text, the notion of “man” takes center stage because the author’s philosophical reflections on justice, governance, and enlightenment are deeply rooted in his understanding of human nature and elaborated upon through this perspective.

To assess the validity of using $\Delta a_i$ as a measure of word importance in texts, we compared $\Delta a_i$ against other widely used metrics for word significance. Specifically, we evaluated the TF-IDF scores and degree centralities [42] for each of the key words used in the multivariate Hawkes process modeling of the considered texts, and investigated how well the values of $\Delta a_i$ aligned with these metrics. The calculated values of TF-IDF and degree centrality are listed in Table 10 and resulting correlation coefficients between $\Delta a_i$ and TF-IDF, as well as between $\Delta a_i$ and degree centrality, are presented in Table 11 for each book.

Table 11. Correlation coefficients between $\Delta a_i$ and TF-IDF and those between $\Delta a_i$ and Degree Centrality for each book.

Text	Δai and TF-IDF	Δai and Degree Centrality
Darwin	0.64292	0.73385
Einstein	0.32818	0.48593
Faraday	−0.04482	0.16863
Freud	0.62809	0.64965
Paine	−0.15629	0.48761
Plato	0.29113	0.34874

Notably, the correlation between $\Delta a_i$ and degree centrality was substantially higher than that between $\Delta a_i$ and TF-IDF, as confirmed by a paired t-test ($p=0.04433$). This outcome is reasonable given the nature of the metrics: TF-IDF is derived purely from the frequency statistics of individual words across documents, whereas degree centrality incorporates information about word co-occurrence. Thus, both degree centrality and $\Delta a_i$ reflect inter-word relationships, which likely explains their stronger correlation.

A key distinction between $\Delta a_i$ and degree centrality lies in their graph structures: $\Delta a_i$ is computed based on directed graphs—where edges, as shown in Figure 10 and Figure 11, are represented by arrows—while degree centrality, when used to measure word importance, is typically calculated using undirected graphs.

Although a comprehensive strategy for evaluating $\Delta a_i$ via comparison with other metrics is not currently available, the observed strong correlation with degree centrality supports the reliability of $\Delta a_i$ as an indicator of word importance.

3.4. Advantages of Analyzing Texts with Multivariate Hawkes Processes

Based on the results outlined so far, the following advantages of utilizing the multivariate Hawkes process have been established:

• As demonstrated in Figure 2 and Figure 3, and Figure 4, Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9, the accuracy of modeling word occurrence signals in documents is significantly improved, making it more reliable when using the multivariate Hawkes process compared to the univariate one.
• A Hawkes graph can be generated from the parameter values obtained through the analysis using the multivariate Hawkes process. This facilitates an intuitive understanding of the relationships among the concepts that emerge in the document.
• The importance of each concept identified in a text can be assessed using the optimized parameters of the multivariate Hawkes process. The most significant concepts suggested for each text analyzed in this study are confirmed to be valid when considering the content of the text.

4. Conclusions

The occurrence signals of important words in six texts (one historical pamphlet and five renowned academic books) were modeled using univariate and multivariate Hawkes processes. The modeling procedure for the univariate Hawkes process was conducted as follows. First, optimized parameter values for the univariate Hawkes process were determined using maximum likelihood estimation (MLE) based on the occurrence signals of a given word in a considered text. Next, word occurrence signals were generated by simulating the univariate Hawkes process with optimized parameters. Finally, the validity of the univariate Hawkes process modeling was confirmed by verifying that the four characteristic quantities derived from the simulated word occurrence signals closely matched those obtained from the real signals observed in the text.

The modeling procedure for the multivariate Hawkes process followed a similar approach. First, optimized parameters for the $d$-dimensional multivariate Hawkes process were evaluated using MLE based on the occurrence signals of $d$ important words, where $d$ represents the number of important words analyzed simultaneously with the multivariate Hawkes process. In this study, the typical value of $d$ was set to be 20. Next, occurrence signals of $d$ important words were simultaneously simulated using the multivariate Hawkes process with the optimized parameters. Finally, the validity of the multivariate Hawkes process modeling was confirmed by ensuring that the four characteristic quantities derived from the simulated word occurrence signals closely matched those obtained from the real signals observed in the text.

By validating the modeling with univariate and multivariate Hawkes processes, we observed that the correlation coefficients of the four characteristic quantities between real and simulated signals are notably closer to one in the case of the multivariate Hawkes process compared to the univariate one. This result demonstrates that the multivariate Hawkes process provides greater accuracy in describing word occurrence signals.

Additional advantages of the multivariate Hawkes process include its ability to intuitively represent the relationships among concepts described in a document through a Hawkes graph and to evaluate the importance of words in the text based on their relationships. For instance, the most important words in the analyzed documents were inferred using the optimized parameters of multivariate Hawkes processes, and these findings were confirmed to be valid.

A key limitation of this study is the relatively low dimensionality used in the multivariate Hawkes process. Specifically, we typically set the dimension to 20, which is insufficient to capture the mutual enhancement of word occurrences in large-scale documents. As previously discussed, this constraint stems from the impractical computational cost of maximum likelihood estimation (MLE) in dimensions beyond 20. Efficient computation using recursive algorithms [43] may help mitigate this issue, and future research exploring this direction would be a compelling subject of investigation. Other potential directions for future research utilizing the multivariate Hawkes process, which proved useful in this study, include:

• Treating the matrix a_ij (defined by Equation (11)) as an adjacency matrix and applying graph theory methods, such as spectral clustering.
• Analyzing texts using advanced stochastic processes, such as the autoregressive-type Hawkes process [44]—a discretized variant of the standard Hawkes model—which is particularly well-suited for high-dimensional analyses (e.g., with more than 20 dimensions) due to its lower computational complexity. An alternative promising framework is the “Flexible Triggering Kernels” model [45], which effectively encodes event history and captures localized excitation dynamics beyond traditional decay-based kernels.