Next Article in Journal
LSE-CVCNet: A Generalized Stereoscopic Matching Network Based on Local Structural Entropy and Multi-Scale Fusion
Previous Article in Journal
Entropy Generation Optimization in Multidomain Systems: A Generalized Gouy-Stodola Theorem and Optimal Control
Previous Article in Special Issue
Scaling Laws in Language Families
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
This is an early access version, the complete PDF, HTML, and XML versions will be available soon.
Article

Long-Range Dependence in Word Time Series: The Cosine Correlation of Embeddings

by
Paweł Wieczyński
1,† and
Łukasz Dębowski
2,*,†
1
Independent Researcher, 80-180 Gdańsk, Poland
2
Institute of Computer Science, Polish Academy of Sciences, 01-248 Warsaw, Poland
*
Author to whom correspondence should be addressed.
These authors contributed equally to this work.
Entropy 2025, 27(6), 613; https://doi.org/10.3390/e27060613
Submission received: 14 May 2025 / Revised: 7 June 2025 / Accepted: 8 June 2025 / Published: 9 June 2025
(This article belongs to the Special Issue Complexity Characteristics of Natural Language)

Abstract

We analyze long-range dependence (LRD) for word time series, understood as a slower than exponential decay of the two-point Shannon mutual information. We achieve this by examining the decay of the cosine correlation, a proxy object defined in terms of the cosine similarity between word2vec embeddings of two words, computed by an analogy to the Pearson correlation. By the Pinsker inequality, the squared cosine correlation between two random vectors lower bounds the mutual information between them. Using the Standardized Project Gutenberg Corpus, we find that the cosine correlation between word2vec embeddings exhibits a readily visible stretched exponential decay for lags roughly up to 1000 words, thus corroborating the presence of LRD. By contrast, for the Human vs. LLM Text Corpus entailing texts generated by large language models, there is no systematic signal of LRD. Our findings may support the need for novel memory-rich architectures in large language models that exceed not only hidden Markov models but also Transformers.
Keywords: word embeddings; mutual information; cosine similarity; power laws; stretched exponential; long-range dependence word embeddings; mutual information; cosine similarity; power laws; stretched exponential; long-range dependence

Share and Cite

MDPI and ACS Style

Wieczyński, P.; Dębowski, Ł. Long-Range Dependence in Word Time Series: The Cosine Correlation of Embeddings. Entropy 2025, 27, 613. https://doi.org/10.3390/e27060613

AMA Style

Wieczyński P, Dębowski Ł. Long-Range Dependence in Word Time Series: The Cosine Correlation of Embeddings. Entropy. 2025; 27(6):613. https://doi.org/10.3390/e27060613

Chicago/Turabian Style

Wieczyński, Paweł, and Łukasz Dębowski. 2025. "Long-Range Dependence in Word Time Series: The Cosine Correlation of Embeddings" Entropy 27, no. 6: 613. https://doi.org/10.3390/e27060613

APA Style

Wieczyński, P., & Dębowski, Ł. (2025). Long-Range Dependence in Word Time Series: The Cosine Correlation of Embeddings. Entropy, 27(6), 613. https://doi.org/10.3390/e27060613

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Article metric data becomes available approximately 24 hours after publication online.
Back to TopTop