Long-Range Dependence in Word Time Series: The Cosine Correlation of Embeddings

Wieczyński, Paweł; Dębowski, Łukasz

doi:10.3390/e27060613

This is an early access version, the complete PDF, HTML, and XML versions will be available soon.

Open AccessArticle

Long-Range Dependence in Word Time Series: The Cosine Correlation of Embeddings

by

Paweł Wieczyński

^1,†

and

Łukasz Dębowski

^2,*,†

¹

Independent Researcher, 80-180 Gdańsk, Poland

²

Institute of Computer Science, Polish Academy of Sciences, 01-248 Warsaw, Poland

^*

Author to whom correspondence should be addressed.

^†

These authors contributed equally to this work.

Entropy 2025, 27(6), 613; https://doi.org/10.3390/e27060613

Submission received: 14 May 2025 / Revised: 7 June 2025 / Accepted: 8 June 2025 / Published: 9 June 2025

(This article belongs to the Special Issue Complexity Characteristics of Natural Language)

Download Versions Notes

Abstract

We analyze long-range dependence (LRD) for word time series, understood as a slower than exponential decay of the two-point Shannon mutual information. We achieve this by examining the decay of the cosine correlation, a proxy object defined in terms of the cosine similarity between word2vec embeddings of two words, computed by an analogy to the Pearson correlation. By the Pinsker inequality, the squared cosine correlation between two random vectors lower bounds the mutual information between them. Using the Standardized Project Gutenberg Corpus, we find that the cosine correlation between word2vec embeddings exhibits a readily visible stretched exponential decay for lags roughly up to 1000 words, thus corroborating the presence of LRD. By contrast, for the Human vs. LLM Text Corpus entailing texts generated by large language models, there is no systematic signal of LRD. Our findings may support the need for novel memory-rich architectures in large language models that exceed not only hidden Markov models but also Transformers.

Keywords: word embeddings; mutual information; cosine similarity; power laws; stretched exponential; long-range dependence

Share and Cite

MDPI and ACS Style

Wieczyński, P.; Dębowski, Ł. Long-Range Dependence in Word Time Series: The Cosine Correlation of Embeddings. Entropy 2025, 27, 613. https://doi.org/10.3390/e27060613

AMA Style

Wieczyński P, Dębowski Ł. Long-Range Dependence in Word Time Series: The Cosine Correlation of Embeddings. Entropy. 2025; 27(6):613. https://doi.org/10.3390/e27060613

Chicago/Turabian Style

Wieczyński, Paweł, and Łukasz Dębowski. 2025. "Long-Range Dependence in Word Time Series: The Cosine Correlation of Embeddings" Entropy 27, no. 6: 613. https://doi.org/10.3390/e27060613

APA Style

Wieczyński, P., & Dębowski, Ł. (2025). Long-Range Dependence in Word Time Series: The Cosine Correlation of Embeddings. Entropy, 27(6), 613. https://doi.org/10.3390/e27060613

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Article metric data becomes available approximately 24 hours after publication online.

Article Menu

Long-Range Dependence in Word Time Series: The Cosine Correlation of Embeddings

Abstract

Share and Cite

Article Metrics

Further Information

Guidelines

MDPI Initiatives

Follow MDPI