Token Trajectories as Knowledge Representation in Large Language Models

Abstract

The subject of this paper is the problem of knowledge revealed by the emergence of advanced natural language processing technology, called large language models (LLMs). It proposes an interpretation of the appearance of this phenomenon as an emergent effect of a complex system, such as LLMs. The meaning of this interpretation is based on the observation of the computational trajectory of computational units (vectors) which, on the other hand, represent discrete semantic units, tokens. To justify the autonomy and relevance of this interpretation of knowledge, the paper invokes the concept of discursive space, which allows, in particular, for the reliance on language as a container of knowledge and for the generalization of the idea of knowledge to other semantic environments. To describe more general units of knowledge, the strong version of the theory proposes the introduction of the institution of gnosemes, of which discourse is a special case. Gnosemes find application in the case of LLMs. Due to the different ontical contexts of the entities described, the paper proposes a transdisciplinary approach, the axis of which is the field of social sciences.

1. Introduction

The subject of this paper is the problem of knowledge revealed by the emergence of advanced natural language processing technology, called large language models (LLMs). It proposes an interpretation of the appearance of this phenomenon as an emergent effect of a complex system, such as LLMs. The meaning of this interpretation is based on the observation of the computational trajectory of computational units (vectors) which, on the other hand, represent discrete semantic units, tokens. To justify the autonomy and relevance of this interpretation of knowledge, the paper invokes the concept of discursive space, which allows, in particular, for the reliance on language as a container of knowledge and for the generalization of the idea of knowledge to other semantic environments.

LLMs technology is based on many original algorithmic solutions, two of which seem to be particularly important, i.e., embedding technology [1] and attention technology [2]. Both of these technologies allowed for an unprecedented possibility of digitizing text, preserving its semantic content, based on the probability distribution model of words in the text.

Specifically, LLMs can generate intelligible text in response to input text, although the functional nature of the output text can vary. This text can be intentionally formulated as a variety of cognitive tasks, such as language comprehension, reading comprehension, natural language inference, logical reasoning, language translation, etc. Various benchmarks used to evaluate LLMs provide good insight into the types of tasks. Naveed et al. provide a comprehensive list of these benchmarks. It also contains systematic information on the development of LLMs, their variants, the solutions on which they are based, etc. [3] p. 29.

Among the tasks performed by LLMs, Naveed et al. explicitly include tasks directly related to the ability to use knowledge. However, it should be emphasized that every cognitive task is based on acquired knowledge. In this situation, in the case of LLMs, knowledge exists as an immanent property of an intelligible artificial text. This issue is discussed in more detail in [4].

The text digitization resulting from the operation of LLMs is based on the idea of language models, which have been supplemented by artificial neural network technologies. As Jurafsky and Martin write “a wide variety” of language models was proposed already in the 1980s and 1990s [5] p. 56. According to the definition given by Jurafsky and Martin, the language model is a model “that assign probabilities to sequences of words” [5] p. 30. Aggarwal more simply explains that “the language model is used to create a probabilistic representation of the text” [6] p. 4. Charniak writes even more generally that “a language model is a probability distribution over all strings in a language” [7] p. 71. Goodfellow et al. provide a similar definition: ‘A language model defines a probability distribution over sequences of tokens in a natural language” [8] p. 461. This solution, supplemented by artificial neural network technology to record this probability distribution, led to the current development of LLMs, e.g., [1]. A comprehensive description of this process is provided by [5,9].

In this way, a goal that had been the subject of many years of effort was achieved: obtaining a useful digital representation of a text that preserved its semantic content, i.e., digitizing the semantics of the text. This solution was considered the digitization of language, that is, the symbolic system realized in practice through text, which serves as the basic component of the digitization procedure. In this sense, the reference to language concerns the phase of its existence that Saussure called parole, in distinction from the language understood as a system with its own rules—langue [10].

The issue of representation is fundamental here. Digitizing the semantics of text (language) means numerically representing its discrete, atomic semantic units, preserving their internal logic. In the case of LLMs, these units are tokens, allowing one to bypass many of the problems associated with words, commonly treated as the smallest semantic units of text. A commonly used solution is the algorithm BPE (Byte Pair Encoding) proposed by Gage et al. [11] and applied in the context of natural language processing procedures, i.e., neural machine translation by Senrich et al. [12].

In the case of LLMs, the aforementioned numerical representation is a set of numbers interpreted as coordinates in a multidimensional space, written in the algebraic form of vectors, which also allows for geometric interpretation. The procedure to assign appropriate numerical values to individual tokens is called embedding—a solution of historical significance was presented by Mikolov et al. [1]. The vector values are calculated as weights for the hidden layer of a neural network trained on extensive text corpora to determine the probability of adjacent words appearing in the corpus. This corpus contains real text instances, most often downloaded from the Internet. This method is based on the so-called distributional hypothesis, which assumes that the words closest to it are crucial for the meaning of a given word (semantic unit) [13,14,15]. The number of neurons in this layer determines the dimension of the vector and the number of dimensions (and coordinates) of the space built on the obtained vectors.

The embedding procedure allows for the storage of semantic relations between tokens (words) as vectors of a given dimension, which allows for computations involving these vectors as equivalent to semantic operations. For example, the geometric distance between vectors is also a measure of their semantic similarity. Very high vector dimension values, on the order of thousands or more, allow for the detection of very subtle semantic relations between tokens (words).

The so-called Attention technology turned out to be a fundamental and key development of the embedding procedure [2]. It became the basis of the LLMs family of LLMs called Transformers, which also includes the model used here as an example, TinyLlama [16]. This model is a type called autoregressive, which also includes the well-known GPT model. It recursively generates the output text by adding one token at a time in a single pass of the algorithm until it is completed. Attention technology, identified here as the second breakthrough solution after embedding, allows the storage of long-range semantic relations of a very subtle and diverse nature in the token vector [2]. It achieves this effect in two main steps: first, by repeatedly repeating the attention head procedure, based on trained, specially selected matrices (in the case of TinyLlama, there are 32 heads). Second, the entire set of head procedures is repeated multiple times in the so-called layers (in the case of TinyLlama, 22 times). The number of heads and layers is usually kept secret in the case of major commercial LLMs. The open source DeepSeek V-3 model contains 128 heads and 61 layers [17]. Using the last version of Llama open model Llama 3.1 405B these numbers are, respectively, 136 and 126 [18]. The Attention procedure uses precomputation to uniquely determine the relative position of tokens using the positional embedding procedure.

The cognitive tasks described above, achieved effectively by LLMs and requiring knowledge, are realized by a variety of computational procedures, among which embedding and attention procedures play a key role. They rely on the identification and use of numerous complex inferences, realized as short- and long-range semantic relations between discrete semantic units, such as tokens. These procedures are computationally very complex due to the unprecedented large amounts of data processed. These procedures simultaneously, in a certain, as yet unexplained way, become a place for knowledge retention and articulation. It should be noted that this latter effect was surprising and can historically be associated with the emergence of OpenAI’s GPT-2 model [19].

It should be emphasized that procedures for extracting semantic relations directly from textual/linguistic resources, interpreted as knowledge, appeared already in the 1960s. Their main goal was to acquire and manage knowledge, based on identifying certain inherent principles of reasoning, e.g., Schank’s conceptual dependency theory [20], semantic networks by Collins and Quillian [21], frames (frame-based systems) by Minsky [22], scripts by Schank and Abelson [23], and ontologies interpreted in a fundamental way by Gruber [24]. All of them were based on the interpretation of words or structures built from them, made arbitrarily by humans, and therefore fundamentally non-objective.

2. Discourse and Discursive Space

From the perspective of the interpretation of knowledge proposed here, it is important that the interdependence of language and knowledge has been the subject of extensive reflection and has an extensive analytical literature and also provides a way of understanding knowledge on which the approach presented here is based. A comprehensive introduction to this issue can be found in [25].

There is no doubt that the use of LLMs in the form of consumer solutions based on them such as ChatGPT or Claude is very common, e.g., [26], which means that the social context of research is perfectly justified. This fact provides practical support for the key theoretical assumption underlying the construction of LLMs, i.e., the reliance on language, realized as text, whose social context is obvious. The problem of knowledge considered in the context of LLMs thus becomes, mutatis mutandis, a social problem.

This interpretation of knowledge is based on the concept of discourse, which is also the subject of numerous studies. The two most important approaches from the perspective of this interpretation are represented by Theun van Dijk and Michel Foucault. The former is the founder of the field of research called discourse studies, inaugurated with a publication in 1997 [27]. They are based on the assumption that discourse can serve as a universal analytical tool, applicable in all disciplines of the humanities and social sciences. Despite its “fuzzy” nature, three key properties of discourse can be formulated: (1) it is based on the use of language, (2) it is a tool intended to communicate beliefs (cognition), and (3) it implements interactions based on social relations [28] p. 2.

The development of this concept leads van Dijk to propose an advanced idea of knowledge: “Discourse presupposes (semantic) situation models of events talked about, as well as (pragmatic) context models of the communicative situation, both construed by the application of general, socially shared knowledge of the epistemic community” [29]. He understands knowledge as a social variable, which means that it is inherently dispersed and not necessarily coherent. It is precisely defined as a local construct, the nature of which can vary and is revealed only at the highest level of generalization, i.e., prevailing beliefs or convictions. It can then be identified as a distinguishing feature of a social group, a knowledge community. This type of knowledge emerges as various discourses based on different knowledge, communicated in different ways. Knowledge is reflected in language as an interpersonal and historical event, simultaneously detaching itself from the human being as a crucial subject. In doing so, it loses its metaphysical foundation, abandoning the claim to truth. This approach opens the possibility of perceiving knowledge “technically” as a certain effect of the functioning of the language (discourse).

However, the most comprehensive and insightful interpretation of discourse in the context of knowledge is provided by Michel Foucault, who is based on the same assumption as van Dijk, i.e., he views it as a social phenomenon. However, he gives it a much more fundamental epistemic character: “an entire field is set free. A vast field, but one that can be defined nonetheless: This field is made up of the totality of all effective statements (whether spoken or written), in their dispersion as events, and in the occurrence that is proper to them. Before approaching, with any degree of certainty. A science, or novels, or political speeches, or the oeuvre of an author, or even a single book, the material with which one is dealing is, in its raw, neutral state, a population of events in the space of discourse in general. One is led therefore to the project of a pure description of discursive events as the horizon for the search for the unities that form within it.” [30] p. 26.

Foucault precisely identifies the realm of discourse in the empirical sense as the real event of formulating an utterance. At the same time, he emphasizes the lack of distinction between utterances realized as speech and writing. This distinction has become the subject of separate theoretical reflection, but it is invalidated in the context provided by Foucault. The only instance of discourse is the utterance, which can be interpreted as an individual subjective occurrence (performance, l’execution) of a linguistic act. Ferdinand de Saussure, a classic of linguistics, calls this phase of language parole [10]. This occurrence is a spontaneous linguistic activity carried out by its users.

Foucault uses the concept of a whole set of linguistic occurrences, whose common feature is their belonging to the space of discourses. In the year of publication of the cited book, i.e., 1969 [31], such a concept could have referred to a collection of a large number of elusive or even impossible-to-grasp elements, practically impossible to utilize. This situation is changing with the emergence and development of digital communication, particularly the Internet. This communication, subject to universal registration and access, even public, is the source of a vast resource of utterances, ultimately taking the form of text.

On the other hand, discourse serves as a container for knowledge. Foucault writes: “there is no knowledge without a particular discursive practice; and any discursive practice may be defined by the knowledge that it forms” [30] p. 182. In this way, the level of discourse is inevitably linked to the level of knowledge, but this link is neither simple nor straightforward. It is achieved through a complex interplay of discourses, which become a key element in the existence of knowledge. In its full realization, this interaction creates a vibrant field of knowledge of a vast cultural scope, which Foucault called the episteme (l’épistémè). This approach allows one to situate the presented reasoning against the backdrop of fundamental social historical analysis [32]. For the present considerations, the key factor remains the level of social practice, realized as an utterance that is both a reflection of a certain discourse (certain discourses) and its (their) production.

Foucault gives the fullest and at the same time the most condensed description of discourse in his lecture delivered at the Collège de France in December 1970, published a year later [33]. To grasp the ambiguous nature of discourse, Foucault proposes basing its description on four key rules (principes, règles), which simultaneously serve as potential characteristics of discourse, defining the perspective for its understanding. This is also a technical way to avoid direct description, which is necessarily discursive and, therefore, subject to fundamental limitations, primarily the historically and socially contextualized realization in the utterance.

The four rules mentioned are as follows: a principle of reversal (principe de renversement), the principle of discontinuity (principe de discontinuité), the principle of specificity (principe de spécificité), and the principle of exteriority (règle de l’extériorité) [34] p. 67. In their light, discourse appears first as diluted and elusive, secondly, as forming a discontinuous and intersecting structure, and thirdly, as an agent producing meanings autonomously and anticipating the movement of interpretation. The fourth rule formulates the necessity and possibility of observing only the external effects of discourse, its structural external shape [35].

The described approach of Foucault allows one to take the next step in our reasoning and interpret discourse based on the concept of dynamical space [36], leading to the idea of so-called discursive space. According to this concept, “discursive space is an n-dimensional dynamical space in which discourses, which are autonomous instances of knowledge, run in time trajectories describing the real state of knowledge in the subject that they concern” [37]. An extension of this idea is the construction of a theory of knowledge based on the idea of discursive space. According to this theory, it was possible to formulate a definition of knowledge limited (weak) to its linguistic (discursive) manifestation: “knowledge is a set of discourses contained in an n-dimensional manifold that can be interpret locally as a discursive space” [37].

This definition was then extended ontologically to other potential carriers of meaning, for which discourse is merely a special case. This extension enabled the formulation of a concept of knowledge space, based on its theoretically anticipated carriers, called gnosemes: “the knowledge space is an n-dimensional dynamic space in which gnosemes, which are autonomous instances of knowledge, run the trajectories in time describing the real state of knowledge in the form they concern”. Consequently, the definition of knowledge has also been generalized: “knowledge is a set of states of gnosemes in an n-dimensional manifold that can be interpreted locally as a knowledge space” [37]. The definitions quoted should be understood as conceptual foundations for the interpretation of knowledge in the context of LLMs.

3. Methods

The key object of interpretation here, knowledge, is understood as a social phenomenon. Its metaphysical and epistemological context serves only as a necessary inspiration. This means that knowledge is a real entity and manifests itself empirically in many ways. Social sciences provide the appropriate tools for analyzing such a phenomenon, which can also naturally draw on the analytical achievements of other sciences. This is necessary in the case of phenomena such as LLMs which are strongly embedded in the social circumstances of their development (e.g., technological, historical, political, cultural, etc.). In this situation, research must be transdisciplinary in nature in the sense given to it by Patricia Leavy: “Transdisciplinarity is an approach to conducting social research. Transdisciplinary research practices are issue- or problem- centered approaches to research that prioritize the problem at the center of research over discipline-specific concerns, theories, or methods. Transdisciplinary research is responsive to (public) needs” [38] p. 14. Another definition of transdisciplinarity is given by Mark Burgin and Wolfgang Hofkirchner: “Transdisciplinary research encompasses problems from different disciplines but goes on a higher level than catch of these disciplines goes in other words, transdisciplinarity treats problems that are at once between the disciplines, across the different disciplines, and beyond any of the individual disciplines involved. It is aimed at understanding of broad spheres of the world directed at the unity of knowledge.” [39] p. 3.

To illustrate the discussed knowledge phenomenon, a small but fully functional LLM called TinyLlama was used, a 1.1B language model pretrained on around 1 trillion tokens, based on the decoder-only Transformer [16]. This model has 22 layers, while the so-called hidden size, i.e., the number of dimensions of the computational space in each layer, is 2048. The selected model represents the family of autoregressive models, which includes, in addition to the Llama 2 model [40], which is the basis of the TinyLlama model, also the GPT model family and the Claude model family. This means that the model predicts the next element in the sequence based on elements that have already appeared, and the full computational trajectory is applied only to the last token in the series. However, in the Attention mechanism, the computation is performed for all tokens of the text processed during the inference phase.

The example presented here uses a famous fragment of text from Ludwig Wittgenstein describing language as a city: “Our language can be seen as an ancient city: a maze of little streets and squares, of old and new houses, and of houses with additions from various periods; and this surrounded by a multitude of new boroughs with straight regular streets and uniform houses” [41] p. 8. This text is 58 tokens long, obtained with the same algorithm used for Llama 2, that is, the byte pair encoding (BPE) algorithm [12], using the implementation of SentencePiece [42]. The UMAP algorithm (Uniform Manifold Approximation and Projection) [43] was used to reduce the number of dimensions, which allowed the representation of token trajectories in three-dimensional space. An illustration of this process is shown in Figure 1.

Figure 1. Example vector trajectories in the TinyLlama model. 22 multidimensional computational spaces were reduced to two-dimensional planes that represent the successive steps of calculations performed on vectors. Each of the 58 tokens has its own trajectory. A significant degree of changes in token values can be interpreted as the effect of deep semantic differences that divide individual spaces (layers). At the same time, the vector bundle maintains coherence, which means that it makes sense. The illustration clearly shows the dynamics of the trajectories.

It should be added that the presented illustration of the token trajectory is only illustrative due to the applied dimension reduction, although some general features, for example the dense shape of the trajectory stream, may have semantic meaning, for example, evidence of similar semantical denseness and consistency.

4. Propositions

We propose to base the interpretation of the phenomenon of knowledge revealed in LLMs on the concept of knowledge as a set of gnosemes, whose ontological form is the computational trajectories of vectors representing tokens, which are semantic signatures. These trajectories traverse all computational spaces of the LLM, distinguished as autonomous computational areas. In the case of the chosen model, this particularly concerns the computational layers of the Attention procedure and is initiated in the embedding procedure. Due to the specific characteristics of these trajectories, they can be considered a complex system, which allows us to interpret the knowledge present in texts generated by LLMs as an emergent effect of this system.

The characteristics of the presented computational processes allow them to be treated as a complex system according to many definitions of this system that I provided in the book [25] pp. 98–99. Here, I will quote a relatively new definition by Kinsner: “In systems theory, complexity means a large number of interacting elements with many degrees of freedom whose individual behavior could not be traced back or predicted. Such systems exhibit self-organization (emergence), thus leading to new features that did not exist before and were not programmed to evolve” [44] p. 277. To fully understand the complexity of a system in LLMs, it is important to consider that the embedding procedure processes the entire large text corpus during training. This allows for mapping a large, complex semantic system, such as language, onto a system of mutually relativized tokens. In other words, the token system inherits the uncontroversial complexity of natural language.

Humans use natural language only partially, pragmatically, that is, within the scope of their needs. Acts of language use of this kind are described, for example, by Foucault. This trend is also continued by the idea of discursive space. The weakness of this approach is the arbitrary nature of decisions regarding identified discourses. These decisions are made by humans and use language as a medium of description. Therefore, they fall into the discursive trap of lack of objectivity, which Foucault wrote extensively about, although this occurs at a higher level of meta-description. In the case of LLMs, language mapping, which, as experience shows, preserves semantic relations enabling the creation of intelligible text, is performed using numbers, allowing them to be represented using mathematical formalism, or more precisely, linear algebra, thanks to which tokens are represented by vectors. Although this formalism is not necessarily completely transparent, it avoids the aforementioned discursive (linguistic) trap.

More formally one can describe the LLM inference process as manifold $\mathcal{M}$, which is a disjoint union of the space $V_i$, for $i$ from 1 to n. Each space $V_i$ has a characteristic computational procedure ${\mathcal{C}}_i$ associated with it, which, based on specified data or transformation rules, determines a point $v_i$ in this space. A point $v_i$ is the result of computing the position of a certain vector in the space $V_i$ according to a specified algorithm (transformation rule) included in the model. The point $v_i$ is not chosen arbitrarily, but is the result of computations specific to the space $V_i$. The set of points $v_i$ forms a trajectory $\gamma$, i.e., a path in the structure of $\mathcal{M}$.

The concept of a manifold should not be understood here in a formal and mathematical way. It is used in its earlier and more original sense. It refers to Riemann’s intuition, but is based on the observations of Deleuze, Bergson, and Husserl. This speculative construction is much broader and more fundamental than the narrow, mathematical definition. In Riemann’s case, it results from the search for a broader genre than space. For Deleuze, it allows us to grasp the cognitive structure of language, as conceived by Foucault. For Bergson, it captures the temporal, interpenetrating character of successive states. Husserl transfers the problem to the level of ontology, which, by the way, can serve as an ontological interpretation of gnosemes; his approach does not exclude a mathematical interpretation. A trajectory in this context is simply a temporally ordered change of values: coordinates, but arbitrarily abstract ones. This issue was extensively described by the author in [25] p. 154, passim.

Consequently, according to a theory of knowledge based on the concept of discursive space, from a formal perspective, LLMs’ computations are a manifold composed of multiple computational spaces, specifically repetitions of the Attention procedure. These spaces are traversed by vectors whose trajectories represent individual semantic signatures, i.e., tokens, functioning in the reality of natural language. These trajectories can be described as gnosemes, the smallest essential components of knowledge. According to this interpretation, computational processes (vector trajectories) can be represented as distinct entities belonging to the reality of knowledge, which in this way acquires its own ontological representation.

Based on Kinsner’s definition, a dynamic trajectory system can be considered complex because (1) it has a large number of interacting tokens with many degrees of freedom, understood as a large number of computational parameters, and (2) whose individual behavior could not be traced back or predicted due to the nonlinear type of computation performed on a vector, an algebraic representation of a token. In this situation, the phenomenon of emergence described by Kinsner, i.e., a kind of self-organization, becomes possible, which, due to the effective mapping of deep language semantics in the LLM system, can be interpreted as knowledge.

It should also be added that from a social perspective, such knowledge exhibits many distinctive features that significantly deviate from the classical, platonic definition. However, they characterize the phenomenon of knowledge viewed through the lens of twentieth-century experience. Knowledge takes on a number of eccentric properties. First, it is deeply contextualized relative to existing spontaneous textual resources, the volume of which leads to their being perceived as representative of “circulation” of knowledge, that is, its retention and articulation. This knowledge is pragmatic in nature and is not subject to claims of truth. It can also be contradictory to other instances of knowledge. In this sense, it realizes and confirms observations related to knowledge interpreted in the context of social sciences as “dispersed, diversified, and decentralized”, leading to the necessity of using the concept of knowledge in the plural: knowledges [45].

5. Discussion

The issue of knowledge in the context of LLMs does not have a very extensive literature on the subject; the literature develops over time, in line with the development of LLMs, and has only become more numerous in recent years. One of the first articles by Grand et al. dealt exclusively with the embedding procedure in the context of meaning formation [46]. Petroni et al. “present an in-depth analysis of the relational knowledge already present (without fine-tuning) in a wide range of state-of-the art pretrained language model” [47]. Wang et al. propose to optimize the handling of factual knowledge in LLMs [48]. Allen-Zhu et al. have no doubt that LLMs store knowledge, but argue that it must be richly represented in the pretraining phase [49]. Recent research addresses more general issues. Kim & Thorne study LLMs as holistic epistemic systems, which is not fully confirmed [50]. Heersmink et al. analyze the phenomenology and epistemology of chatbots [51]. At the same time, the number of publications on knowledge in the context of LLMs is increasing. Several such publications are listed by Fierro et al. [52]. Their list can be supplemented with the following newer publications: [53,54,55,56,57].

Of particular note in the context of the approach proposed here is the concept of Sarfati et al. [56]. They point to the still poorly understood ability of LLMs to perform complex cognitive tasks and cite previous attempts to interpret them. At the same time, they propose a new approach that interprets LLMs as a complex dynamical system. Specifically, they are interested in “in the trajectories, or ‘lines of thought’ (LoT), that embedded tokens realize in the latent space when passing through successive transformer layers” [56] p. 2. These trajectories represent an entity the authors called “lines of thought.” Sarfati et al. based their analysis on token trajectories on a similar proposal by Aubry et al. [58]. Thanks to the development of this interpretative concept, it has become possible to “interpolate token dynamics to a continuous time in the form of a stochastic differential equation” [56] p. 9. This statement allowed us to draw the conclusion that their approach “reveals a surprising dimensionality reduction of token embeddings. It suggests, notably, that the true ‘meaning’ of embeddings is contained within individual variability” [56] p. 10.

It should be noted that Sarfati et al. presented a very interesting formal interpretation of token behavior in LLM, which also allowed conclusions regarding semantic dynamics. Their research is grounded in the widely cited literature on specific parts of reasoning. However, a significant weakness of the publication is the semantic context, which Sarfati et al. reduce to the construct “lines of thought (LoT)”, which is directly identified with token trajectories, but completely devoid of cognitive (semantic) interpretation. The concept of LoT can only be treated as a metaphorical reference to the reality of cognitive processes.

In the case of this text, the situation is the opposite. Its starting point is the field of semantics, extended to the level of knowledge. To this end, it draws on the earlier theory of discursive space, which is a dynamical space that describes a complex system, language, perceived as a set of discourses traversing trajectories within this space. Knowledge is understood as the set of trajectories of these discourses. At the same time, discourse can be treated as a special case of epistemic dynamics, which can also rely on carriers other than discourse.

The extrapolation of the idea of discursive space into the areas of these carriers, which are not limited in nature, leads to the concept of gnosemes as universal carriers of knowledge [37]. In the description presented here, the role of gnosemes is played by token trajectories in a manifold composed of computational spaces traversed by each token, as described in the propositions section. This interpretive proposal is a continuation of an idea that originated much earlier. In particular, it applies the idea of dynamic space to its description and was developed independently of the research described in the paper by Sarfati et al.

The process occurring in the case of LLM differs from the discursive process in two important ways: first, the semantic value of the vector does not result from a human decision, but is the result of statistical calculations concerning the probability distribution of the occurrence of individual tokens (word analogues). This circumstance excludes weaknesses introduced into the process by humans, such as arbitrariness, subjectivity, or determination caused by particular linguistic competences, despite sophisticated methods of discourse research, e.g., [59,60,61,62].

Second, due to its numerical and computational nature, the semantic value of the vector in individual computational phases is uninterpretable and has a hidden character. However, the inference process can be observed as a whole, the computational effects of which are semantic in nature, because the vector value is ultimately returned as a word.

A key factor in the hypothesis posed here is that tokens never occur in isolation. Like words, they are parts of a larger whole, which is the text. This situation is also visible in Figure 1, where a cluster of computational trajectories of vectors representing all the tokens in the sample text appears. This cluster is concentrated, indicating the semantic coherence of its elements, which confirms the meaningfulness and coherence of the sample text.

Each inference process is based on a wave of tokens, represented as a set of vectors that create a kind of cohesive cloud that changes shape in every computational space. In social practice, we encounter a large number of actually implemented inferences, resulting from the enormous popularity of solutions such as ChatGPT (OpenAI), Claude (Anthropic) and Gemini (Google). This means that the dynamic environment of these numerous “clouds” thus created is based on a large number of tokens and can take on a complex nature, and therefore can also be a source of emergent effects.

It should be emphasized that the understanding of semantic processes that underlie the interpretation of knowledge presented here, following Foucault, as described in the Section 2, Discourse and Discursive Space, is based on perceiving them as the effects of internal inferences. In the case of the discursive theory of knowledge, the role of basic units of knowledge, and therefore also of meaning, is fulfilled by discourses which can be generalized to universal representatives of knowledge, i.e., gnosemes, in order to also encompass situations of the existence of knowledge of an extralinguistic nature.

A similar approach to the formation of meanings within LLMs, i.e., based on the mechanism of inference rather than reference, is presented by Piantadosi and Hill [63], based on the earlier theory of meaning proposed by Brandom [64]. In particular, the computational processes occurring in the attention phase can be directly considered as mappings of semantic inferences existing in the text. However, Piantadosi and Hill, in emphasizing the role of inference, rely primarily on conceptual role theory and a cognitivist approach, whereas here the main analytical context is social.

A similar situation, i.e., the tension between the social and subjective approaches, occurs in connection with advanced attempts to analyze the extremely interesting field of human–machine relations in the context of the former’s cognitive capabilities. The problem becomes especially pressing when it concerns the hard, formal descriptions that underpin science, especially within the exact and natural sciences. The problem is epistemological in nature and examines the processes of practicing science in the context of human cognitive capabilities, even when intelligent machines participate in these processes [65]. A characteristic phenomenon in this field is the need to make compromises, specific trade-offs, seeking a balance between the ability of humans to adopt content and the efficiency of the provided diagnoses or interpretations, generated by the mechanism of scientific research itself.

In this context, it can be assumed that the operation of LLMs as knowledge-generating tools is fundamentally based on a trade-off between the cognitive capabilities of a human in terms of internalizing texts facing a machine and a certain, inherent way in which the machine generates content. This approach introduces a deep dependence of human–machine collaboration on this hidden and unknown factor, crucial for human knowledge acquisition or, conversely, for knowledge articulation by a machine, e.g., [66,67]. However, this perspective is alien to the argument presented here, which is situated somewhat earlier and concerns inferences that generate answers based solely on social data provided by language, practically outside the sphere of cognition of the subject participating in generating knowledge. For this reason, although interesting, the trade-offs perspective is not considered here.

6. Conclusions

The problem of knowledge is a well-known and long-standing problem in artificial intelligence research. There are two historical approaches to this problem: symbolic intelligence relies on symbolic representations of knowledge, while computational intelligence relies on numerical representations of knowledge [68]. The spectacular achievements of LLMs are realizations of the second approach. Knowledge is processed using open computational processes, although within these processes it is uninterpretable (hidden).

The paper, in its search for entities that can represent the smallest possible units of knowledge processed by LLMs, is based on the earlier concept of discursive space [37]. It proposes that discourse be considered the smallest unit of knowledge. This approach was proposed by Michel Foucault in the 1960s, e.g., [31,33]. The discursive theory of knowledge extends this approach and proposes that units of knowledge should be considered the trajectories of discourses in dynamic spaces whose dimensions are defined through explicit analysis of texts concerning a given discourse. Knowledge, in this interpretation, is a cloud of trajectories of all discourses under consideration. At the same time, the discursive theory of knowledge provides for the possibility of extrapolating the dynamics of discourse to other, non-linguistic articulations of knowledge, as described in [37]. This generalization was provided with the concept of gnosem, whose ontological basis in the case of LLMs is the token trajectory.

By analogy to the “analog” concept of discursive knowledge, in the concept presented here, knowledge is the sum of trajectories traversed by tokens in computational spaces constructed in each practical realization of the inference phase in LLMs. Such processes are social in nature because they emerge as a mass use of LLMs as specific language processors. The resulting, actually existing cloud of tokens, different in each realization but composed of the same tokens, is the state of knowledge in the social sense.

Complexity, similar to the theory of discursive knowledge, has been proposed as an approach to grasping the nature of these processes. It also comes indirectly from the nature of language itself.

This interpretation requires further elaboration, taking into account such issues as the multiplicity of languages that underlie the creation of tokens, the dispersion of acts of creating trajectories, the large number of texts whose processing by LLMs produces these trajectories, and finally the passage of time, which must induce semantic modifications of individual tokens, which is an effect of the social and historical nature of language.