Mathematical Modeling and Generalization Inference Mechanisms of Large Language Models Under Transformer Architecture

Abstract

Large language models (LLMs) built upon the Transformer architecture have achieved remarkable performance in natural language understanding, text generation and logical reasoning, while their internal working mechanisms remain poorly interpreted. This paper establishes a systematic mathematical analysis framework tailored for decoder-only Transformer LLMs, based on linear algebra, tensor analysis, probability theory, information theory, optimization dynamics and geometric deep learning. We conduct rigorous mathematical modeling and theoretical deduction on core modules including word embedding, position encoding, self-attention, feed-forward networks, training optimization and generalization reasoning, and explore the mathematical nature of semantic representation, contextual correlation, knowledge storage and logical inference within models. In this paper, we strictly distinguish between classic established Transformer theories and our original mathematical derivations and conclusions. Distinct from existing fragmented theoretical studies, this work presents six targeted novel contributions beyond conventional Transformer theories: (1) we construct the first full-process unified mathematical framework covering all core modules and the entire lifecycle of Transformer-based LLMs; (2) we provide strict mathematical proof to verify that single-head self-attention is essentially a kernel weighted average operation in reproducing kernel Hilbert space and derive the low-rank and sparse properties of attention weights; (3) we establish a high-dimensional non-convex optimization dynamics model for pre-training and mathematically prove that model training converges to flat local minima; (4) we derive a tighter upper bound of generalization error and quantify the quantitative relationship among model parameters, sequence length, training data scale and generalization performance; (5) we characterize the latent space as a low-curvature smooth Riemannian manifold and model logical reasoning as geometric transformation on this manifold; (6) we design multi-group controlled experiments on mainstream datasets to quantitatively validate all above theoretical conclusions. This paper further summarizes the inherent mathematical limitations of current Transformer LLMs and proposes feasible theoretical optimization paths, referring to state-of-the-art research published from 2021 to 2026. The outcomes of this research can provide solid mathematical theoretical support for improving model interpretability, optimizing network structures and boosting practical performance, and facilitate the transition of LLM research from empirical engineering practice to theory-driven development.

1. Introduction

1.1. Research Background and Significance

In recent years, large language models represented by GPT series, LLaMA series, and PaLM series have completely reconstructed the technical paradigm of natural language processing, demonstrating unprecedented performance in tasks such as human–computer dialog, code generation, mathematical proof, and multimodal interaction [1]. Benefiting from massive unlabeled text corpora, powerful computing resources and iterative Transformer architecture design, modern LLMs have gradually exhibited emergent capabilities that small-scale models do not possess, which further promotes their large-scale deployment in academia and industry [2]. The current mainstream language models are based on the Transformer decoder architecture, which achieves efficient encoding and representation of language rules, world knowledge, and logical relationships through self-supervised pre-training of massive text data [3].

The basic structural definitions, fundamental formulas of self-attention, feed-forward networks and standard training objectives of Transformer are well-established classic theories proposed in prior work. However, despite the continuous iteration and upgrading of model engineering applications, with parameter quantities jumping from billions to trillions, the corresponding basic theoretical research is seriously lagging behind. The inherent mathematical mechanism of the model has not yet formed a unified, rigorous, and systematic theoretical system, and many core questions have not been answered: What is the mathematical essence of self-attention mechanism to achieve semantic correlation? What geometric characteristics does the semantic representation of model latent space possess? How to quantify the dynamic laws of gradient propagation and parameter optimization during the training process? What is the mathematical root of model emergence ability, generalization performance, and illusion phenomenon [4]?

Deconstructing the intrinsic mechanisms of large language models from a mathematical perspective is a key path to breaking through the technical bottlenecks of existing models, solving core problems such as poor interpretability, insufficient generalization ability, extremely high computational complexity, and inference illusions [5]. At present, most industrial optimization of LLMs relies on empirical tuning rather than theoretical guidance, which leads to high trial-and-error costs and limits the upper limit of model performance [6]. On the one hand, rigorous mathematical modeling and formula derivation can quantify the representation ability, computational complexity, and theoretical boundaries of each module of the model, providing a theoretical basis for the lightweight and efficient design of the model structure; on the other hand, it can break the “black box” of the model, clearly reveal the underlying mathematical logic of semantic understanding, knowledge storage, and logical reasoning and promote the engineering iteration of the large language model from empiricism to scientific innovation guided by theory [7]. Therefore, conducting research on the intrinsic mathematical mechanisms of large language processing models has both important theoretical and engineering application value.

1.2. Current Research Status at Home and Abroad

In the past five years, the international academic community has conducted extensive research on the mathematical mechanisms of large language models, resulting in multi-dimensional and multi-directional theoretical achievements. In terms of mathematical modeling of self-attention mechanisms, researchers have demonstrated through tensor analysis and reproducing kernel Hilbert space theory that self-attention is essentially a kernel weighted average operation in high-dimensional space. They have derived upper bounds on the representational ability and complexity constraints of multi-head attention and analyzed the mathematical relationship between attention weight and semantic similarity [8].

A large number of follow-up studies further expanded the kernel-based interpretation of self-attention. Some works combined kernel PCA to analyze the feature projection characteristics of self-attention and the correlation between value matrices and Gram matrix eigenvectors, but failed to complete the full mathematical proof of the equivalence between softmax self-attention and RKHS kernel averaging [9]. Other studies proposed a multi-head attention kernel combination theory based on statistical mechanics, yet they only focused on attention path analysis under thermodynamic limits and ignored the low-rank and sparse inherent properties of attention matrices [10]. In engineering practice, lightweight Transformer variants adopt kernel functions to optimize attention modules for long-sequence acceleration, while prioritizing computational efficiency over rigorous theoretical derivation of kernel mathematical properties [11].

In terms of model training optimization theory, scholars have explained the convergence characteristics of the loss function of large language models through non-convex optimization dynamics analysis, proving that model training tends to converge to flat minima, which have better generalization performance; meanwhile, based on information theory, a balance between model memory and generalization was constructed [12]. Related research compared the convergence differences between Gaussian kernel and softmax kernel in Transformer training and explored the coupling relationship between optimization landscape and latent space geometry [13]. Existing research has confirmed that adaptive optimizers such as Adam and AdamW can effectively suppress gradient noise and accelerate convergence, but relevant studies have not linked optimization dynamics with the overall mathematical characteristics of the full model [14].

At the level of generalization and inference theory, based on statistical learning theory, researchers have derived an upper bound on the generalization error of large language models and quantified the mathematical relationship between model parameter quantity, training data quantity, and generalization performance; traditional generalization bounds derived from VC-dimension are widely used in early deep learning, but such bounds are too loose and cannot adapt to ultra-large-scale LLMs with hundreds of billions of parameters [15]. Through algebraic topology and manifold learning, the semantic distribution pattern of the hidden space in the model was revealed and logical reasoning was modeled as a geometric transformation of the hidden space.

Existing studies only conducted local geometric analysis on individual network layers of latent manifolds, without constructing a global systematic geometric framework [16]. Some research adopted hyperbolic geometry to model hierarchical semantic structures, but they focused on representation learning rather than connecting manifold characteristics with model generalization and reasoning behaviors [17]. The Linear Representation Hypothesis (LRH) regards latent features as linear directional vectors, which cannot explain the continuous geometric changes during multi-step logical reasoning [18]. Most existing manifold-related interpretability works stay at qualitative description and visual analysis, lacking unified quantitative evaluation metrics [19]. In terms of model interpretability research, the internal processes of mathematical reasoning and knowledge encoding in the model were dismantled through methods such as neuron analysis and causal mediation analysis, and the mathematical roles of core functional modules were identified [20].

Current LLM interpretability research is mainly divided into two branches: local component probing and model behavior analysis. Most probing methods rely on empirical experiments without solid mathematical support; hallucination detection and reasoning evaluation technologies mostly depend on external retrieval or self-consistency strategies and cannot explain internal causes from the perspective of model mathematical mechanisms [21]. The existing theoretical interpretability literature is fragmented and fails to form a complete mathematical system covering the whole lifecycle of LLMs from module computation to training and reasoning [22].

Although existing research has achieved phased results, there are still significant shortcomings: firstly, there is a lack of a unified mathematical framework covering the entire lifecycle of the model, resulting in fragmented theoretical achievements and a failure to form a systematic system; secondly, the formula derivation of the core module is not complete enough, the rigor of the reasoning process is insufficient, and the mathematical proof depth of key mechanisms is lacking; thirdly, there is insufficient integration of the latest cutting-edge research results, which fails to fully reflect the research progress on the mathematical mechanisms of major language models in the past five years.

Against this background, this work distinguishes itself from prior studies: we retain standard Transformer basic definitions and classic formulas as foundations, and focus on completing original systematic mathematical modeling, strict proofs, extended theoretical derivations and quantitative geometric characterizations that have not been fully established in existing literature. This article aims to address the above issues by constructing a comprehensive and full-process mathematical analysis framework and completing rigorous formula derivation and mechanism explanation of the core modules of the large language model.

1.3. Research Content and Framework

This article takes the decoder-only Transformer architecture as the research object and systematically conducts research on the intrinsic mathematical mechanism of large language processing models. The core research content is as follows:

(1)

Sort out the mathematical basic theories related to the large language model, including tensor operations, probability information theory, optimization theory, manifold geometry, etc., and construct the theoretical analysis foundation for the entire text.

(2)

Adopt theoretical derivation and formal proof methods to complete mathematical modeling of each core module of Transformer. We deduce standard formulas, analyze operational mechanisms, and prove the mathematical properties of word embedding, position encoding, self-attention, feed-forward network, residual connection, and layer normalization step by step.

(3)

Establish high-dimensional non-convex optimization dynamic equations for model training. Combining chain rule of derivative and gradient back propagation theory, we analyze the probability essence of self-supervised pre-training, derive loss function and gradient propagation rules, and explore the convergence characteristics of model parameters.

(4)

Elucidate the mathematical essence of model semantic representation and inference, analyze the hidden space semantic manifold structure and geometric transformations in the inference process and derive generalization error constraints by using statistical learning theory and manifold geometric analysis methods.

(5)

Analyze the existing mathematical limitations of the large language model and propose targeted theoretical optimization and structural improvement paths based on international cutting-edge research.

The paper follows the logical sequence of “mathematical foundation → module modeling → training mechanism → inference generalization → limitation outlook” and is divided into eight chapters.

Overall, this research combines theoretical mathematical modeling, strict formal proof, quantitative experimental verification and comparative analysis as the main research methods, integrating theoretical deduction and empirical experiments to ensure the reliability and practicability of all conclusions.

1.4. Innovations of This Paper

This paper focuses on the mathematical modeling and generalization inference mechanisms of Transformer-based large language models and breaks through the existing theoretical framework from the perspectives of mathematical modeling, mechanism analysis, generalization bounds, geometric characterization, and experimental verification. The main innovations are summarized as follows:

It should be noted that the basic viewpoints including RKHS interpretation of self-attention, the correlation between flat minima and generalization, and Riemannian manifold description of latent representations have been discussed in existing literature. Different from these published foundational conclusions, this paper focuses on systematic integration, rigorous extended derivation, quantitative refinement and experimental validation and forms differentiated original contributions as described below:

(1)

A unified, full-process mathematical analysis framework covering word embedding, position encoding, self-attention, feed-forward network, training optimization, and generalization reasoning is constructed for the first time. While separate studies have explored individual modules such as RKHS-based self-attention or manifold features, most existing works are fragmented. This paper organically unifies tensor operation, reproducing kernel Hilbert space, optimization dynamics and manifold geometry into one complete theoretical system covering the whole lifecycle of LLMs, and solves the long-standing fragmentation problem in existing theoretical research.

(2)

The strict mathematical proof that the self-attention mechanism is equivalent to kernel weighted average regression in reproducing kernel Hilbert space is completed. Prior studies only put forward the conceptual equivalence of self-attention and RKHS kernel operation without complete step-by-step mathematical derivation and property verification. Beyond the existing RKHS interpretation, this paper further rigorously proves three core properties (positive definiteness, boundedness and monotonicity) of the exponential kernel and quantitatively derives the inherent low-rank and sparse characteristics of attention weights, which supplements the missing rigorous proof chain in previous research.

(3)

A high-dimensional non-convex optimization dynamics model for large language model pre-training is established. The conclusion that model training converges to flat minima is a well-known finding in the community. On this basis, this paper further deduces the complete gradient propagation law of residual connection and layer normalization, quantifies the convergence rate difference among mainstream optimizers (SGD, Adam, AdamW), and links flat minima with the overall manifold characteristics of latent space, forming an extended dynamic analysis system that existing studies do not involve.

(4)

A tight upper bound of generalization error for large language models is derived based on statistical learning theory and information theory. Traditional VC-dimension-based generalization bounds are widely used in existing work and perform poorly for large-scale LLMs. This paper derives a new tighter error bound tailored for decoder-only Transformer and quantitatively establishes the coupling relationship among model parameters, sequence length and training data scale, providing a more accurate quantitative tool than prior generalization theories.

(5)

The geometric structure of the hidden space of the large language model is characterized as a low-curvature, smooth, nearly flat semantic Riemannian manifold. Some existing studies have adopted Riemannian manifold to describe latent features, but most of them stay at qualitative description. This paper innovatively defines a set of universal quantitative metrics (Local Euclidean Deviation, Manifold Smoothness Index, Average Sectional Curvature) for manifold evaluation. Furthermore, we model multi-step logical reasoning as geodesic motion and affine transformation on the manifold and associate manifold anomalies with model hallucination phenomena, expanding the application scope of manifold theory in LLM interpretability.

(6)

Comprehensive quantitative verification is carried out through multi-scale, multi-dataset controlled experiments. Most previous theoretical papers only put forward qualitative conclusions with insufficient experimental support. This paper designs targeted ablation experiments, cross-dataset tests and practical case studies. All theoretical viewpoints including RKHS properties, optimization dynamics and manifold characteristics are fully verified by numerical results and visualization graphs, and the theoretical conclusions are further applied to long-sequence model optimization and reasoning quality evaluation, realizing the combination of theory and engineering practice.

Compared with the latest fragmented theoretical and interpretability studies on Transformers and LLMs (2023–2026), all six innovations of this paper break through the limitations of existing research. We move from local component analysis to full-process systematic modeling, from empirical phenomenological summary to rigorous mathematical proof, and from qualitative geometric description to quantitative experimental verification. The proposed unified framework also provides a general theoretical tool for follow-up LLM interpretability and optimization research.

1.5. Literature Review and Critical Comparison with State-of-the-Art Work

Based on recent studies (2023–2026) on Transformer theoretical foundations, LLM interpretability, self-attention kernel theory and latent manifold analysis, we conduct a critical review and systematically compare our work with mainstream existing research from four key dimensions: self-attention kernel modeling, latent semantic manifold characterization, optimization dynamics and generalization theory, as well as mechanistic interpretability. We clarify the research gaps of prior arts and emphasize the novelty, uniqueness and advantages of our full-process mathematical framework.

1.5.1. Review of Self-Attention Kernel-Related Research

The idea of explaining self-attention via RKHS has been proposed in the previous literature, but most relevant works only stay at conceptual elaboration without complete formal proof and in-depth analysis of kernel inherent properties.

Many studies derived self-attention from kernel PCA and revealed the connection between value matrices and Gram matrix eigenvectors, but they only focused on feature projection and did not provide complete mathematical proof for the equivalence between softmax self-attention and RKHS kernel averaging. Other works proposed a multi-head attention kernel combination theory based on statistical mechanics, yet they mainly analyzed attention paths under thermodynamic limits and ignored the inherent low-rank and sparse properties of kernel matrices. Some lightweight Transformer variants redesigned attention based on kernel functions to reduce computational complexity, but they prioritized engineering efficiency over rigorous theoretical derivation of kernel mathematical properties.

Our differentiation: Different from the above fragmented kernel research, this paper completes the strict formal proof that single-head self-attention is equivalent to exponential kernel weighted averaging in RKHS. We further systematically prove three core properties (positive definiteness, boundedness and monotonicity) of the adopted kernel, and link kernel characteristics to the low-rank and sparse nature of attention matrices. We combine theoretical derivation with dedicated quantitative experiments to form a closed loop of theory and verification, which is absent in most existing kernel-based attention studies.

1.5.2. Review of Semantic Manifold and Latent Representation Research

Using Riemannian manifold to characterize model hidden representations is not a new research direction, and many prior works have put forward relevant qualitative descriptions.

A series of studies analyzed the static geometric features of LLM latent manifolds, but they only conducted local analysis on individual layers and failed to build a systematic geometric framework for the full model. Some works adopted hyperbolic geometry to model hierarchical semantic structures, but they focused on representation learning rather than connecting manifold characteristics with model generalization and reasoning behaviors. The Linear Representation Hypothesis believes that LLM features exist as directional vectors, which cannot explain the continuous geometric changes during multi-step logical reasoning. Most manifold-related interpretability works stay at qualitative description and visual analysis, lacking unified quantitative evaluation metrics.

Our differentiation: This paper breaks the limitation of local and linear manifold research. We define three universal quantitative metrics (Local Euclidean Deviation, Manifold Smoothness Index, Average Sectional Curvature) for manifold characterization and two geometric indicators for reasoning paths. We verify manifold properties via multi-model, multi-dataset experiments and ablation studies. More importantly, we innovatively model LLM logical reasoning as geodesic motion and affine transformation on the semantic manifold, and use manifold geometric anomalies to explain hallucination phenomena. This realizes the integration of geometric modeling, quantitative evaluation and practical interpretability applications, which is a key expansion of existing manifold theories.

1.5.3. Review of LLM Optimization Dynamics and Generalization Theory

The viewpoint that model training converges to flat minima and brings better generalization performance is a well-established consensus in the field. However, prior studies rarely link this conclusion with the latent manifold geometry of LLMs.

Related work compared the convergence performance of Gaussian kernel and softmax kernel, but did not establish a connection between optimization landscape and latent manifold flatness. Existing generalization error bounds are mostly derived based on traditional VC-dimension theory, which are loose and cannot adapt to large-scale model scenarios. Most interpretability surveys summarized empirical phenomena of model training and generalization, but rarely combined optimization dynamics, manifold geometry and generalization bounds into a unified theoretical system.

Our differentiation: We construct a correlated theoretical chain: model training converges to flat minima (optimization dynamics) → flatter latent manifold (geometric feature) → lower generalization error (generalization performance). We derive a tighter generalization error bound tailored for decoder-only Transformers, and verify the rationality of the bound through comparative experiments. Different from scattered single-point research, our work unifies optimization, geometry and generalization into one full-process mathematical framework.

1.5.4. Review of Mechanistic Interpretability of LLMs

Current mainstream LLM interpretability research is divided into two branches: local component probing (neurons, attention heads) and behavior-based analysis (hallucination detection, reasoning evaluation). Most probing methods are empirical and lack underlying mathematical support; hallucination detection technologies such as self-consistency checking and retrieval augmentation rely on external resources and cannot explain internal causes from the perspective of model mathematical mechanisms. Existing theoretical studies on LLM interpretability are fragmented, lacking a complete mathematical system covering the whole model lifecycle from module calculation to training and reasoning.

Our differentiation: Targeting the “black box” problem of LLMs, we build a full-process mathematical interpretability framework. Starting from the basic formulas of each module, we gradually derive the mathematical essence of self-attention, training dynamics, manifold geometry and reasoning mechanisms. We use mathematical indicators to quantify model behaviors such as reasoning errors and hallucinations, realizing mechanism-driven interpretability rather than empirical summary. This makes up for the lack of systematic mathematical support in existing interpretability research.

2. Mathematical Fundamentals Theory

This chapter introduces classical mathematical tools and basic theories widely adopted in Transformer and deep learning research. These are established foundational knowledge and not original derivations of this paper.

2.1. Tensor and Linear Space Operations

All computations in LLMs rely on tensors and linear algebra. For a language model with embedding dimension d, sequence length L and batch size B, input data is represented as a three-order tensor:

$$X\in {\mathbb{R}}^{B\times L\times d}$$

(1)

For a single text sequence, we use a 2D matrix:

$$X=\left[\begin{array}{c}{x}_1^{\top }\\ x_2^{\top }\\ ⋮\\ x_L^{\top }\end{array}\right]\in {\mathbb{R}}^{L\times d}$$

(2)

where $x_i$ stands for the embedding vector of each token.

We use two basic metrics to measure feature relationships:

• Frobenius norm: measures the overall magnitude of feature matrices.

$$|X{|}_F=\sqrt{{\displaystyle {\sum }_{i=1}^L} {\displaystyle {\sum }_{j=1}^d}|X_{ij}{|}^2}$$

(3)

2.

Matrix inner product: reflects linear correlation between features.

$$⟨A,B{⟩}_F=\mathrm{t}\mathrm{r}\left(A^{\top }B\right)$$

(4)

We also use vector $L_2$ norm and cosine similarity to evaluate semantic similarity between tokens:

$$|x{|}_2=\sqrt{{\displaystyle {\sum }_{i=1}^d}{x}_i^2}$$

(5)

$$cos \left(x,y\right)={\displaystyle \frac{x^{\top }y}{\left|x{|}_2\right|y{|}_2}}$$

(6)

2.2. Softmax Operation and Probability Normalization

The softmax function is the core operation for implementing probability normalization in large language models, widely used in attention weight calculation and output probability generation. Its function is to map any real value vector to a distribution that conforms to the probability axiom. The mathematical expression is:

$$\sigma \left(z\right)i={\displaystyle \frac{\exp \left(z_i\right)}{\sum {j=1}^L\exp \left(z_j\right)}}, {\displaystyle {\sum }_{i=1}^L}\sigma {\left(z\right)}_i = 1$$

(7)

Among them, $z = [z_1,z_2,\dots ,z_L{]}^{\top }\in {\mathbb{R}}^L$ is the input vector and $\sigma (z{)}_i$ is the i-th element in the output probability distribution.

The softmax function is derived to obtain its Jacobian matrix, which is crucial for calculating the back propagation gradient of the model

$${\displaystyle \frac{\partial \sigma {\left(z\right)}_i}{\partial z_j}}=\begin{array}{c}\sigma {\left(z\right)}_i\left(1-\sigma {\left(z\right)}_i\right),i=j-\sigma {\left(z\right)}_i\sigma {\left(z\right)}_j, i\ne j\end{array}$$

(8)

This derivative property ensures the differentiability of softmax operations, allowing the model to achieve parameter optimization through gradient descent.

2.3. Fundamentals of Probability Theory and Information Theory

The essence of the large language model is an autoregressive sequence probability generation model, whose core goal is to fit the joint probability distribution of natural language sequences. Given a text sequence of length L, according to the probability chain rule, its joint probability can be decomposed into:

$$P(S)={\displaystyle {\prod }_{t=1}^L}P\left(x_t\right|x_1,x_2,\dots ,x_{t-1})$$

(9)

Among them, $P\left(x_t\right|x_1,\dots ,x_{t-1})$ represents the conditional probability of the appearance of the t-th word element given the first t − 1 word elements, which conforms to the sequence generation logic of natural language.

Information entropy is used to measure the uncertainty of random sequences and is a core concept in information theory, defined as:

$$H\left(X\right) =-{\sum }_{\mathcal{x}\in \mathcal{X}}P\left(x\right) log P\left(x\right)$$

(10)

Cross-entropy loss is the core loss function for pre-training and fine-tuning large language models, used to measure the difference between the predicted distribution of the model and the true data distribution. The expression is:

$$\mathcal{L}\mathrm{CE} =-{\displaystyle \frac{1}{N}}\sum {i=1}^N{\displaystyle {\sum }_{k=1}^V}{y}_{ik} log {\hat{y}}_{ik}$$

(11)

Among them, N is the number of samples, V is the size of the vocabulary, $y_{ik}$ is the one hot encoding of the true label, and ${\hat{y}}_{ik}$ is the probability value predicted by the model.

KL divergence is used to measure the difference between two probability distributions and can be expressed as:

$$D_{\mathrm{KL}}\left(P|Q\right) =-{\sum }_{x}P\left(x\right) log {\displaystyle \frac{Q\left(x\right)}{P\left(x\right)}}$$

(12)

The essence of training large language models is to minimize the KL divergence between the predicted distribution Q and the true data distribution P of the model.

2.4. Theoretical Basis of Non-Convex Optimization

The training process of a large language model is a parameter optimization process in a high-dimensional non-convex space, with the goal of minimizing the loss function $\mathcal{L}\left(\theta \right)$, where $\mathit{\theta }$ is all learnable parameters of the model.

Stochastic Gradient Descent (SGD) is a fundamental optimization algorithm, and its iterative update formula is:

$${\theta }_{t+1} = {\theta }_t-\eta ·\nabla \mathcal{L}\left({\theta }_t\right)$$

(13)

Among them, $\eta$ is the learning rate and $\nabla \mathcal{L}\left({\theta }_t\right)$ is the gradient of the loss function at the parameter ${\mathit{\theta }}_t$.

The Adam optimizer is a mainstream optimization algorithm for training large language models, which combines first-order moment estimation and second-order moment estimation to achieve adaptive learning rate adjustment. The iteration formula is:

$$\begin{array}{c}{m}_t ={ \beta }_1{m}_{t-1} + \left(1-{\beta }_1\right)\nabla \mathcal{L}\left(\theta t\right) v_t = {\beta }_{2}vt-1 + \left(1-{\beta }_2\right){\left(\nabla \mathcal{L}\left({\theta }_t\right)\right)}^2 {\hat{m}}_t = {\displaystyle \frac{m_t}{1-{\beta }_1^t}},\\ \hat{v}t = {\displaystyle \frac{v_t}{1-{\beta }_2^t}} \theta t + 1 ={ \theta }_t-\eta ·{\displaystyle \frac{{\hat{m}}_t}{\sqrt{{\hat{v}}_t} + \epsilon }}\end{array}$$

(14)

Among them, ${\beta }_1,{\beta }_2$ is the sliding average coefficient, and $ϵ$ is the numerically stable term, and avoids the denominator being zero.

2.5. Regeneration Kernel Hilbert Space and Manifold Geometry

Regenerated kernel Hilbert space (RKHS) is the core theory that analyzes the mathematical essence of self-attention mechanisms. If there are kernel functions $\kappa (x,y)$ and feature maps $\phi :\mathcal{X}\to \mathcal{H}$, they satisfy:

$$\kappa \left(x,y\right) = ⟨\phi \left(x\right),\phi \left(y\right){⟩}_{\mathcal{H}}$$

(15)

Then $\mathcal{H}$ is a reproducing kernel Hilbert space, and the self-attention mechanism can be equivalently modeled as a kernel regression operation on this space.

Riemannian manifolds are used to characterize the geometric structure of the hidden space in large language models. The hidden state vectors of the model form a low-curvature, nearly flat semantic manifold that locally satisfies Euclidean space characteristics $\mathcal{M}$. Samples with similar semantics are closer on the manifold.

3. Mathematical Modeling and Mechanism Analysis of Transformer Core Modules

In this chapter, basic module structures and primitive formulas of Transformer are inherited from the established literature. All extended mathematical analysis, equivalence proof, low-rank/sparsity derivation and in-depth mechanism interpretation are our original work.

3.1. Word Embedding and Position Encoding

3.1.1. Vector Space Mapping of Word Embedding

The mathematical essence of word embedding is the linear mapping of discrete symbols to a continuous high-dimensional vector space, achieving the numerical representation of natural language symbols. Define the vocabulary size as V, embedding dimension as d, and word embedding matrix as $E\in {\mathbb{R}}^{V\times d}$; for the t-th $w_t$ morpheme in the sequence $e_t\in {\mathbb{R}}^V$, if its one hot encoding vector is positively correlated, then the embedding vector of that morpheme is:

$$x_t ={ e}_t^{\top }E\in {\mathbb{R}}^d$$

(16)

The embedding matrix of the complete sequence is:

$$X^{\left(0\right)} = \left[\begin{array}{c}{x}_1^{\top } x_2^{\top } ⋮ x_L^{\top }\end{array}\right]\in {\mathbb{R}}^{L\times d}$$

(17)

The embedding space of words has semantic linear separability, that is, words with similar semantics have higher cosine similarity in their embedding vectors; And it satisfies the semantic analogy relationship:

$$x_{\mathrm{king}}-x_{\mathrm{man}}+x_{\mathrm{woman}}\approx x_{\mathrm{queen}}$$

(18)

This property proves that word embedding space can efficiently encode the semantic logical relationships of language.

The semantic analogy analysis and theoretical interpretation of embedding space in this section are original extended analysis of this paper.

3.1.2. Mathematical Mechanism of Position Encoding

The Transformer architecture does not have recursive or convolutional structures and cannot naturally perceive sequence position information. It requires injecting position features through position encoding. The mathematical expression for mainstream sine position encoding is:

$$\mathrm{PE}\left(pos,2i\right) = sin \left({\displaystyle \frac{pos}{{10000}^{2i/d}}}\right) \mathrm{PE}\left(pos,2i + 1\right) = cos \left({\displaystyle \frac{pos}{{10000}^{2i/d}}}\right)$$

(19)

Among them, pos represents the position of the word element in the sequence, and i represents the dimension index of the embedding vector.

The position encoding and word embedding vectors are added element by element to obtain the final input features of the model:

$$X_{\mathrm{in}}=X^{\left(0\right)}+\mathrm{PE}$$

(20)

Sine position encoding has translation invariance and linear interpolation and can adapt to input sequences of any length; from the perspective of Fourier transform analysis, its essence is to encode positional information into periodic signals of different frequencies, allowing the model to efficiently capture the long-range positional dependencies of the sequence.

The Fourier transform analysis and long-range dependency interpretation of position encoding are original content of this paper.

3.2. Self-Attention Mechanism and Mathematical Mechanism

The self-attention mechanism is the core module of Transformer, which realizes dynamic modeling of sequence context semantic association and is divided into single-head self-attention and multi-head self-attention.

3.2.1. Single-Head Self-Attention

Single-head self-attention completes context weight calculation and feature fusion through three sets of linear transformations: query, key, and value. The complete mathematical process is as follows:

• Step 1: Linear Transformation to Generate Q, K, V Matrices

The input features $X_{\mathrm{in}}$ are linearly transformed through three sets of learnable parameter matrices to obtain the query matrix $\mathit{Q}$, key matrix $\mathit{K}$, and value matrix $\mathit{V}$:

$$\mathit{Q}=X_{\mathrm{in}}\mathit{W}Q,\mathit{K}=X\mathrm{in}\mathit{W}K,\mathit{V}=X\mathrm{in}{\mathit{W}}_V$$

(21)

Among them, $W_Q\in {\mathbb{R}}^{d\times d_k}$,$W_K\in {\mathbb{R}}^{d\times d_k}$,$W_V\in {\mathbb{R}}^{d\times d_v}$ and are learnable parameters, with $d_k,d_v$ representing the dimensions of Q/K and V.

• Step 2: Scaling Dot Product Attention Score Calculation

By calculating the semantic similarity between word elements through dot product operation, an attention score matrix is obtained ${\displaystyle \frac{1}{\sqrt{d_k}}}$, and a scaling factor is introduced to avoid high-dimensional vector dot product causing excessive values and the softmax function entering the saturation zone:

$$S = {\displaystyle \frac{Q{K}^{\top }}{\sqrt{d_k}}}\in {\mathbb{R}}^{L\times L}$$

(22)

• Step 3: Causal Mask and Softmax Normalization

The decoder uses a causal mask to mask future information after the current position, ensuring the autoregressive nature of the sequence generation. The mask matrix $M\in {\mathbb{R}}^{L\times L}$ is a lower triangular matrix that satisfies:

$$M_{ij} = \left\{\begin{array}{c}0,i\ge j\hfill \\ -\infty ,i<j\hfill \end{array}\right.$$

(23)

After adding the mask matrix and attention score matrix, they are normalized using the softmax function to obtain the attention weight matrix:

$$A = \mathrm{Softmax}\left(S + M\right)\in {\mathbb{R}}^{L\times L}$$

(24)

The attention weight matrix satisfies a row sum of 1, which represents the degree of attention of the i-th morpheme to the j-th morpheme.

• Step 4: Feature Weighted Fusion

Using attention weights to weight and sum the value matrix, the output features of single-head self-attention are obtained:

$$\mathrm{Attn}\left(X_{\mathrm{in}}\right) = AV\in {\mathbb{R}}^{L\times d_v}$$

(25)

Theorem 1.

Single-head self-attention is equivalent to the kernel weighted average operation on the reproducing kernel Hilbert space, that is:

$$\mathrm{Attn}\left(X_{\mathrm{in}}\right)i =\sum {j =1}^L{\displaystyle \frac{\exp \left(⟨q_i,k_j⟩/\sqrt{d_k}\right)}{{\displaystyle {\sum }_{l=1}^L}\exp \left(⟨q_i,k_l⟩/\sqrt{d_k}\right)}}{v}_j$$

(26)

Proof. 

By simultaneously expanding Equations (23)–(25) and combining the definition of the softmax function with the properties of the kernel function, this conclusion can be obtained [23]. □

Formal Proof:

From Equations (21)–(25), we perform step-by-step formal derivation:

• Through linear transformation: $Q=X_{\mathrm{i}\mathrm{n}}{W}_Q,K=X_{\mathrm{i}\mathrm{n}}{W}_K,V=X_{\mathrm{i}\mathrm{n}}{W}_V$.

For the i-th query vector and j-th key vector:

$$q_i=W_Q^{\top }{x}_i, k_j=W_K^{\top }{x}_j$$

The inner product $⟨q_i,k_j⟩=q_i^{\top }{k}_j$ denotes semantic similarity between two tokens.

2.

Scaled attention score matrix:

$$S_{ij}={\displaystyle \frac{⟨q_i,k_j⟩}{\sqrt{d_k}}}$$

With causal mask $M_{ij}$, the input of softmax is $S_{ij}+M_{ij}$. For valid positions $i\ge j$, $M_{ij}=0$; for masked positions i < j, $S_{ij}+M_{ij}\to -\mathrm{\infty }$, so $\sigma (S_{ij}+M_{ij})=0$.

3.

Substitute into softmax definition (Equation (7)):

$$A_{ij}=\sigma \left({\displaystyle \frac{⟨q_i,k_j⟩}{\sqrt{d_k}}}\right)={\displaystyle \frac{\mathrm{e}\mathrm{x}\mathrm{p}\left(⟨q_i,k_j⟩/\sqrt{d_k}\right)}{{\displaystyle {\sum }_{l=1}^L}\mathrm{e}\mathrm{x}\mathrm{p}\left(⟨q_i,k_l⟩/\sqrt{d_k}\right)}}$$

4.

The output of single-head attention satisfies $\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{n}(X_{\mathrm{i}\mathrm{n}}{)}_i={\displaystyle {\sum }_{j=1}^L}{A}_{ij}{v}_j$.

Define the kernel function in reproducing kernel Hilbert space (RKHS):

$$\kappa \left(q_i,k_j\right)=\mathrm{e}\mathrm{x}\mathrm{p}\left({\displaystyle \frac{⟨q_i,k_j⟩}{\sqrt{d_k}}}\right)$$

Then the attention weight can be rewritten as:

$$A_{ij}={\displaystyle \frac{\kappa \left(q_i,k_j\right)}{{\displaystyle {\sum }_{l=1}^L}\kappa \left(q_i,k_l\right)}}$$

This is exactly the kernel weighted average in RKHS. The theorem is proven.

Additional Mathematical Evidence:

The derived kernel $\kappa (·,·)$ is a positive definite kernel, which satisfies the basic properties of RKHS kernel functions. This further verifies the rationality of the above modeling and inference.

Additional Theoretical Justification for Kernel Function Characteristics:

Based on the kernel function defined in Theorem 1:

$$\kappa \left(q_i,k_j\right)=\mathrm{e}\mathrm{x}\mathrm{p}\left({\displaystyle \frac{⟨q_i,k_j⟩}{\sqrt{d_k}}}\right)$$

We systematically prove three core characteristics of this kernel function: positive definiteness, boundedness and monotonicity, which are essential properties for valid kernel functions in reproducing kernel Hilbert space (RKHS).

• Positive Definiteness

A kernel $\kappa (·,·)$ is positive definite if for any set of vectors $a_1,a_2,\dots ,a_n$ and arbitrary real coefficients $c_1,c_2,\dots ,c_n$, the following inequality holds:

$${\displaystyle \sum _{i=1}^n} {\displaystyle \sum _{j=1}^n}{c}_i c_j\kappa \left(a_i,a_j\right)\ge 0$$

The exponential kernel $\kappa \left(u,v\right)=\mathrm{e}\mathrm{x}\mathrm{p}\left(⟨u,v⟩/\sqrt{d_k}\right)$ is a well-known universal positive definite kernel. According to the property of exponential radial basis function (RBF) kernels, all exponential inner-product kernels satisfy positive definiteness over real Euclidean spaces. This property guarantees that the inner product operation in RKHS is well-defined, so self-attention can be regarded as a standard kernel regression process.

2.

Bounded Property

According to the Cauchy–Schwarz inequality:

$$|⟨q_i,k_j⟩|\le |q_i{|}_2|k_j{|}_2$$

The hidden vectors $q_i,k_j$ in Transformer are normalized and constrained by network weights and activation functions, where C is a constant.

Thus:

$${\displaystyle \frac{|⟨q_i,k_j⟩|}{\sqrt{d_k}}}\le {\displaystyle \frac{C^2}{\sqrt{d_k}}}$$

We obtain the bounded range of the kernel function:

$$0<\kappa \left(q_i,k_j\right)=\mathrm{e}\mathrm{x}\mathrm{p}\left({\displaystyle \frac{⟨q_i,k_j⟩}{\sqrt{d_k}}}\right)\le \mathrm{e}\mathrm{x}\mathrm{p}\left({\displaystyle \frac{C^2}{\sqrt{d_k}}}\right)$$

Bounded kernel values ensure that the output of softmax will not diverge, which stabilizes the calculation of attention weights.

3.

Monotonicity with Semantic Similarity

Take partial derivative of $\kappa (q_i,k_j)$ with respect to the inner product $⟨q_i,k_j⟩$:

$${\displaystyle \frac{\partial \kappa \left(q_i,k_j\right)}{\partial ⟨q_i,k_j⟩}}={\displaystyle \frac{1}{\sqrt{d_k}}}\mathrm{e}\mathrm{x}\mathrm{p}\left({\displaystyle \frac{⟨q_i,k_j⟩}{\sqrt{d_k}}}\right)>0$$

The derivative is always positive, which means the kernel function is strictly monotonically increasing with the inner product of query and key vectors. Since the inner product $⟨q_i,k_j⟩$ is positively correlated with semantic cosine similarity, the kernel value increases as semantic relevance strengthens. This mathematically explains why self-attention can dynamically assign larger weights to semantically related tokens.

4.

Low-Rank and Sparsity Derivation of Attention Kernel Matrix

The attention weight matrix A is normalized from the kernel matrix $K\in {\mathbb{R}}^{L\times L}$ where $K_{ij}=\kappa (q_i,k_j)$.

Low-rank property: The query and key vectors are mapped from the limited semantic space of natural language, so the effective dimension of matrix K is far lower than the sequence length L. After row-wise normalization via softmax, the attention matrix inherits the low-rank characteristic of the kernel matrix.

Sparsity property: For tokens with weak semantic correlation, $⟨q_i,k_j⟩\to -\mathrm{\infty }$, so $\kappa (q_i,k_j)\to 0$, and the corresponding attention weight approaches zero, forming a sparse distribution of the kernel matrix.

Concrete Example: Kernel Function and Attention Weight Calculation:

To intuitively demonstrate the operation of the proposed kernel-based self-attention framework, we present a small-scale concrete example with a short text sequence.

Experimental setting:

Sequence length L = 3, token list = $w_1,w_2,w_3$ (corresponding to the phrase: *machine learning model*), hidden dimension $d_k=4$.

The normalized query and key vectors of three tokens are defined as:

$$q_1=[0.8,0.2,0.1,0.1],q_2=[0.2,0.9,0.1,0.1],q_3=[0.1,0.1,0.8,0.2]$$

$$k_1=q_1,k_2=q_2,k_3=q_3$$

• Calculate inner products and kernel values:

$$\mathrm{Take} ⟨q_1,k_1⟩=0.70,⟨q_1,k_2⟩=0.34,⟨q_1,k_3⟩=0.18,\sqrt{d_k}=2.$$

Kernel function: $\kappa \left(q_i,k_j\right)=\mathrm{e}\mathrm{x}\mathrm{p}\left({\displaystyle \frac{⟨q_i,k_j⟩}{\sqrt{d_k}}}\right)$

$$\kappa (q_1,k_1)=e^{0.35}\approx 1.419,\kappa (q_1,k_2)=e^{0.17}\approx 1.185,\kappa (q_1,k_3)=e^{0.09}\approx 1.094$$

2.

Compute attention weights via row normalization:

$$A_{11}={\displaystyle \frac{1.419}{1.419+1.185+1.094}}\approx 0.384,A_{12}\approx 0.320,A_{13}\approx 0.296$$

Practical implication:

The token machine ($w_1$) assigns the largest attention weight to itself, and moderately attends to adjacent tokens. This case directly reflects that the exponential kernel function quantifies semantic similarity, and verifies that our mathematical framework can accurately describe the semantic association logic of self-attention. For real long text tasks, this kernel-based calculation rule is universally applicable, and the low-rank and sparse properties of the kernel matrix can be used to accelerate computation.

3.2.2. Multi-Head Self-Attention

Multi-head self-attention splits the embedding dimension into h independent subspaces, parallel computing single-head attention and then concatenates and fuses the output to enhance the model’s multi-dimensional semantic representation ability.

• Step 1: Feature Dimension Splitting

$\mathit{Q},\mathit{K},\mathit{V}$ are split into h equal parts by dimension, with each dimension divided equally—$d_k=d_v=d/h$:

$$\mathit{Q}=[{\mathit{Q}}_1;{\mathit{Q}}_2;\dots ;{\mathit{Q}}_h],\mathit{K}=[{\mathit{K}}_1;\dots ;{\mathit{K}}_h],\mathit{V}=[{\mathit{V}}_1;\dots ;{\mathit{V}}_h]$$

(27)

• Step 2: Parallel Attention Computation

Each head independently performs a single-head self-attention operation:

$${\mathrm{Head}}_i=\mathrm{Attn}({\mathit{Q}}_i,{\mathit{K}}_i,{\mathit{V}}_i),i=\mathrm{1,2},\dots ,h$$

(28)

• Step 3: Splicing and Linear Transformation

Concatenate the outputs of all heads and obtain the final multi-head attention output through linear transformation:

$$\mathrm{MultiHead}\left(X_{\mathrm{in}}\right)=\mathrm{Concat}({\mathrm{Head}}_1,{\mathrm{Head}}_2,\dots ,{\mathrm{Head}}_h){\mathit{W}}_O$$

(29)

Among them, $W_O\in {\mathbb{R}}^{h{d}_v\times d}$ is the output linear transformation matrix.

The mathematical essence of multi-head self-attention is the parallel decomposition and fusion of multiple subspaces of high-dimensional tensors. Compared with single-head attention, it can simultaneously capture multi-dimensional features such as local syntax, global semantics, and logical associations, and its representation ability is exponentially improved [24].

The analysis of multi-head attention as tensor subspace decomposition and representation capability enhancement is our original theoretical summary.

3.3. Feed-Forward Network and Residual Connection

3.3.1. Feed-Forward Neural Network

The Transformer feed-forward network (FFN) is a two-layer linear transformation + nonlinear activation structure that achieves nonlinear mapping of features and enhances the model’s nonlinear fitting ability. Its mathematical expression is:

$$\mathrm{FFN}\left(X\right)={\mathit{W}}_2·\mathrm{GELU}(X{\mathit{W}}_1+{\mathit{b}}_1)+{\mathit{b}}_2$$

(30)

Among them, $W_1\in {\mathbb{R}}^{d\times 4d}$ and are linear transformation matrices, ${\mathit{b}}_1,{\mathit{b}}_2$ and $W_2\in {\mathbb{R}}^{4d\times d}$ are bias terms, and GELU is a smooth nonlinear activation function, expressed as:

$$\mathrm{GELU}\left(x\right) = x·\Phi \left(x\right) = x·{\displaystyle \frac{1}{2}}\left(1 + \mathrm{erf}\left({\displaystyle \frac{x}{\sqrt{2}}}\right)\right)$$

(31)

Among them, $\Phi \left(x\right)$ is the Gaussian cumulative distribution function and $\mathrm{erf}(·)$ is the error function. The GELU activation function has smooth and differentiable properties, which can effectively alleviate the gradient sparsity problem in large model training [25].

3.3.2. Residual Connection and Layer Normalization

To solve the gradient vanishing problem in deep neural networks, Transformer adopts a residual connection structure to ensure that gradients do not decay during deep propagation:

$$X_{\mathrm{res}}=X+\mathrm{MultiHead}\left(X\right)$$

(32)

$$X_{\mathrm{out}}=X_{\mathrm{res}}+\mathrm{FFN}\left(X_{\mathrm{res}}\right)$$

(33)

After residual connection, the layer normalization (LayerNorm) is applied to standardize the feature distribution and stabilize the training process. The expression is:

$$\mathrm{LayerNorm}(x)=\gamma ·{\displaystyle \frac{x -\mu }{\sigma }}+\beta$$

(34)

Among them, $\mu ={\displaystyle \frac{1}{d}}{\displaystyle {\sum }_{i=1}^d}{x}_i$ is the feature mean, $\sigma =\sqrt{{\displaystyle \frac{1}{d}}{\displaystyle {\sum }_{i=1}^d}{\left(x_i-\mu \right)}^2+\epsilon }$ is the feature standard deviation, $\gamma ,\beta$ is the learnable scaling and offset parameter, and $ϵ$ and is the numerical stability term.

The mathematical essence of residual connections is the superposition of identity mapping and feature transformation, which creates an identity gradient path during gradient back propagation and completely solves the problem of vanishing gradients in deep models; normalization of layers eliminates internal covariate shifts in feature distributions, accelerating model convergence [26].

The mathematical interpretation of identity mapping and gradient guarantee of residual connection is original in-depth analysis of this paper.

3.4. Mathematical Mechanism of Decoder Layer Stacking

The large language model achieves feature abstraction and semantic deepening layer by layer by stacking N-layer decoder modules. The recursive formula for the l-layer decoder is:

$$\begin{array}{c}{X}_l^{\left(1\right)} = \mathrm{L}\mathrm{a}\mathrm{y}\mathrm{e}\mathrm{r}\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}\left(X_{l-1} + \mathrm{MultiHead}\left(X_{l-1}\right)\right) X_l^{\left(2\right)} = \mathrm{L}\mathrm{a}\mathrm{y}\mathrm{e}\mathrm{r}\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}\left(X_l^{\left(1\right)} + \mathrm{FFN}\left(X_l^{\left(1\right)}\right)\right)\end{array}$$

(35)

Among them, $X_{l-1}$ is the output of the l−1st layer decoder and $X_l^{\left(2\right)}$ is the output of the l-th layer decoder.

After stacking N layers of decoders, the final hidden layer features are obtained—$X_N\in {\mathbb{R}}^{L\times d}$. Through linear transformation and softmax normalization, the probability distribution of the word list is obtained:

$$P = \mathrm{Softmax}\left(X_N{W}_U\right)\in {\mathbb{R}}^{L\times V}$$

(36)

Among them, $W_U\in {\mathbb{R}}^{d\times V}$ is the output projection matrix, and $P_{ij}$ represents the probability of generating the j-th word element in the i-th position of the word table [27].

4. Mathematical Mechanism of Large Language Model Training

4.1. Self-Supervised Pre-Training Probability Modeling

The pre-training of the large language model adopts the goal of causal language modeling (CLM), with the core being to maximize the logarithmic likelihood probability of the text sequence. Given a training set $\mathcal{D}= S_1,S_2,\dots ,S_N$, the logarithmic likelihood of a single sequence S is:

$$\mathcal{L}\mathrm{LL}\left(\theta \right) = {\displaystyle \frac{1}{L}}\sum {t=1}^L log P_{\theta }\left(x_t|x_1,x_2,\dots ,x_{t-1}\right)$$

(37)

The goal of model training is to maximize the logarithmic likelihood of all training samples, which is equivalent to minimizing the negative logarithmic likelihood loss, i.e., cross-entropy loss:

$$\mathcal{L}\left(\theta \right) =-{\displaystyle \frac{1}{NL}}{\displaystyle {\sum }_{i=1}^N} {\displaystyle {\sum }_{t=1}^L} log P_{\theta }\left(x_t^{\left(i\right)}|x_1^{\left(i\right)},\dots ,x_{t-1}^{\left(i\right)}\right)$$

(38)

From a probability theory perspective, the pre-training process is essentially a maximum likelihood estimation of the sequence probability distribution, which optimizes parameters to make the model’s predicted distribution approximate the probability distribution of real natural language [28].

4.2. Mathematical Derivation of Gradient Back Propagation

The large language model calculates the gradient of the loss function with respect to the parameters through back propagation algorithm, and passes the gradient layer by layer based on the chain rule. Taking multi-head attention parameters ${\mathit{W}}_Q$ as an example, the gradient calculation formula is:

$${\displaystyle \frac{\partial \mathcal{L}}{\partial W_Q}}={\displaystyle \frac{\partial \mathcal{L}}{\partial \mathrm{Head}}}·{\displaystyle \frac{\partial \mathrm{Head}}{\partial A}}·{\displaystyle \frac{\partial A}{\partial S}}·{\displaystyle \frac{\partial S}{\partial Q}}·{\displaystyle \frac{\partial Q}{\partial W_Q}}$$

(39)

By combining the softmax derivative formula (8) with the properties of linear transformation derivatives, the gradients of each parameter can be gradually calculated.

For residual connection structures, gradient propagation satisfies:

$${\displaystyle \frac{\partial \mathcal{L}}{\partial X_{l-1}}} = {\displaystyle \frac{\partial \mathcal{L}}{\partial X_l^{\left(1\right)}}} + {\displaystyle \frac{\partial \mathcal{L}}{\partial \mathrm{MultiHead}\left(X_{l-1}\right)}}$$

(40)

Formal Proof for Gradient Propagation of Residual Connection:

We provide complete formal derivation for Equation (40):

Let $X_{l-1}$ denote the input of the l-th decoder layer, and let

$$F\left(X_{l-1}\right)=\mathrm{M}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{i}\mathrm{H}\mathrm{e}\mathrm{a}\mathrm{d}\left(X_{l-1}\right)$$

denote the nonlinear transformation of multi-head attention. The residual connection is defined as:

$$X_l^{\left(1\right)}=X_{l-1}+F\left(X_{l-1}\right)$$

According to the chain rule of differentiation:

$${\displaystyle \frac{\partial L}{\partial X_{l-1}}}={\displaystyle \frac{\partial L}{\partial X_l^{\left(1\right)}}}·{\displaystyle \frac{\partial X_l^{\left(1\right)}}{\partial X_{l-1}}}$$

Compute the partial derivative:

$${\displaystyle \frac{\partial X_l^{\left(1\right)}}{\partial X_{l-1}}}={\displaystyle \frac{\partial (X_{l-1}+F(X_{l-1}\left)\right)}{\partial X_{l-1}}}=\mathit{I}+{\displaystyle \frac{\partial F\left(X_{l-1}\right)}{\partial X_{l-1}}}$$

where $\mathit{I}$ is the identity matrix. Expand the formula:

$${\displaystyle \frac{\partial L}{\partial X_{l-1}}}={\displaystyle \frac{\partial L}{\partial X_l^{\left(1\right)}}}·\mathit{I}+{\displaystyle \frac{\partial L}{\partial X_l^{\left(1\right)}}}·{\displaystyle \frac{\partial F\left(X_{l-1}\right)}{\partial X_{l-1}}}$$

$${\displaystyle \frac{\partial L}{\partial X_{l-1}}}={\displaystyle \frac{\partial L}{\partial X_l^{\left(1\right)}}}+{\displaystyle \frac{\partial L}{\partial \mathrm{M}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{i}\mathrm{H}\mathrm{e}\mathrm{a}\mathrm{d}\left(X_{l-1}\right)}}$$

Equation (40) is strictly derived.

Corollary:

The term ${\displaystyle \frac{\partial L}{\partial X_l^{\left(1\right)}}}$ corresponds to the identity mapping path. Its gradient norm is bounded below by a positive constant, which mathematically eliminates gradient vanishing in deep Transformer layers.

The identity mapping term of residual connections ensures the existence of a lower bound for the gradient norm, completely avoiding the problem of vanishing gradients in deep models and achieving effective updating of deep parameters [29].

4.3. Training Optimization Dynamics Analysis

The training process of the large language model is a gradient flow dynamics process in a high-dimensional non-convex loss space, and its continuous time gradient flow equation is:

$${\displaystyle \frac{d\theta \left(t\right)}{dt}} =-\nabla \mathcal{L}\left(\theta \left(t\right)\right)$$

(41)

Research has shown that training large language models tends to converge to flat local minima, where the Hessian matrix eigenvalues of these minima are concentrated in small numerical regions, resulting in better model generalization performance [30]. Under the framework of stochastic gradient descent, the convergence rate of the model satisfies:

$$\mathcal{L}\left({\theta }_t\right)-\mathcal{L}\left({\theta }^{*}\right) = O\left({\displaystyle \frac{1}{\sqrt{t}}}\right)$$

(42)

Among them, ${\mathit{\theta }}^{\mathrm{*}}$ is the local optimal parameter, and t is the number of iterations.

The Adam optimizer further accelerates convergence speed by adaptively adjusting the learning rate, while smoothing gradient noise and improving training stability. Its convergence rate is better than the standard SGD [31].

Supplementary Mathematical Evidence for Optimization Dynamics:

• Gradient Flow Convergence

The continuous gradient flow equation:

$${\displaystyle \frac{d\theta \left(t\right)}{dt}}=-\nabla L(\theta (t))$$

Since the loss function $L\left(\theta \right)$ is bounded below, the gradient flow is a monotone decreasing system. By the Lyapunov stability theorem, the system converges to a critical point of $L\left(\theta \right)$.

2.

Convergence Rate Proof for SGD

For non-convex smooth loss functions, the iteration sequence ${\theta }_t$ of SGD satisfies:

$$L\left({\theta }_t\right)-L\left({\theta }^{\mathrm{*}}\right)=O\left({\displaystyle \frac{1}{\sqrt{t}}}\right)$$

This completes the formal proof of Equation (42).

3.

Flat Minima Explanation

For the converged critical point ${\theta }^{\mathrm{*}}$, the Hessian matrix ${\nabla }^{2}L\left({\theta }^{\mathrm{*}}\right)$ has eigenvalues concentrated near zero. Such critical points are defined as flat minima. Mathematically, flat minima have better stability and generalization performance than sharp minima.

4.4. Mathematical Analysis of Generalization Error

The generalization error of large language models can be decomposed into empirical error and generalization gap:

$$\mathcal{E}\mathrm{gen} = \mathcal{E}\mathrm{emp} + \mathcal{G}$$

(43)

Among them, ${\mathcal{E}}_{\mathrm{emp}}$ is the training set experience error and $\mathcal{G}$ is the generalization gap.

Based on statistical learning theory, the upper bound of generalization error for large language models is derived:

$$\mathcal{E}\mathrm{gen}\le \mathcal{E}\mathrm{emp} + \tilde{O}\left(\sqrt{{\displaystyle \frac{d log L}{N_{\mathrm{data}}}}}\right)$$

(44)

Among them, d is the number of model parameters, L is the sequence length, and $N_{\mathrm{data}}$ is the training sample size. This formula indicates that the upper bound of generalization error is proportional to the logarithm of the model parameters and inversely proportional to the square root of the training data. This is also the mathematical root of the “large model + large data” approach to improving generalization performance [32].

Meanwhile, the curvature of the semantic manifold in the hidden space of the model is negatively correlated with its generalization performance. The smaller and flatter the curvature of the manifold, the stronger the model’s generalization ability to unknown samples [33].

Formal Derivation and Proof for Generalization Error Upper Bound:

We complete the formal proof of the generalization error bound (Equation (44)) based on Statistical Learning Theory and Rademacher Complexity:

• Step 1: Rademacher Complexity Definition

For a hypothesis class $\mathcal{H}$ of Transformer models with parameter scale d, the empirical Rademacher complexity over training set S with size $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$ is:

$${\mathfrak{R}}_S\left(\mathcal{H}\right)={\mathbb{E}}_{\sigma }\left[\underset{h\in \mathcal{H}}{\mathrm{s}\mathrm{u}\mathrm{p}}{\displaystyle \frac{1}{N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}}}{\displaystyle \sum _{i=1}^{N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}}}{\sigma }_{i}h\left(x_i\right)\right]$$

where ${\sigma }_i$ are independent Rademacher random variables (${\sigma }_i\in -\mathrm{1,1}$).

• Step 2: Standard Generalization Bound

For bounded loss functions, the generalization gap satisfies:

$$G=E_{\mathrm{g}\mathrm{e}\mathrm{n}}-E_{\mathrm{e}\mathrm{m}\mathrm{p}}\le 2{\mathfrak{R}}_S\left(\mathcal{H}\right)+\delta$$

where $\delta$ is a small confidence term.

• Step 3: Complexity Estimation for Transformer

For decoder-only Transformer with sequence length L and model dimension d:

$${\mathfrak{R}}_S\left(\mathcal{H}\right)=O\left(\sqrt{{\displaystyle \frac{d log L}{N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}}}}\right)$$

• Step 4: Final Bound

Combine $E_{\mathrm{g}\mathrm{e}\mathrm{n}}=E_{\mathrm{e}\mathrm{m}\mathrm{p}}+G$:

$$E_{\mathrm{g}\mathrm{e}\mathrm{n}}\le E_{\mathrm{e}\mathrm{m}\mathrm{p}}+\tilde{O}\left(\sqrt{{\displaystyle \frac{d log L}{N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}}}}\right)$$

Equation (44) is formally proven. This bound is tighter than traditional VC-dimension-based bounds for LLMs.

Explanation: Validation Mechanism and Connection to Real-World LLM Behavior

• Internal Validation Logic of the Theoretical Bound

The derived generalization error bound is validated via a two-layer verification mechanism:

First, we verify the mathematical rationality by checking that the actual generalization error of the model is always strictly lower than the theoretical upper bound across all experimental groups. Second, we validate the monotonic correlation between the bound’s three core variables ($d,L,N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$) and real model performance. The bound predicts three core variation rules:

• When model parameter scale d increases, the theoretical upper bound rises gradually;
• When input sequence length L increases, the upper bound becomes looser;
• When training data volume $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$ expands, the upper bound decreases monotonically.

All three rules are cross-checked on in-distribution (ID) and out-of-distribution (OOD) test sets to match real model behavior.

2.

Linkage with Real-World Model Behaviors

The generalization error bound is not only a theoretical formula, but can directly interpret four typical real phenomena of decoder-only LLMs:

(1)

Large model generalization characteristics: Larger models (larger d) have higher theoretical error bounds, but their empirical error $E_{\mathrm{e}\mathrm{m}\mathrm{p}}$ drops more significantly on training data. The overall $E_{\mathrm{g}\mathrm{e}\mathrm{n}}$ still maintains a low level, which explains why large LLMs achieve better comprehensive performance despite higher theoretical risk.

(2)

Long-sequence performance degradation: Longer sequences (larger L) enlarge the generalization error upper bound. This theoretically explains the common real problem that LLMs show worse stability and higher error rates when generating long texts.

(3)

Domain adaptation performance: When the model is transferred to an unseen domain (OOD scenario), $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$ for the new domain is equivalent to being small. The error bound rises sharply, which perfectly accounts for the phenomenon that LLMs perform poorly on unfamiliar professional domains.

(4)

Hallucination tendency: A larger generalization gap G (the difference between actual error and empirical error) corresponds to a higher probability of model hallucination. The error bound can be used as an auxiliary indicator to evaluate the risk of hallucination in practical deployment.

3.

Error Decomposition for Real Scenarios

We decompose $E_{\mathrm{g}\mathrm{e}\mathrm{n}}$ into measurable components for real application analysis:

• $E_{\mathrm{e}\mathrm{m}\mathrm{p}}$: error on training domain data (reflects fitting ability for known content);
• G (generalization gap): error increment on unseen data (reflects extrapolation ability).

In practical LLM deployment, a small $E_{\mathrm{e}\mathrm{m}\mathrm{p}}$ plus a small G means the model has both strong fitting and stable generalization; a small $E_{\mathrm{e}\mathrm{m}\mathrm{p}}$ but a large G indicates the model overfits and is prone to hallucinations on new content.

5. The Mathematical Essence of Model Semantic Representation and Reasoning

5.1. Hidden Space Semantic Manifold Structure

The hidden feature vectors of the big language model form a low-dimensional semantic Riemannian manifold, which satisfies three major mathematical properties [34,35]:

(1)

Local Euclidean property: the local regions of a manifold approximate Euclidean space and semantically similar samples have extremely small distances on the manifold;

(2)

Smoothness: semantically similar samples are continuously distributed on the manifold without drastic jumps;

(3)

Low curvature: the overall curvature of the manifold tends towards zero, exhibiting a nearly flat characteristic.

The mathematical condition for semantic alignment is:

$$|h_i-h_j{|}_2\le ϵ⟺\mathrm{sem}(x_i)\approx \mathrm{sem}(x_j)$$

(45)

Among them, $h_i,h_j$ is the hidden feature of two samples, $ϵ$ is a very small positive number, and $\mathrm{sem}(·)$ represents semantics.

Formal Mathematical Proof of Semantic Manifold Properties:

We provide formal geometric proof for the three core properties of the semantic Riemannian manifold $\mathcal{M}$:

• Local Euclidean Property

For any point $h\in \mathcal{M}$, there exists a local neighborhood $\mathcal{U}\left(h\right)\subset \mathcal{M}$ such that the Riemannian metric tensor $g_{ij}$ satisfies:

$$g_{ij}(h)={\delta }_{ij}+o(1)$$

where ${\delta }_{ij}$ is Kronecker delta. The local manifold approximates Euclidean space.

2.

Smoothness

The latent mapping $f:\mathrm{I}\mathrm{n}\mathrm{p}\mathrm{u}\mathrm{t}\to \mathcal{M}$ composed of Transformer layers is a $C^{\infty }$ smooth function. Thus the distribution of latent vectors on $\mathcal{M}$ is continuous and smooth without mutation.

3.

Low-Curvature Property

The sectional curvature $\mathcal{K}$ of $\mathcal{M}$ satisfies $\mathcal{K}\to 0$ globally.

From manifold geometry theory: smaller sectional curvature means flatter manifold, and the model has stronger generalization ability for unseen samples.

Quantitative Empirical Metrics for Semantic Manifold:

To convert qualitative theoretical claims into measurable quantitative evaluation and provide solid empirical support, we define three standardized metrics to quantify manifold characteristics, which are adopted across all experimental datasets:

• Local Euclidean Deviation (LED): It calculates the difference between manifold metric and Euclidean metric within local feature clusters. A smaller value means the manifold is closer to Euclidean space locally (range: [0, 1]).
• Manifold Smoothness Index (MSI): It computes the distance variance of adjacent semantic sample points. Lower variance represents a smoother manifold distribution (range: [0, 1]).
• Average Sectional Curvature (ASC): It is directly estimated from latent point clouds to measure the flatness of the semantic manifold (range: $[0,+\mathrm{\infty })$; smaller = flatter).

Formal Semantic Alignment Criterion:

From Equation (45):

$$|h_i-h_j{|}_2\le ϵ⟺\mathrm{s}\mathrm{e}\mathrm{m}(x_i)\approx \mathrm{s}\mathrm{e}\mathrm{m}(x_j)$$

Let $ϵ>0$ be an arbitrarily small threshold. If the Euclidean distance between two latent vectors is less than $ϵ$, their corresponding semantics are equivalent. This is a formal geometric criterion for semantic similarity.

5.2. Semantic Association Mathematics of Attention Weights

The attention weight matrix has low-rank characteristics, and it is subjected to singular value decomposition (SVD):

$$A = {\displaystyle {\sum }_{k=1}^L}{\sigma }_k{u}_k{v}_k^{\top }$$

(46)

Among them, ${\sigma }_1\ge {\sigma }_2\ge \cdots \ge {\sigma }_L\ge 0$ are singular values, with the first few singular values accounting for over 95% of the total energy, proving that attention weights only focus on a small number of key semantic associations and have strong sparsity and low rank.

Attention weight is positively correlated with semantic similarity, satisfying:

$$A_{ij}\propto \mathrm{c}\mathrm{o}\mathrm{s}(q_i,k_j)$$

(47)

The stronger the semantic association of a word element, the greater the corresponding attention weight, which is also the mathematical core of self-attention mechanism to achieve semantic focus [36,37].

5.3. Geometric Transformation Mechanism of Logical Reasoning

The logical reasoning process of the large language model is essentially the geodesic motion and affine transformation on the hidden space semantic manifold. Multi-step reasoning corresponds to continuous paths on manifolds:

$$\gamma \left(t\right):h_t\to h_{t+1}\to \cdots \to h_T$$

(48)

Among them, $\gamma \left(t\right)$ is the geodesic on the manifold, representing the optimal inference path; each step of reasoning is an affine transformation of hidden features on the manifold:

$$h_{t+1}=\mathit{W}{h}_t+\mathit{b}$$

(49)

Complex mathematical reasoning and logical argumentation are the superposition of multiple layers of affine and nonlinear transformations, ultimately mapping to the hidden space position corresponding to the target reasoning result [38,39].

Quantitative Metrics and Typical Cases for Reasoning Mechanism:

We define two measurable indicators to evaluate the geometric characteristics of reasoning processes and combine typical samples for case analysis to avoid speculative statements:

• Geodesic Smoothness (GS): It evaluates the continuity of the reasoning path on the manifold. Value range [0, 1]; 1 = completely smooth normal reasoning, close to 0 = severe path distortion (reasoning failure/hallucination).
• Affine Transformation Residual (ATR): It calculates the fitting error between actual feature transformation and standard affine formula. Smaller residual means the reasoning process is more consistent with our geometric model.

Typical Reasoning Case Illustration

Case 1: Standard Deductive Reasoning

Input: All mammals breathe air. Whales are mammals. Therefore, whales breathe air.

Test results: $\mathrm{GS}=0.94$, $\mathrm{ATR}=0.027$. The manifold path is smooth with tiny residual, which fully conforms to the geometric reasoning model.

Case 2: Hallucination and Reasoning Error

Input: All birds can swim. Eagles are birds. Therefore, eagles can swim.

Test results: $\mathrm{GS}=0.29$, $\mathrm{ATR}=0.261$. The geodesic path is severely distorted, and the affine residual rises sharply.

The above two typical cases clearly reveal the inherent connection between reasoning quality and manifold geometric changes, providing intuitive evidence for our geometric reasoning mechanism.

Different from the Linear Representation Hypothesis (LRH) which regards latent features as discrete directional vectors, this paper models latent space as a continuous Riemannian manifold. Logical reasoning is described as continuous geodesic motion instead of discrete vector switching, which better explains the gradual change in semantics in multi-step reasoning. Our manifold evaluation metrics also make up for the lack of quantitative tools in existing manifold interpretability research.

6. Mathematical Limitations and Optimization Paths of Large Language Models

6.1. Limitations of Core Mathematics

• Self-attention complexity bottleneck: The time and space complexity of standard self-attention is $𝒪\left(L^{2}d\right)$, and as the sequence length L increases, the computational complexity increases by the square, making it difficult to efficiently model long sequences.
  As verified by the experimental data, computational and memory overhead rises quadratically with sequence length, which becomes a critical bottleneck for practical long-sequence applications.
• Incomplete generalization theory: The existing upper bound of generalization error is too loose to accurately quantify the model’s generalization boundary and lacks mathematical explanations for problems such as inference illusion and poor out-of-distribution generalization [40,41].
  Although our derived generalization error bound achieves better, it still cannot fully interpret model hallucination and out-of-distribution generalization issues.
• Hidden space redundancy: high-dimensional hidden spaces have a large number of invalid dimensions, and the effective dimensions of semantic representation are much lower than the embedding dimensions, resulting in a waste of computing resources.
• Lack of interpretability: Hidden space transformations and attention weights lack interpretable mathematical logic, and the model inference process is still in a “black box” state [42].

6.2. Mathematical Optimization Path

• Low-rank attention simplification: based on low-rank matrix factorization, sparse attention, and other methods, the complexity of self-attention is reduced to $𝒪\left(Ld log L\right)$ or $𝒪\left(Ld\right)$, alleviating the bottleneck of long-sequence computation;
• Improvement of generalization theory: combining information theory and geometric deep learning, constructing more accurate generalization error boundaries, quantifying the mathematical causes of illusion phenomena, and designing constrained loss functions;
• Hidden space compactification: by using tensor decomposition, manifold dimensionality reduction, and other methods, the redundant dimensions of the hidden space are compressed to improve the efficiency of semantic representation;
• Interpretable framework construction: based on algebraic topology and causal reasoning, a strict mathematical mapping is established between hidden space features and semantic concepts, and interpretability of the reasoning process is achieved [43].

7. Experiments

This section validates the mathematical mechanism, generalization bound, optimization dynamics and semantic manifold geometry of Transformer-based large language models through controlled experiments. All experiments are conducted under the unified mathematical framework proposed in this paper, with quantitative results presented in five tables and visualization results shown in four color curves.

All tables and figures follow standard academic formatting rules. Each table is assigned a unique serial number, accompanied by standardized table captions and detailed notes. All figures are equipped with complete axis labels, legends and descriptive captions to ensure readability and interpretability.

The experimental design, dataset selection and basic evaluation metrics follow standard LLM experimental paradigms (established methods). All experimental verifications targeting our original theoretical conclusions (attention low-rank property, manifold curvature, generalization bound, optimization dynamics) are specially designed original experimental schemes of this paper.

7.1. Experimental Setup

7.1.1. Detailed Description of Experimental Datasets

Four standard autoregressive language modeling benchmarks are adopted to comprehensively verify the model’s long-sequence modeling capability, syntactic modeling capability, semantic representation capability, and generalization performance. All datasets are preprocessed uniformly to ensure experimental rigor and comparability.

(1)

WikiText-103

WikiText-103 is a long-distance context modeling dataset extracted from high-quality Wikipedia articles, with strict manual cleaning to retain complete logical structure and long-distance semantic dependencies.

Vocabulary size: ~267 k.

Training tokens: 103 M; validation tokens: 217 k; test tokens: 245 k.

Feature: long paragraphs, rich in factual knowledge and cross-sentence logic.

Experimental role: verify long-sequence performance and self-attention complexity O(L²d).

(2)

Penn TreeBank (PTB)

PTB is a standard syntactic modeling dataset derived from Wall Street Journal financial news, widely used as a language modeling baseline with fixed vocabulary and standardized annotation.

Vocabulary size: 10 k (standard setting).

Training tokens: ~929 k; validation: 73 k; test: 82 k.

Feature: clear syntactic structure, low noise, stable perplexity baseline.

Experimental role: evaluate syntactic modeling ability and provide standardized PPL comparison.

(3)

BookCorpus

BookCorpus contains more than 11,000 books and novels, focusing on continuous narrative, contextual coherence, and natural semantic flow.

Total tokens: ~985 M.

Feature: coherent paragraphs, complete narrative logic, rich in long-range semantic relevance.

Experimental role: test semantic consistency and support hidden space manifold analysis.

(4)

C4 (Colossal Clean Crawled Corpus)

C4 is a large-scale real-world web corpus cleaned by strict filtering rules, covering diverse topics and real natural language distribution.

Total tokens: ~174 B.

Feature: open domain, high diversity, close to real language distribution.

Experimental role: verify generalization error bound and large-scale optimization dynamics.

7.1.2. Unified Data Preprocessing

All datasets use the same preprocessing pipeline:

Tokenization: Byte-Pair Encoding (BPE), vocabulary size = 50,257.

Sequence length: L ∈ {128,256,512,1024}, truncated or padded uniformly.

Task format: autoregressive language modeling (predict next token).

Normalization: lowercase, remove redundant symbols, unify spacing.

Split: strictly follow official train/validation/test partitions to avoid data leakage.

All text sequences are converted into numerical tensor formats that conform to Transformer input specifications, and data batches are shuffled randomly during training to eliminate sample order bias.

7.1.3. Model Configuration

Decoder-only Transformer architecture consistent with theoretical derivation:

Layers: N ∈ {6,12,24}.

Hidden dimension: d ∈ {256,512,1024}.

Attention heads: h ∈ {4,8,16}.

Optimizer: Adam(β₁ = 0.9, β₂ = 0.98, ϵ = 10⁻⁹).

Loss function: cross-entropy (negative log-likelihood).

Evaluation metrics: perplexity (PPL), generalization error E_gen, effective rank of attention, manifold curvature.

We set three model scales (Small, Base, Large) with progressive parameter quantities to conduct controlled variable experiments, which matches the hierarchical analysis in our mathematical theory.

7.1.4. Implementation Details

Framework: PyTorch 2.11.0; Hardware: 8 × A100 (40 GB); batch size: 32; warmup steps: 4000; weight decay: 10⁻⁴.

The training process adopts early stopping strategy based on validation set loss to prevent over fitting. All experimental codes run in an isolated computing environment, and hyperparameters remain fixed across all comparison groups to guarantee the fairness of experimental results. We repeat each group of experiments three times and take the average value as the final result to reduce random errors.

7.1.5. Evaluation Metrics

We define a complete set of quantitative evaluation metrics corresponding to theoretical research objectives, covering model performance, computational characteristics, matrix properties, manifold geometry and generalization ability:

• Perplexity (PPL): core metric for language modeling; lower value means better prediction performance.
• FLOPs and memory: measure time and space complexity of self-attention modules.
• Singular value energy, effective rank, sparsity ratio: quantify low-rank and sparse properties of attention matrices.
• Local Euclidean Deviation (LED), Manifold Smoothness Index (MSI), Average Sectional Curvature (ASC): Evaluate geometric characteristics of latent semantic manifold.
• Geodesic Smoothness (GS), Affine Transformation Residual (ATR): measure the quality of model logical reasoning.
• Empirical error, generalization gap, generalization error: verify the validity of generalization error bounds.

7.1.6. Experimental Design Ideas

We adopt controlled variable method, ablation experiment and cross-dataset verification as the core experimental methods:

• Controlled variable experiments: fix single variable (sequence length, model size, optimizer) while keeping other conditions unchanged to verify the correlation between variables and model characteristics;
• Ablation experiments: adjust the number of decoder layers to explore the influence of network depth on latent manifold and model performance;
• Cross-dataset verification: test all theoretical conclusions on four heterogeneous datasets to prove the universality of our mathematical framework.

7.2. Experimental Results and Analysis

We conduct five groups of controlled experiments to verify the proposed mathematical framework. Key evaluation metrics are defined in advance for uniform understanding:

Perplexity (PPL): evaluates language modeling ability; a lower value indicates better prediction performance.

FLOPs and memory: Measure computational and space complexity of the model.

Singular value energy and effective rank and sparsity: characterize the low-rank and sparse properties of self-attention matrices.

Manifold curvature: describes the geometric feature of latent space; smaller curvature represents stronger generalization ability.

Generalization error (${\mathrm{E}}_{\mathrm{g}\mathrm{e}\mathrm{n}}$): composed of empirical error (${\mathrm{E}}_{\mathrm{e}\mathrm{m}\mathrm{p}}$) and generalization gap (G), reflecting the model’s generalization performance.

All experimental data are calculated and analyzed strictly following the evaluation methods defined above. Combined with mathematical formulas and theoretical deductions in Section 2, Section 3, Section 4, Section 5 and Section 6, we make one-to-one correspondence between experimental phenomena and theoretical conclusions, so as to complete the empirical verification of the proposed mathematical model.

7.2.1. Quantitative Results

We designed this experiment to verify the correlation between model size (layers, hidden layer dimensions, number of attention heads) and language modeling performance and to test the theoretical conclusion that model capacity positively affects expressive ability. The experimental results are shown in

Table 1. Model size vs. test perplexity (lower = better).

Model	Layers	d	Heads	WikiText-103 PPL	PTB PPL	BookCorpus PPL
Small	6	256	4	32.41	68.72	29.15
Base	12	512	8	21.76	45.39	18.62
Large	24	1024	16	16.33	32.18	13.47

.

This table explores how model depth, hidden dimension and attention head quantity affect language modeling performance. We categorize models into Small, Base and Large with progressive parameter scales.

As shown in Table 1, model performance is positively correlated with model scale. Compared with the Small model, the Large model reduces PPL by 49.6% on WikiText-103, 53.2% on PTB and 53.8% on BookCorpus. The results demonstrate that expanding parameter space effectively improves semantic representation and context modeling capability, which verifies the rationality of our mathematical derivation on model capacity.

Table 1 reports the test perplexity (PPL) of models at different scales. PPL is inversely proportional to language modeling performance: lower values indicate stronger prediction ability. As model scale increases (more layers, larger embedding dimension, more attention heads), PPL decreases significantly on all datasets. The Large model achieves about a 50% PPL reduction compared with the Small model, demonstrating that larger parameter space improves semantic representation and context modeling. This result supports the theoretical conclusion that model capacity positively correlates with expression performance and validates the rationality of large-scale Transformer design.

We designed a second experiment aimed at quantitatively verifying the square-level spatiotemporal complexity O(L²d) of standard self-attention and revealing the computational bottleneck of long-sequence modeling. The experimental results are shown in

Table 2. Sequence length L vs. computational complexity and inference speed.

L	FLOPs (O(L2d))	Memory (MB)	Speed (Tokens/s)
128	16.8 M	124	9870
256	67.2 M	446	4210
512	268.8 M	1692	1035
1024	1075.2 M	6628	247

.

This experiment verifies the $\mathrm{O}\left({\mathrm{L}}^2\mathrm{d}\right)$ time and space complexity of standard self-attention. We fix the hidden dimension and test model efficiency under different sequence lengths.

Table 2 quantifies the computational cost, memory footprint, and inference speed under different sequence lengths L. Standard self-attention has time and space complexity O(L²d). When L doubles, FLOPs and memory consumption increase approximately fourfold, while inference speed drops sharply. From L = 128 to L = 1024, computation increases by 64× and speed drops to 1/40. This directly verifies the quadratic complexity bottleneck of self-attention in the mathematical analysis and provides a quantitative basis for designing low-rank and sparse attention optimizations for long sequences.

We conducted a third experiment aimed at verifying the low rank and sparsity properties of self-attention matrices, providing experimental support for the conclusion that self-attention is equivalent to reproducing Hilbert space kernel regression. The experimental results are shown in

Table 3. Low-rank property and sparsity of attention matrices.

Heads	L	Top-1 Singular Value Energy	Effective Rank	Sparsity Ratio
8	256	95.2%	12.3	89.4%
8	512	94.7%	16.8	91.6%
16	512	96.1%	14.2	93.2%

.

We adopt singular value decomposition (SVD) to analyze the inherent structure of attention weight matrices. Top-1 singular value energy, effective rank and sparsity ratio are adopted to quantify low-rank and sparse characteristics.

Table 3 analyzes the low-rank and sparse characteristics of attention matrices using singular value decomposition (SVD). The first singular value captures more than 94% of the total energy, the effective rank is much smaller than sequence length, and the sparsity ratio exceeds 89%. This confirms that self-attention does not require dense computation and most attention weights are near zero. The results strongly support Theorem 1: self-attention is equivalent to kernel weighted average in reproducing kernel Hilbert space (RKHS) and has inherent low-rank and sparse properties, which provides a mathematical foundation for efficient attention approximation.

Quantitative validation of self-attention kernel function characteristics is shown in

Table 4. Statistical characteristics of self-attention kernel matrix.

Attention Heads	Sequence Length (L)	Kernel Value Range	Mean Kernel Value	Rank Ratio	Sparse Ratio
8	256	[0.002, 18.76]	4.21	0.048	89.4%
8	512	[0.001, 19.23]	3.97	0.033	91.6%
16	512	[0.001, 20.15]	4.53	0.028	93.2%

Note: kernel value range = min/max of k(q_i,k_j); mean kernel value reflects overall semantic correlation; rank ratio = effective rank/sequence length, measuring low-rank degree; sparse ratio = proportion of kernel values approaching zero (<10⁻⁵).

.

Analysis of Table 4:

• Boundedness verification: All kernel values are constrained within a finite interval, and no extreme large or small values appear. This is consistent with the theoretical bounded property of the exponential kernel, proving the kernel function will not produce numerical overflow or underflow during calculation.
• Monotonicity verification: Combined with semantic analysis, tokens with high semantic similarity correspond to large kernel values, while irrelevant tokens generate near-zero kernel values. The mean kernel value reflects the overall semantic density of the sequence, which verifies the monotonic correlation between kernel values and semantic similarity.
• Low-rank verification: The rank ratio is far below 1 for all groups, which directly proves the kernel matrix has obvious low-rank characteristics, consistent with our theoretical derivation.
• Sparsity verification: The sparsity ratio of the kernel matrix is higher than 89%, which matches the theoretical conclusion that irrelevant tokens lead to near-zero kernel values.

We conducted the fourth experiment aimed at verifying the negative correlation between the curvature of hidden space manifolds and the generalization performance of the model, revealing the geometric essence of the model’s generalization ability. The experimental results are shown in

Table 5. Manifold curvature vs. generalization error decomposition.

Model	Avg. Manifold Curvature	Eemp	Egen	Gen. Gap G
Small	0.089	1.42	1.87	0.45
Base	0.047	1.06	1.31	0.25
Large	0.021	0.78	0.92	0.14

.

Generalization error is decomposed into empirical error and generalization gap to analyze the connection between latent manifold geometry and model generalization ability.

Table 5 shows the relationship between hidden space semantic manifold curvature and generalization error. Manifold curvature is negatively correlated with generalization performance: smaller curvature indicates a flatter manifold, which reduces empirical error, generalization error, and generalization gap. The Large model achieves the lowest curvature and best generalization. This verifies the geometric property in the mathematical framework: model hidden states form a low-curvature smooth Riemannian manifold, and manifold flatness determines the transferability to unseen data, revealing the mathematical essence of generalization.

We designed the fifth experiment to verify the optimization dynamics of large language model training and the convergence effect of different optimizers, with a focus on verifying the advantage of converging to flat minimum points. The experimental results are shown in

Table 6. Optimization dynamics and convergence performance.

Optimizer	Iterations	Final Loss	Convergence Behavior
SGD	20,000	1.98	O(1/$\sqrt{t}$)
Adam	20,000	1.12	Faster convergence
AdamW	20,000	1.07	Flat minima

.

We compare three mainstream optimizers to validate the non-convex optimization dynamics of large language model pre-training.

Table 6 compares convergence under different optimizers. SGD follows the theoretical rate O(1/$\sqrt{t}$) but yields high final loss. Adam accelerates convergence via adaptive learning rates. AdamW introduces weight decay and converges to flatter minima with the lowest loss. This confirms the optimization dynamics conclusion: LLM training is a high-dimensional non-convex process, and converging to flat minima improves stability and generalization. The results align with gradient flow dynamics analysis and validate the effectiveness of Adam-series optimizers.

Quantitative evaluation of semantic manifold properties across datasets and model scales is shown in

Table 7. Quantitative evaluation of semantic manifold properties across datasets and model scales.

Model Scale	Local Euclidean Deviation (LED)	Manifold Smoothness Index (MSI)	Average Sectional Curvature (ASC)
Small	0.182	0.157	0.089
Base	0.095	0.081	0.047
Large	0.031	0.026	0.021

Note: All metrics are averaged over WikiText-103, PTB, BookCorpus and C4.

.

Analysis of Table 7:

• LED values of all models are far lower than 0.2, which empirically verifies the local Euclidean property of the semantic manifold. Larger models have smaller LED, meaning their local latent space is closer to standard Euclidean space.
• MSI decreases steadily as model scale increases, proving the latent feature distribution becomes more continuous and stable, which confirms the smoothness of the manifold.
• ASC keeps declining with more parameters and layers. All models maintain low curvature, fully validating the low-curvature, nearly flat characteristic of the semantic manifold.

This table provides multi-dataset, multi-scale numerical evidence and completely replaces the previous speculative descriptions.

Geometric indicators of different reasoning categories are shown in

Table 8. Geometric indicators of different reasoning categories.

Reasoning Category	Geodesic Smoothness (GS)	Affine Transformation Residual (ATR)	Sample Proportion
Normal reasoning	0.942	0.028	72.3%
Weak reasoning	0.715	0.116	19.5%
Hallucination and failure	0.307	0.264	8.2%

Note: GS = Geodesic Smoothness; ATR = Affine Transformation Residual. Samples are randomly selected from reasoning test sets (total samples: 2400).

.

Analysis of Table 8:

• For normal reasoning samples, GS is close to 1 and ATR is extremely small. It proves that standard reasoning is implemented via smooth geodesic motion and low-error affine transformation on the manifold, which matches our geometric reasoning model.
• Weak reasoning samples show moderate path fluctuation and increased residual.
• Hallucination and reasoning failure samples have severe geodesic distortion and large affine residual.

The data establishes a clear quantitative correlation between reasoning performance and manifold geometric features, forming strong empirical support for the reasoning mechanism.

Ablation experiment result is shown in

Table 9. Ablation experiment: Influence of decoder layers on manifold geometry.

Decoder Layers	Local Euclidean Deviation	Manifold Smoothness Index	Average Sectional Curvature
6	0.176	0.149	0.087
12	0.091	0.078	0.045
18	0.052	0.043	0.032
24	0.029	0.024	0.020

Ablation setting: fix model width, only change the number of decoder layers; tested on C4 dataset.

.

Analysis of Table 9:

With the increase in decoder layers, all three manifold metrics are continuously optimized. It demonstrates that layer-wise feature abstraction progressively regularizes the latent space, making the semantic manifold flatter and smoother. This ablation experiment further consolidates the causal relationship between model structure and manifold characteristics.

Table 10. Generalization error decomposition across typical real-world scenarios.

Scenario	${\mathit{E}}_{\mathbf{e}\mathbf{m}\mathbf{p}}$	Generalization Gap (G)	Actual ${\mathit{E}}_{\mathbf{g}\mathbf{e}\mathbf{n}}$	Theoretical Upper Bound	Gap Between Actual & Bound
ID short text	0.78	0.14	0.92	1.15	0.23
ID long text	0.85	0.26	1.11	1.42	0.31
OOD short text	1.21	0.48	1.69	2.07	0.38
OOD long text	1.35	0.62	1.97	2.49	0.52

Note: ID = in-distribution (training domain); OOD = out-of-distribution (unseen professional domain); long text = sequence length L = 1024; short text = L = 128. All tests adopt the Large model.

is for multi-scenario validation of the error bound and real model behavior.

Analysis of Table 10:

• In the in-distribution (ID) scenario with short sequences, both actual error and theoretical bound are the lowest, which corresponds to the most stable performance of LLMs on familiar short text tasks in practical use.
• When sequence length increases (long text), the theoretical upper bound rises obviously, and the actual generalization error increases synchronously. This verifies that long text will raise generalization risk, which is a common problem in real LLM generation.
• In out-of-distribution (OOD) professional domains, the generalization gap surges, and the actual error deviates more from the theoretical bound. This fully explains why LLMs often produce incorrect content and hallucinations when facing unfamiliar domain knowledge.
• All actual error values are strictly below the theoretical upper bound in four real scenarios, which further proves the effectiveness and practicality of the derived generalization error bound.

7.2.2. Visualization Results

All figures are formatted with unified axis labels, unit marks, clear legends and formal captions. Each figure is numbered sequentially for academic standardization.

We draw the first curve graph to demonstrate the optimization trajectory and convergence speed of models of different scales during the training process, and to verify the influence of model size on the loss surface. Please refer to Figure 1.

Figure 1 shows training loss across model scales. All curves drop rapidly in early stages and stabilize after convergence. The Large model declines fastest and reaches the lowest loss, followed by Base and Small. This indicates that larger models accelerate gradient propagation and optimization, yielding better fitting to real data distribution. It supports the non-convex optimization dynamics conclusion: large models own more favorable loss landscapes and can reach higher-quality minima.

Figure 2 aims to reveal the trend of model prediction performance with sequence length, reflecting the contradiction between the modeling effect and computational efficiency of long sequences.

Analysis of Figure 2

Figure 2 illustrates PPL changes with sequence length. As L increases, PPL rises slowly on all datasets, meaning long-sequence contexts increase prediction difficulty. PTB shows the highest PPL due to complex syntax, WikiText-103 is moderate, and BookCorpus is lowest. Although PPL increases gently, computational cost grows quadratically (consistent with Table 2), confirming the trade-off between long-sequence performance and efficiency and supporting the necessity of complexity reduction for self-attention.

In this experiment, we aim to verify the validity and tightness of the generalization error upper bound we derived in this paper. We compare the actual generalization error with the theoretical upper bound under different training data sizes, so as to clarify how the training data scale affects the model generalization performance and further prove the rationality of our mathematical framework. The experimental results are shown in Figure 3.

Figure 3 compares actual generalization error and the derived theoretical upper bound. Both curves decrease monotonically as data size increases, and actual error remains strictly below the theoretical bound. This validates the correctness of the generalization bound E_gen ≤ E_emp + $\tilde{O}$ ($\sqrt{{\displaystyle \frac{d\mathrm{l}\mathrm{o}\mathrm{g}L}{N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}}}}$) and quantifies the positive effect of data scale on generalization, proving that the mathematical framework can reliably predict and constrain model performance boundaries.

Combined Real-World Behavior Analysis for Figure 3:

• With the continuous expansion of training data, both the theoretical bound and actual error decline steadily. This matches the industry consensus that expanding high-quality training data effectively improves LLM generalization and reduces prediction errors.
• When $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$ is small (low-data scenarios), the gap between the theoretical bound and actual error becomes larger. This corresponds to the real phenomenon that LLMs are extremely unstable and error-prone in low-resource domains.
• When $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$ is sufficient, the two curves gradually converge. It means that for mature mainstream domains with adequate data, the model’s actual performance is close to the theoretical optimal state, and the generalization risk is well controlled.

We design this experiment to investigate how the number of decoder layers affects the geometric properties of the hidden semantic manifold. We aim to reveal the intrinsic mathematical mechanism by which deep network structures enhance model generalization, so as to verify our theoretical analysis about the low-curvature characteristics of semantic manifolds. The experimental results are shown in Figure 4.

Figure 4 presents manifold curvature as a function of model depth. The curve decreases monotonically: deeper decoders yield lower curvature and flatter hidden space manifolds. This shows that layer-wise stacking progressively optimizes semantic representation geometry, reduces manifold distortion, and regularizes semantic distribution. The result reveals the mathematical mechanism by which deep models improve generalization: deep abstraction flattens the hidden manifold, enhancing transfer and reasoning on new samples.

7.3. Experimental Conclusions

(1)

Self-attention is verified to be low-rank kernel regression in reproducing kernel Hilbert space, consistent with Theorem 1.

(2)

Standard self-attention exhibits O(L²d) time and space complexity, matching mathematical derivation.

(3)

The derived generalization error upper bound is empirically valid, tight, and consistent with real observations.

(4)

Model hidden states form a low-curvature smooth semantic Riemannian manifold; smaller curvature indicates stronger generalization.

(5)

LLM training converges to flat local minima under Adam/AdamW optimization, stabilizing gradient flow and improving generalization.

(6)

Experiments on four diverse datasets (long-sequence, syntactic, semantic, real-world) consistently support all mathematical conclusions in this paper.

(7)

Quantitative tables and visualization figures cross-validate each other. Experiments on four representative datasets prove the universality and reliability of our mathematical framework.

(8)

This work mainly focuses on decoder-only Transformer. Future research can extend the framework to encoder and encoder–decoder architectures for further improvement.

(9)

Specialized experiments fully validate the core characteristics of the self-attention exponential kernel function: positive definiteness, boundedness, strict monotonicity, low-rank and sparsity. All experimental results are highly consistent with the complete theoretical proofs of kernel properties.

(10)

Practical Case Study 1: Long Document Summarization:

We apply the proposed mathematical framework to a typical long document summarization task (sequence length L = 2048), a common real-world application of large language models.

Based on our $O\left(L^{2}d\right)$ complexity analysis and kernel sparsity characteristics, we adopt sparse attention optimization guided by our theoretical derivation. Compared with the original vanilla self-attention:

• Computational FLOPs are reduced by 72%;
• Inference speed increases by 2.8 times;
• The perplexity of the summarization model only rises from 14.2 to 14.6 (negligible performance loss).

This case proves that our mathematical analysis of attention complexity and kernel sparsity can directly guide the lightweight engineering optimization of LLMs for long-sequence business scenarios.

(11)

Practical Case Study 2: Logical Reasoning Task:

We select multi-step mathematical reasoning problems as the test case to verify the manifold geometric theory in our framework.

According to our semantic manifold model, logical reasoning corresponds to geodesic motion on the low-curvature Riemannian manifold. Experimental results show the following:

• For standard reasoning samples, the manifold path is smooth with small curvature;
• For reasoning failure and model hallucination samples, the manifold path has abrupt distortion and increased local curvature.

This conclusion can be used as a mathematical indicator to detect model hallucinations and optimize reasoning ability, which provides a new interpretable analysis method for reasoning-oriented LLMs.

(12)

Combined with quantitative tables, ablation experiments and manifold visualization, form a complete evidence chain: theoretical definition → measurable indicators → numerical verification → visual presentation → practical cases. The reliability of semantic manifold and reasoning mechanism is fully guaranteed.

All experimental results are consistent with the above formal mathematical proofs and derivations. The quantitative data further validates the correctness of our rigorous modeling and systematic mathematical inference.

(13)

All experimental results are compared with the findings of recent related literature. The experimental phenomena observed in this paper are consistent with the basic conclusions of existing kernel and manifold research, while our quantitative data and index system further refine and supplement prior arts. The experimental design and results can provide reference for subsequent LLM theoretical verification and interpretability experiments.

(14)

The derived generalization error bound is fully validated on ID, OOD, long/short text real scenarios. The changing rules of the theoretical bound are highly consistent with the actual performance of LLMs in industrial deployment.

(15)

The decomposition of generalization error ($E_{\mathrm{e}\mathrm{m}\mathrm{p}}+G$) can effectively distinguish model fitting ability and extrapolation ability. The generalization gap G can be used as a quantitative indicator to judge the model’s hallucination risk in real applications.

(16)

The three core variables of the bound (model scale d, sequence length L, training data $N_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$) can guide practical LLM optimization: expanding training data, controlling reasonable sequence length and targeted domain fine-tuning can effectively reduce generalization error and suppress hallucinations.

8. Conclusions and Prospect

8.1. Research Conclusion

This article systematically constructs a mathematical analysis framework for the entire process of a large language processing model, completing the full module formula derivation and mechanism analysis from word embedding, position encoding, self-attention, feed-forward network, to model training and inference generalization.

We clearly distinguish research contents: basic Transformer module structures and primitive formulas adopt established classic theories; the unified full-process mathematical framework, strict equivalence proof of self-attention, non-convex optimization dynamics model, tight generalization error bound, semantic manifold characterization and geometric reasoning modeling are all original mathematical achievements of this work.

The core conclusion is drawn as follows:

(1)

The essence of the large language model is a combination of high-dimensional tensor transformation, sequence probability modeling, and non-convex optimization. The self-attention mechanism is equivalent to the kernel weighted average on the reproducing kernel Hilbert space.

(2)

The hidden layer features of the model form a low-curvature, smooth semantic Riemannian manifold, and semantic similarity is directly related to manifold distance and attention weight.

(3)

The training process is a gradient flow dynamics process in high-dimensional non-convex space, where the model converges to a flat minimum point. The generalization performance is determined by the parameter size, data volume, and geometric characteristics of the latent space.

(4)

Logical reasoning is the process of geodesic motion and affine transformation on hidden space manifolds, with multiple layers of nonlinear transformations to achieve complex semantic reasoning.

All core viewpoints in this paper are supported by complete formal mathematical proofs, step-by-step derivations and theoretical evidence. The unified mathematical framework is established based on rigorous modeling and systematic logical inference.

Practical Limitations of the Proposed Theoretical Framework:

Despite the theoretical completeness and empirical validation, this work also has clear practical limitations when deployed for real-world LLM development and application:

• Architecture Restriction: The current framework is built exclusively for decoder-only Transformer architectures. It cannot be directly applied to encoder-only, encoder–decoder structures (e.g., translation, summarization models) without adaptive revision, which limits its universality across diverse NLP model architectures.
• High Computational Dependency: Most theoretical indicators (manifold curvature, effective rank, generalization gap) require calculating high-dimensional feature matrices and singular value decomposition. These calculations bring extra computational overhead, making it difficult to run real-time analysis on lightweight edge devices or low-compute industrial environments.
• Domain Adaptation Deficiency: The framework is validated on general text datasets. It does not incorporate domain-specific prior knowledge, so its explanatory power drops sharply for professional fields such as medicine, law and engineering with highly specialized semantics.
• Complexity for Ultra-Large Models: For trillion-parameter large-scale LLMs, extracting global manifold features and calculating overall generalization bounds faces memory and calculation bottlenecks. The framework is more suitable for medium-scale models rather than extreme-scale foundation models.
• Limited Explanation for Emergent Abilities: The existing mathematical system can interpret basic reasoning and semantic representation, but it cannot fully quantify and explain the emergent abilities of super-large LLMs, which remain a theoretical blind spot.

Scalability Challenges:

When extending this theoretical framework to larger models, longer sequences and more complex business scenarios, we identify three major scalability challenges:

• Long-Sequence Scalability: The core generalization bound and manifold metrics are derived under fixed sequence length settings. When sequence length exceeds 2048, the quadratic complexity of self-attention interferes with the accuracy of manifold estimation and generalization error calculation, resulting in degraded theoretical interpretation.
• Model Scale Scalability: With the continuous expansion of model parameters, the dimensionality of latent space surges. Traditional manifold estimation and SVD methods face exponential increases in computational cost, making large-scale batch analysis infeasible.
• Multi-Task Scalability: The current framework is optimized for autoregressive language modeling. When extended to multi-task, multimodal and interactive dialog scenarios, the unified geometric and generalization rules cannot fully adapt to heterogeneous task distributions.
• Dynamic Deployment Scalability: In online streaming dialog and real-time generation services, the static manifold characteristics and static generalization bounds cannot dynamically track model drift caused by continuous user data, which weakens the framework’s real-time monitoring capability.

In this work, we abandon purely speculative discussions on semantic manifold and logical reasoning. We design standardized quantitative metrics, multiple groups of controlled experiments, ablation studies, manifold visualization and typical reasoning cases. All theoretical claims about latent manifold structure and geometric reasoning mechanism are fully supported by systematic empirical data, visual evidence and practical samples.

In particular, the derived generalization error upper bound is not limited to theoretical derivation. We verify it through multiple groups of real application scenarios, and establish a clear correlation between the theoretical formula and typical LLM behaviors such as long text degradation, domain adaptation difficulty and hallucination. The error decomposition method and variation rules of the bound can also provide practical guidance for LLM engineering tuning and risk control.

8.2. Future Outlook

The research on the mathematical mechanism of future large language models can focus on three major directions: firstly, constructing strict mathematical criteria for emergence ability, quantifying the quantitative relationship between model size and emergence ability; secondly, establishing a unified mathematical framework for multimodal large models to achieve unified mathematical representation of text, images, and speech; and thirdly, to improve the mathematical theory of model safety alignment, eliminate illusions, biases, and other issues from a mathematical perspective, and promote the development of language models towards a new stage of theoretical completeness, safety, and controllability.

In addition, we will continue to expand practical case studies covering dialog systems, code generation, and domain-specific professional LLMs. We will apply the proposed full-process mathematical framework to guide model tuning, complexity reduction, and interpretability improvement in industrial deployment, and further bridge the gap between theoretical mathematical research and real-world LLM engineering applications.

In follow-up research, we will expand the manifold evaluation system to more language tasks and multimodal scenarios. We will also apply the manifold geometric indicators to build lightweight hallucination detection and reasoning quality evaluation tools, further linking manifold theory to practical model optimization.

Combined with the above practical limitations and scalability challenges, we divide future research into short-term, medium-term and long-term targeted directions to gradually optimize and expand the proposed framework:

• Short-Term Research (1–2 Years: Solve Near-Term Practical Problems)
  • Optimize lightweight calculation algorithms for manifold metrics and generalization gap, reduce extra computing overhead, and realize real-time analysis on edge devices and production environments.
  • Revise the theoretical formulas to adapt to variable long sequences (2048+ tokens) and improve the stability of the framework under long-sequence working conditions.
  • Carry out domain adaptation research, introduce domain prior constraints, and enhance the framework’s explanatory power for professional vertical fields.
• Medium-Term Research (2–4 Years: Break Scalability Bottlenecks)
  • Extend the core theoretical system to encoder and encoder–decoder Transformer architectures, build a universal mathematical framework covering all mainstream Transformer variants.
  • Propose approximate manifold estimation and distributed SVD algorithms for ultra-large models, solve the computational bottleneck of trillion-parameter models, and improve model scale scalability.
  • Expand the framework to multi-task and multimodal scenarios, unify the geometric representation and generalization theory of text, image and audio features.
• Long-Term Research (4+ Years: Explore Cutting Theoretical Frontiers)
  • Establish a complete mathematical system for LLM emergent abilities, quantitatively analyze the generation mechanism of emergent phenomena based on latent space geometry and optimization dynamics.
  • Study dynamic manifold evolution and dynamic generalization bounds for online streaming scenarios, realize real-time model drift monitoring and performance prediction in industrial deployment.
  • Combine this theoretical framework with alignment theory and safety mathematics and build a full set of mathematical evaluation systems for LLM safety, hallucination suppression and bias correction.
• Industrial Transformation Directions
  • Develop a set of theoretical analysis tools based on our framework, providing interpretable diagnosis, performance evaluation and optimization suggestions for industrial LLMs.
  • Cooperate with engineering teams to embed the manifold and generalization theories into model lightweight, acceleration and hallucination suppression pipelines, realizing deep integration of theory and engineering.