Architectural and Methodological Advancements in Large Language Models ^†

Abstract

The evolution of large language models (LLMs) has been marked by significant architectural and methodological breakthroughs that have redefined the landscape of natural language processing. This review examines the key techniques driving modern LLMs, including foundational architectures, novel training methodologies, and cutting-edge performance benchmarks. In addition to offering a performance overview, this work presents a focused and up-to-date architectural benchmark that highlights key design differences between the best-performing open-source and closed-source LLMs, providing actionable insights into their underlying components. Beyond the performance comparison, our analysis details the inherent limitations of the monolithic transformer architecture and outlines emerging strategies. By bridging open-source innovations and proprietary advancements, this review offers a balanced resource for researchers and practitioners navigating this rapidly evolving field.

1. Introduction

The pursuit of artificial intelligence (AI) systems capable of understanding and generating human language dates back to Turing’s foundational work [1]. The early approaches relied on Statistical language models (SLMs) [2,3,4], which were effective for simple NLP tasks but limited in capturing language complexity. The introduction of neural language models (NLMs) [5,6,7] significantly advanced the field, using neural architectures to better represent the linguistic context.

A decisive breakthrough emerged with the transformer architecture [8]. Its attention-based mechanism became fundamental for pretrained language models (PLMs) such as BERT [9] and GPT-2 [10], demonstrating superior performance across NLP benchmarks. Scaling this paradigm led to large language models (LLMs), exemplified by GPT-3 [11], which revealed emergent capabilities like zero-shot and few-shot learning [12].

This review contributes by providing not only a performance benchmark of the leading LLMs but also a detailed architectural analysis. Our benchmark is designed to be light yet focused, emphasizing key design components (e.g., normalization methods, positional encoding, and activation functions) that differentiate these models. This dual perspective helps us understand both empirical performance and the inherent architectural innovations that have driven recent advances.

2. Evolution of Language Models

2.1. Statistical to Neural Transition

Statistical language models [2,3,4] initially dominated NLP tasks but were restricted by the “curse of dimensionality”. Neural language models [5,6,7] introduced distributed representations that improved language understanding.

2.2. Transformer Era

The 2017 paper “Attention Is All You Need” [8] proposed the transformer, eliminating recurrent dependencies in favor of self-attention. This led to the rapid rise of PLMs such as BERT [9] and GPT-2 [10]. Subsequent scaling gave rise to LLMs, exemplified by GPT-3 [11], enabling capabilities like few-shot and zero-shot learning [12].

2.3. Open-Source Initiatives

While closed-source models remain highly publicized, open-source equivalents increasingly rival their performance. LLaMA [13] and its variants have facilitated broader research by making advanced weights accessible, and projects like Vicuna [14] illustrate how community-driven fine-tuning can achieve near-state-of-the-art performance.

3. Architectural Innovations and Techniques in Large Language Models

The architecture of LLMs serves as the foundational framework that dictates their capabilities, efficiency, and range of applications. Over the years, a multitude of architectures and techniques have been proposed, each with unique strengths and limitations. Recent advancements have ushered in an era of hyper-scaled models that leverage innovative techniques to push the boundaries of what is computationally feasible and functionally possible.

The performance metrics presented in

Table 1. Comparison of performance metrics between selected LLMs. EM = exact match; pass@1 = top-1 pass rate; “–” = not reported.

Benchmark	DeepSeek-V3	Qwen2.5-72B-Inst	Llama-3.1-405B-Inst	GPT-4	GPT-3.5 (0-shot CoT)	Claude-3.5
MMLU  [15] (EM)	75.9	71.6	73.3	72.6	–	78.0
GPQA-Diamond (pass@1)	59.1	49.0	51.1	49.9	77.3	65.0
MATH 500  [16] (EM)	90.2	80.0	73.8	74.6	94.8	78.3
AIME 2024 (pass@1)	39.2	23.3	23.3	9.3	74.4	16.0
Codeforces (percentile)	51.6	24.8	25.3	23.6	86.0	20.3
SWE-bench (Resolved)	42.0	23.8	23.8	38.8	–	50.8

offer a quantitative comparison between several state-of-the-art LLMs. In particular, differences in exact match scores, pass rates, and problem-solving percentiles reflect both algorithmic innovations and architectural choices. Although performance is crucial, our focus extends to examining how differences in model components, such as normalization methods, positional encoding strategies, and activation functions, affect overall capabilities. This integrated analysis provides researchers with a clearer picture of the trade-offs between model scale, training complexity, and architectural efficiency.

3.1. Mainstream Architectures

The primary architectural paradigms used in LLMs include the following:

• Encoder–Decoder: Pioneered by Vaswani et al. [8], this architecture is specialized for tasks that necessitate both understanding and generation of sequences. It comprises two main components: an encoder that interprets the input and a decoder that produces the output. Models such as T5 [17] and BART [18] exemplify its effectiveness. (see Figure 1)
• Causal Decoder: Predominantly employed for autoregressive sequence generation, causal decoders condition each token on its preceding tokens. The GPT models [10,11] are quintessential examples of this approach. (see Figure 1)
• Prefix Decoder: Designed for tasks requiring simultaneous encoding and decoding within a single step [19], prefix decoders blend the capabilities of causal decoders and encoder-decoder models. An example is GLM-130B [20], which combines bidirectional context encoding with autoregressive token prediction. (see Figure 1)
• Mixture-of-Experts (MoE): This approach extends the aforementioned architectures by activating only a subset of expert networks for each input token (see Figure 2). Notable models leveraging MoE include the following:

  –

  Mixtral 8x7B (27 September 2023): A sparse mixture-of-experts model (SMoE) with open weights licensed under Apache 2.0. that outperforms Llama 2 (70 B) on selected benchmarks.

  –

  DeepSeek-V3 (26 December 2024): A large MoE model with 671B parameters, where only 37B parameters are activated for each token, enabling efficient scaling and performance.

3.2. Detailed Configuration

3.2.1. Normalization Methods

Normalization techniques play a pivotal role in stabilizing the training of large language models. Various methods have been employed:

• LayerNorm: Computes the mean and variance for each layer’s activations; widely used in models like GPT-3 [11] and BERT [9].
• RMSNorm: A computationally efficient alternative that rescales activations based on the root mean square [21]. The LLaMA family [13] leverages RMSNorm for its efficiency gains.
• DeepNorm: Proposed to stabilize extremely deep transformers by scaling residual connections [22]. GLM-130B [20] adopts DeepNorm in its architecture.

3.2.2. Position Embeddings

Position embeddings encode the sequential information of input tokens. Common methods include the following:

• Absolute: Adds a learned position-specific vector to each input token [9].
• Relative: Primarily seen in models like T5 [17], it modifies attention scores based on token distance.
• Rotary (RoPE, ALiBi): PaLM 2 [23] employs advanced rotary position embeddings for handling long sequences.

3.2.3. Activation Functions

Activation functions introduce non-linearity into the model, significantly impacting performance:

• GeLU: A balanced choice for both efficiency and representational power [24].
• SwiGLU and GeGLU: Variants of Gated Linear Units (GLUs) that often yield better empirical performance; used in PaLM 2 [23] and related large-scale models [25].

3.3. Attention Mechanisms

Attention mechanisms form the cornerstone of sequence modeling in modern LLMs:

• Full Attention: The original transformer uses full self-attention, considering all token pairs in a sequence [8].
• Sparse Attention: Reduces computation by attending only to local or patterned subsets of tokens [26,27].
• Multi-query/Grouped-query Attention: Different heads share parameters for keys and values, aiming to save memory without sacrificing too much in performance [28].
• FlashAttention: Reorganizes the computation to use GPU memory more efficiently, preserving exact attention [29].
• PagedAttention: Splits sequences into non-contiguous blocks to further optimize GPU memory utilization in LLM servers [30].

3.4. Training Strategies for Large Language Models

The training of LLMs requires specialized techniques to manage immense computational complexity. The key strategies are outlined below.

3.4.1. Optimization Settings

The following optimization settings are critical for stabilizing and enhancing the training of LLMs:

• Batch Training: Stability often depends on batch sizing. GPT-3, for example, scaled from 32 k tokens up to 3.2 M tokens [11].
• Learning-Rate Scheduling: A linear warm-up followed by cosine decay is common.
• Choice of Optimizer: Adam/AdamW with $({\beta }_1,{\beta }_2)=(0.9,0.95)$ and $ϵ={10}^{-8}$ is standard; Adafactor [31] is also used to reduce memory usage.
• Stabilization Techniques: Gradient clipping and weight decay are widely adopted [32,33].

3.4.2. Scalable Training Methodologies

To efficiently handle the computational demands of large-scale models (see

Table 2. Key architectural differences between selected LLMs.

Model	Size	Category	Normalization	PE	Activation
DeepSeek-V3	671 B	Causal decoder	Pre-LayerNorm	Learned	SwiGLU
Qwen2.5-72B-Inst	72 B	Causal decoder	RMSNorm	RoPE	SwiGLU
Llama-3.1-405B-Inst	405 B	Causal decoder	RMSNorm	RoPE	SwiGLU
GPT-4	500 B	Causal decoder	Pre-LayerNorm	Learned	GeLU
GPT-3.5	175 B	Causal decoder	Pre-LayerNorm	Learned	GeLU
Claude-3.5	100 B	Causal decoder	Pre-LayerNorm	RoPE	SwiGLU

), several scalable training methodologies have been developed:

• 3D Parallelism: Combines data, pipeline, and tensor parallelism to distribute computational loads [34,35,36].
• ZeRO Optimization: Minimizes memory redundancy in data-parallel training by partitioning optimizer states, gradients, and parameters [37].
• Mixed-Precision Training: Employs 16-bit (FP16) or BF16 numerics to accelerate training [38].

4. Conclusions

In the rapidly evolving landscape of large language models (LLMs), understanding their underlying architectures, training strategies, and key technical advances is crucial. This paper has provided a concise yet comprehensive overview of recent progress in open-source LLMs, highlighting essential techniques such as mixture-of-experts, advanced attention mechanisms, normalization layers, and scalable training methodologies.

Moreover, our review underscores a critical limitation inherent to the transformer architecture—the difficulty of integrating its monolithic design with traditional neural network modules. Future research may benefit from advanced approaches to integrate external tools that enhance accuracy and enable richer fact-checking. These directions promise to mitigate current challenges in training efficiency and multimodal integration, paving the way for more adaptable and robust language models.

It is evident that the gap between open-source and closed-source models is narrowing thanks to rapid innovations in architectures and training methods. As we look to the future, it becomes increasingly clear that the democratization of these technologies will spur wider applications in both academia and industry, potentially transforming how we interact with AI-driven language systems.