Large Language Model Architecture
Large language models (LLMs) are a class of deep neural networks built on the transformer architecture, which replaced recurrence entirely with attention mechanisms to process sequences in parallel[^c1]. The transformer, introduced in 2017, consists of stacked encoder and decoder blocks that use self-attention to capture relationships between all positions in a sequence simultaneously, enabling efficient training on vast text corpora and effective modeling of long-range dependencies. Modern LLMs predominantly use decoder-only architectures that generate text autoregressively by predicting the next token given the preceding context[^c2].
The core components of a transformer block include multi-head self-attention, feed-forward networks, layer normalization, and residual connections. Self-attention computes query, key, and value projections from the input and uses scaled dot-product attention to weight the importance of different positions[^c1]. Multi-head attention runs multiple attention operations in parallel, allowing the model to attend to information from different representation subspaces. Feed-forward networks, which account for a large fraction of model parameters and nonlinear expressivity, apply learned transformations to each position independently[^c3]. Layer normalization and residual connections stabilize training in deep networks by maintaining consistent activation distributions and providing direct gradient pathways.
Training large language models involves several stages. Pre-training uses objectives such as causal language modeling (predicting the next token) or masked language modeling (predicting masked tokens from bidirectional context) to learn general linguistic and factual knowledge from enormous text datasets[^c2]. Scaling laws describe the relationship between model size, data size, and compute budget, showing that performance follows predictable power-law trends across many orders of magnitude[^c4]. Post-training techniques including instruction tuning and reinforcement learning from human feedback (RLHF) align model outputs with human preferences and safety requirements.
Efficient training and deployment of LLMs require specialized parallelism strategies. Data parallelism replicates the model across devices and splits the batch, while tensor parallelism partitions individual layers and pipeline parallelism splits the model by depth[^c5]. ZeRO optimization reduces memory consumption by sharding optimizer states, gradients, and parameters across data-parallel processes. Quantization compresses models by reducing the bit-width of weights and activations, enabling deployment on resource-constrained hardware with minimal accuracy loss[^c6].