Latent Space
A compressed, multi-dimensional representation space where a model encodes the essential features of its input data.
Also known as: Representation Space, Embedding Space, Hidden Space, Feature Space
Category: AI
Tags: ai, machine-learning, neural-networks, representations, fundamentals
Explanation
Latent space is the internal, compressed representation that a model creates to capture the essential structure of its input data. When a neural network processes text, images, or other data, it transforms the raw input into points in a high-dimensional space where similar concepts are near each other and meaningful operations become possible.
**What 'Latent' Means**:
The word 'latent' means hidden or concealed. Latent space captures hidden structure — the underlying factors and relationships that aren't directly visible in the raw data but explain how data is organized.
**How Latent Space Works**:
Consider how a language model processes the word "king":
1. The raw input is a token ID (just a number)
2. This maps to an embedding vector (e.g., 4,096 numbers)
3. Through layers of processing, this vector moves through latent space
4. In this space, "king" is near "queen", "monarch", "ruler"
5. The famous example: king - man + woman ≈ queen works because of latent space structure
**Properties of Latent Spaces**:
- **Dimensionality reduction**: Raw data (e.g., a 100K-token document) is compressed into a compact representation
- **Semantic similarity**: Similar meanings cluster together
- **Interpolation**: Moving smoothly between points generates meaningful intermediates
- **Arithmetic**: Vector operations can capture semantic relationships
- **Disentanglement**: Ideally, different dimensions capture different independent factors
**Latent Space in Different Models**:
| Model Type | Latent Space Role |
|-----------|------------------|
| LLMs | Hidden states represent meaning at each layer, evolving as the model "thinks" |
| Autoencoders | Encode input to latent space, then decode back — the bottleneck forces useful compression |
| VAEs | Learn a structured latent space where sampling produces valid new data |
| Diffusion models | Start from noise in latent space and gradually denoise to produce images |
| GANs | Generator maps random points in latent space to realistic outputs |
**Latent Space in LLMs**:
In transformer-based language models, each token is represented as a vector that moves through latent space as it passes through the model's layers. Early layers capture surface-level features (syntax, local patterns), while deeper layers capture abstract features (meaning, relationships, reasoning). The model's 'understanding' lives in these latent representations.
**Practical Applications**:
- **Embeddings and semantic search**: Representing text as vectors in latent space enables finding semantically similar documents
- **Image generation**: Navigating latent space to create or modify images
- **Anomaly detection**: Unusual inputs map to unusual regions of latent space
- **Transfer learning**: Latent representations learned for one task transfer to related tasks
- **Interpretability research**: Analyzing latent space structure to understand what models have learned
Related Concepts
← Back to all concepts