Emergent Abilities
Capabilities that appear in large AI models only beyond a critical scale threshold, absent or near-random in smaller models.
Also known as: Emergent Capabilities, Phase Transitions in AI, Emergent Properties of LLMs
Category: AI
Tags: ai, machine-learning, large-language-models, emergence, research
Explanation
Emergent abilities are capabilities that manifest in large language models (LLMs) only when they reach a sufficient scale of parameters, training data, or compute—capabilities that are absent or near-random performance in smaller models. The term was popularized by a 2022 paper from Google Research ('Emergent Abilities of Large Language Models' by Wei et al.).
**What makes them 'emergent':**
In the sense used here, emergence means that performance on certain tasks does not improve gradually with scale but instead shows a sharp phase transition. Below a certain threshold, the model performs at chance level. Above it, performance jumps dramatically. This unpredictability is what makes emergent abilities both exciting and concerning.
**Examples of emergent abilities:**
- **Chain-of-thought reasoning**: Smaller models cannot follow multi-step logical reasoning, but large models can when prompted to 'think step by step'
- **Arithmetic**: Models below ~100B parameters struggle with multi-digit arithmetic; larger models handle it reliably
- **Code generation**: Producing correct, functional code emerges at scale
- **Translation of rare languages**: Only large models achieve useful quality for low-resource languages
- **Instruction following**: The ability to follow complex, multi-part instructions appears at scale
**The debate:**
The concept of emergent abilities is contested:
- **Supporters** argue that emergence is a genuine phenomenon reflecting qualitative changes in model capabilities, possibly related to the model learning new computational strategies at scale
- **Critics** (notably Schaeffer et al., 2023) argue that apparent emergence is an artifact of measurement—when using continuous metrics instead of threshold-based ones, performance improvements appear smooth rather than sudden
**Implications:**
- **Unpredictability**: If abilities emerge unpredictably at scale, we cannot fully anticipate what larger models will be capable of
- **Safety**: Emergent abilities may include undesired behaviors (deception, manipulation) that appear without warning
- **Investment**: The possibility of emergence incentivizes building ever-larger models in hopes of unlocking new capabilities
- **Evaluation**: We need better benchmarks that test for capabilities we don't yet know to look for
**Connection to scaling laws:**
Emergent abilities complicate neural scaling laws, which predict smooth, predictable improvement. Emergence suggests that scaling can produce qualitative jumps, not just quantitative improvements.
Related Concepts
← Back to all concepts