information-theory - Concepts
Explore concepts tagged with "information-theory"
Total concepts: 12
Concepts
- Noisy-Channel Coding Theorem - Shannon's foundational theorem proving that reliable communication is possible at any rate below channel capacity, even over arbitrarily noisy channels.
- Context Signal-to-Noise Ratio - Proportion of task-relevant versus irrelevant information in an AI agent's context window, serving as the core metric that context engineering optimizes.
- Context Entropy - Natural tendency of AI context systems to degrade toward disorder over time, accumulating contradictions, redundancies, and noise until usefulness declines.
- Shannon Entropy - Information-theoretic measure of the average uncertainty or surprise carried by a random variable, quantified in bits.
- Kolmogorov Complexity - The length of the shortest computer program that produces a given object as output, formalizing the intrinsic information content or 'descriptive complexity' of a string.
- Grossman-Stiglitz Paradox - The paradox that if markets are informationally efficient, there is no incentive to gather information, which undermines that efficiency.
- Cross-Entropy - An information-theoretic measure of dissimilarity between two probability distributions, ubiquitous as the loss function for classification and language modeling.
- Channel Capacity - The maximum rate at which information can be reliably transmitted over a communication channel, measured in bits per channel use or bits per second.
- KL Divergence - An asymmetric measure of how much one probability distribution differs from a reference distribution, foundational to information theory and modern machine learning.
- Huffman Coding - A lossless compression algorithm that assigns shorter binary codes to more frequent symbols, achieving optimal prefix-free encoding for known symbol probabilities.
- Mutual Information - A measure of how much knowing one random variable reduces uncertainty about another, capturing the strength of any relationship — linear or not — between them.
- Information Compression - The process of condensing information into its most essential form while preserving meaning, enabling faster processing and better retention.
← Back to all concepts