Exploring Video Compression

Over the weekend, I was playing badminton with Lemon Reddy, who works on video compression algorithms at Amazon Prime Video. During a break between sets, he was telling me about how he uses Claude Code to research and optimize compression algorithms—diving deep into papers, experimenting with different approaches, and working through complex constraints. His work fascinated me, and it got me curious: How does video compression actually work? What are the mathematical principles behind it? Can I understand and implement these concepts myself?

This blog is my exploration of that curiosity. I started wondering about the fundamentals and dove into understanding video compression from the ground up using Python. Here’s what I discovered along the way.

Why It Matters

Video is data-intensive. A single minute of 1080p video at 30fps can be several gigabytes without compression. Video compression tackles this challenge by:

1. Reducing Storage: Enabling practical storage and archiving of video content
2. Enabling Streaming: Making real-time video delivery over networks feasible
3. Saving Bandwidth: Critical for mobile and internet-constrained environments
4. Preserving Quality: Modern compression algorithms maintain visual quality while drastically reducing file size

Before diving into implementation, it’s important to understand the fundamental concepts:

• The difference between lossy and lossless compression
• Temporal redundancy: Similarity between consecutive frames
• Spatial redundancy: Similarity within a single frame
• Psychovisual redundancy: Limitations of human vision perception

Core Concepts

Let’s start with the fundamental concepts you need to understand video compression:

1. Lossy vs Lossless Compression

Lossless Compression: Perfect reconstruction of original data

• Used when every bit matters (medical imaging, archives)
• Lower compression ratios (typically 2-5x)
• Examples: PNG, FLAC audio

Lossy Compression: Accepts some data loss that’s imperceptible to humans

• Used for streaming video and audio
• Higher compression ratios (10-100x or more)
• Examples: JPEG, MP4, H.265

Video streaming almost always uses lossy compression because:

• Perfect reconstruction is unnecessary for visual content
• Humans can’t perceive certain frequency ranges and color information
• It achieves much better compression ratios

2. Temporal Redundancy

Most video frames are very similar to the previous frame. By exploiting this:

Key Frame: Store full frame (I-frame)
Predicted Frame: Store only differences from previous frame (P-frame)

This is why a scene with minimal motion compresses much better than one with lots of movement.

3. Spatial Redundancy

Within a single frame, nearby pixels have similar colors. Instead of storing each pixel individually:

• Chroma Subsampling: Store color at lower resolution than brightness
  • 4:4:4 = No subsampling (high quality, less compression)
  • 4:2:0 = Color at 1/4 resolution (standard for streaming)
• Transform Coding: Convert pixel blocks into frequency components where you can discard less important information

4. Chroma Subsampling

Human eyes are much more sensitive to brightness changes than color changes:

• 4:4:4: Full resolution for luminance (Y) and chrominance (Cb, Cr)
• 4:2:2: Half horizontal color resolution
• 4:2:0: Half resolution in both dimensions (standard for most streaming)

5. Quantization

Quantization is the step where most of the actual loss happens. Transform coding (like the DCT) by itself is reversible: it just rewrites 64 pixel values as 64 frequency coefficients. Nothing is thrown away until quantization.

The idea is simple: divide each coefficient by a step size and round to the nearest integer. The decoder multiplies back by the same step size, but the rounding error is gone for good.

coefficient = 87,  step = 16  →  87 / 16 = 5.4  →  stored as 5
decoder:        5 × 16 = 80   (error of 7, which we accept)

coefficient = 6,   step = 16  →  6 / 16 = 0.4   →  stored as 0
decoder:        0 × 16 = 0    (small value erased entirely)

Two things make this effective:

• Bigger step size = coarser values. More coefficients round to zero, which shrinks the file but adds visible artifacts like blockiness and banding. This single knob is what the “quality” or QP (quantization parameter) setting in an encoder controls.
• The step size differs per frequency. Low-frequency coefficients (smooth gradients, overall shape) carry what the eye notices most, so they get small steps. High-frequency coefficients (fine detail, noise) are less noticeable, so they get large steps and mostly become zero. That’s what the quality matrix in the code below is.

The long runs of zeros left behind are what make the next stages, run-length and Huffman (entropy) coding, so effective.

Where Codecs Are Today

The techniques above are the foundation of every modern codec, and the codecs differ mainly in how cleverly they apply them (bigger and more flexible blocks, better prediction, smarter entropy coding).

• H.264/AVC is still the most widely compatible baseline.
• VP9 and AV1 are what most large-scale streaming runs on today. They are royalty-free and save significant bitrate over H.264 at the same quality. VP9 is widely used on YouTube, and AV1 adoption is growing quickly as hardware decoding becomes common.
• AV2 is the next-generation standard from the Alliance for Open Media, the successor to AV1. It is not yet what services stream today.

Python Basics

Now let’s implement these concepts in Python to solidify your understanding:

Transform Coding

import numpy as np
from scipy.fftpack import dct

# Example 8x8 DCT block (typical in JPEG/video codecs)
pixel_block = np.array([
    [139, 144, 149, 153, 155, 155, 155, 155],
    [144, 151, 153, 156, 159, 156, 156, 156],
    [149, 153, 156, 159, 164, 160, 162, 162],
    [153, 156, 159, 162, 168, 169, 176, 175],
    [155, 159, 164, 168, 176, 190, 194, 191],
    [155, 156, 160, 169, 190, 205, 214, 215],
    [155, 156, 162, 176, 194, 214, 220, 205],
    [155, 156, 162, 175, 191, 215, 205, 196]
])

# Apply 2D DCT
dct_block = dct(dct(pixel_block.astype(float), axis=0), axis=1)

# Quantization: divide each coefficient by its step size, then round
# Larger step sizes = more zeros = more aggressive compression
quality_matrix = np.array([
    [16, 11, 10, 16, 24, 40, 51, 61],
    [12, 12, 14, 19, 26, 58, 60, 55],
    [14, 13, 16, 24, 40, 57, 69, 56],
    [14, 17, 22, 29, 51, 87, 80, 62],
    [18, 22, 37, 56, 68, 109, 103, 77],
    [24, 35, 55, 64, 81, 104, 113, 92],
    [49, 64, 78, 87, 103, 121, 120, 101],
    [72, 92, 95, 98, 112, 100, 103, 99]
])

quantized = np.round(dct_block / quality_matrix).astype(int)

print("Original block shape:", pixel_block.shape)
print("DCT coefficients (top-left 4x4):")
print(dct_block[:4, :4])
print("\nQuantized coefficients (top-left 4x4):")
print(quantized[:4, :4])

# High-frequency components are heavily quantized to zeros
# This achieves compression while maintaining visual quality
zero_count = np.sum(quantized == 0)
print(f"\nZeros after quantization: {zero_count}/64 ({100*zero_count/64:.1f}%)")
    

# Run-Length Encoding (RLE) - Simple compression for repeated values
# Used for storing quantized DCT coefficients

def rle_encode(data):
    """Encode data using run-length encoding"""
    if not data:
        return []
    
    encoded = []
    count = 1
    prev = data[0]
    
    for i in range(1, len(data)):
        if data[i] == prev and count < 255:
            count += 1
        else:
            encoded.append((count, prev))
            prev = data[i]
            count = 1
    encoded.append((count, prev))
    return encoded

def rle_decode(encoded):
    """Decode run-length encoded data"""
    decoded = []
    for count, value in encoded:
        decoded.extend([value] * count)
    return decoded

# Example: Quantized DCT output often has many zeros
data = [5, 3, 3, 3, 0, 0, 0, 0, 0, 0, 0, 1, 2, 0, 0, 0, 0]

encoded = rle_encode(data)
print("Original data:", data)
print("Encoded (count, value):", encoded)
print(f"Compression ratio: {len(data)} → {len(encoded)*2} bytes")

decoded = rle_decode(encoded)
print("Decoded data:", decoded)
print("Match:", data == decoded)
    

import heapq
from collections import defaultdict, Counter

class HuffmanNode:
    def __init__(self, char, freq):
        self.char = char
        self.freq = freq
        self.left = None
        self.right = None
    
    def __lt__(self, other):
        return self.freq < other.freq

def build_huffman_tree(data):
    """Build Huffman tree from frequency data"""
    frequency = Counter(data)
    
    if len(frequency) == 1:
        return HuffmanNode(list(frequency.keys())[0], 1)
    
    heap = [HuffmanNode(char, freq) for char, freq in frequency.items()]
    heapq.heapify(heap)
    
    while len(heap) > 1:
        left = heapq.heappop(heap)
        right = heapq.heappop(heap)
        
        parent = HuffmanNode(None, left.freq + right.freq)
        parent.left = left
        parent.right = right
        heapq.heappush(heap, parent)
    
    return heap[0]

def build_codes(node, prefix="", codes=None):
    """Generate Huffman codes from tree"""
    if codes is None:
        codes = {}
    
    if node is None:
        return codes
    
    if node.char is not None:
        codes[node.char] = prefix if prefix else "0"
        return codes
    
    build_codes(node.left, prefix + "0", codes)
    build_codes(node.right, prefix + "1", codes)
    return codes

# Example: Compression of quantized DCT coefficients
data = [0, 0, 0, 1, 2, 0, 3, 0, 0, 1, 5, 0, 0, 0, 2]

tree = build_huffman_tree(data)
codes = build_codes(tree)

print("Symbol frequencies:", Counter(data))
print("\nHuffman codes:")
for symbol, code in sorted(codes.items(), key=lambda x: len(x[1])):
    print(f"  {symbol}: {code} (length: {len(code)})")

# Encode
encoded_bits = "".join(codes[symbol] for symbol in data)
print(f"\nOriginal size: {len(data) * 8} bits (8 bits per symbol)")
print(f"Encoded size: {len(encoded_bits)} bits")
print(f"Compression ratio: {len(data)*8 / len(encoded_bits):.2f}x")
    

OpenCV and NumPy

Here’s a practical example using OpenCV to work with video frames:

import cv2
import numpy as np

# Load a video file
cap = cv2.VideoCapture('video.mp4')

# Read first frame
ret, frame = cap.read()

if ret:
    # Get frame dimensions
    height, width, channels = frame.shape
    print(f"Frame size: {width}x{height}, Channels: {channels}")
    
    # Convert BGR to RGB for processing
    frame_rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    
    # Convert to YUV (better for compression than RGB)
    frame_yuv = cv2.cvtColor(frame, cv2.COLOR_BGR2YUV)
    
    # Resize frame (reduces data, useful for streaming)
    small_frame = cv2.resize(frame, (width//2, height//2))
    
    print(f"Original size: {frame.nbytes} bytes")
    print(f"Resized size: {small_frame.nbytes} bytes")
    print(f"Compression ratio: {frame.nbytes / small_frame.nbytes:.2f}x")

cap.release()

This demonstrates basic concepts:

• Video frames are matrices of pixel values
• Color space conversion (RGB → YUV helps with chroma subsampling)
• Resizing trades quality for file size

A Simple Video Processor

Here’s a practical project to apply what you’ve learned:

import cv2
import numpy as np

class SimpleVideoProcessor:
    def __init__(self, input_path, output_path):
        self.input_path = input_path
        self.output_path = output_path
    
    def apply_compression(self, quality=80, resize_factor=1.0):
        """Apply basic compression techniques"""
        cap = cv2.VideoCapture(self.input_path)
        
        # Get video properties
        fps = int(cap.get(cv2.CAP_PROP_FPS))
        width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
        height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
        
        # Create video writer
        fourcc = cv2.VideoWriter_fourcc(*'mp4v')
        new_width = int(width * resize_factor)
        new_height = int(height * resize_factor)
        out = cv2.VideoWriter(self.output_path, fourcc, fps, 
                             (new_width, new_height))
        
        frame_count = 0
        while True:
            ret, frame = cap.read()
            if not ret:
                break
            
            # Apply resize (spatial redundancy reduction)
            if resize_factor < 1.0:
                frame = cv2.resize(frame, (new_width, new_height))
            
            # Convert to YUV for better compression
            frame_yuv = cv2.cvtColor(frame, cv2.COLOR_BGR2YUV)
            
            # Reduce color channels (chroma subsampling simulation)
            frame_yuv[:,:,1:] = cv2.resize(frame_yuv[:,:,1:], 
                                          (new_width//2, new_height//2))
            
            out.write(frame)
            frame_count += 1
        
        cap.release()
        out.release()
        
        return frame_count

# Usage
processor = SimpleVideoProcessor('input.mp4', 'output.mp4')
frames = processor.apply_compression(resize_factor=0.5)
print(f"Processed {frames} frames with 50% size reduction")

This demonstrates key concepts:

• Reading and writing video
• Spatial reduction through resizing
• Chroma subsampling simulation
• Practical trade-offs between quality and file size

Learning Resources

Python Libraries:

• OpenCV Documentation - Video processing and manipulation
• NumPy Documentation - Efficient array operations
• Pillow (PIL) - Image processing
• PyAV - Pythonic binding for FFmpeg

Video Codec Standards:

• H.265/HEVC Codec Standard - Modern standard
• AV1 Codec Specification - Next-generation open codec
• VP9 Codec Documentation - Open-source codec
• H.264 Standard - Widely used baseline

Educational Resources:

• Video Compression Fundamentals - Good overview
• DCT and Quantization - Understand transform coding
• Huffman Coding Tutorial - Entropy encoding

Tools to Experiment With:

• FFmpeg - Industry-standard video processing tool
• MediaInfo - Inspect video file properties and compression
• VLC - Playback and analysis of videos