System Design: URL Shortener at Scale

Requirements

Functional:

• Shorten URLs (custom or generated aliases)
• Redirect short URLs to original
• Analytics (click counts, referrers, geography)
• Link expiration

Non-functional:

• 100M URLs created/day (write: ~1200/s)
• 10B redirects/day (read: ~115,000/s) - 100:1 read/write ratio
• 99.99% availability for redirects
• < 50ms P99 redirect latency

Data Estimation

Storage:
- 100M new URLs/day x 365 days x 5 years = 182.5B URLs
- Per URL: ~500 bytes (ID + original URL + metadata)
- Total: ~90TB

Traffic:
- 115,000 redirects/sec peak
- 80/20 rule: 20% of URLs = 80% of traffic

URL Shortening Algorithm

import string, hashlib

ALPHABET = string.ascii_letters + string.digits  # 62 chars
BASE = 62

def encode(num: int) -> str:
    result = []
    while num > 0:
        result.append(ALPHABET[num % BASE])
        num //= BASE
    return ''.join(reversed(result)) or ALPHABET[0]

def generate_short_code(url: str, user_id: str) -> str:
    input_str = f"{url}{user_id}"
    hash_value = hashlib.md5(input_str.encode()).hexdigest()
    num = int(hash_value[:8], 16)
    return encode(num).ljust(7, ALPHABET[0])[:7]

Database Design

CREATE TABLE urls (
  id          BIGSERIAL PRIMARY KEY,
  short_code  VARCHAR(10) UNIQUE NOT NULL,
  long_url    TEXT NOT NULL,
  user_id     BIGINT REFERENCES users(id),
  created_at  TIMESTAMP DEFAULT NOW(),
  expires_at  TIMESTAMP
);
CREATE INDEX idx_short_code ON urls(short_code);

-- Analytics (separate table/service)
CREATE TABLE url_clicks (
  id          BIGSERIAL PRIMARY KEY,
  short_code  VARCHAR(10) NOT NULL,
  clicked_at  TIMESTAMP DEFAULT NOW(),
  ip_address  INET,
  country     CHAR(2)
);

Architecture

Client -> CDN/Load Balancer -> API Servers (stateless)
                                  |
              Write Path          |          Read Path
                  |               |              |
         Validation Service       |       Redis Cache (TTL=24h)
                  |               |         Hit -> Return URL
           PostgreSQL (primary)   |        Miss -> Read DB -> Cache
                                  |
Analytics: API -> Kafka -> ClickHouse (batch aggregations)

Caching Strategy

async def get_long_url(short_code: str) -> str:
    cache_key = f"url:{short_code}"
    cached = await redis.get(cache_key)
    if cached:
        return cached.decode()

    url = await db.fetchone("SELECT long_url FROM urls WHERE short_code = $1", short_code)
    if not url:
        raise NotFoundException(short_code)

    # Jittered TTL to prevent thundering herd
    import random
    ttl = 86400 + random.randint(0, 3600)
    await redis.setex(cache_key, ttl, url['long_url'])
    return url['long_url']

Scaling Considerations

1. Read scaling: CDN for popular links, Redis cluster for cache
2. Write scaling: Shard DB by short_code range, pre-generate code pools
3. Analytics: Use Kafka + ClickHouse, don't slow down redirects
4. Geographic distribution: Multi-region with local caching

A URL shortener is a great example of a read-heavy system that benefits from aggressive caching.