Design ChatGPT (Conversational AI at Scale)

TL;DR. ChatGPT is a streaming inference system first and a model system second. At 900M+ weekly active users¹, the hard problems are keeping first-token latency under 2 seconds while packing hundreds of thousands of concurrent decode slots onto scarce GPUs, preserving conversation state across regions on a single PostgreSQL primary with ~50 read replicas², and streaming tokens over SSE without proxy buffering eating the experience. The pivotal trade-off is tiered model routing: free users hit GPT-5.4 mini at $0.75/1M input tokens³ while paid users hit GPT-5.5 at $5.00/1M input tokens⁴, cutting free-tier unit cost by ~7x.

Learning Objectives

• Design a multi-tenant LLM gateway that routes free and paid traffic to different model pools without leaking latency between tiers
• Estimate GPU fleet capacity from token throughput requirements using continuous batching and PagedAttention gains
• Build a streaming SSE pipeline from inference server through gateway to browser with backpressure and cancellation
• Justify append-only conversation storage on PostgreSQL with read replicas over a sharded active-active layout
• Place input and output moderation gates without doubling end-to-end latency
• Trade off prompt caching, hierarchical summarisation, and full-context replay for multi-turn conversations

Intuition

A conversational AI system looks like a chat box. Accept a message, generate a response, show it. Handles 10 users fine. At 900 million weekly active users it collapses, and the reason is not the model itself but three constraints that compound.

First, autoregressive decoding is memory-bandwidth bound. An H100 GPU with 3.35 TB/s HBM bandwidth⁵ can push a 70B-parameter model at roughly 24 tokens/sec per stream before bandwidth saturates. You cannot serve 10,000 concurrent users on one GPU. You need a fleet of thousands of GPUs, and every wasted KV-cache byte reduces how many requests share a single forward pass.

Second, every token must reach the user the instant it is decoded. A 500-token response at 50 tokens/sec takes 10 seconds. If you buffer the entire response and send it at the end, the user stares at a blank screen for 10 seconds. Streaming is not optional; it is the product.

Third, conversations are stateful. A user returns tomorrow and expects the model to remember what was said. But the model itself is stateless. Every turn must reconstruct the full conversation context in the prompt, which means the session store, the summarisation layer, and the prompt-assembly pipeline are as critical as the GPU fleet.

The naive single-server design fails on all three axes simultaneously. The architecture that works separates the connection tier (SSE gateway), the inference tier (GPU fleet with paged KV-cache and continuous batching), and the state tier (PostgreSQL + Redis), each scaling independently.

Requirements

Clarifying Questions

• Q: Free tier, paid tier, or enterprise? Assume: All three. Free routes to a cheaper model; paid to the frontier model; enterprise pins a version with data residency.
• Q: Conversation retention policy? Assume: Indefinite for paid users; 30-day for free; enterprise configurable.
• Q: First-token latency target? Assume: p99 < 800 ms for paid, p99 < 2 s for free.
• Q: Multi-region required? Assume: Yes. Data residency in 10 regions including EU, US, Japan⁶.
• Q: Streaming transport? Assume: SSE over HTTP. CDN-friendly, proxy-compatible, auto-reconnect via Last-Event-ID⁷.
• Q: Safety requirements? Assume: Mandatory input and output moderation on every turn. Output moderation must not add to TTFT.

Functional Requirements

• Multi-turn conversations with indefinite history and streamed token responses
• Per-user long-term memory (facts extracted from conversations, retrieved on demand)
• Model selection per tier (free, paid, reasoning) with per-tenant rate limits
• Conversation sharing, organization workspaces, and attachment upload

Non-Functional Requirements

• Users: 900M+ WAU¹, ~10K new turns/sec steady, 30K/sec peak
• Latency: TTFT p99 < 800 ms paid, < 2 s free; TPOT p99 < 80 ms
• Availability: 99.9% control plane; inference fleet degrades gracefully under GPU pressure
• Durability: conversation state at 11 nines; append-only turn log
• Data residency: EU, UK, US, Canada, Japan, South Korea, Singapore, India, Australia, UAE⁶

Capacity Estimation

Key ratios:

• GPU fleet size: at ~8 decode slots per H100 for a frontier model, ~20K H100s for the paid tier alone.
• Egress is connection-bound, not byte-bound: 8M tokens/sec x 4 B/token = 32 MB/s, trivial. The 6M concurrent SSE connections are the bottleneck.
• Cost asymmetry: GPT-5.4 mini is ~7x cheaper per input token than GPT-5.5³⁴. Routing free traffic to the cheap model is the single biggest unit-economics lever.

API and Data Model

API Design

POST /v1/conversations
  Body: { "model": "gpt-5.5", "system_prompt": "..." }
  Returns: 201 { "conversation_id": "uuid", "region": "us-east" }

POST /v1/conversations/{id}/turns
  Headers: Authorization: Bearer <token>
  Body: { "role": "user", "content": "..." }
  Returns: 200 (SSE stream)
    event: token\ndata: {"text": "Hello"}\n\n
    event: done\ndata: {"turn_id": "...", "usage": {...}}\n\n

POST /v1/conversations/{id}/cancel
  Returns: 204 (aborts in-flight generation, releases KV cache)

GET /v1/conversations/{id}?before=<cursor>&limit=50
  Returns: 200 { "turns": [...], "next_cursor": "..." }

PATCH /v1/users/{id}/memory
  Body: { "add": ["prefers metric units"], "remove": ["fact_id_123"] }
  Returns: 200

Rate limiting uses token-bucket counters in Redis, enforcing both RPM (requests per minute) and TPM (tokens per minute) per user, with TPM as the binding constraint for LLM workloads⁸.

Data Model

-- Conversations (PostgreSQL primary, ~50 read replicas)
CREATE TABLE conversations (
  conversation_id  UUID PRIMARY KEY,
  user_id          UUID NOT NULL,
  org_id           UUID,
  model            TEXT NOT NULL,
  region           TEXT NOT NULL,
  created_at       TIMESTAMPTZ DEFAULT now(),
  last_turn_at     TIMESTAMPTZ
);

-- Turns (append-only, partitioned by month)
CREATE TABLE turns (
  turn_id          UUID,
  conversation_id  UUID NOT NULL,
  role             TEXT NOT NULL,  -- user | assistant | tool
  content          TEXT,
  token_count      INT,
  moderation_verdict JSONB,
  created_at       TIMESTAMPTZ DEFAULT now(),
  PRIMARY KEY (conversation_id, turn_id)
) PARTITION BY RANGE (created_at);

-- User memory (vector-indexed for top-k retrieval)
CREATE TABLE user_memory (
  user_id    UUID,
  fact_id    UUID,
  text       TEXT,
  embedding  vector(1536),
  created_at TIMESTAMPTZ,
  PRIMARY KEY (user_id, fact_id)
);

Hot conversations are cached in Redis (conv:{id} -> last N turns + running summary, 24h TTL, promoted on read). Rate-limit state lives in Redis token buckets keyed on (user_id, tier, window).

High-Level Architecture

flowchart LR
    Client[Browser / Mobile<br/>SSE client] --> CDN[CDN / Edge<br/>TLS termination]
    CDN --> GW[API Gateway<br/>auth, rate limit,<br/>conv affinity]
    GW --> Sess[Session Service]
    Sess --> Redis[(Redis<br/>hot cache)]
    Sess --> PG[(PostgreSQL<br/>primary + 50 replicas)]
    GW --> ModIn[Input Moderation<br/>sync, < 50 ms]
    ModIn --> Router[Model Router<br/>tier + feature flag]
    Router --> Free[Inference Pool<br/>GPT-5.4 mini]
    Router --> Paid[Inference Pool<br/>GPT-5.5]
    Router --> Reason[Inference Pool<br/>o1 reasoning]
    Free --> Stream[Token Stream]
    Paid --> Stream
    Reason --> Stream
    Stream --> ModOut[Output Moderation<br/>rolling window, parallel]
    ModOut --> GW
    Stream --> WB[Async Write-back]
    WB --> PG
    WB --> Mem[Memory Extractor]

Write path: Client opens an SSE connection via POST /turns. The gateway authenticates, checks rate limits in Redis, resolves conversation_id to its home region, loads history from the session service, runs synchronous input moderation, routes to the correct inference pool, and streams tokens back as SSE events. Output moderation runs concurrently on a rolling token window.

Read path: History fetches hit PostgreSQL read replicas (50 across regions)². Active conversations are served from Redis cache. Cold conversations fall through to the replica nearest the user.

Async path: Completed turns are appended to the turn log. A memory extractor mines closed conversations for durable user facts, embedded and stored for future top-k retrieval.

Deep Dives

KV cache management and PagedAttention

The KV cache is the dominant memory cost in LLM inference. For each request, the model stores key and value tensors for every layer at every token position. A single 70B-parameter model with a 4K-token context at FP16 consumes roughly 2.5 GB of KV cache per request. At 128K context, that balloons to ~80 GB, exceeding a single H100’s 80 GB HBM⁵.

The naive approach pre-allocates contiguous memory for the maximum sequence length per request. Since actual lengths vary wildly (some responses are 50 tokens, others 4,000), internal fragmentation wastes 60-80% of GPU memory⁹.

PagedAttention (vLLM, Kwon et al. SOSP 2023) solves this by treating KV cache like OS virtual memory⁹. Physical memory is divided into fixed-size blocks (typically 16 tokens). A block table maps each request’s logical blocks to physical blocks. Blocks are allocated on demand and freed immediately when a request completes. This eliminates fragmentation and enables:

• Memory sharing: parallel sampling and beam search share physical blocks via copy-on-write
• Flexible scheduling: the scheduler can preempt low-priority requests by swapping their blocks to CPU memory
• Higher batch sizes: 2-4x throughput improvement over FasterTransformer and Orca baselines⁹

flowchart TD
    subgraph Requests
        R1[Request A<br/>logical blocks 0..3]
        R2[Request B<br/>logical blocks 0..5]
    end
    subgraph Block Table
        BT1[A: phys 4, 7, 2, 9]
        BT2[B: phys 1, 5, 6, 3, 8, 0]
    end
    subgraph GPU HBM Pool
        B0[block 0] --- B1[block 1] --- B2[block 2]
        B3[block 3] --- B4[block 4] --- B5[block 5]
    end
    R1 --> BT1
    R2 --> BT2
    BT1 --> B4
    BT1 --> B2
    BT2 --> B1
    BT2 --> B5

Continuous batching (Orca, Yu et al. OSDI 2022) compounds the gain¹⁰. Static batching pads all requests to the longest sequence and holds GPU slots until the entire batch finishes. Orca’s iteration-level scheduling returns finished requests each step and admits new ones mid-iteration, achieving 36.9x throughput over FasterTransformer at the same latency on GPT-3 175B¹⁰.

gantt
    title Continuous batching vs static batching
    dateFormat X
    axisFormat %L
    section Static batch
    Req A - 8 tokens      : 0, 8
    Req B - 3 tokens pads to 8 : 0, 8
    Req C waits           : 8, 12
    section Continuous batch
    Req A - 8 tokens      : 0, 8
    Req B - 3 tokens      : 0, 3
    Req C admitted step 3 : 3, 9
    Req D admitted step 8 : 8, 12

Speculative decoding adds a third lever. A small draft model proposes K tokens in parallel; the large model verifies all K in a single forward pass. Medusa-2 reports a 2.3-3.6x speedup range across models with jointly-trained decoding heads¹¹. Production engines like vLLM ship speculative decoding (EAGLE, n-gram, Medusa) as configurable options¹².

Quantization trades precision for capacity. AWQ (Activation-Aware Weight Quantization) at INT4 reduces VRAM by ~50% with only 1-3% quality loss¹³, roughly doubling the effective batch size per GPU. TensorRT-LLM on H100 achieves 4.6x the throughput of A100¹⁴, and reaches ~12,000 tokens/sec on H200 for Llama2-13B¹⁴. DeepSpeed Inference reports up to 7.3x latency reduction over prior state of the art¹⁵.

Streaming SSE pipeline

Every major LLM provider converged on Server-Sent Events over HTTP for token streaming⁷. SSE wins over WebSocket for this workload because it is unidirectional (model-to-client), HTTP-native (works through CDNs and proxies without upgrade negotiation), and has built-in browser reconnection via Last-Event-ID.

Architecture: The gateway holds the client SSE connection while consuming a server-side gRPC stream from the inference server. Each decoded token is forwarded as an SSE data: event. The inference server generates at 50-100 tokens/sec; the gateway must not buffer.

Backpressure: At 80-100 tokens/sec, the model can outpace a slow client. The gateway applies cooperative backpressure: when res.write() returns false (Node.js) or the async generator stalls (Python/FastAPI), generation pauses upstream. This prevents memory bloat on the gateway⁷.

Proxy buffering is the #1 production failure. Nginx, Cloudflare, and AWS ALB buffer HTTP responses by default. Without explicit proxy_buffering off and X-Accel-Buffering: no, tokens arrive as a single batch at the end of the response, destroying the streaming experience⁷. Every SSE route requires:

proxy_buffering off;
add_header X-Accel-Buffering no;
proxy_read_timeout 300s;

Cancellation: When a user closes the tab, the gateway must propagate cancellation to the inference server to release KV cache blocks. Without propagation, abandoned requests hold GPU memory until the full completion finishes. The gateway listens for connection close, sends an abort signal upstream, and the inference server frees blocks immediately⁷.

sequenceDiagram
    participant C as Client
    participant G as Gateway
    participant S as Session
    participant M as Moderation
    participant I as Inference
    C->>G: POST /turns (SSE open)
    G->>G: auth + rate limit (Redis)
    G->>S: load history
    S-->>G: prompt assembled
    G->>M: input classify (sync, < 50 ms)
    M-->>G: pass
    G->>I: generate (gRPC stream)
    loop per token
        I-->>G: token
        G-->>C: SSE data: token
        par output moderation
            G->>M: rolling window classify
        end
    end
    I-->>G: done
    G->>S: persist turn (async)
    G-->>C: SSE event: done
    Note over C,I: On client disconnect
    C--xG: connection closed
    G->>I: abort (release KV cache)

Model routing and cost control

The cost spread between model tiers is the single biggest lever on unit economics. GPT-5.4 mini costs $0.75 input / $4.50 output per million tokens³. GPT-5.5 costs $5.00 input / $30.00 output per million tokens⁴. The reasoning model o1 costs $15 input / $60 output per million tokens¹⁶. Routing free-tier traffic to GPT-5.4 mini cuts per-turn cost by ~7x on both input and output compared to GPT-5.5.

Router logic: A gateway component maps (user_tier, feature_flag, experiment_arm) to a model pool. Free tier always routes to GPT-5.4 mini. Paid tier routes to GPT-5.5 by default, with opt-in to o1 for reasoning tasks. Enterprise tier pins a specific model version for reproducibility.

Rate limiting: Redis-backed token buckets enforce both RPM and TPM per user⁸. TPM is the binding constraint for LLM workloads because a single long-context request can consume 100K tokens. The gateway checks the bucket before forwarding to inference; if the user is over quota, it returns 429 immediately without consuming GPU cycles.

Load shedding under GPU pressure: When inference queue depth exceeds a threshold, the router sheds free-tier traffic first (returning a “busy” response), then degrades paid-tier by routing to a smaller model, and only as a last resort queues reasoning-tier requests. This ensures paid latency SLOs hold even during demand spikes.

Prompt caching: Anthropic’s prompt caching demonstrates the economics: cache reads cost 10% of base input price, with up to 90% cost reduction and up to 85% TTFT reduction for long cached prompts¹⁷. In Anthropic’s published example, a 100K-token cached system prompt drops TTFT from ~11.5 s to ~2.4 s (a 79% reduction)¹⁷. For ChatGPT, this means caching the system prompt and tool definitions across turns in the same conversation saves both latency and cost on every subsequent turn.

Real-World Example

OpenAI ChatGPT: from 200M to 900M weekly users in 18 months.

ChatGPT reached 200 million weekly active users in August 2024¹⁸, surpassed 800 million in October 2025, and crossed 900 million weekly active users by February 2026¹. The session store runs on a single Azure PostgreSQL flexible server primary with nearly 50 read replicas spread across regions². This is not a sharded active-active cluster. It is a single-writer architecture where the primary handles all turn appends and replicas offload read traffic globally.

The architecture is deliberately simple at the state layer. Conversations are append-only logs keyed by conversation_id. Writes are sequential appends; regenerations and edits are new turns that reference the original. This design replicates cleanly with PostgreSQL’s physical replication and makes retries auditable².

The inference layer is where complexity concentrates. OpenAI operates tiered model pools: GPT-5.4 mini for free users ($0.75/$4.50 per 1M tokens)³, GPT-5.5 for paid users ($5.00/$30.00 per 1M tokens)⁴, and o1 for reasoning ($15/$60 per 1M tokens)¹⁶. (OpenAI retired GPT-4o-mini from ChatGPT on Feb 13, 2026; the tiered-routing pattern remains canonical even though ChatGPT itself now rate-limits a single default model instead.) The router enforces per-tenant token budgets and sheds cheaper traffic first under GPU pressure.

Data residency spans 10 regions. Enterprise customers using EU-residency projects have API requests handled in-region with zero data retention: requests and responses are not stored at rest¹⁹. This satisfies GDPR-class requirements without a separate EU deployment of the model fleet.

flowchart LR
    subgraph EU Region
        EUG[Gateway EU] --> EUP[(PG primary<br/>EU shards)]
        EUP --> EUR[(PG replica)]
    end
    subgraph US Region
        USG[Gateway US] --> USP[(PG primary<br/>US shards)]
        USP --> USR[(PG replica)]
    end
    EUP <-.async repl.-> USR
    USP <-.async repl.-> EUR
    User((User EU)) --> EUG
    User -.roaming.-> USG

The key insight non-experts miss: the session store is boring by design. A single PostgreSQL primary with read replicas is sufficient because conversation writes are append-only, low-throughput relative to the read fan-out, and partition cleanly by conversation. The hard scaling problem is the GPU fleet, not the database.

Design decisions

The single biggest meta-decision: tiered routing. Without it, free-tier unit economics make the product unsustainable at 900M users. With it, you accept a visible quality gap between tiers but keep the business viable.

Scaling and Failure Modes

At 10x load (80K turns/sec): The PostgreSQL primary saturates on write throughput. Mitigation: shard the turn log by conversation_id hash across multiple primaries, each with its own replica set. The session service routes writes to the correct shard.

At 100x load (800K turns/sec): The GPU fleet becomes the binding constraint. Mitigation: aggressive quantization (AWQ at INT4 cuts VRAM ~50% and roughly doubles batch size¹³), speculative decoding (Medusa-2 at 2.3-3.6x¹¹), and multi-model routing that pushes even more traffic to cheaper models.

At 1000x load: The architecture shifts to a federated model where regional inference fleets operate semi-independently, with a global control plane for routing and billing but no cross-region inference traffic.

Failure mode: GPU node crash mid-generation. The gateway detects the gRPC stream termination, retries on another node in the same pool. The new node re-runs prefill from the cached prompt (no KV cache transfer). The user sees a brief TTFT spike but no data loss. Partially-streamed tokens are already delivered; the retry resumes from the last persisted checkpoint.

Failure mode: Regional PostgreSQL primary failure. The nearest read replica is promoted. New turn writes resume after promotion (seconds to minutes of unavailability on the write path). Read path continues uninterrupted from other replicas. Conversations created during the outage land in the failover region².

Failure mode: Redis cache eviction storm. A sudden spike in new conversations exceeds Redis capacity. Mitigation: the session service falls through to PostgreSQL replicas. Latency increases (cache miss adds ~20 ms) but correctness is preserved. Auto-scaling adds Redis nodes within minutes.

Common Pitfalls

Follow-up Questions

1. How do you handle a zero-downtime model upgrade (GPT-5.5 to GPT-5.5 Instant)?

Deploy the new model version as a separate pool. Route a percentage of traffic via feature flag (canary). Monitor TTFT, output quality (eval sampling), and error rates. Once validated, shift 100% of traffic. In-flight streams on the old pool complete naturally; no mid-stream cutover. The session service does not depend on model version.

2. How would you design the data pipeline so enterprise customers can opt out of training?

Tag all turns with training_eligible: false at ingest for opted-out orgs. The training pipeline filters on this flag. EU-residency projects use zero data retention: requests and responses are not stored at rest¹⁹. Audit logs prove compliance.

3. How do you prevent prompt-injection attacks?

Input moderation classifies the user turn before it reaches the GPU²⁰. A separate system-prompt integrity check detects attempts to override instructions. Output moderation catches leaked system prompts in the response. Defense in depth: no single layer is sufficient.

4. How would you extend this to support voice (speech-in, speech-out)?

Add a speech-to-text service (Whisper) before the session service and a text-to-speech service (TTS) after the token stream. The TTFT budget tightens (voice users expect < 500 ms). Speculative decoding and prompt caching become critical. The SSE transport carries audio chunks instead of text tokens.

5. What changes for a 1M-token context window (Gemini 3.1 Pro scale)?

KV cache per request grows proportionally. At 1M tokens, a single request may consume the majority of HBM on one GPU. Mitigation: sequence parallelism (split the KV cache across multiple GPUs), aggressive summarisation to avoid hitting the full window, and tiered pricing that charges more for long-context requests. Gemini 1.5 Pro maintained 99.7% needle-in-a-haystack recall at 1M tokens²³; Gemini 3.1 Pro (released Feb 19, 2026) retains this 1M-token window with tiered pricing above 200K.

6. How do you handle multi-region active-active for conversations?

Pin each conversation to a home region on creation. The PostgreSQL primary for that shard lives in-region; other regions see it via async replication². A user roaming across regions takes a cross-region routing hop rather than forking state. Failover promotes the nearest healthy replica. This is simpler than active-active and avoids conflict resolution.

Exercise

Exercise 1: GPU fleet sizing

Your product has 1M concurrent users, each generating an average of 1 turn per minute. Each turn has a 500-token prompt and an 800-token completion. Your inference fleet uses H100 GPUs running a frontier model at 50 tokens/sec per decode slot, with 8 concurrent slots per GPU (continuous batching + PagedAttention). How many H100 GPUs do you need for the decode fleet?

Key Takeaways

• ChatGPT is a streaming system first. TTFT and connection management drive gateway design; model quality is table stakes.
• PagedAttention is mandatory. Block-based KV cache allocation delivers 2-4x throughput⁹; without it, GPU memory is 60-80% wasted.
• Tiered routing is the unit-economics lever. Free on GPT-5.4 mini ($0.75/1M) vs paid on GPT-5.5 ($5.00/1M) is a ~7x cost difference³⁴.
• Output moderation runs in parallel, never serial. Serial moderation adds non-trivial latency to TTFT and breaks SLOs²⁰.
• The session store is boring by design. A single PostgreSQL primary with 50 read replicas handles 900M WAU because conversation writes are append-only and low-throughput relative to reads².
• Proxy buffering is the #1 streaming failure. Every SSE route needs explicit proxy_buffering off⁷.

Further Reading

• Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., SOSP 2023). The foundational paper on vLLM; mandatory reading for anyone designing inference serving infrastructure.
• Orca: A Distributed Serving System for Transformer-Based Generative Models (Yu et al., OSDI 2022). Introduces iteration-level (continuous) batching; the technique that unlocked modern LLM serving throughput.
• Anthropic: Prompt caching with Claude. Concrete cost and latency numbers for prompt caching; the best public benchmark of cache economics.
• OpenAI: Scaling PostgreSQL to power 800 million ChatGPT users. Rare public architecture post showing the single-primary + 50-replica layout.
• The Streaming Infrastructure Behind Real-Time Agent UIs. Practical field guide to SSE backpressure, proxy buffering, cancellation, and reconnection patterns.
• Medusa: Simple LLM Inference Acceleration Framework (Cai et al., ICML 2024). Clearest paper on multi-token decoding heads, reporting a 2.3-3.6x speedup range for Medusa-2.
• Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations. The reference input-output safety classifier design for conversational AI.
• NVIDIA TensorRT-LLM performance overview. Hardware-specific throughput numbers for H100 and H200; use as a capacity planning reference.

Flashcards

References