Platform Engineering at AI-Native Companies: What's Actually Different

Meta's Llama 3 training run, 405 billion parameters, used 16,384 H100 GPUs for 54 days. Over those 54 days, the cluster experienced 419 unexpected interruptions — roughly one failure every three hours. And that's the run Meta calls a success. They hit 90 percent effective training time.This is the substrate platform engineers at AI-native companies are operating on.This episode is what's actually different about platform engineering at companies like OpenAI and Anthropic, compared to the traditional shape — Stripe, Netflix, Block, Google. Engineering tone, not hype.The verified primary-source view: OpenAI's two Kubernetes scaling posts at 2,500 and 7,500 nodes (5 API servers, 5 etcd, 70 GB heap per API server, 200,000 IPs in use at peak, MPI gang scheduling via the Coscheduling plugin). OpenAI's Postgres scaled for 800 million ChatGPT users on a single primary plus 50 read replicas. Anthropic's September 2025 postmortem disclosing three serving platforms (first-party, Bedrock, Vertex), three hardware backends (Trainium, NVIDIA, TPU), sticky routing, tens of chips per request.Compute portfolios: Anthropic ~7 GW across AWS Project Rainier (~500K Trainium2), Google-Broadcom (up to 1M TPUs), Microsoft-NVIDIA ($30B / 1 GW Grace Blackwell + Vera Rubin), SpaceX Colossus 1 (220K NVIDIA / 300 MW). OpenAI Stargate at $500B / 10 GW.The new problem classes: training cluster reliability (Meta MTTF 47.7 days at 8 GPUs → 14 minutes at 131,072 GPUs — collapses non-linearly). NCCL collectives. Gang scheduling primitives (Kueue vs Volcano). Inference at p99 (PagedAttention, RadixAttention, continuous batching). Prefill vs decode disaggregation. Heterogeneous fleets across H100, H200, B200, GB200, Trainium2, TPU v5p, Ironwood. HBM and U.S. energy as the binding constraints, not GPU FLOPS.What stays the same: the reliability discipline. SLOs, error budgets, on-call, blameless postmortems, observability. Anthropic's September 2025 postmortem reads like a Google SRE Book chapter. What doesn't transfer: substrate-specific tooling. You can't canary a 16,000-GPU job mid-flight.Three platforms inside one company. Training is a batch-scheduler problem. Inference is a request/response problem. Agents are a durable-workflow problem. Above all three, a chip-portability layer.Same craft. Different physics.RELATED EPISODESHow Netflix, Uber, and YouTube Handle Scale — sister episode, traditional platformHow LLM Inference Actually Works — the inference layer this scalesThe AI Chip War — the H100/Trainium/TPU substrate platforms run onThe Real Cost of AI — economics of running this fleet at 7 gigawattsRAG in Production — the application layer above the platformCHAPTERS00:00 Cold open — Llama 3.1 reliability data00:33 Intro00:59 The traditional platform charter02:24 What's disclosed at OpenAI + Anthropic04:40 Anthropic infrastructure deep dive07:10 Team structure (OpenAI by workload, Anthropic by portability)07:48 The new problem classes08:20 Training cluster reliability + Meta MTTF curve09:52 Gang scheduling — Kueue vs Volcano10:26 Training frameworks — DeepSpeed, FSDP, Megatron11:15 Inference at p99 — PagedAttention, RadixAttention11:58 Prefill vs decode disaggregation12:38 Heterogeneous fleets13:14 Capacity planning + HBM as the binding constraint14:28 What stays the same15:46 Why 'more load-bearing'16:59 Closing thesisSOURCESOpenAI Kubernetes posts (2018, 2021) + Postgres scaling writeupAnthropic September 2025 postmortemAnthropic Managed Agents + Code Execution with MCPAWS Project Rainier, Google-Broadcom, MS-NVIDIA, SpaceX Colossus disclosuresOpenAI Stargate (January 2025)Llama 3.1 paper + Meta cluster MTTF (arXiv 2410.21680)DeepSeek V3 paper · vLLM PagedAttention (SOSP 2023) · SGLang RadixAttentionLatent Space — NVIDIA Dynamo team (prefill/decode disaggregation)Google Borg paper · Netflix Tech Blog (Spinnaker, Atlas, Eureka)