The State of AI Engineering: What a Thousand Companies' Telemetry Reveals

Five Moves for LeadersAdopt a model gateway — centralize routing, failover, governanceBuild deprecation discipline — retire models deliberatelyInstrument agents deeply — especially with frameworksAudit prompt caching — fix layout (stable first, dynamic later)Implement budgets & backpressure — cap loops, build queuesSeven Key TakeawaysMulti-model is the norm (70%+ use 3+ models); use a gatewayLLM tech debt compounds; retire old models deliberatelyFramework adoption doubled; observability burden doubled too69% of tokens are system prompts; only 28% use cachingContext windows exploded but quality beats volumeRate limits are the #1 failure modeAgents are still mostly monoliths; distributed shift is comingKey Quotes"The gap between a good demo and a dependable system is closed by effective evaluation and operational discipline." — Datadog"The next wave of agent failures won't be about what agents can't do. It'll be about what teams can't observe." — Guillermo Rauch, CEO, Vercel"Context quality, not volume, is the new limiting factor for LLM agents." Hosted on Acast. See acast.com/privacy for more information.