S26.E0924 – An AI Said the Function Didn't Exist. It Did. Four Errors in Four Hours.
from Runtime Reality Hosted by Kenneth Henseler
Claude Opus 5 made four confident, plausible, completely wrong claims during a single development session. None were caught by better reasoning. Each was caught by reading a primary source at the moment the answer felt already known. The client was World of Warcraft: Forever, build 69913 — a beta where most legacy Lua APIs have been removed and the survivors were kept inconsistently. The training data describes a game that no longer exists. Error 1 — An API documented as absent. The project's own notes recorded "zero occurrences in this client's surface" as established fact. A three-line type() check found it immediately. Two independent sources of truth were blind in different ways: the API index covers the C API, and a runtime global walk sees functions and C_ namespaces. A Lua table inside a plain global falls through both. Error 2 — A migration that would have destroyed what it was upgrading. Wiping stored data on a schema mismatch reads as reasonable defensive coding. The capability probe had just written its results into that table. Error 3 — The textbook-correct answer that broke every message. ASCII 31, the unit separator, is the character the standard provides for exactly this purpose. The transport library rejects the range it sits in and throws on every send. Error 4 — A rescue plan that had it backwards. Two saved files, and the one visible on screen was empty on disk. SavedVariables flush at reload or logout, so the file lags memory by one reload. The pattern. Not one was caught by reasoning — every error was arrived at by sound reasoning from its inputs. Not by general caution either; being careful is a disposition, it does not tell you which file to open. Measured in this episode: build 69913 · ASCII 31 · CBOR 386 bytes, Deflate 293, Base64 392 — encoded output larger than the input, paying for a chunk boundary rather than for compression · two chunks at 240 · race ID 96 · C_Housing.GetMaxHouseLevel, documented and present at runtime as one of 63 functions, and still access-violating the client. Honest caveat: roughly a third of the addon's fields were never exercised. Sharing has never been tested with a second player; guild and title display have never rendered real data. An episode about verification that overstated its own completeness would be self-refuting. Chapters 00:00 You record it as established fact 02:44 The thesis: plausible is not verified 04:01 The client: build 69913, and what it broke 07:49 Error 1 — two sources of truth, both blind 13:20 Error 2 — the migration that ate its own evidence 25:22 Error 3 — ASCII 31, the textbook answer that failed 30:04 The crash model: why pcall cannot save you 34:07 Error 4 — the rescue plan that had it backwards 39:08 What this means for AI-assisted engineering 44:13 What was never verified Links Source, tooling and commit history — github.com/justadakaje/wow-addons Measured findings for build 69913 — docs/facts-build-69913.md Unedited session, Part 1 — youtu.be/guSsG1I_XX0 Unedited session, Part 2 — youtu.be/lAFS9JJeLPATHE 31-MINUTE CUT — youtu.be/_RP-CQRNKKg Tool used: Claude Code, Opus 5. The "wow-api" MCP server verified API calls against a custom index — correctness, not speed.
Embed this episode
No transcript yet.