“Sanity-checking “Incompressible Knowledge Probes”” by Sturb, LawrenceC

Or, did a chief scientist of an AI assistant startup conclusively show that GPT-5.5 has 9.7 trillion parameters? Introduction Recently, a paper was circulated on Twitter claiming to have reverse engineered the parameter count of many frontier closed-source models including the newer GPT-5.5 (9.7 trillion parameters) and Claude Opus 4.7 (4.0 trillion parameters) as well as older models such as o1 (3.5T) and gpt-4o (720B). The paper, titled “Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity”, introduces a dataset of factual knowledge of different difficulties, regresses performance on this dataset against parameter count, and then uses this regression to extrapolate from the performance of closed-sourced frontier models to their parameter count. A notable fact about this paper is that, unlike most empirical machine learning papers, it's single-authored: Bojie Li, the chief scientist of Pine AI, is the sole author of this piece. These results were suspicious for many reasons, the primary being that it seems like low-effort, hastily-written AI slop. For example, the codebase (https://github.com/19PINE-AI/ikp) was constructed in large part with Claude Code and has many of the flags for code that is almost entirely vibe-coded with little sanity checking (e.g. redundant and inconsistent variable definitions[1] [...] ---Outline:(00:19) Introduction(04:19) Summary of Lis Incompressible Knowledge Probes(08:04) The IKP dataset.(11:59) IKP scoring and Regression Methodology(16:54) Methodological Issues with the IKP paper(17:24) Per-tier floors to the scoring(19:27) Ambiguous/incorrect answers to hard questions(23:21) Corrected model parameter estimates(24:17) Possible methodological issues that mattered less than we thought(24:22) Thinking vs non-thinking(25:59) Different accuracy metrics used in some repository json files(26:31) Conclusion(29:46) Discussion The original text contained 17 footnotes which were omitted from this narration. --- First published: May 1st, 2026 Source: https://www.lesswrong.com/posts/veFMEzDDyWaer2Sms/sanity-checking-incompressible-knowledge-probes --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.