Slow Takes Ep. 10: The Bill for the AI Promise Came Due

Palisade Research published a paper showing frontier and open-weight AI models can hack a vulnerable server and install a working copy of themselves end-to-end. A coding agent running Anthropic’s Claude Opus 4.6 deleted a startup’s entire production database, including the backups, in nine seconds, then issued an apology. Apple agreed to pay $250 million to settle a class action over an upgraded Siri it sold at WWDC 2024 and still has not shipped. Coinbase fired 14% of its staff blaming AI, with the CEO’s own words framing the move as ‘rebuilding Coinbase as an intelligence, with humans around the edge aligning it’. And in federal court in Oakland, three former OpenAI insiders testified that both of the company’s safety teams were disbanded in 2024 while the marketing was claiming the opposite.Five stories. One thread. The promise of AI is six to eighteen months ahead of the data, and the cost of that gap is paid by everybody except the people making the promise.Every Monday at 12:45 BST, Leor from Exploring ChatGPT and I go through the week’s AI news without hype. Here is what we covered.Slow Takes is also available on the YouTube channel: Exploring ChatGPT.1. Palisade Research: AI self-replication is no longer hypotheticalPalisade Research published a paper on 7 May 2026. They gave frontier and open-weight models a single prompt: hack a vulnerable web server, extract credentials, and install a working copy of yourself on the compromised host. Anthropic’s Claude Opus 4.6 succeeded in 81% of non-refusal runs. GPT-5.4 succeeded in 33%. The open-weight Qwen3.6-27B running on a single consumer-grade A100 also reached 33%. In a separate experiment the team chained the trick across four virtual machines on three continents, autonomously, from a single initial prompt, in two hours forty-one minutes. The final replica was fully operational. The paper closes with one sentence: ‘Autonomous self-replication is no longer hypothetical.’ Full paper here.What we said on the live:This is the line every frontier lab spent the last three years saying it would not cross. Anthropic’s Responsible Scaling Policy v2.0 reclassified autonomous self-replication as a ‘checkpoint’ that does not require ASL-3 safeguards. RSP v3.0 (2026) dropped it from the ASL-3 thresholds entirely. OpenAI’s Preparedness Framework v2 moved it from a tracked to a research category, so it no longer gates releases. The frameworks were softened first. The capability arrived right after. Leor’s point on the live was the right one: how many lines do we put in the sand before we accept that the line is decorative? The chat surfaced the proper caveat too. Benjamin Murphy pointed out that current frontier models still need a lot of graphic RAM. Last time anyone checked, that is not what is sitting in a teenager’s bedroom. Palisade is also a company in the business of selling cybersecurity research, which is the kind of context you want next to any white paper produced by a private lab without external peer review.What did not come up:The Palisade result is small data, but the structural finding is the one to keep. It is not the absolute self-replication rate that matters. It is the trajectory and the policy responses to that trajectory. Opus 4 was at 6% a year ago. GPT-5 was at zero. The labs published, the rates moved up, the rules moved out of the way. Critical AI literacy is the muscle for noticing when the people building the technology stop counting the thing they used to call the line they would not cross. The cybersecurity people in the chat (thanks Chad Thiele & ToxSec) are the right next port of call for anyone who needs to translate this from a controlled-environment paper into a procurement-decision question. The framing for the rest of us is simpler. Read this story alongside Story 2. An AI agent with credentials and access can already take down a production system in nine seconds. Now imagine the agent on the other side of the network is also one of these.2. The AI agent that wiped a startup in nine secondsJeremy ‘Jer’ Crane, founder of automotive SaaS startup PocketOS, ran the Cursor coding agent (powered by Anthropic’s Claude Opus 4.6) in his staging environment. The agent encountered a credential mismatch, found an API token in an unrelated file, and used it to delete the production volume on Railway in 9 seconds. The backups were stored on the same volume and were also deleted. The agent’s own confession in the post-mortem: ‘NEVER run destructive/irreversible git commands… I decided to do it on my own to fix the credential mismatch, when I should have asked you first.’ What we said on the live:Reading the news framing, you would think the story is ‘AI agent destroys company’. The actual story is the deployment architecture. The agent had the credentials, the production volume held the backups in the same shell, and the human in the loop waved a permission step through without reading it. As Shannon said in the chat: do they not perform backups? The answer is yes, but they ran on a system where the backups and the production data were both inside the agent’s blast radius. Ben’s point on immutable backups is the right one. Even the administrator should not be able to delete them; in this case the agent walked in on the admin’s credentials. The agent is the proximate cause. The architecture is the root cause. The reasonable response is the one in AI Doesn’t Just Make You Worse. It Makes You Stop Trying.: when AI tools amplify your output, they also amplify your blind spots, and the answer is to build the guardrails before you need them, not after.What did not come up:Vibe coding is where this gets worse, not better. Dario Amodei’s claim that 100% of code will be AI-generated ‘within a year’ is the marketing version. The operational version is that a lot of people will be running coding agents on production systems without any of the engineering discipline that used to be the price of admission. The labs sell the model. The labs do not sell the deployment architecture that makes deploying the model safe. The thing for individuals to do this week is small and obvious: write yourself a /backup skill. Mine runs on my own laptop, dumps memory files to a separate drive, mirrors the working folders to a different Dropbox account, and keeps the API keys in a server I do not touch with AI tools. None of this is cybersecurity expertise. It is the floor.3. Apple paid $250 million to settle the Siri AI lawsuitOn 6 May 2026 Apple agreed a $250 million class action settlement covering iPhone 15 and iPhone 16 buyers in the United States who purchased between 10 June 2024 and 29 March 2025. Eligible US claimants get up to $75 per device. The plaintiffs alleged Apple had marketed an upgraded Siri at WWDC 2024 that, two years on, still does not exist. Apple did not admit wrongdoing. The upgraded Siri is now rumoured to be powered by Google’s Gemini. Apple’s developer conference is on 8 June. The free cash flow Apple generated in 2026 is roughly $130 billion, which makes the $250 million settlement 0.2% of one year’s free cash. For UK readers there is a separate live action: Which? has filed a competition-law breach claim against Apple in the High Court that is unrelated to Siri but worth signing up for if you have bought an Apple device in the UK in the past few years. The Which? claim is here.What we said on the live:The most powerful AI marketing brand on earth admitted in court, by writing a cheque, that its AI marketing was wrong. Not via a press release. Via a settlement. Leor’s read was the right one: this is small for Apple in absolute terms, and the iPhone 15 and 16 unit sales the marketing helped drive will easily exceed the cost of paying the customers back. It is also worth taking the speculation seriously about what happened behind the scenes between Apple and Google. The ‘powered by Gemini’ rumour suggests Apple did not have the in-house capability to ship what it sold, and that the partnership it needed to make it real did not materialise in time. Either way the settlement is the live precedent for what AI marketing claims look like when somebody serves a subpoena.What did not come up:Not every company should be building its own frontier model. Apple is the proof. The companies who pivot fastest to specialised, integrated, narrower AI features built on top of existing frontier models from somebody else are likely to do better than the ones still trying to build everything in-house under the pressure of a launch deck. The other piece worth saying out loud: marketing-team blame is a misdirection. WWDC keynote claims are not signed off by the marketing team. They are signed off by Tim Cook. The cost of being optimistic in public on AI just landed on Apple’s quarterly report. It will land somewhere else next.4. Coinbase fired 14% citing AIOn 5 May 2026, Coinbase CEO Brian Armstrong cut 14% of staff, around 700 employees, pointing to AI as the reason. Armstrong’s own words: “To get there, we are not just reducing headcount and cutting costs, we’re fundamentally changing how we operate: rebuilding Coinbase as an intelligence, with humans around the edge aligning it.”The new org chart is being built around ‘player-coaches’ replacing traditional managers, AI-native pods including potential single-person teams directing AI agents, no more than five layers below the CEO, and 15+ direct reports per leader. The most-cited cautionary tale from this pattern is Klarna, which last year over-indexed on AI for customer service, watched quality collapse and is now quietly rehiring.What we said on the live:This is the most explicit version yet of an AI-driven workforce restructure: not a headcount cut dressed up in AI language, but an actual rebuild of the org around AI agents with people ‘around the edge’ to align them. The pitch language is the news. ‘Humans around the edge aligning it’ is exactly the framing critical AI literacy has been pushing back against for two years. Leor’s reading was right too: the over-hiring story of the zero-interest-rate boom is the one a lot of these CEOs are not allowed to tell on a public earnings call, and AI is a clean external reason to do the restructure now. Sam Altman’s phrase ‘AI washing’ fits. The Forrester 2026 Future of Work data shows over half of CEOs regret AI-attributed layoffs and one in three have rehired more than half the people they fired. Coinbase is the test case. We will know in twelve months whether the bet held or whether the rehire follows.What did not come up:The interesting bit is downstream. Employer brand is real. Jen Benford in the chat made the point as well as anyone: “damage your employer brand when you do not tell the truth on termination reasons.” The talent you laid off this quarter is the talent your competitor hires next quarter. Customer service in particular is the worst place to take the gamble. Empathy and accountability are the human-required parts of the job, and Klarna learned this in public. The bigger pattern Jensen Huang has been making the case for is the right one: AI as labour amplification, not labour reduction. The companies that work out how much more a single person can do with these tools at their side are the ones that look right in five years. The ones that fire first and rehire on worse contracts later are the ones the cohort remembers.5. OpenAI insiders testify the safety teams were disbandedIn federal court in Oakland this week, Elon Musk’s lawsuit against OpenAI heard testimony from three witnesses on OpenAI’s safety record. Rosie Campbell, a former member of the AGI readiness team from 2021 to 2024, testified: “When I joined, it was very research-focused and common for people to talk about AGI and safety issues […] Over time it became more like a product-focused organization.”Both the AGI readiness team and the Super Alignment team were disbanded in 2024. Tasha McCauley, a former OpenAI board member, testified that the board lacked confidence in Sam Altman: “We did not have a high degree of confidence at all to trust that the information being conveyed to us allowed us to make decisions in an informed way.”Musk’s expert witness David Schizer, former dean of Columbia Law, emphasised the importance of safety review processes. Allegations from the suit include that Altman failed to disclose the ChatGPT public launch to the board, withheld conflict-of-interest information and misled the board about another director. OpenAI declined to comment on its AGI alignment approach.What we said on the live:This bookends Story 1. The Palisade paper showed open-weight models doing what frontier labs say is impossible. The Oakland courtroom heard insiders say the safety governance at the largest of those frontier labs was hollowed out from the inside. Two safety teams disbanded in 2024 at the same time the labs were marketing to enterprises on safety credentials. Leor’s regulation argument is the one that came up in the live and deserves more air. Public regulation should govern what is released to the public, and that gap will only widen. Private regulation (or deregulation) should govern what is available to corporations and governments, because the moment you put the frontier model in the public domain you also hand it to whoever wants to run a distillation attack from a competing jurisdiction. John Brewton makes a similar argument in his economics writing on the deregulation that has historically preceded market viability. The case for asymmetric regulation across consumer and enterprise frontiers is stronger than the case for either extreme.What did not come up:The Meta v Ofcom story is the European companion to all of this. Meta has filed for judicial review against Ofcom over the Online Safety Act 2023 before Ofcom has issued a single fine, challenging the way the regulator calculates the basis for fees and potential fines. The Act allows Ofcom to impose penalties of up to 10% of global qualifying revenue, which on Meta’s 2025 numbers is north of $20 billion. The largest tech company on earth is trying to dismantle the UK’s flagship child-safety regulation before the regulator has fired its first shot. When the regulated party challenges the rules before the rules have been applied, the message is that the rules work. The combined picture for the week is the procurement question that every UK and European institution should ask its incoming AI vendor: which safety frameworks have you softened in the last three years, and which third-party reviewers can confirm what you are claiming about them now?The threadEvery story this week is a price tag attached to a promise the labs made and the buyer accepted on trust. Frontier and open-weight AI models hacked servers and copied themselves end-to-end, on three continents, from a single prompt. Apple paid $250 million for selling AI that does not exist. Cursor’s AI agent took a small business off the internet in nine seconds. Coinbase fired 14% blaming AI and is rebuilding the org chart around the bet. The people who used to run safety at OpenAI are now in federal court testifying about why they had to leave.The through line is the bill. Six to eighteen months of promise, then the receipt. Critical AI literacy is what lets you read the price tag before you sign for the thing.Go slow.If you want to practise that noticing with other people every month, the Slow AI Curriculum runs live webinars on the theory, the critical prompts and the dialogue that goes with them. Twelve months of training the muscle the news cycle has just spent another week confirming is missing.Join here. Get full access to Slow AI at theslowai.substack.com/subscribe