#132 - Your Screen Is the Training Data: AI Now Learns Your Job by Watching You Work

Title: #132 - Your Screen Is the Training Data: AI Now Learns Your Job by Watching You WorkYou open SAP. You copy a number into Excel. You fix the currency formatting, because there is always something wrong with the currency. You hit submit. You have done it a thousand times — you could explain it in your sleep. Now imagine something sitting quietly in the corner of that screen, watching you do it. Not to grade you. To learn it — so the next thousand times, it can do it without you.That is not a thought experiment anymore. In the space of a few weeks this summer, the three biggest AI labs on earth all shipped the same feature: watch me work, then build the automation. And the uncomfortable part is not that you will want to use it. It is that your management may decide everyone should.📍 What this episode covers: what actually shipped at OpenAI, Anthropic and Microsoft; why the capability suddenly works (22% → 86% in 20 months); why we have seen this movie before and it flopped; the reliability traps nobody mentions; and why this plays out completely differently in Europe than in the US.🧰 Three labs, one feature, one summer. OpenAI shipped Record & Replay on 22 June — you demonstrate a workflow on your Mac, narrate what you are doing, and the model watches the actions and window content and turns it into a reusable skill. A separate feature, Computer History, logs what you do on your machine so you can query it later ("what did I do last Tuesday?"). Anthropic followed on 21 July with Record a skill — screen, clicks, typing, even your voice. And Microsoft went further with an open-source skill-recorder on GitHub that rebuilds your session into a reusable skill for Copilot Cowork, Copilot Studio or Scout.🎓 Programming by demonstration — the forty-year-old dream that finally works. You do not write the instructions. You just do the thing, and the machine writes the instructions itself. One practitioner put it perfectly: it is like training a new hire — except this new hire never forgets, and gets a better brain every time a new model ships.📈 22% → 86% in 20 months. OSWorld is a benchmark of real desktop tasks; the human baseline is 72%. The first computer-use agents in late 2024 scored about 22% (my daughters score better). Early 2025: 38%. Late 2025: the 60s. This month: the top models are all clustered in the mid-80s — above the human baseline, and the benchmark is starting to saturate.🛑 The contrarian caveat. 86% does not mean 86% of your work is done. Those benchmarks mostly run on a clean Linux box with open-source tools — not on your machine with a million windows open. It still cannot do most of what happens on your SAP screen. But it is getting there, fast.🐴 Why the labs are chasing the boring stuff. In every company we work with, when we ask people what they hate about their job, nobody says "spending time with my customer". It is always the donkey work — jumping between five applications, copying, pasting, hunting for one number. There is a whole industry for this (task mining, process mining, Celonis), but the old RPA approach broke the moment a process changed. LLM-based agents adapt instead. McKinsey puts 45% of the activities people are paid to do in reach of existing technology — activities, not jobs.🎬 We have seen this movie. Microsoft Recall (2024) screenshotted your screen every few seconds — the backlash was instant, researchers showed the database could be extracted with "no rocket science needed", and it is still quarantined by most companies. Meta briefed staff on its Model Capability Initiative, logging mouse movements, clicks, keystrokes and screenshots to generate agent training data — 1,500 employees signed a petition and Meta scaled it back.⚠️ Two traps before you deploy anything. Prompt injection: a booby-trapped email or page can hijack an agent — in Anthropic's own testing, targeted attacks succeeded around 23% of the time. A Trojan horse that works one in four times. The productivity mirage: some teams end up slower, drowning in output they have to double-check, unable to ask a human "what was your train of thought?" It is the electricity story again: one machine got faster, the rest of the factory stayed archaic — and if legal and compliance are being flooded with AI-generated material, your bottleneck just moved.🗣️ The CEOs already said it out loud. Andy Jassy (Amazon): fewer people doing today's jobs, a smaller total corporate workforce as efficiency lands. Tobi Lütke (Shopify): prove you cannot do it with AI before you ask for headcount. Marc Benioff: 30–50% of the work at Salesforce is now done by AI. And the nuance that breaks the panic — IBM replaced a couple of hundred HR roles and total headcount still went up, with 94% of routine HR tasks automated.🇪🇺 Why Europe is a different game. American companies will just do it. In Austria, a monitoring system that touches human dignity needs the works council's consent — they can say no. Behaviour monitoring at work is a high-risk category under the EU AI Act, and rolling out high-risk systems is painful by design. So the adoption gap between Silicon Valley and Europe will widen. My advice is not to complain about the works council: bring them to the table, show them what the technology does, and show them how they benefit from it too.🎯 Three things to try1. Pick a boring workflow, not a flashy one. Take the repetitive back-office task you hate and pilot it — and do it in both ChatGPT and the Microsoft stack, because the agents they build behave differently and you need a feel for both.2. If you cannot do it at work, do it at home. Record a skill on your personal laptop and show your manager. Most managers have not seen this yet.3. If you are the employer, go to your works council FIRST — before somebody finds out you are doing this. Not because you should fear them, but because you need them with you.🔑 The line I would keep: this is not a question of intelligence anymore. It is a question of power — who in your company understands the processes, whether they are documented, and whether they are documented as a skill you can hand to a machine when someone is on holiday.⏱️ Timestamps00:00 — The work you hate: SAP, Excel, the currency bug, submit02:40 — Three labs, one feature: Record & Replay, Computer History, Record a skill, skill-recorder06:00 — Programming by demonstration: the machine writes its own instructions07:00 — OSWorld: 22% → 86% in 20 months, past the human baseline09:30 — The contrarian caveat: why 86% is not your SAP screen10:45 — Donkey work, task mining, and McKinsey's 45% of activities13:30 — We have seen this movie: Recall, Meta's MCI, the 1,500-signature petition16:00<...