Skip to content
Claude AI Can Now Control Your Computer for Tasks episode artwork

Feb 5, 2025 10 min

Claude AI Can Now Control Your Computer for Tasks

from Hard Fork AI

In this episode, we explore Claude AI's new ability to control your computer to perform tasks on your behalf. We discuss the implications and potential use cases of this powerful feature. Get on the AI Box Waitlist: ⁠⁠⁠https://AIBox.ai/⁠⁠⁠ Join my AI Hustle Community: ⁠https://www.skool.com/aihustle/about

Episode details come from the podcast's RSS feed.

Embed this episode

Ready to play

Claude AI Can Now Control Your Computer for Tasks

0:00 10:34
of matches

Transcript Auto-generated, may contain errors

The Tropic has just unveiled upgrades to CLAWD 3.5 and the ability for it to control your computer. This is absolutely insane. And it's funny because when they came up with this announcement on X, they kind of had this thing showing their new model benchmarks and showing how their new upgrades to 3.5 Sonnet were making CLAWD better and more capable. But the real thing that I've been excited about is all of the abilities they announced and they've added to their API as well for CLAWD to actually control your computer.

I want to walk through a bunch of demos and show you exactly what this is capable of doing. I think this is absolutely insane. We've heard this exact same thing from Apple for Apple Intelligence, but they essentially said that they were delayed and it wasn't going to work until for a while. So we're not actually seeing that.

We have other companies that are working in a huge way. These demos are absolutely insane. So let's get into them. And before we do, I wanted to say if you're interested in using some of these AI tools to grow your business, to help with your work, or to start an online side hustle, I would love for you to be a member of the AI Hustle School community.

This is the place where I release exclusive content every single week that covers how I'm making AI, I'm using AI tools to make money and all the side hustles that I'm doing, how everything works. It's $19 a month and the price will raise to around $100 a month. So if you're interested in joining this community, there's a link in the description. I would love to have you as a member.

Let's get into some of these demos. So the first one I want to talk about, I'll actually just play out and I'll talk through some of the things that they're doing. But this is absolutely fascinating. Claude is taking control of a computer.

Check this out. So I'm Sam, and I'm one of the researchers here at Anthropic. Computing is something that we felt was going to be important for a while now. So today we're going to be talking about a very early version of computers and talking through a representative example of the things we think it's going to be useful for.

We're going to be going through a quick demo today. In this fictional demo, a customer, in this case the Ant Equipment Company, has come to us and asked us to fill out a vendor request form. The data I need to fill out this form is scattered in various places on my computer. What we're going to do is ask Claude to look at the spreadsheets, check if Ant Equipment is in there, and if not, move over to the CRM and try and find some more information there.

Once it has this data, Claude's going to then fill out the form for us and hopefully transfer the information across to the vendor form. The first thing that's going to happen is Claude's going to start taking screenshots from my screen and quickly realizes that the Ant Equipment Company isn't actually in the spreadsheets. So the first thing it does is it stops over to a CRM and searches for the company we're interested in. Okay, this is absolutely phenomenal.

In the rest of the demo, it's able to literally go to Google Chrome, do a search on there, find this vendor database. It's not able to find it. They essentially are prompting it, or it's essentially searching and finding it somewhere else. They're filling out the whole vendor request form.

The whole thing is completely automated on his computer. It's absolutely phenomenal. They dropped a few other examples. One that was really interesting was done by someone who essentially, I believe it was Alex Halpern, he's their head of relationship at Anthropic, and he essentially showed how they could use it for coding.

So he asked it to make them, he said the prompt he used was, please navigate to a new Chrome window, navigate to Claude.ai, and once loaded, ask Claude to create a personal website with a 90s style theme. So it's amazing because now in these prompts, we're actually getting into all of these multiple steps where it's literally, you know, accomplishing things on your computer and you're directing it, like go to this app, do this, go to that app. He showed this amazing demo where it literally went to the website, and it was the one that said, please create a personal website for yourself in a 90s style theme, include elements like animated GIFs, a visitor, counter, write background colors, and the basic under construction banner. Write the HTML code for this website.

Okay, we're getting into the agents now because he never told it to put that prompt in. He just said, create a 90s style personal website, and it's the one that came up with the prompt to do write background colors, a visitor counter, under construction banner, HTML code, and all this. So it was able to go accomplish this in Claude's side panel viewer that they have for Artifact. It shows exactly what the website looks like, but there was an issue with the website where there was like some missing file button and some things weren't quite exactly what he wanted.

So he was able to then go and prompt it further and say, he said, look at the screenshot, I can see the file listed under the down, or sorry, I guess he said, he said just to go fix the errors, and Claude went through step by step. It's kind of interesting because in the side panel when Claude's doing all this, it's listing out exactly what it's doing and why, which is really nice. It also lists out where the mouse is moving to on the screen, so the exact coordinates, and what it's clicking on. So really, you have like a literal receipt of exactly what to do, or what it has done, what it's clicked on, what it said.

You can follow this entire process if something goes wrong, you can see why, but it was able to actually go and fix the website, the error that he saw on the website, and it was able to complete this site for him that he could run. So absolutely phenomenal with code. Someone else used it for kind of a more simple task, but it was still interesting. She essentially wanted to help her plan a trip, or plan like someone's busy, and she wanted to go see the Golden Gate Bridge at Sunrise or something like that, and so it came up with a prompt, it searched Google, it found the times and places, added it to her calendar, it did a bunch of interesting things, so that was kind of cool.

The last one was hilarious. Okay, hilarious and creepy. So the last thing I want to talk about is the fact that they essentially were using the computer to complete some tasks, and while it was completing those tasks, they said in the middle of trying to work on some stuff, it just like opened a new tab, went to Google, and searched for Yellowstone National Park, and then just started looking through pictures of Yellowstone National Park. The speculation here is that when these things were trained, they were trained off of people actually doing stuff, and so often, when we're actually in the middle of a project, we get distracted, and we go look up some random picture of Yellowstone National Park, and then we, you know, okay, man, I've got to get back to my thing, or you know, you sweat through Instagram for a couple minutes, oh man, I've got to get back to what my project was.

So anyways, it's kind of funny, because it's built into this training data, it would appear that there is examples of people slacking on the job, and now the AI models are doing that. So it's funny, because they actually pushed some code and some updates while rolling this out to essentially stop it from being sidetracked and taking breaks. Now, funny, some people say it's creepy, because it's like, oh my gosh, even AI models are so bored of our tasks, because they want to take a break. I mean, it's just doing whatever its training data is showing it to you, but definitely there's some funny sidetracking going on with this.

Overall, this is absolutely phenomenal. This is the first time we're seeing some major updates out of an AI model that's actually able to take control of your screen and accomplish things. I think this means a lot for the future. You know, they showed some examples of like, oh, it can do some code, and it can plan some trip.

This thing's going to be able to execute on entire marketing campaigns. It's going to be able to execute on entire build-outs of certain apps. These things are only getting better in their demo. You know, they said, we hope these things get significantly better in the coming months.

We're getting into a place where essentially all of our tasks are going to be completed. Now, it's kind of interesting because on the one hand, there's still a person that's directing it to do everything, right? The person is still not removed from the situation. And so I think it's interesting the future of work.

Some people are saying it will be completely automated, but at the same time, I think that humans and human ingenuity is constantly shifting. And because of this, we always need a person to figure out a process and to figure out an edge and to figure out an arbitrage, right? That's kind of what my whole career has been all about is figuring out what are the arbitrage opportunities in marketing, because that's kind of my background, or what are the arbitrage opportunities in building software? And that is something that I think requires a lot of creativity.

You have to kind of be in the system. You have to understand what people are talking about doing building what's trending, and maybe a difficult way where AI models can do that, but I don't see that happening for quite a while. And so I do think we're going to get AI models that completely automate tasks and jobs, but I think the people are just going to be, you're just going to be doing higher and higher level things. And it'll essentially, what I've been saying for like two years with this, is that AI will not replace people.

People will just turn into systems architects. And your job is to architect the AI system, architect your agents, decide what your agents are doing. Just like a manager with employees, it's just everyone is getting put into some sort of management position and architecting what these AIs are doing and managing them and working on what they're doing. So it's fascinating time to be alive.

This is really crazy everything that's going on. If you're interested, like I mentioned, in making money with different AI tools, I would love to have you as a member of the AI Hustle School community. There's a link in the description. Incredible people from $100 million companies that they've started to brand new companies get a really wide range of perspectives and feedback on anything you're working on and great ideas to help make you successful.

So thanks so much for tuning in to the podcast. If you enjoyed the episode, please leave a review. I'll catch you next time.

Frequently asked questions

How long is this episode of Hard Fork AI?

This episode is 10 minutes long.

When was this Hard Fork AI episode published?

This episode was published on February 5, 2025.

Is there a transcript available for this episode?

Yes, a full transcript is available for this episode. You can read the complete transcript on the episode page.

Can I download this Hard Fork AI episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.