We've been using AI tools for long time and engineering. I believe we were the first company outside of GitHub to use GitHub Co-Pilot. And the reason I know that is because when Thomas Donkade became the CEO of GitHub, the same day I emailed him saying I would like GitHub Co-Pilot, this is 2021, a year before Jatch GBT. He messaged me back saying, it's not available for commercial use, and I said, that's not what I asked. I don't care what you have for commercial use.
I would like this deployed for all Shopify engineers as soon as it came to be possible. I think it took about a month, and we were not charged for two years because there was not excuse, because charges, and we said in exchange we'll give you lots and lots of feedback. And so we were using Co-Pilot for a long time, and then we worked with them as we started looking at the roadmap. We then deployed cursor internally as well, actually we're very new to cursor about a year, into cursor, so we're kind of trying all these things. And as we try them, we want to see what's working, what's not working, and then we let more engineers use it.
The most interesting thing about cursor is that the gross in cursor at Shopify is happening a lot outside of engineering. And outside of R&D, finance, sales, support, those are the teams using cursor. What happens when a company goes all in on AI? Shopify did exactly this a few years ago, and Shopify's head of engineering far on top of R told me exactly how it's going so far. In this episode we discuss, how Shopify works closely with AI Labs, and why far on pair for an hour with an engineer at Atropic working on the Cloud Code team?
Why Shopify is applying to hire 1,000 engineers in a year, and how they incorporated AI into their engineering process? Why Shopify has no cost limit on how much an engineer or team can spend on AI tokens and many more? If you're interested to know what an AI first company operates, like this episode was for you. This podcast was recorded as a live podcast at LDX3 in London. If you enjoyed the podcast, please subscribe on your favorite podcast player, and on YouTube.
So welcome everyone to this very special live podcast with Farhan Tower, and we're gonna talk about AI at Shopify. But before we start about AI, I just want to talk a little bit about you Farhan. So I did a little bit of research. I talked with a couple of engineers at Shopify, and they told me, your role is the head of engineering, and engineers told me that they see you in everything. You have reworked Shopify.
It's an intern hiring program, you define what Shopify does. In fact, you even were in the weeds to deploying an internal hackfest more than getting the Wi-Fi right for a hackfest. So I'd like to know what does a head of engineering do at Shopify, specifically what do you do, or maybe what do you not do? Yeah, so I think what's interesting about Shopify is that we use this line, we're not a swim lane company, which means that we don't try to put people into these roles where you are like, you're in products, so only think about products, or you're an engineering, about how the code is written or architecture. We're very much just curious problem solvers, and so if something's broken, we expect curious people to go and look at the problem, even it doesn't matter what the role is, and try to solve it.
And so last year at our Shopify Summit, which is our employee event, where 7,000 people come to one space, the Wi-Fi did not work super well, and so I spent all my time in that summit for three days, trying to fix the Wi-Fi. I became known as the chief Wi-Fi officer, and so this year, which was two weeks ago, I deployed 300 ubiquity APs, 800 switches, in order to have access ones, to have a much smoother Wi-Fi experience, which is how that all got started in terms of, and the team built lots and lots of memes, because I thought the Wi-Fi was not gonna work, but it did work, and so I did publish all those memes. I need to ask you, because a lot of leaders would say, look, your time is super valuable, you're ahead of an organization, how many people are?
3,000 people, you're kind of up there, and your time is now, if you measure it and dollar it, it'll be expensive, it's a lot better to just get a specialist to do these things, why do you still do these things, and do you disagree with that advice, of focus on the high leverage things? Yeah, so we don't love the, there's that notion of higher smart people and get out of their way, and then what we say is, higher smart people and pair with them on problems. So the difference for us is we really like to bring in, obviously lots of smart people, hands on people, and instead of just saying, hey, you take this area and then go away and just solve it for me and come back, we like to pair with them and say, you're smart, I'm smart, can we work on this problem together? This episode is brought to you by WorkOS. If you're building a SaaS app, at some point your customers will start asking for enterprise features like SAML authentication, skin provisioning, and fine-grained authorization.
That's where WorkOS comes in, making it fast and painless at enterprise features to your app. They're AP as easy to understand, and you can ship quickly and get back to building other features. WorkOS also provides a free user management solution called AuthKit for up to one million monthly active users. It's a drop-in replacement for all zero, and comes standard with useful features like domain verification, role-based access control, ball projection, and MFA. It's powered by RABX components, which means zero compromise in design.
You get limited customizations as well as module templates designed for quick integrations. Today, hundreds of fast-growing startups are powered by WorkOS, including ones you probably know, like cursor, versatile, and perplexity. Check it out at workwells.com to learn more. That is workOS.com. If you want to build a great product, you have to ship quickly.
But how do you know what works? More importantly, how do you avoid shipping things that don't work? The answer, Statsig. Statsig is a unified platform for flags, analytics, experiments, and more, combining five-plus products into a single platform with the univisive data. Here's how it works.
First, Statsig helps you ship a feature via feature flag or config. Then it measures how it's working, from alerts and errors, to replays of people using that feature, to measurement of top-line impact. Then you get your analytics, user-con metrics, and dashboards to track your progress over time, all linked to the stuff you ship. Even better, Statsig is incredibly affordable, with the Super General's Freet's here, a startup program with $50,000 of free credits, and custom plans to help you consolidate your existing spend on flags, analytics, or A.B. testing tools.
To get started, go to Statsig.com.com.com.com that is Statsig.com.com. Happy building. So this leads me really nice into the next thing I heard about you, which is, at Shopify, you have gone kind of AI-first. We're going to talk a little bit about that. But one thing that you do is you have access to some of the top AI labs, and what you told me just before that you pair with them.
You just told me about the other day you were pairing with anthropic engineers. Can you tell me about this whole pairing and how you work with these AI labs? Yeah, because again, we're like super curious problem solvers. And so when there's something interesting happening in the industry, we want to be close to it. So obviously, a chat, a PT comes out, and then the endropics builds their model, Gemini has their model, cohere has their model.
We want to be as close to the leading edge as possible. So we try to be close to those people and what they're working on. And one of the things we do is, when something comes out, this is an example of this cloud code. We deployed inside of Shopify. We saw people using it.
It was in May, right? Three or four weeks ago. So Cloud Code came out before that, like six months ago. And then we started spending time with anthropic. I wanted to see how anthropic was using Cloud Code internally.
Similar to like opening eyes now releasing code X, I want to figure out how they're using it internally. And so what I did was I reached out to anthropic and said, hey, I'd like to pair with one of your apply to AI engineers. We spent an hour building a bunch of things. And I learned how he sees Cloud Code being used internally and anthropic. And I can take that back and say, well, here's how we're using the Shopify.
And we can together figure out where we think this thing can go. Is this just a matter of like, you just asked them? Yeah, I mean, because we are Shopify, we're very lucky. People want to see how we work. We tend to work in an unorthodox way.
And they also are interested in how we use it. But they also use it. And so it was more of a me, again, a pairing. It was a true pairing. It wasn't like me as a customer trying to push them and say, I really want it was we have a shared black channel.
I think they're saying, hey, who would want to pair with me for an hour on something I'm building inside Shopify? So I'm going to put their hand up and apply the engineer and said, we'll show you how we work and we show you how we work. And we can together figure out where we think this product could go, technically with a lot of the providers. And as you're pairing, is it just you pairing and then you later tell people? Or do you do it?
Yeah, so I might record the session and then share it with internally to say, hey, I just have a pairing session on the topic. They might use it for a case study that they're building. But the idea is, again, because I'm just curious when somebody says something, I'm like, how does that work? How can we use this? And so I think it's a way for us to just learn.
And one recent thing that I just wanted to go into before we get into some more of the AI things, it's a really fun fact that most people know. You just came out of a code red for seven months. I think this is the first time we're willing, what was this code red? What was about? And what did people outside and instead of Shopify see of it?
Yeah, so we were internally seeing a lot of things that weren't bubbling to the surface. But that can actually affect the operation of the system. So you would typically call it TechDet. We had a lot of signals showing that the TechDet was growing, not just the general ones, like taking longer to update a piece of software or build a version two of a product, or people having a hard time maintaining some of the software stack. We saw things even not externally, but internally that were shooting out signals, lots of exceptions growing.
Even in some cases, because we built things at the very low level of the stacks. So for example, we patched, we're core contributors to Ruby. We patched MySQL. I believe we have the second largest MySQL fleet in the world outside of meta. And we were seeing things like segfaults and things like that.
That I was not willing to feel like we should move on and just build features as normal. And so we actually got together, spent, we didn't think it was going to be seven months. We thought it was going to be three months. And we took something between 30% to 50% of engineering. We didn't have an actual number.
We just said, these things have to not grow anymore, like exceptions. Even if exceptions counts have to shrink, there should be zero segfaults. We should understand everything that's happening in the system. And so we spent almost seven months doing that over the last, you know, from November till now. How did you make sure that you fixed the right thing?
Did you like talk to the team saying, hey, bring up the things? And also how did you decide when you will stop? Because like, whenever I hear these code read, it's always a lot of companies do code yields code read. But sometimes they can drag on forever. Yeah, so we had multiple metrics that we used.
One was literally what I said, exception counts, unique exception counts, and segfaults. We should really say segfaults are zero now, and exceptions should be going down. But we also looked at other metrics that we have. Like, we have four, we want to have four lines of reliability across our different surfaces, storefronts, merchant admin, point of sale. Like, we look at all these things.
And we also used the 28 day rolling average of those things. And once those were all green, and we saw that, again, segfaults was zero and exception, stopped and was shrinking. We then felt like we were in a place where we can start building features again. And now going on to AI. So you're a huge early adopter of AI coding tools.
Can you tell me on what tools you start to use and what you're using right now? And what you're liking, what engineers are like, and specifically, like, it's a focus more on our software than maybe an outside of a software. Yeah, so we've been using AI tools for a long time in engineering. So we were, I believe we were the first company outside of GitHub to use GitHub Co-Pilot. And the reason I know that is because when Thomas came to the CEO of GitHub, the same day I emailed him saying, I would like GitHub Co-Pilot.
This is 2021, a year before JAPT. He messes me back saying, it's not available for commercial use. And I said, that's not what I asked. I don't care what you have for commercial use. I would like this deployed for all Shopify engineers as soon as humanly possible.
I think it took about a month. And we were not charged, I think, for two years, because there was not a SKU to charge us. And we said in exchange, we'll give you lots and lots of feedback. And so we were using Co-Pilot for a long time. And then we worked with them as we started looking at the roadmap.
We then deployed cursor internally as well. Actually, we were very new to cursor about a year. They were tiny startup back then. Yeah. So I think that's super late.
You're saying it super early. I feel like it's very late to cursor. We knew about cursor for a long time. We like to have one tool at JAPify for the things that we're doing. And so we don't have multiple tools.
So we have Figma. We don't try to have lots of other visualization tools. We have MySQL. We don't have other databases. We try to focus on one tool.
And so our bet was VS code. But because of AI's proliferation and we don't know what's going to happen, we then started making a change in our stance around trying more tools than just one. Maybe we can solve it. Maybe we don't. But right now we have cursor and VS code as our two AI tools.
And then Cloud Code came as the agent workflow. And we started using Cloud Code. We tried Dev in last year. So we were trying all these things. And as we try them, we want to see what's working, what's not working, and then we let more engineers use it.
The most interesting thing about cursor is that the growth in cursor at JAPify is happening a lot outside of engineering. And outside of R&D, finance, sales, support, those are the teams using cursor. What did they use it for? So what's happening is because cursor is actually so ubiquitous on building things, they're using it to build MCP servers outside of regular engineering domains. So for example, Salesforce, Google Calendar, Gmail, Slack.
They're building MCP services and then building home pages for themselves, like your salesperson. Connect to Salesforce, connect to Google Calendar, connect to my email, and tell me what opportunity I should be working on. So with an MCP server, when you have a server, you can put an MCP server, and then you can act with, for example, cursor or with an agent. Do engineers help build that MCP server? Sometimes, yes, but a lot of times, no.
Now that these services are coming out with their own MCP agents, so that you can literally just download GitHub's MCP server, you don't have to actually build it yourself. But a lot of times, they're actually just building it themselves. And they're watching videos. And they're actually just deploying it on their own without any engineering dimension at all. Now it's end of one, right?
They're building software for each individual person, not infrastructure for all people. Does this remind you of anything you've been around in the tech industry for a long time? But the fact that the non-technical people are now building, does it, do you see any parallels from the past? Did this happen before? Or is this new?
This coding part is new? Although, you remember, we see wig editors and people starting to self-serve. I think it reeks of that. But they are, I mean, they're not reading the code, right? They're really much by coding.
And if it doesn't work, they just delete and start over. But even if you read on Thoropics documentation on Cloud Code, they say one third of the time you can get something working with one shot. And so we're starting to see that. And people are getting used to the fact that if it doesn't work, it is to delete and start over and try to get it working. I find this really interesting on how it's happening.
Does it change the dynamics with engineering? Because one thing I noticed when working at PlayZ Like Uber, everyone was a bit jealous of engineering. They wanted engineering resources. They couldn't get it. They really wanted one of these things.
Do you see any of this change? I'm going to say yes and no. So there's a weird thing happening whereby, let's say you're a PM. Let's say you're a technical PM. You now probably have enough knowledge to vibe code.
Some new feature you've been waiting for engineering to build for you yourself. The question is, can you submit a PR and should the PR be accepted by engineering? I'm not your first. And I'll tell you what I think. Well, I think you can always submit a PR.
And engineering will tell you why it can't either be accepted, if it's good, or what is missing. It may not be conceptual, it may be coding standards and all those things. But at least, you know, the speed back for them, they're seeing what this person wants to do. I think it should be a great thing. Yes.
I would say yes with one caveat. And the caveat is the problem with vibe coding today is you might generate 10,000 lines of code for a very simple application feature that now the burden on engineering to read the 10,000 lines is there. So what we say is yes, you can submit a PR. But you have to understand the code you are writing yourself. Right, your stuff.
Exactly. Before you submit it. Because I can do the same thing. I can say, hey, I'm going to write a blog post, generate 20 pages, and then send it to you and be like, hey, edit this and post it on your blog. And now you have the burden of having to.
Now, if I said I read it all and it is my voice, and I agree with it, then you might be more likely to read it. The interesting thing that we're probably going to see this from open source projects, because it's so easy to use AI tools. And they're going to get a lot of these things. I think policies are interesting. Exactly.
Now, I want to get your take on one thing, which is going around in social media. And a lot of people predicting is the end of SAS, because people can create their own SAS. Now, what you told me is some of your non-technical people are creating like small solutions. But themselves. For themselves.
How do you see this potentially changing SAS vendors? Or are you? And you're also someone who actually buys a lot of SAS vendors. Do you understand them? Do you think they change anything or not?
Or is it just more personal software? Yeah. I'd like to think we buy less SAS than those companies, because we try to consolidate down. That being said, there is this notion. We're in this middle zone right now.
The middle zone is basically this idea that you can vibe code something, maybe, for yourself. It's unclear whether you can vibe code as a platform or vibe code, something that gets into the infrastructure layer of what you're building, because you do really not to understand, again, you mentioned earlier, you might be putting yourself in a very precarious situation. If you're building on top of something, you don't understand what you're building, maybe it's a prototype. That's fine. Or a proof of concept.
But if you're building infrastructure for the internet, you likely want to understand how that's being built. So that is not. We're not there yet. Is it coming? Yes.
Is it coming? Yes. It's for sure coming where someone can vibe code something in the entropic, or OpenAI, or Gemini, or who cares, who model. Build something that's actually architecturally elegant in the right architecture for what you need. We're not there today.
So where we are is this notion of still human in the loop. And still start over. And this is not what I meant. And prompt engineering, English as a programming language. And so I do think you can get yourself very far.
But I'm not worried yet about SAS. I think we should be worried, by the way, because let's take another view. How much software do we think should there be in the world? Probably 10,000 times as much as there is now. 100,000.
There's a lot of software that should be in the world. And we are satisfying 0.001% of the demand. So now that everybody can generate software, we should welcome them into the software world. We believe everyone should be writing software just like my sales team is. And everybody becomes more productive.
I'm not worried about the software industry. I still believe this is Jevins Paradox, the more we get, the more we want. Yeah, a really good analogy that has stuck with me is Simon Willison was saying, how has the filming industry changed with this? Everyone has this used to cost like $5,000 or $10,000 a few years ago. This camera, everyone has one there in pockets.
And there's still a professional movie industry. But now there's a bunch of YouTube. There's TikTok. There's all these things. So it has become bigger.
But the barrier for the pros has not really changed. Doo-n and some of these amazing movies are so hard to make. Right. Jevins Paradox, the more you have, the more you want. We all want to be filmmakers now.
We can be filmmakers. And then there's still going to be people that high end of the low end. I would say the controversial, maybe controversial. Spicy take is that these phones actually benefited the experts more, meaning the camera, like the being able to pull out your iPhone and take a video actually benefited those who are really, really good at it already. And the same thing is maybe potentially true of vibe-building.
It's like the AI agents are going to help the best engineers more than the mediocre engineer. So there was a memo that went around the media about a month ago. Toby sent a memo he was meant for internal consumption. It was about reflexive AI usage. And I guess it got leaked then and told me posted the whole thing.
And this is about saying that, hey, everyone is expected to use AI at Shopify. Can you tell me why you felt that memo was sent out? What was the response internally? Has it changed? Anything or was it just kind of stating what was there already?
Yeah, it was a fundamental statement that this is something new. And it was meant to get everyone to move from passive to active. So yes, there's AI in the world and chat, all these things existed. But we started building infrastructure internally to make it easier to use. And we basically said the following.
You don't have to use AI in your workflow. But we're going to expect that you have those plethora of tools available to you, such that the expectation is you're going to be like, your impact is going to be evaluated as if you had the tool. So for example, imagine not having Excel or Google Sheets. And you have to do a complicated analysis. You don't have to use those tools.
You can use a pen and paper. But we're going to expect that you have these tools available. And we expect the level of analysis to be treated as if you had these tools. And so by us doing it, what changed internally was actually probably more of a change outside of R&D like I mentioned. R&D was already leaning in.
Now, of course, there were some folks who were waiting for R&D as engineering. R&D is engineering product data and design. They were more likely to use these tools because they tend to be more forward looking and wanting to use the latest and greatest. However, some people were waiting. It changed their shape because we said, hey, look, we're going to expect that you use these.
But outside of R&D, we saw it even more. Again, sales, finance, customer success, help centers, all those roles started using AI as well. And we, again, gave them the infrastructure to use that. So we have an LLM proxy. It has all of the APIs and all the models available.
You don't have to be worried that your personal information is being leaked to a model. You can kind of use those tools. Let's talk about these things. Let's talk about the tools that you either created or adopted early. I heard a bit more about this.
LLM proxy. And all I heard something about MCP. Yeah. So when I talked to people, let me start with LLM proxy. The problem at the beginning was people would go to chat GBT or Gemini or Claude.
And they would be like, you don't want to put customer data there. Or even employee data. You want to use it to write your employee reviews. You don't want to put, like, Simon is having problems with this and put their first name and last name. Here's a project they worked on in the code names leak.
So we wanted to quickly have an internal LLM proxy. Plus we built on top of LibraChat, which is the open source, they can chat product. And we're core contributors to that as well. And with the LLM proxy, you're sure to be using the enterprise APIs. We can check to see which we get a token.
You ask for a token. We can see which teams are using how much, like what Moss is being incurred by that team. Or by person. We have a leaderboard where we actively celebrate the people who use the most tokens. Oh, really?
Because we want to make sure that they're doing great work. Of course, if they script, that's not what I mean. I mean, they're doing great work with AI. I want to see how did they spend $1,000 a month in credits from Biocersor? Maybe that's something they're building something great.
And they have an agent workforce underneath them. We have this proxy. And again, it's used to build anything that you want to build. And you can ask for a token and use it. And then we are very early on MCP.
We're on the MCP steering committee. We are fans of MCPing all the things. And so when I talk to people about how to get access to some piece of data inside the company, we quickly will spin up an MCP endpoint for them so that they can just use it. So for example, our internal wiki is called Devalt. It has information about all of our projects that are happening.
It has information on every internal talk that has been done. It takes the transcripts of all those. You could see something like, when did the point of sale at Shopify launch? And it will come back and it will say, oh, in 2012, there was a hackathon where the point of sale was started. And then the point of sale was on wiki.
It's on the wiki. And then the 2013 Q1 board letter, it was mentioned. It would tell you all of the history of what happened in Shopify. And so we do that by just having all you do is put up an MCP. And now the Libra chat has access.
It's like an internal perplexity or something like that. Exactly. And so we basically any internal document is crawled by this thing. And now you can have access to it by MCP. And so any new system comes online?
We've got an MCP in front of it. And now everybody has access to it. This episode is brought to you by Sonar, the creators of SonarQ, the initially standard for integrated code quality and code security that is trusted by over 7 million developers at 400,000 organizations around the world. Human written or AI generated, SonarQ reviews all code and provides actionable code intelligence that developers can use to improve quality and security early in the development process. Sonar recently introduced SonarQ advanced security enhancing Sonar's core security capabilities with software composition analysis and advanced static application security testing.
With advanced security, the development team can identify vulnerabilities in third-party dependencies, ensure license compliance, and generate software bills and materials. It's no wonder that SonarQ has been rated the best that it can allow us this tool for five years running. So join millions of developers from organizations like Microsoft, NVIDIA, Mercedes-Benz, Johnson, Johnson, and eBay, and supercharge your developers to build better, faster with Sonar. Visit SonarSource.com slash pragmatic security to learn more. And how many MCP servers do you have?
You have one? Two dozen. Two dozen. And they're growing quickly. They're growing because people are building a new thing.
Again, for Salesforce, there's ones for Figma. There's an MCP in front of anything. But I tell anybody who's looking at building AI systems internally, the way to make this easy is start putting MCP in front of everything. And now your data is accessible. We had a Japanese Leber chat.
You might use something else. But you don't have to be an even in cursor, right? You just use the chat in this. Yeah, you can do the LMs. And I like how you basically have platform team to build the MFROP, to do monitoring, to worry about things like security, decide on models, all those things.
Does this team have a name? Is it an AI platform? Or is it just a platform team? Impertime just took it over. Yeah, data platform.
I don't know if I'm trying to figure out because they're inside the data platform team. I don't know if they have a name. I call them LMS. And what's this more like you or someone upset? Like, hey, we need to do this?
Or are they just like, hey, is it a good idea? That's kind of bottom up. Hey, hey, this team is growing. Is it usage growing? What's happening there?
This person's usage is just a good idea. Number one pushback I get from people or engineering directors working at companies is the data security issue. And obviously, when you host it yourself, it's good, but then you need the infrastructure. So it sounds like at what point did you decide? What was this month ago?
Was this more than a year ago? To like, I'm the best. Probably, I mean, I would say every week it's getting better, right? But I would say probably a year ago, we started thinking about it this way. We had the LM proxy.
When Chat TBT came out, we actively have a warning. If you go to Chat TBT, say, by the way, like you can go to the internal version. That was a long time. No. You mentioned cost.
And I am hearing stories of CTOs, engineering leaders saying, OK, we were paying for GitHub co-pilot. I don't know. It's $10 or $20 per month per engineer. But we have all these seats for me. We have everyone.
And there's only cursor feels a bit too expensive because they're now at $30 or $35. And we're not going to go there. And generally, there's a big pushback on cost. People are saying, look, AI should make engineering more productive. But it's now like, it's kind of ridiculous to spend $1,000 on an engineer.
You think about cost differently. Tell me about it. Yeah, I did a sweet storm on this because I think that people are looking at it differently. So think about it this way. But the answer is yes.
Would you pay $1,000 a month for it as a question? And my hypothesis here is $1,000 a month is too cheap. Like, if I get 10% more, it is way too cheap. Like, we should be $5,000 a month. It's insane that people are looking at this $100 a month, whatever.
Now, I did meet somebody who said they're engineers for spending $10,000 a month per engineer. And I said, I want to talk to them because I want to see. Either doing something very, very smart or very, very stupid. But I want to meet them. And so I let me say, please introduce me.
Because I'm unclear how you can use that much. Either they've got a field of agents building amazing things. Or something is going very, very wrong. But I would like to understand because there's something interesting happening. And you should not be penny pinching on AI tools.
Because actually, the productivity gains are there. We don't know what they are because we actually don't know how to engage in developer productivity. But it is clear that there's something happening there. And you should be very open with your pocketbook there. And so again, we celebrate those who spend more.
And of course, we talked to them. And what did they do? So they were getting the product. Do I understand that even though you cannot measure it right now? Or maybe not as accurately, you believe that there's productivity gains.
And you believe that people need to use it to get these. So you're kind of treating as an investment right now. For a year, for two years, whatever that may be. And then, you're going to be forever. I mean, again, if you're like, you wouldn't turn off spell check, and grammar check, and G Suite, and you wouldn't turn off these tools, right?
I mean, it's probably debated when you turn off startup slack. Because it's like, that could be practically better. But you're seeing the gains. And I don't think that we should be penny pinching. Now, over time, we might realize there's an equilibrium.
And like, oh, this is the amount of tokens people should be using. Actually, if anybody's using cursor here, the number one thing I'll tell you is, please move away from the default model. Because the default model is nowhere close to actually using a sophisticated model. And by default, of course, puts you there. And so we tell people, please move off the small models, move into something more bigger.
Believe me, say things like, please use O1 Pro for this project. Or please use O3 or the Pro, because those are more expensive. They're more expensive. I'll go even further. Mikhail, the CTO, will say something like the following.
If you, you, don't pay personally for O1 Pro or Gemini Ultra, or Claude, whatever it's called, advanced, the $200 month one, you are crazy. Because you can afford it. And actually, you are missing out on actually all of the progress happening at LMs, because you are just stuck to the $10 month or $20 month check GBT. Like, that's how far he would go. One interesting thing that I've heard when I talk with engineers, they're saying leadership is leading by example.
Toby is hacking. And if you want to see it on his Twitter or social media, you're going to see all the stuff that he's doing even a year ago, I want to ask you, what are you doing on the side, or experimenting, or playing with? And what tools are you excited about? Yeah, so when I was building, when I was pairing with that engineer with the on-profit, I was building something for myself. I was building a couple of things.
So one of the ways to do is come and sign. Let me see what they can build, because I didn't want to go super deep into the Shopify API. But we do build a lot of things that, you know, one was actually a commerce flow. I was trying to build a commerce flow via operator, where you can actually record the browser session. And I wanted to see how the session was storing credit card information or secure tokens as a member getting through a commerce flow.
And I was trying to figure out, I was trying to build almost like a subscription product without using our subscription product from Shopify. And so it was an interesting experiment where we were trying to see how do we store those credentials, especially when you want to store credit card as a store in the cloud or as a store by the vendor, are we storing it in the browser session. That's how I kind of stay as close to it as possible. I use cursor pretty often to build my own workflows. In this case, it was Cloud Code.
We actually use Gumb, under the Gumb Loop. Gumb Loop is an automation platform that is very interesting at its own layers. Here's one example. We tried to build, I did something super dumb. I was like, let's figure out when the next Sandisk Xtreme Pro 8 terabyte drive comes out.
The wrong layer. It was trying to read the JSON and try to get the image from the Sandisk side and figure out if the 8 terabyte drive is actually quite complicated. We built it in two minutes and Gumb Loop. Not because they're just. What did Gumb Loop do?
So Gumb Loop is basically like I was browser based. We said, go to this website, go to the search box, using English commands. And it came back right away and said, there's no 8 terabyte drive available. I said, OK, cool. Run it every weekend, send me an email.
It was just different tools for different jobs. And nothing wrong with Gumb it got to drop a code. I used it all the time. It was not meant to do this kind of web scraping, because what it is it was to do. It tries to build a web scraper versus Gumb Loop.
It already knows that on the scrape websites. So it's actually just a different layer of the stack. And what I would encourage people to do is figure out what you're trying to do. And we have a lot of automation happening in Gumb Loop now because people have to do something. I want to scrape a LinkedIn profile.
I want to find out what platform this company is using behind the scenes. I want to scrape a website. All of these things are it's built for the right thing. You have a lot bigger tools. Exactly.
So you should figure out where you want to work. Right? If you're writing code, if you're writing code. If you're writing, if you're writing web scraper, use Gumb Loop. So far it was AI, AI, AI.
And when this letter came out, externally, there has been a lot of some speculation saying, is this a sneaky way for Shopify to freeze hat count because it says you shouldn't hire unless you can check if you can do it with AI. But then you saw something interesting. I'm not sure how big Shopify is. But you told me that you're applying to hire 1,000 interns. Can you put that in context with this size?
Shopify, can you tell me why are you hiring interns where a lot of companies are saying let's not hire entry level people because we have AI. Yeah. What are you seeing that are not seeing? Yeah, a couple of things. One is there's this real notion of generations as they come up and graduate from different programs out of university or even coming from high school.
And you really want to be close to the next generation of people, one, they know things that you don't know. And so we restarted our intern program in full thrust this year. So last year we had 25 interns a term. I convinced it would be the higher 1,000 interns this term. Mostly because I felt that they would say, I know, there would be more AI reflexive than everybody else.
That was our hypothesis. So my tweet actually said, I want you to come to work with an LLM and a brain, not one or the other. The idea was that bringing these like, centaurs, AI centaurs, they know how to work with an LLM to solve their whatever workload is, whether it's a finance person or engineering intern. And we really focused on them coming in and changing our internal culture. And so 1,000 engineers over the year met 350 a term.
And what we saw there was one, they're excited, they're hard working. We also have them as a cohort meeting. They come to the office, we're a remote company. But we have the interns come in because we felt the younger people need to have younger people around them to work together versus just being remote in their condo or whatever. We changed the culture that way as well.
And we did see this. They come, they have AI reflexive. Kind of we saw the same thing in mobile. You were in mobile as well. And so it was in my last decade of work.
And I hired lots of interns because they grew up with mobile phones. And so this generation is growing up with the internet. And they're growing up with phones. And they're growing up with LLMs. So we wanted them to come in and change our culture around that axis.
So do you feel like people are learning from? We always wear interns with a secret weapon. You always learn more from interns. People think about internships as community service. Oh, that's so great.
Jobify your helping the next generation. We don't do it for that. We do it to learn from the interns. That is always the reason. So if you don't have an intern program in early town and I encourage you, you should be bringing those people in because you will learn from them more than you will get out of them.
You will take a net positive benefit from them. And of course, for those who don't say, please don't hire interns, you have to pay them. I'm like, we pay them. I don't know why people think that you don't. We pay them very well, actually.
And we do hire, I think I've hired almost 100 from the last year from the intern tool to come into our program. And we're very much thinking about the only way to get into Shopify as an entry level engineer is the intern program. I fully wish you would on the paying. I never understood why companies are cheating out on this. Because sometimes interns will reject some of the offers because it doesn't cover the living costs, which is wild.
Of course, you have to pay them. Actually, one of the best interns, it was another company's loss. And in Amsterdam, they offered 500 euros per month, no housing. And that intern would have loved to work there. What a different experience.
But they apply to Uber because they said it's just too low. And then we gave them a new grad salary. They could afford housing. They could afford to travel. And that intern could have done amazing things.
That company now, this person is still senior at Uber. They've been there for five years. So they're lost. So think about 350 engineers interns, about 3,000 engineers, about 10% at any one point. So if I joined Shopify as either a software engineer or an engineering manager, can you tell me how stuff gets done?
First as an engineer. What tools do people use? I'm assuming it's just no longer VS Code. There's going to be AI tools here and there. Yeah, well, first on, we have a very specific way of writing software to the point of we built our own project management system.
Like we don't use Jira. We don't use linear. We don't use, yeah. And the reason is because we believe that there's that line, like first you make a tool and then the tool makes you. And so we believe that.
And so we make a tool so that it can make us versus. It's not going to be against those amazing tools. I'm a big fan. Like linear is beautiful. And I'm a pivotal on time.
I'm a pivotal, a phroco guy myself. Being a pivotal person. But we do believe that we have a specific way of writing a building software. And if we use one of those tools, we would be adopting someone else's way. And so instead, we have a tool called GSD, which then forget shit done.
And it is a tool that allows us, it's a program management tool. Which allows us to think about what are we building, who's on the team, what's the latest weekly update. It has metrics. It actually pulls in PR reviews. So you can see it pulls in PRs.
And you can see there's activity happening on this project. Here are the core contributors. There's a long it's been going on for. This tool is how you get work done. And you're forced to update every week.
And now actually we have an AI tool, which will pull in the latest PRs and the latest conversations from the Slack channel. And it'll write an update for you. And then you can look at it and say, this looks good. And you can also tell it what did I not think about. And say, please, emphasize this.
And it'll help you write the project manager update every week. So this is interesting. I want to push you a little bit on this. Because I think there's two schools of thoughts here. One says, your weekly update.
I appreciate that you're doing it. The weekly update, the point should be for you to look back on what you did. Obviously, everyone hates doing this, by the way. And by AI doing it, would you not lose that kind of reflection? Well, this is why we pair you with the AI.
So we do auto-publish it if you don't do anything. But if you auto-publish it and don't do anything, we look at the stats. There is a little bit of loss from you, because you are now losing out on the context that you were supposed to gain as the champion of the project by looking deeply. However, we expect everyone just like a PR written by AI to have read the code and read the update. So you're responsible.
You're responsible. And so we're early days in this. We're literally three weeks in on this project. And we may revert and say, by the way, it turns out everybody has stopped looking at what was happening week to week. But we also, every six weeks at a company level, with Toby, go through every project in the company.
And we see if we think that it's on track, not on track. We kept working on has the right resources. It's not too long. Is it in the right direction? You better know what's in there, because literally we're going to review at the leadership level.
Yeah. I think you have notes. I like how, because I feel like with AI, I want it to automate the stuff that I really don't want to do. And ironically, it's really good at automating stuff that we like to do sometimes, just coding. But writing an update, writing documentation, I don't think it's engineering.
We loved doing that. And so as an engineering, again, we're trying to reduce toil. If we find it reduces actual context, we will revert. I love the experimentation. And going back and forth.
As an engineering manager, what additional things might people use? Do people build their own tools to stay on top of things, to do people go deeper? Because of? Yeah. As an engineering manager and director, we try to surface again metrics so that you can see how your team is doing.
So for example, we have tools that allow you to see things like focus time of your team. How often are they in meetings? AI adoption? Are they using AI tools or not? We're trying to surface this information.
How many of your people are on a GSB project versus not? Because maybe a project ended and they're freed up to work on something else. So we try to surface that. And engineering managers, too, we try to get them to be a lot of the best ones come from IC land, like they started as ICs. So we try to get them to do if they're hired outside.
I started as an IC at Shopify before they move into management. Oh, here's a fun one that I think we should talk about. You mentioned something super interesting to me. When you're hiring engineering directors and above in the past, it was the usual interview. You know, a college service strategy, all that stuff.
You added a coding interview for every single engineering director and above hire. Can you tell me about this? Yeah, so it's interesting. It's maybe shocking to folks. And I know like Rodney sitting there, I guess they work for me.
I did the pairing interview with him. And so what happens is it is shocking for VPs, especially, to be like, whoa, does a coding interview? I'm like, yeah, because we believe that the best leaders here were ones who were not running away from coding. They just felt like they got better leverage from running a team. They still are deeply in love with technology.
And they still, and Rodney still goes on the weekend. And so it worked out super well. But our whole idea is that you're not running away. You're running towards technology. And just a better way for you to express it.
So I pair with the candidates. And they also see that even though I'm not running code every day, I'm still deep in the weeds of technology. I still love technology. I still want to talk about technical topics. And so we pair.
And some people believe that that's not the best for them. And there's lots of great companies out there where that's not the requirement. But at Shopify, we believe people should be as close to details as possible. And we are such a tech nerd company. Like again, our salespeople are by coding now.
We want our entity leaders to be coding as well. And so it doesn't mean coding today. But you should understand code and how code works. And a lot of it comes back. The muscle memory of coding will come back in these pairing interviews.
Yeah, and I guess especially now we have these coding tools when you can generate something as long as you can look through it. Like you can you write it all? This best part about copilots is that a candidate comes, the copilot generates tons of code. You're not running away from it. No.
One of the interesting stories that we just learned cursor has disbanded on the entry process. They're the AI tool company. Right. So you're embracing it. We're embracing it.
How's it working? Tell me. I love it. Because what happens now is the AI will sometimes generate pure garbage. So you're squishing it and you say, look at that every year.
And they're using, I say, copilot. Whatever. Here's what I'll say. If they don't use a copilot, they usually get creamed by someone who does. So they will have no choice but just a copilot.
Sometimes I will shadow an interview. And you do the questions myself. I've never seen them. They wear the copilot and send it to the interviewer and say, demographically that I can't abstract what they do to the Usually. I can't construct themselves.
I can't find the Oh yeah, there's a line. That's wrong. So I want to ask you, what does a standout senior staff engineer today look at Shopify and what has changed, let's say three years ago, when we didn't have any of these tools? I'm trying to prod here, like how you think AI or AI tool users might have changed on what we expect an amazing engineer.
And you think of the person who you think is like this, yeah, fantastic person. Well, I would say the thing that's changing is I'm seeing some of my engineers now actually use these AI tools to like do the infrastructure that they've always wanted to do, but never felt like they had the time. So like tech reduction, refactoring, making things easier to read. Like those are the things I'm seeing the best engineers that Shopify use AI tooling to actually do. And they just never felt like they had the time or the resources.
Like the engineer who also felt like, I wish I had like six months to do a code read. Now can use an AI refactoring tool, like a cloud code or an AI code X to like unleash these agents in parallel to then and then review all the PRs. But then unleash it and feel like they have a team themselves. So that's what I'm seeing. And they're not afraid of trying things that may or may not work.
They might unleash something for 24 hours and like a Devan like to one comes back and like under garbage. They deleted all but they said it was worth trying. Yeah. And that's a close the question. And a lot of people are thinking we'd love to be like I'm either an engineer, I'm either a staff engineer, director of engineering, et cetera.
I'd love to transform my culture to be a bit more AI-first. What would your advice be? How can people start? Like not everyone has a luxury of having to see you saying, okay, I love these things. Do you have tips that you get to your peers?
I know you're a CTO groups, for example. Yeah. So I think that the number one thing is role modeling is the best example. Like I haven't seen anything work better than role modeling. And the way that happens is like you have to do it.
So if you are coding and you are showing people your workflow and you are showing, you are in those same channels asking like, hey, I was trying to get this working and it wasn't working. Or I was able to get this workflow going by this prompt. We have a prompt library internally where you can literally grab prompt and say, hey, this prompt works for somebody. Let me try it and modify it for my use case. That's the number one way in which I see people trying to adopt AI is because they've seen other people do it.
And so we try to share AI use cases. We have, again, prompt libraries. We had a hackathon as a part of Shopify Summit a few weeks ago and we leaned heavily into AI. And it wasn't just junior people learning AI. It was senior people trying to embrace Cloud Code.
And one guy said, I haven't coded in six weeks. I've only used Cloud Code. And I pushed him and he said, okay, I have to make changes. Like, here, and that's a cool. You're 95% AI, but that's the point.
You're not supposed to be 100. And role modeling is another one thing I've seen. That and sharing examples. So role modeling and sharing. Especially because no one has figured it out.
So we don't learn. No one has exactly. No one knows. No one's there yet. Well, this is wonderful.
I love how much you're experimenting. I love how you are not afraid to take some churn to learn more. And I love how much you're sharing internally. I think we got a little window of this, but I think it's just something I hope more and more people and more companies will do. So this was wonderful.
Thank you so much. Thanks for having me. I found it refreshing to see just how practical Shopify is about LLMs. It feels to me that they understand that you need to invest and experiment to get results. One surprising thing for me was understanding my mind towards AI tools and could come up with clever use cases for it.
For more details on Shopify operates, check out DeepDies in the Primatic Engineer link in the shown notes below. If you've enjoyed this podcast, please do subscribe in your favorite podcast platform and on YouTube. This helps more people discover the podcast. And a special thank you if you leave a rating. Thanks and see you in the next one.