Traditional package management systems for JavaScript have faced several inefficiencies related to dependency storage, resolution, and project performance. PNPM is a fast, disk-efficient package manager for JavaScript and TypeScript projects, serving as an alternative to NPM and Yarn. Due to its efficiency and reliability, PNPM is increasingly popular for managing monorepos and large-scale applications. Zoltan Kokan is a full-stack web developer and the creator of PNPM. He joins the show with Josh Goldberg to talk about his background and package management in the web ecosystem.
This episode is hosted by Josh Goldberg, an independent full-time open-source developer. Josh works on projects in the TypeScript ecosystem, most notably TypeScript ES Slint, the tooling that enables ES Slint and Prettier to run on TypeScript code. Josh is also the author of the O'Reilly Learning TypeScript book, a Microsoft MVP for developer technologies, and a live code streamer on Twitch. Find Josh on Blue Sky, Mastodon, Twitter, Twitch, YouTube, and .com as Joshua K. Goldberg.
Zoltan, welcome to Software Engineering Daily. Hi. Hi. I'm happy to be here. As a consistent and now longtime user of PNPM, I'm excited to talk to you. We've got a lot of great stuff to talk about, like architectures and modern repo management. Before we get into all that, though, how did you get into coding? I started coding at school, like at 2004. I didn't have a computer back then, or maybe it was a little bit earlier.
So I kind of had the whole journey from the invention of internet to the invention of AI. And the first couple of years, I was called in without internet. I got internet only in 2008. So I was using just old school books and pirate content, I guess, in Ukraine. Actually, early on, I was planning to be a painter. I was doing a lot of drawing.
I didn't want to join the university or institute it was for painters. But I missed entry days. I mixed up the dates. So accidentally, I failed that gym. And later on, I entered the math faculty to become an engineer. That's, I guess, most of it. I started with Pascal.
at school. And then I used C Sharp and I used C Sharp at my first job. So you're saying there's an alternate universe where instead of creating PNPM, you became an artist, a painter. Yeah, but I don't think I would be very successful at that. I was always painting like real stuff, you know, not some imaginative abstract stuff, which I think is more exciting for a painter. So you got your first job in C Sharp. What was that? So the first job was at an outsource company. It's called Swan.
And I have a little office in my city, Ushgorod, which is in Ukraine. And I had different projects for American companies. I started as a junior programmer there, and I was only doing like... You need tests, which is interesting because people didn't use like test geo development. People were just coding and I was writing the tests for them, which now seems weird, but that's how it worked. And I was so happy when after a few months, I started doing real tasks that are not related to tests. So at that company, I worked almost two years and then I joined JustAnswer, which is a Q&A website where verified experts answer questions. And there I worked as a full stack developer on a .NET stack.
four years maybe it's just answer i started really to look into node.js and frontend and that's where when i found pnpm i have to ask as someone who was in.net before javascript how did you find that transition going from the strongly type strong foundation of c sharp to the at the time even more so wild and wacky and loose world of node.js so there are pros and cons what i really loved about javascript that's before i used typescript is that it's so faster to prototype and to get the working little CLI or tool. For scripting, it's so much easier to use JavaScript because you don't need to deal with all that boilerplate with C-sharp. Even to write a Hello World CLI app with C-sharp, you need to declare a class, declare a function, and stuff like that. Maybe it changed since then, I don't know, because now there is more than 10 years that have passed since then, and I know that C-sharp has improved a lot. Now you can even use C-sharp on all platforms, stuff like that.
But that was my experience back then. And also, you don't need all that IDE, which is actually really expensive to buy a Visual Studio license. You can just use it as an editor for JavaScript. But maybe with C Sharp, it's also possible, but it's not so common. Yeah, these days you can use VS Code for C Sharp, but it is not quite the primary path.
Yeah, it is kind of amazing to think of the trajectory that the .NET area and ethos has gone through. TypeScript recently announced that they are rewriting in Go, which at that time, 10 years ago, would have been completely unimaginable. Microsoft using a Google-oriented or sponsored language for TypeScript development for anything other than .NET. It's an amazing change. But okay, moving on. Node.js. So Node.js typically shifts with NPM, which is their package manager, the Node package manager, as it occasionally is called. Why would you want to create something like PNPM, a different package manager than the default?
how it happened actually at just answer we had a huge monorepo it's about okay it's not a huge monorepo by today's standards but at that time it was like 140 components and the way it worked there was a simple script which in a for loop ran npm install in each directory and maybe they were linked together so sim links or not i don't remember and we had like a sinopia which is the old name of berdacho running in san francisco and we were using it from ukraine This installation was terribly slow. In San Francisco, it was faster. Maybe 10, 15 minutes, I don't remember. But in Ukraine, it was 30 minutes. Just run and install in this monorepo. And also, it used a lot of disk space. So, like, dozens of gigabytes. Because of all these duplications in non-use folders, like, 100 times. I'm not sure how I found the NPM.
But I found it, and I tried it instead of NPM in this monorepo, and it immediately improved speed drastically. So back then, NPM was a lot slower. Not everyone remembers it, because it was like in 2015-16. It was NPM 3. It was before YARN came out.
And even on small projects, NPM was slower than the current NPM, probably 10 times. The current NPM is super fast in comparison. But PNPM was really revolutionary at that time. And I decided to look into it. And at that time, PNPM was not really maintained. It was created by Rico Stack Cruise. I think he's from South Asia. Now he lives in Australia, I think. And it was like a fork or like a clone, I don't know, of another package manager, IED.
created by Alexander Google. And the reason these package managers were created was because in NPM3, the structure of NodeMole use has changed. If you remember, I don't know. In NPM2, NodeMole use used like a nested structure. So every dependency had its own dependencies in a subdirectory inside the NodeMole use in the subdirectory. So you could have really deep tree structure in a NodeMole use. And on Windows, this was causing constant issues because on Windows, there's a limit to the file path. And also the other problem with this structure was that dependencies were duplicated many times inside NodeMole use.
So they wanted to fix these issues. And in NPM3, they changed the NodeModule's layout to a flat layout, where now dependencies are hoisted to the root. And as an alternative to this approach, this IED package manager was developed by Alexander Google. And the idea was to use Simlinks to avoid both nesting real directories and to avoid hoisting everything to the root. So that's the main reason why PNPM was created. You used a few terms there. There's an old joke among developers, a famous cartoon on the internet about how in increasing order of size, there's the planets, the stars, a black hole, and then no modules. So it's nice to see so much work has gone into reducing that size and complexity.
What exactly does hoisting mean in this context? And how does that change or not change how Node.js resolves packages? So by hoisting, we mean that every dependency in the dependency graph is moved to the top of your NodeModules directory. So even if you only have like Webpack in your dependencies, Webpack itself has like hundreds maybe of dependencies of its own. And if you install Webpack with npm and you open your NodeModules directory, you'll see hundreds of directories there. Even though in your packages on, you see only one dependency, which is Webpack.
So this is a simple explanation of hoisting. If in the dependency graph, there are multiple versions of a package, then this hoisting algorithm will pick one of these versions and put it to the root of node modules. And the other ones will be nested in one of the dependency sub node modules directory. Okay. And then you also mentioned symlinks. Now, as a user of PNPM, what we experience is that you can install pretty darn quickly. And oftentimes, if it's the same installation you've done on a previous run on your device, you can just do it offline or even on an airplane. But how does symlinking play into all of this? And what even is symlinking?
I'm not sure the SimLinks have a role in the speed. It just allows us to use an alternative layout for structuring NodeModules. Actually, you also asked about Node.js resolution in the previous question, but I did not answer, so I can mention it now. So the way Node.js resolves dependencies is that it searches for NodeModules directories in the current directory of the JS file that has the required statement, and then also looks in every parent directory as well, and then it finds this dependency. it results it. And an interesting, important point here is that for this resolution, it uses the real location of the module, not the Simlink location. So it means that we can...
have a symlink at one place, but we can have all the dependencies at a different place where the symlink points. So with PNPM, if you run PNPM install with Webpack, for instance, you'll see only Webpack in the root of the NodeModules directory, and it will be actually a symlink. And you also see a hidden directory .PNPM. And if you open up the .PNPM directory, you'll see lots of directories there, like in the form of a package name, add, and version. So this is a flat directory structure.
even flatter than the npm one with the hoisted node modules because it has the version also of the package. So the webpack in the root of node modules points to a directory in this .pnpm directory. And if you open any directory inside this .pnpm, you'll see another node module there, and you'll see all the direct dependencies of each dependency. So if you open webpack in .pnpm, you'll see synlinks to the direct dependencies of webpack. And because when you require webpack from your project, Node.js will resolve the seam link to that hidden directory inside .pnpm, and it will actually look for dependencies of Webpack in that isolated directory inside .pnpm. That's fascinating. Yeah, it's really, I think, a great example of doing less work to achieve just a really clean solution to a common problem. And as a result, to users, it just feels seamless. Like, you run pnpm install, and it feels the same as an npm installation, just faster. But under the hood, it is kind of shockingly different. Yeah, the speed difference is actually achieved by something else. It was also a part of the initial pnpm version when I found it.
So if you open the PNPM website and go to, I think, the motivation page, you'll see there a nice illustration of what makes PNPM so performant. So instead of having separate installation stages, where in the first stage, all the dependencies are resolved, then a separate stage when all dependencies are fetched, and the third stage when all dependencies are written to the NodeModules directory, in PNPM, there are separate independent pipelines for each package. So one of the package will be resolved, fetched, and written to NodeModules at the same time. when others are still being resolved independently at different stages of the installation. Actually, even Yarn has these different installation stages, but still, it's fast enough, I think. Before we move on to the higher-level features like workspaces, it would be great to get your opinions on the current state of package managers. As you noted, Node and MPM themselves have changed a lot, and also Yarn has had quite a few versions. Do you see any particular pros, cons, feature comparisons between the current versions of those three package managers? So I believe currently...
If you check, PNPM is most popular for using in Monorepos. I don't know actually how good are the other package managers at Monorepos. I really stopped comparing at some point. But people say that it currently has the best feature set for Monorepos. You can see it also on our website on the workspace page. At the end of the page, we have usage examples, I think. And there are some big names there.
Like GitHub projects with thousands of stars like Nuxt or Vite, some well-known projects use PNPM for their manual repos. There are some features that are available only in one of the three package managers. Like Yarn has the plug and play feature, which we kind of have also, but it's not really well maintained. So we use Yarn's libraries for that. So if you need plug and play, I would recommend using Yarn. PNPM has some features that others don't have. Like recently we have added configurational dependencies, which is basically a new type of dependencies that run before everything else. And you can load settings from there or use it for plugins.
It's really new, so I think it will be very powerful, but it needs more work to be done. But I think Yarn also has some plugin system. I don't know how it works, but I think they have core plugins, which are installed by default, and they have maybe some community plugins. In PNPM, we have hooks. In PNPM File CJS, you can hook into different parts of PNPM. It's always lovely talking to open source maintainers. Very rarely does anyone who's not an open source maintainer in one answer mention things like, we use one of our direct equivalent packages or projects as a dependency, such as you do with Yarn.
Or for this particular feature, although we have it, you should go use our other equivalent project for certain scenarios. It's really lovely to see that kind of cooperation in the ecosystem. Yeah, well, I've worked on PNPM for like nine years, maybe. So many things have changed since then. But Yarn's core contributor was Myl. He has also worked on Yarn for six years, maybe, or more. So we have talked a lot. And we had, you know, when there is some hate in the community, you can empathize with that. Like, I wouldn't talk about Yarn. I know how much work goes there.
Yeah, there is an unfortunate amount of community virtual, but you're all working towards a better future. And it's really lovely to see. And speaking of that future, let's talk about some of the features of PNPM that you mentioned. So just to take what you said about hooks or the configurational dependencies as an example, why would I want, let's say, a configurational dependency if I'm managing my team's package or projects? Yeah, I think I was a little bit inspired with Bit for configurational dependencies. I worked for Bit for like three years, I think, or more, three, four years. If you haven't heard about Bit, it's like a GitHub for component-based development. It has its own version control system, package manager, like hosting, CI, everything to make component-based development easier.
What makes it interesting is that you don't have to choose between a monorepo and a single package repo because you have dynamic workspaces where you can fetch only those components which you need for a particular feature. So it solves the scaling issue of monorepo, but has all the pros of the monorepo. And one of the ways that Bit makes it possible is that it has this environment for components, and the environment has all the necessary configuration, like not only the dev dependencies, but also any rules for the linter, for the bundler. Think about it like all the .files that you commit to your repo. So this is one of the disadvantages of a multi-repo approach, that you need somehow to synchronize all these .files. But with Bit, you have your environment component, and this way...
you can share all your conflicts by using the environment. I wanted to make it easier with PNPM also. So to be able to publish a package that will contain all your settings, for instance, maybe you use some patches to fix dependencies. If you haven't heard about it, you can use patch files with PNPM. It's also supported in Yarn. And in NPM, you can use a package for that, patch packages, I think. So you can, if you have some fix...
for maybe an unmaintained dependency in your node modules. And it's very deep in your dependency graph, so you can't just fork it. I mean, actually, you can fork it and use an override. But with Apache, it's even easier because you don't need to publish your fork to the registry. So you generate a patch file, and you commit it to your repository, and you tell PNPM to use this patch file.
for the dependency. But if you have many repositories and each repository uses this dependency, you'd have to copy the patch to each repository and make the configuration changes in each repository. With configurational dependencies, you can add this patch file to the configurational dependency and then just use the configurational dependency in each of your repositories and it makes it easier. Also, you can, for instance, have some standard overrides in configurational dependencies or you can have the list of package names that you want to hoist to the root of NodeModules because Sometimes we need that to spend time.
We also have a feature which other package managers don't have currently, which is called catalogs. It's a way to centralize, to synchronize versions of dependencies in a monorepo. So you have these catalogs declared in the PNPM workspace Yarn file at the root of the monorepo. And then inside your projects, you can just put catalog colon in the version field instead of the version. This way, it's easier to control each project in the workspace. We'll use the same versions, which is otherwise possible only using some LinkedIn. I think Yarn has...
constraints it's got, which allows to create LinkedIn rules for this. And there is a third party package for For this, I should think called link sync. But with this catalogs, it's a bit nicer because it's centralized in one place. And it's nicer to see on the change in one file, not in many packages on files throughout the repository. For the future, that we could move these catalogs to these configurational dependencies. And maybe companies could have like catalogs of recommended versions throughout their repositories, stuff like that. Or even like an allow list of dependencies. Maybe I know that some companies have limits to what packages they can use for their code base.
So it would be easier to like... Yeah, there's one developer behavior that's well understood in the web ecosystem. It's that people prefer and enjoy configuration files and avoid and hate linting. So it's nice to see that being moved to a first class supported feature. Bitback brought up your company and we should talk a little bit about it because it's fascinating and also very relevant to this area of repo and component management. So you work at BitDev and BitDev does use things like PNPM. But why would, let's say I have a monorepo with a design system and a bunch of apps and some frontends. Why would I want to use something like BitDev or its Harmony platform to be able to manage all those components and frontends and so on?
From my point of view, the main advantage is actually this possibility to create your own workspace. So your company won't have to decide whether to use multi-package repositories or to use single package repositories or have multiple monorepositories for different things and then somehow link them together. With bits, every component is independent. Actually, you can think about it as a single package repository. So each component has its own version control system. But unlike with a Git-hosted approach, you don't have to...
Like Bits creates a much better user experience, even like in the UI, the website, Bits Cloud. So there is Bits Cloud and Bits Dev. Bits Dev is, I think, mostly about the open source aspects of Bits because you can actually use Bits for free and host it on servers because Bits is a... an open source project but then it has bit cloud which has additional proprietary code and paid features but like with bit like the code you use like when you make changes in multiple components in the pr you will see all the changes to all the components in one place so that's actually not possible currently with github maybe some other companies allow it i don't know and it's very well optimized currently for ui components too so it's like it's like has the how it's discussed project called for UI Library, where you can visually see the components and even click on them. Storybook, yeah. So it's kind of like a mixture of Storybook and GitHub, I think, visually. So the problem with Monorepos is that you can't scale it. No matter what you come up with, at some point it will break. You can scale it to that amount of components that you will need in today's world. I know there are like Turbo and NX.
which improves things a lot for monorepos. But still, at some point, it can't handle that amount. So I think it is a really unique approach because nobody else tries to solve it this way. Nobody wants to create a new version control system and a new GitHub. So yeah, just check it out, I guess. Yeah, creating a new Git does sound rather daunting. There have been some things built on top of Git parentheses as a service that have sprung up, but it's nice to see the innovation here. We don't have too much time left for questions, but real quick towards the end, I do want to ask you, you've talked about monorepos versus multirepos or single. You've talked about...
different forms of package management, why and how newer innovations in PNPM and ARN have happened. For the next, say, five years, are there any more upcoming trends or new areas you're excited about being innovated in? Well, the CEO of Entropic said that developers will not be needed in one year, so I don't know. I think that are actually going to happen, I should clarify. To be honest, the problem with a popular project is that you have so much stuff to do that you can't really make a long term. Like, you can't think about big stuff. We had many, actually. Like, you know, BAN came out, right? There is also Orogin.
which is created by Kat Marchand. I hope I pronounced her name correctly, but she was a core contributor to NPM CLI. There is also Vault, which is created by ex-NPM maintainers and founders. So sometimes we look at these competitors and think about what could we do differently. So for instance, we spent a lot of time on thinking about rewriting PNPM in Rust or some other language because a ban came out and it claimed that it's...
10 times faster than PNPM. So to be honest, for a long time, I thought that maybe the long-term plan would be to rewrite PNPM, right, in Rust. But then we created a proof of concept, and it turned out that the Rust rewrite didn't make it faster. Maybe we just failed, I don't know. But it looked like most of the time is spent on network requests, which are not influenced by the text that we use.
So I think now my big idea for the next few years would be to create some registry integration because we have, I think, optimized almost everything we could on the CLI part. But if we would have our own registry, for instance, I don't know if we would do it. Or we would use some existing registry. But if we could, for instance, move resolution to the registry side, it could make installation faster. Or use some different archiving algorithms for the tarballs that are faster. Or maybe we could fetch not the whole packages, but just the files that are required from code. So I think it would be interesting to look at the registry side. Could be also a way to get some additional funds if we would have like...
our registry. But the competition is really big currently because I know there is JSR from Dino and Vault is coming out and something else. I don't know. Maybe Bambi will have its own registry. I don't know.
very wide in their plans. Would be unsurprising. We have at Software Engineering Daily, an episode from a few months back into 2024, an interview with Luca from Dino that touches on how, yes, once you push a lot of these concerns into the registry, as you said, you can do a lot of optimizations, but then the big focus is TypeScript supports and documentation generation. That'd be fascinating. Sultan, I have one last question for you before we wrap up. What is your take on spicy food and where have you had your favorite pieces of hot food? Thanks. Yeah, I do like spicy food. I'm Hungarian and in Hungary, there is a famous soup, goulash, which should be consumed spicy.
And there are some sauces, a Hungarian sauce called eryus pisto. But then I discovered Asian food, which is even more spicy than Mexican food. And I really, really love Indian food. I have many friends in India from work, from my previous work. For one week, I visited Vengaluru and there, yeah, it was heaven.
A heaven. Food heaven for me. Yeah, a lot of folks from North America, from the West, might not know that there are quite a lot of fantastic, very flavorful Eastern European foods, such as goulash. And it's lovely to hear that you had such a good time. Was this for a conference that you were in India? No, it was a workshop. Just answer has an office in India. So I was visiting colleagues. Well, great. That's all the time we have. One actual final question for you. If people wanted to find out more about you, your projects, anything you're working on, are there any particular locations you'd suggest they go on the internet? They can follow me on Blue Sky, on GitHub, on X.
Most of my activity is in the GitHub repository in PNPM. We also have a Discord for PNPM, which is quite active. Well, thank you so much, Sultan, for coming on. It was fantastic talking to you about the history and features of PNPM and package management and workspace and repo and component management. Looking forward to seeing all the great stuff you're doing with the new features like the dependencies and catalogs. For Software Engineering Daily, this is Josh Goldberg and Sultan Koshan. Cheers, all. Thanks for listening.