JSON API and API Design

I'm Yehuda Katz, and you're listening to The Changelog. Welcome back, everyone. This is The Changelog, and I'm your host, Adam Stacoviak. This is episode 189, and what a show it is.

The first show of the year, kicking it off with Yehuda Katz talking about JSON API. We talked about Yehuda's past, because this is the first time he's been on the show solo, so we had to dive deep with him. We talked about JSON API, JSON API compliance, resource discovery, supporting extensions, future-proofing API design, and so much more. We had three awesome sponsors, CodeShip, Toptal, and also DigitalOcean.

Our first sponsor of the show is CodeShip. Head to resources.codeship.com. Once again, resources.codeship.com. And there you'll find all sorts of free resources from CodeShip, e-books, webinars, email courses, guides, all talking about continuous delivery, Docker containers, getting your code to production in a workflow that makes sense for you and your team.

We've got a couple of guides that are awesome, why containers and Docker are the future, efficiency in development workflows. They have a couple of webinars coming up here in January this month in 2016, continuous delivery and feature flagging, and introduction to web apps with Docker. So those are really awesome free webinars you can join. They also have this really awesome five-day email crash course on continuous delivery.

Once again, head to resources.codeship.com. Enjoy all those free resources from CodeShip, and now on to the show. Hey everyone, we're here with Yehuda Katz. It is the first of the year, January 2016.

We're excited about that for one, but Jared, we're here talking to Yehuda today. And we've had this show, in particular, on the docket for a while, but this recent tweet kicked it off. What was that tweet about? Yeah, that was Yehuda tweeting about JSON API, which is our conversation we'll be about today, where he says that you can think of JSON API.org as ASM.JSON, a low-level, consistent, serialization format and protocol you can build on.

So I'm sure we will tee that one up later in the show as a conversation point, but that was kind of a thought that spurred us to say, hey, we never had him on to talk about JSON API back when it went. I think it was 1.0 was earlier this year. A few months ago, yeah. And so, here we are.

And we've had this, Yehuda, we've had this, so everybody, you've heard Yehuda back there in the wings there, but we've had this in our podcast trouble board for, I don't know, all year, basically, all of last year, technically, since we're actually recording this in 2015, published in 2016. So, you know. Hopefully we'll make it over the hump. There you go.

Well, we had Yehuda and Steve Klavnik on sometime last year talking about Rust, episode 151, which was, yeah, April. And I know at that time, we definitely had, right next to that one in the queue, was JSON API just waiting, so it's definitely been a while. Yeah, and Steve also works on JSON API, so, interestingly. So, obviously, that's the main topic of the show.

I'm sure we'll talk a little bit about Ember. We'll talk a little bit about how those two things play in. And I'm sure you've got some, we have a couple questions in the wings, too, about JSON API and Rails 5 and putting nice together and stuff like that, which I'm sure is what you can talk deeply about. But, something recently that we've done with our guests, especially someone like you, you've been on the show probably four or five times.

I don't even go back and count, honestly. Four times? So, we'll put all the links in the show notes to those past shows that Yehuda's been on. But, Yehuda, we haven't had you on the show by yourself.

So, I don't know how you feel about that. You always come with some sort of guest or some sort of wingman, so to speak. What do you think about that? Yeah, I mean, the reason for that is that, pretty much, I only like to work on open-source projects with other people, because, honestly, I just don't trust myself that much.

And so, there's always at least one other person who I'm like, well, whenever I say the other person will have another perspective that will be useful. And I think if you listen to me talking with almost any other person, there's usually valuable insights that I would have just missed. So, that's why I like to do that. It's just my style.

Well, Jared, should we kick off the show with the mention from Brian Liles? I know we've got a couple questions as well, but that was an interesting takeaway from Beyond Code up in GopherCon. Yeah, so I think we're going to spend a little time even getting to know you from a historical perspective. I think we all have our own interesting roots, and it's fun to find out more about the people behind the awesome open-source software that we all use and love.

We have a video series called Beyond Code, which we shoot at conference after parties. And season 3 was at GopherCon, which is still TBD on the release date on that one. But during that, we had Brian Liles on Beyond Code, and we do ask everybody in that series who their programming hero was. And he mentioned you, and so I actually pulled the quote out, because I think he gave some interesting reasons as to why, and he also seems to be a friend of yours because he kind of takes a couple cheap shots at the end there.

So I was going to read this to you and have you kind of just react to it. So he said, Yehuda Kast is his programming hero. So the reason why is because I remember when Yehuda first started writing code and he wasn't good. And then he started writing more code and he wasn't good.

But then he started writing even more code and now look at him. Not only has he mastered Ruby, he mastered JavaScript. Now he runs his own company and he's like this tall. And he kind of puts his hand up.

And he's kind of weird looking. That's not very short. But he's an amazing person to be around. So high praise from Brian along with a couple of cheap shots.

But can you just react to that? Yeah, I think that's all accurate. I am short, that is true. I think I'm a pretty good programmer these days.

I also was a very bad programmer at the beginning. Like literally every other human being who ever starts programming. And I also think that the process of persevering through struggling with code is the thing that made me better. And I think that I try to tell that to every new person I meet who's struggling.

Basically that's everyone's story. Everybody struggles with code. And if you think, oh, that person's cool, it must have been easy. That's just like, it's like a mythos.

It's not a real thing. I mean, maybe there are some people, but I don't know. That's a good dovetail right into that tweet stream we have mentioned there. And Yehuda, you probably know this because it was recent, but you mentioned the gulf between using programming to automate some repetitive tasks and being able to work on a loose kernel.

And he kind of dove a little bit deeper into struggle versus aptitude. Can you speak to those tweets? We have some links we'll put in the show notes, but I'm sure you can recall that. Yeah, I've been doing a few.

I actually haven't blogged a lot recently because I sort of came to the conclusion that almost anything I have to say could be constructed as a series of at most 10 tweets. There you go. Working on a character, too. Yeah, confirm.

So I guess what I was saying in that tweet stream is that I've met a lot of people over my career who have tried to program. Some people end up being programmers. Other people end up getting to have some aptitude with programming but not necessarily being, quote, unquote, programmers. And I think it's pretty damaging that we give off this sense of programming as a wizardry level thing.

Like you have to be a wizard to be useful at all. And I often like to tell people, like, so many people are very productive in Excel, and Excel is a very simple programming language. Obviously, it's very simple, but there are functions and variables. And some of the more abstract concepts that you need to do to be a programmer are already present.

There's like some amount of variables and things like that. And I like to think that if you could program Excel, that's probably not the end of the story. You could probably do a little bit of scripting. And one of the things I like to think about is one of the first jobs that Leah had, who I work with now at Zelda, and I married her.

I met her at college. But one of the first jobs that she did was a marketing job at a company. And she was basically told, like every single day you're going to get an email from one automated system, and what you're going to do with it is you're going to open it up in Excel, make some changes to it, and then email it to another automated system. She came home and told me this story.

And I said, first of all, can you find out, like, why the person in the first automated system can't just email the second automated system and change the format? And so she came back, like, the next day, and she said, I asked them, and they told me, like, they can do it, but it will be, like, six months to schedule it or whatever That rock hoop that was painted on the other side of the screen, and like my family would play it. So in retrospect, the fact that I was using GOTO and like effectively canvas primitives to draw a animated graphic game when I was eight seems like probably a good indication that I knew how to program. But at the time, I was like, okay, well, QBASIC has first of all, it's interpreted, so I didn't know that, but I knew you have to hit F5, and it was annoying.

I didn't know how to make the game not require that. Although eventually, I discovered that there was like a QBASIC compiler, but I didn't really know how to reliably have access to it. And I was like, oh, I wanna learn more things. This seems like a it's like pretty limited.

And somebody gave me a book on C, which is like, oh, here's the next step. After QBASIC, what you should do is learn C. And I was like, I literally have no idea anything that's going on here. But what I've been told is like, QBASIC is a toy programming language, C is like a real programming language.

And obviously, since I have no idea what's going on with the C book, which I think the book I was given was the K&R C book, which pretty much is not inappropriate book for a small child. I basically just came to the conclusion in my head that like, oh, I can, I'm good enough to like program toy things, but I basically do not have the aptitude to program anything serious. There was no like good bridge between the stuff that I already knew how to program and quote unquote real programming. So at that point, I gave up programming for like a while.

And then I went back, I would say like in my teens, I programmed again, and this is like embarrassing, but the thing I was programming was a StarDate calculator for Star Trek. I found online, I was online pretty early. I found online like a fact that someone had done all the work to reverse engineer all the StarDate algorithm for every single era of Star Trek. And I was like, oh, that seems cool.

Let me like make software out of it. And I used Visual Basic 6, which a lot of people still have fond memories of. And Visual Basic 6 is basically like a WYSIWYG programming environment, but you could like double click on any part of the, any UI elements, and then you could say, effectively pass an event handler to the, to that UI element by just visually, right? So you don't have to write the code to see what it does, but you don't have to do the work of like wiring up the event handler to the visual element and the actual visual work is done, is done using a UI, right?

So you could sort of think of it as Interface Builder, but in Interface Builder, you still have to do, like write the code in the text part, and then you'll have to like, draw, drag the arrow, drag the line from the UI element to the text part, whereas in Visual Basic, you would write, do the UI part, and then instead of having to like, also write the test part, you would just double click on it, and it would effectively scaffold out the, the part that Interface Builder makes you do manually. So it was, it was very, I would say it was very easy to use. And I was able to like, the algorithm, as you can imagine, is not very complicated, so I was able to write it in, in Visual Basic. And at that point, it felt pretty similar to my GW, to my QBASIC time, right?

It was like, it's pretty clear that it's like a toy environment, but um, now the toy environment has gotten better, so I can do, like, I can basically get more done with the amount of work that I already, with the simple programming that I already know how to do. But then I equivalently, I eventually was like, this seems pretty annoying. I wanted to do more things. And what I was told at the time was like, well, the next step is to learn the Win32 API.

And I went to look at the Win32 API, and if anybody wants to look at it, it's horrifically terrible. And well, it's not, it's not terrible. It's just like, equivalently very low level, right? So you get similar to your C, you know, to the experience you had when you were 8, C compared to GW.

Exactly. And the, and the Win32 API is like, it's basically like you have this top level switch statement and every action that happens is basically running, like telling that switch statement what to do. And it has to do low level things like drawing boxes. And it's just, it's basically a massive loop between uh like the nice environment Visual Basic and the quote unquote serious programming that you do with Win32.

So again, when in my teens, I was like, well, that's, that was fun and all, but it doesn't really, I don't seem to have the aptitude and I'm no longer eight years old. Basically the same thing is happening, so I'm pretty sure I should just should not do this. So what I guess when did the professional time happen for you then? Like, was it a struggle for you?

I know you talked about struggle and aptitude there and you said that uh the idea of struggle doesn't mean that you don't have the aptitude. It's, you know, anybody who's learning the program is gonna have some sort of struggle. That's a good thing. Yep.

So what was the, how did things progress for you to get into professional, I guess, programming? So what basically happened is I just gave up. I was like, okay, I, I like basically know the equivalent of like very, very light scripting, but I can't really program. So I, and in college I could have been a CS major, but I was like, doesn't really seem like it's for me.

And I ended up doing a bunch of other stuff, um, accounting and journalism. Um, but uh journalism in college ended up teaching me design. So I learned, I basically did a newspaper design in college. And then I got my first job out of college as a web designer from a person who didn't really understand that web design and print design are not the same thing.

Um, and I also didn't know that at the time, as it turns out. So I was basically given a job to do what I thought was a like pretty straightforward design task that ended up actually being a programming job. And I was like, oh shit, if I want to keep my job, I actually need to learn programming. So I had no choice.

And I was the reason I ended up succeeding was because I was given a couple of applications that people had already written, um, in ColdFusion and PHP. And I was told, like, please update this application for this year. So it was a nonprofit and the nonprofit had like these dinners and they were like, okay, we will, we basically have an application that lets you pay for your dinner ticket and perhaps ask for additional items. It was like not totally trivial.

And they were like, please update it for the changes that we want to make for this year. So, of course, I was able to go in there and either in ColdFusion or PHP, I was able to make the changes because that kind of thing is not so complicated. But I often was, when I was doing that, I was like, oh, well, it seems annoying that you have this giant form and like every time you click next, you have to go back to the server. So maybe I could find a way to make it more interactive.

Uh, again, I was like thinking about it as a designer. And I was like, this was like right around the time AJAX had become a thing. I was like, oh, there's this AJAX thing. I should probably learn that.

Maybe that will help me do it. I was like very enamored at the time of like the idea of um constructing forms that progressively revealed themselves based on the previous information. And the thing that I really was most angry about that still annoys me is the fact that um you would have to put in your city, state and zip code a lot of times. And I was like, if you put in your zip code, you can derive the city and state.

Why are you making me type all these three things? And I didn't really know how to do it, but I was like, I'm pretty confident that this is not actually necessary. Was that the first library you wrote then? Um, I wouldn't say that it was a library, but I figured out how to like download.

At the time there was no libraries really for any of the languages I was writing, but I was able to like download an access database of the city, state and zip. And I was able to figure out how to get it to reveal its secrets to me, um, and not force me to do the thing. And I learned AJAX, so I didn't have to reload the page and things like that. So that was sort of how I got started programming.

Um, but then I was like, well, I can't, like, learning AJAX, quote unquote AJAX all by myself seems very hard. I'm the internet was not so good at the time. There was um, Experts Exchange was like the most popular thing and it was like constantly asking you to pay. I just didn't feel like I was getting a lot of traction.

So I took this class by Thomas Fuchs called, uh, which was like about AJAX. And I effectively, by the time I took that course, I knew most of the things that was in the course. It was actually a course on prototype. But at the very end of the course, to make it feel better is like small things that me 10 years ago could have done.

Right? And someone has to just do it. So one more thing we have to mention, obviously, we can go on and on. You have a deep past with programming open source, and obviously, we'd love to hear all about your past.

One thing I want to talk about before we go into our first break is what I'm calling, I think, might be your brand purpose, your personal brand purpose. And at November this year, I think it was this year or last year, I'm not sure which was, but you said sometime during your talk, and I put this note, I don't think you made explicit that this was your brand purpose, but I want to ask you what you think about this before we transition to this break. But you said, build things that empower people. And it seemed like, you know, rewinding back to your beginning through to your initial comfortable feeling of being a programmer and then moving into open source, like you've been doing, that that's been your mantra.

That's been what you do. What do you think about building things that empower people? I think that's, I would agree. I also agree that it was this year.

I also agree it was not so explicit as a, like me waving the flag around it. But I think that open source does a lot of things for a lot of people. And I think it's good. But for me, the main thing to do with open source is to find places where the gap between someone being empowered and not being empowered is small and just closing it.

And unfortunately, at this point in my career, the gaps that are still good cost benefit for me are usually much bigger, but they're still like, I'm never going to be, I mean, maybe someday, but right now I'm never going to like invent the Rust programming language because it's like the level of low levelness is too much for me. But the gap between like Rust being a good thing for Servo, like a web browser and being good for Ruby programmers is actually much smaller than the gap between like not having Rust in the world and having it at all. And so I, a lot of what I try to do in my, like with my open source work is to try to find things where either like the fact that something hasn't been invented yet is for a stupid reason, like Thor or Ember, or there's something that exists in the world, but it's just inaccessible for some reason like the jQuery doc story or Rust. I'm like thinking about, okay, what, what can we do to make this a thing that a lot of people, a lot more people can use to do something powerful.

And I think, I just think not enough people think about it. And more concerningly, and this was also in my November doc, a lot of people are just pretty cynical about the whole thing. A lot of people are pretty cynical about the level of complexity in programming. A lot of people are pretty cynical about how likely you are to find a good solution for these things.

A lot of people are pretty cynical about the power of like the power of these things in the first place to change what the, to change the nature of what people can do. Right. And I'm very optimistic about being able to change the nature of what a particular person can do at a particular time. And, and, but that requires believing in it.

And I think unfortunately a lot of people don't always believe in it. Well, I think on the other side of the break, we want to take these thoughts, the empowering people and the finding the gaps between empowerment and non-empowerment and use those to cast a light on your work with JSON API. So we'll ask you about that on the other side. We'll be right back.

We know you listen to other podcasts. Don't worry. We're not upset. We know the changelog isn't the only podcast in your list.

And you know what? You may have heard advertisements on other shows about other hiring platforms and other places like our friends at TopTowel. But let me tell you, there is no match to how invested TopTowel is to both sides of the equation. You have companies out there who need great engineers, great designers, and you have great designers and great engineers out there needing great opportunities.

And TopTowel plays both sides of that fence very well, helping make sure the right kind of opportunities are there and the right kind of developers and designers are there. And if you're a CTO listening to this or someone on a team who knows you need to expand and add more people to help reach your goals, TopTowel is the best place to go and find the best designers and the best engineers out there. And if you're one of those best engineers or best designers looking for great places to do great work and freelance and travel the world and have a lot of freedom, but also have the same kind of support you would want from working somewhere, TopTowel is that place. You can blog with them.

You can travel the world with them. They have meetups all around the world. They absolutely love to encourage and to support and help their developers and designers that are in their network all around the world grow and be better and do better. Head to toptal.com to find out more.

That's T-O-P-T-A-L.com. Once again, toptal.com. But if you want a personal introduction on either side of the fence, whether it's an introduction to find the best designers and developers, or if you're a designer or developer looking for great opportunities, get in touch with me. I'd be glad to give you a personal introduction.

Email me at adam at changelog.com. And now back to the show. All right, we are back talking with Yehuda Katz about JSON API. And before the break, we laid out Yehuda how you just try to do two things.

The main purpose is to empower people with open source, and kind of the way you go about that is by finding these gaps between empowerment and non-empowerment and hopefully finding ones that are easy to bridge. Probably they're getting harder and harder to bridge as you get more and more advanced. And we wanted to take those thoughts and apply them to your work behind JSON API because this is something you've been involved in since, I think, May of 2013. So can you cast that light into JSON API and why it's something that you think needs to exist?

Sure. So before I do that, I think it's important to understand what does empowerment actually mean? And one thing that I think is important to keep in mind is that if somebody already knows how to do something, making it easier for them to do that thing is not empowering. It might be nice.

It might help them get their job done faster. And there are some ways in which that kind of thing can be empowering. But the most empowering things are taking something that somebody does not know how to do at all and then making them be able to do it. And fundamentally, in order to do that, you have to use abstractions that hide some of the details.

Because pretty much nobody is going to say, like, oh, I would like to build a web application. So before I do that, let me learn HTTP and Rack and all this stuff, right? They want to, the shortest path to empowering them to be successful, like I was when I started building that CMS for my company, is to hide a lot of those details. And this is actually a fundamental conflict in open source and software zeitgeist, is whether abstracting away details is a good idea.

So, for example, Joel Spolsky has this famous blog post from many years ago called The Law of Leaky Abstractions. And the idea behind this blog post is, like, you can never successfully abstract anything. It will always leak. It will always leak.

So don't even bother. Everybody should just learn everything. I think this is just a silly worldview, in part because we have so, you know, over the past 30 to 50 years, we have succeeded at abstracting so many details that it just seems, like, self-evidently incorrect. Like, whether or not, you know, TCP, which was his example in his blog post, one of them was TCP, which I think is a hilarious example.

The fact that TCP might occasionally leak doesn't necessarily mean that TCP is a bad abstraction. It just means that sometimes if you're doing sufficiently advanced things, you will need to learn TCP to get the sufficiently advanced things done. But that doesn't mean that we should not have TCP and everyone should just use UDP and build their own thing. Although there's a good website called noTCP.io, which is basically making a joke about this entire phenomenon of people not believing in abstractions.

And I guess for me, if you're looking to empower somebody, you have to figure out what parts of the everyday, the 80-20 job that someone does, what parts of it are about things that are not very important. And interestingly, what I have also found, what I found is that when you do that analysis, quite often it's a lot of the thing, right? It's a lot of the stuff people do, like, on a day-to-day basis. If you look at, like, what someone does when they write a backbone application, a huge amount, a huge percentage of that is something that Ember doesn't make you think about.

Or even React or Angular doesn't make you think about. Right? And I think, so I think when you're first looking at empowering people, what you often discover is that sort of the natural zeitgeist, the natural worldview of how people think about whatever task it is that they're setting out to do and what people think is, like, core and fundamental to the task they're trying to do is really pretty abstractable. I think if you're going down this path, it's important to remember that when you abstract something, you disruption in some way, you're trying to figure out, like, okay, how do I make something that a lot of people will not respond negatively to?

And that basically means being pretty wishy-washy. You say, oh, you can do this or you can do that. If you don't like this, you can do this other thing. But the problem with that is that now the clients have to be willing to accept a lot of different things, which makes it hard to write the right clients.

And what we discovered somewhere in the middle of the JSON API journey is that no matter how much people claim that they're willing to adopt something because it has the right aesthetics, the real value prop of something like JSON API is having good servers and clients. So we were getting a lot of people to not whine a lot about the exact aesthetics of JSON API by being pretty wishy-washy, but then every person who was actually trying to write clients would say, you know, this spec seems fine, but I don't know what I'm supposed to do here. This server says you're allowed to use whatever status code you want here. Is that actually, like, what does that mean?

What if the user gives me like a 422? What should I do? Let me just real quick say something that's perhaps obvious to you and I, but just to be super clear with everybody, that the JSON API, and I'm using the word just to delimit its scope, not its usefulness, but it's JSON API, the project that you've been working on that we're talking about, it is just a specification of how you ought to build JSON APIs, correct? Yes, confirm.

Okay. So it's a specific, so just to be clear, it's the only specification that I know of that tells you how to build JSON APIs, both the format and the protocol. In other words, it tells you the HTTP semantics, but it also tells you what the shape of the inputs and outputs. And as far as I know, I'm probably wrong.

There's probably some other thing that exists in the world. But as far as I know, the popular alternatives that people say, why, you know, how or something like that, those things are all either about a protocol or they are about a format and not both. Okay. But yes, I agree in retrospect, probably the name JSON API has some confusing ambiguity here.

And I apologize in retrospect, in hindsight. It's very Googleable though. I mean, you can Google it and find it. So that's a good thing.

Yes, I mean, perhaps surprisingly, but it is true that it is very Googleable. So anyway, so we basically went through this back and forth where at first we were pretty responsive to the aesthetics people. And eventually we realized that the aesthetics people were not actually adopting JSON API, but the tool builders were important for the adoption of JSON API. And they were telling us about all these issues.

So about a year ago, me and Dan Gebhardt, who is probably the main, like the main engine now of this spec, sat down in a few face-to-face sessions. We hashed out what ambiguities people had reported that were making it hard to build tools. And we just made calls about all of them. We just said, the answer is this, the answer is that, the answer is X.

And the consequence of doing that is that now JSON API is very trivial from an aesthetics perspective. It's very easy for someone to look at it and say, I don't like that. A very common thing people say is, is this Wizzle all over again? Actually, it's not Wizzle because it does not have a big schema type thing and it has nothing to do with that.

But it's very easy to look at it and say, you know, I thought the point of JSON was to make this a very like human consumable format, but now you have like this attribute key that has a hash of attributes. Why do you need that? Well, it turns out that the reason you need that is that tools need to know the difference between attributes and associations. And saying that you'll do that by hard coding it in every client is not very good for tools, right?

So there are some things that we did to try to make ambiguities less. We tried very, very hard in the, when we made the calls to not gratuitously add things that, for tools that just, you know, that basically would make it a binary format or something like, like the ugly XML formats that you people are used to seeing. But at the end of the day, there really are some ambiguities that you would like to get out of the spec. And that's sort of why I tweeted that as in JSON thing, was that JSON API is not really fundamentally about being a, it's not about being a bespoke handrollable format.

It's about being, which I think is how a lot of people think about JSON APIs, right? They look at them, they try to construct the most optimal, most aesthetically pleasing shape of their JSON. Right. I think that the JSON, the API or the hand-rolled artisan versus, you know, XML and Wizzle and these things, I think it's this, it's this, it's kind of a yin and yang that we go back and forth between and similar to like static versus dynamic type languages where you want the freedom, but then you lose the tooling and the, you know, these conventions.

And I think it's, it's, it's tricky ground because everybody has opinions on it. And it seems like there's no like black or white answers. So I think the interesting thing for Rails is that DHH seems to really come down on the artisanal side on JSON. But I think I, as a programmer learned the right answer to this question by looking at ActiveRecord, which had a fairly similar question early on, where it basically said, okay, DHH has figured out that the way to produce the best programming experience is by not allowing you to choose whether or not your primary key is called ID and what the foreign keys are called and all this stuff, right?

We're going to tell you what the answer is. And maybe a DBA is going to look at that and say, oh, that's so disgusting. I can't believe the SQL that's being emitted. And you're supposed to say in Rails, I just don't care.

I'm not interested in that level of abstraction. And the story of JSON API is pretty similar, right? It says, what I want to do is I want to describe my models. And yes, there's going to be some impedance mismatches like there is with SQL.

But I mostly want to think at the model level of abstraction. I mostly want to think about the model and something like Ember is pretty similar to like the resource concept in Rails. It's not about database tables. I want to think about models.

And the fact that there's maybe, you know, maybe an extra attributes hash or extra URLs or maybe the fact that every single thing has to have a type and an ID, right? All those questions are just not, they certainly, you want to make sure that the format is debuggable. I think that's one of the things people don't like about more binary formats. It's like you look at it and you're like, as a human, I cannot figure out what this is even trying to say.

And I think that's very bad for debugging. But how many, you know, do I have a type associated with every single one of my records that I've sent over this format? Those are questions that are legitimate aesthetic questions, but they are analogous to, like, I would really not, I don't want ID to be my primary key name. They're just not very important.

And the more you hold on to saying that it's very important that you care about that, the more we push back, we leave open the big gap. So this is the empowerment thing, right? There are people who want to build stuff on the web and making them learn about handcrafting their JSON and HTTP status codes and all this stuff. Even me, when I'm trying to write apps, I don't want to be thinking about this stuff.

And the fact that this is just a part, the normal part of the work of building applications means that there's just a bunch of people who can't do it or can't do it at any particular point. It's a hard weekend project. It's a hard side project for a job, right? And basically the job of JSON API and my philosophy in general is to say, how much of that actually matters?

And in fact, that was also the point of Rails, right? Is to say, how much of the stuff that people are spending all their time on actually matters when it comes down to what your application is doing? Maybe, maybe performance really matters. Maybe you need to handcraft this thing and you can't use JSON API, but why don't we figure that out after you figure out if the thing you're building even matters?

Yeah. And I think that, you know, there are other aspects that you talk about aesthetics. You mentioned one performance. The other thing is like how well your client libraries, whether it's Ember or an iOS library or whatever it happens to be, how well that maps on top of, you know, what JSON API is speaking both in protocol and in format.

So has there been a lot of give and take between, you know, not just suiting Ember's needs, but suiting an iOS library, suiting React or these types of things? Yes, absolutely. So the evolution of JSON API was that it started off being pretty extractive from Ember data. And I agree with the HHH extracting these things is way better than trying to, like, invent them out of your head.

So it started off already being pretty useful, even just to like point people at as if you're trying to build an Ember data thing, it is now documented there. So that's where it started from. I mean, it was like sort of documented, but mostly in We standardized and we must know two interoperable implementations, which sort of implies that the implementations go first. But then, like somewhere in the, you know, somewhere after 2000, a bunch of standards were happening that was attempting to lead by standards.

And I think there's still some of that left, but I just don't think that works at all. And a big part of the pendulum swing that I've been part of in both the TC39 process and to whatever extent I've been involved in the W3C has been an attempt to remind the standards bodies that they should not try to lead. Because that's not what standards are about, right? Standards cannot, standards are about, you know, acquiring social consensus from implementers.

It's about having a nice space where you can work. For W3C, it's very important that the space is not patent encumbered, right? So any spec that's approved by the W3C, by definition, is a promise that all the participants in the W3C, all the members, say that they will not sue anybody for patents because of the technologies that are in the specs. That's pretty important, right?

So the standards bodies are useful, but they really do need to be facilitators, specifically facilitators and implementers and users and not try to have standards people do the work. That's a really good point. I mean, because that's what you're saying here, too, is that, you know, going back to Rails, it was extracted from Basecamp. This was extracted from what you were doing in Ember's, in EmberData.

So it seems like people who are actually doing, not so much that standards aren't doing, but I like your idea of how they're providing this safe environment to come to the consensus of what's out there, what's working, why is it working, who trusts it, why do they trust it, and bring it on to one thing that is under some sort of governance that's sort of community bound in a way. And I think, like, so there always are implementers that are participants. And, like, TC39 is, like, a ton of implementers. And the real thing that you get out of, like, the thing that you get out of that process is not that the implementers are sitting around in a room talking to each other about what to implement, but that the implementers understand that they can, that when they implement something or when they propose something, they have acquired the consensus necessary to feel confident that they can move forward with it.

And that process, like, some people think of that as being an inherently slow process. But first of all, I just don't agree. I think if you look at, like, the ES6 process or even the HTML5 process, we've made pretty good progress on features despite the need for consensus. But I think people forget that there's a massive time cost in not acquiring social consensus, right?

If you ship something a little faster, Firefox could ship something a little faster, but whatever gains you get from shipping something a little faster will be eaten up immediately by the cost of convincing the other browsers to implement it. True. And sometimes you have to go first, right? We always talk about this in TC39.

Sometimes someone has to go first. And usually what will happen is somebody will propose something and everyone will say, I'm not really sure what the value of that is, but I'm not necessarily opposed to it. I will implement it as it turns out. But why don't you go first?

And then usually the person will go first and quite often it will happen. They just draw straws or what? No, it's usually the people who are the most confident that it's a good idea. Well, that's a good place to take a pause.

We have one more break before the show comes to a close. However, don't worry. We have lots more to cover. Compliance, resource discovery, some experimental things, feature-proofing APIs, just to kind of touch on a couple of topics coming up.

But let's take a break. We'll come back and we'll dive deeper with Yehuda on JSON API. And hopefully you'll love it. So we'll be right back.

Our friends at DigitalOcean, simple cloud hosting built for developers, launched a new feature recently that we absolutely love. It's called Droplet Multi-Create for easily launching up to 10 servers at once. You can deploy multiple droplets when you create your droplets with the exact same configuration. And this is a huge feature.

We have to love it. Head to digitalocean.com. And when you sign up, use our code CHANGLE to get a $10 hosting credit when you sign up. All right, everybody, we're back.

And we are diving deep on JSON API. Yehuda, you've mentioned a few features offhand, but I thought we'd take a moment to kind of go through them one by one. The major principles that have shaken out as useful for a JSON API, what's in the spec, and what does a JSON API compliance server look like? Cool.

So, yeah, so I would say the main principle is what I said earlier, which is that what you're doing is you're serializing a graph of objects on your server. What I mean by graph is just that it's possible for one object to be related to multiple of them, which is like sort of like a has-many belongs-to-many relationship. Or you could even have, like, a, you know, a post has many comments, and the comments belongs to a user, but that user might be referenced from other places, right? So when I say graph, I just mean that people often imagine in their head that the data structure is a tree.

They imagine, like, well, there's a post that has many comments, but nothing inside there. And then the comments have users, so you just nest the users inside of there. But in the real world, those objects are related, could be related to other objects. So you don't necessarily want to continue nesting.

And this is actually a core value of JSON API, something that is sort of counterintuitive to a lot of people, but the trying to make it a tree has pretty serious implications on both efficiency and your ability to properly deserialize it. Like, what if you have the same user in multiple places and they don't match? What does that actually mean? Do you have to check that?

So one of the core values of JSON API is that we're talking about a graph of objects, not a tree of objects. And there's always, there is often, not always, but quite often a primary document. So if you ask for post number one, you want to know that you might get back a whole bunch of stuff. You might even get, like, a summary of how many documents there are in the entire system.

But there's a primary response, which is the post object, and then it might have links to comments. And those links to comments could either be referred to other things that are inside the same response. And that's commonly what people do. But they could also be URLs that link to other documents that you can use.

So maybe you're downloading a list of posts just because you want to print the front page of your blog and you don't need the comments until you actually click on a comment. You still want to say, like, if you ever read the comments, here's the URL for those comments. So the idea that we're talking about is a bunch of links, objects. But unlike maybe the more pure rest or the pure, like, linked data philosophy, the assumption is that you will quite often want to include a lot of the linked objects together.

And there should be a very canonical standard way to do that. And it shouldn't always have to be a URL. So one thing that's a little problematic about URLs, first of all, is that it implies that there's a way to get individual pieces. But maybe, for example, there's no way to get comment number 17.

The only way to ever get comment 17 is by going through the post. That's a totally reasonable thing to want to be allowed to do. But if you only link things through URLs, you have to construct a URL for that comment. And maybe, like, a very, you know, URL person would say, that's fine.

You can make a fake URL. But this stuff kind of adds up in complexity. And what we want to be able to do is just express connected data in ways that are most natural for your implementation. It doesn't mean the implementation leaks.

It just should mean that it's natural for the implementation that people have. So we have, you know, we have this post, it has many comments. Maybe the comments are directly embedded in the document. Maybe they're linked through URLs.

Maybe they're not linked through URLs. They're linked in some way. But there's very canonical ways of doing all these different ways of describing things. And they're composable.

And then if you, so all the ways of interacting with that data is done using normal HTTP, API, HTTP verbs. So you get something using the get request. And when you get something, you just might happen to get back other related things. If you want to add a new thing, you post to it.

And there's some rules for exactly how that works. There's also delete to delete something. And there's patch, which allows you to, there's sort of a historical sad ambiguity about what put exactly means. But patch very canonically means the right thing.

So you can patch to make some partial changes to an existing document. Patch and update to an existing document and posts like just replace the entire thing, right? So I will just tell you, having now spent many years writing the spec, that people feel a lot more confident about what put means than I think they should. But people do have religious opinions about what put means.

And because of the fact that you're interacting with infrastructure on the internet, you should probably not run afoul of people's religious opinions about what things mean. Because you're just avoiding I think it will usually work out. So one of the ways we kept up the show is actually kind of deriving where this came from. It started with a tweet from you, which was comparing JSONAPI to ASM.js.

You said you think of JSONAPI.org, which is obviously JSONAPI, as ASM.js, which is a low-level consistent serialization format and protocol that you could build on. Can you compare the two, like you said earlier about having a de facto standard? It seems like, obviously, this is what you're trying to do with it, but then you also have this. Are they competing standards?

What is the differences and do they play together? So ASM.js is this thing that Mozilla invented, Mozilla Research, which is basically just, they noticed that there's a subset of JavaScript that if you squint closely enough, is a fully typed thing that is sufficient to represent basically all C. And so they basically defined a subset of JavaScript that they will validate. You basically tell them that you're using it, and then they validate it, and then it will compile into native code.

It's like somewhere between 20% and 50% slower than just like straight up compiled C code. And the reason I compare it to that is that if you look at ASM.js, what you will see is that no human being wants to write this. It's actually significantly worse than JSONAPI. It's like a horrible format to write in.

But the point of ASM.js is that that's not the point. That's not the goal of ASM.js. The goal of ASM.js is to be fast and also to be a thing that you could build on. You could compile JSIP to ASM.js, and now you have a fast JZIP that you could run in the browser.

And so the idea is you shouldn't be too, people have said on Hacker News a lot of times, like, ASM.js is so ugly. It's like such a hack. Why do you know bytecode? And the point is that that doesn't really, the aesthetics of ASM.js doesn't actually matter.

The thing that matters is that we have now constructed a thing that you could write that ships on the web that is pretty close to being as fast as C. And that is pretty important and effectively supersedes any aesthetic consideration that you might have. And I was sort of trying to get across a similar thing about JSONAPI, that the goal of JSONAPI is not to allow you to build artisanal APIs. The goal of JSONAPI is to create a nice wire protocol and format for communicating graphs of objects between servers and arbitrary servers and clients.

And obviously there's some constraints there, like you should be able to do it on existing servers. So Rails, Express, Java should all be able to do it. You shouldn't have to build your own homebrew server with homebrew server infrastructure. You shouldn't require things like WebSockets to work, right?

So it should work with regular HTTP infrastructure people have already deployed. So those things are all very important. But the exact, whether it's 20% better or worse than the aesthetics that you would have personally designed for a given project is actually just much, much less important than the interoperability. And so that's sort of what I was getting at, was that it's a look.

It's more of a low-level protocol. It's a way of getting everyone on the same page about something that you can share and build on than it is about being a... And again, it needs to not be a format that's impossible for humans to understand. It needs to be a format that a human can look at.

But whether or not there's a top-level attributes key or you leave it off and put all the attributes at the top level is really just not something that needs to be a reason for non-interoperability. That is a bad excuse for non-interoperability. So one last question before we close up the conversation around JSONAPI is, I think 2015 is kind of in the year of Facebook and Netflix kind of launching out these new API, I call them transport patterns or styles. I'm not sure exactly what you call them, but we have GraphQL, we have Falcor, and these seem to be taking a little bit different approach than the traditional REST JSON-based APIs.

Can you compare and contrast, kind of advantage, disadvantage of the JSONAPI style versus these new styles? Yeah, but I think I've been sort of getting at it, but I'll be very explicit. I think that what GraphQL and Falcor are both getting at is the fact that very purist ideological REST is pretty hostile to bulk transport, and it's pretty hostile to customized transport. People don't want you to say, give me this resource, but don't include these attributes.

And they don't want you to say, give me this resource and include these other related resources. In fact, very purist REST people always talk about HTTP2 as allowing you to just continue down the path of having one URL per resource and just firehose them with one HTTP socket, but now multiple requests. And JSONAPI and Falcor and GraphQL all share a philosophy that a client should be able to, you should be able to construct payloads that are appropriate for the access patterns that your client actually has. If you're about to look at a blog, and you know that people are likely to click on a comment, you should be allowed to include all the comments at once.

Or certainly you should be allowed to include the count of comments per post so that you could put in the UI. GraphQL is on a pretty one far end of the spectrum, which is that GraphQL, the idea behind GraphQL is that you derive the exact thing that you need out of the consumption patterns. So you say, I need a post. So every component basically says what it wants.

It says, I want a post. The other guy says, oh, I want a comment. I need the comment count. And the GraphQL library takes all those things, combines it into one request, and it asks for very precisely what the client has asked for.

And Falcor, I would say, is also pretty similar to that, although Falcor is a more flexible... I don't know, people, whoever's listening to me who works on these projects is going to yell at me no matter what, but I would say that both Falcor and GraphQL are designed for consumption patterns. They're designed for building a request for exactly the thing that you need for what the client is asking for. And I, first of all, one of the reasons JSONAPI did not go down this path is that it requires a server that's capable of pretty sophisticated querying.

And I think, again, JSONAPI has as a goal that you should be able to use existing server infrastructure. But perhaps more importantly, that kind of stuff is pretty uncacheable. So if you ask for a very specific subset of the data, it's very hard to say, oh, I basically know what you're asking for. You're asking for all the posts.

No problem. I already cached the whole thing. It's very hard to do that when you're asking for very specific pieces. It's also not really a good fit for what the GraphQL people pejoratively call overfetching, and I positively call overfetching.

Sometimes, like, you have to think about when you're building an application, if you're showing a particular page, what are the likely next things the user is going to do? And if it's very likely that within a few seconds or half a minute, let's say 10 or 15 seconds, the user is going to click on something, it's actually a reasonable thing to give them the thing that they're going to click on together with the original payload sometimes. You don't have to think anymore. You're done.

And I think that's why people like it, and I think it's also why HTTP2... The story about HTTP2 sells better in the realm of the abstract than it does in the real world. The real world is not magically solving the kinds of issues that cause people to, like, concatenate their JavaScript now that HTTP2 exists. People are in fact noticing that it doesn't work as well as they had hoped.

Well, I guess we'll all just have to wait for HTTP3, which I hear will support quantum computing. I mean, I think, honestly, the TLDR is that if you... We just gotta get rid of that speed of light problem, right? Yes.

So, I think the TLDR of all this is that if you... Are you opening the box again and talking about quantum computing? No, I'm not gonna... I think if you do something that is morally equivalent to bundling, like you basically construct the bundle, but then you use HTTP2 to push the bundled components, that's probably fine.

Like, that probably will end up being pretty similar to bundling, except that that's... At that point, you're not really doing the things that people are excited about about HTTP2. Right. It didn't really...

It'll probably happen, right? It'll probably happen when HTTP2 is very ubiquitously rolled out. People will have some format that lets you say, like, here is the quote-unquote bundle. Someone asks for this entry point, give them all the things.

But it doesn't really buy you that much. Well, this has been a fun, deep, long conversation about JSON API's origins. Obviously, you know, the first thing in the call is, you know, focus on you personally and your origins in programming and when you became a professional and obviously open source. But this has been a fantastic call.

Yehuda, you are a man of many hats, though, so we couldn't leave this show without at least touching on Ember. We here at The Changelog have been tracking isemberfastyet.com all of 2015. It's not 2016, so congrats on that actually being fast. But is there anything you can share today about the current state of Ember.js, what's going on, and whatnot there?

Sure. So I've been working on You're constructing a completely new array, but the items inside the array are the same, or maybe they're completely new objects, but the properties of all those objects are exactly the same. And I'm making it so that basically without having to think about it, all those scenarios end up updating fast. And Glimmer 1 did that, I think we showed it at EmberConf demo that we then repeated when we actually shipped Ember 2.0 that showed a speedup of probably like more than 10x for the updating case.

It was to the point where Ember 1.x, the demo basically froze, and Ember 2.x, it effectively shows, it shows fast updates, it shows everything updating many times per second, so instead of being frozen. So very big improvement. However, Glimmer 1.x did not really do much for initial render performance, and in fact, for various and sundry reasons, it regressed initial rendering performance. And unfortunately, because re-rendering was so slow before Glimmer, almost nobody had applications that were doing a lot of re-rendering, which meant that all the work we did to improve re-rendering didn't affect existing apps much, it did let you write new apps better, and that would take advantage of it, but it didn't affect existing apps, but all existing apps got slower because the initial render got slower.

So that made people rightly sad. And we've done a lot of work since then to close the gap just by, you know, picking away at little paper cuts that we had introduced for various reasons, a lot of them were compatibility, but not all of them. And the idea behind Glimmer 2 is to say, okay, now that we have integrated Glimmer into Ember, what have we learned about what the requirements are? So one of the reasons why things were so slow the first time...

I'm being very rough on myself. The reason why performance regressed after Glimmer 1 was that we effectively had to learn what the requirements for Ember compatibility were as we were integrating it, and as you can imagine, like as a programmer, sometimes the way you solve problems like that is you just do whatever. You do whatever you have to do to get past the problem. And we did that enough times that things didn't compose very performantly.

But now that we're done, now that we've done all the integration against the more React style of internals model, we kind of know what all the requirements are. And so the idea behind Glimmer 2 is to rebuild the primitive layer against the new requirements. And we have some benchmarks now that we've been running for a while that show many, many X performance improvement on initial render and things like beating React on equivalent templates. So like beating React by a decent amount on equivalent templates.

So we're definitely pretty excited about that. One of the things that I've been working towards that we haven't quite done yet, but it's going to be pretty awesome, is... So Ember's story about templating the whole time has been that we give you a declarative templating layer. So you just write your HTML and you don't write JavaScript in there, you write basically direct, what I guess Angular called directives, but I mean the English form.

You write directives that say this is an if statement, whatever. And historically, that whole system has been pretty dynamic. Like we sort of compiled it and then we said, we don't really know what this if statement is talking about, so just figure it out at runtime. But we noticed a pattern, which is that a lot of times people will make components.

So there's this project called Font Awesome, which is pretty cool. And people write components that wrap Font Awesome so that instead of having to say like I, class equals fa space fa-bug or whatever, they'll make an Ember component called fa-icon and they'll say like fa-icon equals bug. And it will basically emit the correct code. But if you look at that invocation, icon equals bug, you'll see that there's nothing dynamic about that at all.

They're literally using the string bug. So in an ideal world, you'd be able to take the template that is talking about the layout for that component and you would be able to notice, oh, that is actually not talking about a dynamic thing and compile it into the static thing that is really talking about. And one of the goals of Glimmer 2 is to have a sufficiently flexible compilation architecture that we can effectively do these specializations at runtime, do them only for hot code. So like, probably don't need to do that if it's only, if the IFI only shows up one time, but if it's in a loop, probably matters more.

So basically to have a flexible architecture that lets us say, you know, we noticed that this thing is a lot more static than the code would imply, so let's actually compile it into the static form. And this is all owed to the fact that the Glimmer and Ember templating system is a pretty nice programming language for optimizations in general. So it's more like a referentially transparent programming language. It doesn't allow you to do a lot of do these things that make it hard to do optimizations.

So we can apply like the beginner level optimizations that you learn in school against this very simplified programming language and have, and be able to do that reliably. So that's basically the story behind Glimmer 2, is that we're restructuring the architecture so that it's a flexible compilation architecture and then we can make things that are static behave as if they were static instead of behaving as if they were dynamic. And so far the work that we've done on this kind of stuff has shown very significant improvements. So we're pretty happy about that.

I don't want one of the people that put my hand up saying, I don't know how you do all you do, but nonetheless we are definitely still thankful for one, you know, here at this podcast, just thank you for your time to come on and share your personal story, your story on JSON API and how that plays out for developers and a consistent API and a future-proofing in a way for the future, but also the work you're doing at Ember and your company, all that you do there and the different projects I guess you're working on as well with Plane 2 as well. But if that's, we're about 10 minutes past our time. We're going to close the show today. We did have a couple of questions I want to ask you, but we'll save those for a different day.

I'm sure you'll be back on. This is not your last time on the show, I'm sure. So let's close there. I want to thank our listeners for coming and listening to the show religiously.

It is the new year. We have a great year planned ahead of us. And if you're not a member yet, go check it out, change.com slash membership. For $20 a year, you can support the show and do what we do around here and get an all-access pass back to our members-only Slack channel.

Hang out with us, get behind the scenes. We share lots of stuff with our members before we even share it with the rest of the world. We also had some awesome sponsors sponsor the show, so thanks to those guys. And Yehuda, if you have a second, you can throw a note on this if you'd like to.

But our next show is actually about ZeroDB and end-to-end encrypted database protocol with McLean Wilkinson and Michael. I'm not sure how you say his last name, but that was a really interesting find recently. And that's the next show we're doing. So that's it.

You have any notes you have on that show quickly? That seems cool. I'm generally very pleased about the fact that our community and our ecosystem is pretty encryption-friendly. And I think the government would like it to not be so, but unfortunately, people like encryption.

And I'm happy that that's true. That's cool. All right. Well, everybody, thanks for listening.

That is the end of the show. So everybody on the show, let's say goodbye. Goodbye. Thanks, Yehuda.

No problem.