Twisted and Evented Programming in Python

Foreign. Welcome to The Changelog Episode 0.5.8. I'm Alex Koviak. And I'm Wayne Neville.

This is Changelog. We cover expression new and open source world. Channel 19 draws us on the web at changelog.com or also find ghetto heading out.com explore. Find some training repos, some feature repos on this blog as well as our audio podcasts.

If you're on Twitter, follow Changelog Show. And me, I'm Sack. And I'm Penguin. P E N G W Y N N.

Fun episode. This week talk to Glyph over at Twisted, the granddaddy of all evented non blocking. I'll call it a web framework, but toolkit as it were, big in the Python community. So chat full of white space.

So we like Hamlin Sass in white space, right? Yeah, get a little of that. Cheers. Every time we say Hamlin Sassen now I'm just gonna mix in the clinking sound.

So bottoms up. There you go. So, new jobs read this week. We do have some developments around the sponsorship front that we're just so excited about that we will share in coming weeks.

Yeah, so it's really exciting things here for somebody else as well as. As well as us. As well as us. Kenneth, join the last minute and help us out with this Twisted episode.

I think you're gonna enjoy it. Talked about non blocking frameworks in general, but also kind of the world of Python and really a rich history of this twisted project. 10 years old. Anybody using Tornado might also listen as well.

Yeah, I went into that and some of the, I guess, distinctions between Twisted and Tornado. I think it's why it contains a web server, chat clients and all the fun stuff to do all this. It's really wild. Did you see the success pages?

It powers Hipchat, which we use every day, and also powers Tweetdeck, which we're big fans of some other high profile sites. But he probably gets the most play of the fact that Lucasfilm is using it. There you go. Fun episode.

Should we get to it? Let's do it. How you doing with Glyph Luckowicz from Twisted? Twisted, Matrix Labs.

So Glyph, want to give a quick introduction who you are and a bit about Twisted? Well, I'm the original founder of the Twisted project. I write lots of code in Python, pretty much all of it open source. And Twisted is an event driven framework in Python.

It's a networking engine and has tons of utilities for doing event driven programming of various kinds. Lots and lots of protocol implementations. I could do the whole list, but then we'd be here all day. So yeah, not too much to tell if you know what it is.

You probably have used it because it's one of the few things of its kind. A little bit like event machine or Node, I guess recently they're more popular hip versions of this thing. Twist has been around for 10 years, so we were not the first, but a pretty early on. What was the episode Starting Twisted.

Oh, I love that question. I actually talked about that question recently. The impetus for Starting Twisted was I was making the story actually starts when I was 8, but the first 10 years are so not very interesting. Eventually, as I was writing this video game in Java and I eventually reached the limits of the Java virtual machine, especially because this was around 2000 when there wasn't such a thing as NIO yet.

So I had thread per connection, in fact, four threads per connection, reader, writer, exceptions and logic. And as you might imagine, that got to be a big mess pretty fast. So I wrote it in Python, still a thread per connection thing. And then I discovered the select module.

I was just kind of going through learning each module in the standard library and I had no idea what select meant. So I read the documentation and it didn't make any sense to me. So I did a little prototype and then thought, wow, this is like clearly the best way to do. I owe you.

So much less confusing. Everything just happens in order. And as I was implementing a game, I really wanted things to happen that kind of way because I wanted to do a simulation loop that had discrete phases and discrete ticks and every action that the user took was a discrete object. So I was like, great, I'm going to do this.

I'm going to switch everything over to using Select. Now, where are all the network protocol implementations? Because there's got to be a bunch of them, right? And I found Python develop these great libraries, urlib and httplib and mimelib and imaplib, but none of them work in an asynchronous context.

There was Asyncore at the time, but asyncore didn't actually do anything. It just let you write your own stuff. So I wrote my own vendor in Core and I decided that it should be the one true way to write network protocols in an asynchronous way, so there'd be a common API that people could get a bunch of different protocol implementations and have them all the same process. Because I wanted to make my game accessible to web browsers, which were at the time this new hip thing.

But you know all the other network clients that were popular then telnet clients, and I wanted it to be able to deliver you email. I wanted you to be able to check your email on the game server. And now we actually have all that stuff in Twisted. There's an INAPP implementation, pop, smtp, DNS, HTTP, pretty much all the stuff that I originally wanted to do in that original game.

And the game is nowhere to be seen though that project has become increasingly researchy and it's currently called DivMod imaginary. And if you Google around enough you can probably find the code for that. But it's definitely not as mature or interesting as Twisted. A It's speaking of Google, you own Twisted on Google.

I was impressed. Yeah, there's a couple of people that they're close to a humor site that has Twisted in its name somewhere. But yeah, we've been relentlessly and shamelessly self promoting for a really long time and so Google likes us. So you're mentioning all the different asyncros libraries that weren't available at the time for HTTP.

Do you have any thoughts on gevent and Eventlet and how those are, you know, their relationship with Twisted? Sure. Well, so first of all, Eventlet is great. I don't want people to bring up Eventlet because they so frequently.

I hear it from some programmer who used Twisted is now using Eventlet and they sheepishly admit the betrayal. And the Event was actually the default hub for events that uses twisted for Network IO. And that's pretty much exactly the situation we want to be in. It's just the default choice for network I O.

And the Eventlet presents this API that's different than what Twisted would natively present, but you can still use all the Twisted protocols, presuming that you use that hub and you don't switch to one of the other event hubs, which I don't really understand the point of some of the other Eventlet hubs, because one of the big things that they tell you in their documentation mode which hub you should use is, well, you could use the Twisted Hub, you could use the ePollHub because it's more scalable. But actually you can just use Twisted's EPAul support and it's equally scalable. So I think that there might be a communication issue there that we might need to talk to developers more often. And G Event is like an Eventlet except it's got its own network I O and it's totally compatible with Twisted.

So it's very interesting to me, it kind of seems like Just a step down from what event offers. I realize that it's a little bit simpler, smaller, but things like gevent and Eventlet present this API which is sort of semi synchronous. The code that you write in Eventlet or gevent is more or less indistinguishable from the code you would write if you were just writing a multi threaded server. You just write a protocol implementation that blocks and then transparently in the background, it's made asynchronous.

But you have to do all the same things that you would do. You have to write synchronization logic. You have to make sure that you don't accidentally context switch in the wrong place. So for certain types of applications, and to my mind for most applications, but obviously my taste is a little bit biased here, I think that for most applications something like Twisted is actually simpler because you can.

You don't have to kind of unravel the threads in your head and go and inspect and make sure that nothing you're calling eventually calls a socket function, because that'll cause a context switch that you might not be expecting. With Twisted is all very straightforward. You don't context switch until you return. And so it's very easy to figure out what you're returning.

I've been wanting to get into Twisted for a long time. I just haven't found the excuse to. So I just want to point that out. Nothing.

Deep thoughts by Kenneth. Yeah, sorry. So is HTTP, I guess, the primary protocol people are using when they're using Twisted? Well, of course, but I mean, that's just because HTTP is the primary protocol that people are using when they're using the Internet.

That's Twisted Web, right? Yes, that's Twisted Web. And people who just did tend to use HTTP and then something else. And there's obviously a lot of users who will just use HTTP and write web app.

Especially in these heady days of Comet and WebSockets, where HTTP is an increasingly expanding thing that actually is event driven in two way. But HTTP in combination with DNS or with an email protocol is a very common sort of thing people deal with. Twisted. Was that the primary protocol you had out of the box and how soon did the other protocols trail?

Actually, HTTP was not first at all. I think it might have been third or fourth. I can't remember if NNTP beat it out. The idea was originally Twisted's main protocol was really just a custom remote object access protocol because it was a sort of desktop client for the game that prot became what is now Perspective Broker, which is in the Twisted Spread package and Twisted Spread pb and the idea was you wanted to just publish your objects for access over a network.

So PB was the native protocol of Twisted and all the other things were these kind of degenerate things like oh well, ok, maybe you want to use a browser, but that's not as good. Obviously the PB applications marketplace has not taken off to quite the degree that we expected. So HTTP has become a much bigger part of Twisted's life. But that same idiom kind of pervades still, which is that every protocol is just about publishing your objects to the network somehow.

So HTTP is a little more popular, but it's not really. It doesn't occupy a special position in Twisted's hierarchy. And especially because given that Twisted is not a web framework, people often come to it expecting something like Ruby on Rails, but it's really nothing like that at all. It's a lower level thing that's designed that you would build something like a web framework on top of.

It's because it's not a web framework, people who come to it and expect a web framework are disappointed and leave people who come to it expecting a toolkit to do these kinds of multi protocol things are very happy. And that's what our community is largely made up of. So you mentioned it's 10 years old and I know that we haven't had Mac intel that long. You're on a Mac now, so I'm assuming you weren't on a Mac when this project started.

So you come from a Linux or a Windows Python background. I'm definitely a polyglot. I like to use every platform. I think probably my history is primarily Linux.

A lot of Twisted development was on Linux. But actually I believe at the time that I started the project I was using a public beta of Mac OS X Server. That's kind of what I remember having on my workstation at the time. But yeah, I have lots of Linux servers, I have several Macs, I still play video games, so I run Windows on occasion.

And that's actually part of Twisted's philosophy as well. It's partially just a random comment on my own desire to be portable and be able to work anywhere. But also we want you to be able to write asynchronous code for Twisted. Like the original vision was to be able to write an asynchronous protocol implementation and say there, I've done it.

It's an asynchronous HTTP client. You can use it for any Twisted program. And part of the appeal there is you want to be able to write that on the server and the client potentially. So we have reactors for GUI toolkits of various kinds.

So it runs native in the Mac gui, it can run native in the Windows gui, it can run in GTK on Linux or in qt, which I think works cross platform. Well, obviously QT does. I'm trying to remember how long before it actually does. But yes, you can use Twist to pretty much wherever.

And we try. I think there's a definite bias, especially after many years of trying to fight with the network stack on Windows, that we definitely have a unixie bias and we don't like Windows very much. We try not to let that show through too much. We do have support for I O completion ports, which is a Windows specific asynchronous networking API.

So are there any uses for Twisted other than network driven programming at all? Well, all programming these days is network programming really, but the network is the computer. It's sad that even though those guys were right, they didn't really get the benefit too much. So Twisted has a lot of utilities like for example, Deferred.

So a deferred is a result that doesn't exist yet. That is very useful in network context where you're going to make a call across the network and the result might come back or might not. It's also useful in GUI contest though, because it lets you pop up a dialogue and continue processing. You can have a function which asks a user a question.

And of course, if you're running a web app, that's what you're doing half the time anyway, right? You're just asking the user a question and waiting for the response to come back. And you would make a deferred to say what the response would be. But you can just do that with a dialogue box too.

And that lets you pop up a dialogue box without blocking or having weird reentrant multi level main loops or any of the funky stuff that GUI sometimes have. To facilitate that, you just make a deferred and then you can pop up five dialogue boxes. The user can answer them in any order and you can continue processing the background while you're doing that. So there's that.

There's also some time utilities. One of my favorites is twisted.Internet.task.loopingcall which is really designed for a voiceover IP application. So a networking application, in order to do real time every 10 milliseconds, sending out an audio sample so you can tell people about that. The Next time you hear that Python's not fast enough for something, you can do real time audio processing.

And so that was originally what it's for, but then we realized that we really wanted to have it be able to compensate for falling behind. So the idea is it obviously can't be hard real time because even just forget about Python, even just being a user space process, like not a kernel process on a Unix operating system, you can't really do hard real time. So since it's soft real time, we'll make it so you get called every 10 milliseconds, really reliably, but then if some other thing is processing and delaying the main loop for executing your call again, you get notified. Okay, this is an exact multiple of 10 milliseconds that you're being invoked at, but you've skipped six calls, so you've dropped six frames.

And that's useful for animation. You can use it in a pygame game, which I've seen done. These are obviously less popular uses for Twisted because it is a pretty big piece of code that contains lots and lots of network protocol implementations. So most people just discover it in the first place, come to it because of the networking stuff that's in it.

But lots of people who learn to work with the event abstractions and Twisted and write other kinds of programs do use it for other stuff. So Twisted is extremely performant compared to a lot of other options out there. Can you get a little bit into the whole controversy behind when Tornado came out and how they built the whole. From my understandings, they built a whole framework that was unnecessary because Twisted Web was already there.

Right. But it didn't have Web framework on top of it. I suppose it depends who you ask as to whether it was necessary or not. I think the most definitive answer to this was a couple of so I did a big angry blog post when Tornado came out, largely because.

Not because they wrote it, because people write stuff all the time. It was more that the way they announced it was very strange and revolutionary. Well, it included a comment about Twisted, which was misleading and wrong and kind of weirdly passively aggressive. It said something about Twisted not meeting their requirements and I'd never heard from them about their requirements.

I had no idea what their requirements were. They didn't say what their requirements were. They kind of implied that it wasn't fast enough, but they didn't give any performance numbers or anything. So I.

So I was a little miffed that they were bad mouthing Twisted in this way that made it impossible to Respond and say, no, it's not slow, no it's not whatever. Or even just to constructively respond and say, oh wow, it is slow. We should really fix that. So trusted.

I don't like to harp on performance thing too much because performance is an extremely complex question, especially when you get into a system like Twisted, which allows you to integrate lots of different protocol implementations, lots of different event sources, all firing the same main loop, all sharing resources. What you're doing depends very, very heavily on how it's going to perform and. Or, sorry, how it's going to perform depends very heavily on what exactly you're doing. And one of the things that we have to counsel people over and over again in various support forums for Twisted is write your code, run your code, run a profiler, see what the hotspots are.

Because people get very excited about micro benchmarks and then they focus to the exclusion of actual useful stuff. Especially like performance under scalability. Like if you have a framework that can do 20 connections really really fast and process lots of responses and requests of those 20 connections, but then you scale up to a thousand falls over, is that better or worse than a framework that doesn't do those 20 connections fairly fast, but maintains that performance on a totally consistent ramp up to how we make connections you want. Personally, I tend to go for things that are the latter.

But regardless of what type of performance you're looking for, your application will always be 10 times as much CPU than Twisted. So when you write a Twisted app, you will typically spend your time optimizing things outside of Twisted. I know this because whenever people start talking about performance, I really, really want people to contribute performance patches to help us make Twisted faster. You can check out Twisted performance on speed that twistedmatrix.com but it's fast enough for so many things that we get very few contributions in the performance area.

Because people come to it, they spend a month complaining about performance and doing these tiny little benchmarks and trying to figure out twisted significant and then they decide to use it and then it turns out that actually that was a huge waste of time and all those benchmarks they were doing are not actually measuring their app at all. And when they go and optimize it turns out, oh well, Postgres is 99% of the performance bottleneck. We don't even notice Twist. It doesn't show up in any of our profiles.

So that's a typical performance story. Of course there's stories where, for example, if you're doing voice over IP and you're trying to multiplex a thousand real time audio streams. Then you start to notice the low level networking stuff cropping up. As far as Tornado specifically, it seems to perform kind of to within an epsilon of Twisted.

There are a couple things it doesn't do there. It's a little bit faster. So it's not really clear that there's a huge win on one side or the other. But the most definitive argument in the whole Tornado thing is after I wrote that big angry blog post, a Twisted user came along and just wrote a patch.

I think it was a fork on GitHub, if I recall correctly. That's where they're hosted. And he just took out all of the networking stuff from Tornado, replaced it with Twisted and Web framework API remained exactly identical and it was a patch that deleted like 8,000 lines or something. And Tornado was functionally equivalent on top of Twisted, unless you were writing one of a hook into their IO loop.

So quite the success stories on the twisted website. TweetDeckJustNTV, HipChat, which I use every day. Any of these are you more proud of that you're able to enable someone else's success? Well, I'm proud of all of them.

I'm happy whenever anyone. Anyone's dream to power Lucas in some way, right? Oh, yeah. No, I guess if I had to pick one, it would have to be Lucasfilm.

I got a Christmas card from them once and I was like, I have arrived. They were really great. It's super gracious the folks at Lucasfilm who actually did that and got us a success story, obviously they're a big company and it's difficult to get something like that out past the corporate communications people. And I really appreciate that.

Dave, Pete Colos, who's the guy listed there in Success Story, really works to get us that success story and get on our website. So if I had to choose one, that would be it. But there are so many projects that have used it in some way. Almost proud isn't the right emotion.

In a way, it's honored. I'm honored that they chose the technology that me and the Twisted team worked on so hard. It's just a great validation of our efforts. And there's some on there that if you look on, we've got another wiki page.

Projects using Twisted success stories are just the ones where people can actually put together a little narrative about why they're using it and what's good about it. But another one that I'm really quite happy about is OpenStack, which was the collaboration between NASA and Rackspace. This is the open source web framework, or I guess cloud framework. Yeah, it's a cloud computing thing.

And to be honest, given that it has a word cloud in name, I'm not even really sure what it does, but it's a set of APIs that build on top of the different virtualization layers. So you can control like a VMware stack as well as a different ones. I believe it includes some other things and specs on top of that as well. Storage cats.

Yeah, all that. It's really that some other things where I get a little bit fuzzy. But regardless of what it is, it's used by NASA to control thousands of computers. So that is cool.

You must be a hit at Christmas and Thanksgiving when you're home doing Lucas and NASA. No, you know what? My family is very diverse and eclectic. My sister is an acoustic physicist.

Wow. My other sister is a rock star. You just. It's hard to impress anybody in that family.

In fact, my father, I don't know if you've ever heard him speak, but he was a keynote speaker at PyCon and OSCON. So even if you just restrict my family's achievements, open source, I'm still kind of not necessarily at the top of the heap, so. Yeah, but it's great to be in a family like that. So I was gonna ask, who's your programming hero?

Is your dad in the list? It might sound a little corny, but yeah, I always thought of my dad as my programming hero, but I didn't actually know that much about what he did. He worked a lot on systems in the finance sector and I was a little kid, so I didn't really understand what he did. But actually I worked at a startup a couple years ago with one of his co workers, and that experience was really interesting because apparently I write code very much like my father.

There are similarities between twisters and systems that he worked on. And the more that I've learned about what his career was like and the kind of stuff that he did, the more that he's my programming hero. The voice is strong with this one. Yeah, that's a metaphor.

A little bit. So uncomfortably close to home. So what's coming up on your open source radar? What new projects are you excited about?

You know, it's very hard to choose something. I'm just in the world of open source. I think what I'm really glad about is that we are experiencing this massive renaissance. It's hard to get excited about any single project because every time I do something, I just Type a search into my web browser and there's something that does something like what I want in the open source space.

And even in the relatively small niche of Twisted, there's just tons of libraries that people are writing every day. I'm really kind of excited about the minor renaissance that Twist is enjoying too. The last couple of releases we've gotten out on time. We've gotten new features for a while.

Development was slowing down a bit. We had a lot of bugs to fix for a long time. We transitioned from that process. A process was a wild west kind of commit anything you want to.

Everything has to be unit tested, everything had to be documented. And for a little while that slowed us down. But now that we're reaping benefits of having done that, you can actually see on the twistedmatrix.com highscores the review points that people are accumulating. And if you click on that left arrow I'll go back a couple months.

Deep interface I'm glad you appreciate it. The font was the first thing that went into that web app. Avatar is 8 bit, so I'm stuck in 8 bit world. So glyph is one last question.

Is this your real name or is your name like a symbol? Like Prince Wise where you just had to shorten it to glyph to make it pronounceable or no, glyph as I handle pre created the symbol. When I started using Unix I needed a short handle that was easy to type because I had to type as I started off. When I started using Unix, I was using a Mac OS 8 machine and so I needed to type my username all the time because it wasn't implicit, it wasn't part of my environment.

So every time I connected somewhere I'd type it. So five letters shorter than my real name. But it's not my legal name and I don't talk about my legal name because it's got a little in joke on the open source community. My hypothesis is that nobody really reads licenses or knows what license things are under.

And this is validated by the fact that most people don't know my real name. But for I think 7 out of 10 years the twist has been going. It was at the top of every single file in the posted repository. In the license statement it said copyright to my real name.

So so are you like the y lucky stiff of the Pyton community? I'm not going to randomly disappear from the Internet. Asynchronously, yes, he'll probably be that right. Well, if I did asynchronously I would be the doctor of the five community knock, but I can only hope to be as witty and prolific as Y.

Indeed. Well, thanks so much for joining us, telling the world and all of our listeners about Twisted's been out there for a while, but definitely good stuff. And I wanted to get down with this project just because it seems like we're going to talk about Node js it comes back to oh, that's just like Twisted. So.

Well, thank you for the opportunity and if I might just one last interjection. Since your co host mentioned that he had no reason to get into Twisted, he was looking for a reason to do it. I would just like to leave your listeners with something that they might do. Twisted includes a whole bunch of command line utilities for running all the servers that it includes.

So if you're the average sort of open source nerd who runs a personal server, you can replace all of your personal network infrastructure with Instead of Bind you can run twisty DNS, instead of Apache you can run twisty web and instead of Hybrid you can run twisty words IRC port. So you can pretty much any network service you're interested in playing around with. You can start off with just having one command line. You don't need to write a whole ton of code to get into it.

Fantastic. You need SSH server Twisty Conch. Yep, Twisty conch ssh. Of course with crypto you gotta generate some keys and go over more.

So that's the reason I don't open up with that one. But yeah, it is a functional replacement for OpenSSH. It does authorized keys, authentication, everything. And does that work well with as a client as well?

Yes, you can just run Conch and it's more or less drag and drop or sorry, drop in replacement for things stage. Except it outputs a couple log messages often. So I'm curious, how does that compare to Paramiko? You can run in any AA client in any Twisted server.

That's the difference between that and paramo. Okay, there's 110 Python. There's no dependencies at all. Right, well, there's some secret dependencies, but the application logic is all python.

The network IO is all twisted. It doesn't use any special network I O stuff. It just reads the bytes and does some crypto on. Sounds good.

Sounds like you claimed Ken's upcoming weekend. No, no, no. Excellent. Next month.

Cool. Thanks again. Luckily we really appreciate it and thanks again for the opportunity.