Your Huginn Agents Are Standing By

I'm Andrew Cantino and you're listening to the changelog. It's true site Pulse. Our first sponsor today is our friends at co-chip. They have a new platform called co-chip jet and they've got this webinar coming up to talk about managing test environments with Docker and this new platform called co-chip jet.

Co-chips engineer Brendan Fosbury is going to show you how to use Docker to simplify managing your application across different environments by using Docker and co-chips continuous integration platform jet. This is a free webinar April 28th 2016 at noon Eastern Standard Time. I'll put a link in the show notes, but in the meantime go to co-chip.com slash changelog to learn more about co-chip jet and now on to the show. Welcome back everyone, Jared here.

We want to start the show out with a shout out to a brand new member of the changelog family. That's the baby Stikobiak. So congratulations to Adam and Heather and welcoming Eli to the world and I'm not here on the show today. Taking care of more important things, but we are excited nonetheless to be joined by Andrew Cantino to talk about his project, Hugen.

Andrew, welcome to the show. Thank you. So, Andrew has a way to get started. It helps out and it's fun to understand who the guest is, where you're coming from, how you got to the point where you built Hugen, which is a system for building agents that perform automated tasks for you online.

We're going to dig deep into that. But first of all, I'd like to learn a little bit about you. So can you tell us a little bit about yourself? Sure.

I've been doing programming for probably about 20 years. I started when I was, I think I was about 12th and I took a summer class on robotics. And I actually didn't particularly like the robots because they tended to break a lot of hardware, but I really enjoyed programming for them in basic. And so after that summer I convinced my parents to buy me a Pascal compiler, which I don't think I ever actually used.

I just kept using BASIC and I learned Microsoft Quick BASIC and then I learned something called Real BASIC or Mac. And then I got into Pearl. This was before college and I did a bunch of Pearl scripts, some of which I sold. And then in college, I majored in physics, but I kept doing computer science as well.

I took a bunch of classes and ended up minoring in computational science, applying it to physics. And I think I learned Java and Python and kept using a lot of Pearl for web based projects in college. And then after college, I went to grad school and I switched from physics to pure computer science and I studied machine learning in grad school for a couple years. And then I came out to Silicon Valley and I've been out a bunch of startups and companies out here since then.

Go back to your Pearl scripts. You said you sold some of those. What kind of scripts were you selling there that they were valuable enough people want to pay money for them? This was like before the .com boom and people would buy these CGI scripts for very, what we would now consider very primitive websites.

I had Pearl scripts. I had a guest book that you could leave notes for each other on a website. I had a chat script, which actually got pretty popular. I ran a network of chat servers that were, you could run them with no JavaScript at all.

It was better if you had JavaScript, but they were pure HTML and CSS if you wanted. So we had a lot of users on Web TV, which was a thing. And these really primitive clients that couldn't run JavaScript and got popular in these strange niche communities. I think I ran it for almost 10 years, closed it down right after college.

Web TV, was that the thing where smart TVs are now that it was way before time and you could not get on the web from your television? Is that what it was? Yeah, I think you had a device that had a modem, I believe. And you plugged it into your TV and it was a very poor quality screen because TVs were really low resolution.

And you could very slowly browse the web. And some of them I don't think even ran JavaScript at all. Right, right. So on your website, you say that you're an experimentalist.

What does that mean? I like building things and tinkering with things and experimenting with new ideas. I sort of like building systems from scratch to see how they work. I built a web browser in high school, just for fun to sort of learn what the primitives were.

I'll go back down. They were a lot simpler. And then I was playing with algorithms, playing with toys. What's much better if you have an experimental experience?

I mean, I think perhaps when we get to Hugen, we'll find out that that's probably coming out of that Tinker mindset. But do you have any other examples of things that you just kind of experimented with? I've had a couple of fun projects. I had a project in college.

It might have been before college actually where I was trying to evolve CSS styling for websites. So you could look at a website and it would take the style sheet and mutate it and make a couple potential offspring, kind of like Blind Watchmaker. And you would display them over a proxy version of the site and you pick the one you like. Or if you wanted, you could pick multiple offspring and pick the ones that you like.

And then it would combine them and basically treat lines of CSS as genes that it could flip back and forth. And so do both mutation and combination with the offspring and then you could evolve the style sheet that you like for that site. Very cool. Well, Selector Gadget is a project that I did a while ago.

It still gets some use. It's a tool for building CSS selectors for websites. It's a Chrome extension or a bookmarklet. And you can click on a region of the page that you're interested in and it highlights, it sort of does a best guess of what Selector would make sense there.

So it'll prefer an ID and then if there's no ID, it'll fall back to a class and it'll highlight everything on the page that matches that. And then usually it's not right on the first try. It's not right on something that you don't want in your selection. And it turns red.

And then it tries to figure out the sort of minimal selector that matches everything you do want and nothing that you don't. Yeah, let me just give you props on Selector Gadget. In fact, when I was on your website preparing for this call, I saw that you made Selector Gadget. And I was like, yes, that's awesome because I use that back in the day.

And it was kind of a revelation because I don't know exactly why, but it was so impressive to see it like just, you know, drill down on a specific section and grab the Selector for you because back then we're all kind of learning about CSS selection and how to do it best and how not to do it. It was helping me realize other ways of selecting things that I previously would have never known. So very cool. People are still using that today.

Yeah, it still gets some use. I turned it into a Chrome extension maybe a year ago and it's getting some use. It's never sort of taken off. It's hard to know, you know, because people just use it.

I don't have any great metrics on how much it gets used, but sometimes it shows up on Twitter periodically. Right, right, right. That leads to me one question I've been asking people recently because I'm kind of becoming transfects on this idea of longevity with software projects and really having a lot of respect for things that are, what people might consider legacy, but are just like older, but actually had value and still work today. And so looking back at myself in my own software development career, I look back at stuff that I wrote back in.

I mean, I can only go back about 10 or 12 years because that's how long I've been doing development. But I look at things that I wrote, you know, 10 years ago and ask myself, what's still useful? And so thinking about it like that and you have a longer history than I do. Does anything go past Selector gadget going back?

You said that you had these pro scripts that ran for quite a long time. What's the oldest piece of software that you wrote that's still being used in some capacity today? That's a good question. Well, older than Selector gadget, I was on the Gmail team at Google for a summer internship.

And I'm pretty sure that code I wrote to generate filters in Gmail is still in use as well as possibly some of the code in the search bar that does auto complete. I don't know how much it's evolved since I was there, but that's about eight years old. And then older still, there was a project called absurdly cool freebie finder, which was this search engine for finding free stuff. Nice.

And that's so online. But unfortunately, I sold it, which was not unfortunate, you know, that was great. Unfortunately, the new owner has just sort of let it die, which has actually been my experience in general with stuff that I've sold. I've sold it, and then the next person doesn't care as much as I did.

So it sort of goes away. The site's technically still up, but it's not aggregating anymore. But I ran that since, uh, since about 2005. Older than that, I'm sure there's still some Perl scripts running somewhere.

I mean, I had like a hundred thousand downloads of that guessbook. There's no way they're all gone. But it was, you know, super insecure, terribly written, untested Perl. So I kind of hope they're all gone.

So you told us how you got in the software. What about open source specific? Go back to you. Remember your open source routes and where the idea of open source kind of sprung up in you and you got excited about it.

That's a good question. Um, well, so the, the Perl scripts that I mentioned, most of those were open. There was no realistic way to make them close source because they would, you just have to provide the script for someone to run. Um, so some of those, I think, got changed and you could, most of them were free.

Many of them were free. So that was in high school, um, 20 years ago. Um, before that, a really sort of formative experience for me was this Macintosh application called Hotline. And it was this network of, it was peer to peer and it had trackers and it had servers that would register to the trackers and each server could have a chat room and a forum and file exchange.

Um, and I was probably 13 or something when I found these and I stumbled on this community of other children, roughly my age, somewhere older, somewhere younger, but we're all sort of teenagers who are learning real basic together. And that was the language that I was playing with at that time. Um, and that was really cool because people would just upload examples of something they made and someone else would download them and sort of riff on it and upload a new one. And that's, that was really sort of formative for me to get into real programming and because there were people there who are more experienced than me and I can learn from their tricks.

Um, that's where I sort of, I mentioned that I wrote a simple web browser. It was there and shared and people thought, you know, you get props, you know, people like, this is awesome. How did you do this? Right.

It was really, it was like a fun game. Um, so that sort of proto open source. Like I don't think many things came out of that became real products. I'm sure some did, um, but it was really important for me to sort of work with a community of other programmers.

Just thinking about that and the, even the open nature of the Perl scripts that you were selling makes me wonder because it's so far back. Um, how did you go about? Like, what was the transaction like when you sold a Perl script back then? Was it, uh, mail me some cash?

Was it? Yeah, I think so. I think it was like mail me, mail me a check. Um, that's probably cash sometimes.

Yeah. And we're not talking, you know, this was like a hundred bucks and I was in high school and that was, that was great. A hundred bucks is awesome when you're in high school. Yeah.

I mean, I think I had a few that might have been a couple grand. Like it was, it was certainly a great, great spending money in high school. Very cool. Well, we are going to talk about you again.

Let's take a quick break and we will dive into what that is and why it's awesome. We're back. We're working with our friends at BMC to spread the word about true site calls through real time, monitoring service for apps and infrastructure. I talked to Mike Morandi, senior architect about the idea of dev teams out there, rolling their own monitoring system, using something that's open source or building their own from scratch.

And he had this to say. I think if you want to roll your own and spend the dev effort of having to build that internally, that's great. My only question to you is if you spend your time doing that, are you providing value to your customer? And are you actually moving your product forward or are you holding your product back?

I think a lot of what something like True Site Pulse offers you is we take a lot of that on for you. So you can provide that value to your customer on your product instead. So we have plugins for 30 plugins for different parts of your infrastructure. We have an agent that's been running for three years written in C that takes a very small amount of your resources.

As you add more servers, you're not going to have to worry about the scalability as much. And we've written the chef and the puppet scripts for you. So that's all taken care of. It's letting us worry about it so you can focus on your customers.

That's kind of value that True Site Pulse adds as opposed to you having to do it yourself. We've all been in organizations where we've joined and had to rewrite the entire monitoring stack. And that's just something we didn't want to have to do. We want to come in.

It's not taking care of. And then that way we can focus on the things that are going to matter to our customers. I was Michael Rand, Senior Architect at True Site Pulse. To learn more about True Site Pulse and how it helps you deliver more value to your customers, head to BMC.com slash True Site Pulse.

All the word. And tell them, add them to the chain plug sent you. All right. We are back with Andrew Cantino.

And we want to talk about this really cool project called Hugen that has contractions. Probably more so than select or gadget. I think you got 13,000 stars, 1300 plus forks, 110 contributors, which is quite an accomplishment. We're going to talk about what Hugen does.

But let's first, let's get the name out of the way. Because it's one of these things where, as I was saying before the call, as programmers, we tend to write things all the time and we read words and we all kind of live on our terminals, you know, reading text. And it's not until shows like these where we get together and actually have to talk about concepts. And many things we just pronounce them differently.

And so I thought it was Hugen. Ever since I've heard about it, which was a couple of years ago now, until you sent me your audio clip and I found out you pronounce it Hugen. So can you first give us the inspiration for the name and then kind of describe the pronunciation? Sure.

So Hugen is named after one of the Ravens of the Norse God Odin. In mythology, they flew around the world and reported back on what they saw. So the two Ravens were Hugen and Union thought and memory. And I've always pronounced a Hugen.

And when I've looked online for someone who might actually know what they're talking about, I've heard Hugen. I've never actually heard Hugen, which is what a lot of people say in the open source community. And I think it's kind of cute. So it doesn't bother me.

That's funny because when you come across the name, like I did, I just reading that saying, and you just kind of sounded out. And I don't know anything about that mythology. I think it's a cool name based on that makes a lot of sense. You're like, oh, it's trying to hug you.

I'm like, I'm not really sure why I named it Hugen, but there it is Hugen. So just funny. We kind of construct these things in our minds to fill in the gaps. But that's such a great name based on that mythology.

Was it something where you already knew the story and you're said, well, this makes tons of sense because this is an agent getting information for you, just like in the mythology. Or you dig in for a name and you came across that and you thought, oh, I can use this. How'd that come about? That's a really good question.

And I honestly can't remember. I think that I was looking around for a name and sort of looking for inspiration in mythology. But that may not be right. I can't remember.

Are you ever thought about starting a Munin project? I would, but there is one. Yes, it's a monitoring tool. That's right.

Well, so you're in San Francisco and you're said you worked at some startups. So you're very familiar with the elevator pitch, right? So if you had to describe Hugen to somebody in a paragraph or two and you had to do the elevator pitch, well, that sounded like. So if you're aware of products like ift, which is if this and that or Zapier, then Hugen is really easy to describe because it's basically an open source, self-hosted clone of ift or Zapier or a little bit like Yahoo pipes.

And then the more pure elevator pitch would be use Hugen to monitor the world for you to take data in from interesting sources on the internet and then react to it, filter it, aggregate it and then take actions on your behalf, which is often as simple as sending you an email. But it could be more complicated to like posting something or taking some more interesting action. We might have to take a moment to pause for pouring out a drink for Yahoo pipes. Isn't that dead now?

It is. I was sad to see it go. It was a great product. It really was such an interesting idea and so many uses.

I think a lot of the open sources were sad to see that one finally get closed down. There's been an influx of users after Yahoo pipes closed down to I'd say sort of varying degree of success. Like some of them, Hugen really does meet their needs and some of them want to use Hugen for sort of very deep feed recombination and filtering, which it can do if you sort of squint. But it's not sort of perfect for that at the moment.

So perhaps some more of your disgruntled users or your feature requesters are the people coming from the Yahoo pipes site. First commit March 3rd 2013. So it's three years old now. Was you said you describe it as if you understand if this then that and or Zapier then you can describe Hugen as a clone of those things or an open source version of those things.

If this then that or Zapier the actual inspiration for it was it a clone or it just happened to be like mutual invention or how to how to start. No, it wasn't a clone. I started it a little earlier than that. I think it was December around Christmas of 2012.

I was at home visiting my family and needed a project to tinker on and had this idea for basically just a gooey for con jobs and sort of reusable components that could be wired together because I end up writing so many little scripts in Pearl or Ruby to automate things. And I tend to rewrite them all the time. And I don't have a shared library. And so it was started as a shared library of stuff that I wanted to wire together inside of a Rails app.

I think my first use case which I still actually use was it would be really cool if I got an email in the morning or push notification it was going to rain that day so that I actually remember to take my umbrella because I'm terrible at this. So I have a network of human agents that does that for me and it did that very early on. I think it was the first thing I implemented. I also played with a bunch of location based stuff early on.

So yeah, it wasn't really I don't think it was inspired by any existing product that I was aware of. It was just sort of the result of tinkering. As you became aware of those products did you ever think they perhaps they ripped you off or that you could compete with them or like what was your what was your reaction as you think you're probably newer. I don't know.

I feel like it's newer than if in the marketplace and so maybe they call them them. But what are your thoughts as you see these other things come out and they're like, you know, pretty successful small businesses or startups at this point that are basically providing very similar functionality to what you've been doing in the open source space. I think it's great. They're good products and I have nothing against them.

And if you don't want to host, you know, for something on your own or you're not trying to extend it, they're really good solutions. They have a lot of connectors. They're monitored by, you know, professional people who know how to run services and that's awesome because then I don't need to build a hosting platform. And my interest was always in controlling my own data.

I didn't really want to have my data be in someone else's hands and it doesn't really bother me sort of in an abstract sense, but mostly in sort of the longevity that you mentioned, you know, startups come and go and I want to have, I would love if I have 20, 30 years of history at some point of all this data that I can play with and that seems unlikely if I'm giving it all to a startup. So my motivation was sort of to fold one to control the data, not so much out of paranoia but just sort of to keep my hands on it. And the second was really that I'm a programmer, it's satisfying to write things. And QGIN is a library of reusable components and often I find myself needing to add just one more to make something complicated work.

So it's really satisfying as a developer because you can go in there, use a simple API, write another agent and wire it all together. And that's just not something you can easily do in these hosted solutions. You can't just go write arbitrary code and run it. Yeah, you have to fit inside of whatever framework they provide, which is usually limiting.

Right. Things are, all interfaces are limiting. So what are some of the cool things that you can do with you that you mentioned? The first one, which seems like was your primary use case at first, which is let me know when I need to remember my umbrella.

That's right. What are some other ones? Oh, there's a bunch. And frankly, I think that I am one of the more boring users of you.

It has mostly met my needs for a while. And some of the stuff that people are using it for now are much more complicated than any of my personal use cases. One of my favorite uses that I do use it for routinely is to monitor Twitter. So QGIN can run a Twitter agent, follows the Twitter filtered feed.

You can give it a bunch of terms that you'd like to hear about. And then you can either just, you know, emit an event for every term. So for something rare, I do that, for example, the word you get, not particularly common. So I just see all of them or my last name is actually pretty rare.

So I just see all those. But for anything frequent, I don't want to get an email for every time it's mentioned. I just want to know when it's changed in an interesting way. And so in that case, I use the QGIN mode that has two modes, either events or counts.

And if you do counts and it's basically emitting a histogram bucketed by whatever check frequency you said. So you say to check every five minutes and roll up the number of times that a certain term has occurred. I send those to a peak detector agent, which can filter, basically watches for a high standard deviation spike in that time series, and then triggers its own event if that occurs. And then that I either send to either, you know, a push notification or an email or something else.

So I can give you two examples. For the push notification case, that's something I want to know right now. So I have a couple of agents that watch for like the terms San Francisco tsunami or bomb threat or something like that. And I want to know, like, tell me right away if this happens.

And so I can find out in a couple, you know, in about a minute if there's a sudden spike in conversation about those terms on Twitter. And then luckily those haven't triggered very often. And then the more useful day to day ones are the slightly less frequent, but interesting terms where I basically use you can to watch for interesting stuff to happen on Twitter that I think might happen eventually and just tell me if it does. So I don't have to check myself.

So for example, if I'm waiting for, you know, a call for a call for papers at a conference, I'll put the conference name in Twitter and I'll spike when they announce something and waiting for Mass Effect 4 to be released. So I just have Mass Effect 4. And if there's some announcement, I get an email. NASA announcement, Ruby vulnerability.

A recent one I started using is if there's a movie I'm waiting for on Netflix. I do just movie name Netflix and that will spike when it gets released and I'll get an email. It's basically like a sort of fuzzy natural language API to the world. Like Twitter, you know, people are going to say things in their own words when something happens and it has such a volume that if you just keep an eye on it, almost any term that you pick is going to increase proportionally to the amount of interest.

And so if you just look for spikes in that, it works surprisingly well. That's kind of like, you know, the Google News search alert type of a thing only. I'm not sure how much they put into it, but I love how it's proportionality or based on trends because you tend to get so many false positives or, you know, I used to have one out for my name and because my name is pretty unique at least when the first and last combined and unique enough that there weren't false positives, but all I would get alerts on is whenever I published my blog post. And I'd be like, oh, yeah, well, I already knew that one because I just hit the published button a couple hours ago.

But then if you get more specific on things, if you try to generalize more, you get overflow. You know, you get tons of results. I used to have that problem. So one of the ways that I bootstrapped my consultancy back in, I don't know, 708, whatever.

Twitter was first out there. And I started realizing that a lot of people would, when they turned, when they needed technical help, it was a lot of work to like post a job listing, right? Or to, you know, post an ad in the paper or whatever your traditional means were to post or help. And it was really easy to just like put something out on Twitter.

I know it's a lot of people were starting to do that with specific programming needs. And so I wrote a little monitor, similar to the one that you just mentioned, looking for specific keywords and then paired those with phrases and my turn into an RSS feed because I was just consuming RSS in the morning anyways, and I use that as ways of finding, you know, freelance type stuff. But there was, it was super useful and it ended up being a way that I did get a lot of business that way. But it was full of false positives, especially because like the word Ruby is also a girl's name and, you know, you run into stuff like that.

And it seems like your way of doing trends or waiting for a certain proportional change probably slakes out a lot of the false positives. So it's just higher signal less noise. Is that what you found? Yeah, it does pretty well.

It's not perfect. But it's low enough noise that it's useful. And for those I just have it send me a digest email one day. So I skim through them and formats them so each one's a link to the search results on Twitter.

So I can pretty quickly see like, oh, no, that's, you know, totally unrelated. I can see why it triggered it. But most of them are actually relevant. And so I keep using it.

I love how many ideas you have about ways of using that one single feature. I feel like I wouldn't have, I've never even figured I put Netflix plus a word that I'll know when the movie is available. Like my brain doesn't connect those dots on its own. I'm sure there's a place where at least you can agents where you guys have a list of like here's different ways that you could use it.

But what about your specific Twitter stuff? Is there any word where that lives online? Where somebody could go for inspiration of ways of being notified of an earthquake in Molly? I've written a couple blog posts about it.

There's some tutorials and I know other people are doing it too. But I don't know of the user base of you and how many people are using Twitter. A number of people have said they are. But a bunch of people are using it in different ways than me.

They use it for posting or for following a small group of people and rolling it up into an RSS feed. I also, I'm not sure that we've really sort of explained at a higher level how you get us wired together. If you think that would be useful to talk about. Well, first, just give us a couple more use cases.

I think we'll definitely do that. Sure. When we hear about that architecture, we'll probably talk about the specific bits because I think technically it's interesting. There's lots of moving pieces and you put together into a holistic system.

We want to hear about that. Just to continue to whet our appetites. You have, let me know when it's raining. You have Twitter-based aggregation notifications.

What are some of your main use cases? What are some of the crazier ones that other people have done that you're like, wow, that's really cool. Well, one of my favorites, mostly because it's just a big name, is that the New York Times used it to monitor some of their internal Olympics coverage a couple years ago. So I believe they were using it for sort of fairly traditional monitoring where it was watching their own website and then alerting if it didn't match expectations.

But that resulted also. That was neat. That was cool to see that use case. And also got some contributions from them in terms of addition code.

It's nice when people stay involved and keep adding to it. There was another use case I heard about where someone was using it to download Civic data releases, like if governments released interesting data sets. They would watch those fairly hard to follow feeds and let them know in a more useful way if it happened. I've seen a lot of, you know, there's home automation stuff.

I turn on my porch light at my actual local sunset, stuff like that. You have listed on the readme, just different things you can do with it. One is create Amazon Mechanical Turk workflows as the inputs or outputs of agents. For example, once a day ask five people for a funny cat photo.

Send the results to five more people to be rated, send the top rated five to people. Five other people for a funny caption, send to five final people to rate for funniest captions. Finally, post the best caption photo on my blog. I don't know what's doing that, right?

It's great. I mean, I tried it once and it mostly worked and I decided I didn't need to personally run a fun cat blog. But it did work. It's awesome.

Amazon Turk, if you know anybody using that to create advantage in any way that's not one of these fun kind of like that would be cool. I think it's being used a lot in the machine learning and AI community to build data sets. I think that's the sort of most valuable use case is building a data set of human labeled information, labeling photos or annotating sentiment onto tweets or noting down the text regions of documents, photos of documents, stuff like that, where you just need to build these really large data sets for deep learning. That makes sense.

I think this is a natural place to break. We do want to talk all about the nitty gritty details of how Hugen works. Hugen works. The agents, the peak.

When you call that a peak adapter peak monitor, I don't know. You have a peak detector. You got a nice agent event flow. A few minutes though.

Some graph this fun coming up. Let's take a quick break and we will dive into that on the other side. If you're like me or most people out there, you want to attach highly available expandable blog storage to your droplets on Digital Ocean. And guess what?

The feature is here. You've asked for it. They've delivered. Kind of.

Right now you can request early access. The feature is coming in the summer of 2016. So I heard that the earlier you get on the list, the earlier you get the feature. Head to digitalocean.com slash features slash storage.

Get early access. Do it today. All right. We are back.

And I don't know about everybody else, but I've been my appetite has been sufficiently wetted. I'm very interested. Hugen sounds really cool. It can do lots of different things.

Things that I would never even imagine to even want to be done until I hear about them. And I think, oh yeah, that would be pretty cool. Which is kind of the beauty of these types of systems is that they really are just a bunch of tools and ways that you can do things and you have to bring your ideas to them and make them do cool things. So Andrew, talk to us about specifics, how it was built, how it works.

Do you have agents? It appears to be a Ruby on Rails application. Can you break it apart for us and tell us how it all is wired together? Sure.

So it is a Rails application. It's actually a pretty traditional Rails application. We're not doing anything particularly unusual. It's sort of my focus has always been on ease of use and ease of deployment and ease of development.

And so it's not really intended for high throughput. You know, I wouldn't use Rails for really high throughput application for data processing. But it holds up totally fine for sort of personal business use. It's like small business use.

The basic wiring is that we have these models called agents. They're the core object in the system. There's many types of agents. You know, there's some of the ones I've mentioned.

Like the one that can talk to Twitter and the Peak Detector and the Mechanical Turk Agent. There's a lot of other agents that sort of focus on outputting to systems. Like push-bullet or push-over RSS or Slack. And then there's things like webhook agents that can both send and receive posts to remote systems.

And then one of the more complicated ones is the website agent that can sort of scrape arbitrary websites in. I think at this point RSS and HTML and JSON and XML, it's sort of, it's gotten a little bloated. But it's very powerful for you can give it a set of CSS selectors or XPath selectors. And it can grab all the regions of a page that match and emit those as events.

So that brings us to events. Agents are connected together in an event flow graph. So agents can receive in a mid-event. Events just flow to all the receivers.

And they propagate down until they stop propagating. So, you know, you should avoid making loops. In theory, you could get yourself into a situation where you had an infinite loop. Although in reality, it doesn't happen.

I've never really had it happen. And then, so you have agents, you have events that they emit and receive. And they just propagate along. And then agents are modeled after sort of a simple reaction agent.

So they have state, they have memory that they get to use however they want. You can have many instances of an agent and they each have their own memory. And then they just receive in a mid-event. And events are mutable.

So, you know, that's sufficient to build most types of systems where I need to store some state. I need to react to things. And then it means that there's a pretty simple API for developers to add new agents because it's just basically a simple active-record model that you extend you wire in. If you want to receive events, then you define a receive method.

If you want to check for events on a schedule, sorry, if you want to do something on a schedule, you define a check method and the user can pick what schedule should run. And then you can use your memory if you need it and you can emit events if you need to. Obviously, there's some more complexities. We have types of agents that can run continuously in a background thread and some other stuff like that.

But at a basic level, it's really pretty simple. Yeah. So, your main user interface is the creation and the editing and the hooking up of the different agents to do what you want. And then it shows you kind of the status of those agents and what they've done recently.

How does the scheduling system work? How does the backgrounding work? So, we're running a background. We have two versions that can either run multi-process or multi-thread.

By default, these days it runs multi-thread. We use Rufus Scheduler, which is a Ruby gem that does a good job of simulating both cron-style scheduling as well as sort of exact time scheduling is basically cron. And we use it to trigger any set of registered agents on a certain schedule. So, when you make a new agent that has a check method defined, you can say, you want this to run at midnight every day or every five minutes, et cetera.

And it will do whatever its check method does and do whatever this agent is designed to do. And then we just in the background, use Rufus Scheduler to make sure this run. I imagine, like I said earlier, you have 110 contributors. Most projects will have a lot of long-tail contributors, but only a few core people that work on it.

This seems to be pretty typical in that sense. But is the majority of your contributors besides typos and minor bug fixes? Are they adding agents to the system? Is that the main way that people contribute to the project?

Yeah. I'd say that's the primary way. We do get a lot of small bug fixes, too, for the core system. But there's probably the bulk of the contributors have shown up added one agent that they needed for their use case in a core request, and then they've sort of disappeared.

Some of them show up again a year later with another agent or a bug fix because they've started to push the system. So, yeah, I think that's pretty much true. And one thing you mentioned is that you want to keep it as vanilla or as simple as possible for ease of deployment. And I would say that as an open source advocate and as a developer, as well as a person who's ran many websites and many services over the years, I'm almost allergic to some of these systems because of the maintenance.

And I would almost always use a provided service, even though I'm completely well aware that most of these startups disappear. And I've definitely had them disappear right out from under me. So, everything's a trade-off. But maintenance is a burden.

One that I still run to this day for my business is Airbit, which is a self-hosted version of Airbrake for error reporting. And it's just on Heroku, and they do their best. It's a great project. And I love it.

But I still have to update it. And I'm always afraid it's on MongoDB, which I'm not as familiar with from a maintenance perspective. And I'm just always afraid of this next time when they do a release. And I pull that tag down and push it out to Heroku.

All hell is going to break loose. And now my airy. And now my reporting system's broken. What measures have you taken to make deployment easy?

And just like that idea of not letting it die on the vine, like keeping up with the Hugen Joneses as a user? Have you guys struggled with that? And what have you done to make it so that people can deploy and not feel like it's a huge burden? That's a great question.

It's a hard problem because most people haven't deployed a Rails application before. And that's not trivial. Until fairly recently, we didn't have a great solution. We had tutorials on how to deploy a Rails app with Capistrano.

And we had example, Capistrano scripts, an example, nginx proxy scripts. But it was really for a pretty small set of people who were comfortable doing that. Recently, we've wrapped it in a Docker container. And that's definitely helped.

So now you can use you again with Docker. As long as you link it to an external data source, either Postgres or MySQL in another Docker container, then it's trivial to just sort of redound the new one and launch it. And it should be fine. We definitely make an effort to not break agents in a backwards incompatible way.

So we upgrade that we really try to make sure that the options that your agent has will still work. And we write migrations, if necessary, to make sure that happens. I don't think we've ever broken people in a backwards incompatible way. Except for, you know, I don't think we've ever had data loss issues where we've broken people with an upgrade.

Obviously, there have been bugs that have been introduced. It is an open source project. It's all volunteer. It does break sometimes.

But overall, it's been pretty stable. And I've tried to make very conservative technical choices. We haven't gone very far from traditional Rails. I've resisted pulling in new dependencies like Redis or Mongo, even though especially with Redis.

I'm a fan, but I just don't want people to have to deploy another component. It works with Postgres or MySQL, and we want to just keep it stable with whatever you have running. And the trade-off is then a little hard to make it high throughput and really performance if someone's trying to build a system around it. We have a few users who have really pushed it to its limits with tens of thousands of agents running really, really frequently and monitoring all very complicated flows.

I've never had a need to push it that far. They start to run into issues where they're extending their database, which is too small, or the system isn't high throughput enough. But for the most part, for the types of tasks that I've always wanted to use it for, which is a personal automation or sort of small scale business automation, it holds up pretty well and it's not too difficult to deploy. You can also push it to Heroku.

I think that works last time I checked, and so a bunch of people run it there. Heroku's restructuring of their pricing strategy has limited in the way people deploy hobby to Heroku because it can only be on for 18 hours. That's right. And Hugo is intended to run all the time, so that did affect a lot of our users.

But if you upgrade to their first plan, it's not too expensive. It runs fine at the base plan. It also runs fine on a $5 digital ocean image or something like that. So I mentioned keeping up with the Joneses, and I was just checking out your gem file previous to the call, and you're to be on the most recent version of Rails 4.

All projects that have multiple years go through different versions of Rails, all your dependencies that is complicated, enough application that your gem files, you know, got 100 plus lines in it. I'm not sure exactly how many gems are loaded on a regular basis, but you do have it, even though you're trying to keep it simple, you still have a pretty rich dependency graph. Can you explain any trials and tribulations over the years of just maintaining it, keeping it up to date, security patches? Has that been a struggle for you, or has it been pretty smooth?

It's not trivial, but because we're fairly traditional Rails, the Rails upgrades themselves have not been particularly hard. Sometimes the dependencies are a little more complicated, and we've always toyed with the idea of pulling agents into gems so that you can pick and choose exactly which ones you want. About two years ago, we refactored the gem files, so you can turn a bunch of gems off, and the agents gracefully just sort of disable themselves. So if you're trying to run this on a Raspberry Pi, which sort of barely works, you can turn a bunch of stuff off that you don't need just to reduce your overhead.

And that works fairly well. It's always been this trade-off between, I really have resisted pulling everything into gems, even though it has a sort of structural elegance, because right now, it's sort of the polished monolith where if we make a change to the system, we can update all the agents simultaneously in one commit, and it's going to work. If everything's in gems, then we end up in sort of version management hell, where we need to either own all the gems and update them all ourselves, or if they're owned by third parties, we have to send them a pull request or ask nicely, and I think it's just, I've been resistant to having to manage that. It's a little bit like my experience in the DevOps world between Chef and Ansible, where Ansible shifts with this really rich standard library.

My experience has been that when we moved Ansible, we got to stop using all these community supported things that didn't work super well from Chef and just use the core library that was updated in lockstep with Ansible itself, and it was just much more stable. So that's sort of where I'm trying to keep it. It does limit people's ability if I don't accept a pull request, because the agent brings in a lot of new dependencies that I don't think people are going to, most of the population isn't going to want. It makes it harder for people to get that.

They could obviously use a fork, but if it were in gems, it would be really easy for people to just add exactly which they want. So there's a real problem. You're trying to polish monolith. You might even call it a majestic monolith.

I heard that term recently. I heard that term recently. I heard it. Let me pose this question to you, because I'm looking at the MIT license.

I'm seeing the years of work that's gone into this, and it obviously is working quite well. You have many people using it. You mentioned that some people have come around and push it to certain limits, whether it's the number of agents they run or how often they check, or the types of things that they are doing. Have you ever considered the possibility of somebody coming by and saying, this is really nice as a basis for a new company?

I'm going to take you again. I'm going to wrap it with a new shiny new UI, and I'm going to start some new company, Inc. Is that something that you thought about? Is that something that scares you?

What are your thoughts on that? I would be totally fine with that. I kind of hope it happens. Yeah, I think that it would be great if there's a population of the users who grudgingly run a Rails app, but don't really want to, but they love the power of Eugen, because it is, in a lot of ways, more powerful than if this and that, or Zapier, because both of those you can't chain multiple agents together through a deep flow, as far as I know.

I haven't used either in about six months, but so you can do some very powerful flows of agents that you can't necessarily do somewhere else, and you also can extend it. So there's a population of people who would love to use it, but don't really know how to run a Rails app and don't really want to use Docker and just want to use Eugen. And I've been hesitant to start a business around it, because I don't think it's a huge business, and I don't particularly want to be in the hosting business myself, so although I am still slowly considering it, but if someone wants to take Eugen and build a product around it, and I'm aware of a couple people who have been doing that, I'm totally fine with that. That's great.

More users, more contributions. I mean, it would be awesome if they contribute back to some of their work, which is likely if they want to get the good wealth community. So I think that would be great. Awesome.

Let's talk about your community. There's lots of open source projects out there, and there's only so much limelight. You have selector gadget, which has a little bit of limelight, but like you said, never gained major traction. Eugen seems like it's got the traction.

It's a lot of people active, 110 contributors, like I said, 13,000 stars. People are using it, New York Times is using it, or has used it. Take us back to the launch and the initial reception, or if you had any delusions of grandeur, or if you had a marketing idea, like how did it get traction? So very similar to other projects that I've launched that eventually sort of got attention.

Eugen didn't get any attention when I initially released it. I wrote it, and I put it, I think I posted on Hacker News in March of 2013, or a little earlier, actually. I think it was like December of 2013, maybe, or of 2012, and got like two stars or something like that on Hacker News. It wasn't, no one cared.

But that's the exact same pattern that I followed in other projects. My free-definder site that I built in high school, I posted it at that point. I don't think Hacker News existed. It was Dig.

Dig was what everyone cared about. Yeah. So I posted on Dig, and it didn't go anywhere. And then I'm like, well, that sucks.

And then a month or two later, I reposted on Dig, talks at brands and devoting it up, and it got thousands of digs, I guess, and got really popular. So, you know, Hugo had a somewhat similar trajectory where I posted it. No one cared. I posted it again.

I don't think I, you know, both schemed in this case. I think I just posted it again in March of 2013. That time, for whatever reason, you know, it got to the front page, stayed there for a while, got some users at that point. And then I may have reposted it.

I know it was on Hacker News again in 2014, and I don't remember whether I posted it or someone else did. And then it actually was on Hacker News again to the front page last month, and I definitely didn't post it then, someone else did. So, you know, it keeps, I feel like every couple of years it gets rediscovered in another wave of users and contributors shows up. Again, also with Yahoo Pipes closing down, there was a blog post that talked about it.

And, you know, I've always made a real effort to write good readmees and to sort of invite people in and say, hey, this is everyone's welcome, and here's all the cool things you can do with this project, and we'd love to see what you do with it and to have an approachable readme. And that's always served me well, just like making it approachable. Yeah, I would just say I think this last round of Hacker News coverage was probably what spawned this show because he even had crossed my radar previously, and actually 2012 was probably Pete Hacker News for me, so I made one of your two upvotes there. I was living on the website back then.

Don't check it quite so often nowadays, but people do, and I think it started getting tweeted about again, and somebody mentioned it on our ping repo, and another person emailed us, and so it was kind of like these things just kind of bubble up, and I guess if you're interested enough and you stick with it because you haven't been working on it, it seems like maybe not nonstop, but in a committed way. For a few years, you just kind of get these different rounds of attention. Yeah, that's exactly what I've observed, and that's been true on multiple projects. It's a little hard to predict when people are going to care, and that as soon as a few people care and start talking about it, suddenly everyone notices, and everyone suddenly cares.

And it also needed a certain critical mass of agents and scale, I think, before it met enough use cases that people found it interesting. What does success look like for you again? You can look five years down the road and say, wow, that was a huge success. What would happen between now and then that would make that the case?

That's a good question. I've thought about trying to start a business around it, but I'm not moving in that direction right now. You don't want to be in the hosting business. I don't really want to be in the hosting business, and maybe I sort of have a blind spot, but I'm having trouble seeing a large enough business around it.

It's conceivable that there could be a pro version of Hugen. That's worked well for other open-source projects. I'm thinking about that. That might be one definition of success, but I certainly haven't committed to that yet.

I think a more general definition would be, it still exists, and it's still getting used, and it works. You mentioned longevity earlier, and I really care about survivable software. One of the things I want to build systems that I can build and slowly extend for many years into something that meets my needs. I would be content if Hugen is just continuing to get used and gets continuing to solve problems for people.

Yeah, I like that survivable software. You can go into marketing right now with that term. I think business-wise, just looking at it, MetaBase brings a bell with a recent show that we had in the open-source slash product business. Similar in certain ways, they're doing business intelligence.

They're exposing data to more people in enterprises and small businesses, and it seems like what Hugen provides is an opportunity at so much information that people just don't even know that they need it. But if you show that to them, they immediately see the value. Even myself, I look at this and think, oh, there's probably 10 things that I'm doing manually every week, or they're not even doing it all because they require too much of my time. That a product like this could solve for me, and then you take that time, all the small businesses, and people who actually can't write things themselves.

Because I tend to be the kind of guy that's like, oh, I'll just write this little one off screen each time, you know, to my shame. But many people don't even have that ability, and it seems like unless you want to take over the world, it seems like there'd be enough room for at least a small business. I think there's a huge education side of that, which is expensive, but maybe your zapiers and your ifts are doing some of the education for you. And so I think there's a possibility.

I think that's right. Yeah. And I'm definitely thinking about it. That would certainly be an exciting outcome, and I know some of my other core committers are definitely interested in that.

So we'll see. Yeah, tell me real quick. We're going to take a break. Tell me about your other core committers.

One thing that leads to survivable software or longevity or sustainability in Project is not having to do all the work yourself. And it seems like you have some people who are right up there. In fact, one fellow has more commits than you. Nowadays or total, even though I think you have more lines of code committed.

But how do you get these other contributors and how much are they meant to you? They've been incredibly important. I mean, it's been a team effort. Dominic and Akinori have been written a huge amount of code, especially Dominic, because recently it's been, he did all the Docker work and he's been contributing really important changes around how we handle files, which are upcoming.

In his case, I'm not entirely sure how Dominic first found Hugen. He definitely sent some early pull requests and clearly knew what he was doing and I invited him to be a committer if he would like to be in. And he joined. And then the same with Akinori.

He made some really important early changes. It's been a little less active recently. I think he's busy on his own projects, but he's certainly around. And then we have a couple other less active committers who show up occasionally and do documentation or help with some Docker stuff.

And we're always looking for more. I mean, really all of them. I'm just sort of keeping an eye on pull requests. And if I start to see the same person submit a few well tested, well written pull requests, I make an offer because the more the merrier, it's definitely, I completely agree with you that we need to spread out the load.

Cool. Let's pause here for our final break and we will be right back. Here at the Change Wall, we have two emails we'd love to subscribe to. The first is Change Wall Weekly.

And we've been shipping this email for several years now. We ship it every single Saturday morning. It's everything that hits our open source radar. It's our editorialized take on what happened this week.

It's open source and software development. Go to Change Wall dot com slash weekly to subscribe. And our second email is Change Wall Nightly. Every single night, we ship this email out covering all the top new and top star repos on GitHub at 10 p.m.

Central Time. It's all the latest stuff on GitHub before it blows up. It's often our own radar. We're often creating shows and finding new people, finding new projects, putting things on our own radar.

So we'd love for you to subscribe to that. Hit to Change Wall dot com slash nightly and now back to the show. All right. We are back with Andrew Cantino talking about Huguen.

Let's talk about the roadmap. What's on the immediate future? You mentioned you've had a lot of help with when you're going to contributors working on how you're going to deal with files, which I imagine is tricky and a large feature. Can you tell us about that and about other things that are coming down the pipeline?

Sure. So well, the file stuff has been entirely dominant. He's been figuring out sort of how he wants to handle it. The current plan is to use sort of the concept of a file pointer.

So events are JSON objects that flow between agents. And they're basically schemaless. So he's introducing a little bit of schema where if you declare a file, you annotate your agent as emitting file pointers, then we can look in the event for, I think that's just called file pointer, which will be a reference to either a remote S3 object or a local file object. Or I think there's a third case.

I think you could just put the raw binary or text data if it's small in the event. And then agents that know how to receive that, like a CSV parsing agent or a file pending agent can receive that and do things with it. So opening to the C and writing files kind of two separate things, but they interplay because would that play into an important export type of an idea as well? Are you referring to between systems?

Yeah, I was thinking between systems, but I guess you're thinking between agents. Yeah, well, or between systems because a lot of people use Huguen to, you know, when I post on Facebook, post on Twitter, that kind of thing. And there's often requests for and please move my photos. And that's not something we really can do very well right now.

Also often people are running Huguen in an environment where they can't write files locally, you know, Roku or Docker. And so we need to make sure that you can do things like S3 or remote blob stores. So then you can even do like timed or event based backups or something like that. Yeah, that would be interesting.

You certainly could fetch a photo on a schedule and do something with it. You know, make a, I don't know, make a time lapse or something would be cool. What else do you got? So files, I think that sounds like it opens up a world of possibilities.

Anything else that you guys definitely want to get done in the next six months, a year that you're thinking about? I think the two most important next steps for us are extending this concept. It's called a scenario. I haven't mentioned it yet, but you can take your agents and basically tag them with a label.

We call it a scenario and then you can export them and hand someone else a JSON file, which has a set of configured agents in them and they can import it and use it. And then one of the cool things is you can actually peer to peer subscribe to their scenario from your human instance. And if you click the update button, it'll go fetch the embedded URL for their scenario and their system, assuming it's public, and to do a diff and merge it into yours. So you can actually sort of subscribe to other people's agents in scenarios, which are basically just collections of agents.

The next step for that to really make it much more powerful would be to very belize it so that you can have a set of options that you fill out when you first subscribe to a scenario, such as your API key for something that isn't embedded already in the options of those agents or your personal location preference or something like that. Right now you could do it by editing the options of the agents once they've come in, and that mostly works, but it would be really cool to make a library of these, which leads to the second thing that we really need, which is a community site, to share these scenarios. Yeah, that was where I was just waiting for my turn to talk, because I have a sense of where your HQ or your place where people can just share their agents, and I love that you'd be able to just merge your own fields into one you're subscribed to, be super powerful, and actually kind of necessary if you're going to have that kind of sharing going on. Here's a random question that may seem off topic, but here we go anyways.

How do you guys deal with expiring off tokens when you're doing agent or background based tasks? Lots of times you know off token will expire, and then usually a user would have to get involved and refresh their browser to the redirect flow again. What does it do in those situations? So it depends on the system.

We use Omniaf embedded in Rails to manage the actual request for the OF2 token, and then it depends on the system. So for example, Twitter and Dropbox, I've never seen them expire. I think they have perpetual access tokens. Facebook definitely expires after it might be a month, it might be a couple months, and they don't offer refreshed tokens, so you do need to get involved.

It will just break. Agents have a concept of whether or not they're working, so they'll turn red if whatever that means for an agent, often it means they've successfully received an event or successfully created an event in a certain time window. They'll turn red, and then you can update them. They also have a log of their own errors.

Other agents that watch the agents? Yeah, there's agents that can control their agents either reconfiguring them or checking if they're right. We'll notify you that you need to come back and fix this thing. Yeah.

Cool. So that's where we've got files. We've got the community stuff. Anything else you want to mention before we move on?

I think the community site's the really big one where we certainly could use help if someone wanted to get involved. It's its own chunk of work, and it would be really exciting if it was tied back in some way to the core human system, so that you could either preview the networks of agents or, you know, have conversations about them and how they work, and it'd be even more interesting if it was distributed, so that, you know, if I run much like Hotline, which I mentioned earlier, if I run a tracker and it knows about a bunch of scenarios, someone else could replicate it with a feed and run their own, and it would be a little bit decentralized a little bit, so that we don't have to run a core one. Or you'd set up a commerce system, you could sell your agents, and then take your all the way back to the days of your pro scripts. There you go.

There you go. Surround selling little scripts, y'all. But open source wins. So, let's talk about roads to getting involved from two angles.

First of all, what's the happiest path to becoming a Hugon user? So deploying my instance and then setting it up, and then, secondly, from the development site, if I want to get involved from that angle, where do I start and where do I go? So start with as a user. So if you're a user and you're not planning to develop on Hugon, I would recommend either Docker or Heroku.