Go in the Modern Enterprise and Go Kit

Welcome back everyone. This is the Change Log and I'm your host, Adam Stikoviak. This is episode one, 63 on today's show. We got Peter Bagon joining us to talk about Go in the modern enterprise.

And today we're talking about GoKit, a Go toolkit for microservices. Great call today with Peter. And speaking of Peter, he's going to be at Go4Con along with us and so many other people. 1,500 people are going to be at Go4Con 2015.

So we're going to be there with cameras in hand, covering everything we possibly can with Go4Con. So say hello if you're there, hit us up on Twitter, whatever it takes. We got free awesome sponsors for this show, co-chip, digital ocean, and top-town. Our first sponsor is co-chip, a hosted continuous delivery service focused on speed, security, and customizability.

You can set up continuous integration in your application today in a matter of seconds and automatically deploy your code when you're tested past. Co-chip supports your GitHub and your Bitbucket projects, and you can get started today, totally and absolutely for free. She decided to go with a premium plan. You can save 20% off any plan you choose for three months by using our special code.

It's a change-all podcast. Use that code and save 20% off any plan you choose for three months head to co-chip.com slash the change-all to get started and now on to the show. All right, we're back. Peter, but not with us.

He's a software engineer living in Berlin, Germany. It's 10 at night over there, by the way. It's 3 p.m. on nighttime across this pond, but he focuses on large-scale distributed systems.

Peter, what can do the show? Thank you very much. Thanks for having me. So Peter, the conversation we're going to have is based around GoKit, but there's lots of other separate conversations that happen to describe why and what GoKit is and why you're doing it.

Before we do all that, for those who may not know who you are, can you describe what you do and what you work on makes in your background? Oh, sure. So I got my start actually in the telecom industry. I was doing embedded software like CNC++ on, I guess it was like routers and switches that you would normally stick into telecom switching centers at sort of the national level.

So I kind of grew up as an embedded networking guy. And then over some time, moved into more distributed systems at a higher application level. I worked at Bloomberg for a while. I did a search product there.

I sort of a federated search, and I worked for some NGOs for a little while. I worked at another telecom company doing monitoring product, sort of global scale monitoring product, and wound up at SoundCloud here in Berlin. And at SoundCloud, I worked on quite a few things. The search product there, I rebuilt that.

I moved into the stream activities feed kind of area, which I guess is something that a lot of web companies are doing in one way or another. I got into distributed systems theory there, and I did a couple of projects related to that. I guess the one that got the most attention in the world is a distributed sort of quasi time series database system called Roshi, which is based on CRDTs, which is a fun distributed systems data type conflict free replicated data types. It's all the rage.

It's the hipster data type industry system theory. And then after that, I, and currently I'm working for WeaveWorks, which is a company that does software defined networking for the container space, so for Docker containers. So we currently are giving away this product that can network Docker containers up together in the easiest way possible. And then we're building out stuff around that, basically.

Very interesting. I mean, speaking of Docker, and what Docker can't just happen, they just had a huge announcement about the app container spec becoming, I guess, federated is maybe a way to say it. Yeah, the two houses combined. Yeah, I mean, that's that's good news for the users.

So that must be some good news for Weave as well. Yeah, definitely. We're we've works as a company. Weave is the product products.

I was looking at the page, I'm like, I just said, Weave, I don't want to mispronounce it. I'm like, Okay, is that right or wrong? Yeah, this seems really interesting. So you've definitely had, you know, the full gamut of interesting software development experience embedded.

I mean, doing something on just a device itself, how different is that than what you're using now, which is distributed in, you know, multi-layered services, how much different is it to just sort of embed software onto a never, you know, never car wears like that. Yeah, I mean, there's similarities and differences. A lot of the theory applies pretty directly, actually, no matter what scale you're working at. I would say the major differences are the sorts of things you have to care about.

When you're focused on something so small, you your scope is narrowed, I would say. So you spend a lot of time counting all the bytes and counting all the ops. And when you're working at a broader level, that stuff sort of fades into the background and you're more concerned with correctness, especially in the face of failure. I think probably the biggest shift for me in my career is that I become more and more concerned with building correct things quickly or rather than sort of eking out the last little bit of performance, because that seems to be what's important for perhaps where the industry is going.

Like we can buy servers to scale horizontally, but if the systems are fundamentally broken, no amount of servers is going to fix that. So that's kind of in my experience. That's very true. So I guess the meat of this conversation, as we mentioned a little earlier, was really right around GoKit.

And this was originally a talk that you gave at Fosdam at the Google campus there in London at a meetup earlier this year. So back in February, so this is relatively a new idea or at least publicly a new idea? Yeah, yes and no. What happened there was Andrew Jurans who I know through the Go community reached out to me and asked me if I might have anything to say at Fosdam since I'm not too far away and I spoke in at Go events in the past.

And he kind of lifted up to me what I should say or what I wanted to talk about. And I kind of did a survey of my mind and the state of the sort of Go community at the time. And at the time I was working at SoundCloud and we had just sort of, or we were maybe in the middle of this period of change, maybe the story will resonate with a lot of people working in organizations where when we were kind of young and excited and full of optimism and opportunity, we had a lot of languages that different teams at the company were free to implement things in. And that was great.

It was like a Cambrian explosion and we were very productive and very happy. But we were small also. We were maybe 20, 50 up to 75 engineers or something when that was still going on. And so obviously that carries a cost as you get bigger, support costs go up and there's a natural tendency to kind of isolate to or like pare down to just a few sort of core supported languages.

For example, at Google, I believe the supported core languages are C++, Java, Python, and now go. But that was the most recent addition. So that's the choices that you have when you want to write something there. And so similarly at SoundCloud we started saying, you know, we can't really be putting Haskell services into production forever.

Let's think about where our strengths lie. Let's think about the sorts of things we want to be supporting long term. And it came down to a few final candidates. We had a lot of people who were into Scala, which was great.

A lot of services in Scala. We had a lot of people into Go. We built a lot of stuff and Go and it was running, you know, really quite well. There's a little bit of closure.

There's a little bit of Java. There's quite a lot of Ruby as these things go. And those were kind of the languages that we settled on. But that process continued.

And what I found was that the Scala camp was kind of winning. And more and more new things were being written in Scala, where Scala was being chosen for more and more and more new projects, where in my opinion, Go would have been an equally good or perhaps even better choice. So I started reflecting on why this was. And what I kind of came up with was that product managers and owners, people who weren't invested in the long term success of projects and teams, they wanted some sort of sense of security or support from a language, from a framework, from the technology decisions that they were investing in for a long time, right?

Like a year or two years, maybe indefinite sort of time horizon. And while Go was like proving itself technically, it wasn't mature enough. And it didn't quite have the library support that we needed in our architecture. So it was an easier or maybe safer choice to pick Scala, even though it might have performance characteristics that weren't as good as it could have been or might have a cost of complexity that was higher than it needed to be.

So that got me thinking and maybe realized maybe something that Go is missing is this collection of like higher order idioms and tools and a hesitate to use the word framework, but maybe something like a framework that can be used in these sort of, well, I call it a modern enterprise, these sort of organizations, like SoundCloud is to give these middle managers and technical leaders confidence in choosing Go a sort of long term language for their application layer, for their business logic. And so I sort of rolled that idea around in my head a little bit and decided that would probably make a pretty good talk. And GoKit was born. So Kit certainly makes a lot more sense when you reflect back on the idea of going framework or not framework.

And Kit certainly makes a lot more sense too, especially as you start to break it down into all these different components that may or may not fit into this blessed framework that somebody might use sort of piecemeal, right? It's sort of choose your own, choose your own things you need for your infrastructure, your architecture. Exactly. One of the fundamental ideas behind the GoKit is that we're coming in, we expect to be coming into an organization that already is a little bit big, successful, they have some inertia, they have some momentum, they have some existing infrastructure that they're not going to just throw away overnight.

We want to kind of slide in where Go makes sense and we want to work with the stuff that's already there. So yeah, we're not a framework that you have to buy into 100% and you can only talk to other GoKit services or something like that. We really want to be good neighbors and we want to be able to have a story that you can tell to your boss or to your engineering director that's like, look, we're just going to use a little bit right now, if it makes sense, we're going to, there's a future there. And I just want to make that future kind of look right.

Exactly. This also seems like a, not so much a fistfight, but definitely an attempt, a try, a fight for Go to win over Java or Scala or other options in the process. You mentioned closure, Ruby, and Java. So it seems like this is your attempt to, because you're, you seem to go champion and you want to use Go where you can use it and not let Scala or others win the battle for the next project.

The battle for Mindshare, yeah. I'm definitely a Go advocate and that's a personal thing that was immediately apparent to me when the initial release happened and the more energy I put into Go, the more I got out of it. It really just aligns with my personal philosophies and my preferences when it comes to programming. But I want to be, and of course this is going to sound a little political, but I'm really being honest when I say it's not for me a fight at all.

I don't want to see Go beat Java or Scala or any of these things. All I want to do is make Go a viable option for people who want to use it. And if there are Go champions at your organization who think that they can be productive and Go, we think that there are a set of services that would work really well with Go. I want to lower barriers to adoption from people who maybe don't know a lot about the language or would tend to stick with safer, safer choices like Java or the JVM.

So yeah, it's really not about winning so much as just getting Go to the same playing field, I would say. So for those, you mentioned middle managers, product managers that wanted security and support from the language, and you mentioned that Go hadn't been quite as mature as it could be to win some of those over. Since February and since you've gotten more of an advocate for Go, what have you learned about Go in terms of security and support from the language that's changed that would change those people's minds? So a lot of it I think is purely a matter of time.

I mean, Go has only been in the wild for I think five years. I think we're coming up on a five year anniversary if I'm not mistaken. Maybe six years now in 2009. I just had Andrew Duran on the show not long ago.

I think we're talking about the five years then because to rewind way back, I think episode, if I'm correct, I keep missing the number, but that was so three, I believe, of the change law we had Rob Pike on to talk about Go and that was November 27, 2009. So long before that, that Go was born. Exactly. That sounds true to me.

So yeah, five and a half years or something, which to someone who's been in the language like 100% basically from the beginning, that seems like a long time. But of course, to somebody who is responsible for the engineering department of a business, that's almost no time at all. So I totally understand that they are a bit perhaps cautious about investing in something so young on a broad time scale. And I think that's actually totally rational.

I think it's just a matter of time for a lot of people and they need to see not just one or two successful projects at super techy companies, but a series of successful projects and companies that are not so techy, that are consumer oriented, that are maybe B2B companies who can tell a good business success story from choosing Go rather than just a technical success story. And so I want to help the people who are capable of becoming those good stories, becoming Go advocates from a perhaps less technical and more business oriented kind of domain. So that's like my impetus here. Well, let's not have deep into Go kit, then let's figure out what it is while you built it.

And I guess what problems it aims to solve for the inputs modern enterprise. Yeah, exactly. So the most important piece of context to kind of make transparent is that I'm designing this explicitly for companies who have chosen to go with the so-called microservice architecture or service oriented architecture. And in my experience, and I think correctly, that is something that a company or organization should do only once they reach a certain size.

So we were talking about containers a little bit earlier. And often, I think containers and microservices end up in the same bag of tricks on the same like Hacker News homepage. And they are related, but I feel like they serve orthogonal kind of ends. They do different things.

Containers are a solution for a technical problem of continuous integration, of continuous deployment, of getting your actual code onto machines and running it in a predictable and reproducible way. And they're great for that, right? Whether it's, whether it's apps, open container format, Docker, like they solve this very technical problem. Microservices, on the other hands, they can't help with technical things, but fundamentally, they're not solving a technical problem.

They're solving an organizational problem. They're solving the problem of too many cooks in the kitchen, too many engineers wanting to touch the same fundamentally, the same software project. And what they allow you to do is decouple workflows and component life cycles in a way that increases the forgive the like agile speak for a moment, but increases the velocity of your organization. So in my opinion, companies that are adopting microservice architectures are of a minimum size.

And I sort of point this out on their GoKit website. They're probably at least 100 engineers. Maybe you can bring that down to 50, but probably not fewer than that. GoKit isn't really targeting organizations that are fewer than 50 engineers.

Not to say you can't use it, but that's just not where we're kind of focusing our energy. And it's at that scale, in my opinion, that microservices really begin to make sense when you can decouple teams from each other's life cycles when you don't have to have these deployment dependencies when you can sort of treat other teams and other services within your organization has in some sense a black box with an API that you can talk to and interact with to accomplish your own specific business goals. This is the context where GoKit is kind of designed to shine. So that's sort of a microservice architecture.

So let's pause there for just a second. So you mentioned microservices or service oriented architecture. Some other options you have, monolith, SOA, can, does it make sense to break down, I guess, the different options to developers and different options to those who are architecting services inside organizations that are 50, 200 or more developers? Yeah, sure.

Obviously, like the buzzword is monolith versus microservices. Right. It just leads out to a really awesome article, not Chad Fowler, Martin Fowler. I think Martin Fowler first, it was his title of it.

So it's catchy. Yeah, yeah. And I think he's on to something kind of glossing over the details in general, I would agree that if you're a small team trying to find market fit or whatever you're doing with your venture capital money, it makes a lot of sense to start with the monolith because the velocity you can get with four people crammed around a table is super high. And microservices carry real frictional costs that only make sense to pay when the benefits you get are bigger than the costs.

And that only happens when your team is quite large, I think. Yeah, so the question is like sort of the evolution, right? I think when teams start on a product or an idea, monolith makes a lot of sense. And to be clear, monolith is the idea that all of your code is deployed in effectively a single binary as a single application to one or more application servers.

And any change you make means you rebuild and redeploy the whole thing. Even if you just change the search function, you're going to rebuild and redeploy the entire application. And that's fine for a long time, actually. I think a lot of companies can stay in that mode for years until they grow to a certain size.

What ends up happening is when you get to a certain size changes that you want to apply. And I feel like I'm just kind of reciting a history now that everybody's already heard that a thousand times. But just for the sake of completeness, the changes that you want to make start conflicting with the changes that other people want to make. And you start having this sort of friction deployment related friction.

And at this point, you start thinking, wouldn't it be nice if I could deploy the search function totally independently of the user graph function or whatever the front end or whatever it may be. And when you start having these conversations around the water cooler, then microservices begin to make sense. There's a whole lot of costs that come with decoupling into independent processes that communicate over the network. The complexity is actually really, in my opinion, undervalued by people who are microservice champions.

But despite all that, I think they still do make sense once you get past a certain scale. And yes, that's the evolution as I see it. So that's that and then you got the, I guess the focus here you said is 50, is as far down as if you've actually been to enrolls, you said 100 first and then you say 50, but that's okay because we're trying to get go to an even playing field. And it's really focused towards the modern enterprise.

And you mentioned, if those out there listening to this have heard or talked, you mentioned that it's not comes like Apple, IBM, Bank of America, Goldman Sachs, things like that. It's not your large corporations that typically you might think of when you think modern enterprise is more like Google, Amazon, Twitter, Netflix, Facebook, Spotify, or even your previous on a model, which is SoundCloud. And they sort of set the tone for the industry in terms of the way they build software that is consumer focused and user experience focused and now you're growing your team. So when when you had this idea for go kit and it was back, it fuzzed them, you think, well, what can I talk about go kit is what was born, what exactly is go kit when you break it down?

Yeah, so go kit fundamentally is a collection of pieces that you play well together, but that you can kind of opt into one by one. And the idea is you are in an organization that's pretty big and you're in a team that is deploying a bunch of services or microservices to accomplish specific business goals. So maybe you have a search service, maybe you have a user service, maybe you have an authentication service or something like this. These services as they exist today, maybe written in a number of languages, maybe Ruby, maybe Scala, whatever.

And you have some idea that you'd like to use go. And I think that idea makes a lot of sense because go is in many, in my opinion, in many ways, sort of the perfect language for microservices. So what go kit is a collection of things you can kind of compose in and produce what I call a well-behaved microservice. And well-behaved means a lot of things.

It means well-behaved sort of inside of itself. That means proper logging, proper telemetry and instrumentation, proper life cycle management of components, stuff that keeps your process on your on your Linux system happy and keeps your telemetry and your metrics and sort of the business layer up to date and correct and that sort of thing. But playing nice also means talking, playing nice with other services in your infrastructure. And so here, there's a whole suite of things that are very subtle, very complex and can fail in a lot of terrible ways.

And this is the field of distributed programming or distributed systems theory. And go kit really simplifies a lot of it down. We choose as our messaging pattern, so-called messaging pattern, we're doing RPC right now. So if you know anything about distributed systems, you know, there's a lot of ways that processes can talk to other processes, pub sub, request response, async, message boss, blah, blah, blah.

There's a whole of these, like, zero MQ actually implements a lot of these and nano message, the spiritual successor implements, sort of a different set. We've chosen to isolate down and focus purely on RPC for the time being, emphasis on the time being. That's just sort of an optimization so that we can iterate quickly and get a product to market, so to speak, very quickly. So your service is going to play nicely on the system.

It's going to be an RPC service. It's going to call other services using the RPC messaging pattern. And it's going to play nicely with them. And so that means go get is going to give you idioms like circuit breakers on client side, rate limiters on both client and service, client and server side.

It's going to give you integration with service discovery components, whether you're using DNS records, DNS or V records, console at CD, however you, or just manual configuration, however you get services to find out about each other and talk to each other. We're going to have integrations for that. We're going to have integrations with distributed tracing systems. In fact, we already have an integration with Zipkin, which is the Twitter sort of distributed tracing infrastructure.

We have several more sort of planned. So things like this, things that are making services be well-behaved neighbors and these sort of infrastructures that I hope, and I hope other people will agree, is something that product owners and engineering directors want and need to see before they can kind of sign off and go as an implementation language. This stuff exists in Scala in a number of different ways. For example, Twitter has something called finagle, which is kind of what I'm driving towards.

It's a similar collection of components, load balancers and so forth that make services play nicely in these kind of environments. Netflix has a whole suite of tools. I guess the most important one to search would be something called ribbon, but there's also other correlated things that can compose together in the same way and kind of accomplish the same thing. Although I think most people believe that the Netflix stack is a little bit more tailor-made to the Netflix architecture.

Twitter stack is a bit more generic, and in that sense, Goka is aiming to be a bit more generic even than finagle. There's a couple other ones that do this piecemeal. Airbnb has something called smart stack, which is mostly concerned with service discovery and load balancing and that sort of thing, but that's a bit more infrastructure, a bit less libraries that you put in your service. So that's what Goka is a collection of these things that you can take in one by one, but they play nicely together and kind of give a positive story and a bright future to organizations that are buying in to go perhaps for the first time or perhaps just more than they were before.

Very interesting. So I guess the question I think I come up with after this is you mentioned a lot of big companies there with a lot of firepower, and I'm not saying you're one little guy, but I'm wondering what part we've worked plays into this, and I guess just your ability to build what needs to be built to compete, even on light scale to other organizations building some of the things in the languages, like why will you win? Why will this not so much win, but how will you succeed? Yeah, exactly.

One person, how are you doing it? Well, this really gets to the core of what I wanted to talk about today, actually, and what I'm really happy to be on the changelog for, which is the open source community, right? It's one of my favorite things about the Go community, and one of the things that probably has kept me around for longer than I've been part of anything in sort of the technologies here, is that the quality and the intelligence and the friendliness and just the level of people involved in the Go open source community has been one of the nicest things for me, and nothing I've seen in any other sort of sphere that I've been in. And so you asked me, I'm just one guy, that's definitely true.

This is absolutely a passion project. It's something that I want to see happen. It's not something that I'm being funded for. We've worked is not directly sponsoring any of this work.

It's really a nice and weak end sort of gig for me, and I'm motivated purely out of sort of this, hold on, I want to make sure I say the right word, but Netherlands. That's the good one, right? But Netherlands is the bad one. Yeah.

Okay. Oh, is it? There was a bad or good to educate me. Well, whichever one is good, that's the one I have in my heart.

Yeah. And so in combination, so I feel like my job is kind of to carry the banner and say, hey, if Go is going to get to the next level of its success, which I'm sure it will, and I hope it will. We're going to need stories like GoKit, we're going to need things like GoKit to get it there. We're going to need buy-in in sort of like the next tier of developers, which are the developers that are working at these somewhat larger companies that maybe aren't refreshing Hacker News eight times a day, and that still deserve the benefits that Go can provide.

So I'm going to be on myself as, at the moment, definitely the primary developer, but more of a sort of community leader, and I'm hoping with the help of the changelog, with the help of the talk that I'm going to do at GopherCon in about a week, and with the sort of publicity, for lack of a better word, than I'll be doing throughout the year, to attract people who have similar ideas. And I definitely got some of this when I gave the two talks in Brussels and in London. I was approached by several people in those communities and saying, you know, I've been saying the same thing to my guys at my organization, my girls at my university, and we really feel that the time is right for this, and we're excited to contribute. And indeed, we've had a couple of really great, powerful contributors so far.

Top of my mind is Chris Hines, who's been really incredible in helping me with the log package. We iterated on a few API designs. We're continuing to iterate. I think we're going to produce definitely, even if you use nothing else in GoKit, the GoKit log package is going to be the premier value ad log package in the Go universe.

And there's already, that's a very crowded field, so it's a big statement. But I think it's going to get there, absolutely. My friend here in Berlin, Tomassen Arts, has helped me a lot with iterating on the endpoint, API design, and the core components there. Canonical friend of mine, Roger Peppe, I hope I'm saying his name correctly, he's helped me a lot with the rate limiter.

And we have, like, a core stable of contributors, a couple of people from DigitalOcean are very interested as well and have helped me out. So, I hope this grows. And you're absolutely right to say, just one guy isn't going to be able to, you know, produce code at the same level that Twitter is able to do. But I hope that my instinct is correct and that this is something that people want.

And with a little bit of help, I'll be able to attract the kind of people that can make patches and push it forward with me. Absolutely. Well, that's what we're here for. And I'm glad you mentioned DigitalOcean because it is time for a sponsor break.

And I'm going to change the order of a top top top was supposed to be the first sponsor of the show. But, well, technically the slot number two, I'm going to move DigitalOcean up just because you mentioned them. So, we're going to take a break and hear from our friends at DigitalOcean who love this show and obviously love what Peter's doing with with GoKit and is supporting it. So, take a listen and we'll be back.

I have yet to meet a single person who doesn't love DigitalOcean. If you've tried DigitalOcean, you know how awesome it is. And here at the Chinese vlog, everything we have runs on blazing fast SSD cloud servers from DigitalOcean. And I want you to use the code CHANGELOG when you sign up today to get a free month, run a server with one gig of RAM and 30 gigs of SSD drive space totally for free on DigitalOcean.

Use the code CHANGELOG. Again, that code is CHANGELOG. Use that when you sign up for our new account, head to DigitalOcean.com to sign up and tell them the CHANGELOG sent you. All right, we're back.

And I guess Peter, since you mentioned DigitalOcean, we just had that super awesome sponsor spot for DigitalOcean. And you also mentioned your upcoming talk, which is we're recording this. Let's see what's today. Today is June 30th.

And we'll publish this on July 3rd. And on July 8th or 9th, I'm not sure which day you speak, you'll be at Go for calling to be talking about GoKit. So on one of those days, we'll be giving a talk going even more in depth. So if you're a CHANGELOG listener and you're not going to Go for calling it, there's still probably about half a second to buy a ticket.

Maybe they're all going, I don't know, but we'd love to see you there. And if you see us say hello, but we're work with GoforCon, we're doing video work within this year. If you see us run around, we got cameras in our hands, say hello. It's me, myself, which is me and myself, right?

We got Jared and we also have Donald, I call him DK. But if you see us around, say hello, well, every day we have plenty of them, but every day we'll be wearing a CHANGELOG teacher because what other, you know, from what we wear. But you'll be there. GoforCon is awesome.

1,500 people. How excited are you to come back over to the US? You were just in California, right? Yeah, that's right.

I was just there for DockerCon. Yeah. And I think that was, I don't know how many people that was, but it was on the same order of magnitude. Wow.

DockerCon is huge. It was really gigantic, gigantic conference. I'm looking forward to, I've never been to, well, I guess I have been a Rails Conf, stuff like that. And I think that might have borderline on like 800, 900-ish at the time.

Now it's much more. But it's been a while since I've been to such a large conference and then to be in the role of documenting what's happening, it kind of blows my mind that we'll have to like catch up with 1,500 people somehow. Yeah, it's going to be great. But anyway, it's enough to say, GoforCon is awesome.

You'll be there. You'll be speaking there. That's right. And that's yet another plug, but let's dive deep.

Now we're back from that break. Let's dive deep into GoKit and its components and their statuses and kind of what makes up GoKit itself, not just this idea of it. Yeah. Where should we begin there?

So probably conceptually the place that makes sense to start is package endpoint, which is kind of, there's not much there. I think there's like three things there. But the idea is to define a common interface or a common function signature that everything else can be built around. And this gets back to what I said.

The way we're starting is by assuming RPC, the RPC messaging pattern. So in package endpoint, we have what I was able to figure out is the lowest common denominator RPC method signature. And it's kind of what you'd expect. It takes a request return to a response.

But because it's in Go, it returns a response and an error. And because we need to thread a lot of things through this, it also takes a context. And a context is a thing that was released not too long ago, maybe a year, maybe a little bit more. From Google, it's this value add package.

And it is a little bit awkward when you first see it. But it's a parameter that you can thread through or rather you should thread through all of the functions or all of the points in your request path. And it allows you to do a lot of really handy stuff at a basic level, it allows you to thread information sort of across stack boundaries, across thread boundaries, across context, basically across bounded contexts within not only a single process, but from one service to another. And it also lets you set up nice, I forget exactly what the term is, sort of like a hierarchy of component ownership, process ownership, lifecycle ownership.

So if you're a component and you want to scatter down their 10 requests to 10 other components, you sort of can set a sort of timer send those requests off. And then the first one that comes back can, for example, cancel all the other ones and they'll terminate and clean up nicely. So the context is something that gives these sort of nice semantics. And it's been around out in the wild for I guess about a year.

A lot of things have made good use of it. And so GoKit has chosen that as its mechanism of taking information across context boundaries. So that's a good place to start. And once you get your head around that, everything else kind of opens up like a flower.

So you got a package endpoint in a particular where I'm reading down from the read me package log package metrics, Patrick package input, which we just talked about, I'll not say package anymore because it preferences all then but transport, circuit breaker, which we sort of talked about a little bit in theory earlier, load balancer rate limiting or rate limit, those are all implemented. And then in prototyping, you've got tracing, you've also got client patterns, which seems to be, I'm not really sure what that makes up it. And you got service discovery, which is impending. And then you got some other, I'm sure some other ideas, ad services is, is implement as well.

Yeah. So let's start at the top, I guess, log in metrics are maybe the simplest ones in the sense that they just do a very specific thing. Package log is like many other log packages kind of floating out there in the wild. It encodes some opinions about how microservices should do logging.

And I really have Chris Hines to thank for pretty much all of this, but something that we both agree on and was in the initial RFC, which was contributed by a digital edition guy, by the way, was that microservices or rather package log and go kit, enforces strictly the idea of structured logging. So if you're familiar with the standard go logging package, you can write log dot printf and then just sort of an arbitrary string. And it's go kits opinion that that's kind of bad practice. And what all of your logs should look like is key value pairs.

So if you wanted to log, for example, starting a thrift server on this host import with this particular piece of debug information, you wouldn't write that sentence out in a print off string and drop your variables in. Rather, you would log the structured data, transport equals thrift, address equals whatever it is, debug equals blah blah blah, and then log that sort of collection of pairs. And you do pay a cost in sort of human readability, but it's go to the opinion that that cost is more than made up for. And the ability to programmatically read those logs, parse those logs, and hopefully at 2 a.m.

got forbid, makes sense of those logs. So that's sort of the one core opinion about logging in. We have a lot of stuff that kind of wraps that core opinion and gives nice value add stuff, level logging is in pipeline, contextual logging, different output formats. Another important thing about package log is that it's not only for application logging, it's also equally usable for so-called structured log format data.

This is the kind of stuff that you might push into a Kafka instance. This is click tracking, this is general analytics, it's anything that needs to have a stronger sort of QOS than simple application logging that might end up in an elk stack and elastics or log stash kabana stack. So log is good for all of these things. And a natural sister or brother would be metrics.

Precisely. And metrics is also simple in the sense that it does sort of one thing that's independent from other things. It lives in your process. And actually, maybe we should take a step back and ask what is metrics, what is instrumentation?

Lots of people have. I love it. I should do it. Yeah.

There are different ideas. There are services like airbreak, right, where you sort of trap all the exceptions or errors in your code. And then whenever you see one, you emit a piece of information to a third party server that collects them and tells you what parts of your system are crashing or whatever. That's kind of a type of instrumentation, I guess.

But it's not what metrics is. Something a bit closer would be a system like graphite or satsy where rather than trapping errors, what you're doing is going through your code and you're instrumenting all of the important bits. And what is an important bit? And one thing might be the number of requests that kit you on an HTTP service, the duration of those requests.

So average mean, mean, median, max, man, this sort of thing. Not only the basic sort of statistics, but also bucketed over quantiles. So mean 50th percentile latency, 99th percentile latency, this sort of thing. So that's kind of more what I'm getting at here.

And metrics, package metrics in GoKit represent sort of a distilled, boiled down version of what I and many of my contributors have found is important. We expose three core concepts. That is the counter, the histogram, and the gauge. And we provide different backends that implement each of these counters, or each of these metrics, you can put them all together in a single sort of API, you can use one or as many as you want.

And so the idea is you should aggressively instrument your code using the package metric sort of interfaces. And then once the program started up in your phone domain, you wire up the interface to the back end of your choice. And this is in keeping with our sort of philosophy, GoKit philosophy of working with the infrastructure that you have, you almost certainly as an organization are going to have a stats D server or graphite server, or whatever it is, existing and receiving metrics if you're a team of 50 engineers. So we want to work with that.

And we have adapters for many of the common ones. The one I'll plug is Prometheus. It was also developed at SoundCloud. And yeah, we think that's probably the best model for this kind of thing.

We've done a little bit more coverage on it too. Definitely. If you need to ensure, let me know. I know the developers quite well, and they're also going to be a go for come.

Well, that would be awesome. Let's make it happen. Yeah, definitely. No, it's a really cool piece of software.

And it does its job quite well. So a data is into some of the most common metric packages. It's an exp bar, as I pronounce it. Yeah, as far as one of the standard library packages in Go.

And it's really like bare-bones simple stuff, but pretty cool. You can export kind of instantaneous view of certain types of metrics, and just kind of dump them by hitting a specific HTTP endpoint. So it's a good like entry-level bare-bones type of exposition. And development or something like that.

Yeah, sure. Sure. Okay. Very cool.

That makes sense. So counters, gauges, histograms, those all dump data into these known metrics packages that help you sort of dive deeper into them as you need to. What's next? So if we walk down the stack, we can kind of start looking at the value add components.

And this is stuff that benefits from the common endpoint API or the common endpoint interface. This is things like circuit breaker load balance or rate limiting. And these are things that you probably can intuitively guess that a microservice needs. But it actually turns out it takes some thinking to get it right.

And this is part of the value proposition of GoKit. We have contributors who have thought about it and have made mistakes implementing circuit breakers and load balancers and prevent you from making those same mistakes, hopefully. So each of them is something that you want to wire into your microservice, either at the client side when you're connecting to other services or on the server side when you're receiving connections from other services. And they prevent you from behaving badly.

So let's start with circuit breaker. This is something that you would typically put in the client side. And what it does is, if it detects that requests to a specific backend to a specific other service are failing regularly above some threshold, let's say, it will prevent other requests from going out until it detects sort of a healthy state. And the idea here, the reason it's called circuit breakers, it's kind of like a fuse, right?

If you put too many amps through a fuse, it's going to explode and prevent you from starting a fire in your house, right? Great naming. I like the naming. Yeah, it's a cool name.

It's a pretty common pattern. But until you've had this sort of thundering herd where a failure in one service brings down your entire company and it takes like hours to get everything restarted and then back online, you don't really understand how important it is to have these things. And it is important and go get shipped with one. Actually, several, you can kind of pick your favorite type of implementation and wire it into hopefully every request that you make.

And it's important to note that circuit breakers aren't really a mechanism of solving load problems. So if you're under provisioned in your infrastructure, circuit breakers aren't going to fix that. What they do is prevent bad problems from becoming terrible problems. And for that reason, they're very important.

Does this tie in to load balancer then? Yeah, in a way, you would certainly wire circuit breakers and load balancers together. Where circuit breaker is kind of something that only comes in play when things go wrong. Load balancer is something that's in play all the time when things are going right.

The idea is that if you have a set of services that are horizontally scaled, multiple instances have the same thing to support the amounts that the millions of requests per second that I'm sure your startup is getting. Load balancer is something you're going to install in your client side to distribute the load across all of those instances. And there's a lot of ways you can do that. You can do very simple things like picking a random instance every time you can do well-known algorithms like Robin.

You can use an algorithm that sort of weights each instance based on criteria like average response times. You send more requests to the healthier instances. You can weight them based on locality, based on data center. So package load balancer is something that allows you to implement those types of algorithms across sets of otherwise identical instances of services.

And we provide a lot of hooks for getting sets of instances. And so this is sort of the service discovery component. Given you know you want to talk to the user service and how do you translate the string user service to a set of instances? Well, this is a whole probably a multi-hour change log podcast in itself.

But we provide hooks for a number of common solutions to that problem. And so that's also wrapped up into the load balancer package. How is that different from rate limiting? So rate limiting is something that you can stick either on the client side or the server side of your service.

And it's something that maybe is a little bit less useful or at least less generally applicable. But if you know that, for example, you're calling out to the Facebook API and you know that you don't want to do more than 10 requests a minute or 100 requests a second or whatever it happens to be, you can declare that front. You can stick a rate limiter on that client connection. And you can say, I only want to do 100 requests a second max.

And then you can determine what to do with any requests that happen to go above that limit. You can kill them immediately with an error. You can tell them to wait a while, whatever is right for you. That package is pretty barebuns at the moment.

But there's hooks for doing things like cooking into a central lock store so that you can enforce, for example, a consistent rate limit across multiple instances. Yeah, this guy's the limit there. But that's what that does. So just to summarize, we've got a lot of metrics, endpoint, transport, circuit breaker, load balancer, and rate limit.

And those are all implemented and ready to use today. And tracing is in part of that being staged along with client patterns. Can we talk through those two pieces there? Sure.

So tracing is something that maybe a lot of people, some people know about some people don't. The idea is when you have a microservice architecture and you have a bunch of things talking to each other. Typically, what happens is, a request is going to hit your website, for example, it's going to go to some service. That service is going to need information from a bunch of other services.

Each of them in turn may need information from more services. And so what happens is you create this sort of tree, this sort of hierarchy of requests that was all spawned by a single incoming request. And once you get past a certain size or a certain level of complexity, it's really important to be able to look at that call graph and see how long each individual thing takes, where your hotspots are, this sort of thing. And the way to do that is with a so-called distributed tracing framework.

The canonical one in the open source world is, well, I should take a step back. This all became sort of public knowledge quite a long time ago, I think maybe 10 years ago at this point, when Google released a paper called Dapper, Dapper is the name of the Google internal system that does this. And not to say it was the first system that does this, but that sort of became en vogue. And so in Google fashion, they released a paper but not the implementation.

The Apache folks, I think, maybe it was Twitter initially. I don't have my history right on this. Somebody said about creating an open source implementation of the Dapper paper, which when completely called Zipkin, and Zipkin is what you can kind of download and use today if you're running on the JVM. There's a number, I guess I was three or four years ago that that became public knowledge.

There's a number of similar systems now in other languages. There's a number of Zipkin implementations in other languages. And so GoKit provides a package tracing that does this sort of thing. We have a Zipkin implementation at the moment.

So if you have a Zipkin infrastructure in your organization, we can interact with that. We have planned support for a similar system from a company called Sourcegraph, which is also another prominent member of the Go community. They have a system called AppDash, which does basically the same thing as far as I understand. I plan on tackling that at some point in the near future.

And there's actually a couple of other ones. There's another Apache project called HTrace. There's something from Netflix with a funny name that begins with us that I can't remember. Actually, during GopherCon, there's a distributed tracing working group that's going on in Budapest.

Unfortunately, I'm not going to be able to be a part of, but they're sort of having a symposium and talking about the future of this sort of thing. So keep an eye out in the next month or two. I'm sure we'll see some interesting news there. Very interesting.

So a lot of stuff happening around that. I mean, especially the support with a lot of new things I've heard there too, Zipkin, AppDash, Dapper. And like you mentioned, it is similar to how Kubernetes came out recently. It came out as a paper version.

Well, sort of what was a Kubernetes, the same thing, but then the other thing, they recently announced similar Borg. Yeah, Borg. That's what I was saying. I think it helped me there.

It's like they released this paper, but then you always have Kubernetes, which is similar to what's happened with Borg. Exactly. I think they would say that Kubernetes is like the if they were able to redo Borg and fix all of the problems that they discovered that that would be what Kubernetes is and kind of made it a bit more straightforward for the open source kind of public world at large. Yeah, exactly.

You got Zipkin, AppDash, and what was the other one? HTrace. HTrace. HTrace.

And then there was another thing you mentioned about that one down. Yeah, I didn't get a chance to remember the name of it either. All right. We'll go back a list to the show notes.

I'll just think of the show notes sakes to make sure we get it all in there. Yeah. Let's see where we at on the list here. I guess I close that list.

I close that list. Let me get back to the reviews I can get back there. Next is so client patterns. Exactly.

Well, this has no link. There's no description. Help me out. So, yeah, the idea is you write a service and your service has an implementation and people can call it.

And this is something I haven't mentioned yet, actually, but it's actually also pretty core to the go kit idea. You're going to have a service and it's going to be implemented and go. And you're going to do your implementation once. But you probably want to be able to expose that service on any number of different transports.

Maybe your company is using thrift exclusively behind the scenes. Maybe it's using just standard HTTP JSON semantics. Maybe it's using bleeding edge GRPC, which is actually very similar to Kubernetes. It's like the open source, the Google open source version of an internal project called stubby, an internal RPC framework.

So, I don't know, we have a bunch of so-called transports that allow you to expose your service in different ways. And a core idea of go kit is that you should write your implementation once using sort of the RPC pattern. But then you should be able to expose it on an arbitrary number of transports simultaneously in the same process. And so, yeah, that's an important thing that I sort of forgot to mention.

So, in support of that, you have the service running on some transport. And you can hand write the code to talk to that. If it's HTTP JSON, it's probably pretty straightforward. But if it's a thrift server, for example, it's a little bit more laborious.

And so, client patterns is just my way of saying, given your exposing a service on any one of our supported transports, we want to be able to show you an example client you can build that gives you the same semantics, the same go language level RPC semantics to talk to that service over the chosen transport. So, if you have some service that adds two numbers together and you expose it on thrift, gRPC, and HTTP, then we want to be able to say you can create a thrift, gRPC, or HTTP client that can talk to that service. And you just get the really straightforward, basic call response semantics on it. So, client patterns are just building those things up, basically, and making good examples.

It sounds like service discovery might play a little bit into that in terms of discovering different services. Exactly, exactly. So, yeah, it's a bit of a tautology. But, yeah, precisely, you're going to, as part of a client package, you're going to figure out how to get to the services you want to talk to.

And that's a question that's answered differently in different infrastructures. But, yeah, absolutely, you're going to wire in service discovery into your load balancer all on the client side and then use that in combination with a circuit breaker or whatever else to talk to those remote services. That's interesting. I mean, as we walk down this entire component status list and everything that you're doing with it, it starts to really make a lot more sense in terms of what GoKit is trying to accomplish.

And, reminding all the listeners too that, you know, quite a bit of this is implemented. We didn't even talk about the ePAST ability documents and that piece yet there. But, if you want to talk about ad service real quick and then just sort of go through the ePAST ability, which also talk about the status that you got adopted, implemented, prototyping, pending. So, several different statuses that sort of trick you to navigate around as an outsider.

Yeah, yeah. And that's totally my bad. This is also sort of in a pre-office state. And the words don't really mean a lot.

Well, you're one person, we're trying to give you some slack here. So, I want to repeat the fire, Peter. Yeah, much appreciated. Yeah, it's just a sort of a signal to anyone who might stumble by us to where we are sort of in a percentage wise in accomplishing the goals we set out for ourselves.

So, yeah, pretty much everything is implemented with a basic API. We're not, personally, GoKit itself is not in an API stable state. If someone came along and filed an issue and said, Hey, here's a much better API for the metrics package, take a look. If indeed they're right, then that's going to change.

And, yeah, we're going to be that way for a little while. So, this is definitely like alpha quality stuff right now, alpha stage stuff. But that said, the example ad service is sort of the proving ground for a lot of this. It's just a directory of a fake microservice that implements all of the transports, the implements, all of the little value ad composed in features, just as a way to feel, get a sense of how all these things play together.

And it's really an opportunity for me to see if I'm accomplishing my goals with this thing. If I design my APIs correctly, this stuff should sort of fall out naturally and feel very good to write on the page. And if that's not the case, I'm going to see it in ad service and I'm going to need to change things. That's interesting.

So, the ad service, essentially, is a good example for anyone following up with steps in this conversation to look at how it should be implemented, how it should work. Exactly. And is your working sort of test proving ground? Yeah, exactly.

And I think the thing I'm most proud about is that when you look at the main function in ad service, there's no magic at all. It's very comparatively long, but it's all straightforward. There's no package globals that get sort of magically pulled in. Everything is composed in this very declarative bump, bump, bump, step by step way that is really, for me, a joy to read.

It's so easy to figure out what's being wired together, how the information is flowing through the object graph. This is really nice to me and something I'm really seeking to preserve. On the notion of the API stability, you'd mention that the API is stable, but it's not like, in quotes, stable to the point where you couldn't come by and change it if someone wanted to listen to this podcast or pick up what you're doing and step in and help out in some way. Does it make sense to talk about that API stable quote you have on the docs there?

Yeah. Do you know how hard do we read it for you? No, no, I've got it in front of me too. And this is actually sort of a conversation that touches in the broader Go ecosystem.

One of the most long-standing sort of trouble spots or maybe concerns in the Go world is this idea of how do you manage dependencies and how do you make reproducible builds and how do you enforce things like API stability policies. And there's been a number of solutions to that that have been more or less successful. Luckily in Go 1.5, which is due for release, I think in about a month, we have sort of a canonized, blessed approach to this process of so-called rendering that's going to be baked into the Go tool. But for a Go kit, really, the API stability policy comes in two parts.

It says, first of all, if we're going to bring in a dependency, we prefer to have one that has a stable API. So that users of our software don't get unexpected breakage whenever they import us and we in turn import something else. So we prefer to do that given a choice between two packages. We'll pick the one with a stable API.

The other side of the coin is the API of Go kit itself is not currently stable. So if you want to use Go kit, what you should do is vendor it in. And this is sort of a process by which in your service, you shouldn't import GitHub Go kit kit sort of directly. You should use some sort of rendering tool or some sort of rendering process to vendor that code into your repo directly and use it from that place so that you control the lifecycle.

I guess let's take a pause there. We'll have one last sponsor break when we come back. We'll talk a bit, I guess, about the closing piece of it. I want to talk to you about the potential working group and what's going on at Go for Con to see if there's any sort of get-together for Go kit enthusiasts or those who want to step up and help out.

So let's break. We'll come back and we'll talk about that. TopTow is by far the best place to work as a freelance software developer. I had a chance to sit down and talk with Brendan Beneschott, the co-founder and COO of TopTow and I had some Brendan to share some details about the foundation of TopTow, what makes TopTow different and what makes their network of the engineers so strong.

Take a listen. I'm one of the co-founders and I'm an engineer. I studied chemical engineering and to pay for this super expensive degree. I was freelancing as a software developer the by the time I finished realized that being a software developer was pretty awesome and so I kept doing that.

And my co-founder is in a similar situation as well and so we wanted to solve a problem as engineers and do it from as a network of engineers kind of four engineers by engineers and having that perspective and consistently bringing on new team members who also share this really makes TopTow different and that if the network of engineers not kind of like you have TopTow and then the developers it's never about us and them it's always us like everybody at TopTow for the most part refers to TopTow as their company and they feel like it's their company and everybody acts like a core team member even though they're freelancers within the TopTow network and all of these things are extremely important to us. Alright if you're interested in learning more about what TopTow is all about at to toptow.com slash developers that's t-o-p-t-a-l dot com slash developers to learn more and make sure you tell them to change what I've sent you. Alright we're back I guess Peter this has been quite an enlightening trip down go kit slash go slash microservices lane. Good here.

I know I've certainly learned quite a bit and I'm thinking for the listener's sake you know how can they get involved and that's one of our quick questions after telling this podcast which we'll get to here in just a minute or so but I'm thinking is our working group at Fozdom you got a lot of enthusiasm wrong go kit and I'm one of what's happened since then that that was February this is obviously June now so not too much time but enough to have some things change and get more you got Chris and you got some others who else is sort of stepping up and what's happening in terms of a working group. Yeah so yeah there was a lot of ideas that came into the into the into the sphere right away we set up a mailing list and so that would be a good place to start you can see that sort of I think at the top of the read me or the top of the website you can subscribe to that is just a typical Google group mailing list and so that's that's good for a sort of longer form discussion there's also a slack channel which we set up recently on the gophers.slack.com slack group is that what's called slack organization? I have no idea what to call these days I just they're just out there so many slack rooms there's so many things so many but yeah that's actually a well managed and moderated slack group and there's a channel in there go kit you have to get an invite for that but invites are freely available and I provide a link for invites on the go kit website as well. Go for that right now quick second right now gophers.slack.com okay and if you want to invite that is bit.ly slash go dash slack dash sign up and that'll get you one in an email right away as far as I'm aware.

Awesome. Yeah so those are the online forums I'm pretty active on this thing so you have ideas that's probably the easiest way to reach me and there's also several contributors that hang out there. In terms of go for con we're definitely going to do a go kit sort of hack day on the hack day I think that's going to be Wednesday is that right? Thursday is Tuesday is the is the training I feel at the con workshops and so that's Tuesday Wednesday Thursday is the conference days and Friday is that right?

Yeah so I was at the hack day last year and that was actually probably the coolest part of the conference because it was an opportunity to meet people you kind of only knew by their Twitter or GitHub handles or whatever and the level of stuff that was produced there was actually really cool. I'm not really much of a hackathon guy myself but the hack day was a much different experience than that it was a bit more heads down bit more focused and really like a good a good sort of atmosphere and a really productive space so I'm really looking forward to that we're gonna have a good table. We had Eric and Brown on the show recently and they on that note they said that you know you mentioned hackathon that it's not hackathon at all it's more like for those 1500 people come to the conference you got all these speakers and people there that are building and contributing into the go ecosystem this is a chance to sit down with you know you're like you mentioned the avatar you only know or your hero so to speak and sit down and hack with them on your favorite library or ask them questions and sort of fill with code together and that's more or less what the is that what you got out of as well? Yeah definitely I met some really cool people who there for the first time who I'm still talking to you on a weekly basis today and I'm looking forward to meeting them again this year.

Very cool so yeah Chris signs mentioned in the millis room that your millis that a go kit birds what feather might happen so is that around the same hack day yes first I understand I don't know if there's a formal process if there is I'll look into it and I'll be sure to set something up but I think it's pretty informal maybe day of I'll set something up and I'll make myself visible so yeah if anybody's at all interested this show up in the right room and you'll be able to find me I'll make sure that's possible and while we're talking about gopher con if you're listening to this you're not a gopher con but you know somebody who is and they listen to this show and they want to get on camera they want to say hello we'll be at the first after party the second after party the hack day pretty much everything you see there we'll be there cameras in hand the change blog has a new division called change blog films we're going to conferences and helping conferences like gopher con document what's going on in the community and it's a huge part of what we're moving towards so if you're listening to this you're not there hopefully we'll see you next year but if you've got friends that are there hit them up let them know we're there if they don't already know and tell them come check us out because we want to uh we want to see everybody and we'll have you on camera as well at some point Peter so you can't uh you can't hide you can't hide and I guess uh it's true fashion at the end of the show let's let's wrap with two awesome questions the first question is if we haven't already said it obviously we've talked quite a bit about several things inside of go and go kit but for those out there that are thinking man this is super cool how can I get involved what is the where are the places you need help to make go kit a real thing and keep moving forward and get better adoption yeah so this is a great question um if you're an accomplished go developer and everything i've been saying is really like struck a chord with you then uh dive right into the issues list i've making i've been making an effort and i'll continue to do uh to make an effort to um sort of keep my roadmap up to date in there and um have a list of things that need to be implemented that i'll try to make as straightforward as possible if you want to claim ownership of them please feel free a lot of them is pretty straightforward a lot of them are going to be a little bit subtle but you know it's nothing we we can't like talk through so if you're uh go for already and you want to contribute that would be amazing um but that's not even really necessary the thing that i think would help quite a lot and especially this early stages if you're somebody in an organization and you want to use go but you're feeling some friction or you're you're you're missing something in ecosystem something that go kit might be able to provide or even help provide um filing a sheet with that let me know somehow and let me sort of fold it into my like idea of what this thing is that i'm building um use cases like that user stories uh sorry again for the agile terminology i've been working in startups for too long i guess um but information like that is super super helpful to me then i'll also don't forget the gophers.slack.com too so you said you're there all the time and so i guess issues would be a good place to go and kind of uh find things off yourself but at the same time get an invite yeah because that's library very nicely awesome well cool and i guess our our final question which to some i don't know peter you might like as well but some absolutely hate this question some love it we'll see what line you fall upon but uh we're curious who your programming hero or heroes are yeah that's interesting it's sort of like with me it's it's like a favorite like i try not to have favorites i try not to do things like that but i will say that there's been several people in my like career that have been very heroic to me and it's not specific people or specific personalities rather it's been people who have been mentors to me uh not only when i was in school or fresh out of school although those people have played like a really important role in in growing me and and making me a better person and a better programmer and all these fun things but even like on a day-to-day basis today people who take time out of their day to contribute to me and to like lend a little wisdom to me if if you can get into a mentor mentee is that right apprentice master sort of thing this kind of relationship with somebody in a professional context i think this is really the way that information flows this is really the way people get better and to everyone who's ever been a mentor to me that's people like Gary Sumar back at Bloomberg Sean Treadway at SoundCloud more people than i could possibly name actually but these are the people that are like heroes to me and the people that have really allowed me to level up and and get to where i am today speaking of mentoring and menting if that's i'm not sure that's the thing i'm maybe not i'm gonna follow your lead on that one are you mentoring anybody uh not sort of officially right now and that's a great sort of call out of me i should really find a way to do that um yeah i'm gonna i'm gonna look into the community here because Berlin is a great place for this lots of junior devs lots of people looking to get into the industry so yeah it's definitely something i could look into yeah uh i'm trying to remember the the ads what i did for the resolution but uh in that ads what i talked about the huge community for um startups there because they just opened up uh f-r-a-1 yeah french version so that yeah it's on that trying to remember the words for some some exchange some exchange it's huge that feeds basically all of your up and it makes it super fast that's where the digitalization servers are at yeah um not the plug them much more but you can maybe think about how that spot talked about the thriving start community there and ecosystem there so yeah definitely and when they came into their press tour i was actually like helping them coordinate that there's a lot of fun stuff happening in german and berlin especially so yeah definitely and you said there's some folks at digital ocean helping with go kit um yeah there's a couple of infrastructure engineers who've contributed to the log package a bit to the uh to the server side stuff um with luck uh they've already contributed at sort of a staging testing uh server for me to help me test the zipkin stuff it turns out if you want to start as a consumer you need a pretty beefy machine uh zipkin is written in on the jdm and it uses a lot of memory more than my little thinky uh vps can handle so thanks to them for that um with luck that'll continue uh yes a digitalization has been a great partner for go kit so far i guess the one question that eluded my mind to even think about but mentioned is ocean there and asked about their uh work into the into the software itself is who is there anybody out there who's adopted go kit so far and using go kit in the wild or even in like a test phase yeah um there's a couple of organizations that i can't really name by name for a variety of reasons that are using pieces of go kit um the the log package is attracting a lot of attention metrics is attracting a lot of attention because these are sort of the most feature complete kind of self-contained packages at the moment um but yeah a couple of others are starting to play around with um structuring microservices using go kit components and then hopefully the goal is their feedback is going to drive further go kit development so that's the idea fantastic well peter it's been such an honor to hang on the show me i know we've been what playing twitter dm tag for a little bit and then email a little bit and then you were traveling and then it was a good time for us so we finally got you on we wanted to get you on the show after we had that conversation with andrew yeah uh because that sort of set the the new tone for go on the show go to large yeah exactly yep we sort of stepped back into go on a year to your bases and you know we started with rob pike by himself way back and go very first started and then about two years back had andrew and rob back on and then we had andrew back on and i knew that it was time to to talk to you because he was like hey when i'm done on this show you got to get peter online because he's got something very cool happening that everyone in the go can he has to know about nice so so there you go so you got to get all the slap on the back and blessing from andrew for that as well good to hear and i hope to hear even more go people on the changelog in the future i'm definitely gonna tune in if that happens well you definitely want to talk about bermiti at some point so when we get off your lap to get some it's gonna happen in your future but peter it's it's been awesome any least you want to mention as we close here to follow you on you got your github.com slash peter and your last name anywhere else it's best to kind of catch up with yeah yeah i mean i'm probably easiest to reach on twitter that's just my full name peter gone no spaces um github sorry go kit.io takes you to the github repo and um yeah i guess that's that's about it i'm pretty pretty simple guy and for listeners sake we'll have all those in the show notes so this is episode 163 of the changelog go to changelog.com slash one six three you'll find all the show notes and all the details about everything we talked here so don't feel like you got to wreck your car if you're listening in the car or jumping out of that airplane to get to i don't know just make funny jokes but don't go crazy just go to the show notes everything's there for you we make it easy but peter thanks so much for doing us today on the show and let's let's say goodbye all right yeah no thank you very much for the pleasures online and i had a lot of fun