Episode #58: Observing Serverless Observability with Erica Windisch

About Erica Windisch Erica Windisch was the co-founder and CTO of IOpipe where she helped organizations maximize the benefits of serverless applications by minimizing debug time and providing correlation between business intelligence and application metrics. She is now a Principal Engineer at New Relic. As an advocate and pioneer of cloud computing since 2001, Erica is always pushing forward as technology and the industry adapt. She was an early contributor to OpenStack and maintainer of the Docker project where she worked on hardening Linux containers and establishing corporate and community security policies. Erica is a champion of AWS Lambda and serverless technologies, and she speaks frequently at conferences about AWS Lambda and other AWS solutions. She's passionate about systems architecture, security, and the future she sees for machine-automated, low-code development. Twitter: twitter.com/ewindisch New Relic: newrelic.com Personal email: [email protected] Professional email: [email protected] Watch this episode on YouTube: https://youtu.be/T1t_P_zqOiE Transcript Jeremy: Hi, everyone. I'm Jeremy Daly and this Serverless Chats. Today I'm joined by Erica Windisch. Hey, Erica, thanks for joining me. Erica: Hello. Hi. Thanks for having me. Or thank you for having me. Jeremy: So you are a principal engineer and architect at New Relic. And you're also an AWS Serverless Hero. So I'd love it if you could tell the listeners a bit about your background and what you've been doing at New Relic. Erica: Oh, gosh. Okay, well, my background is pretty deep. So, I'm at New Relic now. Before New Relic, I was the founder and CTO of IOpipe, which was an observability product for serverless applications. Now, I am working as an architect and principal engineer for New Relic. And if we're going to rewind history a little bit. I previously was a security engineer working at Docker, where I founded their security team and their security processes. I was involved in OpenStack from very early, since its founding. And then before that I had actually had my first company and we had built a cloud. We actually had our own cloud services. We were building from 2003 actually, building out horizontally scalable cloud services. And I said, "Well." We bought really early into that pets versus cattle idea. Jeremy: Nice, nice. Well, so obviously you're doing a lot with observability. And you're doing that in New Relic, that's sort of what New Relic does. IOpipe was all about that. I know a lot of the team has gone over from IOpipe to New Relic, to continue to work and expand their services. And I'd love to talk to you about that today. We've done a number of shows where we've talked about observability. But that was probably almost a year ago at this point. And I'd love to get a sense of sort of, where things have gone, where things are going. You know maybe what the future is going to look like. I got a bunch of other things I want to talk to you about. But maybe you could just start, just in case listeners don't know, what do we mean by observability? Erica: Oh, gosh. The way I see it is, being able to really see what's happening in your applications, in your infrastructure and doing that... I would say early monitoring. Things like Nagios was, I would not consider observability that was monitoring. It was very much, very reactive. There was zero product... It was not productive at all using something like Nagios. Logging products give you some ability to start getting into, being able to be proactive. And I think that observability kind of ties in some of the concepts from logging, and ties it in with your metrics and ties in being highly correlated. And also deeper into your application, having traces in your application, having contacts for your applications. For instance, just having a trace and knowing that, say an API gateway triggered a Lambda is one piece of information that you can have, but knowing say, the resource path, the HTTP method, things like that. That's a deeper set of insight that I think is necessary. And definitely fits within an observability picture that is very much say different and distinct from something like Nagios. Or even just plain text logs. Jeremy: Right. Yeah. And we've talked about on the show the three pillars, right? You've got monitoring, tracing and logging. And so monitoring, like you said, is that sort of general like just something goes wrong, maybe you get an alert, something like that. The logging bit is obviously logging data. But let's get more into tracing a little bit. What do we mean by tracing? Erica: Sure. The way I look at tracing as being able to see the relationship between various components, and not just the components. And I think this also where maybe tracing generally... in our industry historically, has been this service talks to this or that service. And that service talks to another service, etc. I think of it as this function communicates to this other function. And that is true, even outside of serverless, where functions are the primitive. Serverless was a really great place for us to start because it's already segmented into functions. But if you're looking at a microservice, there's no reason that you can't think about your code, about say this functions or this component or this resource path is communicating to this other function and also, contextually. So, for instance, maybe this service only calls DynamoDB when it's inserting data. Or when the API gateway, there's a put request, right? That triggers a put into DynamoDB. You don't get a put into DynamoDB when you do a delete on a API gateway. So that's the kind of context that I think is really interesting for things like tracing. That is a little bit more I think, beyond what traditional tracing solutions have been doing. Jeremy: Right. Because I mean, it's a lot different in these distributed systems. I mean, even if you're just talking to one microservice, it's usually you talk to one microservice and maybe you want to see that continuity there, or service X called service Y. But now with serverless specifically, we have you function X calling service Y, which generates an event that then gets picked up by EventBridge, and then another service picks it up and so forth. So we can get into why it's important in serverless applications, but is there anything else, where observability is different in the sense of monitoring? I mean, you mentioned the idea of being a little bit proactive. What do you mean by being more proactive? Erica: Well, by being proactive I guess I'm referring to the fact that things again, rewinding history a little bit and going to something very distinctly not observably like Nagios. Again saying, "This very reactive." Something went down and now we then asked it, "If it's up?" And I said, "It's..." And we couldn't reach it, so then we determined that it's down. I think that kind of step two of that kind of journey towards observability, would say, "Okay. Well, we have logs, we have a logging product. And the logs told me that when, I don't know. This service tried running to the Syslog Server, they got an error." Well, when I get that error, I know that at least this sys...