Episode #41: Communication Patterns in Serverless with Paul Swail

About Paul Swail Paul is an independent cloud architect who helps development teams make the transition to serverless. He has almost 20 years’ experience delivering software solutions to clients across a wide variety of industries. In addition to client consulting, Paul has been running his small SaaS business on AWS for the past 6 years and is slowly migrating it to a fully serverless stack. He writes in-depth articles on serverless in his email newsletter and on his blog at winterwindsoftware.com. Twitter: @paulswail LinkedIn: https://www.linkedin.com/in/paulswail/ Blog/Newsletter: http://serverlessfirst.com/ WATCH THIS EPISODE ON YOUTUBE: https://www.youtube.com/watch?v=gf__z3K8LBI Transcript: Jeremy: Hi, everyone. I'm Jeremy Daly, and you're listening to Serverless Chats. This week I'm chatting with Paul Swail. Hey, Paul. Thanks for joining me. Paul: Hey, Jeremy. It's great to be here. Thank you. Jeremy: You are a cloud architect at Winter Wind Software. So, why don't you tell the listeners a little bit about yourself and what you do? Paul: Yeah, sure. I'm an independent cloud architect. I work primarily with AWS and I specialize in helping development teams ship their first serverless application into production. I've been focused specifically on serverless for two years or so now, although I have been doing software development professionally for about 19 years in total. Jeremy: Wow. Still not as old as me, but that's okay. One of the things that you've been focusing on lately... Or, I think you're making this transition into serverless first. So, why don't you tell us a little bit about that? Paul: Yeah, yeah. My website is currently in the process of being moved to ServerlessFirst.com, so I think that... Basically, serverless first is a methodology, which I've been taking with my clients that I've been working with over the past couple of years. They're used to more traditional ways of building serverless apps, and they see the value of using serverless, but they can't use it for everything. By default, your architectural decisions start with serverless services unless you can justify using something else. Jeremy: Awesome. Alright, I wanted to talk to you today because... I don't know what happened, but somehow you've become one of the most prolific writers in serverless over the last couple of months, releasing a few articles or a few blog posts every week. It's been awesome because every time we get more content, and you answer more questions, and you get deep onto one particular subject, I think it's super helpful. One of the things that you focused on quite a bit, I think, has been this idea of these communication patterns in serverless applications. You wrote two articles recently. One was called Seven Ways to Do Async Message Processing in AWS, and another one was Interservice Communication Channels for Serverless Microservices in AWS. Both great articles. Definitely go to WinterWindSoftware.com, check those out. Very, very interesting stuff. Why don't you tell us a little bit? Maybe you can get us started, in terms of what are the main communication patterns in serverless, and why is it so different, maybe, than a traditional application? Paul: Yeah. I think a lot of my clients have come from the monolithic architectural background and asynchronous stuff. They may be aware of it but they haven't used it a lot. AWS has a lot of services which does... Around asynchronous messaging patterns and it can be hard to understand what to choose. So, a lot of it is just documenting questions that clients have had for me. A lot of the writing is around that area. We can go through the details of lots of individual services that I discussed in the article. Jeremy: Yeah. Let's start, though, maybe just thinking about the difference between asynchronous and synchronous because I think most people are very familiar with that monolithic approach of... I should maybe take a step back. They're used to that request-response type mechanism, right? I make a request to a website or to an API and that data comes back to me. There's that one part of that immediate response. That's not going to change whenever you have a customer-facing or a web-facing side of things, but it's where the backend... The backend is what gets different. That is one of those things where, I think, when people are familiar with monolithic applications, they think, hey, I've got 15 different methods, or functions, or whatever, that are all in one big application and I can say, "hey, I need to process the order. I need to pull the inventory. I need to send the message." And that's all in one app, or one, I guess, big chunk of code, really. But when you start moving to this asynchronous thinking, we're starting to separate out these components separately. So what do people have to think about when they start building that type of application? Paul: There are quite a lot of things to think about. I guess based on the workloads, firstly. If it's like a task or a job-based type workflow, you may want to do, or maybe that you need to notify a lot of other systems, so based around... That's a decision in itself, around what service you use, based on the nature of the workload. There are a lot of operational considerations just around throughput, and concurrency requirements, and latency requirements, and scalability of any downstream systems that you may need to talk to, and message durability, and your error handling, and retry mechanisms. These are all things... Oh, cost, of course, as well. These are all things that you need to consider around how you would structure any asynchronous messaging patterns that your workload requires. Jeremy: Yeah. I think that makes a lot of sense. I think what you get people coming from the monolithic space or the traditional system space, and they move into distributed systems. Now, there's this whole different idea of passing messages around, right? We're no longer using a single system, so now we're trying to communicate with multiple systems, and as you said, things like durability of messages, that becomes a huge concern. Or, something like the error handling. What happens when you send a message off into the ether and you don't know what happens to it? Does it ever get to its destination? How do you know a message was even sent if that information isn't recorded correctly. I think maybe that's another thing that is interesting to me though, is that even people who maybe come from distributed systems and think about, oh, I've got to set up a Kafka cluster, or I've got to do something like that. Traditionally, there has been a ton of stuff that you would need to do just to create the messaging components between these distributed systems, but serverless changes that quite a bit. Paul: Yeah, it does. I can echo those sentiments you said, I used to, back in my Microsoft.net developer days, setting up BizTalk servers, something actually I knew how to do, and we did it in a few projects, but just the provisioning and management overhead of doing it just... even though it was a nice distributed messaging patte...