LLMs at Scale: Infrastructure That Keeps AI Safe, Smart & Affordable // Marco Palladino// # 341

LLMs at Scale: Infrastructure That Keeps AI Safe, Smart & Affordable // MLOps Podcast #341 with Marco Palladino, Kong's Co-Founder and CTO.Join the Community: https://go.mlops.community/YTJoinInGet the newsletter: https://go.mlops.community/YTNewsletter// AbstractWhile conversations around AI regulations continue to evolve, the responsibility for AI continues to be with developers. In this episode, Marco Palladino, CTO and co-founder of Kong Inc., explores what it means to build and scale AI responsibly when the rulebook is still being written. He explains that infrastructure should be the frontline defense for enforcing governance, security, and reliability in AI deployments. Marco shares how Kong’s technologies, including AI Gateway and AI Manager, help organizations rein in shadow AI, reduce LLM hallucinations, improve observability, and act as the foundation for agentic workflows.// BioMarco Palladino is an inventor, software developer, and internet entrepreneur. As the CTO and co-founder of Kong, he is Kong’s co-author, responsible for the design and delivery of the company’s products, while also providing technical thought leadership around APIs and microservices within both Kong and the external software community. Prior to Kong, Marco co-founded Mashape in 2010, which became the largest API marketplace and was acquired by RapidAPI in 2017. // Related LinksWebsite: https://konghq.com/ https://www.youtube.com/watch?v=odpPVeQZjHU https://www.thestack.technology/the-big-interview-kong-cto-marco-palladino/ ~~~~~~~~ ✌️Connect With Us ✌️ ~~~~~~~Catch all episodes, blogs, newsletters, and more: https://go.mlops.community/TYExploreJoin our Slack community [https://go.mlops.community/slack]Follow us on X/Twitter [@mlopscommunity](https://x.com/mlopscommunity) or [LinkedIn](https://go.mlops.community/linkedin)] Sign up for the next meetup: [https://go.mlops.community/register]MLOps Swag/Merch: [https://shop.mlops.community/]Connect with Demetrios on LinkedIn: /dpbrinkmConnect with Marco on LinkedIn: /marcopalladino/Timestamps:[00:00] Agent-mediated interactions shift[01:17] Kong connectivity and agents[04:36] Transcript cleanup request[08:11] MCP server use cases[12:37] Agent world possibilities [15:55] Business communication evolution[18:55] System optimization[25:36] AI gateway patterns[31:30] Investment decision making[35:54] Building conviction process[41:34] Polished customer conversation[46:37] AI gateway R&D future[50:52] Wrap up