The Hard Truth About AI Agents: What We Learned Building in Open Source

In this episode of Humans of AI, I sit down with my friend Eric Zhu to unpack what are the frameworks and architectures powering modern AI products today.From our times building Semantic Kernel, Autogen, and GraphRAG at Microsoft, we offer a retrospective on the evolution of this space and where we think it'll go in 2026 and beyond.We dive into how AI systems are actually being built today, why early agent frameworks were brittle, and what’s changing as models, tooling, and abstractions improve. Eric shares insights from hands-on experience building AI systems, including lessons around memory, orchestration, open source, and why building has become dramatically faster than even a few years ago.We also talk about the human side of all this—how builders should think about careers in AI, avoiding burnout, and navigating a fast-moving field without getting overwhelmed.Whether you’re an AI engineer, founder, researcher, or just curious about where things are headed, this conversation goes deep into the why behind modern AI systems.⏱️ Chapters00:00 – Intro02:02 - Welcoming Eric Zhu! Experiences building open-source systems from PhD to Industry11:18 – Early agent systems, frameworks, and brittle abstractions20:18 – Multi-agent frameworks and why it was hard to go beyond a demo25:47 – From autonomous agents to workflow orchestration - a trough of disillusionment?30:00 – Breathing new life into agents with Claude Code34:35 – "Harness" Engineering and what goes into building agent systems?39:08 – The importance of data and data representations (i.e knowledge graphs, structured data)45:57 – Lessons from building in Open Source56:54 – Office Hours: How to not get left behind in AISubmit your question to the Humans of AI Office Hours and we may cover it in a future episode!https://forms.gle/iztuQCjLyUe8kXYRA🎙️ Connect with the hosts:Alex Chao 📧 Newsletter: https://alexchao.substack.com/ 💼 LinkedIn: https://www.linkedin.com/in/alexchao56/ 🐦 X: https://x.com/alexchaomander📱 TikTok: https://www.tiktok.com/@alexchaomanderEric Zhu💼 LinkedIn: https://www.linkedin.com/in/ekzhu/🐦 X: https://x.com/ekzhuAI moves fast. We keep it human.