Arch Gateway: Add AI To Your Apps Without Custom Development

Summary In this episode of the AI Engineering Podcast Adil Hafiz talks about the Arch project, a gateway designed to simplify the integration of AI agents into business systems. He discusses how the gateway uses Rust and Envoy to provide a unified interface for handling prompts and integrating large language models (LLMs), allowing developers to focus on core business logic rather than AI complexities. The conversation also touches on the target audience, challenges, and future directions for the project, including plans to develop a leading planning LLM and enhance agent interoperability. Announcements Hello and welcome to the AI Engineering Podcast, your guide to the fast-moving world of building scalable and maintainable AI systems Your host is Tobias Macey and today I'm interviewing Adil Hafeez about the Arch project, a gateway for your AI agents Interview Introduction How did you get involved in machine learning? Can you describe what Arch is and the story behind it? How do you think about the target audience for Arch and the types of problems/projects that they are responsible for? The general category of LLM gateways is largely oriented toward abstracting the specific model provider being called. What are the areas of overlap and differentiation in Arch? Many of the features in Arch are also available in AI frameworks (e.g. LangChain, LlamaIndex, etc.), such as request routing, guardrails, and tool calling. How do you think about the architectural tradeoffs of having that functionality in a gateway service? What is the workflow for someone building an application with Arch? Can you describe the architecture and components of the Arch gateway? With the pace of change in the AI/LLM ecosystem, how have you designed the Arch project to allow for rapid evolution and extensibility? What are the most interesting, innovative, or unexpected ways that you have seen Arch used? What are the most interesting, unexpected, or challenging lessons that you have learned while working on Arch? When is Arch the wrong choice? What do you have planned for the future of Arch? Contact Info LinkedIn GitHub Parting Question From your perspective, what are the biggest gaps in tooling, technology, or training for AI systems today? Closing Announcements Thank you for listening! Don't forget to check out our other shows. The Data Engineering Podcast covers the latest on modern data management. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story. To help other people find the show please leave a review on iTunes and tell your friends and co-workers. Links Arch Gateway Gradient Boosting Envoy LLM Gateway Huggingface Katanemo Models Qwen2.5 Rust Clippy The intro and outro music is from Hitman's Lovesong feat. Paola Graziano by The Freak Fandango Orchestra/CC BY-SA 3.0