Orchestrating cost-efficient data pipelines at Insignis
from The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
Running Airflow with a one-person data engineering team means every decision about cost, tooling, and best practices has to earn its keep. In this episode, Kenten sits down with Aniruddha Sengupta, Staff Data Engineer at Insignis, to talk through how the UK cash-savings fintech runs a low-cost, scalable Airflow platform. The conversation covers observability, DAG-level best practices, the internal tools Insignis built to control costs, and where AI fits into the roadmap. Key Takeaways: 00:00 Introduction. 02:22 Insignis data team as a "newspaper" for the business. Aniruddha describes the data team's role: consuming data from operational systems and external sources like Salesforce and Ringover, and delivering insight without owning business logic. 08:07 Airflow as the orchestrator, not the executor. Airflow sits at the heart of the platform, orchestrating SQL pipelines, API ingestion, and even acting as an observability layer over real-time pipelines built elsewhere. 12:19 Observability as definition of done. Why a green Airflow UI is not observability, and how Insignis builds independent dashboards to catch silently dropped data. 14:38 Choosing managed Airflow. How abstracting away the scheduler, workers, and database frees a small team to focus on code and documentation. 20:04 DAG patterns that keep costs down. CI/CD enforcement, failure callbacks, mindful DAG boundaries, and periodic purging of logs and metadata. 22:06 Tariff and Maestro. Two internal tools Insignis built: Tariff to surface SQL pipeline costs, and an AI-powered tool called Maestro to recommend worker, scheduler, and database sizing. 25:40 Exploring AI with Airflow. MLOps use cases around client behavior and the appeal of self-healing pipelines. 28:20 Wishlist for Airflow. Event-based DAG parsing so code changes surface in the UI closer to instantaneously. Resources Mentioned: Annual Airflow survey - Insignis - Apache Airflow - Astronomer - Building a Low-Cost Scalable Airflow Platform for Small Teams - Google BigQuery Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow
Embed this episode
What this episode covers
Running Airflow with a one-person data engineering team means every decision about cost, tooling, and best practices has to earn its keep. In this episode, Kenten sits down with Aniruddha Sengupta, Staff Data Engineer at Insignis, to talk through how the UK cash-savings fintech runs a low-cost, scalable Airflow platform. The conversation covers observability, DAG-level best practices, the internal tools Insignis built to control costs, and where AI fits into the roadmap. Key Takeaways: 00...
No transcript yet.