PODCAST · technology
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
by Astronomer
Welcome to The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI— the podcast where we keep you up to date with insights and ideas propelling the Airflow community forward.Join us each week, as we explore the current state, future and potential of Airflow with leading thinkers in the community, and discover how best to leverage this workflow management system to meet the ever-evolving needs of data engineering and AI ecosystems. Podcast Webpage: https://www.astronomer.io/podcast/
Episodes
-
Orchestrating cost-efficient data pipelines at Insignis
Running Airflow with a one-person data engineering team means every decision about cost, tooling, and best practices has to earn its keep. In this episode, Kenten sits down with Aniruddha Sengupta, Staff Data Engineer at Insignis, to talk through how the UK cash-savings fintech runs a low-cost,...
-
Scaling to 1,000+ Airflow Dags at Descartes / Idelic
Scaling Airflow to over 1,000 active DAGs on a team of 3-4 engineers takes structure. In this episode, Kenten Danas talks with Matt Stavinga, Data Engineer at Idelic (now part of Descartes), about how his team standardized DAG creation and logging, built a Jenkins-driven testing workflow with...
-
Orchestrating local LLMs for sensitive data with Airflow
Running LLMs over sensitive data means external APIs are off the table, but open-weight models still need reproducibility, validation, and a paper trail before downstream systems can depend on their output. In this episode, Kenten is joined by Chhayank Jain, Senior ML Engineer at Visa, to talk...
-
Improving security in Airflow 3
Security is one of the least visible parts of Airflow for most users, but it shapes what features teams can turn on in production. In this episode, Kenten Danas talks with Anish Giri, Senior Software Engineer and active Apache Airflow contributor focused on security, about three recent...
-
Orchestrating data across 30 companies at itti
Orchestrating data across more than 30 companies means most Airflow users on the platform aren't data engineers. In this episode, Kenten Danas talks with Lucas Trubiano, Data Engineer at itti, the technology company within Grupo Vázquez in Paraguay. Lucas walks through the custom YAML framework…
more -
Building self-healing Airflow pipelines at ATC Drivetrain
Manufacturing data pipelines can't afford silent failures. When quality decisions and shop-floor visibility depend on Airflow, a broken DAG at midnight can cost real money. In this episode, Kenten Danas talks with Kumuda Sreenivasa, Founder of Receitly and Senior Data Architect at ATC Drivetrain,...
-
Orchestrating strictly sequential ETL pipelines at Synechron
Strict sequential execution across DAGs sounds simple until you have scheduled pipelines and event-driven pipelines writing to the same MongoDB collections. Ivana Isailovic, Senior Big Data Engineer at Synechron, joins Marc Lamberti to walk through the three-layer DAG architecture her team built…
more -
Managing financial datasets with Airflow at Wise
Financial data pipelines have to be right the first time. On this episode, Kenten sits down with [Antonello Benedetto](linkedin.com/in/anbento4), Staff Data Engineer at Wise, to talk about how the central data and analytics engineering team runs Airflow for critical financial datasets, tiers...
-
Managing travel platform data with Airflow at Headout
Running a travel platform means dealing with fast-moving inventory, real-time fraud detection, and heavy performance marketing attribution, all while keeping data trustworthy for a data-driven org. In this episode, Mrinalini Singh, Data Platform Engineer at [Headout](headout.com), walks through…
more -
How Airflow orchestration decisions impact Spark performance
Airflow and Spark are one of the most common combinations in modern data platforms, but the orchestration decisions made on the Airflow side often determine whether Spark jobs run fast and cheap or slow and expensive. In this episode, Kenten is joined by [Meni...
-
Using Airflow for diverse client projects at Accion Labs
When a real-time loan eligibility scoring pipeline is built on cron jobs, midnight pages are inevitable. In this episode, [Chandan Gowda](linkedin.com/in/chandan-gowda-a-h-744908195), Data Engineer at [Accion Labs](accionlabs.com), joins Kenten to discuss how his team uses Airflow across client...
-
Consolidating legacy job schedulers with Airflow at Trading Technologies
Migrating two decades of legacy job orchestration is the kind of project most teams quietly avoid. In this episode, Marc Lamberti is joined by [Sanket Patel](linkedin.com/in/sanket-patel-a2a57b3a), Director of Engineering at Trading Technologies, to talk through how his team is consolidating 20...
-
Orchestrating AI video intelligence and evaluation pipelines at Firework
How do you orchestrate AI workflows when LLM outputs are non-deterministic and evaluation costs can quietly exceed compute costs? Shawn Feng, Head of Data at [Firework](firework.com), joins the show to walk through how his team uses Airflow to coordinate ingestion, embeddings, and evaluation...
-
Orchestrating Retail Data Pipelines at Saks Global
Saks Global runs one of the largest retail data operations in the US, with around 8 million SKUs flowing across point-of-sale, e-commerce, catalog, and fraud detection systems into Snowflake. In this episode, [Shailesh Kadam](linkedin.com), Architect at [Saks Global](saks.com), joins Kenten to…
more -
What's New in Apache Airflow® 3.3
Airflow 3.3 is here, with a set of features to help with the messy realities of production pipelines: persisting state across retries, reacting intelligently to different failure types, and partitioning assets by more than just time. In this episode, Marc Lamberti, Education Content Lead at...
-
Running Airflow 3 in a regulated environment at OTPP
Running Apache Airflow at a major pension fund means balancing strict compliance requirements with the need to move fast on new capabilities. On this episode, Kowsy Narayan, Cloud Data Platform Lead, Data Engineering at [Ontario Teachers' Pension Plan](otpp.com), joins host Kenten Danas to walk...
-
Managing a Customer Analytics Platform with Airflow at Skimlinks
Skimlinks runs a reporting platform that serves around 2,000 weekly publisher users, and the data infrastructure behind it runs on Airflow. In this episode, Julian Larralde, Director of Data Engineering at Skimlinks, walks through the stack, the migration from external task sensors to…
more -
Building a custom Tableau provider for Airflow at JLR
JLR is the UK's largest automotive manufacturer, behind brands like Range Rover, Jaguar, Defender, and Discovery. In this episode, Najeeb Sulaiman, Senior Data Engineer at JLR, walks through how Airflow orchestrates data across manufacturing, supply chain, and finance — including a custom Tableau...
-
Orchestrating 2,000 Airflow pipelines at Luiza Labs with Mateus Ferreira
Running Airflow at the scale of a national retailer means more than just scheduling. It means giving non-engineers a path to ship DAGs, and classifying thousands of runs to know which ones need attention. In this episode, Mateus Ferreira, Senior Data Engineer at Luiza Labs (the technology arm of...
-
Enhancing DAGs for Data Processing with William Orgertrice III at Cargill
In the data engineering world, the difference between a pipeline that works and one that's truly production-ready often comes down to a handful of deliberate decisions. William Orgertrice III, Data Engineer at Cargill, joins us to share the DAG design and monitoring practices he presented at...
-
Getting Into Data Engineering with Shrividya Hegde, Data and AI Engineer
In this episode, we take a step back from implementation-specific topics to explore what it actually takes to build a career in data engineering — and how AI is reshaping that path. Shrividya Hegde, a data and AI engineer and an Airflow champion in Astronomer’s Champions program, joins us to...
-
Orchestrating DBT With Cosmos and Airflow with Filip Kunčar at ShipMonk Product Development
We explore how a third-party logistics platform built its entire data orchestration layer on Airflow, and what that makes possible for developer teams and merchant-facing products alike. Filip Kunčar, Platform Director at ShipMonk Product Development, discusses migrating from a closed source tool...
-
Building Airflow CTL with Buğra Öztürk at Mollie
Buğra Öztürk, Senior Data Engineer at Mollie and Committer and PMC member on the Apache Airflow project, joins us to walk through Airflow CTL — what it is, how it differs from the existing Airflow CLI and where it is headed under AIP-94. Key Takeaways: 00:00 Introduction. 03:10 Buğra has...
-
Introducing Airflow’s Common AI Provider with Pavan Kumar Gopidesu and Kaxil Naik
In this episode, we explore the newly released Apache Airflow common AI provider — what problem it solves, how it was built and what's coming next. Kaxil Naik, Senior Director of Engineering at Astronomer and Apache Airflow PMC member, and Pavan Kumar Gopidesu, Lead Data Engineer at Experian and...
-
Building AI Debugging Agents Into Airflow DAGs at Jeppesen ForeFlight with Samantha Blaney Cuevas
Aviation data pipelines run on strict 28-day publication cycles, and the margin for error is zero. In this episode, we're joined by Samantha Blaney Cuevas, Software Engineer at Jeppesen ForeFlight, to explore how her team orchestrates a complex, time-sensitive data pipeline with Airflow and where...
-
Introducing Airflow 3.2
We introduce Airflow 3.2 and its updates for teams that build and operate data pipelines. Astronomer’s Head of Customer Education, Marc Lamberti, and Senior Manager of Developer Relations, Kenten Danas, break down what’s new, from asset partitioning to Async Python tasks and DAG versioning. They...
-
Reflections on a Decade of Data Engineering at Seattle Data Guy
Lessons from the past decade of data engineering reveal how much the ecosystem has changed and what has stayed surprisingly consistent. In this episode, Benjamin Rogojan, Owner and Data Consultant at Seattle Data Guy, joins us to reflect on how the data engineering landscape has evolved alongside...
-
Managing Data Quality and Governance With Airflow at Credit Karma with Ashir Alam
Data quality is not optional when you manage credit data at scale. In this episode, Ashir Alam, Senior Data Engineer at Credit Karma, joins us to share how his team acts as the gatekeeper for credit data ingestion, how they standardize data quality with Airflow and DAG Factory and how they scale...
-
Open Source Airflow Contributions and Performance Improvements at G-Research with Christos Bisias
Modern Airflow isn’t just orchestration. It's a contribution. In this episode, we explore how open source investment drives real performance gains and deeper observability. We’re joined by Christos Bisias, Open Source Software Engineer, Apache Airflow at G-Research, to discuss how his team…
more -
Automating Threat Intelligence Using Airflow with Karan Alang
In this episode, Karan Alang, Principal Software Engineer at Versa Networks, joins the conversation to discuss how Airflow can be used to automate threat intelligence in modern cybersecurity environments. He explains the growing scale of cloud computing, the profitability of hacking and the...
-
Using Plugins To Customize Airflow at Ponder Labs with Egor Tarasenko
In this episode, we explore how teams scale Apache Airflow in complex environments and what it takes to make orchestration work across many stakeholders. We look at real-world challenges around visibility, ownership and predictability as data platforms grow. Egor Tarasenko, Data and AI Engineer…
more -
Scaling Airflow at Wix for Analytics and AI with Ethan Shalev
Modern data orchestration at scale demands reliability, speed and thoughtful adoption of new tooling. As organizations grow, keeping pipelines efficient while supporting more teams becomes a critical challenge. In this episode, we’re joined by Ethan Shalev, Data Engineer at Wix, to discuss how…
more -
Using Airflow To Orchestrate Billions of Events at Addi with Carlos Daniel Puerto Niño
Strong data orchestration is as much about culture and visibility as it is about technology. As data platforms scale, teams need systems that reduce cognitive load while increasing reliability and observability. In this episode, Carlos Daniel Puerto Niño, Senior Analytics Engineer and Data…
more -
Building Event-Driven Data Pipelines With Airflow 3 at Astrafy with Andrea Bombino
Real-time data expectations are reshaping how modern data teams think about orchestration and dependencies. As event-driven architectures become more common, teams need to rethink how pipelines react to data changes, rather than schedules. In this episode, Andrea Bombino, Co-Founder and Head of...
-
Uphold’s Approach to Orchestrating Modern Data Workflows with Jaime Oliveira
A strong data-driven mindset underpins how fintech teams scale analytics, infrastructure and decision-making across the business. In this episode, Jaime Oliveira, Lead Data Engineer at Uphold, joins us to discuss how Uphold structures its data organization and orchestration strategy. Jaime shares...
-
Modern Airflow Best Practices for Scalable Data Pipelines with Bhavani Ravi
Building reliable data pipelines at scale requires more than writing code. It depends on thoughtful design, infrastructure trade-offs and an understanding of how orchestration platforms evolve over time. In this episode, Airflow best practices shaped by real-world implementation are examined....
-
Inside Conviva’s Decision To Power Its Data Platform With Airflow with Han Zhang
Conviva operates at a massive scale, delivering outcome-based intelligence for digital businesses through real-time and batch data processing. As new use cases emerged, the team needed a way to extend a streaming-first architecture without rebuilding core systems. In this episode, Han Zhang joins...
-
Why Airflow Became the Scheduling Backbone at Condé Nast Technology Lab with Arun Karthik
Data platforms are moving from batch-first pipelines to near real-time systems where orchestration, observability, scalability and governance all have to work together. In this episode, Arun Karthik, Director, Data Solutions Engineering at Condé Nast Technology Lab, joins us to share how data...
-
The Role of Airflow in Building Smarter ML Pipelines at Vivian Health with Max Calehuff
The integration of data orchestration and machine learning is critical to operational efficiency in healthcare tech. Vivian Health leverages Airflow to power both its ETL pipelines and ML workflows while maintaining strict compliance standards. Max Calehuff, Lead Data Engineer at Vivian Health,...
-
Scaling Airflow to 11,000 DAGs Across Three Regions at Intercom with András Gombosi and Paul Vickers
The evolution of Intercom’s data infrastructure reveals how a well-built orchestration system can scale to serve global needs. With thousands of DAGs powering analytics, AI and customer operations, the team’s approach combines technical depth with organizational insight. In this episode, András...
-
How Covestro Turns Airflow Into a Simulation Toolbox with Anja Mackenzie
Building scalable, reproducible workflows for scientific computing often requires bridging the gap between research flexibility and enterprise reliability. In this episode, Anja MacKenzie, Expert for Cheminformatics at Covestro, explains how her team uses Airflow and Kubernetes to create a…
more -
Building Secure Financial Data Platforms at AgileEngine with Valentyn Druzhynin
The use of Apache Airflow in financial services demands a balance between innovation and compliance. Agile Engine’s approach to orchestration showcases how secure, auditable workflows can scale even within the constraints of regulatory environments. In this episode, Valentyn Druzhynin, Senior…
more -
How Redica Transformed Their Data With Airflow and Snowflake with Shankar Mahindar
The life sciences industry relies on data accuracy, regulatory insight and quality intelligence. Building a unified system that keeps these elements aligned is no small feat. In this episode, we welcome Shankar Mahindar, Senior Data Engineer II at Redica Systems. We discuss how the team...
-
How Airflow and AI Power Investigative Journalism at the Financial Times with Zdravko Hvarlingov
The Financial Times leverages Airflow and AI to uncover powerful stories hidden within vast, unstructured data. In this episode, Zdravko Hvarlingov, Senior Software Engineer at the Financial Times, discusses building multi-tenant Airflow systems and AI-driven pipelines that surface stories that...
-
Inside Vinted’s Code-Generated Airflow Pipelines with Oscar Ligthart and Rodrigo Loredo
The shift from monolithic to decentralized data workflows changes how teams build, connect and scale pipelines. In this episode, we feature Oscar Ligthart, Lead Data Engineer, and Rodrigo Loredo, Lead Analytics Engineer, both at Vinted, as we unpack their YAML-driven abstraction that generates...
-
Transforming Data Pipelines at XENA Intelligence with Naseem Shah
The shift from simple cron jobs to orchestrated AI-powered workflows is reshaping how startups scale. For a small team, these transitions come with unique challenges and big opportunities. In this episode, Naseem Shah, Head of Engineering at Xena Intelligence, shares how he built data pipelines...
-
Scaling Geospatial Workflows With Airflow at Overture Maps Foundation and Wherobots with Alex Iannicelli and Daniel Smith
Using Airflow to orchestrate geospatial data pipelines unlocks powerful efficiencies for data teams. The combination of scalable processing and visual observability streamlines workflows, reduces costs and improves iteration speed. In this episode, Alex Iannicelli, Staff Software Engineer at...
-
Scaling Airflow for Enterprise Data Platforms at PepsiCo with Kunal Bhattacharya
PepsiCo’s data platform drives insights across finance, marketing and data science. Delivering stability, scalability and developer delight is central to its success, and engineering leadership plays a key role in making this possible. In this episode, Kunal Bhattacharya, Senior Manager of Data...
-
Building a Unified Data Platform at Pattern with William Graham
The orchestration of data workflows at scale requires both flexibility and security. At Pattern, decoupling scheduling from orchestration has reshaped how data teams manage large-scale pipelines. In this episode, we are joined by William Graham, Senior Data Engineer at Pattern, who explains how…
more -
How Astronomer Turns Proactive Monitoring Into Customer Success with Collin McNulty
The evolution of Airflow continues to shape data orchestration and monitoring strategies. Leveraging it beyond traditional ETL use cases opens powerful new possibilities for proactive support and internal operations. In this episode, we are joined by Collin McNulty, Sr. Director of Global Support...
We're indexing this podcast's transcripts for the first time — this can take a minute or two. We'll show results as soon as they're ready.
No matches for "" in this podcast's transcripts.
No topics indexed yet for this podcast.
Loading reviews...
ABOUT THIS SHOW
Welcome to The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI— the podcast where we keep you up to date with insights and ideas propelling the Airflow community forward.Join us each week, as we explore the current state, future and potential of Airflow with leading thinkers in the community, and discover how best to leverage this workflow management system to meet the ever-evolving needs of data engineering and AI ecosystems. Podcast Webpage: https://www.astronomer.io/podcast/
HOSTED BY
Astronomer
CATEGORIES
Loading similar podcasts...
Frequently Asked Questions
How many episodes does The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI have?
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI currently has 119 episodes available on PodParley. New episodes are automatically indexed when they're published to the podcast feed.
What is The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI about?
Welcome to The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI— the podcast where we keep you up to date with insights and ideas propelling the Airflow community forward.Join us each week, as we explore the current state, future and potential of Airflow with leading thinkers...
How often does The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI release new episodes?
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI publishes regularly with 119 episodes available. Check the episode list for the typical release schedule.
Where can I listen to The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI?
You can listen to The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI on PodParley by clicking any episode. We provide an embedded audio player for direct listening, and you can also subscribe via your preferred podcast app using the RSS feed.
Who hosts The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI?
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI is created and hosted by Astronomer.
URL copied to clipboard!