Orchestrating Retail Data Pipelines at Saks Global

What this episode covers

Saks Global runs one of the largest retail data operations in the US, with around 8 million SKUs flowing across point-of-sale, e-commerce, catalog, and fraud detection systems into Snowflake. In this episode, [Shailesh Kadam](linkedin.com), Architect at [Saks Global](saks.com), joins Kenten to walk through how Airflow acts as the nervous system tying it all together, why they moved from self-managed Kubernetes to Astro, what is driving their Airflow 3 upgrade, and how they are approaching agentic AI, MCP, and credential security.Key Takeaways:00:00 Introduction.01:31 Saks Global today. Shailesh describes the business after separating e-commerce from brick and mortar and acquiring Neiman Marcus, and the modern cloud-native stack on AWS, Snowflake, and Airflow.02:50 8 million SKUs in motion. Why every name, image, inventory, and price change has to flow in near real time across operational systems.04:50 What the pipelines look like. Point-of-sale ingestion, fraud signals to third parties like Fiserv, and hourly product catalog feeds out to Meta and Google.07:30 Moving off self-managed Kubernetes to Astro. Shailesh contrasts past experience with Kubernetes, IBM Tivoli, and Control-M against running on Astro.09:35 Upgrading to Airflow 3. Event and asset-based scheduling, DAG versioning, task isolation, and using Otto to convert DAGs in a phased rollout.13:13 Agentic AI and MCP on the roadmap. How Saks plans to use Airflow's MCP for LLM-driven product classification and to feed Snowflake analyses like churn and spend.18:01 Securing PII and credentials. Secrets backends, cloud secret manager integration, key rotation, and keeping credentials out of DAG code.21:02 Wishlist for Airflow. Interactive data lineage across DAGs and a UI-based debugging interface for support teams.Resources Mentioned:[Apache Airflow](airflow.apache.org)[Astro](astronomer.io/product)[Otto, the Astronomer data engineering agent](astronomer.io)[Snowflake](snowflake.com)[Saks Global](saks.com)Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow

PodParley-generated summary based on available episode metadata and transcript content.