Running Airflow 3 in a regulated environment at OTPP

Running Apache Airflow at a major pension fund means balancing strict compliance requirements with the need to move fast on new capabilities. On this episode, Kowsy Narayan, Cloud Data Platform Lead, Data Engineering at [Ontario Teachers' Pension Plan](otpp.com), joins host Kenten Danas to walk through OTPP's cloud migration, their move to Airflow 3, and going fully live on remote execution.Key Takeaways:00:00 Introduction.01:18 Inside the OTPP data platform team and what they're responsible for across cloud migration, standards, and enablement.02:33 What's driving OTPP's multi-year move off on-prem to a cloud architecture built around scalability and resilience.02:57 The new stack: Snowflake as the enterprise data platform, dbt for transformation, and Airflow as the orchestrator in the middle.04:15 Why OTPP chose Astronomer: active contributions to the Airflow OSS project, fast runtime releases, and built-in monitoring, observability, and RBAC.05:50 Evolving from dbt core with Bash operators to dbt Cosmos for model-level granularity, lineage, and precise failure recovery, plus a performance boost from watcher mode.08:00 Upgrading from Airflow 2.9 to Airflow 3, using the Astro CLI and linters to catch deprecations quickly.09:32 The drivers behind adopting remote execution: keeping data inside the security perimeter and scaling workloads on their own Kubernetes cluster.11:35 How remote execution replaced a complex network architecture of VPN tunnels and firewall rules, removing latency along the way.12:53 The POV process, success criteria, and a six week timebox to validate remote execution before going to production.14:14 Going fully live: OTPP's last hosted deployment was sunset just before recording.15:06 What Kowsy wants next from Airflow: AI orchestration capabilities and continued maturation of remote execution.Resources Mentioned:[Ontario Teachers' Pension Plan](otpp.com)[Apache Airflow](airflow.apache.org)[Astronomer](astronomer.io)[Cosmos](astronomer.io/cosmos)Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow