The Role of Airflow in Building Smarter ML Pipelines at Vivian Health with Max Calehuff

The integration of data orchestration and machine learning is critical to operational efficiency in healthcare tech. Vivian Health leverages Airflow to power both its ETL pipelines and ML workflows while maintaining strict compliance standards.Max Calehuff, Lead Data Engineer at Vivian Health, joins us to discuss how his team uses Airflow for ML ops, regulatory compliance and large-scale data orchestration. He also shares insights into upgrading to Airflow 3 and the importance of balancing flexibility with security in a healthcare environment.Key Takeaways:00:00 Introduction.04:21 The role of Airflow in managing ETL pipelines and ML retraining.06:23 Using AWS SageMaker for ML training and deployment.07:47 Why Airflow’s versatility makes it ideal for MLOps.10:50 The importance of documentation and best practices for engineering teams.13:44 Automating anonymization of user data for compliance.15:30 The benefits of remote execution in Airflow 3 for regulated industries.18:16 Quality-of-life improvements and desired features in future Airflow versions.Resources Mentioned:Max Calehuffhttps://www.linkedin.com/in/maxwell-calehuff/Vivian Health | LinkedInhttps://www.linkedin.com/company/vivianhealth/Vivian Health | Websitehttps://www.vivian.comApache Airflowhttps://airflow.apache.org/Astronomerhttps://www.astronomer.io/AWS SageMakerhttps://www.google.com/aclk?sa=L&ai=DChsSEwj3-fbz1tiQAxWXlKYDHXUBBVoYACICCAEQABoCdGI&ae=2&aspm=1&co=1&ase=2&gclid=Cj0KCQiA5abIBhCaARIsAM3-zFWbfj2olUvX4dqoiYNaE3q2fMf_ZifRjmbKNQCVX7D6ZMClaUXUkFkaAuwmEALw_wcB&cid=CAASQuRoMccxWhBvMq-1Uez3XOZti1ul7mTDotKvSMoDHv0q2xCsyS2FzMptO5dJf3tmfkLRu22TtD8ChTmdjvs6YetTjQ&cce=2&category=acrcp_v1_35&sig=AOD64_2xE2xolEEVbpDb56qXQluxTzs-Aw&q&nis=4&adurl&ved=2ahUKEwj7le3z1tiQAxWXcvUHHfZePbAQ0Qx6BAgUEAEdbtLabshttps://www.getdbt.com/Cosmoshttps://github.com/astronomer/astronomer-cosmosSplithttps://www.split.io/Snowflakehttps://www.snowflake.com/en/Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow