Orchestrating Dynamic Data Flows, AgileData Engineering Pattern #1 - Episode #67

Episode 67 of the Agile Data N’ Info podcast, hosted by AgileData, titled "Orchestrating Dynamic Data Flows, AgileData Engineering Pattern #1 - Episode #67" was published on June 30, 2025 and runs 22 minutes.

June 30, 2025 ·22m · Agile Data N’ Info

0:00 / 0:00

Summary

Join Shane Gibson and Nigel Vining as they describe and discuss the AgileData Engineering Pattern of Orchestrating Dynamic Data Flows.The Dynamic Data Flow Orchestration pattern dynamically generates and self-heals data flow manifests (DAGs) at runtime from a central context database, enabling adaptive, cost-effective, and low-latency data processing primarily for micro-batching workloadsAn AgileData Engineering Pattern is a repeatable, proven approach for solving a common data engineering challenge in a simple, consistent, and scalable way, designed to reduce rework, speed up delivery, and embed quality by default. If you want a copy of the pattern template head over to: https://agiledata.substack.com/i/167144707/pattern-name Discover more AgileData Engineering Patterns over at https://agiledata.substack.com/s/agiledata-engineering-patterns If you just want to talk about making magic happen with agile and data you can connect with Shane @shagility on LinkedIn. Subscribe: Apple Podcast | Spotify | Google Podcast | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser | Deezer | Podcast Addict | Buy the Green Book now! Simply Magical Data Ways of Working

Episode Description

Join Shane Gibson and Nigel Vining as they describe and discuss the AgileData Engineering Pattern of Orchestrating Dynamic Data Flows. The Dynamic Data Flow Orchestration pattern dynamically generates and self-heals data flow manifests (DAGs) at runtime from a central context database, enabling adaptive, cost-effective, and low-latency data processing primarily for micro-batching workloads An AgileData Engineering Pattern is a repeatable, proven approach for solving a common data engineering challenge in a simple, consistent, and scalable way, designed to reduce rework, speed up delivery, and embed quality by default.

If you want a copy of the pattern template head over to:

https://agiledata.substack.com/i/167144707/pattern-name

Discover more AgileData Engineering Patterns over at https://agiledata.substack.com/s/agiledata-engineering-patterns

If you just want to talk about making magic happen with agile and data you can connect with Shane @shagility on LinkedIn.

Subscribe: Apple Podcast | Spotify | Google Podcast | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser | Deezer | Podcast Addict |

Buy the Green Book now!

Simply Magical Data Ways of Working

Share this episode

Similar Episodes

How to Stay Consistent With Habits Using Habit Stacking

Apr 11, 2026 ·1m

Mar 24, 2026 ·30m

Mar 15, 2026 ·10m

2 Corinthians 11-13

Mar 15, 2026 ·11m

2 Corinthians 6-10

Mar 13, 2026 ·13m

2 Corinthians 1-5

Mar 11, 2026 ·15m

Similar Podcasts

PlanningBI PlanningBI Além de especialista em OKR e BSC é gestor de TI e cientista de dados, com mais de 14 anos conectando tecnologia a planejamento estratégico, indicadores e resultados. Lidera times multidisciplinares (Infra, BI, SAP, RPA e Processos), estruturando Data Lake, pipelines de BI e KPIs executivos para decisão ágil. É criador da plataforma PlanningBI, que integra OKR + BSC para alinhar metas, executar com governança e medir impacto. Ao longo da carreira, automatizou rotinas críticas, reduziu horas manuais e elevou a performance operacional e financeira, sempre com foco em previsibilidade e ROI. Agile: Make your clients benefit from Scrum Gilbert NZEKA Become an expert in Agile & Scrum project management.To register for the full video course, visit: https://bit.ly/udemyscrum.——————————————————————💙 If you haven't yet, don't hesitate to follow me!——————————————————————◾ Web : https://www.ilaria-academy.com/◾ Youtube : http://bit.ly/2xwrpEf ◾ Medium : https://medium.com/@gilbertnzeka ◾ Facebook : https://www.facebook.com/GilbertNzeka ◾ Instagram : https://www.instagram.com/gilbertnzeka/ ◾ Twitter : https://www.instagram.com/gilbertnzeka◾ LinkedIn : https://www.linkedin.com/in/gilbertnzeka/ ◾ Eventbrite : http://bit.ly/gilbertnzeka-eventbrite ◾ Udemy : https://www.udemy.com/user/gilbert-nzeka/ Agile Engineering Podcast Agile Engineering Podcast Join our hosts Liam Gulliver, Louise Paling, Mica Bell, Jonathan Relf and Pete Gallagher for Agile Engineering Podcast! Get insights on Agile, DevOps, software engineering, and team leadership from subject experts that draw from their experience in their day jobs in wide and varied backgrounds across multiple industries such as aviation, IoT, communications, retail, oil and gas, FinTech and recruitment. Do Agile Be Agile Podcast: Agile Practices | Soft Skills | Agile Mindset Mario RT: Agile Coach, Instructor y Consultor Do Agile Be Agile, se transmite cada 15 días, hablamos sobre tener una perspectiva integral, no solo hacer ágil (Do agile), si no, ser ágil (Be agile). Sigan a Mario Rodríguez (Mario RT) Agile Coach, Instructor y Consultor a través de: prácticas y mindset ágil, cultura, personas y cambio. Algunos de los temas: Agile Mindset, Change Management, Soft Skills (Equipos de Alto Rendimiento, Negociación, Autogestión, Productividad, Comunicación), Frameworks Ágiles (Scrum, Kanban, Design Thinking, Lean Startup) Herramientas y más, con entrevistas a Agilistas, Coaches e Instructores.

URL copied to clipboard!