Episode 56: Pipeline chaining, meta pipelines and automation (Part 1 of 2)

Episode 56 of The Nextflow Podcast (March 2026) focuses on pipeline chaining and meta pipelines in Nextflow, with guests Ben Sherman and Edmund Miller. The episode was split into two parts: this first part surveys current solutions, while part two will cover future Nextflow language changes. The discussion defines meta pipelines as importing pipelines (e.g., nf-core/rnaseq) as subworkflows to form one DAG with parallelization and full resume, versus pipeline chaining using external orchestration to run Pipeline A then feed outputs to Pipeline B. They cover obstacles to meta pipelines in nf-core, including tooling, parameter/config clashes, and tight coupling of pipeline code and configuration. Current chaining approaches include bash/Makefiles, Python, Seqera Platform APIs, nf-cascade (running Nextflow inside Nextflow), wrapping Nextflow with Snakemake, and automation/orchestration tools like Node-RED, n8n, Dagster, and Temporal, including event-driven patterns on AWS.00:00 Nextflow Podcast, Episode 5600:08 Welcome01:42 Introduction to meta pipelines and pipeline chaining05:01 What makes importing pipelines difficult?06:55 CLI tooling to import pipelines09:13 Overlapping config scopes10:53 Subworkflows or pipelines?12:38 Pipeline chaining13:36 nf-cascade16:42 Nextflow in Snakemake22:26 Automating Nextflow runs24:10 Event-driven bioinformatics26:32 Node-RED + Seqera30:40 Node-RED flexibility33:45 Glue code35:31 Other automation frameworks37:02 Bioinformatics pipelines vs. ETL workflows38:56 Tangent: What makes Nextflow special44:11 Dagster automation demo47:29 Temporal automation demo51:13 Wrap up