Spark Job Definitions: Notebooks Are for Humans, SJDs Are for Machines

Spark Job Definitions: Notebooks Are for Humans, SJDs Are for Machines Episode 6 • 2026-02-06 Duration: 10:23 Matthias and Fabia break down Spark Job Definitions in Microsoft Fabric — the production wrapper most teams skip until their scheduled notebook fails at 3 AM. They cover the notebook-to-SJD promotion path, why state leakage kills overnight runs, retry policies, and when a plain notebook schedule is actually fine. What we discuss A real-world mistake from a pre-Fabric era The one question that reframes the architectural debate How we got here — predecessor products and evolution Why the "obvious" answer is often wrong A real Reddit/Microsoft Q&A question unpacked The concrete recommended architecture F-SKU realism — what this actually costs When the rejected approach is actually right Risks of the recommended path What Microsoft is shipping that changes the calculus The architectural principle to take home Key takeaways And wrap it in a Pipeline. Even if just for the alerts. Fair. Solo workflows where the output includes visualizations — scheduled notebook is the right call. The line I draw: the moment a second person depends on that output, or it feeds a downstream system, promote it. Solo analyst with a... Every SJD run starts clean. No leftover variables, no stale session. That's — I mean, that's the whole point. Reproducibility by design, not by discipline. Resources What is a Spark Job Definition? Create SJD Run SJD SJD Git Integration Pipeline SJD Activity Streaming Data into Lakehouse Spark Best Practices Job Queueing Decision Guide: Pipeline vs Dataflow vs Spark Fabric vs Synapse Spark Comparison MS Q&A Reddit r/MicrosoftFabric Stack Overflow About the show Built on ElevenLabs voice synthesis. Matthias — cloned voice. Fabia — designed AI co-host. See Matthias live on YouTube (Fabric Friday), at his meetups, and at conferences like FabCon. Hosted by Matthias Falland — Microsoft Data Platform MVP and community architect behind the Fabric Periodic Table. New episodes every Friday. Submit your case Have an architecture decision you are wrestling with? DM Matthias on LinkedIn — find him as Matthias Falland. Three to five sentences about the decision, your team size, and your current stack. We anonymize before airing. Built on ElevenLabs voice synthesis. Brand design based on fabricperiodictable.com.