Vibe Check: Can AI Actually Write Spark? Live Coding w/ Databricks & Snowflake

Watch the video version for the demos and screen shares: https://www.youtube.com/live/jCQx4XvrEUkHolden Karau (Snowflake, formerly Databricks) and Lisa Cao (Databricks DevRel) joined us to dig into what it actually looks like to write Spark code with AI agents in 2026.We worked through a real example from Holden's High Performance Spark Second Edition — a Goldilocks-style exact percentiles problem across hundreds of columns — and watched Claude Code handle it from a cold-start repo, then again with proper context and a reference solution.Lisa walked through using a skill file to get Claude to write declarative pipelines the right way instead of hallucinating the syntax, and we talked about why local mode, Arrow UDFs, and the upcoming Python UDF transpilation work in Spark 4.3 matter for AI-assisted Spark development.We also covered: why the "agents trying to shut you up" failure mode shows up in code generation, the case for property-based testing as a guardrail for agentic coding, Spark 3 to Spark 4 porting as a more reliable path than one-shotting Spark 4, and where agents still struggle (dependency management, Spark internals, real stack trace debugging vs. pattern matching from Stack Overflow).Resources:Apache Spark: https://spark.apache.org & https://github.com/apache/sparkHigh Performance Spark (O'Reilly): https://www.oreilly.comSpark testing base: https://github.com/holdenk/spark-testing-baseDatabricks Spark: https://www.databricks.com/spark/aboutData + AI Summit (June 15–18, San Francisco): https://www.databricks.com/dataaisummitSnowflake Summit (June 1–4, San Francisco): https://www.snowflake.com/summitPySpark Data Sources: https://github.com/allisonwang-db/pyspark-data-sourcesPySpark SDP: https://github.com/lisancao/pyspark-sdpDistributed Computing 4 Kids: https://distributedcomputing4kids.com/What's working for you when you write Spark with agents? Drop your patterns and gotchas below. And let us know what you want to see on future episodes.Originally streamed May 13, 2026 on Vibe Check Temporal YouTube channel and you can watch the full episode at: https://www.youtube.com/live/jCQx4XvrEUk