Session Description

As Spark DataFrames have become the default for large-scale batch processing, many teams quietly accept a dangerous trade-off: great performance and flexibility at the cost of weak guarantees about data correctness. Meanwhile, AI agents are starting to write more and more of these pipelines, making it easier to introduce subtle bugs that tests and reviews miss.

In this talk, we’ll examine a realistic Spark job written with the standard Spark DataFrame API and surface concrete failure modes: mis-typed column names caught only at runtime, silent schema and type mismatches in joins, invalid aggregations, and nulls propagating unnoticed into critical business metrics. We’ll then rewrite the same pipeline using Wick, a type-safe Spark API that models schemas as Scala types, tracks nullability explicitly, and constrains aggregations and joins at compile time.

We’ll explore how Wick integrates with Spark’s Catalyst optimizer to retain full query performance while making illegal states unrepresentable for your pipelines. Finally, we’ll connect this to AI-assisted development: how stronger types, precise compile-time errors, and better tooling give both humans and AI agents the context they need to generate correct Spark pipelines faster and with fewer iterations.