The time it takes to run the tests for your Spark application matter. Have you ever had a build held up by slow or sometimes unreliable Spark integration tests? Been frustrated by how long it takes to develop or debug Spark application tests? This talk looks at practical configurations for making your tests faster and strategies to solve common problems with test data and state management. Learn how to configure your JVM and local Spark instances for maximum speed in your CI environment. Review different test fixture strategies for Spark sessions and when to use each. Power up with test frameworks that not only eliminate boilerplate but also combat common causes of flaky tests. Conquer test log spam. Consider how to make external dependencies in your pipeline, like reading from and writing to cloud storage, more resilient. A top to bottom, front to back guide to making your Spark tests run clean.
Source code spelunker, build file whisperer, query plan rat terrier, Discord lurker, JVM hedge witch. Just your average backend engineer lifer, really.