Apache Spark was created as a unified analytics engine in the context of "big data". The past few years have witnessed the meteoric rise of machine learning. Data scientists and engineers are now building sophisticated ML applications with tool sprawl. In this talk, we will discuss the challenges organizations face in this new world, and how we envision to tackle these challenges with two new open source projects: Delta and MLflow.
Reynold Xin is a cofounder and Chief Architect at Databricks. In the open source community, Reynold is known as the top contributor to the Apache Spark project, having designed its core user-facing APIs and execution engine. Prior to Databricks, Reynold was pursuing a PhD at the UC Berkeley AMPLab, where he worked on large-scale data processing.