My Notes From Spark+AI Summit 2020 (Application-Specific Talks)

Eugene Yan · 6 years ago

The article summarizes technical talks from Spark+AI Summit 2020 covering data engineering frameworks and machine learning infrastructure, including Airbnb's Zipline for point-in-time feature engineering, Airbnb's Sputnik for reducing boilerplate in Spark jobs, Gojek's Feast feature store for decoupling ML pipelines, Netflix's data quality approaches using statistical tests, and applications of unsupervised and reinforcement learning. Zipline reduces computation time from O(N²) to O(NLogN) for non-reversible aggregations using binary trees, while Sputnik achieved 15% storage cost savings through better detection of unused data. These tools enable organizations to scale machine learning systems by improving code reusability, data consistency between training and serving, and proactive error detection in data pipelines.