Building a Unified Feature Platform with DuckDB and Arrow
Transforming raw data into features to power machine learning models is one of the biggest challenges in production ML. At Tecton, we’re building a Feature Platform to help Data Scientists and ML Engineers tackle this problem. Recently, we’ve launched a new version of our product based on DuckDB and Arrow to complement the existing Spark- and Snowflake-based versions. Our goals were to provide a fast, delightful local development experience, decrease time-to-deployment for our customers, and provide flexibility in integrations with external systems. In this talk, I will cover our decision to build this new product, our experiences with the implementation, and the outcomes to date. I will dive into technical details of the implementation, including: Strategies for scaling up to larger dataset sizes without a distributed query engine Implementing and distributing DuckDB extensions Building interoperability with DeltaLake, outside of the Spark ecosystem Performance comparisons with Spark Leveraging cloud resources to scale up beyond laptop-size datasets without losing the laptop developer experience