Flyte: Cloud Native Machine Learning & Data Processing Platform
Flyte is the backbone for large-scale Machine Learning and Data Processing (ETL) pipelines at Lyft. It is used across business critical applications ranging from ETA, Pricing, Mapping, Autonomous, etc. At its core is an open source workflow engine that executes 15M+ containers per month as part of thousands of workflows. The talk will focus on:- Architecture of Flyte and its specification language to orchestrate compute and manage data flow across disparate systems like Spark, Flink, Tensorflow, Hive, etc.- Data Provenance and Lineage in Flyte- How to leverage Flyte in various parts of the machine learning pipelinesThe talk will conclude with a demo of a machine learning pipeline built using the open source version of Flyte.