Appifying Data Science Workflows to Create Composable, User-Friendly Data Pipeline Products
At Quid, our flagship product is a SaaS research platform which allows users to gain full-picture insights about written content like news. Over the past year, we built an internal framework so that data scientists and data engineers can encapsulate tailored product use cases (e.g., "understand how a company and its competitors are being covered in the media" or "find key opinion leaders in the news related to some topic") into single apps. This framework has allowed for the modularization of data science capabilities and has dramatically lowered the effort needed to see data science realized in application form. We use Netflix's open-source orchestration system, Conductor, to isolate infrastructure concerns and allow data science teams to safely own products from prototype to production. In this talk I'll cover: * Big-picture transferable lessons that we’ve learned the hard way, that you can apply yourself to solve your team’s data science workflow challenges * How the structure of our data science and data engineering team has evolved to utilize this framework * How we leverage data from our flagship product to discover what apps to build next