Building a Control Plane for Data

Keynote: March 28th @ 9:00am Data Discovery, Data Observability, Data Quality, Data Governance, Data Management have traditionally been addressed as individual problems with distinct solutions. However, business challenges like access management, data retention, cost attribution and optimization, and end-to-end data quality require these tools to work together. Without a harmonizing layer across systems, these issues are impossible to manage in a uniform manner across the stack. We are calling this harmonizing layer the ‘Control Plane for Data’ - powered by the common thread across these systems: Metadata. In this talk, we’ll describe what the control plane of data looks like and how it fits into the reference architecture for the deconstructed data stack: a data stack that includes operational data stores, streaming systems, transformation engines, BI tools, warehouses, ML tools and orchestrators. We’ll dig into the fundamental characteristics for a control plane: Breadth (completeness) Latency (freshness) Scale Source of Truth Auditability We’ll discuss what use-cases you can accomplish with a unified control plane and why this leads to a simpler, more flexible data stack. Finally, we’ll share thoughts on how the industry should evolve to enable this vision and how a constellation of projects including DataHub are working towards making this a reality. If you’re a data leader that has accumulated a ton of specialized data tools in a short amount of time and are wondering how to restore order back into your data teams’ daily lives, this talk is for you.