Panel: Data Lineage We’ve Come a Long Way
Lineage has long been a requirement for anyone processing data - whether for complying with regulations, ensuring data reliability or, to quote Marvin Gaye, plainly just knowing what’s going on from provenance to impact analysis. However, our industry has historically had difficulties collecting data lineage reliably. From the early days of lineage powered by spreadsheets, we’ve come a long way towards standardizing lineage. We have evolved from painful, manual approaches to automated operational lineage extraction across batch and stream processing. Now, we’re on the brink of a new era when lineage will be built into every data processing layer - whether ETL, data warehouse or ai - and not an afterthought. In this panel, OpenLineage project lead Julien Le Dem will ask professionals from the Data catalog and data observability space how their experience building products that rely on lineage has evolved over the past 10 years.