Using GIT as a NoSQL Database for Fine-Grained Control Over the Data Pipeline

In this talk, Hung will reveal how Y42, an all-in-one data pipeline tool, has leveraged Git as a noSQL database to foster unparalleled collaboration opportunities between data engineers and data analysts. Y42’s decision to abandon all classical databases within their platform in favor of using Git has brought a series of major innovations — but this cutting-edge approach hasn’t been without challenges. This talk will highlight the extraordinary benefits of using Git-as-a-database to power an all-in-one data workspace, such as: Data-pipeline-as-code (including dashboard-as-code and integrations-as-code) Browser-based, high-performance implementation of Git using WebAssembly, meaning the Y42-data-pipeline can be implemented using no-code, low-code and/or code Simple end-to-end templating Easy version control + rollback + environment of job status, settings and data warehouse tables Coherent pipeline automations with one orchestration layer And it doesn’t end there… However, deciding to use this unconventional approach meant they were in for a bumpy ride. The Y42 team found themselves facing a series of challenges, including: Performance issues to save thousands of jobs inside Git Access control issues using folder paths as ids Getting the web app to seamlessly integrate Git and code The Y42 team invites you to attend this talk and learn more about using Git as a noSQL database for a new data revolution. It’s a bold statement, but they believe this talk has the potential to significantly shape the data industry for the years to come.