Functional Data Engineering - A Set of Best Practices

Batch data processing (also known as ETL)  is time-consuming, brittle, and often unrewarding. Not only that, it’s hard to operate, evolve, and troubleshoot. In this talk, we’ll discuss functional programming paradigm and explore how applying it to Data Engineering can bring a lot of clarity to the process. It helps solving some of the inherent problems of ETL, leads to more manageable and maintainable workloads and helps to implement reproducible and scalable practices. It empowers data teams to tackle larger problems and push the boundaries of what’s possible.