Building a Scalable Real-Time Data Pipeline

Delivery Hero is a company running in 40+ countries in which several entities run their own systems and storages around the world. Building new global micro-services with real-time data processing gets complicated in such an environment. Therefore, we built a global data pipeline, called “Data Fridge” which provides data normalization and data validation to downstream consumers. Our data pipeline is fully serverless and provides three different ways to consume data (HTTP push, HTTP pull and SQL batch for big data processing). In this session, I will cover in detail how we have built this data pipeline using several AWS and GCP services. I will describe why we chose these services and what are the challenges and limitations we have faced so far.