Twitter Heron: The Path Towards Elastic Streaming
Twitter is all about real-time at scale. To achieve real-time performance, Twitter has developed, deployed and open-sourced Heron, the next-generation cloud streaming engine. The amount of data that need to be processed in Twitter’s data centers changes significantly due to expected and unexpected global events. For example, during the Super Bowl, there are spikes of tweets that all need to be processed in real-time. Similarly, unexpected events such as natural disasters can generate very large volumes of data. In this talk we will describe how Twitter and Microsoft have been collaborating to transform Heron into a truly elastic system that can support dynamic load changes. We'll present how we adapted several components of the system, such as the scheduler and resource manager, to make them able to seamlesly support elasticity. We will also present our future plans for scaling that current and future Heron contributors could tackle.