The Observatorium - Using Machine Learning and Observability Together to Reduce Incident Impact
Service organizations often measure themselves on keeping customer downtime to a minimum. In the complex distributed architectures inherent to many modern tech companies, however, blips are bound to occur, rendering the effectiveness of incident response critical to the customer experience. KPIs such as MTTD and MTTR (Mean Time to Detection/Resolution, respectively) are used to better understand the efficiency of said incident response, and maturing organizations would be wise to leverage tooling to improve these metrics. In a maturing global company such as DigitalOcean, distributed systems reign supreme, and with them the myriad microservices that generate metrics and data (and duly need to be observed effectively). Accordingly, we’ve built a platform named The Observatorium, whose primary goal is to reduce MTTD/MTTR across our cloud; we do so by curating and shepherding information in creative-yet-efficient ways, which I’ll discuss in more depth in this talk.