Getting ROI from Experimentation: How AB Experimentation plays out in Organizations

Experimentation is a critical part of the modern product development lifecycle. In order to implement experimentation effectively, data teams must invest in robust, scalable assignment systems, metrics computation, and complex statistical analysis frameworks. Most companies get started with decentralized analysis, in Jupyter or mode notebooks. Others centralize piecemeal with a looker dashboard and homegrown feature flagging, but leave crucial workflows to be done ad hoc or not at all. This talk will review the core elements of the modern experimentation stack, which components must be closely vetted, and examples of how bad experimentation can be more damaging to an organization than no experimentation at all. Listeners will leave with an understanding of how to spot a faulty experimentation stack with a review of alternative solutions.