Roll-out phantoms - When results look worse than they are
This article explains an often-overlooked effect that can distort metrics during feature roll-out. The primary audience are software engineers and managers working on infrastructure and libraries. I want to motivate this problem with a fictive scenario: Alex wrote a replacement for an old and inefficient component in their app. It took the team a few weeks to implement and they carefully measured on test devices that it improves all core metrics. It is faster, uses less memory, all the good stuff. Following procedure, Alex’s new code is bundled with the next release and they open it to 1% for an A/B test. ...