Roll-Out Phantoms - When Results Look Worse Than They Are

This article explains an often-overlooked effect that can distort metrics during feature roll-out. The primary audience are software engineers and managers working on infrastructure and libraries. I want to motivate this problem with a fictive scenario: Alex wrote a replacement for an old and inefficient component in their app. It took the team a few weeks to implement and they carefully measured on test devices that it improves all core metrics....

2021-01-12 · 4 min · Daniel

Advanced A/B Test Concepts

This article explains advanced A/B test concepts. The primary audience are engineers encountering such A/B tests for the first time. A/B tests allow engineers to validate fixes, test for regressions, and measure improvements. In most situations the standard approach is to create a control and a test group of equal size (e.g. both 5%). Users within the test population (e.g. 10%) are then randomly assigned to one of these. However, sometimes more intricate methods can improve the user experience and make deployment safer and more effective....

2021-01-11 · 6 min · Daniel