A fundamental problem of hypothesis testing with finite inventory in e‐commerce. (7th October 2020)

• Record Type: Journal Article
• Authors: Bohle, Dennis; Marynych, Alexander; Meiners, Matthias
• Other Names: Srivastava Gautam guestEditor.
• Is Part Of: Applied stochastic models in business and industry. Volume 37:Number 3(2021)
• Page Start: 454
• Page End: 474
• Publication Date: 2020-10-07
• Subjects: A/B test -- conversion rate -- functional limit theorem -- two‐dimensional random walk in a semiinfinite strip; Stochastic analysis -- Periodicals; Stochastic processes -- Periodicals; Business mathematics -- Periodicals; Finance -- Mathematical models -- Periodicals; Industrial management -- Mathematical models -- Periodicals; 338.00151923
• DOI: 10.1002/asmb.2574
• Languages: English
• ISSNs: 1524-1904

Abstract

Abstract: We draw attention to a problem that is often overlooked or ignored by companies practicing hypothesis testing (A/B testing) in online environments. We show that conducting experiments on limited inventory that is shared between variants in the experiment can lead to high false‐positive rates since the core assumption of independence between the groups is violated. We provide a detailed analysis of the problem in a simplified setting whose parameters are informed by realistic scenarios. The setting we consider is a two‐dimensional (2D) random walk in a semiinfinite strip. It is rich enough to take a finite inventory into account, but is at the same time simple enough to allow for a closed form of the false‐positive probability. We prove that high false‐positive rates can occur, and develop tools that are suitable to help design adequate tests in follow‐up work. Our results also show that high false‐negative rates may occur. The proofs rely on a functional limit theorem for the 2D random walk in a semiinfinite strip.