Sliced Designs for Multi-Platform Online Experiments. Issue 3 (2nd July 2020)

• Record Type: Journal Article
• Authors: Sadeghi, Soheil; Chien, Peter; Arora, Neeraj
• Is Part Of: Technometrics. Volume 62:Issue 3(2020)
• Page Start: 387
• Page End: 402
• Publication Date: 2020-07-02
• Subjects: A/B testing -- Blocking -- Design of experiments -- Digital marketing -- Email testing -- Web experiments; Statistical physics -- Periodicals; Chemistry -- Statistical methods -- Periodicals; Engineering -- Statistical methods -- Periodicals; 519.5
• DOI: 10.1080/00401706.2019.1647288
• Languages: English
• ISSNs: 0040-1706

Abstract

Abstract: Multivariate testing is a popular method to improve websites, mobile apps, and email campaigns. A unique aspect of testing in the online space is that it needs to be conducted across multiple platforms such as a desktop and a smartphone. The existing experimental design literature does not offer precise guidance for such a multi-platform context. In this article, we introduce a multi-platform design framework that allows us to measure the effect of the design factors for each platform and the interaction effect of the design factors with platforms. Substantively, the resulting designs are of great importance for testing digital campaigns across platforms. We illustrate this in an empirical email application to maximize engagement for a digital magazine. We introduce a novel "sliced effect hierarchy principle" and develop design criteria to generate factorial designs for multi-platform experiments. To help construct such designs, we prove a theorem that connects the proposed designs to the well-known minimum aberration designs. We find that experimental versions made for one platform should be similar to other platforms. From the standpoint of real-world application, such homogeneous subdesigns are cheaper to implement. To assist practitioners, we provide an algorithm to construct the designs that we propose.