Stochastic approximation versus sample average approximation for Wasserstein barycenters. (3rd September 2022)

• Record Type: Journal Article
• Authors: Dvinskikh, Darina
• Is Part Of: Optimization methods and software. Volume 37:Number 5(2022)
• Page Start: 1603
• Page End: 1635
• Publication Date: 2022-09-03
• Subjects: Empirical risk minimization -- stochastic approximation -- sample average approximation -- Wasserstein barycenter -- Fréchet mean -- stochastic gradient descent -- mirror descent; Mathematical optimization -- Periodicals; Algorithms -- Periodicals; 519.7
• DOI: 10.1080/10556788.2021.1965600
• Languages: English
• ISSNs: 1055-6788

Abstract

Abstract : In the machine learning and optimization community, there are two main approaches for the convex risk minimization problem, namely the Stochastic Approximation (SA) and the Sample Average Approximation (SAA). In terms of the oracle complexity (required number of stochastic gradient evaluations), both approaches are considered equivalent on average (up to a logarithmic factor). The total complexity depends on a specific problem, however, starting from the work [A. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro, Robust stochastic approximation approach to stochastic programming, SIAM. J. Opt. 19 (2009), pp. 1574–1609] it was generally accepted that the SA is better than the SAA. We show that for the Wasserstein barycenter problem, this superiority can be inverted. We provide a detailed comparison by stating the complexity bounds for the SA and SAA implementations calculating barycenters defined with respect to optimal transport distances and entropy-regularized optimal transport distances. As a byproduct, we also construct confidence intervals for the barycenter defined with respect to entropy-regularized optimal transport distances in the ℓ 2 -norm. The preliminary results are derived for a general convex optimization problem given by the expectation to have other applications besides the Wasserstein barycenter problem.