Mathematical constraints on FST: multiallelic markers in arbitrarily many populations. (6th June 2022) - Alcala, Nicolas; Rosenberg, Noah A.
Abstract : Interpretations of values of the F ST measure of genetic differentiation rely on an understanding of its mathematical constraints. Previously, it has been shown that F ST values computed from a biallelic locus in a set of multiple populations and F ST values computed from a multiallelic locus in a pair of populations are mathematically constrained as a function of the frequency of the allele that is most frequent across populations. We generalize from these cases to report here the mathematical constraint on F ST given the frequency M of the most frequent allele at a multiallelic locus in a set of multiple populations. Using coalescent simulations of an island model of migration with an infinitely-many-alleles mutation model, we argue that the joint distribution of F ST and M helps in disentangling the separate influences of mutation and migration on F ST . Finally, we show that our results explain a puzzling pattern of microsatellite differentiation: the lower F ST in an interspecific comparison between humans and chimpanzees than in the comparison of chimpanzee populations. We discuss the implications of our results for the use of F ST . This article is part of the theme issue 'Celebrating 50 years since Lewontin's apportionment of human diversity'.