Mathematical Constraints on FST: Biallelic Markers in Arbitrarily Many Populations. Issue 3 (1st July 2017) - Alcala, Nicolas; Rosenberg, Noah A
Abstract: F ST is one of the most widely used statistics in population genetics. Recent mathematical studies have identified constraints that challenge interpretations of F ST as a measure with potential to range from 0 for genetically similar populations to 1 for divergent populations. We generalize results obtained for population pairs to arbitrarily many populations, characterizing the mathematical relationship between F ST, the frequency M of the more frequent allele at a polymorphic biallelic marker, and the number of subpopulations K . We show that for fixed K, F ST has a peculiar constraint as a function of M, with a maximum of 1 only if M = i / K, for integers i with ⌈ K / 2 ⌉ ≤ i ≤ K − 1. For fixed M, as K grows large, the range of F ST becomes the closed or half-open unit interval. For fixed K, however, some M < ( K − 1 ) / K always exists at which the upper bound on F ST lies below 2 2 − 2 ≈ 0.8284. We use coalescent simulations to show that under weak migration, F ST depends strongly on M when K is small, but not when K is large. Finally, examining data on human genetic variation, we use our results to explain the generally smaller F ST values between pairs of continents relative to global F ST values. We discuss implications for the interpretation and use of F ST .