## The Annals of Applied Probability

### When can one detect overdominant selection in the infinite-alleles model?

#### Abstract

One of the goals of this paper is to show that the infinite-alleles model with overdominant selection "looks like" the neutral infinite-alleles model when the selection intensity and mutation rate get large together. This rather surprising behavior was noticed by Gillespie (1999) in simulations. To make rigorous and refine Gillespie's observations, we analyze the limiting behavior of the likelihood ratio of the stationary distributions for the model under selection and neutrality, as the mutation rate and selection intensity go to $\infty$ together in a specified manner. In particular, we show that the likelihood ratio tends to 1 as the mutation rate goes to $\infty$, provided the selection intensity is a multiple of the mutation rate raised to a power less than $3/2$. (Gillespie's simulations correspond to the power 1.) This implies that we cannot distinguish between the two models in this setting. Conversely, if the selection intensity grows like a multiple of the mutation rate raised to a power greater than $3/2$, selection can be detected; that is, the likelihood ratio tends to 0 under neutrality and $\infty$ under selection. We also determine the nontrivial limit distributions in the case of the critical exponent $3/2$. We further analyze the limiting behavior when the exponent is less than $3/2$ by determining the rate at which the likelihood ratio converges to 1 and by developing results for the distributions of finite samples.

#### Article information

Source
Ann. Appl. Probab., Volume 13, Number 1 (2003), 181-212.

Dates
First available in Project Euclid: 16 January 2003

https://projecteuclid.org/euclid.aoap/1042765666

Digital Object Identifier
doi:10.1214/aoap/1042765666

Mathematical Reviews number (MathSciNet)
MR1951997

Zentralblatt MATH identifier
1011.62115

Subjects
Primary: 62F05: Asymptotic properties of tests
Secondary: 60G42: Martingales with discrete parameter 60G15: Gaussian processes

#### Citation

Joyce, Paul; Krone, Stephen M.; Kurtz, Thomas G. When can one detect overdominant selection in the infinite-alleles model?. Ann. Appl. Probab. 13 (2003), no. 1, 181--212. doi:10.1214/aoap/1042765666. https://projecteuclid.org/euclid.aoap/1042765666

#### References

• ETHIER, S. N. and KURTZ, T. G. (1994). Convergence to Fleming-Viot processes in the weak atomic topology. Stochastic Process. Appl. 54 1-27.
• GILLESPIE, J. H. (1999). The role of population size in molecular evolution. Theoretical Population Biology 55 145-156.
• GRIFFITHS, R. C. (1979). On the distribution of allele frequencies in a diffusion model. Theoretical Population Biology 15 140-158.
• GRIFFITHS, R. C. (1988). Distribution of F in the infinitely-many-alleles model. Statistics Research Report 183, Dept. Mathematics, Monash Univ.
• JOy CE, P., KRONE, S. M. and KURTZ, T. G. (2002). Gaussian limits associated with the Poisson- Dirichlet distribution and the Ewens sampling formula. Ann. Appl. Probab. 12 101-124.
• KINGMAN, J. F. C. (1977). The population structure associated with the Ewens sampling formula. Theoretical Population Biology 11 274-283.
• MOSCOW, IDAHO 83844 E-MAIL: joy ce@uidaho.edu S. M. KRONE DEPARTMENT OF MATHEMATICS UNIVERSITY OF IDAHO
• MOSCOW, IDAHO 83844 E-MAIL: krone@uidaho.edu T. G. KURTZ DEPARTMENTS OF MATHEMATICS AND STATISTICS UNIVERSITY OF WISCONSIN
• MADISON, WISCONSIN 53706-1388 E-MAIL: kurtz@math.wisc.edu