## The Annals of Applied Probability

- Ann. Appl. Probab.
- Volume 13, Number 1 (2003), 181-212.

### When can one detect overdominant selection in the infinite-alleles model?

Paul Joyce, Stephen M. Krone, and Thomas G. Kurtz

#### Abstract

One of the goals of this paper is to show that the infinite-alleles model with overdominant selection "looks like" the neutral infinite-alleles model when the selection intensity and mutation rate get large together. This rather surprising behavior was noticed by Gillespie (1999) in simulations. To make rigorous and refine Gillespie's observations, we analyze the limiting behavior of the likelihood ratio of the stationary distributions for the model under selection and neutrality, as the mutation rate and selection intensity go to $\infty$ together in a specified manner. In particular, we show that the likelihood ratio tends to 1 as the mutation rate goes to $\infty$, provided the selection intensity is a multiple of the mutation rate raised to a power less than $3/2$. (Gillespie's simulations correspond to the power 1.) This implies that we cannot distinguish between the two models in this setting. Conversely, if the selection intensity grows like a multiple of the mutation rate raised to a power greater than $3/2$, selection can be detected; that is, the likelihood ratio tends to 0 under neutrality and $\infty$ under selection. We also determine the nontrivial limit distributions in the case of the critical exponent $3/2$. We further analyze the limiting behavior when the exponent is less than $3/2$ by determining the rate at which the likelihood ratio converges to 1 and by developing results for the distributions of finite samples.

#### Article information

**Source**

Ann. Appl. Probab., Volume 13, Number 1 (2003), 181-212.

**Dates**

First available in Project Euclid: 16 January 2003

**Permanent link to this document**

https://projecteuclid.org/euclid.aoap/1042765666

**Digital Object Identifier**

doi:10.1214/aoap/1042765666

**Mathematical Reviews number (MathSciNet)**

MR1951997

**Zentralblatt MATH identifier**

1011.62115

**Subjects**

Primary: 62F05: Asymptotic properties of tests

Secondary: 60G42: Martingales with discrete parameter 60G15: Gaussian processes

**Keywords**

Infinite-alleles model overdominant selection homozygosity mutation rate likelihood ratio Ewens sampling formula Poisson-Dirichlet weak limits

#### Citation

Joyce, Paul; Krone, Stephen M.; Kurtz, Thomas G. When can one detect overdominant selection in the infinite-alleles model?. Ann. Appl. Probab. 13 (2003), no. 1, 181--212. doi:10.1214/aoap/1042765666. https://projecteuclid.org/euclid.aoap/1042765666