The Annals of Applied Probability

Coalescence in a random background

N. H. Barton, A. M. Etheridge, and A. K. Sturm

Full-text: Open access


We consider a single genetic locus which carries two alleles, labelled P and Q. This locus experiences selection and mutation. It is linked to a second neutral locus with recombination rate r. If r=0, this reduces to the study of a single selected locus. Assuming a Moran model for the population dynamics, we pass to a diffusion approximation and, assuming that the allele frequencies at the selected locus have reached stationarity, establish the joint generating function for the genealogy of a sample from the population and the frequency of the P allele. In essence this is the joint generating function for a coalescent and the random background in which it evolves. We use this to characterize, for the diffusion approximation, the probability of identity in state at the neutral locus of a sample of two individuals (whose type at the selected locus is known) as solutions to a system of ordinary differential equations. The only subtlety is to find the boundary conditions for this system. Finally, numerical examples are presented that illustrate the accuracy and predictions of the diffusion approximation. In particular, a comparison is made between this approach and one in which the frequencies at the selected locus are estimated by their value in the absence of fluctuations and a classical structured coalescent model is used.

Article information

Ann. Appl. Probab., Volume 14, Number 2 (2004), 754-785.

First available in Project Euclid: 23 April 2004

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 60J80: Branching processes (Galton-Watson, birth-and-death, etc.) 60J85: Applications of branching processes [See also 92Dxx] 60J70: Applications of Brownian motions and diffusion theory (population genetics, absorption problems, etc.) [See also 92Dxx] 60K35: Interacting random processes; statistical mechanics type models; percolation theory [See also 82B43, 82C43]

Coalescent selection recombination identity by descent random environment


Barton, N. H.; Etheridge, A. M.; Sturm, A. K. Coalescence in a random background. Ann. Appl. Probab. 14 (2004), no. 2, 754--785. doi:10.1214/105051604000000099.

Export citation


  • Barton, N. H., Depaulis, F. and Etheridge, A. M. (2002). Neutral evolution in spatially continuous populations. Theoretical Population Biology 61 31–48.
  • Barton, N. H. and Etheridge, A. M. (2004). The effect of selection on the distribution of coalescence times. Genetics. To appear.
  • Barton, N. H. and Navarro, A. (2002). Extending the coalescent to multilocus systems: The case of balancing selection. Genetical Research 79 129–139.
  • Darden, T., Kaplan, N. L. and Hudson, R. B. (1989). A numerical method for calculating moments of coalescent times in finite populations with selection. J. Math. Biol. 27 355–368.
  • Donnelly, P. J. and Kurtz, T. G. (1999). Genealogical processes for Fleming–Viot models with selection and recombination. Ann. Appl. Probab. 9 1091–1148.
  • Ethier, S. N. and Kurtz, T. G. (1986). Markov Processes: Characterization and Convergence. Wiley, New York.
  • Feller, W. M. (1966). An Introduction to Probability Theory 2. Wiley, New York.
  • Herbots, H. M. (1997). The structured coalescent. In Progress of Population Genetics and Human Evolution (P. Donnelly and S. Tavare, eds.) 231–255. Springer, New York.
  • Kaplan, N. L., Darden, T. and Hudson, R. B. (1988). The coalescent process in models with selection. Genetics 120 819–829.
  • Kingman, J. F. C. (1982a). On the genealogy of large populations. J. Appl. Probab. 19A 27–43.
  • Kingman, J. F. C. (1982b). The coalescent. Stochastic Process. Appl. 13 235–248.
  • Krone, S. M. and Neuhauser, C. (1997). Ancestral processes with selection. Theoretical Population Biology 51 210–237.
  • Nagylaki, T. (1989). Gustave Malecot and the transition from classical to modern population genetics. Genetics 122 253–268.
  • Nordborg, M. (1997). Structured coalescent processes on different timescales. Genetics 146 1501–1514.
  • Notohara, M. (1990). The coalescent and the genealogical process in geographically structured populations. J. Math. Biol. 31 841–852.
  • Przeworski, M., Charlesworth, B. and Wall, J. D. (1999). Genealogies and weak purifying selection. Molecular Biology Evolution 16 246–252.
  • Simmons, G. F. (1972). Differential Equations. McGraw-Hill,\goodbreak New York.
  • Stroock, D. W. and Varadhan, S. R. S. (1979). Multidimensional Diffusion Processes. Springer, New York.