The Annals of Applied Probability

Dynamics of cancer recurrence

Jasmine Foo and Kevin Leder

Full-text: Open access


Mutation-induced drug resistance in cancer often causes the failure of therapies and cancer recurrence, despite an initial tumor reduction. The timing of such cancer recurrence is governed by a balance between several factors such as initial tumor size, mutation rates and growth kinetics of drug-sensitive and resistance cells. To study this phenomenon we characterize the dynamics of escape from extinction of a subcritical branching process, where the establishment of a clone of escape mutants can lead to total population growth after the initial decline. We derive uniform in-time approximations for the paths of the escape process and its components, in the limit as the initial population size tends to infinity and the mutation rate tends to zero. In addition, two stochastic times important in cancer recurrence will be characterized: (i) the time at which the total population size first begins to rebound (i.e., become supercritical) during treatment, and (ii) the first time at which the resistant cell population begins to dominate the tumor.

Article information

Ann. Appl. Probab., Volume 23, Number 4 (2013), 1437-1468.

First available in Project Euclid: 21 June 2013

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 60J80: Branching processes (Galton-Watson, birth-and-death, etc.)

Branching processes population genetics cancer


Foo, Jasmine; Leder, Kevin. Dynamics of cancer recurrence. Ann. Appl. Probab. 23 (2013), no. 4, 1437--1468. doi:10.1214/12-AAP876.

Export citation


  • [1] Billingsley, P. (1999). Convergence of Probability Measures, 2nd ed. Wiley, New York.
  • [2] Chmielecki, J., Foo, J. et al. (2011). Optimization of dosing for EGFR-mutant non-small cell lung cancer with evolutionary cancer modeling. Science Translational Medicine 3 90ra59.
  • [3] Coldman, A. J. and Goldie, J. H. (1986). A stochastic model for the origin and treatment of tumors containing drug-resistant cells. Bull. Math. Biol. 48 279–292.
  • [4] Foo, J. and Michor, F. (2010). Evolution of resistance to anti-cancer therapy during general dosing schedules. J. Theoret. Biol. 263 179–188.
  • [5] Goldie, J. H. and Coldman, A. J. (1979). A mathematic model for relating the drug sensitivity of tumors to their spontaneous mutation rate. Cancer Treat. Rep. 63 1727–1733.
  • [6] Goldie, J. H. and Coldman, A. J. (1983). Quantitative model for multiple levels of drug resistance in clinical tumors. Cancer Treat. Rep. 67 923–931.
  • [7] Harnevo, L. and Agur, Z. (1992). Drug resistance as a dynamic process in a model for multistep gene amplification under various levels of selection stringency. Cancer Chemotherapy Pharmacology 30 469–476.
  • [8] Harnevo, L. E. and Agur, Z. (1991). The dynamics of gene amplification described as a multitype compartmental model and as a branching process. Math. Biosci. 103 115–138.
  • [9] Iwasa, Y., Michor, F. and Nowak, M. A. (2003). Evolutionary dynamics of escape from biomedical intervention. Proc. Roy. Soc. Lond. B 270 2572–2578.
  • [10] Jagers, P., Klebaner, F. C. and Sagitov, S. (2007). Markovian paths to extinction. Adv. in Appl. Probab. 39 569–587.
  • [11] Jagers, P., Klebaner, F. C. and Sagitov, S. (2007). On the path to extinction. Proc. Natl. Acad. Sci. USA 104 6107–6111.
  • [12] Komarova, N. (2006). Stochastic modeling of drug resistance in cancer. J. Theoret. Biol. 239 351–366.
  • [13] Komarova, N. L. and Wodarz, D. (2005). Drug resistance in cancer: Principles of emergence and prevention. Proc. Natl. Acad. Sci. USA 102 9714–9719.
  • [14] Michaelson, J. S., Halpern, E. and Kopans, D. B. (1999). Breast cancer: Computer simulation method for estimating optimal intervals for screening. Radiology 212 551–560.
  • [15] Michor, F., Nowak, M. A. and Iwasa, Y. (2006). Evolution of resistance to cancer therapy. Curr. Pharm. Design 12 261–271.
  • [16] Oller, A. R., Rastogi, P., Morgenthaler, S. and Thilly, W. G. (1989). A statistical model to estimate variance in long term-low dose mutation assays: Testing of the model in a human lymphoblastoid mutation assay. Mutat. Res. 216 149–161.
  • [17] Pakes, A. G. (1989). Asymptotic results for the extinction time of Markov branching processes allowing emigration. I. Random walk decrements. Adv. in Appl. Probab. 21 243–269.
  • [18] Pao, W. and Chmielecki, J. (2010). Rational, biologically based treatment of EGFR-mutant non-small-cell lung cancer. Nat. Rev. Cancer 10 760–774.
  • [19] Sagitov, S. and Serra, M. C. (2009). Multitype Bienaymé–Galton–Watson processes escaping extinction. Adv. in Appl. Probab. 41 225–246.
  • [20] Sehl, M., Zhou, H., Sinsheimer, J. S. and Lange, K. L. (2011). Extinction models for cancer stem cell therapy. Math. Biosci. 234 132–146.
  • [21] Seshadri, R., Kutlaca, R. J., Trainor, K., Matthews, C. and Morley, A. A. (1987). Mutation rate of normal and malignant human lymphocytes. Cancer Res. 263 407–409.
  • [22] Wu, X. and Kimmel, M. (2010). A note on the path to extinction of critical Markov branching processes. Statist. Probab. Lett. 80 263–269.