The Annals of Applied Statistics

Automatic estimation of flux distributions of astrophysical source populations

Raymond K. W. Wong, Paul Baines, Alexander Aue, Thomas C. M. Lee, and Vinay L. Kashyap

Full-text: Open access


In astrophysics a common goal is to infer the flux distribution of populations of scientifically interesting objects such as pulsars or supernovae. In practice, inference for the flux distribution is often conducted using the cumulative distribution of the number of sources detected at a given sensitivity. The resulting “$\log(N>S)$–$\log(S)$” relationship can be used to compare and evaluate theoretical models for source populations and their evolution. Under restrictive assumptions the relationship should be linear. In practice, however, when simple theoretical models fail, it is common for astrophysicists to use prespecified piecewise linear models. This paper proposes a methodology for estimating both the number and locations of “breakpoints” in astrophysical source populations that extends beyond existing work in this field.

An important component of the proposed methodology is a new interwoven EM algorithm that computes parameter estimates. It is shown that in simple settings such estimates are asymptotically consistent despite the complex nature of the parameter space. Through simulation studies it is demonstrated that the proposed methodology is capable of accurately detecting structural breaks in a variety of parameter configurations. This paper concludes with an application of our methodology to the Chandra Deep Field North (CDFN) data set.

Article information

Ann. Appl. Stat., Volume 8, Number 3 (2014), 1690-1712.

First available in Project Euclid: 23 October 2014

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Broken power law CDFN X-ray survey interwoven EM algorithm likelihood computations $\log N$–$\log S$ Pareto distribution


Wong, Raymond K. W.; Baines, Paul; Aue, Alexander; Lee, Thomas C. M.; Kashyap, Vinay L. Automatic estimation of flux distributions of astrophysical source populations. Ann. Appl. Stat. 8 (2014), no. 3, 1690--1712. doi:10.1214/14-AOAS750.

Export citation


  • Aue, A. and Lee, T. C. M. (2011). On image segmentation using information theoretic criteria. Ann. Statist. 39 2912–2935.
  • Baines, P. D. (2010). Statistics, science and statistical science: Modeling, inference and computation with applications to the physical sciences. Ph.D. thesis.
  • Baines, P. D., Meng, X. L. and Xie, X. (2014). The interwoven EM algorithm. Unpublished manuscript.
  • Baines, P. D., Udaltsova, I. S., Zezas, A. and Kashyap, V. L. (2012). Bayesian estimation of $\log N$–$\log S$. In Statistical Challenges in Modern Astronomy V (E. D. Feigelson and G. J. Babu, eds.) 469–472. Springer, New York.
  • Cappelluti, N., Hasinger, G., Brusa, M., Comastri, A., Zamorani, G., Böhringer, H., Brunner, H., Civano, F., Finoguenov, A., Fiore, F., Gilli, R., Griffiths, R. E., Mainieri, V., Matute, I., Miyaji, T. and Silverman, J. (2007). The XMM–Newton Wide-field survey in the COSMOS field. II. X-ray data and the $\log N$–$\log S$ relations. Astrophys. J., Suppl. Ser. 172 341–352.
  • Dempster, A. P., Laird, N. M. and Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. J. Roy. Statist. Soc. Ser. B 39 1–38.
  • Friel, N. and Pettitt, A. N. (2008). Marginal likelihood estimation via power posteriors. J. R. Stat. Soc. Ser. B Stat. Methodol. 70 589–607.
  • Guetta, D., Granot, J. and Begelman, M. C. (2005). Constraining the structure of gamma-ray burst jets through the $\log{N}$–$\log{S}$ distribution. Astrophys. J. 622 482–491.
  • Hewish, A. (1961). Extrapolation of the number-flux density relation of radio stars by Scheuer’s statistical methods. Monthly Notices of the Royal Astonomical Society 123 167–181.
  • Hickox, R. C. and Markevitch, M. (2007). Can Chandra resolve the remaining cosmic X-ray background? Astrophys. J. 671 1523–1530.
  • Jordán, A., Côté, P., Ferrarese, L., Blakeslee, J. P., Mei, S., Merritt, D., Milosavljevifá, M., Peng, E. W., Tonry, J. L. and West, M. J. (2004). The ACS Virgo cluster curvey. III. Chandra and Hubble space telescope observations of low-mass X-ray binaries and globular clusters in M87. Astrophys. J. 613 279–301.
  • Kenter, A. T. and Murray, S. S. (2003). A new technique for determining the number of X-ray sources per flux density interval. Astrophys. J. 584 1016–1020.
  • Kiefer, J. and Wolfowitz, J. (1956). Consistency of the maximum likelihood estimator in the presence of infinitely many incidental parameters. Ann. Math. Statist. 27 887–906.
  • Kitayama, T., Sasaki, S. and Suto, Y. (1998). Cosmological implications of number counts of clusters of galaxies: $\log N$–$\log S$ in X-ray and submm bands. Publ. Astron. Soc. Jpn. 50 1–11.
  • Kouzu, T., Tashiro, M. S., Terada, Y., Yamada, S., Bamba, A., Enoto, T., Mori, K., Fukazawa, Y. and Makishima, K. (2013). Spectral variation of hard X-ray emission from the Crab Nebula with the suzaku hard X-ray detector. Publ. Astron. Soc. Jpn. 65 74-1–74-11.
  • Mateos, S., Warwick, R. S., Carrera, F. J., Stewart, G. C., Ebrero, J., Della Ceca, R., Caccianiga, A., Gilli, R., Page, M. J., Treister, E., Tedds, J. A., Watson, M. G., Lamer, G., Saxton, R. D., Brunner, H. and Page, C. G. (2008). High precision X-ray $\log N$–$\log S$ distributions: Implications for the obscured AGN population. Astron. Astrophys. 492 51–69.
  • Mathiesen, B. and Evrard, A. E. (1998). Constraints on $\Omega_{0}$ and cluster evolution using the ROSAT $\log N$–$\log S$ relation. Mon. Not. R. Astron. Soc. 295 769–780.
  • Moretti, A., Campana, S., Lazzati, D. and Tagliaferri, G. (2003). The resolved fraction of the cosmic X-ray background. Astrophys. J. 588 696–703.
  • Ryde, F. (1999). Smoothly broken power law spectra of gamma-ray bursts. Astrophys. Lett. Commun. 39 281–284.
  • Scheuer, P. A. G. (1957). A statistical method for analysing observations of faint radio stars. Proc. Cambridge Philos. Soc. 53 764–773.
  • Segura, C., Lazzati, D. and Sankarasubramanian, A. (2013). The use of broken power-laws to describe the distributions of daily flow above the mean annual flow across the conterminous U.S. J. Hydrol. 505 35–46.
  • Trudolyubov, S. P., Borozdin, K. N., Priedhorsky, W. C., Mason, K. O. and Cordova, F. A. (2002). On the X-ray source luminosity distributions in the bulge and disk of M31: First results from the XMM–Newton Survey. Astrophys. J. Lett. 571 17–21.
  • Wald, A. (1949). Note on the consistency of the maximum likelihood estimate. Ann. Math. Stat. 20 595–601.
  • Wong, R. K. W., Baines, P., Aue, A., Lee, T. C. M. and Kashyap, V. L. (2014). Supplement to “Automatic estimation of flux distributions of astrophysical source populations.” DOI:10.1214/14-AOAS750SUPP.
  • Yao, Y.-C. (1988). Estimating the number of change-points via Schwarz’ criterion. Statist. Probab. Lett. 6 181–189.
  • Yu, Y. and Meng, X.-L. (2011). To center or not to center: That is not the question—an ancillarity-sufficiency interweaving strategy (ASIS) for boosting MCMC efficiency. J. Comput. Graph. Statist. 20 531–570.

Supplemental materials