The Annals of Applied Statistics

Analysis of spatial distribution of marker expression in cells using boundary distance plots

Kingshuk Roy Choudhury, Limian Zheng, and John J. Mackrill

Full-text: Open access


Boundary distance (BD) plotting is a technique for making orientation invariant comparisons of the spatial distribution of biochemical markers within and across cells/nuclei. Marker expression is aggregated over points with the same distance from the boundary. We present a suite of tools for improved data analysis and statistical inference using BD plotting. BD is computed using the Euclidean distance transform after presmoothing and oversampling of nuclear boundaries. Marker distribution profiles are averaged using smoothing with linearly decreasing bandwidth. Average expression curves are scaled and registered by x-axis dilation to compensate for uneven lighting and errors in nuclear boundary marking. Penalized discriminant analysis is used to characterize the quality of separation between average marker distributions. An adaptive piecewise linear model is used to compare expression gradients in intra, peri and extra nuclear zones. The techniques are illustrated by the following: (a) a two sample problem involving a pair of voltage gated calcium channels (Cav1.2 and AB70) marked in different cells; (b) a paired sample problem of calcium channels (Y1F4 and RyR1) marked in the same cell.

Article information

Ann. Appl. Stat., Volume 4, Number 3 (2010), 1365-1382.

First available in Project Euclid: 18 October 2010

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Euclidean distance transform smoothing functional data analysis curve registration image texture


Roy Choudhury, Kingshuk; Zheng, Limian; Mackrill, John J. Analysis of spatial distribution of marker expression in cells using boundary distance plots. Ann. Appl. Stat. 4 (2010), no. 3, 1365--1382. doi:10.1214/10-AOAS340.

Export citation


  • Ambler, S. K., Poenie, M., Tsien, R. Y. and Taylor, P. (1988). Agonist-stimulated oscillations and cycling of intracellular free calcium in individual cultured muscle cells. J. Biol. Chem. 263 1952–1959.
  • Anderson, T. (2003). An Introduction to Multivariate Statistical Analysis, 3rd ed. Wiley, Hoboken, NJ.
  • Bewersdorf, J., Bennett, B. and Knight, K. (2006). H2AX chromatin structures and their response to DNA damage revealed by 4Pi microscopy. Proc. Natl. Acad. Sci. USA 103 18137–18142.
  • Callinan, L., McCarthy, T., Maulet, Y. and Mackrill, J. (2005). Atypical L-type channels are down-regulated in hypoxia. Biochemical Society Transactions 33 1137–1139.
  • Fabbri, R., Costa, L., Torelli, J. and Bruno, O. (2008). 2D Euclidean distance transform algorithms: A comparative survey. ACM Computing Surveys 40 2:1–2:44.
  • Fernandez-Gonzalez, R., Munoz-Barrutia, A., Barcellos-Hoff, M. and Ortiz-De-Solorzano, C. (2006). Quantitative in vivo microscopy: The return from the ‘omics.’ Current Opinion in Biotechnology 17 501–510.
  • Friedman, J. (1989). Regularized discriminant analysis. J. Amer. Statist. Assoc. 84 165–175.
  • Gomez-Ospina, N., Barreto-Chang, O., Hu, L. and Dolmetsch, R. (2006). The C terminus of the L-type voltage-gated calcium channel Ca(V)1.2 encodes a transcription factor. Cell 127 591–606.
  • Hastie, T., Tibshirani, R. and Friedman, J. (2001). The Elements of Statistical Learning. Springer, New York.
  • Jahne, B. (2005). Digital Image Processing, 6th ed. Springer.
  • Knowles, D., Sudar, D., Bator-Kelly, C., Bissell, M. and Lelièvre, S. (2006). Automated local bright feature image analysis of nuclear protein distribution identifies changes in tissue phenotype. Proc. Natl. Acad. Sci. USA 103 4445–4450.
  • Mackrill, J. J. (1999). Protein–protein interactions in intracellular Ca2+-release channel function. Biochem. J. 337 345–361.
  • Ramsay, J. and Silverman, B. (2002). Functional Data Analysis, 2nd ed. Springer, New York.
  • Silverman, B. (1984). Spline smoothing: The equivalent variable kernel method. Ann. Statist. 12 898–916.
  • Silverman, B. (1985). Some aspects of the spline smoothing approach to non-parametric regression curve fitting. J. Roy. Stat. Soc. Ser. B 47 1–52.
  • Wahba, G. (1975). Optimal convergence properties of variable knot, kernel, and orthogonal series methods for density estimation. Ann. Statist. 3 15–29.