We introduce and study several measures of complexity of functions from the convex hull of a given base class. These complexity measures take into account the sparsity of the weights of a convex combination as well as certain clustering properties of the base functions involved in it. We prove new upper confidence bounds on the generalization error of ensemble (voting) classification algorithms that utilize the new complexity measures along with the empirical distributions of classification margins, providing a better explanation of generalization performance of large margin classification methods.

We propose new bounds on the error of learning algorithms in terms of a data-dependent notion of complexity. The estimates we establish give optimal rates and are based on a local and empirical version of Rademacher averages, in the sense that the Rademacher averages are computed from the data, on a subset of functions with small empirical error. We present some applications to classification and prediction with convex function classes, and with kernel classes in particular.

Boosting is one of the most significant advances in machine learning for classification and regression. In its original and computationally flexible version, boosting seeks to minimize empirically a loss function in a greedy fashion. The resulting estimator takes an additive function form and is built iteratively by applying a base estimator (or learner) to updated samples depending on the previous iterations. An unusual regularization technique, early stopping, is employed based on CV or a test set.

This paper studies numerical convergence, consistency and statistical rates of convergence of boosting with early stopping, when it is carried out over the linear span of a family of basis functions. For general loss functions, we prove the convergence of boosting’s greedy optimization to the infinimum of the loss function over the linear span. Using the numerical convergence result, we find early-stopping strategies under which boosting is shown to be consistent based on i.i.d. samples, and we obtain bounds on the rates of convergence for boosting estimators. Simulation studies are also presented to illustrate the relevance of our theoretical results for providing insights to practical aspects of boosting.

As a side product, these results also reveal the importance of restricting the greedy search step-sizes, as known in practice through the work of Friedman and others. Moreover, our results lead to a rigorous proof that for a linearly separable problem, AdaBoost with ɛ→0 step-size becomes an L¹-margin maximizer when left to run to convergence.

We propose a novel approach to sufficient dimension reduction in regression, based on estimating contour directions of small variation in the response. These directions span the orthogonal complement of the minimal space relevant for the regression and can be extracted according to two measures of variation in the response, leading to simple and general contour regression (SCR and GCR) methodology. In comparison with existing sufficient dimension reduction techniques, this contour-based methodology guarantees exhaustive estimation of the central subspace under ellipticity of the predictor distribution and mild additional assumptions, while maintaining $\sqrt{n}$-consistency and computational ease. Moreover, it proves robust to departures from ellipticity. We establish population properties for both SCR and GCR, and asymptotic properties for SCR. Simulations to compare performance with that of standard techniques such as ordinary least squares, sliced inverse regression, principal Hessian directions and sliced average variance estimation confirm the advantages anticipated by the theoretical analyses. We demonstrate the use of contour-based methods on a data set concerning soil evaporation.

Variable selection is fundamental to high-dimensional statistical modeling. Many variable selection techniques may be implemented by maximum penalized likelihood using various penalty functions. Optimizing the penalized likelihood function is often challenging because it may be nondifferentiable and/or nonconcave. This article proposes a new class of algorithms for finding a maximizer of the penalized likelihood for a broad class of penalty functions. These algorithms operate by perturbing the penalty function slightly to render it differentiable, then optimizing this differentiable function using a minorize–maximize (MM) algorithm. MM algorithms are useful extensions of the well-known class of EM algorithms, a fact that allows us to analyze the local and global convergence of the proposed algorithm using some of the techniques employed for EM algorithms. In particular, we prove that when our MM algorithms converge, they must converge to a desirable point; we also discuss conditions under which this convergence may be guaranteed. We exploit the Newton–Raphson-like aspect of these algorithms to propose a sandwich estimator for the standard errors of the estimators. Our method performs well in numerical tests.

We consider the statistical analysis of data on high-dimensional spheres and shape spaces. The work is of particular relevance to applications where high-dimensional data are available—a commonly encountered situation in many disciplines. First the uniform measure on the infinite-dimensional sphere is reviewed, together with connections with Wiener measure. We then discuss densities of Gaussian measures with respect to Wiener measure. Some nonuniform distributions on infinite-dimensional spheres and shape spaces are introduced, and special cases which have important practical consequences are considered. We focus on the high-dimensional real and complex Bingham, uniform, von Mises–Fisher, Fisher–Bingham and the real and complex Watson distributions. Asymptotic distributions in the cases where dimension and sample size are large are discussed. Approximations for practical maximum likelihood based inference are considered, and in particular we discuss an application to brain shape modeling.

This paper deals with projective shape analysis, which is a study of finite configurations of points modulo projective transformations. The topic has various applications in machine vision. We introduce a convenient projective shape space, as well as an appropriate coordinate system for this shape space. For generic configurations of k points in m dimensions, the resulting projective shape space is identified as a product of k−m−2 copies of axial spaces ℝP^m. This identification leads to the need for developing multivariate directional and multivariate axial analysis and we propose parametric models, as well as nonparametric methods, for these areas. In particular, we investigate the Frećhet extrinsic mean for the multivariate axial case. Asymptotic distributions of the appropriate parametric and nonparametric tests are derived. We illustrate our methodology with examples from machine vision.

This paper explores a class of empirical Bayes methods for level-dependent threshold selection in wavelet shrinkage. The prior considered for each wavelet coefficient is a mixture of an atom of probability at zero and a heavy-tailed density. The mixing weight, or sparsity parameter, for each level of the transform is chosen by marginal maximum likelihood. If estimation is carried out using the posterior median, this is a random thresholding procedure; the estimation can also be carried out using other thresholding rules with the same threshold. Details of the calculations needed for implementing the procedure are included. In practice, the estimates are quick to compute and there is software available. Simulations on the standard model functions show excellent performance, and applications to data drawn from various fields of application are used to explore the practical performance of the approach.

By using a general result on the risk of the corresponding marginal maximum likelihood approach for a single sequence, overall bounds on the risk of the method are found subject to membership of the unknown function in one of a wide range of Besov classes, covering also the case of f of bounded variation. The rates obtained are optimal for any value of the parameter p in (0,∞], simultaneously for a wide range of loss functions, each dominating the L_q norm of the σth derivative, with σ≥0 and 0<q≤2.

Attention is paid to the distinction between sampling the unknown function within white noise and sampling at discrete points, and between placing constraints on the function itself and on the discrete wavelet transform of its sequence of values at the observation points. Results for all relevant combinations of these scenarios are obtained. In some cases a key feature of the theory is a particular boundary-corrected wavelet basis, details of which are discussed.

Overall, the approach described seems so far unique in combining the properties of fast computation, good theoretical properties and good performance in simulations and in practice. A key feature appears to be that the estimate of sparsity adapts to three different zones of estimation, first where the signal is not sparse enough for thresholding to be of benefit, second where an appropriately chosen threshold results in substantially improved estimation, and third where the signal is so sparse that the zero estimate gives the optimum accuracy rate.

Let y=Aβ+ɛ, where y is an N×1 vector of observations, β is a p×1 vector of unknown regression coefficients, A is an N×p design matrix and ɛ is a spherically symmetric error term with unknown scale parameter σ. We consider estimation of β under general quadratic loss functions, and, in particular, extend the work of Strawderman [J. Amer. Statist. Assoc. 73 (1978) 623–627] and Casella [Ann. Statist. 8 (1980) 1036–1056, J. Amer. Statist. Assoc. 80 (1985) 753–758] by finding adaptive minimax estimators (which are, under the normality assumption, also generalized Bayes) of β, which have greater numerical stability (i.e., smaller condition number) than the usual least squares estimator. In particular, we give a subclass of such estimators which, surprisingly, has a very simple form. We also show that under certain conditions the generalized Bayes minimax estimators in the normal case are also generalized Bayes and minimax in the general case of spherically symmetric errors.

This article develops, and describes how to use, results concerning disintegrations of Poisson random measures. These results are fashioned as simple tools that can be tailor-made to address inferential questions arising in a wide range of Bayesian nonparametric and spatial statistical models. The Poisson disintegration method is based on the formal statement of two results concerning a Laplace functional change of measure and a Poisson Palm/Fubini calculus in terms of random partitions of the integers {1,…,n}. The techniques are analogous to, but much more general than, techniques for the Dirichlet process and weighted gamma process developed in [Ann. Statist. 12 (1984) 351–357] and [Ann. Inst. Statist. Math. 41 (1989) 227–245]. In order to illustrate the flexibility of the approach, large classes of random probability measures and random hazards or intensities which can be expressed as functionals of Poisson random measures are described. We describe a unified posterior analysis of classes of discrete random probability which identifies and exploits features common to all these models. The analysis circumvents many of the difficult issues involved in Bayesian nonparametric calculus, including a combinatorial component. This allows one to focus on the unique features of each process which are characterized via real valued functions h. The applicability of the technique is further illustrated by obtaining explicit posterior expressions for Lévy–Cox moving average processes within the general setting of multiplicative intensity models. In addition, novel computational procedures, similar to efficient procedures developed for the Dirichlet process, are briefly discussed for these models.

We consider a time series model involving a fractional stochastic component, whose integration order can lie in the stationary/invertible or nonstationary regions and be unknown, and an additive deterministic component consisting of a generalized polynomial. The model can thus incorporate competing descriptions of trending behavior. The stationary input to the stochastic component has parametric autocorrelation, but innovation with distribution of unknown form. The model is thus semiparametric, and we develop estimates of the parametric component which are asymptotically normal and achieve an M-estimation efficiency bound, equal to that found in work using an adaptive LAM/LAN approach. A major technical feature which we treat is the effect of truncating the autoregressive representation in order to form innovation proxies. This is relevant also when the innovation density is parameterized, and we provide a result for that case also. Our semiparametric estimates employ nonparametric series estimation, which avoids some complications and conditions in kernel approaches featured in much work on adaptive estimation of time series models; our work thus also contributes to methods and theory for nonfractional time series models, such as autoregressive moving averages. A Monte Carlo study of finite sample performance of the semiparametric estimates is included.

We consider the estimation of the location of the pole and memory parameter, λ⁰ and α, respectively, of covariance stationary linear processes whose spectral density function f(λ) satisfies f(λ)∼C|λ−λ⁰|^−α in a neighborhood of λ⁰. We define a consistent estimator of λ⁰ and derive its limit distribution Z_λ⁰. As in related optimization problems, when the true parameter value can lie on the boundary of the parameter space, we show that Z_λ⁰ is distributed as a normal random variable when λ⁰∈(0,π), whereas for λ⁰=0 or π, Z_λ⁰ is a mixture of discrete and continuous random variables with weights equal to 1/2. More specifically, when λ⁰=0, Z_λ⁰ is distributed as a normal random variable truncated at zero. Moreover, we describe and examine a two-step estimator of the memory parameter α, showing that neither its limit distribution nor its rate of convergence is affected by the estimation of λ⁰. Thus, we reinforce and extend previous results with respect to the estimation of α when λ⁰ is assumed to be known a priori. A small Monte Carlo study is included to illustrate the finite sample performance of our estimators.

An exact form of the local Whittle likelihood is studied with the intent of developing a general-purpose estimation procedure for the memory parameter (d) that does not rely on tapering or differencing prefilters. The resulting exact local Whittle estimator is shown to be consistent and to have the same $N(0,\frac{1}{4})$ limit distribution for all values of d if the optimization covers an interval of width less than $\frac{9}{2}$ and the initial value of the process is known.

We establish the Bahadur representation of sample quantiles for linear and some widely used nonlinear processes. Local fluctuations of empirical processes are discussed. Applications to the trimmed and Winsorized means are given. Our results extend previous ones by establishing sharper bounds under milder conditions and thus provide new insight into the theory of empirical processes for dependent random variables.

We study the problem of ignorability in likelihood-based inference from incomplete categorical data. Two versions of the coarsened at random assumption (car) are distinguished, their compatibility with the parameter distinctness assumption is investigated and several conditions for ignorability that do not require an extra parameter distinctness assumption are established.

It is shown that car assumptions have quite different implications depending on whether the underlying complete-data model is saturated or parametric. In the latter case, car assumptions can become inconsistent with observed data.