Function estimation/approximation is viewed from the perspective of numerical optimization in function space, rather than parameter space. A connection is made between stagewise additive expansions and steepest-descent minimization. A general gradient descent “boosting” paradigm is developed for additive expansions based on any fitting criterion.Specific algorithms are presented for least-squares, least absolute deviation, and Huber-M loss functions for regression, and multiclass logistic likelihood for classification. Special enhancements are derived for the particular case where the individual additive components are regression trees, and tools for interpreting such “TreeBoost” models are presented. Gradient boosting of regression trees produces competitive, highly robust, interpretable procedures for both regression and classification, especially appropriate for mining less than clean data. Connections between this approach and the boosting methods of Freund and Shapire and Friedman, Hastie and Tibshirani are discussed.

We study the rates of convergence of the maximum likelihood estimator (MLE) and posterior distribution in density estimation problems, where the densities are location or location-scale mixtures of normal distributions with the scale parameter lying between two positive numbers. The true density is also assumed to lie in this class with the true mixing distribution either compactly supported or having sub-Gaussian tails. We obtain bounds for Hellinger bracketing entropies for this class, and from these bounds, we deduce the convergence rates of (sieve) MLEs in Hellinger distance. The rate turns out to be $(log n)^\kappa /\sqrt{n}$, where $\kappa \ge 1$ is a constant that depends on the type of mixtures and the choice of the sieve. Next, we consider a Dirichlet mixture of normals as a prior on the unknown density. We estimate the prior probability of a certain Kullback-Leibler type neighborhood and then invoke a general theorem that computes the posterior convergence rate in terms the growth rate of the Hellinger entropy and the concentration rate of the prior. The posterior distribution is also seen to converge at the rate $(log n)^\kappa /\sqrt{n}$, where $\kappa$ now depends on the tail behavior of the base measure of the Dirichlet process.

Mixture models for density estimation provide a very useful set up for the Bayesian or the maximum likelihood approach.For a density on the unit interval, mixtures of beta densities form a flexible model. The class of Bernstein densities is a much smaller subclass of the beta mixtures defined by Bernstein polynomials, which can approximate any continuous density. A Bernstein polynomial prior is obtained by putting a prior distribution on the class of Bernstein densities. The posterior distribution of a Bernstein polynomial prior is consistent under very general conditions. In this article, we present some results on the rate of convergence of the posterior distribution. If the underlying distribution generating the data is itself a Bernstein density, then we show that the posterior distribution converges at “nearly parametric rate” $(log n) /\sqrt{n}$ for the Hellinger distance. If the true density is not of the Bernstein type, we show that the posterior converges at a rate $n^{1/3}(log n)^{5/6}$ provided that the true density is twice differentiable and bounded away from 0. Similar rates are also obtained for sieve maximum likelihood estimates.These rates are inferior to the pointwise convergence rate of a kernel type estimator.We show that the Bayesian bootstrap method gives a proxy for the posterior distribution and has a convergence rate at par with that of the kernel estimator.

The consistent estimation of mixture complexity is of fundamental importance in many applications of finite mixture models. An enormous body of literature exists regarding the application, computational issues and theoretical aspects of mixture models when the number of components is known, but estimating the unknown number of components remains an area of intense research effort. This article presents a semiparametric methodology yielding almost sure convergence of the estimated number of components to the true but unknown number of components. The scope of application is vast, as mixture models are routinely employed across the entire diverse application range of statistics,including nearly all of the social and experimental sciences.

A special semiparametric model for a univariate density is introduced that allows analyzing a number of problems via appropriate transformations. Two problems treated in some detail are testing for the presence of a mixture and detecting a wear-out trend in a failure rate. The analysis of the semiparametric model leads to an approach that advances the maximum likelihood theory of the Grenander estimator to a multiscale analysis. The construction of the corresponding test statistic rests on an extension of a result on a two-sided Brownian motion with quadratic drift to the simultaneous control of “excursions under parabolas” at various scales of a Brownian bridge. The resulting test is shown to be asymptotically optimal in the minimax sense regarding both rate and constant, and adaptive with respect to the unknown parameter in the semiparametric model. The performance of the method is illustrated with a simulation study for the failure rate problem and with data from a flow cytometry experiment for the mixture analysis.

We present a new, natural way to construct nonparametric multivariate tolerance regions. Unlike the classical nonparametric tolerance intervals, where the endpoints are determined by beforehand chosen order statistics, we take the shortest interval that contains a certain number of observations. We extend this idea to higher dimensions by replacing the class of intervals by a general class of indexing sets, which specializes to the classes of ellipsoids, hyperrectangles or convex sets.The asymptotic behavior of our tolerance regions is derived using empirical process theory, in particular the concept of generalized quantiles. Finite sample properties of our tolerance regions are investigated through a simulation study. Real data examples are also presented.

We present a non-parametric survival model with two time-scales. The time-scales are equivalent up to a constant that varies over the subjects. Covariate effects are modelled linearly on each time scale by additive Aalen models. Estimators of the cumulative intensities on the two time-scales are suggested by solving approximate local maximum likelihood estimating equations. The local estimating equations necessitate only the choice of one bandwidth. The estimators are provided with large sample properties. The model is applied to data on patients with myocardial infarction, and used to describe the prognostic effect of covariates on the two time scales, time since myocardial infarction and age.

In the problem of testing the equalityof k regression curves from independent samples, we discuss three methods using nonparametric estimators of the regression function. The first test is based on a linear combination of estimators for the integrated variance function in the individual samples and in the combined sample. The second approach transfers the classical one-way analysis of variance to the situation of comparing non-parametric curves, while the third test compares the differences between the estimates of the individual regression functions by means of an $L^2$-distance.We prove asymptotic normality of all considered statistics under the null hypothesis and local and fixed alternatives with different rates corresponding to the various cases. Additionally,consistency of a wild bootstrap version of the tests is established. In contrast to most of the procedures proposed in the literature, the methods introduced in this paper are also applicable in the case of different design points in each sample and heteroscedastic errors. A simulation studyis conducted to investigate the finite sample properties of the new tests and a comparison with recently proposed and related procedures is performed.

Let $(\mathcal{X}_1, \mathcal{Y}_1),\dots,(\mathcal{X}_n, \mathcal{Y}_n)$ be a random sample from a bivariate distribution function $F$ in the domain of max-attraction of a distribution function $G$. This $G$ is characterised by the two extreme value indices and its spectral or angular measure. The extreme value indices determine both the marginals and the spectral measure determines the dependence structure of $G$. One of the main issues in multivariate extreme value theory is the estimation of this spectral measure. We construct a truly nonparametric estimator of the spectral measure, based on the ranks of the above data. Under natural conditions we prove consistency and asymptotic normality for the estimator. In particular,the result is valid for all values of the extreme value indices. The theory of (local) empirical processes is indispensable here. The results are illustrated by an application to real data and a small simulation study.

The asymptotic classification risk for nearest neighbor procedures is well understood in the case of i.i.d. training sequences. In this article, we generalize these results to a class of dependent models including hidden Markov models. In the case where the observed patterns have Lebesgue densities, the asymptotic risk takes the same expression as in the i.i.d. case. For discrete distributions, we show that the asymptotic risk depends on the rule used for breaking ties of equal distances.

We examine the way in which empirical bandwidth choice affects distributional properties of nonparametric density estimators. Two bandwidth selection methods are considered in detail: local and global plug-in rules. Particular attention is focussed on whether the accuracy of distributional bootstrap approximations is appreciably influenced by using the resample version $\hat{h}*$,rather than the sample version $\hat{h}$, of an empirical bandwidth. It is shown theoretically that,in marked contrast to similar problems in more familiar settings, no general first-order theoretical improvement can be expected when using the resampling version. In the case of local plug-in rules, the inability of the bootstrap to accurately reflect biases of the components used to construct the bandwidth selector means that the bootstrap distribution of $\hat{h}*$ is unable to capture some of the main properties of the distribution of $\hat{h}$. If the second derivative component is slightly undersmoothed then some improvements are possible through using $\hat{h}*$, but they would be difficult to achieve in practice. On the other hand, for global plug-in methods, both $\hat{h}$ and $\hat{h}*$ are such good approximations to an optimal, deterministic bandwidth that the variations of either can be largely ignored, at least at a first-order level.Thus, for quite different reasons in the two cases, the computational burden of varying an empirical bandwidth across resamples is difficult to justify.

This paper proposes a test for selecting explanatory variables in nonparametric regression. The test does not need to estimate the conditional expectation function given all the variables, but only those which are significant under the null hypothesis. This feature is computationally convenient and solves, in part, the problem of the “curse of dimensionality” when selecting regressors in a nonparametric context. The proposed test statistic is based on functionals of a $U$-process. Contiguous alternatives, converging to the null at a rate $n^{-1/2}$ can be detected. The asymptotic null distribution of the statistic depends on certain features of the data generating process,and asymptotic tests are difficult to implement except in rare circumstances. We justify the consistency of two easy to implement bootstrap tests which exhibit good level accuracy for fairly small samples, according to the reported Monte Carlo simulations. These results are also applicable to test other interesting restrictions on nonparametric curves, like partial linearity and conditional independence.

We consider the problem of parameter estimation for multidimensional continuous-time linear stochastic regression models with an arbitrary finite number of unknown parameters and with martingale noise. The main result of the paper claims that the unknown parameters can be estimated with prescribed mean-square precision in this general model providing a unified description of both discrete and continuous time process. Among the conditions on the regressors there is one bounding the growth of the maximal eigenvalue of the design matrix with respect to its minimal eigenvalue. This condition is slightly stronger as compared with the corresponding conditions usually imposed on the regressors in asymptotic investigations but still it enables one to consider models with different behavior of the eigenvalues. The construction makes use of a two-step procedure based on the modified least-squares estimators and special stopping rules.