Tree ensembles are looked at in distribution space, that is, the limit case of "infinite" sample size. It is shown that the simplest kind of trees is complete in D-dimensional $L_2(P)$ space if the number of terminal nodes T is greater than D. For such trees we show that the AdaBoost algorithm gives an ensemble converging to the Bayes risk.

The notion of boosting originated in the Machine Learning literature in the 1980's [VALIANT, L.G. (1984). A theory of the learnable. In Proc. 16th Annual ACM Symposium on Theory of Computing 436-445. ACM Press, New York]. The goal of boosting is to improve the generalization performance of weak (or base) learning algorithms by combining them in a certain way. The first algorithm of this type was discovered by Schapire [SCHAPIRE, R.E. (1990). The strength of weak learnability. Machine Learning 5 197-227] and then the second one by Freund [FREUND, Y. (1995). Boosting a weak learning algorithm by majority. Inform. and Comput. 121 256-285]. Schapire and Freund [FREUND, Y. and Schapire. R.E. (1997). A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. System. Sci. 55 119-139] came up with the idea of a more practical version of boosting and invented the algorithm called AdaBoost that combines simple classification rules into much more powerful and precise classification algorithms. For a fixed number of iterations, AdaBoost runs the weak (or base) learning algorithm on resampled original data sets in a sequential manner and then combines the resulting learning algorithms through a weighted summation at the end of the iteration. Gradually, it became clear that AdaBoost is a special case of a more general statistical methodology of combining simple estimates in classification or regression into more complex and more precise ones. The study of statistical properties of these methods has been conducted in several directions since then in both the machine learning and statistics communities.

The problem of consistency of AdaBoost is posed by Leo Breiman in the first paper in this issue of The Annals of Statistics. Breiman studies one ingredient needed to prove the consistency, the convergence properties of AdaBoost as a numerical method in the population case. This paper has been circulated for a couple of years as a preprint and its results were also covered in the Wald Lectures delivered by Breiman at the IMS Annual Meeting in 2002 in Banff, Canada. The papers by Jiang, Lugosi and Vayatis, and Zhang, published below with discussions, consider various versions of boosting and give answers to the consistency question posed by Breiman.

Recent experiments and theoretical studies show that AdaBoost can overfit in the limit of large time. If running the algorithm forever is suboptimal, a natural question is how low can the prediction error be during the process of AdaBoost? We show under general regularity conditions that during the process of AdaBoost a consistent prediction is generated, which has the prediction error approximating the optimal Bayes error as the sample size increases. This result suggests that, while running the algorithm forever can be suboptimal, it is reasonable to expect that some regularization method via truncation of the process may lead to a near-optimal performance for sufficiently large sample size.

The probability of error of classification methods based on convex combinations of simple base classifiers by "boosting" algorithms is investigated. The main result of the paper is that certain regularized boosting algorithms provide Bayes-risk consistent classifiers under the sole assumption that the Bayes classifier may be approximated by a convex combination of the base classifiers. Nonasymptotic distribution-free bounds are also developed which offer interesting new insight into how boosting works and help explain its success in practical classification problems.

We study how closely the optimal Bayes error rate can be approximately reached using a classification algorithm that computes a classifier by minimizing a convex upper bound of the classification error function. The measurement of closeness is characterized by the loss function used in the estimation. We show that such a classification scheme can be generally regarded as a (nonmaximum-likelihood) conditional in-class probability estimate, and we use this analysis to compare various convex loss functions that have appeared in the literature. Furthermore, the theoretical insight allows us to design good loss functions with desirable properties. Another aspect of our analysis is to demonstrate the consistency of certain classification methods using convex risk minimization. This study sheds light on the good performance of some recently proposed linear classification methods including boosting and support vector machines. It also shows their limitations and suggests possible improvements.

Discussions of: "Process consistency for AdaBoost" [Ann. Statist. 32 (2004), no. 1, 13-29] by W. Jiang; "On the Bayes-risk consistency of regularized boosting methods" [ibid., 30-55] by G. Lugosi and N. Vayatis; and "Statistical behavior and consistency of classification methods based on convex risk minimization" [ibid., 56-85] by T. Zhang. Includes rejoinders by the authors.

Classification can be considered as nonparametric estimation of sets, where the risk is defined by means of a specific distance between sets associated with misclassification error. It is shown that the rates of convergence of classifiers depend on two parameters: the complexity of the class of candidate sets and the margin parameter. The dependence is explicitly given, indicating that optimal fast rates approaching $O(n^{-1})$ can be attained, where n is the sample size, and that the proposed classifiers have the property of robustness to the margin. The main result of the paper concerns optimal aggregation of classifiers: we suggest a classifier that automatically adapts both to the complexity and to the margin, and attains the optimal fast rates, up to a logarithmic factor.

The depth of multivariate data can be used to construct weighted means as robust estimators of location. The use of projection depth leads to the Stahel-Donoho estimator as a special case. In contrast to maximal depth estimators, the depth-weighted means are shown to be asymptotically normal under appropriate conditions met by depth functions commonly used in the current literature. We also confirm through a finite-sample study that the Stahel-Donoho estimator achieves a desirable balance between robustness and efficiency at Gaussian models.

Location estimators induced from depth functions increasingly have been pursued and studied in the literature. Among them are those induced from projection depth functions. These projection depth based estimators have favorable properties among their competitors. In particular, they possess the best possible finite sample breakdown point robustness. However, robustness of estimators cannot be revealed by the finite sample breakdown point alone. The influence function, gross error sensitivity, maximum bias and contamination sensitivity are also important aspects of robustness. In this article, we study these other robustness aspects of two types of projection depth based estimators: projection medians and projection depth weighted means. The latter includes the Stahel-Donoho estimator as a special case. Exact maximum bias, the influence function, and contamination and gross error sensitivity are derived and studied for both types of estimators. Sharp upper bounds for the maximum bias and the influence functions are established. Comparisons based on these robustness criteria reveal that the projection depth based estimators enjoy desirable local as well as global robustness and are very competitive among their competitors.

Proportionality of covariance matrices of n independent p-dimensional normal distributions with the same type of linear restrictions of the inverse covariances is considered. Conditions for existence and uniqueness of the maximum likelihood estimator are obtained through the development of general results for scale-invariant natural exponential families.

This article provides a historic review of the forward and backward projection pursuit algorithms, previously thought to be equivalent, and points out an important difference between the two. In doing so, a small error in the original exploratory projection pursuit article by Friedman [J. Amer. Statist. Assoc. 82 1987) 249-266] is corrected. The implication of the difference is briefly discussed in the context of an application in which projection pursuit density estimation is used as a building block for nonparametric discriminant analysis.

A random variable X is symmetric about 0 if X and -X have the same distribution. There is a large literature on the estimation of a distribution function (DF) under the symmetry restriction and tests for checking this symmetry assumption. Often the alternative describes some notion of skewness or one-sided bias. Various notions can be described by an ordering of the distributions of X and -X. One such important ordering is that $P(0<X<X\le x)-P(-x\le X<0)$ is increasing in $x<0$. The distribution of X is said to have a Type II positive bias in this case. If X has a density f, then this corresponds to the density ordering $f(-x)\le f(x)$ for $x<0$. It is known that the nonparametric maximum likelihood estimator (NPMLE) of the DF under this restriction is inconsistent. We provide a projection-type estimator that is similar to a consistent estimator of two DFs under uniform stochastic ordering, where the NPMLE also fails to be consistent. The weak convergence of the estimator has been derived which can be used for testing the null hypothesis of symmetry against this one-sided alternative. It also turns out that the same procedure can be used to estimate two cumulative incidence functions in a competing risks problem under the restriction that the cause specific hazard rates are ordered. We also provide some real life examples.

A general doubly adaptive biased coin design is proposed for the allocation of subjects to K treatments in a clinical trial. This design follows the same spirit as Efron's biased coin design and applies to the cases where the desired allocation proportions are unknown, but estimated sequentially. Strong consistency, a law of the iterated logarithm and asymptotic normality of this design are obtained under some widely satisfied conditions. For two treatments, a new family of designs is proposed and shown to be less variable than both the randomized play-the-winner rule and the adaptive randomized design. Also the proposed design tends toward a randomization scheme (with a fixed target proportion) as the size of the experiment increases.

The optimality of CUSUM under a Lorden-type criterion setting is considered. We demonstrate the optimality of the CUSUM test for Itô processes, in a sense similar to Lorden's, but with a criterion that replaces expected delays by the corresponding Kullback-Leibler divergence.

It is known that both the optimal exponentially weighted moving average (EWMA) and cumulative sum (CUSUM) control charts are based on a given reference value $\delta$, which, for the CUSUM chart, is the magnitude of a shift in the mean to be detected quickly. In this paper a generalized EWMA control chart (GEWMA) which does not depend on $\delta$ is proposed for detecting the mean shift. We compare theoretically the GEWMA control chart with the optimal EWMA, CUSUM and the generalized likelihood ratio (GLR) control charts. The results of the comparison in which the in-control average run length approaches infinity show that the GEWMA control chart is better than the optimal EWMA control chart in detecting a mean shift of any size and is also better than the CUSUM control chart in detecting the mean shift which is not in the interval $(0.7842\delta ,1.3798\delta )$. Moreover, the GLR control chart has the best performance in detecting mean shift among the four control charts except when detecting a particular mean shift $\delta,$ when the in-control average run length approaches infinity.

A unified approach to maximum likelihood inference for a broad, new class of contingency table models is presented. The model class comprises multinomial-Poisson homogeneous (MPH) models, which can be characterized by an independent sampling plan and a system of homogeneous constraints, h(m) = 0, where m is the vector of expected table counts. Maximum likelihood (ML) fitting and large-sample inference for MPH models are described. The MPH models are partitioned into well-defined equivalence classes and explicit comparisons of the large-sample behaviors of ML estimators of equivalent models are given. The equivalence theory not only unifies a large collection of previously known results, it also leads to useful generalizations and many new results. The practical, computational implication is that ML fit results for any particular MPH model can be obtained directly from the ML fit results for any conveniently chosen equivalent model. Issues of hypothesis testability and parameter estimability are also addressed. To illustrate, an example based on statistics journal citation patterns is given for which the data can be used to test the hypothesis that a certain model holds, but they cannot be used to estimate any of that model's parameters.

The Bradley-Terry model for paired comparisons is a simple and much-studied means to describe the probabilities of the possible outcomes when individuals are judged against one another in pairs. Among the many studies of the model in the past 75 years, numerous authors have generalized it in several directions, sometimes providing iterative algorithms for obtaining maximum likelihood estimates for the generalizations. Building on a theory of algorithms known by the initials MM, for minorization-maximization, this paper presents a powerful technique for producing iterative maximum likelihood estimation algorithms or a wide class of generalizations of the Bradley-Terry model. While algorithms for problems of this type have tended to be custom-built in the literature, the techniques in this paper enable their mass production. Simple conditions are stated that guarantee that each algorithm described will produce a sequence that converges to the unique maximum likelihood estimator. Several of the algorithms and convergence results herein are new.