We develop minimax optimal risk bounds for the general learning task consisting in predicting as well as the best function in a reference set $\mathcal{G}$ up to the smallest possible additive term, called the convergence rate. When the reference set is finite and when n denotes the size of the training data, we provide minimax convergence rates of the form $C(\frac{\log|\mathcal{G}|}{n})^{v}$ with tight evaluation of the positive constant C and with exact 0<v≤1, the latter value depending on the convexity of the loss function and on the level of noise in the output distribution.

The risk upper bounds are based on a sequential randomized algorithm, which at each step concentrates on functions having both low risk and low variance with respect to the previous step prediction function. Our analysis puts forward the links between the probabilistic and worst-case viewpoints, and allows to obtain risk bounds unachievable with the standard statistical learning approach. One of the key ideas of this work is to use probabilistic inequalities with respect to appropriate (Gibbs) distributions on the prediction function space instead of using them with respect to the distribution generating the data.

The risk lower bounds are based on refinements of the Assouad lemma taking particularly into account the properties of the loss function. Our key example to illustrate the upper and lower bounds is to consider the L_q-regression setting for which an exhaustive analysis of the convergence rates is given while q ranges in [1; +∞[.

We propose a general maximum likelihood empirical Bayes (GMLEB) method for the estimation of a mean vector based on observations with i.i.d. normal errors. We prove that under mild moment conditions on the unknown means, the average mean squared error (MSE) of the GMLEB is within an infinitesimal fraction of the minimum average MSE among all separable estimators which use a single deterministic estimating function on individual observations, provided that the risk is of greater order than (log n)⁵/n. We also prove that the GMLEB is uniformly approximately minimax in regular and weak ℓ_p balls when the order of the length-normalized norm of the unknown means is between (log n)^κ₁/n^1/(p∧2) and n/(log n)^κ₂. Simulation experiments demonstrate that the GMLEB outperforms the James–Stein and several state-of-the-art threshold estimators in a wide range of settings without much down side.

We consider the classical problem of estimating a vector μ=(μ₁, …, μ_n) based on independent observations Y_i∼N(μ_i, 1), i=1, …, n.

Suppose μ_i, i=1, …, n are independent realizations from a completely unknown G. We suggest an easily computed estimator μ̂, such that the ratio of its risk E(μ̂−μ)² with that of the Bayes procedure approaches 1. A related compound decision result is also obtained.

Our asymptotics is of a triangular array; that is, we allow the distribution G to depend on n. Thus, our theoretical asymptotic results are also meaningful in situations where the vector μ is sparse and the proportion of zero coordinates approaches 1.

We demonstrate the performance of our estimator in simulations, emphasizing sparse setups. In “moderately-sparse” situations, our procedure performs very well compared to known procedures tailored for sparse setups. It also adapts well to nonsparse situations.

We show that, under a sparsity scenario, the Lasso estimator and the Dantzig selector exhibit similar behavior. For both methods, we derive, in parallel, oracle inequalities for the prediction risk in the general nonparametric regression model, as well as bounds on the ℓ_p estimation loss for 1≤p≤2 in the linear model when the number of variables can be much larger than the sample size.

We consider the problem of model selection and estimation in situations where the number of parameters diverges with the sample size. When the dimension is high, an ideal method should have the oracle property [J. Amer. Statist. Assoc. 96 (2001) 1348–1360] and [Ann. Statist. 32 (2004) 928–961] which ensures the optimal large sample performance. Furthermore, the high-dimensionality often induces the collinearity problem, which should be properly handled by the ideal method. Many existing variable selection methods fail to achieve both goals simultaneously. In this paper, we propose the adaptive elastic-net that combines the strengths of the quadratic regularization and the adaptively weighted lasso shrinkage. Under weak regularity conditions, we establish the oracle property of the adaptive elastic-net. We show by simulations that the adaptive elastic-net deals with the collinearity problem better than the other oracle-like methods, thus enjoying much improved finite sample performance.

We consider the problem of locating a jump discontinuity (change-point) in a smooth parametric regression model with a bounded covariate. It is assumed that one can sample the covariate at different values and measure the corresponding responses. Budget constraints dictate that a total of n such measurements can be obtained. A multistage adaptive procedure is proposed, where at each stage an estimate of the change point is obtained and new points are sampled from its appropriately chosen neighborhood. It is shown that such procedures accelerate the rate of convergence of the least squares estimate of the change-point. Further, the asymptotic distribution of the estimate is derived using empirical processes techniques. The latter result provides guidelines on how to choose the tuning parameters of the multistage procedure in practice. The improved efficiency of the procedure is demonstrated using real and synthetic data. This problem is primarily motivated by applications in engineering systems.

We consider a bivariate process X_t=(X_t¹, X_t²), which is observed on a finite time interval [0, T] at discrete times 0, Δ_n, 2Δ_n, …. Assuming that its two components X¹ and X² have jumps on [0, T], we derive tests to decide whether they have at least one jump occurring at the same time (“common jumps”) or not (“disjoint jumps”). There are two different tests for the two possible null hypotheses (common jumps or disjoint jumps). Those tests have a prescribed asymptotic level, as the mesh Δ_n goes to 0. We show on some simulations that these tests perform reasonably well even in the finite sample case, and we also put them in use for some exchange rates data.

In this paper, we propose a covariate-adjusted nonlinear regression model. In this model, both the response and predictors can only be observed after being distorted by some multiplicative factors. Because of nonlinearity, existing methods for the linear setting cannot be directly employed. To attack this problem, we propose estimating the distorting functions by nonparametrically regressing the predictors and response on the distorting covariate; then, nonlinear least squares estimators for the parameters are obtained using the estimated response and predictors. Root n-consistency and asymptotic normality are established. However, the limiting variance has a very complex structure with several unknown components, and confidence regions based on normal approximation are not efficient. Empirical likelihood-based confidence regions are proposed, and their accuracy is also verified due to its self-scale invariance. Furthermore, unlike the common results derived from the profile methods, even when plug-in estimates are used for the infinite-dimensional nuisance parameters (distorting functions), the limit of empirical likelihood ratio is still chi-squared distributed. This property eases the construction of the empirical likelihood-based confidence regions. A simulation study is carried out to assess the finite sample performance of the proposed estimators and confidence regions. We apply our method to study the relationship between glomerular filtration rate and serum creatinine.

We present a new methodology for sufficient dimension reduction (SDR). Our methodology derives directly from the formulation of SDR in terms of the conditional independence of the covariate X from the response Y, given the projection of X on the central subspace [cf. J. Amer. Statist. Assoc. 86 (1991) 316–342 and Regression Graphics (1998) Wiley]. We show that this conditional independence assertion can be characterized in terms of conditional covariance operators on reproducing kernel Hilbert spaces and we show how this characterization leads to an M-estimator for the central subspace. The resulting estimator is shown to be consistent under weak conditions; in particular, we do not have to impose linearity or ellipticity conditions of the kinds that are generally invoked for SDR methods. We also present empirical results showing that the new methodology is competitive in practice.

An important issue in survival analysis is the investigation and the modeling of hazard rates. Within a Bayesian nonparametric framework, a natural and popular approach is to model hazard rates as kernel mixtures with respect to a completely random measure. In this paper we provide a comprehensive analysis of the asymptotic behavior of such models. We investigate consistency of the posterior distribution and derive fixed sample size central limit theorems for both linear and quadratic functionals of the posterior hazard rate. The general results are then specialized to various specific kernels and mixing measures yielding consistency under minimal conditions and neat central limit theorems for the distribution of functionals.

We consider maximum likelihood estimation for both causal and noncausal autoregressive time series processes with non-Gaussian α-stable noise. A nondegenerate limiting distribution is given for maximum likelihood estimators of the parameters of the autoregressive model equation and the parameters of the stable noise distribution. The estimators for the autoregressive parameters are n^1/α-consistent and converge in distribution to the maximizer of a random function. The form of this limiting distribution is intractable, but the shape of the distribution for these estimators can be examined using the bootstrap procedure. The bootstrap is asymptotically valid under general conditions. The estimators for the parameters of the stable noise distribution have the traditional n^1/2 rate of convergence and are asymptotically normal. The behavior of the estimators for finite samples is studied via simulation, and we use maximum likelihood estimation to fit a noncausal autoregressive model to the natural logarithms of volumes of Wal-Mart stock traded daily on the New York Stock Exchange.

We provide a nonparametric method for the computation of instantaneous multivariate volatility for continuous semi-martingales, which is based on Fourier analysis. The co-volatility is reconstructed as a stochastic function of time by establishing a connection between the Fourier transform of the prices process and the Fourier transform of the co-volatility process. A nonparametric estimator is derived given a discrete unevenly spaced and asynchronously sampled observations of the asset price processes. The asymptotic properties of the random estimator are studied: namely, consistency in probability uniformly in time and convergence in law to a mixture of Gaussian distributions.

In this paper we examine the implications of the statistical large sample theory for the computational complexity of Bayesian and quasi-Bayesian estimation carried out using Metropolis random walks. Our analysis is motivated by the Laplace–Bernstein–Von Mises central limit theorem, which states that in large samples the posterior or quasi-posterior approaches a normal density. Using the conditions required for the central limit theorem to hold, we establish polynomial bounds on the computational complexity of general Metropolis random walks methods in large samples. Our analysis covers cases where the underlying log-likelihood or extremum criterion function is possibly nonconcave, discontinuous, and with increasing parameter dimension. However, the central limit theorem restricts the deviations from continuity and log-concavity of the log-likelihood or extremum criterion function in a very specific manner.

Under minimal assumptions required for the central limit theorem to hold under the increasing parameter dimension, we show that the Metropolis algorithm is theoretically efficient even for the canonical Gaussian walk which is studied in detail. Specifically, we show that the running time of the algorithm in large samples is bounded in probability by a polynomial in the parameter dimension d and, in particular, is of stochastic order d² in the leading cases after the burn-in period. We then give applications to exponential families, curved exponential families and Z-estimation of increasing dimension.

We consider the problem of constructing optimal designs for model discrimination between competing regression models. Various new properties of optimal designs with respect to the popular T-optimality criterion are derived, which in many circumstances allow an explicit determination of T-optimal designs. It is also demonstrated, that in nested linear models the number of support points of T-optimal designs is usually too small to estimate all parameters in the extended model. In many cases T-optimal designs are usually not unique, and in this situation we give a characterization of all T-optimal designs. Finally, T-optimal designs are compared with optimal discriminating designs with respect to alternative criteria by means of a small simulation study.