We consider the least angle regression and forward stagewise algorithms for solving penalized least squares regression problems. In Efron, Hastie, Johnstone & Tibshirani (2004) it is proved that the least angle regression algorithm, with a small modification, solves the lasso regression problem. Here we give an analogous result for incremental forward stagewise regression, showing that it solves a version of the lasso problem that enforces monotonicity. One consequence of this is as follows: while lasso makes optimal progress in terms of reducing the residual sum-of-squares per unit increase in L₁-norm of the coefficient β, forward stage-wise is optimal per unit L₁ arc-length traveled along the coefficient path. We also study a condition under which the coefficient paths of the lasso are monotone, and hence the different algorithms coincide. Finally, we compare the lasso and forward stagewise procedures in a simulation study involving a large number of correlated predictors.

We provide a new algorithm for the treatment of inverse problems which combines the traditional SVD inversion with an appropriate thresholding technique in a well chosen new basis. Our goal is to devise an inversion procedure which has the advantages of localization and multiscale analysis of wavelet representations without losing the stability and computability of the SVD decompositions. To this end we utilize the construction of localized frames (termed “needlets”) built upon the SVD bases.

We consider two different situations: the “wavelet” scenario, where the needlets are assumed to behave similarly to true wavelets, and the “Jacobi-type” scenario, where we assume that the properties of the frame truly depend on the SVD basis at hand (hence on the operator). To illustrate each situation, we apply the estimation algorithm respectively to the deconvolution problem and to the Wicksell problem. In the latter case, where the SVD basis is a Jacobi polynomial basis, we show that our scheme is capable of achieving rates of convergence which are optimal in the L₂ case, we obtain interesting rates of convergence for other L_p norms which are new (to the best of our knowledge) in the literature, and we also give a simulation study showing that the NEED-D estimator outperforms other standard algorithms in almost all situations.

The positive false discovery rate (pFDR) is a useful overall measure of errors for multiple hypothesis testing, especially when the underlying goal is to attain one or more discoveries. Control of pFDR critically depends on how much evidence is available from data to distinguish between false and true nulls. Oftentimes, as many aspects of the data distributions are unknown, one may not be able to obtain strong enough evidence from the data for pFDR control. This raises the question as to how much data are needed to attain a target pFDR level. We study the asymptotics of the minimum number of observations per null for the pFDR control associated with multiple Studentized tests and F tests, especially when the differences between false nulls and true nulls are small. For Studentized tests, we consider tests on shifts or other parameters associated with normal and general distributions. For F tests, we also take into account the effect of the number of covariates in linear regression. The results show that in determining the minimum sample size per null for pFDR control, higher order statistical properties of data are important, and the number of covariates is important in tests to detect regression effects.

A new class of Marginal Structural Models (MSMs), History-Restricted MSMs (HRMSMs), was recently introduced for longitudinal data for the purpose of defining causal parameters which may often be better suited for public health research or at least more practicable than MSMs (6, 2). HRMSMs allow investigators to analyze the causal effect of a treatment on an outcome based on a fixed, shorter and user-specified history of exposure compared to MSMs. By default, the latter represent the treatment causal effect of interest based on a treatment history defined by the treatments assigned between the study’s start and outcome collection. We lay out in this article the formal statistical framework behind HRMSMs. Beyond allowing a more flexible causal analysis, HRMSMs improve computational tractability and mitigate statistical power concerns when designing longitudinal studies. We also develop three consistent estimators of HRMSM parameters under sufficient model assumptions: the Inverse Probability of Treatment Weighted (IPTW), G-computation and Double Robust (DR) estimators. In addition, we show that the assumptions commonly adopted for identification and consistent estimation of MSM parameters (existence of counterfactuals, consistency, time-ordering and sequential randomization assumptions) also lead to identification and consistent estimation of HRMSM parameters.

We consider the problem of binary classification where one can, for a particular cost, choose not to classify an observation. We present a simple proof for the oracle inequality for the excess risk of structural risk minimizers using a lasso type penalty.

This paper studies oracle properties of ℓ₁-penalized least squares in nonparametric regression setting with random design. We show that the penalized least squares estimator satisfies sparsity oracle inequalities, i.e., bounds in terms of the number of non-zero components of the oracle vector. The results are valid even when the dimension of the model is (much) larger than the sample size and the regression matrix is not positive definite. They can be applied to high-dimensional linear regression, to nonparametric adaptive regression estimation and to the problem of aggregation of arbitrary estimators.

Linear structural error-in-variables models with univariate observations are revisited for studying modified least squares estimators of the slope and intercept. New marginal central limit theorems (CLT’s) are established for these estimators, assuming the existence of four moments for the measurement errors and that the explanatory variables are in the domain of attraction of the normal law. The latter condition for the explanatory variables is used the first time, and is so far the most general in this context. It is also optimal, or nearly optimal, for our CLT’s. Moreover, due to the obtained CLT’s being in Studentized and self-normalized forms to begin with, they are a priori nearly, or completely, data-based, and free of unknown parameters of the joint distribution of the error and explanatory variables. Consequently, they lead to a variety of readily available, or easily derivable, large-sample approximate confidence intervals (CI’s) for the slope and intercept. In contrast, in related CLT’s in the literature so far, the variances of the limiting normal distributions, in general, are complicated and depend on various, typically unknown, moments of the error and explanatory variables. Thus, the corresponding CI’s for the slope and intercept in the literature, unlike those of the present paper, are available only under some additional model assumptions.

Longitudinal data tracking repeated measurements on individuals are highly valued for research because they offer controls for unmeasured individual heterogeneity that might otherwise bias results. Random effects or mixed models approaches, which treat individual heterogeneity as part of the model error term and use generalized least squares to estimate model parameters, are often criticized because correlation between unobserved individual effects and other model variables can lead to biased and inconsistent parameter estimates. Starting with an examination of the relationship between random effects and fixed effects estimators in the standard unobserved effects model, this article demonstrates through analysis and simulation that the mixed model approach has a “bias compression” property under a general model for individual heterogeneity that can mitigate bias due to uncontrolled differences among individuals. The general model is motivated by the complexities of longitudinal student achievement measures, but the results have broad applicability to longitudinal modeling.

We provide sensitivity comparisons for two competing versions of the dimension reduction method principal Hessian directions (pHd). These comparisons consider the effects of small perturbations on the estimation of the dimension reduction subspace via the influence function. We show that the two versions of pHd can behave completely differently in the presence of certain observational types. Our results also provide evidence that outliers in the traditional sense may or may not be highly influential in practice. Since influential observations may lurk within otherwise typical data, we consider the influence function in the empirical setting for the efficient detection of influential observations in practice.

Smoothing methods and SiZer are a useful statistical tool for discovering statistically significant structure in data. Based on scale space ideas originally developed in the computer vision literature, SiZer (SIgnificant ZERo crossing of the derivatives) is a graphical device to assess which observed features are ‘really there’ and which are just spurious sampling artifacts.

In this paper, we develop SiZer like ideas in time series analysis to address the important issue of significance of trends. This is not a straightforward extension, since one data set does not contain the information needed to distinguish ‘trend’ from ‘dependence’. A new visualization is proposed, which shows the statistician the range of trade-offs that are available. Simulation and real data results illustrate the effectiveness of the method.

A coordinate system free definition of complex structure multidimensional item response theory (MIRT) for dichotomously scored items is presented. The point of view taken emphasizes the possibilities and subtleties of understanding MIRT as a multidimensional extension of the “classical” unidimensional item response theory models. The main theorem of the paper is that every monotonic MIRT model looks the same; they are all trivial extensions of univariate item response theory.

In this article, we study rates of convergence of the generalization error of multi-class margin classifiers. In particular, we develop an upper bound theory quantifying the generalization error of various large margin classifiers. The theory permits a treatment of general margin losses, convex or nonconvex, in presence or absence of a dominating class. Three main results are established. First, for any fixed margin loss, there may be a trade-off between the ideal and actual generalization performances with respect to the choice of the class of candidate decision functions, which is governed by the trade-off between the approximation and estimation errors. In fact, different margin losses lead to different ideal or actual performances in specific cases. Second, we demonstrate, in a problem of linear learning, that the convergence rate can be arbitrarily fast in the sample size n depending on the joint distribution of the input/output pair. This goes beyond the anticipated rate O(n⁻¹). Third, we establish rates of convergence of several margin classifiers in feature selection with the number of candidate variables p allowed to greatly exceed the sample size n but no faster than exp(n).

We consider the regression model with (known) random design. We investigate the minimax performances of an adaptive wavelet block thresholding estimator under the L^p risk with p≥2 over Besov balls. We prove that it is near optimal and that it achieves better rates of convergence than the conventional term-by-term estimators (hard, soft,…).

This paper deals simultaneously with linear structural and functional error-in-variables models (SEIVM and FEIVM), revisiting in this context generalized and modified least squares estimators of the slope and intercept, and some methods of moments estimators of unknown variances of the measurement errors. New joint central limit theorems (CLT’s) are established for these estimators in the SEIVM and FEIVM under some first time, so far the most general, respective conditions on the explanatory variables, and under the existence of four moments of the measurement errors. Moreover, due to them being in Studentized forms to begin with, the obtained CLT’s are a priori nearly, or completely, data-based, and free of unknown parameters of the distribution of the errors and any parameters associated with the explanatory variables. In contrast, in related CLT’s in the literature so far, the covariance matrices of the limiting normal distributions are, in general, complicated and depend on various, typically unknown parameters that are hard to estimate. In addition, the very forms of the CLT’s in the present paper are universal for the SEIVM and FEIVM. This extends a previously known interplay between a SEIVM and a FEIVM. Moreover, though the particular methods and details of the proofs of the CLT’s in the SEIVM and FEIVM that are established in this paper are quite different, a unified general scheme of these proofs is constructed for the two models herewith.

In a two-stage cluster sampling procedure, n random populations are drawn independently from independent populations and a sub-sample of observations is taken in each of them. The estimator of the general mean of the observed variables is asymptotically Gaussian and the asymptotic distributions of several bootstrap versions of the normalized and studentized statistics are studied. A weighted population resampling provides a good approximation and its accuracy depends on the convergence rate of the sample size of the populations.

Dynamic factor models have a wide range of applications in econometrics and applied economics. The basic motivation resides in their capability of reducing a large set of time series to only few indicators (factors). If the number of time series is large compared to the available number of observations then most information may be conveyed to the factors. This way low dimension models may be estimated for explaining and forecasting one or more time series of interest. It is desirable that outlier free time series be available for estimation. In practice, outlying observations are likely to arise at unknown dates due, for instance, to external unusual events or gross data entry errors. Several methods for outlier detection in time series are available. Most methods, however, apply to univariate time series while even methods designed for handling the multivariate framework do not include dynamic factor models explicitly. A method for discovering outliers occurrences in a dynamic factor model is introduced that is based on linear transforms of the observed data. Some strategies to separate outliers that add to the model and outliers within the common component are discussed. Applications to simulated and real data sets are presented to check the effectiveness of the proposed method.

We use rescaled Gaussian processes as prior models for functional parameters in nonparametric statistical models. We show how the rate of contraction of the posterior distributions depends on the scaling factor. In particular, we exhibit rescaled Gaussian process priors yielding posteriors that contract around the true parameter at optimal convergence rates. To derive our results we establish bounds on small deviation probabilities for smooth stationary Gaussian processes.

We consider a multivariate density model where we estimate the excess mass of the unknown probability density f at a given level ν>0 from n i.i.d. observed random variables. This problem has several applications such as multimodality testing, density contour clustering, anomaly detection, classification and so on. For the first time in the literature we estimate the excess mass as an integrated functional of the unknown density f. We suggest an estimator and evaluate its rate of convergence, when f belongs to general Besov smoothness classes, for several risk measures. A particular care is devoted to implementation and numerical study of the studied procedure. It appears that our procedure improves the plug-in estimator of the excess mass.

Consider a compound Poisson process with jump measure ν supported by finitely many positive integers. We propose a method for estimating ν from a single, equidistantly sampled trajectory and develop associated statistical procedures. The problem is motivated by the question whether nerve cells in the brain exhibit higher-order interactions in their firing patterns. According to the neuronal assembly hypothesis (Hebb [13]), synchronization of action potentials across neurons of different groups is considered a signature of assembly activity, but it was found notoriously difficult to demonstrate it in recordings of neuronal activity. Our approach based on a compound Poisson model allows to detect the presence of joint spike events of any order using only population spike count samples, thus bypassing both the “curse of dimensionality” and the need to isolate single-neuron spike trains in population signals.

We establish a functional limit law of the logarithm for the increments of the normed quantile process based upon a random sample of size n→∞. We extend a limit law obtained by Deheuvels and Mason [12], showing that their results hold uniformly over the bandwidth h, restricted to vary in [h'_n,h''_n], where {h'_n}_n≥1 and {h''_n}_n≥1 are appropriate non-random sequences. We treat the case where the sample observations follow possibly non-uniform distributions. As a consequence of our theorems, we provide uniform limit laws for nearest-neighbor density estimators, in the spirit of those given by Deheuvels and Mason [13] for kernel-type estimators.

We characterize and study variable importance (VIMP) and pairwise variable associations in binary regression trees. A key component involves the node mean squared error for a quantity we refer to as a maximal subtree. The theory naturally extends from single trees to ensembles of trees and applies to methods like random forests. This is useful because while importance values from random forests are used to screen variables, for example they are used to filter high throughput genomic data in Bioinformatics, very little theory exists about their properties.

We want to recover the regression function in the single-index model. Using an aggregation algorithm with local polynomial estimators, we answer in particular to the second part of Question 2 from Stone (1982) on the optimal convergence rate. The procedure constructed here has strong adaptation properties: it adapts both to the smoothness of the link function and to the unknown index. Moreover, the procedure locally adapts to the distribution of the design. We propose new upper bounds for the local polynomial estimator (which are results of independent interest) that allows a fairly general design. The behavior of this algorithm is studied through numerical simulations. In particular, we show empirically that it improves strongly over empirical risk minimization.

The effect of vigorous physical activity on mortality in the elderly is difficult to estimate using conventional approaches to causal inference that define this effect by comparing the mortality risks corresponding to hypothetical scenarios in which all subjects in the target population engage in a given level of vigorous physical activity. A causal effect defined on the basis of such a static treatment intervention can only be identified from observed data if all subjects in the target population have a positive probability of selecting each of the candidate treatment options, an assumption that is highly unrealistic in this case since subjects with serious health problems will not be able to engage in higher levels of vigorous physical activity. This problem can be addressed by focusing instead on causal effects that are defined on the basis of realistic individualized treatment rules and intention-to-treat rules that explicitly take into account the set of treatment options that are available to each subject. We present a data analysis to illustrate that estimators of static causal effects in fact tend to overestimate the beneficial impact of high levels of vigorous physical activity while corresponding estimators based on realistic individualized treatment rules and intention-to-treat rules can yield unbiased estimates. We emphasize that the problems encountered in estimating static causal effects are not restricted to the IPTW estimator, but are also observed with the G-computation estimator, the DR-IPTW estimator, and the targeted MLE. Our analyses based on realistic individualized treatment rules and intention-to-treat rules suggest that high levels of vigorous physical activity may confer reductions in mortality risk on the order of 15-30%, although in most cases the evidence for such an effect does not quite reach the 0.05 level of significance.

When a series of (related) linear models has to be estimated it is often appropriate to combine the different data-sets to construct more efficient estimators. We use ℓ₁-penalized estimators like the Lasso or the Adaptive Lasso which can simultaneously do parameter estimation and model selection. We show that for a time-course of high-dimensional linear models the convergence rates of the Lasso and of the Adaptive Lasso can be improved by combining the different time-points in a suitable way. Moreover, the Adaptive Lasso still enjoys oracle properties and consistent variable selection. The finite sample properties of the proposed methods are illustrated on simulated data and on a real problem of motif finding in DNA sequences.

We observe a large number of functions differing from each other only by a translation parameter. While the main pattern is unknown, we propose to estimate the shift parameters using M-estimators. Fourier transform enables to transform this statistical problem into a semi-parametric framework. We study the convergence of the estimator and provide its asymptotic behavior. Moreover, we use the method in the applied case of velocity curve forecasting.

A set of curves or images of similar shape is an increasingly common functional data set collected in the sciences. Principal Component Analysis (PCA) is the most widely used technique to decompose variation in functional data. However, the linear modes of variation found by PCA are not always interpretable by the experimenters. In addition, the modes of variation of interest to the experimenter are not always linear. We present in this paper a new analysis of variance for Functional Data. Our method was motivated by decomposing the variation in the data into predetermined and interpretable directions (i.e. modes) of interest. Since some of these modes could be nonlinear, we develop a new defined ratio of sums of squares which takes into account the curvature of the space of variation. We discuss, in the general case, consistency of our estimates of variation, using mathematical tools from differential geometry and shape statistics. We successfully applied our method to a motivating example of biological data. This decomposition allows biologists to compare the prevalence of different genetic tradeoffs in a population and to quantify the effect of selection on evolution.