On the basis of N i.i.d. random variables with a common unknown distribution P we wish to estimate a functional $\tau_N(P)$. An obvious and very general approach to this problem is to find an estimator $\hat{P}_N$ of P first, and then construct a so-called substitution estimator $\tau_N (\hat{P}_N)$ of $\tau_N(P)$. In this paper we investigate how to choose the estimator $\hat{P}_N$ so that the substitution estimator $\tau_N (\hat{P}_N)$ will be consistent.

Although our setup covers a broad class of estimation problems, the main substitution estimator we have in mind is a general version of the bootstrap where resampling is done from an estimated distribution $\hat{P}_N$. We do not focus in advance on a particular estimator $\hat{P}_N$, such as, for example, the empirical distribution, but try to indicate which resampling distribution should be used in a particular situation. The conclusion that we draw from the results and the examples in this paper is that the bootstrap is an exceptionally flexible method which comes into its own when full use is made of its flexibility. However, the choice of a good bootstrap method in a particular case requires rather precise information about the structure of the problem at hand. Unfortunately, this may not always be available.

We will introduce a generic approach for solving problems in pattern recognition based on the synthesis of accurate multiclass discriminators from large numbers of very inaccurate "weak" models through the use of discrete stochastic processes. Contrary to the standard expectation held for the many statistical and heuristic techniques normally associated with the field, a significant feature of this method of "stochastic modeling" is its resistance to so-called "overtraining." The drop in performance of any stochastic model in going from training to test data remains comparable to that of the component weak models from which it is synthesized; and since these component models are very simple, their performance drop is small, resulting in a stochastic model whose performance drop is also small despite its high level of accuracy.

In model selection, usually a "best" predictor is chosen from a collection ${\hat{\mu}(\cdot, s)}$ of predictors where $\hat{\mu}(\cdot, s)$ is the minimum least-squares predictor in a collection $\mathsf{U}_s$ of predictors. Here s is a complexity parameter; that is, the smaller s, the lower dimensional/smoother the models in $\mathsf{U}_s$.

If $\mathsf{L}$ is the data used to derive the sequence ${\hat{\mu}(\cdot, s)}$, the procedure is called unstable if a small change in $\mathsf{L}$ can cause large changes in ${\hat{\mu}(\cdot, s)}$. With a crystal ball, one could pick the predictor in ${\hat{\mu}(\cdot, s)}$ having minimum prediction error. Without prescience, one uses test sets, cross-validation and so forth. The difference in prediction error between the crystal ball selection and the statistician's choice we call predictive loss. For an unstable procedure the predictive loss is large. This is shown by some analytics in a simple case and by simulation results in a more complex comparison of four different linear regression methods. Unstable procedures can be stabilized by perturbing the data, getting a new predictor sequence ${\hat{\mu'}(\cdot, s)}$ and then averaging over many such predictor sequences.

The principal result is that, under conditions, to any nonparametric regression problem there corresponds an asymptotically equivalent sequence of white noise with drift problems, and conversely. This asymptotic equivalence is in a global and uniform sense. Any normalized risk function attainable in one problem is asymptotically attainable in the other, with the difference in normalized risks converging to zero uniformly over the entire parameter space. The results are constructive. A recipe is provided for producing these asymptotically equivalent procedures. Some implications and generalizations of the principal result are also discussed.

Signal recovery in Gaussian white noise with variance tending to zero has served for some time as a representative model for nonparametric curve estimation, having all the essential traits in a pure form. The equivalence has mostly been stated informally, but an approximation in the sense of Le Cam's deficiency distance $\Delta$ would make it precise. The models are then asymptotically equivalent for all purposes of statistical decision with bounded loss. In nonparametrics, a first result of this kind has recently been established for Gaussian regression. We consider the analogous problem for the experiment given by n i.i.d. observations having density f on the unit interval. Our basic result concerns the parameter space of densities which are in a Hölder ball with exponent $\alpha > 1/2$ and which are uniformly bounded away from zero. We show that an i. i. d. sample of size n with density f is globally asymptotically equivalent to a white noise experiment with drift $f^{1/2}$ and variance $1/4 n^{-1}$. This represents a nonparametric analog of Le Cam's heteroscedastic Gaussian approximation in the finite dimensional case. The proof utilizes empirical process techniques related to the Hungarian construction. White noise models on f and log f are also considered, allowing for various "automatic" asymptotic risk bounds in the i.i.d. model from white noise.

We wish to estimate the probability density $g(y)$ that produced an observed random sample of vectors $y_1, y_2, \dots, y_n$. Estimates of $g(y)$ are traditionally constructed in two quite different ways: by maximum likelihood fitting within some parametric family such as the normal or by nonparametric methods such as kernel density estimation. These two methods can be combined by putting an exponential family "through" a kernel estimator. These are the specially designed exponential families mentioned in the title. Poisson regression methods play a major role in calculations concerning such families.

Compared to traditional approaches to curve estimation, such as those based on kernels, wavelet methods are relatively unaffected by discontinuities and similar aberrations. In particular, the mean square convergence rate of a wavelet estimator of a fixed, piecewise-smooth curve is not influenced by discontinuities. Nevertheless, it is clear that as the estimation problem becomes more complex the limitations of wavelet methods must eventually be apparent. By allowing the number of discontinuities to increase and their size to decrease as the sample grows, we study the limits to which wavelet methods can be pushed and still exhibit good performance. We determine the effect of these changes on rates of convergence. For example, we derive necessary and sufficient conditions for wavelet methods applied to increasingly complex, discontinuous functions to achieve convergence rates normally associated only with fixed, smooth functions, and we determine necessary conditions for mean square consistency.

Let a function f be observed with a noise. We wish to test the null hypothesis that the function is identically zero, against a composite nonparametric alternative: functions from the alternative set are separated away from zero in an integral (e.g., $L_2$) norm and also possess some smoothness properties. The minimax rate of testing for this problem was evaluated in earlier papers by Ingster and by Lepski and Spokoiny under different kinds of smoothness assumptions. It was shown that both the optimal rate of testing and the structure of optimal (in rate) tests depend on smoothness parameters which are usually unknown in practical applications. In this paper the problem of adaptive (assumption free) testing is considered. It is shown that adaptive testing without loss of efficiency is impossible. An extra log log-factor is inessential but unavoidable payment for the adaptation. A simple adaptive test based on wavelet technique is constructed which is nearly minimax for a wide range of Besov classes.

We define a minimum distance estimate of the smoothing factor for kernel density estimates, based on a methodology first developed by Yatracos. It is shown that if $f_{nh}$ denotes the kernel density estimate on $\mathbb{R}^d$ for an i.i.d. sample of size n drawn from an unknown density f, where h is the smoothing factor, and if $f_n$ is the kernel estimate with the same kernel and with the proposed new data-based smoothing factor, then, under a regularity condition on the kernel K, $$\sup_f \limsup_{n \to \infty} \frac{E \int | f_n - f|dx}{\inf_{h>0} E \int |f_{nh} - f|dx} \leq 3.$$ This is the first published smoothing factor that can be proven to have this property.

We shall study the general regression model $Y = g_0 (X) + \varepsilon$, where X and $varepsilon$ are independent. The available information about $g_0$ can be expressed by $g_0 \epsilon \mathscr{G}$ for some class $\mathscr{G}$. As an estimator of $g_0$ we choose the least squares estimator. We shall give necessary and sufficient conditions for consistency of this estimator in terms of (basically) geometric properties of $\mathscr{G}$. Our main tool will be the theory of empirical processes.

A general constrained minimum risk inequality is derived. Given two densities $f_{\theta}$ and $f_0$ we find a lower bound for the risk at the point $\theta$ given an upper bound for the risk at the point 0. The inequality sheds new light on superefficient estimators in the normal location problem and also on an adaptive estimation problem arising in nonparametric functional estimation.

Since not all $2^{n-1}$ fractional factorial designs with maximum resolution are equally good, Fries and Hunter introduced the minimum aberration criterion for selecting good $2^{n-1}$ fractional factorial designs with the same resolution. We modify the concept of minimum aberration and define weak minimum aberration and show the usefulness of this new design concept. Using some techniques from finite geometry, we construct $2^{n-1}$ fractional factorial designs of resolution III with weak minimum aberration. Further, several families of $2^{n-1}$ fractional factorial designs of resolution III and IV with minimum aberration are obtained.

A general result is obtained that relates the word-length pattern of a $2^{n-k}$ design to that of its complementary design. By applying this result and using group isomorphism, we are able to characterize minimum aberration $2^{n-k}$ designs in terms of properties of their complementary designs. The approach is quite powerful for small values of $2^{n-k} - n - 1$. In particular, we obtain minimum aberration $2^{n-k}$ designs with $2^{n-k} - n - 1 = 1$ to 11 for any n and k.

Consider search designs for identifying at most k nonzero effects under a search linear model. In the noisy case, the procedure for identifying these effects, which is based on the sum of squares due to error, is considered and its stochastic properties are studied. The errors of observations are assumed to be distributed independently with the normal distribution $N(0, \sigma^2)$ and, in particular, the case of $k = 1$ is considered. Some results on a searching probability under the procedure are presented. Probabilities with respect to search designs which are main effect plus one plans constructed in earlier works, are presented. A comparison between two search designs is given by using a criterion based on probability.

An experiment records stimulus and response for a random sample of cases. The relationship between response and stimulus is thought to be linear, the values of the slope and intercept varying by case. From such data, we construct a consistent, asymptotically normal, nonparametric estimator for the joint density of the slope and intercept. Our methodology incorporates the radial projection-slice theorem for the Radon transform, a technique for locally linear nonparametric regression and a tapered Fourier inversion. Computationally, the new density estimator is more feasible than competing nonparametric estimators, one of which is based on moments and the other on minimum distance considerations.

A structured linear regression model is one in which there are permanent dependencies among some p row vectors of the $n \times p$ design matrix. To study structured linear regression, we introduce a new class of robust estimators, called D-estimators, which can be regarded as a generalization of the least median of squares and least trimmed squares estimators. They minimize a dispersion function of the ordered absolute residuals up to the rank h. We investigate their breakdown point and exact fit point as a function of h in structured linear regression. It is found that the D- and S-estimators can achieve the highest possible breakdown point for h appropriately chosen. It is shown that both the maximum breakdown point and the corresponding optimal value of h, $h_{\mathrm{op}}$, are sample dependent. They hinge on the design but not on the response. The relationship between the breakdown point and the design vanishes when h is strictly larger than $h_{\mathrm{op}}$. However, when h is smaller than $h_{\mathrm{op}}$, the breakdown point depends in a complicated way on the design as well as on the response.

We obtain strong Bahadur representations for a general class of M-estimators that satisfies $\Sigma_i \psi (x_i, \theta) = o(\delta_n)$, where the $x_i$'s are independent but not necessarily identically distributed random variables. The results apply readily to M-estimators of regression with nonstochastic designs. More specifically, we consider the minimum $L_p$ distance estimators, bounded influence GM-estimators and regression quantiles. Under appropriate design conditions, the error ratesobtained for the first-order approximations are sharp in these cases. We also provide weaker and more easily verifiable conditions that suffice for an error rate that is suboptimal but strong enough for deriving the asymptotic distribution of M-estimators in a wide variety of problems.

Given data $X_1, \dots, X_n$ and a kernel h with m arguments, Serfling introduced the class of generalized L-statistics (GL-statistics), which is defined by taking linear combinations of the ordered $h(X_{i_1}, \dots, X_{i_m})$ where $(i_1, \dots, i_m)$ ranges over all $n!/(n - m)!$ distinct m-tuples of $(1, \dots, n)$. In this paper we derive a class of incomplete generalized L-statistics (IGL-statistics) by taking linear combinations of the ordered elements from a subset of ${h(X_{i_1}, \dots, X_{i_m})}$ with size $N(n)$. A special case is the class of incomplete U-statistics, introduced by Blom. Under very general conditions, the IGL-statistic is asymptotically equivalent to the GL-statistic as soon as $N(n)/n \to \infty \as n \to \infty$, which makes the IGL much more computationally feasible. We also discuss various ways of selecting the subset of ${h(X_{i_1}, \dots, X_{i_m})}$. Several examples are discussed. In particular, some new estimates of the scale parameter in nonparametric regression are introduced. It is shown that these estimates are asymptotically equivalent to an IGL-statistic. Some extensions, for example, functionals other than L and multivariate kernels, are also addressed.

Maximum likelihood estimators for record-value data with a linear trend are quite sensitive to misspecification of the error distribution. Indeed, incorrect choice of that distribution can lead to inconsistent estimation of the intercept parameter and produce estimators of slope that do not enjoy the asymptotic convergence rate prescribed by the information matrix. These properties and the importance of linearly trended records lead us to suggest a distribution-free approach to inference. We show that the slope and intercept parameters and the entire error distribution can be estimated consistently, and that bootstrap methods are available. The latter may be employed to estimate the variance of estimators of slope, intercept and error distributions. The case of trends that increase faster than linearly is also considered, but is shown to be relatively uninteresting in the sense that the natural estimators have rather predictable properties.

Let $\hat{F}_n$ be the Kaplan-Meier estimator of a distribution function F computed from randomly censored data. It is known that, under certain integrability assumptions on a function $\varphi$, the Kaplan-Meier integral $\int \varphi d \hat{F}_n$, when properly standardized, is asymptotically normal. In this paper it is shown that, with probability 1, the jackknife estimate of variance consistently estimates the (limit) variance.

To compare two samples of possibly right-censored failure times, a location-scale model, without assuming the distribution form, is considered for the log-transformed data. This new accelerated failure time model is introduced to accommodate the possible heterogeneity between within-treatment variations of the two groups considered. One distinct feature of this model is that, with the presence of heterogeneity, statistical inferences on both location and scale parameters based on log-rank tests or linear rank tests will become inappropriate. In this paper we propose the empirical process approach to construct a regression setup derived from the theory of strong approximation of the Kaplan-Meier product-limit empirical quantile process. A generalized least squares (GLS) estimator is obtained and shown to be semiparametric efficient. Also it is shown that this estimation is adaptive for a special case. At the end, results on the two-sample setting are applied to the K-sample problem.

Linear regression with doubly censored responses is considered. Buckley-James-Ritov-type estimators are proposed. Semiparametric information and projective scores are discussed. An expansion of the estimating equations is obtained under fairly general assumptions. Sufficient conditions are given for the asymptotic consistency and normality of the estimators.

Universal Gaussian approximations are established for empirical cumulative hazard and product-limit processes under random censorship. They hold uniformly up to some large order statistics in the sample, with the approximation rates depending on the order of these statistics, and require no assumptions on the censoring mechanism. Weak convergence results and laws of the iterated logarithm follow on the whole line if the respective processes are stopped at certain large order statistics, depending on the type of result. Some new consequences and negative results for confidence-band construction are discussed. Some new uniform consistency rates up to large order statistics are also derived and shown to be universally best possible for a wide range of tail order statistics.