Additive principal components are a nonlinear generalization of linear principal components. Their distinguishing feature is that linear forms $\sum_ia_i X_i$ are replaced with additive functions $\sum_i\phi_i(X_i)$. A considerable amount of flexibility for fitting data is gained when linear methods are replaced with additive ones. Our interest is in the smallest principal components, which is somewhat uncommon. Smallest additive principal components amount to data descriptions in terms of approximate implicit equations: $\sum_i\phi_i(X_i) \approx 0$. Estimation of such equations is a data-analytic method in its own right, competing in some cases with the more customary regression approaches. It is also a diagnostic tool in additive regression for detection of so-called "concurvity." This term describes degeneracies among predictor variables in additive regression models, similar to collinearity in linear regression models. Concurvity may lead to statistically unstable contributions of variables to additive models. As an example, we show in a reanalysis of the ozone data from Breiman and Friedman that concurvity does indeed exist in this particular data, a fact which may impact the interpretation of the additive fits. In the second half of this paper, we give some second-order theory, including the description of null situations and eigenexpansions derived from associated eigenproblems. We show how ACE and additive principal components are related, and we outline some analytical methods for theoretical calculations of additive principal components. Lastly we consider methods of estimation and computation. Additive principal components have had a long tradition in psychometric research and correspondence analysis. They have started receiving attention by statisticians only in recent years.

We consider the fitting of generalized linear models in which the link function is assumed to be unknown, and propose the following computational method: First, estimate regression coefficients using the canonical link. Then, estimate the link via a kernel smoother, treating the direction in the predictor space determined by the regression coefficients as known. Then reestimate the direction using the estimated link and alternate between these two steps. We show that under fairly general conditions, $n^{1/2}$-consistent estimates of the direction are obtained. A small Monte Carlo study is presented.

Several Markov chain methods are available for sampling from a posterior distribution. Two important examples are the Gibbs sampler and the Metropolis algorithm. In addition, several strategies are available for constructing hybrid algorithms. This paper outlines some of the basic methods and strategies and discusses some related theoretical and practical issues. On the theoretical side, results from the theory of general state space Markov chains can be used to obtain convergence rates, laws of large numbers and central limit theorems for estimates obtained from Markov chain methods. These theoretical results can be used to guide the construction of more efficient algorithms. For the practical use of Markov chain methods, standard simulation methodology provides several variance reduction techniques and also give guidance on the choice of sample size and allocation.

In the problem of estimating an unknown distribution function $F$ in the presence of censoring, one can use a nonparametric estimator such as the Kaplan-Meier estimator, or one can consider parametric modeling. There are many situations where physical reasons indicate that a certain parametric model holds approximately. In these cases a nonparametric estimator may be very inefficient relative to a parametric estimator. On the other hand, if the parametric model is only a crude approximation to the actual model, then the parametric estimator may perform poorly relative to the nonparametric estimator, and may even be inconsistent. The Bayesian paradigm provides a reasonable framework for this problem. In a Bayesian approach, one would try to put a prior distribution on $F$ that gives most of its mass to small neighborhoods of the entire parametric family. We show that certain priors based on the Dirichlet process prior can be used to accomplish this. For these priors the posterior distribution of $F$ given the censored data appears to be analytically intractable. We provide a method for approximating this posterior distribution through the use of a successive substitution sampling algorithm. We also show convergence of the algorithm.

Preexperimental frequentist error probabilities are arguably inadequate, as summaries of evidence from data, in many hypothesis-testing settings. The conditional frequentist may respond to this by identifying certain subsets of the outcome space and reporting a conditional error probability, given the subset of the outcome space in which the observed data lie. Statistical methods consistent with the likelihood principle, including Bayesian methods, avoid the problem by a more extreme form of conditioning. In this paper we prove that the conditional frequentist's method can be made exactly equivalent to the Bayesian's in simple versus simple hypothesis testing: specifically, we find a conditioning strategy for which the conditional frequentist's reported conditional error probabilities are the same as the Bayesian's posterior probabilities of error. A conditional frequentist who uses such a strategy can exploit other features of the Bayesian approach--for example, the validity of sequential hypothesis tests (including versions of the sequential probability ratio test, or SPRT) even if the stopping rule is incompletely specified.

Asymptotic expansions of posterior distributions are derived for a two-dimensional exponential family, which includes normal, gamma, inverse gamma and inverse Gaussian distributions. Reparameterization allows us to use a data-dependent transformation, convert the likelihood function to the two-dimensional standard normal density and apply a version of Stein's identity to assess the posterior distributions. Applications are given to characterize optimal noninformative priors in the sense of Stein, to suggest the form of a high-order correction to the distribution function of a sequential likelihood ratio statistic and to provide confidence intervals for one parameter in the presence of other nuisance parameters.

The problem of Bayes estimation of the scale parameter is considered. Lower bounds for the asymptotic Bayes risk are given as the restricted parameter space increases to the positive half-line. The results are next applied to establish the second-order minimax estimator of the scale parameter. Surprisingly, the least favorable distribution coincides with that for the corresponding location parameter problem.

We consider the relationship between the finite-dimensional distributions of a stationary time series model and its asymptotic behavior in the framework of interval-valued probability (IVP), a simple generalization of additive probability measures. By Caratheodory's theorem, the specification of a countably additive probability measure on the algebra of cylinders $\mathscr{C}$ uniquely defines its behavior on $\sigma(\mathscr{C})$ (containing the tail events). If the measure is stationary, then the ergodic theorem indicates that its extension assigns zero probability to the tail event consisting of all sequences for which the time averages diverge (the divergence event). This link between the marginals and the tail behavior is no longer valid in IVP, and we can reconcile arbitrary finite-dimensional distributions and tail behavior for stationary IVP-based models. The linking mechanism between the marginals of a time series model and its asymptotic behavior turns out to be continuity, not stationarity or even additivity. We prove that any stationary finitely additive probability (charge) defined on cylinders has a stationary charge extension that can assign the divergence event any prescribed probability. Moreover, on the space of binary sequences, we consider IVP models that incorporate: (i) Stationarity. (ii) Continuity along $\mathscr{C}$. (iii) Almost sure support for divergence. (iv) Estimability of the divergence event from cylinders. (v) Nearly additive finite-dimensional distributions. We enhance the previous constructions of IVP's satisfying (i)-(iv) so that they satisfy (v) by agreeing with a stationary measure either exactly on one-dimensional cylinders or arbitrarily closely on a given class of bounded-dimensional cylinders. Our time series constructions follow from the observation that the algebra of cylinders and the tail $\sigma$-algebra are mutually nonsingular. We use the same idea to prove the existence of joints for general marginal IVP's. These constructions have implications for frequentist interpretations of probability.

This paper indicates that a minor modification of the maximum likelihood estimate of Vardi, Shepp and Kaufman can be regarded as a step in the standard nonparametric MLE by the method of sieves and establishes the asymptotic consistency for it. This method of sieves suggests that the number of pixels needs to be in line with the number of detectors in order to avoid poor image reconstructions. A simulation study is also presented to support this suggestion.

Hidden Markov models are today widespread for modeling of various phenomena. It has recently been shown by Leroux that the maximum-likelihood estimate (MLE) of the parameters of a such a model is consistent, and local asymptotic normality has been proved by Bickel and Ritov. In this paper we propose a new class of estimates which are consistent, asymptotically normal and almost as good as the MLE.

Suppose one observes independent samples of size $n$ from both the mixture density $\int p(x \mid z) d\eta(z)$ and from the distribution $\eta$. The kernel $p(x \mid z)$ is known. We show asymptotic normality and efficiency of the maximum likelihood estimator for $\eta$.

Stochastic regression models of the form $y_i = f_i(\theta) + \varepsilon_i$, where the random disturbances $\varepsilon_i$ form a martingale difference sequence with respect to an increasing sequence of $\sigma$-fields $\{\mathcal{G}_i\}$ and $f_i$ is a random $\mathcal{G}_{i - 1}$-measurable function of an unknown parameter $\theta$, cover a broad range of nonlinear (and linear) time series and stochastic process models. Herein strong consistency and asymptotic normality of the least squares estimate of $\theta$ in these stochastic regression models are established. In the linear case $f_i(\theta) = \theta^T\psi_i$, they reduce to known results on the linear least squares estimate $(\sum^n_1\psi_i\psi^T_i)^{-1}\sum^n_1\psi_i y_i$ with stochastic $\mathcal{G}_{i - 1}$-measurable regressors $\psi_i$.

Let $(X_{ij}, \mathbf{z}_i), i = 1, 2, \ldots, k, j = 1, 2, \ldots, n_i$, be independent observations such that $\mathbf{z}_i$ is a fixed $r \times 1$ vector $\lbrack r$ can be 0 (no $\mathbf{z}$'s observed) or $1, 2, \ldots, k - 1\rbrack$, and $X_{ij}$ is distributed according to a one-parameter exponential family which is log concave with natural parameter $\theta_i$. We test the hypothesis that $\theta = \mathbf{Z}\beta$, where $\theta = (\theta_1, \ldots, \theta_k)', \mathbf{Z}$ is the matrix whose $i$th row is $\mathbf{z}'_i$ and $\beta = (\beta_1, \ldots, \beta_r)'$ is a vector of parameters. We focus on $r = 2$ and $\mathbf{z}'_i = (1, z_i), i = 1, 2, \ldots, k, z_i < z_{i + 1}$. The null hypothesis on hand is thus of the form $\theta_i = \beta_1 + \beta_2z_i$. In such a case the model under the null hypothesis becomes logistic regression in the binomial case, Poisson regression in the Poisson case and linear regression in the normal case. We consider mostly the one-sided alternative that the second-order differences of the natural parameters are nonnegative. Such testing problems test goodness of fit vs. alternatives in which the natural parameters behave in a convex way. We find classes of tests that are unbiased and that lie in a complete class. We also note that every admissible test of constant size is unbiased. In some discrete situations we find the minimal complete class of unbiased admissible tests. Generalizations and examples are given.

A new criterion is proposed for the evaluation of variable selection procedures in multiple regression. This criterion, which we call the risk inflation, is based on an adjustment to the risk. Essentially, the risk inflation is the maximum increase in risk due to selecting rather than knowing the "correct" predictors. A new variable selection procedure is obtained which, in the case of orthogonal predictors, substantially improves on AIC, $C_p$ and BIC and is close to optimal. In contrast to AIC, $C_p$ and BIC which use dimensionality penalties of 2, 2 and $\log n$, respectively, this new procedure uses a penalty $2 \log p$, where $p$ is the number of available predictors. For the case of nonorthogonal predictors, bounds for the optimal penalty are obtained.

Random coefficient regression models are important in modeling heteroscedastic multivariate linear regression in econometrics. The analysis of panel data is one example. In statistics, the random and mixed effects models of ANOVA, deconvolution models and affine mixture models are all special cases of random coefficient regression. Some inferential problems, such as constructing prediction regions for the modeled response, require a good nonparametric estimator of the unknown coefficient distribution. This paper introduces and studies a consistent nonparametric minimum distance method for estimating the coefficient distribution. Our estimator translates the difficult problem of estimating an inverse Radon transform into a minimization problem.

Limit theorems for an $M$-estimate constrained to lie in a closed subset of $\mathbb{R}^d$ are given under two different sets of regularity conditions. A consistent sequence of global optimizers converges under Chernoff regularity of the parameter set. A $\sqrt n$-consistent sequence of local optimizers converges under Clarke regularity of the parameter set. In either case the asymptotic distribution is a projection of a normal random vector on the tangent cone of the parameter set at the true parameter value. Limit theorems for the optimal value are also obtained, agreeing with Chernoff's result in the case of maximum likelihood with global optimizers.

Recent work of several authors has focussed on first-order properties (e.g., consistency) of general bootstrap algorithms, where the numbers of times that data values are resampled form an exchangeable sequence. In the present paper we develop second-order properties of such algorithms, in a very general setting. Performance is discussed in the context of distribution estimation, and formulae for higher-order moments and cumulants are developed. Arguing thus, necessary and sufficient conditions are given for general resampling algorithms to correctly capture second-order properties.

In this article, the construction of confidence regions by approximating the sampling distribution of some statistic is studied. The true sampling distribution is estimated by an appropriate normalization of the values of the statistic computed over subsamples of the data. In the i.i.d. context, the method has been studied by Wu in regular situations where the statistic is asymptotically normal. The goal of the present work is to prove the method yields asymptotically valid confidence regions under minimal conditions. Essentially, all that is required is that the statistic, suitably normalized, possesses a limit distribution under the true model. Unlike the bootstrap, the convergence to the limit distribution need not be uniform in any sense. The method is readily adapted to parameters of stationary time series or, more generally, homogeneous random fields. For example, an immediate application is the construction of a confidence interval for the spectral density function of a homogeneous random field.

This paper studies the weak convergence of the sequential empirical process $\widehat{K}_n$ of the estimated residuals in ARMA$(p, q)$ models when the errors are independent and identically distributed. It is shown that, under some mild conditions, $\widehat{K}_n$ converges weakly to a Kiefer process. The weak convergence is discussed for both finite and infinite variance time series models. An application to a change-point problem is considered.

Verifiable conditions are given for the validity of formal Edgeworth expansions for the distribution of sums $X_1 + \cdots + X_n$, where $X_i = F(Z_i, \ldots, Z_{i + p - 1})$ and $Z_1, Z_2, \ldots$ is a strict sense stationary sequence that can be written as $Z_j = g(\varepsilon_{j - k}: k \geq 0)$ with an $\operatorname{iid}$ sequence $(\varepsilon_i)$ of innovations. These models include nonlinear functions of ARMA processes $(Z_i)$ as well as certain nonlinear AR processes. The results apply to many statistics in (nonlinear) time series models.

In this paper we propose a Capon-type estimator for the spectrum of a stationary time series. This estimator may be viewed as an alternative to classical periodogram-based estimators. Its advantage is that it copes with the "leakage effect" by using implicitly automatic adaptive windowing. We show its asymptotic equivalence to a random variable which is a quadratic form in the observations, thus obtaining the asymptotic normality of the Capon estimator. We also study its asymptotic bias and variance.

We describe kernel methods for estimating the covariance function of a stationary stochastic process, and show how to ensure that the estimator has the positive semidefiniteness property. From a practical viewpoint, our method is significant because it does not demand a parametric model for covariance. From a technical angle, our results exhibit a striking departure from those in more familiar cases of kernel estimation. For example, in the context of covariance estimation, kernel estimators can have the same convergence rates as maximum likelihood estimators, and can have exceptionally fast convergence rates when employed to estimate variance.