New upper bounds are developed for the ${\mathit{L}}_2$ distance between $\mathit{\xi }/Var{[\mathit{\xi }]}^{1/2}$ and linear and quadratic functions of $\mathit{z}\sim \mathit{N}(\mathbf{0},{\mathit{I}}_{\mathit{n}})$ for random variables of the form $\mathit{\xi }={\mathit{z}}^{\top }\mathit{f}(\mathit{z})-div\mathit{f}(\mathit{z})$. The linear approximation yields a central limit theorem when the squared norm of $\mathit{f}(\mathit{z})$ dominates the squared Frobenius norm of $\nabla \mathit{f}(\mathit{z})$ in expectation.

Applications of this normal approximation are given for the asymptotic normality of debiased estimators in linear regression with correlated design and convex penalty in the regime $\mathit{p}/\mathit{n}\le \mathit{\gamma }$ for constant $\mathit{\gamma }\in (0,\infty )$. For the estimation of linear functions $⟨{\mathit{a}}_0,\mathit{\beta }⟩$ of the unknown coefficient vector β, this analysis leads to asymptotic normality of the debiased estimate for most normalized directions ${\mathit{a}}_0$, where “most” is quantified in a precise sense. This asymptotic normality holds for any convex penalty if $\mathit{\gamma }<1$ and for any strongly convex penalty if $\mathit{\gamma }\ge 1$. In particular, the penalty needs not be separable or permutation invariant. By allowing arbitrary regularizers, the results vastly broaden the scope of applicability of debiasing methodologies to obtain confidence intervals in high dimensions. In the absence of strong convexity for $\mathit{p}>\mathit{n}$, asymptotic normality of the debiased estimate is obtained for the Lasso and the group Lasso under additional conditions. For general convex penalties, our analysis also provides prediction and estimation error bounds of independent interest.

The iterated conditional sequential Monte Carlo (i-CSMC) algorithm from Andrieu, Doucet and Holenstein (J. R. Stat. Soc. Ser. B Stat. Methodol. 72 (2010) 269–342) is an MCMC approach for efficiently sampling from the joint posterior distribution of the T latent states in challenging time-series models, for example, in nonlinear or non-Gaussian state-space models. It is also the main ingredient in particle Gibbs samplers which infer unknown model parameters alongside the latent states. In this work, we first prove that the i-CSMC algorithm suffers from a curse of dimension in the dimension of the states, D: it breaks down unless the number of samples (‘particles’), N, proposed by the algorithm grows exponentially with D. Then we present a novel ‘local’ version of the algorithm which proposes particles using Gaussian random-walk moves that are suitably scaled with D. We prove that this iterated random-walk conditional sequential Monte Carlo (i-RW-CSMC) algorithm avoids the curse of dimension: for arbitrary N, its acceptance rates and expected squared jumping distance converge to nontrivial limits as $\mathit{D}\to \infty$. If $\mathit{T}=\mathit{N}=1$, our proposed algorithm reduces to a Metropolis–Hastings or Barker’s algorithm with Gaussian random-walk moves and we recover the well-known scaling limits for such algorithms.

Local differential privacy has recently received increasing attention from the statistics community as a valuable tool to protect the privacy of individual data owners without the need of a trusted third party. Similar to the classical notion of randomized response, the idea is that data owners randomize their true information locally and only release the perturbed data. Many different protocols for such local perturbation procedures can be designed. In most estimation problems studied in the literature so far, however, no significant difference in terms of minimax risk between purely noninteractive protocols and protocols that allow for some amount of interaction between individual data providers could be observed. In this paper, we show that for estimating the integrated square of a density, sequentially interactive procedures improve substantially over the best possible noninteractive procedure in terms of minimax rate of estimation.

In particular, in the noninteractive scenario we identify an elbow in the minimax rate at $\mathit{s}=\frac{3}{4}$, whereas in the sequentially interactive scenario the elbow is at $\mathit{s}=\frac{1}{2}$. This is markedly different from both, the case of direct observations, where the elbow is well known to be at $\mathit{s}=\frac{1}{4}$, as well as from the case where Laplace noise is added to the original data, where an elbow at $\mathit{s}=\frac{9}{4}$ is obtained.

We also provide adaptive estimators that achieve the optimal rate up to log-factors, we draw connections to nonparametric goodness-of-fit testing and estimation of more general integral functionals and conduct a series of numerical experiments. The fact that a particular locally differentially private, but interactive, mechanism improves over the simple noninteractive one is also of great importance for practical implementations of local differential privacy.

We study the classical problem of recovering a multidimensional source signal from observations of nonlinear mixtures of this signal. We show that this recovery is possible (up to a permutation and monotone scaling of the source’s original component signals) if the mixture is due to a sufficiently differentiable and invertible but otherwise arbitrarily nonlinear function and the component signals of the source are statistically independent with ‘nondegenerate’ second-order statistics. The latter assumption requires the source signal to meet one of three regularity conditions which essentially ensure that the source is sufficiently far away from the nonrecoverable extremes of being deterministic or constant in time. These assumptions, which cover many popular time series models and stochastic processes, allow us to reformulate the initial problem of nonlinear blind source separation as a simple-to-state problem of optimisation-based function approximation. We propose to solve this approximation problem by minimizing a novel type of objective function that efficiently quantifies the mutual statistical dependence between multiple stochastic processes via cumulant-like statistics. This yields a scalable and direct new method for nonlinear Independent Component Analysis with widely applicable theoretical guarantees and for which our experiments indicate good performance.

We study mean-field variational Bayesian inference using the TAP approach, for ${\mathbb{Z}}_2$-synchronization as a prototypical example of a high-dimensional Bayesian model. We show that for any signal strength $\mathit{\lambda }>1$ (the weak-recovery threshold), there exists a unique local minimizer of the TAP free energy functional near the mean of the Bayes posterior law. Furthermore, the TAP free energy in a local neighborhood of this minimizer is strongly convex. Consequently, a natural-gradient/mirror-descent algorithm achieves linear convergence to this minimizer from a local initialization, which may be obtained by a constant number of iterations of Approximate Message Passing (AMP). This provides a rigorous foundation for variational inference in high dimensions via minimization of the TAP free energy.

We also analyze the finite-sample convergence of AMP, showing that AMP is asymptotically stable at the TAP minimizer for any $\mathit{\lambda }>1$, and is linearly convergent to this minimizer from a spectral initialization for sufficiently large λ. Such a guarantee is stronger than results obtainable by state evolution analyses, which only describe a fixed number of AMP iterations in the infinite-sample limit.

Our proofs combine the Kac–Rice formula and Sudakov–Fernique Gaussian comparison inequality to analyze the complexity of critical points that satisfy strong convexity and stability conditions within their local neighborhoods.

f-DP has recently been proposed as a generalization of differential privacy allowing a lossless analysis of composition, post-processing, and privacy amplification via subsampling. In the setting of f-DP, we propose the concept of a canonical noise distribution (CND), the first mechanism designed for an arbitrary f-DP guarantee. The notion of CND captures whether an additive privacy mechanism perfectly matches the privacy guarantee of a given f. We prove that a CND always exists, and give a construction that produces a CND for any f. We show that private hypothesis tests are intimately related to CNDs, allowing for the release of private p-values at no additional privacy cost, as well as the construction of uniformly most powerful (UMP) tests for binary data, within the general f-DP framework.

We apply our techniques to the problem of difference-of-proportions testing, and construct a UMP unbiased (UMPU) “semiprivate” test which upper bounds the performance of any f-DP test. Using this as a benchmark, we propose a private test based on the inversion of characteristic functions, which allows for optimal inference on the two population parameters and is nearly as powerful as the semiprivate UMPU. When specialized to the case of $(\mathit{ϵ},0)$-DP, we show empirically that our proposed test is more powerful than any $(\mathit{ϵ}/\sqrt{2})$-DP test and has more accurate type I errors than the classic normal approximation test.

Modeling univariate block maxima by the generalized extreme value distribution constitutes one of the most widely applied approaches in extreme value statistics. It has recently been found that, for an underlying stationary time series, respective estimators may be improved by calculating block maxima in an overlapping way. A proof of concept is provided that the latter finding also holds in situations that involve certain piecewise stationarities. A weak convergence result for an empirical process of central interest is provided, and further details are examplarily worked out for the probability weighted moment estimator. Irrespective of the serial dependence, the asymptotic estimation variance is shown to be smaller for the new estimator. In extensive simulation experiments, the finite-sample variance was typically found to be smaller as well, while the bias stays approximately the same. The results are illustrated by Monte Carlo simulation experiments and are applied to a common situation involving temperature extremes in a changing climate.

The problem of learning graphons has attracted considerable attention across several scientific communities, with significant progress over the recent years in sparser regimes. Yet, the current techniques still require diverging degrees in order to succeed with efficient algorithms in the challenging cases where the local structure of the graph is homogeneous. This paper provides an efficient algorithm to learn graphons in the constant expected degree regime. The algorithm is shown to succeed in estimating the rank-k projection of a graphon in the ${\mathit{L}}_2$ metric if the top k eigenvalues of the graphon satisfy a generalized Kesten–Stigum condition.

We analyze the singular values of a large $\mathit{p}\times \mathit{n}$ data matrix ${\mathbf{X}}_{\mathit{n}}=({\mathbf{x}}_{\mathit{n}1},\dots ,{\mathbf{x}}_{\mathit{n}\mathit{n}})$, where the columns $\{{\mathbf{x}}_{\mathit{n}\mathit{j}}\}$ are independent p-dimensional vectors, possibly with different distributions. Assuming that the covariance matrices ${\mathrm{\Sigma }}_{\mathit{n}\mathit{j}}=Cov({\mathbf{x}}_{\mathit{n}\mathit{j}})$ of the column vectors can be asymptotically simultaneously diagonalized, with appropriately converging spectra, we establish a limiting spectral distribution (LSD) for the singular values of ${\mathbf{X}}_{\mathit{n}}$ when both dimensions p and n grow to infinity in comparable magnitudes. Our matrix model goes beyond and includes many different types of sample covariance matrices in existing work, such as weighted sample covariance matrices, Gram matrices, and sample covariance matrices of a linear time series model. Furthermore, three applications of our general approach are developed. First, we obtain the existence and uniqueness of the LSD for realized covariance matrices of a multi-dimensional diffusion process with anisotropic time-varying co-volatility. Second, we derive the LSD for singular values of data matrices from a recent matrix-valued auto-regressive model. Finally, we also obtain the LSD for singular values of data matrices from a generalized finite mixture model.

Structural matrix-variate observations routinely arise in diverse fields such as multilayer network analysis and brain image clustering. While data of this type have been extensively investigated with fruitful outcomes being delivered, the fundamental questions like its statistical optimality and computational limit are largely under-explored. In this paper, we propose a low-rank Gaussian mixture model (LrMM) assuming each matrix-valued observation has a planted low-rank structure. Minimax lower bounds for estimating the underlying low-rank matrix are established allowing a whole range of sample sizes and signal strength. Under a minimal condition on signal strength, referred to as the information-theoretical limit or statistical limit, we prove the minimax optimality of a maximum likelihood estimator which, in general, is computationally infeasible. If the signal is stronger than a certain threshold, called the computational limit, we design a computationally fast estimator based on spectral aggregation and demonstrate its minimax optimality. Moreover, when the signal strength is smaller than the computational limit, we provide evidences based on the low-degree likelihood ratio framework to claim that no polynomial-time algorithm can consistently recover the underlying low-rank matrix. Our results reveal multiple phase transitions in the minimax error rates and the statistical-to-computational gap. Numerical experiments confirm our theoretical findings. We further showcase the merit of our spectral aggregation method on the worldwide food trading dataset.

We consider the estimation of two-sample integral functionals, of the type that occur naturally, for example, when the object of interest is a divergence between unknown probability densities. Our first main result is that, in wide generality, a weighted nearest neighbour estimator is efficient, in the sense of achieving the local asymptotic minimax lower bound. Moreover, we also prove a corresponding central limit theorem, which facilitates the construction of asymptotically valid confidence intervals for the functional, having asymptotically minimal width. One interesting consequence of our results is the discovery that, for certain functionals, the worst-case performance of our estimator may improve on that of the natural ‘oracle’ estimator, which itself can be optimal in the related problem where the data consist of the values of the unknown densities at the observations.

We study the properties of nonparametric least squares regression using deep neural networks. We derive nonasymptotic upper bounds for the excess risk of the empirical risk minimizer of feedforward deep neural regression. Our error bounds achieve minimax optimal rate and improve over the existing ones in the sense that they depend polynomially on the dimension of the predictor, instead of exponentially on dimension. We show that the neural regression estimator can circumvent the curse of dimensionality under the assumption that the predictor is supported on an approximate low-dimensional manifold or a set with low Minkowski dimension. We also establish the optimal convergence rate under the exact manifold support assumption. We investigate how the prediction error of the neural regression estimator depends on the structure of neural networks and propose a notion of network relative efficiency between two types of neural networks, which provides a quantitative measure for evaluating the relative merits of different network structures. To establish these results, we derive a novel approximation error bound for the Hölder smooth functions using ReLU activated neural networks, which may be of independent interest. Our results are derived under weaker assumptions on the data distribution and the neural network structure than those in the existing literature.

It is of soaring demand to develop statistical analysis tools that are robust against contamination as well as preserving individual data owners’ privacy. In spite of the fact that both topics host a rich body of literature, to the best of our knowledge, we are the first to systematically study the connections between the optimality under Huber’s contamination model and the local differential privacy (LDP) constraints.

In this paper, we start with a general minimax lower bound result, which disentangles the costs of being robust against Huber contamination and preserving LDP. We further study four concrete examples: a two-point testing problem, a potentially diverging mean estimation problem, a nonparametric density estimation problem and a univariate median estimation problem. For each problem, we demonstrate procedures that are optimal in the presence of both contamination and LDP constraints, comment on the connections with the state-of-the-art methods that are only studied under either contamination or privacy constraints, and unveil the connections between robustness and LDP via partially answering whether LDP procedures are robust and whether robust procedures can be efficiently privatised. Overall, our work showcases a promising prospect of joint study for robustness and local differential privacy.

We study the covariate shift problem in the context of nonparametric regression over a reproducing kernel Hilbert space (RKHS). We focus on two natural families of covariate shift problems defined using the likelihood ratios between the source and target distributions. When the likelihood ratios are uniformly bounded, we prove that the kernel ridge regression (KRR) estimator with a carefully chosen regularization parameter is minimax rate-optimal (up to a log factor) for a large family of RKHSs with regular kernel eigenvalues. Interestingly, KRR does not require full knowledge of the likelihood ratio apart from an upper bound on it. In striking contrast to the standard statistical setting without covariate shift, we also demonstrate that a naïve estimator, which minimizes the empirical risk over the function class, is strictly suboptimal under covariate shift as compared to KRR. We then address the larger class of covariate shift problems where likelihood ratio is possibly unbounded yet has a finite second moment. Here, we propose a reweighted KRR estimator that weights samples based on a careful truncation of the likelihood ratios. Again, we are able to show that this estimator is minimax optimal, up to logarithmic factors.

We consider the task of estimating a conditional density using i.i.d. samples from a joint distribution, which is a fundamental problem with applications in both classification and uncertainty quantification for regression. For joint density estimation, minimax rates have been characterized for general density classes in terms of uniform (metric) entropy, a well-studied notion of statistical capacity. When applying these results to conditional density estimation, the use of uniform entropy—which is infinite when the covariate space is unbounded and suffers from the curse of dimensionality—can lead to suboptimal rates. Consequently, minimax rates for conditional density estimation cannot be characterized using these classical results.

We resolve this problem for well-specified models, obtaining matching (within logarithmic factors) upper and lower bounds on the minimax Kullback–Leibler risk in terms of the empirical Hellinger entropy for the conditional density class. The use of empirical entropy allows us to appeal to concentration arguments based on local Rademacher complexity, which—in contrast to uniform entropy—leads to matching rates for large, potentially nonparametric classes and captures the correct dependence on the complexity of the covariate space. Our results require only that the conditional densities are bounded above, and do not require that they are bounded below or otherwise satisfy any tail conditions.

We study the rank of the instantaneous or spot covariance matrix ${\mathrm{\Sigma }}_{\mathit{X}}(\mathit{t})$ of a multidimensional process $\mathit{X}(\mathit{t})$. Given high-frequency observations $\mathit{X}(\mathit{i}/\mathit{n})$, $\mathit{i}=0,\dots ,\mathit{n}$, we test the null hypothesis $rank({\mathrm{\Sigma }}_{\mathit{X}}(\mathit{t}))\le \mathit{r}$ for all t against local alternatives where the average $(\mathit{r}+1)$st eigenvalue is larger than some signal detection rate ${\mathit{v}}_{\mathit{n}}$.

A major problem is that the inherent averaging in local covariance statistics produces a bias that distorts the rank statistics. We show that the bias depends on the regularity and spectral gap of ${\mathrm{\Sigma }}_{\mathit{X}}(\mathit{t})$. We establish explicit matrix perturbation and concentration results that provide nonasymptotic uniform critical values and optimal signal detection rates ${\mathit{v}}_{\mathit{n}}$. This leads to a rank estimation method via sequential testing. For a class of stochastic volatility models, we determine data-driven critical values via normed p-variations of estimated local covariance matrices. The methods are illustrated by simulations and an application to high-frequency data of U.S. government bonds.

Conformal prediction is a popular, modern technique for providing valid predictive inference for arbitrary machine learning models. Its validity relies on the assumptions of exchangeability of the data, and symmetry of the given model fitting algorithm as a function of the data. However, exchangeability is often violated when predictive models are deployed in practice. For example, if the data distribution drifts over time, then the data points are no longer exchangeable; moreover, in such settings, we might want to use a nonsymmetric algorithm that treats recent observations as more relevant. This paper generalizes conformal prediction to deal with both aspects: we employ weighted quantiles to introduce robustness against distribution drift, and design a new randomization technique to allow for algorithms that do not treat data points symmetrically. Our new methods are provably robust, with substantially less loss of coverage when exchangeability is violated due to distribution drift or other challenging features of real data, while also achieving the same coverage guarantees as existing conformal prediction methods if the data points are in fact exchangeable. We demonstrate the practical utility of these new tools with simulations and real-data experiments on electricity and election forecasting.

High-dimensional time series data appear in many scientific areas in the current data-rich environment. Analysis of such data poses new challenges to data analysts because of not only the complicated dynamic dependence between the series, but also the existence of aberrant observations, such as missing values, contaminated observations, and heavy-tailed distributions. For high-dimensional vector autoregressive (VAR) models, we introduce a unified estimation procedure that is robust to model misspecification, heavy-tailed noise contamination, and conditional heteroscedasticity. The proposed methodology enjoys both statistical optimality and computational efficiency, and can handle many popular high-dimensional models, such as sparse, reduced-rank, banded, and network-structured VAR models. With proper regularization and data truncation, the estimation convergence rates are shown to be almost optimal in the minimax sense under a bounded $(2+2\mathit{ϵ})$th moment condition. When $\mathit{ϵ}\ge 1$, the rates of convergence match those obtained under the sub-Gaussian assumption. Consistency of the proposed estimators is also established for some $\mathit{ϵ}\in (0,1)$, with minimax optimal convergence rates associated with ϵ. The efficacy of the proposed estimation methods is demonstrated by simulation and a U.S. macroeconomic example.