Estimating high-dimensional covariance and precision matrices under general missing dependence

Abstract

A sample covariance matrix S of completely observed data is the key statistic in a large variety of multivariate statistical procedures, such as structured covariance/precision matrix estimation, principal component analysis, and testing of equality of mean vectors. However, when the data are partially observed, the sample covariance matrix from the available data is biased and does not provide valid multivariate procedures. To correct the bias, a simple adjustment method called inverse probability weighting (IPW) has been used in previous research, yielding the IPW estimator. The estimator can play the role of S in the missing data context, thus replacing S in off-the-shelf multivariate procedures such as the graphical lasso algorithm. However, theoretical properties (e.g. concentration) of the IPW estimator have been only established in earlier work under very simple missing structures; every variable of each sample is independently subject to missingness with equal probability. We investigate the deviation of the IPW estimator when observations are partially observed under general missing dependency. We prove the optimal convergence rate $O_p(\sqrt{logp∕n})$ of the IPW estimator based on the element-wise maximum norm, even when two unrealistic assumptions (known mean and/or missing probabilities) frequently assumed to be known in the past work are relaxed. The optimal rate is especially crucial in estimating a precision matrix, because of the “meta-theorem” [26] that claims the rate of the IPW estimator governs that of the resulting precision matrix estimator. In the simulation study, we discuss one of practically important issues, non-positive semi-definiteness of the IPW estimator, and compare the estimator with imputation methods.