## The Annals of Statistics

### Regularized estimation in sparse high-dimensional time series models

#### Abstract

Many scientific and economic problems involve the analysis of high-dimensional time series datasets. However, theoretical studies in high-dimensional statistics to date rely primarily on the assumption of independent and identically distributed (i.i.d.) samples. In this work, we focus on stable Gaussian processes and investigate the theoretical properties of $\ell_{1}$-regularized estimates in two important statistical problems in the context of high-dimensional time series: (a) stochastic regression with serially correlated errors and (b) transition matrix estimation in vector autoregressive (VAR) models. We derive nonasymptotic upper bounds on the estimation errors of the regularized estimates and establish that consistent estimation under high-dimensional scaling is possible via $\ell_{1}$-regularization for a large class of stable processes under sparsity constraints. A key technical contribution of the work is to introduce a measure of stability for stationary processes using their spectral properties that provides insight into the effect of dependence on the accuracy of the regularized estimates. With this proposed stability measure, we establish some useful deviation bounds for dependent data, which can be used to study several important regularized estimates in a time series setting.

#### Article information

Source
Ann. Statist., Volume 43, Number 4 (2015), 1535-1567.

Dates
Revised: January 2015
First available in Project Euclid: 17 June 2015

https://projecteuclid.org/euclid.aos/1434546214

Digital Object Identifier
doi:10.1214/15-AOS1315

Mathematical Reviews number (MathSciNet)
MR3357870

Zentralblatt MATH identifier
1317.62067

#### Citation

Basu, Sumanta; Michailidis, George. Regularized estimation in sparse high-dimensional time series models. Ann. Statist. 43 (2015), no. 4, 1535--1567. doi:10.1214/15-AOS1315. https://projecteuclid.org/euclid.aos/1434546214

#### References

• Bańbura, M., Giannone, D. and Reichlin, L. (2010). Large Bayesian vector auto regressions. J. Appl. Econometrics 25 71–92.
• Basu, S. (2014). Modeling and estimation of high-dimensional vector autoregressions. Ph.D. thesis, Univ. Michigan, Ann Arbor, MI.
• Basu, S. and Michailidis, G. (2015). Supplement to “Regularized estimation in sparse high-dimensional time series models.” DOI:10.1214/15-AOS1315SUPP.
• Bernanke, B. S., Boivin, J. and Eliasz, P. (2005). Measuring the effects of monetary policy: A factor-augmented vector autoregressive (FAVAR) approach. Q. J. Econ. 120 387–422.
• Bickel, P. J. and Levina, E. (2008). Covariance regularization by thresholding. Ann. Statist. 36 2577–2604.
• Bickel, P. J., Ritov, Y. and Tsybakov, A. B. (2009). Simultaneous analysis of lasso and Dantzig selector. Ann. Statist. 37 1705–1732.
• Chen, X., Xu, M. and Wu, W. B. (2013). Covariance and precision matrix estimation for high-dimensional time series. Ann. Statist. 41 2994–3021.
• Chudik, A. and Pesaran, M. H. (2011). Infinite-dimensional VARs and factor models. J. Econometrics 163 4–22.
• Davis, R. A., Zang, P. and Zheng, T. (2012). Sparse vector autoregressive modeling. Preprint. Available at arXiv:1207.0520.
• De Mol, C., Giannone, D. and Reichlin, L. (2008). Forecasting using a large number of predictors: Is Bayesian shrinkage a valid alternative to principal components? J. Econometrics 146 318–328.
• Dobriban, E. and Fan, J. (2013). Regularity properties of high-dimensional covariate matrices. Preprint. Available at arXiv:1305.5198.
• Fan, J. and Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. J. Amer. Statist. Assoc. 96 1348–1360.
• Fan, Y. and Lv, J. (2013). Asymptotic equivalence of regularization methods in thresholded parameter space. J. Amer. Statist. Assoc. 108 1044–1061.
• Fan, J., Lv, J. and Qi, L. (2011). Sparse high-dimensional models in economics. Annual Review of Economics 3 291–317.
• Friston, K. (2009). Causal modelling and brain connectivity in functional magnetic resonance imaging. PLoS Biol. 7 e1000033.
• Giraitis, L., Koul, H. L. and Surgailis, D. (2012). Large Sample Inference for Long Memory Processes. Imperial College Press, London.
• Grenander, U. and Szegö, G. (1958). Toeplitz Forms and Their Applications. Univ. California Press, Berkeley.
• Hamilton, J. D. (1994). Time Series Analysis. Princeton Univ. Press, Princeton, NJ.
• Han, F. and Liu, H. (2013). Transition matrix estimation in high dimensional time series. Proceedings of the 30th International Conference on Machine Learning (ICML-13) 28 172–180.
• Kumar, P. R. and Varaiya, P. (1986). Stochastic Systems: Estimation, Identification and Adaptive Control. Prentice Hall, New York.
• Liebscher, E. (2005). Towards a unified approach for proving geometric ergodicity and mixing properties of nonlinear autoregressive processes. J. Time Series Anal. 26 669–689.
• Loh, P.-L. and Wainwright, M. J. (2012). High-dimensional regression with noisy and missing data: Provable guarantees with nonconvexity. Ann. Statist. 40 1637–1664.
• Loh, P.-L. and Wainwright, M. J. (2013). Regularized M-estimators with nonconvexity: Statistical and algorithmic theory for local optima. Preprint. Available at arXiv:1305.2436.
• Lütkepohl, H. (2005). New Introduction to Multiple Time Series Analysis. Springer, Berlin.
• Michailidis, G. and d’Alché-Buc, F. (2013). Autoregressive models for gene regulatory network inference: Sparsity, stability and causality issues. Math. Biosci. 246 326–334.
• Negahban, S. and Wainwright, M. J. (2011). Estimation of (near) low-rank matrices with noise and high-dimensional scaling. Ann. Statist. 39 1069–1097.
• Negahban, S. N., Ravikumar, P., Wainwright, M. J. and Yu, B. (2012). A unified framework for high-dimensional analysis of $M$-estimators with decomposable regularizers. Statist. Sci. 27 538–557.
• Parter, S. V. (1961). Extreme eigenvalues of Toeplitz forms and applications to elliptic difference equations. Trans. Amer. Math. Soc. 99 153–192.
• Priestley, M. B. (1981). Spectral Analysis and Time Series. Vol. 2. Multivariate Series, Prediction and Control, Probability and Mathematical Statistics. Academic Press, London.
• Raskutti, G., Wainwright, M. J. and Yu, B. (2010). Restricted eigenvalue properties for correlated Gaussian designs. J. Mach. Learn. Res. 11 2241–2259.
• Rosenblatt, M. (1985). Stationary Sequences and Random Fields. Springer, Boston, MA.
• Rudelson, M. and Vershynin, R. (2013). Hanson–Wright inequality and sub-Gaussian concentration. Electron. Commun. Probab. 18 no. 82, 9.
• Rudelson, M. and Zhou, S. (2013). Reconstruction from anisotropic random measurements. IEEE Trans. Inform. Theory 59 3434–3447.
• Seth, A. K., Chorley, P. and Barnett, L. C. (2013). Granger causality analysis of fMRI BOLD signals is invariant to hemodynamic convolution but not downsampling. NeuroImage 65 540–555.
• Shojaie, A. and Michailidis, G. (2010). Discovering graphical Granger causality using the truncating lasso penalty. Bioinformatics 26 i517–i523.
• Sims, C. A. (1980). Macroeconomics and reality. Econometrica 48 1–48.
• Smith, S. M. (2012). The future of FMRI connectivity. NeuroImage 62 1257–1266.
• Song, S. and Bickel, P. J. (2011). Large vector auto regressions. Preprint. Available at arXiv:1106.3915v1.
• Stock, J. H. and Watson, M. W. (2005). Implications of dynamic factor models for VAR analysis. Working Paper No. 11467, National Bureau of Economic Research, Cambridge, MA.
• van de Geer, S. A. and Bühlmann, P. (2009). On the conditions used to prove oracle results for the Lasso. Electron. J. Stat. 3 1360–1392.
• van de Geer, S., Bühlmann, P. and Zhou, S. (2011). The adaptive and the thresholded Lasso for potentially misspecified models (and a lower bound for the Lasso). Electron. J. Stat. 5 688–749.
• Vershynin, R. (2010). Introduction to the non-asymptotic analysis of random matrices. Preprint. Available at arXiv:1011.3027.
• Wu, W. B. (2005). Nonlinear system theory: Another look at dependence. Proc. Natl. Acad. Sci. USA 102 14150–14154 (electronic).
• Wu, W.-B. and Wu, Y. N. (2014). High-dimensional linear models with dependent observations. Preprint.
• Xiao, H. and Wu, W. B. (2012). Covariance matrix estimation for stationary time series. Ann. Statist. 40 466–493.
• Zhang, C.-H. (2010). Nearly unbiased variable selection under minimax concave penalty. Ann. Statist. 38 894–942.
• Zhou, S. (2010). Thresholded Lasso for high dimensional variable selection and statistical estimation. Technical Report 511, Dept. Statistics, Univ. Michigan, Ann Arbor, MI. Available at arXiv:1002.1583.

#### Supplemental materials

• Supplement to “Regularized estimation in sparse high-dimensional time series models”. For the sake of brevity, we moved the appendices containing many of the technical proofs and detailed discussions to the supplementary document [Basu and Michailidis (2015)].