## The Annals of Statistics

### Divide and conquer in nonstandard problems and the super-efficiency phenomenon

#### Abstract

We study how the divide and conquer principle works in non-standard problems where rates of convergence are typically slower than $\sqrt{n}$ and limit distributions are non-Gaussian, and provide a detailed treatment for a variety of important and well-studied problems involving nonparametric estimation of a monotone function. We find that for a fixed model, the pooled estimator, obtained by averaging nonstandard estimates across mutually exclusive subsamples, outperforms the nonstandard monotonicity-constrained (global) estimator based on the entire sample in the sense of pointwise estimation of the function. We also show that, under appropriate conditions, if the number of subsamples is allowed to increase at appropriate rates, the pooled estimator is asymptotically normally distributed with a variance that is empirically estimable from the subsample-level estimates. Further, in the context of monotone regression, we show that this gain in efficiency under a fixed model comes at a price—the pooled estimator’s performance, in a uniform sense (maximal risk) over a class of models worsens as the number of subsamples increases, leading to a version of the super-efficiency phenomenon. In the process, we develop analytical results for the order of the bias in isotonic regression, which are of independent interest.

#### Article information

Source
Ann. Statist., Volume 47, Number 2 (2019), 720-757.

Dates
Revised: May 2017
First available in Project Euclid: 11 January 2019

https://projecteuclid.org/euclid.aos/1547197236

Digital Object Identifier
doi:10.1214/17-AOS1633

Mathematical Reviews number (MathSciNet)
MR3909948

Zentralblatt MATH identifier
07033149

Subjects
Primary: 62G20: Asymptotic properties 62G08: Nonparametric regression
Secondary: 62F30: Inference under constraints

#### Citation

Banerjee, Moulinath; Durot, Cécile; Sen, Bodhisattva. Divide and conquer in nonstandard problems and the super-efficiency phenomenon. Ann. Statist. 47 (2019), no. 2, 720--757. doi:10.1214/17-AOS1633. https://projecteuclid.org/euclid.aos/1547197236

#### References

• [1] Banerjee, M. (2005). Likelihood ratio tests under local and fixed alternatives in monotone function problems. Scand. J. Stat. 32 507–525.
• [2] Banerjee, M. (2007). Likelihood based inference for monotone response models. Ann. Statist. 35 931–956.
• [3] Banerjee, M. (2008). Estimating monotone, unimodal and U-shaped failure rates using asymptotic pivots. Statist. Sinica 467–492.
• [4] Banerjee, M., Durot, C. and Sen, B. (2019). Supplement to “Divide and conquer in nonstandard problems and the super-efficiency phenomenon.” DOI:10.1214/17-AOS1633SUPP.
• [5] Banerjee, M. and McKeague, I. W. (2007). Confidence sets for split points in decision trees. Ann. Statist. 35 543–574.
• [6] Banerjee, M. and Wellner, J. A. (2005). Confidence intervals for current status data. Scand. J. Stat. 32 405–424.
• [7] Brown, L. D., Low, M. G. and Zhao, L. H. (1997). Superefficiency in nonparametric function estimation. Ann. Statist. 25 2607–2625.
• [8] Brunk, H. D. (1955). Maximum likelihood estimates of monotone parameters. Ann. Math. Stat. 26 607–616.
• [9] Brunk, H. D. (1970). Estimation of isotonic regression. In Nonparametric Techniques in Statistical Inference 177–197. Cambridge Univ. Press, London.
• [10] Chernoff, H. (1964). Estimation of the mode. Ann. Inst. Statist. Math. 16 31–41.
• [11] Durot, C. (2002). Sharp asymptotics for isotonic regression. Probab. Theory Related Fields 122 222–240.
• [12] Durot, C. (2008). Monotone nonparametric regression with random design. Math. Methods Statist. 17 327–341.
• [13] Durot, C. and Lopuhaä, H. P. (2014). A Kiefer–Wolfowitz type of result in a general setting, with an application to smooth monotone estimation. Electron. J. Stat. 8 2479–2513.
• [14] Durot, C. and Thiébot, K. (2006). Bootstrapping the shorth for regression. ESAIM Probab. Stat. 10 216–235.
• [15] Grenander, U. (1956). On the theory of mortality measurement, part II. Skand. Aktuarietidskr. 39 125–153.
• [16] Groeneboom, P. and Jongbloed, G. (2014). Nonparametric Estimation Under Shape Constraints: Estimators, Algorithms and Asymptotics. Cambridge Series in Statistical and Probabilistic Mathematics 38. Cambridge Univ. Press, Cambridge.
• [17] Groeneboom, P. and Wellner, J. A. (1992). Information Bounds and Nonparametric Maximum Likelihood Estimation. DMV Seminar 19. Birkhäuser, Basel.
• [18] Huang, J. and Wellner, J. A. (1995). Estimation of a monotone density or monotone hazard under random censoring. Scand. J. Stat. 3–33.
• [19] Kim, J. and Pollard, D. (1990). Cube root asymptotics. Ann. Statist. 18 191–219.
• [20] Li, R., Lin, Dennis, K. J. and Li, B. (2013). Statistical inference in massive data sets. Appl. Stoch. Models Bus. Ind. 29 399–409.
• [21] Manski, C. F. (1975). Maximum score estimation of the stochastic utility model of choice. J. Econometrics 3 205–228.
• [22] Massart, P. (1990). The tight constant in the Dvoretzky–Kiefer–Wolfowitz inequality. Ann. Probab. 18 1269–1283.
• [23] Prakasa Rao, B. L. S. (1969). Estimation of a unimodal density. Sankhyā, Ser. A 31 23–36.
• [24] Revuz, D. and Yor, M. (2013). Continuous Martingales and Brownian Motion. Grundlehren der mathematischen Wissenschaften 293. Springer, Berlin.
• [25] Robertson, T., Wright, F. T. and Dykstra, R. L. (1988). Order Restricted Statistical Inference. Wiley Series in Probability and Mathematical Statistics: Probability and Mathematical Statistics. Wiley, Chichester.
• [26] Rousseeuw, P. J. (1984). Least median of squares regression. J. Amer. Statist. Assoc. 79 871–880.
• [27] Shi, C., Lu, W. and Song, R. (2018). A massive data framework for M-estimators with cubic-rate. J. Amer. Statist. Assoc. To appear.
• [28] Tsybakov, A. B. (2009). Introduction to Nonparametric Estimation. Springer Series in Statistics. Springer, New York. Revised and extended from the 2004 French original; translated by Vladimir Zaiats.
• [29] Wright, F. T. (1981). The asymptotic behavior of monotone regression estimates. Ann. Statist. 9 443–448.
• [30] Zhang, Y., Duchi, J. and Wainwright, M. (2013). Divide and conquer kernel ridge regression. In Conference on Learning Theory 592–617.
• [31] Zhao, T., Cheng, G. and Liu, H. (2016). A partially linear framework for massive heterogeneous data. Ann. Statist. 44 1400–1437.

#### Supplemental materials

• Supplement to “Divide and conquer in nonstandard problems and the super-efficiency phenomenon”. The supplementary material contains elaborate proofs of some of the more technical results used in the main body of the paper.