Electronic Journal of Statistics

On the asymptotic efficiency of selection procedures for independent Gaussian populations

Royi Jacobovic and Or Zuk

Full-text: Open access


The field of discrete event simulation and optimization techniques motivates researchers to adjust classic ranking and selection (R&S) procedures to the settings where the number of populations is large. We use insights from extreme value theory in order to reveal the asymptotic properties of R&S procedures. Namely, we generalize the asymptotic result of Robbins and Siegmund regarding selection from independent Gaussian populations with known constant variance by their means to the case of selecting a subset of varying size out of a given set of populations. In addition, we revisit the problem of selecting the population with the highest mean among independent Gaussian populations with unknown and possibly different variances. Particularly, we derive the relative asymptotic efficiency of Dudewicz and Dalal’s and Rinott’s procedures, showing that the former can be asymptotically superior by a multiplicative factor which is larger than one, but this factor may be reduced by proper choice of parameters. We also use our asymptotic results to suggest that the sample size in the first stage of the two procedures should be logarithmic in the number of populations.

Article information

Electron. J. Statist., Volume 11, Number 2 (2017), 5375-5405.

Received: July 2017
First available in Project Euclid: 28 December 2017

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62F07: Ranking and selection
Secondary: 62L99: None of the above, but in this section

Selection procedures asymptotic statistics extreme value theory discrete events simulation

Creative Commons Attribution 4.0 International License.


Jacobovic, Royi; Zuk, Or. On the asymptotic efficiency of selection procedures for independent Gaussian populations. Electron. J. Statist. 11 (2017), no. 2, 5375--5405. doi:10.1214/17-EJS1375. https://projecteuclid.org/euclid.ejs/1514430422

Export citation


  • [1] Ahmed, M. and Alkhamis, T. (2002). Simulation-based optimization using simulated annealing with ranking and selection., Computers & Operations Research, 29(4):387–402.
  • [2] Alzer, H. (2001). Mean-value inequalities for the polygamma functions., Aequationes Mathematicae, 61(1–2):151–161.
  • [3] Banks, J. (1998)., Handbook of simulation: principles, methodology, advances, applications, and practice. John Wiley & Sons.
  • [4] Bechhofer, R. (1954). A single-sample multiple decision procedure for ranking means of normal populations with known variances., The Annals of Mathematical Statistics, 25(1):16–39.
  • [5] Bechhofer, R., Goldsman, D., Dunnett, C., and Hartmann, M. (1990). A comparison of the performances of procedures for selecting the normal population having the largest mean when the populations have a common unknown variance., Communications in Statistics-Simulation and Computation, 19(3):971–1006.
  • [6] Billingsley, P. (2013)., Convergence of probability measures. John Wiley & Sons, New York.
  • [7] Boesel, J., Nelson, B., and Kim, S. (2003). Using ranking and selection to “clean up” after simulation optimization., Operations Research, 51(5):814–825.
  • [8] Branke, J., Chick, S., and Schmidt, C. (2007). Selecting a selection procedure., Management Science, 53(12):1916–1932.
  • [9] Chen, C.-h. and Lee, L. H. (2011)., Stochastic simulation optimization: an optimal computing budget allocation, volume 1. World scientific.
  • [10] Chen, C. H., Lin, J., Yücesan, E., and Chick, S. E. (2000). Simulation budget allocation for further enhancing the efficiency of ordinal optimization., Discrete Event Dynamic Systems, 10(3):251–270.
  • [11] Chick, S. E. (2006a). Bayesian ideas and discrete event simulation: why, what and how. In, Simulation Conference, 2006. WSC 06. Proceedings of the Winter, pages 96–106. IEEE.
  • [12] Chick, S. E. (2006b). Subjective probability and bayesian methodology., Handbooks in Operations Research and Management Science, 13:225–257.
  • [13] Chow, Y. and Robbins, H. (1965). On the asymptotic theory of fixed-width sequential confidence intervals for the mean., The Annals of Mathematical Statistics, 36(2):457–462.
  • [14] Dembo, A., Karlin, S., and Zeitouni, O. (1994). Limit distribution of maximal non-aligned two-sequence segmental score., The Annals of Probability, pages 2022–2039.
  • [15] Dudewicz, E. (1971). Non-existence of a single-sample selection procedure whose p (cs) is independent of the variances., South African Statistical Journal, 5(1):37–39.
  • [16] Dudewicz, E. and Dalal, S. (1975). Allocation of observations in ranking and selection with unequal variances., Sankhyā: The Indian Journal of Statistics, Series B, pages 28–78.
  • [17] Ein-Dor, L., Zuk, O., and Domany, E. (2006). Thousands of samples are needed to generate a robust gene list for predicting outcome in cancer., Proceedings of the National Academy of Sciences, 103(15):5923–5928.
  • [18] Ferguson, T. (1996)., A course in large sample theory. Chapman & Hall London.
  • [19] Forgionne, G. (1983). Corporate management science activities: an update., Interfaces, 13(3):20–23.
  • [20] Frazier, P. I., K. A. M. (2011). Guessing preferences: A new approach to multi- attribute ranking and selection. In, In Simulation Conference (WSC), Proceedings of the 2011 Winter, pages 4319–4331.
  • [21] Frazier, P. I. (2014). A fully sequential elimination procedure for indifference-zone ranking and selection with tight bounds on probability of correct selection., Operations Research, 62(4):926–942.
  • [22] Fu, M. (1994). Optimization via simulation: A review., Annals of operations research, 53(1):199–247.
  • [23] Fu, M. C. et al. (2015)., Handbook of simulation optimization, volume 216. Springer.
  • [24] Fu, M. C., Glover, F. W., and April, J. (2005). Simulation optimization: a review, new developments, and applications. In, Simulation conference, 2005 proceedings of the winter, pages 13–pp. IEEE.
  • [25] Galambos, J. (1979)., The asymptotic theory of extreme order statistics. John Wiley & Sons, New York.
  • [26] Ghosh, B. (1975). On the distribution of the difference of two $t$-variables., Journal of the American Statistical Association, 70(350):463–467.
  • [27] Gibbons, J., Olkin, I., and Sobel, M. (1999)., Selecting and ordering populations: a new statistical methodology. SIAM.
  • [28] Goldsman, D. and Nelson, B. (1998). Comparing systems via simulation., Handbook of simulation: Principles, methodology, advances, applications, and practice, pages 273–306.
  • [29] Grigelionis, B. (2013)., Student’s $t$-distribution and related stochastic processes. Springer.
  • [30] Gupta, S. and Panchapakesan, S. (2002)., Multiple decision procedures: theory and methodology of selecting and ranking populations. SIAM.
  • [31] Harpell, J., Lane, M., and Mansour, A. (1989). Operations research in practice: A longitudinal study., Interfaces, 19(3):65–74.
  • [32] Jacobson, L. and Schruben, S. (1989). Techniques for optimization response optimization., Operations Research Letters, 8:1–9.
  • [33] Kim, S.-H. and Nelson, B. L. (2001). A fully sequential procedure for indifference-zone selection in simulation., ACM Transactions on Modeling and Computer Simulation (TOMACS), 11(3):251–273.
  • [34] Kim, S. H. and Nelson, B. L. (2006a). On the asymptotic validity of fully sequential selection procedures for steady-state simulation., Operations Research, 54(3):475–488.
  • [35] Kim, S. H. and Nelson, B. L. (2006b). Selecting the best system., Handbooks in operations research and management science, 13:501–534.
  • [36] Kleijnen, J. P. (2008)., Design and analysis of simulation experiments, volume 20. Springer.
  • [37] Lane, M., Mansour, A., and Harpell, J. L. (1993). Operations research techniques: A longitudinal update 1973–1988., Interfaces, 23(2):63–68.
  • [38] Leadbetter, M., Lindgren, G., and Rootzén, H. (2012)., Extremes and related properties of random sequences and processes. Springer Science & Business Media.
  • [39] Mukhopadhyay, N. (1979). Some comments on two-stage selection procedures., Communications in Statistics-Theory and Methods, 8(7):671–683.
  • [40] Mukhopadhyay, N. and Solanky, T. (1994)., Multistage selection and ranking procedures: second order asymptotics, volume 142. CRC Press.
  • [41] Nelson, B., Swann, J., Goldsman, D., and Song, W. (2001). Simple procedures for selecting the best simulated system when the number of alternatives is large., Operations Research, 49(6):950–963.
  • [42] Rinott, Y. (1978). On two-stage selection procedures and related probability-inequalities., Communications in Statistics-Theory and methods, 7(8):799–811.
  • [43] Robbins, H. and Siegmund, D. (1967)., Iterated logarithm inequalities and related statistical procedures. Department of Statistics, Stanford University.
  • [44] Shannon, R., Long, S., and Buckles, B. (1980). Operation research methodologies in industrial engineering: A survey., AIIE Transactions, 12(4):364–367.
  • [45] Stein, C. (1945). A two-sample test for a linear hypothesis whose power is independent of the variance., The Annals of Mathematical Statistics, 16(3):243–258.
  • [46] Wang, H. and Kim, S. (2013). Reducing the conservativeness of fully sequential indifference-zone procedures., IEEE Transactions on Automatic Control, 58(6):1613–1619.
  • [47] Zuk, O., Ein-Dor, L., and Domany, E. (2007). Ranking under uncertainty. In, Proceedings of the Twenty-Third Conference on Uncertainty in Artificial Intelligence, pages 466–473. AUAI Press.