The Annals of Statistics

Nonlinear sequential designs for logistic item response theory models with applications to computerized adaptive tests

Hua-Hua Chang and Zhiliang Ying

Full-text: Open access

Abstract

Computerized adaptive testing is becoming increasingly popular due to advancement of modern computer technology. It differs from the conventional standardized testing in that the selection of test items is tailored to individual examinee’s ability level. Arising from this selection strategy is a nonlinear sequential design problem. We study, in this paper, the sequential design problem in the context of the logistic item response theory models. We show that the adaptive design obtained by maximizing the item information leads to a consistent and asymptotically normal ability estimator in the case of the Rasch model. Modifications to the maximum information approach are proposed for the two- and three-parameter logistic models. Similar asymptotic properties are established for the modified designs and the resulting estimator. Examples are also given in the case of the two-parameter logistic model to show that without such modifications, the maximum likelihood estimator of the ability parameter may not be consistent.

Article information

Source
Ann. Statist., Volume 37, Number 3 (2009), 1466-1488.

Dates
First available in Project Euclid: 10 April 2009

Permanent link to this document
https://projecteuclid.org/euclid.aos/1239369028

Digital Object Identifier
doi:10.1214/08-AOS614

Mathematical Reviews number (MathSciNet)
MR2509080

Zentralblatt MATH identifier
1160.62073

Subjects
Primary: 62L05: Sequential design
Secondary: 62P15: Applications to psychology

Keywords
Sequential design computerized adaptive testing item response theory Rasch model logistic models Fisher information maximum likelihood recursion martingale local convergence consistency asymptotic normality

Citation

Chang, Hua-Hua; Ying, Zhiliang. Nonlinear sequential designs for logistic item response theory models with applications to computerized adaptive tests. Ann. Statist. 37 (2009), no. 3, 1466--1488. doi:10.1214/08-AOS614. https://projecteuclid.org/euclid.aos/1239369028


Export citation

References

  • Chang, H. and Stout, W. F. (1993). The asymptotic posterior normality of the latent trait in an IRT model. Psychometrika 58 37–52.
  • Chang, H. and Ying, Z. (1996). A global information approach to computerized adaptive testing. Appl. Psychol. Meas. 20 213–229.
  • Chang, H. and Ying, Z. (1999). a-stratified multistage computerized adaptive testing. Appl. Psychol. Meas. 23 211–222.
  • Chang, H., Qian, J. and Ying, Z. (2001). a-stratified multistage CAT with b-blocking. Appl. Psychol. Meas. 25 333–341.
  • Cheng, Y., Chang, H. and Yi, Q. (2007). Two-phase item selection procedure for flexible content balancing in CAT. Appl. Psychol. Meas. 31 467–482.
  • Chow, Y. S. (1965). Local convergence of martingales and the law of large numbers. Ann. Math. Statist. 36 552–558.
  • Chow, Y. S. and Teicher, H. (1988). Probability Theory, 2nd ed. Springer, New York.
  • Hambleton, R. and Swaminathan, H. (1985). Item Response Theory: Principles and Applications. Kluwer Nijhoff, Boston.
  • Lord, M. F. (1971). Robbins–Monro procedures fora testing. Educ. Psychol. Meas. 31 3–31.
  • Lord, M. F. (1980). Applications of Item Response Theory to Practical Testing Problems. Erlbaum, Hillsdale, NJ.
  • Mislevy, R. and Chang, H. (2000). Does adaptive testing violate local independence? Psychometrika 65 149–165.
  • Mislevy, R. and Wu, P. K. (1996). Missing responses and Baysian IRT ability estimation: Omits, choice, time limits, and adaptive testing. Research Report RR-96-30-ONR. Educational Testing Service, Princeton, NJ.
  • Owen, Z. J. (1975). A Bayesian sequential procedure for quantal response in the context of adaptive mental testing. J. Amer. Statist. Assoc. 70 351–356.
  • Pollard, D. (1984). Convergence of Stochastic Processes. Springer, New York.
  • Rasch, G. (1960). Probabilistic Models for Some Intelligence and Attainment Tests. The Danish Institute of Educational Research, Copenhagen. Expanded edition. Univ. Chicago Press, Chicago (1980).
  • Robbins, H. and Monro, S. (1951). A stochastic approximation method. Ann. Math. Statist. 22 400–407.
  • Samejima, F. (1973). A comment on Birnbaum’s three-parameter logistic model in the latent trait theory. Psychometrika 38 221–233.
  • van der Linden, W. J. (1998). Bayesian item-selection criteria for adaptive testing. Psychometrika 63 201–216.
  • van der Linden, W. J. and Chang, H. (2003). Implementing content constraints in alpha-stratified adaptive testing using a shadow test approach. Appl. Psychol. Meas. 27 107–120.
  • Wainer, H. (2000). Computerized Adaptive Testing: A Primer, 2nd ed. Erlbaum, Hillsdale, NJ.
  • Weiss, D. J. (1976). Adaptive testing research in Minnesota: Overview, recent results, and future directions. In Proceedings of the First Conference on Computerized Adaptive Testing (C. L. Clark, ed.) 24–35. United States Civil Service Commission, Washington, DC.
  • Wu, C. F. J. (1985). Efficient sequential designs with binary data. J. Amer. Statist. Assoc. 80 974–984.
  • Yi, Q. and Chang, H. (2003). a-stratified multistage CAT design with content-blocking. British J. Math. Statist. Psych. 56 359–378.
  • Ying, Z. and Wu, C. F. J. (1997). An asymptotic theory of sequential designs based on maximum likelihood recursions. Statist. Sinica 7 75–91.