The Annals of Applied Statistics
- Ann. Appl. Stat.
- Volume 13, Number 1 (2019), 420-443.
The role of mastery learning in an intelligent tutoring system: Principal stratification on a latent variable
Students in Algebra I classrooms typically learn at different rates and struggle at different points in the curriculum—a common challenge for math teachers. Cognitive Tutor Algebra I (CTA1), an educational computer program, addresses such student heterogeneity via what they term “mastery learning,” where students progress from one section of the curriculum to the next by demonstrating appropriate “mastery” at each stage. However, when students are unable to master a section’s skills even after trying many problems, they are automatically promoted to the next section anyway. Does promotion without mastery impair the program’s effectiveness?
At least in certain domains, CTA1 was recently shown to improve student learning on average in a randomized effectiveness study. This paper uses student log data from that study in a continuous principal stratification model to estimate the relationship between students’ potential mastery and the CTA1 treatment effect. In contrast to extant principal stratification applications, a student’s propensity to master worked sections here is never directly observed. Consequently we embed an item-response model, which measures students’ potential mastery, within the larger principal stratification model. We find that the tutor may, in fact, be more effective for students who are more frequently promoted (despite unsuccessfully completing sections of the material). However, since these students are distinctive in their educational strength (as well as in other respects), it remains unclear whether this enhanced effectiveness can be directly attributed to aspects of the mastery learning program.
Ann. Appl. Stat., Volume 13, Number 1 (2019), 420-443.
Received: July 2017
Revised: June 2018
First available in Project Euclid: 10 April 2019
Permanent link to this document
Digital Object Identifier
Mathematical Reviews number (MathSciNet)
Zentralblatt MATH identifier
Sales, Adam C.; Pane, John F. The role of mastery learning in an intelligent tutoring system: Principal stratification on a latent variable. Ann. Appl. Stat. 13 (2019), no. 1, 420--443. doi:10.1214/18-AOAS1196. https://projecteuclid.org/euclid.aoas/1554861655
- Supplement to “The role of mastery learning in intelligent tutoring systems: Principal stratification on a latent variable”. We provide modeling details, Stan code, and an extensive set of model goodness-of-fit and sensitivity analyses and plots.