## The Annals of Applied Statistics

- Ann. Appl. Stat.
- Volume 13, Number 1 (2019), 420-443.

### The role of mastery learning in an intelligent tutoring system: Principal stratification on a latent variable

Adam C. Sales and John F. Pane

#### Abstract

Students in Algebra I classrooms typically learn at different rates and struggle at different points in the curriculum—a common challenge for math teachers. Cognitive Tutor Algebra I (CTA1), an educational computer program, addresses such student heterogeneity via what they term “mastery learning,” where students progress from one section of the curriculum to the next by demonstrating appropriate “mastery” at each stage. However, when students are unable to master a section’s skills even after trying many problems, they are automatically promoted to the next section anyway. Does promotion without mastery impair the program’s effectiveness?

At least in certain domains, CTA1 was recently shown to improve student learning on average in a randomized effectiveness study. This paper uses student log data from that study in a continuous principal stratification model to estimate the relationship between students’ potential mastery and the CTA1 treatment effect. In contrast to extant principal stratification applications, a student’s propensity to master worked sections here is never directly observed. Consequently we embed an item-response model, which measures students’ potential mastery, within the larger principal stratification model. We find that the tutor may, in fact, be *more* effective for students who are more frequently promoted (despite unsuccessfully completing sections of the material). However, since these students are distinctive in their educational strength (as well as in other respects), it remains unclear whether this enhanced effectiveness can be directly attributed to aspects of the mastery learning program.

#### Article information

**Source**

Ann. Appl. Stat., Volume 13, Number 1 (2019), 420-443.

**Dates**

Received: July 2017

Revised: June 2018

First available in Project Euclid: 10 April 2019

**Permanent link to this document**

https://projecteuclid.org/euclid.aoas/1554861655

**Digital Object Identifier**

doi:10.1214/18-AOAS1196

**Mathematical Reviews number (MathSciNet)**

MR3937435

**Zentralblatt MATH identifier**

07057434

**Keywords**

Causal inference principal stratification item response theory latent variables Bayesian educational technology

#### Citation

Sales, Adam C.; Pane, John F. The role of mastery learning in an intelligent tutoring system: Principal stratification on a latent variable. Ann. Appl. Stat. 13 (2019), no. 1, 420--443. doi:10.1214/18-AOAS1196. https://projecteuclid.org/euclid.aoas/1554861655

#### Supplemental materials

- Supplement to “The role of mastery learning in intelligent tutoring systems: Principal stratification on a latent variable”. We provide modeling details, Stan code, and an extensive set of model goodness-of-fit and sensitivity analyses and plots.Digital Object Identifier: doi:10.1214/18-AOAS1196SUPPSupplemental files are immediately available to subscribers. Non-subscribers gain access to supplemental files with the purchase of the article.