## Statistical Science

- Statist. Sci.
- Volume 30, Number 3 (2015), 352-371.

### A Closer Look at Testing the “No-Treatment-Effect” Hypothesis in a Comparative Experiment

#### Abstract

Standard tests of the “no-treatment-effect” hypothesis for a comparative experiment include permutation tests, the Wilcoxon rank sum test, two-sample $t$ tests, and Fisher-type randomization tests. Practitioners are aware that these procedures test different no-effect hypotheses and are based on different modeling assumptions. However, this awareness is not always, or even usually, accompanied by a clear understanding or appreciation of these differences. Borrowing from the rich literatures on causality and finite-population sampling theory, this paper develops a modeling framework that affords answers to several important questions, including: exactly what hypothesis is being tested, what model assumptions are being made, and are there other, perhaps better, approaches to testing a no-effect hypothesis? The framework lends itself to clear descriptions of three main inference approaches: process-based, randomization-based, and selection-based. It also promotes careful consideration of model assumptions and targets of inference, and highlights the importance of randomization. Along the way, Fisher-type randomization tests are compared to permutation tests and a less well-known Neyman-type randomization test. A simulation study compares the operating characteristics of the Neyman-type randomization test to those of the other more familiar tests.

#### Article information

**Source**

Statist. Sci., Volume 30, Number 3 (2015), 352-371.

**Dates**

First available in Project Euclid: 10 August 2015

**Permanent link to this document**

https://projecteuclid.org/euclid.ss/1439220717

**Digital Object Identifier**

doi:10.1214/15-STS513

**Mathematical Reviews number (MathSciNet)**

MR3383885

**Zentralblatt MATH identifier**

1332.62065

**Keywords**

Causal effects completely randomized design finite-population sampling theory Fisher vs. Neyman Fisher’s exact test Horvitz–Thompson estimator nonmeasurable probability sample permutation tests potential variables process-based inference randomization-based inference randomization tests selection-based inference

#### Citation

Lang, Joseph B. A Closer Look at Testing the “No-Treatment-Effect” Hypothesis in a Comparative Experiment. Statist. Sci. 30 (2015), no. 3, 352--371. doi:10.1214/15-STS513. https://projecteuclid.org/euclid.ss/1439220717