An Empirical Study of Cross-site Variation in Treatment Effects of Educational Interventions

Date of Award

6-2024

Degree Name

Doctor of Philosophy

Department

Educational Leadership, Research and Technology

First Advisor

Jessaca K. Spybrook, Ph.D.

Second Advisor

Ya Zhang, Ph.D.

Third Advisor

Nianbo Dong, Ph.D.

Keywords

Cross-site impact variation, generalizability, Institute of Education Sciences (IES), k-12 educational interventions, multisite cluster randomized trials, statistical power

Abstract

In the past two decades, we have seen a dramatic increase in the number of randomized controlled trials (RCTs) to assess the efficacy of educational interventions. In the field of education, it is common to randomize intact groups of students such as entire schools or individual teachers and their classrooms to conditions (treatment vs. control). This leads to a hierarchical, or nested, research design known as cluster randomized trial (CRT) design. Many impact studies of educational interventions use a specific type of CRT, known as multisite cluster randomized trial (MSCRT) design. In a MSCRT, random assignment of clusters of individuals to conditions occurs within sites, which can be natural units (e.g., schools and districts) or be formed and embedded in the design (e.g., pairs and groups of schools). Most MSCRTs in education to date focus on estimating the average treatment effect (ATE). Although the ATE is of primary interest, in a MSCRT it is also important to estimate and report the cross-site impact variation as reporting the ATE by itself can mask important differences across sites.

This study provides empirical estimates of the variation in the treatment effects across sites in a set of completed large-scale MSCRTs funded by the Institute of Education Sciences (IES), the leading federal funder of RCTs in education. The MSCRTs assess the efficacy of various educational interventions on K-12 student achievement in mathematics, reading, and behavior outcomes. I empirically estimate the cross-site impact variation using a three-level fixed intercepts and random treatment coefficient (FIRC) model. In addition, I examine the relationships between select study/intervention characteristics (intervention intensity, intervention type (computer-based vs. non-computer-based), outcome domain (mathematics vs. reading), and targeted grade level (elementary vs. other grade levels)) and the magnitude of cross-site impact variation using a regression model. I also demonstrate the ways education researchers can use the empirical estimates of cross-site impact variation provided by this study to conduct power analyses and planning for future MSCRTs.

The findings from this study suggest the magnitude of cross-site variation in treatment effects ranges from 0.09𝜎 to 0.37𝜎. The treatment effects of computer-based interventions and interventions targeting elementary school students are found to vary more substantially across sites than non-computer-based interventions and interventions targeting other grade levels. While there is a limit to the comprehensiveness and generalizability of the set of estimates of cross-site impact variation provided by this study due to data constraints and model selection, I believe it represents the best information that is currently available about the magnitude of cross-site impact variation in MSCRTs in education. The results can be leveraged by education researchers to design future MSCRTs with greater precision.

Access Setting

Dissertation-Abstract Only

Restricted to Campus until

6-1-2034

This document is currently not available here.

Share

COinS