Estimating and Applying Parameters Necessary to Plan Cluster Randomized Trials (CRTs) and Multisite Cluster Randomized Trials (MSCRTs)

Date of Award

6-2024

Degree Name

Doctor of Philosophy

Department

Educational Leadership, Research and Technology

First Advisor

Jessaca Spybrook, Ph.D.

Second Advisor

Eric Hedberg, Ph.D.

Third Advisor

Ya Zhang, Ph.D.

Keywords

Cluster-randomized trials (CRTs), design parameters, evaluation, intraclass correlations (ICCs), multisite cluster-randomized trials (MSCRTs), power analysis

Abstract

Cluster randomized trials (CRTs) are commonly used to study the effectiveness of educational interventions. During the design phase of a study, it is critical for researchers to ensure their studies are adequately powered to detect meaningful treatment effects, including both main and moderator effects. Designing CRTs with adequate power to detect main and moderator effects requires accurate estimates of design parameters. This research aims to advance the literature on design parameters for power analyses, specifically focusing on empirical estimates of intraclass correlations (ICCs). The work consists of three research papers that examine the role of including the teacher level in CRTs and practical considerations on how teacher- and school-level ICCs vary with school and district characteristics.

Paper 1 demonstrates the extent to which including teacher-level ICCs in multilevel designs matters. In this paper, power for main and moderator effects are calculated using ICCs and R2 values from three-level models with students nested within teachers within schools in Michigan, Kentucky, North Carolina, and Maryland. These design parameters are compared to the common rule of thumb where one-third of the original school-level ICC from a two-level model with students nested in schools is allocated to the teacher level in a three-level CRT. The findings show that, when using empirical versus the rule-of-thumb ICCs for different effects of interest, there are differences in the minimum detectable effect size or the number of clusters needed to detect a certain effect. Furthermore, using empirical teacher-level ICCs may improve the accuracy of power analyses for three-level designs in which students are nested in teachers nested in schools.

Paper 2 aims to provide evidence for how teacher ICCs for student science, math, and reading scores systematically vary by school characteristics. In this paper, using administrative state datasets, unconditional teacher ICCs within schools in Michigan, Kentucky, and North Carolina for grades 3–8 are computed. In addition, data on school characteristics is collected from the Common Core of Data (CCD). Using random-effects meta-analysis, this paper systematically investigates how teacher ICCs relate to school characteristics. The results show that teacher ICCs vary by school characteristics, grades, and subjects, across states, and that states exhibit different patterns in terms of teacher ICCs.

Paper 3 aims to provide evidence for how teacher and school ICCs for student achievement in science, math, and reading systematically vary by district characteristics. The unconditional teacher and school ICCs for districts in Michigan and Kentucky are computed, and district characteristics are collected from the CCD. Multivariate meta-regression is employed to examine whether ICCs vary with district characteristics. The findings show significant variability in ICCs among teachers and schools across districts within a state. They also support the hypothesis that teacher and school ICCs systematically vary across districts, and they differ by grade across districts, and in some cases, by subject.

Access Setting

Dissertation-Abstract Only

Restricted to Campus until

6-1-2026

This document is currently not available here.

Share

COinS