Date of Award

4-2018

Degree Name

Doctor of Philosophy

Department

Educational Leadership, Research and Technology

First Advisor

Dr. Jessaca Spybrook

Second Advisor

Dr. Gary Miron

Third Advisor

Dr. Jianping Shen

Keywords

Cluster randomized trials (CRT), Power Analyses, Asian countries, Intra-class correlation (ICC), covariates, effect sizes (MDES)

Abstract

A cluster-randomized trial (CRT) is one type of randomized controlled trial (RCT) that is often more feasible for evaluating the effectiveness of an educational intervention due to the multi-level structure of the data (e.g., students nested in schools). In order for a CRT to generate rigorous evidence, it is essential that is it adequately powered to detect an effect of a given size. As such, it is common for researchers to conduct a priori power analyses to determine how large a sample the CRTs will require to detect an effect of a given magnitude. Previous work has shown that two design parameters, the intra-class correlation coefficient (ICC) and the percent of variance explained by covariate(s) R2, are the keys to conducting accurate power analyses in CRT studies and are used at the outset to plan robust CRTs. Good estimates of these two design parameters are critical. However, literatures indicate that the ICCs and R2, are context specific. The absence of empirical estimates of design parameters from Asian contexts leads to increased potential for inaccurate estimates of design parameters and hence misleading power analyses. Therefore, this study attempts to improve the design of CRTs of education interventions by improving the accuracy of the parameter estimates used for a priori power analyses in Asia based on the data from Programme for International Student Assessment (PISA) 2012. Two main parameters, ICCs and R2, were estimated in this study.

The sample was from 1,999 schools and consisted of 65,219 15-16-year-old students in twelve Asian countries/regions. Students’ reading, mathematics and science achievement were used as outcomes. For each outcome, a series of two-level hierarchical linear models where students are nested within schools were built to estimate to the ICCs and R2. The index of economic, social, and cultural status (ESCS), gender, school type, and school size were selected as covariates to estimate the proportion of variance explained by covariates R2.

Major findings of the study strengthen the argument that the ICC and R2 are context specific. Findings demonstrate large variability exists in ICC estimates across Asian countries. Further, the mean of the ICCs was greater than in the United States, potentially a result of the different ways schools are organized in Asia. Findings also indicate that the R2 varied by countries, and the most influential covariates in Asian countries might be different in the United States given the differences in social and culture backgrounds. Power calculations, in the form of the minimum detectable effect size, were also calculated given the empirical estimates of the ICC and R2. In general, more schools are needed to detect the same MDES in Asian contexts than in the United States. More empirical work in Asia is recommended for further studies, including estimating the explanatory power of meaningful covariates in a specific country, examining the ICCs and R2 using another database, including different outcomes (e.g., teacher), or including another level (e.g., students nested within teachers nested within schools).

Access Setting

Dissertation-Open Access

Download

Share

COinS