Date of Award

12-2022

Degree Name

Doctor of Philosophy

Department

Evaluation

First Advisor

Jessaca Spybrook, Ph.D. (Co-chair)

Second Advisor

Brooks Applegate, Ph.D. (Co-chair)

Third Advisor

Susan Kowalski, Ph.D.

Abstract

Over the last four decades, meta-analysis has proven to be a vital analysis strategy in educational research for synthesizing research findings from different studies. When synthesizing studies in a meta-analysis, it is common to assume that the true underlying effect varies from study to study, as studies will differ in design, participants, interventions, or sample size that can lead to heterogeneity in their underlying effects. The magnitude of this heterogeneity between studies can be quantified as τ2 in a random-effects meta-analysis. Estimating the between-study heterogeneity (τ2) becomes an important part of random effects meta-analysis reporting, since this quantity plays a vital role in understanding how the effect sizes in the studies are dispersed around the mean effect size. Unfortunately for practitioners, there are a multitude of τ2 estimators that have been derived. Moreover, there are few studies that have compared the different forms of τ2 estimators, thus understanding their differences and similarities is needed and the conditions under which observed τ2 values might vary among the different estimators.

The purpose of this quantitative study is to investigate the performance of five τ2 estimators commonly used in education-related random-effects meta-analysis. These five estimators are DerSimonian and Laird (DL), Two-step DerSimonian and Laird (DL2), Maximum Likelihood (ML), Restricted Maximum Likelihood (REML), and Sidik and Jonkman (SJ). Estimator performance was operationalized in this dissertation as: (a) the magnitude of τ2, (b) bias, (c) mean squared error (MSE), and (d) the coverage of 95% confidence interval (CI). A Monte Carlo simulation study compared the performance of the five heterogeneity estimators varying the following experimental conditions: (1) number of studies included in a meta-analysis, (2) study sample size, and (3) level of heterogeneity among the enrolled studies. The data analysis was conducted using multivariate analysis of variance (MANOVA) and multivariate logistic regression, followed by post-hoc analyses controlling for a family-wise type I error of .05.

Findings in this study suggest that estimates of heterogeneity magnitude, bias, MSE and coverage derived from different estimators can be notably different as a function of experimental conditions. These findings have important implications for educational researchers who wish to report the between-study heterogeneity among studies via a τ2 estimator in random-effects metaanalysis. Recommendations for researchers on the selection of heterogeneity estimators and areas for future research about between-study heterogeneity estimation are discussed.

Access Setting

Dissertation-Open Access

Download

Share

COinS