Date of Award

4-2018

Degree Name

Doctor of Philosophy

Department

Educational Leadership, Research and Technology

First Advisor

Dr. Brooks Applegate

Second Advisor

Dr. Kim Nimon

Third Advisor

Dr. Jessaca Spybrook

Fourth Advisor

Dr. David Hartman

Keywords

Measurement in variance, response shift bias, self-reported change

Abstract

Traditional pre-test (TpT)/post-test (PT) and retrospective pre-test (RpT)/post-test (PT) designs are used to collect data on self-reported measures to assess the magnitude of change that occurs from interventions. If measurement invariance does not exist across the measurement occasions within these research designs, it is inappropriate to compare mean group differences that result from the intervention and derive inferences about change. The theory of response-shift suggests a subject’s understanding of a subject matter at TpT may not be the same as their understanding of a subject matter at PT, and that as a result, the construct measured at these measurement occasions may not be the same, or may not have the same structural components (factor loadings and scale). RpT/PT research designs have been suggested as an alternative to TpT/PT research designs to control for response-shift bias. Unfortunately, measurement invariance is rarely investigated in either of these research designs, it is merely assumed. Given this, it is important to understand the extent to which both TpT/PT and RpT/PT research designs demonstrate measurement invariance in various contexts and the impact measurement invariance may have on effect size estimates. The principle aim of this dissertation study was to examine the theory of response-shift bias by testing if an instrument administered in a TpT/PT design evidences the same structural meaning as the same instrument administered in a RpT/PT design, and the extent to which the observed (raw) scores obtained in the context of these designs are the same as the latent means. This study examines self-reported change of communication reticence using a longitudinal measurement invariance model. In this study, the measurement occasions in the TpT/PT research design are invariant to the level of strong. This indicates that there is not a shift in the understanding of the construct in either research design. However, the measurement occasions in the RpT/PT research design were only found to be invariant to the level of weak. A partial invariance analysis of the RpT/PT model revealed that when the intercept of an indicator associated with knowledge was unconstrained, strong invariance was achieved.

In the TpT/PT research design an effect size analysis revealed that the raw scores underestimate the effect, however the difference between the raw and latent means is not statistically significantly different. This study would suggest, that in this context, the TpT/PT and RpT/PT research designs resulted in the explication of the same construct at pre-test and post-test and given this both research designs could be used to derive valid inferences about the constructs being measured. Because measurement invariance in the RpT/PT was not found beyond the level of weak, only the TpT/PT research design could be used to derive valid inferences about the magnitude of pre-test and post-test scores, unless the constraint for on the intercept for the knowledge indicator was relaxed. Hence in this context, only the TpT/PT research design should be used to derive valid inferences about the magnitude of change that resulted from the intervention.

Access Setting

Dissertation-Open Access

Download

Share

COinS