Date of Award

4-2024

Degree Name

Doctor of Philosophy

Department

Evaluation

First Advisor

Brooks Applegate, Ph.D.

Second Advisor

Ya Zhang, Ph.D.

Third Advisor

Alyssa Counsell, Ph. D.

Keywords

Confirmatory factor analysis, equivalence test, hypothesis test, measurement invariance, partial measurement invariance, simulation

Abstract

This dissertation compares the performance of equivalence test (EQT) and null hypothesis test (NHT) procedures for identifying invariant and noninvariant factor loadings under a range of experimental manipulations. EQT is the statistically appropriate approach when the research goal is to find evidence of group similarity rather than group difference; despite this, the conventional approach to measurement invariance analysis relies upon NHT. EQT has proved effective for invariance detection using global model-data fit statistics in simulated and real-world data (Counsell et al., 2020) but its use in partial measurement invariance (PMI) analysis for evaluation of factor loading differences between groups has not been previously investigated.

A Monte Carlo simulation is used to investigate test performance under experimental manipulations of power, model complexity, degree of noninvariance, and choice of EQT bound. Jung and Yoon (2017) use a similar approach to demonstrate that the direct comparison of factor loading differences across groups using NHT confidence intervals effectively detect population patterns of invariant and noninvariant loadings. I compare the performance of NHT to EQT using both conventional benchmarks of factor loading equivalence inferred from the work of Counsell and Cribbie (2015), Nye, Bradburn, Olenick, Bialko, and Drasgow (2019), and Shi et al. (2019) as well as EQT bounds selected post hoc to maximize recovery of the population pattern of invariance or partial invariance. I also investigate the likelihood of false invariance detections for truly noninvariant loadings and false noninvariance detections for truly invariant loadings over a range of EQT bounds, sample sizes, and magnitudes of noninvariance.

The results of this simulation indicate that EQT can detect population patterns of invariance or partial invariance, but requires high power (i.e., large sample sizes and strong communalities) as well as optimal selection of the EQT bound. In experimental conditions where these requirements are met, EQT recovers the correct population pattern of invariant and noninvariant factor loadings in more simulation replicates than the NHT approach (Jung & Yoon, 2017). Inspection of individual loading detection rates reveals that false noninvariance errors occur most frequently when power is low, while false invariance errors are common whenever the true magnitude of noninvariance is small and the selected EQT bound is larger than the loading difference. In the EQT framing, false invariance errors in the presence of minor noninvariance might also be interpreted as accurate detections of practically equivalent loadings.

These findings support the use of EQT for PMI analysis and reinforce previous findings that EQT is an effective but power-intensive procedure in the multivariate context (Alter & Counsell, 2023). The choice of EQT bound has implications for both the power of the test and the consequences of accepting loading differences falling within the selected EQT bound as invariant. Researchers using EQT to detect PMI should collect large samples in order to maximize the likelihood of detection, and should consider how much misspecification their selected EQT bounds might tolerate. Future development of PMI simulation tools could support the selection and justification of EQT bounds in applied settings.

Access Setting

Dissertation-Open Access

Download

Share

COinS