The Role of Evaluation Terms in the Quality of Final Evaluation Reports in International Development Cooperation: A Multi-Phase Mixed Methods Study

Date of Award

6-2024

Degree Name

Doctor of Philosophy

Department

Evaluation

First Advisor

Gary Miron, Ph.D.

Second Advisor

Brooks Applegate, Ph.D.

Third Advisor

Ramona Lewis, Ph.D.

Fourth Advisor

Ada Ocampo, M.A.

Keywords

Evaluation terms of reference, final evaluation report, international development cooperation, multi-phase mixed methods

Abstract

In international development cooperation there is constant discussion about how to improve evaluation practices. Too little is known about where and which factors influence the quality of evaluation reports. Arguably, one influential factor is the quality of the terms of reference (TORs), which, in many sectors, are referred to as Requests for Proposals (RFPs). This study examines the role of the quality of TORs in positively affecting the design and conduct of evaluations, as well as the overall quality of final evaluation reports. This study also examines other factors that may contribute to the quality of final evaluation reports.

The researcher used a multiphase mixed-method design and mixed purposeful sampling strategies. The United Nations Children’s Fund (UNICEF) was chosen as the case study. Key source data came from final evaluation reports and TORs from Global Evaluation Reports Oversight System (GEROS) meta-evaluations. Primary and secondary data were derived from multiple sources and methods. The primary data was comprised of interviews with evaluators, evaluation managers, and program specialists, as well as surveys of evaluators. Secondary data included evaluation reports, their corresponding TORs, quality feedback, and evaluation framework documents.

Findings from this study suggest that the TOR quality influences the efficiency and effectiveness of the evaluation process. The relative quality of TORs could incentivize or demotivate prospective contractors to participate in the bidding process. Related to this, well prepared TORs can reduce guessing about the evaluation parameters, which results in better proposals submitted by evaluators that more fully address the needs of end users. The results suggest that the quality of TORs does affect the quality of final evaluation reports. The findings indicate that other key factors that positively influence the quality of final evaluation reports include the evaluator’s qualifications and the oversight provided by an evaluation manager or program officer.

The findings have a broad range of potential implications including improvement of the use of peer-reviewed quality criteria from a TOR quality scoring rubric. The findings could also inform revisions of current TOR quality standards. Although this study focused on the international development cooperation sector, the findings are relevant for other sectors and should have relevance for both large and small organizations. A few of the key recommendations address means to enhance practices for assessing TOR quality and more research to better understand the factors that support or hinder evaluations. Key contributions of this study include the development and refining of tools and instruments that can help practitioners prepare higher quality TORs. High quality TORs can then help organizations as they work to recruit, select, and oversee evaluators. Improving these practices will lead to higher quality final reports that are more likely to be used and more likely to inform beneficial decisions about programs.

Access Setting

Dissertation-Abstract Only

Restricted to Campus until

6-1-2026

This document is currently not available here.

Share

COinS