Date of Award

12-2008

Degree Name

Doctor of Philosophy

Department

Evaluation

First Advisor

Dr. Paul Clements

Second Advisor

Dr. Michael Scriven

Third Advisor

Dr. Liliana Rodriguez-Campos

Abstract

This document describes the metaevaluation of aid evaluation reports generated in a single fiscal year (FY 2004). Sample reports (n=102) were chosen basically by stratified random sampling from the pool of evaluation reports (N=1034). The pool consists of ex post evaluation reports (and some extensive midterm and termination evaluation reports) obtained from Web sites.

The Key Aid Evaluation Checklist (KAEC) was developed based on the well-known Key Evaluation Checklist (KEC) (Scriven, 2005, 2006). A supplemental checklist, Subdimensional Checklist (SDC) was also applied for the metaevaluation.

A key conclusion of this metaevaluation is this: Quality of the current aid evaluation reports in general is regarded as “satisfactory” but far from “excellent,” although some are marked as high quality; i.e., they successfully examine a set of subevaluations and offer overall evaluative conclusions (such as a single grade), while others are of very poor quality (they do not determine overall or at least some dimensional evaluative conclusions but just present analytic conclusions or fact-findings).

Some serious flaws exist in the current aid evaluation practice, as follows: (1) Primary operation of evaluation (subevaluation of each dimension) is relatively well conducted, but secondary operation (integrating them into a single judgment) is very weak or is simply not conducted. (2) Value identification/verification, which is usually conducted by stakeholder participation or political inclusion, is very weak. (3) Cost-efficiency subevaluation (either cost-benefit subevaluation or cost-effectiveness subevaluation) is generally weak or simply not conducted, especially if outcome-based evaluation is conducted. On the other hand, the evaluation report is generally regarded as good quality if it conducts serious and extensive cost-efficiency subevaluation. (4) Metaevaluation (evaluation of the evaluation report by the evaluator him/herself) has not been accepted widely as an essential part of aid evaluation.

Based on the identified flaws, several recommendations are proposed.

This study examined whether an issue was treated or not at each checkpoint on the checklist, not whether the content was correctly addressed. Also, this metaevaluation does not examine the relationship between the collected reports and the overall development portfolio of individual donors. These will require further research.

Access Setting

Dissertation-Open Access

Download

Share

COinS