What's the First Step in an Evaluation, Really?

Let's play a game. Place the following early-stage evaluation steps in sequential order.

Select methods / Conduct stakeholder consultations / Draft inception report / Define evaluation questions / Confirm scope and budget

We're not huge gamblers, but if we were, we'd wager a fair amount that you ordered them something like this:

1. Confirm scope and budget
2. Conduct stakeholder consultations
3. Define evaluation questions
4. Select methods
5. Draft inception report

For many, this is a complete and reasonable sequence. The problem, of course, is that several steps are missing, and they happen to be the most important ones.

The Step Nobody Built

What we consider the first step in an evaluation is a factor of our positionality. For funders, the first step is determining what the evaluation is allowed to cost. For organizations, it’s determining the purpose and scope of the evaluation. For evaluators, it's validating what the pre-determined scope, purpose, and budget mean for what questions get answered and how.

Few evaluations have a step for determining who makes decisions about the evaluation itself.

This decision-making defaults to whoever is already in the room, typically the funder and the organization, and the process moves forward with the support of an evaluator. The community affected by the program may be consulted, but by then, decision rights over the evaluation itself were calcified in the hands of the organization and funder.

What We’re Losing

Most people who have experienced an evaluation have seen the consequences of forgoing shared decision rights with communities: 

• Evaluations answering questions that don’t matter as much as people hoped. While the questions reflect the existing needs of funders and organizations, the information needs of the people who live the program results go unaddressed. Meanwhile, results mean less to the organization as their priorities shift in response to the times (or new funder priorities). 

• Findings have poor reach. Organizations receive a final report, ship it off to a funder, and maybe publish it online. Only people proximate to the organization receive the new knowledge, and in worst-case scenarios, there is not even meaningful communication back to the people who directly contributed to the evaluation (survey respondents, data collectors, etc.).

• Recommendations don't get implemented. The program team goes back to other parts of implementing the program. Without an internal mechanism to operationalize evaluation findings, formalized channels for communicating progress, or external pressure to develop either, there is no one to hold the organization accountable for implementing recommendations and reporting their progress. 

The field has been good at identifying the problem. What needs to happen now is a radical shift in our thinking about just outcomes for evaluations. That starts with answering difficult questions about governance. 

Asking Governance Questions

Before deciding the scope or the questions to be answered, funders and organizations commissioning an evaluation must start explicitly asking and rethinking their implicit answers to these questions:

• Who holds decision rights over what this evaluation studies? 
• Who has power over how the data is interpreted? 
• Who determines what the findings are allowed to say? 
• Who determines how the knowledge will be held and carried forward?
• Who owns this evaluation?

On most evaluations, the answers to these questions are assumed to be the organization or the funder. Yet, we’ve been made acutely aware that funders can vanish overnight. When they last, their priorities shift. Organizations, meanwhile, change their priorities to attract funding or close altogether when funding dries up. The only constant is the communities left behind, the same community that most evaluations extract from.

Managing Governance in Practice

Advice for Funders: Whether community ownership has what it needs to thrive starts with funders. You control how money can be spent, timelines, and often approve evaluators. Funders establish the conditions that affect all downstream decisions. Funders should adopt a posture that shares power from the start, and encourage their grantees to do the same. This requires work well beyond the evaluation, but as far as evaluations are concerned, this includes encouraging (if not requiring) organizations to share decision-making power with the communities they serve. 

At the same time, select evaluators/evaluation approaches that pay explicit attention to power, with justice as an intended outcome. “Participatory,” “equitable,” and “community-centered” approaches aren’t enough. 

This may “sound political”; however, evaluation is political, because absolutely everything surrounding organizations and programs is political. Keen attention should be paid not only to avoiding perpetuating historical wrongs, but also to evaluation as a tool for addressing those wrongs.

Advice for Organizations: Even organizations with progressive funders fall into some of the traps we’ve outlined. Recognize that while trained evaluators bring a necessary expertise to evaluations, so do the people who live the data we work with. Wherever possible, design RFPs that explicitly ask evaluators to facilitate power sharing and ensure resources are conducive to it.

At the same time, organizations must push back against extraction when it is encouraged by their funder, while presenting the benefits of sharing power with communities. The benefits are undeniable, now more than ever. 

Advice for Evaluators: Most professional evaluation associations (e.g., AEA, EEA, AfREA, APEA) name interpersonal skills as a core competency for evaluators. To make ownership possible, evaluators must be skilled in facilitation, managing conflict, and be willing to take on roles APEA names as Guardians and Change Agents. Our role in community-owned evaluation extends beyond providing technical expertise to being advocates, negotiators, and confidants who facilitate justice throughout the evaluation. 

Evaluators are not powerless actors in this system. Outside of evaluations, we must advocate directly to donors and organizations to make space for community ownership. During evaluations, we should find and take advantage of any opportunity to make shared ownership meaningful, while actively pushing for shared decision-making. 

See how we structure ownership in practice.

The Revised Sequence

In an ideal world, below is how an evaluation begins. Of course, in practice, some of these steps (particularly budgeting) are determined before an evaluator enters the room. That doesn't make governance impossible. It’s just a known constraint that needs to be discussed openly with community decision makers who should have the opportunity to challenge and offer alternatives.

1. Identify and mobilize the Affected Community (AC)
2. Determine evaluation governance
3. Clarify constraints, opportunities, and safeguards
4. Define the evaluation purpose
5. Define an appropriate scope and budget 
6. Develop evaluation questions and methods
7. Draft inception documentation 

It's a longer list than the previous one, but it's also a list that produces evaluations people trust, findings that get used, and knowledge that communities can use to hold people in power accountable for upholding their end of the bargain.

If you're planning an evaluation and haven't had the governance conversation yet, start there. Before the RFP goes out. Before questions are drafted. Before anyone is hired.

Want support thinking through what that looks like in your context, we'd love to hear from you.