The Evaluation Mindset: Methods and Frameworks

Methods and frameworks are how we turn our professional actions and philosophies into practical strategies.

Once formed, they can be shared and shaped by other hands. They can evolve and grow. They can sometimes take on a life of their own.

Methods and frameworks are two different things. I have a tendency of bundling them together when speaking of our tools. But thanks to the amazing Jara Dean-Coffey, I’m going to try harder to keep the two separate via my communications.

This is great. Point of clarification… @equitableeval is a framework that shifts axiology (purpose) and this affects methods.

Jara Dean-Coffey via Twitter in reply to last week’s post

It might seem pedantic (ivory towery) but it’s useful. Because all sorts of methods and frameworks can be used in parallel. Not even just that, they can build off of one another.

Just like Batman doesn’t have to choose between his utility belt and his bat-mobile, we don’t have to choose between developmental evaluation and the equitable evaluation framework.

But like all the things I like to talk about, let’s leave the nuanced terminology behind for the moment. Instead, let’s talk about these three things.

• What we do.
• How we do.
• How we see.

Before we jump in though, let’s take one short pandemic related tangent.

Back to normal…

When all sorts of things are changing rapidly, it’s natural to ask yourself, “when will things get back to normal?”

But is normal what we really want? Or does it just feel safer?

Our normal is fueling global warming. Our normal is not looking out for the well-being of our neighbors. Our normal is unsustainable consumption. Our normal is the continued strengthening of structures that enable global inequality. Our normal is divisive and sometimes violent.

A global pandemic is not the societal shock that anybody would choose. It is catastrophic and will have the greatest impact on the people who are already the most vulnerable.

But it is a shock.

There is very little doubt that our society will change in response to this crisis. That this will change us.

But whether that change is towards a better future, or not, that is on us to mold and shape.

It is certain, in any case, that ignorance, allied with power, is the most ferocious enemy justice can have.

James Baldwin from No Name in the Street

What we do.

Understanding our actions is important.

What we do is the action that leads to our desired consequence. Sending out a survey. Holding a focus group. Analyzing a dataset. We usually call these methods.

Most of the time we group strings of methods together in a series of steps. Then we might call this an approach.

Or we create a collection of methods and approaches designed so you can adapt to the needs of your evaluation. This might be called a toolkit.

Approaches (on this site) refer to an integrated package of options (methods or processes). For example, ‘Randomized Controlled Trials’ (RCTs) use a combination of the options random sampling, control group and standardised indicators and measures.

How Better Evaluation defines an approach.

How we do.

How we do our methods is also important.

The methods can be evidence-based and deployed systematically with fidelity (a.k.a. by the book). But that won’t necessarily build the trust required to convince the people who will feel the consequences.

How we do things is inextricably tied to who we are.

In some circles and professions, subjective is thrown around as if it were an insult. As if history and perspective are things we should ignore and pretend don’t matter. But subjectivity is purely a reality of the human condition.

The values we hold, the skills we have, the trust we’ve built, our failures, our successes, our connections, our culture, our mental lapses, our history, and the future we perceive, all shape the way we do our work.

There are a number of evaluation approaches designed to mitigate this natural challenge by focusing on the role and placement of the evaluator. They are usually compatible with all sorts of methods (what we do) and fit within broader frameworks (how we see).

Empowerment evaluation is a stakeholder involvement approach designed to provide groups with the tools and knowledge they need to monitor and evaluate their own performance and accomplish their goals.

Better Evaluation page on Empowerment Evaluation, written by David Fetterman

When I think about Indigenous evaluation (or in my case Kaupapa Maori evaluation), it is about evaluation by Indigenous, for Indigenous, with Indigenous, and as Indigenous; and where there is no assumed role for non-Indigenous people, unless by invitation.

Nan Wehipeihana from Increasing Cultural Competence in Support of Indigenous-Led Evaluation: A Necessary Step toward Indigenous-Led Evaluation

How we see.

Some people see the glass half full.

Other people see the glass half empty.

The evaluator considers how this evidence will inform policy.

The researcher objectively measures 8 ounces of water in a 16 ounce glass. They write down their observation and share it with an exclusive group of their researcher friends.

Critical thinking without hope is cynicism. Hope without critical thinking is naïveté.

Maria Popova – Brainpickings

The way in which we see the world changes what we see.

One organization’s view of success is another organization’s view of failure.

If you believe that we have influence over climate change, and that it is essential we exercise that influence to reduce global warming, then that view will likely influence how you act.

If you see the coronavirus pandemic as a sham, no more than the flu but hyped up by the fake news left to try to take down the president, the likelihood that you are social distancing (action/method) right now is probably fairly low.

How we see the world, changes what we see, and subsequently what we do.

Adopting a framework that aligns with your values can provide the foundation you need to select appropriate methods and approaches.

Our premise is that evaluators have a moral imperative to contribute to equity. Evaluators who work with foundation and nonprofits who are working on equity have a special obligation to ensure that their evaluation practices don’t reinforce or even exacerbate the inequities that efforts seek to address.

The Equitable Evaluation Framing Paper

The Eval Central UnWebinar for March 25, 2020 at 10AM Eastern US Time

I’m sometimes asked, “Why are there so few people of color in evaluation?” I flip the question: “Why is evaluation so white?” And answer: “Because our labor is actively erased.”

Vidhya Shanker from her AEA365 blog post: The Invisible Labor of Women of Color and Indigenous Women in Evaluation

Join us on March 25 for only our second ever Eval Central UnWebinar.

Hook up your web cam, put on your headset, and bring your expertise.

Each week we’ll facilitate a conversation. Starting with a seed topic, then allowing the conversation to evolve naturally.

Vidhya Shanker will be this week’s special guest. The seed topic: The Invisible Labor of Women of Color and Indigenous Women in Evaluation. Want to prep your mind before the call, I suggest starting with Vidhya’s AEA365 post.