freshspectrum

In 2018, 87 years after it was completed, Zora Neale Hurston’s Barracoon was released to the public.

Reading the book today the story of Cudjo Lewis feels completely relevant to our contemporary times. It is heartbreaking, illuminating, and honest.

In the way she writes, Hurston allows Cudjo to tell his own story. They are his words, told in his way. She is an active participant in the collection of those words, a supporting character in the telling, and a translator through her authorship.

And because of Hurston’s insistence that the book remain that way, and not adapted to meet the desires of publishers wanting the writing altered from dialect to language, the words were almost lost to time.

Poor Zora. An anthropologist, no less! A daughter of Eatonville, Florida where truth, what was real, what actually happened to somebody, mattered. 

Alice Walker’s blog post introducing Barracoon

Who owns the data?

A few years ago I had the honor of attending a presentation by Stafford Hood and Nicole Bowman. And something from the Q&A has been stuck in my head ever since.

For me, it’s a no brainer…If they are saying they don’t want that data published, it’s over.

Stafford Hood – 2016 EERS Eleanor Chelimsky Forum [YouTube recording! Thanks EERS!]

Data is too simple a word for what it represents. As if collections of characteristics, thoughts, feelings, and stories are not reflections of humanity and filled with complexity.

Of course simplifying something so personal allows it to become marketable. If you detach data from people, turn the observations into a series of ones and zeros few can understand, all of a sudden it’s a product that can be sold?

Disconnecting “data” from people doesn’t make it objective. It just hides the origin. Just like giving a cell a name like HeLa, and allowing it to be bought and sold, doesn’t mean it wasn’t lifted without permission from a person named Henrietta Lacks.

Putting Research before People

For long before you and I, science has been used as a justification for both ethical and unethical practices.

The human subjects protections we have today don’t exist because scientists and policymakers were forward thinking. They exist because of past atrocities and unethical studies that put the pursuits of science over the rights and well-being of people.

The men were never given adequate treatment for their disease. Even when penicillin became the drug of choice for syphilis in 1947, researchers did not offer it to the subjects. 

The CDC’s Page on the Tuskegee Syphilis Study

Today’s Data Challenge

There’s a good case for using smartphone data in the COVID-19 response, but Americans deserve an explanation.

Casey Newton for The Verge

Do you want to know how to make a member of an Institutional Review Board freak out? Just show them the amount of personal data available to your run-of-the-mill online marketer.

Want to know who clicked what, when, and where, well our digital world keeps a record. And the records can be compiled together to paint a bigger picture. Ultimately you can get a story of a person who likely had little idea they were ever being followed.

Most of the time the data being collected seems harmless enough. At least until it’s not.

Right now the same marketing data is being engaged in efforts to combat the spread of a pandemic. Does that make the social media industry’s collection, storage, and commercialization right? Does contact tracing justify China’s development of a surveillance state?

These are open questions, and not always easy ones to answer.

Being a data steward

Evidence has value, but it also comes with a price tag (societal cost). Our values, and the values of our organizations, guide our usage.

That is, if it’s on anyone’s radar.

As an evaluator it is often not your role to choose, but it is your role to guide choice. Who does data belong to? Where did it originate? How is it being used?

These are all questions we need to help answer. Because there is no guarantee that anyone else will.

Methods and frameworks are how we turn our professional actions and philosophies into practical strategies.

Once formed, they can be shared and shaped by other hands. They can evolve and grow. They can sometimes take on a life of their own.

Methods and frameworks are two different things. I have a tendency of bundling them together when speaking of our tools. But thanks to the amazing Jara Dean-Coffey, I’m going to try harder to keep the two separate via my communications.

This is great. Point of clarification… @equitableeval is a framework that shifts axiology (purpose) and this affects methods.

Jara Dean-Coffey via Twitter in reply to last week’s post

It might seem pedantic (ivory towery) but it’s useful. Because all sorts of methods and frameworks can be used in parallel. Not even just that, they can build off of one another.

Just like Batman doesn’t have to choose between his utility belt and his bat-mobile, we don’t have to choose between developmental evaluation and the equitable evaluation framework.

But like all the things I like to talk about, let’s leave the nuanced terminology behind for the moment. Instead, let’s talk about these three things.

• What we do.
• How we do.
• How we see.

Before we jump in though, let’s take one short pandemic related tangent.

Back to normal…

When all sorts of things are changing rapidly, it’s natural to ask yourself, “when will things get back to normal?”

But is normal what we really want? Or does it just feel safer?

Our normal is fueling global warming. Our normal is not looking out for the well-being of our neighbors. Our normal is unsustainable consumption. Our normal is the continued strengthening of structures that enable global inequality. Our normal is divisive and sometimes violent.

A global pandemic is not the societal shock that anybody would choose. It is catastrophic and will have the greatest impact on the people who are already the most vulnerable.

But it is a shock.

There is very little doubt that our society will change in response to this crisis. That this will change us.

But whether that change is towards a better future, or not, that is on us to mold and shape.

It is certain, in any case, that ignorance, allied with power, is the most ferocious enemy justice can have.

James Baldwin from No Name in the Street

What we do.

Understanding our actions is important.

What we do is the action that leads to our desired consequence. Sending out a survey. Holding a focus group. Analyzing a dataset. We usually call these methods.

Most of the time we group strings of methods together in a series of steps. Then we might call this an approach.

Or we create a collection of methods and approaches designed so you can adapt to the needs of your evaluation. This might be called a toolkit.

Approaches (on this site) refer to an integrated package of options (methods or processes). For example, ‘Randomized Controlled Trials’ (RCTs) use a combination of the options random sampling, control group and standardised indicators and measures.

How Better Evaluation defines an approach.

How we do.

How we do our methods is also important.

The methods can be evidence-based and deployed systematically with fidelity (a.k.a. by the book). But that won’t necessarily build the trust required to convince the people who will feel the consequences.

How we do things is inextricably tied to who we are.

In some circles and professions, subjective is thrown around as if it were an insult. As if history and perspective are things we should ignore and pretend don’t matter. But subjectivity is purely a reality of the human condition.

The values we hold, the skills we have, the trust we’ve built, our failures, our successes, our connections, our culture, our mental lapses, our history, and the future we perceive, all shape the way we do our work.

There are a number of evaluation approaches designed to mitigate this natural challenge by focusing on the role and placement of the evaluator. They are usually compatible with all sorts of methods (what we do) and fit within broader frameworks (how we see).

Empowerment evaluation is a stakeholder involvement approach designed to provide groups with the tools and knowledge they need to monitor and evaluate their own performance and accomplish their goals.

Better Evaluation page on Empowerment Evaluation, written by David Fetterman

When I think about Indigenous evaluation (or in my case Kaupapa Maori evaluation), it is about evaluation by Indigenous, for Indigenous, with Indigenous, and as Indigenous; and where there is no assumed role for non-Indigenous people, unless by invitation.

Nan Wehipeihana from Increasing Cultural Competence in Support of Indigenous-Led Evaluation: A Necessary Step toward Indigenous-Led Evaluation

How we see.

Some people see the glass half full.

Other people see the glass half empty.

The evaluator considers how this evidence will inform policy.

The researcher objectively measures 8 ounces of water in a 16 ounce glass. They write down their observation and share it with an exclusive group of their researcher friends.

Critical thinking without hope is cynicism. Hope without critical thinking is naïveté.

Maria Popova – Brainpickings

The way in which we see the world changes what we see.

One organization’s view of success is another organization’s view of failure.

If you believe that we have influence over climate change, and that it is essential we exercise that influence to reduce global warming, then that view will likely influence how you act.

If you see the coronavirus pandemic as a sham, no more than the flu but hyped up by the fake news left to try to take down the president, the likelihood that you are social distancing (action/method) right now is probably fairly low.

How we see the world, changes what we see, and subsequently what we do.

Adopting a framework that aligns with your values can provide the foundation you need to select appropriate methods and approaches.

Our premise is that evaluators have a moral imperative to contribute to equity. Evaluators who work with foundation and nonprofits who are working on equity have a special obligation to ensure that their evaluation practices don’t reinforce or even exacerbate the inequities that efforts seek to address.

The Equitable Evaluation Framing Paper

The Eval Central UnWebinar for March 25, 2020 at 10AM Eastern US Time

I’m sometimes asked, “Why are there so few people of color in evaluation?” I flip the question: “Why is evaluation so white?” And answer: “Because our labor is actively erased.”

Vidhya Shanker from her AEA365 blog post: The Invisible Labor of Women of Color and Indigenous Women in Evaluation

Join us on March 25 for only our second ever Eval Central UnWebinar.

Hook up your web cam, put on your headset, and bring your expertise.

Each week we’ll facilitate a conversation. Starting with a seed topic, then allowing the conversation to evolve naturally.

Vidhya Shanker will be this week’s special guest. The seed topic: The Invisible Labor of Women of Color and Indigenous Women in Evaluation. Want to prep your mind before the call, I suggest starting with Vidhya’s AEA365 post.

This post is a day late as recent events hit me with blogger’s block.

I didn’t want to just write another post, seemingly oblivious to the very real global pandemic that is on everyone’s mind. But I didn’t just want to add to the COVID-19 noise.

But the more I thought about it, the more I started to consider all of the super useful ways evaluators can contribute in times of crisis.

This post is a reflection of that brainstorm. It is what I see as evaluator roles during a crisis. How we can use our evaluation expertise and skills to support our society in an unprecedented time.

It is not comprehensive, after you read it, I would love to hear your thoughts and additions.

Remembering to Breathe

Breathe in, breathe out.

When you breathe in, you bring oxygen into your body.

When you breathe out, you expel carbon dioxide.

Hold your breath and your body doesn’t get the amount of oxygen it needs to work. This is particularly challenging when physically active and your body needs more oxygen.

On the other end, breathing in too much oxygen can also be bad. Hyperventilation causes our oxygen levels to go up and carbon dioxide to go down. But if the carbon dioxide goes down too far we suffer, as we have nothing to breath out.

Healthy breathing is about balance.

Evaluation is the oxygen that powers decision making. Too little and we are likely to make poor decisions. And when faced with big challenges, we need more than usual.

Too much evaluation without action leads to hyperventilation. Analysis paralysis.

As an evaluator, it is your responsibility to keep the breathing steady.

Informing decision makers

Dr. Bicknell believed “public health is much more dangerous than medicine. We deal with populations, doctors deal mainly with individuals. So bad public health professionals are far more dangerous than bad doctors. We can kill more people with less accountability.”

From Rationing, Data, and the Ethics of our Decisions by Amanda Makulec

During times of crisis decisions must be made, with or without data.

Acting quickly, with the right decisions, can save lives.

Acting quickly, with the wrong decisions, can cost lives.

Most of the time decision making isn’t about yes or no, black or white, and wrong or right. It’s about making a choice at the time, with the information you have at hand.

There are decision makers who stay uninformed by choice. Deciding to follow their gut or only listen to others who are similarly uninformed.

But there are others who prefer to be informed. To show up with at least a sense of the decision that seems most supported by the information we have at hand. It isn’t always going to be the right choice, but it is less likely to be the wrong choice.

Informing those who wish to be informed, is also with the domain of evaluation.

Causality without control groups

Contrary to popular opinion, there are actually a lot of options [to measure attribution]. They range from the high-powered ones to some fairly low-tech common-sense options you can use even in small-scale community projects, and even if ALL of your evidence is qualitative. Yes, really!

From Understanding Causes by Jane Davidson

There is not always a control group.

When the situation at hand is complex and emerging, there might not be a proper comparison group. But that doesn’t mean we can’t determine how certain actions lead to desired consequences.

The often touted “gold standard” RCT methodology is not the only evaluation method capable of determining if something works. Especially for emerging social responses to crises.

Creating feedback loops, opportunities to understand and learn from our successes and mistakes, are critical to fueling rapid improvement.

As an evaluator, you have a role in the iterative process of rapid response.

Speaking for the vulnerable

Social vulnerability refers to the resilience of communities when confronted by external stresses on human health, stresses such as natural or human-caused disasters, or disease outbreaks. 

From the CDC’s Social Vulnerability Index

Vulnerable populations need advocates even at the best of times.

But when the worst happens, the most vulnerable populations are likely to be most at risk. Vulnerability amplified.

When all the restaurants close their doors, a tip-reliant single-mother already living paycheck to paycheck, can get left without any options. Her child who might have been getting most of her meals at school, is now sheltering at home without that benefit.

When decisions are made to support the greater good, it is right to ask, “just what is that greater good?”

“Who doesn’t have a seat at the table?”

“Who is likely to see the most negative consequences from our actions?”

Out of mind, out of sight. As an evaluator, you can use your talents to help keep those with the highest needs stay in mind and in sight.

Not just an afterthought

Developmental Evaluation (DE) is an evaluation approach that can assist social innovators develop social change initiatives in complex or uncertain environments.

Better Evaluation’s Overview of Developmental Evaluation

Even among the most data savvy of organizations, I think there is still a prevailing thought on the timing of evaluation. That it’s something you plan at the beginning, and do at the end.

But the role an evaluator, as a guide, is one that can be useful during every stage of a program or activity. It’s not just about lessons learned, but informing change.

Our methods are not simply about saying whether or not a program worked. They are not simply about improving designs. Our methods are also useful in discovering the right way to move forward.

In challenging times, we need step up.

Evaluation is an anytime activity.

Staying true to your values

Everyday narratives that continue to marginalize, minimize, and disrespect people of color and those with less privilege could be replaced with ones that do not demonize and place blame on the individual. They could instead lift up the historical, contextual, and powerful dynamics that create and sustain oppression and shed light on the strategies and solutions which can shift the “rules of the game” so that equity is achievable.

From the Equitable Evaluation Initiative

It is easy to live our values when everything is going along swimmingly. When the economy is solid and the uncertainty level is low.

As evaluators we have the responsibility to hold our organizations accountable for core values, especially when times are tough.

***

Facilitating Online Meetings

Let’s be honest. A poorly designed run meeting is a waste of time, energy and resources. This is true face to face and online.

From So You Want to Host a Web Meeting? by Nancy White, with Pete Cranston, Susan Stewart and Bonnie Koenig

The guide above is from a few years ago. But it’s well written and full of useful insight on hosting effective web meetings.

Wherever you are, stay safe. Flatten the curve. Lookout for your neighbors.

We’ll get through this.

“Let’s take a shortcut back to the car.”

I was 19 and hiking through the woods with one my best friends. We were both eagle scouts, and had spent much of our young lives wandering through the woods. The area we were exploring was one of our favorites and we had hiked it a dozen times over the preceding couple of years.

But as we traveled down our little mountain shortcut, things seemed different. We were walking by a camp we had never visited. Then we came to roads we had never traveled and a few small country houses we had never seen.

We spotted a county maintenance pickup truck sitting on the side of the small road with two guys seemingly catching a break. And we asked them, “do you happen to know which way it is to the Wolf Rock parking lot?”

They looked at each other then pointed to the back of the pickup and said, “we’ll take you.” It was as we drove out towards the main highway that my friend and I came to a realization that almost seemed too obvious.

Our shortcut had taken us down the wrong side of the mountain. And lucky for us, we stumbled upon two county workers who had the time to provide a ride for a couple of clearly lost hikers.

Even in the best of circumstances.

When the origin and the destination are both well-known. And the path between the two is one that has been ventured over and over again. And the people on the journey have ample experience.

We are always just one wrong turn from ending up on the wrong side of the mountain.

The World in the Weeds

Our world is complex.

The faraway problems that were once obscured by distance are now delivered right to your social media feed.

And our society, it’s changing rapidly. Symptoms of overwhelm can be seen everywhere. Things like increasing anxiety, depression, and a general sense of disconnect.

If you have ever worked in a restaurant you might have heard the phrase, “in the weeds.” It’s the line you use when a fellow server gets overwhelmed. Their eyes and actions communicate that overwhelm as a silent call for help.

As I walk around my co-working space I can’t help but notice that look. I see the same look in the eyes of colleagues and friends. And I see the look in the eyes of the people who are tasked with changing the world.

Our society is in the weeds.

And when you’re in the weeds, when the origin is fuzzy, when the destination is unknown, when the path is unclear, and when design is really just a series of educated guesses, it is not the best of circumstances.

Our society needs help.

Our society needs evaluators.

Evaluation as Wayfinding

As the voyages became longer, they developed a highly sophisticated navigation system based on observations of the stars, the ocean swells, the flight patterns of birds and other natural signs to find their way over the open ocean.

Wayfinders, A Pacific Odyssey [PBS]

Thousands of years before Europeans on their big ships began conquering and colonizing the world (a.k.a. “exploring”), the indigenous peoples of the Pacific were discovering and populating new lands.

The Europeans were surprised when “discovering” Pacific islands already inhabited by people who only seemed to possess the simplest of tools. The Polynesians didn’t have the metal, maps, or fancy instruments that the Europeans believed to be prerequisites to cross such distances. But yet, here they were.

The traditional way of navigating used all the natural science to guide the canoe. And that would be in the heavens, the stars and the moon at night, the sun and the moon during the day, as well as the ocean waves. The system incorporated reading the signs of the clouds and also the animals, especially the seabirds to determine where land is. It was an ingenious system that required tremendous observation of the elements of nature to make successful landfall.

Nainoa Thompson

In the years that I have spent working as a UX designer and visual data scientist I have noticed something peculiar. We are faced with an overwhelming abundance of data and evidence. And our response is usually to develop something technical and complex (AI/Machine Learning/Blockchain) in an effort to make the complex digestible.

But while inventors work to develop the modern equivalents of sextants and compasses, much of the evidence remains accessible but untapped.

We don’t need the expensive tools to navigate our way through the modern world. We just need the discipline to see the evidence, understand our origin, imagine our destination, and plot our course.

This is the role of the evaluator.

To leverage evidence and expertise in order to help our society navigate out of the weeds and toward a better future.

Describing how a process works is valuable for two reasons. It forces you to make sure you know how it works. Then it forces you to take the reader through the same sequence of ideas and deductions that made the process clear to you.

William Zinsser

The whole point of having an evaluation mindset is that by knowing how something works (or does not work) you can try to make it work better. Or it can inform you as to when you should abandon your plans and take a different path. Or it can give you insight on other pathways to reaching your target consequences.

But mindset doesn’t mean just keeping the thoughts in your head. There is incredible value in putting ideas down on paper.

Modeling your evaluation thought process creates something that can be built upon. It makes your thoughts and evidence tangible, so they can be measured and tweaked.

And it gives you something you can share with others. A representation of your thought process.

All Models are Wrong

But some are useful.

I have always liked that quote by the late British statistician George Box.

In the field of evaluation, models are important. We use them to ground our work and communicate our assumptions.

They carry different names depending on your context, sometimes called logic models or log frames and other times theories of change. Over time these concepts have naturally evolved, increasing complexity and decreasing accessibility.

But for your evaluation mindset, I encourage you to think of them as something else.

Representations.

They can be simple depictions of the actions and consequences discussed in my last post.

And like all models, they will be wrong.

But our goal isn’t to make them right.

Our goal is to make them useful.

Doodle Modeling

So let’s draw some useful models.

We’ll start with a basic action – consequence pair. Then we’ll expand up on it, adding additional details and space for measurements/context.

We’ll doodle it out on a piece of paper using arrows and boxes to connect different pieces.

Action: We Teach ______
Consequence: They Learn _____

Now let’s expand upon it a bit.

Action: We Teach _____
Measure: How much did we actually teach? How many people did we teach? How long did we spend teaching them?

Consequence: They Learn _____
Measure: Did they actually learn _____? How much of what was taught did they learn _____?

Now let’s go a little bit further with more detail.

Goal: We want to teach all high schoolers in our county personal finance.

Context: We are given an hour once a week to speak to a single class of high school seniors. This class is mandatory and we will see the same students over and over again. We were awarded a local government grant to put on a single class and cover a little bit of prep time.

Action: We teach practical personal finance lessons for students who will soon be off to higher education or fully into the working world.
Measure: How many lessons were taught? Were all of the students present for all lessons?

Consequence: They learned how to create a budget and tips for staying on that budget. They also learned how to use tools to monitor their progress.
Measure: Did they actually learn how to create a budget? Will they start creating/following budgets on their own after the program has ended?

Doodle Modeling an Indie Business

Alright, so let’s do another example.

Action: Create Websites
Consequence: Get Paid

It doesn’t take long before it dawns on you that just creating websites doesn’t get you paid.

You need to find people who will hire you to create websites. Then you can create the websites and get paid. So the tricky part of the business, most of the time, is finding clients.

It’s not that getting paid to create websites isn’t the work. It’s just not particularly useful to focus a lot on that aspect. So instead of building out that model, let’s build out another one.

Action: Attending Networking Events
Consequence: Get Clients

Okay, so this pair is a bit more useful in my opinion, at least for most of the indie businesses I know who are constantly struggling to find new clients.

Let’s expand it out a little.

Action: Attending Networking Events
Measure: How many networking events did you go to? How many prospective customers did you meet?

Consequence: Get Clients
Measure: How many follow-up coffee chats were setup? How many new clients were found that can be directly connected to that event? How many indirectly connected new clients?

Models make actions and consequences tangible.

The goal of most models isn’t academic.

It’s about translating a bunch of ideas, evidence, goals, and strategies into something tangible. Something that you can wrap your mind around. Something that you can use to discuss your plans with other people.

You are creating a useful representation.

You don’t need a special degree or certification to create a model. It doesn’t have to be super fancy or made with super fancy software.

A pen and paper is a great place to start.

Where Doodle Models meet Logic Models

So for all of you professional evaluators and nonprofit program managers who are trying to tie this to the work you do right now. This is how I connect the two.

Outputs are measurable actions.

Outcomes are measurable consequences.

Tip: Start Simple then Expand

The action-consequence pair is the heart and soul of the model.

If you begin with that, you can then start to expand and add more detail. You can add more measurements, more context, more detail.

And while in our complex world, the idea of boiling our projects down to action-consequence pairs might seem wrong, it can certainly be useful.

Practice Time

Time to doodle some of your own models. To get you started, here are some prompts.

Let’s say you want to have a fantastic weekend (consequence) what actions will help make that consequence a reality? How would you measure both your actions and consequences?

Maybe you would like to run, not walk, an entire 5K (consequence). What actions would help make that consequence a reality? How would you measure both your actions and consequences?

Let’s say you want to help the people who are important in your life feel more appreciation (consequence). What actions would help make that consequence a reality? How would you measure both your actions and consequences?

Thank You!

As our collective attention gets pulled in every which direction, it becomes increasingly meaningful to have your work read, discussed, and shared.

So thank you for reading, commenting, and sharing, it really means a lot.