freshspectrum

This post was originally written by Andrea Guerrero- Guajardo as a direct response to Michael Quinn Patton’s blog post “Rules for Privileged Straight White Males.” It was published to Google Docs on June 6, 2020 as a personal blog post and shared on EvalTalk as a contribution to the overall discussion.

The words in this post belong to Andrea and Michael. Andrea has graciously allowed me to re-post her words here on this site. The cartoons are my contribution (-Chris).

Hi Evaltalk-I have some thoughts:

I know and acknowledge that this was an earnest attempt at writing these rules. Upon first and second and third reading, I became increasingly aware that the rules continue to frame the straight white male perspective from a passive posture without advocating for an active role in dismantling white supremacy.

Each of these rules can be accomplished from the safety of the straight white male’s living room. None of these rules obligates the straight while male to actually do anything to elevate the voices of Black and African-American people. (Side note: I intentionally prioritize Black and African-American people because the moment demands it. The use of “women and people of color” minimizes the notion that Black Lives Matter. As a Latina and a Cherokee woman, I can declare that Black Lives Matter without asking – but what about me? I digress). 

I have offered my own revisions to the rules that center how straight white males can be actively anti-racist instead of simply being satisfied with declaring that they are not themselves a racist. Dr. Ibram X. Kendi states that “…the problem with being ‘not racist’? It is a claim that signifies neutrality: I am not a racist, but neither am I aggressively against racism. But there is not neutrality in the racism struggle. The opposite of ‘racist’ isn’t ‘not racist.’ It is ‘antiracist.;”

I also note that MQP has cited three of our colleagues: Nicky Bowman, Vidya Shenker, and Rita Fierro. I admire and respect and absolutely value both scholarship and friendship that comes from these women. But in a moment where we are screaming Black Lives Matter from the pits of our souls with the pain and anguish of 400 years, why did MQP not find the time to publish the names of Black Evaluator Lives that Matter? I’ll start, and I’m sure that I will miss some.

Please reply and add more. Hazel Symonette, Stafford Hood, Melvin Hall, Rodney Hopson, Dominica McBride, Ayesha Boyce Akynruly, Katrina Bledsoe, Denise Ward Hood, Tamara Bertrand-Jones, Kevin Favor, Hank Frierson, Leah Peoples, Cindy Crusto, Krystle Tomlin, Maurice Samuels.  Seriously, add names here. (Update, additional names provided via crowdsource): Anna. M Madison, James E. Davis, Geri Lynn Peak, Veronica Thomas, Elmina Johnson, Ricardo Millett, Jara Dean Coffey, Michael Arnold.

Every single one of these people has been my teacher in some way. I either read their book or paper, heard their keynote, attended their sessions, or shared a story over a cocktail. Their lives matter just like all other Black Lives Matter, and our eval community should be saying that out loud right now. 

Similarly, I notice that MQP was guided mostly by water protectors at Standing Rock. This is another missed opportunity to center Black and African American voices.  There are innumerable local, state, and national Black Lives Matter chapters and organizations. Reclaim the Block, the Black Visions Collective, Color of Change, Dignity and Power Now, and many others have been on the front lines of protest and can offer guidance in the same way that Standing Rock can. The difference is this: you are centering Black voices and is why I had to offer my revisions of MQP’s rules. They remove the straight white male frame while offering action-oriented strategies for how to be a straight white male working to dismantle white supremacy. 

MQP’s differentiation of rules versus principles to create contextual guardrails operates under the assumption that privileged straight white males follow all the rules. Principles are subjective based on the social conditioning of the straight white male, and it has been proven time and again that he chooses to disregard the rules, even rules as simple as “no means no” and “stop sign means stop.” Often, the straight white male weighs the cost of breaking the rule against the consequence he might suffer. The framework of white supremacy that tells us that the consequence will likely be lenient. When confronted with “no means no,” the privileged straight white male must ask himself, “Will I be fired for sexually harassing my colleague?” or when he encounters a stop sign, “can I afford to just pay a ticket?” A hard and fast rule does not equate to cognitive or behavior change and is ineffective if the straight white male is not willing to, or does not understand, the principle that should influence his decision-making. 

MQP draws on his own “personal experiences” to inform the rules for straight white males. I suggest that straight white males are not the ones who should be policing themselves. Additionally, it would be irresponsible of me, as a Cherokee Latina woman, to accept rules for straight white males that have been written by straight white males. MQP declares that these rules do not follow a commonly accepted process for co-construction, are absolute, and not up for discussion. The restriction disallows the introduction of input and critique from any source other than the privileged straight white male and discounts legitimate knowledge of communities of color.

MQP acknowledges that these rules are likely incomplete, and invites others who want to add their own to “bring them on.” These revisions are my own critical read of the Rules as written by MQP. I have spent the last few weeks (and a lifetime, quite frankly) listening to my *white allies* pat themselves  on the back for just knowing that white supremacy exists but doing nothing about it. MQP has most certainly been doing the *work* in his scholarship and practice, but if we are writing rules for other straight white males, the rules have to be explicit. 

1. Explore the state of the world through the eyes of a Black or African-American person.

Before you express your opinion and preface it by saying, “In my opinion,” or “From my perspective,” find ways to put yourself in the shoes of a Black person and then see if you should still be sharing your opinion at all. Instead, elevate the voices of Black and African American people. 

Do not make general authoritative statements about the state of the world, or any parts of it (like evaluation). Do not pontificate in general authoritative terms about how things are and/or how things ought to be. An example would be to state unequivocally, as was recently done: “Evaluation is not a tool for social change.”

The appropriate way to express opinions is to preface those opinions with some conditional introduction like, “From my perspective,” “In my view,” or “Based on my experiences…..”

You do this not to alert others that you are expressing a personal opinion but to remind yourself of that fact because, straight white males often confuse their opinions about the world with how the world actually is because, in their arrogance, self-assurance, and cluelessness, they actually believe they speak truth and that others ought to feel privileged to hear their truth. So, it can be helpful to remind yourself that you are expressing an opinion not speaking truth by prefacing your opinions conditionally. Others already know that anything and everything you say is conditioned by your being a privileged straight white male. They don’t need to be reminded of that. You do.

Moreover, the sub-rule of this rule, is never add IMHO. By invoking that particular conditional, you are communicating not humility, but the exact opposite. The language of “my humble opinion” is arrogant, grating, insensitive, and infuriating because it tells the listener that you are the exact opposite of humble, but you think you can disguise your arrogance, insensitivity, and cluelessness with that juvenile phone text shorthand.

From Michael Quinn Patton’s blog post “Rules for Privileged Straight White Males.”

2. Use your white male privilege and the benefits you receive from it, to actively help Black and African-American people.

Leverage wealth, influence, social capital to enrich or enhance the work of Black and African-American people, their careers, initiative, foundations, scholarship, etc. Furthermore, the privileged straight white male needs to be willing to say, “I am an oppressor.”

Don’t deny you are privileged. Privileged straight white males, which is all  white males in a society that inherently privileges straight white maleness, understandably feel uncomfortable with having their privilege named and exposed. The Standing Rock guidance states: “Standing Rock challenges allies to be aware of their white privilege, and to occasionally be comfortable sitting in your own discomfort.” Protesting that you are not privileged simply demonstrates your cluelessness about your privileges and the systemic and structural nature of those privileges. Instead of resisting the privileged status, put your energy into thinking about and identifying the nature of your privilege. A simple example.

Growing up, my father paid me an allowance to mow the grass, shovel snow, and other “male” tasks. My sisters were required to do housework without pay because as housewives they would not be paid, so they should not come to expect to be paid for housework. I was exempt from housework. I was allowed to deliver newspapers to make money. My sisters were denied such opportunities.

In elementary school the black students regularly got detention for talking to each other in class, but the white students never did. Several detentions led to suspension; suspension led to failing a grade; being held back a grade and then another, led to dropping out of school; and onward into poverty and prison. Such are the small, trivial beginnings of disadvantage versus privilege that build and accumulate and become systemic and make you believe that you’ve earned the position of privilege. The rule: Don’t deny you are privileged. Understand your privileges. Search and you will find them, aplenty.

From Michael Quinn Patton’s blog post “Rules for Privileged Straight White Males.”

3. Please do not ever, under any circumstances go around stating that you are a privileged straight white male.

Full stop. We are probably already aware of that status and don’t need to be told. Instead, actively work toward not doing harm to Black and African-Americans with words or deeds. (Note: I get that MQP meant this as one does at a conference or in a paper, but it sounded ridiculous to just state that you are a privileged white male and NOT offer some context for having said it.)

Avoid adding contingencies, explanations, conditionals, and limitations to your straight white male identity and corresponding privilege. What does that mean? Should you find yourself in a situation where people are introducing themselves, or in providing a context for something you write, you state, appropriately: “I am a privileged straight white male.” End the statement there. Full stop. Don’t rush to add: However…, But…, Nevertheless…, Despite that… or any of scores of such conditionals that communicate that you actually are not aware of the implications of being a privileged straight white male as you rush to explain away the very significance of that revelation. Your conditionals will simply reveal your cluelessness. Omitting the conditionals won’t eliminate your cluelessness. That would require adherence to rule number two. But omitting the conditionals would at least keep you from overtly exposing your cluelessness.

From Michael Quinn Patton’s blog post “Rules for Privileged Straight White Males.”

4. Make it your job to learn about the experiences of Black and African-American people.

Still, don’t profess to understand them because, as MQP correctly points out, you can’t and never will. That doesn’t mean that you shouldn’t acknowledge that the white-washed history you were fed in school was incomplete and often erroneous. Study up, straight white male. But keep it to yourself. No straight white male-splaining to people about their own history. Don’t presume to understand what is best for Black and African American people, and don’t force your allyship on others. 

Don’t profess to understand the experiences of women and people of color. You don’t. I don’t. You can’t. I can’t. They know it. You and I need to know it. For example, you are rarely even aware that you are a straight white male. It scarcely, if ever, even enters your consciousness that whatever is happening at any moment is a consequence of and has to do with your being a straight white male. In contrast, women and people of color regularly explain that they are always aware of how their status, whatever it is, is present in the moment at some level in some way. Part of being privileged, indeed, a big part of being privileged, is not having to think about your identity as a straight white male and the privilege that comes with that identity.

From Michael Quinn Patton’s blog post “Rules for Privileged Straight White Males.”

5. The revision to this rule relates to the second half of it. Yes, please listen and ask questions rather than speaking over people.

However, do not force labor on Black and African-American people. It is not their responsibility to educate you or be available for your inquiry stance. Take it to the google or the library. There is a immense body of literature on injustice, racism, and sexism – and all the intersections (shout out Dr. Crenshaw) of those oppressive forces. The people doing the work are exhausted from having to explain for the umpteenth time why growing up poor does not negate your white privilege. Don’t give them more work to do. 

“Speak to understand not to be heard.” This is another rule gently articulated by the Native American leaders who organized the protests against the pipeline at Standing Rock. Straight white males have a long history of dominating conversations, not listening, not asking questions, not listening, interrupting others, did I mention not listening, especially interrupting women and people of color, taking up too much air time, and being in love with hearing themselves talk. These tendencies flow from socialization as a privileged straight white male. These tendencies are not congenital, but become deeply embedded through reinforcement and practice. Learn to ask questions rather than make statements. Take an inquiry stance. You and I have nothing to teach about injustice, racism, and sexism. You and I have everything to learn.

Extra credit homework personal growth assignment for privileged straight white male evaluators: Read Vidhya Shanker’s (2019) dissertation on “the construction of race in and through evaluation,” 

Read it appreciatively, to learn. This is not an evaluation assignment. It is a personal growth assignment. Read without critique, without judgment, without automatically agreeing or disagreeing. Don’t evaluate it. It has already been evaluated by her doctoral committee. Absorb it. Examine your reactions. Seek to understand. Ask clarifying questions. But don’t express opinions about it. You don’t actually understand enough to have opinions about it. A great leap forward would be to actually identify things you don’t understand and questions you have.

From Michael Quinn Patton’s blog post “Rules for Privileged Straight White Males.”

6 – 8. Honestly, 6-8 are kinda similar, so here’s the revision: yes, don’t complain or whine about any of those things, but you are obligated to take it a step further.

When you hear your friends, family, colleagues, co-worker, barber, grocery store clerk, dry cleaner, uber driver, or (and especially elected officials) complain or whine about any of these things – CALL IT OUT. Maybe they’ll listen or maybe they won’t, but you don’t get a W just because you manage not to complain or whine. It’s a standard that most children learn by the time they leave kindergarten, so you have to be better and do better.

Don’t complain about political correctness, identity politics, affirmative action, intersectionality, diversity initiatives (Does that include me?), inclusion concerns (ditto), and equity (ditto). Just don’t. It’s tiresome and enraging.

7. Don’t ever complain that you are being treated unfairly. The world is not fair. Your sense of unfairness pales in comparison to the unfair challenges and obstacles faced by those on the front lines working against racism and for justice. During the protests about the murder of George Floyd, a major, I would even say dominant, theme in news interviews with black mayors, female mayors, black female mayors, and other mayors representing minority groups has been that twice as much is expected of them as of the white male mayors that they replaced and they get 10 times the criticism. When things go well, the successes are attributed to others; when things go badly, they get the full blame. Fair? African-Americans have twice the infection rates from the coronavirus as whites and twice the death rate from Covid-19. Job losses due to shutting down the economy are significantly greater for women and people of color. Poor people are significantly more affected by pollution of water, land, and air. The list goes on and on. These are indicators of systemic and structural injustice and unfairness.

8. Don’t whine about being misunderstood, misrepresented, and misinterpreted. In so complaining you simply demonstrate a deep, embarrassing cluelessness about your own actions and statements. Instead of lamenting your ill-treatment, think about how you presented yourself and what you said that has led to the reactions you have received that feel unfair. Don’t ask others to behave differently. Your only way to affect their reactions of others is for you to behave differently.

From Michael Quinn Patton’s blog post “Rules for Privileged Straight White Males.”

9. This one works. No revisions here.

Focus on systemic and structural racism and injustice. It’s not about you personally as a privileged straight white male. It’s about the system that you are part of and are either upholding or working to change. These rules are about changing behaviors to change relationships, perspectives, dynamics, and boundaries to change systems. The affliction of being a privileged straight white male is making everything about you. You are a cog in the system. Get cognitive.

From Michael Quinn Patton’s blog post “Rules for Privileged Straight White Males.”

10. Yes, rules are different for Black and African-American people than they are for straight white males. We see this in practice every day, but it’s not just enough to know that it happens.

Find opportunities to leverage your straight white maleness in service to others. Use your time, talent, and treasure to undo the unspoken rules.  Put your body between a person who would assault a Black body. Walk in protest of white supremacy and all its progeny that pervades every single institution of power that exists on turtle island. Real change means sacrifice, and it’s your turn to give up the comfort and safety of your straight white maleness. 

Don’t complain about these rules. Of course, you resist and detest such disrespectful, authoritarian, and demeaning rules. The very scope and depth of your anger at such rules is evidence of your long-standing privilege of not being subject to arbitrary and capricious rules. Four years ago, Philando Castile, a 32-year-old African American man, was stopped while driving and fatally shot by a police officer in Saint Paul, Minnesota. He was carefully following the rules he had learned about how black men should behave when pulled over by police for “driving while black.” At the time I was involved with a team in evaluating an African-American after school program in the inner city of St. Paul. Teachers and parents worked together to teach six, seven, eight, nine-year-old black boys the rules for how to behave to avoid racist retaliation by white teachers, white police, and white people generally. The teachers and parents struggled with how to offer their children a vision of opportunity and hope while alerting them to be careful and follow the rules, unfair rules, disrespectful rules, outrageous rules, but potentially lifesaving rules.

From Michael Quinn Patton’s blog post “Rules for Privileged Straight White Males.”

Guajardo, A. G. (2020, June 6). Andrea’s Rewrite of the Rules for Privileged Straight White Males.  EvalTalk, Personal Blog Post

A Special Thanks to Andrea for allowing me to re-post her words here! – Chris

Eval Central UnWebinar for June 10, 2020 @3PM Eastern (12PM Pacific)

Special Guest: Thomas Archibald

This week’s seed topic: Anti-Racism, White Fragility, White Supremacy, White Privilege, and White People Questions

Join us! > https://www.crowdcast.io/e/evalcentral/register

Ahmaud Arbery. Sean Reed. Breonna Taylor. Tony McDade. George Floyd. When does this racial terror end?

For the past three months, our communities have been inundated with painful stories and images of Black people targeted, harassed, arrested, and killed by police and racist vigilantes. As Black people all across the country grapple with the aftermath of an unprecedented global pandemic, somehow Black people also have to figure how to stay alive while jogging, buying groceries, and yes — even while sleeping.

Color of Change – State of Emergency: End the War on Black people!

It should never have taken this amount of publicly shared violence to engage white people like me in active anti-racism. I appropriately sit with that discomfort thinking of moments across my life where I could have, and should have, said something or done something to show opposition.

I know I am not alone, and many of my dear readers are thinking the same. I’ve spent the last week signing petitions, following up with friends, donating to anti-racist causes, and calling out racist posts from extended family and childhood friends. Going to facebook for me is like visiting my hometown and feeling uncomfortable in the overwhelming whiteness. I hate it, but burying it deep within and pretending it’s not a part of me won’t make it go away.

<Someone’s comment on my comment> Calling someone or something racist, is the last resort of an idiot.”

<My response> in my defense, it was my first resort to call the string of racist memes racist.”

Excerpt from a recent comment conversation I had on Facebook. Basically I called a cousin out on a string of racist memes he was sharing, then had an opportunity to exchange pleasantries with him and his friends…

But as my wife and I sat discussing what we could do to be better allies and accomplices, I started to think about how we could embrace our evaluation skill-set to support anti-racism efforts in a systematic and practical way.

So that’s what this post is. It is a brainstorm on action-oriented contributions. It’s not complete, so if you would like to add your own thoughts, please do.

I offer these observations as one human clinging onto her hopes for humanity.

I invite each of us to remember and unleash our humanity in the work we need to do to make us worthy of this planet and each other.

My simmering rage both fuels and exhausts me. It keeps me alive with a deeper level of understanding in my body and in my soul not just in my head of what is at stake: Everything.

Jara Dean-Coffey from her post on the Equitable Evaluation Initiative website: Embodied Knowledge: Simmering Rage | A note from Luminare Group Founder & EEI Director

To my BIPOC readers, colleagues, and friends.

I pledge to be a supportive ally and accomplice.

Your life matters. Your family’s lives matters. Your friend’s lives matter.

This is no little problem that will away go away when things “return to normal.” The evidence of inequality between BIPOC and white members of this country is pervasive a deep.

I am nowhere near perfect, but I will always listen. Even if the conversation is hard.

Reminder: check on your Black friends, family, colleagues, and students. We are not ok. A few have asked me what could/should this look like? Be honest, be vulnerable, be compassionate. Here are screen shots of an email I sent to my students and colleagues of African descent. pic.twitter.com/KVVJayflDi

— Ayesha Boyce (@AyeshaBoyce) June 1, 2020

My digital office door is always open. I usually say that I offer one 30 minute consultation for free. But that’s pretty much bunk, we can chat longer and it could be more sessions.

In other words, if you ever want to chat, I’m here. And I’m not going to try to sell you on anything. But I can make a pretty good thought partner.

Things I’m particularly good at for thought partnering:

• Coming up with digital strategies.
• Designing digital training programs.
• Evaluating digital efforts.
• Actionable data reporting.

Breaking the Cycle

It’s not a new problem.

It’s just being caught on tape.

Breaking the cycle starts with understanding that a cycle exists.

What is James Baldwin thinking more than 50 years after this interview, which could be written today? (You will notice that nothing I post anywhere in my own words started with me–it is common knowledge among BIPOC thinkers and writers and artists. White investment in individualism means that white leadership always tries to individuate themselves from these repeated patterns of behavior–they are somehow different–even when they demonstrate the documented behavior.)

A post from Vidhya Shanker on LinkedIn.

Working within the Evaluation Guiding Principles

You want to know something neat about being evaluator. Most of the time we work for a client who is paying us to evaluate a program or activity. Or we work for an agency that is paying us to evaluate a whole set of programs or activities. The majority of the time, we receive compensation for our work.

But if you read our guiding principles, you’ll see very little specifically connected to how we serve our bill-paying clients. Our client is only one part of the broader stakeholder groups we serve.

You don’t need a client to evaluate a program and deliver professional services. You just need stakeholders.

So why not use our skills as evaluators to systematically evaluate programs as we strive to contribute to the common good, regardless of whether or not the targeted programs have requested our services.

We just need stakeholders.

Short Version of the AEA Guiding Principles

A. Systematic Inquiry: Evaluators conduct data-based inquiries that are thorough, methodical, and contextually relevant.

B. Competence: Evaluators provide skilled professional services to stakeholders.

C. Integrity: Evaluators behave with honesty and transparency in order to ensure the integrity of the evaluation.

D. Respect for People: Evaluators honor the dignity, well-being, and self-worth of individuals and acknowledge the influence of culture within and across groups.

E. Common Good and Equity: Evaluators strive to contribute to the common good and advancement of an equitable and just society.

Stakeholders – individuals, groups, or organizations served by, or with a legitimate interest in, an evaluation including those who might be affected by an evaluation.

Sharing Evidence to Guide Practice

I get asked a lot for best practices in evidence sharing. Organizations that do a good job of sharing data in an actionable format.

Campaign ZERO’s website is that (JoinCampaignZERO.org). This is information design at its best. If you have data and evidence to share that you hope could become actionable, I suggest treating this site as a blueprint for how it’s done.

My favorite page on the site is the solutions page. Here is how it’s structured.

Campaign ZERO was developed with contributions from activists, protesters and researchers across the nation. This data-informed platform presents comprehensive solutions to end police violence in America. It integrates community demands and policy recommendations from research organizations and President Obama’s Task Force on 21st Century Policing. Together, we will win.

The Campaign Zero Planning Team (a.k.a. amazing people you should really follow on Twitter):
@deray, 33, is a protestor, dedicated to ending police and state violence.
@samswey, 29, is data scientist who leads the development of research, digital tools and platforms to end police violence and systemic racism in America.
@MsPackyetti, 33, is a St. Louis native raised in a tradition of social justice.

Localizing Critical Data

I want to mention two more sites from the same team who brought us Campaign Zero.

The first is MappingPoliceViolence.org

This site uses secondary sources to map police violence across the country. The data combines products of crowd-sourced sources with original research to paint a picture not currently shared by law enforcement agencies. The methodology is fully transparent and the full database is available for download.

The second is PoliceScorecard.org

The Police Scorecard provides a deep dive into the actions and policies for police forces in California. The score card, which grades individual police departments, is systematic and transparent in its evaluative approach. The overall mission is to expand the work nationwide.

Crime data is collected and reported much in the same way as it has for the last two decades. Seriously, not an overstatement. In the year 2000 I was a criminology major in undergrad. Looking at the Uniform Crime Reporting data and the National Victimization Crime Survey data creates flashbacks to my college days.

Sites like Mapping Police Violence exist because our current national data systems do not answer important evaluative questions in a meaningful way (if at all).

Ask some questions and try to find the answers. You shouldn’t need a Masters and two decades of data experience to find them. We also can’t wait for the bureaucracy to get their data act together.

Start local. Find the answers. Share the answers.

Break down the cryptic national data and put it into local context. Then working with social justice organizations in your area, share the data in a digestible format. Here are some examples of questions that should be easy to answer, but often are not.

• What police departments exist in your area?
• Is the police department demographically representative of the local area?
• Has the police department been involved in a shooting?
• Is there evidence that suggests racial profiling in arrests?

Adapting Data for Social Sharing

Lots of potentially useful data gets buried in poorly designed public websites and warehouses.

But if you are the kind of person who can adapt this information into useful guidance, it’s worth time and effort to explore these public sources. Yes, it would be great if our public organizations prioritized useful dissemination efforts. But we can’t wait for that happen.

Here is one way you can take data you find and adapt it into a useful format.

• Think of an audience you would like to reach. For example, a local mayor’s office or county executive.
• Think of a finding you believe might influence or move them to action.
• Find a simple template in Canva for a social media post or presentation.
• Create an image that shares data for that specific audience. Annotate that data, mention the audience and others who you believe could help the chart reach its intended target.

Rubric Supported Digital Content Analysis

In the modern era, websites and social media streams are the digital manifestations of a police department’s strategy. You “should” be able to find quite a bit about the policies and overall composition of an agency through reviews of publicly available information.

A scroll through a website is not a bad place to start deciphering a department’s strategy. But as an evaluator, you can do it systematically.

Create a basic rubric. It could be as simple as a list of questions you have about the department’s policies. (Such as, does this police department employ the use of body cameras or does it train on de-escalation techniques?). Then systematically go question to question, sourcing the specific answers as you go. Also rate based on the completeness of the answers (such as answered, partially answered, incomprehensible, or no information provided).

You could then take a further step, sharing your findings with the police department and asking them to fill in the blanks. Or you could report as it is, since complete or not this is the public face of the department.

Wondering what you might want to review? Building off the work of others is always the best place to start when available. Again I will advocate for Campaign ZERO.

Modeling Systems of Oppression and White Supremacy

Evaluators love logic models.

But for the most part we tend to develop models based on program activities designed to bring positive change.

But what if you believe that the system is not broken and in need of a fix. It’s working exactly as designed (to help the rich grow richer and maintain white supremacy).

Therefore the working systems of oppression need to be broken.

Use the skills you have to model white supremacy. Instead of creating a theory of change, develop a theory of oppression. Understanding how our systems and societal structures perpetuate inequality and maintain white supremacy is a step towards breaking the harmful causal mechanisms.

Systematically Documenting Community Stakeholder Experience

The world is complex.

One way to jump into that is to develop super sophisticated, and often utterly confusing, diagrams or algorithms in an attempt to map out our analyze through the complexity.

But an easier way I find is to put a focus on the stakeholders of interest. Tell the story of a person showing how they navigate a system you are trying to change. Then do it over and over, with different people.

Be intentional, and systematic. Document your approach and your methods.

For instance, what if you started with a black male high school sophomore with college aspirations. Given his age and home life, start pulling together the descriptive data on outcomes and challenges.

Think through current events. What is the likelihood that he would join his peers in a local peaceful protest near a downtown church? Then maybe face tear gas and rubber bullets as the police attempt to clear the path for a presidential photo opportunity.

Human lives are complex. Stories are data.

Single stories can be powerful. Collections of stories built systematically over time integrating numbers, images, and video can be hard to ignore.

With cell phones everywhere, more individual stories are coming to light. Reporters report and move on, it’s the way their field is designed. But as an evaluator, you can collect and synthesize.

Evidence Supported Political Activism

You can, and should, be in touch with your political representatives.

Let them know what you think.

But if you want to strengthen your argument. Bring data to your calls, to your emails, to your tweets, and your comments.

What other ways do you think we can leverage our skills as evaluators to support anti-racist actions?

This was all just a brainstorm. There is so much more we can do. Would love to hear your thoughts.

Post originally published as a collection of 13 evaluation cartoons on March 25, 2014. Updated with way more context on May 27, 2020.

When you search “What is Evaluation?” on Google, you get the kinds of responses you might expect.

• The Wikipedia page…if a topic is important enough it will have a Wikipedia page. And whatever you may think about Wikipedia, it’s one of Google’s favorite sites.
• Dictionary pages. We did ask for a definition, right?
• Evaluation association definitions.
• Large government and NGO definition pages.

They are somewhat formulaic. Usually the definition is tied directly to a prominent author or theorist. Or it’s sourced to a recent article which sourced a prominent author or theorist. Then the organization expands upon that definition.

But why are you asking? Is a direct answer really all that important?

If it is, well then, here is a good one.

What is Evaluation?

Evaluation is the process of determining the merit, worth and value of things, and evaluations are the products of that process.

Michael Scriven – Evaluation Thesaurus, Page 1

Now for the indirect answer, let’s dive deeper.

Evaluation can be hard to explain.

Back in 2014 the American Evaluation Association put together a task force with the purpose of defining evaluation.

 The statement is meant to encourage dialogue — so based on comments and responses it will be revised periodically. Thus, your reactions and comments are encouraged (see comment section below).The Task Force was comprised of both long-time evaluation professionals and AEA members newer to the profession. All have experience and expertise in communicating to others about evaluation. The task force included:

Michael Quinn Patton, Chair, Edith Asibey, Jara Dean-Coffey, Robin Kelley, Roger Miranda, Susan Parker, and Gwen Fariss Newman

The American Evaluation Association’s “What is Evaluation?”

The Canadian Evaluation Society took a similar root to find their own.

(Through a reflective process, the CES Board of Directors, supported by a consultation of members, has crafted and adopted the following as the CES definition of evaluation. PDF version. Cheryl Poth, Mary Kay Lamarche, Alvin Yapp, Erin Sulla, and Cairine Chisamore also published Toward a Definition of Evaluation Within the Canadian Context: Who Knew This Would Be So Difficult? in the Canadian Journal of Program Evaluation, vol. 29, no. 3.)

The Canadian Evaluation Society’s “What is Evaluation?”

For most of us “What is Evaluation” is an open question. It evolves over time and adapts based on context. But I am not sure evaluators would have it any other way.

Research vs Evaluation

From this perspective, evaluation “is a contested term”, as “evaluators” use the term evaluation to describe an assessment, or investigation of a program whilst others simply understand evaluation as being synonymous with applied research.

The Evaluation Wikipedia Page (accessed May, 27, 2020)

I’m pretty sure the Wikipedia evaluation page editor is trying to call us out.

But honestly, there are a lot of converted researchers in evaluation. And there are a lot of “evaluators” who are really just doing research.

I was a converted researcher. I saw the similarities in the methods and thought that it was pretty much the same thing. But it’s not what you do.

The how, where, who, and why really matter in evaluation.

Evaluation is not…

Research: The purpose of research is to generate new knowledge, while evaluation is about making evaluative claims and judgments that can be used for decision making and action

From FSG’s What is Evaluation, Really?

Figuring out the Type of Evaluation

Just like there is not one definition, there is not just one type of evaluation or one way to do an evaluation.

Evaluation can range from being very simple service evaluations to complex evaluative research projects. Each service will require a different approach depending on the purpose of the evaluation; evidence base, stage of development, context of the service; and the resources and timescales for the evaluation.

Evaluations can focus on implementation and learning (formative evaluation), how a service works (process evaluation) and whether it has worked (outcome/summative evaluation) – or all of these aspects over the life cycle of a project.

A lot of organization’s boil evaluation down to these three kinds of pursuits.

On BetterEvaluation, we use the word ‘evaluation’ in its broadest sense to refer to any systematic process to judge merit, worth or significance by combining evidence and values. 

But there are all sorts of types of evaluation out there in the world.

Here are some thoughts on the topic by Emily Elsner, a current member of the freshspectrum Panel of Experts.

Michael Quinn Patton defines evaluation thus:

Evaluation involves making judgements about the merit, value, significance, credibility, and utility of whatever is being evaluated: for example, a program, a policy, a product, or the performance of a person or team.’

Evaluation Facilitation

Evaluation seems to raise two assumptions in people: firstly, that there is an easy ‘off-the-shelf’ solution, and second, that evaluation is going to be critical and negative. The angle of judgement, and (as Patton elaborates in his book) the association of judgement with values, is a crucial aspect that can be forgotten. Yes, evaluation can be critical, but it can also provide strategic guidance, support decision-making, and more – all positive, useful things for projects and organisations. 

Linked to this, if evaluation is to be supportive of projects and organisations, then it needs to be tailored to the project/organisation, otherwise it is just measuring for the sake of it, as Muller, in his book ‘The Tyranny of Metrics’, reminds us: ‘

There are things that can be measured. There are things that are worth measuring. But what can be measured is not always worth measuring; what gets measured may have no relationship to what we really want to know […] The things that get measured may draw effort away from the things we really care about. And measurement may provide us with distorted knowledge – knowledge that seems solid but is actually deceptive. 

The Tyranny of Metrics

Emily Elsner is an impact and evaluation consultant based in Zurich, Switzerland. She has spent the last few years working in the migration-entrepreneurship-social enterprise space, and is now independent, balanced between the social and environmental spheres.

Overcoming the inherent tension between the evaluator and program being evaluated.

Who gets to say what works and what does not work? What does it all really mean? It’s definitely not hard to move back and forth between evaluation and philosophy.

Evaluation and evaluative work should be in service of equity.

The first principle of the Equitable Evaluation Framework

 Focusing on the Right Outcomes

We are what we measure.

UNEG’s definition of evaluation further states that evaluation “should provide credible, useful evidence-based information that enables the timely incorporation of its findings, recommendations and lessons into the decision-making processes of the organizations and stakeholders

The United Nations Office of Drugs and Crime – What is Evaluation?

 Connecting the Dots

Evaluative reasoning is the process of synthesizing the answers to lower- and mid-level questions into defensible judgements that directly answer the high-level questions. All evaluations require micro- and meso-level evaluative reasoning… not all require it at the macro level.

Jane Davidson’s UNICEF Methodological Brief on Evaluative Reasoning

Making Comparisons

In its simplest form, counterfactual impact evaluation (CIE) is a method of comparison which involves comparing the outcomes of interest of those having benefitted from a policy or programme (the “treated group”) with those of a group similar in all respects to the treatment group (the “comparison/control group”), the only difference being that the comparison/control group has not been exposed to the policy or programme. 

EU Science Hub’s Counterfactual Impact Evaluation

 Systematic Assessment

Effective program evaluation is a systematic way to improve and account for public health actions by involving procedures that are useful, feasible, ethical, and accurate. Several key documents guide program evaluation at the CDC.

The CDC Approach to Evaluation

At various times, policymakers, funding organizations, planners, program managers, taxpayers, or program clientele need to distinguish worthwhile social programs from ineffective ones, or perhaps launch new programs or revise existing ones so that the programs may achieve better outcomes. Informing and guiding the relevant stakeholders in their deliberations and decisions about such matters is the work of program evaluation.

From Rossi, Lipsey, and Henry’s book Evaluation, A Systematic Approach

Evaluation is the systematic assessment of the operation and/or the outcomes of a program policy, compared to a set of explicit or implicit standards, as a means of contributing to the improvement of the program or policy.

From Carol Weiss’ book on Evaluation from 1998.

Impact Assessments

Impact analysis is a component of the policy or programming cycle in public management, where it can play two roles:

Ex ante impact analysis. This is part of the needs analysis and planning activity of the policy cycle. It involves doing a prospective analysis of what the impact of an intervention might be, so as to inform policymaking – the policymaker’s equivalent of
business planning;

Ex post impact assessment. This is part of the evaluation and management activity of the policy cycle. Broadly, evaluation aims to understand to what extent and how a policy intervention corrects the problem it was intended to address. Impact assessment
focuses on the effects of the intervention, whereas evaluation is likely to cover a wider range of issues such as the appropriateness of the intervention design, the cost and efficiency of the intervention, its unintended effects and how to use the experience from this intervention to improve the design of future interventions.

OECD’s What is Impact Assessment

We need to accept the fact that what we are doing is measuring with the aim of reducing the uncertainty about the contribution made, not proving the contribution made.

John Mayne Addressing Attribution Through Contribution Analysis: Using Performance Measures Sensibly

Want more evaluation cartoons?

If you were brought here because of the evaluation cartoons, you can find lots more by checking out the following post: 111 Evaluation Cartoons for Presentations and Blog Posts. The post will also provide you with information on licensing terms and use.

So I decided to create what I hope will eventually become an ultimate list of evaluation blogs. Not just a boring bullet point list, but one that gives you a sense of the human being on the other side of the internet.

This post is designed to grow. Meaning I plan to come back and update it. There are already a bunch of evaluation blogs I know well that are not on this page yet. But give it time and they will be added.

This post is in three sections.

• Section I includes recommendations from the FreshSpectrum Panel of Experts
• Section II includes all of the bloggers currently part of the main Eval Central blog feed. I’ve pulled a quote from a post for each blogger, and subsequently cartoon illustrated the quote.
• Section III will eventually include other evaluation blogs.

If all goes as planned, eventually this particular blog post will get very, very, large.

Section I: Blog Recommendations from the FreshSpectrum Panel of Experts

Alright, so these recommendations come from my awesome panel of experts (which is still open for you to join).

Sue Sing Lim

One of my favorite blogs is “We All Count”, a Canadian evaluation firm, who provides useful and practical tips on how to implement equitable evaluation. The founder, Heather Krause, is very generous in sharing her experiences on her successes and challenges when she tried to implement that. 

At We All Count, we agree that Big Data is a valuable resource but we think there are some very important concerns that Big Data alone won’t fix. We think that what’s really exciting about Big Data is the ability to combine the efficiency and power of large datasets with the intentionality of small, curated data samples. 

Why Big Data Needs Small Data

Her blog posts stand out because I found her voice easy to understand and also practical. I don’t feel it is overwhelming or too abstract. I feel like I can take away or do something after reading them and this gives me a sense of empowerment. 

Highly recommend! 

Sue Sing Lim joined Kansas State Research and Extension (KSRE) SNAP-Ed program in 2016 in the role of program evaluator. She is responsible for designing evaluation plan, overseeing data collection, creating evaluation training and workshops, analyzing data, and creating reports to disseminate the results of program impacts. 

Jon Prettyman

Many thanks to Chris for the opportunity to share (and the chance to add lots of new content to my feedly subscriptions)! I recently found Marcus Jenal’s blog, where he comments on complexity and its application in social change processes. It’s worth a look if you’re interested in applying complexity concepts to your evaluation practice. 

I know that in complex systems things are never that neat and never linear causal – there is not one thing in one box that leads to another thing in another box or to an observed behaviour. Reality is messier. I also missed the dynamics in these diagrams – how are these structure created, how do they persist, how do they change?

Systemic change: A dance between structures and events

Jon Prettyman joined the monitoring and evaluation team at Climate-KIC in 2019. He designs and manages evaluation strategies for the portfolio of systems innovation projects. Before joining Climate-KIC, he worked with Mercy Corps on efforts to use emerging technologies for evaluation. 

Christina Gorga

While not entirely related to evaluation, I’m really digging data-based design that’s created across different platforms. Judit Bekker out of Budapest has been producing some killer work with Figma, Tableau, and Adobe Illustrator that gives me lots of inspiration for my own work and how to push beyond default settings. She also gives insight on web safe fonts for Tableau as well as color considerations for passing contrast ratio tests.

Using fonts in Tableau can be a tricky thing because only Tableau web safe fonts will show up the same for everyone. A web safe font is a font that is considered to be a ‘safe bet’ to be installed on the vast majority of computers. Every computer that has a browser installed has default fonts built in so that it can display the text on the web.

How to use fonts in Tableau?

Christina Gorga is a data visualization designer and strategist in Booz Allen’s Health account. She has experience designing reports and interactive Dashboards for program evaluations, state healthcare agencies, HHS, CMS, and VA. She is also an active Tableau community member and loves training teams how to use it. You can reach out to her on LinkedIn or follow her on Twitter at @styleSTEAMed.

Marianne Brittijn

I follow Zenda Ofir, a South African evaluator based in Geneva (https://zendaofir.com/) who plays an active role in the South African Monitoring & Evaluation Association and blogs about current trends and debates in the sector. I discovered the quirky and wise developmental evaluator Carolyn Camman, based in Vancouver,  through their Eval Café Podcast and only read their first blog post today (but have been following them on Twitter for a while).

Two – We have to do much more to display the full value of evaluation for a new era. As the COVID-19 pandemic races around the world, evaluation struggles for space. Research studies and data overwhelm, yet evaluation professionals and studies are not present at influential tables. We have fumbled in proving the value of evaluation for the challenges facing humankind. Let us do our best to show the value of evaluation once the immediate heat of the pandemic is over and we move into sense-making in a changed world.

Transforming Evaluations and COVID-19, Part 4. Accelerating change in practice

Evaluators interested in anything related to developmental evaluation and equity should pay attention to Carolyn (http://www.camman-evaluation.com/).  I also love the ARTD blog (https://www.artd.com.au/read/our-blog/). They’ve been putting out highly practical content that speaks to how organisations can (and should) adapt their M&E during the COVID-19 crisis. The same is true for Feedback Labs (https://feedbacklabs.org/blog/).

Marianne Brittijn is a Monitoring & Evaluation (M&E) practitioner in the development and social justice sector. She conducts external evaluations (ideally participatory and developmental ones), develops organisational M&E systems and facilitates M&E training courses. In addition to her consultancy, she works as a part-time PMEL Officer for CORC and the South African Shack Dwellers International Alliance.

Section II: Eval Central Bloggers

All of the bloggers in this section can be followed via Eval Central.

In order to be part of the main Eval Central blog feed I require permission from the blogger. If you have a blog that you want included, you can submit it for consideration.

I also included myself, because it felt weird not to include myself. Although it also felt weird including myself. I ended up just highlighting a really old blog post (circa 2012).

Amanda Klein

It’s not always easy to measure the impact of family and community engagement efforts. Some aspects of education — like test scores or report card grades — (notwithstanding the wide variety of controversies around their use) are pretty straightforward to measure. They’re already quantified. They’re known entities. We can tell that story. But when we talk about measuring the impact of, say, a super successful family science night, our minds go blank. 

What Will Be Your Story?

Ann K Emery

We also wanted the information to be actionable (duh). We wanted to design a one-page meeting handout that was not only clear but would also give the leaders something to talk about together.

How to aggregate information across sites.

Ann W Price

That’s why I find logic models so darn helpful. They may be despised by some, but I believe they are despised because they are oftentimes overly complicated. (I certainly have been guilty of creating a few that were way too complicated myself). But I have experienced over and over again a situation in which the program staff and leaders just knew they could explain their program clearly. Until we went through a logic model process, and they couldn’t.

A Failure to Plan…You Know the Rest of the Story

Betsy Block

Maybe because I’ve only ever seen hot air balloons from a distance, my memory of them leans towards vibrant orbs, sometimes illuminated, gracefully soaring in the air. I kept working through this image of the hot air balloon, and thinking about what goes into a successful flight. What came to mind was the construction of the balloon itself, how it is sewn, the importance of fabric; and that my personal mission is to be a weaver of a fabric for a stronger community.

It’s in the cut of the cloth.

Beth Snow

a lot of the value of clarifying a program theory comes from the process. Finding out that people aren’t on the same page as one another about what the program is doing and why, identifying gaps in your program’s logic, surfacing assumptions that people involved in the program have – all of this can lead to rich conversations and shared understanding of the program among those involved and you just don’t get that by handing someone a description of a program theory that was created by just one or two people.

Evaluator Competencies Series: Program Theory

Chris Lysy

The null hypothesis is immensely powerful. It doesn’t have to be proven, it just is.

You don’t have to explain why you’re using Word to write a report or Power Point to give a presentation. You don’t have to explain why you present at conferences or write for a journal. They are already accepted, they are the null.

Creative approaches are never the null.

I am the null hypothesis.

Carlos Rodriguez-Ariza

This moment is also a challenge for those of us who work in evaluation functions in the field of international development, where we generally have the luxury of time, operate within clear theories of change, and do our best work when we can mix methods, use multiple data sources and conduct in-depth interviews with a variety of stakeholders .
[translated from Spanish]

Evaluando en tiempos de pandemia

Carolyn Camman

About a year ago I was chatting with someone who had been learning to work with wood. He said found it powerfully healing because you do make mistakes and you can’t reverse them. The mistakes become part of it. You just keep going. I’ve also been taking improv classes and learning the same thing. Whatever happens, you work with it. Fix it by moving forward, not by trying to roll it back and erase it.

Working from the Mistake

Cameron D. Norman

Health. Lastly, how well are we? When the effects of being inside, isolated, and perhaps exposed to a virus are real, present and pervasive, your audience might not be in the state where the depth and quality of thought are what we need to get the responses we want. Many of us are not our usual selves these days and our responses will reflect that.

Better Data Collection

Dana Wanzer

RoE is not conducted for the sake of conducting it, nor is an evidence base of research important unless it is useful and used by the intended audience—in this case, practicing evaluators.

What is Research on Evaluation (ROE)?

Elizabeth Grim

For four years I’ve been pondering – Who am I as an evaluator? Am I even an evaluator? Or am I a social worker with an evaluation and data-driven mindset? Can I be both? Am I a policy analyst who implements data-driven approaches? Am I an advocate that uses data to drive change? How many hats can I wear before my professional identity is so dispersed that it is nonexistent? Does claiming a professional identity even matter?

Wearing Many Hats: Where Does the Role of Evaluator End?

Eval Academy

Recently I was asked by a client about an evaluation literacy course for its board. The client’s board members had just attended a strategic planning day and through that discussion felt they needed education on evaluation and metrics. On one hand I thought “bravo, they want to know more about evaluation!”; on the other hand I thought “shit…., I’ve totally failed them as their evaluator – what have I been missing?”

Strategic Learning and Evaluation – What Boards Need to Know

Katherine Haugh

Michelle Molina

Having a clear understanding of the purpose of the sort of data you are collecting will help you focus on the type of data you should be collecting and the sorts of conversations you should be having when you are making sense of that data.

Three Types of Evaluation for Nonprofits (Simple Overview)

Nicole Clark

Before transitioning, I knew what the end game was: to live life on my terms and help organizations raise their voices for women and girls of color. Once that transition happened, it became harder to stay motivated because everything I was doing was for someone else.

What I’m learning along the way is that it’s ok to question a dream. Which can be difficult when you’ve had a tunnel vision on that dream for so long. I’ve also learned that it’s ok to give yourself permission to try.

Ask Nicole: Why Am I Doing This?

RKA

Numbers had their purpose (they are easily gathered and understood), but they have outlived their usefulness. The silver lining: with nary a visitor to count inside the building, is now not the perfect time to rethink and change how your museum measures success? How does a museum arrive at metrics that will stand the test of time?

Zero

Thomas Winderl

Instead of action language, use change language.Change language reports on the results of an action instead of the action itself.

An example of action language is:

“150,000 girls know how to protect themselves against HIV infection with the support of XYZ”

Use Change Language in Reports

Section III: Other Evaluation Blogs

Alright, so I know that right now I am missing a bunch of evaluation world favorites. And I could bounce around from list to list finding and pulling them together for this post.

But I think blogs are better shared when they come with advocates. Either the bloggers themselves, or others in the evaluation community who love their work.

So if you want your favorites listed here, be their advocate and write a comment.

Like I said at the beginning of this post. This page is intended to grow.

Social media is not a monologue. It’s a collection of voices seen (or unseen) based almost entirely on who you do or do not follow.

But the world we see is based, as it always has been, on the people that surround us. Historically this has been primarily determined by where we live, and the communities where we interact. But increasingly, what we see it is based at least somewhat on who we choose to follow in the digital realm. And who we choose to invite into our own conversations.

I am an on and off lurker* of the American Evaluation Association’s Evaltalk listserv and recently stumbled into this comment by Doug Fraser.

It raises the same question Bill Fear has tried to raise a couple of times, namely: why, at this of all times, when the world faces the biggest and most vital evaluation challenge in its recent history, is the evaluation profession missing in action?

I don’t see it that way.

I see evaluators across the globe stepping up. They are using their approaches, methods, and other tools in their toolkits to make contributions. And they are using their voices to share their perspectives and guidance across the globe.

In this post I wanted to create a compilation of some of those viewpoints. And as I collected, I found that I could have kept going and going.

If you have read (or written) something recently that inspired you, please share in the comments.

*EvalTalk has a long history, but listserv technology has a way of elevating the perspectives of those members who prefer shouting to discourse. Subsequently, I find many of the conversations lacking sufficient diversity in both participation and perspective. But I digress.

When Was There Certainty?

The pandemic has laid bare that which many of us (meaning people of the global majority that live in the United States) have always known: this country is designed to maintain a power dynamic that privileges white male power and wealth at the sacrifice of most everything else.

Jara Dean-Coffey in A note from EEI’s Director: “I have feelings.”

On May 13, at 12PM Eastern/3PM Pacific, Jara will be joining me for an Eval Central UnWebinar to talk about being of service when you are not essential. Join us!

The Lines Between Work and Life and Life and Work

The resources designed to help us adapt to Covid-19 don’t match up with our lives right now. Every day, I sit down at my kitchen table — the same table where I (used to) host dinners and put together puzzles — in front of a make-shift workstation where I do my job. My living space is also my workspace. We’re managing a whole new definition of work-life balance right now and it turns out work is part of our lives.

Alissa Marchant in 18 Resources helping me in work and life with Covid-19

Don’t Overthink, Just Do

@timbidey/@traversepeople Tim is an experienced qualitative researcher with a passion for helping charities explore what works (and what doesn’t) and why to inform their project design and practice.

It’s clear that many of the voluntary and community sector organisations that Traverse works with are struggling at the moment in the UK. Charities have had to adapt to continue delivering frontline services or develop new ones to meet emerging needs in a world of new public health restrictions – all amid a catastrophic loss of funding and, in some cases, lack of staff where people have been furloughed.

In some cases, evaluation has fallen fast down the list of priorities – but it’s important to remember that ‘evaluation’ in itself is not a homogenous practice. Sure, now is not the right moment for continued impact evaluation of multi-year programmes, but evaluation has many faces and need not be so traditional or comprehensive.

My advice to organisations has been to keep it simple. Evaluation is so often seen as a ‘mystifying practice’, but now, more than ever, it’s better to collect something than not collect anything through fear of being seen as unsystematic.

After all, evaluation’s value lies in its utility – it needs to serve the information needs of its users. Right now, the needs of voluntary and community sector organisations demand real-time data to inform weekly decisions about delivery, rather than demonstrating the differences that they’ve made for funders or members of the public. 

Evaluation for this purpose does not need to be theory-based with perfectly rounded edges, it just needs to capture ‘good enough’ data about the essentials on a regular basis. For new or developed services these might include: what is the problem? Who is affected, how? Who are we reaching or not reaching – why? What do people think? What, if any differences are we making? What do we need to do to improve?

A sense of intended outcomes, basic monitoring data, simple service user feedback or even reflective, anecdotal data from staff can all provide ‘good enough’ insights into these questions for the situation at hand – so long as people remain honest with themselves (and others) about how insights were generated and what limitations sit behind them. 

So don’t overthink, just do.

Tim Bidey of Traverse is part of the FreshSpectrum Panel of Experts. These were his words.

Becoming Developmental Evaluators

All evaluators must now become developmental evaluators, capable of adapting to complex dynamics systems, preparing for the unknown, for uncertainties, turbulence, lack of control, nonlinearities, and for emergence of the unexpected. This is the current context around the world in general and this is the world in which evaluation will exist for the foreseeable future.

Michael Quinn Patton in the Evaluation Implications of the Coronavirus Global Health Pandemic Emergency

Because Our Decisions Have Consequences

 Some of us want to do everything we can to stop the spread of the pandemic and minimize the overall harm it will cause. Others of us are more concerned with managing the indirect effects of the crisis on communities or causes we care about. Still others of us are just trying to figure out the role we can and ought to play. All of us, though, can benefit from approaching these challenges with thoughtfulness and rigor.

Ian David Moss in Deciding Well in Tumultuous Times

Changing Definitions

But now, with the uncertainty of what visitation will look like over the coming months and potentially years as museums phase into reopening with limitations on visitor capacity and new social distancing measures, I wonder what does a “representative sample” mean now?  

Katie Chandler in Sampling: What does “representative” mean during and after coronavirus?

Who is Afraid of Rigor?

Bio: @b3consults/www.b3consults.com  Betsy brings all the tools of data & program evaluation in harmony with the heart and intuition-led world of coaching, to increase the impact of results-driven organizations.

My gut reaction, though I’m scared to put it out there, is two fold.  One, what about the organizations that were ready for the pandemic? Two, why shouldn’t we see a call for increased rigor?

To the first: I believe some organizations planned for something along the lines of a global pandemic.  As early as 2012, some organizations were making the case that a COVID-19 like event was coming.  The crisis was foreseen, and some organizations were ready for it. 

To the second: I’ve seen implied that we should be more adaptive and iterative, in a sense softening the rigor as organizations lean into change and adapt.  
But what if COVID-19 isn’t a one-time event (and most experts will tell you it isn’t)? And what if our ability to measure, with a high degree of rigor, how organizations served critical populations during this time will be crucial to inform what we need to do next time? 

I’m an unlikely architect of highly rigorous evaluation design, I tend to focus in the PrEvaluation space. However, when I think of my clients, the ones that know that their ability to rise up and serve critical populations well NOW will prevent further exacerbating equity issues in the future, I want to hear the kind of evaluation for which they would strive.  And I would hesitate to assume they want less rigor, or that they are afraid of rigor in a crisis. 

Ask yourself: what if they crave rigor right now?

Betsy Block of B3 Consults is part of the FreshSpectrum Panel of Experts. These were her words.

Responsible COVID-19 Data Visualization

Amanda said, “There are different points in which we make decisions about how and what we visualize, and then how we publish and share. What are we creating and doing more for our own exploration and understanding? And what are doing so that we can share it with the public to help others make sense of information?”

From Ann K Emery’s interview with Amanda Makulec in Visualizing COVID-19 Data Responsibly: An Interview with Amanda Makulec

Evaluation Contingency Plan

We’re often providing funders an evaluation plan that includes best case scenario and prioritize in-person interactions. Rarely do plans require the evaluation team to offer contingency options. As many of us are now tailoring methods to respond to social distancing and travel recommendations, we’re switching to virtual interviews and other methods of web-based data collection. Building contingencies into future evaluation plans will leave us better prepared to pivot and could save time and resources on creating post-hoc plans.

Martena Reed in Reflex or Reflection: Three Lessons for Evaluators Amid COVID-19

Changing Your Data Strategy

Consider what sort of data collection activities are going to help you use your data and collect data that is useful.

Michelle Molina in her Video on Nonprofit Data Adjustments to COVID-19

Forget Returning to Normal

So no, I don’t want us to return to normal. I want us to use this an opportunity to change, to create systems and social structures that create deep and lasting equity and a world where we work together for the common good. One can dream, right? If anything, this crisis should teach us that we are all connected. 

Ann Price in There are words I really hate right now.

The Importance of Our Work

We work to support better evaluation globally. Good evaluation helps people identify the information they need and make sense of it.  It helps inform decisions about what to do and how to improve results. Good evaluation is essential to guide the best use of resources and to ensure accountability and learning. During this pandemic and in the post-pandemic world our work is more important than ever.

Patricia Rogers in BetterEvaluation COVID-19 Statement

Compensating for a Lack of Monitoring

Una evaluación en tiempo real (RTE) está diseñada para proporcionar retroalimentación inmediata (en tiempo real) a aquellos que planifican o ejecutan un proyecto o programa, para que puedan realizar mejoras. Esta retroalimentación generalmente se proporciona durante el trabajo de campo de la evaluación, en lugar de después.

Carlos Rodriguez-Ariza in La evaluación en tiempo real en emergencias

[English Translation] A real-time evaluation (RTE) is designed to provide immediate (real-time) feedback to those who plan or execute a project or program, so they can make improvements. This feedback is generally provided during the assessment fieldwork, rather than afterwards.

Access for On-Site Data Collection

On 1 April 2020, USAID and IDEAL hosted a webinar on ‘Challenges and Strategies for Monitoring and Evaluation (M&E) in the Time of COVID-19’. The virtual meeting was attended by 500 M&E professionals.

The participants of webinar completed a poll that yielded the following results.

Ann-Murray Brown in A New Dawn: Monitoring and Evaluation during COVID-19

Before Expanding Boundaries

This is the smallest visible system (SVS) in which you can make a difference. Once you can act wisely on this system, you can expand the boundaries and scope to work larger.

Cameron D. Norman in Acting in Complex Times

Looking to the Desired Present

I’m looking to the desired present instead of a desired future. Not because I have no hopes or aspirations for the future, but because I don’t find it helpful right now to aim for something I can’t see. I don’t know what the future will hold. I don’t know where this moment goes. I’m hoping there’s a future out there so different from this one that I can’t even imagine it fully much less trace a path to it by design. All I want to do is find the best part of whatever moment I am in, and work with that.

Carolyn Camman in Entering the Clearing