Welcome to our monthly roundup of new and noteworthy evaluation news and resources – here is the latest.

Have something you’d like to see here? Tweet us @EvalAcademy or connect on LinkedIn!

---

New and Noteworthy — Reads

---

Last month we highlighted Khalil Bitar’s blog post from May 11 where he talked about how the evaluation community is not immune to prejudice, discrimination and racism. He went on to say that the evaluation community is in fact practicing racism by primarily producing and sharing evaluation knowledge by white men from the Global North. Two weeks after this blog post George Floyd was killed. George Floyd’s killing ignited protests across the world, but it also ignited more reflection and conversation about what could be done about systemic racism in our societies. Evaluators reflected through their blogs on how evaluators can begin to move the needle. Some relevant blog posts to check out include:

 

Jara Dean-Coffey – Musings + Machinations

Jara Dean-Coffey is the Founder and Director of the Equitable Evaluation Initiative (EEI). If you haven’t checked out the EEI then make a point to do so. There are resources that evaluators can access to better understand how evaluation can and should be “utilized in a manner that promotes equity.” But as Jara stated on LinkedIn, she felt the need to write outside of EEI and recently began a blog called musings + machinations  – writings by jdc. In June she posted 12 posts! Read them all and make sure to subscribe to this blog – her musings are insightful and and her machinations inspiring.  

 

Engage R+D –  It’s Time to Let Go of Tired Narratives about Talent in Evaluation

We hear a lot about narratives these days. In this blog post on Engage R+D, Clare Nolan talks about narratives – what they are, the power they can have, and how they have been used to support oppression. She talks about some of the tired narratives about talent and expertise in evaluation that “get in the way of effective and equitable solutions” and then counters those with ideas of potential new narratives that can help advance equity. For example, we’ve all heard the “diverse applicants don’t meet our standard qualifications.” A potential new narrative is looking at how implicit bias and white-dominant norms are constraining our ability to recognize valuable knowledge, experience and credibility.

 

Michael Quinn Patton’s Rules for Privileged Straight White Males and Andrea Guerrero-Guajardo’s Rewrite

MQP published a blog post on his website that outlined rules for straight white males. He outlined ten rules that privileged white males don’t necessarily have to like or agree with but do have to follow. If you scroll down to the comments you can find many people who definitely disagree and not in a constructive manner. Someone who did provide constructive feedback was Andrea Guerrero-Guajardo who put time and thought into rewriting MQP’s rules, which can be found here.

 

Chris Lysy –  Evaluation, Compassion, Fatigue and Health Inequity

Chris’ blog post talks about a number of different topics, as the name suggests. He explores how as evaluators we need to not only do something but stand for something. He suggests the Equitable Evaluation Initiative as a resource and approach for how evaluators can channel their efforts and keep the momentum moving forward because “failing to channel our efforts can quickly lead to fatigue.” In this blog post, he also explores inequity in public health data.

---

New and Noteworthy — Tools

---

Code for America’s Quantitative Research Practice Guide

Code for America has come out with a timely guide that can be used by anyone looking to conduct qualitative research “in ways that can help everyone ensure that their products and services are as inclusive as possible.” The guide outlines specific methods for conducting research and analysis, but to be honest my favourite part of this guide is Code for America’s core research philosophy and guiding principles. The fact that this organization has a research philosophy and guiding principles is swoon-worthy enough, but how they have articulated their philosophy and guiding principles is so on point I found myself re-reading it several times. They conclude their research philosophy by stating: “Ultimately, research is a tide that lifts all boats. It is fundamental to developing government services that better and more equitably meet the needs of communities. Raising the bar on quality of research raises the bar on quality and effectiveness for everything that we seek to do for the world.” Well said.

---

New and Noteworthy — Courses, Events and Webinars

---

July 2020

1. Results-based Management & Theory of Change Workshop during and after COVID-19

   Instructor: Mosaic.net International Inc.

    

2. IPDET Evaluation Hackathon

   July 7 – 13

3. Claremont Graduate University – The Evaluator’s Institute

   A variety of courses starting July 20 that are being conducted by various instructors, including some big names like Michael Quinn Patton, Ann K. Emery, and Ann Doucette.

---

We have a free guide:

Applying the JCSEE Evaluation Standards in Practice

Whether you’ve read The Program Evaluation Standards cover to cover or not, you may be wondering how to ensure you’re applying them to your evaluation practice. This free digital download will give you the reflective prompts you need to ensure your next evaluation project incorporates all 30 Standards.

---

Subscribe to Our Newsletter

Sign up with your email address to receive news and updates from Eval Academy every month.

Email Address
Sign Up

We respect your privacy.

Thank you!

---

 

 

A recent study in the American Journal of Evaluation showed how three simple visual design principles could be applied to logic models to make them more effective and understandable. This article summarizes the findings of that study so you can improve your logic model.

 

What are logic models?

Logic models are widely used in evaluation to visually summarize how a program is expected to work: what resources will be used, what activities will be undertaken, and how those activities will cause desired outcomes. These are often a staple in our evaluation work, whether we are starting with a logic model that has already been created, or we are creating one for the program from scratch.

 

Why visual design?

Visualizing information effectively is important for evaluators because we are “communicators and knowledge brokers” at our core. The way we visualize and present information affects how it is used, and who is able to use it. Therefore, logic models could be made more useful and understandable by improving their visual design. The goal of the study was to make a logic model better by applying visual design principles. A good visualization:

• Is clear, useful, and memorable;

• Supports audience understanding;

• Is easy to mentally process; and

• Is organized into memorable chunks.

These visualization principles, along with the relevant research on data visualization and visual design, were used to make some simple (yet effective) improvements to the basic logic model.

 

Visual improvements to the logic model

The original logic model looked like this:

This is a fairly standard logic model that evaluators are used to seeing. It outlines the program’s inputs, activities, reach, and outcomes, and shows the interconnections between them.

This logic model was revised by incorporating the following visualization best practices:

1. Colour: Colour was used to group similar items together (i.e., each column was a different colour)

2. Proximity: Elements that were connected to each other were moved closer together, which reduced the emphasis on arrows

3. Reducing ink: unimportant elements were de-emphasized or removed to keep the focus on the most important elements. For example, by removing the black borders around boxes.

After making these improvements, the revised logic model looked like this:

 

The revised logic model was easier to understand

After testing these logic models with a survey of the general U.S. population, the researchers found that the revised logic model was:

• Faster to review

• Interpreted more accurately

• Easier to understand

• More aesthetically pleasing

• Perceived as more credible

 

With relatively simple visual changes, which can be implemented in Microsoft Word or PowerPoint, the researchers were able to improve their audience’s understanding of the logic model. We can leverage these 3 design tweaks (colour, proximity, and reducing ink) to help our audiences read our logic models faster and more accurately.

---

Notes on study methodology

The original study (linked below) has more detail about how exactly the research was conducted, but in short:

1. A survey was conducted with the general U.S. public.

2. Respondent were shown one of six possible variations of the logic model, which were combinations of:

   • Original vs. revised logic model

   • With vs. without a written narrative

   • Greyscale vs. colour logic model

3. Respondents were then asked a series of questions about the logic model they were shown, including their understanding of the program, the effort it took to understand, their perception of the credibility, and the aesthetic qualities.

4. Responses to the different types of logic models were compared to determine if the visual design elements helped improve the original logic model (and it did!).

 

The full study can be found here:

Jones, N. D., Azzam, T., Wanzer, D. L., Skousen, D., Knight, C., & Sabarre, N. (2019). Enhancing the Effectiveness of Logic Models. American Journal of Evaluation, 1098214018824417. https://doi.org/10.1177/1098214018824417

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---

 

 

Utilization of evaluation results can be underwhelming to the well-intentioned evaluator. Time and time again, we hear of people going through an evaluation only to be disappointed that the findings didn’t give them the answers they wanted. It is such a big problem that Michael Quinn Patton decided to design an entire approach to evaluation called Utilization-Focused Evaluation (UFE). I’m going to save you from reading all 600-and-some pages of it and instead share three ways we at Three Hive Consulting help clients use the results from our evaluations.

 

1. Identify your A-Team and get them involved!

People will use information if it is the right information — in other words, the information they want or need for decision making. However, you need to find the “right” stakeholders to work with and to provide the right information to. It sounds simple enough, but the more complex the initiative, the more complex the number and type of stakeholders. You can try to meet the needs of everyone, but doing so often leads to not adequately meeting anybody’s needs. Unless you have unlimited resources, which in my experience is never the case, you will need to identify your A-team. In UFE these are called ‘primary intended users.’ Primary intended users are the stakeholders that have the willingness, authority and ability to use the findings. When you have identified your A-team, it will be much easier to design your evaluation  — people drive purpose and purpose drives design.

A stakeholder matrix is a simple tool that organizes:

• stakeholders,

• the group they belong to,

• what they see as the purpose for the evaluation,

• how they will use the results, and;

• how they want to be involved throughout the evaluation.

 

Below is an example:

Notice the last column, “Nature of Involvement.” Can you guess who the A-team is? Chances are your A-team are the service providers. They are the ones that have multiple ways they can and should be involved throughout the evaluation. I say should because the more involved your A-team is throughout the evaluation process, the greater chance they will use the findings in a meaningful way.

 

2. Tailor reporting to needs

One size does not fit all when it comes to evaluation reporting. Your findings need to be accessible and relevant to each stakeholder group, which means some extra work tailoring your reporting. One way to help meet the information needs of various stakeholder groups is to layer your content. In Kylie Hutchinson’s “A Short Primer to Innovative Evaluation Reporting”, she uses the analogy of a burger to explain that not everyone can digest the entire burger, so we need to layer in the fixings (i.e. executive summaries, one-page overviews, appendices, etc.) if people only want to digest some of the burger.

A few years ago, we worked on a project where we needed to do just that. We ended up layering our reporting by providing the full burger (i.e. the final report), but also producing a variety of other reports throughout the evaluation:

3. Stop being so boring! Facilitate use through interactive strategies

One of Three Hive’s core values is “intelligence having fun.” We don’t believe that evaluation should be a boring make-work project where some outsider comes in and tells someone what needs to be done, asks for data, tells them what is wrong and then recommends a bunch of things that aren’t feasible or relevant. There is very little chance that any learning will result from that approach. As we know, most people do not truly learn through passively listening or reading  — we learn by doing. This means that if we want people to truly understand and transform what they are doing, they need to be involved in the evaluation process. 

Jean King and Laurie Stevahn’s “Interactive Evaluation Practice” is a book I frequently refer to when I am looking for facilitation ideas. In it, they lay out what they call “An Evaluator’s Dozen of Interactive Strategies.” They go on to describe the materials needed, instructions for how to conduct the strategy, facilitation tips, variations on the strategy, and when they can be used throughout the evaluation process (see the figure below).

People may initially moan and groan at the idea of interactive activities, but in the end, they really do enjoy them. After putting the time in to use these strategies, I have received feedback that described the strategies as “engaging,” “productive,” and “approachable” ways to be involved in evaluations.

---

You’ll notice that the title of this article isn’t “Three simple ways to increase the chances your evaluation results will actually get used.” You do have to build in extra time and resources to take the time to: 

1. identify your A-team and involve them throughout the process,

2. tailor your reporting, and;

3. make things fun through interactive strategies.

However, the investment in these three areas will make a difference for how your clients understand and utilize findings and, as a result, your credibility as an evaluator.

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---

 

 

Evaluators need to write clearly for their work to be used. Although not preferable, the written evaluation report must stand on its own, clearly conveying the key findings and messages. The 9 mistakes below are ones that I’ve come across in my years of writing, editing, and reading evaluation reports. The first three are more evaluation specific, while the middle three relate more to writing style, and the final three are all about grammar. I’ve provided tips to help you prevent or correct these common errors.

1. Using Jargon

To explain why not to use jargon I, ironically, must first explain it. The Oxford Dictionary defines jargon as “special words or expressions that are used by a particular profession or group and are difficult for others to understand.” 

What is considered jargon depends on who your target audience is. If your audience is expected to have some technical expertise, using terms others may consider jargon can help them quickly identify what you are talking about. But, if your audience is general, it may be best to eliminate the jargon or clearly explain it and avoid repeated use of the word.

Jargon can be hard to identify. In evaluation, common jargon includes topics related to:

• Methods: e.g. mixed methods, formative evaluation, summative evaluation, systematic sampling, convenience sampling

• Analysis: e.g. data saturation, logistic regression, triangulation

• Interpretation: e.g. benchmark, generalizability

Tip: If you need to use jargon in a report that includes a general audience, explain the concept first, then use the jargon-y word. Using jargon first and explaining second can cause general audience members to disengage and feel like concepts are over their heads.

 

2. Using analogous words

In evaluation there are several words that can be used relatively interchangeably.

Common analogous words include:

• Program/project/initiatives

• Staff/employee/provider

• Patient/client

• Participant/respondent

Sometimes, which term is being used depends on your audience or stakeholder. Check if you are unsure. In rare cases, multiple words can be used within one program in which case its best to have your stakeholders agree which term will be used in the evaluation report (and to prevent a tracked changes war between stakeholders. Trust me, I’ve seen it happen).

Tip: ctrl+F can help you find and replace mistakes that slip by or when multiple people are involved in writing sections of the report.

3. Floating or repetitive quotes

When using quotes, tie them into the text without restating what the quote is about. Quotes should fit seamlessly into the narrative and be embedded within the surrounding text. Do not just restate what the quote says in the body of your writing.

For example, I often see:

 

“…Participants were happy with the training, especially the extra attention given to policies and procedures.      

‘I liked the training, especially the part where we learned about the policies and procedures.’ (Program participant)”

 

Tip: Quotes should be used for emphasis and to showcase the participants’ voice, not to explain an entire concept.

 

4. Exaggerating with Adverbs and Adjectives

Be very cautious about using hyperbole in your writing (see what I did there!). Modifiers that exaggerate your statements should be used with intention. Writing that is littered with exaggerative adjectives and adverbs tend to make people skeptical. Show rather than tell your audience why something is the most/best/greatest/very.

For example, instead of writing:

“Participants were very frustrated about x,” show the reader what you mean. “Participants consistently expressed frustration about x” tells us that this topic came up many times, while “Participants recalled specific times where they were frustrated about x” tells that this issue was described specifically rather than people just stating “x was frustrating.” In both cases, the examples provide a clearer picture of the frustration than just ‘very’ frustrated. Being more specific lends greater clarity to your writing and provides important details for your audience.

Tip: Writing with exaggeration is a habit that is best caught during the editing process. If you can catch yourself while writing, ask yourself what other descriptive word could be used. Most often, the adjective or adverb is unnecessary and can be removed.

 

5. Being Verbose

In school I often filled my papers with extra words and explanations to meet the page or word count or an assignment. In writing for evaluation, you want to do the opposite. The time and energy you put into creating clear and readable reports is wasted if you leave your readers to wade through your long-winded writing. Being succinct is a learned skill.

Tip: Use active voice (see more below) and edit, edit, edit.

 

6. Using Passive Voice

Active voice is when the person or thing doing the action comes first. Passive voice is when the person or thing being acted on comes second. Writing evaluation reports using active voice gives your writing a sense of energy and reduces your word count. It also makes it is clear who or what is doing the action.

For example:

• The participants were told about the interview by the program director. (Passive)

• The program director told the participants about the interviews. (Active)

Tip: Always think who did what. Be assertive in your writing.

 

7. Data is/are

No matter what side of the data is/are debate you fall on, pick one side in your writing and stick to it. Sometimes your choice is dictated by your audience. A more academic of formal environment may prefer that you refer to data as plural (and use datum when referring to a singular piece of data) while a general audience may feel this style is too formal. The APA 7th edition citation guide defines data as a plural noun, requiring the use of plural verbs. However, in many contexts ‘data’ is considered a mass noun, like butter, water, or sugar.

Tip: Whether you’ve chosen to refer to data in the plural or singular, an easy way to check that you are using the correct verb when editing your writing is to replace the word ‘data’ in your head with another word which is more obviously either singular or plural. I use ‘cats’. Reading my methods section aloud in my head sounds like “The program staff captured the administrative cats in an Excel spreadsheet. The cats were checked for completion and missing data.”

 

8. Oxford Comma

Highly contested just like referring to data as a singular noun, the use (or not) of the oxford comma often elicits strong opinions. Being consistent with your usage sends a message to your reader about grammatical precision. When multiple people are writing the report, it is important to be consistent in whether you use (or not) the oxford comma or not throughout the report.

Tip: If you chose not to use the oxford comma be aware that there are certain instances where a comma is necessary before the word ‘and’ in a list greater than two things to convey the correct meaning.

Probably the most common example: ‘The strippers, Hitler and Stalin…’ implies, or is ambiguous at best, that Hitler and Stalin are the strippers in this sentence. While ‘The strippers, Hitler, and Stalin…’ makes it very clear that there are three groups of people being referred to in this sentence: 1) strippers, 2) Hitler, 3) Stalin.

           

9. Unclear pronoun reference

No, I’m not referring to the use of gender pronouns ‘they’ and ‘them’, rather I’m referring to having an unclear link between the pronoun and it’s antecedent (the noun it is referring to). Pronouns such as ‘it,’ ‘they,’ and ‘that’ should link clearly to one noun.

For example: “Laura and Marissa liked the ice cream she made.”

• Who made the ice cream? She is ambiguous in this context as the reader is unsure which of the two women made the ice cream.

A more common example I see in reporting: “The participants and program staff had differing opinions about the pizza lunch that was provided. They said it was greasy and soggy.”

• Who said it was greasy and soggy? The participants or the program staff? Since they have differing opinions, we are assuming both groups didn’t think the pizza was greasy and soggy.

Tip: Nothing can fix this mistake other than judicious editing. Match your pronoun to one clear noun.

Hopefully these common errors and tips have given you a starting place to improve your next evaluation report.

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---

 

 

Evaluations are inherently political, which means they are fraught with ethical choices and decisions along the way. There have been many instances throughout my career where I get that uneasy feeling, my gut talking to me and telling me to slow down and re-think what I am doing. I’ve learned that when I do, a devil appears on my left shoulder and starts yelling:

 

“WHAT ARE YOU DOING? YOU DON’T HAVE TIME FOR THIS! YOU’RE GOING TO GO WAY OVER BUDGET DOING THIS – KEEP GOING! SO WHAT IF YOU DON’T HEAR FROM THOSE PEOPLE? SO WHAT IF THAT PUTS THEM AT RISK?”

 

And then the angel on my right shoulder appears. Thank goodness! We all know from the movies that I’m supposed to listen to her. I slow down even more and listen intently:

“Do the right thing,” she whispers.

Uh-huh, okay. I keep listening for more….

That’s it?! What in the hell (excuse my language, angel), does that mean? In my evaluation experience I have been faced with numerous situations where I know I should do “the right thing,” but more times than not the decisions we face are not black and white or right and wrong.

For example:

….during evaluation planning, we don’t have the resources to answer all questions from all stakeholders – how do we decide how to focus the evaluation? Whose needs will we address and whose will we leave out? Is it appropriate to focus evaluation questions on funder needs and burden program staff and participants with collecting and reporting information that is not important or useful to them? Is it enough for funders to check the box and say, “the program was a success –  good for us,” but has little benefit for program staff and participants? 

….during data collection, we may be inclined to take the easier path to get the data we need to answer our questions. We may ask ourselves, “do I really need to include that group?” Trying to access that group and get their informed consent could burn a lot of time and resources. Is it right to exclude them and only provide the perspective of others?

….during analysis and reporting, what do we do if our stakeholders suggest we present findings  in an alternative (i.e. more favourable) way? This happens far more often than I would like, but stakeholders are called stakeholders for a reason  – they have a stake in whatever it is you are evaluating. We conducted an evaluation of a program a few years ago and from the start were informed that the livelihood of this program depended on favourable evaluation results. When we uncovered and reported what they perceived as negative findings, the stakeholders of course pushed back on those findings. As evaluators, we have a responsibility to present the data (both positive and negative); however, the reality is, anyone who has worked as an evaluator knows that data isn’t just data (yes, even quantitative data). As Michael Quinn Patton states in his book Utilization-Focused Evaluation, “data always requires interpretation. Interpretation is only partly logical and deductive; it’s also value laden and perspective dependent.” Conversations are never really about reporting if the results get presented, but more of a back and forth of how and to what extent.

So, what is an evaluator to do?

Here are some things that have helped me silence the devil on my left shoulder and figure out what my angel means:

Program Evaluation Standards

There are standards that have been created to help guide the way for evaluators. The Joint Committee on Standards for Educational Evaluation (JCSEE) developed Program Evaluation Standards that have been adopted by evaluation associations in the United States and Canada. The Standards provide guidance both for evaluators in planning and implementing their program evaluation projects, and for evaluation users in knowing what to expect from the evaluation process and products. Check out our free 6-page resource that provides evaluators with reflective questions for each of the Standards. 

---

Alberta Innovates ARRECI

– Most researchers have access to a research ethics board to review and approve — or not — their research projects. However, evaluators are often turned away by research ethics boards – after all they are “research” ethics boards. Nonetheless, evaluation still involves people, information and potential risk to participants. In our home province, Alberta Innovates has created an ethical framework for evaluation and other innovation projects in Alberta. ARECCI stands for A pRoject Ethics Community Consensus Initiative. This collaborative initiative has developed a screening tool that helps identify what your project is (i.e. research, evaluation, QI) and an ethics guideline tool. Even if you are an evaluator outside of Alberta, these tools are useful for identifying risk to participants and helping you make decisions that will protect people and their information.  

---

Evaluator colleagues and community

When you’re not sure, ask! At Three Hive, we are lucky to work on a team where we can bounce ideas and questions off each other. In addition, my business partner, Shelby Corley, is what Alberta Innovates calls a Second Opinion Reviewer, which means she has received additional training on reviewing projects for ethical risk and ways to mitigate risk. If you’re an independent consultant and don’t have your own in-house ethical angel, then reach out to the evaluator community. As you know, evaluators love giving recommendations! Start an #evalTwitter on Twitter, post a question on EvalCentral, or join some evaluation groups on LinkedIn and post your questions. While standards are useful, you will get far more real-world practical advice from your fellow evaluators.

If you’re still at a loss, sometimes you just have to re-visit that gut feeling. Maybe my angel wasn’t saying “do the right thing” but “do what feels right” (it is so hard to hear with that noisy devil sometimes!). As Ernest Hemingway said:

 

“So far, about morals, I know only that what is moral is what you feel good after and what is immoral is what you feel bad after.”

 

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---