Welcome to our monthly roundup of new and noteworthy evaluation news and resources – here is the latest.

Have something you’d like to see here? Tweet us @EvalAcademy or connect on LinkedIn!

New and Noteworthy — Reads

Last month we highlighted Khalil Bitar’s blog post from May 11 where he talked about how the evaluation community is not immune to prejudice, discrimination and racism. He went on to say that the evaluation community is in fact practicing racism by primarily producing and sharing evaluation knowledge by white men from the Global North. Two weeks after this blog post George Floyd was killed. George Floyd’s killing ignited protests across the world, but it also ignited more reflection and conversation about what could be done about systemic racism in our societies. Evaluators reflected through their blogs on how evaluators can begin to move the needle. Some relevant blog posts to check out include:

Jara Dean-Coffey – Musings + Machinations

Jara Dean-Coffey is the Founder and Director of the Equitable Evaluation Initiative (EEI). If you haven’t checked out the EEI then make a point to do so. There are resources that evaluators can access to better understand how evaluation can and should be “utilized in a manner that promotes equity.” But as Jara stated on LinkedIn, she felt the need to write outside of EEI and recently began a blog called musings + machinations  – writings by jdc. In June she posted 12 posts! Read them all and make sure to subscribe to this blog – her musings are insightful and and her machinations inspiring.  

Engage R+D –  It’s Time to Let Go of Tired Narratives about Talent in Evaluation

We hear a lot about narratives these days. In this blog post on Engage R+D, Clare Nolan talks about narratives – what they are, the power they can have, and how they have been used to support oppression. She talks about some of the tired narratives about talent and expertise in evaluation that “get in the way of effective and equitable solutions” and then counters those with ideas of potential new narratives that can help advance equity. For example, we’ve all heard the “diverse applicants don’t meet our standard qualifications.” A potential new narrative is looking at how implicit bias and white-dominant norms are constraining our ability to recognize valuable knowledge, experience and credibility.

Michael Quinn Patton’s Rules for Privileged Straight White Males and Andrea Guerrero-Guajardo’s Rewrite

MQP published a blog post on his website that outlined rules for straight white males. He outlined ten rules that privileged white males don’t necessarily have to like or agree with but do have to follow. If you scroll down to the comments you can find many people who definitely disagree and not in a constructive manner. Someone who did provide constructive feedback was Andrea Guerrero-Guajardo who put time and thought into rewriting MQP’s rules, which can be found here.

Chris Lysy –  Evaluation, Compassion, Fatigue and Health Inequity

Chris’ blog post talks about a number of different topics, as the name suggests. He explores how as evaluators we need to not only do something but stand for something. He suggests the Equitable Evaluation Initiative as a resource and approach for how evaluators can channel their efforts and keep the momentum moving forward because “failing to channel our efforts can quickly lead to fatigue.” In this blog post, he also explores inequity in public health data.

New and Noteworthy — Tools

Code for America’s Quantitative Research Practice Guide

Code for America has come out with a timely guide that can be used by anyone looking to conduct qualitative research “in ways that can help everyone ensure that their products and services are as inclusive as possible.” The guide outlines specific methods for conducting research and analysis, but to be honest my favourite part of this guide is Code for America’s core research philosophy and guiding principles. The fact that this organization has a research philosophy and guiding principles is swoon-worthy enough, but how they have articulated their philosophy and guiding principles is so on point I found myself re-reading it several times. They conclude their research philosophy by stating: “Ultimately, research is a tide that lifts all boats. It is fundamental to developing government services that better and more equitably meet the needs of communities. Raising the bar on quality of research raises the bar on quality and effectiveness for everything that we seek to do for the world.” Well said.

New and Noteworthy — Courses, Events and Webinars

July 2020

1. Results-based Management & Theory of Change Workshop during and after COVID-19

   Instructor: Mosaic.net International Inc.

    

2. IPDET Evaluation Hackathon

   July 7 – 13

3. Claremont Graduate University – The Evaluator’s Institute

   A variety of courses starting July 20 that are being conducted by various instructors, including some big names like Michael Quinn Patton, Ann K. Emery, and Ann Doucette.

We have a free guide:

Applying the JCSEE Evaluation Standards in Practice

Whether you’ve read The Program Evaluation Standards cover to cover or not, you may be wondering how to ensure you’re applying them to your evaluation practice. This free digital download will give you the reflective prompts you need to ensure your next evaluation project incorporates all 30 Standards.

Email Address
Sign Up

We respect your privacy.

Thank you!

A recent study in the American Journal of Evaluation showed how three simple visual design principles could be applied to logic models to make them more effective and understandable. This article summarizes the findings of that study so you can improve your logic model.

What are logic models?

Logic models are widely used in evaluation to visually summarize how a program is expected to work: what resources will be used, what activities will be undertaken, and how those activities will cause desired outcomes. These are often a staple in our evaluation work, whether we are starting with a logic model that has already been created, or we are creating one for the program from scratch.

Why visual design?

Visualizing information effectively is important for evaluators because we are “communicators and knowledge brokers” at our core. The way we visualize and present information affects how it is used, and who is able to use it. Therefore, logic models could be made more useful and understandable by improving their visual design. The goal of the study was to make a logic model better by applying visual design principles. A good visualization:

• Is clear, useful, and memorable;

• Supports audience understanding;

• Is easy to mentally process; and

• Is organized into memorable chunks.

These visualization principles, along with the relevant research on data visualization and visual design, were used to make some simple (yet effective) improvements to the basic logic model.

Visual improvements to the logic model

The original logic model looked like this:

This is a fairly standard logic model that evaluators are used to seeing. It outlines the program’s inputs, activities, reach, and outcomes, and shows the interconnections between them.

This logic model was revised by incorporating the following visualization best practices:

1. Colour: Colour was used to group similar items together (i.e., each column was a different colour)

2. Proximity: Elements that were connected to each other were moved closer together, which reduced the emphasis on arrows

3. Reducing ink: unimportant elements were de-emphasized or removed to keep the focus on the most important elements. For example, by removing the black borders around boxes.

After making these improvements, the revised logic model looked like this:

The revised logic model was easier to understand

After testing these logic models with a survey of the general U.S. population, the researchers found that the revised logic model was:

• Faster to review

• Interpreted more accurately

• Easier to understand

• More aesthetically pleasing

• Perceived as more credible

With relatively simple visual changes, which can be implemented in Microsoft Word or PowerPoint, the researchers were able to improve their audience’s understanding of the logic model. We can leverage these 3 design tweaks (colour, proximity, and reducing ink) to help our audiences read our logic models faster and more accurately.

Notes on study methodology

The original study (linked below) has more detail about how exactly the research was conducted, but in short:

1. A survey was conducted with the general U.S. public.

2. Respondent were shown one of six possible variations of the logic model, which were combinations of:

   • Original vs. revised logic model

   • With vs. without a written narrative

   • Greyscale vs. colour logic model

3. Respondents were then asked a series of questions about the logic model they were shown, including their understanding of the program, the effort it took to understand, their perception of the credibility, and the aesthetic qualities.

4. Responses to the different types of logic models were compared to determine if the visual design elements helped improve the original logic model (and it did!).

The full study can be found here:

Jones, N. D., Azzam, T., Wanzer, D. L., Skousen, D., Knight, C., & Sabarre, N. (2019). Enhancing the Effectiveness of Logic Models. American Journal of Evaluation, 1098214018824417. https://doi.org/10.1177/1098214018824417

Email Address
Sign Up

We respect your privacy.

Thank you!