Welcome to our monthly roundup of new and noteworthy evaluation news and resources – here is the latest.

Have something you’d like to see here? Tweet us @EvalAcademy or connect on LinkedIn!

---

New and Noteworthy — Resources

---

Michael Quinn Patton’s evaluation criteria for evaluating transformation

The American Journal of Evaluation recently published an article by Michael Quinn Patton (@MQuinnP) that outlines six criteria that are required for evaluating transformation. MQP argues that fundamental systems transformations are needed to address global emergencies brought on by climate change and the COVID-19 pandemic. MQP states that current evaluation criteria and practices are inadequate for evaluating transformation and that “evaluating transformation means transforming evaluation.” The six criteria are: 1) Transformation Fidelity 2) Complex Systems Framing, 3) Eco-Efficient Full-Cost Accounting, 4) Adaptive Sustainability, 5) Diversity/Equity/Inclusion, 6) Interconnected Momentum.

 

For more MQP you can also check out his recent webinar on YouTube – Evaluation and the Challenge of Our Times (a follow-up to his American Evaluation Association 2020 presentation).

This is why evaluation is a tool for systems-change

Speaking of systems change and transformation, check out Pablo Vidueira’s article (@PabloVidueira.) This article succinctly outlines why evaluation needs to transform. It highlights a few examples of how Blue Marble Evaluation is being applied. Pablo concludes by saying, “Transforming our society means transforming our systems and our way of thinking – moving away from a project-focused mindset and toward one that recognizes the fragile, finite and interconnected nature of the Blue Marble we inhabit.”

Living history: A developmental evaluation tool

Cameron Norman (@cdnorman) recently wrote a post for the Tamarack Institute that outlined a method called The Living History. Cameron discussed how developmental evaluation can provide vital information to inform the evolution of a program by connecting what is done with what is happening. The Living History method is a means for capturing and documenting all the actions, decisions and assumptions that occur throughout the life a program so you can answer “why did we do this?” or “what led to this action.” The Living History can also document environmental or cultural shifts that might affect a program. For more information, you can go to Cameron’s original post on his website.

Tamarack Institute’s guide for meaningfully engaging youth

The Tamarack Institute has co-developed with youth organizations a guide that outlines the principles and practices for meaningful youth engagement. The guide is not specific for evaluation; however, these are useful best practices evaluators can incorporate when involving youth in their evaluations.

 

The practice of evaluation: partnership approaches for community change

This book was recently released by SAGE Publishing (authored by Ryan Kilmer and James Cook from the University of North Carolina.) The book provides content on evaluation concepts, approaches and methods with an emphasis on the use of evaluation and partnership approaches to effect change for program improvement.

 

A stakeholder view of the development of national evaluation systems in Africa

This article sought to document and compare developments of national evaluation systems (NESs) that are being set up across Africa. This article focused specifically on Ghana, Kenya, Rwanda, South Africa, Uganda and Zambia. The findings show that governments are progressively investing in M&E as a practice and its infrastructure. However, different M&E systems in the six countries are shaped by state architecture, political administration priorities, government capacity, resources available and other enabling environmental factors. The study offers insights that can be used to start dialogue about how to close existing knowledge gaps, improve M&E capacity building offerings and increase the use of M&E evidence in policy and program implementation.

---

New and Noteworthy — Courses, Events and Webinars

---

December 2020

Evaluation Planning – Setting the Stage for Successful Evaluations

Sponsor: EnCompass  – Ayesha Boyce
Date: December 7 – 10
Venue: Online Course

 

Data Storytelling

Sponsor: EnCompass  – Andy Krackov
Date: December 15 -17
Venue: Online Course

---

Keep the roundups coming? Sign up for our monthly roundup here:

Subscribe to Our Newsletter

Sign up with your email address to receive news and updates from Eval Academy every month.

Email Address
Sign Up

We respect your privacy.

Thank you!

---

 

 

In this video tutorial, we show you how to create a column chart in Microsoft Excel. Learn the basics of creating and transforming your column charts into something clean, informative, and impactful:

To download a PDF version of this tutorial to save for future use, click here.

To download the Excel spreadsheet and use it with your own data, click here.

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---

 

 

Six hacks for renovating your evaluation report

Part 4

This series of posts walks you through how to reno your evaluation reports using six of Canva’s design lessons. In this series of articles, Part 1 focused on how to take your audience on a journey using storytelling techniques. Part 2 focused on how to format your report with a consistent, cohesive look using colour and font. Part 3 dove into grouping and spacing elements in your reporting. Part 4 in the series is going to explore how to make elements in your report pop using focal points.

 

The importance of focal points

I heard recently “if you’re marketing to everyone, you’re marketing to no one.” The same applies when it comes to creating emphasis – if everything pops, then nothing pops. A focal point is something that draws our readers to a specific element on a page. It is different or unique and something that is distinct from the rest of the information on a page. It could be a title, a graph, a picture or even a link, but ultimately focal points create something called dominance.

Creating a dominant element in your report shows your reader where to look first. Dominant elements reveal what is most important in your report. If used properly, these elements can even direct how your audience processes the information in your report. For example, in the image below your eyes likely go to the “look here” box first and then down to the “now here” box and then to the bottom left “finish here.” The reason our eyes follow this path has to do with visual weights – our eyes are attracted to elements with greatest visual weight.

Creating elements with visual weight helps our readers to easily process information. Look at the two pages below. If you are anything like me, you will look at the left and automatically feel overwhelmed. Without a focal point, like a heading, we are creating unnecessary work for our reader that may cause them to close the report. The page on the right is still text heavy, but we have added in some emphasis using headings and graphs that draws the reader’s attention to what is contained in the text. You will increase readability and have a higher likelihood of engaging your audience using focal points that break up your text.

Creating focal points using visual weights

You should create focal points to emphasize what you want your audience to take away. The page on the right contains focal points that were created using some simple elements that you can incorporate in your reporting.

 

Size

Large elements have more visual weight than small elements. In the example above, “Timely Access to Care” has the largest font (i.e. more visual weight than other headings) so it draws the eyes to it. Likewise, the graphs are the largest elements on the page so they draw the eyes there.

 

Position

Elements located higher on the page are perceived to weigh more than elements located lower on the page. Generally, our eyes will flow from the top left to right in a sweeping pattern as illustrated below. Therefore, your most important information should be positioned in the top left of the page (at least for languages based on the Latin alphabet where we read left to right.)

 

Colour

Darker elements have more visual weight than lighter elements. You’ll notice in the example above we’ve chose blue to highlight titles, headings and key takeaways from the graph. We then muted the other colours on the page to make the blue pop (hint: contrast also creates visual weight.)

 

Saturation

Saturated colours appear heavier than desaturated. We have bolded our heading of “Timely Access to Care” to create more visual weight.

 

Intrinsic interest

Some things are more interesting than others – graphs and pictures are some examples of elements that attract the eye. By inserting the graphs in our report, we are creating another focal point to draw the eye to. 

---

The elements described above are just five different elements you can use to create focal points in your report; however, there are many others that you can incorporate as well (i.e., textures, shape, orientation, complexity.) When you start considering all of the different elements it begins to get confusing wondering what has more visual weight. The bottom line is there are no hard and fast rules – ultimately, creating visual weight depends on how all the elements are used together. Some might say the title on our page has the most visual weight while others would say the graphs. The most important thing is that you use these elements together to create focal points that draw your audience’s attention to key takeaways in your report.

Take a look at some of the reports you are creating. Are there opportunities to make something pop? Try it out and make sure to stay tuned for the fifth article in our six-part series, “Photo Love” where I talk about the importance of imagery and show you how to use images in your reporting.

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---

 

 

Defining evaluation purpose. Writing evaluation questions. Deploying data collection tools. These topics can all seem abstract on their own. To put the pieces in context, we’re offering this series on how we evaluated to show you what real-world evaluation looks like in practice.

This post explores how we at Three Hive Consulting worked with REACH Edmonton Council and other agencies to evaluate a unique initiative called Bridging Together. You’ll see how they developed and carried out an evaluation plan that yielded actionable information.

 

The initiative: Bridging Together

“When I’m focused on the day-to-day details, it’s easy to forget how many lives the collaborative is reaching. For me, evaluation helps to keep things in perspective.”

With funding from Immigration, Refugees and Citizenship Canada, REACH Edmonton Council acted as the backbone organization for this collective of youth-serving non-profits. Each of the partner organizations already offered programming for immigrant and refugee youth outside of school hours. Their after-school and summer programs varied in focus, but common elements included academics, sports, life skills, culture and recreation. These partner organizations met regularly to share resources, discuss common problems and share solutions, and REACH arranged for relevant training opportunities.

 

Intended outcomes

Bridging Together aimed to enhance outcomes for immigrant and refugee children and youth, their families, and the partner organizations.

Immigrants and refugees face many well-documented challenges when arriving in Canada, not limited to linguistic, cultural and environmental differences, physical and mental health, socialization, education and justice. While many reveal resilience and integrate well into Canadian society, a significant number do not fare so well. Through out-of-school time programming, partner organizations intended to help children and youth develop healthy relationships, improve self-efficacy, become involved in community, improve academic performance, and perhaps most importantly, have fun.

 

Developing the evaluation plan

Convening thirteen organizations to work toward a common goal is no small task. Having them agree on intended outcomes and evaluation processes was a smoother process than expected. We held a large group session to begin defining evaluation purpose, use and focus areas. From this meeting, we drafted four focus areas and posed several questions to attendees:

• What would you like to know about your program?

• What has worked before with evaluations you have been involved in?

• What is your one piece of advice for how to make this a successful evaluation?

• What difference should we see in a child or youth after participating in your program?

 

This stakeholder engagement process showed a need for a data collection approach that acknowledged commonalities while accommodating the uniqueness of different programs. We confirmed four common focus areas:

1. Program description and participation

2. Child, youth and family outcomes

3. Collaboration

4. Social return on investment

The social return on investment (SROI) was non-negotiable requirement. It is not a method we would have suggested, but as evaluators we know that sometimes we just have to do what we’re told. We’ll reflect on that SROI below.

Partners reviewed and made suggestions on draft versions of the evaluation plan until we arrived at a final version to guide the next two years.

 

Adapting data collection approaches

We mentioned above that it was important to partners that the evaluation reflected their individual programs. There was quite a bit of variation to address; one organization delivered their programming entirely in French, one provided free sports leagues for children in grades four through six, while others delivered more of a “homework club” program. Some organizations offered multiple programs through Bridging Together. Participant ages ranged from six to 24. In the first year, 390 children and youth participated.

Our methods, obviously, needed to accommodate different program activities, different languages, different reading levels, and very different logistics. So here’s what we did:

1. Interactive, arts-based feedback sessions with youth in summer programs

2. Program experience surveys for older children and youth

3. Self-efficacy surveys for older children and youth

4. Video-recorded, small group interviews with children at sports leagues

5. Parent/caregiver program experience surveys

6. Interviews with organization staff

7. Social network analysis

8. Administrative data analysis

9. Social return on investments, requiring detailed funding and spending information from all organizations

 

Project ethics

We’re big fans of ARECCI, a project ethics review process we can access in Alberta. We made sure to include an ARECCI project ethics review in our proposal to REACH, and incorporated their suggestions into our processes.

 

Collecting data

We expected challenges in implementing the nine approaches above. In our monthly status updates, we tracked what we had done, what we planned to do next, what risks emerged and how we were mitigating them. 

Completing the summer feedback sessions required some support from sub-contractors. Our plan was to schedule these sessions, where we would also support the survey administration for older children and youth, as close to the end of their summer program as possible. Not surprisingly, many programs ended in the same week, so deploying evaluation assistants to all sites was tricky but we were able to accommodate those programs that agreed to participate.

Collecting data from this many sites also required support from program staff and volunteers. Contacting some organizations was easy; others’ capacity was so stretched that returning phone calls and emails did not always happen. Most were quite willing to support survey administration, with guidance provided. We did find, though, that sometimes younger children were completing surveys intended for older children and youth.

Getting parents and caregivers to complete surveys was challenging for some programs, and smooth for others. To make it easier for parents and caregivers to complete surveys, we provided both an online option and paper surveys, and kept the survey as short as possible while collecting the meaningful data we needed. Overall, our sample size for parents and caregivers was lower than we had hoped for—that’s a challenge many working in non-profit evaluation will be familiar with.

The SROI calculation required detailed information about program inputs and spending. Most partner organizations were running multiple programs, some of which had funding from the same sources. Many programs also relied on funding from other sources, volunteers, and subsidized facility rentals. We were fortunate to have support from REACH to create a spreadsheet for organizations to identify all financial and in-kind resources needed to run their Bridging Together program and all associated spending. Completing that spreadsheet represented a great deal of time for partner organizations.

 

Sharing findings

We produced a few different reports throughout this contract. The major products were comprehensive written reports for Year 1 and Year 2. Each yearly report addressed the first two focus areas, program reach and outcomes, and one additional focus area. Following the preparation of the draft reports, we attended meetings with partners to review findings and gather their perspectives and suggestions.

These comprehensive reports addressed Bridging Together as a whole, but we also wanted to provide individual organizations with results that they could use to inform program changes, organizational reporting and further advocacy. We therefore provided short summaries of results for each partner organizations.

 

Informing our practice

As evaluators, we learn from every project we undertake. The Bridging Together project spanned two years and showed us the importance of strong working relationships with clients and stakeholders. This project showed us how valuable a convener or coordinator is in collective impact projects—we would have needed to invest more resources in project management if REACH had not so capably undertaken that role.

This project also demonstrated how vital data management practices are when working with multiple sites across multiple timepoints. A good spreadsheet or other tool to track which data has been received from which site supports sounds project management.

We’ve always been pretty flexible, but this project reinforced how important is to be able to adapt processes to fit different contexts. For example, our youth feedback sessions looked different across sites. In some, we used classrooms with structured space; in others, we set up in a hallway and had children and youth move through a sort of drawing and writing gauntlet. One method, the mini-interview, was used for just one program because it was simply the only feasible way to collect data from busy kids running on and off the field. Seeing how this variation in methods led to a richer knowledge product has reinforced for us that adaptability is key in real-world evaluation.

And finally, the SROI. The calculation showed that for every dollar invested, Bridging Together created at least $3.30 in returned social value. This figure is powerful in reporting and future funding applications. Obtaining the data to inform this calculation was A LOT of work for partner organizations. Many organizations’ accounting systems were not set up to track costs for individual programs; the work required to set up overhead calculations and other bookkeeping details for many different programs cannot often be accommodated through non-profit administrative allocations. We have always viewed this method with skepticism, and questioned the need for it at all. The value of improving outcomes for children and youth has been well documented. We already know that investing in children saves money later. We approached this project with the view that requiring resource-limited programs to undertake this complex and imprecise calculation is an undue burden and does not yield new findings; that view hasn’t changed.

 

Client perspective

“ Having an unbiased third-party report to show our success is so important to be able to justify the worth of this collaborative to the funder we had, as well as potential future funders.”

How has this evaluation been applied at REACH? Evaluation use is a topic many contracted evaluators wonder about. Is the report just living on a server somewhere, never to be consulted again? Or have the findings and recommendations been used to drive program changes, to advocate for funding, to share a story of impact?  

Overall, REACH is dedicated to evaluating its work. “We know that nothing is perfect and evaluation results help to inform the project as it unfolds and influences decisions,” notes Project Manager Lisa Kardosh. “When I’m focused on the day-to-day details, it’s easy to forget how many lives the collaborative is reaching. For me, evaluation helps to keep things in perspective.”

For REACH, early results were useful in ongoing planning. “The interim report gave the collaborative a chance to assess if we were on the right track or not, and thankfully for the most part we were,” says Kardosh. “One benefit of the report was that it helped to shed light on gaps that were popping up, like more training being needed, so we could address it.”

Interim reporting also yielded an early opportunity to demonstrate the value of the program to the funder. “It was useful to share the Year 1 results with our funder so that they could see that their investment was making a difference.”

REACH and the Bridging Together partners have used the final evaluation report for advocacy and communication. “We’ve shared the Year 2 results quite broadly among our networks,” says Kardosh. “Having an unbiased third-party report to show our success is so important to be able to justify the worth of this collaborative to the funder we had, as well as potential future funders.”

 

Watch for more in our How We Evaluated series.

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---

 

 

Welcome to our October roundup of new and noteworthy evaluation news and resources – here is the latest.

Have something you’d like to see here? Tweet us @EvalAcademy or connect on LinkedIn!

---

New and Noteworthy — Resources

---

Evaluation issues and recommendations for change

The tables have turned and now evaluators need to examine how we can do better. The Funder & Evaluator Affinity Network (FEAN) developed a Call to Action series: five briefs, decision tools, open letters, discussion guides, and more that address urgent issues and recommendations for change. The five briefs propose tangible steps to make changes in the field: 1) Strategy and Practice; 2) Evaluators of Color; 3) Knowledge Sharing; 4) Global Transformation, and; 5) Collaboration and Partnership. We have a lot of work to do. The first step is reading these.

Engaging community in evaluation through empowerment evaluation

The Tamarack Institute and Fetterman & Associates co-facilitated an online session that explored how we ensure evaluation is inclusive, equitable and community-driven. The first part of the session focuses on criteria central to relationship building in participant-driven community-based evaluation. The second half focuses on empowerment evaluation as an evaluation approach to foster community engagement, improvement and self-determination.

Principles Focused Evaluation: A quick overview

CoLab Dudley has pulled together a quick case study guide for anyone interested in Principles Focused Evaluation (PFE). If you don’t have the time to read Michael Quinn Patton’s 435 page book on PFE then check out this resource. The guide contains links to insights from CoLab Dudley team members telling you why and how PFE works for them in their role. It also outlines the steps they took to implement PFE and tips if you are thinking of using it.

How to incorporate gender in your evaluation

Khulisa Management Services breaks down how they incorporate gender into their work through six best practices. Incorporating gender means incorporating analyses of institutions and structures that limit the realization of gender equity. One best practice that Khulisa implements is developing gender-related evaluation questions like, “have stigma and discrimination against people who do not follow traditional gender norms and behaviours been reduced?” Check out the blog and the accompanying slide deck here. 

EvaluATE’s tips for better communication

One-pagers are a necessity when it comes to quickly conveying findings. However, creating a one-pager is often more difficult than a 25-page report! EvaluATE has put together a number of resources to help you create knockout one-page reports. The resources include fillable planning sheets, examples to inspire you, template grids to use in your one-pager and even a couple videos.

EvaluATE also recently posted a blog that outlines strategies for communicating in virtual settings. It outlines some really easy, practical tips for how we can improve our communication in virtual settings. A tip I never thought about – “if you have books in the background turn the spines of the books away. The titles of the books can be distracting and can communicate unintended messages.”

Planning and conducting virtual workshops

Jan Noga (@jannoga) recently contributed to AEA’s A Tip-a-day by publishing a three-part series of articles focused on helping us plan and conduct virtual workshops. In the posts she talks about how online workshops are not the same as in person and gives some helpful tips, tricks and tools that can be used to make sure they are just as effective. The third articles lists “rad resources” like books, training materials, and software to reference when planning your next workshop.

---

New and Noteworthy — Courses, Events and Webinars

---

November 2020

Using data analysis and visualization to drive social impact

Sponsor: Clear Horizon Academy 
Date: November 9, 2020
Venue: Online course

 

MEL Project

Sponsor: Clear Horizon Academy 
Date: November 30, 2020
Venue: Online course

Evaluation and the Challenge of Our Time – AEA Panel Follow-up Conversation

Sponsor: Blue Marble Evaluation
Date: November 13, 2020
Venue: Online

Gender Transformative Program Design and Evaluation

Sponsor: Encompass – Tessie Catsambas
Date: November 3 – 12
Venue: Online Course 

December 2020

Evaluation Planning – Setting the Stage for Successful Evaluations

Sponsor: EnCompass  – Ayesha Boyce
Date: December 7 – 10
Venue: Online Course

Data Storytelling

Sponsor: EnCompass  – Andy Krackov
Date: December 15 -17
Venue: Online Course

---

Keep the roundups coming? Sign up for our monthly roundup here:

Subscribe to Our Newsletter

Sign up with your email address to receive news and updates from Eval Academy every month.

Email Address
Sign Up

We respect your privacy.

Thank you!

---