Welcome to our July roundup of new and noteworthy evaluation news and resources – here is the latest.

Have something you’d like to see here? Tweet us @EvalAcademy or connect on LinkedIn!

---

New and Noteworthy — Reads

---

International Program for Development Evaluation Training – Evaluation Hackathon

For those of you on Twitter, you likely have been following IPDET’s Evaluation Hackathon. The Evaluation Hackathon took place from July 7-13 and was “a playground for creative individuals from around the world to unite their skills, knowledge and inspirations to find creative solutions to challenges of our times.” These solutions are ones that might help to empower the field of evaluation. Check out all the cool ideas on the project page.   

Capacity4dev – Evaluation in Crisis

Capacity4dev is the European Commission’s platform for sharing information related to International Cooperation and Development. Its Evaluation Support Service team created the DEVCO/ESS Evaluation in Crisis Initiative. This initiative curates resources (documents, webinars, videos, blogs and podcasts) to help evaluators evaluate in crisis. Some of the topics covered include: How do we need to adapt our processes to move quickly? What data collection techniques are best suited in a crisis situation? Do we need to review our evaluation ethics? How do we check facts when using remote techniques? Can we still contribute to sustainability and if so, how?

Eval Forward – Evaluation in Times of COVID-19

If you are looking for more insights about evaluation during times of crisis then check out Eval Forward’s three-part blog series that describes reflections from leaders and managers currently engaged in humanitarian-development evaluations. Evaluation leaders from Action Against Hunger and the World Food Programme were interviewed and asked to reflect on how the pandemic is affecting the practice of evaluation and what they think it will mean for evaluation going forward. Interestingly, some speculated on a greater mix of national and international evaluators on evaluation teams as a result of COVID-19 (check out Engage R+D’s report mentioned below for why this is so important in our field.)

Engage R + D – Listening for Change: Evaluators of Color Speak Out About Experiences with Foundations & Evaluation Firms

In Engage R+D’s Listening for Change learning brief they state, “foundation staff and evaluators tasked with planning and assessing social change efforts do not reflect the demographics and cultures of the communities they serve” – there needs to be more attention to diversity, equity and inclusion (DEI). To make progress when it comes to DEI, we need to start by listening to ideas, insights and experiences of professionals of color. The learning brief reports on four key themes on what it will take to support leaders of color in philanthropic evaluation: 1) Outreach is key to opening a career pathway; 2) Attitudes and dynamics in the workplace affect retention of evaluators of color; 3) Demonstrated commitment to DEI attracts evaluators of color to evaluation firms and clients, and; 4) Employers have an active role to play in retaining staff.

Stanford Social Innovation Review – Ten Reasons Not to Measure Impact and What to do Instead

While we’re talking about ways to transform the evaluation field, let’s talk more about impact evaluation. There is a continued push for more and more impact measurement; however, this is not always appropriate and even problematic in a lot of circumstances. In this article, Mary Kay Gugerty and Dean Karlan outline the ten reasons or circumstances not to measure impact and the alternatives that can be adopted instead. Ultimately these reasons fall into four categories: 1) Not the right tool; 2) Not now; 3) Not feasible, and; 4) Not worth it.

---

New and Noteworthy — Tools

---

EvaluATE – Key Resource by Evaluation Topic

EvaluATE has many resources on its site; however, we all know clicking through and navigating to numerous resources can quickly lead you wondering where you are and how you got there. Instead, EvaluATE has compiled its resources into one PowerPoint file, organized according to evaluation topic areas, so you can quickly navigate to the resources you need. Topic areas include Finding and Selecting an Evaluator, Integrating Evaluation in Proposals, Getting Started with Evaluation, Evaluation Design, Data Collection and Analysis, and Reporting and Use.

Inspiring Impact – Review your existing data worksheet

Inspiring Impact created a worksheet that outlines a step-by-step process to help review data. The worksheet is helpful in determining what information you should continue to collect, what to stop collecting and what to start collecting. The worksheet is available in both Word and Excel formats.

Khulisa Management Services – Visual Methodologies in Evaluations

My favourite part of being an evaluator is when I can combine my analytical and creative sides – (so much so I coined the term Evalucreator!) For all you Evalucreators out there, check out this deck on visual methodologies and how to incorporate them into your evaluation practice.

DC Fiscal Policy Institute – Style Guide for Inclusive Language

We’ve mentioned DEI above in our New and Noteworthy reads. There are steps you can take for greater DEI when you write. This style guide provides guidelines for ways that we can employ inclusive language and integrate a racial equity lens in our writing. While the guide is targeted for the DC geography, it still provides useful terms and principles that can be applied in different settings.

---

New and Noteworthy — Courses, Events and Webinars

---

August 2020

Claremont Graduate University – The Evaluator’s InstituteA variety of courses that are being conducted by various instructors, including some big names like Michael Quinn Patton, Ann K. Emery, and Ann Doucette.

Australian Evaluation Society – Fundamentals of Good Evaluation Reporting and PracticeFacilitator: Anne Markiewicz

Date and Time: August 17 & August 24; 9:30am – 11:00am AEST

Venue: Online 

September 2020

Evalpalooza I: Evaluation Failures with Kylie Hutchinson and Thought Leaders

Presenters: Kyle Hutchinson & Libby Smith

Date and Time: September 24; 12pm CDT

Venue: Online

---

We have a free guide:

Program Evaluation Scoping Guide

This is a free digital download. The guide outlines questions evaluators can ask program managers or other stakeholders to better understand the scope of the program and its evaluation. The questions in the guide are intended to help evaluators begin formulating a quote and/or an evaluation plan; however, it can also be used identify disagreements or gaps in what is known about the program and/or the boundaries of the evaluation.

---

---

 

 

Recently, we at Three Hive Consulting used outcome harvesting as part of a developmental evaluation with an organization who builds connections and helps facilitate community change. As with most developmental and participatory techniques, using this method was a bit time intensive, but the results were worth it. Along the way, we realized that although there are research and examples about how to use this methodology, we wished we could find a real account of the ups and downs of implementing the methodology in a real world setting. Here we share how we used the methodology and what we wished someone had told us before we started.

 

What is outcome harvesting?

To begin, let’s quickly review what outcome harvesting is. Outcome harvesting is a participatory evaluation methodology that was developed by Richard Wilson-Grau and colleagues*. In this methodology, change is monitored by collecting evidence of what has happened (gathering outcomes) and then looking back to understand how a program or intervention has contributed to these changes.

Outcome harvesting helps to understand what has happened due to actions taken in the past. It is particularly useful in programs or interventions which are targeting community- or population-level changes or in complex situations where the change seen in the beneficiaries cannot be directly tied back to one action or program or actor. It is also useful when the goals of a program or intervention are broad and flexible; and thus can be a helpful tool in developmental evaluations, where the actions and intended outcomes may change over the course of the evaluation. The findings of an outcome harvest can be used to understand how a program or initiative contributes to change and can be used as a planning tool to course-correct or modify program approaches.

Who is involved?

A successful outcome harvest is a participatory process. By involving those who have experienced change and those who can use the findings, the outcome harvest will be more successful. There are three groups of people who need to be involved in the outcome harvest.

1. Informants: The people who were part of or who witnessed the outcomes.

   In our case, these were the partners that the community initiative worked with.

2. Harvest user: The person using the findings to make a decision or take action. They will help guide the approach used to ensure the data they need is collected. Sometimes there are multiple harvest users (e.g. funding organizations and the funding recipients who provide programming).

   In our case, the organization and the evaluation sub-committee was the harvest user.

3. Harvester: the person(s) leading the outcome harvest. They support the process and suggest strategies to improve the credibility and reliability of the data.

   That’s us! Our role was to consider how to make the data tools, collection, analysis, and reporting processes as credible and rigorous as possible.

 

How did we do it?

While Better Evaluation describes 6 main steps for outcome harvesting, in reality our approach had 4 simple steps.

1.    Design

In the design phase, the focus is on clarifying what the harvest user needs to know and how they want to use the information. This is basically the same as step 1 in the 6-step approach.

We asked:  What activities, events, or programs has the organization contributed to? How has the organization contributed? What is the impact of these activities, events, or programs? Who did they impact?

2.    Draft outcome descriptions

Typically, this is done by the harvester (evaluator) through document review. The organization we were working with was not a direct service provider and for many outcomes there was little documentation to review. So, we started with the organization rather than with the informants. We reviewed what documents were available, but also invited the organization to list activities, initiatives, and outcomes they helped to contribute to. The organization also provide us with the contact information for each of the partners (informants) who were involved.

We ended up with nearly 30 outcomes and almost 20 partners.

3.    Expand and corroborate

Because we started with the organization drafting outcomes, the next step in our approach was to connect with the partners listed in step 2 to hear what they thought about the outcomes the organization generated.

In a short interview we asked the partners:

• “What have you worked on with the organization?” 

• “The organization identified that you worked on X together, can you tell me a bit more about that. How did the organization contribute?”

• “What was the significance of that event/activity/program/collaboration?”

• “What impact do you think it had on you/the community?”

If the partners’ accounts differed from the outcome description, we made sure to probe further and seek additional sources of information. Some partners suggested additional partners who were also involved in the outcomes that we then reached out to.

4.    Analyze and use

Finally, the data analysis and ensuing conversations were part of using the findings. We classified the outcomes based on the organization’s priority areas where they aimed to make change. In our case, collecting information about the outcomes was a tool in itself to engage the partners and have them reflect on what they have accomplished with the organization. Discussing the outcomes that were achieved also helped the organization to identify that they needed to do some further work to understand how their activities were linked to outcomes (i.e., logic modelling). We also recognized that the outcome harvest had not captured the community members’ perspectives on the outcomes. We used the stories of most significant change technique to gather participants’ perspectives on the activities and events and compared the outcomes and findings from the two techniques.

What did we learn?

Through this outcome harvest, we learned a lot of important lessons about using this method effectively.

1. Clear guiding questions set you up for success. As the harvester, communicate the strengths and limitations of this method with your harvest user to help them understand what questions to ask.

2. Use multiple sources of information to collect and substantiate your outcomes. In our case, multiple partners often contributed to a single outcome. Asking all of them about their experience rather than just one provided a range of perspectives.

3. Get creative with how you collect and verify outcomes. Most examples of substantiating outcomes used surveys, which seemed too impersonal and quantitative for the level of understanding we were hoping for. Instead, we set up short interviews with informants.

4. Get both the organization and the informants’ point of view on the outcomes. Understanding both perspectives enriched our data. We also found some unintended consequences and negative feedback, which helped to provide actionable results.

5. This method takes time. While the phone calls with informants only took 20 minutes, it took 6 weeks to reach out to all of them. Additionally, it took 2-3 weeks to draft outcome statements with the organization.

6. Generally, people like sharing about the work they have done. As an evaluator (or harvester), this was a rewarding experience and helped us to better understand the organization and how they work.

7. The outcomes are limited to those that the organization and informants identify. A more diverse pool of informants leads to a wider perspective about the outcomes. Ask your informants if there is anybody else they think you should be talking with.

8. Sometimes the process is more important than the final reveal. The act of harvesting outcomes and the ensuing conversations with informants and users can inspire more action than the final report.

---

Hopefully this article has given you another perspective on outcome harvesting. It can be a powerful methodology to understand complex situations. Don’t be afraid to make the modifications necessary to best suit your harvest users and informants.  

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---

 

 

Recently I made the decision to buy a bike. Like many others, I’ve put on some COVID weight and thought incorporating some exercise into my daily routine would help. I didn’t have a bike, so I started shopping for one.

Google is good, but not that good; it could bring up a bunch of bikes when I searched but I still needed to filter my search in order to figure out which bike to pick. What type of bike? (e.g. commuter, electric, road, etc.) Which brand of bike? What size of bike? How much did I want to pay? What features and accessories did I want? Admittedly, I know little about bikes, but what I did know was what I wanted to use it for. I wanted a bike to commute 5 km to and from work, on a non-hilly bike lane. In addition to that, I didn’t want to spend more than $750.

I ended up choosing a 3-speed commuter bike, with hand brakes, and a step-through frame. I added a bell and a box that sits on a carrier on the back of the bike. Two months in and there are no regrets with the purchase – it fits my needs to a tee. I can wear a skirt to work, carry my laptop back and forth, and switch gears if needed when my legs are feeling tired.

 

The two most important questions when scoping an evaluation

When you are asked to conduct an evaluation for a program it can be like shopping for a bike – there are various types, sizes and budgets. Designing an evaluation that meets your stakeholders’ needs begins with a scoping process. There are a number of questions that you should ask to scope an evaluation (refer to our scoping guide for a complete list of questions), but ultimately scoping an evaluation focuses on purpose.

1.     What does the initiative do (and for what purpose)?

2.     Who wants to know what (and for what purpose)?

 

What does the program do and for what purpose?

With scoping you want to get a detailed understanding of the program by asking stakeholders about the program’s:

• history

• stage of implementation

• activities

• intended results

• target population

• scale

• budget

• measurement activities

When I ask stakeholders about the program, I am listening for details that help me understand the scope and complexity of the program. However, what is equally important during the scoping process is listening for any disagreements or gaps in what is known about the program and its purpose. Does the program’s theory of change pass the sniff test? In other words, is the program designed to bring about its intended results? Listening for these clues are important to note, as they will help guide your conversation for the next scoping question.

Who wants to know what and for what purpose?

There are many different reasons and uses for a bike. Similarly, there can be many reasons for conducting an evaluation and how those evaluation findings get used. However, the more focused you can be on whose needs you are meeting through the evaluation, the more useable the evaluation will be.  

When I ask stakeholders about the evaluation and its purpose, I am listening for details that help me understand who needs what information. However, I am also listening to see if expectations for the evaluation align with the details that were given regarding the program. For example, if you discover the primary purpose for the evaluation is to determine the value and future of the program then a summative evaluation is likely the right type of evaluation. For a summative evaluation we need a strong theory of change and program data to monitor changes over time. However, if you heard through your discussion to the first scoping question that the program does not have a clear, established theory of change and has not collected data then there is likely a discrepancy in expectations that you should address. Addressing the discrepancy means a more detailed discussions with stakeholders questioning them about the readiness of the program for a summative evaluation and advising on more appropriate evaluation support given the stage of the program.

Focusing your scoping process on understanding the program and evaluation purpose will equip you with the knowledge you need to develop a quote and/or an evaluation design. However, it also provides a framework for expectation setting with stakeholders that will result in more accurate quotes and appropriate, focused and useable evaluation designs.

 

Check out our guide to program scoping here

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---

 

 

Sometimes online resources just aren’t enough. Some questions call for discussion, for a second opinion – a coach. That’s why Eval Academy is launching its new evaluation coaching service.

Check it out

What is it?

An evaluation coach works with you on your evaluation goals, helping you to understand where and how to improve. You can evaluate – and you can do it better with a coach. Evaluation coaching can help you with:

• Developing your logic model

• Understanding stakeholder resistance

• Strengthening your evaluation plan

• Choosing your methods

• Editing your interview guides and surveys

• Crafting your recruitment strategy

• Overhauling your charts

• Transforming your evaluation reports

 

Who is it for?

An evaluation coaching session is ideal for those who need a little help in defining their evaluation plans, who aren’t sure if they’re choosing the right method, or who want an extra layer of review for their data collection tools. If you’re not sure how to get started with your logic model, evaluation coaching is for you. If you don’t want to hire an evaluator to conduct your evaluation but need a little inspiration, evaluation coaching is for you. Emerging evaluators can benefit by having a second opinion on their plans.

 

What’s involved?

In an evaluation coaching session, you set the direction. You’ll tell your coach what you want help with, and your coach will support your growth by answering questions and posing a few of their own. Your coach may send you additional resources or tips by email after your session is over. You may have just one session, or you may schedule a series to work through your questions in greater depth.

 

Who is the coach?

Your coach is a Credentialed Evaluator – that means they have the skills, knowledge and practical experience to be recognized as a competent evaluator. Your coach lives and breathes evaluation (well, at least most of the time) and loves to talk about the discipline. With over 15 years of experience in the research and evaluation fields, you’ll be well supported by your evaluation coach. Learn more about your coaches here, or see LinkedIn profiles for Shelby Corley and Kristy Madsen.

 

How does this work?

Start by following this link to our Teachable page to fill out the intake form. You’ll pay for and schedule your appointment, then receive a link to a video call platform. If you have documents to share, you can send them by email to your coach, or share your screen during your coaching session.

 

Evaluation coaching is not…

Coaching is not a replacement for basic training – there are more cost-effective ways to get the basics down first. Don’t think of a coaching session as a replacement for a research ethics board or an institutional review board, either – while your coach can certainly help to strengthen your project’s ethics, they can’t grant approval for anything (sorry!). Evaluation coaching is not a replacement for in-depth training in analysis – if you need a statistics refresher, this is not the place. Evaluation coaching is not a replacement for a consultant – your coach won’t do your evaluation for you, but they will help you do it better yourself.

 

What does it cost?

A lot less than hiring a consultant! Each session is a full hour, and you’ll only invest $200 Canadian.

 

Are there discounts available?

Discount codes are sometimes included in the Eval Academy newsletter – sign up so you don’t miss out.

 

Still not sure?

If you’re not sure if evaluation coaching is right for you, send us a quick email and we’ll help you decide.

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---

 

 

Evaluators have several options for transcribing audio from qualitative interviews, including voice-to-text software, outsourcing, and doing it ourselves. Depending on the budget for the project, you may not be able to afford software licenses or the cost to hire someone else to transcribe your interviews, so you might be left transcribing interviews yourself.

If you are taking the DIY approach, here are four tips to get you transcribing like a pro!

1. Decide on the appropriate level of accuracy

Before you start, you need to decide how accurate your transcriptions need to be. You have a range of options for how precise to transcribe, and your choice depends on how the data will be analyzed. Some commonly used levels of accuracy are (from most accurate to least accurate):

• Exactly verbatim: Type exactly what is said, including every “um,” “uh,” and “hmmm.”

• Intelligent verbatim: Type exactly what is said, EXCEPT for filler words that do not change the meaning. At this level, you would skip phrases like “um,” “you know,” and “like” when appropriate.

• Edited transcription: Skip irrelevant sentences that don’t relate to the evaluation. For example, someone might start telling a tangential personal story – if it is not relevant to the purpose of the interview, you might not transcribe it.

• Summary transcription: Only important passages are transcribed. For example, you may only require the answers to the questions in the interview guide, and any other comments are skipped.

When choosing a level of accuracy, you face a trade-off between the time it takes and the level of detail. Exactly verbatim transcriptions take the longest to type, while summary transcriptions are the fastest. However, summary transcriptions necessarily exclude some information, so you risk missing important data. It is up to you as the evaluator along with the stakeholders involved to navigate this trade-off between accuracy and resource-use in a way that makes sense for the particular evaluation.

2. Document interview meta-data

At the top of your Word document where you will transcribe the interview, you should include relevant meta-data (i.e., data about the interview). This could be:

• Interview ID#

• Project name

• Participant’s name or initials (remember to maintain appropriate levels of confidentiality)

• Interview group, if applicable (e.g., client, program manager, funder)

• Interviewer’s name

• Date of interview

• Location of interview

• Transcriber’s name

 

In the metadata section, also identify any abbreviations that will be used in the transcription. For example, you may simple use “P” to denote the participant, and “I” to denote the interviewer. My transcriptions usually look like this:

I: Can you tell me about your use of the program?

P: Sure, I started using this program when I first moved to the city.

I: When was that?

P: About a year ago.

 

3. Use audio software

Rather than using the default audio player on your computer, like iTunes or QuickTime, it can greatly speed up your transcribing to use an audio software with more advanced features. I recommend ExpressScribe because it is reasonably priced and has some really important capabilities:

1. Fine-tuned control over playback speed. It might seem counterintuitive, but slowing down the audio playback speed can actually increase the speed of your transcription. Most of us can’t type as quickly as people talk, so by slowing the audio down to about 50-60%, you will be able to type at the same pace as the speaker. This reduces the amount of time you spend pausing and rewinding the audio!

2. Global hot-keys. This was a game-changer for me: ExpressScribe allows you to control the audio playback (play, pause, rewind) using keyboard shortcuts that work even when the program is running in the background (that’s why they’re called global hotkeys). The set-up I use is:

   • Ctrl/Cmd + down arrow = Stop

   • Ctrl/Cmd + up arrow = Play

   • Ctrl/Cmd + left arrow = Step back 3 seconds

   • Ctrl/Cmd + right arrow = Step forward 3 second

With these advanced controls, you can play/pause and rewind without leaving the Word document you’re typing in!

4. Include time stamps

While you transcribe, it is helpful to include a time stamp in the transcription about every 5 minutes. This allows someone reading the transcript to easily find passages in the audio recording if necessary. I keep an eye on the audio player for when I reach an interval of five minutes, and I document it in the text using square brackets:

I: Can you tell me about your use of the program?

P: Sure, I started using this program when I first moved to the city.

[5:00]

I: When was that?

P: About a year ago.

Time stamps should also be included when you reach a passage you can’t understand. Especially when people refer to place names, people’s names, or words in another language, it can be hard to get the word right. This should be flagged inside square brackets along with the time stamp. For example:

P: After that, we went to [place name 12:05] to gather berries. 

---

After incorporating all of these tips, you can expect verbatim transcriptions to take about 3-6 hours for each hour of recorded audio. Keep this in mind while planning the project: 10 interviews, each an hour long, could take up to 60 hours to fully transcribe!

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---