evalacademy

I’ve been thinking recently about bias in evaluation methodology. There’s so many different types of bias it’s hard to keep them all straight, never mind mitigating them all!

One particular area has me interested: 

What bias does offering an incentive to participate introduce?

So, let’s broaden the question and explore incentives for participation generally (don’t worry, we’ll cover bias too).

As evaluators, one of our primary activities is data collection. Some of the most common methods include asking other people to share their thoughts or experiences in surveys, interviews, or focus groups. Sometimes it can be difficult to get people to agree to participate. It’s very enticing to offer an incentive to boost that response rate or sample size. 

---

Why offer an incentive?

Many argue, myself included, that those with lived experience are experts and knowledge keepers, and should be compensated for sharing their expertise. So, incentives can serve a moral purpose, offering compensation in exchange for expertise.

Incentives can also serve the function of attempting to mitigate non-response bias. Theoretically, by increasing response rates, your data will be that much more robust and representative of the population you are hoping to report on.

---

Is cash king?

The two most common forms of incentives are cash or gift cards. So, which should you use, when, and why? 

When I was a graduate student, I ran a study that collected data from individuals experiencing challenges with substance use. The Research Ethics Board (REB) at the university would not let me offer cash as an incentive to participate. It never sat well with me. Ethically, who am I, as a researcher or evaluator, to decide whether a participant “deserves” cash or not? Who am I to declare that they can only spend earnings at gift card locations?  Besides, there’s nothing stopping anyone from re-selling gift cards for cash. It seemed like a misstep to me.

More recently I offered a cash incentive to participants for an interview. Many of those participants voluntarily expressed appreciation for the cash support. That project felt better for me.

So, morally, I am a believer that cash is appropriate. Functionally, research says that money is more motivating than other gifts. So, yes, cash may actually be king.

---

Low budget options

What happens if you don’t have the budget for cash or gift cards? There are still ways to offer incentives.  Alternatives could be:

• Access to products or services (e.g., 1 month of free access to the services of the organization)

• Company swag or merchandise 

• Discounts or coupons (e.g., a 50% discount on your next program registration)

Or perhaps you don’t have the budget to offer everyone an incentive, but you can afford something. In these cases, I’ve used the “chance to win” approach. This is more common in surveys, whereby filling out a survey the participant is entered into a draw to win something, usually a gift; but there’s no reason this couldn’t also be cash. 

CAUTION: Make sure that your cash-alternative prizes (e.g., gift cards or products) don’t introduce new forms of bias!  If you advertise that participants get gift cards to a specific store or restaurant you may be biased toward people who frequent those businesses. If you advertise that participants get a discount on products or services, you may be biased toward people interested in those products or services. 

---

What about coercion?

I said we’d talk about bias. When offering an incentive, the risk is that it introduces coercion: the practice of persuading someone to do something. It’s a fine balance between motivating someone to spend their time sharing their experience with you in exchange for a token of appreciation, versus inducing a feeling of being compelled because the incentive is so enticing. Coercion exists when a participant accepts otherwise intolerable risk to receive the incentive. I think this risk is real, however, some research has shown that any risk associated with participation is not influenced by incentives. This study found no relationship between the size of the risk and the size of the incentive. 

It may be impossible to prove that an incentive isn’t coercive. The best way to protect participants from risk isn’t by making changes to your incentive, but by examining policy, protocol, and consent within your project or program. Probably the best mitigation strategy is informed consent, letting participants know about their rights to refuse, clearly outlining any risk to participation, and describing how participation does or does not affect their access to services.  Related to this, refusing access or service based on participation is definitely coercive!

---

What impact does offering an incentive have on your data?

One reason to offer an incentive is to increase the response rate. Increasing response rates should improve your data quality through more valid, representative data. However, there are negative impacts as well – including introducing profile bias based on the specific incentive (as cautioned above). Incentives could also change the responses themselves, either because participants feel compelled to respond positively to receive the incentive, or with even more subtle changes, like improved mood. However, some research suggests that these influences are unlikely.

Ideally, we want people to participate because they care about the outcome, or they have a vested interest in the program or organization. However, motivations to participate cover 3 primary reasons: 1) altruism 2) topical reasons, e.g., the desire to share a specific positive or negative experience, or interest in the topic or 3) egoist reasons, i.e., for the incentive. Most evaluations probably have a mix of these motivations. It can be difficult or impossible, and probably unnecessary to tease them apart. Ensuring strength in your data collection tools and processes will help to mitigate any risk associated with egoist participation.

---

What is participation worth?

A colleague of mine recently had to counsel a client on whether $100 for survey participation was coercive or not. While it’s not common to have to debate if the incentive is too much, it’s still a good lesson in ethical practice. And begs the question, what is participation worth?

There’s no one answer. In fact, there isn’t even a “right” answer. It depends on several factors including:

• time burden: is it a 5-minute survey, or a 3-hour focus group?

• how intense is the participation process?

• how specific or niche, and therefore possibly difficult to reach, is your target participant group?

• how specialized is the knowledge or experience the participant group brings? 

• your project budget!

The higher the burden or intensity, or the more specialized the nature of participation, the higher the perceived value.

Wellesley Institute, out of Toronto, did some exploration in 2018 and found that in research (i.e., not evaluation), participants were paid, on average, $30 for an interview, $25 for a focus group, and $20 for a survey. In my own experience, this is pretty close in evaluation as well. Depending on budget and level of burden (i.e., time commitment) I’ve offered up to $50 for interviews. Wellesley suggests that a good rule of thumb is $25 per hour of participation.

---

Appropriateness of incentives

While cash may be king, there are other considerations when deciding on incentives. I’ve recently been working on a project where paying for services may introduce trauma for some participants. I’ve worked closely with this project team to develop a plan where the participants are compensated for sharing their experiences, but without the transactional nature of giving an incentive after participation. Specifically, persons with lived experience are invited to participate in an interview; regardless of participation, they are also offered extra support to meet any emergent needs they may have. That extra support is not based on their participation, but on the invitation.

Vulnerable populations may respond to incentives differently, by being more likely to accept risk. Vulnerable populations are also frequent target populations for research and evaluations.  This is a great resource that offers more guidance on compensation for persons with lived experience, including: discussing compensation clearly and upfront, offering options, paying in cash, and offering to pay for additional costs (like child care).

Depending on your participant population there are groups and cultures where exchanging gifts is an act of respect. Consider reviewing your incentive practice from a lens of diversity and intersectionality.

Finally, there is some grey area about who is offered an incentive. For example, you are evaluating a program, commissioned by the organization that runs the program. Part of your evaluation plan is to interview the program participants. I think we can safely agree that offering some form of incentive is justified. Part of the same evaluation plan is to interview the organization leaders and the facilitators of the program. Does it make sense to offer them an incentive? Or is it “part of their job” to participate? To pay them for participation is essentially paying them out of their own budget. In my experience, I have differentiated between those participating in a professional capacity (i.e., not incentivizing staff) and those volunteering to participate (i.e., incentivizing lived experiences or program participants).

A key to ensuring your incentives are appropriate is an open dialogue with your project team, which hopefully includes a representative from your target population. Engage in your own reflective practice as an evaluator and seek input from your colleagues to identify risks you may have missed.

---

Other Considerations

1. To offer an incentive, you may need to collect some contact information. With so much data collection being done virtually these days, many incentives are offered electronically, where email addresses are required. This has at least two impacts:

   • Is your participant population likely to have an email address and the capacity to retrieve and access an e-transfer or electronic gift card?

   • Does collecting contact information introduce any ethical risk around identification or breach of confidentiality?

2. In my experience where most data collection is happening virtually, the idea of bringing food and/or beverages is a moot point, but in the past, I have offered focus group participants snacks and drinks. These “perks” to participating are also an incentive and should be considered thoughtfully.

3. Arguably outside the discussion of incentives specifically, is reimbursement. Some participants may need to travel or enlist child or elder care to accommodate their participation. It is common practice to offer reimbursement for such expenses, which can take many forms, including cash, bus tickets, cab fare, mileage payment, etc.

4. I’ve had discussions that the incentive isn’t an incentive, per se, but a “thank-you”. This can be true if the thank-you is given after participation without prior knowledge, or perhaps if the value is so low that it would in no way incentivize anyone. Similarly, some participants have regular hourly rates and can be offered an honorarium to compensate them directly for their time. These distinctions between a thank-you, an honorarium, and an incentive are very blurred. As always, data collection processes and informed consent are the best ways to mitigate risks!

---

There’s a lot of research about incentives, much more than I’ve touched on here. Hopefully, I’ve covered some of the important ground to guide you in deciding whether to offer an incentive and what form and value that incentive should be.

Remember to include the use of incentives in your evaluation contracts. The expense can be significant!

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address


Sign Up

We respect your privacy.

Thank you!

---

A Log Frame is a tool that has mainly been used for designing, monitoring, and evaluating international development projects. Using this tool is a way of structuring the main elements in a project or program, and highlighting the logical connections between them. 

In this article, we explain what a Log Frame is (spoiler: it’s not a logic model), why they can be useful tools for program planning and evaluation outside of international development, and how to make one of your own.

---

What is a Log Frame?

The Log Frame was developed back in 1969 for the U.S. Agency for International Development (USAID). It was developed as a planning tool for international development programs in the form of a matrix which presents a program’s main goal, activities, and what these activities are expected to lead to.

This visual approach helps you to think about the relationships between available resources, planned activities, and the desired changes or results. This structure also helps to explain the linkages between a program’s components.

It follows this main idea:

1. The successful completion of these activities = the production of these outputs

2. The production of these outputs = these changes through outcomes

3. These changes through outcomes = the achievement of these objectives

4. The achievement of these objectives = contributions to the larger goal

Therefore, this tool can help program planners to explain how they believe change will be realized. For evaluation purposes, the Log Frame also identifies the measures and indicators that will help to monitor the program’s anticipated results. 

Although there are a few models that focus on logical links between a program and its contribution to success, be careful not to confuse Log Frames with the following:

• Results Frameworks: Log Frames are focused more on how you will get to your program’s goal. Results Frameworks focus more on explaining the program’s results.

• Logic Models: although the two are very similar, a Log Frame is depicted using a matrix or table, whereas a Logic Model is shown using a flow chart.

• Theory of Change (ToC): ToC is also used for program design, whereas Log Frames are useful mainly for evaluation only. A ToC is also more explanatory and for explaining more complex initiatives, whereas Log Frames are descriptive and better placed for small to medium sized projects. Log Frames don’t easily capture the how and why in the same way a ToC does.

---

When and why are Log Frames used?

Since their development for USAID, Log Frames have become a standard approach required by donors for grant applications. For international development programs, they have become a staple of project planning and evaluation. Due to their utility and accessibility, they’re now also being used for programs outside of the international development realm.

Most Log Frames are developed during program design and are updated throughout the program’s lifespan. Log Frames are not set in stone and should be flexible to the program’s needs and any changes happening on the ground. Developing a Log Frame at the program planning stages helps to involve the whole team and allow key stakeholders to provide ideas about how they see the program operating to reach its goals.

Log Frames can be used for the following:

• To confirm the theory of why your program will result in the desired change. Log Frames can help you to see whether your program will really work and identify any flaws in the theory. It’s all about being logical!

• Help you in making strategic decisions that align with the logic of how your program will contribute to change and reach its goal. It can also help you to allocate resources to where they will be used most effectively.

• To support transparency by clearly describing what your program aims to achieve and how. This can help your program to be more attractive to donors by clearly and logically explaining your ideas. It can also help you to make sure that all staff are on the same page.

• By including indicators of how you will measure change, a Log Frame helps you to develop how you will monitor and evaluate your program. This provides the groundwork for how an evaluation can measure the impact of your program. 

---

What are the key components of a Log Frame?

A Log Frame is commonly presented as a 4×4 table with 16 cells, though this can be modified if needed. Each row represents types of events that take place to help the program achieve its goals. Although the wording may differ slightly, these include:

• Main Goal

• Outputs

• Outcomes

• Activities

The first column describes how the program will reach its objective. The second and third columns summarise how the program’s achievements will be tracked through indicators (i.e., measurement of change) and sources of information (i.e., information needed that will allow indicators to be measured). The last column lists the assumptions or risk analysis. These are the factors outside of the program’s control that are necessary to ensure the program’s success.

---

How do I develop a Log Frame?

It is important to develop a Log Frame with key stakeholders included in the program. Collaborating helps to make sure that it is not developed through a “top-down” approach and leads to better program planning by ensuring that everyone is on the same page in terms of shared objectives. Developing a Log Frame often works best through a mind-mapping session.

Firstly, it is important that there is a clearly established problem your program aims to address. You should also be very aware of the context within which your program will operate. It helps to think about setting your goals first and then working backwards to the grassroots to see what you need to do to reach these goals.

The main questions to consider in the development of your Log Frame include:

• What will our program achieve?

• What activities will we complete?

• What resources will we need to do this?

• What are the problems and challenges we might face along the way?

• How will we measure the progress and outcomes of the program?

There can be more than one indicator and assumption listed for each level, but it is good to keep these manageable by using tools such as SWOT analysis (see our free template here), Stakeholder Analyses, and a Risk Matrix.

A Log Frame should also be tested to check the logic. A simple test is to ask the following:

1. IF these activities are undertaken AND the assumptions hold true, THEN the intended outputs will be created

2. IF these outputs are delivered AND the assumptions hold true, THEN the purpose will be achieved

3. IF the purpose is achieved AND the assumptions hold true, THEN the intervention will have contributed to the overarching goal

(Adapted from DFID, 2011)

Once developed, make sure that the Log Frame doesn’t just sit on a shelf throughout program implementation. Review it frequently and use it to support decision-making, manage activities, assess progress, and keep stakeholders aware of plans to monitor progress.

---

Pros & Cons of using a Log Frame

Benefits of a Log Frame:

• It ensures objectives are clear and measurable

• It ensures concrete evidence for achievement is collected

• Because risks and assumptions are made explicit, problems can be analyzed systematically

Weaknesses of a Log Frame:

• It may cause rigidity in program management if it is not viewed as something that can be updated throughout program implementation

• It is a “one size fits all approach” which does not always capture the complexity and context of a program

---

Have you worked with Log Frames before, or have questions? Comment on this article or connect with us on LinkedIn or Twitter!

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address


Sign Up

We respect your privacy.

Thank you!

---

Kirkpatrick is probably one of those names/methods you’ve heard about in your evaluation career, but have you ever used it? I’m surprised how many evaluators I talk to that haven’t because I find it pretty useful and straightforward with tonnes of resources to support you. 

I love experiential descriptions, that is, reading about how someone else applied a method in a real-world scenario: the ups and downs, the backtracking and lessons learned.  

So, having used Kirkpatrick a handful of times on a few initiatives, here is my account of how to use the Kirkpatrick model in your evaluation planning, implementation, and reporting.  

---

What is the Kirkpatrick Model?

The Kirkpatrick model was originally developed in the 1950s but gained popularity in the 1970s as a way to evaluate training programs. Donald Kirkpatrick proposed 4 levels:

Kirkpatrick can apply to evaluating any type of educational endeavour – where participants or attendees are intended to learn something and implement those learnings.

---

How do I use the Kirkpatrick Model in Evaluation?

Step 1: Do some basic research. 

The model has spawned a website of Kirkpatrick Partners. They host training and events; they have a newsletter, blog and resources to support your use of Kirkpatrick. You can even get certified in using Kirkpatrick. (Disclaimer: I am not certified.) My goal for the rest of this article will be to show you how I’ve actually used Kirkpatrick and some of my thoughts along the way. 

Step 2: Incorporate Kirkpatrick in your evaluation plan. 

Unless you are exclusively evaluating a training program, I’ve found that Kirkpatrick is often a part of the evaluation plan, but not the only part. 

I’ve used Kirkpatrick on an initiative that trained people to facilitate quality improvement in primary care practices. I used Kirkpatrick for the training but had lots of other evaluation questions and data sources for the quality improvement efforts and outcomes in the primary care clinics. 

Like RE-AIM, the good news is that Kirkpatrick gives you a solid head start on your training-related evaluation questions. 

Your overarching evaluation question might be something like: 

To what extent did training prepare people to [make the intended change]? 

Or 

How effective was the training at improving [desired behaviour]? 

From there, use the four levels of the model to ask specific questions or craft outcome statements (See below for a detailed explanation of each level):

• Level 1: Reaction

• Level 2: Learning

• Level 3: Behavior

• Level 4: Results

Step 3: Reporting  

I don’t think I’ve ever used the Kirkpatrick levels explicitly in my reporting. I think most audiences are not interested in the theory of evaluating a training program but are more interested in answering “did the training work?” As I mentioned, Kirkpatrick has usually been only a part of my evaluation planning, so, similarly, reporting on the effectiveness of training is usually only one part of my reporting.  

Because Level 1 of Kirkpatrick assesses formative questions about training – things that you could change or adapt before running the training session again, I have often produced formative reports or briefs that summarize just Levels 1 and 2 of Kirkpatrick. This promotes utilization of evaluation results! 

Using surveys (or a test!) likely gives you some quantitative data for you to employ your data viz skills on. But keep in mind that it’s not necessary for you to report all of the data you gather. You likely don’t need a graph showing your reader that the participants thought the room was the right temperature. Sometimes less is more and a couple of statements like “Participants found the training environment to be conducive to learning and found the training to be engaging. However, the days were long, and they recommended more breaks.” can cover a lot of your Reaction results. We’ve got lots of resources to help you with your report design. 

Sometimes reports, particularly interim reports, are due before you can get to behaviour change or results. This is actually one of the criticisms of Kirkpatrick – that many evaluations will cover Levels 1 and 2 thinking them sufficient but fail to invest the time and resources into ensuring the behaviour changes and outcomes are captured. This is one of the reasons that planning an evaluation concurrently with project design is helpful and can prevent these shortcuts. 

---

How to use the 4 levels of the Kirkpatrick Model

LEVEL 1: REACTION 

Reaction is about all those things you think immediately after you’ve attended a session. These are less about what you learned, and more about: Was the trainer effective? Was the environment supportive to learning? Was the day interactive and fun, or didactic and tiring? 

Reaction is all of those things that influence learning. They may seem of less importance but contribute a lot to how much a person learns, retains and acts on. 

Let’s use a scenario where you are evaluating a training program designed to teach participants how to implement COVID-19 safety protocols in the workplace. 

For Reaction, the evaluation is almost entirely content agnostic, so it matters less what the training program is about, and more about the delivery, for example: 

The venue was appropriate. 

The presenter was engaging. 

The training was relevant to my work. 

This is almost always captured in a post-training survey, which of course could be paper for in-person events, or QR codes/links for virtual events. It could be emailed out to participants after the session, but we all know that response rates are much better when you carve out 5 minutes at the end of the last session to complete the evaluation.  

The learning here is about how you can tweak the delivery of the training. Was the room too cold? Was the presenter about as engaging as Ferris Bueller’s homeroom teacher? 

LEVEL 2 – LEARNING 

As the name says, level 2 is about assessing what the participant learned. I like to think: Knowledge, Skills, and Attitudes. Assessing learning can (and arguably should) touch on each of these. So, your questions might be: 

I can explain our COVID-19 safety policies. [knowledge] 

I understand why our COVID-19 safety policies are important. [knowledge] 

I am confident that I can enforce our COVID-19 safety policies. [attitude] 

I learned 3 ways to build buy-in about our COVID-19 safety policies. [skills] 

As these examples imply, it’s very common for these questions to be included on a post-workshop survey. I usually embed them with the Reaction survey, so participants fill out one survey after training. I try to keep the total question count under 20, usually Likert Scale, with some opportunity for qualitative feedback. 

TIP: If it’s important to you to be able to say that the training was the reason for the results you get, you’ll want to consider a baseline survey – that is, a survey with the same learning questions as the post-training survey, but it’s completed before training. That way, any change that you see can be more strongly linked to the training that they received, as opposed to pre-existing knowledge, skills, or attitudes. 

If your training has learning objectives, this is a good place to look for what knowledge, skills or attitudes the program is intended to impact. 

Personally, I’ve always lacked confidence in results that come from self-assessed ratings of knowledge, skills, and attitude. We know there are several biases in play – including a tendency to not use the full range of a scale, and to rate yourself positively. One way to mitigate these confounders is to test the learning. Instead of asking for an opinion about what they learned, the post-training survey could be formatted to actually test the learning: 

List three COVID-19 safety protocols implemented at your workplace. 

Which of the following is a reason why these COVID-19 protocols were selected. [multiple choice] 

Describe one way to build buy-in with your staff around COVID-19 policies. 

The downside is that not only are these potentially more resource-intense to analyze and report on, but most programs and organizations are worried about the impression it gives to test participants or workshop attendees. We all hate tests, right!? I haven’t had much luck convincing a program to use testing as opposed to a self-rated survey. Let me know if you’ve fared any better! 

LEVEL 3 – BEHAVIOUR 

Here is where it gets a bit tricky. To measure behaviour you need to have access to the participants after a set amount of time; your participants need an opportunity to put their learning into action.  

Assuming you have email addresses or perhaps a reason to bring this group together again, I’ve assessed behaviour change through surveys (yes, surveys again). These surveys are usually sent over email after a pre-determined amount of time. The time interval depends on the learning you are hoping they achieved and the opportunities to implement it. I’ve used anywhere from 4 weeks to 6 months. 

In the last [time interval] I have: 

Explained why COVID-19 safety protocols are important. 

Used techniques learned at training to deal with an individual not wishing to follow COVID-19 safety protocols. 

It’s possible that you could re-administer the baseline/post-training survey as a [time interval] follow-up. This would help you to assess retention of learning and may get at some behaviour change too. The difference though is that the assessment of learning was likely opinion-based or in-the-moment, while assessment of behaviour change is retrospective – it’s not about what they have knowledge, attitude or skills to do, or what they intend to do, it’s about what they did do. 

Another option is to change your data collection strategy: 

• Observation: can you watch attendees implementing the training? 

• Interviews: can you gather data from people who didn’t attend the training to understand what changes they have seen? Or interview the training attendees to understand what they’ve put into practice (and in what context) 

• Role playing: If you don’t anticipate being able to reach the attendees after a time period, perhaps role playing (and observation) could be a part of the training curriculum. Can attendees demonstrate what they have learned? 

• Evidence of action: perhaps you can access evidence that proves the training resulted in action – maybe participants were asked to write business plans (how many were written?) or were asked to design and implement a communication strategy about COVID-19 policies (was it done?) 

In our scenario, observation may look like observing building entrance processes and observing, counting or noting the number of times a COVID-19 policy is explained or enforced. Or, you could survey staff to see how well they understand the COVID-19 policies (assuming their knowledge is a result of the training to select staff). The key to Level 3 is the demonstration of the behaviour. 

LEVEL 4: RESULTS 

Results brings everything full circle. It drives at the purpose of the training to begin with – what is the impact of the training. In our scenario, the ultimate goal may have been to have staff and patrons compliant with COVID-19 safety protocols. Assessment of Results will be a reflection of this. 

# of encounters with non-compliant staff 

# of encounters with non-compliant patrons 

# of COVID-19 outbreaks at the workplace 

Assessment of Results may start to blur the lines with the rest of your evaluation plan. Perhaps the training is part of a larger program that is designed to create a safe environment for staff and patrons, part of which was implementing COVID-19 policies.  

Assessment of Results likely requires an even longer time frame than Level 3, Behaviour.  

---

Drawbacks of the Kirkpatrick Model

I’ve found Kirkpatrick to be useful to ensure that evaluation of a training program goes beyond simply measuring reactions, however, the temptation to cut the method short and only measure reaction and learning is common practice. 

In perhaps the simplest approach, Kirkpatrick is a survey-heavy method – which relies on adequate response rates and is littered with biases. More rigorous methods to measure the four levels – observation, interviews – are likely more time consuming and resource intensive. It’s a tradeoff. If you go the survey route, here are some tips on how to use Likert scales effectively. 

Another common criticism of Kirkpatrick is the assumption of causality. The model takes the stance that good, effective training is a positive experience (Level 1), results in new learning (Level 2), and drives behaviour change (Level 3), which leads to achievement of your desired outcomes (Level 4). It fails to account for the environmental, organizational, and personal contexts that play a role. Whether or not an organization supports the behaviour change or empowers attendees to make change matters, regardless of how fantastic the training was. The further you move through the four levels of Kirkpatrick, the looser the link to causality. 

---

Kirkpatrick Model Return on Investment 

At some point, an unofficial 5th Level was added to Kirkpatrick – the Phillips Return on Investment (ROI) model (and sometimes this is not a new level but tacked onto Level 4 – Results). The idea here is that the cost of running training should be absorbed by the positive financial impact on the organizational improvements that come from the training and subsequent behaviour change.  

I’ve never actually used the ROI, but there are lots of resources out there to help you. 

---

Other Uses of the Kirkpatrick Model

Here at Eval Academy we often talk about the importance of planning your evaluation right along with program design. Kirkpatrick is no exception. The four levels can offer guidance and perspective on how to actually design a training program from the start. By working backwards, you can ask questions that ensure your training program has the right curriculum and approach to achieve your goals: 

1. What are we trying to achieve? (Level 4 – Results) 

2. What behaviours need to happen to realize that goal? (Level 3 – Behaviour) 

3. What do attendees need to know or learn to implement those behaviours (Level 2 – Learning) 

4. How can we package all of this into a high-quality, engaging workshop? (Level 1 – Reaction). 

I’ve only ever used Kirkpatrick as a planned-out evaluation approach. It may be trickier to use this model for training that has already happened: you’ll have missed the opportunity for baseline data collection, and importantly, depending on the time that has passed the Reaction results may be less reliable. 

---

I like Kirkpatrick because it’s simple and straightforward. This simplicity is actually one of the main criticisms about it, and its failure to recognize context. However, in my experience, the guidance offered by the four levels helps to shape my thinking about how to evaluate a training program. Because a training program is often only part of a larger initiative, perhaps the lack of context hasn’t been as blatant for me. 

I’d love to know your experiences with Kirkpatrick. Have you evaluated all four levels? Have you assessed ROI? 

For other real-life accounts of how to use an evaluation methodology, check out our articles on RE-AIM, Developmental Evaluation and Outcome Harvesting. 

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address


Sign Up

We respect your privacy.

Thank you!

Sources: 

Graham, P., Evitts, T. & Thomas-MacLean, R. (2008) “Environmental Scans: How useful are they for primary care research?”. Can Fam Physician. 54(7): 1022-1023. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2464800/ 

Polanin, J.R., Pigott, T.D., Espelage, D.L. & Grotpeter, J.K. (2019) “Best practice guidelines for abstract screening large-evidence systematic reviews and meta-analyses”. Res Synth Methods. 10(3): 330-342. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6771536/ 

---

Whether starting a new program or making changes to an existing one, you’re going to be faced with numerous questions about how best to move forward. And some of those questions can feel pretty daunting! Many organizations use environmental scanning as a method of strategic planning to gain insights and gather information on how to put their best foot forward. 

This article is aimed at those who are new to environmental scanning and are looking for new ways to support program planning and improvement. 

---

What is an environmental scan?

Environmental scans started out as a tool for businesses to find and organize information that could be used for decision-making.

The process of environmental scanning includes finding, gathering, interpreting, and using information from the internal and external environments of an organization to help direct future action.

This method uses multiple strategies for information collection such as focus groups, in-depth interviews, surveys, literature reviews, and reviewing personal communications, policy analyses, and internal documents.

The results of an environmental scan can be very useful in helping an organization or program shape its goals and strategies.

---

What’s the difference between an environmental scan and a literature review?

Unlike a literature review that searches for published, peer-reviewed articles, an environmental scan also examines unpublished literature and publicly available information.

An environmental scan also incorporates methods such as interviews and focus groups not used within literature reviews.

---

Why should I complete an environmental scan?

Scanning the environment is an important part of strategic planning and has been linked to improved organizational performance.

Environmental scans can help plan for the future, provide evidence about potential directions for an organization or program, raise awareness of risks or issues, or help to initiate a new program.

The diversity of information sources and types of information gathered through environmental scans have helped organizations to effectively plan and implement programs across a variety of sectors.

---

When should I do an environmental scan?

You can take a reactive approach (e.g., a challenge has arisen that needs to be addressed) or a proactive approach (e.g., a new program is being implemented and you want to ensure its success) to scanning.

Scanning is typically more frequent when an organization or program has higher levels of perceived uncertainty, such as when taking a new direction or during its start-up.

---

How do I complete an environmental scan?

To complete an environmental scan, we typically follow six main steps: 

1. Identify the purpose of the environmental scan and your topics of interest 

2. Identify the research question(s) 

3. Identify the activities you will complete and where you will look for information 

4. Create a list of keywords and search terms 

5. Catalogue the information systematically  

6. Present the information in a way that is useful for your organization 

 

Step 1: Identify the topics of interest and the purpose of the environmental scan 

Before jumping onto the internet to search for information, it is first important to specify the purpose for the environmental scan and identify your topics of interest. This will help to anchor the process, focus your time and resources, and avoid those rabbit holes! Although an environmental scan should remain flexible to allow for new questions that might arise, having a clear scope will help you stay focused. 

For example, let’s say we’re looking to create a new program in the Vancouver region that aims to provide mental health support to isolated and elderly individuals, but we’re not sure what similar programs are already out there. To avoid duplicating services and making sure we reach a portion of the target population that isn’t already being served, the purpose of our environmental scan would be to: 

“Support the development of a program that aims to improve the mental health of the isolated and elderly by learning about available programs and identifying their strengths and weaknesses.” 

Our topics of interest would then be: 

• Examples of programs that are currently available in the Vancouver region which aim to improve the mental health of people aged 75+ who are isolated (e.g., live alone) 

• Identifying what the programs currently do well including effective modalities, treatment options, and who they are reaching 

• Identifying gaps in the programs such as treatment options that are not provided and target populations that are missed 

Step 2: Identify the research question(s) 

Now that you’re clear on your environmental scan purpose, you can narrow this even further by identifying your research questions. This is the broad rule for knowing when to stop your search and will help you to decide whether an article or website is worth continuing to explore.  I tend to recommend having between 1 and 3 research questions for an environmental scan that dig a little deeper into your topics of interest to really pull out all the juicy details you’re looking for! 

The research questions for our example environmental scan could be: 

1. What programs are currently implemented in the Vancouver region for the isolated and elderly (75+ years of age) which focus on improving their mental health? 

2. How do these programs operate and what makes them effective? 

   • Who is the audience that they’re reaching and how do they reach them? 

   • What modalities do they use (e.g., group settings, individual therapy)? 

   • What treatment options do they implement? 

3. Are there any gaps in these programs? 

   • Are there any of the target population that are not effectively captured by the program? 

Step 3: Identify what environmental scan activities you will complete and where you will look for the information  

Once we have our topics of interest, purpose, and research question, next it’s time to identify what activities you’ll complete and where you’ll look for the information to answer your questions. In an environmental scan, your activities can focus on understanding the internal (your organization’s) or external (other organization’s) environment to the particular topic. This will help to provide input into strategic thinking, decision making, and planning. 

For understanding the internal environment, appropriate activities could include: 

• Reviewing organizational documents (e.g., organizational strategy, policies, or internal communications) 

• Interviewing members of staff  

For understanding the external environment, appropriate activities could include: 

• Grey literature searches on the internet (e.g., Google search) 

• Interviews or surveys with organizations or individuals of interest identified through your online search 

Don’t forget that you’ll also need to present your findings in an appropriate format for your organization whether that’s a report, infographic, or a presentation! (See Step 6 for further info) 

Although looking online for information is important in an environmental scan, involving stakeholders can be the key to its success. Frequently grey literature searches like reviewing websites can leave you falling short on all the details needed to answer your research questions. Reaching out to key individuals at the organizations of interest with an invitation to take part in an interview or survey can help you to fill these gaps. For example, many Indigenous organizations will have reports and summaries or have participated in other scans and engagements, so having the opportunity for Indigenous organizations to recommend grey literature is important to help fill gaps in knowledge and understanding. 

Before reaching out to stakeholders, make sure you have a clear understanding of what information is needed from them. This can help with engagement fatigue and make sure you’re asking the right questions. You should only ask them for information that you are not able to uncover on your own. Create a clear plan for conversations with participants, such as having a clear set of questions or requests for further information. Be prepared to answer further questions about the environmental scan process and why you’re interested in learning more about their work and organization. Lastly, be aware of who you are interviewing and methods for appropriately and respectfully engaging them. For our example, if you are interviewing Indigenous Elders then be aware of the protocol with visiting Elders and what you need to bring to respect the knowledge and teachings you will receive from Elders. 

The type and volume of activities you’ll complete are often affected by the amount of time that you have to finish the environmental scan. If a timeline isn’t established for you by a funding agency or by your leadership, it is important that you create your own timeline from the outset to help you plan and stay on task. If interviews of surveys are part of your environmental scan, make sure you allocate yourself plenty of time for creating the survey tool and interview guides, reaching out to your participants, collecting and analyzing the data, and interpreting your results.  

The activities for our example environmental scan could be: 

• Environmental scan of grey literature looking for examples of similar programs in Vancouver. Best practice examples from other geographies may also be included if “gold standards” are discussed 

• Contacting organizations who have implemented similar programs for follow-up questions through telephone interviews 

• In-person interviews with members of the target audience to learn how future programs could be improved for them 

• Presenting a selection of similar programs and treatment options through a final report 

Step 4: For online searches, you’ll need to create a list of key words and search terms that you will use 

To make sure you’re looking for information in the right places while searching online, it’s important that you create a list of key words and search terms to help guide your search. 

To start this process, make a note of all the key terms that will help you to capture your topics of interest. Cast your net wide to make sure you don’t miss anything. Then break down these topics into clear concepts and keywords. 

For our example environmental scan, a list of key terms could be:  

• Mental health programs 

• Isolated 

• Elderly 

• Vancouver 

• Effective modalities 

• Treatment options 

Using key terms such as “Indigenous”, “Female”, “LGBTQ2S+”, “Newcomer”, “Black” amongst others will help ensure an intersectional lens to your search. 

The next step is to identify any of these concepts that could be expressed using related terms or synonyms. You can make a list of these under each of the key terms. 

Boolean searching through creating search strings is a common way to look for information in online databases and can also help to frame internet searches. Boolean searching uses connector works such as AND, OR, and NOT to create phrases based on rules and search logic.  

We’ll dive deeper into how you can systematically search online databases in our upcoming article on literature reviews! 

Step 5: Catalogue the information systematically 

Once you’ve found your sources, it is important to catalogue the information in a systematic way that links back to your research questions. Using a table like the one below that breaks the sources down by your topics of interest can make it very clear and help you when having to present your findings. 

Once you’ve catalogued your information systematically, you can then reach out to organizations and key stakeholders to request their participation in an interview or a survey to help you fill in the gaps.

Step 6: Present the information in a way that is useful for your organization 

Once you’ve organized, analyzed, and interpreted your information, it is crucial that you present your findings in a way that is useful for your organization. If an appropriate means of presenting the findings is not given to you by leadership, consider your audience and how the information will be used. This will help you to decide the best way of presenting your results. Reflect again on your topics of interest and make sure you present your information in a way that answers the research questions whether that’s in a summary report, infographic, or presentation to your organization. Lastly, make sure your work is disseminated and shared with all relevant stakeholders! This includes disseminating your work to anyone who participated in the scan, including any Indigenous stakeholders to align with OCAP principles. 

---

What’s your experiences with Environmental Scans? Comment on this article or connect with us on LinkedIn or Twitter! 

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address


Sign Up

We respect your privacy.

Thank you!

Sources: 

Graham, P., Evitts, T. & Thomas-MacLean, R. (2008) “Environmental Scans: How useful are they for primary care research?”. Can Fam Physician. 54(7): 1022-1023. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2464800/ 

Polanin, J.R., Pigott, T.D., Espelage, D.L. & Grotpeter, J.K. (2019) “Best practice guidelines for abstract screening large-evidence systematic reviews and meta-analyses”. Res Synth Methods. 10(3): 330-342. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6771536/ 

---

In this follow-up article, we will walk you through a step-by-step process of how you can develop your own Theory of Change (ToC) diagram for evaluation purposes.

You can also find our Theory of Change template here which you can use to support the development process. However, there is no one right way of developing a ToC.

Some people prefer working backwards from the program purpose, and it often depends on the program and the information that is already available to what method would work best. The method described below is one that I have found particularly useful from past experience.

---

When developing a ToC, it is important to keep the following points front of mind: 

• A ToC should be credible. It should be based on the insight and experience from different key stakeholders  

• A ToC should be achievable. You need to have the necessary resources to complete the desired change processes. Setting out to solve world hunger with a small budget isn’t the way to go! 

• A ToC should be supported. All key stakeholders should have some level of involvement in defining a ToC. Key stakeholders also need to agree with the ToC to ensure buy-in and understanding across all levels 

• A ToC should be testable. The diagram and accompanying narrative need to be a complete picture of the program. However, they also need to be clear, concise, and not over-complicate the description of the program’s work. Making sure you have indicators of achievement for each outcome will help you to collect the necessary data to test your ToC 

---

How to Host a Theory of Change (ToC) Workshop

Developing a ToC should be a highly collaborative process. The best method I have found to develop a ToC is by holding a 2-hour workshop (which can be broken up into two parts if needed) with key stakeholders. The first step in your process should be to identify and gather all of the relevant stakeholders and agree on a time to come together for the workshop.

As previously mentioned, the ToC needs to be owned and used by stakeholders, so it is crucial that they believe in it and have a clear understanding of how it works. I have held ToC workshops both in-person (my preferred method) and online.

Some useful materials to gather prior to the workshop include:

• If holding the workshop in-person: bring some big pieces of paper or have a whiteboard handy. Make sure you have lots of marker pens and different coloured sticky notes so that everyone can take part in workshopping the ToC 

  *Top tip: We suggest using different coloured sticky notes for each step in the ToC diagram (e.g., blue for activities, yellow for outputs, etc.) 

• If holding the workshop online: there are lots of types of online collaborative software you can use to workshop a ToC. My personal favourite is Miro which allows everyone to participate in adding to the ToC, moving things around, and editing the diagram 

Your main job in the ToC workshop is to act as a facilitator to help the key stakeholders develop the ToC. In this article, we’ve provided a list of steps and some key questions that you can use to facilitate the ToC workshop.

Note that this process is for developing a ToC for evaluating an already established program. For planning and designing a new program, the questions can be reframed slightly to change the tense from “what we’ve done” to “what we’ll do.” 

---

Key definitions to display for all key stakeholders throughout the ToC workshop

One last point before we get started; it is essential that you clearly define what a ToC is and is not for all stakeholders before you start to make sure they understand the intent of the workshop. This will make sure that everyone is on the same page and that there is a clear understanding of what the final product will be. 

*Top tip: Make sure you have a note-taker on hand in the workshop to capture all the key details that aren’t listed on the sticky notes 

Activities 

• The actions that will be taken by the program that are expected to contribute to the change. For example, this could include interviews with patients, training of clinicians, etc. 

Outputs 

• The products or the services that the program will create through the activities. For example, certain pieces of new knowledge (e.g., patients’ opinions) or new processes (e.g., a new clinic workflow).  

Outcomes 

• The changes in knowledge (e.g., clinicians have a greater understanding of a condition), attitudes (e.g., patients change their perspectives about exercise), or skills (e.g., MOA’s learn how to use TNA) of the key actors that are expected to lead to a change in their behaviour (e.g., clinicians use their knowledge to support patients, patients exercise 3 times a week, MOA’s use TNA for tracking appointments) as a result of the program. Outcomes are sometimes broken down into intermediate, end of program, and high-level depending on when they are expected to be realized in the program life cycle.  

Impacts 

• The top-level changes that result wholly or in part from the outcomes (e.g., the change process) to which the program contributed to. 

Purpose 

• The overall goal/aim the program aims to contribute to; it explains the program’s reason for existence. 

Impact Pathways 

• The pathways through which the program is expected to contribute to change. These are often actor specific to make them more tangible (e.g., Clinician Capacity Building Pathway, Clinic Workflow Pathway). 

Causal Logic 

• Describes the how and why between each step in a ToC. In the ToC diagram it is often presented using arrows.  

Indicators 

• A marker of accomplishment that can be used to measure the success of a program (e.g., did the program contribute to a certain outcome?).  

Assumptions 

• The factors outside of the program’s control that are necessary to ensure the program’s success. 

---

Creating an Agenda for your ToC Workshop

The way I workshop and develop a ToC is not linear. Here is an agenda of what to expect: 

1. Purpose: What was the long-term goal of the program? 

2. Target audiences: Who were the main target audiences of the program? 

3. Activities: What were the program’s main activities and how were they carried out? 

4. Outputs: What outputs did these activities result in? 

5. Outcomes: What were the main outcomes of the program? 

6. Impacts: What impacts did these outcomes lead to and how are they connected to the purpose? 

7. Impact Pathways: What types of impact pathways were implemented? 

8. Indicators: How do we measure the outcomes? 

9. Assumptions: What are the underlying assumptions for each outcome? 

I have also numbered each stage of the workshop process in the following ToC diagram. The steps that should be captured outside of the diagram (e.g., in a ToC tracking table, see part 1; “What is Theory of Change in Evaluation?”) and will be useful for evaluating the program and testing the diagram have also been included below for your reference. 

*Top tip: take a photo of your whiteboard, sticky wall, or online workshop board after the workshop and keep it safe! You’ll need it to create a more consolidated version of the ToC diagram

---

Consolidating the Theory of Change (ToC)

The first ToC to come out of the workshop will look very messy, but don’t panic! Your notes will help you to consolidate and make sense of the ToC.

There is no right way to consolidate the ToC and it will take practice, but a useful technique is to think about the different target audiences identified and make sure each step in the change process (e.g., outcomes) is well established; if not, check to see if outcomes can be merged or if any logical steps are missing.

Drafting a supporting narrative is also something that takes practice, but it’s key to focus on identifying the causal logic. You want to clearly explain the theory behind the diagram so that everyone can understand it, even those outside of the program; focus on explaining who is doing what differently and why because of the program and its activities? 

*Top tip: Probe stakeholders during the workshop to identify and determine the causal logic between each step and outcome, and make sure the note-taker writes this down in detail! 

---

Validating the Theory of Change (ToC)

Once you have a consolidated ToC and draft narrative, the most important thing is to validate it with key stakeholders. Share both the model and narrative with them, walk them through it, and make sure it makes sense to everyone.

Have a list of questions ready to ask them if anything doesn’t quite make sense to you. Don’t forget to make sure that your ToC is credible, achievable, supported, and testable! 

---

Have you worked with ToC before, do you have a different way of developing a diagram, or have questions? Comment on this article or connect with us on LinkedIn or Twitter! 

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address


Sign Up

We respect your privacy.

Thank you!

---