cplysy

Kirkpatrick is probably one of those names/methods you’ve heard about in your evaluation career, but have you ever used it? I’m surprised how many evaluators I talk to that haven’t because I find it pretty useful and straightforward with tonnes of resources to support you. 

I love experiential descriptions, that is, reading about how someone else applied a method in a real-world scenario: the ups and downs, the backtracking and lessons learned.  

So, having used Kirkpatrick a handful of times on a few initiatives, here is my account of how to use the Kirkpatrick model in your evaluation planning, implementation, and reporting.  

---

What is the Kirkpatrick Model?

The Kirkpatrick model was originally developed in the 1950s but gained popularity in the 1970s as a way to evaluate training programs. Donald Kirkpatrick proposed 4 levels:

Kirkpatrick can apply to evaluating any type of educational endeavour – where participants or attendees are intended to learn something and implement those learnings.

---

How do I use the Kirkpatrick Model in Evaluation?

Step 1: Do some basic research. 

The model has spawned a website of Kirkpatrick Partners. They host training and events; they have a newsletter, blog and resources to support your use of Kirkpatrick. You can even get certified in using Kirkpatrick. (Disclaimer: I am not certified.) My goal for the rest of this article will be to show you how I’ve actually used Kirkpatrick and some of my thoughts along the way. 

Step 2: Incorporate Kirkpatrick in your evaluation plan. 

Unless you are exclusively evaluating a training program, I’ve found that Kirkpatrick is often a part of the evaluation plan, but not the only part. 

I’ve used Kirkpatrick on an initiative that trained people to facilitate quality improvement in primary care practices. I used Kirkpatrick for the training but had lots of other evaluation questions and data sources for the quality improvement efforts and outcomes in the primary care clinics. 

Like RE-AIM, the good news is that Kirkpatrick gives you a solid head start on your training-related evaluation questions. 

Your overarching evaluation question might be something like: 

To what extent did training prepare people to [make the intended change]? 

Or 

How effective was the training at improving [desired behaviour]? 

From there, use the four levels of the model to ask specific questions or craft outcome statements (See below for a detailed explanation of each level):

• Level 1: Reaction

• Level 2: Learning

• Level 3: Behavior

• Level 4: Results

Step 3: Reporting  

I don’t think I’ve ever used the Kirkpatrick levels explicitly in my reporting. I think most audiences are not interested in the theory of evaluating a training program but are more interested in answering “did the training work?” As I mentioned, Kirkpatrick has usually been only a part of my evaluation planning, so, similarly, reporting on the effectiveness of training is usually only one part of my reporting.  

Because Level 1 of Kirkpatrick assesses formative questions about training – things that you could change or adapt before running the training session again, I have often produced formative reports or briefs that summarize just Levels 1 and 2 of Kirkpatrick. This promotes utilization of evaluation results! 

Using surveys (or a test!) likely gives you some quantitative data for you to employ your data viz skills on. But keep in mind that it’s not necessary for you to report all of the data you gather. You likely don’t need a graph showing your reader that the participants thought the room was the right temperature. Sometimes less is more and a couple of statements like “Participants found the training environment to be conducive to learning and found the training to be engaging. However, the days were long, and they recommended more breaks.” can cover a lot of your Reaction results. We’ve got lots of resources to help you with your report design. 

Sometimes reports, particularly interim reports, are due before you can get to behaviour change or results. This is actually one of the criticisms of Kirkpatrick – that many evaluations will cover Levels 1 and 2 thinking them sufficient but fail to invest the time and resources into ensuring the behaviour changes and outcomes are captured. This is one of the reasons that planning an evaluation concurrently with project design is helpful and can prevent these shortcuts. 

---

How to use the 4 levels of the Kirkpatrick Model

LEVEL 1: REACTION 

Reaction is about all those things you think immediately after you’ve attended a session. These are less about what you learned, and more about: Was the trainer effective? Was the environment supportive to learning? Was the day interactive and fun, or didactic and tiring? 

Reaction is all of those things that influence learning. They may seem of less importance but contribute a lot to how much a person learns, retains and acts on. 

Let’s use a scenario where you are evaluating a training program designed to teach participants how to implement COVID-19 safety protocols in the workplace. 

For Reaction, the evaluation is almost entirely content agnostic, so it matters less what the training program is about, and more about the delivery, for example: 

The venue was appropriate. 

The presenter was engaging. 

The training was relevant to my work. 

This is almost always captured in a post-training survey, which of course could be paper for in-person events, or QR codes/links for virtual events. It could be emailed out to participants after the session, but we all know that response rates are much better when you carve out 5 minutes at the end of the last session to complete the evaluation.  

The learning here is about how you can tweak the delivery of the training. Was the room too cold? Was the presenter about as engaging as Ferris Bueller’s homeroom teacher? 

LEVEL 2 – LEARNING 

As the name says, level 2 is about assessing what the participant learned. I like to think: Knowledge, Skills, and Attitudes. Assessing learning can (and arguably should) touch on each of these. So, your questions might be: 

I can explain our COVID-19 safety policies. [knowledge] 

I understand why our COVID-19 safety policies are important. [knowledge] 

I am confident that I can enforce our COVID-19 safety policies. [attitude] 

I learned 3 ways to build buy-in about our COVID-19 safety policies. [skills] 

As these examples imply, it’s very common for these questions to be included on a post-workshop survey. I usually embed them with the Reaction survey, so participants fill out one survey after training. I try to keep the total question count under 20, usually Likert Scale, with some opportunity for qualitative feedback. 

TIP: If it’s important to you to be able to say that the training was the reason for the results you get, you’ll want to consider a baseline survey – that is, a survey with the same learning questions as the post-training survey, but it’s completed before training. That way, any change that you see can be more strongly linked to the training that they received, as opposed to pre-existing knowledge, skills, or attitudes. 

If your training has learning objectives, this is a good place to look for what knowledge, skills or attitudes the program is intended to impact. 

Personally, I’ve always lacked confidence in results that come from self-assessed ratings of knowledge, skills, and attitude. We know there are several biases in play – including a tendency to not use the full range of a scale, and to rate yourself positively. One way to mitigate these confounders is to test the learning. Instead of asking for an opinion about what they learned, the post-training survey could be formatted to actually test the learning: 

List three COVID-19 safety protocols implemented at your workplace. 

Which of the following is a reason why these COVID-19 protocols were selected. [multiple choice] 

Describe one way to build buy-in with your staff around COVID-19 policies. 

The downside is that not only are these potentially more resource-intense to analyze and report on, but most programs and organizations are worried about the impression it gives to test participants or workshop attendees. We all hate tests, right!? I haven’t had much luck convincing a program to use testing as opposed to a self-rated survey. Let me know if you’ve fared any better! 

LEVEL 3 – BEHAVIOUR 

Here is where it gets a bit tricky. To measure behaviour you need to have access to the participants after a set amount of time; your participants need an opportunity to put their learning into action.  

Assuming you have email addresses or perhaps a reason to bring this group together again, I’ve assessed behaviour change through surveys (yes, surveys again). These surveys are usually sent over email after a pre-determined amount of time. The time interval depends on the learning you are hoping they achieved and the opportunities to implement it. I’ve used anywhere from 4 weeks to 6 months. 

In the last [time interval] I have: 

Explained why COVID-19 safety protocols are important. 

Used techniques learned at training to deal with an individual not wishing to follow COVID-19 safety protocols. 

It’s possible that you could re-administer the baseline/post-training survey as a [time interval] follow-up. This would help you to assess retention of learning and may get at some behaviour change too. The difference though is that the assessment of learning was likely opinion-based or in-the-moment, while assessment of behaviour change is retrospective – it’s not about what they have knowledge, attitude or skills to do, or what they intend to do, it’s about what they did do. 

Another option is to change your data collection strategy: 

• Observation: can you watch attendees implementing the training? 

• Interviews: can you gather data from people who didn’t attend the training to understand what changes they have seen? Or interview the training attendees to understand what they’ve put into practice (and in what context) 

• Role playing: If you don’t anticipate being able to reach the attendees after a time period, perhaps role playing (and observation) could be a part of the training curriculum. Can attendees demonstrate what they have learned? 

• Evidence of action: perhaps you can access evidence that proves the training resulted in action – maybe participants were asked to write business plans (how many were written?) or were asked to design and implement a communication strategy about COVID-19 policies (was it done?) 

In our scenario, observation may look like observing building entrance processes and observing, counting or noting the number of times a COVID-19 policy is explained or enforced. Or, you could survey staff to see how well they understand the COVID-19 policies (assuming their knowledge is a result of the training to select staff). The key to Level 3 is the demonstration of the behaviour. 

LEVEL 4: RESULTS 

Results brings everything full circle. It drives at the purpose of the training to begin with – what is the impact of the training. In our scenario, the ultimate goal may have been to have staff and patrons compliant with COVID-19 safety protocols. Assessment of Results will be a reflection of this. 

# of encounters with non-compliant staff 

# of encounters with non-compliant patrons 

# of COVID-19 outbreaks at the workplace 

Assessment of Results may start to blur the lines with the rest of your evaluation plan. Perhaps the training is part of a larger program that is designed to create a safe environment for staff and patrons, part of which was implementing COVID-19 policies.  

Assessment of Results likely requires an even longer time frame than Level 3, Behaviour.  

---

Drawbacks of the Kirkpatrick Model

I’ve found Kirkpatrick to be useful to ensure that evaluation of a training program goes beyond simply measuring reactions, however, the temptation to cut the method short and only measure reaction and learning is common practice. 

In perhaps the simplest approach, Kirkpatrick is a survey-heavy method – which relies on adequate response rates and is littered with biases. More rigorous methods to measure the four levels – observation, interviews – are likely more time consuming and resource intensive. It’s a tradeoff. If you go the survey route, here are some tips on how to use Likert scales effectively. 

Another common criticism of Kirkpatrick is the assumption of causality. The model takes the stance that good, effective training is a positive experience (Level 1), results in new learning (Level 2), and drives behaviour change (Level 3), which leads to achievement of your desired outcomes (Level 4). It fails to account for the environmental, organizational, and personal contexts that play a role. Whether or not an organization supports the behaviour change or empowers attendees to make change matters, regardless of how fantastic the training was. The further you move through the four levels of Kirkpatrick, the looser the link to causality. 

---

Kirkpatrick Model Return on Investment 

At some point, an unofficial 5th Level was added to Kirkpatrick – the Phillips Return on Investment (ROI) model (and sometimes this is not a new level but tacked onto Level 4 – Results). The idea here is that the cost of running training should be absorbed by the positive financial impact on the organizational improvements that come from the training and subsequent behaviour change.  

I’ve never actually used the ROI, but there are lots of resources out there to help you. 

---

Other Uses of the Kirkpatrick Model

Here at Eval Academy we often talk about the importance of planning your evaluation right along with program design. Kirkpatrick is no exception. The four levels can offer guidance and perspective on how to actually design a training program from the start. By working backwards, you can ask questions that ensure your training program has the right curriculum and approach to achieve your goals: 

1. What are we trying to achieve? (Level 4 – Results) 

2. What behaviours need to happen to realize that goal? (Level 3 – Behaviour) 

3. What do attendees need to know or learn to implement those behaviours (Level 2 – Learning) 

4. How can we package all of this into a high-quality, engaging workshop? (Level 1 – Reaction). 

I’ve only ever used Kirkpatrick as a planned-out evaluation approach. It may be trickier to use this model for training that has already happened: you’ll have missed the opportunity for baseline data collection, and importantly, depending on the time that has passed the Reaction results may be less reliable. 

---

I like Kirkpatrick because it’s simple and straightforward. This simplicity is actually one of the main criticisms about it, and its failure to recognize context. However, in my experience, the guidance offered by the four levels helps to shape my thinking about how to evaluate a training program. Because a training program is often only part of a larger initiative, perhaps the lack of context hasn’t been as blatant for me. 

I’d love to know your experiences with Kirkpatrick. Have you evaluated all four levels? Have you assessed ROI? 

For other real-life accounts of how to use an evaluation methodology, check out our articles on RE-AIM, Developmental Evaluation and Outcome Harvesting. 

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address


Sign Up

We respect your privacy.

Thank you!

Sources: 

Graham, P., Evitts, T. & Thomas-MacLean, R. (2008) “Environmental Scans: How useful are they for primary care research?”. Can Fam Physician. 54(7): 1022-1023. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2464800/ 

Polanin, J.R., Pigott, T.D., Espelage, D.L. & Grotpeter, J.K. (2019) “Best practice guidelines for abstract screening large-evidence systematic reviews and meta-analyses”. Res Synth Methods. 10(3): 330-342. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6771536/ 

---

Whether starting a new program or making changes to an existing one, you’re going to be faced with numerous questions about how best to move forward. And some of those questions can feel pretty daunting! Many organizations use environmental scanning as a method of strategic planning to gain insights and gather information on how to put their best foot forward. 

This article is aimed at those who are new to environmental scanning and are looking for new ways to support program planning and improvement. 

---

What is an environmental scan?

Environmental scans started out as a tool for businesses to find and organize information that could be used for decision-making.

The process of environmental scanning includes finding, gathering, interpreting, and using information from the internal and external environments of an organization to help direct future action.

This method uses multiple strategies for information collection such as focus groups, in-depth interviews, surveys, literature reviews, and reviewing personal communications, policy analyses, and internal documents.

The results of an environmental scan can be very useful in helping an organization or program shape its goals and strategies.

---

What’s the difference between an environmental scan and a literature review?

Unlike a literature review that searches for published, peer-reviewed articles, an environmental scan also examines unpublished literature and publicly available information.

An environmental scan also incorporates methods such as interviews and focus groups not used within literature reviews.

---

Why should I complete an environmental scan?

Scanning the environment is an important part of strategic planning and has been linked to improved organizational performance.

Environmental scans can help plan for the future, provide evidence about potential directions for an organization or program, raise awareness of risks or issues, or help to initiate a new program.

The diversity of information sources and types of information gathered through environmental scans have helped organizations to effectively plan and implement programs across a variety of sectors.

---

When should I do an environmental scan?

You can take a reactive approach (e.g., a challenge has arisen that needs to be addressed) or a proactive approach (e.g., a new program is being implemented and you want to ensure its success) to scanning.

Scanning is typically more frequent when an organization or program has higher levels of perceived uncertainty, such as when taking a new direction or during its start-up.

---

How do I complete an environmental scan?

To complete an environmental scan, we typically follow six main steps: 

1. Identify the purpose of the environmental scan and your topics of interest 

2. Identify the research question(s) 

3. Identify the activities you will complete and where you will look for information 

4. Create a list of keywords and search terms 

5. Catalogue the information systematically  

6. Present the information in a way that is useful for your organization 

 

Step 1: Identify the topics of interest and the purpose of the environmental scan 

Before jumping onto the internet to search for information, it is first important to specify the purpose for the environmental scan and identify your topics of interest. This will help to anchor the process, focus your time and resources, and avoid those rabbit holes! Although an environmental scan should remain flexible to allow for new questions that might arise, having a clear scope will help you stay focused. 

For example, let’s say we’re looking to create a new program in the Vancouver region that aims to provide mental health support to isolated and elderly individuals, but we’re not sure what similar programs are already out there. To avoid duplicating services and making sure we reach a portion of the target population that isn’t already being served, the purpose of our environmental scan would be to: 

“Support the development of a program that aims to improve the mental health of the isolated and elderly by learning about available programs and identifying their strengths and weaknesses.” 

Our topics of interest would then be: 

• Examples of programs that are currently available in the Vancouver region which aim to improve the mental health of people aged 75+ who are isolated (e.g., live alone) 

• Identifying what the programs currently do well including effective modalities, treatment options, and who they are reaching 

• Identifying gaps in the programs such as treatment options that are not provided and target populations that are missed 

Step 2: Identify the research question(s) 

Now that you’re clear on your environmental scan purpose, you can narrow this even further by identifying your research questions. This is the broad rule for knowing when to stop your search and will help you to decide whether an article or website is worth continuing to explore.  I tend to recommend having between 1 and 3 research questions for an environmental scan that dig a little deeper into your topics of interest to really pull out all the juicy details you’re looking for! 

The research questions for our example environmental scan could be: 

1. What programs are currently implemented in the Vancouver region for the isolated and elderly (75+ years of age) which focus on improving their mental health? 

2. How do these programs operate and what makes them effective? 

   • Who is the audience that they’re reaching and how do they reach them? 

   • What modalities do they use (e.g., group settings, individual therapy)? 

   • What treatment options do they implement? 

3. Are there any gaps in these programs? 

   • Are there any of the target population that are not effectively captured by the program? 

Step 3: Identify what environmental scan activities you will complete and where you will look for the information  

Once we have our topics of interest, purpose, and research question, next it’s time to identify what activities you’ll complete and where you’ll look for the information to answer your questions. In an environmental scan, your activities can focus on understanding the internal (your organization’s) or external (other organization’s) environment to the particular topic. This will help to provide input into strategic thinking, decision making, and planning. 

For understanding the internal environment, appropriate activities could include: 

• Reviewing organizational documents (e.g., organizational strategy, policies, or internal communications) 

• Interviewing members of staff  

For understanding the external environment, appropriate activities could include: 

• Grey literature searches on the internet (e.g., Google search) 

• Interviews or surveys with organizations or individuals of interest identified through your online search 

Don’t forget that you’ll also need to present your findings in an appropriate format for your organization whether that’s a report, infographic, or a presentation! (See Step 6 for further info) 

Although looking online for information is important in an environmental scan, involving stakeholders can be the key to its success. Frequently grey literature searches like reviewing websites can leave you falling short on all the details needed to answer your research questions. Reaching out to key individuals at the organizations of interest with an invitation to take part in an interview or survey can help you to fill these gaps. For example, many Indigenous organizations will have reports and summaries or have participated in other scans and engagements, so having the opportunity for Indigenous organizations to recommend grey literature is important to help fill gaps in knowledge and understanding. 

Before reaching out to stakeholders, make sure you have a clear understanding of what information is needed from them. This can help with engagement fatigue and make sure you’re asking the right questions. You should only ask them for information that you are not able to uncover on your own. Create a clear plan for conversations with participants, such as having a clear set of questions or requests for further information. Be prepared to answer further questions about the environmental scan process and why you’re interested in learning more about their work and organization. Lastly, be aware of who you are interviewing and methods for appropriately and respectfully engaging them. For our example, if you are interviewing Indigenous Elders then be aware of the protocol with visiting Elders and what you need to bring to respect the knowledge and teachings you will receive from Elders. 

The type and volume of activities you’ll complete are often affected by the amount of time that you have to finish the environmental scan. If a timeline isn’t established for you by a funding agency or by your leadership, it is important that you create your own timeline from the outset to help you plan and stay on task. If interviews of surveys are part of your environmental scan, make sure you allocate yourself plenty of time for creating the survey tool and interview guides, reaching out to your participants, collecting and analyzing the data, and interpreting your results.  

The activities for our example environmental scan could be: 

• Environmental scan of grey literature looking for examples of similar programs in Vancouver. Best practice examples from other geographies may also be included if “gold standards” are discussed 

• Contacting organizations who have implemented similar programs for follow-up questions through telephone interviews 

• In-person interviews with members of the target audience to learn how future programs could be improved for them 

• Presenting a selection of similar programs and treatment options through a final report 

Step 4: For online searches, you’ll need to create a list of key words and search terms that you will use 

To make sure you’re looking for information in the right places while searching online, it’s important that you create a list of key words and search terms to help guide your search. 

To start this process, make a note of all the key terms that will help you to capture your topics of interest. Cast your net wide to make sure you don’t miss anything. Then break down these topics into clear concepts and keywords. 

For our example environmental scan, a list of key terms could be:  

• Mental health programs 

• Isolated 

• Elderly 

• Vancouver 

• Effective modalities 

• Treatment options 

Using key terms such as “Indigenous”, “Female”, “LGBTQ2S+”, “Newcomer”, “Black” amongst others will help ensure an intersectional lens to your search. 

The next step is to identify any of these concepts that could be expressed using related terms or synonyms. You can make a list of these under each of the key terms. 

Boolean searching through creating search strings is a common way to look for information in online databases and can also help to frame internet searches. Boolean searching uses connector works such as AND, OR, and NOT to create phrases based on rules and search logic.  

We’ll dive deeper into how you can systematically search online databases in our upcoming article on literature reviews! 

Step 5: Catalogue the information systematically 

Once you’ve found your sources, it is important to catalogue the information in a systematic way that links back to your research questions. Using a table like the one below that breaks the sources down by your topics of interest can make it very clear and help you when having to present your findings. 

Once you’ve catalogued your information systematically, you can then reach out to organizations and key stakeholders to request their participation in an interview or a survey to help you fill in the gaps.

Step 6: Present the information in a way that is useful for your organization 

Once you’ve organized, analyzed, and interpreted your information, it is crucial that you present your findings in a way that is useful for your organization. If an appropriate means of presenting the findings is not given to you by leadership, consider your audience and how the information will be used. This will help you to decide the best way of presenting your results. Reflect again on your topics of interest and make sure you present your information in a way that answers the research questions whether that’s in a summary report, infographic, or presentation to your organization. Lastly, make sure your work is disseminated and shared with all relevant stakeholders! This includes disseminating your work to anyone who participated in the scan, including any Indigenous stakeholders to align with OCAP principles. 

---

What’s your experiences with Environmental Scans? Comment on this article or connect with us on LinkedIn or Twitter! 

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address


Sign Up

We respect your privacy.

Thank you!

Sources: 

Graham, P., Evitts, T. & Thomas-MacLean, R. (2008) “Environmental Scans: How useful are they for primary care research?”. Can Fam Physician. 54(7): 1022-1023. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2464800/ 

Polanin, J.R., Pigott, T.D., Espelage, D.L. & Grotpeter, J.K. (2019) “Best practice guidelines for abstract screening large-evidence systematic reviews and meta-analyses”. Res Synth Methods. 10(3): 330-342. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6771536/ 

---

In this follow-up article, we will walk you through a step-by-step process of how you can develop your own Theory of Change (ToC) diagram for evaluation purposes.

You can also find our Theory of Change template here which you can use to support the development process. However, there is no one right way of developing a ToC.

Some people prefer working backwards from the program purpose, and it often depends on the program and the information that is already available to what method would work best. The method described below is one that I have found particularly useful from past experience.

---

When developing a ToC, it is important to keep the following points front of mind: 

• A ToC should be credible. It should be based on the insight and experience from different key stakeholders  

• A ToC should be achievable. You need to have the necessary resources to complete the desired change processes. Setting out to solve world hunger with a small budget isn’t the way to go! 

• A ToC should be supported. All key stakeholders should have some level of involvement in defining a ToC. Key stakeholders also need to agree with the ToC to ensure buy-in and understanding across all levels 

• A ToC should be testable. The diagram and accompanying narrative need to be a complete picture of the program. However, they also need to be clear, concise, and not over-complicate the description of the program’s work. Making sure you have indicators of achievement for each outcome will help you to collect the necessary data to test your ToC 

---

How to Host a Theory of Change (ToC) Workshop

Developing a ToC should be a highly collaborative process. The best method I have found to develop a ToC is by holding a 2-hour workshop (which can be broken up into two parts if needed) with key stakeholders. The first step in your process should be to identify and gather all of the relevant stakeholders and agree on a time to come together for the workshop.

As previously mentioned, the ToC needs to be owned and used by stakeholders, so it is crucial that they believe in it and have a clear understanding of how it works. I have held ToC workshops both in-person (my preferred method) and online.

Some useful materials to gather prior to the workshop include:

• If holding the workshop in-person: bring some big pieces of paper or have a whiteboard handy. Make sure you have lots of marker pens and different coloured sticky notes so that everyone can take part in workshopping the ToC 

  *Top tip: We suggest using different coloured sticky notes for each step in the ToC diagram (e.g., blue for activities, yellow for outputs, etc.) 

• If holding the workshop online: there are lots of types of online collaborative software you can use to workshop a ToC. My personal favourite is Miro which allows everyone to participate in adding to the ToC, moving things around, and editing the diagram 

Your main job in the ToC workshop is to act as a facilitator to help the key stakeholders develop the ToC. In this article, we’ve provided a list of steps and some key questions that you can use to facilitate the ToC workshop.

Note that this process is for developing a ToC for evaluating an already established program. For planning and designing a new program, the questions can be reframed slightly to change the tense from “what we’ve done” to “what we’ll do.” 

---

Key definitions to display for all key stakeholders throughout the ToC workshop

One last point before we get started; it is essential that you clearly define what a ToC is and is not for all stakeholders before you start to make sure they understand the intent of the workshop. This will make sure that everyone is on the same page and that there is a clear understanding of what the final product will be. 

*Top tip: Make sure you have a note-taker on hand in the workshop to capture all the key details that aren’t listed on the sticky notes 

Activities 

• The actions that will be taken by the program that are expected to contribute to the change. For example, this could include interviews with patients, training of clinicians, etc. 

Outputs 

• The products or the services that the program will create through the activities. For example, certain pieces of new knowledge (e.g., patients’ opinions) or new processes (e.g., a new clinic workflow).  

Outcomes 

• The changes in knowledge (e.g., clinicians have a greater understanding of a condition), attitudes (e.g., patients change their perspectives about exercise), or skills (e.g., MOA’s learn how to use TNA) of the key actors that are expected to lead to a change in their behaviour (e.g., clinicians use their knowledge to support patients, patients exercise 3 times a week, MOA’s use TNA for tracking appointments) as a result of the program. Outcomes are sometimes broken down into intermediate, end of program, and high-level depending on when they are expected to be realized in the program life cycle.  

Impacts 

• The top-level changes that result wholly or in part from the outcomes (e.g., the change process) to which the program contributed to. 

Purpose 

• The overall goal/aim the program aims to contribute to; it explains the program’s reason for existence. 

Impact Pathways 

• The pathways through which the program is expected to contribute to change. These are often actor specific to make them more tangible (e.g., Clinician Capacity Building Pathway, Clinic Workflow Pathway). 

Causal Logic 

• Describes the how and why between each step in a ToC. In the ToC diagram it is often presented using arrows.  

Indicators 

• A marker of accomplishment that can be used to measure the success of a program (e.g., did the program contribute to a certain outcome?).  

Assumptions 

• The factors outside of the program’s control that are necessary to ensure the program’s success. 

---

Creating an Agenda for your ToC Workshop

The way I workshop and develop a ToC is not linear. Here is an agenda of what to expect: 

1. Purpose: What was the long-term goal of the program? 

2. Target audiences: Who were the main target audiences of the program? 

3. Activities: What were the program’s main activities and how were they carried out? 

4. Outputs: What outputs did these activities result in? 

5. Outcomes: What were the main outcomes of the program? 

6. Impacts: What impacts did these outcomes lead to and how are they connected to the purpose? 

7. Impact Pathways: What types of impact pathways were implemented? 

8. Indicators: How do we measure the outcomes? 

9. Assumptions: What are the underlying assumptions for each outcome? 

I have also numbered each stage of the workshop process in the following ToC diagram. The steps that should be captured outside of the diagram (e.g., in a ToC tracking table, see part 1; “What is Theory of Change in Evaluation?”) and will be useful for evaluating the program and testing the diagram have also been included below for your reference. 

*Top tip: take a photo of your whiteboard, sticky wall, or online workshop board after the workshop and keep it safe! You’ll need it to create a more consolidated version of the ToC diagram

---

Consolidating the Theory of Change (ToC)

The first ToC to come out of the workshop will look very messy, but don’t panic! Your notes will help you to consolidate and make sense of the ToC.

There is no right way to consolidate the ToC and it will take practice, but a useful technique is to think about the different target audiences identified and make sure each step in the change process (e.g., outcomes) is well established; if not, check to see if outcomes can be merged or if any logical steps are missing.

Drafting a supporting narrative is also something that takes practice, but it’s key to focus on identifying the causal logic. You want to clearly explain the theory behind the diagram so that everyone can understand it, even those outside of the program; focus on explaining who is doing what differently and why because of the program and its activities? 

*Top tip: Probe stakeholders during the workshop to identify and determine the causal logic between each step and outcome, and make sure the note-taker writes this down in detail! 

---

Validating the Theory of Change (ToC)

Once you have a consolidated ToC and draft narrative, the most important thing is to validate it with key stakeholders. Share both the model and narrative with them, walk them through it, and make sure it makes sense to everyone.

Have a list of questions ready to ask them if anything doesn’t quite make sense to you. Don’t forget to make sure that your ToC is credible, achievable, supported, and testable! 

---

Have you worked with ToC before, do you have a different way of developing a diagram, or have questions? Comment on this article or connect with us on LinkedIn or Twitter! 

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address


Sign Up

We respect your privacy.

Thank you!

---

 

We get it, Theory of Change (ToC) doesn’t really sound like the most fun, interesting, or appealing evaluation tool. I certainly wasn’t buzzing with excitement when I first heard the phrase ToC! 

If this also sounds like you, then you’ve come to the right place. This article aims to introduce you to ToC and is aimed at those who may have heard of the tool before but aren’t quite sure as to what it is and how it can be useful for evaluation. After putting my preconceptions of the tool to one side and learning more about its uses, I have found that ToC has been one of the most valuable tools to me so far as an evaluator. 

This article is the first part of our discussion on ToC and answers the question of “what is theory of change in evaluation?”

The second part of our discussion will focus on how to develop and use ToC for evaluation. 

---

So, what is Theory of Change (ToC)? 

Theory of Change (ToC) explains how a set of activities will solve a problem. It does this by clearly describing the problem and the change you (and the program) intend to create.

Most importantly, ToC depicts change that is not linear. Processes of change actually have multiple feedback loops that need to be understood and described.

In short, ToC is a diagram to describe “if you do this, then these are the intended results.” When we break it down a little more we get … 

Although ToC has similar components to other evaluation tools such as logic models that mainly focus on the what, ToC goes into more detail about the why.

It does this by identifying processes called “causal logic” or “causal mechanisms” which are represented in the diagram using arrows. We will go into further detail about causal logic below.

While logic models also highlight contextual factors and assumptions, they are often focused at a high-level and don’t look at each specific step within the change process (e.g., between each outcome box). This is something that is key to a ToC. 

ToC allows us to map each step towards a long-term goal which provides an explicit and testable diagram of how and why a change is expected to happen in a particular context.

It is not just a list of activities with arrows linking them to their intended outcomes like a logic model, but it explains how these changes will happen and how the program will contribute to this change at each little step in the change process.

You guessed it, ToC creates its own program theory. It also looks at what other external factors outside of the program will affect the desired change. 

ToC can be developed for any level of an intervention – an event, a project, a program, a policy, a strategy, or an organization.

This makes it both useful for program planning and in monitoring and evaluation where it is often captured through a one-page diagram that contains multiple boxes and arrows. This diagram often has a supporting written narrative that explains each step of the theory in more detail to walk the reader through the diagram.

A ToC is a living document; it is not set in stone and should be revisited and revised as the program progresses to make the most of opportunities and address challenges that arise. Depending on the lifespan of the program, a 6-monthly or yearly review of the ToC diagram and narrative might be appropriate. 

---

What are the main components of a Theory of Change (ToC)? 

There are several main components of a ToC. Some of these components are included in the ToC diagram itself, while others should be captured in the supporting narrative or an alternate table. This has been explained in further detail below. 

• Activities are the actions that will be taken by the program that are expected to contribute to the change. For example, this could include interviews with patients, training of clinicians, etc. 

• Outputs are the products or the services that the program will create through the activities. For example, certain pieces of new knowledge (e.g., patients’ opinions) or new processes (e.g., a new clinic workflow). 

• Outcomes are the changes in knowledge (e.g., clinicians have greater understanding of a condition), attitudes (e.g., patients change their perspectives about exercise), or skills (e.g., clinicians learn how to use new software) of the key actors that are expected to lead to a change in their behaviour (e.g., clinicians use their knowledge to support patients, patients exercise 3 times a week, MOA’s use TNA for tracking appointments). Outcomes are sometimes broken down into intermediate, end of program, and high-level depending on when they are expected to be realized in the program life cycle. However, there is no right number of outcomes which entirely depends on the program itself and the change process they are describing.  

• Impacts are the top-level changes that result wholly or in part from the outcomes (e.g., the change process) to which the program contributed to. 

• Purpose is the overall goal/aim the program aims to contribute to; it explains the program’s reason for existence. Unlike a Vision Statement which is more high-level and focuses on the what, the program’s purpose focuses on the why. For example, why is your program on the journey it’s on? 

• Impact Pathways are the pathways through which the program is expected to contribute to change. These are often actor specific to make them more tangible (e.g., Clinician Capacity Building Pathway, Clinic Workflow Pathway) 

• Causal Logic describes the how and why between each step in a ToC. In the ToC diagram it is often presented using arrows. More detail about the causal logic is captured within the supporting narrative. 

Main components that need to be captured outside of the TOC diagram: 

• Key Actors are the people who are crucial to the program. Those developing a ToC should be aware of the key actors’ roles within the context of the program, and how the program will work with these key actors to create change. Key actors can be identified using the framing of “who is doing what differently and why as a result of the program?” and captured using a Stakeholder Matrix (our template can be found here).  

• Causal Logic describes the how and why between each step in a ToC. Detail about the causal logic which links each step in the ToC diagram is captured using the descriptive narrative that accompanies the diagram. The narrative walks the reader step-by-step through the diagram. 

• Indicators are a marker of accomplishment that can be used to measure the success of a program (e.g., did the program contribute to a certain outcome?). These play a key role in the evaluation plan and can be captured and listed using the suggested ToC tracking table below.  

• Assumptions are the factors outside of the program’s control that are necessary to ensure the program’s success. This is one of the most valuable parts of the ToC process as stakeholders get to hear each other’s understanding of the goals, challenges, and what is needed in the program’s context for it to succeed. Like indicators, it is very important to list and understand these key assumptions as they relate to each individual outcome. 

---

Why should I use a Theory of Change (ToC)? 

ToC can help you to understand why change is expected to occur by altering the way you think about your program from what you are doing to what you want to achieve. The tool can also help programs respond to changes and guide stakeholder engagement, communication, and support decision-making.

By defining long-term goals and then mapping backwards to identify the necessary preconditions, ToC can provide the basis for arguing that a program is making a difference whilst identifying weaknesses in the argument and providing the opportunity to make changes.

It is well suited to complex programs that are influenced by multiple systems and actors due to its fluid, yet comprehensive approach. 

---

Why is ToC useful for evaluation? 

• ToC essentially provides a diagram that can be tested by analyzing relevant indicators to see if the program actually contributed to the intended change 

• ToC builds a solid foundation for an impact evaluation by providing a reporting framework and identifying what data needs to be gathered to test the theory 

• A well-developed ToC can help to create better key evaluation questions, identify indicators for monitoring change, and identify gaps in available data 

• This can help to focus new data collection on areas where there are gaps leading to more effective use of evaluation resources 

• By identifying the why and how of a program, ToC can help to reflect on what has worked or not worked to understand the past, identify opportunities for learning, and plan for the future 

However, developing a ToC can be challenging as it involves facilitating collaboration with all key stakeholders, synthesizing a range of views and information sources, as well as obtaining agreement and buy-in from stakeholders. The collaborative aspect of developing a ToC is crucial and will be further discussed in part 2 of our series on ToC. 

---

How has Theory of Change (ToC) helped me in my evaluations? 

ToC has played a BIG part in my role as an evaluator so far in helping me to deliver meaningful results to stakeholders. It has also helped me to build an understanding of change processes and increase the capacities of key stakeholders to do the same. 

Collaboratively developing a ToC before the start of a program has supported my clients in identifying how and why the program is expected to lead to the desired change. It has helped my clients to become clearer in their own understanding, highlight gaps they may not have considered, and document their own underlying assumptions. ToC helped my clients to better understand risk and opportunity within their program’s context to ensure success.  

Collaboratively developing a ToC after a program has been launched for evaluation purposes has also benefitted my clients by helping them to understand why they made certain decisions and why they collaborated with certain actors. As well as providing me with a clear evaluation framework, ToC has helped me to provide actionable recommendations and results. Using ToC as an evaluation tool has also helped my clients to learn more about the benefits of having a clear ToC from the program’s start to help them make decisions and guide their program through challenges. 

---

Have you worked with ToC before, or have questions? Comment on this article or connect with us on LinkedIn or Twitter! 

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address


Sign Up

We respect your privacy.

Thank you!

---