If you’ve been delving into the world of evaluation and Theory of Change (ToC), you are likely to have come across the concept of “assumptions.” Assumptions have a reputation for causing confusion in evaluation. Articulating assumptions can be a tricky task. 

Documenting assumptions about why a program will contribute to change is very important in order to evaluate the success of a program and its outcomes. In this article, we: 

• Describe what assumptions are in evaluation; 

• Explain why you should document assumptions; and, 

• Briefly explain how to reflect on your assumptions when collecting and analyzing evaluation results. 

We also provide some practical examples of how to include assumptions in your own evaluations. 

What are assumptions?

Assumptions are ideas about how a program is expected to influence or contribute to change. These beliefs focus on how change is expected to happen in the short, medium, or long term of a program and its context (i.e., the conditions of the system in which the program is operating). Assumptions are often informed by worldviews and a stakeholders’ prior experience. They can also sometimes be situated in scientific evidence.  

An example of an assumption for a Health Promotion Program is: 

• Outcome: Health Promotion Program participants lead healthier lives.

• Assumption: Once people have the information, they will make better choices.

This assumes that participants currently do not have the information needed to make better choices about their health. By creating and sharing this information with participants, the Health Promotion Program is expected to contribute to change. 

For a deeper dive into the many different types of assumptions (e.g., contextual, normative, diagnostic, prescriptive, etc.), check out Apollo M. Nkwake’s book which includes a detailed discussion on working with assumptions in international development program evaluation. 

Assumptions lie outside the direct control of the program. Assumptions are also outside the program’s interventions and activities, but they are critical to the program’s success. 

Why should I document assumptions when evaluating or discussing a program’s logic?

There are a few reasons why an evaluation should clearly document assumptions: 

1. Documenting assumptions will pinpoint external factors that can affect how a program contributes to change. 

2. By making assumptions clear from the start of an evaluation they become hypotheses which can be tested, challenged, and refined by different stakeholder perspectives. Doing so can support learning on how change is expected to happen. 

3. Documenting assumptions can also inform future activities, collaborations, and a program’s strategic decisions through the increased awareness of how change is expected to happen. 

4. Documenting assumptions provides clarity on whether change will be realized in the future, therefore helping to focus activities and identify opportunities. 

5. Documenting assumptions are therefore critical in helping to articulate, validate, and assess the strength of a program’s theory. 

6. Documenting assumptions can support explanations if things do not go as planned, for better or worse.  

How and when should I document assumptions?

Documenting and testing assumptions is central to developing a ToC or a Log Frame. See our previous ToC tip sheet and ToC template. When creating an evaluation plan and formulating evaluation questions, assumptions can be identified about the change the program is expected to contribute to. The perfect time to start documenting assumptions is at the program planning stages (e.g., a ToC development workshop). If the program is already underway and you’re starting evaluation partway through, make sure that your evaluation kick-off meeting includes the identification of program assumptions.

Here is our 5-step process for identifying assumptions in your next meeting: 

1. Have each stakeholder/member write and share their assumptions about the expected change. 

2. You can use facilitating questions such as “Why is this change expected to happen?” and “What needs to happen for this change to be realized?” 

3. Discuss each stakeholder’s assumptions as a group; refine and aggregate each of the assumptions as needed. This participatory process of documenting assumptions supports clarity among stakeholders about how a program is expected to contribute to change. It also allows stakeholders to identify factors that are critical to achieving desired results. 

4. Identifying assumptions in this participatory way can be challenging. Thinking about change in relation to underlying assumptions may be a new perspective for stakeholders. Therefore, evaluators should recognize these difficulties and be prepared to ask the right questions to support stakeholders in articulating their assumptions. 

5. Assumptions can be actor-specific to ensure they are grounded. For instance, in the example above, we focus on Health Promotion Program participants as the actor group. This helps us when thinking about who is expected to be doing something differently because of the intervention. 

Documenting assumptions and justifying change is a continuous process. It should not be forgotten about following the initial kick-off meetings! Assumptions should be re-visited and re-questioned throughout the evaluation. 

Considering assumptions when analyzing results

It is important that your assumptions are also considered in the data collection processes and when analyzing your results. Collecting evidence on your assumptions can highlight common pitfalls about change underlying a program in contrast to the realities of how change happened. This provides the opportunity for a great learning process! 

Evidence for assumptions can be collected through methods such as interviews, focus groups, and/or surveys. Targeted questions for each of the assumptions can be added to any of these methods that are already planned for your evaluation. 

For the example above, a targeted interview question to gather data on the assumption could be: “Why have you started to make better choices about your health?” 

This will uncover some insight into the causal processes that contributed to participants making better choices about their health. It will also help you to understand whether the information provided by the Health Program played a role in this. 

For those wishing to dive a little deeper into the world of assumptions, check out the book Working with assumptions in International Development Program Evaluation by Apollo M. Nkwake.

Have you worked with assumptions before, or have questions? Comment on this article or connect with us on LinkedIn or Twitter! 

Email Address


Sign Up

We respect your privacy.

Thank you!

Sources: 

• Belcher, B., Davel, R., & Claus, R. (2020) “A refined method for theory-based evaluation of the societal impacts of research”.  MethodsX. 7(1). Available from: https://www.sciencedirect.com/science/article/pii/S221501612030008X 

• Hearn, S. (2009) “Outcome Mapping: Monitoring and Evaluating Policy Influencing” Presentation for Overseas Development Institute. Available from: https://www.slideshare.net/sihearn/om-for-policy-influencing 

• Mitroff, I., & Bonoma, T. V. (1978). Psychological assumptions, experimentation, and real-world problems: A critique and an alternate approach to evaluation. Evaluation Quarterly, 2(2), 235–260. Available from: https://doi.org/10.1177/0193841X7800200204 

• Nkwake, A.M. (2012) Working with Assumptions in International Development Program Evaluation: With a Foreword by Michael Bamberger. Springer Science & Business Media: Berlin, Germany. 

• Stein, D., & Valters, C. (2012) “Understanding Theory of Change in International Development”. Justice and Security Research Programme at the London School of Economics and Political Science. Available from: http://eprints.lse.ac.uk/56359/1/JSRP_Paper1_Understanding_theory_of_change_in_international_development_Stein_Valters_2012.pdf 

Methods such as interviews and focus groups can be an excellent way to collect qualitative data for your evaluation. Check out our previous article on how to conduct interviews. However, there are lots of methods for making sense of qualitative data. These include narrative analysis, discourse analysis, grounded theory, constant comparison analysis, interpretive phenomenological analysis (IPA), and qualitative content analysis. Thematic analysis is one such method that can be used to break down the process into manageable steps. It can also help you to represent your data as honestly as possible.  

This article supports evaluators who are new to qualitative data analysis. We start by defining thematic analysis, then give you a 5-step process to complete your own analysis. We end the article by highlighting some common challenges with thematic analysis. 

Our top tips for Thematic Analysis

1. Read, read, and read your data again! Coding is only efficient when you are completely familiar with your entire data set. 

2. Although this is an exploratory method, keep the purpose of your research in mind. This will stop you from getting carried away and over-coding data that is not relevant. 

3. Don’t rush! Leave yourself plenty of time for qualitative data coding and analysis. It takes practice and you won’t get it right on the first go. This is definitely a learn-by-doing method. 

4. If possible, walk through your process and codes with a colleague or team member. Everyone looks at qualitative data slightly differently. Having someone review it will support validation of your codes and themes, and reduce bias. 

5. Transparency is key! Your methods for analysis should be clearly described and documented. This ensures that your readers are aware of the process you followed. Make sure you manage your data by keeping a master list of codes (if coding by hand) and backing up any work you complete in analysis software. 

What is Thematic Analysis?

Completing a clear and organized analysis of qualitative data is extremely important. Thematic analysis provides you with the opportunity to go through your data in a methodical and thorough way to identify themes and patterns. 

In short, thematic analysis is a way of producing themes from texts such as interview or focus group transcripts. The method makes sense of large amounts of information so that responses to a research question can emerge. 

This method is very flexible. It is also a good method to follow when you want to find out people’s views, opinions, knowledge, or experience on a topic. The most common method of thematic analysis follows a 5 or 6 step process: 1) familiarization; 2) coding; 3) generating themes; 4) reviewing themes; 5) defining and naming themes; and 6) reporting. These steps were defined by Braun & Clarke (2008) in this article which is paywalled.

The method is suitable for both inductive and deductive studies which are described below. However, some steps will not be as long in a deductive process. 

Deductive Studies:

Deductive is coming to the data with predetermined themes that you expect to find based on existing knowledge or established evaluation questions. In our experience, deductive analysis is more common than inductive analysis in evaluation. It starts with a predefined set of codes which are then assigned to the qualitative data set. These codes might come from previous research, or you might already know what themes you are interested in looking for. It is important that other ‘unexpected’ themes are not missed. These can be captured by creating a code such as “Unexpected info” or “Misc. theme.” Deductive coding can save time and help guarantee your areas of interest are coded, but you also need to be careful of bias. 

Inductive Studies:

Inductive is a method of coding that allows the data to determine your themes. This is also known as ‘open coding.’ This is an iterative process that involves lots of refinement and multiple rounds of analysis. It often takes longer but can be more thorough and exploratory than deductive coding. If you’ve ever used a grounded theory approach, you’ve probably done inductive analysis.   

It is important that you decide which method (either inductive or deductive) to use before you start the thematic analysis! 

How to use Thematic Analysis

Step 1: Familiarize yourself with the data 

Having a large qualitative data set can be overwhelming. So where are you supposed to start?  

 The first step is to become familiar with the data. If you collected the data, you may have already started to make notes on areas of interest discussed by the participants. Another great way to become familiar with the data is to complete the transcribing process. (See our tips on how to transcribe interviews like a pro.) But if you outsourced the transcribing or maybe weren’t the one completing the data collection, it is important to spend time reviewing (either reading or listening to) your data. Take notes, memos, and start to jot down some ideas of potential codes. But don’t start coding just yet! At the end of this process, you should be very familiar with the entire body of data.  

Step 2: Generate an initial set of codes from a first review, and code your data 

What exactly is a code? A code is a brief description of what is being said in the interview or focus group extract. It’s a description, not an interpretation (we save this part for later!). 

A code is a word or a short phrase that captures the meaning of specific quotes. What codes you use depends on what is being said within your data and on the purpose of your research. It also depends on whether you are performing an exploratory analysis (i.e., inductive) where the themes depend on the data. Or a deductive analysis where you search for specific themes. At this stage, your codes might look something like this: 

• Job security 

• Learning resources 

• Horizontal violence 

• Unions 

• Wages 

Overall, you should start to identify codes by reading through your data and applying the same code to sections of the text that represent the same meaning. You should also start to create a codebook to keep track of the codes. If you are coding by hand, you can use sticky notes and colour coding to start to organize your data in a meaningful and systematic way. 

There are also different software packages that can support coding such as NVivo (my personal favourite) and MAXQDA. Be thorough at this initial stage and don’t be afraid of over-coding. We will refine our codes in the later steps. Be careful not to lose context by coding too little of your data. 

In my own experience, coding at this general level first is a step towards organizing the data into meaningful categories. It also allows me to discover reoccurring concepts that could be further refined. Coding in this exploratory way enables me to recognize and recontextualize the data to give a fresh perception of what was already visible. 

Consistent coding is accurate coding, so establishing coding procedures and guidelines from the start is crucial.

Step 3: Start to search for themes in your codes across the entire data set 

This step is all about organizing your codes and starting to identify reoccurring themes. If doing this by hand, cut out all of the data extracts pertaining to the specific codes and start to group the codes together. If using software, the software will automatically do this for you. At this stage, you might decide that some codes are unclear or not relevant to your study. Now is the time to modify and update codes that may not align with the purpose of your project. You can also reflect on whether there is any missing data. For example, is there anything that wasn’t discussed which you found surprising? 

What is a theme? Themes are broader than codes (often combining several codes into a theme) and involve interpretation of the codes and the data. A theme captures something important about the data in relation to the research purpose. It also represents a pattern or relationship across the data set. Searching for common themes across codes is an iterative process where you move back and forth between the codes to identify commonalities.  

Once you have a list of main themes you can start to reflect on the relationships between them and how the themes fit together to tell a bigger story. 

Step 4: Review and refine your themes 

Review your themes by (once again) reading through your data excerpts and ensuring that there are identifiable differences between your themes. Themes need to be clear and distinctive, as well as tell a story. This review process will help to identify new themes you might have missed and make sure that your themes are useful and accurate representations of the data. Ensure that each of your themes have enough data to be convincing and show patterns across the data set.  

Some overarching questions can help you to review your themes: 

• Do the themes make sense? 

• Does the data support the themes? 

• Is the theme too broad or too narrow? 

• If themes overlap, are they really separate themes? 

• Are there themes within themes (subthemes)? 

• Am I missing any themes? 

Define exactly what you mean by each theme. You can also construct a thematic map like the one below to help refine your themes. Discussing your themes with a colleague or your team can help to ensure agreement across findings. You can also use intercoder reliability i.e., having a colleague code the data set using your codebook to see whether you are capturing the same thing. The general rule of thumb is that each theme should not have more than 4-5 codes. It is also better to have 6-10 broad themes with sub-themes rather than lots of really detailed themes.

Step 5: Reporting 

It is crucial that you clearly communicate the methods and steps that you took in your thematic analysis. You may also want to share a copy of your codebook and related themes in an appendix of the report. 

Reporting needs to go beyond just describing your data and should include your own analysis to make an argument for the claims and the story you present. Don’t just paraphrase your data. Make sure you tell a coherent story about your data and choose quotes that back up your points. You can also create an “exemplar quote” code to highlight quotes that you might want to use that help to capture the essence of themes. This helps to make quotes easy to find and will save you re-reading the excerpts another time! 

Ensure any ethical processes are followed e.g., removing identifiable features of the quotes. In projects with small populations, we’ll sometimes even remove any unique turns of phrase or colloquialisms that seem like they could identify the speaker.  

Be aware of the following challenges in thematic analysis: 

• Confirmation bias: thematic analysis is subjective. It often relies on the evaluators/researcher’s judgement. Reflect on the bias of your own interpretations. Document what these might be and how they might affect the results. 

• If you’re completing a deductive process, look explicitly for contradictory evidence in the data to minimize bias or missing any important points. 

• Don’t just organize, make sure you actually analyze the data! Don’t use the main interview questions as themes, a deeper level of analysis is required. 

This can be a long process, especially for first-time coders! So make sure you leave plenty of time in your evaluation plan/timeline to complete the thematic analysis. 

Why not give it a go and let us know how you got on! 

Email Address


Sign Up

We respect your privacy.

Thank you!

Sources: 

• Braun, V. & Clarke, V. (2006) “Using thematic analysis in psychology”. Qualitative Research in Psychology. 3(2): 77-101. Available from: https://www.tandfonline.com/doi/abs/10.1191/1478088706qp063oa 

• Ryan, G. & Bernard, H.R. (2003) “Techniques to Identify Themes”. Field Methods. 15(1): 85-109. Available from: https://www.researchgate.net/publication/241176170_Techniques_to_Identify_Themes