The May 13 Group

the next day for evaluation

You are here: Home / Archives for allblogs / evalacademy

evalacademy

The importance of articulating assumptions

If you’ve been delving into the world of evaluation and Theory of Change (ToC), you are likely to have come across the concept of “assumptions.” Assumptions have a reputation for causing confusion in evaluation. Articulating assumptions can be a tricky task. 

Documenting assumptions about why a program will contribute to change is very important in order to evaluate the success of a program and its outcomes. In this article, we: 

  • Describe what assumptions are in evaluation; 

  • Explain why you should document assumptions; and, 

  • Briefly explain how to reflect on your assumptions when collecting and analyzing evaluation results. 

We also provide some practical examples of how to include assumptions in your own evaluations. 

What are assumptions?

Assumptions are ideas about how a program is expected to influence or contribute to change. These beliefs focus on how change is expected to happen in the short, medium, or long term of a program and its context (i.e., the conditions of the system in which the program is operating). Assumptions are often informed by worldviews and a stakeholders’ prior experience. They can also sometimes be situated in scientific evidence.  

An example of an assumption for a Health Promotion Program is: 

  • Outcome: Health Promotion Program participants lead healthier lives.

  • Assumption: Once people have the information, they will make better choices.

This assumes that participants currently do not have the information needed to make better choices about their health. By creating and sharing this information with participants, the Health Promotion Program is expected to contribute to change. 

For a deeper dive into the many different types of assumptions (e.g., contextual, normative, diagnostic, prescriptive, etc.), check out Apollo M. Nkwake’s book which includes a detailed discussion on working with assumptions in international development program evaluation. 

Assumptions lie outside the direct control of the program. Assumptions are also outside the program’s interventions and activities, but they are critical to the program’s success. 

Why should I document assumptions when evaluating or discussing a program’s logic?

There are a few reasons why an evaluation should clearly document assumptions: 

  1. Documenting assumptions will pinpoint external factors that can affect how a program contributes to change. 

  2. By making assumptions clear from the start of an evaluation they become hypotheses which can be tested, challenged, and refined by different stakeholder perspectives. Doing so can support learning on how change is expected to happen. 

  3. Documenting assumptions can also inform future activities, collaborations, and a program’s strategic decisions through the increased awareness of how change is expected to happen. 

  4. Documenting assumptions provides clarity on whether change will be realized in the future, therefore helping to focus activities and identify opportunities. 

  5. Documenting assumptions are therefore critical in helping to articulate, validate, and assess the strength of a program’s theory. 

  6. Documenting assumptions can support explanations if things do not go as planned, for better or worse.  

How and when should I document assumptions?

Documenting and testing assumptions is central to developing a ToC or a Log Frame. See our previous ToC tip sheet and ToC template. When creating an evaluation plan and formulating evaluation questions, assumptions can be identified about the change the program is expected to contribute to. The perfect time to start documenting assumptions is at the program planning stages (e.g., a ToC development workshop). If the program is already underway and you’re starting evaluation partway through, make sure that your evaluation kick-off meeting includes the identification of program assumptions.

Here is our 5-step process for identifying assumptions in your next meeting: 

  1. Have each stakeholder/member write and share their assumptions about the expected change. 

  2. You can use facilitating questions such as “Why is this change expected to happen?” and “What needs to happen for this change to be realized?” 

  3. Discuss each stakeholder’s assumptions as a group; refine and aggregate each of the assumptions as needed. This participatory process of documenting assumptions supports clarity among stakeholders about how a program is expected to contribute to change. It also allows stakeholders to identify factors that are critical to achieving desired results. 

  4. Identifying assumptions in this participatory way can be challenging. Thinking about change in relation to underlying assumptions may be a new perspective for stakeholders. Therefore, evaluators should recognize these difficulties and be prepared to ask the right questions to support stakeholders in articulating their assumptions. 

  5. Assumptions can be actor-specific to ensure they are grounded. For instance, in the example above, we focus on Health Promotion Program participants as the actor group. This helps us when thinking about who is expected to be doing something differently because of the intervention. 

Documenting assumptions and justifying change is a continuous process. It should not be forgotten about following the initial kick-off meetings! Assumptions should be re-visited and re-questioned throughout the evaluation. 

Considering assumptions when analyzing results

It is important that your assumptions are also considered in the data collection processes and when analyzing your results. Collecting evidence on your assumptions can highlight common pitfalls about change underlying a program in contrast to the realities of how change happened. This provides the opportunity for a great learning process! 

Evidence for assumptions can be collected through methods such as interviews, focus groups, and/or surveys. Targeted questions for each of the assumptions can be added to any of these methods that are already planned for your evaluation. 

For the example above, a targeted interview question to gather data on the assumption could be: “Why have you started to make better choices about your health?” 

This will uncover some insight into the causal processes that contributed to participants making better choices about their health. It will also help you to understand whether the information provided by the Health Program played a role in this. 

For those wishing to dive a little deeper into the world of assumptions, check out the book Working with assumptions in International Development Program Evaluation by Apollo M. Nkwake.

Have you worked with assumptions before, or have questions? Comment on this article or connect with us on LinkedIn or Twitter! 


We respect your privacy.

Thank you!

Sources: 

 

Interpreting themes from qualitative data: thematic analysis

Methods such as interviews and focus groups can be an excellent way to collect qualitative data for your evaluation. Check out our previous article on how to conduct interviews. However, there are lots of methods for making sense of qualitative data. These include narrative analysis, discourse analysis, grounded theory, constant comparison analysis, interpretive phenomenological analysis (IPA), and qualitative content analysis. Thematic analysis is one such method that can be used to break down the process into manageable steps. It can also help you to represent your data as honestly as possible.  

This article supports evaluators who are new to qualitative data analysis. We start by defining thematic analysis, then give you a 5-step process to complete your own analysis. We end the article by highlighting some common challenges with thematic analysis. 

Our top tips for Thematic Analysis

  1. Read, read, and read your data again! Coding is only efficient when you are completely familiar with your entire data set. 

  2. Although this is an exploratory method, keep the purpose of your research in mind. This will stop you from getting carried away and over-coding data that is not relevant. 

  3. Don’t rush! Leave yourself plenty of time for qualitative data coding and analysis. It takes practice and you won’t get it right on the first go. This is definitely a learn-by-doing method. 

  4. If possible, walk through your process and codes with a colleague or team member. Everyone looks at qualitative data slightly differently. Having someone review it will support validation of your codes and themes, and reduce bias. 

  5. Transparency is key! Your methods for analysis should be clearly described and documented. This ensures that your readers are aware of the process you followed. Make sure you manage your data by keeping a master list of codes (if coding by hand) and backing up any work you complete in analysis software. 

What is Thematic Analysis?

Completing a clear and organized analysis of qualitative data is extremely important. Thematic analysis provides you with the opportunity to go through your data in a methodical and thorough way to identify themes and patterns. 

In short, thematic analysis is a way of producing themes from texts such as interview or focus group transcripts. The method makes sense of large amounts of information so that responses to a research question can emerge. 

This method is very flexible. It is also a good method to follow when you want to find out people’s views, opinions, knowledge, or experience on a topic. The most common method of thematic analysis follows a 5 or 6 step process: 1) familiarization; 2) coding; 3) generating themes; 4) reviewing themes; 5) defining and naming themes; and 6) reporting. These steps were defined by Braun & Clarke (2008) in this article which is paywalled.

The method is suitable for both inductive and deductive studies which are described below. However, some steps will not be as long in a deductive process. 

Deductive Studies:

Deductive is coming to the data with predetermined themes that you expect to find based on existing knowledge or established evaluation questions. In our experience, deductive analysis is more common than inductive analysis in evaluation. It starts with a predefined set of codes which are then assigned to the qualitative data set. These codes might come from previous research, or you might already know what themes you are interested in looking for. It is important that other ‘unexpected’ themes are not missed. These can be captured by creating a code such as “Unexpected info” or “Misc. theme.” Deductive coding can save time and help guarantee your areas of interest are coded, but you also need to be careful of bias. 

Inductive Studies:

Inductive is a method of coding that allows the data to determine your themes. This is also known as ‘open coding.’ This is an iterative process that involves lots of refinement and multiple rounds of analysis. It often takes longer but can be more thorough and exploratory than deductive coding. If you’ve ever used a grounded theory approach, you’ve probably done inductive analysis.   

It is important that you decide which method (either inductive or deductive) to use before you start the thematic analysis! 

How to use Thematic Analysis

Step 1: Familiarize yourself with the data 

Having a large qualitative data set can be overwhelming. So where are you supposed to start?  

 The first step is to become familiar with the data. If you collected the data, you may have already started to make notes on areas of interest discussed by the participants. Another great way to become familiar with the data is to complete the transcribing process. (See our tips on how to transcribe interviews like a pro.) But if you outsourced the transcribing or maybe weren’t the one completing the data collection, it is important to spend time reviewing (either reading or listening to) your data. Take notes, memos, and start to jot down some ideas of potential codes. But don’t start coding just yet! At the end of this process, you should be very familiar with the entire body of data.  

Step 2: Generate an initial set of codes from a first review, and code your data 

What exactly is a code? A code is a brief description of what is being said in the interview or focus group extract. It’s a description, not an interpretation (we save this part for later!). 

A code is a word or a short phrase that captures the meaning of specific quotes. What codes you use depends on what is being said within your data and on the purpose of your research. It also depends on whether you are performing an exploratory analysis (i.e., inductive) where the themes depend on the data. Or a deductive analysis where you search for specific themes. At this stage, your codes might look something like this: 

  • Job security 

  • Learning resources 

  • Horizontal violence 

  • Unions 

  • Wages 

Overall, you should start to identify codes by reading through your data and applying the same code to sections of the text that represent the same meaning. You should also start to create a codebook to keep track of the codes. If you are coding by hand, you can use sticky notes and colour coding to start to organize your data in a meaningful and systematic way. 

There are also different software packages that can support coding such as NVivo (my personal favourite) and MAXQDA. Be thorough at this initial stage and don’t be afraid of over-coding. We will refine our codes in the later steps. Be careful not to lose context by coding too little of your data. 

In my own experience, coding at this general level first is a step towards organizing the data into meaningful categories. It also allows me to discover reoccurring concepts that could be further refined. Coding in this exploratory way enables me to recognize and recontextualize the data to give a fresh perception of what was already visible. 

Consistent coding is accurate coding, so establishing coding procedures and guidelines from the start is crucial.

Step 3: Start to search for themes in your codes across the entire data set 

This step is all about organizing your codes and starting to identify reoccurring themes. If doing this by hand, cut out all of the data extracts pertaining to the specific codes and start to group the codes together. If using software, the software will automatically do this for you. At this stage, you might decide that some codes are unclear or not relevant to your study. Now is the time to modify and update codes that may not align with the purpose of your project. You can also reflect on whether there is any missing data. For example, is there anything that wasn’t discussed which you found surprising? 

What is a theme? Themes are broader than codes (often combining several codes into a theme) and involve interpretation of the codes and the data. A theme captures something important about the data in relation to the research purpose. It also represents a pattern or relationship across the data set. Searching for common themes across codes is an iterative process where you move back and forth between the codes to identify commonalities.  

Once you have a list of main themes you can start to reflect on the relationships between them and how the themes fit together to tell a bigger story. 

Step 4: Review and refine your themes 

Review your themes by (once again) reading through your data excerpts and ensuring that there are identifiable differences between your themes. Themes need to be clear and distinctive, as well as tell a story. This review process will help to identify new themes you might have missed and make sure that your themes are useful and accurate representations of the data. Ensure that each of your themes have enough data to be convincing and show patterns across the data set.  

Some overarching questions can help you to review your themes: 

  • Do the themes make sense? 

  • Does the data support the themes? 

  • Is the theme too broad or too narrow? 

  • If themes overlap, are they really separate themes? 

  • Are there themes within themes (subthemes)? 

  • Am I missing any themes? 

Define exactly what you mean by each theme. You can also construct a thematic map like the one below to help refine your themes. Discussing your themes with a colleague or your team can help to ensure agreement across findings. You can also use intercoder reliability i.e., having a colleague code the data set using your codebook to see whether you are capturing the same thing. The general rule of thumb is that each theme should not have more than 4-5 codes. It is also better to have 6-10 broad themes with sub-themes rather than lots of really detailed themes.

Step 5: Reporting 

It is crucial that you clearly communicate the methods and steps that you took in your thematic analysis. You may also want to share a copy of your codebook and related themes in an appendix of the report. 

Reporting needs to go beyond just describing your data and should include your own analysis to make an argument for the claims and the story you present. Don’t just paraphrase your data. Make sure you tell a coherent story about your data and choose quotes that back up your points. You can also create an “exemplar quote” code to highlight quotes that you might want to use that help to capture the essence of themes. This helps to make quotes easy to find and will save you re-reading the excerpts another time! 

Ensure any ethical processes are followed e.g., removing identifiable features of the quotes. In projects with small populations, we’ll sometimes even remove any unique turns of phrase or colloquialisms that seem like they could identify the speaker.  

Be aware of the following challenges in thematic analysis: 

  • Confirmation bias: thematic analysis is subjective. It often relies on the evaluators/researcher’s judgement. Reflect on the bias of your own interpretations. Document what these might be and how they might affect the results. 

  • If you’re completing a deductive process, look explicitly for contradictory evidence in the data to minimize bias or missing any important points. 

  • Don’t just organize, make sure you actually analyze the data! Don’t use the main interview questions as themes, a deeper level of analysis is required. 

This can be a long process, especially for first-time coders! So make sure you leave plenty of time in your evaluation plan/timeline to complete the thematic analysis. 

Why not give it a go and let us know how you got on! 


We respect your privacy.

Thank you!

Sources: 

 

Evaluation Facilitation Series: Facilitation Activity #1 (Making Metaphors)

Evaluators need to be many things; a good facilitator is high up on the list. We facilitate all sorts of conversations throughout the evaluation process (or at least we should!).

Sometimes just posing questions to a group and having a discussion works, but other times we may need to put in a little more effort to elicit the rich discussion and insights we are looking for.

After all, evaluation may get you jazzed up, but that probably is not the case for most people! To make sure stakeholders are engaged and get just as excited about evaluation as us, we need to use all the tools we have to support evaluation use.

Earlier in my career, I found myself searching the Internet for facilitated activity ideas I could apply in my evaluations. In 2014, I came across Jean A. King and Laurie Stevahn’s interactive evaluation practice workshop that was being offered at the American Evaluation Association conference.

In this workshop, they introduced interactive strategies that could be applied across various evaluation aims, approaches, and contexts. Jean and Laurie based this workshop on their book called Interactive Evaluation Practice: Mastering the Interpersonal Dynamics of Program Evaluation. I have revisited this book many times since that workshop when I need some inspiration for facilitation activities.

This evaluation facilitation series will highlight some of these facilitation activities and how I have applied them over the years. This article focuses on their “Making Metaphors” activity, along with some step-by-step instructions you can use to try it in your own evaluations.

Making Metaphors

When to use

Metaphors and similes are effective at taking an abstract or hard-to-understand concept and comparing it to a simple, concrete concept. As a result, good metaphors are effective at helping people understand something they otherwise might not have (check out this article where we compared writing an evaluation report to cooking). I like using the Making Metaphors activity to: 

  • Assist people with important or difficult conversations 

  • Integrate views and develop a shared understanding on a topic 

  • Articulate what is known to the group 

  • Spark humour and playfulness 

In particular, I like to use this activity when I start an evaluation. At the start of an evaluation, I want to understand the ideas, perceptions, and experiences those I am working with have around evaluation so I can adjust my approach accordingly. Sometimes I incorporate the Making Metaphors activity in the evaluation kick-off meeting when people introduce themselves (see our What conversations do you need to have at the start of an evaluation? article for more kick-off meeting ideas). As part of their introduction, I will ask them to think about the metaphor or simile (e.g., “Evaluation is like…”) and then ask them to complete it.  

This activity is also good to use when completing a logic model or theory of change with a group and you need them to clearly articulate what success looks like.  

The following outlines directions for the activity, along with an example and tips to consider. 

Directions 

Group Size 

This activity can work with any group size, but works best if you break participants into groups of 2 – 5 people. 

Time Required 

5 -10 minutes per metaphor 

 

Instructions 

  1. Present an open-ended metaphor or simile (using the word like) stems. For example: 

    • “My previous evaluation experiences have been like…” 

    • “Evaluation is like…” 

    • “A successful evaluation looks like…” 

    • “Success of the program looks like…” 

  2. Ask participants to complete the metaphor or simile. The quickest and simplest way is to ask participants to respond using words. However, you can also ask participants to create a response by selecting an object (see Tips below). If you have the time, I would suggest trialing the object over the words, because words can sometimes be constraining and using an object depends less on verbal acuity. 

  3. (If an object is selected) Give participants time to find an object until they find one that relates to and completes the metaphor or simile. Connections to the object can be anything – literal, rational, emotional, intuitive, symbolic etc. 

  4. (If an object is selected) Ask participants to take a few minutes to 1) write down the details of the object and then, 2) the connection they made between the object and the metaphor. 

  5. Ask participants to share back the word or details of the object and how it relates to the metaphor. 

  6. In groups, identify patterns, similar qualities, and/or unique outliers across the responses. 

  7. Have each group share back their new insights with the larger group and what it means going forward. 

 

Example script for “Evaluation is like…” metaphor stem: 

  1. I want you to look around the room for an object or something you feel represents what you think of when you hear the term “evaluation.” 

  2. Write a few notes about the object. Describe it in detail first and then describe how it relates to evaluation.  

  3. In your group, share back why you chose that object. Take time to describe the object and its connection to the following simile: “Evaluation is like…” 

  4. After you have all shared, have a discussion in your group about any patterns, similarities, and differences in responses.  

  5. After your discussion, have your group identify one idea or suggestion for how to make this the best evaluation experience for the group. 

Tips 

Objects can be random items you collect from around your house. Store the collection of items in a plastic box, tote bag, or other easily transportable container. 

You can also ask participants to select items in their pockets, wallets, handbags, briefcases, or things they are wearing or from items they see in the room (selecting from a room works best with virtual meetings). 

Instead of objects, you can also use sets of visual images (e.g., pictures, art cards, photos, postcards, magazine illustrations, etc.). At My Best are a set of cards that contain words on one side and a picture on the other. I have used these in workshops and they are more compact to carry than a box of random items!  

Display sets of visual images or tangible objects on a designated table in the room. Participants gather around the table to look at the items and select one to complete the metaphor. You can line up the images or scatter – doesn’t matter!

Each set of words, images, or objects should contain at least 10 to 15 items more than the total number of people participating (e.g., if 25 people participate, then the set should contain about 35–40 items).

If you’re needing to infuse a bit more creativity into your evaluation practice then try out the Making Metaphors activity.

If you haven’t already, subscribe to our newsletter for more facilitation inspiration!


We respect your privacy.

Thank you!

 

Evaluation Question Examples by Type of Evaluation

We’ve defined and written about How to Write Good Evaluation Questions before. We’ve even shared some examples of evaluation questions based on the sector or content area. But there is another way to think about evaluation questions. Let’s look at how using different evaluation strategies or frameworks can help you to craft those perfect evaluation questions.

To describe how evaluation questions differ based on type of evaluation, let’s make up a hypothetical program to evaluate.

You’ve been hired to evaluate a program that has been operating for one year. The program aims to engage youth in an after-school program with the goal of keeping at-risk youth safe. They offer physical activity programs (e.g., basketball), tutoring, and life-skills programs (e.g., cooking classes), as well as lounge areas.

Scenario 1: Formative Evaluation 

The purpose of formative evaluation is to assess how the program is being implemented. Key evaluation questions should focus on enablers and barriers of implementation. For example: 

Awareness 

  • Are youth aware of the program?  

  • Are families aware of the program? 

Utilization 

  • What are program utilization rates? 

  • Why are youth attending, or not? 

  • What barriers exist in attending various programs? 

  • What parts of the program are most desirable? 

Implementation 

  • What is working well?  

  • What could be improved? 

I’ve found that looking to the field of implementation science can help to outline themes and even specific questions that might be relevant to formative evaluation. 

Scenario 2: Summative Evaluation 

After a year of operations, the program leaders believe it is time to assess the impact of the program. Your summative key evaluation questions should focus on outcomes: 

Utilization 

  • Who used the program? What is the profile of the youth who attended our program? 

Outcomes 

  • Did more youth graduate high school? Did fewer youth drop out of school? 

  • Did youth experience fewer disciplinary actions at school? In the community? 

  • Did youth learn new life skills? Do those life skills support them in gaining employment? 

Scenario 3: Using the RE-AIM framework

First a quick note: most of these scenarios are not mutually exclusive. RE-AIM is well suited to both formative and summative evaluation. For an overview of RE-AIM read this. RE-AIM essentially comes with its own questions, that only need to be adapted to your specific program.  The key evaluation questions are based on the 5 focus areas of RE-AIM: 

  • Reach – Who is coming to the program? Who is not? 

  • Effectiveness – Is the program engaging at-risk youth? Are more youth finishing high school? 

  • Adoption – How does the program collaborate and integrate within the community? What partnerships enable the program’s success? How is awareness being spread? 

  • Implementation – How is the program being implemented? Is it implemented as intended? What barriers exist? 

  • Maintenance – Are the outcomes sustained over time? 

Scenario 4: Developmental Evaluation 

The program is interested in really understanding how they can be adaptable to the current context and how they can continue to develop and grow to have a sustainable impact on the complexity presented in at-risk youth. In the last year they haven’t yet landed on a stable program – constantly adapting to new information and changing environments. They want to use Developmental Evaluation to support rapid growth and emergent innovations. Key evaluation questions should focus on emergent learning and opportunities for development: 

  • What are we learning that informs our development? 

  • How are we engaging the community? 

  • What evidence of effectiveness is useful to our development? 

  • What opportunities are emerging? 

For more great examples of DE evaluation questions, check out Developmental Evaluation Exemplars: Principles in Practice edited by Michael Quinn Patton, Kate McKegg and Nan Wehipeihana. 

Scenario 5: Utilization-focused evaluation 

Again, Utilization-focused evaluation is not at all mutually exclusive from previously described scenarios. In this scenario, you determine that the primary purpose of evaluation is to give the staff something they can action (or use). In this case, the approach to development of the evaluation questions is key: engage your stakeholders often, early, and deliberately. What do they need to know? What decisions do they need to make? Key evaluation questions may include: 

  • How can the program attract more at-risk youth? 

  • How can the program retain/continue to engage at-risk youth? 

  • Which programs have the most demand? 

  • Which programs have the best return on investment? 

Scenario 6: Most Significant Change Evaluation 

Most Significant Change evaluation moves away from standard indicators and measuring. It uses storytelling to evaluate the success of a program through the lens of various stakeholders. For this evaluative approach questions may be tailored to each audience: 

For program staff: 

  • What is the biggest success you have seen? 

For youth: 

  • What is the biggest change you have experienced since participating in this program? 

Scenario 7: Outcome Harvesting 

Outcome Harvesting is a participatory evaluation methodology. Instead of key evaluation questions, Outcome Harvesting focuses on Outcome Descriptions. You may ask questions of the stakeholders like: 

  • What have you worked on? (or participated in?) 

  • What was the significance of the work you did? 

  • What impact do you think it had? 

Scenario 8: Principles-Focused Evaluation 

Principles-focussed evaluation aims to assess how an applied set of principles translated into action or behaviour. Your methodologies may be very similar to other evaluation approaches, but the questions may centre on a pre-defined list of principles. Let’s say the program had a list of core values that included: 

  • Person-centred supports. The program aims to offer support to each youth, based on individual, specific needs. 

  • Welcoming. The program has no exclusionary criteria. 

  • Fun! The program encourages staff and youth to have fun and try new things. 

Your evaluation questions may then look like: 

  • Do youth feel supported? 

  • What barriers exist to accessing the program? 

  • What is the experience of participating? 

Again, Michael Quinn Patton has an entire guide to Principles-Focussed Evaluation that can help you develop the right evaluation questions. 

Hopefully, this has helped to showcase how some evaluation approaches and frameworks can shape your evaluation questions.

Program evaluation is certainly not limited to one dominant question. Most evaluations can assess several domains. The rule of thumb is 3 – 5 key evaluation questions, with nuanced sub-questions embedded if needed.

Check out our checklist that will help to ensure you’ve considered some important details in crafting your evaluation questions.


We respect your privacy.

Thank you!

 

Book Review: Developmental Evaluation by Michael Quinn Patton

I recently finished reading Michael Quinn Patton’s Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use (2011) cover-to-cover. I am working on a project using Developmental Evaluation (DE) and felt like I wanted to take a deeper dive into my understanding.

I decided to read it slowly, taking lots of notes, using my highlighter (until it ran out of fluid!), and writing notes to myself in the margins. I read only a handful of pages a day, and sometimes only a handful a week. It took me just over 3 months to finish, but I wasn’t in any rush. 

Patton has a conversational style of writing. It’s casual, full of puns, jokes, tangents, and many, many stories. It’s no surprise, then, that his teaching style is by example. He doesn’t so much tell you how to do DE as he shares real-life stories and examples. He’s offering you the privilege of experiencing his experiences through his storytelling. The book is not light on examples to draw from.

Though he tries his best to lay out how the narrative showcases DE, it’s tricky. DE is many different things depending on your context. In fact, I struggled to even find a concise definition of DE! Briefly, DE is about “bringing evaluative thinking and data to bear, as a project is conceptualized and developed.” (page 3); “DE is designed …to nurture development, emergent, innovation and transformative processes.” (page 7). 

If he’s not teaching through example, he’s teaching through repetition. Which, let’s admit it, is annoying when you’re in it, but super helpful in the long run. It makes those key messages sticky! He has dedicated ample pages to helping the reader identify situations in which DE may apply and explaining what about those situations make them a good fit for DE. 

A quick rundown of the book goes like this: early chapters are dedicated to “What is DE?” and the role of the evaluator. It then moves into identifying situations to use DE – it doesn’t apply to every evaluation, so how does one know?

The book takes a bit of a detour (in my opinion) in Chapter 5 to talk at length about systems thinking, before going back to how DE applies to the lifecycle of complex programs. It’s not until Chapter 8 that we start to get a little more practical about how to do DE. But note that I say “a little.” One of the key points of the book is that there is no one way or one right way to do DE. It’s not about the methods or the approach so much as the principles and desire to develop that are foundational to DE. However, in Chapter 8, he does offer a practical list of inquiry frameworks that can be used in DE.

Chapter 9: “Bricolage” (meaning “something constructed or created from a diverse range of available things”) is like the junk drawer of DE information. It’s full of “other things to know about DE” before the book is wrapped up in Chapter 10: an excellent conclusion, summarizing places to use DE, core characteristics of DE and necessary characteristics of the evaluator. 

If you don’t have the time to read this text cover-to-cover, I might suggest Chapters 1 and 2 for a comprehensive overview, and Chapters 8, maybe 9, and 10 for more practical application tips. 

The tricky thing about DE is that one can’t really explain how to do it. It is not, itself, a methodology. Reading this book and hoping to gain a step-by-step how-to is a misstep. This book explains why DE is a necessary tool in the evaluator tool kit – why formative and summative evaluation are sometimes insufficient. The storytelling throughout the book is the point. DE is about going on a journey to arrive in a place that maybe you’d thought of, but likely didn’t predict. I found reading the book slowly to be helpful. Instead of blazing past the stories, I allowed myself time to think about them, to reflect, and I think that helped me to understand DE just a little bit more. 

It was great to have a project that I’m actively working on to apply some of the strategies, to shape my thinking, and to reinforce key messaging to my clients using Patton’s own words. “I need to be included in strategic planning sessions and discussions” and “my job is to provide you with useful information in as-close-to-possible real-time, and for you to use that information. Let me facilitate a discussion with the team about what this means for you” have become my regular catchphrases. 

I think the intended audience for the book is the evaluator: the user of DE, but it could also be a valuable read for those interested in social change, and those who don’t know how evaluation can help in their innovative, unplanned, complex situation. 

Have you read it? Let me know your thoughts! If you haven’t read it and you may use DE, I think this book is worth your time. 

For more on DE, check out our Six Lessons from Practicing DE.


We respect your privacy.

Thank you!

 

en English
af Afrikaanssq Shqipam አማርኛar العربيةhy Հայերենaz Azərbaycan dilieu Euskarabe Беларуская моваbn বাংলাbs Bosanskibg Българскиca Catalàceb Cebuanony Chichewazh-CN 简体中文zh-TW 繁體中文co Corsuhr Hrvatskics Čeština‎da Dansknl Nederlandsen Englisheo Esperantoet Eestitl Filipinofi Suomifr Françaisfy Fryskgl Galegoka ქართულიde Deutschel Ελληνικάgu ગુજરાતીht Kreyol ayisyenha Harshen Hausahaw Ōlelo Hawaiʻiiw עִבְרִיתhi हिन्दीhmn Hmonghu Magyaris Íslenskaig Igboid Bahasa Indonesiaga Gaeilgeit Italianoja 日本語jw Basa Jawakn ಕನ್ನಡkk Қазақ тіліkm ភាសាខ្មែរko 한국어ku كوردی‎ky Кыргызчаlo ພາສາລາວla Latinlv Latviešu valodalt Lietuvių kalbalb Lëtzebuergeschmk Македонски јазикmg Malagasyms Bahasa Melayuml മലയാളംmt Maltesemi Te Reo Māorimr मराठीmn Монголmy ဗမာစာne नेपालीno Norsk bokmålps پښتوfa فارسیpl Polskipt Portuguêspa ਪੰਜਾਬੀro Românăru Русскийsm Samoangd Gàidhligsr Српски језикst Sesothosn Shonasd سنڌيsi සිංහලsk Slovenčinasl Slovenščinaso Afsoomaalies Españolsu Basa Sundasw Kiswahilisv Svenskatg Тоҷикӣta தமிழ்te తెలుగుth ไทยtr Türkçeuk Українськаur اردوuz O‘zbekchavi Tiếng Việtcy Cymraegxh isiXhosayi יידישyo Yorùbázu Zulu