evalacademy

Jul 31 2023

Creating a Qualitative Codebook

This article is rated as:

What is a codebook?

A codebook for qualitative research is a stand-alone document that contains a list of themes, codes, and definitions that you are using in your qualitative analysis. A codebook helps to keep the entire team on the same page about the qualitative analysis and supports turning your analysis into a clear summary, where you can easily add or consolidate details as needed. A codebook can be used with both inductive and deductive coding approaches. In inductive coding, you aim to develop a theory or develop codes and themes as you go. With deductive coding, you aim to test an existing theory, meaning you have a coding structure developed in advance.

Having a codebook can help you keep track of themes and codes, especially when you have multiple coders or emerging themes. While some qualitative analysis software provides you with a space to include the information contained within a codebook, the information isn’t all contained in one place. Having a separate codebook allows you to have one easily updatable document that contains all the information in one place. I like to keep my codebook open in a separate tab while I am coding regardless of whether I’m coding in fancy software or in Excel or Word.

I like to build my qualitative codebook in Excel, adding columns and rows to fit whatever analysis I’m completing. Typically, my codebooks start with the same 4 key components:

Theme and code headings
Code definition and description
Examples or key quotes
References to source documents

How to create a codebook

To help you understand how to create and use a codebook, I’ll walk through each of these key components, then provide you with two examples.

Theme and code headings

The first few columns in my codebook are dedicated to the theme, subtheme (if necessary) and code. Each row in the codebook is dedicated to one code. If you are using a deductive coding approach, you would build these columns and rows into your codebook before you start coding, populating the themes and codes in advance.

Code definition and description

Next up are the code definition and description columns. Sometimes I just create one combined definition and description column, but in more detailed projects it is helpful to keep them separate. With a deductive coding approach, you can complete the code definition column in advance, while if you are taking an inductive approach, you will complete these as you go. Codes should be clearly defined to avoid overlap between codes. Even with a deductive coding approach, the definition of what is included in each code will likely evolve over time or need clarification as you dive into coding. When coding as a group, the group should discuss all changes to code definitions before they are made. The code description section is where you can jot down some high-level details of what you are finding within the code. This section also helps you summarize your findings and write your reports. Just remember, the code definition is where you put what the code is about, while the code description is where you put the details of what you are finding in each code.

Examples or key quotes

This section is where I put high-quality examples of findings in a code. Sometimes I have an ‘examples’ column and a ‘quotes’ column and sometimes just one column will do. Make sure to include references to your source documents (e.g., which interview or focus group the quote came from) when including quotes in these columns.

References

This is one of the most subjective sections that will vary from project to project, but typically I use this space to indicate which interviews or focus groups the code was discussed in. If I’m analyzing interviews or focus groups that fall into different categories, you can use the reference columns to indicate whether the code was discussed in each category or not. If your code doesn’t include directionality (e.g., a code about “wait times” doesn’t indicate if the discussion was positive or negative) it can be helpful to note if certain groups discussed the code in different ways. This section can also help you quantify your qualitative data if necessary (see our “How to Quantify Qualitative Data” and “3 Easy Ways to Quantify Your Qualitative Data” articles for more on this!).

Example Codebooks

Here are two example codebooks to help you understand how the excel sheets can be set up and tailored to your specific analysis.

Example 1

In the first example, the Examples column is used for lengthier, more descriptive quotes or examples, while the Quotes column contains the shorter, punchier quotes that I might want to include in a report. The References section highlights which groups discussed the code and specifically denotes which focus groups the code was referenced in.

Example 2

In the second example, the Examples column serves as both the Examples and Quotes column. The References section shows which focus groups the code was discussed in and the directionality of the discussion (positive or negative). In Example 2, the focus groups may have been heterogenous, and included a mix of all participants, hence the sub-columns were unnecessary.

While codebooks are useful tools to support your qualitative analysis, they don’t help you understand how to code. If you are looking for more resources to help you learn about qualitative coding, check out some of our articles:

– Definition of qualitative analysis

– Definition of thematic analysis

Let us know if you have other questions about qualitative analysis that we can help answer!

Jul 31 2023

New infographic: 10 tips for running a focus group

This article is rated as:

Focus groups are a valuable method of qualitative data collection in evaluation, enabling in-depth exploration of participant perspectives, experiences, and opinions. By leveraging group dynamics and interactions, focus groups provide a platform for participants to build on ideas, challenge assumptions, and generate new insights. In this article, we present our top tips for running successful focus groups that are efficient and gather quality data. You can download the infographic here.

Plan and prepare by defining objectives and creating a discussion guide.

Before organizing your focus group, take some time to clearly define the purpose and objectives of the group. Determine the specific information or insights you aim to gather from participants and articulate the evaluation questions or topics you want to explore.

You will also want to develop a detailed plan that outlines the logistics, including the number of sessions, duration, location, and participant recruitment criteria. This will ensure a well-organized and smoothly executed focus group.

You will also want to prepare a discussion guide that serves as a roadmap for the conversation. Discussion guides can be structured, semi-structured, or unstructured. The guide should outline the key topics, questions, and prompts you want to cover during the focus group to keep the discussion focused and on track. Check out our focus group moderation guide to help plan and prepare for your next focus group: https://www.evalacademy.com/resources/focus-group-moderation-guide-template

2. Recruit participants. Pay close attention to potential group dynamics and power imbalances.

When recruiting participants for your focus group, select individuals who align with your target audience or possess relevant experiences and perspectives. It can be beneficial to include a diverse group of participants to obtain varied insights. The focus group can be comprised of individuals from different backgrounds, perspectives, and experiences related to the topic to ensure a richer discussion and minimize bias. One of the main values of a focus group is that participants can respond to what others are saying; they get to hear others’ perspectives and agree or disagree. However, it is important to be aware of potential power imbalances amongst your focus group participants. For instance, when conducting a focus group to explore perceptions of insecurity between managers and staff, including participants from both managerial and staff positions in the same group may create a reluctance among individuals to express their thoughts openly in front of one another.

Select a focus group time that you think will work best for participants and send the invites as early as possible to decrease scheduling conflict and allow participants to prepare for the discussion.

To facilitate meaningful discussions, aim for a group size of 3 to 8 participants. Smaller groups allow for deeper engagement and individual participation, while larger groups can be more challenging to manage and may restrict individual contributions. You can learn more about sampling and recruitment on Eval Academy here: https://www.evalacademy.com/articles/sampling-and-recruitment-101

3. Create a welcoming environment conducive to open discussion and take time to build rapport.

Whether meeting in person or online, set up a comfortable space that promotes open discussion. In a physical setting, arrange seating in a circle or semi-circle to encourage eye contact and engagement. Choose a neutral and private venue, such as a dedicated focus group facility or quiet meeting room. For online meetings, advise participants to find a quiet and comfortable location with minimal distractions, and encourage camera usage (if they are comfortable to do so) to enhance personal connection.

Establish rapport by beginning the session by introducing participants and engaging in icebreaker activities. These activities will create a comfortable atmosphere, help to make group members feel at ease, and foster a positive and collaborative environment.

4. Establish ground rules to encourage honest conversation, including confidentiality measures.

Set clear ground rules at the beginning of the session that emphasize respect, active listening, and open dialogue. Encourage participants to freely express their honest opinions. Assure participants that their responses will be confidential, and their personal information will be protected. Obtain informed consent for their participation and adhere to privacy standards. You can learn more about consent on Eval Academy and access our Focus Group Information Letter and Consent Form to use in your next focus group. By creating a safe and confidential space, participants will feel comfortable engaging in open and honest discussions.

5. Record and document with the participant’s consent.

Consider recording the focus group with participants’ consent to alleviate the need for extensive notetaking and allow you to focus on facilitating the discussion. Ensure all participants are comfortable with the conversation being recorded before proceeding. It can be helpful to ask a colleague to join the focus group as an assigned notetaker, particularly if consent is not given to record the conversation. The notetaker should document key points, participant responses, non-verbal cues, and any noteworthy interactions, including power dynamics. These comprehensive notes will be instrumental for later analysis and reporting purposes.

6. Act as a facilitator as opposed to a conversation leader. Practice active listening.

Foster an open and non-judgmental discussion by asking open-ended questions and encouraging participants to freely share their thoughts and experiences. Frame questions in an open-ended format, avoiding leading or biased prompts that may sway responses. Utilize probing techniques to delve deeper into their responses. Remember, as the facilitator, your role is to guide rather than lead the conversation. Your goal is to steer the discussion, encourage participation, and manage conflicts or dominant voices. Practice active listening, paying attention to both verbal and non-verbal cues. Demonstrate genuine interest and engagement by asking follow-up questions and seeking clarifications.

7. Manage group dynamics.

Take measures to ensure equal participation among participants by effectively managing dominant or shy individuals. Encourage quieter participants to express their opinions while carefully preventing any individual from monopolizing the conversation. Address dominant participants by gently redirecting the conversation and inviting quieter individuals to share their perspectives. This will help to create an environment where all participants have an equal opportunity to contribute their insights. Employ effective moderation techniques to maintain focus and stay within the allotted time. For example, redirect tangential discussions and ensure that all planned topics are adequately covered.

8. Be adaptable and allow for flexibility in conversation.

Maintain adaptability during the focus group session, allowing the conversation to naturally evolve and adjusting the discussion guide as necessary. Embrace the potential for unexpected insights and ideas that may arise during the discussion as long as the discussion stays on topic. Stay open-minded and receptive to new perspectives, allowing for a dynamic and organic conversation.

9. Summarize and validate with participants.

Summarize the key points raised by participants and seek their validation to ensure accuracy and demonstrate respect for their input. This process helps you verify your understanding of their perspectives and ensures that their voices are accurately represented. If participants share highly sensitive and private information during the session you can offer them an opportunity to see the results and reports before they are finalized to ensure anonymity and accuracy. See our article on Member Checking: https://www.evalacademy.com/articles/what-you-need-to-know-about-member-checking. By actively involving participants in the validation process, you foster a collaborative and inclusive environment that values their contributions.

10. Be appreciative and consider remuneration.

Show your gratitude to the participants for their time and valuable contributions at the conclusion of the focus group session. Take a moment to thank each participant individually after the session to acknowledge their valuable insights and perspectives. Consider providing incentives as a gesture of gratitude for their involvement and willingness to share their experiences. This expression of appreciation reinforces the importance of their contribution and fosters a positive relationship for future engagements. You can learn more about incentives for participation in evaluation here: https://www.evalacademy.com/articles/incentives-for-participation

Lastly, it is important to reflect on the focus group process after its completion and gather feedback from participants and anyone else supporting the facilitation process. Use this feedback to improve future focus groups.

Have you facilitated a focus group? What are your top tips for ensuring a successful data collection activity? Comment on this article or connect with us on LinkedIn or Twitter!

Jul 04 2023

New Template: Style Guide Template!

This article is rated as:

Eval Academy just released a new template, “Style Guide Template”

Who’s it for?

Whether you’re new to evaluation or if evaluation is your main role, this Style Guide Template is for anyone looking to create consistency among project documents.

What’s the purpose?

A style guide is a time-saving tool that helps you be consistent in your formatting when creating client and public-facing products, from evaluation plans to reports and presentations. When working with a team, style guides ensure that team members are working efficiently as they create evaluation products.

What’s included?

This Word template provides each of the main elements needed in a style guide (e.g., fonts, heading styles, colours, imagery, and charts) and space for you to enter your own project details. This template also includes a bonus section about writing styles.

Learn more: related articles and links:

You can learn more about designing an evaluation report on Eval Academy through the following links:

You can also find many other templates in our list of resources to support you in designing evaluation reports. Some of our most popular templates include:

What do you think of our new Style Guide template? Let us know in the comments below!

Jul 04 2023

The Art of Writing Evaluation Questions

This article is rated as:

It seems so simple – just ask a question! But many new evaluators or side-of-desk evaluators struggle with confidence in creating good evaluation questions. Here are a few tips to accompany some content we’ve already shared (How To Write Good Evaluation Questions; Evaluation Question Examples by Type of Evaluation; Evaluation Question Examples). Then, I’ll share an example of how I go from a client meeting to drafting evaluation questions.

Structure Tips

Evaluation questions often have similar lead-ins, that is, that starting part of the question. Evaluation questions are intended to elicit a narrative response, not a simple yes/no. Therefore, the question (usually) shouldn’t be “Are clients satisfied?” or “Did the program reach the target?” These can be answered in one word: yes or no. Usually, evaluation questions are open-ended questions that leave room for context, exploration, or explanation.

Try starting your evaluation questions with:

“To what extent….”

This one is a favourite. Starting your question with “To what extent” leaves room for a range of responses. Often it addresses program effectiveness. The end of that question could be an outcomes statement, e.g., To what extent did the program provide equitable access to housing services?

“Why….”

Why questions can help a program to understand the results they are getting. It can explore processes that usually aren’t documented, e.g., Why are clients choosing this program over that program?

“How….”

How questions are excellent for process or formative evaluation. How questions help a program to understand what works in what context and can identify enablers or barriers, e.g., How do clients learn about our services?

“In what ways.…”

In what ways questions can be used when there is a specific feature that you want to explore, e.g., In what ways did self-referral impact program outcomes?

“What..
- ..contribution”
- ..impact”
- ..factors”

What questions can also explore specific features, e.g., What impact did the email campaign have on client access?

Or, what questions can help to identify barriers or enablers, e.g., What factors contribute to client success rates?

What about Who?

In my opinion, questions that start with “Who” are rarely evaluation questions. Having said that, I often include them in my evaluation plan. I do this because it clearly and transparently shows clients that in addition to answering their key evaluation questions, I’ll also provide descriptions or profiles of who is accessing their service. Where ethical and possible, I’ll use the “who” information to further explore answers to the evaluation questions, e.g., How did satisfaction vary by demographics?

Sometimes I’ve seen these referred to as “Descriptive Evaluation Questions”, and I think they’re important.

Client Requests

Despite my argument earlier that yes/no questions are generally not great evaluation questions, sometimes I do include them in my evaluation plan, often as sub-questions under a key evaluation question. I do this to show clients that I intend to answer their burning questions, which may be yes/no.

For example:

A key evaluation question may be:

“To what extent were clients satisfied with the service?”

And then underneath that, I’ll include:

“Did the program reach at least 80% of clients satisfied?”

You’ll see what I mean about a key evaluation question and sub-questions in my example below.

However, I have found that with a little wordsmithing magic many yes/no questions can be made stronger using the lead-ins described above.

For example:

Did the program stay within budget? -> How well did the program align with the budget?

Are clients satisfied with the program? -> To what extent are clients satisfied with the program?

Did changing the intake process impact outcomes? -> In what ways did changing the intake process impact outcomes?

From client meetings to evaluation questions

Now that we have some structure around evaluation questions, let me share a real example. At the start of each new client contract, I hold a kick-off meeting (here are some great resources for that: Evaluation Kick-Off Meeting Agenda (Template); How to Kick Off Your Evaluation Kick-Off Meeting. A primary goal of that meeting is to get me started on drafting evaluation questions.

You can try asking your clients “what are your evaluation questions” directly, but in my experience, you’ll be left with blank stares. Clients (usually) aren’t evaluators. So instead of the direct pathway, I facilitate discussions about what questions they have about their program, what they want to learn, and what they (might) do with those learnings. Evaluation questions will often flow from this understanding.

In this example, I was working with a new client to evaluate outcomes of a mental health program for youth. After a quick orientation to evaluation that we call “The Fastest Evaluation 101 Ever”, where I frame evaluation questions as providing the roadmap for where the evaluation goes, I move into a discussion where I transition from capacity builder to listener.

Before the meeting I figure out what I need to know, and therefore what questions I’ll ask to generate discussion; determining this is tailored to each client and depends on what I already know, which is usually from documents they’ve already shared with me. Here are some guiding questions I use that enable me to hear from the client:

1. Why am I here? What do you want evaluated and why? What are your expectations?

This drives at purpose and scope. Sometimes I link this discussion to content in my evaluation orientation. This is why that orientation (The Fastest Evaluation 101 Ever) is so useful; now they know that evaluations can be used to make judgments, to learn, to grow and expand, or to monitor (to name a few), and they can use that knowledge to describe the current evaluation. As a group, we spend a fair bit of time understanding why they’re evaluating, and why now.

In this youth mental health program example, the notes I took said:

They want to explore their flow of service and better understand effectiveness and quality
They want to know what’s going well, and what gaps and challenges they have
They want to learn and act on those learnings
They want to know about access, quality, and impact
They want to understand the client trajectory or journey

2. What questions do you have about this program? What decisions do you make and what informs those decisions?

This section is pretty clearly about evaluation questions but also looks at gaps/opportunities and how they may act on them. Sometimes you’ll find it’s hard for your client to think of questions per se, so asking what decisions they make day-to-day, or monthly, or quarterly and then following up with “what evidence or data do you use to inform those decisions”, will help move things along.

In this example, the notes I took said:

What gaps do we have in terms of who accesses the service?
Are there gaps by certain populations (e.g., socioeconomic status) or certain regions of the province?
Who are we serving? And who are we not serving?
What are the barriers to access?

3. How do you define success for this program?

This question gets at outcomes and program effectiveness, which are usually closely related to evaluation questions. This also starts to give you an idea of how you may measure (and answer) those evaluation questions.

In this example, the notes I took said:

Self-assessment scores improve pre to post
Evidence of fidelity to the model
Families report success
Improved mental health
Demand for services is high

If a client is particularly interested in outcomes, I’ll ask: What does your program make better? What changes do you expect to see? I didn’t need to do that in this example, though.

From those notes, I drafted only three key evaluation questions. I really liked how one of the staff talked about three domains: Access, Quality, and Impact, so I mirrored that thinking back to them in the development of the questions.

1. To what extent are we serving the families who need our services? (ACCESS)

Under this key question, I had 9 more specific questions. Here are some highlights:

a. Who was referred to our program? (Notice the inclusion of a descriptive evaluation question!)

b. How convenient were services to access?

c. To what extent do program participants use what they learn in other scenarios?

2. What is the experience of participating in the program? (QUALITY)

Under this question, I had 5 more specific questions. Here are some highlights:

a. To what extent do families report a positive experience with the program?

b. What supports enable program implementation?

3. What impact does the program have on participating families? (OUTCOMES)

Under this, I had 6 more specific questions. Here are some highlights:

a. What impact does this program have on the mental health of youth?

b. What impact does this program have on families?

c. To what extent are outcomes sustained?

In this case, categorizing the evaluation questions added a lot of clarity, and actually helped me to structure the final report as well.

Sometimes, clients will say they’re interested in a logic model, or theory of change, or, as mentioned above, in developing new outcome statements. In those cases, the development of those products will inform and guide your evaluation questions.

Evaluation questions can also be tied to a framework. I’ve used RE-AIM or Proctor and built evaluation questions mapped onto specific domains (which is pretty similar to the categorization of evaluation questions in the example shared above).

Sometimes the clients have shared enough documentation with me before the kick-off meeting that I could show them some example evaluation questions and assess their reaction/reception. I really like this approach and use it whenever I can. I find that oftentimes reacting to something is easier and more efficient than starting from scratch – you’ll know right away if you’re on- or off-track.

I hope this helps you in your process to create evaluation questions. Use our evaluation question checklist for some additional considerations.

Jul 04 2023

SWOT Analysis: What is it and how do I use it?

This article is rated as:

Have you ever participated in a strategic planning session or helped develop or start a new program? A SWOT is a tool commonly used at the beginning of a new venture or as part of a new strategic planning process. The real value in a SWOT analysis is in the gathering of perspectives across and outside of the organization to better reveal potential obstacles and find the less obvious solutions. It is frequently used because it is simple to understand. A SWOT in evaluation could be used as a data collection tool or to present evaluation findings and create recommendations.

What is a SWOT analysis?

SWOT stands for Strengths, Weaknesses, Opportunities, and Threats. SWOT started as a business and marketing tool that is now commonly used in strategic planning sessions. It is often used at the organizational level but can be used to assess a program or project. A SWOT is normally depicted as a 2 x 2 grid or matrix, like the one below.

Strengths and weaknesses examine what is going on internally. These are the areas that the organization can control and change. Strengths look at what the project excels at and is good at doing. Weaknesses are of course the other side to that coin; weaknesses look at what the project is not good at. These could be areas to improve. Evaluation findings can uncover unexpected strengths and weaknesses that those who are internal to the project just might not be able to see.

Opportunities and threats are external facing. Opportunities look at the current environment that the organization exists in and where it could grow to fill service gaps in the community or ways the organization can expand. In evaluation findings, opportunities might include external factors that could support identified program improvements or external context that has led to positive program findings. Threats look at the current environment to see areas where the growth and sustainability of the organization might experience barriers to change or be negatively impacted by external context.

How to do a SWOT analysis

I have most often taken part in a SWOT analysis in a workshop format, where someone facilitates the conversation, gathering feedback and insight from people with a vested interest in the project. However, a SWOT could also be a useful way to design interview questions that look at internal and external factors impacting a project or to structure qualitative analysis from interview findings.

When gathering people to complete a SWOT analysis, consider who you are inviting. Is this an exercise with just the evaluation advisory committee or is this an opportunity to expand beyond that group to gain outside perspectives? Should leadership, frontline staff, or other groups with vested interests be included?

When doing an in-person SWOT analysis, I like the use of flipcharts. I would normally have Strengths, Weaknesses, Opportunities, and Threats placed around the room. If doing a virtual SWOT analysis virtually, some tools that could be useful include Canva charts, Microsoft OneNote, or maybe PowerPoint with a slide for each factor. Use the tools that you already have access to, in order to gather the insights of your group.

There are a few things to keep in mind when facilitating a SWOT analysis. This is meant to be participatory, and everyone in the group is encouraged to give their insights. This can be done through a full group discussion of each of the factors, or depending on group literacy, sticky notes could be given to each participant to write their own ideas onto and stick onto the sheet with the corresponding factor.

When planning for your group, create guiding questions that are relevant to your project and use them to help stimulate conversation. Some questions to help facilitate the discussion:

What types of activities does our organization do best?
Where is our biggest success?
What type of skills do we have?
Where are we most effective?
Who are our strongest allies?
What processes do we need to improve?
What assets do we need to build or get?
What gaps do we have in skills?
How do clients perceive our organization?
How is technology changing how we deliver our activities?
Are there any funding changes that may affect our ability to continue running the organization’s activities?
What are current donor trends and how might those affect the organization?

These are just a few ideas of questions that can help to get some ideas flowing, but in no way is this an exhaustive list.

Once the group has completed listing their SWOT analysis, it is time to set priorities. Each group may do this a bit differently, maybe each member individually ranks their priorities. Again, this could be done with sticky notes with numbers one to five and each group member places these numbers beside their priority areas. The group would then discuss and come to a consensus to define which items are priorities. It is possible to skip the individual prioritization, but I find that quieter group members tend to not speak up in the prioritization within out having done the individual priorities first. These priorities would go into the completed SWOT matrix and are commonly listed in order of priority.

The following is an example of what a SWOT might look like for a small soup kitchen.

After priorities are set, an action plan or actionable recommendations are created to follow up and pursue these priorities. The SWOT matrix itself is not the goal but a tool to help set priorities. A SWOT is a simple tool to help set priorities for a project, program, or organization. A SWOT can help foster group ownership through involvement in SWOT analysis gathering. It can also be a great way to share information as it is easy to understand.

Creating a Qualitative Codebook

What is a codebook?