evalacademy

This article is rated as:

---

In every research and evaluation project, it is important to identify and address sources of error that may impact the accuracy of your findings and the relevance of your recommendations.

Two common sources of error in evaluation are bias and confounding.

Here, we will look at what bias and confounding are (and are not), the differences between them, and important considerations to take to prepare for and address both in your next evaluation project.

---

What is bias?

In research and evaluation, bias refers to systematic error in the way data are collected, analyzed, or presented resulting in incorrect interpretation of the findings.

There are many types of bias that can occur in evaluation which fall into two broad categories: selection and information biases.

Selection bias

Selection bias refers to systematic differences between those who engage in a program and those who do not, or when only certain groups are given the opportunity to give their opinion. Specific types of selection bias include:

• Sampling bias results when groups of individuals are over- or under-represented during the data collection process.

• Allocation bias refers to scenarios when researchers or evaluators don’t appropriately randomize participants to experimental and control study groups. 

• Attrition bias refers to systematic differences between participants who stay in a program and those who leave the program.

Information bias

On the other hand, information bias refers to systematic differences in the way data are collected from participants. For instance, if an evaluator sits in on a particular session to assess how an intervention is being delivered, facilitators and participants may alter their behaviour to ‘impress’ the evaluator without even realizing they are doing it. This is an example of one type of information bias called Observer Bias. Other types of information bias include:

• Interviewer bias which occurs when an interviewer has preconceived ideas about the person they are interviewing which clouds or distorts their perception of the interviewee’s responses.

• Recall bias occurs when participants do not accurately remember an experience and leave out or alter details when reporting about it. 

• Non-Response bias is the skewing of results due to differences between respondents who answer specific questions and those who skip questions. 

• Social Desirability bias refers to the tendency of participants to answer questions in a way that makes them or their actions seem more appropriate or desirable to the person asking the questions.

---

What is Confounding? 

Confounding refers to situations where there is a real relationship between a program or intervention and the outcome you are measuring, but it is affected by the presence of another factor called the confounder.

A straightforward way to think about this is to think of the synonym of confound, which is ‘confuse’. In other words, confounding variables confuse your findings.

Example 1
A classic and simple example of confounding is the relationship between increased ice cream sales and decreased rates of the common cold.

While it might appear that ice cream is protective against contracting the common cold, what really explains this association is the confounding variable of the weather.

Warmer weather is related to higher ice cream sales and lower common cold rates even though ice cream sales and common colds aren’t directly related to one another.

In this case, the weather confounds the association between ice cream sales and rates of the common cold. 

Example 2
To understand confounding in the context of program evaluation, consider evaluating the impact of a new light activity program to increase mobility in older adults.

Although it is possible that the program is successful in improving mobility on its own, it is also possible that adults who live within walking distance of a favourite café may be more motivated to attend the program so they can maintain their mobility and continue walking to the café.

If their regular walks to the café (which occur outside of the structured activity sessions) also play a role in improving their mobility, the walkability of their neighbourhood may be confounding the association between the activity program and improvements in mobility. 

When confounding isn’t addressed, there are four primary ways that it can impact our results. The presence of a confounding variable may result in:

• A spurious association: An apparent association despite no real association.

• A hidden association: An apparent absence of association despite a real association existing.

• Positive confounding: Enhancing (or overestimating) a true association.

• Negative confounding: Masking (or underestimating) of a true association. 

---

What is the difference between bias and confounding?

In short, bias refers to systematic error in how we measure or report data, while confounding refers to real but misleading associations.

The ability to distinguish between biasing and confounding factors can be helpful in evaluating the true impact of a program or public health initiative on the desired outcome.

While neighbourhood walkability is a possible confounding variable in the example above, if only adults who live in walkable neighbourhoods were included in the light activity program, then this would be an example of sampling bias rather than confounding. What is important to note here is that you can often account for confounding, but not bias, in the way you analyze and report your findings. 

For instance, to account for how confounding may impact your results, you could compare the improvements in mobility between the ‘walkable neighbourhood’ and ‘non-walkable neighbourhood’ participants. This would allow you to estimate how much of an effect the activity program had compared to the impact of neighbourhood walkability on mobility. However, if your sample were biased to include only participants from walkable neighbourhoods, you would not be able to examine how living in a walkable area may affect mobility. In this case, you would need to report this bias and how it potentially skewed your results.

---

How can I predict and avoid bias and confounding in my next research or evaluation project?

I have found that the key step is to continually remind yourself and your team that the issue you are investigating is almost always part of a larger, more complex system than what your project can assess.

Throughout a project’s lifespan, there are practices that can help you and your team predict, identify, and overcome sources of bias and confounding to unveil accurate and actionable results. 

During project development and outcome operationalization

In these phases, ensure you are specific and clear in your definitions of variables and aims. For instance, if you aim to measure how a certain intervention impacts participants’ health, be clear about what health means for your evaluation.

It can also be useful to collect information about an outcome of interest in a few different ways (for example, through surveys and participant interviews) to reduce the chance that you are missing areas of bias or confounding.

For an evaluation within the healthcare field, some questions to consider when developing your measures could include:

• What type of health outcome are you assessing? Are you looking at overall wellbeing, or something more specific such as blood glucose levels or scores on an anxiety screening test?

• How will you measure the type of health you are interested in assessing? The way that we collect information about someone’s health can greatly impact the results of an analysis. Using objective measures that have been validated through repeated trials will increase the accuracy of your evaluation and allow you to compare the results to other similar programs.

• Are there meaningful thresholds that indicate distinct levels of this type of health? Depending on the type of health outcome you are assessing and the measures you are using to assess it, small improvements in measures could be meaningful or arbitrary. It is best to select a threshold that is supported by existing literature before you begin your project to avoid biasing your results.

• What are some possible unintended findings of this evaluation? It is always a good idea to discuss possible findings with your team and stakeholders. Sometimes, even after thorough discussion, we find that our results tell us something completely unexpected! Be open to these findings and plan for how you will present them in your report to stakeholders or clients.

During data analysis

Although more common in research, depending on the size and scope of your evaluation you may be able to ‘control’ for confounding variables through statistical modelling. ‘Controlling’ a variable means holding it constant while assessing the changes in the variable(s) of interest. The method you use to control your confounding variable(s) will depend on your analytic method, but in the right settings, it can allow you to look specifically at the independent impact of the program or intervention on your outcome of interest.

After analysis

It is also important to not just take the data at face value but to ask why an association may exist. Consider what other factors, including ones that you did not or could not measure, may have played a role in the observed outcome. For instance, was one group more likely to meaningfully engage in an intervention compared to another group? Is the sample who is accessing your program more likely to face hardships not addressed by your program that may impede their progress toward the target outcome? Talking through preliminary results with your team, stakeholders, and/or clients may help to explore hidden confounding or uncover biases that would otherwise go unnoticed.  

When reporting your findings

At this stage, it is always important to note the limitations of what you can conclude from the data collected. Often single sources of information are not enough to give us answers that span across diverse groups and populations, and that is okay! By reporting information about the demographic groups of participants, the program methods, and how outcomes were measured, you can more accurately draw conclusions from your findings without overgeneralizing the results.

---

Associations in programs and studies involving humans are often complex and involve more factors than we can assess in any single research or evaluation project.

Understanding how to identify sources of bias and confounding can help you and your team to draw more well-informed conclusions from your analyses and provide realistic and actionable recommendations for future projects. 

This article is rated as:

---

Eval Academy just released a new Tip Sheet, “Tips for Conducting Interviews”

Who’s it for?

If you will be conducting interviews in your evaluation, then this tip sheet is for you! Whether you’re new to the process or have conducted interviews before, this tip sheet provides a good overview and refresher to make your next interview experience a great one.

What’s the purpose?

This tip sheet highlights our 5 top tips to ensure:

1. Your interviews collect rich data.

2. Your interviewees have a good experience.

It also includes links to other Eval Academy resources to guarantee informed consent within your interviews.

What’s included?

A printable 1-page tip sheet that defines our 5 top tips.

Get the tip sheet

---

---

Learn more: related articles and links

You can learn more about collecting qualitative data on Eval Academy through the following links:

• How to conduct interviews

• How to use Calendly to schedule interviews like a pro

Some helpful Eval Academy resources to collect and track your qualitative data include:

• Interview Tracking Log Template

• Excel Interview Tracking Log Template

• Standard Interview Guide Template

• Standard Interview Information Letter Template

• Standard Interview Consent Form Template

• Standard Interview Templates Bundle

---

What do you think of our new tip sheet? Let us know in the comments below!

This article is rated as:

---

Eval Academy just released a new infographic, “Qualitative Data Saturation”

Who’s it for?

This infographic is for those who collect or who will be collecting qualitative data and are looking to support the validity of their results. It’s a helpful resource for those who are both new and experienced in evaluation!

What’s the purpose?

This infographic defines qualitative data saturation, lists why it is important and identifies how you know when you’ve reached saturation. It also includes links to further resources for those who are looking to dive a little deeper into the theoretical concept of qualitative data saturation.

What’s included?

A printable 1-page infographic that outlines qualitative data saturation.

Get the infographic

---

---

Learn more: related articles and links

You can learn more about qualitative data on Eval Academy through the following links:

• How to conduct interviews

• How to “Quantify” Qualitative Data

• How to Transcribe Interviews Like a Pro

• Interpreting themes from qualitative data: thematic analysis

• How to use Calendly to schedule interviews like a pro

• 3 Easy Ways to Quantify your Qualitative Data

• Sampling and Recruitment 101

Some helpful Eval Academy resources to collect and track your qualitative data include:

• Interview Tracking Log Template

• Excel Interview Tracking Log Template

• Standard Interview Guide Template

• Standard Interview Information Letter Template

• Standard Interview Consent Form Template

• Standard Interview Templates Bundle

• Focus Group Information Letter and Consent Form Template

• Focus Group Moderation Guide Template

---

What do you think of our new infographic? Let us know in the comments below!

This article is rated as:

---

Member checking is a technique often used with qualitative methods to help validate findings. It can be used in evaluation to help validate, interpret, and analyze findings from interviews, focus groups, and other forms of qualitative data.

While commonly used in qualitative research, especially as it is included in qualitative research checklists, like COREQ, it’s less commonly used in evaluation and we think that should change!

In this article, we’ll review what member checking is and why, when, and how you should use it.

---

What is member checking?

In essence, member checking is when participants in qualitative research validate their data, checking for accuracy and validity. Member checking can happen informally during data collection when the researcher or evaluator summarizes and confirms their interpretation of what a participant said during data collection. This can look like an interviewer summarizing a participant’s statement mid-interview, allowing the participant to affirm or correct the statement. My personal favourite lead into this is, “So what I’m hearing you say is…”. 

This article will focus instead on formal member checking, whereby the researcher or evaluator reaches back out to participants after data collection to check their findings. This method is attributed to Lincoln and Guba, who argued its purpose is to validate, verify, and assess the trustworthiness of results. I’d argue that it can also be used to build trust with your participants, correct assumptions you have made, and lead to new insights and deeper understandings of the data.

Formal member checking can encompass a range of activities and can occur at different points during data collection and analysis.

• Immediately after data collection: participants can review their transcripts and are invited to check the accuracy of the data. Participants can be given the opportunity to add, remove, or clarify their statements.

• During data analysis: participants can be invited to review preliminary themes and how their quotes fit into these themes, leaving room for further discussion and (re)interpretation of data. Participants can also be involved in the theming and sense-making process.

• After data analysis: participants can review draft reports with their quotes and contributions highlighted for their review. Often at this stage, participants are limited in their ability to change their quotes or provide additional context, but it gives them a clear idea on how their data is presented and used and provides an opportunity for participants to review if the right emphasis was placed on topics and themes.

---

When should you use member checking?

Now you may be thinking, ‘sounds great! How do I know when it’s a good idea to use member checking?’

Consider using member checking when:

• You need to build trust with your participants. giving them the chance to edit or clarify what they said can help participants trust you, the evaluator. This can also be a valuable step if you know you’ll need to return to this same group for future data collection.

• Your sample is small, and it might be easy to know who said what. Despite your best efforts to maintain confidentiality, it can be difficult if your sample is small. Giving participants the opportunity to redact statements, especially how they will appear in context, can help to preserve their confidentiality, or ensure that they are aware of what will be shared and how.

• When the topic or content is sensitive. Member checking can support participant confidentiality, allowing them to review how you’ve used their information.

• When you aren’t familiar with the context. Member checking can aid in interpretation and analysis and provide additional context.

• You need enhanced validity. Member checking can add rigour and validity. Consider using it when other validation techniques, such as triangulation, are not possible. Using member checking can provide evidence that your interpretation and analysis are appropriate, accurate, and reflect the content of the discussion.

• You are conducting a participatory evaluation. Member checking can be a participatory technique to include participants in data collection, analysis, and reporting. Member checking can support community buy-in for the evaluation and its results.

Avoid using member checking when:

• You have a short timeline or small budget. Member checking takes extra time and therefore more money. Make sure you have the time and budget to do it justice.

• Your participants are short on time or doing so will add undue burden. Member checking adds an additional burden to participants. For those that are busy or do not have a lot of capacity, member checking is another activity you are adding to their plate.

• Participants will struggle with the concept of themes. For those of us in the research and evaluation world, the concept of summarizing information into themes is easy to comprehend. Avoid doing member checking using themed or summarized information with groups who may not understand how and why themes are used.

• When your participants may have low (English) literacy levels. For those who are not comfortable or able to read well in English, member checking by providing a written summary or transcript is likely not a viable option. Feel free to get creative, though, if member checking is important for this group. Short videos or audio clips may be a way to get in touch with this audience.

• You are not able to receive input or incorporate feedback. Member checking rests on the principle that participants can modify the transcripts or analysis. If you are unable to make changes, member checking is not an appropriate tool for that project.

• When you won’t be able to do member checking close to the interview. Sometimes it takes a long time to gather and analyze data. Member checking loses its benefits when it is done a long time after data collection has occurred as participants may not recall the purpose or context as clearly.

• When there are power imbalances between the evaluator and participant. In cases where the participant may not feel comfortable contradicting the evaluator or the findings, member checking may not provide the value or validity you are hoping for. In some cases, you may be able to use other people or methods to even out the power imbalance.

• If reviewing their input may pose a risk to participants. In some cases, asking participants to review their contributions can be distressing, especially if the data are gathered about a sensitive topic. Additionally, if participants don’t see their contributions reflected in a summary or the themes, it may leave them feeling isolated or unheard.

---

What you need to consider when member checking

I’ve described some of the what, why, and when, but the real question remains, how do you actually do it? There’s a range of options for how to member check. You need to decide on a few key things: what to send them, how you want them to interact with what you’ve sent, what you will do with their contributions, and how much time they have.

Let’s start with what to send them. You can send participants their data in a range of forms, from their raw data (e.g., their transcripts), to a summary of their contribution, or their quotes in themes or in the context of the report. 

Next up, is how you will connect with them. Individual member checking can be done through a 1:1 conversation using a set of interview questions, an email asking them to reply with comments, or a survey asking for feedback on themes. Member checking can also be done in a group setting, especially when the data were gathered in a group (e.g., focus groups). You can host a group focus group or discussion, or even structure your session like a data party.

Note: while member checking can be done in a data party format, not all data parties are considered member checking. Member checking is specifically done with those who provided data. Data parties can include those who were not involved in data collection to support interpretation.

Next, you need to be clear with what you expect them to do with the information. Are they able to ask for information to be removed or to edit a quote? Can they offer additional context? What if they disagree with the theme or title?

Next, you need a plan for what you are going to do with the feedback. This should be determined in advance. Will you make the edits and re-send them for approval? What will you do with conflicting information? What happens if they don’t respond? Who has the final say about the findings and interpretation?

Finally, be clear on timelines. You should give participants a clear indication of when they should expect to receive the information you are asking them to check and how long they will have to provide feedback. The last thing you want is for someone to be away or not prepared to set aside time to review what you are provided.

---

In the comments, let us know about a time you’ve used member checking. How did it go?

This article is rated as:

---

Eval Academy just released a new template, “Canva design templates for creating your own Logic Model”

Who’s it for?

Whether you’re new to evaluation or if evaluation is your main role, these Canva design templates are for anyone who wants to design a Logic Model to be more visually appealing. To use these templates, you will need to set-up or log in to your own account in Canva.

What’s the purpose?

These Canva design templates can be used to present your own Logic Model for any type of intervention in a more distinct format. They’re great for sharing your Logic Model with a wider audience, such as when you want to share your model on your website.

What’s included?

Two customizable Canva templates that provides you with the space you need to input your Logic Model components.

Get the templates

---

---

Learn more: related articles and links

You can learn more about Logic Models on Eval Academy through the following links:

• The definition of Logic Models

• Differences between Theory of Change, Log Frames, Results Frameworks and Logic Models – what are they and when to use them

• Improve Your Logic Model Using 3 Simple Design Principles

• Developing a Logic Model Template 

You can also find many other templates in our list of resources to support you in planning and implementing an evaluation. Some of our most popular templates include:

• Theory of Change Template

• Evaluation Plan Template

• Evaluation Kick-Off Meeting Template

---

What do you think of our new design templates? Let us know in the comments below!