cplysy

This article is rated as:

Bias in evaluation is inevitable. Reflection helps us to identify our bias and when we do, it is necessary to identify sources of bias in our processes, eliminate which bias we can, and acknowledge which bias we cannot.

For simplicity, I will focus on the data collection process and the associated sampling bias: ‘Bias that results from certain outcomes or responses being favoured over others.’ Sampling bias results from groups of individuals being over- or under-represented during the data collection process. This results in weak, or incorrect, conclusions being drawn from the data. Incomplete data lead to incomplete results.

Identifying sampling biases

Sampling bias encompasses any biases that originate during the data collection process. Here we’ll focus on a few common sources of bias in data collection.

1. Undercoverage

Commonly specific groups of individuals are underrepresented in our data; this is undercoverage. Often minority or marginalized groups are absent from collected data or these groups are poorly represented so that their responses are diluted. This leads to poor conclusions about specific populations, as groups underrepresented may provide insights not captured by the majority.

Key points:

• A segment of the population of interest in not represented (or underrepresented)

• People captured in the data differ systematically from those not captured (e.g., demographic, social, or cultural differences)

Example: You want to understand how newcomers to Canada navigate the Canadian healthcare system by conducting English-language interviews. However, this will result in undercoverage as newcomers to Canada may be less than confident in their English fluency or may not speak English at all. The resulting data will skew towards English-speaking newcomers only and will not properly represent all newcomers’ thoughts and opinions on navigating the Canadian healthcare system. Conclusions drawn from these results may not generalize to or represent the underrepresented population, thereby reinforcing barriers and continuing to marginalize these groups.

2. Non-response bias

Inevitably some individuals will not respond to a survey or will back out of a scheduled interview last minute. Like undercoverage, non-response bias may result in some groups being underrepresented, but non-respondents opt not to respond, whereas in undercoverage, specific groups are missed and not provided the opportunity to respond. Non-respondents may simply have forgotten to respond or may refuse to respond. If the non-respondents differ from the respondents in some meaningful way, the subsequent results will be inaccurate. 

Key points:

• Some people may forget to respond or be unwilling to respond to a survey (or participate in an interview)

• Survey fatigue may result in increasing non-responses for latter survey questions

• Technology is convenient but has limitations (e.g., some people do not have access to an email account or, if they do, emails are too often redirected to the spam folder)

Example: You are evaluating patient satisfaction using a survey following visits with a family medical doctor. Surveys are emailed to patients, and they are asked to complete the survey at their convenience. Guaranteed, there will be several non-responses as surveys are lost (email addresses are incorrect; email sent to spam), or patients forget to respond. Non-respondents may represent a specific group, such as single parents with limited time to complete the survey or elderly people that are less likely to access email. And again, conclusions drawn from these results would not necessarily generalize to the population of interest as non-respondents may differ in some meaningful way relative to respondents.

3. Voluntary response bias

This is where specific individuals volunteer to participate in a survey or interview. These individuals usually have a strong interest in the survey/ interview topic, either positive or negative. This results in a biased data sample that is crucially missing people of middling opinions.

Key points:

• Participants self-select to complete a survey or interview

• Strong opinions, positive or negative, are disproportionately represented in the data

Example: You want to receive feedback on ways to improve your organization by providing an anonymous survey link to all employees in your organization. Most responses will be from people that are completely satisfied with the organization or people that are completely dissatisfied with the organization. Responses will skew to the extreme and valuable information in the middle will be lost, which could result in changes to the program or organization that may be unwarranted if most staff feel neutral.

4. Survivorship bias

Survivorship bias results when we consider only surviving (i.e., existing) observations and fail to consider all other potential observations. This may result from focusing in on only the successes of an evaluation but ignoring the failures or roadblocks that built the foundation for said success. That is, we may be prone to focus on people that start and finish a program, but neglect information from those that did not start or complete a program. Results would therefore reflect the thoughts and opinions of those completing a program, and potentially miss critical information on why people did not complete a program. 

Key points:

• Reliance on existing data, “the survivors”, at the expense of other data (e.g., historical data, attrition data, etc.)

• Generally, overemphasizes positive outcomes over negative or null outcomes

Example: You could administer a survey to participants of a physical fitness and healthy eating program to gauge their engagement and satisfaction with the program. You may find that these participants are engaged and satisfied with the program, but the data fails to capture the participants that started and did not complete the program. There may be underlying reasons as to why they did not complete the program, which could provide insight into how to improve the program to reach more people in the future.

5. Recall bias

Our recollection of past experiences diminishes over time. This results in inaccuracies when recalling past experiences. The longer the time between an experience and when an individual is asked about said experience, the more likely those recollections will be inaccurate, influenced by current events and experiences.

Key points:

• Everyone forgets details of past experiences over time

• Longer durations between experiences and recollection, the greater the likelihood to misremember specific details

Example: You wrap with a mental health initiative for seniors with mild depression and conduct follow-up interviews with the participants. Immediately after the program conclusion, participants provide more detailed accounts of their experiences. However, six-months down the road, you interview these same participants. They may have vague impressions of the program but are no longer able to provide concrete details of their experience of the program.

Avoiding sampling biases

Sampling biases will always be present during the data collection process and, oftentimes, multiple biases will be present. The goal is to eliminate which bias we can and reduce the rest. Here are a few steps to reduce bias in your next evaluation:

• If in doubt, try to sample more people than you think you need. For minimum estimates, consider using a sample size calculator.

• Use random sampling protocols. Random samples improve the likelihood of capturing a sample representative of your population of interest.

  • You may also consider maximum variation sampling to avoid sampling the same homogenous groups and better capture underrepresented or marginalized groups.

• Keep your surveys and interviews concise and accessible. People are more likely to participate if the barrier for participation is low.

• Follow up with non-responders to increase the robustness of your data. If you are still having trouble getting responses, consider providing incentives for non-respondents.

This article is rated as:

New Template: Developing a Logic Model

Eval Academy has just released a new template, “Developing a Logic Model”.

• Who’s it for?

  • Whether you’re new to evaluation or if evaluation is your main role, this template is for anyone who wants to develop a Logic Model.

• What’s the purpose?

  • This template can be used to develop and create your own Logic Model for any type of intervention.

• What’s included?

  • The template offers an overview of what a Logic Model is and provides a step-by-step process using facilitating questions for developing your own.

  • It also includes a blank, colour-coded template that can be personalized and modified to your needs.

Learn more: related articles and links

You can learn more about Logic Models on Eval Academy through the following links:

• The definition of Logic Models

• Differences between Theory of Change, Log Frames, Results Frameworks and Logic Models – what are they and when to use them

• Improve Your Logic Model Using 3 Simple Design Principles

You can also find many other templates in our list of resources to support you in planning and implementing an evaluation. Some of our most popular templates include:

• Theory of Change Template

• Evaluation Plan Template

• Evaluation Kick-Off Meeting Template

Have you used any of our templates? Let us know what you think, and which one is your favourite!

This article is rated as:

As an evaluator, I’m passionate about making sure evaluation results actually get used. Too many times have I seen evaluations take place as just a tick-box activity or to meet a funding requirement.

Whether it’s by building capacity through the evaluation process or by developing realistic, actionable recommendations, evaluation plays an important part in making a difference on both a small and a grand scale! We see evaluation changing the world by providing direction for program or organizational improvement (e.g., by showing success and failure), demonstrating value and need, and by uncovering credible, reliable knowledge of how change happens to support learning and decision-making.

Through our sister company, Three Hive Consulting, we support healthcare and non-profit organizations to achieve their mission by uncovering insights that drive impact through our three-staged evaluation approach. 

1. Understand: we collaboratively define who needs to know what, when, and how we will collect, analyze, and report on the necessary information

2. Uncover: guided by the evaluation plan developed in step 1, we collect data to provide answers to the evaluation questions

3. Utilize: we put the information to work by presenting the data in ways most useful and appropriate for our clients and their audiences

Through these steps, here are just a few ways evaluation has helped change the world.

Human trafficking:

Human trafficking is a serious and complex crime that exploits people of all ages and genders. Data are difficult to report in human trafficking because it is a hidden crime –victims often don’t realize they are being trafficked and/or are afraid to report their traffickers. Despite this, human trafficking is one of the fastest-growing crimes in Canada, and reports are rising in Alberta.

In 2021, Three Hive began an ongoing evaluation to understand the impact of Alberta’s regionalized approach to combat human trafficking and provide support to victims. By comparing the two different models used in Northern and Southern Alberta, the evaluation helped to uncover enablers and barriers, and reviewed the role of the Safety Network Coordinator (SNC) in responding appropriately and promptly to the needs of victims. The evaluation included interviews with team members, pulse surveys with advisory circles and the core project team, and a document review.

The evaluation found that after a year of the SNC being operationalized, 90 victims of human trafficking have been supported. The evaluation work was able to highlight the success of this role while describing critical success factors and areas for improvement. Overall, ensuring that the SNC role continues is an important goal for Northern and Southern Alberta teams. Both teams identified several opportunities to improve supports provided to victims of human trafficking, including mental health supports that specialize in trauma, housing availability with wrap-around services, and making further inroads into rural Alberta.

On October 2, 2022, it was announced that the Alberta government is committing $20.8 million in new funds over the next four years to implement the Alberta Human Trafficking Task Force’s five recommendations to step up the fight against human trafficking. 

The funding will:

• Establish and implement an Office to Combat Trafficking in Persons and a centre of excellence for research and data collection.

• Support a new grant for coordinated community support and Indigenous-led and culturally appropriate services.

• Fund civilian positions through the Alberta Law Enforcement Response Teams whose roles would be focused on supporting victims and survivors throughout the investigation process.

• Collaboratively implement other related task force recommendations.

The third point about a “civilian position” indicates that the SNC role will be sustained.

You can read more about this story here!

Communities United

Communities United (CU) was a collaborative, community development initiative in Northeast Edmonton with an overt poverty reduction message. It was a multiyear initiative with a broad mandate, pulling together community stakeholders to collaboratively support and build initiatives and make the most of each partner’s strengths and resources. 

In 2019-2020, CU worked with Three Hive on an evaluation report. As a result of a recommendation from a Three Hive evaluation, CU created an impact and lessons learned report, as well as a Theory of Change. These outputs are an attempt to mobilize knowledge and share what was achieved, highlight what was learned, and inspire others to utilize some of the strategies and approaches listed to contribute to the reduction of poverty. 

You can read more about this here!

EndPovertyEdmonton

EndPovertyEdmonton (EPE) is a broker organization that brings together existing charities, social services, the private sector, and government agencies to identify how different groups can work together to tackle broader social problems. In 2016, Edmonton City Council approved funding for the launch of EndPovertyEdmonton to steward the vision of eliminating poverty in a generation. To achieve this vision, EPE adopted a modified Collective Impact approach to address the complex challenge of ending poverty and guide how EPE partners work together.  

Following on from a previous evaluation by the City of Edmonton in 2018, Three Hive completed an evaluation in 2022 to measure the advancement of EPE on the Collective Impact conditions, as well as equity. The evaluation included a Group member online survey, interviews, focus group discussions, and document review. The evaluation found that EPE has achieved many successes in the past three years and has made progress in some aspects of Collective Impact efforts. However, some issues have persisted such as employees’ lack of understanding of EPE’s overall goals and how their work fits in, as well as concerns around governance and leadership. EPE’s long-term systemic goal of ending poverty in a generation is also difficult to measure.

A story about EPE appeared in the online edition of the Edmonton Journal on September 21, 2022, regarding a review of EPE’s funding by the Edmonton City Council. Council increased funding for the agency in the fall of 2021 but withheld $600,000 pending a review set for the end of September 2022. The Edmonton Journal article discusses the 2018 evaluation, as well as the Three Hive evaluation, and how evidence from these evaluations will help to guide the discussion around EPE’s strengths and limitations, and fundamentally help to determine whether withheld funds will be released. 

You can read more about this story here!

Do you have an example of how evaluation has helped to change the world? Let us know in the comments!