cplysy

I’ve been thinking recently about bias in evaluation methodology. There’s so many different types of bias it’s hard to keep them all straight, never mind mitigating them all!

One particular area has me interested: 

What bias does offering an incentive to participate introduce?

So, let’s broaden the question and explore incentives for participation generally (don’t worry, we’ll cover bias too).

As evaluators, one of our primary activities is data collection. Some of the most common methods include asking other people to share their thoughts or experiences in surveys, interviews, or focus groups. Sometimes it can be difficult to get people to agree to participate. It’s very enticing to offer an incentive to boost that response rate or sample size. 

---

Why offer an incentive?

Many argue, myself included, that those with lived experience are experts and knowledge keepers, and should be compensated for sharing their expertise. So, incentives can serve a moral purpose, offering compensation in exchange for expertise.

Incentives can also serve the function of attempting to mitigate non-response bias. Theoretically, by increasing response rates, your data will be that much more robust and representative of the population you are hoping to report on.

---

Is cash king?

The two most common forms of incentives are cash or gift cards. So, which should you use, when, and why? 

When I was a graduate student, I ran a study that collected data from individuals experiencing challenges with substance use. The Research Ethics Board (REB) at the university would not let me offer cash as an incentive to participate. It never sat well with me. Ethically, who am I, as a researcher or evaluator, to decide whether a participant “deserves” cash or not? Who am I to declare that they can only spend earnings at gift card locations?  Besides, there’s nothing stopping anyone from re-selling gift cards for cash. It seemed like a misstep to me.

More recently I offered a cash incentive to participants for an interview. Many of those participants voluntarily expressed appreciation for the cash support. That project felt better for me.

So, morally, I am a believer that cash is appropriate. Functionally, research says that money is more motivating than other gifts. So, yes, cash may actually be king.

---

Low budget options

What happens if you don’t have the budget for cash or gift cards? There are still ways to offer incentives.  Alternatives could be:

• Access to products or services (e.g., 1 month of free access to the services of the organization)

• Company swag or merchandise 

• Discounts or coupons (e.g., a 50% discount on your next program registration)

Or perhaps you don’t have the budget to offer everyone an incentive, but you can afford something. In these cases, I’ve used the “chance to win” approach. This is more common in surveys, whereby filling out a survey the participant is entered into a draw to win something, usually a gift; but there’s no reason this couldn’t also be cash. 

CAUTION: Make sure that your cash-alternative prizes (e.g., gift cards or products) don’t introduce new forms of bias!  If you advertise that participants get gift cards to a specific store or restaurant you may be biased toward people who frequent those businesses. If you advertise that participants get a discount on products or services, you may be biased toward people interested in those products or services. 

---

What about coercion?

I said we’d talk about bias. When offering an incentive, the risk is that it introduces coercion: the practice of persuading someone to do something. It’s a fine balance between motivating someone to spend their time sharing their experience with you in exchange for a token of appreciation, versus inducing a feeling of being compelled because the incentive is so enticing. Coercion exists when a participant accepts otherwise intolerable risk to receive the incentive. I think this risk is real, however, some research has shown that any risk associated with participation is not influenced by incentives. This study found no relationship between the size of the risk and the size of the incentive. 

It may be impossible to prove that an incentive isn’t coercive. The best way to protect participants from risk isn’t by making changes to your incentive, but by examining policy, protocol, and consent within your project or program. Probably the best mitigation strategy is informed consent, letting participants know about their rights to refuse, clearly outlining any risk to participation, and describing how participation does or does not affect their access to services.  Related to this, refusing access or service based on participation is definitely coercive!

---

What impact does offering an incentive have on your data?

One reason to offer an incentive is to increase the response rate. Increasing response rates should improve your data quality through more valid, representative data. However, there are negative impacts as well – including introducing profile bias based on the specific incentive (as cautioned above). Incentives could also change the responses themselves, either because participants feel compelled to respond positively to receive the incentive, or with even more subtle changes, like improved mood. However, some research suggests that these influences are unlikely.

Ideally, we want people to participate because they care about the outcome, or they have a vested interest in the program or organization. However, motivations to participate cover 3 primary reasons: 1) altruism 2) topical reasons, e.g., the desire to share a specific positive or negative experience, or interest in the topic or 3) egoist reasons, i.e., for the incentive. Most evaluations probably have a mix of these motivations. It can be difficult or impossible, and probably unnecessary to tease them apart. Ensuring strength in your data collection tools and processes will help to mitigate any risk associated with egoist participation.

---

What is participation worth?

A colleague of mine recently had to counsel a client on whether $100 for survey participation was coercive or not. While it’s not common to have to debate if the incentive is too much, it’s still a good lesson in ethical practice. And begs the question, what is participation worth?

There’s no one answer. In fact, there isn’t even a “right” answer. It depends on several factors including:

• time burden: is it a 5-minute survey, or a 3-hour focus group?

• how intense is the participation process?

• how specific or niche, and therefore possibly difficult to reach, is your target participant group?

• how specialized is the knowledge or experience the participant group brings? 

• your project budget!

The higher the burden or intensity, or the more specialized the nature of participation, the higher the perceived value.

Wellesley Institute, out of Toronto, did some exploration in 2018 and found that in research (i.e., not evaluation), participants were paid, on average, $30 for an interview, $25 for a focus group, and $20 for a survey. In my own experience, this is pretty close in evaluation as well. Depending on budget and level of burden (i.e., time commitment) I’ve offered up to $50 for interviews. Wellesley suggests that a good rule of thumb is $25 per hour of participation.

---

Appropriateness of incentives

While cash may be king, there are other considerations when deciding on incentives. I’ve recently been working on a project where paying for services may introduce trauma for some participants. I’ve worked closely with this project team to develop a plan where the participants are compensated for sharing their experiences, but without the transactional nature of giving an incentive after participation. Specifically, persons with lived experience are invited to participate in an interview; regardless of participation, they are also offered extra support to meet any emergent needs they may have. That extra support is not based on their participation, but on the invitation.

Vulnerable populations may respond to incentives differently, by being more likely to accept risk. Vulnerable populations are also frequent target populations for research and evaluations.  This is a great resource that offers more guidance on compensation for persons with lived experience, including: discussing compensation clearly and upfront, offering options, paying in cash, and offering to pay for additional costs (like child care).

Depending on your participant population there are groups and cultures where exchanging gifts is an act of respect. Consider reviewing your incentive practice from a lens of diversity and intersectionality.

Finally, there is some grey area about who is offered an incentive. For example, you are evaluating a program, commissioned by the organization that runs the program. Part of your evaluation plan is to interview the program participants. I think we can safely agree that offering some form of incentive is justified. Part of the same evaluation plan is to interview the organization leaders and the facilitators of the program. Does it make sense to offer them an incentive? Or is it “part of their job” to participate? To pay them for participation is essentially paying them out of their own budget. In my experience, I have differentiated between those participating in a professional capacity (i.e., not incentivizing staff) and those volunteering to participate (i.e., incentivizing lived experiences or program participants).

A key to ensuring your incentives are appropriate is an open dialogue with your project team, which hopefully includes a representative from your target population. Engage in your own reflective practice as an evaluator and seek input from your colleagues to identify risks you may have missed.

---

Other Considerations

1. To offer an incentive, you may need to collect some contact information. With so much data collection being done virtually these days, many incentives are offered electronically, where email addresses are required. This has at least two impacts:

   • Is your participant population likely to have an email address and the capacity to retrieve and access an e-transfer or electronic gift card?

   • Does collecting contact information introduce any ethical risk around identification or breach of confidentiality?

2. In my experience where most data collection is happening virtually, the idea of bringing food and/or beverages is a moot point, but in the past, I have offered focus group participants snacks and drinks. These “perks” to participating are also an incentive and should be considered thoughtfully.

3. Arguably outside the discussion of incentives specifically, is reimbursement. Some participants may need to travel or enlist child or elder care to accommodate their participation. It is common practice to offer reimbursement for such expenses, which can take many forms, including cash, bus tickets, cab fare, mileage payment, etc.

4. I’ve had discussions that the incentive isn’t an incentive, per se, but a “thank-you”. This can be true if the thank-you is given after participation without prior knowledge, or perhaps if the value is so low that it would in no way incentivize anyone. Similarly, some participants have regular hourly rates and can be offered an honorarium to compensate them directly for their time. These distinctions between a thank-you, an honorarium, and an incentive are very blurred. As always, data collection processes and informed consent are the best ways to mitigate risks!

---

There’s a lot of research about incentives, much more than I’ve touched on here. Hopefully, I’ve covered some of the important ground to guide you in deciding whether to offer an incentive and what form and value that incentive should be.

Remember to include the use of incentives in your evaluation contracts. The expense can be significant!

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address


Sign Up

We respect your privacy.

Thank you!

---

A Log Frame is a tool that has mainly been used for designing, monitoring, and evaluating international development projects. Using this tool is a way of structuring the main elements in a project or program, and highlighting the logical connections between them. 

In this article, we explain what a Log Frame is (spoiler: it’s not a logic model), why they can be useful tools for program planning and evaluation outside of international development, and how to make one of your own.

---

What is a Log Frame?

The Log Frame was developed back in 1969 for the U.S. Agency for International Development (USAID). It was developed as a planning tool for international development programs in the form of a matrix which presents a program’s main goal, activities, and what these activities are expected to lead to.

This visual approach helps you to think about the relationships between available resources, planned activities, and the desired changes or results. This structure also helps to explain the linkages between a program’s components.

It follows this main idea:

1. The successful completion of these activities = the production of these outputs

2. The production of these outputs = these changes through outcomes

3. These changes through outcomes = the achievement of these objectives

4. The achievement of these objectives = contributions to the larger goal

Therefore, this tool can help program planners to explain how they believe change will be realized. For evaluation purposes, the Log Frame also identifies the measures and indicators that will help to monitor the program’s anticipated results. 

Although there are a few models that focus on logical links between a program and its contribution to success, be careful not to confuse Log Frames with the following:

• Results Frameworks: Log Frames are focused more on how you will get to your program’s goal. Results Frameworks focus more on explaining the program’s results.

• Logic Models: although the two are very similar, a Log Frame is depicted using a matrix or table, whereas a Logic Model is shown using a flow chart.

• Theory of Change (ToC): ToC is also used for program design, whereas Log Frames are useful mainly for evaluation only. A ToC is also more explanatory and for explaining more complex initiatives, whereas Log Frames are descriptive and better placed for small to medium sized projects. Log Frames don’t easily capture the how and why in the same way a ToC does.

---

When and why are Log Frames used?

Since their development for USAID, Log Frames have become a standard approach required by donors for grant applications. For international development programs, they have become a staple of project planning and evaluation. Due to their utility and accessibility, they’re now also being used for programs outside of the international development realm.

Most Log Frames are developed during program design and are updated throughout the program’s lifespan. Log Frames are not set in stone and should be flexible to the program’s needs and any changes happening on the ground. Developing a Log Frame at the program planning stages helps to involve the whole team and allow key stakeholders to provide ideas about how they see the program operating to reach its goals.

Log Frames can be used for the following:

• To confirm the theory of why your program will result in the desired change. Log Frames can help you to see whether your program will really work and identify any flaws in the theory. It’s all about being logical!

• Help you in making strategic decisions that align with the logic of how your program will contribute to change and reach its goal. It can also help you to allocate resources to where they will be used most effectively.

• To support transparency by clearly describing what your program aims to achieve and how. This can help your program to be more attractive to donors by clearly and logically explaining your ideas. It can also help you to make sure that all staff are on the same page.

• By including indicators of how you will measure change, a Log Frame helps you to develop how you will monitor and evaluate your program. This provides the groundwork for how an evaluation can measure the impact of your program. 

---

What are the key components of a Log Frame?

A Log Frame is commonly presented as a 4×4 table with 16 cells, though this can be modified if needed. Each row represents types of events that take place to help the program achieve its goals. Although the wording may differ slightly, these include:

• Main Goal

• Outputs

• Outcomes

• Activities

The first column describes how the program will reach its objective. The second and third columns summarise how the program’s achievements will be tracked through indicators (i.e., measurement of change) and sources of information (i.e., information needed that will allow indicators to be measured). The last column lists the assumptions or risk analysis. These are the factors outside of the program’s control that are necessary to ensure the program’s success.

---

How do I develop a Log Frame?

It is important to develop a Log Frame with key stakeholders included in the program. Collaborating helps to make sure that it is not developed through a “top-down” approach and leads to better program planning by ensuring that everyone is on the same page in terms of shared objectives. Developing a Log Frame often works best through a mind-mapping session.

Firstly, it is important that there is a clearly established problem your program aims to address. You should also be very aware of the context within which your program will operate. It helps to think about setting your goals first and then working backwards to the grassroots to see what you need to do to reach these goals.

The main questions to consider in the development of your Log Frame include:

• What will our program achieve?

• What activities will we complete?

• What resources will we need to do this?

• What are the problems and challenges we might face along the way?

• How will we measure the progress and outcomes of the program?

There can be more than one indicator and assumption listed for each level, but it is good to keep these manageable by using tools such as SWOT analysis (see our free template here), Stakeholder Analyses, and a Risk Matrix.

A Log Frame should also be tested to check the logic. A simple test is to ask the following:

1. IF these activities are undertaken AND the assumptions hold true, THEN the intended outputs will be created

2. IF these outputs are delivered AND the assumptions hold true, THEN the purpose will be achieved

3. IF the purpose is achieved AND the assumptions hold true, THEN the intervention will have contributed to the overarching goal

(Adapted from DFID, 2011)

Once developed, make sure that the Log Frame doesn’t just sit on a shelf throughout program implementation. Review it frequently and use it to support decision-making, manage activities, assess progress, and keep stakeholders aware of plans to monitor progress.

---

Pros & Cons of using a Log Frame

Benefits of a Log Frame:

• It ensures objectives are clear and measurable

• It ensures concrete evidence for achievement is collected

• Because risks and assumptions are made explicit, problems can be analyzed systematically

Weaknesses of a Log Frame:

• It may cause rigidity in program management if it is not viewed as something that can be updated throughout program implementation

• It is a “one size fits all approach” which does not always capture the complexity and context of a program

---

Have you worked with Log Frames before, or have questions? Comment on this article or connect with us on LinkedIn or Twitter!

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address


Sign Up

We respect your privacy.

Thank you!

---