cplysy

This article is rated as:

---

Eval Academy just released a new template, “Canva design templates for creating your own Logic Model”

Who’s it for?

Whether you’re new to evaluation or if evaluation is your main role, these Canva design templates are for anyone who wants to design a Logic Model to be more visually appealing. To use these templates, you will need to set-up or log in to your own account in Canva.

What’s the purpose?

These Canva design templates can be used to present your own Logic Model for any type of intervention in a more distinct format. They’re great for sharing your Logic Model with a wider audience, such as when you want to share your model on your website.

What’s included?

Two customizable Canva templates that provides you with the space you need to input your Logic Model components.

Get the templates

---

---

Learn more: related articles and links

You can learn more about Logic Models on Eval Academy through the following links:

• The definition of Logic Models

• Differences between Theory of Change, Log Frames, Results Frameworks and Logic Models – what are they and when to use them

• Improve Your Logic Model Using 3 Simple Design Principles

• Developing a Logic Model Template 

You can also find many other templates in our list of resources to support you in planning and implementing an evaluation. Some of our most popular templates include:

• Theory of Change Template

• Evaluation Plan Template

• Evaluation Kick-Off Meeting Template

---

What do you think of our new design templates? Let us know in the comments below!

This article is rated as:

---

At Three Hive Consulting, we pride ourselves on being led by three Credentialled Evaluators. But what is a CE? What does it mean? 

Let’s begin by talking about what the Credentialed Evaluator (CE) Designation is. The CE designation is a product of the Canadian Evaluation Society (CES), but the concept of competencies and credentialing does not belong to the CES alone. Many evaluation societies have published evaluation competencies (e.g., the American Evaluation Association, the European Evaluation Society, the International Development Evaluation Association, Japan Evaluation Society, South African Monitoring and Evaluation Association, Thailand Evaluation Network and United Nations Evaluation Groups), but the CES is one of the few societies or groups who have taken the competencies a step further and provides a credentialing service to its members. 

---

What is the Credentialed Evaluator designation?

Members of the CES can voluntarily go through the credentialing process. Achieving the designation signifies that the holder has adequate knowledge, skills, and experience in the 36 competencies and the ability to produce sound evaluations. To be credentialled, a member is assessed on both their education (a graduate degree is required) and experience through submitting written explanations of how they meet the competencies. 

---

What the Credentialed Evaluator designation is not

What the CE designation is not:

• A certificate or license which formally tests skills or knowledge. There is no written or oral test used to demonstrate knowledge and understanding. 

• A requirement. The CE is not a licensing board which restricts an evaluator’s ability to practice evaluation.

• Extra courses or academic work (so long as you meet the academic requirements).

---

Why should you become a Credentialed Evaluator?

From the CES’s point of view, the CE designation was developed to help with the professionalization of evaluation and to improve the quality of evaluations being produced. To learn about why evaluators get their CE, I talked with three individuals who either have their CE or are in the process of receiving it. For these individuals, the CE designation is:

• A learning tool.

The process of achieving your CE requires reflection on your experiences and highlights areas where you may need to brush up on. Going through the process can help you identify your strengths and weaknesses and inform future professional development.

• A celebration of your achievements and competence.

Describing how you meet the 36 competencies is an exercise in reflection and can help you to articulate what you have achieved thus far in your evaluation experience.

• A signal to others about your ability and professionalism.

There is some recognition and understanding of the CE designation within the evaluation field. Having your CE can signal to others that you have a certain level of experience and knowledge to be able to complete quality evaluations. This can be helpful when job searching, applying for proposals, or being required to demonstrate competency in your practice. More and more evaluation job descriptions are also beginning to list the CE as a desired trait.

• Contributing to the standardization process and raising the standards of evaluation in Canada.

This last point is more about contributing to the professionalization of evaluation and raising awareness of the CE designation by obtaining it and hopefully having conversations with others about its value or meaning.

---

Why shouldn’t you become a Credentialed Evaluator?

Although I didn’t talk to anyone directly who didn’t have the designation for the article, I have talked with many evaluators throughout the course of my career as I personally hemmed and hawed about whether to get the designation or not myself.

For some, the designation provides little additional value to their careers and is not necessary to continue to advance in their field. For others, the process and benefits are too vague to justify putting in the time and effort. Finally, some evaluators weren’t clear on what the process entailed and thought that obtaining their CE would require extra courses or education.

---

Now that we’ve covered the big questions – what it is and why would you (or wouldn’t you) obtain your CE designation – we’ll dig deeper into the process of getting the CE designation and hear more from those who are in the process of getting it or who already have it in the next article in this series.

What do you think about the CE process? Let us know in the comments below!

This article is rated as:

---

You’ve got your evaluation plan; you’ve developed your data collection tools and you’re ready to go live with collecting the data you need to answer your evaluation questions. Step 1: Identify your sample. Step 2. Recruitment. But how do you get participants to take part in the data collection process?

Recruitment is often difficult. In my experience, it’s been a rare but welcome occurrence that my target sample size is reached easily. This means I’m often hustling and employing a number of strategies to boost my recruitment efforts.

Sufficient recruitment for your data collection methods is important. Poor recruitment can lead to:

• Bias

• insufficient sample size (poor power)

• under or over-representation of specific groups

• costly delays 

• drawing inappropriate conclusions

• limited generalizability

I’ll share with you some of the strategies and methods I’ve used that help me to identify and reach my target recruitment levels for the data collection process.

---

Identify your Sample

Identifying your sample size and characteristics of the sample ahead of your data collection will likely help you to select your recruitment methods and will definitely help you to identify when you can stop recruiting.

Let’s start with inclusion and exclusion criteria. For example, if you’re evaluating a program, you may only want to include people who actually participated in that program (this is the inclusion criteria). If you want to evaluate the long-term impact of a service, you may want to exclude anyone who didn’t receive the full service or dropped out (this is the exclusion criteria). 

Identifying these characteristics about your sample is important, but it’s also wise to reflect on what bias this may be introducing. You likely don’t want to create such strict inclusion/exclusion criteria as to end up with a sample that doesn’t actually reflect the population you are exploring. Evaluation is not research; evaluation often benefits from a more heterogenous, representative sample.

For example, you may choose not to include attendance rates of a program as an inclusion criterion to allow you to explore barriers faced in participation.

Similarly, you can’t control for everything and there may be unknown characteristics introduced in your sample. 

For example, you likely can’t create an inclusion criterion for participant motivation to take part in a program, but participant motivation may impact your evaluation findings. 

When identifying your sample, you’ll also want to consider if there are any inherent ethical considerations for your participant groups. For example, are they youth, or could they have any cognitive disabilities? This may require close attention to your consent process. 

Access is another key consideration.  Does your identified sample have access to the medium you’re using, like a phone or computer; are they likely to be available at the times you may be offering; and, are they comfortable participating in the primary language of the evaluator? If not, you may want to consider using an interpreter in your data collection process. Check out our article on Top Tips for Using a Real-Time Interpreter for Interviews and Focus Groups.

Recruiting participants who dropped out, quit, or never started a program can be very difficult. If this is part of your identified sample, you will likely need to work with program staff to implement a process where there is an exit survey, or a few questions asked by intake or administrative staff at the time of contact with (potential) participants.

---

Sampling Methods

There are several ways to sample! Here are some of the more common ones used in evaluation:

1. Random – include all individuals who fit your inclusion criteria.  Random sampling means everyone in the population has an equal chance of participation. Random sampling is simple and mitigates risks of bias; however, the simplicity often means it’s difficult to get people to volunteer.

2. Convenience – you recruit those who are most accessible to you. For example, you may attend a program session and use the participants from that session as your sample, or you may sit in the waiting room of an office and use people who have appointments that day as your sample. Convenience sampling is also simple and likely quick, but it may introduce bias or generate a sample that doesn’t represent the population.

3. Snowball – using word of mouth you build your sample starting from the first participant. If you can identify one or only a small number of participants, you can use the assumption that your first participants likely know others that fit your inclusion criteria as they did. In snowball sampling, you ask your participants to help you recruit by spreading the word, or at least to help you identify other means to recruit your sample. You likely want to have recruitment cards or flyers available to give out (see an example below). 

4. Maximum Variation – you intentionally recruit for variation. Not only do you have an identified overall sample size, but you have it broken down – for example, perhaps you want 10 participants from each of the three program sites or spanning certain ages, genders, or backgrounds. This is a favourite of mine in evaluation because I often try to capture varied experiences. The disadvantage is that sometimes recruitment can take longer, and you may need very targeted recruitment strategies.

5. Deviant Case – if you’re limited in your ability to recruit a sufficient sample size, you may target individuals for whom things went particularly well (the gold star cases) or those whose experiences were off track. These “deviant” cases offer experiential insight into what worked well and contributed to success, or what not to do to avoid failure in achieving intended outcomes. 

6. Typical Case – you may choose the alternative to the deviant case, which is the “typical case” which is targeting your recruitment to describe the average experience. This can create a profile that describes what a “normal” experience is like.

Remember when I said recruiting sometimes involves hustling and using a number of strategies? You can definitely use a combination of these sampling methods!

---

Recruitment Strategies

Recruitment isn’t as easy as picking your sampling methods, from there, you still need to actually identify and make contact with your intended sample. Some concrete methods for doing so include:

1. Posters – Particularly if you are recruiting from individuals who attend a certain office or program, hanging posters in a waiting room can be effective. If you’re looking for a broader sample, like the general public, you could consider coffee shops, libraries, or community centres.

2. Virtual posters – most people are online, so using social media is probably one of the strongest recruitment tools. You could post to a program’s Facebook or Instagram account, or work with a communications team to Tweet about opportunities to participate.  

3. Website pop-ups – like virtual posters, if your program has a website you can work with the web team to include a pop-up for visitors to the site inviting them to participate.

4. Newsletters – If you’re planning is established ahead of time, blurbs in newsletters can alert participants to upcoming opportunities. 

5. Collaborators – I often tell my clients at our kickoff meeting that I expect them to be champions of evaluation, which includes making connections or introductions, and advocating the importance of participation in evaluation. Your evaluation advisory committee or working group, if one exists, can probably do a lot in terms of identifying staff to talk to, or customers to recruit. Just keep in mind that they can introduce their own bias and direct you to the more favourable participants. 

6. Attendance – suitable to convenience sampling, sometimes asking permission to wait in a waiting room on a certain day or attending a program session will help in your recruitment efforts. I do find that putting a face to a name can make potential participants feel more comfortable signing up. I’ve also attended program graduation or celebration events where many participants come together. I come armed with recruitment flyers to leave on tables or hand out directly to people I meet. Here’s an example of one I’ve used:

7. Existing databases – many programs have existing databases of clients’ names and contact information. Be careful. You want to ensure you have the right to access contact information and can contact them for evaluation purposes. Often, I get program staff to do the first cold call, introducing me.

8. Recruiting for Mixed Methods – sometimes you need to recruit for multiple methods. I’ll often send out a survey and include a question at the end about a participant’s willingness to take part in a follow-up interview or focus group, or even gauge their willingness for future data collection. That way I have permission to contact them directly, with the contact information they provide.  

Again, using many of these strategies will make your recruitment faster and hopefully get you the sample you need.

In most of these recruitment methods and strategies, having a link where participants can access more information or even sign up directly for an interview or focus group using tools such as Calendly will boost your chances of reaching your sample, rather than asking participants to email or phone you.

Adherence to ethical practice is important throughout recruitment. Make sure you reflect on your strategies for accessibility and inclusion, but also look for potential coercion, including reviewing your use of an incentive. 

---

Do you have any go-to recruitment methods? Or have you tried any of these before? Comment on this article or connect with us on LinkedIn or Twitter! 

This article is rated as:

---

In my role as an evaluator specializing in qualitative methods, I use interviews and focus groups A LOT. Interviews and focus groups are strategies used to gather in-depth qualitative data from stakeholders. They’re great methods for capturing perspectives and stories. For further insights on how to conduct interviews, check out this Eval Academy article.

However, interviews and focus groups also have their participation challenges. Stakeholder groups such as those who don’t have access to technology are often excluded from virtual interviews, and individuals who can’t afford travel costs or arrange childcare for example can be excluded from in-person interviews. Stakeholder groups who are unable to communicate in the interviewer’s language or would prefer to speak their own are also easily excluded from participating.

In Canada, the primary language for over 10% of the population is a language other than English or French (Statistics Canada, 2022). A real-time interpreter to translate the interview or focus group conversation can help ensure these individuals’ voices are heard within your evaluation. 

In this article, I share my top tips for running an interview or focus group using a real-time interpreter. I do this by reflecting on a recent online focus group I completed which utilized an Arabic interpreter.

---

How to decide when to use an interpreter

So firstly, how do you decide when to use an interpreter? Some evaluations may request it within their RFP. In others, when developing the stakeholder matrix and identifying your target groups in the planning stage, it might become apparent that some key groups cannot communicate in the evaluator’s language, or they may be more comfortable conversing in their own. To help you decide whether an interpreter is needed, you can also take a look at this article on ethical decision-making in evaluation.

Once it’s determined that a real-time interpreter is required, a good place to start in recruitment is to ask the organization or program for recommendations of anyone they have used before or of someone they think may be appropriate. In my experience, a professional translator is not always required as we recruited an individual with bi-lingual capabilities identified by the program. In that case, your local university can also be a good place to turn to for ideas and support. Qualities to look for in a real-time interpreter include:

• A professional communicator; someone who is comfortable and thrives when speaking to new people

• This includes fluency and experience in conversing in the evaluator’s language and the participant’s language. It is particularly useful if the interpreter has knowledge of the social and cultural background of the participants

• A professional listener 

• This includes attention to detail as well as the ability to engage others

It is important to consider whether the interpreter will receive payment for their services. This should be discussed with the organization or program you’re evaluating. If the interpreter is a volunteer from the organization or program, or from a local university, consider offering an incentive that aligns with incentives provided to interview or focus group participants. Check out this Eval Academy article to learn more about incentives for participation in evaluation.

When using an interpreter within an interview or focus group, I suggest 3 main steps:

1. Preparation

2. Data collection and real-time interpretation

3. Debrief

---

Step 1: Preparation

Using an interpreter within an interview or focus group will mean the session requires more time. If an interview is expected to take 30 minutes in the evaluator’s language, I would suggest setting aside an hour using a real-time interpreter. If you planned a focus group for 90 minutes, consider scheduling 3 hours. If you planned several interviews with individuals that all require interpretation, you could consider scheduling as a focus group for efficiency. Due to the increased time needed, it’s best to keep these focus groups to a maximum of 3 to 4 people to allow for in-depth conversation and real-time interpretation.

It’s important to meet with your interpreter before jumping into data collection. Your interpreter should understand the purpose of the data collection and ideally have some knowledge of the cultural and social background of the participants. This first step is vital to ensure accurate interpretations from the qualitative data collected in the interview or focus group. This will also provide you with the opportunity to assess the interpreter’s experience with translating interviews and focus groups. In this preparation meeting, you should:

• Provide the interpreter with an overview of the project or program you’re evaluating, why you’re evaluating, why you’re collecting data from participants, and how that data will be used

• Share all necessary materials with the interpreter in advance (e.g., interview guide, consent form, etc.) so that they can translate it into the appropriate language

• Run through the question guide to recap each question and the associated probes to ensure the interpreter understands the purpose and intent of the questions

• Participants should be made aware that an interpreter will be available at the interview or focus group and they can communicate in their own language. Therefore, it is helpful to translate all pre-interview/focus group documentation such as the invitation, information sheet, and consent form, and include a brief introduction to the interpreter in your invite

• Pre-determine how you will run the interview or focus group (see step 2). You should also make it clear to the interpreter that you would like them to translate the conversation as accurately as possible to avoid any potential judgemental language or interpretation bias. This will help to minimize misunderstandings by relaying the message in its entirety

---

Step 2: Data collection and real-time interpretation

Now you’ve completed the preparation, it’s time to start collecting data. When facilitating the interview or focus group, it is helpful to record the conversation so you can listen back to it later and transcribe it. Make sure you gain all the participant’s and interpreters’ approval to record the interview or focus group.

When using an interpreter in the interview or focus group, I’ve found that it’s best to break up any large portions of text into blocks of a couple of sentences, then have the interpreter translate each block. It’s important, like with all interviews or focus groups, that participants are given the opportunity to ask any questions before the interview begins. Make sure participants fully understand their rights and that participation in the discussion is voluntary.

If there are multiple participants in a focus group setting, I suggest asking each interview question to each participant and then asking the interpreter to translate their response before moving on to the next individual. This helps to ensure that the interpreter relays the participant’s response in as true a form as possible.

Your main job as the facilitator is to influence the direction of the focus group or interview and interpret the meaning of the participants’ responses. Therefore, while the interpreter translates the response, it is important to take notes for any probes you want to ask the individual. You can also ask the interpreter whether they themselves have any follow-up questions.  Within a focus group, it is important to allow participants to build upon each other’s answers by using probing questions such as “did you have anything to add to x participant’s answer?”. For focus groups, don’t forget to take notes on any relevant body language you view as the conversation is happening like whether there are any dominant voices for example.

The diagram below by Quintanilha et al (2015) provides a visual of the role of the facilitator (i.e., researcher) and interpreter within the focus group setting. 

---

Step 3: Debrief

Immediately after the interview or focus group, it is useful to hold a quick debrief with the interpreter to recap the discussion and expand on the main discussion points. This can support the validity of findings by ensuring meaning has been interpreted correctly. 

If you had permission to record the discussion, it’s important to consider the higher costs of transcribing when using a real-time interpreter. To reduce costs, I suggest listening back to the recording and only transcribing the sections translated into the facilitator’s language. To do this, you could use audio trimming software like iMovie or Audacity. Take a look at our previous article on how to transcribe interviews like a pro.

Once the audio has been transcribed, it can be useful to verify the transcript with the interpreter. I suggest sending the transcript to the interpreter and asking them to review it for accuracy and meaning. 

When analyzing and presenting your results, make sure you discuss how a real-time interpreter was used to collect and make sense of the data, as well as any possible bias or limitations as a result. This might include power imbalances through interpreter-researcher-participant dynamics or challenges encountered when the flow of the discussion is hindered using back-and-forth interpretation. 

---

What’s your experience of using a real-time interpreter in an interview or focus group?  Comment on this article or connect with us on LinkedIn or Twitter!