allblogs

This article is rated as:

Eval Academy just released a new template, “Canva design templates for creating your own Logic Model”

Who’s it for?

Whether you’re new to evaluation or if evaluation is your main role, these Canva design templates are for anyone who wants to design a Logic Model to be more visually appealing. To use these templates, you will need to set-up or log in to your own account in Canva.

What’s the purpose?

These Canva design templates can be used to present your own Logic Model for any type of intervention in a more distinct format. They’re great for sharing your Logic Model with a wider audience, such as when you want to share your model on your website.

What’s included?

Two customizable Canva templates that provides you with the space you need to input your Logic Model components.

Get the templates

Learn more: related articles and links

You can learn more about Logic Models on Eval Academy through the following links:

• The definition of Logic Models

• Differences between Theory of Change, Log Frames, Results Frameworks and Logic Models – what are they and when to use them

• Improve Your Logic Model Using 3 Simple Design Principles

• Developing a Logic Model Template 

You can also find many other templates in our list of resources to support you in planning and implementing an evaluation. Some of our most popular templates include:

• Theory of Change Template

• Evaluation Plan Template

• Evaluation Kick-Off Meeting Template

What do you think of our new design templates? Let us know in the comments below!

This article is rated as:

At Three Hive Consulting, we pride ourselves on being led by three Credentialled Evaluators. But what is a CE? What does it mean? 

Let’s begin by talking about what the Credentialed Evaluator (CE) Designation is. The CE designation is a product of the Canadian Evaluation Society (CES), but the concept of competencies and credentialing does not belong to the CES alone. Many evaluation societies have published evaluation competencies (e.g., the American Evaluation Association, the European Evaluation Society, the International Development Evaluation Association, Japan Evaluation Society, South African Monitoring and Evaluation Association, Thailand Evaluation Network and United Nations Evaluation Groups), but the CES is one of the few societies or groups who have taken the competencies a step further and provides a credentialing service to its members. 

What is the Credentialed Evaluator designation?

Members of the CES can voluntarily go through the credentialing process. Achieving the designation signifies that the holder has adequate knowledge, skills, and experience in the 36 competencies and the ability to produce sound evaluations. To be credentialled, a member is assessed on both their education (a graduate degree is required) and experience through submitting written explanations of how they meet the competencies. 

What the Credentialed Evaluator designation is not

What the CE designation is not:

• A certificate or license which formally tests skills or knowledge. There is no written or oral test used to demonstrate knowledge and understanding. 

• A requirement. The CE is not a licensing board which restricts an evaluator’s ability to practice evaluation.

• Extra courses or academic work (so long as you meet the academic requirements).

Why should you become a Credentialed Evaluator?

From the CES’s point of view, the CE designation was developed to help with the professionalization of evaluation and to improve the quality of evaluations being produced. To learn about why evaluators get their CE, I talked with three individuals who either have their CE or are in the process of receiving it. For these individuals, the CE designation is:

• A learning tool.

The process of achieving your CE requires reflection on your experiences and highlights areas where you may need to brush up on. Going through the process can help you identify your strengths and weaknesses and inform future professional development.

• A celebration of your achievements and competence.

Describing how you meet the 36 competencies is an exercise in reflection and can help you to articulate what you have achieved thus far in your evaluation experience.

• A signal to others about your ability and professionalism.

There is some recognition and understanding of the CE designation within the evaluation field. Having your CE can signal to others that you have a certain level of experience and knowledge to be able to complete quality evaluations. This can be helpful when job searching, applying for proposals, or being required to demonstrate competency in your practice. More and more evaluation job descriptions are also beginning to list the CE as a desired trait.

• Contributing to the standardization process and raising the standards of evaluation in Canada.

This last point is more about contributing to the professionalization of evaluation and raising awareness of the CE designation by obtaining it and hopefully having conversations with others about its value or meaning.

Why shouldn’t you become a Credentialed Evaluator?

Although I didn’t talk to anyone directly who didn’t have the designation for the article, I have talked with many evaluators throughout the course of my career as I personally hemmed and hawed about whether to get the designation or not myself.

For some, the designation provides little additional value to their careers and is not necessary to continue to advance in their field. For others, the process and benefits are too vague to justify putting in the time and effort. Finally, some evaluators weren’t clear on what the process entailed and thought that obtaining their CE would require extra courses or education.

Now that we’ve covered the big questions – what it is and why would you (or wouldn’t you) obtain your CE designation – we’ll dig deeper into the process of getting the CE designation and hear more from those who are in the process of getting it or who already have it in the next article in this series.

What do you think about the CE process? Let us know in the comments below!

This article is rated as:

You’ve got your evaluation plan; you’ve developed your data collection tools and you’re ready to go live with collecting the data you need to answer your evaluation questions. Step 1: Identify your sample. Step 2. Recruitment. But how do you get participants to take part in the data collection process?

Recruitment is often difficult. In my experience, it’s been a rare but welcome occurrence that my target sample size is reached easily. This means I’m often hustling and employing a number of strategies to boost my recruitment efforts.

Sufficient recruitment for your data collection methods is important. Poor recruitment can lead to:

• Bias

• insufficient sample size (poor power)

• under or over-representation of specific groups

• costly delays 

• drawing inappropriate conclusions

• limited generalizability

I’ll share with you some of the strategies and methods I’ve used that help me to identify and reach my target recruitment levels for the data collection process.

Identify your Sample

Identifying your sample size and characteristics of the sample ahead of your data collection will likely help you to select your recruitment methods and will definitely help you to identify when you can stop recruiting.

Let’s start with inclusion and exclusion criteria. For example, if you’re evaluating a program, you may only want to include people who actually participated in that program (this is the inclusion criteria). If you want to evaluate the long-term impact of a service, you may want to exclude anyone who didn’t receive the full service or dropped out (this is the exclusion criteria). 

Identifying these characteristics about your sample is important, but it’s also wise to reflect on what bias this may be introducing. You likely don’t want to create such strict inclusion/exclusion criteria as to end up with a sample that doesn’t actually reflect the population you are exploring. Evaluation is not research; evaluation often benefits from a more heterogenous, representative sample.

For example, you may choose not to include attendance rates of a program as an inclusion criterion to allow you to explore barriers faced in participation.

Similarly, you can’t control for everything and there may be unknown characteristics introduced in your sample. 

For example, you likely can’t create an inclusion criterion for participant motivation to take part in a program, but participant motivation may impact your evaluation findings. 

When identifying your sample, you’ll also want to consider if there are any inherent ethical considerations for your participant groups. For example, are they youth, or could they have any cognitive disabilities? This may require close attention to your consent process. 

Access is another key consideration.  Does your identified sample have access to the medium you’re using, like a phone or computer; are they likely to be available at the times you may be offering; and, are they comfortable participating in the primary language of the evaluator? If not, you may want to consider using an interpreter in your data collection process. Check out our article on Top Tips for Using a Real-Time Interpreter for Interviews and Focus Groups.

Recruiting participants who dropped out, quit, or never started a program can be very difficult. If this is part of your identified sample, you will likely need to work with program staff to implement a process where there is an exit survey, or a few questions asked by intake or administrative staff at the time of contact with (potential) participants.

Sampling Methods

There are several ways to sample! Here are some of the more common ones used in evaluation:

1. Random – include all individuals who fit your inclusion criteria.  Random sampling means everyone in the population has an equal chance of participation. Random sampling is simple and mitigates risks of bias; however, the simplicity often means it’s difficult to get people to volunteer.

2. Convenience – you recruit those who are most accessible to you. For example, you may attend a program session and use the participants from that session as your sample, or you may sit in the waiting room of an office and use people who have appointments that day as your sample. Convenience sampling is also simple and likely quick, but it may introduce bias or generate a sample that doesn’t represent the population.

3. Snowball – using word of mouth you build your sample starting from the first participant. If you can identify one or only a small number of participants, you can use the assumption that your first participants likely know others that fit your inclusion criteria as they did. In snowball sampling, you ask your participants to help you recruit by spreading the word, or at least to help you identify other means to recruit your sample. You likely want to have recruitment cards or flyers available to give out (see an example below). 

4. Maximum Variation – you intentionally recruit for variation. Not only do you have an identified overall sample size, but you have it broken down – for example, perhaps you want 10 participants from each of the three program sites or spanning certain ages, genders, or backgrounds. This is a favourite of mine in evaluation because I often try to capture varied experiences. The disadvantage is that sometimes recruitment can take longer, and you may need very targeted recruitment strategies.

5. Deviant Case – if you’re limited in your ability to recruit a sufficient sample size, you may target individuals for whom things went particularly well (the gold star cases) or those whose experiences were off track. These “deviant” cases offer experiential insight into what worked well and contributed to success, or what not to do to avoid failure in achieving intended outcomes. 

6. Typical Case – you may choose the alternative to the deviant case, which is the “typical case” which is targeting your recruitment to describe the average experience. This can create a profile that describes what a “normal” experience is like.

Remember when I said recruiting sometimes involves hustling and using a number of strategies? You can definitely use a combination of these sampling methods!

Recruitment Strategies

Recruitment isn’t as easy as picking your sampling methods, from there, you still need to actually identify and make contact with your intended sample. Some concrete methods for doing so include:

1. Posters – Particularly if you are recruiting from individuals who attend a certain office or program, hanging posters in a waiting room can be effective. If you’re looking for a broader sample, like the general public, you could consider coffee shops, libraries, or community centres.

2. Virtual posters – most people are online, so using social media is probably one of the strongest recruitment tools. You could post to a program’s Facebook or Instagram account, or work with a communications team to Tweet about opportunities to participate.  

3. Website pop-ups – like virtual posters, if your program has a website you can work with the web team to include a pop-up for visitors to the site inviting them to participate.

4. Newsletters – If you’re planning is established ahead of time, blurbs in newsletters can alert participants to upcoming opportunities. 

5. Collaborators – I often tell my clients at our kickoff meeting that I expect them to be champions of evaluation, which includes making connections or introductions, and advocating the importance of participation in evaluation. Your evaluation advisory committee or working group, if one exists, can probably do a lot in terms of identifying staff to talk to, or customers to recruit. Just keep in mind that they can introduce their own bias and direct you to the more favourable participants. 

6. Attendance – suitable to convenience sampling, sometimes asking permission to wait in a waiting room on a certain day or attending a program session will help in your recruitment efforts. I do find that putting a face to a name can make potential participants feel more comfortable signing up. I’ve also attended program graduation or celebration events where many participants come together. I come armed with recruitment flyers to leave on tables or hand out directly to people I meet. Here’s an example of one I’ve used:

7. Existing databases – many programs have existing databases of clients’ names and contact information. Be careful. You want to ensure you have the right to access contact information and can contact them for evaluation purposes. Often, I get program staff to do the first cold call, introducing me.

8. Recruiting for Mixed Methods – sometimes you need to recruit for multiple methods. I’ll often send out a survey and include a question at the end about a participant’s willingness to take part in a follow-up interview or focus group, or even gauge their willingness for future data collection. That way I have permission to contact them directly, with the contact information they provide.  

Again, using many of these strategies will make your recruitment faster and hopefully get you the sample you need.

In most of these recruitment methods and strategies, having a link where participants can access more information or even sign up directly for an interview or focus group using tools such as Calendly will boost your chances of reaching your sample, rather than asking participants to email or phone you.

Adherence to ethical practice is important throughout recruitment. Make sure you reflect on your strategies for accessibility and inclusion, but also look for potential coercion, including reviewing your use of an incentive. 

Do you have any go-to recruitment methods? Or have you tried any of these before? Comment on this article or connect with us on LinkedIn or Twitter!