evalacademy

This article is rated as:

 

 

Data are only useful when used! They do no good buried in reports, sitting on shelves (or shared drives) hidden away. Data, particularly data from an evaluation, are begging to be discussed, contemplated, and put into action!

Let’s chat about some ways to make sure your data are used. One place to start is to think about why an evaluation was conducted to begin with. Evaluations can serve many purposes – check out these 10 reasons to evaluate.  If you can articulate why an evaluation was conducted, you can review your data with a focused lens.

Another place to look is to explore your program goals, objectives, or intended outcomes and see how your data can help you to meet them or speak to them. Or perhaps you have targets or Key Performance Indicators that will help to frame your review of the data. It goes without saying that your data should be answering your evaluation questions, but still, this isn’t helping the move into the “now what” phase of an evaluation.

One important step is to put together an engaged and passionate team of representatives from across your organization, like described here: Three Ways to Increase the Chances your Evaluation Results will Actually Get Used. This team can help to interpret the data (often called sense-making) and can help to identify actions that can be taken on data. They can also help build organizational engagement and spread key messages.

Once you have your dream team, check out our new list of reflective questions that can help to uncover new insights buried in your data.

Trying to answer all of the questions in this checklist will be too many to tackle at once. But pulling out, say, 5 can get some discussion started. I’ve often used these questions in my sense-making sessions or even final presentations to stakeholder audiences to get them talking about the data and really thinking about what it means.

Of course, talking about the data assume that your data are high quality and presented in a way that your audience can learn from them. Here are some final tips to make sure your data are working for you: From Data to Actionable Insights

---

Let us know what you think about our new infographic Questions to Get you Thinking about your Data below!

This article is rated as:

 

 

We’ve written before about Ethical Decision Making in Evaluation, which describes those grey areas in evaluation planning, data collection and analysis, and reporting, and we’ve offered Program Evaluation Standards in Practice as a guiding tool.

We’ve also shared first-hand experience when faced with real-time ethical decision-making in My Interviewee is Drinking Vodka: An Evaluation Ethics Case. So why am I  writing about ethics again? Because ethical practice, to me, is a cornerstone of my practice. Not only does ethical practice ensure we are doing right by everyone involved, but conducting evaluations ethically adds to the professionalization of our work.

---

What is ethical practice?

Ethical practice in evaluation is ensuring your work is guided and driven by standards of conduct, that promote integrity, honesty, and respect, where the potential for harm is minimized. This starts with determining whether the evaluation is worth doing at all, considering, for example, the burden placed on participants.

Ethical practice comes into play at every level of your evaluation: from planning to designing data collection tools, data collection, analysis, and reporting.  Take a look at each section of your evaluation plan and see if you can identify the risks. If you can’t, keep reading!

The Canadian Evaluation Society offers some Guidance for Ethical Evaluation Practice. They suggest that ethical practice is based on 3 values:

• Rights and well-being of persons and peoples

• Truth-seeking, honesty and transparency

• Responsibility to stakeholders and society

In addition, many ethical guidelines draw from the 1979 Belmont Report, created by the National Commission for the Protection of Human Subjects of Biomedical and Behavioural Research. The Belmont report provides three guiding principles:

• Respect for persons, including informed consent

• Beneficence: do no harm

• Justice: equality, without bias or discrimination

Typically, evaluations don’t require review from a Research Ethics Board, but often there is nowhere else to turn for an ethical review. In Alberta, Canada where Eval Academy is based, we’re fortunate to have a program called ARECCI that can provide ethical review to evaluation and quality improvement projects. This article is intended for those without a formalized review process, to help you apply your own ethical review to your own evaluation work.

---

How do I ensure I apply an ethical lens to my evaluation?

The first step in ethical evaluation is awareness. Take a look at our infographic of questions to ask to ensure your evaluation is ethical.

Ethical practice is about self-awareness, honest understanding of limitations and self-reflection, knowing your own biases, assumptions, and values, and considering how they influence an evaluation. Ultimately, asking your self “Am I doing the right thing?” will force you to consider some of these questions.

---

Types of Risk

Now that we’re aware of the potential for risk in evaluations, and where to look for risk, what exactly are you looking for? As you’d expect, there are many types of risk. Usually, we’re talking about risk to participants so let’s start there.

Risks to participants include:

• Mental and emotional risk, including re-traumatization and distress

• Power imbalances, real or perceived coercion

• Reputational risk through breached confidentiality or privacy

• Physical risk, safety from the nature of participation

• Legal risk through disclosure of information

• Financial risk through disclosure of information

Vulnerable populations require additional consideration. These could include children, equity-deserving populations, those with limited capacity, or those in power imbalances, but often this includes any group who are regular targets for research, evaluation, or any data collection. The burden of participation or risk of exploitation is not insignificant for many populations.

There are also risks to your project. You’ll want to consider if your evaluation plans would put the project or funding at risk by not meeting timelines, or perhaps your data collection strategies risk going substantially over budget. This is an example of where the principle of justice comes in – your desire to gain knowledge must be balanced against what would be lost if the knowledge were not gained. That is, are your data collection strategies putting the potential learnings at risk in any way?

There are also risks to systems. Is it possible through participating in your evaluation, that participants may access additional supports or services, which may create system delays, and capacity issues or cause access trouble for others needing that service?

The consequences of including high risk in a project can be significant. Participants may experience extreme distress and require access to additional supports (and they’d likely look to you to connect them with those supports). There are also risks to programs including loss of trust, funding, negative publicity, or even fines.

Finally, but importantly, there may be there may be other ethical guidance available to you. For example, the First Nations Principles of OCAP ® offers support for information governance with respect to the ownership, control, access, and possession of data from First Nations.

---

Summary

All projects have some level of risk. Eliminating risk isn’t necessarily the goal. There is certainly a balance between collecting information you need to answer important questions that drive knowledge gain, best practice, and informed decision-making, with some acceptable risk. The question is, what tolerance for risk do you (and your clients or organizations) have? We want to not only minimize risk but also maximize benefit. The key is that the risks have been given deliberate consideration.

---

What are your top tips for assessing and managing risk in evaluation? Share your ideas below!

This article is rated as:

 

 

How often have you said the phrase “your information will be kept confidential”?  “Confidential” and “anonymous” are words we use quite a bit in the evaluation world. But do you know what they actually mean? Let’s explore some of these concepts.  

---

Confidentiality

Confidentiality is when personally identifying information is collected but is not linked to the individual’s responses; in other words, the personally identifying information is kept separately. Confidentiality can be maintained through password-protected files and/or through proper record storage and destruction. Maintaining confidentiality is also a key consideration in reporting.

Anonymity

Anonymity is when the data collection process does not collect any personally identifiable information. Anonymity is when there is no opportunity to link responses back to a specific individual.

If you are interviewing an individual or conducting a focus group, your data collection cannot be anonymous. If you are sending out a link to an online survey that collects no identifying information, then you have anonymity.

Personally Identifying Information

Notice this phrase pops up in both of those definitions? This phrase, sometimes shortened to PII, unfortunately doesn’t have a standard definition. It means different things to different organizations, in different policies or legislation, and across different sectors.

Generally, it is understood to mean any singular datum or group of data that could identify an individual. It may be obvious that a name is personally identifying, but many other demographic or profile data depend on the context. For example, age can be identifying if you are collecting data from a sample with an average age of 50, but one participant is 87. Gender may be identifying if your sample is predominately one gender.

Combinations of data can also lead to identification. For example, perhaps in your sample, organizational role is not identifying – you have lots of program leads and coordinators, but if you include duration of time employed and role together you may identify that newly hired coordinator, or the program lead whose been in the organization longer than the rest.

Most countries or jurisdictions have policies or laws around the protection of personally identifying information. Here in Canada where Eval Academy is based we have PIDEDA: The Personal Information Protection and Electronic Documents Act. In the United States HIPAA (Health Insurance Portability and Accountability Act) is a well-known regulation.

Staff roles within an organization is a common one I come across that is potentially identifying.

---

Difference between legal confidentiality and ethical confidentiality

For an evaluation, the question of confidentiality or anonymity is ethical, not legal. It is our duty to put in extra effort to preserve confidentiality, having thought of the ethical risks of data breaches. This is not the same as lawyer-client confidentiality.

---

Disclosure

Some professions have bodies or policies that guide if and when confidential information can be disclosed: think psychologists, physicians, or lawyers. Unfortunately, as of yet, evaluators do not have this, and yet we are often working with vulnerable populations who may disclose a number of concerning matters to us.

Given that there are no official rules, what you do about disclosing information is ultimately up to you and your organization or client. It is prudent to anticipate potentially risky scenarios. Here are some scenarios you may want to consider:

If a participant shares or discloses a risk to themselves (e.g., plans of suicide): if you believe the situation is an emergency, you may choose to call emergency services, like 911, and disclose the information you have.

If a participant discloses non-emergent risks about problems with mental health or behaviour: you may wish to have a list of local resources available to you that you could share. Your resource list may include things like housing options, the food bank, a crisis line, and other mental health supports or social support agencies relevant to your population.

Of course, a person may disclose criminal activity or other abuses. You will need to rely upon your own discretion to determine if there is anybody at urgent risk and what to do in these situations.

In non-emergent cases, and depending on the resources available to you, seeking consent to disclose information is always a good choice:

“What I’d like to do is share [details, e.g., name and phone number] with [description of where it will be shared, e.g., our social worker], so that [describe intent]. Is that ok?”

If they decline, you should not disclose any details. If consent is given, when you disclose confidential information, ensure you are only sharing information relevant to the reason for disclosure, and ensure the disclosure is through confidential channels.

Check out our Tips for Conducting Interviews infographic and our posts about consent for extra guidance here.

---

How to maintain Confidentiality

Most organizations have a records storage and retention policy. It is your duty to be familiar with them regarding confidential information. A storage policy will guide you on how and where to store specific types of information. Importantly, you should also be aware of who else has access to that storage location and if they should have access to any confidential information.  A retention policy will likely categorize the type of record, how long it will be kept for, and how it will be destroyed. These policies can be referenced in your data collection introductions, for example:

“Information you share will be kept for [X years] on a password-protected server owned by [organization]. After [X years] we delete all copies.”

 Some other places to consider confidentiality:

• Transcription: you could not include names in transcription, thus when the recording is deleted, the transcript has (less) personally identifying information

• File naming: ensure you don’t name recordings, transcripts or notes using personally identifying information, e.g., “Beth Smith’s Interview”

• In reporting: Using quotes can add such rich information to your reporting, but quotes should be considered from the lens of confidentiality, and where possible use Member Checking to ensure explicit consent

 

The key here is to be a good data steward, which includes ethical data collection, transparency in the use of data, sound data management (storage, retention, destruction) and reporting that maintains confidentiality.  

---

Let us know your tips for maintaining confidentiality below!

This article is rated as:

 

 

Eval Academy just released a new checklist, Information Request Checklist

Who’s it for?

Whether you’re new to evaluation or if evaluation is your main role, this checklist is for anyone who’s about to start a new evaluation project to make sure you’re gathering the information necessary to support your evaluation endeavour!

---

What’s the purpose?

While not all projects will have all of these documents available, this checklist can act as a supporting tool to make sure you have the context you need when starting an evaluation. Use this checklist prior to your Evaluation Kick-Off Meeting to gain as much detail as you can about the project. We like to say that an evaluator can rarely have too much program information. You don’t know what you don’t know!

---

What’s included?

A downloadable and printable checklist to help you gather information on the project, client details, and evaluation-specific details.

 

 

Learn more: related articles and links

You can learn more about planning an evaluation through the following links:

• The Components of an Evaluation Plan

• How to Kick Off Your Evaluation Kick-Off Meeting

• Scoping an Evaluation: Begin with the Purpose

---

You can also find many other resources to support you in planning and implementing an evaluation. Some of our most popular resources include:

• Evaluation Plan Template

• Evaluation Kick-Off Meeting Agenda Template

• Program Evaluation Scoping Guide

What do you think of our new checklist? Let us know in the comments below!

This article is rated as:

---

In the first article of this installment, we covered what the Credentialed Evaluator (CE) designation is and is not, and talked a little bit about why you might get it. In this article, we will explore what you need to do to gain your CE designation. 

At Three Hive Consulting, we believe the designation has value, but recognize that getting your CE is a daunting task.

---

The How

Let’s dig in and explore how you actually obtain the designation. To be eligible to gain the CE designation, evaluators who are members of the CES must have:

1. Evidence of a graduate-level degree or a graduate certificate or diploma in Program Evaluation.

2. Two years of full-time evaluation-related work experience or the equivalent within the last 10 years.

3. Demonstrated appropriate education or experience across the 5 domains of practice. (This last piece is where we’ll focus on the most).

When you’re ready to apply, we suggest getting all of your documents ready first, before applying and spending ~$500 on your application fee (and starting the clock on your three-year time limit). 

To demonstrate your education:

To demonstrate your education, you will be required to upload a copy of your certificate or graduate degree. Mine was hanging out in my parent’s house in a different province, and so obtaining ‘proof’ took a bit more coordination than most of you might need. 

To highlight your work experience:

You will need to provide a project CV and references who can attest to your two years (or 24 non-consecutive months) of evaluation experience. You will have to fill out a reference/sponsor declaration form for each reference and have them sign it. This sponsor form is available on the CES website without paying your application fee.

To showcase your competency:

Finally, to show your competency, you must prove that you have experience in 70% of each of the five domains. You can download the Competencies for Canadian Evaluation Practice from the CES website.

From the CES website, the 5 domains of practice are:

1. Reflective Practice

   • Focusing on knowledge of evaluation theory and practice; adhering to standards, guidelines, and ethics; reflection, learning and self-awareness.

2. Technical Practice

   • Focusing on the development, implementation, and completion of an evaluation.

3. Situational Practice

   • Focusing on attending the context in evaluation.

4. Management Practice

   • Focusing on the project management and facilitation aspects of an evaluation

5. Interpersonal Practice

   • Focusing on the people-skills required to conduct an evaluation.

Each domain has 7-15 competencies within it and to gain the CE designation, a member must demonstrate competency in at least 70% of each domain by providing written examples of how they have achieved each competency.

A reviewer reads what you have written and assigns you a pass or fail for each competency. As such, we don’t recommend you only provide examples for 70% of the competencies in each domain, otherwise failing just one competency means you do not obtain your CE and you must revise and re-submit your application. I ended up filling out all of the competencies, although I’ve heard some suggestions to just fill out an extra one or two competencies per domain.

The trick with the competencies is that you only have 1,000 characters to demonstrate your experience in each competency.

To assist you with this process, we’ve created a competency template, which aligns with the new competencies released in 2019. The template walks you through the domains and competencies, pointing you to relevant resources where applicable. It also gives you a warning when you are nearing or over your 1,000-character limit.

Get the Template

The main takeaway from this how to apply section is that you can do a lot of the time-consuming preparation work on your CE before you pay and formally begin the application process.

---

So, what’s stopping you from applying?

We’ve covered the technical details, now let’s talk about what might be stopping you from sitting down and filling out your experience, paying the application fee, and submitting your application.

To help us understand what was helping or hindering others from obtaining their CE, we talked to some evaluators who were considering obtaining their CE and a few who already had theirs.

Barriers to obtaining a Credentialed Evaluator designation included:

• Limited value-add to their career – few workplaces place explicit value on the CE designation, as such it’s a big commitment without much return on investment.

• Low priority – without external stimulus, it can be hard to set time aside to write how you have achieved all 36 competencies. 

• Expensive – the application fee is $485 and you must be a member of the CES, which is another $195.

• Time-consuming – writing 36 concise 1,000-character descriptions of how you meet a competency takes time! A lot of time.

• Unclear about the process and details – while the CES has a few webinars about the application process, the process still seems a bit obscure. How exactly does one demonstrate competency in just 1,000 characters? What does an adequate description look like?

---

Let us help you break down those barriers!

Stay motivated

Once you’ve decided that you want to obtain your CE designation, write down the reasons why you are applying and keep those reasons at the forefront during this process. It can be easy to lose motivation, so having a strong ‘why’ (perhaps even tying it into your performance or development goals) will help keep you motivated.

Connect with your evaluation community! There are likely other members in your CES Chapter who are also working on their CE, see if you can connect with them. Other Chapters may offer mentoring support so that you can be paired with a CE who can support you in the process.

I’m also a big fan of ‘treat yourself’. Set up external motivation and rewards. There’s nothing like taking yourself out for a fancy latte and pastry to work on your CE.

Be strategic with your time

Make sure to set aside dedicated time to work on your CE. Take it from me, it takes a lot longer if you continually jump in and out of writing up your competencies. I took the whole three years and wasted a lot of time re-writing the same competencies over and over again.

You can use our template to help keep you focused on what you have completed and what needs your attention.

Use your words sparingly

It can be tricky to include all of the relevant details of how you have demonstrated a competency in 1,000 characters. The good news is that you are encouraged to create a narrative throughout your competencies. This means that once you introduce a project, you don’t have to re-introduce it if you use it in another example.

Pick a few strong evaluation projects and use them to demonstrate the various competencies. Even better, once you’ve introduced a project, give it an acronym and save yourself a few characters of space.

Our template has a column that tracks which projects you reference for each competency so you can check that you’re creating a consistent narrative. It will help you to make sure that you aren’t re-using an example within a project.

---

Hopefully, by breaking down the process and providing you with a template to use, we’ve made the CE application process feel a little more attainable. Let us know how it works for you!