cplysy

This article is rated as:

 

 

Data are only useful when used! They do no good buried in reports, sitting on shelves (or shared drives) hidden away. Data, particularly data from an evaluation, are begging to be discussed, contemplated, and put into action!

Let’s chat about some ways to make sure your data are used. One place to start is to think about why an evaluation was conducted to begin with. Evaluations can serve many purposes – check out these 10 reasons to evaluate.  If you can articulate why an evaluation was conducted, you can review your data with a focused lens.

Another place to look is to explore your program goals, objectives, or intended outcomes and see how your data can help you to meet them or speak to them. Or perhaps you have targets or Key Performance Indicators that will help to frame your review of the data. It goes without saying that your data should be answering your evaluation questions, but still, this isn’t helping the move into the “now what” phase of an evaluation.

One important step is to put together an engaged and passionate team of representatives from across your organization, like described here: Three Ways to Increase the Chances your Evaluation Results will Actually Get Used. This team can help to interpret the data (often called sense-making) and can help to identify actions that can be taken on data. They can also help build organizational engagement and spread key messages.

Once you have your dream team, check out our new list of reflective questions that can help to uncover new insights buried in your data.

Trying to answer all of the questions in this checklist will be too many to tackle at once. But pulling out, say, 5 can get some discussion started. I’ve often used these questions in my sense-making sessions or even final presentations to stakeholder audiences to get them talking about the data and really thinking about what it means.

Of course, talking about the data assume that your data are high quality and presented in a way that your audience can learn from them. Here are some final tips to make sure your data are working for you: From Data to Actionable Insights

---

Let us know what you think about our new infographic Questions to Get you Thinking about your Data below!

This article is rated as:

 

 

We’ve written before about Ethical Decision Making in Evaluation, which describes those grey areas in evaluation planning, data collection and analysis, and reporting, and we’ve offered Program Evaluation Standards in Practice as a guiding tool.

We’ve also shared first-hand experience when faced with real-time ethical decision-making in My Interviewee is Drinking Vodka: An Evaluation Ethics Case. So why am I  writing about ethics again? Because ethical practice, to me, is a cornerstone of my practice. Not only does ethical practice ensure we are doing right by everyone involved, but conducting evaluations ethically adds to the professionalization of our work.

---

What is ethical practice?

Ethical practice in evaluation is ensuring your work is guided and driven by standards of conduct, that promote integrity, honesty, and respect, where the potential for harm is minimized. This starts with determining whether the evaluation is worth doing at all, considering, for example, the burden placed on participants.

Ethical practice comes into play at every level of your evaluation: from planning to designing data collection tools, data collection, analysis, and reporting.  Take a look at each section of your evaluation plan and see if you can identify the risks. If you can’t, keep reading!

The Canadian Evaluation Society offers some Guidance for Ethical Evaluation Practice. They suggest that ethical practice is based on 3 values:

• Rights and well-being of persons and peoples

• Truth-seeking, honesty and transparency

• Responsibility to stakeholders and society

In addition, many ethical guidelines draw from the 1979 Belmont Report, created by the National Commission for the Protection of Human Subjects of Biomedical and Behavioural Research. The Belmont report provides three guiding principles:

• Respect for persons, including informed consent

• Beneficence: do no harm

• Justice: equality, without bias or discrimination

Typically, evaluations don’t require review from a Research Ethics Board, but often there is nowhere else to turn for an ethical review. In Alberta, Canada where Eval Academy is based, we’re fortunate to have a program called ARECCI that can provide ethical review to evaluation and quality improvement projects. This article is intended for those without a formalized review process, to help you apply your own ethical review to your own evaluation work.

---

How do I ensure I apply an ethical lens to my evaluation?

The first step in ethical evaluation is awareness. Take a look at our infographic of questions to ask to ensure your evaluation is ethical.

Ethical practice is about self-awareness, honest understanding of limitations and self-reflection, knowing your own biases, assumptions, and values, and considering how they influence an evaluation. Ultimately, asking your self “Am I doing the right thing?” will force you to consider some of these questions.

---

Types of Risk

Now that we’re aware of the potential for risk in evaluations, and where to look for risk, what exactly are you looking for? As you’d expect, there are many types of risk. Usually, we’re talking about risk to participants so let’s start there.

Risks to participants include:

• Mental and emotional risk, including re-traumatization and distress

• Power imbalances, real or perceived coercion

• Reputational risk through breached confidentiality or privacy

• Physical risk, safety from the nature of participation

• Legal risk through disclosure of information

• Financial risk through disclosure of information

Vulnerable populations require additional consideration. These could include children, equity-deserving populations, those with limited capacity, or those in power imbalances, but often this includes any group who are regular targets for research, evaluation, or any data collection. The burden of participation or risk of exploitation is not insignificant for many populations.

There are also risks to your project. You’ll want to consider if your evaluation plans would put the project or funding at risk by not meeting timelines, or perhaps your data collection strategies risk going substantially over budget. This is an example of where the principle of justice comes in – your desire to gain knowledge must be balanced against what would be lost if the knowledge were not gained. That is, are your data collection strategies putting the potential learnings at risk in any way?

There are also risks to systems. Is it possible through participating in your evaluation, that participants may access additional supports or services, which may create system delays, and capacity issues or cause access trouble for others needing that service?

The consequences of including high risk in a project can be significant. Participants may experience extreme distress and require access to additional supports (and they’d likely look to you to connect them with those supports). There are also risks to programs including loss of trust, funding, negative publicity, or even fines.

Finally, but importantly, there may be there may be other ethical guidance available to you. For example, the First Nations Principles of OCAP ® offers support for information governance with respect to the ownership, control, access, and possession of data from First Nations.

---

Summary

All projects have some level of risk. Eliminating risk isn’t necessarily the goal. There is certainly a balance between collecting information you need to answer important questions that drive knowledge gain, best practice, and informed decision-making, with some acceptable risk. The question is, what tolerance for risk do you (and your clients or organizations) have? We want to not only minimize risk but also maximize benefit. The key is that the risks have been given deliberate consideration.

---

What are your top tips for assessing and managing risk in evaluation? Share your ideas below!