Welcome to our August roundup of new and noteworthy evaluation news and resources – here is the latest.

Have something you’d like to see here? Tweet us @EvalAcademy or connect on LinkedIn!

---

New and Noteworthy

---

Evaluation Podcasts

While podcasts aren’t new it seems like podcasts are a new medium for the evaluation field. Recently there was a Twitter thread asking about evaluation podcasts. Below is a summary of current evaluation podcasts that were mentioned and what they are all about: 

Eval Cafe – is co-hosted by Carolyn Camman (@c_camman) and Brian Hoessler (@strongRoots_SK). The premise for this podcast is “informal chats on evaluation-related topics – the kind you might overhear at your favourite coffeeshop, if it was frequented by evaluators.” There have been 35 episodes produced since its first episode in June 2017. Previous guests have included Khalil Bitar (@khalilbitar), Kylie Hutchinson (@evaluationmaven), Dana Wanzer (@danawanzer), Sarah Farina (@broadleafc) and Michael Quinn Patton (@MQuinnP). When there isn’t a guest, Brian and Carolyn discuss and explore various evaluator topics like: how do we explain what we do? What is so scary about evaluation? And other evaluator questions you may grapple with. 

The Glass Frog Podcast – This podcast started in 2018 by GlassFrog Solutions (@glassfrogtweets) – a research and evaluation firm. It has produced 20 episodes with a wide-variety of evaluation-related content, including a) empathy in evaluation, b) survey design, c) evaluating collaboratives, d) photography and evaluation, and so much more! 

EvalCrisis – is produced by the DEVCO/ESS Evaluation in Crisis Initiative. The purpose of the podcast is to share teachings for the evaluation community to learn from so they can adapt evaluations in crisis situations, like the current COVID-19 pandemic. The podcast series began in June 2020 and have eight episodes posted on its site. Topics discussed so far have included, safety, remote data collection, ethical issues, and shifting evaluation foci. 

Evaluland – is hosted by Dana Wanzer (@danawanzer). The podcast discusses the land of evaluation. It is a relatively new podcast that started this year; so far there have been six episodes produced. The focus of the podcast thus far has been on evaluation theory and teaching evaluation, which is a natural fit for Dana who is an Assistant Professor teaching evaluation at the University of Wisconsin-Stout. 

EvalEdge – The European Evaluation Society started a podcast series last month. So far there have only been two episodes produced. The first episode introduced the podcast and co-hosts @1marcolorenzoni, @hurhassnain and @saskia_sbr and the second podcast hosted Linda Raftree (@meowtree), co-founder of MERL Tech, about innovative examples of using big data and the ethical considerations to be aware of.

---

New and Noteworthy — Tools

---

Charles and Lynn Schusterman Family Foundation – A guide toward diversity, equity and inclusion in data collection

At Three Hive Consulting, have been discussing how we can be more mindful and intentional when collecting our data to promote inclusion and equity – this guide is a great resource. It was produced “as a starting point to spark inquiry, conversation, disruption and, ultimately, better data collection practices within organizations.” In this guide they provide examples for collecting survey data related to gender identity, sexual orientation, race and ethnicity, and disability. It also includes a list of additional resources to access.

Clear Horizon Academy – Tools and resources

I stumbled across Clear Horizon’s Academy where they have a tools and resources section. This section of the site includes “must haves” for change makers. It includes tools and approaches for measurement, evaluation and learning; guides for different approaches like Most Significant Change, and; real-life examples on how these tools and approaches can be applied to support change.

Khulisa’s #evaltuesdaytip for photo elicitation and evidence maps

Each Tuesday Khulisa outlines, you guessed it, tips for evaluators. This month Khulisa created a blog post that unpacks photo elicitations and how it can be applied in youth or children in the education sector by providing seven insightful resources. Another helpful post was its #evaltuesdaytip that provided 14 links to evidence maps. Check out Beam Exchange for an example of what an evidence-map is and how it can be used.

---

New and Noteworthy — Courses, Events and Webinars

---

September 2020

Asian Evaluation Week: Evaluating for a Better Future

Sponsor: Asia-Pacific Finance and Development Institute and the Asian Development Bank’s Independent Evaluation Department (IED)

Dates: September 7 – 11

Venue: Virtual event

 

Group deliberations (focus groups, etc) in realist evaluations and reviews

Presenter: Dr. Ana Manzano (RealismLeeds)

Date and Time: September 11; 02:30 – 03:30 MDT

Venue: Online

Evalpalooza I: Evaluation Failures with Kylie Hutchinson and Thought Leaders

Presenters: Kyle Hutchinson & Libby Smith

Date and Time: September 24; 12pm CDT

Venue: Online

 

October 2020

Evaluation 2020

Sponsor: American Evaluation Association

Dates: October 27 – 30

Venue: Virtual Event

---

Subscribe to Our Newsletter

Sign up with your email address to receive news and updates from Eval Academy every month.

Email Address
Sign Up

We respect your privacy.

Thank you!

---

 

 

Part 1 – Take them on a journey

Evaluators are notorious for bad reporting. According to Jane Davidson it has to do with our training. Often evaluators are trained in social sciences – a world that prepares people for academic style research and how to write scientifically. The problem is this doesn’t work in the “real world.” Leaders and decision makers don’t want to comb through pages and pages of text to try and find findings and what they should do about them. #TLDR

At Three Hive Consulting we took Jane’s advice and started unlearning these bad reporting habits. Instead we looked to creative fields like graphic design and visual storytelling to inform our reporting. Canva is a great place to start learning about design. They have free courses and tutorials that walk you through design basics you can apply to your reporting. In this series of posts, I am going to walk you through how I reno’d draft evaluation reports using six of Canva’s design lessons. This first post relates to Canva’s design lesson “Take them on a journey.”

 

The importance of storytelling

The best way to engage an audience is to tell them a story. Our brains love stories and are much more likely to remember information is we activate our imagination and our emotions through story, as opposed to showing facts alone.

A well-crafted story will take people on a journey. If you think back to your favourite stories they likely have two common characteristics. The first is, good stories feature transformation – there was a problem, someone (or something) overcame that problem and now there is some sort of resulting change. The second is, good stories have a clear structure – a beginning, middle and end. This journey is what activates our imagination and emotions and gives our message staying power. Not only this, a good story will motivate people to act.

As evaluators we want our stakeholders to use our findings (i.e. act). The problem is evaluators often take people on a really boring journey full of facts and data in our reports. As Jane describes in her editorial, Unlearning Some of our Social Scientist Habits:

“Executive Summary (lots introductory information, methodology, sampling, random snippets of findings that fail to give a clear sense of the program’s quality or value, plus something incomprehensible about moderator variables), Introduction, Literature Review, a theoretical model and detailed explanation of the relevant social science theory explaining the links along some variables (unfortunately not a program logic model, and not even remotely linked to an evaluation question—this part contributed by a university faculty member with no evaluation expertise), Methodology, Findings (about 20 pages of raw data, all presented separately by source and data type with virtually no explanatory narrative, none of it linked back to the questions), Conclusions (some glimmers of hope in here, but by now we are 37 pages into the report and have lost most of our audience), Appendices.”

It is no wonder the only action stakeholders take when reading these types of evaluation reports is to shut their eyes! We need to stop this useless reporting if we want them to engage and act on the findings. Unlearning this is hard, but once you give yourself the freedom to let go you will never look back. Instead, let’s look at how to craft a story to take our stakeholders on a more interesting journey.

 

Crafting your story

Get to know your audience

Before you begin crafting your story you need to get to know your audience (aka your stakeholders) in order to make that report accessible. Who are they? What are their needs and how can this information address them? How can the information you discovered solve their problem? What do you want them to do with the information? Asking questions like these are critical for crafting your story so the content is relevant and resonates with your audience. Refer to my post, Three Ways to Increase the Chances Your Evaluation Results Will Actually Get Used, where I dig into stakeholder analysis and how to tailor reporting needs to different stakeholders.

 

Figure out the point(s)

In this post Stephanie Evergreen talks about the most important question in data viz, “what’s your point?” This question is just as important with your overall evaluation reporting. Jane discusses in her editorial how evaluators have a bad habit of producing reports with pages and pages of data but little acknowledgement of how the findings were used to answer any question of value; “when, oh when are they going to get to the point?” she asks. Bottom line when it comes to reporting – have a point. It seems somewhere along the way we forgot about those thesis statements and may need to re-learn how to incorporate those.

But since we’re trying to unlearn our scientific writing style perhaps we should think of our thesis statement as a ‘big idea’ – term coined by master communicator, Nancy Duarte. A big idea is the core message you want your audience to hold on to. A big idea contains two components:

1. Point of view – your point of view on a subject

2. Stakes – the reason why the audience for your report should care.

 

For your evaluation, the point of view should come from the findings – what is the data telling you? The stakes should be driven by who the evaluation stakeholders are (specifically for this evaluation report) and how do they intend to use the findings? Combining these will help you figure out your point or big idea. Everything you put in your report should then focus on that big idea.

We recently conducted a Social Network Analysis with a client. Ultimately, this client wanted to understand what its collaborative network looked like and how it could be strengthened. Throughout the report we presented the findings, so they answered the evaluation questions. In addition, as results were presented, we identified the meaning behind the findings and by calling out the ‘so what’ in italics (see below). You’ll notice we also didn’t use generic graph titles like, “Relationship between degree centrality and trust.” Instead, we told people the point – “A higher degree centrality was related to higher trust.”

 

Lead the audience to a destination

Taking an audience on a journey means we need to lead them to a destination. You were hired to evaluate something for a purpose. Hopefully you’ve discovered answers to questions that your stakeholders didn’t know before they hired you. Leading our stakeholders to a destination means we need to not only outline what they should they do with the information (i.e. recommendations), but in order to motivate people to act we need to outline why they should act (or not). What will happen to them as a result? What will happen if they don’t act? Nancy Duarte describes this as a “new bliss.” For evaluators this means not just listing off recommendations but why our stakeholders should implement those recommendations. For example, if we are recommending a department reorganization it is important to outline what that new bliss will look like after the reorganization (e.g. a saner workload.)

So how can we re-jig how we report our findings, so we incorporate these storytelling principles and take our stakeholders on a journey that motivates them to act? Below is a reporting structure we encourage you to try. It gets rid of that long-winded scientific style that forces people to flip through pages and pages of information searching for the point. Instead, this structure presents an alternative that weaves those two important characteristics of storytelling – transformation (what, so what, now what) and a clear structure (beginning, middle and end).

 

Try it out and make sure to stay tuned for the next article in our six part series, “Consistency is cool” where I show you how I reno’d an evaluation report using consistent design (e.g. font, colours, heading levels).

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---

 

 

Evaluators are equipped to ask and answer some pretty complex questions. We are not afraid to tackle abstract ideas and make sense of messy data. However, there is one question that can be tough to succinctly answer – what is evaluation?

As evaluators, you can likely recall sitting at family dinners or catching up with a friend when they ask you “what is evaluation?” You may have even had this conversation with colleagues or clients. My initial reaction is “where do I begin?” or “how do I explain evaluation to a non-evaluator?”

Many evaluators and evaluation societies have attempted to answer this question. The Canadian Evaluation Society (CES) consulted members and published their findings on this very topic. This is the definition they came up with:

Evaluation is the systematic assessment of the design, implementation or results of an initiative for the purposes of learning or decision-making.

The need to explain evaluation is likely to come up again and again for evaluators. Fortunately, the American Evaluation Association (AEA) recently created a series of videos and short summaries to explain evaluation. Their post is intended to help us define our profession and better explain the value of evaluation to outsiders.

AEA presents evaluation in the context of three sectors: Organizational, Teaching, and Independent or Consultant. To better understand the work of evaluation in the sectors, AEA interviewed three evaluators. The remainder of this article will review learnings from the AEA post as well as some potential implications for the future.

Evaluation Sectors

Organizational Evaluation

This sector refers to working internally within an organization. Some examples may include: government, non-profits, charities, foundations, or even industry. AEA interviewed Eric Barela, the first Measurement & Evaluation professional hired by Salesforce. Here is a summary of what he had to say about being an internal evaluator:

• Approach evaluation by asking three questions:

  • What? To gain an understanding of what an organization is doing

  • So what? To see if anything is happening

  • Now what? To use the information to make change or move from “good” to “great”

• Evaluation is becoming more applicable to other industries, including for-profit organizations. He feels this is driven by the desire to focus more on social impact.

• As an internal evaluator, he helps his organization ask questions, understand data, and then act on it. He poses two other questions that help him in his evaluation role at Salesforce:

  • How do we know this is happening?

  • What do we need to know in order to improve what we’re doing?

 

Teaching Evaluation

The teaching of evaluation within academic settings is viewed by AEA as a main sector of practice. AEA recognizes that evaluators come from diverse backgrounds and there is currently no set educational pathway nor is there a licensing body for the profession. However, there are academic institutions that offer evaluation courses to help students prepare for a career in evaluation.

Karen Jackson, from the Leadership Studies and Adult Education department at North Carolina A&T State University, was interviewed to gain more insight into the teaching of evaluation. Here are some key takeaways from her interview:

• Defining evaluation is a complicated task, but the definition should be flexible to allow for adapting in diverse settings.

• Academic institution, like hers, offer evaluation courses that focus on the methodology, cultural competency and application of skills.

• She often starts a course by asking her students “what is evaluation?” The initial perspective is that evaluation is a means of assessment, testing, or a way to meet funder requirements.

• As her courses progress she encourages her students to ask themselves “why they are doing an evaluation.” This leads to answers that include learning, improvement and trust – which she argues are at the root of evaluation.

 

Independent Evaluator or Consultant

The final sector described by AEA refers to external evaluations that are conducted by independent evaluators or consultants. AEA views independent evaluation as a way to boost public confidence or verify what an organization is doing. AEA interviewed two consultants about their experience in independent evaluation: Jara Dean-Coffey and Karen Jackson. Both interviewees aim to discover insights to help their clients make decisions. Here is a summary of what they had to say:

Jara Dean-Coffey

• Relies on six areas to help her clients:

  1.     Evaluative thinking

  2.     Evaluative practices

  3.     Strategy

  4.     Communication

  5.     Capacity building

  6.     Coaching

• She recommends always starting with evaluative thinking to understand: “the context, the assumptions, the values, and the outcomes” of her client. This includes reviewing the organization’s theory of change.

• One strategy she offers is to start with what the client feels is most important for them to answer at the moment.

 

Karen Jackson

• Karen Jackson views her purpose as “help[ing] organizations learn more about themselves.”

• She seeks to understand if clients are achieving their goals and if not, help them take the steps to do so.

• She believes that independent evaluators have the opportunity to give people a voice and work with communities.

Next Steps

The breakdown of the definition by sector combined with the three evaluator perspectives was a helpful way for AEA to explain evaluation. There were also common themes that emerged across the sectors, such as: learning, helping others to ask meaningful questions, and identifying ways to improve. Revisiting Eric Barela’s use of the question “now what?”, I have some suggestions on how to use the AEA post and insight moving forward.

• Ask yourself the same questions to understand your practice: What does evaluation mean to you? How do you approach evaluation? And what do you find rewarding about your work?

• Look for sectors that may be missing from AEA’s overview. For example, should ‘Evaluation Research’ be included in the definition?

• Create your own definition of evaluation. Using AEA’s recent post and/or other definitions (such as the one offered by CES), create your own definition of evaluation and what you do in your role.  

• Advocate for evaluation in these three sectors and beyond. Large for-profit companies, like Salesforce, are starting to see the importance of evaluation – let’s keep that momentum going!

 

Conclusion

AEA’s post does a good job at explaining evaluation in a succinct way. For me, I have never worked in the Learning Evaluation sector, but I still found Karen Jackson’s approach and perspective on evaluation to be both relatable and useful. Likewise, I think movement between these sectors is expected – both in terms of the definitions put forward, the skillsets used and the nature of evaluators having the ability to work in multiple sectors. 

AEA’s explanation of evaluation is similar to many other definitions put forward by societies (i.e CES). I think what differentiates this post from other definitions is that it incorporates the experiences and perspectives of multiple evaluators by effectively using video. The use of interviews result in an overview of evaluation that is meaningful to evaluators as well as outsiders.

I think the question “what is evaluation?” and our attempt to define it is an on-going process with no end in sight. I think this is what makes evaluation interesting – it continues to expand and evolve by incorporating new ideas and entering new settings. A flexible definition is key to our profession and I think AEA’s post does a good job at showing the complexity, yet inclusivity of evaluation. 

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---

 

 

This article is a summary and discussion of Caldwell and Bledsoe’s 2019 paper in the American Journal of Evaluation called “Can Social Justice Live in a House of Structural Racism? A Question for the Field of Evaluation.” The original article contains the research and citations to back up these claims.

Racists vs. Racism

Before we can talk about possible strategies to address racism in evaluation, we need to make a very important distinction between individual “racists” and structural “racism.”

A racist individual is someone who holds racist beliefs, such as biases against certain races or negative opinions based on race. For example, the belief that one race of people is less intelligent than another race of people. A racist (or someone with racist beliefs) may act on these beliefs by committing acts of bigotry, hate crimes, or even violence.

Structural racism, on the other hand, occurs at the level of social systems (not individuals). In the words of Ibram X. Kendi, it is like a rain that falls on everyone in a society – no one is immune to or exempt from structural racism. Racism is present in institutions and systems of power, such as unfair laws or the discriminatory practices of schools, workplaces, or government agencies. Racism can also be present within a society’s culture in the form of ideologies or myths that systematically advantage white people and disadvantage people of colour. For example, the overwhelming depiction of people of colour in mainstream media as criminals encourages discrimination and unequal treatment of individuals.

This distinction matters because when we talk about racism, we are usually not talking about racist individuals – we are talking about structural racism in our culture and institutions.

This means that people can participate in systems of racism without holding racist beliefs themselves. For example, research on implicit bias suggests that many of our decisions around racial stereotypes happen in a split second without our awareness. Because we are all living in the “rain” of structural racism, it is likely we have internalized some of these biases and stereotypes, even if we believe ourselves to be nice, fair, unbiased individuals. We are also bound by procedures, policies, and laws that may be racist – so even if we are not racist individuals, our actions are limited by structural racism.

Racism in evaluation

Caldwell and Bledsoe trace the history of evaluation as an academic field, and identify the ways it, too, has been soaked in the “rain” of racism. For example:

• Western perspectives and assumptions are ingrained in evaluation methods and theory (e.g., what is defined as credible and valid data).

• Evaluation as a field has historically excluded diverse, non-Eurocentric ways of knowing (e.g., cultures that use a logic system that is circular rather than linear).

• In evaluation, it is generally accepted that issues can be defined and “solved” by social scientists who do not understand or value the life experiences of people of colour.

• Evaluations are based on an academic discipline that was born out of colonialism, slavery, segregation, and apartheid (e.g., research and evaluation was used to support the racist claim that Black people were inherently less intelligent than White people).

This is not to say all of evaluation is racist, because that would ignore the many contributions of evaluators who are black, indigenous, and people of colour (BIPOC).

Some responses to these issues of racism in the field include culturally responsive evaluation, indigenous evaluation, and equitable evaluation (EE). However, these perspectives are generally seen as “optional” and are not necessarily the norm in evaluation. It will take intentional work to undo these systems of racism and make the field anti-racist.

 

Strategies to eradicate racism

If we accept that “all evaluators, regardless of demographic designation, are subject to perpetuating structural and institutional racism, found in the history and systems of the profession,” the question becomes: what can we do about it?

The authors propose a suite of strategies to unravel racism within evaluation, such as:

1. Include culturally diverse perspectives in evaluation theory, practice, and education.

2. Normalize social justice methods and theories in the field of evaluation.

3. Make changes to professional organizations (like the AEA or CES), for example:

   • Only nominate professionals for awards and presentations who have applied a social justice framework in their work,

   • Provide training programs with a social justice perspective,

   • Reject manuscripts that do not address social justice, equity, or culture; and

   • Make social justice a criterion for accreditation of evaluators.

4. Funders of evaluations require a social justice statement or equitable evaluation methods in their requests for proposals.

5. Expand training programs, conference themes, and workshops focused on racial equity and inequality.

Part of the solution to racism in the evaluation profession is improving the standards and expectations of our professional organizations so they align with social justice and equity. Changing these sorts of systems is difficult and complex, but it can only happen if people demand it. I encourage you to think about the professional organizations you are a part of, and how you can use your power to move these conversations forward.

These may sound like radical changes (because they are!), but they are necessary to eradicate structural racism in our profession and society.

 

Source

Caldwell, L. D., & Bledsoe, K. L. (2019). Can Social Justice Live in a House of Structural Racism? A Question for the Field of Evaluation. American Journal of Evaluation, 40(1), 6–18. https://doi.org/10.1177/1098214018815772

 

A note about the author

This article was written by Nick Yarmey, a white settler living in Treaty 6 territory of what is currently called Canada. I write this with the acknowledgement that I do not have the lived experience of being black, indigenous, or a person of colour (BIPOC). However, I based this article on what I have learned from BIPOC authors and researchers in the interest of taking on some of the labour of explaining these concepts, especially to other white folks. I invite questions, critiques, additions, and comments.

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---

 

 

On a summer morning, after several attempts to interview clients for an evaluation project, I arrived with a social worker at an overnight shelter. Finally, we had located Jules, who wanted to share her experiences with the program I was learning about. When we approached her and her friends, we noticed that she was sipping from a bottle of vodka. 

Now this is certainly not an everyday occurrence in my life as an evaluator. Most of my days involve evaluation planning, liaising with stakeholders, and polishing up reports. The majority of my firm’s primary data collection is done by other members of my team, although occasionally I can’t keep my curiosity in check and I conduct a few interviews myself. For this project, I wanted to be closely connected to the program beneficiaries and ensure I had a very detailed understanding of their experience.  

This project, to me, was a very big deal. 

I had heard about it well before we were contracted to explore its impact on clients; I knew that it was incredibly innovative, that it was doing hard work that appeared vitally needed, and that I wanted to know more. It was without question that I wanted to claim the role of interviewer for myself. My academic background is in anthropology. While I knew that a life in academia was not in my future, I have been grateful for my training in social science methods. What I was doing in this project was close to participant ethnography and it rang true to me. Though short-term, as so many evaluation contracts are, I spent several days with a social worker trying to make meaningful contact with participants in this program.  

The initiative we were evaluating was designed to support the most vulnerable residents of our city. Participants were invited to the program because they were found to be the most intensive users of publicly available supports. They all had very frequent interactions with police and the courts, relied regularly on overnight shelters and other housing supports, and were heavy users of emergency departments and emergency transport services. All had experienced a great deal of trauma in their lives and were living with addictions and mental health concerns, along with various severe physical ailments.  

All of this background is leading up to a question of professional ethics — but a few more details first.  

While we had relatively unprecedented access to various quantitative datasets – thanks to stakeholders determined to collaborate and establish the right consent and information sharing agreements – this evaluation would have been incomplete without investigating the lived experience of the clients. The quantitative data, compelling though it would prove, would not speak for itself. To really tell the story of this initiative, we needed to understand the clients’ journeys through services (or lack thereof) and how this initiative was different. 

Jules’ social worker shared some of what she knew about her client. Jules had lived a frankly horrifying childhood in another province before moving to our city with her boyfriend; this boyfriend was also her abuser and her pimp. She was still young, in her early twenties, addicted to alcohol, spending nights in shelters, often beaten by people she considered friends, and yet still believing that she would be loved. Animal therapy was helpful for her, and she hoped to have children someday.  

In the previous three years, Jules had been transported by ambulance 53 times. She had visited the emergency department 81 times; 11 of those visits were for head injuries. Her hospital admissions – times where she stayed overnight on an inpatient unit – were for reasons including pneumonia, a leg ulcer, pancreatitis and alcohol withdrawal. In one year, she had critical levels of ethanol in her blood eight times. 

If it is not already clear, alcohol was an integral part of Jules’ life. 

When planning our interviews, with the project team’s advice, we aimed to schedule times that would work best for clients who may be actively using intoxicants. For most, that ideal time was between 9 am and noon. Most of our interviews were scheduled at that time, although few were conducted smoothly. One client didn’t come home the night before and the on-site staff at his permanent support house couldn’t locate him. Another was able to share some of his experiences with me but became overwhelmed and we stopped the interview; his social worker spent more time with him after I left. Yet another client needed emergency care when we arrived at his apartment. 

I should note that social workers were very involved in developing our consent and interview protocol. We did not invite any clients who had guardianship orders in place, or those who the social workers felt were not cognitively or emotionally capable of participating. Social workers discussed the evaluation and the interviews with clients before scheduling, and we reviewed information about the nature of the project, the risks and benefits of participating, before obtaining informed consent. Mental health support was always immediately available if it became necessary. 

The “sober window” we were aiming for simply didn’t exist for Jules.

At this point in her life, she was drinking alcohol continuously, and substituting hand sanitizer or mouthwash when necessary. She regularly stole alcohol and products containing alcohol, leading to regular interactions with police and several warrants. At night, she was sleeping at a shelter; in the mornings, she would vomit repeatedly before starting to drink again. Without alcohol, she could not function. 

So, when I met Jules at 9:30 on a summer morning, prepared with my recorder and questions, Jules was sipping vodka on a sidewalk surrounded by friends. We were both faced with decisions. Jules needed to decide if she wanted to speak with me, and where. I needed to decide if interviewing a client who was actively drinking alcohol was ethical. 

I’ve shared this story with participants in project ethics training courses, and at an evaluation conference. People feel very strongly about their stance on whether I made the right decision. I was rapidly running through a mental decision tree.  

Perhaps the most obvious question: can an intoxicated person give informed consent? In general, I would say no. If I was interviewing middle managers about their experiences of a workplace mentoring program and one of them was drinking or using other drugs, I would very likely reschedule. But what about when a person’s most functional state requires alcohol? Without alcohol, Jules was violently ill and could not function. 

How could I be certain that Jules was participating voluntarily? She had previously told her social worker that she wanted to participate and was telling me that she still wanted to. But how confident could I be that she knew what she wanted? We were providing gift cards as honoraria – perhaps Jules was so desperate for that money that she would participate even though she didn’t really want to. 

If I chose not to interview Jules, who would tell her story?  

As an evaluator, I feel that one of our responsibilities is to give voice to people who may not have the opportunity or ability to share their stories.

You may have already guessed which path I chose. And you may be thinking that I am a clearly unethical evaluator and no researcher worth their degrees would be so cavalier. If that’s the case, I can’t blame you. Nowhere in my training did anyone suggest that I should pursue data collection with an intoxicated person. Although, to be fair, none of my early-2000s research training had really addressed this particular scenario. It had certainly provided me with examples of how researchers have exploited vulnerable people and done irreparable harm to them.  

But I would like to defend my decision to proceed with my interview. I did consider several ethical points: 

• Voluntary participation: I truly believed that Jules wanted to talk to me. She had told her social worker so in previous days and repeated that desire when we met. At the end of the interview, she thanked me for speaking with her. The gift card, though helpful, was of a value less than what she could make through prostitution in the same time – I don’t think it was unduly coercive. 

• Informed consent: Again, Jules heard about the intention of our evaluation project and how the results would be used on more than one occasion. I do believe that though she was drinking, she did provide informed consent. I confirmed this more than once verbally before proceeding. 

• Respect for vulnerable persons: All the clients served in this initiative were highly vulnerable. By having Jules’ social worker at the interview, we ensured that any potential new disclosures could be followed up on, and that appropriate mental health supports could be immediately provided. This interview was one more opportunity for Jules to continue her relationship with her social worker, for whom she expressed gratitude many times during our conversation.   

• Giving voice: This point, of course, was what tipped the scales. If I chose not to interview Jules that morning, her voice and her journey would not have been included in our evaluation. As I’ve stated above, there simply was not a point during the day when Jules was not either violently ill or drinking; these were her two states of being at the time. To say that her voice was not valid, that it was less valuable than a sober voice, would have been unethical. You may disagree with me, but I stand firm that Jules’ voice needed to be amplified.  

Had I not proceeded with the interview, I would not have heard her perspective on how much the program team had helped her. I would not have seen how dedicated her social worker was – our interview ended with Jules vomiting and her social worker calmly and compassionately cleaning up both her and the floor around her. I would not have heard about the severe trauma Jules experienced throughout her life. Understanding the deeply disturbing past that led to Jules, and other clients, being served by this initiative was fundamental to telling the story of its impact.  

“I’m a hooker. I’m a drunk and a junkie.”

That’s how Jules described herself to me. She was physically and sexually abused for as long as she could remember. She feared seeking police support when beaten because her street friends who hurt her were who she considered her family. She did not want permanent supportive housing because she was terrified of being alone, even though when she was with others she was regularly beaten, robbed – even urinated on in the days before our interview. She drank until she blacked out, and often became violent during those blackouts. She desperately missed her incarcerated boyfriend and hoped that when he was released, he would love her the way she loved him. She enjoyed pet therapy and was effusive with praise and love for her workers.   

Without that story, Jules’ collection of statistics looks frankly unimpressive. While her interactions with police services in the six months she had been receiving supports from this initiative decreased by 26%, her visits to the emergency department increased by 71% and her use of ambulance services more than doubled. Sharing Jules’ story puts context to those results. 

I haven’t been faced with such a challenging decision since this project. But when it happens again, I am confident that I will rely on both my training and inherent respect for human dignity to guide my choice. 

---

For more on the ethics that guide evaluators, here are a few resources: 

American Evaluation Society Guiding Principles 

Australian Evaluation Society Code of Ethics and Professional Practice 

Canadian Evaluation Society Ethics 

United Nations Evaluation Group Norms and Standards  

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---