Evaluators ask questions. All the time. We ask questions in focus groups, we write questions in surveys, we pose questions to our datasets. But the questions that really drive our work are evaluation questions.

What are evaluation questions?

Evaluation questions focus data collection. They are what our stakeholders need to answer. When they have the answer to these questions, they can tell their stories. As we’ve written, evaluation questions are the high-level questions an evaluation is designed to answer.

Knowing the definition of “evaluation question” is one thing; writing them is another. It can be challenging to write questions at just the right level, that will provide guidance for choosing methods and developing data collection tools, and will actually yield the information to satisfy stakeholders.

Keep these points in mind, and you’ll be off to a good start.

Evaluation questions are informed by the evaluation purpose

Why are you doing this evaluation? Is it to support new policy development? Is it to inform a decision about spreading or contracting a program? Whatever the reason, that purpose will guide the evaluation question development. For example, an evaluation that is intended to demonstrate accountability will likely have an evaluation question around meeting the funder’s requirements.

Write evaluation questions with your stakeholders

Stakeholder engagement is key throughout evaluation projects. Working closely with program leaders and operational staff will ensure that the questions you develop together are the right questions. There is no point in writing what you think are great questions if they don’t meet stakeholders’ needs. Group writing is hard—in your evaluation planning session, don’t worry about getting every word perfect. Make sure you understand the concept that is important, then finesse the language on your own.

Stay open

Evaluation questions should be open-ended (except when they don’t need to be… see our post on why the answer to so many evaluation methodology questions is “it depends”). Open-ended questions give room for a range of possible answers.

• Close-ended question: Did participants enjoy the program?

• Open-ended question: How do participants characterize their experience?

See how that second question gives room for a range of responses beyond “yes” and “no”? This second question brings the opportunity for nuanced data that yields deeper insights; that depth is what makes a good evaluation question.

Evaluation questions are not survey questions

Survey questions are very focused, while evaluation questions are broader. Multiple survey questions may be used to answer an evaluation question. If the question you write feels like something you’ve answered before in a survey, you haven’t written an evaluation question. Climb up a level and rewrite.

• Survey question: How satisfied are you with the timeliness of the email from your support worker?

• Evaluation question: To what extent are services delivered in a timely fashion?

The data from that survey question can be one of the indicators you use to answer the evaluation question.

Evaluation questions may have multiple indicators

Strong evaluations employ triangulation; that is, multiple views on the same question. One evaluation question may be answered by a combination of two, three or more indicators, relying on multiple methods of data collection.

• Evaluation question: To what extent is the program having a positive impact on families?

• Indicators:

  • Parents’ self-reported ability to attend training classes

  • Youth mental health scores

  • Changes in number of hours spent together each week

Together, this suite of indicators provides more reliable insight into the program’s impact than one indicator alone.

How many evaluation questions?

Well, it depends. For a very comprehensive evaluation of a major initiative, more evaluation questions may be required. You may need fewer questions for a simpler project. A general guideline is between five and seven evaluation questions, but it’s not uncommon to see between three and ten. Remember, every evaluation project is different—the main goal is to ensure that stakeholders’ information needs are met, but we must also consider feasibility. If your capacity to collect data, whether through existing resources or by hiring external help, you will likely need to stick with fewer evaluation questions.

Themes can help

Evaluation questions can be clustered in themes that are relevant to the purpose of the evaluation and the nature of the initiative. For example, my evaluation firm has worked on several healthcare projects that rely on a quality matrix for health. That matrix provides a common language and shared concepts throughout healthcare partners, so we use themes like accessibility and appropriateness to guide our evaluation questions. If your organization has a strategic plan or shared goals, those may be key to guiding your evaluation question development. Or look to other frameworks, like the RE-AIM framework, for inspiration on evaluation question themes.

Edit, edit, edit, then step back

Language matters when writing good evaluation questions. Changing just one word can mean the difference between clarity and ambiguity. Use the writing process that works for you, whether that’s working on paper, consulting with a colleague, or staring each word down until you find the absolute perfect alternative. If you’re a true evaluation nerd, there is immense satisfaction in writing the very best question you can. But remember, perfection is not always possible or practical, and just like that last literature review you wrote, sometimes you just need to call it done and move on.

How do you know when you have it right?

You’ll know you have near-perfect evaluation questions when:

• Together, their answers will tell a high-level story of the initiative being evaluated

• You have between three and ten questions

• The questions cannot be answered by a simple yes/no, or by a number

• Indicators and methods are already suggesting themselves

• Your stakeholders (and you!) breathe a sigh of relief when they read them

• For an extra round of review, try this checklist from the CDC 

What comes next?

After you’ve crafted fantastic evaluation questions, you’ll move on to selecting indicators and data collection methods. In doing so, you may need to revisit your evaluation questions and make minor modifications, or even add or remove questions altogether.

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---

 

 

Recently, Jennica Nichols and Maya Lefkowich (of AND Implementation) hosted a Canadian Evaluation Society (CES) webinar about using art as a data collection method. The webinar was fun and interactive and included (you guessed it) hands-on examples of how to use arts-based techniques and how to modify them for an online audience. Without rehashing the entire webinar (CES members can re-watch it here: Using Art in Creative Data Collection and Evaluation), I wanted to share the most salient points and how we, here at Eval Academy and Three Hive Consulting, have and will put them to use. 

Why

Arts-based techniques can be used to get audiences to open-up or explore topics that can be hard to put into words. Jennica and Maya suggested using art-based methods for exploring relational meaning. In other words, they are important when: a) exploring concepts in context is important; b) needing to make connections between two distinct ideas (e.g. how the social determinants of health may mediate a program’s impacts), or; c) exploring emotions or experiences that are hard to put into words. They also noted that arts-based methods allow for many ways of knowing, moving beyond text and words to think about how things are connected in space or time or can be represented in a tactile manner.

Arts-based methods also allow participants to make more spontaneous or out-of-the-box associations between ideas. They push us out of our comfort zones and encourage different forms of expression.

What

Arts-based data collection techniques are inherently participatory methods, involving the artist in the creation and interpretation of data. They are inductive techniques, meaning that they are meant to be used for exploring ideas or describing concepts. These techniques start with observations (the art!) then work with the participants to understand the meanings and conclusions that can be drawn from the art.

There are 5 main arts-based data collection techniques:

1. Literary (e.g. poetry)

2. Performative (e.g. interpretive dance, theatre)

3. Visual (e.g. pictures, collage)

4. Audiovisual (e.g. film, video)

5. Multimedia (e.g. graphic novel, art installation)

Multimethod techniques make use of two or more arts-based methods.

In the webinar, Maya and Jennica stressed that arts-based data collection techniques are not art nor art therapy, as they aim to answer specific questions and take the information outside of the data collection space to inform decisions. In arts-based data collection techniques the description or explanation of the art is used as data, rather than the art itself.

How

Like other data collection techniques, arts-based methods require consent from participants. Because participants won’t know what they’ve created and how they feel about it being used before they’ve made it, Maya and Jennica suggest obtaining consent before the data collection begins and again once it is completed. They also suggest creating a clear consent checklist to provide participants options for how their art is used, including a discussion of if/how the participant want to be credited for what they’ve created (authorship). Check out Eval Academy’s information sheets and consent forms in our tools section – they can be downloaded and modified for this! Because the narrative behind the art is what is being evaluated, it is important to present the description alongside the art.

Before diving into creating art, it is important to develop a solid foundation with the participants. Give participants permission to be silly and creative. Maya and Jennica suggested setting the tone from the beginning of the session, tell a joke and set appropriate boundaries. Discuss the purpose of the session, let participants know what is expected of them, and how the art they create will be used. Before starting the activity, provide participants with clear prompts or questions they are to focus on when creating their art and set appropriate amounts of time for each of the activities. Too much time can cause participants to become stressed about adding more details or filling the space. Consider providing visual or auditory reminders of the prompt or question during the session to re-focus participants.

Once participants have created their artwork, the important work begins. Remember, when using arts-based data collection, the narrative or description behind the art is the data we are seeking to collect. Follow up with interviews or focus groups to understand the meaning or outcomes that came from the process of creating. Ask questions to illuminate underlying connections, assumptions, values, or ideas. 

After the process is complete, revisit consent with each of the participants. Check if and how they are ok with you sharing their art and the narrative that goes along with it. Be clear about how and where the information will be shared.

Organization Tips:

• Make sure you have all the tools you need before your session

• Don’t assume that participants have access to items such as cameras, markers, glue, or other supplies

• Prepare your questions and test the timing of your activities in advance

Tips for conducting online sessions:

• Consider supplying the questions and supplies in advance of the session. Mail participants packages or provide the log in information for online platforms so that people can become familiar with them in advance

• Build in extra time to orient people to using the online software.

• Use Zoom polls or break out rooms to encourage reflection

• Consider whether to follow up in groups or one-on-one; much like deciding between a focus group and an interview, the nature of the data you wish to collect should drive your decision

---

How We’ve Used Art-Based Techniques

Here at Eval Academy and Three Hive Consulting, we are big fans of using creative approaches in our evaluation practice. Our core values include being creative in our work, both to engage our clients and evaluation participants, and as a way to generate new ideas.

To get un-stuck and re-imagine the evaluation experience for our clients.

Recently, we opened our annual team retreat with an activity designed to help us channel our inner four-year-old to get silly and lower our creative inhibitions. Next, we doodled our way through a visioning exercise to help us re-imagine the evaluation experience for our clients. While a small flood prevented us from completing the second half of the exercise, we gained a pretty clear picture of the barriers to evaluation our clients might face.

As part of focus-groups and workshops.

We’re also a big fan of using the At My Best strengths cards which have pictures on the front and a single word on the back to do photo-elicitation techniques. We’ve used them in workshops to get participants to open up, to help jumpstart the outcome mapping process with program funders, and with health partners to develop an approach for complex patients. Interestingly, in our experiences using these cards, in every session at least one person can’t get the idea of using the photo and must flip the card over to use the words.

To understand the impacts of a program on children.

We used visual data collection methods in a feedback session with children and youth, allowing participants to give visual and verbal feedback. Children rotated through a series of flip charts with a question posed at the top. Facilitators helped the children interpret the questions and clarified the meanings of the images. One big thing we learned at these sessions were to use washable markers with children.

How we might be using these methods in the future

In a previous article, we explored using virtual reality tools to augment evaluation, including in data collection (check out: Visual Storytelling Though Augmented and Virtual Reality) and I know we are just waiting for the right project to try this out with.

As we may not be meeting in person any time soon, we can use arts-based data collection techniques to better understand our participant’s experiences. Literary, visual and audiovisual methods can create a starting point to capture and understand participants’ stories in today’s virtual world. Because we can’t be in person to build rapport, arts-based techniques can create a common and safe starting point to explore ideas with participants.

And finally, in a recent team-building effort, we took a few hours off to play Pictionary and some other online drawing games as a team. After the fun and games, we noticed a few of us were in creative and out of the box mindsets and had a pile of new ideas. Using arts-based techniques to break out of our routines and explore new ways to approach our evaluation practice will be a trick we continue to use with our team.

---

Connect with AND Implementation on social media:

Instagram: @andimplementation | Twitter: @and_implement | Website: andimplementation.ca

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---