“Start where you are. Use what you have. Do what you can.”

Working in the evaluation field is appealing in that it can take place across various sectors, systems and geographic locations. One area of particular interest for some evaluators, like myself, is program evaluation in low-resource settings (LRS). LRS, sometimes referred to as “resource poor” or “resource strained,” indicates countries or regions that lack the financial means to cover the costs associated with infrastructure, healthcare and/or trained professionals – as well as other system or societal needs.  These evaluations are often requested by funders or granting agencies as a means of providing evidence for effective use of funds. Likewise, evaluations are particularly important in LRS as programs need to operate as efficiently as possible due to limited human and material resources.

Program evaluations in a LRS can be challenging in that the program staff may not have the skills or capacity to see it through. However, agencies often require formal evaluation of their funded programs which may lead to the hiring of a contract evaluator. Take USAID for example, they provide funding to LRS programs all over the world and emphasize the need to embed evaluation in program planning. USAID values evaluation so much that they have developed policies to enforce and support agencies in meeting evaluation requirements (see USAID Evaluation Policies).

In my (very biased) opinion, I think that LRS evaluations can be complex, but in the long run will help programs as well as the population and country in which they take place. Moreover, evaluation is often viewed as something that must be done for funders, but in many/most cases is just as valuable to the program itself and it provides a source of evidence to inform program planning. As evaluators we need to advocate for the use of evaluations for both funders and their funded programs – the best way is to show what we can do by conducting meaningful evaluations. We already have the skillset to conduct evaluations of all shapes and sizes (impact, development, process, etc.) – but in a LRS we have to be a bit more creative and thrifty in our approach.

As such, I offer four strategies for success when evaluating within a LRS: Be Tech Savvy, Consider Capacity, Be Ready to Adjust, and Use Yourself as a Resource. As you explore these strategies, I am confident that many will already be a part of your practice in all settings (not just LRS). However, I would encourage you to lean in to the strategies even more when you find yourself in a LRS. Lastly, all evaluations in both high- and low-resource regions are susceptible to being thrown curveballs (take COVID-19 as an example) – meaning that as evaluators we need to be ready to engage new strategies on short notice.

Be Tech Savvy

So you find yourself in a LRS or perhaps a very low budget evaluation, but you still need to exchange information and data with stakeholders. In this case, you could even sub out LRS for “pandemic” and the following advice will still apply.

Even if the program has been using the same spreadsheet since 1995 or has a VPN that takes what seems like 2 days to connect – try to utilize existing systems. When capacity is limited, implementing new systems may result in lost time or frustration.

If you find yourself in a setting where staff are eager to have better software or have requested new systems for their information – start by being proud of your forward-thinking team/client but be cautious of available capacity for both finances and time. What I mean by this is to utilize free (or low cost) and easily accessible software before signing the organization up for a pricey software package.

• Sharing Information: Consider free tools such as G Suite or DropBox to share files and work on documents (reports, spreadsheets, presentations) simultaneously. These are less expensive (or free for basic accounts) compared to platforms such as Office 365. Note that many of these tools have offline capability meaning that you can work on the documents even when internet connection is limited.

• Communicating: The COVID-19 pandemic showed us the many available resources for connecting. In my experience, many of the free services are just as good. Zoom or GoToMeeting are popular, but only offer a free trial period or require a subscription for longer meetings. Alternatively, well-known platforms such as Skype, Google Hangouts (part of G Suite), or even Facebook through their Messenger Rooms can be used for free video calls. WhatsApp and Slack are also great apps for communicating with teams or clients. These are just a few examples of the many communication tools available to evaluators.

Consider Capacity

Evaluations can often be perceived as a capacity strain, whether it be through using staff time or the cost of hiring an external consultant. Given the capacity limitations inherent to LRS, evaluators should keep capacity at the forefront of their minds from the first meeting to the final iteration of the report. Some examples of capacity considerations during an evaluation include:

• Developing the work plan and timeline: Try to plan meetings so that they occur only when necessary and include relevant stakeholders. Regular check-in meetings may not be helpful and may take staff away from program delivery – consider 1-2 page status reports instead.

• Creating evaluation plan/questions: We often evaluate efficiency of programs, but it would also be helpful if evaluators put more of an emphasis on capacity or even a sub-section focused on capacity. Develop questions that are feasible in a LRS and likely to uncover actionable findings (not just funder-mandated metrics). Check out Eval Academy’s How to Write Good Evaluation Questions for guidance on writing evaluation questions.

• Collecting data: Try to pull from existing data, add questions to regularly administered surveys, or plan focus groups for days where stakeholders are already in the same location. If appropriate, consider methodologies such as Participatory Action Research (PAR) to build evaluation/research capacity of staff (Baum et al., 2006).

• Presenting findings & recommendations: Include capacity building or capacity considerations for all recommendations – after all, recommendations are not likely to be adopted or sustained without capacity.

Whether you are an internal or external evaluator in a LRS, the program or agency should feel like you contributed capacity whether it be through the new knowledge, sound recommendations or development of internal evaluation capacity.

Be Ready to Adjust

As evaluators we like plans and go into projects with a clear timeline and guide of what we would like to accomplish. Does it always go as planned? Definitely not! In all engagements we need to be ready to pivot or adjust our plan – this is even more true for evaluations in LRSs.

As I write this article, we are still deep in the uncertainty of COVID-19. Although many evaluators are already comfortable with remote working, many aspects of conducting an evaluation have had to change. Couple this with the complexities of a LRS and there is no choice but to be flexible and ready to adjust evaluations accordingly.

I expect that evaluators will have many successes (and some failures) to share about their evaluation practice during these times. For now, I will offer a couple suggestions that I have found to be helpful working on a LRS project during a pandemic.

• Information gathering: Focus groups, interviews and surveys may need to be facilitated online. Likewise, program staff may need to extract data and share with the evaluator remotely. Workshop facilitation may require the evaluator to search for facilitation tools embedded in video chat platforms (such as surveys) or collaborative tools (such as a shared whiteboard).

• Focus on the most important aspects: Some key evaluation topics or focus areas may need to change. Consider resource constraints or competing priorities (pandemic related or not) and the impact on the program you are evaluating. Ask questions like:

  • Are the evaluation topics/questions still valid?

  • Are the various evaluation phases feasible remotely and/or with less access to staff and the population being served?

  • How can evaluation evidence support the program as it adapts for COVID-19 or other resource constraints?

• Revisit the timeline: Expect the timeline to change – whether it is rescheduling some of the meetings, changing them to virtual meetings, or completely revising the project work plan. For example, if the evaluation capacity is severely limited due to a pandemic or other LRS obstacles, look at postponing until there is more capacity. It may be better to postpone or extend the timeline rather than sacrificing the quality of the evaluation.

Use Yourself as a Resource

In a LRS, answers to your questions or material to inform both the planning and execution of an evaluation may not be easily accessible (or in some cases may not be known). As an evaluator for a LRS program, internal or external, you will likely need to be your own resource when it comes to finding data, identifying stakeholders or to develop a general understanding of the program. Hopefully the program or agency at the center of the evaluation will be willing to share all relevant documents and offer some context – but in most cases you will need to be ready to dive in to uncover more. Here are a couple of tips to further explain this strategy:

• Hands-on learning: Rather than taking the capacity (resources or staff) to learn about the program or agency being evaluated, consider shadowing or observing the program (activities and meetings). This will prevent the evaluation from eating up too much staff time and I would argue that it will also provide you with a richer understanding of program being evaluated.

• Self-led professional development: There may be minimal options for professional development in LRS, especially if you are working as an internal evaluator. Connecting with other evaluators or professionals working on LRS projects is a great place to start. There may be existing Communities of Practice (in-person or virtual) or in my experience individuals working in the same region are more than happy to share their experiences. For evaluation specific education or new methods (even seasoned evaluators need some inspiration every once in a while!) consider online resources, like EvalAcademy.

These strategies were summarized with LRS evaluations in mind, but I am confident they can be adopted for evaluation projects in all settings. If you have any other resources or strategies for evaluating in a LRS, please comment below.

 

Resource

Baum F, MacDougall C, Smith D. Participatory action research. J Epidemiol Community Health. 2006;60(10):854–857. doi:10.1136/jech.2004.028662

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---

 

 

To skip right to the free guide, check out our New Products page.

Many evaluators will already be familiar with the Program Evaluation Standards developed by the Joint Committee on Standards for Educational Evaluation (JCSEE). For those newer to the field, take comfort in knowing that evaluation has this set of Standards to guide your way forward. The Standards provide guidance both for evaluators in planning and implementing their program evaluation projects, and for evaluation users in knowing what to expect from the evaluation process and products.

Evaluation practitioners come to their work through diverse academic and practice backgrounds. We may identify primarily as evaluators, or as a myriad of other job titles: program manager, executive director, assessment coordinator, quality improvement director, research assistant or data scientist, to name a few. The Program Evaluation Standards help to bring all of that professional diversity together into a field that has a common language and common expectations.

Developed and revised by experts, and with sponsoring organizations including the American Evaluation Association and the Canadian Evaluation Society, the Standards are published in a comprehensive guide that we recommend all evaluators keep in their library.

Through developing and delivering evaluation training, we know the value of short guides for translating concepts to practice. That’s why we developed this free resource that helps evaluators reflect on whether and how they are applying the Standards to their practice.

Our 6-page resource provides evaluators with reflective questions for each of the Standards. We suggest that you read through these questions as part of your evaluation planning process or use them to guide a self-reflective exercise after your evaluation project has concluded. For anyone working toward their Credentialed Evaluator designation through the Canadian Evaluation Society, this guide will support you to consider the Reflective Practice competencies.

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---

Reference: Yarbrough, D.B., Shula, L.M., Hopson, R.K., & Caruthers, F.A. (2010). The Program Evaluation Standards: A guide for evaluators and evaluation users (3rd. ed). Thousand Oaks, CA: Corwin Press.

 

 

Recently I was asked by a client about an evaluation literacy course for its board. The client’s board members had just attended a strategic planning day and through that discussion felt they needed education on evaluation and metrics. On one hand I thought “bravo, they want to know more about evaluation!”; on the other hand I thought “shit…., I’ve totally failed them as their evaluator – what have I been missing?”

Boards need quality information to make strategy and leadership decisions; however, the reality is this board isn’t getting the information it needs to inform its decisions. As their evaluator, it is my responsibility (and also opportunity) to show them the way forward, so they are no longer left with answers that are “a definite ‘maybe,’” but instead have data and insights that they can use to inform their decision making. This means a more systematic, coordinated, and intentional approach to evaluation and learning – a strategic learning and evaluation system (SLES), as is described in FSG’s Building a Strategic Learning and Evaluation System for your Organization. The course I developed therefore focused not only on evaluation literacy, but also how evaluation can support a SLES. The course has three overall objectives for its board members:

1. Understand the necessity of and advocate for strategic learning;

2. Understand the basics of evaluation and how it can support strategic learning, and;

3. Begin developing the building blocks for a strategic learning and evaluation system.

Here are some of the key learnings and action imperatives for the board: 

Understand that evaluation is one piece of the learning pie

While evaluation is important for learning and improvement, it is only one of many information-gathering approaches that can be used to inform decision making about strategy. Organizations also collect information through performance measurement, audits, research, case studies, discussions at the water cooler and a number of other ways. So, if the ultimate goal is for organizations to learn and use that information to improve, boards need to shift thinking from “leading with evaluation to leading with learning” (Centre for Evaluation Innovation).

---

Understand and advocate for strategic learning

I am lucky that my client’s board has a desire to learn more about evaluation and how it can support strategy development. The Centre for Evaluation Innovation recently conducted a survey to collect data on evaluation and learning practices in foundations and found that senior management often communicate support for evaluation, but their behaviours do not demonstrate support for evaluation. So what are the board behaviours that would demonstrate support for evaluation? And even more specifically, evaluation for strategic learning? 

The Centre for Evaluation Innovation produced a report titled, Evaluation to Support Strategic Learning: Principles and Practices, in this report they explain that “designing data collection and evaluation specifically to support strategy decisions requires shifts in thinking about what questions get asked, the role the evaluator plays, how data collection is timed, and the framing of the findings” (pg. 3). They go on to articulate nine principles of evaluation for strategic learning that boards can advocate for within their organization:

1. A support for strategy

2. Integrated and conducted in partnership

3. Emphasizes context

4. Client focused

5. Places high value on use, and helps to support it

6. Data to inform strategy can come from a wide variety of sources and methods

7. Must take place within a culture that encourages risk taking, learning and adaptation

8. Is flexible and timely and ready for the unexpected

9. Is constructivist

Understand the basic evaluation terms and steps

Part of advocating for strategic learning is understanding basic evaluation concepts and terms. If learning and evaluation efforts are to inform an organization’s decision-making practices, then boards need a clear vision for evaluation – what it is and is not.

I’m not going to lie – I don’t believe in reinventing the wheel, and Chris Lovato and Kylie Hutchinson put together an Evaluation for Leaders course. Much of what I covered in the evaluation basics module of my course follows what they outline for evaluation terms, types and steps. However, as Chris and Kylie say in their course:

 

“Leaders and decision makers don’t need to be evaluation experts, just expert supporters and users.”

 

So, in my course there is an emphasis on the first evaluation step – focus. For boards it is important to understand this step. An evaluation can’t be all things to all people – focusing it provides a clear direction of who needs what information and how the user(s) are going to use it.

Understand what is credible evaluation evidence

Thinking has changed on what constitutes “credible” evaluation evidence. Many board members from this organization come from a science background that believes RCTs (randomized control trials) are the gold standard for evaluation evidence. This is a common misconception, but one that is particularly important to address with board members. As the Evaluation Yoda, Michael Quinn Patton states:

 

“Despite the recognition more than 35 years ago that the reductionist approach to complex problems is likely to fail, many still persist in believing that we must rigorously apply the scientific method to problems in medicine and public health.”

 

If a board is trying to impact systems and trying to shift the conditions that are holding problems in place (i.e. systems change) then it is important that the board shift their thinking from measuring and proving against some sort of fixed model to understanding and improving – in other words, to adopt a systems thinking lens. The evaluation methods that are selected to evaluate that change comes back to how appropriate the methods are given the purpose of the evaluation, the questions it needs to answer, and how technically adequate the findings are given the time and cost constraints. As the United States General Accounting Office Program Evaluation and Methodology Division (1991, pg. 17) quotes:

 

“A strong study is technically adequate and useful – in short, it is high quality”

 

Know what you want/need and communicate it to your organization

If a board is not clear on what information it needs to inform its decisions, you can be sure the rest of the organization won’t either. A board gets a lot of information, but it may not:

• Contain the right information,

• Be presented in a useful and useable format,

• Be received on time (i.e. after a decision has been made), or;

• Be connected to organizational strategy, which means the findings aren’t getting used (or at least not fully). 

An easy first step to enhance use of evaluation findings is for boards to make their timelines and reporting preferences known. A more difficult next step a board should consider when trying to enhance usability of findings is to implement a SLES.

According to FSG’s Building a Strategic Learning and Evaluation System for your Organization, a SLES contains:

1. A clear vision for evaluation,

2. A culture that fosters individual, group and organizational learning,

3. A compelling and strong strategy,

4. Coordinated evaluation and learning activities, and;

5. A supportive environment.

Bottom line – a SLES will provide guidance and align organizations on who, what, when, where, why and how to measure and report.

I left my client a lot to chew on. As I mentioned above, implementing a SLES will be difficult, but ultimately should provide the board with an evaluation strategy that increases the value of evaluation for its organization.

Interested in learning more? Sign up for our newsletter and we’ll notify you when the online version of this course is available.

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---

 

 

Evaluators ask questions. All the time. We ask questions in focus groups, we write questions in surveys, we pose questions to our datasets. But the questions that really drive our work are evaluation questions.

What are evaluation questions?

Evaluation questions focus data collection. They are what our stakeholders need to answer. When they have the answer to these questions, they can tell their stories. As we’ve written, evaluation questions are the high-level questions an evaluation is designed to answer.

Knowing the definition of “evaluation question” is one thing; writing them is another. It can be challenging to write questions at just the right level, that will provide guidance for choosing methods and developing data collection tools, and will actually yield the information to satisfy stakeholders.

Keep these points in mind, and you’ll be off to a good start.

Evaluation questions are informed by the evaluation purpose

Why are you doing this evaluation? Is it to support new policy development? Is it to inform a decision about spreading or contracting a program? Whatever the reason, that purpose will guide the evaluation question development. For example, an evaluation that is intended to demonstrate accountability will likely have an evaluation question around meeting the funder’s requirements.

Write evaluation questions with your stakeholders

Stakeholder engagement is key throughout evaluation projects. Working closely with program leaders and operational staff will ensure that the questions you develop together are the right questions. There is no point in writing what you think are great questions if they don’t meet stakeholders’ needs. Group writing is hard—in your evaluation planning session, don’t worry about getting every word perfect. Make sure you understand the concept that is important, then finesse the language on your own.

Stay open

Evaluation questions should be open-ended (except when they don’t need to be… see our post on why the answer to so many evaluation methodology questions is “it depends”). Open-ended questions give room for a range of possible answers.

• Close-ended question: Did participants enjoy the program?

• Open-ended question: How do participants characterize their experience?

See how that second question gives room for a range of responses beyond “yes” and “no”? This second question brings the opportunity for nuanced data that yields deeper insights; that depth is what makes a good evaluation question.

Evaluation questions are not survey questions

Survey questions are very focused, while evaluation questions are broader. Multiple survey questions may be used to answer an evaluation question. If the question you write feels like something you’ve answered before in a survey, you haven’t written an evaluation question. Climb up a level and rewrite.

• Survey question: How satisfied are you with the timeliness of the email from your support worker?

• Evaluation question: To what extent are services delivered in a timely fashion?

The data from that survey question can be one of the indicators you use to answer the evaluation question.

Evaluation questions may have multiple indicators

Strong evaluations employ triangulation; that is, multiple views on the same question. One evaluation question may be answered by a combination of two, three or more indicators, relying on multiple methods of data collection.

• Evaluation question: To what extent is the program having a positive impact on families?

• Indicators:

  • Parents’ self-reported ability to attend training classes

  • Youth mental health scores

  • Changes in number of hours spent together each week

Together, this suite of indicators provides more reliable insight into the program’s impact than one indicator alone.

How many evaluation questions?

Well, it depends. For a very comprehensive evaluation of a major initiative, more evaluation questions may be required. You may need fewer questions for a simpler project. A general guideline is between five and seven evaluation questions, but it’s not uncommon to see between three and ten. Remember, every evaluation project is different—the main goal is to ensure that stakeholders’ information needs are met, but we must also consider feasibility. If your capacity to collect data, whether through existing resources or by hiring external help, you will likely need to stick with fewer evaluation questions.

Themes can help

Evaluation questions can be clustered in themes that are relevant to the purpose of the evaluation and the nature of the initiative. For example, my evaluation firm has worked on several healthcare projects that rely on a quality matrix for health. That matrix provides a common language and shared concepts throughout healthcare partners, so we use themes like accessibility and appropriateness to guide our evaluation questions. If your organization has a strategic plan or shared goals, those may be key to guiding your evaluation question development. Or look to other frameworks, like the RE-AIM framework, for inspiration on evaluation question themes.

Edit, edit, edit, then step back

Language matters when writing good evaluation questions. Changing just one word can mean the difference between clarity and ambiguity. Use the writing process that works for you, whether that’s working on paper, consulting with a colleague, or staring each word down until you find the absolute perfect alternative. If you’re a true evaluation nerd, there is immense satisfaction in writing the very best question you can. But remember, perfection is not always possible or practical, and just like that last literature review you wrote, sometimes you just need to call it done and move on.

How do you know when you have it right?

You’ll know you have near-perfect evaluation questions when:

• Together, their answers will tell a high-level story of the initiative being evaluated

• You have between three and ten questions

• The questions cannot be answered by a simple yes/no, or by a number

• Indicators and methods are already suggesting themselves

• Your stakeholders (and you!) breathe a sigh of relief when they read them

• For an extra round of review, try this checklist from the CDC 

What comes next?

After you’ve crafted fantastic evaluation questions, you’ll move on to selecting indicators and data collection methods. In doing so, you may need to revisit your evaluation questions and make minor modifications, or even add or remove questions altogether.

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---

 

 

Recently, Jennica Nichols and Maya Lefkowich (of AND Implementation) hosted a Canadian Evaluation Society (CES) webinar about using art as a data collection method. The webinar was fun and interactive and included (you guessed it) hands-on examples of how to use arts-based techniques and how to modify them for an online audience. Without rehashing the entire webinar (CES members can re-watch it here: Using Art in Creative Data Collection and Evaluation), I wanted to share the most salient points and how we, here at Eval Academy and Three Hive Consulting, have and will put them to use. 

Why

Arts-based techniques can be used to get audiences to open-up or explore topics that can be hard to put into words. Jennica and Maya suggested using art-based methods for exploring relational meaning. In other words, they are important when: a) exploring concepts in context is important; b) needing to make connections between two distinct ideas (e.g. how the social determinants of health may mediate a program’s impacts), or; c) exploring emotions or experiences that are hard to put into words. They also noted that arts-based methods allow for many ways of knowing, moving beyond text and words to think about how things are connected in space or time or can be represented in a tactile manner.

Arts-based methods also allow participants to make more spontaneous or out-of-the-box associations between ideas. They push us out of our comfort zones and encourage different forms of expression.

What

Arts-based data collection techniques are inherently participatory methods, involving the artist in the creation and interpretation of data. They are inductive techniques, meaning that they are meant to be used for exploring ideas or describing concepts. These techniques start with observations (the art!) then work with the participants to understand the meanings and conclusions that can be drawn from the art.

There are 5 main arts-based data collection techniques:

1. Literary (e.g. poetry)

2. Performative (e.g. interpretive dance, theatre)

3. Visual (e.g. pictures, collage)

4. Audiovisual (e.g. film, video)

5. Multimedia (e.g. graphic novel, art installation)

Multimethod techniques make use of two or more arts-based methods.

In the webinar, Maya and Jennica stressed that arts-based data collection techniques are not art nor art therapy, as they aim to answer specific questions and take the information outside of the data collection space to inform decisions. In arts-based data collection techniques the description or explanation of the art is used as data, rather than the art itself.

How

Like other data collection techniques, arts-based methods require consent from participants. Because participants won’t know what they’ve created and how they feel about it being used before they’ve made it, Maya and Jennica suggest obtaining consent before the data collection begins and again once it is completed. They also suggest creating a clear consent checklist to provide participants options for how their art is used, including a discussion of if/how the participant want to be credited for what they’ve created (authorship). Check out Eval Academy’s information sheets and consent forms in our tools section – they can be downloaded and modified for this! Because the narrative behind the art is what is being evaluated, it is important to present the description alongside the art.

Before diving into creating art, it is important to develop a solid foundation with the participants. Give participants permission to be silly and creative. Maya and Jennica suggested setting the tone from the beginning of the session, tell a joke and set appropriate boundaries. Discuss the purpose of the session, let participants know what is expected of them, and how the art they create will be used. Before starting the activity, provide participants with clear prompts or questions they are to focus on when creating their art and set appropriate amounts of time for each of the activities. Too much time can cause participants to become stressed about adding more details or filling the space. Consider providing visual or auditory reminders of the prompt or question during the session to re-focus participants.

Once participants have created their artwork, the important work begins. Remember, when using arts-based data collection, the narrative or description behind the art is the data we are seeking to collect. Follow up with interviews or focus groups to understand the meaning or outcomes that came from the process of creating. Ask questions to illuminate underlying connections, assumptions, values, or ideas. 

After the process is complete, revisit consent with each of the participants. Check if and how they are ok with you sharing their art and the narrative that goes along with it. Be clear about how and where the information will be shared.

Organization Tips:

• Make sure you have all the tools you need before your session

• Don’t assume that participants have access to items such as cameras, markers, glue, or other supplies

• Prepare your questions and test the timing of your activities in advance

Tips for conducting online sessions:

• Consider supplying the questions and supplies in advance of the session. Mail participants packages or provide the log in information for online platforms so that people can become familiar with them in advance

• Build in extra time to orient people to using the online software.

• Use Zoom polls or break out rooms to encourage reflection

• Consider whether to follow up in groups or one-on-one; much like deciding between a focus group and an interview, the nature of the data you wish to collect should drive your decision

---

How We’ve Used Art-Based Techniques

Here at Eval Academy and Three Hive Consulting, we are big fans of using creative approaches in our evaluation practice. Our core values include being creative in our work, both to engage our clients and evaluation participants, and as a way to generate new ideas.

To get un-stuck and re-imagine the evaluation experience for our clients.

Recently, we opened our annual team retreat with an activity designed to help us channel our inner four-year-old to get silly and lower our creative inhibitions. Next, we doodled our way through a visioning exercise to help us re-imagine the evaluation experience for our clients. While a small flood prevented us from completing the second half of the exercise, we gained a pretty clear picture of the barriers to evaluation our clients might face.

As part of focus-groups and workshops.

We’re also a big fan of using the At My Best strengths cards which have pictures on the front and a single word on the back to do photo-elicitation techniques. We’ve used them in workshops to get participants to open up, to help jumpstart the outcome mapping process with program funders, and with health partners to develop an approach for complex patients. Interestingly, in our experiences using these cards, in every session at least one person can’t get the idea of using the photo and must flip the card over to use the words.

To understand the impacts of a program on children.

We used visual data collection methods in a feedback session with children and youth, allowing participants to give visual and verbal feedback. Children rotated through a series of flip charts with a question posed at the top. Facilitators helped the children interpret the questions and clarified the meanings of the images. One big thing we learned at these sessions were to use washable markers with children.

How we might be using these methods in the future

In a previous article, we explored using virtual reality tools to augment evaluation, including in data collection (check out: Visual Storytelling Though Augmented and Virtual Reality) and I know we are just waiting for the right project to try this out with.

As we may not be meeting in person any time soon, we can use arts-based data collection techniques to better understand our participant’s experiences. Literary, visual and audiovisual methods can create a starting point to capture and understand participants’ stories in today’s virtual world. Because we can’t be in person to build rapport, arts-based techniques can create a common and safe starting point to explore ideas with participants.

And finally, in a recent team-building effort, we took a few hours off to play Pictionary and some other online drawing games as a team. After the fun and games, we noticed a few of us were in creative and out of the box mindsets and had a pile of new ideas. Using arts-based techniques to break out of our routines and explore new ways to approach our evaluation practice will be a trick we continue to use with our team.

---

Connect with AND Implementation on social media:

Instagram: @andimplementation | Twitter: @and_implement | Website: andimplementation.ca

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---