As evaluators, we are rarely organizational decision-makers; it is our job to provide those decision-makers with actionable insights. In this article I highlight how you can translate data into meaningful findings, or insights, so you can support decision-makers to drive action within their organizations.

---

Asking the right questions

The process of deriving actionable insights from data starts with asking the right evaluation questions. What do your clients need to answer to tell their stories? Evaluation questions are the starting point to any analysis and the answers are the end point. As the anchors of your analysis, it is crucial to dedicate time with your clients to iron out these evaluation questions; they will provide needed context to all results garnered from the analysis. 

If you are struggling with writing evaluation questions, we have previously written about how to write evaluation questions (with sample evaluation questions). Refer to these articles for more details on establishing effective evaluation questions. 

Start with the data that you have

Make life easier on yourself: start with the data that you have. Data collection takes time. Rather than expending a bunch of resources on data collection, evaluate whether current data sources are sufficient to answer the evaluation questions.  

However, there will be instances when the data required to address the evaluation questions do not exist. In this case, you may need to develop data collection tools (e.g., surveys, interviews) to collect relevant data. Keep things simple and focus on the evaluation questions. Anchoring the data and analysis in your evaluation questions maintains focus, limiting the ability for a project’s scope to creep beyond what was originally agreed upon.  

Which data collection tool you select will depend on the evaluation question and your evaluation design or approach. However, surveys are usually a quick and cost-effective method for collecting data. For example, your evaluation question may ask: “To what extent do patients have a positive experience with primary care programs and services?” To answer this question, we could design a patient survey. The survey could include questions on satisfaction with specific programs and services or overall satisfaction with primary care. The survey should ask questions that will directly address the overall evaluation question. Fewer direct questions are recommended over many tangential or unrelated questions. 

While this is a simplified example, a survey is not limited to answering a single evaluation question. Survey tools can be designed to capture data for one or many evaluation questions. The key is to make sure all questions align with your evaluation questions; this will focus the survey and capture data relevant to the overall goal of your evaluation. 

“Garbage in, garbage out”

The results of any data analysis are only as good as the data themselves. If data quality is not ensured, the results of your analysis will be suspect and likely invalid. It is critical that data are scrutinized prior to analysis to establish confidence in the results and insights drawn from the analysis. Therefore, prior to analysis, data quality needs to be evaluated on: 

• Completeness – are the data sufficiently complete to address your evaluation questions? 

• Accuracy – do the data correctly reflect the data being collected? 

• Consistency – do data reflect the same information within and across data sources? 

• Validity – do the data align with pre-determined conditions/ formats? 

• Uniqueness – are data represented once within a given data set? 

• Timeliness – are data up to date to adequately address your evaluation questions? 

Likely, data will not meet all dimensions of data quality right away. Some dimensions of data quality can be fixed with simple data cleaning (e.g., correcting minor typos and formatting dates). Other times data points may be excluded from the analysis. However, it is crucial that the data meet all dimensions of data quality prior to analysis to ensure accurate results.

Data to information

Spreadsheets, regardless of their size and complexity, only store data. That is, a spreadsheet does not provide any meaningful information until the data are structured and organized in a meaningful way. Analysis takes the data building blocks and structures them into something useful (i.e., information). This information will, again, be tied back to the evaluation questions outlined prior to the analysis. 

Information may be summarized as numbers (e.g., proportions, tables) or images (e.g., charts, infographics). How information is structured and presented are dependent on the context of the evaluation questions asked. The key is to provide information that is simple and easy to interpret. 

Sample “information”: 

Information should focus on meanings. What do the data illustrate? How does the information connect to the evaluation questions? This is accomplished by focusing the information. That is, focus on one major point per piece of information. By narrowing the focus, you are better able to communicate that information with decision-makers.

Actionable insights

Now that data have been converted into information, it is time to take that information and transform it into actionable insight. Actionable insights come from taking the information gleaned from an analysis and getting at the “so what?” 

 Getting at the “so what?” is not always easy. But there are a few approaches to move insight to actionable insight, including: 

• Segmenting (or grouping) the results 

• Using data visualizations to support the results 

• Comparing to benchmarks (e.g., time series, norms) 

• Adding additional context 

Segmentation 

Segmenting data into discernable groups can help get at the “so what?” Segments, such as demographics, split the results of the analysis into comparable groups. Which segments you investigate are dependent on the evaluation questions asked. 

Looking within an organization? Segment by department to derive insight into potential departmental differences. 

Looking at financial literacy outcomes? Segment by age or gender to derive insight into potential learner differences. 

Segmenting the information derived from your analysis may help identify patterns in the results. Patterns may identify important differences between segments that will allow for the client to better develop an action plan. 

 

Data visualizations 

Data visualizations, such as charts and infographics, do not inherently provide actionable insights. However, they can provide additional support for the key findings of an analysis. Effective data visualizations can highlight key messages within the data and help identify areas for action. 

Take this result: 80% of patients were satisfied with their last visit.  

On its own, we only have one piece of the story. Did the remaining 20% of patient feel neutral about their last visit? Or were they very dissatisfied? For this example, providing a chart with the statement can provide additional context. Knowing that 20% of patients were dissatisfied with their last visit is likely to spur more action than if the patients had neutral feelings about their last visit. 

 

Benchmarks 

Further insights may be gleaned from benchmarks. These may be internal (e.g., comparing between time points) or external (e.g., comparing to standard norm). Using benchmarks can get at the “so what?” and provide valuable context to the results of an analysis. 

Looking at the previous example, exploring the results over time could provide additional context. For example, if 100% of patients were satisfied with their last visit in 2020 and 80% of patients were satisfied with their last visit in 2021, we can immediately identify a decrease in patient satisfaction. However, if 60% of patients were satisfied with their last visit in 2020, we would likely see a different response from the client. Providing results with the additional context of a benchmark has the potential to turn information into an actionable insight. 

 

Additional context 

As evaluators, it is not necessarily within the scope of our role to expand beyond what is provided in the data. Sometimes the data do not fully lend themselves to actionable insights. These cases require additional context beyond the data.  

At this point, it is time to hand the results off to your client. Your client will have a better understanding of internal operations, processes, or biases within their organization. Their expertise can provide additional context not apparent from the data alone and the client can come up with their own conclusions based on the results. 

---

The roadmap for transforming data into actionable insights starts and ends with asking the right evaluation questions. These questions guide the entire analysis process, moving data to information and information to actionable insight. The goal is derive meaning from data and answer the “so what?” questions to help organizations target areas for action.

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---

Sources:

How to Write Good Evaluation Questions 

Evaluation Question Examples 

How to Conduct Interviews 

Scope Creep: When to Indulge It, and When to Avoid It  

Dial Down Your Data 

7 Tips for Better Data Visualizations 

How (and Whether) to Write Recommendations

What Is Data Quality and Why Is It Important?  

 

 

Want to hear my elevator speech about what an evaluator does? It goes something like this: 

“What do you do?” 

“I’m an evaluator.” 

“What’s that?” 

“When an organization runs a program or initiative and they want to know how well they did, they hire an evaluator to help them measure and report back.” 

“Cool, so you tell them how to do better?” 

“Uhh…sometimes.” 

See how it kind of falls apart at the end there? It’s not that I don’t know my own work, it’s that the scope of the evaluator role has been bouncing around in my brain for a while now. Do we tell programs how to improve? Or do we simply share the data with them and let them draw their own conclusions? 

---

Does an evaluator make recommendations?

I had a mentor for the first few years of my evaluation career. He had an answer to this: “No.”  He firmly believed that an evaluator’s role is to stay neutral, design data collection strategies, implement them, analyze them and report back in a utilization-focused way.  

Michael Quinn Patton would argue there are definitely situations where an evaluator should be anything but neutral, like in evaluating complex innovations where evaluation feeds into design, but that’s not really what I’m driving at here. So, in a traditional formative/summative evaluation, what is the scope of an evaluator’s role: should an evaluator make recommendations? 

Let’s argue both sides of this: 

Pro Recommendations: 

• Clients look for recommendations, it is expected 

• Recommendations are the “now what” to reporting results.  

• You’ve gathered and analyzed data, you’ve interpreted the data (the “so what”), so recommendations are the next logical progression 

Con Recommendations: 

• The evaluator is not the expert  

• Imagine recommending that a program change approach (think harm reduction vs. abstinence, or patient-centred care vs. physician-directed care), undermining the expertise and approach being tested 

• The evaluator does not know the entire context  

• Imagine recommending that an organization hire a Communications Advisor to help in getting their message out and build awareness, only to learn that their funding has recently been cut in half 

So where does that leave us?

One option is to side-step the issue: hearing my mentor’s voice in my head, I have written reports that stop just shy of “recommendations” but definitely include “Key Lessons Learned.”

So, I might say: 

“Training attendance was better on weekday mornings.”  

Rather than saying: 

“Recommendation #1. Run training sessions on weekday mornings.” 

ARE WE splitting hairs? Is there an actual difference here? 

Part of the difference is in human psychology. Not only do people generally not like being told what to do, but change management literature says that people will engage in change more readily if they come up with the idea themselves. So, if you say, “Training was better in the morning” and the project lead says, “Hey, we should run all sessions in the morning” – they get the credit, and the change is more likely to happen and be sustained. 

Another option might be to use softer language, so instead of: 

“Recommendation #1. Run training sessions on weekday mornings.” 

I might try to soften the language: 

“Consider alternative delivery dates and times. Review methods for optimizing the schedule of delivery based on participant needs and trainer capacity.” 

Again, this feels a little like cheating. And it’s not answering the questions about should an evaluator make recommendations. I think it’s because ultimately it depends on the relationship between the evaluator and the client, the strength of the evidence, the level of knowledge of the evaluator, and the integration of the evaluator within the operations.  

Michael Scrivens tells us that “lessons learned—of whatever type—should be sought diligently, expressed cautiously, and applied even more cautiously.” Scrivens suggested that “micro recommendations” which offer commentary or suggestions about implementation or operational details may be very appropriate within a formative evaluation while macro-recommendations – think “adopt, adapt or abandon”  –  are not necessarily the role of the evaluation and should not be made unless the evaluator has extensive knowledge and knows the context well.  

Micro and macro recommendations describe two types of recommendations, but there are others. Not all recommendations are created equal. Recommending that the coffee vendor at the conference be swapped for better quality coffee is not really the same as recommending that a program close down due to poor outcomes. Yet another type of recommendation (to avoid) is the less-than-helpful “More research is needed,” which can be valid but probably isn’t what your client is looking for.   

So, if your client wants recommendations, or perhaps your role as the evaluator is integrated and knowledgeable enough to warrant recommendations, consider the Following….recommendations (see what I did there?):

---

Drafting Recommendations

1. Plan early! 

• When you are scoping an evaluation with your clients ask, “Are you seeking recommendations?” Knowing their expectations can frame the purpose of the evaluation and prepare you to deliver a product that meets their needs. 

• When you are working with your client to develop the key evaluation questions to frame your evaluation plan, ask your stakeholders “What would you do differently if the outcome is A? What if it was B?” 

2. Engage stakeholders in coming up with recommendations

• Share results early and often with your key stakeholders. Conduct a sense-making session to review the findings and gather their insights on what actions may be feasible as a result of lessons learned. This is sort of the best of both worlds, technically your report will include recommendations but also technically they didn’t just emerge from your brain! 

3. Validate draft recommendations

• Are they actionable? Feasible? Under the control of the stakeholders? 

• Consider conducting a brief literature review, or comparing to an existing review. 

4. Make sure they are justifiable

• Link your recommendations directly and clearly to the data that drives it. 

• If possible, seek out multiple data sources that align around the same path forward. This includes perspectives from multiple stakeholders at multiple levels. Think about who is the decision-maker and who would be responsible for implementation. 

---

Presenting Recommendations

1. Don’t bury them in the narrative of a report! 

• Think about where to include your recommendation in the final report. Some writers like to showcase them upfront as a quick highlight or summary of the report. Some people like to save them for the end after the results have been presented in full. Other people will embed recommendations throughout the report, often below the data that informed them. In any case, ensure they stand out clearly as recommendations and not hard-to-find, half-hearted suggestions. 

2. Consider if there is a way to group recommendations by urgency, by ease, by impact, or by content theme

3. Though ideally you’ll have engaged stakeholders in crafting the recommendations, it’s always good practice to facilitate a discussion afterward as well

• Help the team to do some visioning around what would happen if the recommendation was actioned. Is there evidence you can share? Are there indicators you can offer to help the client measure the impact? 

4. Be concise

• Be clear, but not directive.  

• Try to limit your recommendations to those that are most valid and actionable. 

• Avoid lumping several recommendations into one. 

---

Other Considerations 

1. If possible, give options

• Perhaps the data showed that mornings are better for training. One recommendation could be to run training in the morning, but alternatively, if you address the barrier about why training was better in the morning you may find an alternative recommendation is “Offer lunch vouchers and free parking to training sessions in the afternoon.” 

2. If you are an internal evaluator, try to build in time to follow up after your recommendations have been made 

---

Making poor recommendations can undermine your credibility as an evaluator, but conversely, strong, relevant recommendations are valuable and likely exactly what your client is looking for.  

If you’re looking for more tips on how to craft an amazing evaluation report, check out our 6-part series on Renovating Your Evaluation Report or how to deliver less-than-stellar results to your clients. 

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---

 

 

Evaluators do not often share their evaluation reports. The organization or client that the evaluator works for usually owns the report, which can make sharing them difficult. But sometimes it can be helpful to see what and how information is presented in an evaluation report.

A few years ago, we completed an evaluation for a breast cancer clinic. In honour of Breast Cancer Awareness month, we thought we would highlight some excerpts from that report to help inspire your next evaluation report!

---

1. Describing the evaluation

In our reports we like to acquaint the audience with an overview of the evaluation near the beginning of the report. In the evaluation overview, we try to provide a high level understanding of the evaluation plan and how it was developed.

For this evaluation, the stakeholders requested we include an abbreviated stakeholder matrix that we co-created at the beginning of our evaluation planning.  

 An evaluation can’t be all things to all people, and by including a stakeholder matrix identifying the primary intended users, we showed that all stakeholders were considered and involved in the evaluation in some way (as outlined in the “nature of involvement” column).

This evaluation had four focus areas:  

1. Describing the model of care,  

2. Reporting on patient outcomes,  

3. Reporting on health provider outcomes, and  

4. Describing the costs.  

Each focus area had corresponding evaluation questions. These questions were answered using mixed methods. We don’t want to overwhelm the reader by going into the detailed minutia of our methods, so usually we append that at the back of the report and include a high level overview of our data sources near the beginning of the report.

2. Describing the clinic

In this evaluation we needed to describe the clinic and its model of care (see evaluation questions under model of care). We did so using narrative and a number of figures, including the ones below.

This figure was created using PowerPoint and shows a breast cancer patient’s journey and the clinic’s role in that journey.

Logic Models can be a great communication tool to describe a program. There was a more detailed logic model created for this clinic; however, for the purposes of this report we simplified it to this one-page overview using SmartArt. (We know SmartArt isn’t for everyone! Check out this article by Dr. Echo Rivera for some SmartArt alternatives.)

3. Describing the findings

We collected survey data and interview data from patients and family members. We wanted to present that information in a cohesive way instead of using headings like “patient survey findings,” “patient interview findings,” and “family interview findings.” Instead, we used the Canadian Partnership Against Cancer’s pillars of a positive patient experience frame how the results were reported.

We showed the relevant survey findings and incorporated interview findings from both patients and family members. Family member perspectives were highlighted with a call out box and an icon to cue the reader that the information was from family members and not patients.

---

Unfortunately, we can’t share with you excerpts from the report where we helped interpret the findings and recommended next steps, since these are specific to the clinic and their stakeholders. However, we hope we have provided you with some inspiration to help inspire your next evaluation report. 

If you’re looking for more reporting ideas and inspiration, check out our Six Hacks For Renovating Your Evaluation Report series of articles. Or feel free to reach out to one of our Eval Academy coaches.  

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---

 

 

Is Consent Always Needed?

Consent to participate in an evaluation project may not be required if the risks posed to an individual are normal, or no more than routine care. A good rule of thumb is that “the level of consent should match the level of risk.” A thorough ethical review may help if you’re unsure what this means. Quality improvement projects where participant information is used to inform improvements but kept confidential may not require consent, whereas interviewing members of a vulnerable population may require a comprehensive consent process. 

Some questions to consider when assessing the risk of participation may include: 

• What are the impacts of a breach of confidentiality? 

• Is the topic sensitive in nature? Does it address a stigmatized behaviour or population, religious, culture or legal issues? 

• What is being asked of the participant? Does it cause any burden? 

• Is there risk of causing psychological distress? 

There are a few more considerations and principles to cover: 

1. Capacity

Capacity is the ability to understand and appreciate the information being provided. Those with diminished capacity may still (and are encouraged to) be involved in the consenting process. It is important that each consenting participant is fully aware and capable of providing consent. Commonly, this means being over the age of 18, free from the influence of drugs or alcohol, and free from any mental or physical illness that may impair decision-making or understanding.  Check out our article where one of the Eval Academy team encountered problems with capacity and how she dealt with it.  

Importantly, capacity may change over time and there are projects where you may need to assess capacity multiple times. 

2. Coercion

It is possible that the invited participants feel obligated to participate. Falagas found that only 47% of participants truly understood the voluntary nature of participation. This is alarming and suggests that reasons for participation may rise from fear of being denied services or a power imbalance between the project lead and the invitee. Asking yourself about perceived or real power imbalances or conflicts of interest between participants and project leadership is a good place to start.

3. Revisiting Consent

Importantly, consent can change; it must be obtained and maintained. In projects that have a longitudinal time component, consent should be revisited at each touchpoint throughout a project. Reminders can be given about what has been previously consented to, the nature of remaining participation, changes in capacity can be assessed, the right to withdraw may be presented and/or a new opportunity to ask questions may be offered.

4. Evidence of Consent

Evidence of consent is an important part of the process. Evidence may be important should adverse events occur within the project to show that participants were fully informed prior to participation. As described in Part 1 (link), each form of consent has a means of providing evidence.  

Evidence of consent may be: 

• a (signed) consent form 

• a documented process or script for obtaining oral consent 

• a recording of verbal consent 

• field notes from the data collector about the consent process 

• through the actions of the participant (i.e., implied consent) 

---

There is limited information available to guide the consent process in evaluation work. The ARECCI guidelines and screening tool are an excellent resource to get project leads to question the need for consent and the ethical issues surrounding consent (e.g., power imbalances, conflicts of interest, information sharing, or the confidentiality associated with participation). 

Ultimately, thoughtful consideration of the consent process is an important project design step. If consent is required, documentation of the consenting process allows for evidence that the consent process has been planned and will be applied systematically. 

Consent can be a huge part of an evaluator’s role, but it isn’t always scary. If you still have questions reach out to us – we’d love to chat about this more!  

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---

 

Welcome to our monthly roundup of new and noteworthy evaluation news and resources – here is the latest.

Have something you’d like to see here? Tweet us @EvalAcademy!

New and Noteworthy — Reads

The Complexity Evaluation Toolkit

The Centre for the Evaluation of Complexity Across the Nexus recently published its complexity evaluation toolkit. The toolkit collates information about complex evaluation and covers key issues in commissioning, designing, and managing an evaluation. The toolkit is targeted for those with some understanding of evaluation but needing guidance on how to handle complexity. 

12 Foundational Principles for Data Story Design

Juiceanalytics has resources on its site for data professionals. They have a number of resources on data visualization, data storytelling, dashboards, and data products, but its most recent resource is on principles for data story design. The resource presents 12 principles in 12 slides that evaluators should know and practice when presenting data.

Reflections from BetterEvaluation’s outgoing CEO, Patricia Rogers

BetterEvaluation’s CEO, Patricia Rogers, has passed on the reins. After 12 years in her role of CEO she shares some of her highlights and reflections, including: how and why BetterEvaluation started; how its Rainbow Framework was developed; the global nature of BetterEvaluation’s audience, and; the different ways people find BetterEvaluation useful.

Reflections on the Funder & Evaluator Affinity Network

The Funder & Evaluator Affinity Network (FEAN) was created to try and address how evaluators and funders work together to deepen social impact. Four years later it is “sunsetting” and, like good evaluators, asking “so what?” and “now what?” This article outlines the difference FEAN made to its members and four reflections of what is needed going forward.

New and Noteworthy — Courses & Events

We need to talk about evaluation

• Organized by: Centre for Cultural Value

• Date: October 13, 2021

• Type: Webinar

Virginia Tech Speaker Series

• Organized by: Virginia Tech

• Date(s): October 14, 2021 (Grant Evaluation as a Career); November 18, 2021 (Culturally Responsive STEM Evaluation)

• Type: Webinar

Participatory Evaluation: Community-Based Assessment and Strategic Learning Practices

• Organized by: Tamarack Institute

• Date: November 17, 2021

• Type: Virtual Workshop