Understanding what you need to know. Designing your approach to finding the answers. Easy enough in theory, but real-world examples can show you have to apply theory to practice. That’s why we’re sharing this series on how we evaluated. 

 In this post, I describe how Three Hive Consulting supported the evaluation of a home health monitoring project for patients with chronic conditions. 

 

The Project: Home Health Monitoring in Primary Care

This project rolled out in the summer of 2020, in the throes of the Covid-19 pandemic. At that time, there was enough experience of the pandemic to know that limiting in-person interactions was the key to reducing the spread. The world was working to find ways to deliver services remotely, and healthcare was no exception. 

 Patients with chronic conditions often require more frequent medical care and are at greater risk of developing severe complications or needing hospitalization with Covid-19 infection. Monitoring symptoms and delivering care virtually was an obvious need.  

 In this project, patients with chronic conditions were provided with free monitoring kits for 90 days. The kits included blood pressure cuffs, pulse oximeters, thermometers, scales, and tablets. Patients input their information into tablets loaded with a platform enabling the reporting of results to primary care nurses. Primary care nurses then checked the platform regularly for alerts for clinically significant results. They called the patients directly or consulted with family physicians when appropriate. Many concerns could be addressed by nurses over the phone, while others required the patient to attend either a virtual or in-person visit with their family physician.  

Intended Outcomes

This home health monitoring project aimed to: 

• Improve patients’ access to care 

• Decrease patients’ risk of exposure to Covid-19 

• Detect worsening symptoms earlier 

• Optimize physicians’ time 

• Improve patients’ ability to self-manage 

• Reduce hospital admissions and emergency department visit 

• Maintain or improve patients’ health-related quality of life 

Developing the Evaluation Plan

This project involved many partners. Primary care in this region (central Alberta, Canada) is delivered by family physicians and supported by Primary Care Networks (PCNs). Three Central Zone PCNs received funding and in-kind resources from several partners, including Health City, Alberta Innovates, and Boehringer-Ingelheim. Each of these partners had input into the evaluation plan. 

 With limited resources available to evaluate the first phase of this potentially multi-year project, partners agreed to four focus areas: 

1. Project reach (using document review and administrative data) 

• Did the project achieve its aim of enrolling at least 30 patients? 

• Who are the participating patients? 

• What challenges were encountered in patient recruitment? 

• What was helpful or successful in patient recruitment? 

• Was PCN and provider participation maintained through Phase 1? 

2. Provider experience (using surveys at two points in time)

• What challenges did providers encounter in working with the HHM model? 

• What worked well for providers? 

3. Patient experience (using surveys at program discharge) 

• What challenges did patients encounter in working with the HHM model? 

• What worked well for patients? 

• What suggestions did patients have for improvement? 

4. Patient health outcomes (using the EQ-5D-5L at intake and discharge, patient surveys at discharge, and clinical data) 

• How, if at all, did patient-reported quality of life change over the duration of the project? 

• To what extent were clinically significant results identified through the HHM platform? 

• How quickly were patient results reviewed? 

• To what extent did patients utilize other health services during their participation in the project? 

We designed the evaluation plan to minimize the workload on busy providers and take advantage of information already documented for clinical care; asking nurses or physicians to track additional data elements was not feasible, particularly during the pandemic. Patient health outcome data was limited to what patients could reliably report themselves and the clinical information captured in the home health monitoring platform.  

The evaluation plan included mid-term reporting of provider experiences and final reporting of all available data. The evaluation findings would need to provide information to inform decisions about spread and scale, as well as point to a more robust evaluation approach supporting the expansion of this model of care after the initial phase.  

“It’s safe to say this is a new way of doing business,” says Central Zone PCN Committee Operations Lead Jodi Thesenvitz. “We need to see if this is going to be a legitimate model of doing work going forward.”  

Collecting Data

In a year of shifting priorities, collecting data went fairly well but did not yield the sample sizes we intended. All data collection was undertaken online to ensure safety and convenience, but we struggled with uptake.  

 Invitations and links to complete the EQ-5D-5L and patient experience survey were embedded within the home health monitoring platform, and so were seen by all participating patients. Of the 37 patients enrolled, only 11 completed the EQ-5D-5L at both intake and discharge, leaving us with results that were not entirely reliable. Similarly, only 17 completed the patient experience survey at discharge. 

 

Our Learnings

Low response rates are always a cue for evaluators to re-examine their recruitment processes. Upon reflection, the language we used in the home health monitoring platform to invite patients to complete both survey tools was not as compelling as it could be. In our subsequent second phase, we’ve adjusted the invite to provide more information about the importance of patient feedback and used plainer language to describe the EQ-5D-5L—it’s a very simple tool with a rather technical name. 

We’ve also increased engagement with providers, letting them know early about the role of evaluation in the project and their role in supporting that evaluation.  

The evaluation findings have been already been helpful to the project partners. “It helped validate the expansion of the initiative,” according to Health City CEO Reg Joseph. “One of our key goals is to scale the initiatives that make sense to scale, in that they drive health adoption of innovation, improve health outcomes and drive economic growth in the health sector. To do this often requires policy change that itself requires evaluation and data.” 

The findings have been helpful for PCNs and physicians considering joining the second phase of the project. The evaluation report was “used as a communication tool,” said Thesenvitz. “It was a source of evidence for those considering participation in the next wave. It legitimized and explained the effort.” 

Future Evaluation

The second phase of the project allows us to expand our evaluation approach. With a larger group of patients and access to system-level health data such as emergency department visits, hospital admissions, and physician visits, we will be able to build the body of evidence needed to make decisions not just at the operational level, but at policy and funding levels, too.  

We hope this example has helped illustrate what evaluation of a virtual health initiative, or even just program evaluation in general, looks like in practice. For another tool to apply evaluation theory to practice, see our free resource on applying the JCSEE Program Evaluation Standards in practice. 

 

To learn more about applying evaluation in practice, check out more of our articles, or connect with us over on Twitter (@EvalAcadmey) or LinkedIn.

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---

 

Six hacks for renovating your evaluation report

PART 6

This article is Part 6 in a six-part series that walks you through how to reno your evaluation reports using six of Canva’s design lessons.

• Part 1 focused on how to take your audience on a journey using storytelling techniques.

• Part 2 focused on how to format your report with a consistent, cohesive look using colour and font.

• Part 3 dove into grouping and spacing elements in your reporting.

• Part 4 explored how to make elements in your report pop using focal points.

• Part 5 explored how to use images as focal points.

This last article explores how to simplify data presented in graphs and tables in your report.

Simplifying data starts by having a clear message to convey

In the past, I have been guilty of putting any and all data I could into a report. I’m talking pages of charts to show ALL the results. If I’m being honest, in some instances, I didn’t know what the point was. I put in as much detail as I could to shift the burden of deciphering the meaning behind the data to my reader. Or, I fell back on my training that ingrained in my brain that I should be objective and not provide any insights into what the data might mean (see Part 1).

Regardless, what I learned the hard way is that I need a big idea (see Part 1), position, thesis, point, message, whatever you want to call it, when I am pulling my evaluation report together. Your big idea is your filter for determining what should and should not be included in your report; it will also help you determine how to present it. 

 

Choosing how to present your data is no longer limited to simple column and bar graphs. In fact, the number of different charts and ways to present the data in your report is sometimes overwhelming. Should you use a line chart? Bar chart? Column chart? Bubble chart? Scatter plot? Tree map? Heat map – the list goes on.

There are a number of different chart chooser tools out there you can find. Stephanie Evergreen has both quantitative and qualitative chart chooser tools. The Data Visualization Catalogue is another great online resource for selecting and understanding the various chart available to you. What you’ll notice though, is regardless of the tool you choose, they all rely on you being able to identify your story.  

Once you know what it is you are trying to convey, selecting the right chart becomes a whole lot easier – and more impactful. Take for example the chart below.

It probably took you some time to figure out the message – engagement within the HR department has tanked. The column chart is not doing anything to highlight that story. Compare that column chart to this slope chart.

The slope chart immediately highlights the change in scores from 2015 to 2017. Of course, there are many other formatting elements that help to convey our message, which brings me to my next point….

Don’t default to the defaults

Your software is smart, but not smart enough to know the message you are trying to convey. The column chart example highlights how default charts will only get you so far. Choosing the right chart (i.e., the slope chart as opposed to the column chart) helps to convey your message; however, formatting your chart to hone your message is where the magic happens. The slope chart highlights three ways we did that. We….

1. Got rid of distractions – We removed the gridlines from the default column chart. I tend to always remove those since I find them distracting to the eye. We also removed the y-axis. If you include data labels, then often one of the axes becomes redundant and can be removed. You can also remove the tick marks on the remaining axes to dial down another default distraction. 

2. Used colour intentionally – The blue and orange columns tell our eyes to look at the columns, but the colours compete with each other and for our eyes’ attention. The slope chart uses red intentionally to highlight the HR department and mutes out the rest of the data with grey. Immediately, the red draws attention to the message we want to highlight without being distracted by competing colours. 

3. Stated the story – The column chart contains a generic title “Employee Engagement Scores Over Time.” The slope chart clearly states the key message the audience needs to know. 

Dialing down your data means you have a dialed message. That dialed message frames not only what data goes into your report, but how that data is presented. Take a look at some of the reports you are creating. Are there opportunities to dial down your data? Try it out! And don’t forget about the other five hacks outlined in this series.

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

Welcome to our monthly roundup of new and noteworthy evaluation news and resources – here is the latest.

Have something you’d like to see here? Tweet us @EvalAcademy!

New and Noteworthy — Reads

Applying Evaluation Criteria Thoughtfully

The Organization for Economic Co-operation and Development (OECD) and the Network on Development Evaluation (EvalNet) laid out six evaluation criteria to support consistent, high-quality evaluation in the early 2000s. However, there was never a document to help evaluators and others understand these criteria and improve their use. OECD just released Applying Evaluation Criteria Thoughtfully. This document is meant to explore the criteria in detail by explaining what they are and how they are meant to be used.

Tools and Tips for Implementing Contribution Analysis

The Centre for Evaluation Innovation recently published a quick guide for implementing contribution analysis. The guide outlines what contribution analysis is and the six steps practitioners can take to implement it. The author also outlines challenges they have seen when conducting these types of evaluations and some ideas of how to mitigate those challenges. As always, the Centre for Evaluation Innovation has provided a very simple, practical document to help guide your practice.

Indigenous Made in Africa Evaluation Frameworks

The most recent American Journal of Evaluation published this article by Bagele Chilisa and Donna M. Mertens. This article discusses how issues of culture, ethics, and values from an Indigenous paradigm perspective have largely been ignored by international agencies when framing evaluations. This article describes an “Indigenous paradigmatic framework and then narrows the focus to a Made in Africa approach to evaluation that is designed to redress the epistemic violence perpetrated by the use of a Western Cultural lens to determine evaluation approaches.”

Using Twitter Data for Development Research and Evaluation

The World Bank and IEG staff shared their experience in a webinar using sentiment analysis (including using Twitter data) as a tool for collecting data for development research and evaluation. This article is a summary of the lessons from that webinar. If you are interested in using social media data for data collection, this is a good high-level overview of the benefits, opportunities, and risks.

New and Noteworthy — Events

A Conversation on Evaluative Thinking: A discussion with Chari Smith (Evaluation into Action) and Hayat Askar (EvalJordan)

Organized by: Evaluation into Action and EvalJordan 

Date: April 9; 9:00-10:00am (Pacific Standard Time) 

Facilitator: Chari Smith and Hayat Askar 

Using Art in Creative Data Collection and Evaluation

Organized by: Canadian Evaluation Society 

Date: April 15; 12:00-1:00pm (Eastern Daylight Time) 

Facilitators: Jennica Nichols and Maya Lefkowich 

Decolonizing ‘Development’ Evaluation

Organized by: Virginia Tech 

Date: April 15; 12:00-1:00pm (Eastern Time) 

Speaker: Candice Morkel  

Evaluation for Transformative Change

Organized by: Tamarack Institute  

Dates: April 20, 22, 27 and 29 

Facilitators: Michael Quinn Patton and Mark Cabaj 

Courses

Most Significant Change

Instructor: Clear Horizon Academy 

Start Date: April 16, 2021 

Evaluation Systems Change and Place-Based Approaches 

Instructor: Clear Horizon Academy 

Start Date: May 21, 2021 

 

This past year, our team at Three Hive Consulting worked with a community development initiative to evaluate their activities. A key function of the initiative was to serve as a connector and integrator, bringing together different individuals, groups, and organizations who were all interested in strengthening the lives of the citizens in a specific community. 

One of the goals of the initiative was to create community linkages and foster collaboration. We chose social network analysis (SNA) as the methodology to understand how the initiative had affected the relationships and collaborations in the community. 

Before we jump into what we learned, and what the client thought, let’s back up and review what an SNA is and how our team used the methodology for this client. 

 

What Is A Social Network Analysis?

An SNA is a quantitative approach to measure the strength and types of relationships within a network. It examines how different players relate to each other and assesses the characteristics of those relationships. 

An SNA focuses on network structure, rather than the individual characteristics of the partners. 

Data about who is included in the network, how they are related, and how they are working together is collected to create a picture of the relationships in that specific network. 

The focus of a SNA is to understand the network structure rather than the individual characteristics of the partners in the network. 

 

What Is Social Network Analysis Used For?

An SNA can be used in an evaluation to measure how groups of people are working together to achieve a common goal. It’s a particularly useful tool when addressing complex issues where multiple players are working towards a common outcome.

When the functioning of the network can impact whether the group achieves its goal, an SNA analysis can uncover insights about how the players are working together and pinpoint areas for improvement- where relationships can be strengthened or built.

A Social Network Analysis is also useful for community development or population health initiatives where the actions of a group of players contribute to change.

 Social network analyses can help to answer questions such as:

• Who is connected to whom?

• How well are the existing relationships working?

• How connected are different sectors within the network?

  • How strong are these connections?

  • Do these connections have directionality?

  • Who are the central players and outliers?

• Where do relationships need to be built or strengthened?

• How is power shared within a network?

• Who shares resources with whom? What kind of resources?

How Did We Use Social Network Analysis?

One of the first steps in a social network analysis is bounding the network; which means deciding who is or isn’t considered part of the network. 

In our case, the evaluation sub-committee, which was made up of community partners and staff from the initiative, listed who the initiative was directly working with and categorized each partner by sector (e.g. faith, education, recreation) and by the type of community capital they provided (e.g. natural, built, social, financial). 

We used the PARTNER Platform by Visible Network Labs to send the survey and analyze the data.

Although the use of the PARTNER platform simplified the process, gathering the data was still time-intensive for all involved to achieve the high response rate (>75%) required for data accuracy.

While using the PARTNER platform also simplified data analysis, sense-making was, again, time-consuming. 

The data from the survey was complex. We as evaluators looked to find the overarching trends in the data, then worked with the evaluation sub-committee to delve into the nuances and understand what the data meant to them.

For example, we generated a map of which partners were sharing resources with each other (see graph below). The colour coding represented the partner’s sector. 

The SNA generated many maps like this, which helped the client visually see which organizations were connecting with each other and who was being left out. However, when it came to understanding what the map meant for contributing to the initiative’s success, we needed more context. 

We presented these maps to the evaluation sub-committee and together we worked to make sense of the data. Some maps and insights were more relevant and inspired ideas for action based on who could be collaborating more or which two unconnected partners might work well together.

We also generated value, trust, and connectedness scores for the network as a whole and for each individual partner. 

When we compared each partner’s degree of trustworthiness with how well connected they were, an interesting pattern emerged. Visualizing these results allowed us to identify partners who were highly trusted but not well connected (black dots in the graph below). 

Again, it was up to the evaluation sub-committee to put these results into context, using their knowledge of the partners to decide this was an expected result or whether connecting these highly trusted but not well-connected partners was feasible and relevant to achieving the network’s goals. 

In some cases, a highly trusted but not well-connected partner may only be involved in the network’s work tangentially and it wouldn’t have made sense to connect them with more organizations.

To learn more about using the results, we asked our client how the social network analysis was useful to them. 

Client Perspective

About the overall experience:

“Being invited to design and create a Social Network Analysis was a meaningful experience. It was helpful to consider different variables of our network, that perhaps had only been tacitly considered – like value and trust. Or, we had considered these things, as they come up regularly in partnerships, but we hadn’t formalized the concepts or explored how they could be measured. Learning more about this was insightful and empowering.”

 

About visualizing the connection in the network as a map:

“Practically, I had a visual understanding of our network and saw right away that meaningful connections needed to be made between some of our partners. Having partners collaborating with us does not mean that they are collaborating with each other, or even aware of each other. We have used this insight to connect more people to [one organization], as well as linking some of our partners from the business sector into what have traditionally been agency-dominated conversations exploring food security. “

 

About using the trust and value scores:

“In some ways, the obvious is profound and humbling. We were well connected, in our network, but we were also shown to be trusted and valued. This was immediately encouraging and affirming, especially when considering that we haven’t always delivered, or moved forward with clarity, or met the expectations of all stakeholders.”

About using the recommendations:

“The recommendations in the final report have been very valuable and will inform strategy, especially over the next 11 months…[for example]  we are working to improve our communication with partners and employing a number of different methods. We have intentionally started… to share information and to host projects and groups. It is working.”

What we Learned From Conducting A Social Network Analysis

A social network analysis must be a collaborative effort

Everyone needs to be on board with taking a collaborative approach to the project; from deciding who is included in the network and how to categorize them, to encouraging partners to participate in the survey, and understanding the data in context.

Interpret the data with the client to make it more meaningful

Understanding how groups work together requires an understanding of the context and a level of detail that an external evaluator doesn’t have. The data only became useful once we started asking the client what it meant to them. If a client isn’t willing to work through the data and generate insights and recommendations together, the SNA loses much of its value.

Your social network analysis will take time

The biggest lesson here was how much time and effort it takes to conduct a meaningful social network analysis. It takes communication and follow-up with network members to get buy-in and a high response rate on the network survey. Analyzing the results and making sense of the data must be done in conjunction with those involved in the network.

Clearly defining your network is extremely important

Without a clearly defined network, an SNA can quickly become bloated and confusing. A network should have clear inclusion or exclusion criteria and include a clear definition of the goal the network is working towards. An SNA may not be very useful when a group of organizations or people are not working towards a specific goal.

 

To learn more about applying evaluation in practice, check out more of our articles, or connect with us over on Twitter (@EvalAcadmey) or LinkedIn.

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---

 

Welcome to our monthly roundup of new and noteworthy evaluation news and resources – here is the latest.

Have something you’d like to see here? Tweet us @EvalAcademy!

New and Noteworthy — Reads

A Social Equity Assessment Tool

Khalil Bitar (@KhalilBitar) recently published a paper on ResearchGate titled “A Social Equity Assessment Tool (SEAT) for Evaluation.” The paper discusses how there is a lack of tools for equity assessment in evaluation. Bitar proposes a tool with 13 social equity aspects that evaluators can use to examine equity and social justice issues in various contexts. The SEAT consists of eight demographic aspects and five cross-cutting aspects. This is a comprehensive tool to assess equity more inclusively in your practice – give it a try!

Adapting Evaluation Questions to the COVID-19 Pandemic

The UNFPA Evaluation Office recently released “Adapting Evaluation Questions to the COVID-19 pandemic.” The increasingly complex and dynamic context of the pandemic has meant the UNFPA programmes are being reassessed and redesigned to respond to the emerging needs. Given this, there is a need to adapt evaluation questions to assess the extent to which UNFPA has adapted its interventions. UNFPA developed this document to encourage evaluators to reflect on their existing questions and to formulate meaningful questions that take into account COVID-19 factors. Questions centre around: a) relevance, b) effectiveness, c) efficiency, d) coordination and coherence, e) sustainability.

Reflections – Lessons from Evaluations: Learning from Past Crises for Recovering from COVID-19

The Independent Evaluation Office (IEO) of the United Nations Development Programme (UNDP) has a reflections series, which are lessons learned from past UNDP evaluations. This first volume is a compendium of the various reflection papers. The 61 lessons presented in this first volume can inform how to design and implement responses to the COVID-19 crisis based on what has worked well in previous crises.

Measuring Our Impact: Evaluation Framework for Measuring the Impact of Community Development Work Across Local Government in Western Australia

The Community Development Network of Local Government Professionals of Western Australia worked in partnership with a number of partners to create a Community Development Evaluation Framework and Toolkit. Community practitioners are the intended target audience. The purpose of the resource is to strengthen community practitioners’ understanding of evaluation and to provide them with practical steps to implement evaluation in their practice. The resource is basic evaluation content but is very comprehensive and contains a number of toolkits and templates.

New and Noteworthy — Events

Virtual Workshops

Transformation Narratives: Storytelling in the Service of Evaluation and Organizational Learning Through a Gender Lens 

Organized by: Institute of Social Studies Trust 

Date: March 23, 2021; 6 pm Indian Standard Time 

Facilitator: Hamutal Gouri 

Evaluation for Transformative Change 

Organized by: Tamarack Institute  

Dates: April 20, 22, 27, and 29, 2021 

Facilitators: Michael Quinn Patton and Mark Cabaj 

Webinars

Evaluation Speaker Series 

Organized by: Virginia Tech 

Presenter: Sheila Robinson – Designing Quality Survey Questions 

Dates: March 25, 2021; 1 – 2 pm 

Presenter: Candice Morkel – Decolonizing ‘Development’ Evaluation 

Dates: April 15, 2021; noon – 1 pm 

Courses

Most Significant Change

Instructor: Clear Horizon Academy 

Start Date: April 16, 2021 

Evaluation Systems Change and Place-Based Approaches 

Instructor: Clear Horizon Academy 

Start Date: May 21, 2021