evalacademy

This article is rated as:

---

In this article, we provide the answers to some of your FAQs about evaluation including:

• Why do people conduct evaluations?

• Where can I find an evaluator?

• What makes a good evaluator?

• How can I get training in evaluation?

• What is the difference between research and evaluation?

• How much does an evaluation cost?

---

Why do people conduct evaluations?

The “why” for an evaluation depends on the “who.” Generally, an evaluation is conducted because a person or group of people (i.e., stakeholder(s)) need to systematically answer questions about a project, program, initiative, and/or system. The information discovered through an evaluation can be used for different purposes depending on the stakeholders involved, but may include:

• Assessing merit/worth (i.e., judgement)

• Improvement (i.e., learning)

• Demonstrating resources are well spent (i.e., accountability)

• Early identification of problems (i.e., monitoring)

• Adapting and informing design of the initiative (i.e., development)

---

Where can I find an evaluator?

Recommendations from colleagues and business contacts who have used an evaluator in the past is always the best place to start. If you’ve asked around and still can’t find anyone, then you can search the online directories. Evaluation is a distinct discipline and profession, which means there are professional associations all over the world.

The Canadian Evaluation Society, the American Evaluation Association, the European Evaluation Society, the Australian Evaluation Society and African Evaluation Association all have directories of evaluators you can search for according to geography.

If you’ve found some names on the directories, it’s a good idea to do some background research on that person or firm. LinkedIn is a good place to find out about their qualifications.

---

What makes a good evaluator?

In our article The Six C’s of a Good Evaluator, we list credibility, curiosity, communication, creativity, and common-sense orientation as five important skills and traits to look for in an evaluator. All of these are important in determining the sixth “C” which is competency. A competent evaluator will demonstrate all these traits and will sometimes even have a professional designation that shows they have the requisite skills, knowledge, and practical experience. For example, the Canadian Evaluation Society has a Credentialed Evaluator designation and a listing of those people on their site.

Of course, “good” is a value-based term that ultimately will be determined by who is interpreting it. If one person values one particular skill, trait, method, approach, or type of evaluation over another, then it is best to find an evaluator who is competent in that area and prioritize that over other areas to best meet your need. 

---

How can I get training in evaluation?

If you are looking for formal education in evaluation, several universities offer high-quality, graduate-level education in evaluation. In Canada, The Consortium of Universities for Evaluation Education (CUEE) lists courses and programs. The American Evaluation Association also has a listing of university programs all over the world.

If you are not looking for formal education, but self-directed learning then there are a number of online courses, webinars, articles, and podcasts you can start with. Eval Academy is a great place to start. Check out our available courses. The professional associations mentioned above also have listings of courses.

Of course, some of the best learning and training occur by doing. Do you know someone who is an evaluator? Are they willing to mentor you? Do you work in an organization that contracts evaluators? Can you find a way to learn from them? Most of us learn by doing, so the best training is some formal learning with a lot of real-world applications.

---

What’s the difference between research and evaluation?

Research and evaluation share methods but differ in their purpose. Research is curiosity-driven and often aims to generalize or develop theories that apply broadly. Evaluation is utilization driven and focuses on understanding a particular project, program, initiative, etc. While research aims to control settings and often uses an experimental design to support that (e.g., Randomized Control Trials); evaluation occurs in real-world settings which often inhibits the use of experimental design.

---

How can I become an evaluator?

There are many paths to becoming an evaluator. Although such programs exist, you do not need a degree in evaluation to be an evaluator. In Canada, The Consortium of Universities for Evaluation Education (CUEE) lists courses and programs. The American Evaluation Association also has a listing of university programs all over the world.

Having a thesis-based graduate degree may help an evaluator’s technical practice (i.e., competencies related to methodological decision-making). Many evaluators have degrees in social sciences, education, healthcare, or statistics that are complimented with shorter, targeted training courses.

---

How much does an evaluation cost?

Asking how much an evaluation costs is like asking how much a house costs. When you buy a house there are lots of factors that determine cost – location, size, level of finishing in the house, lot size, etc. The same applies to an evaluation. How many evaluation questions do you have? How many stakeholder groups? How often do they want to be involved? How do they want to be involved? How do they want the findings reported? Presentation? Video? Report? How many reports? What length is the report? And this doesn’t even get into the questions around methodology and evaluation approach!

If you need to fill in a line item on your budget, then some common rules of thumb are 5% – 20% of an initiative’s overall budget; however, your best bet is to go to a credible evaluator with as many details regarding your initiative and evaluation so they can provide you with an informed quote.

---

Do you have other evaluation questions you want answered? Drop them in the comments below or connect with us over Twitter!

This article is rated as:

---

Creating clean and consistent charts does not need to be complicated or time-consuming. With some forethought and a style guide, you can draft templates for all your charting needs. Chart templates will alleviate the pains of trying to reformat every new chart from scratch. From adjusting text size and font to formatting bar width and colour, a template will save you a significant amount of time in drafting your next evaluation report.

Investing in creating chart templates for your next report will:

• Save you hours in your reporting. While you will spend time upfront to draft your templates, it will save you time in the long run when formatting your other charts.

• Apply “most” chart formatting to all charts of the same type (i.e., a bar chart template will apply best to other bar charts).

  Note: I say most and not all because you may have some manual formatting (e.g., a custom title) that will not always apply to subsequent charts.

• Give you peace of mind knowing that your charts are consistent and professional looking. This will allow you to focus on better telling the story of the data and not worrying about minor formatting issues.

Learning to create chart templates is best done by doing. And we want to go beyond generic Excel chart templates and create something visually engaging for our evaluations. Therefore, here I will walk through how to format an appealing and impactful chart to use as the backbone for your first chart template.

---

Creating a Chart Template

Sometimes the best data visualizations are the simplest (here are some tips to help improve your data visualizations), and few charts are as simple and effective at presenting data as the bar chart. And creating a standard template for your bar charts will speed up the reporting process.

---

The Excel Default

1. Highlight the data from which you want to draft a bar chart. In this example, we will plot the top ten coffee-consuming countries.

2. Navigate to Insert > Charts > Clustered Bar Chart

---

 Drafting a Better Template

1. Trim the fat by removing unnecessary labels and gridlines.

• Click on the data labels along the horizontal axis and hit Delete.

• Click on the vertical gridlines and hit Delete.

2. Change the font to your style of preference.

• Select the chart and navigate to the Theme Fonts and chose your favourite font (here I will use Tw Cen MT).

• Change the Font Colour to black. While font colours can be adjusted, using a standard black font is often best for the readability of your chart.

• Adjust the Font Size to your preferred size (here I adjusted the font to size 11).

3. Draw attention to the bars by increasing their width and changing their colour.

• Right-click any bar > Format Data Series…

• Adjust the Gap Width to 30%

  Note: a smaller percentage decreases the gap between bars and increases the overall size of each bar.

• Change the Fill to the colour of your choice.

  • For extra emphasis, you can mute most bars to focus on your key findings (e.g., highlight the top three in a Dark Blue and mute the remaining in a Light Grey).

  • If opting to mute some colours and emphasize others, you will need to manually click on the bars you want to change the colour of one-by-one, and change their colour by right-clicking or using the Format option for the chart.

4. Add in data labels.

• Click on any bar to highlight them all > Right-click > Add Data Labels OR

• Click on the + symbol at the top right of your chart (when highlighted) > Toggle on Data Labels

5. Sort the data in descending order (largest to smallest).

• Data are often entered or sorted from largest to smallest value. However, by default, Excel will plot bar charts with the top value in a table as the bottom bar and the bottom value in a table as the top bar.

• To correct this, right-click on the vertical axis labels > Format Axis > Categories in reverse order

6. Make final touches to the bar chart template by removing the border and improving the title.

• Right-click on the chart > Outline > No Outline

• Click on the title and write a title that tells the story of the data being presented. Bold the import pieces and increase the font relative to the rest of the chart to draw attention to the title.

---

Saving and Applying a Chart Template

Once you’ve put in the effort to format a single chart to your desired preferences, it is only a matter of saving the template and applying the template to other charts without having to go through all the previous steps.

1. Right-click on the chart created in the ‘Drafting a Better Template’ section > Save as Template…

2. Next, create another default Excel bar chart with new data (see ‘The Excel Default’ section).

3. In the unformatted bar chart, right-click > Format Chart Type… > Templates > Select your bar chart template.

4. While the template will transfer over, you will need to make minor edits to the chart title (e.g., to highlight to top three tea consuming countries) but all other formatting will be done automatically.

---

A Quick Hack for Copying Over a Chart Template

Saving and applying your chart templates is always recommended. But sometimes you need to apply the same template to multiple, consecutive charts. You could follow the above process and upload your chart template to each chart individually, but there is a faster approach and one that I use often.

Note that in the following example I use a slightly modified template, in that I no longer highlight the top three countries and I did not modify the title. However, this approach works with any formatting or template previously created.

1. Create all the default Excel charts for the charts to which you want to apply a chart template.

2. Format a single chart to your preferred style or apply a saved chart template.

3. Next click on the formatted chart and copy (CTRL + C or Right-click > Copy) the chart.

4. Click on the next unformatted chart and on the Home tab select Paste > Paste Special… > Formats (or use the keystrokes ALT + E, S, T and Enter); this will copy the chart format to the unformatted chart.

5. Follow the same Paste Special… process or keyboard shortcut for the next chart, or simply press F4 after the initial format has been pasted over once. F4 will redo the previous command, which will allow you to simply click on each subsequent unformatted chart and format it with the press of a single key.

---

You can create templates for all chart types, and it will save you an inordinate amount of time in your future evaluations. Even creating a basic template to build off will save you time in the long run.

Not all charts need to be identical, but a framework with consistent fonts, bar widths, etc. will improve the look and readability of your report charts. Add in your personal touches and experiment with colour, size, and labelling to create charts to wow both your employer and clients.

Do you use chart templates in your evaluation work? Let us know in the comments!

To celebrate the end of the year, we thought we’d highlight Eval Academy’s top ten posts from 2022.

What posts or resources would you like to see in 2023? Comment below, tweet us @EvalAcademy or connect with us on LinkedIn!

10. Differences between Theory of Change, Log Frames, Results Frameworks and Logic Models – what are they and when to use them

 Noticed how there seem to be lots of evaluation tools doing very similar things? This article selects a few of our favourite tools to compare and contrast.

 

9. Finding the Right Sample Size (the Hard Way)

For quantitative data collection, this article helps you to calculate sample sizes by hand and understand the math behind sample size calculations.

8. How to “Quantify” Qualitative Data

This framework for quantifying qualitative data is one you’ll want to bookmark!

 

7. Evaluation Question Examples by Type of Evaluation

This article looks at how using different evaluation strategies or frameworks like Formative Evaluation and Developmental Evaluation can help you to craft those perfect evaluation questions.

 

6. Reflexivity in Evaluation

We’ve all heard of reflective evaluation practice, but what is reflexive evaluation practice? This article breaks it down for you and outlines tools and ideas for reflexion.

 

5. Everything You Need to Know about Likert Scales

The Likert Scale is one of the more commonly used rating scales in surveys. As evaluators, we should know a thing or two about it, and how to navigate some of the decisions involved in using a Likert Scale, and this article does exactly that!

 

4. But really, how do I use the RE-AIM Framework?

Ever heard of the RE-AIM Framework? This article provides an account of what it is and how to use the framework in your evaluation planning, implementation, and reporting.

 

3. Evaluation Questions Examples

At number 10 last year and moving up through the ranks, this article provides you with some examples of evaluation questions to inspire your own.

 

2. How to Write Good Evaluation Questions

Last year’s top post has moved into second place! This article outlines what an evaluation question is and points to consider when drafting your evaluation question. An ongoing top contender!

1. Interpreting themes from qualitative data: thematic analysis

Our top post for 2022 supports evaluators who are new to qualitative data analysis. We start by defining thematic analysis, then give you a 5-step process to complete your own analysis. We end the article by highlighting some common challenges with thematic analysis. It’s not one to miss!

This article is rated as:

---

Much like you could argue that research and evaluation are related, perhaps members of the same family tree, I like to think of Implementation Science as a distant cousin of evaluation; one that comes for a really fantastic visit once in a while.

I was granted the opportunity to take a course a couple of years ago that opened my eyes to the world of Implementation Science. The course introduced me to new approaches, frameworks, and tools that can be used in evaluation.

So, what is this world? Let’s start with a definition and some background. I think you’ll begin to see why Implementation Science is a great relative of evaluation. 

---

What is Implementation Science? 

Implementation Science is a field that examines the methods and strategies that enable the successful implementation of practice. It was only established in the early 2000s as a response to the too-common gap between best practice research and behaviour change.

Do we all know the story of handwashing? You can read a summary of it here, but quickly: Ignaz Semmelweis made the connection between handwashing and deaths on a maternity ward in 1847, but Semmelweis was ineffective at communicating and spreading his learning (i.e., no behaviour change was implemented!) It wasn’t until the 1860s-1880s when germ theory was established that handwashing became more commonplace and mortality rates decreased. Twenty years of unnecessary death. So, what was the problem? Implementation Science tries to answer that.

While the handwashing story sounds like a story that would happen in the 19th century, this persists today. The research community and academia are regularly determining the efficacy of treatments and interventions that are not spread to common practice. It’s now a commonly cited fact that it takes 17 – 20 years for clinical innovations to become practice. This paper written in the year 2000 shows how long it took for some effective therapies to reach even minimal rates of use:

Implementation Science was developed as means to address this research/practice gap.

So, when a new, evidence-based program or intervention is designed and is ready to be operationalized, Implementation Science directs you to focus on how best to do that:

• Which stakeholders should you engage? 

• What barriers or obstacles can you anticipate and mitigate? 

• What enablers can you put in place? 

• How can you be sure the program is implemented with fidelity? 

• How can you implement in a way that promotes sustainability, or can uncover lessons for spread and scale? 

Implementation Science aims to bridge the gap from what you know to what you do and offers frameworks and structure to do this.

Now, despite being a relatively young field, there’s still a lot to dig through in Implementation Science. There are whole courses on Implementation Science (like the one I took) and it even has its own journal. I’ll focus on how it relates to evaluation and why you might use it, after just a little more context.

---

A Snapshot of the Implementation Science Toolbox

Implementation Science has an overwhelmingly large toolbox. There are many, many frameworks, models, and tools that can be applied in various contexts. I’ll summarize just a few, that are likely the most relevant to evaluation. I’m sharing some links that will lead you to more details if you want to dive in. After a brief description of a few models, I’ll follow with some real-world examples.

1. Knowledge to Action:

A process model used to describe and/or guide the process of translating research into practice. It has been adopted by the Canadian Institute of Health Research (CIHR) as a core component of their Knowledge Translation.

2. Determinant Frameworks:

Describe general types of determinants that are hypothesized to influence implementation outcomes (e.g., fidelity, skillset, reinforcement).

• PARIHS (Promoting Action on Research Implementation in Health Services) uses the Organization Readiness Assessment to explore identified determinants.

• Theoretical Domains Framework (TDF) integrates several theories into 14 core domains.

• CFIR: (Consolidated Framework for Implementation Research) is a practical guide for assessing barriers and enablers during implementation. CFIR has a website dedicated to its use that includes guidance for use in evaluation, and a question bank.

3. Classic Theories

Rogers’ Diffusion of Innovation was one of the first theories to suggest that implementation, or diffusion of behaviour change, is a social process.

COM B (Capability, Opportunity, Motivation, Behaviour) uses a behaviour change wheel to support design of interventions.

NPT (Normalization Process Theory) aims at assessing how behaviour change is embedded into regular routine. It includes a 16-item assessment scale centered on four core constructs.

Obviously, that was a pretty high-level run-through of just a few models, but all those links will give you more detail should you wish to learn more about any of those models.

---

Where does evaluation fit into all of this?

When you read those questions posed above about how to implement a new intervention, did it make you think of anything in evaluation? It did for me! FORMATIVE EVALUATION! When it comes time to ask your formative evaluation questions, Implementation Science can be a great guide.  

When you think about conducting formative evaluation, aside from “what’s working” and “what’s not working”, it may be difficult to ensure you are asking questions about the right factors (or determinants!) that may impact successful program implementation. Much like RE-AIM provides structured guidance to ensure you pay attention to five core domains of public health interventions, these Implementation Science models, frameworks, and tools are offering us tips and tricks about potentially overlooked factors that contribute to program success or failure – things that we can be evaluating.

Let’s use CFIR as an example because it’s my favourite. I have often navigated to their question bank and asked myself, “Do my evaluation plan and data collection tools collect enough information to be informative about….”

• the intervention itself? 

  • How might the strength and quality of the evidence for the intervention impact the outcomes I am evaluating?

• the external context (or outer setting)?

  • How might policies or incentives impact fidelity to the intervention?

• the inner setting?

  • How might team cultures or readiness for change affect the speed in which we expect to see the outcomes met?

In a recent project, the CFIR guide helped me to think about the importance of having a champion of their initiative. Without a champion in the project, who is passionate and promoting the importance of the work, this side-of-desk, in-kind-funded program could easily lose momentum. So, in a pulse survey tool I had created for the operational team I added a question asking the team if they felt they could identify a project champion. I’ve used the answers as indicators to discuss with the team whether the project is on track and adequately resourced.

Normalization Process Theory (NPT) is another one I’ve used. When an apparently great intervention isn’t getting any traction or is failing to spread or scale, why is that? It wouldn’t be out of scope to get an evaluator to help to answer that question. NPT offers guidance about where to look. Can you assess or measure:

1. Coherence: does the team understand the intervention? 

2. Cognitive Participation: is there sufficient direction and messaging to support this intervention?

3. Collective Action: is the team empowered to act? Do they have the right tools and resources?

4. Reflexive Monitoring: what are the team’s reactions to doing things differently?

On a project evaluating implementation clinical pathways in primary care, NPT helped to guide evaluation of the mental models of physicians. That is, how do physicians normally implement pathways: what enables use and what poses a barriers? The four core constructs of NPT helped to ensure we were evaluating actual behaviour change.

---

The distinction between formative (or process) evaluation and Implementation Science is blurry, for me anyway. I think there’s some evidence to say I’m not alone in this thinking. Implementation Science even claims RE-AIM as an implementation model, like here, here and here. But formative evaluation and Implementation Science are different. Formative or process evaluation aims to determine if an initiative is on track to meet outcomes, whereas Implementation Science doesn’t look at effectiveness overall, only of the implementation strategy. Implementation Science assumes the intervention is already evidence-based, proven best practice, whereas evaluation might be looking to build that evidence base.

I think being knowledgeable about Implementation Science can only make our evaluation work stronger. I don’t think we need to implement an entire framework with academic-level rigor for it to be useful. I like the “borrow and steal” approach, where I feel like Implementation Science is giving me insider information, pointing me to look at proven determinants of program success that might sometimes be overlooked in our traditional evaluation frameworks.

Do you have other fields that you like to borrow and steal from? Let me know what they are!

This article is rated as:

---

As you might know, through our sister company, Three Hive Consulting, we support healthcare and non-profit organizations to achieve their mission by uncovering insights that drive impact through evaluation. 

Recently, we’ve had several clients ask us about Developmental Evaluation (DE) including what it is and how it’s done. So we’ve pulled together a few main points on DE below.

---

How does Developmental Evaluation work?

• Developmental Evaluation (DE) facilitates close-to-real-time feedback about what is working well and what’s not, therefore facilitating parallel streams of planning, acting, and doing

• DE tailors your evaluation to your needs and the specific context

• In contrast to formative or summative evaluation, DE supports the creation, development, or adaptation of a program as it is being implemented to inform development

• As part of DE, the evaluator is embedded in the program as a member of the team

---

What Developmental Evaluation is NOT

• Developmental Evaluation (DE) is not an evaluation method, there is no one right way of doing it

• DE is not about product or service Improvement

---

When should Developmental Evaluation be used?

• Developmental Evaluation (DE) is for emergent, volatile, and exploratory programs, those that don’t know what activities to do or steps to follow to meet their goals

• DE is most appropriate when working in complex environments where the route to change is non-linear and cannot be easily predicted

• DE is suited to socially complex situations, innovation, radical program re-design, and crises

---

“Developmental Evaluation (DE) is an investigative evaluation approach that supports the development of social change initiatives in complex or uncertain environments.”

---

---

Have you used Developmental Evaluation before? Let us know your thoughts in the comments below!