In most evaluation plans, we try to define scope. Defining evaluation scope means putting boundaries around things like:  

• Time – defining what timeframe of the initiative you will include in your evaluation, or how long you will be undertaking evaluation work 

• People – deciding which clients’ or stakeholders’ perspectives will be included, and which won’t 

• Components – clarifying which parts of a program or initiative to evaluate, and which to exclude 

The scope of your evaluation project should be clearly documented in your evaluation plan. That documentation will be important throughout the life of the evaluation project. As an example, here is a scope statement from one of our recent evaluation projects: 

The scope of this evaluation will vary between operational phases. Generally, the scope will be limited to activities related to this initiative from Red Deer and north in Alberta, although some alignment of evaluation activities with activities in southern Alberta may be incorporated. While past programs and systems inform the initiative, this evaluation will not investigate any of those previous programs or services. This evaluation plan addresses Phases II (April 2021 – September 2022) and III (October 2022 – March 2024). 

 And another:  

 All evaluation projects face limitations in resourcing and feasibility. This evaluation will include all patients and providers participating in the initiative. Data elements will include what can be reliably reported by patients and providers through surveys and through patient interviews, as well as data captured in the pharmacy information platforms. All activity between project initiation (March 2021) and project close (March 2022) is within scope for this evaluation. The perspective of others in the healthcare system, including family physicians and health authority staff, are considered out of scope for this evaluation. Patient health outcomes are not in scope for this evaluation. 

Ideally, our evaluation projects would proceed as planned. But as all project managers know, sometimes things change. Actually, most of the time, things change! 

In some situations, our evaluation approach can be modified to adapt to the changing context, but in others, we have to say no to scope creep. 

---

What is scope creep?

The term “scope creep” comes to us from the project management world. Definitions about, but essentially scope creep is when the project work begins to extend, or “creep,” beyond what was originally agreed.  

 Many evaluators will have received requests from clients to add in a thing or two here and there after the evaluation project is already underway – maybe another data source has surfaced, such as a survey run by a partner organization that seems relevant. Maybe they would like to add another stakeholder group to be interviewed. Or maybe they would like to include activity data from the last fiscal year, despite originally wanting to focus only on the current year. Some scope creep requests relate to reporting – an extra presentation here, a customized sub-report there.  

 Or perhaps it’s the evaluator that wants to expand the scope. You may have identified a new opportunity to strengthen your evaluation project by incorporating a new analytic approach, or early findings suggest there is merit in pursuing an additional line of questioning. There are times when expanding the original scope of an evaluation project can open up important insights that may have seemed inaccessible during the planning phase.  

 But beware! Scope creep isn’t always so obvious. It can creep up slowly, and before you know it, you’ve added a couple of survey questions, provided a third draft of a slide deck, and interviewed management staff that you hadn’t planned to.  

What to ask yourself when you discover scope creep

When you see scope creep happening, take a quick pause to reflect. You might need to say to your client or team: “Let’s just take a few minutes to see where this fits in our project scope.” Referring to the scope statement defined in your evaluation plan will support the conversation or introspection that follows. Whether you are acting as an external evaluation consultant or as an internal evaluator, there are a few questions you can ask to help make the “to scope creep or not to scope creep” decision.  

• What is motivating this request? 

• Are there ethical implications to this new task? 

• Does this new request add value?  

• Do we have access to the data source required for this request? 

• Do we have the time to complete this extra task? 

• Is there room in the budget? Or, can additional funds be found? 

There’s no easy algorithm for making your choice about project scope. Here are a few considerations that can help you make the best decision.  

When should you indulge in scope creep?

We can’t anticipate everything when we’re designing an evaluation plan, and there is a very real possibility that a meaningful opportunity to create a better evaluation will arise. If the option to strengthen your evaluation approach arises and:  

• it is ethical,  

• it doesn’t unfairly burden any individual or group, 

• it is in alignment with the overall purpose of the evaluation,  

• it can be done well, and  

• it is feasible within the timelines and budget allocated… 

then go ahead and creep! Your original evaluation plan doesn’t need to restrict innovation—one of the points we’ve made about evaluation project management is that you shouldn’t live and die by the plan. 

When to avoid scope creep

On the other hand, some project expansions are best avoided. If the potential expansion appears to: 

• be motivated by a desire to bring a more “positive” conclusion, 

• be too far outside the original purpose, 

• have no ready, reliable data source, 

• place too much burden on a particular individual or group, or 

• require more investment than it’s worth, 

then maybe you need to push back. And to be very practical, sometimes you just can’t accommodate new requests because of important deadlines or budget limitations.  

 If you indulge scope creep when any of those red flags are present, you run the risk of damaging relationships or delivering a late or sub-par evaluation product.  

---

If you do decide to expand your scope, make sure you document that decision (Remember the JCSEE Program Evaluation Standard on Evaluation Documentation?). Savvy project managers will have a change log to note any changes and their reasoning. You should probably update the evaluation plan, noting the date of the scope change, and share with the team. There may even be a place to include the rationale in your evaluation report, perhaps in an appendix.  

We hope these points are helpful the next time you need to make a decision about scope creep. 

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---

 

 

As an evaluator, a big part of our role is to collect data. Sometimes that data comes from administrative databases that require data sharing agreements, but sometimes it comes from people, who need to consent to sharing their information (or data) with you. So let’s talk about what consent is (and isn’t).

I’ll share the punch line with you up front: consent doesn’t have to be a scary, overly burdensome process. If you keep some key principles in mind, obtaining the right consent can be pretty straightforward.

There are different types of consent, many of which don’t apply in an evaluation context: consent for medical procedures, legal consent, etc. It is this legal consent, and the “signed consent form” that most people think of when they think of consent.

Consent, broadly, is giving permission for, or agreeing to, something happening. Informed consent means individuals consent with full knowledge of the nature of participation, alternative options, and potential benefits, risks, or consequences to them (more on this below!). As the evaluator, you need to respect an individual’s choice to participate in your evaluation. For an individual to participate, they need to know what those pros and cons or benefits and risks are! Finally, assent is given on by a person unable to provide their own consent (e.g., children or adults with disabilities).

As an evaluator, it is best practice to consider the consent process through an ethical lens, and then to document that process and the decision-making rationale. The documentation serves to provide evidence of the process.

---

What is Informed Consent?

Informed consent requires that the participant be informed of: 

• all the known or suspected risks and benefits of participation 

• the complete nature of participation (that is, what are they being asked to do or share) 

• the withdrawal process (if there is one) 

• their rights to access their own information 

• benefits to the project 

• what will be done with the information 

• who will have access to their information and confidentiality will be maintained 

• compensation or expenses 

However, writing this out in a consent form is not the end of informed consent. An important part of consent is that the participant is provided the opportunity to ask questions or clarify information. This dialogue is important to ensure the potential participant has all the information they need to make an informed decision, but also for the project lead to ensure that the participant properly understands the information.  

A review of consent literature has shown that there are major opportunities for improving the consent process! Falagas et al1 found that only 54% of consenting participants understood the study’s aim, 47% understood the voluntary nature of participation, 44% understood the right/ability to withdraw, 50% understood the associated risks, and 57% understood the benefits to participating. These results highlight the importance of an open discussion where it is the aim of the evaluator not only to inform, but to ensure understanding. You are not collecting informed consent if your participant is not informed!

---

Methods of Obtaining Consent

1. Consent forms (aka written consent)

The purpose of a written consent form is to provide all the information regarding participation to the participant in a clear, systematic way. The written consent, whether signed or not, can provide evidence that there was, at minimum, a presentation of information to the participant.

Reminder: Having a consent form does not necessarily mean you have informed consesnt, and informed consent does not necesarily require a consesnt form.

A systematic review by Nishimura et al2 found that an enhanced consent form or an extended discussion around consent were the most effective methods at improving participant understanding. An enhanced consent form aims to simplify consent to optimize understanding. It can include increased font size, enhanced literacy level, or additional materials including leaflets. Microsoft Word has features that can help you assess readability and literacy levels or look into an app like Hemingway.

If a paper consent form is used, it is important to draft the consent with the target audience in mind. Health Canada recommends a reading level of grades 6 – 8 for the general population3. Here’s a quick checklist for what to include in a consent form:

• purpose for the data collection 

• description of the complete nature of participation, which may include time    commitment 

• risks 

• benefits 

• assurance of the voluntary nature and the right to withdraw 

• use of information 

• confidentiality 

• contact person 

• any appropriate legal disclaimers, or reference to research grants or ethic approvals 

Reading Level Tips: To reduce reading level, try to make sentences shorter by removing unnecessary words or breaking a long sentence into 2-3 separate sentences. Choose words with as few syllables as possible.

Try to avoid jargon and acronyms and assess the literacy level (aim for Grades 6 – 8).

It is common for the project lead to keep a copy of the consent, and also provide a duplicate copy for the participant to keep.

A word on signatures: A signature on a consent form has its roots in legal consent. A signature may act as evidence not only that the person acknowledges the information in the consent form but that they have, in fact, provided consent. However, unless you have funding, legal, or organizational policies around signatures, they are not necessary to obtain informed consent. In fact, if you are collecting data from vulnerable populations, there can be risk in keeping a record of a participant’s name, especially if it can be linked to their data. An ethical principle of evaluation is to collect the bare minimum data required, so consider carefully if you absolutely need a signature. 

2. Verbal consent

Verbal consent can be an ethically sound method of obtaining consent. Scripting a consent process will ensure that each participant is given the same information about the project. Or, a consent form may be used as an aid or resource. As in a written consent, the language used in a verbal consent process should use lay language and keep the target audience in mind. A script or consent form can act as evidence that a consent process was documented and that each participant engaged in a conversation before giving or denying consent.

3. Implied consent

Implied consent is when a participant does not explicitly express consent but consents via action or behaviour. It can also be a valid option in appropriate circumstances. 

In some evaluation projects, the project leads do not interact with participants. For example, a lead would like a survey handed out by front desk staff at a service centre to each customer. In this case, the project lead is not available to obtain consent and placing the onus of obtaining consent on the front desk staff may be inappropriate. Once a project has been deemed ethically sound, it is possible that a participant can give informed consent by action. For this to occur, the participant must have access to the same information as one would receive in a written or verbal consent process. This information may come as a detached written document or the information may be included at the start of any paper data collection (e.g., a survey). 

Example Text: By completing this survey you are consenting to have your responses used in aggregate and shared back with program staff to be used in quality improvement.

There should be a statement that indicates that by moving forward with participation (e.g., by completing this survey), that the participant understands and accepts the nature of participation and provides consent. In this case, the completed participation, along with documentation of the consenting process, would act as evidence of consent. Because capacity to consent cannot be assessed in this way, ethical review of the project is recommended.

---

In all cases, the project leads should have a clear, documented process for how consent will be obtained, and how (if necessary) it will be reassessed throughout the project. If you are unsure if there are any policies around how you obtain consent, check in with your local health authorities, associations/colleges, or governmental acts that may have policies around consent. 

If a consent process is used, the participant should be informed of the intended use of the data, including an intent to publish data. The intent to publish, however, does not necessitate requiring consent. Publishing data while maintaining confidentiality is ethically acceptable. 

Stay tuned for Part 2 of this discussion that dives more into how to obtain consent and a list of principles to consider! 

In the meantime, check out Eval Academy tools for some consent form templates we have for interviews and focus groups.

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---

 

 

Technological advances have resulted in the collection of large amounts of data and the availability of data continues to skyrocket. To give you a perspective, more data has been collected in the past two years (2019 & 2020) than the entire human history before that.

This post isn’t about big data – we know there are more than enough articles about big data out there. Here, we’ll focus on how evaluators can (and should) clarify details about the data being used for evaluation. In other words, how and why build an evaluation-specific data dictionary.

---

What is a Data Dictionary?

Definitions of “data dictionary” vary but it is generally understood to be a common language for quantitative data. Data dictionaries provide a precise vocabulary for specific data elements and help to standardize a dataset and ensure that the relevance, and quality of data elements, are the same for all users. Data dictionaries describe the meaning and purpose of data elements within the context of a project and provide guidance on interpretation.

Why Use a Data Dictionary?

It is ideal to have a data dictionary whenever you have quantitative data that will be used and shared by multiple people or groups. Without precise definitions, it is very easy to arrive at different results while using the same dataset. Confusions can be avoided by documenting data definitions and parameters and sharing them with all stakeholders.

Although creating a data dictionary is time-consuming, having precise documentation that can be used by all stakeholders promotes efficiency. Look at the following example for an online health education program:

• The program team defines the “number of program participants per week” as the total number of participants that completed the online module per week.

• The IT team defines the “number of program participants per week” as the total number of participants that accessed the online module per week.

As evaluators, if we didn’t examine the definition of “number of program participants per week” meant, we might draw some incorrect conclusions, and risk making unreliable or even dangerous recommendations.

A data dictionary that is prepared collaboratively between the evaluator and stakeholders can prevent confusion and promote alignment. In short, data dictionaries can:

• Provide consistency in the collection and use of data across multiple users;

• Make data analysis easier;

• Promote usability of data; and

• Increase confidence in the data, results, and decisions.

How to Prepare a Data Dictionary

Before embarking on the task of creating a data dictionary, ask the program team if there’s an existing data dictionary for the dataset. It is a common practice to share a dataset with the data dictionary if there is one. However, the project team/client might not think to share the data dictionary with you. If there is a common, vetted, and documented data dictionary, it may not be necessary to create a new one.  

 The built-in active data dictionary can be used in most data management systems including MS Access and SPSS to generate documents as needed. Below is an image of a simple SPSS codebook output.  

Alternatively, if your data set is in MS Excel, you can use MS Excel or Word for documentation. Creating and managing a data dictionary is an iterative process; the definitions for the data dictionary categories and the relationships need to be revised regularly. Often data dictionaries in program evaluation contain the following:  

• A list of data objects: names, metrics (measurement units) and definitions; 

• Inclusion and exclusion criteria: specify cases to be included or excluded; 

• Data Source(s): specify the source of data;  

• Data Update: state how frequently the data is updated and available (e.g., weekly, monthly, annually);  

• Limitations: specify any considerations that would impact the use of the indicator (can comment on reliability and validity of the data and include any other detail); 

• Missing data: state if there are any missing values and how they were handled;  

• Technical notes: provide technical details which help interpret the data presented; and 

• Approval and sign-off: a data dictionary should be created collaboratively with approval from all those that will use the dictionary. After revisions and edits, and it should be signed off by the team leads to finalize the document.   

---

In summary, a data dictionary is a great evaluation tool for projects with quantitative data. A data dictionary is time-consuming to prepare; however, it can promote efficiency and accuracy in the long run. Try building one for your next evaluation project.

While you’re creating definitions, check out our Performance Measures Definitions template.  

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---

 

Welcome to our monthly roundup of new and noteworthy evaluation news and resources – here is the latest.

Have something you’d like to see here? Tweet us @EvalAcademy!

New and Noteworthy — Reads

The Ultimate Guide to Effective Data Collection

A free 30-page eBook titled, “The Ultimate Guide to Effective Data Collection” can be downloaded from Atlan. The eBook focuses on survey design and administration. The nine chapters in the eBook are intended to help you design a survey that will give you high-quality data.

Evaluation Plan Rubric

EvaluATE recently released an Evaluation Plan Rubric. This rubric was used as part of study EvaluATE was conducting to assess evaluation-related content of its ATE proposals. It lists criteria to consider when assessing evaluation proposals and therefore is a useful resource for those of you involved in selecting an evaluator. However, the rubric is also a helpful reference document when writing evaluation proposals.

Defining and Measuring Diversity, Equity, and Inclusion

EvaluATE has published its survey findings of 210 ATE evaluators. One subsection of the survey examined how evaluators define and measure equity, diversity and inclusion in their projects. The survey found that more participants indicated that they measured diversity than equity and inclusion. Surveys were the most frequently reported data collection tool used across all constructs. Surveys asked diversity questions related to gender most often, then ethnicity, then race.

New and Noteworthy — Events

Most Significant Change

Organized by: Clear Horizon Academy

Date: September 10, 2021

Instructor: Jess Dart

Program Monitoring: The Key to Successful Implementation

Organized by: EnCompass

Date: September 21 & 23, 2021

Instructor: Kerry Bruce

Data Storytelling

Organized by: EnCompass

Dates: September 27 & 30, 2021

Instructor: Andy Krackov

 

Early in my career as a consulting evaluator, I landed a major contract. The contract was to evaluate a nationally-funded province-wide quality improvement program in health care. The funder specified that I evaluate using the RE-AIM framework. 

Enter: Googling about RE-AIM. 

I know we’ve all had to-do items either personally or professionally where we just want someone to “tell me what to do.” Sure, I can read all the peer-reviewed articles and evaluation textbooks, but this framework has been used in projects all over the world – hasn’t someone put together the “RE-AIM for Dummies” book? Surely someone somewhere can point me in the direction of the first steps and key lessons learned. If it existed at the time, I didn’t find it. 

Years later, I was working on an academic research team using Framework Analysis to analyze a huge set of qualitative data. Same scenario: someone please just tell me how to start and what to do! The difference this time was that someone had: I found it, it did exist, and it was awesome. Parkinson et al had published a detailed description of their use of Framework Analysis, complete with missteps, backtracking, and all (Parkinson, Eatough, Holmes, Stapley, Midgley, 2015). 

I love these experiential descriptions. I love reading about moving from knowledge gathering to action. I love sharing failures. What better way to learn? 

So, having used RE-AIM a handful of times on a few major initiatives, here is my account of how to use the RE-AIM framework in your evaluation planning, implementation, and reporting. 

---

What is RE-AIM?

In quick summary, for those less familiar, RE-AIM was originally developed to assess the public health impact of interventions, based on five domains: 

• R – REACH 

• E – EFFECTIVENESS 

• A- ADOPTION 

• I – IMPLEMENTATION 

• M – MAINTENANCE 

RE-AIM provides great structure to an intervention that is well defined, but it may not pair well with something like Developmental Evaluation. It can certainly be used with a Utilization-Focused Evaluation approach.  

---

Step 1: Do some basic research.

I recommend the following: 

• Where it all started, the first paper to describe RE-AIM. 

• The academic version of this article, describing what it means to use RE-AIM.  

• A more recent update, describing the evolution and application of RE-AIM in 2019. 

• And, lucky readers, now there is a full website dedicated to RE-AIM, complete with a comprehensive list of resources. 

But I did say I was going to be more practical than just telling you to do all the academic background reading I was so desperately trying to avoid so many years ago, but truthfully, you can’t get around needing some background knowledge. My goal is to share with you a summary of the content on the RE-AIM website but also share some learnings from my own experiences.  

Step 2: Build (and implement) your evaluation plan.

The good news is that the RE-AIM framework gives you your key evaluation questions. Of course, you can (and should) supplement and add detail. 

So, where RE-AIM says, “Have I reached my target population?” you may adapt to “How many clients participated?” or “How many patients had access to the program?”  

Where RE-AIM says, “Was my intervention effective?” you may add in detail “Did my intervention improve patient-reported outcome scores?” or “Did my intervention improve [insert primary outcome measure]?” 

I like to structure the data matrix section of my evaluation plan right around the RE-AIM domains, like in the table below. 

One thing to note early on here, is that RE-AIM is a clever acronym ordered so that it can be read or pronounced easily, but, in my opinion, it’s a little misguiding, giving a sense of purpose in order where there is none. From my experience, it probably goes something more like ARIEM, a little less catchy. (Interestingly, in researching for this article I found a small hidden footnote on the RE-AIM website that admits this exactly!) 

RE-AIM can be effective in the actual planning of the intervention. As an evaluator, I always advocate for being part of the design team. With RE-AIM your goal is to get the team thinking about these five dimensions: 

• How will they be recruiting (Reach/Adoption)?

• Will it be representative (Reach/Adoption)?

• What are their goals or outcome measures (Effectiveness)?

• Do they have a clear plan of how they will achieve those goals (Implementation)? And so on. 

I think RE-AIM lends itself well to formative and summative evaluations. Several times I have drafted formative and summative evaluation plans for a single project by splitting up the RE-AIM acronym.

Adoption, Reach, and Implementation are those things in an intervention that can be course-corrected. Think of these as your process evaluation metrics. If you aren’t reaching your target population, don’t have organizational adoption, or aren’t implementing according to plan, you won’t be effective or maintain anything of worth. This is your formative evaluation. Then, Effectiveness and Maintenance can assess outcomes and sustainability as part of your summative evaluation. 

REACH:

Reach is likely just a count, but can be supplemented with qualitative data captures for a deeper understanding. 

e.g., We designed a training program aimed at Grade One teachers. Our city has 500 teachers and 286 participated in our training. Our reach was 286, or 57%. 

You could then go on to describe the demographics and how they differed (or not) from those who did not participate. Be sure to be clear about any inclusion/exclusion criteria! Reach is also where you can include questions that address access, equity, diversity, and inclusion: are participants representative of the population? Are we reaching those who would benefit most from the intervention? 

EFFECTIVENESS:

Think of this as a “traditional” evaluation – this is: Did the program work? and What difference did it make? This domain is where you will report outcome measures. Any number of methodologies would be appropriate here, depending on your specific intervention. Effectiveness may well be the bulkiest section of your evaluation plan. As in any evaluation, triangulation is a good idea to aim for. 

e.g., Our training program had a goal of training teachers to use a new method of teaching reading to Grade One students. Our effectiveness measures may include: # of trained teachers using the method (process measure or output) and % of students with improved reading skills (outcome measure), or the actual % improvement in reading score. 

Side note: RE-AIM is not mutually exclusive with other frameworks. I have often–in the evaluation of training programs–embedded the Kirkpatrick evaluation framework into the “E” and “I” of RE-AIM. The RE-AIM website actually recommends layering with PRISM. 

ADOPTION: 

It is easy to confuse reach and adoption. I struggled with this at first. For me, it helps to think of them as the same concept but at different setting levels: Reach is about individuals or participants, whereas Adoption is about groups or organizations. Adoption is asking: What organizational support do you have? So, similar to Reach, this is likely also a count. And, like Reach, you can supplement with additional data for a deeper understanding. 

e.g., How many schools supported teachers to participate in the training. How many schoolboards supported the schools to support the teachers? Our city has 300 schools; 120 supported teachers to participate: 40% adoption. Our analysis shows that there was an underrepresentation of rural schools and overrepresentation of inner-city schools. 

Like Reach, you could go on to describe characteristics of these organizations and how they supported the initiative. I have often used Adoption formatively to understand why these organizations endorsed the project or bought in. This exploration can help with spread and scale, or, if things aren’t going well, it is a great way to course correct. I have also included interviews or focus groups with the organizations that did not engage, to understand key barriers. 

IMPLEMENTATION:

Implementation is huge. There are whole fields about Implementation Science. In RE-AIM evaluation, you are primarily concerned with fidelity to the plan: Was the intervention implemented as intended? What adaptations were made? How consistent was implementation? Completion rates may also be an appropriate measure here. 

e.g., Interviews or surveys with the trainers identified barriers and enablers for successful training sessions. Interviews or surveys with the operational team identified barriers and enablers for recruitment, training the trainers, developing curriculum, building engagement and buy-in, etc.  

You could definitely layer on any number of implementation science frameworks here, but likely this is not the key area of interest for your stakeholders and doing so would make this beast unwieldy and hard to manage. Key tips here include considering how each level contributes to implementation: What did the adopted organizations do? What did your organization do? Don’t forget that your own design and operational team are key data sources too! 

Your “results” here are likely descriptions of barriers and enablers along with formative lessons learned and resultant adaptations. 

MAINTENANCE:

I’ll be honest: I have definitely turned in a final evaluation report before the program has reached a stage to be evaluated for maintenance. New initiatives tend to focus on implementation and first-round outcomes. I have, however, been fortunate enough that this hasn’t always been true. In one initiative, I used annual data reviews to look at maintenance of outcomes. In this particular initiative, we were happy to see maintenance, but we also learned that there was a significant plateau or ceiling effect of both outcomes and reach. This isn’t a huge surprise given what we know about the Diffusion of Innovation. As an evaluator, I could then facilitate discussions like: How will (or should) we attempt to reach those laggards? Will they take up 80% of the resources? 

In fact, this is an example of why applying a framework to your evaluation is helpful. If you build in a maintenance evaluation from the start, your team will know that this is planned and you will have the capacity to do the work when the time comes. 

So, if you are fortunate enough to be able to evaluate maintenance, it is likely a repetition of many of the measures that came before – you may take a look an ongoing reach and adoption: have you plateaued or continued to spread? You may look at effectiveness outcomes: have you sustained the gains you made? 

e.g., Annual check-in of reading scores in Grade One (and now Two) children. Updated participation counts to assess spread and scale. 

There is a handy Checklist for some key questions and considerations for each domain. It’s also worth noting that there is nothing holding you to evaluate all five dimensions. I say that begrudgingly though because RE-AIM was developed so that evaluators didn’t overlook key dimensions that are essential to program success. But sometimes there are valid reasons that one of these dimensions may not be relevant for your intervention.   

Step 3: Reporting

The RE-AIM website asks you to consider quantifying or scoring the five dimensions for a visual display:

I’ve never done this. This may be informative to you, as the evaluator, but I find that most stakeholders are less interested in the details of the evaluation framework you’ve applied and more interested in the “So what? Now what?” I certainly have used the RE-AIM structure to guide my reporting, but I don’t think it’s required. The key here is to know your audience – how aware are they of RE-AIM? If you were involved in the planning and they built key evaluation questions into the RE-AIM framework, using the five dimensions in your reporting may be appropriate, but in my experience, you can also draft a really great evaluation report that was based on RE-AIM without being tied to the domains as your section titles. Eval Academy has some great articles on how to draft that killer evaluation report.

---

Things have changed in the 10 years since I first used RE-AIM. There have been a lot more examples published and a lot more content is available. My goal today was to try to synthesize the key points in one place for you, and to share some lessons from my own experiences. I have found RE-AIM to be both highly structured, providing directed guidance, but also flexible enough to allow you to explore in greater depth the key areas of interest for your evaluation. 

So, hopefully, you aren’t as in the dark as I felt when I was first tasked with using RE-AIM. It’s one of many tools for evaluators to consider and one that I’ve had lots of success with! If you want to talk more about whether or how to use RE-AIM in your next evaluation project, consider booking some time with one of our evaluation coaches.  

Speak with an Evaluation Coach

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---