evalacademy

To celebrate the end of the year, we’ve highlighted Eval Academy’s top ten posts and resources from 2024.

What posts or resources would you like to see in 2025? Comment below, tweet us @EvalAcademy or connect with us on LinkedIn!

---

Top posts of 2024

10. How to link surveys in Qualtrics with a participant ID

This is a step-by-step guide for linking your surveys in Qualtrics with a participant ID to assess change over time. Linking surveys together means that you’re connecting the data from all your surveys in a way that lets you track an individual respondent.

9. How to write good evaluation questions

The article outlines essential considerations for writing effective evaluation questions, emphasizing the need for questions to align with the evaluation’s purpose and involve stakeholder collaboration. The article provides guidance on the optimal number of questions, suggests the use of thematic clusters, and underscores the importance of language precision in crafting questions, concluding with indicators of well-crafted questions and next steps in the evaluation process.

8. What you need to know about member checking

While member checking is commonly used in qualitative research, it’s less commonly used in evaluation and we think that should change! In this article, we review what member checking is and why, when, and how you should use it.

7. Evaluation question examples by type of evaluation

This article takes a look at how using different evaluation strategies or frameworks can help you to craft perfect evaluation questions.

6. Differences between Theory of Change, Log Frames, Results Frameworks and Logic Models – what are they and when to use them

You might have noticed how there seem to be quite a few evaluation tools doing very similar things. It’s no surprise that these tools can sometimes be confused, and the lines can be blurred around when and how to use them. To help you on your way to deciding which tool is best suited to our evaluation, this article compare and contrasts few of our favourites.

5. Creating a qualitative codebook

A codebook for qualitative research is a stand-alone document that contains a list of themes, codes, and definitions that you are using in your qualitative analysis. This article outlines the structure of a qualitative codebook as well as steps to follow to create your own.

4. Finding the right sample size (the hard way)

For those interested in calculating sample sizes by hand, or getting a better understanding of the math behind many sample size calculators, this article outlines the formulae used to calculate sample sizes.

3. How to complete an environmental scan: avoiding the rabbit holes

This article is aimed at those who are new to environmental scanning and are looking for new ways to support program planning and improvement.

2. Everything you need to know about Likert scales

The Likert scale is one of the more commonly used rating scales in surveys. As evaluators, we should know a thing or two about it, and how to navigate some of the decisions involved in using a Likert scale. This article introduces Likert scales and how to use them.

1. Interpreting themes from qualitative data: thematic analysis

This article supports evaluators who are new to qualitative data analysis. We start by defining thematic analysis, then give you a 5-step process to complete your own analysis. We end the article by highlighting some common challenges with thematic analysis.

---

Top resources of 2024

Complete Reporting Bundle

CA$0.00

10. Complete reporting bundle

Discover the Complete Reporting Bundle from Eval Academy, your all-in-one resource for reporting on your evaluation. This bundle equips you with the tools to excel in reporting including templates and tools for styling and structuring your next report. Download now and elevate your reporting to new heights!

9. Evaluation status update template

A status template to keep stakeholders informed on how the evaluation is progressing (Project Management tool).

8. Outcome statement tipsheet

This tip sheet lists the key elements you need to include in your outcome statement. 

7. Survey questions infographic

This infographic outlines 8 types of survey questions you should avoid and explains why you should avoid them.

6. Stakeholder matrix tipsheet

This tip sheet provides basic tips for creating a stakeholder matrix to better understand who your stakeholders are and how they should be involved.

5. Theory of Change tipsheet

Tips on how to create a theory of change.

4. Outcome harvest template

This Outcome Harvesting Substantiation Form streamlines the documentation of a program’s impact by capturing outcome statements and assessing observed changes. It enables you to evaluate significance, determine program contribution, and classify outcome types, facilitating informed decision-making and demonstrating program effectiveness.

Evaluation Question Checklist

CA$0.00

3. Evaluation kick-off meeting agenda template

This customizable resource provides an agenda template for an “Evaluation Kick-off Meeting” for a program or project. The meeting aims to initiate the evaluation process and includes various activities such as introductions, identification of team members and their roles, clarification of program goals, evaluation planning, and communication strategies. The template downloads as a Word document.

2. A beginner’s guide to evaluation

This infographic is for people who are new to the field of evaluation and want to learn more about it. The purpose of the infographic is to provide newcomers with valuable resources that answer three important questions about evaluation. The infographic includes a one-page PDF with clickable links to relevant resources, making it easy for beginners to explore and learn more about evaluation.

1. Evaluation question checklist

This checklist acts as a tool to score your evaluation questions. This resource downloads as a Word document.

This article is rated as:

 

 

Eval Academy just released a new, free template: “Top 10 tips for facilitating an evaluation planning meeting”

Top 10 Tips for Facilitating an Evaluation Planning Meeting

CA$0.00

Curious about how to facilitate effective evaluation planning meetings? Our new template provides clear, actionable steps to ensure your meetings are productive, inclusive, and result-oriented.

Who’s it for?

This template is perfect for:

• Evaluation facilitators

• Program managers

• New and seasoned evaluators

• Project leads and decision-makers

What’s Inside?

This infographic outlines:

• The top 10 tips for facilitating evaluation planning meetings

• Key preparation steps to ensure success

• Practical advice for conducting and wrapping up meetings effectively

 

Learn more: related articles and links

• How to Kick Off Your Evaluation Kick-Off Meeting

• Evaluation Kick-Off Meeting Agenda (Template)

• What conversations do you need to have at the start of an evaluation?

• Information Request Checklist

This article is rated as:

 

 

Recently, I took over as an evaluation lead for a large and complex evaluation project. The transition happened suddenly, much earlier than the scheduled date. Thus, we lost much of our preparation time, and I had to take over immediately.

Taking over a complex project has made me realize the importance of preparation and robust project management processes. Managing a program evaluation project requires careful planning, comprehensive communication, and a deep understanding of the project’s objectives, partners, and methodologies. Thus, when I took over the project, I had to learn about all aspects of the project and the evaluation quickly to maintain continuity, quality, and trust. This article outlines my reflections and essential steps for a smooth transition.

---

1. Preparing for the Transition 

Establish a Transition Plan: Before the transition begins, both the outgoing and incoming evaluation leads should collaboratively create a comprehensive transition plan. Of course, this is best practice and not always feasible during emergencies and/or unplanned staff changes. This plan should include a specific timeline for the handover process and key milestones. It should also clearly outline the duties of both the outgoing and incoming leads during the transition period.

Share Essential Documents with the Incoming Evaluation Lead: If it isn’t already accessible to the incoming evaluation lead, the outgoing lead should compile all relevant documentation. This can include:

• Project scope and objectives: project description, literature review, Terms of Reference.

• Evaluation plan and tools: evaluation plan, data collection tools, data files, reports and presentations generated by the evaluation team to date.

• Project management tools: project management documents such as the evaluation budget, invoices with deliverables and timelines, a list of project partners with roles and responsibilities, and contact information.

In addition, ensure that the incoming evaluation lead has access to all necessary digital tools and platforms used during the project. This includes project management software and data management systems (databases, survey tools, analytics, and reporting software vital for the evaluation process). Lastly, the incoming evaluation lead must have the appropriate permissions and clearance to access the project data.

---

2. Knowledge Transfer

During evaluation planning and implementation, evaluation leads learn a lot about the project and have an in-depth understanding of the project. It is critical to pass on this knowledge to the incoming evaluation lead as much as possible. An effective way of such knowledge transfer is through a series of meetings between the outgoing the incoming evaluation leads. During these meetings, the following topics should be addressed:

• Project Background: Discuss the history of the project, including its inception, any changes in direction, and important decisions made along the way.

• Partner/Client Engagement: Review clients and key project partners, their roles, expectations, and work styles (e.g., how the lead connects with them, how responsive and engaged they are). Discuss any past interactions, covering any formal and informal feedback from clients and partners and future engagement strategies.

• Timeline of Activities: A timeline visualizing completed and upcoming project milestones.

• Challenges and Risks: Discuss any ongoing challenges, unresolved issues, and risks associated with the project.

• Pending Tasks: Identify tasks that are pending or upcoming tasks that require immediate attention.

If it isn’t possible to arrange a series of meetings with the outgoing evaluation lead, you can set up regular meetings with the other evaluation team members. When I took over the project, having a regular meeting with the team was valuable for getting an in-depth understanding of the project and upcoming deliverables. Some details came out during conversations, and I feel those details have enhanced my transition experience, allowing me to feel confident and well-informed. The points mentioned above are still relevant topics to discuss with the team.

If the project was handled solely by the outgoing lead and no one else from your organization was involved in the evaluation, then review their notes and files to learn more about the topics mentioned above.

---

3. Internal Team Integration

If the incoming evaluation lead is new to the organization, establish a formal introduction opportunity and informal connections to build team rapport. Clearly outline the evaluation team’s roles and responsibilities, including the incoming evaluation lead’s. Discuss as a team the preferred work styles and expectations.

Again, encourage team meetings during the transition to promote open communication. Consider having a frequent team check-in to gain insights, address concerns, and foster team cohesion. Establish a mechanism for team members to provide feedback on the transition process.

I didn’t have to go through the team integration part, as I worked closely with the evaluation team supporting the project (Three Hive Consulting is a small and closely-knit organization). However, for larger organizations, building the evaluation team and making sure they have a good working relationship is essential, as team members need to collaborate and provide guidance to higher-ranking staff.  

---

4. Client and Partner Communication

Identify key project partners and clients and establish a communication plan to inform them about the upcoming changes. You don’t want project clients and partners to accidentally find out about the transition without following proper protocol. If you are working as an evaluation contractor, swapping staff might be easy and non-issue for some clients, but others might want detailed information about the incoming lead and how they fit in the role. Thus, plan to inform clients and partners as early as possible and per your contract and arrangement. When announcing the change, consider the following steps:

• Announcement Message: Craft a formal announcement regarding the transition, introducing the new evaluation lead and outlining their qualifications.

• Engagement Strategy: Discuss how the project will continue engaging clients and partners, ensuring that their needs and feedback are prioritized during the transition.

• Meetings: Schedule meetings between the new evaluation lead, clients and key partners to strengthen relationships and address concerns.

In my first meeting with the clients and partners, I summarized my background and experience and highlighted my working style. Since I had worked closely with the outgoing lead in the past, I was able to articulate how my evaluation management style differs from and aligns with the outgoing lead’s style. I believe this reassured the clients and partners gave them an idea of my working style.  

---

5. Continuous Project Management

• If possible, once the incoming lead has taken over the evaluation project, establish a period of continued support from the outgoing lead to provide guidance and answer questions as they arise. This has been a lifesaver for me as the outgoing lead I was covering for availed themselves to answer my questions after I took over. Sometimes, a 1-minute conversation or a text message can save many hours of frustration and inefficiency.  

• Solicit feedback regularly from clients, partners and the evaluation team to identify gaps and areas for improvement in the transition process for future evaluations. Incorporate lessons learned into best practices.

---

Conclusion

Overall, having a good project management and record-keeping system is valuable during planned and unplanned transitions. As the incoming evaluation lead, I was initially overwhelmed by the amount of information and complexity of the projects I inherited. However, my colleague’s meticulous project management system and our organization’s record-keeping processes were a saving grace.

At Three Hive, there are a few things we do as part of project management that made my life easier when I suddenly took over an evaluation project, which include:

• Having a detailed and updated evaluation plan. Like most evaluators, we are diligent in preparing a comprehensive evaluation plan, with methods, timeline and roles and responsibilities.

• We often write detailed meeting notes in a shared notebook, highlighting decisions made and actions required.

• If it’s a large evaluation, we sometimes prepare a monthly status update report outlining risks and mitigation strategies.

• We keep other team members up-to-date and engaged, often through quick project team huddles.

• We have a cohesive and supportive team that is willing to provide support as needed. We prioritize work based on need, urgency and time sensitivity.

Overall, transitioning to an evaluation project is hard and time-consuming. It takes a long time for the incoming evaluation lead to settle into their role. I found inheriting a project much more stressful than starting an evaluation project from the beginning. So, prepare as much as possible and give it time and grace!

This article is rated as:

 

 

At Three Hive Consulting, we recently embarked on an innovative journey to pilot a 4-Day Work Week (4DWW). Because we’re evaluators, we tasked ourselves with assessing the impact of this new work schedule. Here, we share the methods and insights from our evaluation process.

---

Evaluation Methods

Our team of evaluators employed a mixed-methods approach to evaluate our experience of the 4DWW. This included both quantitative and qualitative data collection techniques to provide a comprehensive understanding of its impact.

Quantitative data was gathered through surveys distributed to all employees twice: just before we started, and about 6 months later. These surveys included Likert scale questions to measure various aspects such as work-life balance, productivity, and overall job satisfaction. The quantitative data provided a broad overview of the general sentiment and measurable outcomes associated with the 4DWW.

To complement the survey data, we discussed the 4DWW at our regular weekly and quarterly team meetings. This less formal qualitative approach allowed us to delve deeper into individual experiences, uncovering detailed feedback on the benefits and challenges of the 4DWW. We have also held informal conversations with our clients about the 4DWW.

The combination of survey data and discussion findings offered a well-rounded perspective on the initiative.

---

Key Findings

The evaluation revealed several significant benefits of the 4DWW. One of the most notable advantages was the improvement in work-life balance. Employees reported having more time for personal activities, which contributed to better mental and physical health. As one team member noted, “Having Fridays to get stuff done at home has made Mon-Thurs less chaotic”. This additional day off allowed employees to recharge, leading to more focused and productive work during the remaining four days.

In terms of productivity, our team reported that the need to complete tasks within a shorter timeframe encouraged us to focus more on each task. “I have been able to focus in more on each task, as the 4DWW adds a slight pressure and artificial deadline to get more done in less time,” shared another team member. This focused approach not only enhanced productivity but also allowed for more flexibility to enjoy life outside of work.

We also explored the impact on employee retention. While retention has never been a problem at Three Hive, our team members shared that the 4DWW does make Three Hive a more appealing organization to remain with and that moving back to a 5DWW would be unattractive.

Not all results were positive. In our conversations with clients, one reported that our team being unavailable on Fridays was too great a challenge, and contributed to them not renewing their contract when the term ended. Most clients, however, have been supportive and curious.

---

Challenges and Recommendations

Despite the numerous benefits, the team identified some challenges associated with the 4DWW. One of the main difficulties was adjusting to completing all the work in 80% of the time. This made workdays more intense and required careful planning and time management. “Doing 100% of my work in 80% of the time is a challenge that takes adjusting to and does make the workdays more intense,” shared one team member.

To address these challenges, we implemented strategies such as time blocking and front-loading the week. Scheduling important tasks early in the week ensures that critical work is completed on time. “I have found that scheduling my work and time blocking has worked well for project work,” a team member recommended.

What has proven to be more difficult is understanding the impact on our corporate revenue and profit. Project-based work is highly variable, and many factors influence whether we win new work through competitive award processes, and the extent to which each project yields a profit.

---

Conclusion

The evaluation of the 4DWW by our team of evaluators provided valuable insights into its benefits and challenges. We’ve learned that the 4DWW offers a promising approach to improving work-life balance and productivity, and requires careful planning and time management. We’re happy to announce that based on our evaluation efforts, we have made the 4DWW permanent at Three Hive Consulting. Our team now work a standard day Monday through Thursday, with no reduction in pay.

We now let clients know early on in the engagement process that we are generally not available on Fridays; we do make some exceptions, in the same way that we must sometimes accommodate data collection or knowledge translation in the early or late hours of weekdays, or on weekends. Our email signatures include a reminder that we are away from our inboxes on Fridays. So far, we have had no negative feedback about our 4DWW from potential clients. There may be times when we are not selected and we aren’t aware that the 4DWW was a factor, but that’s okay!  

---

See more examples of evaluation in practice with our other “How We Evaluated” articles here and here.

This article is rated as:

 

 

Eval Academy just released a new, free infographic: “Amplifying the Voices of Emerging and Experienced Evaluators – AEA Poster”!

---

Amplifying the Voices of Emerging and Experienced Evaluators – AEA Poster

CA$0.00

Curious about how Eval Academy is democratizing evaluation knowledge? Dive into our new infographic, a downloadable copy of the poster presented at the 2024 AEA Conference in Portland, Oregon.

Who’s it for?

This resource is perfect for anyone passionate about expanding access to evaluation knowledge, including:

• Students

• Program Managers

• New and Emerging Evaluators

• Seasoned Evaluators

• CEOs and Decision-Makers

What’s Inside?

This infographic reveals:

• The mission and impact of Eval Academy

• How content is crafted to support evaluators at all levels

• Eval Academy’s reach and resources at a glance

---

 

 

---

Learn more: related articles and links

• About Eval Academy

• Reflections from the AEA Conference 2024: Amplifying and Empowering Voices in Evaluation

• Information Overview