Let’s be clear: sums and frequencies are not the desired product of qualitative questions. In qualitative approaches, we want to describe, to present details and nuances and interesting outliers. But as evaluators, we need to do more than just report what is—we need to comment on what it means. In familiar evaluation terms, moving from the “what” to “so what?”

Qualitative purists may hiss at the idea of quantifying qualitative data. But as evaluators, our job is to apply evaluative thinking to our qualitative findings. Not all findings are as material as others—in other words, the one respondent who thought their nutrition class provided just the right amount of detail is likely overshadowed by the eleven who described feeling overwhelmed at the volume of information. Evaluators would be remiss not to introduce an element of quantification to their qualitative data.

Caveat: I do not intend to suggest that a higher number of respondents reporting a similar answer is always more important. Outliers and small groups matter, and understanding those outliers is a major part of why qualitative approaches are used.

But we do need to be able to describe the proportion of respondents who report similar answers.

The key to quantifying qualitative findings is consistency. Editing reports where descriptions of qualitative data included words like “a lot,” “the majority,” “many” and “most” left me wondering why those particular words were chosen. How is “a lot” different from “many?” Are “the majority” and “most” roughly the same number of respondents? And if I was asking those questions, I know our stakeholders would be asking them, too.

To give my staff concrete guidance, I found this framework… online… somewhere… maybe in 2013? (If this is your framework, or you know who created it, please let me know! I’ve been using these definitions in evaluation and reporting workshops for a few years, and have seen it used in Government of Canada documents, but without attribution.)

Few	Less than 10% of participants
Several	Less than 20%
Some	More than 20%
Many	Nearly 50%
A majority	More than 50%, but fewer than 75%
Most	More than 75%
Vast majority	Nearly all participants, with some still having different views
Unanimous, or almost all	All participants, or the vast majority gave similar answers and the rest did not comment

---

These definitions may work for you. Or you might take issue with some of the ranges and want to create your own. As I said before, consistency is key! Try using this framework in your next report, and include it in your methods appendix.

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---

 

 

Reflective practice is the first competency domain for the Canadian Evaluation Society (CES) Credentialed Evaluator (CE) Designation. It is a part of the foundation that makes up a sound evaluation practice. Reflective practice involves using a learning mindset to stay up to date on new and best practices, integrating evaluation standards and ethics into practice, providing transparency and a balanced perspective, contributing to the profession of evaluation and using self-awareness and reflective thinking to continually improve practice. This last point is reflexivity, or continually reflecting on how oneself and one’s learnings impact the broader context within which a person works.

While most of us generally understand the concept of reflectivity and how to be reflective (learn, stay up to date, apply professional standards and ethics), this last concept of reflexivity can be elusive. Reflexivity often falls into that category of things we know we should be doing, but aren’t really doing, or things we might be doing, but aren’t so sure we’re doing right.

To try and ease your stress about reflexivity and to help guide your practice, we’ve compiled our best tips below. To start, we define reflexivity and outline why you should develop a reflexive practice before jumping in to help you become more reflexive.

What is reflexivity

What really is reflexive practice and how do you do it in a practical manner? If we look to the CES Credentialed Evaluator domain, reflective practice is about having a deep understanding of evaluative theory and practice, applying evaluation standards and ethics, and having an awareness of self and reflection on one’s practice. In essence, it’s about the cycle of learning and growth, both about the field of evaluation and yourself as an evaluator, and using critical insights to improve your practice. Reflexivity can take on many forms, but it is essentially the practice of examining ones’ self as an evaluator, how you have been shaped by the evaluative process and how your values and viewpoints have shaped your evaluations.

Why develop a reflexive practice

You might be wondering why you should be concerned with developing a reflexive practice. In evaluation, we are often tasked with defining or providing information for decision-making about the value or merit of an evaluand (e.g. a program or project). We must pay attention to the needs of different stakeholders, outside political influences, and our own biases. Reflexivity gives us the space to process these elements and critically examine these influences on our evaluations.  How can you measure value if you are not aware of your own values? Through reflexivity, we learn and grow from our mistakes.

How to be reflexive

Reflexivity sounds like a daunting task. How does one regularly and thoroughly critically appraise ones’ self and evaluation practice? In today’s busy world where we are being pulled in many directions, here are some strategies to help you develop your own practical reflexive practice.

1.     Be reflexive often.

Reflexivity should be a continuous process. To ensure that it doesn’t fall to the wayside, carve out time in your calendar and stick to it. Figure out what works best for you — do you prefer to work in larger chunks of time, or to split up your reflexion over the space of a few days or weeks? Being reflexive is something that needs to fit within your current work practices. If you are always rushing out of the office on a Friday afternoon, don’t schedule your reflexive time then. 

In addition to scheduling reflexive time regularly, include it as part of your evaluation plans. Ensure you include time at the end of every major evaluation phase for some project-specific reflexion time. This practice can help you implement improvements in your current evaluation project and save you from future pitfalls in similar projects. A quick check-in about what you did well, what you could improve on, and what changes you will make with this information counts as reflexion.

2.     Be reflexive in a structured manner.

Left to your own devices and without a plan, you can easily use up your whole reflexion time googling how to be reflexive (if that Google search led you to this article, Hello! And welcome to the end of your search.) Set out your questions or focus in advance and stick to them. There are some practical tools to help with this in the tools section at the end of this article.

Bring structure into your reflective practice in a way that makes sense for you. Perhaps it’s a weekly set of questions, free drawing time with a focus or intention in mind, or a daily project journal.

3.     Be reflexive alone.

Reflexion is about reflecting on your own processes, questioning your attitudes, thought processes, values, assumptions and habitual actions in order to understand our roles in complex situations. Nobody else can do this work for you. Ensure that some of your reflexion is done alone.

4.     Be reflexive together.

While you need to be able to think critically about yourself, bringing others into your reflexive practice from time-to-time can help you gain a deeper understanding. Getting feedback from others can challenge your assumptions about yourself. Your coworkers, clients, and colleagues are sources of information to promote learning and growth. Offer to take them out for a coffee, meet up for a walk, or schedule an informal phone call.

If you work as part of a team, consider bringing everyone together for a project debrief. Ask them what worked well in the project, what external facilitators and barriers contributed to your final product, and what internal processes could be improved for next time. You can also examine how you worked together as a team and if there are areas where the team can grow.

5.     Record it!

Part of reflexion is looking back at your growth. Recording your thoughts will help you to look back and see patterns over time. It also helps you to be accountable. Find a method of recording your reflexion that works for you. Keep it simple and don’t overthink it.

6.     Get meta about reflexion.

Don’t do this too often, but every once in a while it doesn’t hurt to be reflexive about your reflexion. How are your chosen processes and tools working?  Are you actually making the time to be reflexive on a regular basis? Are there questions you are avoiding? Have you implemented any of the steps you outlined to make improvements?

Tools and Ideas for Reflexion

• At the end of an evaluation, once you have reported on the findings, go back to the data collection tools you use and consider what changes you would make in hindsight. If your evaluation included audio-recorded interviews or focus groups, go back and listen to them again, this time focusing on yourself. How did your involvement in the data collection impact the participants and your interpretations of the findings?

• If you are looking for a structured process for reflexion, check out the DATA model:

  • Describe what is or has been happening in practice.

  • Analyze the current state of practice; why is this happening in this way?

  • Theorize why things are occurring.

  • Act; make a specific action plan to make changes.

•  Feeling more creative? Carolyn Camman created an adult colouring book with reflexive questions to get your mind going.

• Do something with your hands that you are good at but doesn’t required a lot of concentration. Doodle, bake, work on a puzzle, or play with playdough while you mull over your reflexive questions. Personally, I’ve baked dozens of cookies and a couple of pies as a way to carve out time and keep my hands busy while giving my brain space to think.

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---

 

 

You have developed your evaluation plan, written your questions, and deployed your tools. And now, you have just received your raw data. You are excited to jump in and analyze the data.

But wait.

You note several inconsistencies in the raw data. Three different date formats are used within the same column. You take a closer look and several categorical values are spelled and punctuated differently. Also, several cells contain both numerical and text data. And dispersed randomly across the spreadsheet are blank open cells.

These are a few common data inconsistencies we face as evaluators and analysts when receiving a fresh set of raw data. Data cleaning is often the first step, unless receiving the unicorn of all spreadsheets with perfect formatting, consistency, and accuracy. However, more likely than not, you will experience one or several of these data inconsistencies in your raw data.

This article addresses four common mistakes made with data entry and offers suggestions to improve data entry to increase the overall productivity and, let us not forget, happiness of your colleagues, collaborators, and evaluators.

Date Data

Many may think that “a date is a date,” but this is not always the case within a spreadsheet. When several date formats are used (e.g., 2020-04-13; April 13th, 2020; Apr 13, 2020), the spreadsheet, and your evaluator, may be confused trying to make sense of these values. In Excel, dates are stored in each cell as a numerical value and converted to a pre-selected date format. However, if text values are included with your dates, Excel may be unable to accurately interpret each cell entry.

Best case scenario: a few data points are missed when analyzing or visualizing your data.

However, this best case is not ideal when you, as an evaluator, are needing to present accurate results to your clients, such that they can digest and act on the results of your evaluation.

How do we improve date entry?

• Enter dates in a YYYY-MM-DD format.

• If you prefer a different date format (see Ablebits article for ideas on date formatting), be consistent. Dates entered consistently, regardless of format, are much easier to work with.

• Consider using data validation to prevent improperly formatted dates or text from being entered into your spreadsheet.

• Communicate with data entry personnel and agree on a single date format for data entry.

Text Data

For anyone that works largely with quantitative data, your heart may have just sunk as you read those two words: “Text data.” In this article, we are not talking about open-ended questions for our qualitative analyses (this may be a topic for another article). No, we are referring to text data in the form of Likert scales (e.g., Strongly agree to Strongly disagree) and categorical variables (e.g., gender, ethnicity).

But why does the thought of analyzing text data result in mild anxiety? It boils down to spelling and grammar. When cell entries vary in their spelling and grammar, your spreadsheet software may count each variant of a cell entry as a discrete value. Your five-point Likert scale now becomes a ten-point scale and your categorical response variables now has three unique variants for each potential response.

How do we ensure that text data is entered accurately?

• Consider converting Likert scale data to numbers (e.g., Strongly agree = 1; Strongly disagree = 5).

• Similarly, categorical variables can be coded as numbers (e.g., Female = 1; Male = 2).

• If converting text data to numerical data, provide a guide to data entry personnel to follow (and for you to review when analyzing the data).

• If you prefer to go the route of using text, consider using data validation. Lock cells to accept only values spelled correctly and avoid misspellings and other errors.

Multiple Values per Cell

It is tempting to include several pieces of related data in a single cell. You have created a spreadsheet and asked the person entering the data to record a few biometrics: height and blood pressure.

You ask that height be measured in centimeters. You receive the data and each result is entered with the unit “cm” at the end of each value. When glancing at the data, it is useful to understand the unit of measurement. However, this creates complications when analyzing the data. Your spreadsheet will read each height measurement as a text value. Try to complete a simple calculation and you will receive an error. Units are important but should be included in the column title rather than included with individual cell values.

Now for blood pressure. You are measuring systolic over diastolic blood pressure. A patient gets a reading of 120/80 and the value is entered under a single column titled “Blood Pressure”. While this may save a few seconds on data entry, this can complicate analysis as two values share a single cell. It is better to split the data into two columns: (1) “Systolic Blood Pressure” and (2) “Diastolic Blood Pressure”. Each cell should contain a single, discrete value. It is better to add an extra column to your spreadsheet than attempt to cram multiple data into a single cell. 

How do we avoid multiple values per cell?

• Communicate with data entry personnel and explain that each cell should contain a single, discrete data value.

• If a value has a unit of measurement, include the measurement in the column title and only insert numerical values into your cells.

• With related data, split the results into two or more columns with each column relating to a single value (see Microsoft Office Support to see how to split delimited data into multiple cells in Excel).

• Keep notes separate from raw data values. If you need to clarify or explain a value, add a separate column to include your note.

Missing Data

Now to address missing data. Missing data can come in many guises, from blank cells to NA’s to numerical place holders (0’s or 99’s). While each method of dealing with missing data has its merits, consistency in handling missing data is the key to clean data.

Within spreadsheet software, such as Excel, blank cells will not interfere with calculations. Looking to average a column? Excel will skip the blanks and average the present data. However, insert a numerical place holder, such as 0 or 99, and your averages will now be skewed by the presence of these values. It is possible to work around these place holders, but it is easy to forget about these place holders when working with numerical data as they are not immediately identified as missing values.

An alternate is to use NA to fill in the blanks. This makes it immediately clear that data is missing from the cell. However, a hiccup occurs if calculating values in Excel. Your calculations are likely to error out due to the NA’s present in your data. This can be worked around by excluding NA’s in your formulae. An advantage of using NA’s is when exporting your data to statistical software (such as R Statistical software), which are often designed to handle NA’s effectively. If you import your raw data into such software frequently, consider replacing missing data with NA.

How do we handle missing data?

• Leave no cell blank. While blank cells may work for calculating values in your spreadsheet software, it is unclear whether these cells reflect an absent value (i.e., there is no value available) or that the cell was missed during data entry (i.e., human error).

• Avoid using numerical place holders to code for missing data. These are easily overlooked and will skew data calculations.

• Use a code like NA. While you may be required to adjust your formulae in your spreadsheet software, it will be clear that these cells represent missing data.

• Additionally, NA works well with external statistical software. If you use your spreadsheets for data storage only, use NA as it will work best with your statistical software of choice.

• Regardless of your choice, be consistent with coding missing data and communicate to data entry personnel on the preferred method. 

Conclusion

It is near impossible to eliminate all data entry mistakes; humans make mistakes. However, it is possible to be aware of common mistakes and create procedures to reduce data entry mistakes. Consistency and communication are key when sharing data between teams and collaborators. If everyone is on the same page, it is possible to reduce many of these mistakes. Address these mistakes early, and you will have much cleaner data to work with come analysis time. And clean data results in accurate, efficient, and actionable results.

Tips

• Consistency. Consistently entered data will result in clean, readable data that will significantly improve the efficiency of your data analysis.

• Communication. Communicate data entry protocols and codes to everyone that will interact with the spreadsheet.

• Data validation. Consider using data validation to ensure that only valid data are entered into each cell.

• One piece of data per cell. Each cell should contain a single, discrete value. If you have related data, consider splitting the data into multiple columns.

• No blank cells. Missing data can cause issues in data analysis. Find a code to represent missing data and use it consistently throughout your data.

Further reading

Date formatting: https://www.ablebits.com/office-addins-blog/2015/03/11/change-date-format-excel/

Split text to multiple columns: https://support.office.com/en-us/article/split-text-into-different-columns-with-the-convert-text-to-columns-wizard-30b14928-5550-41f5-97ca-7a3e9c363ed7

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---

 

 

---

New and Noteworthy — Reads

---

A framework for adapting evaluation designs in times of COVID-19

Last month’s roundup focused specifically on COVID-19 and its implications for evaluation. Evaluators continue to produce COVID-19 related resources as the global crisis continues.

The Independent Evaluation Group of the World Bank recently produced a framework organized around four questions to address the evaluation challenges during the COVID-19 pandemic, which include:

1. Should we adapt our evaluation questions and scope? In short, of course. As they state, “we must consider what evaluation scope or angle can bring the most value at this time.”

2. Can we improve what remains feasible? Yep – lots of good ideas on how evaluators can improve how we review and synthesize existing knowledge, including experimenting with AI.

3. Can we find ways around what is infeasible? Lockdown conditions means findings creative ways to engage with stakeholders and collect data (see below under New and Noteworthy tools for a remote survey toolkit).

4. Can we tap into alternative sources of evidence? There are opportunities to incorporate existing sources of data that evaluators do not typically tap into (e.g. geospatial, financial or social media).

Tapping into big data: Lessons from evaluators working in the food and agriculture sector

Speaking of opportunities to incorporate existing sources of data, Eval Forward recently posted a blog post titled Evaluation in the age of big data: Opportunities and challenges in agriculture and food security. In this post, they explore the challenges evaluators face (i.e. time and cost constraints and the trade-offs that are made because of that) and how big data and data analytics can strengthen evaluations. The post provides a list of some widely used big data techniques and their actual or potential applications in food and agriculture evaluation. These techniques include satellite and drones, remote sensors, GPS location data, social medial, Internet search data, integrated data platforms and biometric data. Interested in learning more about these techniques? Check out The UN Global Pulse website for over 100 case studies of how they have been applied.

---

New and Noteworthy — Tools

---

Remote Surveying – A toolkit on how to do it right

60 Decibels (@60_decibels) created a Remote Survey Toolkit to help you navigate phone survey best practices, survey providers, survey questions and more. Now that nearly everyone is practicing social distancing, face-to-face data collection has gone the way of the Dodo bird (well maybe not, but for the time being). The toolkit has lots of user-friendly tips, cheat sheets, decision trees, example questions, and other resources 60 Decibels has collected over the years that they have generously curated for all of us to use and benefit from!

Tools for capturing activities, outcomes, and learnings

Evaluation Support Scotland produces a number of free tools that can be downloaded from its site. Under its COVID-19 page it lists not only tips for evaluating during COVID-19 but practical tools. New tools include: 

• Using contact forms to gather evidence during a call

• Taking stock in a time of change (method sheet)

• Using social media to evaluate other activities (method sheet)

---

New and Noteworthy — Courses and Webinars

---

April 30, 2020

Krazy Glue Messaging: Making Your Evaluation and Research Findings Sticky – Webinar

• Presenter: Kylie Hutchinson (@EvaluationMaven)

May 2020

Evaluation in a time of change – Webinar (May 12, 2020)

• Presenter: Evaluation Support Scotland

Facilitating Evaluation – Online Course

• Instructor: Michael Quinn Patton (@MQuinnP)

Dynamic Data – Mastering Pivot Tables for Engaging Data Viz – Online Course

• Instructor: Carolyn Hoessler (@carolynhoessler)

June 2020

Feminist Evaluation: Not your standard gender-responsive approach! – Online Course

• Instructor: Donna Podems (@DonnaPodems)

Transformative Mixed-Methods Evaluation – Online Course

---

Subscribe to Our Newsletter

Sign up with your email address to receive news and updates from Eval Academy every month.

Email Address
Sign Up

We respect your privacy.

Thank you!

---

 

 

The first evaluation book I read was Michael Quinn Patton’s (aka MQP) Utilization-Focused Evaluation. It was an intimidating introduction to evaluation. The book is 667 pages of subject matter that isn’t exactly light. However, it is full of anecdotes, case studies, stories, pictures and metaphors that inspired me to keep reading. His use of metaphors helped me transcend literal concrete evaluation concepts and theories and instead helped me create images that were easier to understand, process and ultimately use (talk about utilization-focused!)

I am now a full-time evaluation consultant who co-founded Three Hive Consulting – a company that provides people with evaluation expertise they need to learn and improve. Eleven years later, I still refer to that book and a multitude of other evaluation resources MQP has bestowed upon our professional niche over the years. He is the one I, along with many other evaluators, listen to and learn from. Like Yoda, MQP is revered for his wisdom and his power to inspire thinking differently about evaluation ideas. And while he isn’t a small, green humanoid alien there are striking similarities between the two (that go beyond the bald head and unique sweaters).

Yoda trained Jedi for 800 years and had a hand in training almost every Jedi master in the galaxy

When you visit MQP’s website it is obvious he is evaluation’s Jedi master. He has worked in the field for almost 50 years – beginning just when evaluation was emerging as a profession. During that time, he has authored numerous books that include Principles-Focused Evaluation (2018), Facilitating Evaluation (2018), Developmental Evaluation (2010) and Utilization-Focused Evaluation (2008). He has also edited or contributed articles to numerous books and journals.

MQP is an active trainer, workshop presenter and evaluation consultant. He has trained evaluation Jedis all over the world and across different organizational sectors. In fact, his Blue Marble Evaluation is all about training evaluators to engage with and evaluate global change efforts.

Yoda is revered for his wisdom and has tremendous power in the force

Over his 50 years in the field, MQP has accumulated a wealth of knowledge, but his power in the evaluation world doesn’t come from information gathering. Knowledge relates to learned facts, but wisdom means putting that knowledge to good use. Wisdom is about knowing which facts are relevant and being able to thoughtfully apply that information.

Through his work over the years, MQP found that traditional evaluation approaches were not meeting the needs of social innovators – they needed another approach to match the nature and scope of innovations. Instead of imposing logic models and rigid methods and approaches on these innovations, MQP developed an evaluation approach called Developmental Evaluation. With Developmental Evaluation it is okay to begin an evaluation without clear, measurable outcomes when you’re in new territory with massive challenges and you don’t know the solution.

We are all creatures of habit; as evaluators we have our go-to methods. Yoda taught us that to achieve growth and success, sometimes “you must unlearn what you’ve learned.” While MQP isn’t saying we must unlearn traditional evaluation approaches, he is cautioning that to successfully evaluate complexity we need to be ready to change our attitudes and leave the comfort zone that our logic models provide.

Yoda warns Luke won’t be a Jedi until he confronts Darth Vader

MQP says that evaluation “grew up in the projects” and has a project mentality. But evaluators are increasingly tasked to move beyond evaluating projects and programs and instead are called to evaluate non-traditional and new directions like systems change, policy, strategy and transformation.

Evaluators are being challenged to evaluate these new directions because our world is so much more connected. Our interconnected world means global challenges like the COVID-19 pandemic, climate change, migration and inequality, among others, are not confined to borders. MQP developed Blue Marble Evaluation “to ensure that evaluators are prepared to engage with and evaluate these global change efforts” which he says require special perspectives and competencies:

“It means bringing the full arsenal of evaluation thinking, tools, methods, and processes to bear at a global level — and creating new approaches appropriate to the challenges of global systems evaluation.”

Much like when Yoda warns Luke he wouldn’t be a Jedi until he confronts Darth Vader, MQP is cautioning that evaluators won’t be full Jedi evaluators until we confront global challenges (our Darth Vader) by thinking globally, acting locally and evaluating globally.

Yoda is a comical, eccentric, provocative character

MQP is a comical, eccentric, provocative character. Case in point….

Thank you MQP for being our Yoda. Thank you for your wisdom, but more importantly thank you for passing it on. As Yoda taught us, the wisdom we gain in life is a gift to pass along—not to keep to ourselves. To all you evaluation Jedis out there, share your insights with others as Yoda and MQP did with us, for

“in a dark place we find ourselves, and a little more knowledge lights our way”

---

Sign up for our newsletter

We’ll let you know about our new content, and curate the best new evaluation resources from around the web!

Email Address
Sign Up

We respect your privacy.

Thank you!

---