Evaluator Competencies Series: Program Evaluability

2.2 Assesses program evaluability.

I can’t remember exactly when or where this was, but at some point during my career as an evaluator, I saw the phrase “Evaluability Assessment” and thought “what’s that?” I know I wasn’t a brand new evaluator, as when I looked it up and learned what it was I thought, “Oh! I’ve been doing that at the start of every evaluation that I’ve done. I didn’t know there was a name for it!”

An evaluability assessment is, much like the name suggests, assessing “the extent to which an activity or project can be evaluated in a reliable and credible fashion” (OECD-DAC 2010; p.21 cited on Better Evaluation). As I learned to be an evaluator, this seemed to be a thing that I naturally needed to do.

An example would be where a client says “I want an evaluation that tells me if the program is achieving its goals”. The first question one would ask is: “What are the program goals?” because I certainty can’t tell you to what extent you are achieving your goals if you don’t have any goals. Similarly, a program may ask an evaluator to conduct an evaluation on whether the program has improved some particular thing for their program participants (e.g., their health, their knowledge of a topic, their social connectedness – whatever thing the program is trying to help its participants improve). In that case, one would naturally ask “how were the clients doing on that something before they started the program?” (i.e., do you have any baseline data we can use to compare to?). Or perhaps it’s a case where the client says “I want to know if my program is working?”, in which case I would ask “What does the program “working” mean to you?”. And that might lead to some work around developing a program theory, or figuring out if they want to know what outcomes are achieved, or if they want to know if their processes are efficient, or whether they are concerned about negative unintended consequences (or maybe all of the above). In my experience as an evaluator, when I ran into situations like this, my first course of action would be to work with the clients to figure out how to get their program into a state in which they are evaluable.

What I didn’t realize in my early years as an evalulator is that in some cases, an evaluability assessment could be a project unto itself. (Check out the Better Evaluation page on Evaluability Assessment if you want to read more about it.)

This may be due to the fact that I’ve always been an internal evaluator (or, as I think of myself on the program I’m currently working on, an external evaluator who is embedded in the program for the long term). So I’ve always had the luxury to be able to work with my “clients” to get them into an evaluable state as part of the work I do with them. Perhaps if I were an external evaluator, I may have come across stand alone evaluability assessments as potential projects.

I couldn’t find any images online that I felt represented “evaluability assessment” (probably not surprising… spell check doesn’t even believe that “evaluability” is a word!). So instead I give you this picture of my cats: