cplysy

Sep 22 2019

Evaluator Competencies Series: Program Theory

2.3 Clarifies the program theory.

I really like helping programs figure out what their theory of change is. Early in my career as an evaluator, I was surprised how often I would work with a program that had no idea what its theory was. Like, you’d sit down with them and ask questions about what they were trying to achieve and how what they thought what they were doing was going to help them achieve it – and they didn’t know. They had never really thought about it. The program was the way it was by some combination of it having been started by someone in some way at some time for some reason and then it had been adapted over the years in response to funding cuts/new funding opportunities/new leadership/new research/[enter all sort of other possible factors here]. While talking about this with my class this weekend (I’m teaching a Program Planning & Evaluation course in a Masters of Health Administration program), one student described the programs that she’s worked on as having been MacGyvered and I absolutely love that description!

Perhaps way back when a program started there had been an idea of a program theory – or possibly not – but it’s been MacGyvered over the years and often there us no record of any original program theory. And so I discovered that an important part of work as an evaluator is often to help the program make explicit the theory of why they think the program will result in changes to achieve whatever it is trying to achieve. Because even if a program doesn’t have an explicit program theory, there is some implicit theory underneath.

And there are many benefits about making your program ‘s theory of change explicit. As an evaluator, I want to know what the program’s theory is so I can design an evaluation to test the theory. But it can also be quite helpful to the program itself – helping them to get everyone on the same page about what the program is actually trying to achieve and getting them to think about whether what their program does is likely to get them there. Also, sometimes mapping out a program theory helps a program to identify that it is doing activities that are not likely to help them achieve their goals. It’s surprising how often programs do things because “we’ve always done these things”, even though they may no longer be needed or relevant. Working through a program theory can help identify those things.

Oftentimes, I work with those involved in the program to clarify the theory by developing a logic model together. There is a debate about whether a logic model is or is not a program’s theory of change. According to Michael Quinn Patton (2012), a logic model is simply a description of a logical sequence, but “specifying the causal mechanisms transforms a logic model into a theory of change”, i.e., you need to “explicitly add the change mechanism” to make it a theory of change. I like this explanation because it reminds us that a logic model on its own isn’t quite enough to be a “theory of change” so we need to think about what is the actual mechanism that is believed to lead to the change.

Thinking about how I do the work of clarifying program theory, I think my tips would be:

however you choose to clarify a program’s theory of change, do it collaboratively with as many people who have an interest in the program as possible. This is important because:
- different people bring different perspectives and thus can help us to more fully understand how the program operates and the effects it can have
- a lot of the value of clarifying a program theory comes from the process. Finding out that people aren’t on the same page as one another about what the program is doing and why, identifying gaps in your program’s logic, surfacing assumptions that people involved in the program have – all of this can lead to rich conversations and shared understanding of the program among those involved and you just don’t get that by handing someone a description of a program theory that was created by just one or two people.
a program theory should be thought of as a living thing. You can’t just map out a program theory once and think “well, that’s done!” Programs change, contexts change, people change… and our theories of change need to change to keep up with all of that!

This topic is also a good time to plug the free online logic modelling software that my sister, her partner, and I created: Dylomo (short for DYnamic LOgic MOdels). You can sign up for free and play around with it. Apologies in advance for any bugs – we created it off the side of our desks, so haven’t had time to add all the features we would like. If you do have any issues with it – or feedback about it – do get in touch!

References

Patton, M. Q., (2012). Essentials of Utilization-Focused Evaluation. Thousand Oaks, CA: Sage.

Image Source

Photo of leaves was opsted on Flickr by Mehul Antani with a Creative Commons licence. Again, I couldn’t find a good free-to-use image for what I was searching for (program theory, theory of change, logic model), but while searching for “change” I found that image of leaves changing colour and thought it was beautiful.
Dylomo logo was designed by my amazing sister, Nancy Snow.

Sep 13 2019

Chief Learning Officer

C-suite leadership roles focus on an organization’s most important functions. Time to introduce the role of the Chief Learning Officer.

Learning — easier said than done. Yet, learning is vital to the success of an organization that seeks to innovate to gain advantage or merely survive – which is most human service organizations these days.

Learning opportunities abound, yet these require energy and attention in order to take advantage of them. Organizationally, this requires leadership and resources to support people across the organization to learn within their areas of focus and across the institution and networks.

Everyone is responsible for learning and there are some great resources to support that effort, but without someone taking explicit leadership on making sure learning happens within the institution, it’s less likely to happen — at least happen in a way that is designed for innovation.

Introducing: the Chief Learning Officer.

Leading Learning

With the alphabet soup of C-s that we are seeing among organizations’ leadership teams adding another might not seem helpful. What we propose is less about formalizing the title of CLO and more about creating the function of what they can do within an organization.

We envision a CLO role as one that does the following:

Establishing a learning plan for the organization and the data structure to support that learning. This means instilling and building a culture of evaluation across the organization, which provides data and feedback on what is happening to allow staff at all levels to learn from what is being done. It involves showing what evaluation can do and co-creating ways to do it across the enterprise to add learning value.
Ensures that staff roles and functions include the ability to study and reflect on the work being done and its impact. This means establishing practices and procedures that link evaluation data to program activities. It also involves creating the means to bring in insights from outside sources (e.g., published research and reports, networks, professional communities, customers and clients). Structuring what we do and how and when we do it is part of this function to ensure that roles and learning needs are fit-for-purpose.
Organizing the evaluation of program activities and ensuring that staff at all relevant levels of the organization close to each program have access to the information about those programs and can make decisions about those programs without having to go through cumbersome layers of bureaucracy.
Create sensemaking channels and opportunities throughout the organization. This allows intra- and cross-departmental/unit collaboration to understand the bigger picture of what’s going on in the organization and industry.
Supports the development of self-sustaining communities of practice or learning groups on topics relevant to the organization, yet without specific roles or functions. Topics might include emerging technologies, leadership, creative thinking, or professional development.

Creating your CLO Office

A CLO would link the activities of the organization to the monitoring and evaluation data about those activities with the literature and trend data from outside the organization into a culture of learning within the organization.

This could be a full- or part-time position or something like a fractional CLO role like we can play at Cense.

Whether you create a CLO within your organization or choose to recruit learning support from outside, having a dedicated person shepherding your culture into a learning organization is something that will increase your innovation capacity exponentially.

For more information about how you can build this learning culture within your organization or the fractional CLO role, contact us. We’d love to help you out.

Sep 09 2019

drowning in the data

“Can we even get that data?”

That question, or some version of it, is usually one of the first, if not the first, question I hear when planning or discussing a new evaluation project. People want to know if it’s possible to collect data on a particular outcome or from a particular group. There’s often an undertone of, “I bet we can’t,” in the question too.

From the outside, I think evaluation must look a lot like survey + interviews = report. Those are the parts of the process that most people actually get to interact with after all. (Which is too bad, because they are also the least impactful parts to experience in terms of process use.) So I can understand why questions about data collection (and reporting deliverables) often dominate early conversations I have.

But data collection is not the hardest part of evaluation. My answer to, “Can we even get that data?”, is “Yep, probably.” In some way, in some form, we can get data on that or from that group of people. But there are so many more essential questions to be asked first. Do we need that data? What are we going to use it for? Do we have a plan for how to use it? Do we have the people, resources, and processes in place to make sure it’s going to get used?

Because here is the thing. If you have gasoline, you generally also want—at minimum—a gas-powered vehicle, a capable driver, and a destination in mind or at least a general direction to start in, otherwise you’ve got a lot of something you can’t use very effectively. Same goes for data. And while you want to factor stopping for gas into your road trip plans, it’s probably not where you start the planning process.

A flaw in that analogy though is that it doesn’t reflect just how much of a problem it can when we focus on data collection to the detriment of data use. When we collect data we don’t need and can’t use, or *do* need but fail to use well, the consequences can range from wasteful to, in the most extreme circumstances, deadly.

Julia Coffman wrote up one of the most compelling examples of this in her article, Between the Devil and the Deep Blue Sea: The Consequences of Small Failures in Learning. In it, she explores a fatal shipwreck caused by what should have been an avoidable collision with a hurricane, had it not been for the persistent misuse of data (in this case meteorological data about the course of the incoming storm). There was enough of the necessary data to avoid the collision as well as people highly qualified to interpret it on the ship, but fatal decisions were still made. Julia explains the cognitive and interpersonal biases as well as situational factors that may have contributed to the misinterpretation and misapplication of data, though we can’t know for certain as the main decision-maker—the captain—died along with his crew.

Just having data doesn’t mean we will use it well, and having too much data or the wrong kind of data can be misleading as well as a waste of resources. As this article from the Stanford Social Innovation Review points out, with technological advances it’s getting easier and easier (and cheaper) to collect, store, and analyze data. But a tool like a dashboard only produces higher-level data of your raw data, it can’t tell you what it all means or how to use it effectively. That requires human-level interpretation and application, which isn’t getting any faster or cheaper or easier and isn’t helped by a flood of irrelevant information without a meaningful practice for making sense or use of it.

The latest example I’ve found of how easily and chronically we end up in these bad habits with data is a is New Yorker review of various books on the history of spies and intelligence agencies, which sums up the main takeaway as, “The history of espionage is a lesson in paradox: the better your intelligence, the dumber your conduct; the more you know, the less you anticipate.” Yep, that’s right—there’s a long and storied tradition of intelligence-gathering being self-defeating and counter-productive.

Espionage and evaluation differ many ways , but fundamentally we’re still talking about data-gathering to inform decision-making and we’re in the territory of human fallibility, so many of the problems surfaced in the article sounded awfully familiar. For example, “Not for the first or the last time, the point of spying—to know what the other side is likely to do—had been swallowed up by the activity of spying,” reminds me of this statement from another article on over-measurement practices, “Micro-measuring what we have done seems to be more important than what we actually do.” Process subverting purpose!

The problem of volume and data overload comes through in this comment, “if you have any secret information at all, you often have too much to know what matters”. And the issue of focusing on the wrong kind of data, “The two agencies were so busy spying on each other, it almost seems, that they forgot to spy on each other’s government. Knowing what the K.G.B. was doing wasn’t the same thing as knowing where the Soviet state was heading, and the rise of Mikhail Gorbachev and the fall of the Soviet Union came as a complete surprise to the C.I.A.” There’s also a repeated pattern of accurate, useful intelligence being suppressed or ignored because of “… confusion, political rivalry, mutual bureaucratic suspicions, intergovernmental competition, and fear of the press (as well as leaks to the press), all seasoned with dashes of sexual jealousy and adulterous intrigue.” (Okay, those last two probably come up less often in evaluation, but the rest are commonplace!)

The role of trust and transparency in the misuse of intelligence fascinated me too. “The universal law of unintended consequences rules with a special ferocity in espionage and covert action, because pervasive secrecy rules out the small, mid-course corrections that are possible in normal social pursuits. When you have to prevent people from finding out what you’re doing and telling you if you’re doing it well, you don’t find out that you didn’t do it well until you realize just how badly you did it.”

Maybe this is less relevant to the use of evaluation findings since there technically we’re not required to operate under a shroud of secrecy the same way spies are. But don’t we end up doing so much of the time anyway? Evaluation is political, and a lot of it happens quietly, in-house, with little-to-no publicly available documentation, and minimal participation by most people affected by the process in anything other than providing data and maybe having access to the final report. While participatory approaches to evaluation seem to be gaining in usage, I’m not sure they’re anywhere near being the norm. There can be a real culture of fear around data and how it might be used to reflect badly on organization or program that prevents, or at least seriously limits, actual useful discussion and application of it.

I won’t minimize the political nature of data and how it can be weaponized, but I’ll offer that blanket secrecy is not necessarily the most effective way to manage that risk, especially with the cost to utility. This is where the value of having a clear evaluation strategy—a direction and purpose behind why the evaluation is being conducted and how it will be translated into use—can benefit again. It ensures that data are collected with purpose and intention and can be interpreted and applied in light of that purpose, with a clear explanation of why that data, that interpretation, and that use. The more robust the evaluative reasoning, the harder anyone else is going to have to work to offer a counter. And, bonus, you end up with a better evaluation either way.

Sep 08 2019

Evaluator Competencies Series: Program Evaluability

2.2 Assesses program evaluability.

I can’t remember exactly when or where this was, but at some point during my career as an evaluator, I saw the phrase “Evaluability Assessment” and thought “what’s that?” I know I wasn’t a brand new evaluator, as when I looked it up and learned what it was I thought, “Oh! I’ve been doing that at the start of every evaluation that I’ve done. I didn’t know there was a name for it!”

An evaluability assessment is, much like the name suggests, assessing “the extent to which an activity or project can be evaluated in a reliable and credible fashion” (OECD-DAC 2010; p.21 cited on Better Evaluation). As I learned to be an evaluator, this seemed to be a thing that I naturally needed to do.

An example would be where a client says “I want an evaluation that tells me if the program is achieving its goals”. The first question one would ask is: “What are the program goals?” because I certainty can’t tell you to what extent you are achieving your goals if you don’t have any goals. Similarly, a program may ask an evaluator to conduct an evaluation on whether the program has improved some particular thing for their program participants (e.g., their health, their knowledge of a topic, their social connectedness – whatever thing the program is trying to help its participants improve). In that case, one would naturally ask “how were the clients doing on that something before they started the program?” (i.e., do you have any baseline data we can use to compare to?). Or perhaps it’s a case where the client says “I want to know if my program is working?”, in which case I would ask “What does the program “working” mean to you?”. And that might lead to some work around developing a program theory, or figuring out if they want to know what outcomes are achieved, or if they want to know if their processes are efficient, or whether they are concerned about negative unintended consequences (or maybe all of the above). In my experience as an evaluator, when I ran into situations like this, my first course of action would be to work with the clients to figure out how to get their program into a state in which they are evaluable.

What I didn’t realize in my early years as an evalulator is that in some cases, an evaluability assessment could be a project unto itself. (Check out the Better Evaluation page on Evaluability Assessment if you want to read more about it.)

This may be due to the fact that I’ve always been an internal evaluator (or, as I think of myself on the program I’m currently working on, an external evaluator who is embedded in the program for the long term). So I’ve always had the luxury to be able to work with my “clients” to get them into an evaluable state as part of the work I do with them. Perhaps if I were an external evaluator, I may have come across stand alone evaluability assessments as potential projects.

I couldn’t find any images online that I felt represented “evaluability assessment” (probably not surprising… spell check doesn’t even believe that “evaluability” is a word!). So instead I give you this picture of my cats:

Watson & Crick in a Costco box — Watson (the tabby) and Crick (the grey and white cat).

Sep 03 2019

Evaluator Competencies Series: Clarifying Purpose and Scope

The next domain of competence is technical practice.

2. Technical Practice competencies focus on the strategic, methodological, and interpretive decisions required to conduct an evaluation.

And the first competency in this domain is:

2.1 Clarifies the purpose and scope of the evaluation.

Let’s begin at the beginning. That may seem trite, but I think it’s such a common saying because so often, people want to start somewhere other than the beginning. I cannot tell you how many times I’ve been consulted about an evaluation and the person seeking my advice starts with something like:

I have a set of indicators and I need to do an evaluation using them.
I want to do an evaluation of my program but I can’t figure out how to make it into a randomized controlled trial (RCT) because the program is already run.
I need your help to create a survey to evaluate my program.
I need to do a developmental evaluation [or whatever the latest trend in evaluation is at the time] of my program.

These are all examples of not beginning at the beginning. Many people seem to think that an evaluation requires a specific method (e.g., a survey) or a specific design (e.g., an RCT). Or they think that whatever the latest trend in evaluation must be the best approach, because it’s new. Or they have data already and they want to use it ¹I just noticed that I‘ve written about this before, more than 4 years ago! Past Beth would be sad to hear that I’m still experiencing this!. But where an evaluation needs to start is with its purpose. Why, exactly, do you want an evaluation? What will you use the findings of the evaluation for? These are the types of questions that I will ask (usually preceded by me saying “Let’s back up a second!”). Because the purpose of the evaluation will guide the choice of approach, design, and methods. For example, if you are interested in an evaluation that will help you to determine to what extent you’ve achieved your goals, and none of your current indicators relate to your goals, then starting with “I have a set of indicators and I need to do an evaluation using them” is not going to get you where you want to be. Similarly, if you want an evaluation that will help surface unanticipated consequences (and I tend to think that evaluations should usually be on the look out for them), then having a set of pre-defined indicators is not going to be what you need (after all, to create an indicator, you have to have been anticipating that it might be affected by the program!). If the purpose of your evaluation is not a developmental one, then developmental evaluation might not be the best approach for you. So clarifying the purpose (or purposes) of an evaluation is something that I do at the start of every evaluation – and something that I check in on during the evaluation, both to see if what we are doing in the evaluation is helping to meet its purpose and to see if the purpose changes (or new purposes emerge) along the way.

Clarifying the scope of an evaluation is also really important, and something that I struggle with. I’m am an infinitely curious person and I want to know all the things! But there just isn’t enough time and resources to look at every possible thing in any given evaluation, so it’s important to be able to clarify what the scope of any given project is. Like purpose, it’s important to clarify the scope of the evaluation with your client at the start, and to keep tabs on it throughout the evaluation. If you don’t have a clear scope, it’s very easy to fall into the trap of the dreaded “scope creep” – where extra things get added to the project that weren’t initially agreed to and then either the costs go up – or the timeline gets extended. It’s not to say that the scope can’t change during an evaluation, but just that any changes to scope should be done mindfully and in agreement between the client and the evaluator.

Working in a large organization like I do, I also find it useful to understand the scope of other departments that do similar work to evaluation (like quality improvement and performance management). This is helpful in ensuring that we aren’t duplicating efforts of other teams, and also that we aren’t stepping on anyone else’s toes. Also, I’ve had the experience of taking on work that really should have been done by another team (i.e., the dreaded scope creep!) and had we not figured this out by clarifying scope, it would have really impaired our ability to deliver on the work that we needed to deliver on.

My team and I have done some work on clarifying what the scope of evaluation is relative to these other groups and I was about to say “and that’s a topic for another blog posting”, but then I remembered that I’m presenting a webinar (based on a conference presentation I gave last year) on that in a couple of weeks! So here’s my shameless plug: if you want to hear me pontificate on the similarities, differences, and overlaps between evaluation and other related approaches to assessing programs and services, register for my webinar, hosted by the Canadian Evaluation Society’s BC Chapter on Friday, September 13 (that’s right, Friday the 13th!) at 12 pm Pacific Time.

Footnotes [ + ]

↑

I just noticed that I‘ve written about this before, more than 4 years ago! Past Beth would be sad to hear that I’m still experiencing this!