Another stellar reflection, Dr. Beth. Thanks for the introduction to Dylomo, too. I have had fun trying it out.
allblogs
Innovation: Why Starting Points Matter

If you’re looking to measure some form of progress or impact connected to your innovation (a product, service, or policy) then paying attention to the starting point is critical.
Evaluators call this a baseline and it’s maybe the most important line you can draw. A baseline is really the point of comparison for all you do. When speaking about improvements or change, this is the point you refer to when making those claims.
For something so important, it’s remarkable how few organizations capture baselines well. Let’s look at what it means and how you can do a better job of determining your innovation’s baseline.
Setting a baseline
An ideal baseline is set as far back from the present as possible at the start of your innovation journey. However, as many journeys have starts, stops, and tangents it might be that the start of the innovation journey actually ‘begins’ mid-way through a timeline.
If you are already started your innovation journey, the best time to set a baseline is now. It’s possible in some cases to use retrospective data (looking backward) to assess a baseline, however that can be fraught with certain biases that are unhelpful. If looking retrospectively, consider neutral data points like dates and times, concrete descriptions of product work, and use verifiable sources of data (e.g., work activities, prototypes, expenditures) to support that work.
When setting a baseline, there are some other tips we advise to enable you to capture the most possible useful data you can. If you are innovating in a human system, it’s possible that the innovation may have many effects that go beyond the most obvious so collecting the right data to capture these effects at the beginning is key.
- List out the resources that have been assembled to develop the innovation such as people, space, and other capital (e.g. funds). These are your starting inputs into the project.
- Gather a project plan or schedule of activities early to help determine what happens after the project begins. This will help determine where deviations from the plan take place, when, and help you trace back what happens if or when those changes take place to the strategy. Capturing deviations is critical because it helps you go back to see what adaptations you make at the end. Without this data, these activities might appear to be random or haphazard.
- Capture cultural/environmental factors. Using the STEEP-V (Social, Technological, Economic, Environmental, Political, and Values) model is helpful in knowing what to pay attention to. One of our clients experienced a major, unexpected removal of funding due to rapidly changing political priorities of a government that was supporting their work. By capturing these broader situational variables you can place your innovation work in a context.
- Document the state of your organization’s readiness and preparedness, which may also include an assessment of innovation readiness. Many innovations fail not out in the market, but within the design studio. Changes to organizational priorities, resources, and personnel can scupper, delay, or change the plans for an innovation. Capturing the state of the organization is an important point as it will allow you to see where things go off track or where they are enabled because of the organization.
- Develop a project charter and theory of change. While a project may change direction many times, a baseline assessment can help you reflect the desired outcomes and original purpose of the innovation — which are quite likely to change over time. Having this in place can help explain what changes take place and what adaptations take place.
Baselines are the key point for making any claims of change, improvement, or transformation. They are the point where we say “in relation to what?” when speaking about change.
Give yourself some time and use the baseline assessment as a chance to spur reflective and strategic planning about your innovation. You will be grateful you did and amazed at the results later on.
If you’re interested in learning more about baseline development and its role in supporting innovation evaluation, contact us and we’ll gladly help.
Photo by Suad Kamardeen on Unsplash
Evaluator Competencies Series: Program Theory
2.3 Clarifies the program theory.
I really like helping programs figure out what their theory of change is. Early in my career as an evaluator, I was surprised how often I would work with a program that had no idea what its theory was. Like, you’d sit down with them and ask questions about what they were trying to achieve and how what they thought what they were doing was going to help them achieve it – and they didn’t know. They had never really thought about it. The program was the way it was by some combination of it having been started by someone in some way at some time for some reason and then it had been adapted over the years in response to funding cuts/new funding opportunities/new leadership/new research/[enter all sort of other possible factors here]. While talking about this with my class this weekend (I’m teaching a Program Planning & Evaluation course in a Masters of Health Administration program), one student described the programs that she’s worked on as having been MacGyvered and I absolutely love that description!
Perhaps way back when a program started there had been an idea of a program theory – or possibly not – but it’s been MacGyvered over the years and often there us no record of any original program theory. And so I discovered that an important part of work as an evaluator is often to help the program make explicit the theory of why they think the program will result in changes to achieve whatever it is trying to achieve. Because even if a program doesn’t have an explicit program theory, there is some implicit theory underneath.
And there are many benefits about making your program ‘s theory of change explicit. As an evaluator, I want to know what the program’s theory is so I can design an evaluation to test the theory. But it can also be quite helpful to the program itself – helping them to get everyone on the same page about what the program is actually trying to achieve and getting them to think about whether what their program does is likely to get them there. Also, sometimes mapping out a program theory helps a program to identify that it is doing activities that are not likely to help them achieve their goals. It’s surprising how often programs do things because “we’ve always done these things”, even though they may no longer be needed or relevant. Working through a program theory can help identify those things.
Oftentimes, I work with those involved in the program to clarify the theory by developing a logic model together. There is a debate about whether a logic model is or is not a program’s theory of change. According to Michael Quinn Patton (2012), a logic model is simply a description of a logical sequence, but “specifying the causal mechanisms transforms a logic model into a theory of change”, i.e., you need to “explicitly add the change mechanism” to make it a theory of change. I like this explanation because it reminds us that a logic model on its own isn’t quite enough to be a “theory of change” so we need to think about what is the actual mechanism that is believed to lead to the change.
Thinking about how I do the work of clarifying program theory, I think my tips would be:
- however you choose to clarify a program’s theory of change, do it collaboratively with as many people who have an interest in the program as possible. This is important because:
- different people bring different perspectives and thus can help us to more fully understand how the program operates and the effects it can have
- a lot of the value of clarifying a program theory comes from the process. Finding out that people aren’t on the same page as one another about what the program is doing and why, identifying gaps in your program’s logic, surfacing assumptions that people involved in the program have – all of this can lead to rich conversations and shared understanding of the program among those involved and you just don’t get that by handing someone a description of a program theory that was created by just one or two people.
- a program theory should be thought of as a living thing. You can’t just map out a program theory once and think “well, that’s done!” Programs change, contexts change, people change… and our theories of change need to change to keep up with all of that!
This topic is also a good time to plug the free online logic modelling software that my sister, her partner, and I created: Dylomo (short for DYnamic LOgic MOdels). You can sign up for free and play around with it. Apologies in advance for any bugs – we created it off the side of our desks, so haven’t had time to add all the features we would like. If you do have any issues with it – or feedback about it – do get in touch!

References
Patton, M. Q., (2012). Essentials of Utilization-Focused Evaluation. Thousand Oaks, CA: Sage.
Image Source
-
Photo of leaves was opsted on Flickr by Mehul Antani with a Creative Commons licence. Again, I couldn’t find a good free-to-use image for what I was searching for (program theory, theory of change, logic model), but while searching for “change” I found that image of leaves changing colour and thought it was beautiful.
-
Dylomo logo was designed by my amazing sister, Nancy Snow.
Chief Learning Officer

C-suite leadership roles focus on an organization’s most important functions. Time to introduce the role of the Chief Learning Officer.
Learning — easier said than done. Yet, learning is vital to the success of an organization that seeks to innovate to gain advantage or merely survive – which is most human service organizations these days.
Learning opportunities abound, yet these require energy and attention in order to take advantage of them. Organizationally, this requires leadership and resources to support people across the organization to learn within their areas of focus and across the institution and networks.
Everyone is responsible for learning and there are some great resources to support that effort, but without someone taking explicit leadership on making sure learning happens within the institution, it’s less likely to happen — at least happen in a way that is designed for innovation.
Introducing: the Chief Learning Officer.
Leading Learning
With the alphabet soup of C-s that we are seeing among organizations’ leadership teams adding another might not seem helpful. What we propose is less about formalizing the title of CLO and more about creating the function of what they can do within an organization.
We envision a CLO role as one that does the following:
- Establishing a learning plan for the organization and the data structure to support that learning. This means instilling and building a culture of evaluation across the organization, which provides data and feedback on what is happening to allow staff at all levels to learn from what is being done. It involves showing what evaluation can do and co-creating ways to do it across the enterprise to add learning value.
- Ensures that staff roles and functions include the ability to study and reflect on the work being done and its impact. This means establishing practices and procedures that link evaluation data to program activities. It also involves creating the means to bring in insights from outside sources (e.g., published research and reports, networks, professional communities, customers and clients). Structuring what we do and how and when we do it is part of this function to ensure that roles and learning needs are fit-for-purpose.
- Organizing the evaluation of program activities and ensuring that staff at all relevant levels of the organization close to each program have access to the information about those programs and can make decisions about those programs without having to go through cumbersome layers of bureaucracy.
- Create sensemaking channels and opportunities throughout the organization. This allows intra- and cross-departmental/unit collaboration to understand the bigger picture of what’s going on in the organization and industry.
- Supports the development of self-sustaining communities of practice or learning groups on topics relevant to the organization, yet without specific roles or functions. Topics might include emerging technologies, leadership, creative thinking, or professional development.
Creating your CLO Office
A CLO would link the activities of the organization to the monitoring and evaluation data about those activities with the literature and trend data from outside the organization into a culture of learning within the organization.
This could be a full- or part-time position or something like a fractional CLO role like we can play at Cense.
Whether you create a CLO within your organization or choose to recruit learning support from outside, having a dedicated person shepherding your culture into a learning organization is something that will increase your innovation capacity exponentially.
For more information about how you can build this learning culture within your organization or the fractional CLO role, contact us. We’d love to help you out.
drowning in the data

Photo by Lianhao Qu on Unsplash
“Can we even get that data?”
That question, or some version of it, is usually one of the first, if not the first, question I hear when planning or discussing a new evaluation project. People want to know if it’s possible to collect data on a particular outcome or from a particular group. There’s often an undertone of, “I bet we can’t,” in the question too.
From the outside, I think evaluation must look a lot like survey + interviews = report. Those are the parts of the process that most people actually get to interact with after all. (Which is too bad, because they are also the least impactful parts to experience in terms of process use.) So I can understand why questions about data collection (and reporting deliverables) often dominate early conversations I have.
But data collection is not the hardest part of evaluation. My answer to, “Can we even get that data?”, is “Yep, probably.” In some way, in some form, we can get data on that or from that group of people. But there are so many more essential questions to be asked first. Do we need that data? What are we going to use it for? Do we have a plan for how to use it? Do we have the people, resources, and processes in place to make sure it’s going to get used?
Because here is the thing. If you have gasoline, you generally also want—at minimum—a gas-powered vehicle, a capable driver, and a destination in mind or at least a general direction to start in, otherwise you’ve got a lot of something you can’t use very effectively. Same goes for data. And while you want to factor stopping for gas into your road trip plans, it’s probably not where you start the planning process.
A flaw in that analogy though is that it doesn’t reflect just how much of a problem it can when we focus on data collection to the detriment of data use. When we collect data we don’t need and can’t use, or *do* need but fail to use well, the consequences can range from wasteful to, in the most extreme circumstances, deadly.
Julia Coffman wrote up one of the most compelling examples of this in her article, Between the Devil and the Deep Blue Sea: The Consequences of Small Failures in Learning. In it, she explores a fatal shipwreck caused by what should have been an avoidable collision with a hurricane, had it not been for the persistent misuse of data (in this case meteorological data about the course of the incoming storm). There was enough of the necessary data to avoid the collision as well as people highly qualified to interpret it on the ship, but fatal decisions were still made. Julia explains the cognitive and interpersonal biases as well as situational factors that may have contributed to the misinterpretation and misapplication of data, though we can’t know for certain as the main decision-maker—the captain—died along with his crew.
Just having data doesn’t mean we will use it well, and having too much data or the wrong kind of data can be misleading as well as a waste of resources. As this article from the Stanford Social Innovation Review points out, with technological advances it’s getting easier and easier (and cheaper) to collect, store, and analyze data. But a tool like a dashboard only produces higher-level data of your raw data, it can’t tell you what it all means or how to use it effectively. That requires human-level interpretation and application, which isn’t getting any faster or cheaper or easier and isn’t helped by a flood of irrelevant information without a meaningful practice for making sense or use of it.
The latest example I’ve found of how easily and chronically we end up in these bad habits with data is a is New Yorker review of various books on the history of spies and intelligence agencies, which sums up the main takeaway as, “The history of espionage is a lesson in paradox: the better your intelligence, the dumber your conduct; the more you know, the less you anticipate.” Yep, that’s right—there’s a long and storied tradition of intelligence-gathering being self-defeating and counter-productive.
Espionage and evaluation differ many ways , but fundamentally we’re still talking about data-gathering to inform decision-making and we’re in the territory of human fallibility, so many of the problems surfaced in the article sounded awfully familiar. For example, “Not for the first or the last time, the point of spying—to know what the other side is likely to do—had been swallowed up by the activity of spying,” reminds me of this statement from another article on over-measurement practices, “Micro-measuring what we have done seems to be more important than what we actually do.” Process subverting purpose!
The problem of volume and data overload comes through in this comment, “if you have any secret information at all, you often have too much to know what matters”. And the issue of focusing on the wrong kind of data, “The two agencies were so busy spying on each other, it almost seems, that they forgot to spy on each other’s government. Knowing what the K.G.B. was doing wasn’t the same thing as knowing where the Soviet state was heading, and the rise of Mikhail Gorbachev and the fall of the Soviet Union came as a complete surprise to the C.I.A.” There’s also a repeated pattern of accurate, useful intelligence being suppressed or ignored because of “… confusion, political rivalry, mutual bureaucratic suspicions, intergovernmental competition, and fear of the press (as well as leaks to the press), all seasoned with dashes of sexual jealousy and adulterous intrigue.” (Okay, those last two probably come up less often in evaluation, but the rest are commonplace!)
The role of trust and transparency in the misuse of intelligence fascinated me too. “The universal law of unintended consequences rules with a special ferocity in espionage and covert action, because pervasive secrecy rules out the small, mid-course corrections that are possible in normal social pursuits. When you have to prevent people from finding out what you’re doing and telling you if you’re doing it well, you don’t find out that you didn’t do it well until you realize just how badly you did it.”
Maybe this is less relevant to the use of evaluation findings since there technically we’re not required to operate under a shroud of secrecy the same way spies are. But don’t we end up doing so much of the time anyway? Evaluation is political, and a lot of it happens quietly, in-house, with little-to-no publicly available documentation, and minimal participation by most people affected by the process in anything other than providing data and maybe having access to the final report. While participatory approaches to evaluation seem to be gaining in usage, I’m not sure they’re anywhere near being the norm. There can be a real culture of fear around data and how it might be used to reflect badly on organization or program that prevents, or at least seriously limits, actual useful discussion and application of it.
I won’t minimize the political nature of data and how it can be weaponized, but I’ll offer that blanket secrecy is not necessarily the most effective way to manage that risk, especially with the cost to utility. This is where the value of having a clear evaluation strategy—a direction and purpose behind why the evaluation is being conducted and how it will be translated into use—can benefit again. It ensures that data are collected with purpose and intention and can be interpreted and applied in light of that purpose, with a clear explanation of why that data, that interpretation, and that use. The more robust the evaluative reasoning, the harder anyone else is going to have to work to offer a counter. And, bonus, you end up with a better evaluation either way.
