cplysy

Pilots have a pre-flight check before they fly a plane. A developmental evaluation journey requires the same kind of preparedness.

Developmental Evaluation (DE) is an approach to strategic learning designed for understanding and supporting innovation. This approach to data-driven learning requires a specific set-up to do well. While we need a mindset, skillset, and toolset to do DE, we also need a receptive system to make it all work.

Here are the things you can do to lay the groundwork for doing DE and making it a success. Think of this as your pre-flight checklist for a journey of learning together.

These are attributes and may be found within individuals, but it’s better if they are distributed widely across the organization.

Mindset

The most important aspect of a Developmental Evaluation is creating the right mindset. This mindset is focused on complexity, adaptation, and development (vs. improvement, linear growth). A list of statements to examine whether you have the right mindset is available on Censemaking and can provide a start.

Some other mindset questions include:

1. Openness to experience. A DE will likely take a journey that can’t be fully predicted, much like a road trip. An openness to a non-linear pathway toward a preferred destination and the willingness to adapt the path and the destination based on evaluation data and evidence is key.
2. Tolerance for ambiguity. Few of us enjoy uncertainty, but we all need to deal with it when working on a DE.
3. Self-awareness. We often get in our own way. Understanding how we think and the biases and perspectives we hold is important to knowing when they serve us and when they are a barrier to innovation. Mindfulness is a part of this quality.
4. Patience. This is a skill and mindset quality. Knowing that things will unfold at a time and pace that might change is useful — and requires patience in just knowing that.
5. Evaluative thinking. This is a form of thinking that gets us connecting our activities, outcomes, and effects and asks three key questions about how we know something (and what to do about it).

Skillset

Developmental evaluation is not for the unskilled. While many of the qualities inherent in a DE are natural to people, they are not commonly practiced in organizations. DE is an approach, and while a skilled evaluator might have many methodological skills to do much of what is in a DE, without the following skills — with the evaluator, the evaluation team, the organization, or all of them — you won’t likely succeed.

• Facilitation. A DE is a collaborative endeavour. You need facilitation skills to engage the diversity of perspectives in the system and coordinate the discussion, feedback, and opportunities for everyone to engage with these perspectives.
• Sensemaking. Sensemaking is a participatory process that involves taking data that may be difficult to understand, ambiguous in its conclusions, or incomplete and helps people make sense of what it means for the organization within the present context and strategic needs. Sensemaking for DE helps guide decision-making and strategy.
• Complexity and Systems Thinking. A distinguishing feature of DE is the use of applied thinking from systems science and complexity to guide the entire process. This means creating designs and processes that are sensitive to detecting emergent properties, network effects, organizational behaviour (like self-organization), and non-linear or algorithmic change dynamics.
• Dynamism. Complexity also requires that an evaluation be designed with consideration of the dynamics of the program, actors, and system. This means adapting or developing from a moving position, not a static position. It also involves designing evaluations that take into account the growth pattern and evolution of a program that suits the system. It’s about a design for changing conditions.
• Visualization. System complexity is made more so by the inability to effectively ‘see’ the systems in our heads – there’s too much information. The ability to create system maps, visualize the dynamics and relationships within it, and facilitate discussion of those systems is critical to understanding where a program fits within it all. Visualization can be sophisticated or relatively simple in nature.
• Design. What makes DE distinct is that it is about making modifications to the program as it unfolds using evaluation data to guide the strategy. These modifications require design and the skills — like with evaluation — require designing a program while it is moving. These developmental design skills are often overlooked and bring together service design and innovation with elements of systemic design.

Toolset

The toolset is the least specific of the three ‘sets’ for DE. Almost any method can support a DE, although unlike other approaches, rarely will there be a case where a single method — interview, survey, observation — will work on its own. A multi-method or mixed-method approach to data collection is almost always necessary.

Some additional tools or methods that are not common are ones we’ve advocated for and either developed ourselves or borrowed from other contexts. Three of the most important of these are:

• The Living History Method. This meta-method brings together artifacts that inform the program’s early development and evolution. It recognizes that the present program is shaped by where it came from and that such a history can influence where things are going. Unless these conscious or unconscious patterns are recognized it is unlikely they will change. This method also helps record decisions and activities that often get missed when creating an innovation.
• Dashboards. The wealth of potential information generated from a DE and available at our fingertips can overwhelm even the most skilled evaluator. Creating dashboards using simple tools and technologies can help organize the various streams of data. It also can allow someone to gain a sense of where there may be simple patterns laying within a complex layer of data.
• Spidergrams. The spider diagram is a simple visualization tool that can work well in a DE as it helps pull various data points together in a way that allows comparison between them AND the ability to dive deep within each data stream.

Some additional perspectives on doing DE can be found in the guide (PDF) published by the McConnell Foundation by Elizabeth Dozois and colleagues.

Next time you are considering a DE, look at your present inventory of mindsets, skillsets, and toolsets to see if you have the right resources in place to do the work you need to do.

Need help? Contact us. We can help you set up a Developmental Evaluation, do one, or both. We’d love to hear from you.

Photo by Lucia Otero on Unsplash

Sketch diagrams are powerful tools for visualizing complex programs and systems. This simple technique literally gets people on the same page.

Mapping out a service or product ecosystem can be a complicated endeavour. There is the entire field of systemic design that focuses on tools and strategies to engage users for starters. There are approaches like synthesis mapping and service design canvases that can help us walk through the various aspects of a system to find points of leverage, threat, and opportunity.

Nearly all of these methods and tools require user orientation and training — sometimes a little, sometimes a lot. However, in many of our engagements, particularly in healthcare, we find the time (and attention) is so limited it becomes difficult to engage participants using methods that require considerable instruction.

It’s here that we introduce a technique and tool called the system sketch map.

Sketch mapping

A sketch map is a multimedia system map that is created by using any form of representation such as blocks and arrows, stick figures, or elaborate illustration and can be developed at any moment. A sketch map gets participants past the challenge of having to learn a technique or representation convention and can be particularly useful for those participants who feel unskilled at drawing or visualizing their thoughts.

This works well with professionals who may find themselves uneasy about using visual media or do not consider themselves ‘creative’. (Note: Everyone can draw. It’s important to emphasize that this is not an art project).

The exercise works like this:

Begin with the instructions: Draw your system.

That’s it.

Some guidelines: 

• Any visual formalism can be used. One can even combine visual approaches together.
• Emphasize the simplest media possible: Pens and paper (or crayons), sticky notes (or stickers) are among the best tools because they are flexible, colourful, and can be combined easily. These are also inexpensive and easy to obtain.
• Large format paper (e.g., newsprint) or whiteboards are best to use as a canvas to facilitate group participation
• Group participation is key
• There are no right or wrong ways to do this. Whatever participants wish to include in that system is all that matters.
• Give participants a time frame (usually 30 – 50 minutes works best) and try and ensure there are between 4 and 6 people in the group.
• Emphasize DOing over THINKing. It’s easy for groups to try and do this ‘right’ and analyze everything. The use of simple, inexpensive materials allows people to create ‘do-overs’ easily, erase material, revise and recreate things.
• Lastly, strive for ‘good enough’ and ‘coherent’ over ‘excellent’ and ‘complete’ (which are highly relative in this context).

What is interesting is that the participants define what their system is and what goes in it. In making these choices it becomes evident what they see as most essential, important, or relevant.:

A completed sketch map then allows everyone (the facilitator and participants) afterward to ‘interrogate’ the map (not the map makers) and ask questions like “does this choice of colour mean anything?“, “is the distance between these two things represent some kind of scale?“, “what might be missing from all of this?”

This interactive discussion process allows everyone to explore what gets placed at different positions, sizes, in different colours, and what gets included and left out of the map. It allows for the use of symbolism (conscious or not), metaphor, and representation without having to shape or bias the participants toward using a particular way of visualizing the system.

What it is, is what it is.

This simple technique can yield enormous insights into the assumptions, structures, relationships, actors, and core components associated with a system and do so within one or two hours and with a small budget.

For more information about sketch mapping and how it can help you with your work or just for more on innovation methods, tools, and strategies, feel free to contact us. We’d love to hear from you and can help.

Photo by Danae Paparis on Unsplash and Kaleidico on Unsplash

If you’re looking to measure some form of progress or impact connected to your innovation (a product, service, or policy) then paying attention to the starting point is critical.

Evaluators call this a baseline and it’s maybe the most important line you can draw. A baseline is really the point of comparison for all you do. When speaking about improvements or change, this is the point you refer to when making those claims.

For something so important, it’s remarkable how few organizations capture baselines well. Let’s look at what it means and how you can do a better job of determining your innovation’s baseline.

Setting a baseline

An ideal baseline is set as far back from the present as possible at the start of your innovation journey. However, as many journeys have starts, stops, and tangents it might be that the start of the innovation journey actually ‘begins’ mid-way through a timeline.

If you are already started your innovation journey, the best time to set a baseline is now. It’s possible in some cases to use retrospective data (looking backward) to assess a baseline, however that can be fraught with certain biases that are unhelpful. If looking retrospectively, consider neutral data points like dates and times, concrete descriptions of product work, and use verifiable sources of data (e.g., work activities, prototypes, expenditures) to support that work.

When setting a baseline, there are some other tips we advise to enable you to capture the most possible useful data you can. If you are innovating in a human system, it’s possible that the innovation may have many effects that go beyond the most obvious so collecting the right data to capture these effects at the beginning is key.

• List out the resources that have been assembled to develop the innovation such as people, space, and other capital (e.g. funds). These are your starting inputs into the project.
• Gather a project plan or schedule of activities early to help determine what happens after the project begins. This will help determine where deviations from the plan take place, when, and help you trace back what happens if or when those changes take place to the strategy. Capturing deviations is critical because it helps you go back to see what adaptations you make at the end. Without this data, these activities might appear to be random or haphazard.
• Capture cultural/environmental factors. Using the STEEP-V (Social, Technological, Economic, Environmental, Political, and Values) model is helpful in knowing what to pay attention to. One of our clients experienced a major, unexpected removal of funding due to rapidly changing political priorities of a government that was supporting their work. By capturing these broader situational variables you can place your innovation work in a context.
• Document the state of your organization’s readiness and preparedness, which may also include an assessment of innovation readiness. Many innovations fail not out in the market, but within the design studio. Changes to organizational priorities, resources, and personnel can scupper, delay, or change the plans for an innovation. Capturing the state of the organization is an important point as it will allow you to see where things go off track or where they are enabled because of the organization.
• Develop a project charter and theory of change. While a project may change direction many times, a baseline assessment can help you reflect the desired outcomes and original purpose of the innovation — which are quite likely to change over time. Having this in place can help explain what changes take place and what adaptations take place.

Baselines are the key point for making any claims of change, improvement, or transformation. They are the point where we say “in relation to what?” when speaking about change.

Give yourself some time and use the baseline assessment as a chance to spur reflective and strategic planning about your innovation. You will be grateful you did and amazed at the results later on.

If you’re interested in learning more about baseline development and its role in supporting innovation evaluation, contact us and we’ll gladly help.

Photo by Suad Kamardeen on Unsplash