carolyncamman

Do yourself a favour and check out this post from Mariah Brothe Gantz, “The Realm of Possibility in Evaluation”. (And once you’ve clicked through and read it in its entirety, also click through and read all the posts that Mariah links to! They’re all brilliant.) I love how Mariah embraces her “youth” as an evaluator as a way to call on all of us to re-examine and renew our spirits and our practices. There’s a surge of energy in the field around questions like these, spurred by the young and young-at-heart among us. It’s a beautiful and critical time to be an evaluator as we hash out what that can and should mean.

In the article, Mariah pushes back on the idea that evaluation is “simply an accountability mechanism, something to keep you in line and something that might show you failed even without meaning to” when what it can be is “a mechanism that can help to advance social justice and equity”. I’ve been in a lot of conversations with people about the capacity for evaluation to foster and support transformative change through learning that is purposeful, collaborative, and rooted in justice. Last year on the Eval Cafe podcast, guest Chris Corrigan offered up a desire for us to “recover learning from the trauma of being judged”, and it’s a statement that’s stuck with me as I explore what it looks like for evaluation to be a healing practice instead of a harming one.

And.

And.

Let’s not give up on accountability.

Let’s take it back.

Bear with me. Accountability does not feel like a fun and delightful topic. It does not sound deep and meaningful, it sounds kind of boring, maybe even mean. It sounds like report cards and audits and scrutiny and being called out and dressed down and judged and found wanting. It sounds like punishment. It sounds like all the things that most evaluators I know try to distance ourselves from, because we don’t WANT to be the scary ogres in the room who freak everyone out or bring the mood crashing down with wagging fingers and shaking heads. It sounds like when you have to take all the beautiful, important, complex, sacred things that you do and squeeze them into a series of boxes in a fillable PDF form generated by a distant and inflexible bureaucracy in order to justify your existence and the worthiness of the work that you do to someone who won’t see or know anything about it or about you outside of that but still gets to decide if you and/or your staff get paid or not.

Yikes.

But we don’t have to concede to a definition of accountability as mindless, wasteful, status quo-justifying, weaponized compliance activities rooted in fear and mistrust and impersonal tick-boxes (even though that is absolutely how it often appears and operates, as evidenced by this recent article on the long history of dysfunctional government funding of non-profits in Canada). And learning and accountability don’t have to be separate and competing. It’s not a zero-sum game where one has to be sacrificed or diminished for the other to thrive. The more time I have spent thinking about accountability and learning, the more I have realized that in evaluation they are and must be the same thing, part of the same process of transformation and flourishing. We need to recover them both from the traumatic ways we’ve gone about them.

What if I said accountability—being accountable and being held accountable—is an act of love? Of trust? A profoundly relational and relationship-building act? Something that enriches rather than diminishes? Lifts up, witnesses, and heals? That’s how I see it, when I think of it in its ideal.

“How do you like to be held accountable?”

A question offered by Eroc Arroyo-Montano on an episode of the Healing Justice podcast. The episode is called Tender Masculinity, and I’m mindful of the way that masculinity is something else with a bad reputation right now, not undeservedly so within a context of patriarchy and toxic masculinity. But masculinity itself is not the problem, and I offer that accountability isn’t either, although we are certainly surrounded by ‘toxic accountability’.

“How do you like to be held accountable?”

The question is both powerful and tender. The word “held” is significant. How do you like to be held? is a question for intimates and lovers. And accountability is a form of holding. When liberated from punishment and blame, it can be steady and uplifting, rather than crushing.

Eroc goes on to say this:

I think that for me, the process of accountability has been a gift. And the ways in which I can be accountable to myself and the people I love in my community increase my capacity and ability to be whole, to be healthier, and to be able to walk in the world in the way that I wanna walk in the world. Which makes me someone who can be proud of the example that I’m giving, not just the rhetoric I’m spittin, you know? … I had to realize that there are some people who love me so much that their critique was not actually an attempt to tell me that I wasn’t worthy. It was actually them showing my worth. You’re worthy of hearing ways in which you can be a better person in this relationship, in this community, in this world.

He’s not talking about accountability the way it comes up in evaluation work, when we’re thinking about deliverables and reporting back to funders or out to communities. The “we did what we said we would do” version of accountability. He’s talking about the accountability of being in relationship, of addressing harms in acts and omissions, of the ways in which we fail each other and ourselves and have to deal with what that causes. If anything, it’s a much more terrifying type of accountability, where the stakes are high and very direct and personal. When we talk about accountability in the context of transformative justice, it means we are in the midst of injury and pain, recent, on-going, and historical, and need processes that allow us to come through it whole, as individuals and communities. The work being done on accountability in this spaces is next level, because it has had to be. It is the incredibly difficult work of creating and holding human spaces that refuse to let people be disposable, whether they have caused or been subjected to harms (or both), particularly those people most likely to have their humanity denied to make it more convenient to discard them.

It may seem worlds apart from what accountability represents in evaluation discussions, but to me it all connects. I don’t believe that accountability is about ticking off a check-list of things we said we’d do. It’s fundamentally about how we are responsible to and in integrity with each other and ourselves. A recent tweet from Kai Cheng Thom, a luminary writer in this space and author of one of the articles linked above, summed it up like this: “Integrity is our accountability to self. Honour is our accountability to others.”

In her article, Mariah, using phrasing I often find myself using and hear regularly from many in our field, notes the “disconnect in how most people think about what evaluation is for (just accountability) and what evaluation could be (a mechanism that can help to advance social justice and equity)”, and I ask myself—are these not the same thing? Is accountability not a crucial part of advancing social justice and equity? A recent report on the use of culturally-responsive evaluation in Wisconsin began with a quote that included this statement: “People who educate the next generation of academics and award contracts, grants, keynotes, or presidential sessions MUST be held accountable for structurally ensuring and requiring diversity in curricular content, human resources, funding priorities, contract/grant awards, keynotes, publications, etc., or things won’t change.” (Italics mine, capitals not.)

We recognize that accountability is an essential part of the fight for change. It’s also something I find to be in high demand, when I listen closely.

When I start an evaluation project, I try to find out why it matters to the people who will be involved in it (including but definitely not limited to the person who commissioned the evaluation). What’s important about it to them? Why evaluate? Answers vary, and there’s usually a mix of references to both learning and accountability. “It’s a funding requirement” comes up a lot, but not as the main goal. And “learning” blurs with “accountability”. When I ask what people want to learn, they want to know what is going well, but as much (sometimes more) they want to know what’s not working. Where the challenges and problems are. I am sometimes pressed for reassurance that negative feedback will be heard, that disappointing results won’t be hidden or glossed over, that the data will be credible and speak to what’s really going on no matter what. I notice that these questions often come most earnestly from those doing direct service delivery or are otherwise working closely with (or are part of) the communities and people they’re trying to support. It’s the kind of responsibility you feel when you have to look someone in the eye, be connected with them, and know that you want to be offering something that is real and meaningful and true to what you say it is.

Another way it appears is a strong focus on how learning will be shared throughout networks, with partners, and the community. It’s the sense of responsibility to one’s community, to building and strengthening the shared practice space and the relationships among those working for common goals. “Learning in plain sight” is a lovely turn of phrase often shared by Trilby Smith from her work at the Vancouver Foundation. Is that not also accountability? When we show up and share what we’re learning, as we’re learning it, with authentic vulnerability to receiving the judgement of others, is that not an effort to be accountable? “Here I am. Here is what I’m doing. What do you think? Is this useful to you?”

Judgement is scary. So is falling in love and being human and any number of important things we do. One of my favourite quotes, “If we want the rewards of being loved we must submit to the mortifying ordeal of being known”, for me resonates deeply with Eroc’s words above, “… critique was not actually an attempt to tell me that I wasn’t worthy. It was actually them showing my worth.” Judgement is part of evaluation. We need judgement, discernment in order to move ahead with what we learn. Part of a learning culture is a good accountability culture, one that is about being better humans together instead of just something that keeps us “in line”. The struggle is not in accountability itself, but in how the dominant accountability processes we have and are subjected to come from a place of fear and control that is both grounded in and reinforcing of systemic injustice and oppression, so that we end up with structures that require excessive “accountability” from those with the least power and influence and little, if any, from those with the most.

And where this leads me is that if we are to have meaningful accountability, in addition to transforming our own relationships with it, we must also challenge those with the most power, influence, and resources—including funders—to be accountable for that power. Because how can we ask for accountability for anyone if we aren’t asking for it from everyone? And in equivalent proportion to the power and authority they have? Reclaiming accountability is also a call to step into our role as advocates for change, for justice and equity. To de-weaponize accountability. To take it back. To see doing so as a profound act of love.

So the pivot in my practice is to lean into accountability, rather than away from it. To seek out understandings of accountability that are founded in trust, love, and respect. To learn about, be thoughtfully attentive to, and boldly interrogative of dynamics of power and influence—especially those sourcing from historically-embedded and on-going systemic oppression, including settler colonialism and white supremacy culture—within my evaluation practice and the contexts in which I practice to resist replicating and reinforcing unjust dynamics with control-oriented compliance activities.

To that end, here are some of the questions I’m learning to attend to when thinking about accountability within evaluation:

• Who is accountable to whom? For what and why and by what processes and with what consequences?

• Who is not required to be accountable? Who is able to determine their level of participation in accountability processes, and with what consequences?

• Who is making decisions about what constitutes accountability?

• What is the relationship between the answers to those questions and how power, influence, and resources are distributed and concentrated within the relevant context?

• What would constitute meaningful accountability (definitions and processes) to different individuals and groups within the relevant context?

• How would we like to be accountable to ourselves? To each other?

And, of course, how do I like to be held accountable?

Acknowledgements

I have been influenced by many people in my thinking and actions in this space, and have to give credit and gratitude where it is due.

• to Kim van der Woerd, of Reciprocal Consulting, and the teaching she offers on culturally responsive evaluation in the Transformative Evaluation Landscapes course

• to Jara Dean-Coffey, of the Luminare Group, and her work with the Equitable Evaluation Initiative

• to Vu Le, and his blog NonprofitAF, a catalogue of all the things we could and should be doing (and not doing) in the non-profit and funding worlds

• to Kris Archie, Executive Director of The Circle on Philanthropy, for so much (and one of her recent powerful twitter threads on structural racism, settler philanthropy, and what is needed is too good not to link)

• to Shawn Wilson, for his book, Research is Ceremony: Indigenous Research Methods, which was my introduction to relational accountability

• to Edgar Villanueva, for his book, Decolonizing Wealth, which offered me new frames of critique and ways of thinking about using power

• to Kai Cheng Thom, for her inspirational writing and speaking and being

• to adrienne maree brown, for her blog, books, and general brilliance

• … and to many countless others who have educated and shared with me in community. Thank you!

Also a particular gratitude to Mariah for her words and thoughts. Seriously, if you haven’t read her beautiful article on transformative possibility in evaluation yet, go do that!

Email Address
Sign Up

We respect your privacy.

Thank you!

Yesterday afternoon I spent a few delightful hours in a spoon carving workshop learning how to carve a cedar spoon with a bush knife and some hot coals. (Thank you to Delmar Williams of Squamish and Lil’wat Nations for teaching us and providing all the supplies and tools, and to Sharon Kallis of EartHand Gleaners Society for organizing and getting that all-important burn permit!)

I’m new to carving, and working with both knives and fire was exciting. As Delmar explained, when he teaches kids these techniques, it’s fun because he’s the guy who gets to come in and show them how to work with all the ‘dangerous’ things they’re supposed to avoid. The methods he showed us were not complicated on the face of them, but the actual execution took patience and the slow building up of skill through practice. We started with the coal burn-out method for the bowl of the spoon, which involves holding a piece of burning coal on your block of cedar (using another smaller piece of cedar to hold the coal in place), and feeding it air with your breath—or a helpful breeze—to keep it hot until you get a nice burned-in dish.

Easier said than done. Most of us struggled to keep our coals lit and burning long enough to make a dent. I kept getting a better light on my top stick than the one that was meant to be a spoon. When I did finally get a piece of coal going, I had to figure out how to keep a consistent breath on it and learn the rhythm of breathing and how to manoeuvre the wood and my head so that I didn’t accidentally suck in a big lungful of smoke or singe my lips on the occasional flame that caught. Eventually I got a good big piece of hot coal in there, found a place to stand where I could let the wind do some of the work for me, learned how to pace my breath and also how to shift the coal around so that I could get the size and depth of burn that I wanted.

When we were finally ready to begin carving, that was also a whole new skill set. I’ve never carved anything more substantial than a ham before this, but Delmar broke it down for us. As one of the other participants noted, it was more about the knife than the spoon. Delmar explained the anatomy of the knife and took us through the stages of spoon production and the different techniques needed for each. He told us that the material teaches you and, boy, did it. My biceps were throbbing after with all the learning imparted.

The application of knife blade to wood is something that would have unnerved me once upon a time. The piece of red cedar I was working felt important and special already (Delmar had said he never gives wood to anyone, he asks them to pick the piece that speaks to them). I’d put so much time into getting the bowl right. I wanted to be respectful of the material, of the teaching, and of my own investment of time, effort, and the tears I’d shed after many face-fulls of smoke. I expected to be awkward and unskilled with the blade, to make many poor cuts. I was hoping I wouldn’t do any significant “feeding the blade” (i.e., bleeding on it). There was nothing to do but start trying and hope I got better.

I did make many poor, awkward cuts, leaving divots as the blade skipped or caught or cracking off big pieces when I went in the wrong direction. I fed the blade just a little bit when I was working on rounding the the tip of the spoon and the knife slipped and I poked myself. As time in the afternoon got shorter, I tried not to rush, but I also wanted to make enough progress to learn all the techniques, knowing I’d need to do some finishing work on my own. I also tried to let the material teach me, both the wood and the knife. The wood was more dense in some places than others, harder to cut through, possibly because it had come from closer to the heart of the tree. Each new knife technique took time to wrap my head and my hand around, but repetition slowly eked out competence. The grain my knife strokes revealed was beautiful and I began to experiment with trying to highlight it, bringing the creativity of design along with the gradual mastery of technique. I made errors over and over and over again, sometimes setting back good work already done, but kept going.

About a year ago I was chatting with someone who had been learning to work with wood. He said found it powerfully healing because you do make mistakes and you can’t reverse them. The mistakes become part of it. You just keep going. I’ve also been taking improv classes and learning the same thing. Whatever happens, you work with it. Fix it by moving forward, not by trying to roll it back and erase it. Maybe you learn something from it, like what a better angle for the knife edge might be and how that should (or shouldn’t) feel in your hand. Maybe it’s a “happy failure”, in improv lingo, that creates an even funnier moment than the one you’d intended. You don’t barrel through oblivious to consequences (especially not when the consequence might result in stitches and not the “audience in stitches” kind), but hanging back isn’t an option because you only learn by trying, failing, learning, and trying again.

This goes for any space of uncertainty, which is most things we want to learn about. Even though carving might seem like a technical exercise with a specific tool and a set of techniques that can be learned and even (eventually) mastered, every piece of wood is different, holding different parts of the life of the tree it came from, still living and changing in moisture content as it seasons. The fire of the coal and the air that helps it burn, both breeze and breath, have their own life, and improvise with the wood to shape the bowl. Every knife cut works with the wood and the cut that came before, influencing the conditions for the cut that will come next. The hands that hold the wood and the knife, the arms that hold those hands, and the body that holds it all, are all changing, moving, getting tired as the time passes, getting stronger and more skillful too, learning from the teacher, learning from the other students, and learning from the material—knife, wood, and self.

The learning is in the doing, and the doing includes mistakes. If we aren’t making mistakes, we aren’t doing enough and we won’t be learning much either.

Shout-out to my cousin, Eric Mummery, who carves beautiful spoons and other kitchen utensils out of locally-sourced and found wood, which is part of what inspired me to take this workshop! Check out him out on Instagram or Facebook.

wood yearning to be a spoon!

delicious wood-coal sandwich

scraped-out bowl, ready for carving

not quite there, but well on its way to spoon-dom

Email Address
Sign Up

We respect your privacy.

Thank you!

I imagine evaluators are reflective people by default, whether by nature or practice. I definitely find myself obsessed with milestones and checking back to think about what changes and progress I’ve made in whatever timespan I’m reflecting on. The end of the Gregorian calendar year is a very exciting time for me! This one in particular because I’m coming up on my five-year anniversary of moving to Vancouver, which means catching up on my five-year plan that I had when I came out here. (People are sometimes very impressed when I say that I moved to Vancouver with a five-year plan and then I tell them the plan was, “check in at five years and see how things are going and if I should keep sticking it out or not”. Very developmental!) I’m happy to report that my hope to have some form of sustaining work with a promising future and not an over-bearing cost of living on top of it was well-exceeded by year four, so year five has been nothing but bonus learning and discoveries!

This year has been one of the most transformative of my life. I’m writing this at the end of a holiday home in Ontario where I’ve been seeing lots of family, and have gotten feedback that they notice a difference in me, even in my presence and how I carry myself (thank you, Alexander Technique!). It’s exciting and intimidating. I’ve also had a few people this year reference how I’ve seemed to “find my voice”, but I don’t think I’m done with that yet. I think I’ve got a lot more transformation to come. In the words of a wise sage from my youth, “This isn’t even my final form!”

I feel similarly about evaluation itself as a field and a practice. We are stretching, learning, pushing, and transforming, holding up the familiar to a series of critical lenses and asking ourselves and each other to try different ways of evaluative thinking and being than the ones we are most comfortable and practiced with. We’re asking questions about what evaluation is for, and who, and what it takes to do it. I’m both alive with the energetic possibility of it all, and a little overwhelmed by the responsibility and urgency of it too.

One of the on-going conversations I’ve had this year is with a friend who talks about risk and taking risks and how and why we do, not because we aren’t scared but because we want it enough we can’t imagine not doing it, whether it’s launching ourselves off a mountain (she’s a paraglider), falling in love, or upending our professional practice. I’ve been playing with risks all year, scaling my way up along the intimidating rock face of my fears and insecurities, looking for a path I’m capable of following, that isn’t too hard, while still taking me higher and higher. I like a rock-climbing metaphor because it’s a reminder that pursuit of risk shouldn’t be reckless. Anxious paralysis can be an obstacle to innovation and change, but fear and caution are gifts too. There are real dangers to be minded and navigated, and launching forward with headlong abandon (a tempting antidote to years of timid hesitation) can produce disastrous results. Particularly as an evaluator, my responsibilities are not just for my own concerns but for everything and everyone my work impacts. This isn’t an argument for halting, half-hearted ‘progress’ though—change is needed and the pressures are real—but a reminder that leaps of faith can be thoughtful and calculated in their own way. Paragliders test the wind and chose their moments.

Standing at the precipice of a new year though, I’m still invigorated by the uncertainty. In 2019, I learned to relax into the not-knowingness of it all by deepening my trust in myself and in the capacity of human resilience and love to see things through regardless. I took many risks, only some of them reckless. I opened myself up to possibility and imagination, sometimes plunging and sometimes wading, always swimming deeper and deeper. I’m full of plots and plans and schemes for 2020 and I have no idea which, if any, will actually come to fruition as intended by 2021. I kind of hope they all surprise me.

Here’s something I’ve been thinking about.

Imagine a big knotted clump of yarn, the kind of thing that happens when you leave a crafting project at the bottom of your bag for too long. (Alternatively, what happens to your headphones if you leave them in your pocket for five seconds.) It’s a solid mass of strands going every which way at the centre and a bunch of loops and looser tangles hanging off it. If you’re lucky you might know where the two ends are (assuming there’s only two!), but you probably have no idea what else is going on in there and how it’s gotten this tangled and useless. The fastest way to get rid of it would be to cut it (ah, Occam’s Razor), but that wouldn’t help you finish the crafting project, the scarf or hat or sweater that this thing is supposed to be.

Every knot is its own beast, so there’s no single specific sequence of steps to follow to untangle this one. But there are some basic things you’ll probably do, either from trial-and-error learning or past experience if you’ve done this before. To start, you’ll look over the whole thing and try to see if there’s a likely place to begin, a bit that seems amenable to a little tugging. You might give the whole thing a quick shake to see what that loosens up. You’ll want to track down those end strands if you can and keep an eye on them. As you pull a little bit at one spot and then another, you’ll keep looking back at the whole thing to see what’s shifting, what you might pull on next. There might be some false starts before you finally get some traction. With a bit of luck, things will begin to pull apart and the knots-within-knots will start to surface. There may be parts that have been pulled so tight it seems like you’ll never pry them apart again. You’ll be confronted with moments of guessing whether something is a slipknot, where one sharp yank will pull it through, or not, and the same yank will fix the knot in place for good. You might find yourself suddenly at an impasse after a whole lot of good progress with a something that just won’t give. You may end up with several separate equally confounding tangles, smaller but no more straightforward than the one you began with. At various points in the whole frustrating process you’ll wonder why you’re even bothering, why you don’t switch to watercolours or needle-felting. Or you might hit a flow where you press on and the act of disentangling feels as much a part of your hobby as the deliberate ‘entangling’ you do with hook or needle. (It’s definitely a good way to get familiar with the material of your craft.) Hopefully you hit a point where the tangles start to fall away on their own, when you find that little loop that got stuck inside another loop and then another, kicking the whole mess off, and then all of a sudden you’ll have a lap full of soft strands, ready to be rolled up into a ball so that crafting can continue, like the knot never existed. (Or maybe you transcend the paradigm, give up on the sweater, and turn the knot into a cat toy or cut it up and use it for stuffing.)

What earthly relevance is all of this?

One of the things I’ve been working on in my evaluation practice is my own internalized framing of what my work is—what it’s about, what it’s for, why I’m doing it—and shifting that from a focus on specific programs, “is it good or not? Was it a success or failure?”, and into something that is more contextual and strategic—“what is happening here? How does this relate to the larger change effort?”. Some of this is from moving deeper into developmental evaluation, driven by the needs of clients working in spaces of deep complexity and uncertainty, who are adapting and inventing practices and approaches as they go. But as I go further into this space, I also find that traditionally-focused summative and formative evaluations benefit from being contextualized within this wider lens. I can’t think of anyone I’ve spoken to recently in the social sector who didn’t have a sense that big shifts are needed and that individual programs and services are meant to be part of larger transformations. This is where I find the developmental perspective to be liberating and necessary, even when not doing strictly developmental evaluation work, because it moves out of the either/or paradigm of ‘success/failure’ that informs the summative and formative models and offers a different frame in which to think about and judge what we’re doing.

Here’s another way I think about it:

This is my understanding of an ‘ideal’ evaluation scenario, with evaluation involved at several points in a program development and refinement process. In the planning and design phase, evaluation helps surface questions about models and theories and assumptions, lay the groundwork for data collection, and gather early impressions and context. Then throughout the implementation, there are formative assessments, asking, “How can this be better?”, checking in on implementation quality and fidelity and beginning to look at outcomes, first the early ones and then later ones, seeing how the program theory is panning out. Finally, at the summative phase, going in-depth into those outcomes, what it takes to make the program work and whether it can and should be scaled and replicated, asking, “Does it work and is it worth it?” The underlying assumptions of this process are that things that ‘work’ can be reliably expected to keep ‘working’ in their own context and even in other contexts, and that what ‘working’ means will also be relatively stable. If something is good, then it is good, by whatever measure of ‘good’ we have agreed on. (And how “good” and “working” are defined and by whom and to what ends are separate questions that are often under-explored.)

Here’s another picture, one that captures more of what I have experienced and witnessed working in spaces of change, both in the formal social sector and in activism, and what another ‘ideal’ evaluation process might look like:

Looks a lot more like a tangled ball of yarn, right? The evaluation line is running parallel to the whole development process (which isn’t neatly distinguished into “design” and “implementation” either, since the whole thing is a constant exercise of doing and re-working and adapting). Learning and doing are happening together, and the nature and focus of the learning is shifting and adapting as much as the doing is. There’s an overall guiding direction (from one side to the other), but the moment-to-moment directions change, including sometimes going back over itself. It could just as easily be a journey-map of someone lost in a mall, or of the way we handle our yarn knot.

To me the disentangling of a yarn knot is metaphor of how big change happens, scaled down to something I can hold in my hands and my head. I zoom in to individual stands and knots-within-knots, then zoom back out again to assess the whole, then zoom in again. My immediate outcomes, their indicators, and my interventions are shifting and relative. I tug on a strand and notice the way it leads to movement on the other side of the knot, so I move to that part next and see what headway I can make there, looking back to the part I was working on before as well as keeping an eye out for what else might be happening elsewhere in the knot (and to ‘scale up’ the metaphor, imagine a knot the size of a Buick and a group of people working on different parts of it simultaneously, affecting each other’s work and hopefully also communicating effectively). This is the systems-view understanding that everything is connected and influencing each other, often in ways we can’t immediately decipher, with the need for timely data and insights to inform and support the work as it’s happening as an integral part of what the work is. There are formative and summative queries embedded through out this developmental process (“is this working? Could it be better?”), but they are contextual rather than absolute, and don’t require complete, definitive answers each time because the context will keep shifting anyway as we continue to interact with the system.

In these spaces, it’s not about “what’s good?” or even the more pragmatic “what’s good enough?” It’s about “what can we do right now that might suit our present situation as well as work in service of (albeit not always immediately directly toward) our ultimate purpose?” It’s strategic and deeply, deeply contextual, to time, place, purpose, resources, people, opportunities, etc. In these spaces ‘successes’ aren’t always successes and ‘failures’ aren’t always failures. Success and failure are relative, dynamic, and sometimes just irrelevant.

What does this look like in practice?

I find it challenging to bring up specific examples from my practice that exemplify the difference I’m talking about because so far I’ve found it quite rare to work on projects that fully embody this understanding. Even in developmental projects, the formative/summative way of thinking is deeply entrenched and often reinforced by funding models that serve short-term, linear, and compartmentalized thinking and ways of working (and I can’t single clients and projects out until their findings and process are at a stage for public sharing). But fortunately activism abounds in examples of working in strategic adaptation supported by deep evaluative practice (despite not being formally recognized as such).

Here’s one: sharing our pronouns. You might have experienced people sharing what pronouns they use, maybe in their email signature, on a button on their lapel, or as part of introductions at a workshop or event. You might have a practice of sharing your own pronouns or asking people for theirs. As someone who uses ‘non-standard’ pronouns (they/them), I end up advertising mine a lot and have written a whole blogpost on the topic which I share often, partly as an end-run around having the same conversation about them on repeat. Pronoun sharing is a fairly recent phenomenon, taking off within the last few years, though it’s far from universal. It’s also garnered a fair share of critique, both on form and function, and some of these have produced modifications. One early criticism was the language of “what are your preferred pronouns?”, which was argued to suggest that some people’s pronouns are merely ‘preferences’, and we’ve generally moved away from that language to just, “what are your pronouns?”. We’ve also pushed for pronoun-sharing to become a normalized activity for everyone so that we aren’t singling out the people with ‘non-standard’ or ‘non-obvious’ pronouns.

If we took a formative lens to this, we might be thinking, “yes, we’re tweaking the intervention to improve it, make it the best version of itself that it can be”. But some of the criticisms aren’t easily resolved. By making pronoun-sharing a routine activity, we’ve also fostered scenarios in which people have to choose between being ‘out’, with all the risk that entails and the infringement on self-determination, being misgendered, or, if they chose not to disclose at all, being perceived of as less ‘progressive’ than their cis and binary-gender peers who can put “she/her” and “he/him” in their email signatures without concern. I actually dislike having to constantly advertise my pronouns. I find it distracting and a constant reminder of the awkwardness of not fitting in. When I do it, it’s because I want to find and connect with other nonbinary people and because I believe it, to some extent, normalizes the existence and presence of nonbinary people (“to some extent” because it also runs the risk of turning my uniqueness into a ‘standard’ if people see me into a reference point for nonbinary-ness instead of as a single datum in massively diverse constellation of people). But to me it’s not a solution, it’s a tactic—part of a larger, emerging strategy.

The risk is that we settle into complacency, “Okay, we’ve got pronoun-sharing now, so that solves that problem and there’s nothing else we need to do to structurally shift our harmful, rigid, binarist conceptualizations of gender that are also bound up in white supremacy, colonialism, and misogyny” (which is the direction here, the purpose, the knot being untangled). Pronoun-sharing is a useful intervention to the extent that it gets people thinking differently, creates opportunities for generative conversation, expands imaginative horizons, eases the pain of erasure and being made invisible, and demonstrates respect for a wider expression of what being human looks like. It also represents and reflects progress made so far to trouble and disrupt entrenched understandings of gender in this particular social, political, temporal context. It is also imperfect and imperfectible. Pronoun-sharing only makes sense in this given context because of where we are coming from and where we are trying to go and it comes with trade-offs and limitations. It is not a solution we should expect to scale or replicate ad infinitum. In fact, if we did we would only become committed to upholding or even creating the conditions in which pronoun-sharing makes sense—i.e., the binarism, colonialism, racism, misogyny, etc. (Which is what happens when we assume that ‘progressive’ gender concepts that are specific to white, Western, Anglophone contexts apply without modification to all other linguistic and cultural contexts.) ‘Success’ in the immediate would mean failure in service of our larger goals. And the critiques of pronoun-sharing are actually part of what makes it valuable, because these are conversations that help us move beyond the present moment into the next opportunity. When we can see and talk about what we don’t want, we can also have new inspirations about what we do. So ‘failure’ carries the seeds of success (or change).

And what kind of change? I often see incremental and transformative change presented as mutually-exclusive and oppositional approaches—slow and steady gains versus the total upheaval, the progressive versus the radical—but in practice I have only ever seen real transformative change emerge from a collection of strategically-leveraged incremental changes, climbing a rock face handhold by handhold. To go back to the yarn analogy (I think almost entirely in metaphors and puns, you’re welcome), the need to work with the yarn in whatever state it is in at any given moment is to look for what smaller shifts can give way to the release of the whole knot. The last few disentanglings may seem like the most transformative, but it was the whole process that mattered. We can also get caught up in those incremental changes in ways that work against our guiding purpose—going back and forth over the same bit of freely-moving strand because it’s easier to work with even though it’s not actually helping unravel anything else, or finding that the process of disentangling one part tightens up another part and makes the whole thing worse. (Honestly, there so much in this metaphor for me that I’m not even going to try to get at all of it in one blog post. Hopefully some of you reading this are inspired to take it and run with it.)

(Also, for another example of what it looks like to pursue change adaptively over time, with an emerging and shifting strategy that unfolds in unpredictable ways and through collective and collaborative efforts, watch this video of an event that took place in Vancouver recently, Decolonizing the City: The Future of Indigenous Planning in Vancouver, where a panel of Indigenous planners describe and discuss their work from over the past decade, what’s shifted and how and what is yet to come. Also basically any activist movement from the US Civil Rights movement to the current Hong Kong protests is a case study of adaptive, emergent strategy. What’s harder to find is explicit examples of evaluators being useful in these scenarios, though evaluative thinking abounds.)

Why does this matter?

I know we’re not always able to (or feel capable of) working in spaces that are acknowledged to be about systems change or complexity (though that doesn’t mean that we aren’t still in those spaces), but I think it’s ultimately a more hopeful, helpful frame to recognize that our formative and summative evaluations are often actually us working on just one brief segment of that bigger tangly process that happened before we got there and will keep going after we leave, and we can make our work more useful by accounting for that. In fact I think many of us already try to, if we are attending at all to the contextual factors informing the spaces we’re working in and people we’re working with, crafting questions and processes and recommendations that speak to strategic needs (having sussed these out through thoughtful discussion and exploration with various stakeholders, since they’re not necessarily the ones outlined in the 5-year strategic plan that’s been gathering dust for the last three years) and helping people take “the next elegant step” in their work, as a friend of mine says. Hopefully we’re illuminating the strategic nature of the work at the same time, and using evaluation as an opportunity for people to connect dots between micro- and macro-level shifts and see their work in context, which can be both humbling and inspiring. Keeping the whole knot in mind even while working on one part of it at a time reminds us to think in terms of context and to hold onto our definitions of “success” and “failure”, “working” and “not working” lightly.

It can also help us not fall in love with our interventions (and our designs, tactics, and even our strategies) to the detriment of our purposes. Some of the inspiration for this post came from a recent experience I had attending a presentation and watching someone totally shut down in the face of a valid critique of the intervention she was working on. She was presenting the intervention as a model practice, eager for it to be scaled and replicated, and was committed to seeing it as a ‘solution’ where any shortcomings were actually features, not bugs, so when the critiques came up, she had nowhere to go with them. (She literally ended the presentation and shooed us from the room!) But when I heard more about the reasons behind certain ‘design’ features of the intervention and the constraints they were operating within from her co-presenter, it all sounded reasonable to me—I could see what they were trying to do and why it was happening the way it was at the moment and what it might lead to in the future. I wondered if the ‘model practice’ couldn’t have been viewed instead as a useful workaround for a specific time and place, from which valuable learnings could be shared with people working on similar challenges in their own contexts. With so much invested in this particular intervention being the way it is right now though, I wonder how adaptable it will be to future opportunities, and I’m trying to hold this lesson in mind for the next time I’m tempted to fall in love with a particular change effort.

The structures we work within (looking at you, current funding models! And you, characteristics of white supremacy culture!) pressure us to come up with best practices and replicable models and scaleable results and all sorts of things that fit snugly within the success/failure, “either it works or it doesn’t work” paradigm (hm, sounds like a mix of either/or thinking, perfectionism, sense of urgency, quantity over quality, and progress is bigger and more… seriously, characteristics of white supremacy culture). We don’t always have the power to resist these pressures fully when they are structurally embedded—that’s a knot we are still untangling—but we can resist internalizing the assumptions that keep us in this narrow frame. Otherwise we end up fighting off despair with false hope while we struggle through, or redefining our goals into things so achievable as to be irrelevant and we never have to worry. Working with a systems view doesn’t have to be overwhelming, abstract, or impractical. We work with what’s in front of us, we think about what we’re ultimately trying to do, we get as real as we can about the relationship between the two, we change it up when we need to, and (outside of imperfect-but-useful tangled yarn analogies) we work with others to do it.

Gratitude

There were a lot of different sources of inspiration for this particular post and I want to name some of them. I wrote the first draft of this post right before attending the 2019 American Evaluation Association conference and was delighted to find that my conference experience ended up reflecting back, reinforcing, and deepening so many of the ideas in it, including (among MANY) the plenary with Glenda Eoyang of the HSD Institute, Khalil Bitar of EvalYouth, and Dr. Dominica McBride of Become Center, on Prioritizing What Matters In Evaluation, and a session with Dr. Donna Mertens, Dr. Katrina Bledsoe, Dr. Dominica McBride (yup, twice) and Jen LoPiccolo, and Dr. Gail Dana-Sacco, called, An Intersectional and Transformative Future for Evaluation. These sessions (and others!! but for this post these stand out particularly) dropped the mic on using evaluation to collectively disrupt, strategize, advocate, and transform in service of love and justice, which is about as big picture as I can imagine. I’ve also had my attention tuned to the need to integrate evaluation and strategy as a result of work being done by Zena Sharman and Julia Langton of Michael Smith Health Research Foundation, Jara Dean-Coffey and Jeanne Bell in their recent Non-Profit Quarterly webinar, Developing Practical Systems: A Masterclass on Linking Strategy and Evaluation, and adrienne maree brown’s incomparable Emergent Strategy. Many, many conversations with friends also shaped this post (you know who are—thank you for letting me dry-run it on you), and of course the reason I actually finished it was the encouragement from friends, old and new, at a recent Equitable Evaluation workshop, for whom the yarn metaphor will live on. 😉 Thanks, friends.

h/t to Zena Sharman for the title of this post. Alternate titles included: a good yarn, honing the ‘craft’, from wicked problems to ‘knotty’ ones, and, simply, tangled

Coda

If you’re hungry for more metaphorical forays into what it looks and feels like to do systems change in complexity, read the chapter “A Mother’s Work”, in Robin Wall Kimmerer’s Braiding Sweetgrass. And then read the rest of Braiding Sweetgrass. It’s all there.

I know what you’re thinking. Do we really need yet another “The Art of”? Can’t we let this naming convention go and find something more creative for our workshop titles? I sympathize, I do, yet here we are. Because this time it’s my turn and, darn it, it’s just such a good description of what we’re up to!

Some of you may have seen me tweet a teaser about this recently, but here’s the full promotional package. From October 16th to November 6th, Rita Fierro and I will be leading an engaging online workshop on practical approaches to developmental evaluation, hosted by our lovely friends at Beehive Productions. There’s more info, including pricing and timing details, on the registration page: Developmental Evaluation: The Art of Learning

It’s a 4-week online course of four weekly 2-hour interactive sessions. We’re offering a mix of teaching, stories based on our own experiences, new tools, and small- and large-group conversations. This is not a “basics of DE” course (though we’ve got resources to help people get up to speed if they’re brand-new to it). It’s a space for those who are interested in building and expanding their own DE practice to look at what it’s like to “do” developmental evaluation, grounded in real-life stories and examples. We’re going to be talking about things like working in complexity vs. complicated realms, readiness for and navigating change at organizational and personal levels, sifting through multiple layers of learning, and building responsive learning frameworks. If you’re new to DE, it’s a chance to explore what your own practice might look like. If you’re already in the thick of doing DE, there will be conversations to deepen your thinking, other practitioners to connect with, and new tools and resources to try!

Sessions are recorded and released after each session, so if you’re interested but not available on all of the posted dates (Wednesday mornings PDT, from Oct 16 to Nov 6), you will have a way to catch up and stay involved. Please feel free to share this widely and contact me if you have any other questions about the course.

Hope to see you there!

While I’m self-promoting, here are some other cool opportunities from folks I know!

From Kim van der Woerd, Billie Joe Rogers, and Sofia Vitalis – Culturally-Responsive Evaluation workshops in Victoria (Oct 15) and Vancouver (Oct 18)

From Chris Corrigan, Caitlin Frost, Pawa Haiyupis, Kelly Foxcroft Poirier, and Amanda Fenton – The Art of Hosting Spirited Dialogue: Inviting Diverse Perspectives in Complex Times (Whitehorse, YT, Oct 29-31) and The Art of Hosting Meaningful Conversations and Participatory Leadership (Bowen Island, BC, Nov 10-14)

From Gary Hutton and Anastasia Gaisenok – Systems Thinking with Indigenous Lens, a 3-hour to full-day workshop offering (contact Anastasia at connect@anastasiaconsulting.ca for more information)

And we made a video! And I’m practicing embedding videos on my website!