carolyncamman

My friend Chris Corrigan recently wrote a great blog post on weather and complexity, riffing off a statement from a retiring weather forecaster to talk about how to navigate complexity. One of my favourite COVID-era hobbies was tracking weather patterns with Chris and our friend Amanda. As systems swept in and out over the coast, we would announce in our group text the moment when rain reached our respective locations, from Nex̱wlélex̱wm/Bowen Island to East Van to New Westminister. Chris always has a fascinating app or person he follows on Twitter with cool maps and data about what is actually happening and the three of us got quite nerdy about it. (I’ll never forget on the first night of the heat dome, when he showed me a heat map visualizing that column of hot, red air going straight up to the highest levels of the atmosphere, sitting on top of us with nowhere to go. Terrifying.)

So when I need a simple way to illustrate how data alone is not the answer to our evaluation challenges, I often find myself using weather forecasting. It’s something that we’re all familiar with in a general sense, but often don’t have a full appreciation of what’s actually going on behind the scenes. It also speaks to the very practical circumstances of our day-to-day lives, including the fact that we live in a world of increasingly dangerous weather events and climate change.

Weather is also something we can collect fairly concrete data about. These are physical phenomena that we can measure directly with reasonable precision and reliability. We can then subject this data to some pretty sophisticated mathematical modelling and make decently accurate predictions about what will happen at least briefly into the future. The meteorologist that Chris’s post quotes foresees a future into which our statistical clairvoyance can still improve by leaps and bounds with new technological breakthroughs.

Yet data collection and even data analysis are only one part of what we need. A sentence that stood out to me from the quote at the start of that post, “The atmosphere is a nonlinear system, meaning our ability to forecast it is extremely sensitive to knowing the exact condition of every breath of air”, really gets at the scope of the challenge. Highly-connected, interdependent complex systems are limitless and irreproducible in models. We cannot capture every single small element that might affect what happens next. Not to mention, because these systems are non-linear, where inputs and outputs are not proportionate (i.e., ‘the butterfly effect’), even the tiniest unmeasured element, like a breath of air, might end up being an integral part of a complex emergence. Add to that the logistical cost and complexity of collecting, managing, and analyzing high volumes of data, which is is limited not just by the human capacity to do so but literally the physical computer processing power available. And this again is relatively concrete data that lends itself to this kind of statistical modelling (versus the rigamarole of developing abstract proxy indicators of non-quantitative concepts, like “wellbeing” or “motivation” or “knowledge”, which do not have universally-agreed upon definitions much less direct modes of measurement).

On top of all of that, even with all of the data we can get and the physical ability to analyze and make sense of it, and the historical and theoretical knowledge to build reasonably accurate predictive models about it, not only is that prediction still always going to be inherently limited and fallible (which does not detract from the work of the forecasters), but having all of that in hand still only at best offers us somewhat more information with which we must still ultimately make a decision about what to do. And even that information will be highly context-specific and have a fairly short shelf-life of relevance. Alongside getting a heads-up that my area is at risk of an extreme weather event within the next six hours, I need to have my own contingencies in place and hope I live in a place with a well-resourced and well-designed emergency response plan and the capacity and political wherewithal to carry it out. That also

No shade to the meteorologists, of course! Their job is to make sure the information is timely, accessible, and reliable, and that’s important. But data will never tell us what to do or how to do it or make sure that it’s acted on well—that’s on us. 

And you may think that evaluation isn’t about forecasting, it’s about accountability and learning and looking to the past to describe and report on what happened and what has been accomplished. But in practical terms, many of us approach evaluation with the idea (rightly or wrongly) that what has happened in the past will happen again the same way in the future. When we ask, “Does the program work?” (a question situated in a generic present tense), the logic of evaluation is to look at how it has (and hasn’t) worked already and extrapolate into a present (and presumably future) tense based on that. When program sponsors decide to fund or support a particular initiative, they are doing so with an eye to the future and what they hope or believe or want to see happen, usually with a lot less concrete data to go on than what the weather forecasters are working with. We look to the past the same way we use a mirror to look at the back of our heads—to see what we can’t see.

If you think the solution to the uncertainty and guesswork of this process is “data-driven decision-making”, I refer you again to the sentence, “our ability to forecast it is extremely sensitive to knowing the exact condition of every breath of air”, and ask, who will build and fund this data infrastructure and make sure it is available to and appropriate for everyone (so as not to design inequity into the system from the start)? Even the meteorologist notes, “we crudely sample the atmosphere directly with instruments that aren’t precise and numerous enough, and make even more approximations with remote sensing like satellites”. These are multi-billion investments over decades that are still underfunded and less developed than they could be. 

As Chris makes the point in his post, these models do not give us much insight into hyper-local conditions, which are greatly impacted by the geological specificities of our exact contexts. Saying, “This program works. It’s a great intervention.” does not account for the particularities of how a program might play out at another time or in another place. And the answer to that is not ‘more data’ and ‘better models’, but better situational awareness and attention to the context of the present moment, more acknowledgement of our agency and responsibility in the decisions we make about our social interventions, and looking to the data available to us with a critical eye for useful insights rather than definite answers.

Want to learn more about evaluation with me? Check out my upcoming offerings, including the WEAVING IT IN evaluation workshop.

What a time to be alive. Back when we could still gather in groups to sing together, that was a lyric from a song I loved in the drop-in choir group I sang with every week. It is time now, and what a time to be alive.

Yesterday was Juneteenth. Tomorrow is National Indigenous Peoples Day. It’s the height of Pride season. It’s the summer solstice. It’s the first day of Cancer season. There’s a full moon in Capricorn on Thursday, which is also the Strawberry Moon, which is also the last supermoon of the year. Somewhere in there it’s also going to be my 35th birthday. And we are still living within crisis within crisis within crisis within crisis within crisis. What a time to be alive.

I mark the passing of time with reflection. I love to skip back through my calendar apps and my photo albums and remind myself where I was, what I was doing, what I was thinking about and trying to get done a month ago, a year ago, a decade ago. I realized yesterday that I have just passed the ten-year mark of being an evaluator.

Ten years ago last month, I began leading my first evaluation project. Still a grad student at the time and would be for another couple of years, but that was the real threshold for me. That was the moment I stepped into the role of evaluator and took on that responsibility. The work I started doing then had all the portents and shades of what I would spend the next ten years plus trying to realize in my practice. I wanted to do work that was meaningful, helpful, accessible, and exciting for people pouring their hearts into transforming their corner of the world.

It’s not yet quite the three year anniversary of when I felt truly inspired in how to do that. Three years ago, in September 2018, when I took a course called Transformative Evaluation Landscape from Kim Van Der Woerd, Sofia Vitalis, and Elders Lillian Howard and Roberta Price, and with a cohort of peers experienced for the first time a complete immersion into a space where we were asking each other and ourselves, why are we evaluating, who are we evaluating for, and according to whose values and worldviews?

I’ve told that story many times, but I don’t think I’ve shared yet the phrase that got stuck in my head and my heart by the end of that course, that I’ve been carrying ever since. I’ve realized that this phase is the thesis statement of my practice journey, my theory of change, my core principle. Evaluation is for everyone.

It took me over a year to even unpack what that sentence means to me. Evaluation is for everyone. I didn’t know what it meant in my head so much as I felt it with the certainty of my whole body. Evaluation is for everyone. That is the beating heart of my practice.

Evaluation is for everyone

means that evaluation is supposed to serve everyone. Not just the people with the money. Not just the people with the power to set policy. The power dynamics in evaluation are intense and so much of the decision-making revolves around appeasing and appealing to those with a very specific kind of power. Rarely have I seen the energy expended on engaging and satisfying funders matched with what will engage and satisfy the people directly impacted by the program and the evaluation. This is a systemic issue, not the fault of individual organizations and people trying to keep their work alive and funded, but as evaluators we play into it and collectively seem to have resigned ourselves to it being an inevitability. I remember getting into a Twitter dust-up with someone who argued that the people with the money ultimately should get to decide what they do with that money, and evaluation therefore is there to serve them first and foremost. Others out there have already done the work of unpacking a statement like that, including Edgar Villenueva in Decolonizing Wealth and the entire Equitable Evaluation framework, so I’ll mostly leave it at that, but I will also add that being able to pay for evaluation and then having evaluation influence funding is an embarrassingly obvious “success to the successful” system trap that we should be a lot more concerned about as an equity issue in evaluation. Evaluation cannot serve equity when it is extractive, taking data from some to generate recommendations and reports for others, and when it uncritically reinforces and upholds the inequitable distributions of power already present.

Evaluation is for everyone

means that everyone can and does do evaluation. Everyone is an evaluator. Evaluation is a fundamental human behaviour of learning from experience and making judgement calls based on what you were trying to do, what happened, and what you want to happen next. When I do evaluation capacity-building work, I always come from the standpoint of, “You already know how to think and work this way, you do it all the time; I’m going to offer some language and framing so that we can talk about how we’re doing it together, do it more intentionally, and feel more confident and less lost”. There is nothing about evaluation that requires a technical degree or a particular credential. I became a professional evaluator when I started doing paid evaluation work and I brought to that a host of skills and life experience I had acquired up to that point, only a subset of which came from formal evaluative training (and only a subset of that training was ultimately useful to me as a practitioner). To be a professional evaluator is to step into the responsibility of a particular role in a project (which is not always the role we think it is), but it is not to lay exclusive claim to what evaluation is as an essential human endeavour. I have never once entered a project as an evaluation consultant and found that no evaluation was happening. People are already in there, working with the data they have, using the critical and creative evaluative thinking that comes naturally to them, generating, refining, implementing, and adapting their theories of change. The quality and utility of my work increased vastly when I stopped doing evaluation like a social science researcher, mediating and creating knowledge for people, and started doing it like a participatory facilitator, hosting the spaces and opportunities for people to create the knowledge they need to do their work better and recognizing their inherent capacity and capability to do that.

Evaluation is for everyone

means that evaluation needs everyone to be part of it. It’s for all of us to do together. We can evaluate as individuals and reach individual conclusions and choices, but when the change requires collective work, our evaluation must also be collective. The things we are struggling with are huge, bigger than we can address on our own or with individual programs and organizations. For evaluation to be more than an institutionally-mandated surveillance-and-compliance exercise and to not get stuck in a space of organizational navel-gazing and toxic intellectualization, we have to notice and challenge our individualistic approaches to evaluation. This is difficult, in no small part because our funding models and the dominant cultural frameworks we’re all being forced to work within reinforce individualism at every step. But every evaluation we do is a chance to see the paradigm we’re in and look for opportunities to play with it, subvert it, push back on it, and imagine something different. What would it change about our work if we entered it with the fundamental belief that evaluation is a collective, communal act? That it’s not enough to circulate reports and products after the fact, but that the process, the evaluative act itself, belongs to and depends on a community of people invested in a change and a state of being in the world? That this is not a niche form of evaluation appropriate to special circumstances but reflective of a worldview and a belief that we deserve to live in systems that are arranged to respect our agency and our interdependence.

Evaluation is for everyone. Put it on a t-shirt.

Happy solstice, friends.

Reading this Twitter thread and thinking about the conversations we’re having in evaluation about being human and bringing our whole selves to our work made me really want to talk about being disabled and an evaluator.

But I have no idea if I’m disabled.

Am I?

I’m writing this in bed, with my trusty heating pad and medication I will take every four hours, the maximum safe dosage allowable, for the next few days. I am experiencing pain that, when it peaks, makes it hard to focus or stay coherent or do anything but gasp. And when it subsides again it stays uncomfortable enough to be distracting and make me not want to do much besides lie in bed with a heating pad, a snack, and a distraction. The pain comes bundled with other symptoms that make me want privacy and safety and assured access to the things I need to be comfortable. Before learning about these chemical heating pads a couple years ago, something like this would have meant cancelling any in-person meetings or taking a day home from work. (One more reason I prefer not to work in offices, even before the pandemic.)

This is an improvement. When I was a teenager, the pain was vicious. I missed school. I threw up. I blacked out. I lost control of my body. I cried and told my mom not to stroke my hair because it hurt. Everything hurt. The medication I took then was stronger, so strong I can’t take it anymore. But mercifully the pain has tapered down from a regular 10 on the debilitating pain scale to a 7-8.

This is the only diagnosis that I have. Primary dysmenorrhea, cause unknown. It might as well be a question mark. Because it’s a condition that affects primarily women, and the reproductive organs typically associated with women, it’s under-researched and poorly understood. When I was fifteen, my family doctor joked, “Well, the one guaranteed way to clear it up is to get pregnant!” I waited it out instead, mostly in silence and in private. I rarely speak about it outside of careful euphemisms, and only then out of the sheer necessity of asking for accommodations. (I made the mistake once of mentioning to some fellow staff at a job that I had my period and wasn’t feeling great, and watched a man I respected and cared about turn abruptly and silently walk away as if I had uttered something so offensive he couldn’t bear it. He never apologized and I never told him how much it hurt.) I built my life around the (only semi-predictable) rhythm of it, the management of it, staying safe. I’m not a fan of long camping trips and uncertain accommodations. I always have a strategy around needing to cancel or bail on something last minute. I look for privacy and predictability in how I travel. I’ve had enough privilege to make that happen so far.

So that’s the thing I’ve had diagnosed.

Then there’s the rest of it.

I was eleven or twelve the first time I told someone (my mom) about being depressed and suicidal. I remember because I knew it was a big deal to bring it up. She was concerned. She talked to her minister. No one knew what to do, least of all me. Nothing came of it. My grades were good and my pain was quiet. And when it wasn’t, I was a “good kid” (a.k.a., white, middle-class), so I got a pass. No stigma, no punishment, and no help either. I learned to navigate panic attacks without knowing what they were. I shrugged off things I didn’t know were PTSD symptoms as weird teenager stuff. I kept it to myself. (Too bad the dentist never mentioned the severe teeth-grinding as a possible symptom of untreated anxiety, or even mentioned it at all—all that damage is going to come back and bite me in the ass someday. Or gum it, at least.) When I went through that ‘weird’ phase in my twenties where I just couldn’t stop crying but had no idea what I was crying about, the EAP counsellor I saw through my mother’s work benefits suggested loneliness and that I join a club. I did community theatre. I loved it. Didn’t actually help, but I enjoyed it.

Dissociating helped though. School was great for that. Lots of structure, plenty of distractions, a reward system that I was rocking (being white and financially-privileged helped a lot here too), and a good long-term solution since it could occupy decades and decades of my life if I took it all the way. And psychology turned out to be a fantastic discipline for looking out at the world without reflecting on yourself or remembering that you have a body, past, or identity (I mean, you can, but no one will make you and some people will grade you down for it). I stopped writing poetry and short stories and focused my creativity on designing experiments and developing research programs. I swapped speculative fiction for textbooks. I took classes in clinical and abnormal psychology and blissfully never once recognized myself in any of the abstracted diagnoses I studied, despite qualifying for a few of them. No one said I had to do school this way, but it was easier than being in pain all the time and my grades were great (for a while), so no one was complaining either, least of all me.

If you’d asked me at any age up to about twenty-six if I’d ever significantly struggled with my mental health, I’d have confidently assured you that I hadn’t. I’d learned about real disorders and no one had ever diagnosed me with one of those. I’d have also assured you that I didn’t grow up in an abusive household until I was thirteen, that I was a woman, and also as neurotypical as can be. (Oh geez, and I didn’t even touch on the disordered eating and dysmorphia, did I? It gets all tangled up in the trauma and gendering and sensory processing and executive dysfunction. Never showed up as neatly as it did in the books I was given as a slightly chubby pre-teen in what I imagine must have been some kind of preventative intervention that mostly just confused me and gave me cool tips on how to hide having an eating disorder.)

Anyway. Turns out trauma, dysphoria, and feeling like a constantly-overwhelmed alien creature in a strange and hostile environment are good predictors of uncontrollably crying on buses. Also it turns out that knowing these things are part of the fabric of your life doesn’t unweave them from it.

Is this disability?

Other things come up. It’s all tangled together. Treating myself like a brain in a jar all the way through grad school turned into chronic back pain. Pairing that with the unleashed tidal wave of repressed and ignored emotions and my retreat into the safety and comfort of prone positions (a.k.a. lying down all the time) on top of a lifetime of under-treated chronic physical ailments (itself the product of it being hard to have a healthy relationships with your body when you exist in a societal context that is hostile to bodies in general and also when your body has nearly always been an scary and confusing place to be for several reasons all at once) caused the muscles of my feet to seize up so badly I was unable to walk without pain for months (and the effects have lingered for years after that). Walking is one of my go-to ways of managing my mental health, and losing access to that made things harder but also forced me to find other ways to cope, including talking to people about it and asking for help. That’s been powerful and empowering, and at the same time exhausting. I spend a lot of time coping and compensating. I am careful about boundaries and rest because I’m aware of the consequences if I’m not, and because I can be.

Coming to work with my whole self means coming with my whole self. The parts that hurt. The parts that struggle to function. That can’t sometimes. That need to lie down. That need time. The parts that remember crying, blacking out, and not feeling safe or together or okay. The parts that know we don’t speak of these things, of any of these things. And when I do the work the way I feel called to do it, when I bring my whole self, it brings all of this and asks me to reckon with it, hold it, sit with it. Healing is an endurance test. There’s a tissue-box-to-human ratio in my house of 5:1 for a reason and it’s not seasonal allergies. Falling apart on my way to being whole.

Maybe. Or maybe it doesn’t go away. Some things are just the things we live with. Seeing every disability as a condition to be cured and a tragedy if it isn’t or can’t be is part of ableism.

But is this disability?

One of the weird things for me about the pandemic was how somethings actually got a little better, a little easier. Suddenly there were people were talking about feeling the way I feel a lot of the time. The world got slower (for some). Staying home all the time and wearing comfy clothes became normal (for some). It became more okay to talk about being scared, angry, upset, overwhelmed, and to “be human”. Suddenly, at least for a little while, it was mainstream to organize your life around surviving instead of producing (though in a capitalist context, our main mechanisms for “surviving” still strongly depend on exploited labour, we just temporarily reduced the burden of exploitation for some while maintaining and increasing the burden for others in ways stratified by race, class, gender and more).

My friend Erin wrote about something similar in an Instagram post shortly after the global pandemic was announced:

“Vulnerability has blossomed into a popular virtue in the last while. The guru of vulnerability, Brene Brown, has given us a way to respect and admire, and cultivate it—when it’s emotional. When it’s optional. But when it’s physical? When it’s forced on us? Ahhh…. well now we’re in the territory of the disabled. We’re the gurus. The social isolation, the process of coming up with creative ways to handle uncertain and indefinite constraints, the constant mental math of wondering how seriously to take a thing while still being able to take pragmatic precautions but also not torture yourself with disappointment, panic and overwhelm. The immutable reality that you have no control and no way left to convincingly pretend you have control—it’s familiar territory to us. We are watching the world around us contend with these experiences for the first time and feel for the stress and pressure and bewildering level of personal responsibility people now have to navigate.”

We spoke about how it was in a way delightful to see how people who didn’t normally need to were adapting and figuring things out and making it work. To watch mutual aid and pods and systems of interdependence flourish. I found I had useful advice to give about crying self-care (HYDRATE) and navigating emotional breakdowns, and it felt fine to talk about it. Publicly. On my ‘professional’ twitter account. The one I link to in my email signature that goes out to clients. And it didn’t feel wrong. As Erin had noted, emotional vulnerability has been experiencing a renaissance (Brenaissance?), and the realities of the pandemic pushed that further.

But… when the emotional vulnerability isn’t optional? Or temporary? When it revolves around things that are taboo, stigmatized, or uncomfortable? When we’re not in the first few months of the pandemic and “all in this together” anymore? When the momentarily-stalled machines of productivity-above-survivability growl back to life? When they never stopped for you? When it’s tied up in racial injustice, systemic violence, and other institutions of privilege that would very much like to carry on “as normal”? Whose vulnerability? When? About what? (I am emphatically not speaking only of myself and my experiences here. My whiteness and access to financial security have kept me alive and given me opportunities to be vulnerable with less risk and fear than others contend with and I know it.)

And am I disabled? I don’t know. Maybe. I could be more disabled, I know that. And the emotional pain I experience is as weird as the menstrual pain, in that I can switch from feeling like I’m going to die, or I’d like to die if it means not feeling that way anymore, from a 10 on the debilitating pain scale all the way back down to a 0 in an instant. Like it never happened. But it has, it does, and it will again. That I’ve learned. So I design my life around the (even less) predictable rhythms of it, and hope for the best. I anticipate. I learn from it. Is that disability? I don’t even know where all this stands relative to other people. Who can know when so much of the game is not talking about playing it.

I showed a draft of this post to Erin and her comment was, “It feels like you’re digging into what is disability and what is human in a deep way here. And the idea that we’ve separated disability from being in a way that makes parts of being human feel disabling.”

I have no idea if I’m disabled. I’m probably even less clear now then when I started this. But the point that has surfaced for me in all the messiness of this is if we’re going to be human in our work, that absolutely requires deep structural changes in how we work and the spaces we work in. Inside-change needs outside-change, and vice versa. Being whole and human is a lot. If I were to attempt to draw a ‘vent diagram’ for it, I would probably end up with a lot of overlapping circles, with one saying, “Being whole and human is necessary for me to do the work I need to do well” on one, “Feeling like I’m able and allowed to be whole and human, including the messy and painful parts, is critical for my own mental wellness” on another, “Being truly whole and human carries its own emotional toll and potential for mental harm” on the third, then “Explicit displays of vulnerability or personal disclosures aren’t required for authentic wholeness and humanity”, and another with “The culture of silence and denial about struggles, especially stigmatized and taboo ones, as well as the limited framing afforded by an individualist view of the world limits our collective awareness of what wholeness and humanity entails”, and one last one that’s a bigger circle enveloping all of these, “The context of social injustice and inequity affects what makes up each of our wholenesses and also what we must navigate in our choices to show up in it”.

It’s a lot. It doesn’t resolve neatly. It doesn’t find a natural conclusion. The other writing advice Erin passed along to me was “resist arriving”. She’s kind of brilliant that way.

Extra Reading

Want more of Erin’s brilliance? You’re in luck! Her first book is coming out NEXT WEEK. The deliciously-titled, If You Really Love Me, Throw Me Off The Mountain, a memoir about risk, love, disability, romance, paragliding, and, oh yes, there’s French fries. Still time to pre-order!

Elizabeth Grim wrote a beautiful post about her experiences with her experiences with Type 1 Diabetes and how that’s informed her work as an evaluator.

There is so much more to be said about disability and its intersections and entanglements with racial justice and social transformation, and Mia Mingus and Leah Lakshmi Piepzna-Samarasinha are two brilliant people who have written and spoken about this extensively. Check them both out.

A friend asked me today how I was figuring out the current crisis, and we had a lovely clarifying conversation about our respective struggles and journeys. A few ideas surfaced for me that I want to preserve and share.

One is that I’m looking to the desired present instead of a desired future. Not because I have no hopes or aspirations for the future, but because I don’t find it helpful right now to aim for something I can’t see. I don’t know what the future will hold. I don’t know where this moment goes. I’m hoping there’s a future out there so different from this one that I can’t even imagine it fully much less trace a path to it by design. All I want to do is find the best part of whatever moment I am in, and work with that.

My friend asked if I thought humanity would better or worse after this moment and I realized don’t think there is an “after” in the sense of a threshold, a “before and after”. I think the change is already here, I think “better” or “worse” will be a mix, and what happens next will play out on vastly different timescales. Some of the realities of today will be a memory a year from now. Some of the implications won’t be apparent for generations. My grandmother was shaped by the Depression, and through her it shaped her children, and through them her grandchildren. So I’m not navigating by a future vision, I’m looking to the immediate moment and trying to find the need in it and serve that need with whatever I have to give. As my friend put it, thinking of her one-year-old, this moment is requiring us to model the kind of people we need to be right now in order to get through this, and that’s what the future really needs from us.

I’m also trying to pace myself. In all ways I’m trying to find the rhythm I can live by, to pace my sorrows and my joys, to let there be swells of activity and ebbs of rest. I’m thinking about the last time I sang in a crowd, one of the evenings of song and harmonizing led by Vanessa Richards at her Van Van Song Society weekly drop-in choir (now moved online where she’s exploring new ways to keep community connected in song), and what it feels like to be part of a throng of people feeling each other out by voice. One of the things I love about Vanessa’s choir is the freedom to simply to sing together, without any particular aspiration beyond joy, community, and finding what sounds good to our ears and our bodies. We vocalize and play and find harmonies and fit in where we feel comfortable. That’s the kind of pacing I’m seeking, feeling out where I fit amidst all the other voices and bodies, in a way that’s comfortable to me while lending myself to the song. This is a long road, but we’re walking it together.

And that’s the other thing I’m working on. To learn to see myself always as connected. To see my work not as amplifying and serving individuals or organizations (though I may be working with individuals and organizations), but as amplifying and serving connections, communities, wholes. That what I do (or don’t do) ripples outward, and to feel a sense of responsibility to that, but not as an individual burden, because I am connected. That’s terrifying to me because I don’t know if I’m up for it. I’ve spent a lot of time trying to be as self-sufficient as possible, to feel as safe as possible that I won’t inconveniently need something I can’t provide or acquire for myself, to avoid the terror and (and sometimes literal danger) of vulnerability. But I’ve only been able to do that by leaning into capitalism, ableism, settler colonialism, and white supremacy culture (and everything else inextricably tied up in all of that), which means lessening the terror and danger for myself, at least temporarily, but only by pushing it off onto others. There’s a curve of inequality that desperately needs to be flattened.

So the seeds I am planting now, in the upturned, upheaved soil of my life, are ones of kindness, connection, and trust. I can’t engineer the future, but I can fill the present moment with what matters most to me and know that it will ripple outward from there.

One way to keep things rippling out is to support fundraising and mutual aid efforts in your area if you are in a position to do so. Here are some of the ones I’ve been supporting. We are connected and we need each other.

Here’s the poem that Vanessa closed out our most recent evening of choir with, which spoke to my heart in perfect, bell-like tones:

Clearing

Do not try to save
the whole world
or do anything grandiose.
Instead, create
a clearing
in the dense forest
of your life
and wait there
patiently,
until the song
that is your life
falls into your own cupped hands
and you recognize and greet it.
Only then will you know
how to give yourself
to this world
so worth of rescue.

Martha Postlewaite

I promised Jara I wouldn’t overthink this one and just get this off Twitter and into a blog post, and I’m really trying hard to live up to that. (I might have overthought “not overthinking it” though.) I also know that with how fast things are moving now, there’s a balance to be struck between deliberateness and irrelevance. The post I’m writing now isn’t the one I would have written last week, nor is it the one I’d write a month from now. It’s the one I’m writing in this moment, sitting on my couch in my apartment where I’ve been mostly alone for the two and a half weeks, since the global pandemic came crashing down on these shores.

It’s an interesting time to be an evaluator. There’s a real and present (and needed?) challenge to the relevance of our work. I was speaking to a friend the other day who is working with teams to develop logic models for their programs. I asked her if they were including COVID-19 in their logic models and she said yes, under the “assumptions” section (i.e., “things we assume to be true about our context that would affect our program). I asked her if they were including potential global disasters like COVID-19 but whose manifestations are as yet unknown. She said no, they were just trying to wrap their heads around what was already happening.

Honestly, can’t blame them. I’m trying to do that too. Program logic models are about diagramming knowns, not unknowns. They are representations of what we believe is happening and why. But they hinge on the assumption that the relevant causality of our contexts is predictable, describable, and repeatable, and that’s just not always true. And, worse than untrue, it’s often not useful. Evaluating in complexity means evaluating in uncertainty. In a space where we fundamentally can’t be sure exactly what is happening, why, or what will happen next. Where knowing what has already happened does not give us predictive confidence over the future. This is the nature of complexity—irreducible uncertainty. Assumptions of certainty don’t apply.

Uncertainty can give rise to anxiety and fear, but also surprise and delight. We wouldn’t gather with friends if we knew exactly what was going to happen each time, what conversations would emerge, what unexpected jokes would make us laugh, what poignant turns there might be. We gather with friends to be immersed in the complexity of human dynamics, confident in our collective ability to navigate whatever happens in service of greater joy and connection. We look forward to what is unknown.

Certainty and uncertainty also came in our recent Eval Cafe episode with Nora Murphy Johnson and Andy Johnson of Inspire to Change, and Chris Corrigan of Harvest Moon Consultants. We talked about how evaluators are often hired and expected to bring certainty, but in actuality when we are at our best, we bring more uncertainty. We bring it with the question that we ask, with the invitation to look more deeply into what is being done, why, and to what effect. Our role then is not to eliminate the uncertainty, but to accompany people well as they move into and through it.

Why is it so hard though? That’s a question I ask a lot. I don’t believe it’s just that uncertainty and complexity is inherently a more difficult place to work in. It has challenges, but it also has a rhythm and principles. “Human beings are built for complexity” (a phrase I can attribute to Chris). We navigate complexity all the time—our lives are improvisations from the moment we wake up to the moment we fall asleep each day. We raise children. We form societies. We host gatherings. We learn and create languages. We make art. We are literally adapted for these things.

So why is it so hard to work this way? Why is it that folks who teach developmental evaluation (which is just a form of evaluating according to the principles of complexity) have to warn us (and truthfully) that we will need to constantly be helping our clients and stakeholders “stay the course”, not panic and retreat back into the familiarity of formative and summative evaluation? Why is it so hard for many of us (myself included) to even understand what developmental evaluation is and do it, consistently and coherently?

There isn’t a single answer to that question (beyond “it depends”, the ultimate single answer to any question), but there’s one part of the answer that has been on my mind lately, and it speaks to the idea that “injustice is rooted in uncertainty”, a hypothesis that Chris offered on the podcast.

Here’s what I tweeted about it (slightly edited for clarity and links added for reference):

Here is my take: the conversation about complexity is inextricable from the one about justice and equity.

The point on which my whole practice has turned lately is the understanding that the aversion to complexity is not just because complexity is challenging, but because settler colonialism and white supremacist culture reject complexity. Complexity isn’t just “hard” inherently. It’s specifically hard for particular ways of being and acting. It’s hard to consolidate power in complexity, because it requires diversity and collaboration, acceptance of partial and multiple truths, and openness to ongoing change.

Applications of complexity thinking aren’t inherently just. A complexity-informed approach can be used to colonize better, exploit better, oppress better. As @jdeancoffey pointed out, complexity work must intentionally serve equity, not just be assumed to, “And I would add @MQuinnP that we need to do DE in pursuit of equity, liberation and justice. We have to be for something.”

And still I see a resonance in the obstruction and devaluation of the ways of knowing that serve in complexity and the epistemic injustice and epistemicide levied at Indigenous ways of knowing, which also tend to be relational and contextual. Insistence on certainty at a level inappropriate to context is both irrational and, when combined with institutionalized inequity capable of enacting and sustaining harm through violence or neglect on specific groups of people, unjust.

In more practical terms what this means for me as an evaluator is that I have a responsibility to be competent both in certainty and uncertainty, discerning when each is appropriate, and supporting the people I work with in navigating both. Creating certainty is not my job. AND that ongoing questions at all times in my work are, “Whose way of knowing is being centered here? Why? Is it named or presented as the default? In what contexts does that way of knowing operate?” These are @equitableeval questions, and I need them to evaluate in complexity.

And from Jara in response:

I would add not only to evaluate in complexity but to understand complexity and the ways in which it may or may not be grounded in or be in service of equity, liberation and justice – probably need to add healing.

I also followed up with a response to myself that while it might be difficult to consolidate power in complexity, it’s not impossible to do so. Lots of bad things also operate in complexity. Racism, colonialism, exploitation capitalism, these are all things that have thrived and continued to thrive within the parameters of the complexity of human social organization. Complexity can be navigated to many different ends, hence the need for an explicit ethic or axiology of justice to inform where we are headed and why and how. As Jara said, we have to be for something.

Human beings aren’t innately good or bad, we’re just human. We’re self-determining and we exist in contexts that influence us and are influenced by us. If it were just a case of deciding we want to work in complexity and skilling up in how to do so, it wouldn’t be so hard. There’s a paradigm shift involved, but it’s doable. There are complementary new practices and techniques to learn, but they’re available. Any muscle takes time and effort to be strengthened, every craft can be practiced and honed. But it’s harder than it needs to be because we’re also reinforced in so many systemic ways not to work in complexity, every time there is a demand for certainty that’s disconnected with the actual context being operated in.

There are RFPs and funder expectations that equate to “buying success” because we only want to fund what is ‘guaranteed’ to work with a ‘proven track record’ (the language of certainty). The under-resourcing and over-burdening of the social sector where the demands for responsibility and accountability are not remotely matched with the necessary support, leading to fear and aversion to risk and being punished for not delivering exactly what is asked for. The ways that the epistemologies of certainty and their accompanying methods (e.g., randomized control trials) as accepted and positioned as more credible, valid, and valuable than epistemologies and methods of complexity and uncertainty (even in casual language like, “well ideally we’d have harder data on this, but for now this is what we’re seeing on the ground”—anyone obsessing over exponential charts lately knows that the most useful data is the data you have to work with and that numbers don’t bring certainty, just a different kind of perspective best complemented with other kinds of data, like stories about what other people and countries are doing to cope and manage in highly emergent circumstances). There’s a whole fantastic article from Tanya Beer about all the institutionalized impediments to working in complexity within evaluation itself and within funding organizations. On a grander scale, there’s straight-up capitalism (economic stability, remember?). And white supremacy culture. These are all barriers to just and equitable evaluation as much as to complex evaluation.

The argument here isn’t that complexity and complexity-informed approaches are superior or universally appropriate. I don’t want a complexity-informed approach to making a vaccine. I want the vaccine figured out according to the known good practices we have in that space because that’s a context where certainty, predictability, and established practices apply. And I want to know that we are prepared to have adaptive, responsive, uncertainty-appropriate practices for how we’re going to keep ourselves together as a species over the coming months, which is a very different situation to grapple with. And I want that adaptivity and responsivity to be consciously, explicitly, and fundamentally in service of justice and equity.

And for my fellow white and settler evaluators (and anyone else who wants to), for us to keep asking, “What and whose ways of knowing are being reflected in this evaluation work? Why? Whose values and worldviews are determining what ‘counts’ as valid knowledge? And appropriate ways of generating and using that knowledge?”

And for those of us operating in complexity, to be asking those questions as well, and also what we are navigating complexity in service of, and how does that show up in our work?

And for us to look to the work that has been and is being done particularly by people of colour and Indigenous evaluators working in community, in relationship, and in complexity (and who are persistently erased in the evaluation community, particularly when they are women, as has been thoroughly researched and described by Vidhya Shanker and referenced in this recent AEA365 post), not to invent or reinvent evaluating in complexity but to see how it is already being done, listen, and learn.