cplysy

Have you ever had that experience of being really seen? Paid attention to in that deep way where the other person notices things about you that no one else ever seems to, maybe even sees things in you that you didn’t know were there, but now you see them too? Maybe with a therapist, or a romantic partner, or a really sensitive, observant friend or family member? There’s something tingly and terrifying about being seen that way, but also deeply satisfying and rewarding. The pay-off of that vulnerability is intimacy.

When we talk about evaluation being an intimate experience (something I’ve heard several people mention over the last few months and experienced myself), that’s what I think about. Really, evaluation is saying, “I want to see all of you. The parts you love, the parts that make you tremble, the parts you don’t even know are there. I want us to look at them together, and I want to know you inside and out.” When we evaluate, we’re asking the people we work with to be completely open and vulnerable with us about things that matter a lot to them, to lay it all out there for close observation and follow-up commentary and judgement, often for a paying audience. Evaluators are there to see you, to help you see yourself, and help others see you too. Terrifying. And thrilling, in the right context.

I’ve had the chance to talk about intimacy a lot lately with a friend of mine, Erin Clark. Erin is a writer, performer, athlete, and all-around sparkly, amazing person. She has the gift of being an elegant and soulful thinker and you can find out more about how she’s been navigating intimacy in her own life by reading her digital memoir, Love All The Way (not entirely safe for work, I’ll warn you now), and checking out her recent guest appearance on the Free Her Spirit podcast.

When it comes to intimacy, one of the things Erin talks about in the podcast is how people are constantly forcing intimacies on her when they see that she uses a wheelchair. She gives an example of an exchange where a woman in an airport abruptly transitions from asking about outlets to charge her phone to probing Erin about why she’s in the wheelchair and whether she’s experienced some kind of trauma. Here’s what Erin said about this kind of interaction:

“It’s a very loaded and violating exchange that happens so frequently it becomes mundane, which is very weird. … A lot of the heartbreak I experience, or the struggle for me in having a disability, comes down to intimacies being forced on me so frequently that it shuts down my ability to feel intimacies, so that strangers are having conversations with me that only lovers should, and people are carrying me or touching me or taking me over in ways that only people I’ve developed that trust with, who’ve earned that, should. But because it feels so casual to people, so right to them, so, you know, allowed and permissed, that there aren’t a lot of ways for me to maintain those boundaries and protect my chosen intimacies that don’t involve being shut down completely.”

(You should definitely listen to the whole episode, or read the transcript, because Erin has magical things to say about being a sex icon, a world-champion pole-dancer and a paragliding pilot in Spain, and her insights into experiencing life through risk, desire, and intuition. Erin is my #LivingInComplexity icon. You can find out more about her by following her on Instagram, checking out her website, and also by reading her soon-to-be-published memoir about love, sex, risk, family, intimacy, travel, adventure, self-discovery, and so much more. I’m halfway through reading a draft of it and it’s already made me cry a lot, write poetry, and dream up a list of fabulous risks I can take.)

The context that Erin is talking about isn’t the same as program evaluation, but it still got me thinking. Evaluation is an intimate experience, so there are implications for how little autonomy organizations and communities have to enter into it authentically. Even for evaluations that aren’t explicitly mandated or when participation in evaluation processes is meant to be consent-based, there can be a sense of, “Well, we need to do it because it’s what’s expected. We don’t really have a choice. It’s not up to us. This is how the system works.” And the system works that way because of assumptions made about how to manage social services and distribute funding and resources—that accountability and efficacy of services and policies comes from externally-imposed ‘objective’ scrutiny and surveillance (whether from funders to programs, or service providers to service users), and not the fostering of authentic, mutual relationships founded on trust and respect. (Even though a lot of us working in this area know that the people closest to and most invested in the thing being evaluated are the best-positioned for informed and insightful critique of it and the most motivated to hold it to the highest standard of meaningful impact.)

Real intimacy isn’t possible when you aren’t allowed to have control over your boundaries and your privacy. Real trust doesn’t grow in a culture of mistrust. The most accountability is demanded from those with the least structural power, dragged upward instead of flowing downward, and evaluation is under constant pressure to be co-opted into maintaining this arrangement of who is scrutinized and who does the scrutinizing, for which the quality and meaning of our work suffers. Vulnerability is a gift, and gifts have to be given, not taken. Like Erin says, we are supposed to earn our intimacies, not assume we are entitled them, and we do all kinds of harm when we do.

It’s a bad system, but there’s promise here too. When we do find ways to foster the conditions for meaningful, enthusiastic, consensual participation in evaluation, when we collectively resist, disrupt, and subvert the power dynamics that rear up in so many places throughout the process, when we make spaces for each other and ourselves to “heal from the trauma of being judged” (thank you forever for that phrase, Chris—it’s definitely going on a t-shirt), then we can participate in the magic of intimacy in evaluation. We can let down our guards and set our masks aside for a little while (that means us too, fellow evaluators), and share our scars and fears, hopes and delights, questions and insights, and dive into discovery together.

Blogging is hard. I’m not sure why I find it such a struggle, though I know I’m not the only one who does. I marvel at the folks who seem to be able to write quickly, eloquently, and insightfully (I can usually manage 1-2 at a time but rarely all three). But I’m choosing to assume that blogging ability is a learnable skill so I keep practicing and looking for ways to make it easier on myself.

One thing I’m trying is to look for synergies between blogging and other creative work that I find easier, to see if I can borrow some of that momentum and inspiration. In this case, that’s the podcast I’ve been producing with my good friend and co-conspirator Brian Hoessler (of Strong Roots Consulting) for the last couple of years. (If you haven’t checked it out yet, the podcast is called Eval Cafe and it’s semi-regular series of casual conversations about things we find interesting in evaluation, frequently with guests! To quasi-quote our introduction, “it’s the kind of thing you might overhear in your local coffee shop if your local coffee shop were frequented by evaluators”.)

(Here is some fun tangential trivia about me and Brian: we have no idea how we met each other. It was definitely in Saskatoon and probably circa 2012-2013, but other than that, neither of us can remember a specific point in time of meeting. We just appeared in each other’s lives as buddies, and the rest is history! There was a coffee shop hang-out early on though.)

Working on a podcast is all kinds of fun and delightful and it gives me a really good excuse to randomly reach out to interesting people who are doing and saying cool things and invite them onto the show. A perfect example of that is back in January I read an AMAZING article on Better Evaluation LINK, called, “What does it mean to ‘un-box’ evaluation?”, by Jade Maloney, an Australia-based consultant doing evaluation and design work in the disability sector. The article is part of the lead-up for the 2019 Australian Evaluation Society conference and it explores their conference theme of ‘Evaluation Un-boxed’, which is officially my favourite conference theme ever (and there have been some good ones lately!). I loved the article so much that Brian and I reached out to Jade right away and recorded a fabulous episode where we all dug into that theme even more.

Definitely go and check out the article as well as the write-up of the theme on conference website and our podcast episode, ‘What’s in the mystery box?’, while you’re at it (don’t forget to scroll down to see what we link in the show notes as well). And also the follow-up article that Jade wrote after we recorded the episode!

We covered a lot of ground in that conversation, but there was one part that I’ve really wanted to go back to because it spoke to something that’s been on my mind quite a lot lately, which is the idea that evaluation is (or can be) a gift.

Here’s what Jade wrote in her first article:

As evaluators, we see evaluation as a gift. We see evaluation’s potential to support effective policy and program design, guide ongoing program development, provide insight into on-the-ground practice, and identify whether intended (and any unintended) outcomes are being realised. We see how evaluation can support better public policy and, thereby, better individual, social and environmental outcomes.

I was so excited to see Jade writing about this, because a few weeks earlier I’d scribbled some notes in my reflection journal along the same lines and was glad to see I wasn’t alone in thinking about it. This is what I wrote and shared again on the podcast:

If I believe evaluation expertise is a gift that can be given, that has three implications for my practice: 1) I have something of value to give. Thinking of something as a gift means assuming it is not worthless. 2) Good gifts are thoughtful and personalized. They are selected and given based on a judgement call that they will be meaningful, useful, and valuable to the recipient. They are also presented in a way that emphasizes the assumption of their value. Bad gifts are generic, impersonal, and presented without thought or attention to how they will be received. 3) While you can prepare a gift to be received well, you cannot assume control over what happens with it once given. It is up to the recipient to decide what it means to them and what they will do with it.

It was that last thought there that really caught me—this recognition that while we might try to influence it, we can’t control how people receive what we do and that’s the way it’s supposed to be. I may work really hard for an evaluation (design or process or deliverable) to be good and take a lot of pride and satisfaction in doing what I do well, and I might do everything I can for it to be useful and used well, but gifts are meant to be given, and giving something means letting go of it. I don’t know that I felt especially possessive of my evaluation work or projects before now, but I definitely didn’t think of them with that degree of… I’m grasping for a word here and the closest one to me at the moment is “liberation”.

There’s an ephemerality in doing evaluation work as an external consultant, which I’d noticed but hadn’t thought through all the implications of. Paying attention to it now is shifting my thinking in subtle ways. It makes me feel even more strongly about the centrality of relationships and collaboration in the work. It nudges my thinking about responsibility, what I am responsible and not responsible for, and reminds me that the people I’m working with are partners who share the responsibility for how evaluation is used, rather than it being my burden to carry alone. It moves me further away from that expert-driven model. It also reinforces the importance of asking, “Who will be using these findings? And how?”, to make sure the gift is given widely and to those who will use it well and be generous themselves. Acknowledging that I don’t control what is done with a gift doesn’t erase my responsibilities in giving it. If anything it heightens them.

I can’t control how evaluation is used, but I can make damn sure that if I give a gift, I’m giving it well, giving it in a way that reflects the value it’s meant to have. If bad gifts are generic, impersonal and presented thoughtlessly, then good gifts are meaningful, personalized, and presented with care and attention. That doesn’t always mean fanfare and fireworks and big glitzy reports that are designed up the wazoo. Some of the best gifts I’ve received have been ones that were just handed to me, a book with a special inscription inside that made it clear it was chosen for me with love and intention or a something small given with warmth and a smile. The key is that it’s a gift that you know is meant for you with thought to what will make it meaningful and useful for you, in what it is and how it’s given. In evaluation we already know that this is important. It’s the reason we care about utilization-focused evaluation and have embraced data visualization and effective reporting strategies. This is just a further reminder for me to keep it personal and personalized. Heck, somebody out there might even want and need the 400-page wordy tome of a report (after all, some of us like socks for Christmas!), though I’m probably not the person to give that particular gift.

So if evaluation can be a gift, if it can be given in a way that is meaningful and useful to someone, then that means it has value and it’s okay to think of it that way. I worry about that sometimes. I ask myself, “Do I really have something to offer here? Is this helpful? Is it doing anything good?” Sometimes evaluation feels like a burden, a challenge, a distraction, a threat, a shadow, or just a very tall mountain to climb. And it can be all of those things too, but it’s still a gift. It’s the gift of permission, time, space, and a way to ask just those kinds of questions—“Is this helping? Is this good?”—and learn something about that.

Just writing this and thinking about evaluation as a valuable gift that deserves to be treated that way in how we do it brings me right back around again to that question of, “Who gets given this gift? Who is it for? Personalized and thoughtfully chosen and presented for whom?” Any time there is distribution of a valuable resource, it becomes important to ask about and examine the equity of that distribution, and these are questions I’m pushing myself to always ask.

I’m happy to report that this is one of the faster blog posts I’ve written. Yay for building on podcast inspiration! And with 19-going-on-20 episodes posted, maybe I’ll even make a regular blogger out of myself yet. (Do not hold me to this.)

Speaking of podcasts (and gifts), here are some of the ones I’m finding the most enlivening and enlightening right now, and maybe you’ll enjoy them too!

• MEDIA INDIGENA – Rick Harp leads a vibrant weekly roundtable on current affairs from Indigenous perspectives

• Secret Feminist Agenda – Hannah McGregor furthers a nefarious feminist agenda with a host of fabulous guests

• The Good Ancestor Podcast – Layla Saad connects with culture-shapers and change-makers to find out what makes a good ancestor

• Living Myth – Michael Meade draws links between current affairs in troubling times with timeless, mythic stories

There’s a really great device that Michael Quinn Patton offers for use in developmental evaluation called ‘sensitizing concepts’. He’s borrowed it from qualitative research methods as a way of providing guidance to inquiry in complexity. Here’s a definition he gives in his qualitative methods book that came out a few years ago:

“Sensitizing concepts are terms, phrases, labels, and constructs that invite inquiry into what they mean to people in the setting(s) being studied. … Qualitative inquiry using sensitizing concepts leaves terms purposefully undefined to find out what they mean to people in a setting. Sensitizing concepts are windows into a group’s worldview.”

Elsewhere he elaborates:

“The observer moves between the sensitizing concept and the real world of social experience, giving shape and substance to the concept and elaborating the conceptual framework with varied manifestations of the concept. Such an approach recognizes that although the specific manifestations of social phenomena vary by time, space, and circumstance, the sensitizing concept is a container for capturing, holding, and examining these manifestations to better understand patterns and implications.” (from “Process Use as a Usefulism”, in New Directions for Evaluation, issue 116, 2007)

So if I was studying evaluation or evaluators with this technique in mind, some of the sensitizing concepts I might unearth could be “use”, “accountability”, “learning”, and “stakeholders”. These are all terms and concepts that evaluators use a lot in shaping and describing our work, but we don’t always define them or agree on definitions of them and the act of exploring our definitions and what they mean to us in a given context can be very enlightening as to our underlying assumptions and values. The point of a sensitizing concept is not to nail down exactly what it means but to use it as a jumping-off point for inquiry, a flexible container to give some shape and direction to our learning process. A sensitizing concepts points to something and says, “Whoa, hey, there’s something going on here. This is important to the people involved in this. Watch this space.”

(Another similar device I’ve come across is Arnold Mindell’s “quantum flirts”, or signals and insights thrown at us by the universe as something we should pay attention to, described here by Kate Sutherland. I gravitate a little more toward the sensitizing concepts framing, but it’s useful to have different ways of engaging with and thinking about this idea.)

True to form, I can’t resist applying sensitizing concepts in my own life. I’ve been noticing all those words, terms, ideas, and concepts that keep popping up in my conversations and my field of awareness, like the non-musical equivalent of an earworm. Lately I’ve taken to putting them on post-its as I notice them so that I can spend more time in active reflection with them. Some of the ones on my wall right now are “gifts”, “boundaries”, “habits”, “abundance”, and “comfort/discomfort”.

That last one has been on my mind a lot lately. I started noticing it well over a year ago, coming out of an evaluation I was working on where some curiously-contrasting findings were emerging. One set of findings was about how much the participants (English-language learning older adults) valued how the program was relaxing and low stress to be in. The other set of findings was about how some of the actual outcomes of the program (social connections, language learning) were occurring really strongly around one of the most stressful and un-relaxing parts of the program (public performances). These findings weren’t in conflict though. There wasn’t a split among the participants themselves and it wasn’t a case of participants wanting one thing and the program implementation pushing something else. Rather the facilitators were creating classroom experiences that were comfortable, low stress, and welcoming while also providing opportunities for participants to take on different levels of challenge at their own pace. The participants, while enjoying the relaxed atmosphere where nobody had to be an expert, would then often set their own standards for achievement, asking for more practices and rehearsals and setting challenges for themselves like memorizing their scripts. The performances were still stressful, but rehearsing and overcoming the difficulty together helped the participants bond and gain confidence in their ability to speak without being perfect. The comfortable and uncomfortable aspects of the program worked together.

Not an earth-shattering conclusion, actually! It seems obvious in retrospect and in keeping with theory around group cohesion and principles of adult learning. It’s also a good example of emergent program design, since we never set out to make performances part of the program nor would we have likely induced the same people to participate by advertising it that way. And a great reminder of the power of moving from questions like, “How do we create X outcome?” to ones like, “What conditions tend to support the emergence of X outcome and why?”, since an answer such as, “Have people do public performances to enhance social bonds and language learning”, speaks to what you’re trying to do but, “Create environments where people feel comfortable and safe and then give them opportunities to step into discomfort and challenge on their own terms”, tells you how to get there (since what mattered was not the performances themselves so much as how they were experienced).

It also speaks to the flexibility and utility of a principle over a rule, which Michael Quinn Patton distinguishes between in his Principles-Focused Evaluation, where a rule tells you exactly what to do in a specific situation while a principle gives you less specific but still guiding advice that can be adapted across many different contexts and situations. It’s this quality of a principle that I’ve found with the comfort/discomfort concept and the way it keeps popping up to guide my practice and self-learning. I’ve started seeing it everywhere, this use of comfort/discomfort as a managed experience for learning and change, either by creating a space that has proportional elements of both or an iterative process of moving between comfort and discomfort cyclically as part of the learning process. I see it in any theory that talks about “optimal zones” for learning/performing, like the inverted-U of the Yerkes-Dodson theory or Vygotsky’s zone of proximal development. I see it in eco-cycle planning and the chaordic path. There was a conversation on the Art of Hosting listserv about cognitive load theory in instructional design and it came up there as well in reference to comfort zones and balancing cognitive load. I noticed it in my own guiding principles of “be kind” and “be curious” (where kindness tells me to get comfortable and curiosity tells me to get uncomfortable by leaving the known in order to open up to the unknown). It turns up in a somewhat different way in activist spheres through ideas like, “Art should comfort the disturbed and disturb the comfortable” (which can be attributed to activist and poet Cesar A. Cruz and not Banksy, though Cruz was also possibly riffing off the original satirical version about newspapers).

Having tuned into comfort/discomfort as a generative conceptual space for me with regard to learning and change, I can use it intentionally as a way to explore that deeper. I can ask myself questions around it, both in planning (“how can I use this concept in the design of learning experiences for myself and others?”) and in reflection (“what am I learning about this concept through interacting with it?”).

When it comes to my own learning process, I can ask myself:

Am I comfortable or uncomfortable? What am I comfortable with? What am I uncomfortable with? What does “comfort” look like for me? And “discomfort”? What’s the relative balance of these two experiences in my life? Do I need more comfort? What would help me feel more comfortable? What opportunities can I give myself to be uncomfortable? In what ways am I moving between states of comfort and discomfort? What is shaping and directing this movement? How is this impacting the quality of my learning experiences?

And when am I working with others, I can think about:

Who is comfortable? Who is uncomfortable? Why? What are they comfortable (or uncomfortable) with? What might lend itself to more comfort? What opportunities are available for them to be uncomfortable? What opportunities can I offer for people to engage with discomfort? How can I create spaces where people can manage their own comfort levels? In what ways are people moving between comfort and discomfort, and to what ends? How is that interacting with the learning experience?

As I spend more time playing with this idea, my understanding of it will get deeper and more sophisticated (at least, that’s the hope) and it gives me a way of organizing a lot of incoming data I’m receiving, a way to focus in on “How are these things connecting or not connecting with this sensitizing concept? What are the patterns and themes? What doesn’t hang on this concept entirely or at all and needs something else?” Or I may move beyond it entirely (stop being “sensitized” by it) as I encounter more useful and engaging concepts that take my learning to another level.

(And if you’re wondering, “But, Carolyn, how do you avoid getting so caught up in these ideas that you force them onto situations where they don’t fit or see patterns and connections that aren’t really there? How do you know it’s not just all in your head?”, then I applaud your critical questioning and offer that this is why working in collaborative, interdependent ways is so important because by doing things like bringing these ideas up in conversation, like putting them in a blog post where other people can interact with them, ask questions, and offer their own insights, they can be tested and built on and strengthened and discarded through discussion. So in a more formal evaluative process, these concepts would be identified and explored within the group of people engaged in the evaluation, not by one person alone.)