cplysy

If you listened to our recent episode of Eval Cafe with Michael Quinn Patton on principles-focused evaluation, you’ll remember him sharing his new favourite example of principles in action. It’s from the introductory article of a recent special issue of The American Statistician, which is all about moving beyond the use of p < 0.05 as the threshold for determining statistical significance. The article offers an impassioned explanation of why abandoning the entire concept of statistical significance is necessary and also outlines the beginnings of an alternative practice for valuing and interpreting statistical findings. The reason it showed up in the podcast is because the authors ground this new framework in principles, or flexible advice that can guide decisions and give direction, but must be adapted and interpreted in context. In comparison, p < 0.05 is a rule—it is applied the same way regardless of any contextual factors. (Check out the podcast and also Michael’s book, Principles-Focused Evaluation, to learn more about the implications of principles for evaluative work.) Specifically, the principles that the authors offer are, “Accept uncertainty. Be thoughtful, open, and modest” (or “ATOM”, as a mnemonic), and the remainder of the issue (43 articles worth!) goes on to offer more depth around the issues of p < 0.05 and the discussion of alternatives.

For an academic publication about statistics, it is, frankly, stirring. Read this:

“At times in this editorial and the papers you’ll hear deep dissonance, the echoes of ‘statistics wars’ still simmering today (Mayo 2018). At other times you’ll hear melodies wrapping in a rich counterpoint that may herald an increasingly harmonious new era of statistics. To us, these are all the sounds of statistical inference in the 21st century, the sounds of a world learning to venture beyond ‘p < 0.05.’ This is a world where researchers are free to treat ‘p = 0.051’ and ‘p = 0.049’ as not being categorically different, where authors no longer find themselves constrained to selectively publish their results based on a single magic number. … As we venture down this path, we will begin to see fewer false alarms, fewer overlooked discoveries, and the development of more customized statistical strategies. Researchers will be free to communicate all their findings in all their glorious uncertainty, knowing their work is to be judged by the quality and effective communication of their science, and not by their p-values.”

(Has a paper on inferential statistical testing ever brought tears to your eyes? This brought tears to mine. The freedom in it, the vision of it, the clarion call to a remembered sacred purpose of meaningful scientific discovery—it’s poetry. And, honestly, the whole article is a delight to read and it’s open access. Treat yourself!)

So what’s wrong with p < 0.05? Why devote an entire issue of an journal to explaining why it should be done away with?

The problem is that it’s a seductively simple idea that was never equipped to be used the way that it has been. We took something that might have been an okay guideline for thinking about whether to explore a statistical relationship further and turned it into something so hard-and-fast that careers are made and broken on it, that people are incentivized to do bad science because of it (whether that’s bad actors manipulating results or more subtly the cumulative, unintentional harm of something like the “file drawer problem”). We took p < 0.05 and applied it thoughtlessly, rigidly, imperiously, and with total disregard for context. To the point that the statisticians tell us that the solution to p < 0.05 is not to tweak it, to start using “p < 0.10” instead or confidence intervals or to come up with a more complicated system of rules that let us keep doing essentially the same thing as we always have but “better” this time. Instead they ask us to recognize that the entire concept of a fixed threshold for statistical inference is flawed and we must shift to a way of thinking that incentivizes nuance, humility, and care. It is a call to transformation, to a world beyond p < 0.05.

So why am I bringing this up, since this post isn’t actually about statistics? (Surprise!) Bear with me—we’re about to go on a bit of journey.

I bring up p < 0.05 and its critiques because I realized that to me it speaks to the same issues I have with sex and gender binaries*. We have taken what is a fluid, complex interplay based around complementary elements that are meant to be more like tent poles, lifting up a fabric of possibility that drapes around and between them, and we have stripped them of nuance and severed what connects them. We have sacrificed depth and breadth in our understanding and experience of sex and gender for convenience, control, and predictability, leaving ourselves with two bare stakes in the ground. Because just like with p < 0.05, there’s something seductive about the notion of a fundamental, highly predictive, nearly-universal binary division of sex and gender, something deeply appealing about the idea that this dichotomy is rooted in biology and manifested at all individual and sociocultural levels. We can see the evidence of the attraction in how often it shapes the most basic of our everyday activities—going to the washroom, putting on clothes, talking about other people, filling out forms, etc. All of which feeds back into the perception that these are innate, meaningful, nigh-universal differences, which is why it is so powerful to take note of where the contrasts, contradictions, and variations arise, in language, biology, psychology, culture, and elsewhere. And how these variations are not statistical noise and or mere outliers but vital parts of the whole picture.

Because I’m not saying there is no pattern to sex and gender. I’m saying that the pattern has been reduced, over-simplified, and blown completely out of proportion, with individual components being mistaken for the entire phenomenon, like a painting being the sum of the pots of paints used to produce it. Now imagine a world in which every piece of art was classified based on its relative proportion of red and blue hues just because those are two primary colours. Whole galleries divided into “red wings” and “blue wings”. Vast swathes of art from across the colour spectrum lumped together without regard to the rest of their palettes (and never mind all the other defining characteristics one could consider about them). New technology devised for the sole purpose of pinpointing with stunning accuracy the exact amount of red and blue present in a given piece. The stubbornly unclassifiable relegated to storage because it’s only a small proportion anyway and it just upsets what is otherwise such an elegant, simple binary scheme. Because establishing clear, bright lines around sex and gender categories does make life easier in many ways. But convenience comes with a cost (think about Amazon and Uber), usually a profound human cost that is disproportionately distributed along lines of power and influence although we each pay our own price for it.

Rachel Pollack, a science fiction and comic book author and trans woman, captures the heart of the struggle with in her recent essay, “Trans Central Station”, where she shares her experience of coming out as a trans woman in the 1970s (a time before the language of “transition”, “transgender”, and “trans woman” even existed in English):

“What I felt, what I desired was unspeakable because, for me, at least, the words did not exist. Or rather, the telling did not exist. … The mind could not form the thought. I did not wish to tell people and didn’t dare. I simply could not imagine doing it. … I was not trapped in the wrong body, I was trapped in the wrong universe. In order to become who I was, I had to break the world open. I had to embrace a kind of science fiction life. … The physical world may be made out of elementary particles (and dark matter) but the world of our lives is made out of language. With the wrong language, one of strict categories and confinement, the world becomes a fake, a stage set whose actors don’t know they are in a play … Most people do not notice this because their own sense of self, of language, more or less fits the received version of existence. They still suffer, for in a world of strict and very limited categories, they must constantly check themselves against the model of a ‘real man’ or a ‘real woman’. The ones who reveal the fake are the ones who simply cannot make themselves fit. To not fit can bring great pain and often very real danger, yet who else can discover the light behind the screen?”

As I shared when I wrote about my pronouns, the words to describe my gender (and my sex and the relationship between them and my relationship with them) don’t exist in my language. I borrow words like “queer” and “trans” and “nonbinary”, because they come as close as they can, but they are a finger pointing at the moon, not the moon itself. I suppose all of language is a finger pointing at the moon, but there are varying degrees of remembering and forgetting that the finger is not the moon. With p < 0.05, we forgot the moon existed at all, and I think we do the same with “male” and “female”, “man” and “woman”, the lonely tent poles that they are. Because I don’t think my gender is any more mysterious or complex than anyone else’s or any less coherent, save for the misfortune of being born in the wrong language, one that has the idea that these things are governed by rules, clear and delineated and precise. Sex and gender are messy, multifaceted, ill-defined, and exhilarating. They’re never going to boil down to simple rules so we should start thinking about what kinds of principles might help us here. We need to embrace the ambiguity rather than keep trying to erase it, because it belongs to all of us, not just those of us who simply cannot make ourselves fit.

I don’t have 43 articles to share of deep exploration into this issue and possible alternatives. But even if I did, our friends at The American Statistician still had to warn their readers, “What you will NOT find in this issue is one solution that majestically replaces the outsized role that statistical significance has come to play. The statistical community has not yet converged on a simple paradigm for the use of statistical inference in scientific research—and in fact it may never do so. A one-size-fits-all approach to statistical inference is an inappropriate expectation, even after the dust settles from our current remodeling of statistical practice (Tong 2019).” A one-(or-two)-size-fits-all approach to sex and gender is unlikely to be forthcoming either. I also don’t have principles to offer for a better understanding of sex and gender right now, though, “Accept uncertainty. Be thoughtful, open, and modest”, are pretty good ones to start with.

We’re going to keep wrestling with these issues, standing in the world of now while we look to a world beyond. (And that world is coming—you can watch it happen in the flux and growth of the language and the ideas that are becoming available to us.) One thing we need to do while we experience this uncertainty is to not try to negate it or reduce it or look for a newer, better p < 0.05 that will let us carry on as usual. It’s not about tacking on an extra category or two. It’s about letting go of the seductive simplicity altogether, and finding a way forward that allows for nuance and wholeness.

That’s what I find so inspiring about the p < 0.05 article. It’s a beautiful exploration of how (and why) to move from disastrous over-simplification to an adaptive embrace of complexity (and within an institution that is itself complex, with a life and momentum of its own that resists change out of an impulse for self-preservation which is understandable even as we recognize that to resist change is also an act of self-destruction through obsolescence**). It may contain no absolute answers (that would be too simple), but it has an abundance of hope, compassion, and courage, as well as a frank reckoning of the systemic, institutional challenges to such a profound shift. I want to see the same combination of depth, hope, and strategy brought to our conversations around sex and gender as well.

The transformation is out there. What starts as science fiction can become the art that life imitates.

Happy Pride, y’all.

*I’m referring to “sex and gender” throughout because while they can be thought of as different things, they are BOTH complex and BOTH socially-constructed. It’s not a case of “sex is simple, gender is complex”. If you want to know more on that, check out the readings I reference at the end of my pronoun post.

**I think I’m going to start summarizing this paradox as “change and/or die”.

BONUS:

Podcast episode recommendation! Check out the Indigiqueer episode of the All My Relations podcast for a look into gender and sexuality from Indigenous viewpoints.

Transformation is awkward.

I mean, we all know that change is hard, but it’s also awkward.

It’s been a big year for me. The last ten months in particular have included a lot of things I never saw coming and just got to run with as they came my way. I’ve made a lot of new friends. I’ve gotten to have some incredibly transformative learning experiences (the Evaluation for Social Change and Transformational Learning program through SFU and the annual Art of Hosting training on Bowen Island being two I can strongly recommend to others!). I’ve been graced with some amazing opportunities to step in and push myself to new levels. I’ve had to deal with some setbacks, disappointments, and missteps, though I contend that all my best mistakes are still ahead of me. (I can only aspire to be a future contributor to a follow-up volume to the wonderfully honest and generously insightful, Evaluation Failures: 22 Tales of Mistakes and Lessons Learned.)

When I think back to myself a year ago, I’m astounded. It’s not that I was doing so terribly before, but it’s such an unexpected, qualitative difference. In less than a year, I’ve transformed my practice. I’ve clarified (at least for now) my guiding principles. I’ve discovered personality traits I didn’t know I had. I’ve changed things I thought were immutable personality traits! (This is why it makes so much more sense to think of personality as a system, as an aside.) My social and professional networks (around here those are basically the same thing) quadrupled at least (I’d give my right arm for a time-lapse network analysis of this, truly). I grew out my dang hair even, which is a big deal! I had that buzzcut longer than I’d lived most places. And, weirdly, I’ve felt more like myself than I can recall for a very long time.

But it’s been uncomfortable. It’s been hard. The biggest, most joyful and breath-taking breakthroughs also brought on periods of grief and despair—grief over the loss of a perceived self, grief over feelings of lost time spent travelling down paths that led away from the core self, despair over how long it takes to find a way back again. I spent a lot of time pondering this visualization of artist block and reminding myself that I am improving all the time, it just doesn’t feel like it when I’m also increasing my capacity to see what else I could be doing.

And even now that I’m acclimating to the emotional waves of change and not getting knocked about quite so much, it’s still awkward. Ungainly. I don’t get to be coolly possessed and confident. My intuition will run ahead of my conscious understanding so I’ll end up feeling strongly about something without being able to articulate why, and then having to backtrack later and explain once I’ve figured it out. Or I’ll find myself entangled in the extinction burst of a past habit and at cross-purposes with myself. A moment of insight and clarity will take the impossible knot I’ve been wrestling with and dissolve it into irrelevance, and I’ll find myself wondering how I managed to waste so much time on something that was resolved so easily. I spend a lot of time feeling incompetent, then realizing that I’m way more competent than I think, and then feeling incompetent for thinking I’m incompetent. Argh! I feel inconsistent and inconstancy still reads as a character flaw or moral failing to me. (Another habit yet to be unpicked and re-stitched! Coherency is preferable to constancy, if nothing else.)

Change is just like that, whether it’s coming in drips, tides, or tsunamis. You don’t get to sail into it, smooth and suave, confident that you know exactly what you’re doing and how to do it. Because you don’t! That’s the point. If we knew what we were doing, it wouldn’t be change. But not knowing exactly what you’re doing but kind of maybe knowing what you’re supposed to be doing and possibly doing some of it wrong but going ahead doing it anyway and trying to pull it off with some level of grace and integrity and humility is kind of where it’s at. So thank goodness for having a really high tolerance for embarrassment (not a skill I’d planned to lean on so much in adulthood, oh well) and a good circle of folks to commiserate and celebrate with, awkwardness and all.

(This is an evaluation blog, or at least the blog of an evaluator on their professional evaluation website, so I always feel compelled to draw some kind of evaluation-specific connections or insights or lessons out of what I write about, but I feel like “transformation is awkward” speaks for itself, y’know? Let’s all go a little easier on ourselves.)

I tweeted a joke about making an adult colouring book with reflective questions on each page for evaluators and it found a VERY receptive audience, so….. I made a prototype! You can download the PDF here. I hope you all colour your way to some satisfying insights! Comment or tweet and tell me what you think, or show off your masterpieces for us all to enjoy. 😀

Shout-out to André Luiz (@andreluizgollo) for putting up some great repeating pattern icons on The Noun Project that I was able to turn into the artwork for this colouring book.

I can’t recall for certain, but I think the first person I “had coffee” with in a professional capacity was my friend Brian Hoessler, a fellow evaluation consultant and now my co-host on our evaluation-themed podcast, Eval Cafe. And of course the podcast is all about, as we say in our intro, “informal chats on evaluation-related topics. The kind you might overhear a your favourite coffee shop, if your favourite coffee shop was frequented by evaluators.” I guess we really set the tone with that first conversation!

When I moved to Vancouver, I didn’t know anyone here. I hadn’t done much networking outside of Saskatchewan and was mostly a quiet observer at the CES and AEA conferences I’d attended. Fortunately Brian had been more social and pointed me at Sarah Farina, so I sent her an email, stiffly titled, “looking for professional evaluation contacts in Vancouver”. We were sitting down for coffee the next week. I remember being struck by how easy it was, reaching out to a total stranger and within a week having a conversation about the local evaluation landscape and where a newcomer might look to get involved.

After that coffee, Sarah put me in touch with Diana Tindall who was looking for volunteers to help out with planning the 2015 CESBC one-day conference. Michelle Naimi was also on the planning committee and she and I went on to co-found the Student & Emerging Evaluator Network (SEEN), organizing events and opportunities for new evaluators in Vancouver, including a monthly meet-up at, you guessed it, a coffee shop. I’ve now worked with Diana and Michelle many times in various capacities and spent a lot of time in coffee shops with both of them.

Sensing a trend?

Somewhere along the line, the coffee shop connections have become a cornerstone of my professional practice. This year in particular, it’s gone from meeting up with someone once or twice a month to once or twice a week (or more!), according to the rich historical data source that is my Google calendar. Even my vacations become an opportunity to grab coffee with someone new (when I was down in San Francisco this April I got to spend a couple of hours talking about evaluation and coaching with Betsy Baum Block at a coffee shop in Berkeley—perfect vacation activity! Also pleasingly alliterative). Last week, I had four separate coffee engagements, including spending my whole Saturday in eight consecutive hours of conversation, starting in a coffee shop on Commercial Drive and ending up in another one in Hastings-Sunrise (with a three hour yarn-store-adventure-and-chat with my sister in between).

I meet with people I know well and see regularly, I meet with people I’ve just met or only talked to online. I have a few “standing” coffee dates and a lot that come up spontaneously. Sometimes I ask and sometimes I’m the person getting the email with a subject line like, “looking for professional evaluation contacts in Vancouver” (except less awkward than that). Sometimes the “coffee shop” is a zoom room, occasionally it’s a pub or a bar or someone’s front porch. Sometimes I’m close to home, other times I’m venturing out to New West or North Van or hopping on a ferry, island-bound. Rarely is coffee actually consumed, at least by me (too much caffeine), but the coffee is largely metaphorical anyway. It’s about the connections, and, increasingly, the conversations.

Conversation is at the heart of it for me. There’s definitely a conventional networking component to it as well and I’m always aware of how work opportunities can emerge unexpectedly through these connections (almost all of the work I do right now has come from a “somebody introduced me to somebody who introduced me to somebody” type of situation). But the conversations themselves are key.

One of the things I’ve learned about myself this year is that I think in conversation, in dialogue. I was surprised, because “reflective practice” always signified to me something quiet and peaceful—walking alone in nature, sitting quietly in meditation, or writing bent over a journal. But when someone asked me to describe my reflective practice I realized that it looks like pacing around my office or my apartment and talking to myself. Occasionally I will write things on post-it notes, stick them to the wall, and then carry on a conversation with the post-it notes. (This is a lifelong habit. Once I was talking to myself while out walking and gesticulated so much I accidentally hailed a cab.)

I wasn’t sure if this counted as “reflective practice” until someone shared a quote with me from the feminist economic geographers who co-published under the name, “J.K. Gibson-Graham”. The quote was:

“… our seldom-inspected common sense posits a separation—or even an opposition—between thought, understood as cerebral reflection, and action, understood as embodied engagement with the world. This makes it hard to see thinking itself as a kind of action—that we are doing thinking, in other words, touching the world and being touched by it and in the process things (and we) are changing.” (A Postcapitalist Politics)

It was a wonderful observation and broke open my assumptions about what did and did not count as reflection (or as action), and gave me a new way to appreciate why my particular reflective practice works for me—because it helps me break down the artificial distinction between my mental processes and my embodied experiences. For me things live differently when I speak them out loud. Who hasn’t had the experience of, “Wow, that sounded better in my head”? Putting it out there lets me engage with it, interact with it, disagree and argue with it, be inspired by it. To paraphrase the above, “to touch the world and be touched by it and, in the process, change”.

I think this is the core of any reflective practice, really. Whether it’s journalling or nature walks or quiet contemplation. All of these practices are meant to create opportunities for another kind of interaction, through a change of scenery, seeing words on a page, or finding the space inside to hear things in a different way.

Where do coffee shop conversations come into it again?

Complex adaptive systems thrive on a diversity of inputs, a source of the variations that allow for adaptive resilience and flexibility. Talking to myself (or my post-its, or my whiteboard, or the book I’m reading) is part of how I inject a little variety into my own system. I don’t know exactly what I’m going to say until I say it, or quite how it’s going to land with me until I hear it. Letting my harvest of post-its accumulate on the wall for months becomes a source of unexpected inspiration when I am reminded of something or see it in a new light on a different day.

But it only goes so far. I may contain multitudes but those multitudes have a lot in common with each other. And while I’m pretty good at taking myself by surprise (I used to play whole games of cribbage against myself when I was a kid and learned to “forget” what I knew and operate as if I didn’t hold all the cards), there’s nothing like the true mystery of interacting with another person. Even if we have a topic picked out in advance, I rarely go into these conversations with any particular goal beyond discovery and relationship-building, and I’m always surprised and delighted and stimulated by what happens and what comes up. And it’s usually, as far as I know, reciprocal. Good conversations are collaborative and co-creative. It matters who I’m talking with—different people mean different conversations.

It’s also resource of integration and embodiment for me. I’ve learned that it’s not just enough to write my thoughts down. The written word is a temporary holding space, an extension of my working memory. The true archive is me, my person and my personality. I don’t truly know something until I don’t have to hold it deliberately in my head and my hands anymore, but have it available to me in the moment that I need it. Conversations become a rehearsal space for my being—what do I say? What do I think about? What do I hear? What do I notice? What do I discover? What do I remember? Anyone who’s had a conversation with me lately probably heard me utter something like, “Oh, I was just talking with someone the other day about this, and…” at least ten times. Every conversation reinforces the associative network of embodied knowledge that I am weaving into myself. (And that’s not even counting the more traditional, “Oh, I just read this the other day…”, which has always been a constant.)

And on that metaphor, on the relationship-building side of it, it’s more than just fixing myself as a well-connected node in a network, it also feels like I’m weaving together and reinforcing the fabric of my community. I always want all the cool people I know to know each other and to share their brilliance with the world and a lot of these conversations result in contact referrals and introductions (both ways). If I had one frustration with the coffee shop conversations it’s that I’m always thinking at some point or another, “Oh, so-and-so should be here for this!”, and truly there isn’t a coffee shop big enough for what is effectively a friend-convention that I want to throw. I am trying to level up to dinner parties at least. (Also even as I’m writing this, I’m realizing, “Oh, this is so obviously the trajectory of the four-fold practice that is the basis of the Art of Hosting. Of course.” Also, this is why folks should check out The Art of Hosting. Tell them Carolyn sent you!!)

It wasn’t always this way for me. I’m not joking when I describe myself as “a shy, socially anxious introvert who drastically overcompensated”. Once when I was a teenager I managed to eek out five minutes of small talk with someone waiting at the same bus stop and it was such a remarkable and unprecedented accomplishment for me that I positively glowed all the way home. I truly believe that talking about the weather is humanity’s greatest invention (even greater than post-it notes). I don’t know what changed exactly. I think I just finally became more curious than I was shy. And it was easier than I thought it was going to be when I stopped trying to be so very smart about it and got humble.

The coffee is a metaphor and the coffee shop is too. The conversations have percolated* into nearly every facet of my work. The podcast has been both an invitation for conversation and a platform to share that conversation with a wider audience and have new conversations because of it. (And, let me tell you, there’s no better ice-breaker at a conference than, “Do you want to be on my podcast?”) Twitter has become an important space of conversation for me. Though conversing in tweets can be a little fragmented and chaotic, one of these 280-character flurries led to the forming of a co-operative inquiry group of evaluators (we’ve been meeting monthly to have generative conversations around the topic of “boundaries in evaluation”). This blog is also an attempt to be in conversation in another way, though I find it trickier and am trying to bring in more conversational elements to make it easier. The ability to be in good conversation is a core evaluative practice for me as well, from the big boardroom meetings to the casual hallway chats. In fact, especially the casual chats—that’s where the magic is.

It will be interesting to see how my coffee shop conversational practices shift (or don’t) as time goes on. Right now they feel absolutely integral, as much as any other professional development practice I’m engaged in. It was an absolutely giddying realization that “hanging out with cool people in coffee shops” is a totally valid and awesome part of my job description. Now, to remember to save the receipts.

(Also, if you think I use a lot of parenthetical asides while I’m writing, you should experience firsthand some of the exciting “thought journey” detours I go on in person, which a good 70-80% of the time will reconnect back with the original point, or at least end up in verdant pastures of inspiration in their own right, hopefully. Conversational off-roading.)

(It is possible that mostly talking to myself and therefore needing to hold up two sides of a conversation at once has instilled some terrible habits, I mean, interesting quirks!)

(I’m done now.)

*Yes, I forced the pun a little and, no, I don’t regret it.