In reply to Melanie Hunsaker.

As I get older and become more experienced, and after my conversations on boundaries and thinking about the relationships between evaluation and research, I’ve almost become a bit less inclined towards boundaries. In the beginning, it was nice to set boundaries: this evaluation, and that is research. Now, I see things as being a bit more amorphous.

This semester, I started using Perusall for all my assigned
readings in my evaluation course. In this course, students are often learning
for the first time what evaluation is while at the same time working with a
local organization to design an evaluation for them. There is a lot of ground
to cover while simultaneously giving them enough time to work with their
clients.

What is Perusall?

Perusall is a collaborative
reading application. Students and instructors read articles or websites or
watch videos on the app and can comment chunks of text or time stamps in the
video. These comments—or annotations, as they’re called on Perusall—can have
simple text formatting, code snippets, emojis, images, links, and videos. Then other
students or instructors can reply to that comment to make it a thread or
conversation. Comments can be upvoted, and annotations that are questions are
automatically recognized and other students can indicate they’d like the answer
to this question too. In that way, the scoring is a bit “gamified” to encourage
high quality annotations that others find useful or insightful.

Helpfully, the Perusall platform is free for students,
instructors, and educational institutions. Although I do not use a textbook for
this course, students can purchase their textbooks through Perusall. This would
be problematic at our institution since we do textbooks via rentals to students
to balance textbook costs across students. However, I do not plan on using this
platform (yet) with my undergraduate students and I currently do not use
textbooks with my graduate students.

Why use Perusall?

After seeing a demonstration of Perusall by Eric Mazur, I
was hooked. This made reading fun, engaging, and not so lonely of a process. During
my first year of graduate school I did not quite know how to read effectively,
and I imagine most of my students feel much the same way.  Perusall helps students learn to read complicated
texts together. They ask questions, they make comments, and I can monitor and
contribute to the conversation to keep everyone on track.

As a result, this year’s cohort seems to have grasped what
evaluation is far quicker than last year’s cohort. To be fair, there could be many
other possible explanations; there are some differences in the make-up of the
cohorts or perhaps my teaching has improved from last year to this year.
However, personally I think Perusall has made a world of a difference. Here are
some benefits that I have noticed:

1. I can save time in class by focusing on
   activities applying what they learned in the readings rather than
   discussing the readings themselves. We sometimes go over the readings in class,
   but this has freed me up to focus on other things which has also helped them
   grasp evaluation much more quickly. However, note that this will take them more
   time out of class and to plan the workload accordingly.
2. Students can read at any time but still
   benefit from reading “with” peers and me. I encourage students to check
   into Perusall once a day to catch up on the latest conversation. They also get
   notified when someone tags them or responds to their conversation. As a result,
   this does not become a task that students just read and complete. It also leads
   to more conversation than the typical LMS discussion board provides. If this
   were an entirely asynchronous course, this would be highly beneficial to
   promote student-to-student and student-to-instructor interaction.  
3. The conversations we are having are deeper
   than most conversations I ever experienced in class-wide or small-group discussions
   during graduate school. Even if the entire class period gave time to have a
   discussion, it was never enough to really get to everything. This lets us
   review and discuss every piece of the reading: not only the big major take-aways
   but also some of the nitty gritty or fun tidbits.
4. Although I can guide the conversation,
   students can also take leadership of the conversation. Sometimes I will start
   conversations to get students thinking about the major take-aways of the
   articles. However, I’ve done this less and less as the semester has progressed
   because I’ve found they are grasping the take-aways themselves better over
   time. They can also introduce new and exciting conversations that I had not
   even thought of myself!  

What are the potential downsides?

I think there are a few, and some that are specific to my particular
situation, but overall I think the benefits above outweigh the costs.

First, this is going to take students more time to complete.
Average reading time per article ranges from 30-90 minutes per student, and I assign
multiple articles a week. I had to adjust my out-of-class activities
accordingly. However, I also teach another course these students take and, except
for the first few weeks, there is far less reading in that other course.
However, I think the benefits of understanding the reading more deeply outweigh
the costs here.

Second, it adds another thing students must learn. Currently
students have to learn our LMS, Microsoft teams (for online class meetings),
and now also Perusall. That can be an added challenge. However, I think the
benefits of a platform that provides a better reading experience outweigh the
costs here.

Third, my institution will not enable the integration into
our LMS. My hope is if more faculty begin adopting this program, they will
investigate an integration in the future, but for now this becomes a separate
website that students have to check regularly. Part of the issue is that
Perusall will not meet individually with institutions to ensure accessibility
regulations are being met; instead, they provide a website with detailed information
because they are faculty-run and supported and are a free platform.

How can I get started?

I recommend watching a previously recorded webinar or signing
up for an upcoming webinar here. If
you’d like to read research about using Perusall, check out their list of
research here. Otherwise, create an
account, create a course, add your readings and students, and get started!
There is plenty of support
content to help you get started and manage Perusall.

In reply to Jimmy.

I personally don’t. The education plan is free, and so my students are able to also create free accounts. The teams pricing looks like it is $10 per member per month at the lower tier, and I wouldn’t want to ask my students to pay for that.

Here’s from Teams: “The free Personal Plan is designed for individual use, but allows up to 5 unique guests. These are friends, family members, and others you can invite to collaborate with you privately on any page. You can also share a page publicly and turn on commenting or editing access.”

I don’t know if it’s because I have the Education plan, but I’m at the “Personal Pro” level and have unlimited guests, so I don’t think this will ever become an issue unless they begin charging education accounts.

This blog post was originally posted in the American Evaluation Association July 2020 newsletter. Thank you to AEA for asking me to reflect on what the AEA values mean to me and how they guide my work.

This summer has been a time of reflection due to physical distancing, transitioning to remote teaching, and the growth in the national visibility of and support for the movement for Black lives. This time of reflection was a pause on “walking the talk” because I first had to grapple with what walking and talking meant to me personally. I had to sit down and think through what my personal values were and how I would embody them in my everyday life. 

I have never explicitly stated or evaluated my personal values because I merely adopted the ones our white dominant society presented to me: civility, perfectionism, a sense of urgency, defensiveness, dichotomous thinking, paternalism, etc. As a white, heterosexual, cisgendered woman with a privileged academic background and job, it was easy to accept these as-is without reflection. It was comfortable. It was white supremacy. 

The first challenge to my personal values occurred during my job talk for my now position at UW-Stout. A graduate student asked me, “How does social justice affect your evaluation practice?” My initial gut reaction was: it doesn’t. And sadly, that was true. I never thought explicitly about social justice beyond the idea that there was a fourth branch on Mertens and Wilson’s (2019) adaptation of the evaluation theory tree. That began a quest to bolster—and in some ways challenge—my formal education. I began reading, discussing these topics with friends and colleagues, and slowly adding what I learned to what and how I taught evaluation to students. 

A year and a half later, George Floyd was murdered by police, and I left the comfort of my home to go protest in the little rural town I live in. I had always supported the Black Lives Matter movement, but never had I truly walked the talk until I stood at the corner of our biggest intersection with my sign. I began to realize how much easier it was to read books on antiracism than to practice antiracism. 

After attending a couple demonstrations, I wanted to continue walking the talk. I wanted to help change the world and make a difference! I dove head-first into new projects and ideas, wrote and disseminated things, and realized quickly that I had re-submerged into the pool of the white dominant society’s values, not my own. I felt the urgency to get things done, to do things big, perfectly, and mostly by myself. I quickly became defensive when people who cared pointed out that my actions were the characteristics of white supremacy in action. 

This was the juncture at which I realized I needed to pause, slow down, and reflect. So instead of walking the talk lately, I’ve been figuring out and explicitly determining my set of personal values, seeking inspiration alignment with scholars, friends, and colleagues, primarily those who are Black, indigenous, or people of color. 

Some values were easy to determine, like valuing openness and transparency, which I embody through open science practices, sharing resources and research through my blog, and being honest with my students about what I am teaching and why. Other values have been far more difficult, presumably because they more directly challenge the white supremacist culture ideology I had internalized for so long. For example, the work of the Equitable Evaluation Initiative has emphasized to me that it is not enough to have diversity and inclusion if equity is not front and center. 

As I continue to reflect on my values, I look in part to AEA’s values. There are some values that resonate deeply with me, such as having evaluations that are  “high quality, ethically defensible, and culturally responsible”. There are others that I would modify, such as valuing equity in addition to diversity and inclusion. And there are yet others that I would add, such as valuing advocacy efforts on behalf of the evaluation field (AEA Professional Practice Competency 1.9). 

More than anything, I am realizing that “walking the talk” and truly living out AEA’s values is an enduring process that is constantly evolving as I pause, slow down, reflect, and challenge my role in white supremacist culture.

Background

On May 25, 2020, George Floyd was killed by police officers.
This news is not new. George’s cries of “I can’t breathe” echo the same cries
by Eric Garner, who was also killed by police officers on July 17, 2014. Black,
indigenous, and people of color (BIPOC) are all disproportionately more likely to be stopped,
arrested, incarcerated, and killed by police than White people: indigenous
people are 3.5 times more likely and Black people are 3 times more likely to
be killed by police than White people.

The problem is not just with police or prisons. Segregation still exists, with both racially homogeneous schools and neighborhoods. Black and Latinx families are less likely to live in owner-occupied housing compared with White families. Voter suppression disproportionately affects BIPOC. The employment rate is around 15% lower for Black people than for White people. The end of affirmative action policies across states and colleges have led to significant declines in the number of BIPOC at large state schools. Black and Brown students are much less likely to graduate college with a degree compared to White and Asian students. People of color have poorer health outcomes (e.g., diabetes, heart disease, worse life expectancy) and are more likely to die from Covid-19. The list goes on.

I bring up this information to show the racial disparities
present in a variety of areas: health, education, voting, policing and incarceration,
housing, and more. These areas are many of those that we evaluators study.
And those programs and policies disproportionately
serve BIPOC. When a white person evaluates a program serving BIPOC, then
there is even greater need for a culturally responsive evaluation approach.

Culturally Responsive Evaluation

In 2011 the American Evaluation Association issued a statement on cultural competence in
evaluation. In it, they describe that culture affects all parts of an
evaluation, and “evaluations cannot be culture free.” It is part of our ethical
duty as evaluators to attend to culture in our evaluations and to make valid
inferences. However, to do so requires a “shared understanding within and
across cultural contexts.”

Although an evaluator can develop such a shared
understanding through deliberate practice and training, a number of
studies in cross-cultural evaluation suggest that culturally competent
evaluation is best done by evaluators who share the same cultural identity as
the evaluands. Therefore, the field of evaluation should focus on training
and promoting BIPOC evaluators so that our field can promote culturally
responsive evaluation.

Indeed, culturally responsive evaluation should be the default
evaluation approach. If culture is everywhere, and no evaluation can be culture-free,
then culturally responsive evaluation is good evaluation. If we truly
care about the validity of our evaluation findings, then we need to employ
culturally responsive evaluation practices in all our evaluations.

White Culture in Evaluation

Yet this can be difficult for White evaluators like myself.
We naïvely believe that we have no culture, but we come to that conclusion
because the White culture is the dominant culture in society. White culture has
become the one in power, the one that becomes the “default” and the “norm.”

What are the characteristics
of a White dominant culture? Perfectionism, a sense of urgency,
defensiveness, quantity over quality, worship of the written word, paternalism,
either/or thinking, power hoarding, fear of open conflict, individualism,
objectivity, and a right to comfort for those in power. One need only peruse
EvalTalk to see all these characteristics come full bloom once the White
dominant culture is threatened (as evident in the early June 2020 conversations).

When White culture is the dominant culture, then the
evaluations designed by White evaluators become the default evaluation. If you
want to see the White dominant frame in evaluation, look at our predominantly
White evaluation theory tree. Look out who we call the “fathers” of evaluation.
Look at who is being cited (and,
therefore, read). Look at how we call non-White evaluation culturally
responsive evaluation (instead of just evaluation).

When AEA put together the Race and Class Dialogues in 2017,
there was incredibly low attendance across the sessions. The 2018
Member Survey revealed that of the 1,484 respondents who answered the
question 582 (39%) were not aware of them and 629 (42%) did not participate. Of
the 273 (%) that did respond, 15.8% found them “not at all useful,” 29% said
they were “somewhat useful,” and 55% said they were “useful.” If evaluators
were not oblivious, then they decided not to participate. And for those that
did participate, nearly half decided they were not very useful, despite our
statement on cultural competence. However, further reading of the 2018 Member
Survey shows that 18% of members were not aware of the statement on cultural
competence, 24% had not read it, but at least most who did take the time to
read it found it useful (68%). White evaluators need to become familiar with
these resources and incorporate them into their regular evaluation practices.

Stepping Up

We live in a society[1] in
which the dominant frame is that of the White, male, Christian, heterosexual,
well-educated, affluent, and able-bodied person. Each of us may identify with
none, one, some, or all of these dominant frames, and each of these dominant frames have
a form of oppression associated with it: racism, sexism, religious oppression,
heterosexism, classism, ageism, and ableism.

We should first recognize our identities for what they are.
For example, I have the privileges and systematically receive the benefits of
being a White, heterosexual, well-educated, middle-class, and able-bodied
person. I may be systematically disadvantaged by my gender, and perhaps my age,
but my privileges are numerous. And because these privileges are part of the
dominant culture, they are often taken for granted and considered “normal.”

It is the responsibility of the privileged to use that power
and privilege. Too often, we have placed the burden of change on those
experiencing the oppression. There are many things we can do to improve our
evaluation practices as White evaluators:

1. Learn about Your White Privilege

Take time to learn about the privileges you hold. This blog
post focuses on white privilege, and so here are some resources to begin
understanding your white privilege. The first step we can take as White
evaluators is to understand our White dominant culture.

Here are a list of books I have read or are highly
recommended by my evaluator colleagues:

1. White
   Fragility: Why It’s So Hard for White People to Talk About Racism by Robin
   DiAngelo
2. How to Be An
   Antiracist by Dr. Ibram X. Kendi
3. So
   You Want to Talk About Race by Ijeoma Oluo
4. Why
   Are All the Black Kids Sitting Together in the Cafeteria? And Other
   Conversations About Race by Beverly Daniel Tatum
5. Me
   and White Supremacy: Combat Racism, Change the World, and Become a Good
   Ancestor by Layla F. Saad
6. Witnessing
   Whiteness: The Need to Talk About Race and How to Do It by Shelly Tochluk
7. My
   Grandmother’s Hands: Racialized Trauma and the Pathway to Mending Our Hearts
   and Bodies by Resmaa Menakem

If you’re interested in podcasts, articles, films and TV
series, or organizations to follow, I highly recommend this
resource.

I also recommend building racial equity habits. Eddie Moore
Jr. and Debby Irving created a 21-day racial equity habit
building challenge. They suggest for 21 days that you do at least one
action to further your understanding of power, privilege, supremacy,
oppression, and equity. They provide suggestions in the categories of things to
read, listen, watch, notice, connect, engage, act, reflect, and stay inspired. Although
they named it a 21-day challenge, it does not have to be 21 days nor do the
days have to be continuous. Just like we can never become culturally competent
(rather we can only do culturally competent actions), our work here is never
done.

2. Learn about and incorporate Culturally Responsive Evaluation Practices

The AEA statement on cultural competence points to a need to
have a shared understanding. Although evaluators who share the culture of the
program and its participants are best positioned to do culturally responsive
evaluation, White evaluators can still deliberately practice and receive
training to better do culturally responsive evaluation in contexts they do not
share the culture.

There is a great deal of humility needed to do this type of
work. A humility that the program knows what is best for them. A humility to
set aside the White dominant culture and all the assumptions and norms you have
internalized from birth. It is hard work, and you will mess up. Think of
yourself as a beginner in this work and be open to learning.

3.  Critically reflect on your
evaluation practices

As you are incorporating culturally responsive evaluation
practices, you should be attending to the defaults, assumptions, and norms you
have carried into your evaluation practice. You will have many thoughts and
feelings going through this process. Create a space with other White
evaluators to discuss these thoughts and feelings and help each other through
this process. Do not rely on your BIPOC colleagues to help you through
this. They have told us for years, and they tell us now they are tired. Listen
to them and do your own work.

Some ideas of questions and comments you may want to ponder
regarding your evaluation practices:

1. Think critically about for whom in your
   evaluation work. For whom does the program work? For whom does the program unintentionally
   (or intentionally!) harm?
2. When you think of evaluation of a program, what
   comes to mind? When you think of evaluation of the sameprogram
   serving BIPOC, what comes to mind? Are they the same, or different? What do
   you think that says about you and your evaluation practice?
3. If you work with other evaluators, who are you hiring?
   Do they primarily look like you? Does your evaluation team share the lived
   experiences of the people in the programs you are working with?
4. Who do you include in the evaluation process? Madison
   concluded in her review of 20 years of New Directions for Evaluation volumes
   that “underrepresented groups continue to be presented as subjects of
   evaluation rather than as invested stakeholders.” The stakeholders most often
   to be included in the evaluation consist primarily of the program leaders who,
   like evaluators, are less likely to
   be BIPOC.
5. What choices do you routinely make in your
   evaluation practices? Some major choices include the design and methods you
   use. For each of those choices, examine how race, class, gender, sexual
   orientation, and more affect the decisions you make. Refrain from thinking that
   they do not make a difference. For example, the creators
   of modern statistics did so to promote a eugenics agenda and data is not value-neutral,
   equitable, or unbiased.

4. Critically Reflect on the Field of Evaluation

Recognize that racism is everywhere, including at the AEA conference,
on EvalTalk, and more. BIPOC evaluators have called out evaluation spaces for
being racist and unsafe spaces for them for years. Start listening to them,
believing them, and supporting them.

The AEA conference in 2019 made one step forward when it
offered “A
Respite & Healing Space” for BIPOC attendees to rest, heal, and connect
with other BIPOC attendees. White allies and accomplices were invited to
contribute financially as their way to support the space. Listen and act when
other opportunities are presented for you to show your allyship for our BIPOC
colleagues.

If you are a teacher in evaluation, think critically
about who you accept as students and what you teach. Recruit and accept BIPOC
students into your program. Encourage your institution to hire and create
safe work environments for BIPOC. Check your syllabus: who do you assign as
readings? What topics are covered? How are race, class, gender, and other
topics infused into your curriculum? A week on culturally responsive evaluation
is a starting point, but it is not enough.

If you are a researcher of evaluation, think critically
about who you cite. Our BIPOC colleagues are not cited at the same rates as
White evaluators. Check your reference list that you have a balance of
representation by race, gender, nationality, and more.  

Conclusion

Listening, stepping back, and critically reflecting is hard work. But what is harder and just as necessary is for us to speak up, stand up, and step up in alliance of our BIPOC colleagues. Advocate for change in your institution, in our association, and in our field in general. Advocate for change with the organizations you partner with, citing our guiding principles and statement of cultural competence when you write an evaluation proposal. Speak up when you see prejudiced behaviors or racist systems. It is time. Step up.

Do you have comments, questions, feedback, or suggestions? Add a comment below or use the contact form to email me on my website.

[1] I focus on the United States in this blog post because that is the nation I know best, but this is likely true for other nations as well. I mention because Khalil Bitar rightfully pointed out that these conversations are largely USA-centered, mostly reactive, and there is limited participation by the often unheard voices.